Handling Digital Brains: A Laboratory Study of Multimodal Semiotic Interaction in the Age of Computers (Inside Technology)

Handling Digital Brains

Inside Technology edited by Wiebe E. Bijker, W. Bernard Carlson, and Trevor Pinch Janet Abbate Inventing the Internet Atsushi Akera Calculating a Natural World: Scientists, Engineers and Computers during the Rise of U.S. Cold War Research Morana Alacˇ Handling Digital Brains: A Laboratory Study of Multimodal Semiotic Interaction in the Age of Computers Charles Bazerman The Languages of Edison’s Light Marc Berg Rationalizing Medical Work: Decision-Support Techniques and Medical Practices Wiebe E. Bijker Of Bicycles, Bakelites, and Bulbs: Toward a Theory of Sociotechnical Change Wiebe E. Bijker and John Law, editors Shaping Technology/Building Society: Studies in Sociotechnical Change Wiebe E. Bijker, Roland Bal, and Ruud Hendricks The Paradox of Scientific Authority: The Role of Scientific Advice in Democracies Karin Bijsterveld Mechanical Sound: Technology, Culture, and Public Problems of Noise in the Twentieth Century Stuart S. Blume Insight and Industry: On the Dynamics of Technological Change in Medicine Pablo J. Boczkowski Digitizing the News: Innovation in Online Newspapers Geoffrey C. Bowker Memory Practices in the Sciences Geoffrey C. Bowker Science on the Run: Information Management and Industrial Geophysics at Schlumberger, 1920–1940 Geoffrey C. Bowker and Susan Leigh Star Sorting Things Out: Classification and Its Consequences Louis L. Bucciarelli Designing Engineers Michel Callon, Pierre Lascoumes, and Yannick Barthe Acting in an Uncertain World: An Essay on Technical Democracy H. M. Collins Artificial Experts: Social Knowledge and Intelligent Machines Park Doing Velvet Revolution at the Synchrotron: Biology, Physics, and Change in Science Paul N. Edwards The Closed World: Computers and the Politics of Discourse in Cold War America

Andrew Feenberg Between Reason and Experience: Essays in Technology and Modernity Michael E. Gorman, editor Trading Zones and Interactional Expertise: Creating New Kinds of Collaboration Herbert Gottweis Governing Molecules: The Discursive Politics of Genetic Engineering in Europe and the United States Joshua M. Greenberg From Betamax to Blockbuster: Video Stores and the Invention of Movies on Video Kristen Haring Ham Radio’s Technical Culture Gabrielle Hecht Entangled Geographies: Empire and Technopolitics in the Global Cold War Gabrielle Hecht The Radiance of France: Nuclear Power and National Identity after World War II Gabrielle Hecht The Radiance of France: Nuclear Power and National Identity after World War II, New Edition Kathryn Henderson On Line and On Paper: Visual Representations, Visual Culture, and Computer Graphics in Design Engineering Christopher R. Henke Cultivating Science, Harvesting Power: Science and Industrial Agriculture in California Christine Hine Systematics as Cyberscience: Computers, Change, and Continuity in Science Anique Hommels Unbuilding Cities: Obduracy in Urban Sociotechnical Change Deborah G. Johnson and Jameson W. Wetmore, editors Technology and Society: Building Our Sociotechnical Future David Kaiser, editor Pedagogy and the Practice of Science: Historical and Contemporary Perspectives Peter Keating and Alberto Cambrosio Biomedical Platforms: Reproducing the Normal and the Pathological in Late-Twentieth-Century Medicine Eda Kranakis Constructing a Bridge: An Exploration of Engineering Culture, Design, and Research in Nineteenth-Century France and America Christophe Lécuyer Making Silicon Valley: Innovation and the Growth of High Tech, 1930–1970 Pamela E. Mack Viewing the Earth: The Social Construction of the Landsat Satellite System Donald MacKenzie Inventing Accuracy: A Historical Sociology of Nuclear Missile Guidance

Donald MacKenzie Knowing Machines: Essays on Technical Change Donald MacKenzie Mechanizing Proof: Computing, Risk, and Trust Donald MacKenzie An Engine, Not a Camera: How Financial Models Shape Markets Maggie Mort Building the Trident Network: A Study of the Enrollment of People, Knowledge, and Machines Peter D. Norton Fighting Traffic: The Dawn of the Motor Age in the American City Helga Nowotny Insatiable Curiosity: Innovation in a Fragile Future Ruth Oldenziel and Karin Zachmann, editors Cold War Kitchen: Americanization, Technology, and European Users Nelly Oudshoorn and Trevor Pinch, editors How Users Matter: The Co-Construction of Users and Technology Shobita Parthasarathy Building Genetic Medicine: Breast Cancer, Technology, and the Comparative Politics of Health Care Trevor Pinch and Richard Swedberg, editors Living in a Material World: Economic Sociology Meets Science and Technology Studies Paul Rosen Framing Production: Technology, Culture, and Change in the British Bicycle Industry Richard Rottenburg Far-Fetched Facts: A Parable of Development Aid Susanne K. Schmidt and Raymund Werle Coordinating Technology: Studies in the International Standardization of Telecommunications Wesley Shrum, Joel Genuth, and Ivan Chompalov Structures of Scientific Collaboration Charis Thompson Making Parents: The Ontological Choreography of Reproductive Technology Dominique Vinck, editor Everyday Engineering: An Ethnography of Design and Innovation

Handling Digital Brains A Laboratory Study of Multimodal Semiotic Interaction in the Age of Computers

Morana Alacˇ

The MIT Press Cambridge, Massachusetts London, England

© 2011 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. For information about special quantity discounts, please email special_sales @mitpress.mit.edu This book was set in Stone Sans and Stone Serif by Toppan Best-set Premedia Limited. Printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Alacˇ, Morana. Handling digital brains : a laboratory study of multimodal semiotic interaction in the age of computers / Morana Alacˇ. p. cm. — (Inside technology) Includes bibliographical references and index. ISBN 978-0-262-01568-4 (hardcover : alk. paper) 1. Human-computer interaction—Case studies. 2. Semiotics. 3. Neuroscientists. 4. Brain mapping. 5. Brain—Magnetic resonance imaging. I. Title. QA76.9.H85A53 2011 612.8′2—dc22 2010039561 10

9

8 7

6 5

4 3 2

1

To those from whom I learn

Contents

Acknowledgments

xi

1

In the fMRI Laboratory

2

fMRI Brain Visuals as Fields for Interaction

3

fMRI Brain Imaging and the Experience of Sound

4

fMRI Brain Visuals and Semiotic Bodies

5

The Semiotic Mind in the fMRI Laboratory

6

Materiality of Digital Brains

7

Publishing fMRI Visuals

8

Conclusion

Notes 169 References 179 Index 195

159

1

147

121

23

67 93

49

Acknowledgments

This book reports on my walks through the laboratories of cognitive neuroscience that use functional magnetic resonance imaging technology as their primary mean of investigation. I am grateful to the inhabitants of those spaces who opened their everyday life to my visits: the cognitive neuroscientists at the University of California, San Diego (UCSD) and the Salk Institute. From them I learned not only about cognitive neuroscience, brain imaging, and the excitements of their work, but also about collaboration, multidisciplinarity, and generosity. I am enormously indebted to Marty Sereno. Special thanks go to Wendy Ark, Geoff Boynton, Marisa Brandt, Karen Emmorey, Don Hagler, Ed Hubbard, Dan Kennedy, Silvia Paparello, Brianna Paul, Susan Peppas, Melissa Saenz, Ayse Saygin, Joan Stiles, Stephen McCullough, Hsin-Hao Yu, and all the participants in my study. I am grateful for the intellectual support and engaging scholarship of my colleagues and students in the Department of Communication at UCSD. I am particularly appreciative to Lisa Cartwright, Mike Cole, Ivana Guarrasi, Dan Hallin, Chandra Murkeji, and Carol Padden, who read and commented versions of this manuscript. I also immensely enjoyed the exchanges that the Program in Science Studies, Program in Math and Science Education, and California Institute for Telecommunication and Information Technology have provided. My undergraduate and graduate adviser in Philosophy and Semiotics at the University of Bologna, Umberto Eco, and my graduate adviser in Cognitive Science at UCSD, Ed Hutchins, have irreversibly shaped the way I ask questions. The members of my Ph.D. committees—Lisa Cartwright, Seana Coulson, Charles Goodwin, Gilles Fauconnier, and Patrizia Violi— provided a huge amount of advice, challenge, and encouragement.

xii

Acknowledgments

I learned a great deal from Anne Beaulieu, Monika Buscher, Giovanna Franci, Philip Lieberman, Michael Lynch, David Mindell, Lucy Suchman, and Elizabeth Wilson. Thank you for your generosity and the work you do. I am thankful to Maurizio Marchetti for helping me prepare the visuals for this book. Portions of this text, reworked and rewritten for the purposes of this book, have been previously published in Journal of Cognition and Culture, Journal of Social Epistemology, Science Images and the Popular Images of the Sciences, Semiotica, and Social Studies of Science. I am grateful to the editors, publishers, reviewers, and my co-authors who influenced this book through their contribution to those articles. My study has also benefited from discussions and teaching of Aaron Cicourel, Elena Collavin, Jeff Elman, Yrjö Engeström, David Groppe, Deborah Forster, John Haviland, Jim Hollan, Ron Langacker, Marta Kutas, Javier Movellan, Katrina Petersen, Leigh Star, Mark Turner, and Ying Wu. I feel so privileged to be part of the UCSD community with you. Marguerite Avery at MIT Press has been the best imaginable editor. I am also grateful to Katherine Almeida and Johna Picco for their editorial help with the manuscript, and to the three reviewers who provided invaluable suggestions and comments. I am deeply thankful to my family—Mirjana, Žarko, Josipa, Gorana, Sandro, Federico, and Miro—for their “nema zime,” and to Maurizio and Lucy for always leaving with me.

1 In the fMRI Laboratory

It is 2002, and we are in a cognitive neuroscience laboratory at the University of California, San Diego. There, we encounter two researchers seated in front of a computer screen (figure 1.1). One of them, the laboratory director, Paul (a professor with a distinguished record of publishing and teaching in the field of cognitive neuroscience), is talking with a graduate student named Jane (a promising Ph.D. candidate in cognitive science) seated next to him (featured on the right in figure 1.1). The two researchers are engaged in the practice of functional magnetic resonance imaging (fMRI). fMRI, together with its forerunner, MRI, is a key modern digital imaging technology used for medical and scientific purposes. The goal of MRI is to provide detailed static renderings of the anatomic structure of internal body parts, such as the brain. This technique uses radiofrequency, magnetic fields, and computers to create visual renderings (“visuals”) based on the varying local environments of water molecules in the body. To obtain such visuals, a person (or, in fMRI practitioners’ jargon, an experimental subject or a subject) is scanned. During a brain MRI scanning session, hydrogen protons in brain tissues are magnetically induced to emit a signal that is detected by the computer. Such signals, represented as numerical data, are then converted into visuals of the brain as the brain anatomy of the experimental subject is imaged. The mapping of human brain function by use of fMRI represents a new dimension in the acquisition of physiologic and biochemical information with MRI. The technique is used to observe dynamic processes in the brain that are demonstrated by visualization of the local changes in magnetic field properties occurring in the brain as a result of changes in blood oxygenation. The role of fMRI visuals is, thus, to display the degree of activity

2

(a)

Chapter 1

(b)

Figure 1.1 fMRI researchers working on a laboratory computer.

in various areas of the brain; if the experimental data are obtained while a subject is engaged in a particular cognitive task, the visual can indicate which parts of the brain are most active during that task. To show these results, however, fMRI visuals require extensive analysis in the laboratory. During analysis sessions, such as the one in which Jane and Paul are involved, fMRI practitioners use computers to engage their data, shaping the appearance of fMRI visuals. The engagement with digital material allows the practitioners to enhance their understanding of the imaged biological matter. This means that observation by an fMRI practitioner is multimodal and cyborg-like; it is accomplished by coordinating the eyes with digital technology, an array of instruments, graphical inscriptions, and actions of the hands.1 Rather than passively gazing at static visual images, fMRI researchers interact with each other and the technology, engaging their experiential and semiotic bodies. They manipulate, listen to, and touch computers and other instruments, and they also talk, gesture, interactionally engage their heads, necks, and torsos as they attend to each other in the work of cognitive neuroscience. One of the ways practitioners can engage digital material to enhance their understanding is to flatten computationally the imaged cerebral cortex to see not only what is shown on its ridges (or gyri) but also what otherwise would be hidden in the fissures (or sulci) that surround such ridges. This is exactly what Jane and Paul are doing when we join their data analysis session. Because Jane still needs to acquire skills in data analysis, Paul, while talking about the brain visuals displayed on the

In the fMRI Laboratory

3

computer screen, shows Jane how to use fMRI software to generate visuals that appear flattened. When the computer screen displays a set of data as an “inflated surface,” Paul identifies portions of the imaged visual cortex: “That’s the center of the gaze and that’s that other thing that I said don’t look at it. That’s up, right there.” As he points out a section that does not look like he thinks it should, Paul chuckles while jokingly warning Jane not to pay attention to it. During the entire sequence, as Paul holds the computer mouse in his right hand and skillfully directs the changes of the displayed visuals, he indicates specific brain areas by using the cursor and pointing with his left hand. While attentively listening, Jane takes notes and shows her understanding by nodding. When Paul begins to explain how laboratory members look at the imaged visual cortex, his description invokes physical actions: “Usually how we look at it is, we put a cut down there and another cut over here, and then we flatten the whole thing out.” To make their seeing (as the capacity to discern the cortical organization) more powerful, laboratory members perform the actions of “cutting” and “flattening” of the digital matter. When the screen displays the desired view, Paul says: “So you made a cut right here then like spread that out.” Just before saying “spread that out,” he gets closer to the screen and carefully places both hands over it; his palms face the screen while the right hand is tilted slightly to the right (figure 1.2a). As he utters “spread that out,” Paul stretches the hands apart (see figure 1.2b, c), generating a clearly marked but patiently performed gesture. To conclude the action, he turns toward Jane and says, “That’s actually how you are looking at it when you look at it flattened.” What just happened—Paul’s stretching his hands over the digital display while talking about physical actions and changing the appearance of the brain visuals—is not treated as an extraordinary event. Quite to the contrary, after the completion of Paul’s action, the two researchers continue with their discussion as if nothing unusual had happened. Paul’s animating of the visual matter by selecting computer commands, gesturing, and talking about physical actions is an ordinary and frequently encountered feature of laboratory work and communication. Scientists routinely engage their hands by typing commands on keyboards and touching the computer screen while gesturing in front of brain visuals. Does this mean that such acts are of no particular importance? Is the engagement of hands with

4

Chapter 1

(a)

(b)

(c) Figure 1.2 fMRI researcher coordinates his gesture and talk with a brain visual.

digital visual matter not of interest for an understanding of scientific practice? In this book, I argue that these routine and ubiquitous interactional enactments deserve attention. To illustrate how the interactional events constitute scientific work and generate understanding of scientific data, this book focuses on scientific visuals—a topic that has been extensively discussed in social studies of science and technology (e.g., Cartwright, 1995; Daston & Galison, 2007; Knorr-Cetina & Amann, 1990; Latour, 1995; Lynch, 1985b, 1990; Lynch & Woolgar, 1990; Rheinberger, 1998; Rudwick, 1976). In attending to the articulation of fMRI brain visuals in the

In the fMRI Laboratory

5

laboratory, I am interested in how digital scientific visuals achieve meaning through the engagement of hands. As seen in the interaction between Paul and Jane, fMRI practitioners operating computers often gesture over screens to communicate with each other and make sense of the visual data with which they are dealing. These multimodal engagements with the digital matter are central for turning experimental material into what can be seen and understood. Rather than being only a contextual scene for the digital visuals, they are their essential component. The participation of the hands in work and multimodal semiotic interaction illustrates the specific ways in which scientific practice is rooted in the body. Michael Lynch distinguishes two orders of laboratory “space,” opticism and digitality: “The paradigm for the former is the lensed instrument and the scrutinizing eye, while the latter is embodied by the play of fingers (digits) on a keyboard instrument” (Lynch, 1991: 56). Lynch points out that digitality does not displace opticism; rather, the two orders coexist and overlap with each other across historical periods. I deal with the interplay between opticism and digitality at the level of multimodal action and interaction. As exemplified by the interactional moment in Paul’s laboratory, cognitive neuroscientists, in the age of computers (e.g., Mindell, 2002, 2008), gesture, talk, orient their bodies, and gaze to accomplish their work. These actions not only show how they understand the objects of their inquiry but also allow for discussing the assumptions behind fMRI. fMRI technology implies a model of the mind and embodiment where the mind is confined to the internal workings of an individual, and embodiment concerns the grounding of the mind in the brain. Conversely, the subtle and skillful handling of the digital substance in the laboratory suggests that the mind in action cannot be solely understood in terms of the brain without taking into account the entire body and the sociocultural world that such a body experiences. In the context of laboratory work, fMRI scientists think by bringing together their lived semiotic bodies with digital screens. Embodiment and the Body Since the 1990s—famously named the “decade of the brain”2—cognitive neuroscience has occupied the central stage in the scientific study of the human mind: Human mental processes are to be studied in terms of brain

6

Chapter 1

processes. This grounding of the mind in neuronal activity places cognitive neuroscience apart from earlier efforts in the study of human cognition. The approaches that typically fell under the umbrella of cognitivism were based on the analogy between the mind and the computer program, where the mind was seen in terms of the manipulation of symbols defined independently of the material substrate in which such manipulation is instantiated (or, on which the program runs). This implied a disregard for the details of the biological underpinning of cognition; the mind was considered to be a logical rather than biological machine. In contrast with the approaches grounded in Cartesian modernism (e.g., Damasio, 1994), the availability of imaging technologies—particularly fMRI—has allowed biological processes to become the focal point. By visualizing brain processes, fMRI is framed as a technique for the study of the mind that concerns the human body. Consequently, the turn to embodiment, shaped by the availability and constraints of fMRI technology, presupposes an equation between the brain and the body; when talking about embodiment, cognitive neuroscience refers to the brain. Feminist and postphenomenological studies of science and technology, on the other hand, conceptualize embodiment and the body in a somewhat different manner. Embodiment is not only about the brain but also the lived body immersed in the complexities of the sociocultural and technologymediated world. These approaches, rooted in Foucault’s (1977)discussion of surveillance and self-surveillance, Lacanian psychoanalysis (Lacan, 1977, 1982), and the phenomenological writings of Heidegger (1962, 1977, 1982) and Merleau-Ponty (1962, 1968), argue against restricting our understanding of the body to naturalistic and scientific modes of explanation. Feminist scholars (e.g., Butler, 1993; Grosz, 1987, 1994; Irigaray, 1985; Marshall, 1996), thus, see the body as a fluid site of potential. The body is neither—while also being both—the private or the public, self or other, natural or cultural, psychical or social, instinctive or learned, genetically or environmentally determined. In the face of social constructionism, the body’s tangibility, its matter, its (quasi) nature may be invoked; but in opposition to essentialism, biologism, and naturalism, it is the body as cultural product that must be stressed. (Grosz, 1994: 23–24)

In focusing on the developments of new scientific instruments and information technologies, feminist scholars have shown that once the relationship between the body and technology are brought to the surface,

In the fMRI Laboratory

7

the idea of the body as a self-contained biological object that obeys mathematical-causal laws is irreversibly disrupted (e.g., Barad, 2003, 2007; Cartwright, 2008; Haraway, 1988, 1991; Hartouni, 1991; Mol, 2002; Sobchack, 2004; Suchman, 2007; Wilson, 2004). Similarly, postphenomenology, in indicating how technologies allow us to see what would otherwise be invisible, explores how the experience and sense of the body are affected by such an engagement (e.g., Ihde, 1990, 1993, 1998, 2002). In science, the early use of optical technologies—telescopes and microscopes, for example—transformed what was seen, enhancing features of the observed object while reducing the visual field to which that object belongs. The current book draws upon these theoretical developments to explore how scientists experience their bodies while interfacing with digital computers, other imaging technologies, their colleagues, and the imaged bodies. I describe the complexities of the thinking body in cognitive neuroscience by specifically focusing on the digital visuals and interactional organization of everyday scientific activity. How do fMRI practitioners gesture, coordinate their talk, and orient to each other and to their computers? How do they touch the objects that surround them and how do they sense their bodies as they accomplish their work? Dealing with these questions is a way to discuss how fMRI technology implies, constrains, and enables the body in cognitive neuroscience. Whereas the technology and the theoretical positions associated with it assume that a thinking body can be reduced to the brain, I focus on the dynamics of interaction in fMRI practice to talk about the constitutive outside, questioning the proper distinction between brain and body, the scientists and the sociomaterial world of their practice. Studying Multimodal Interactional Organization of Scientific Practice To capture embodied interaction in scientific practice, I draw from ethnomethodology (Garfinkel, 2002) and conversation analysis (CA; Jefferson, 2004; Sacks, 1995; Sacks, Schegloff, & Jefferson, 1974). In particular, I ground my approach in a recent research trend aimed at recovering fine details of the multimodal interactional organization of everyday practices (e.g., Goodwin, 1994, 2000b; Heath & Hindmarsh, 2002; LeBaron, 2007; Koschmann et al., 2007; Mondada, 2007; Ochs, Gonzales, & Jacoby, 1996; Streeck, 2009; Suchman, 2000).3 Similar to ordinary language philosophy (Austin, 1962;

8

Chapter 1

Wittgenstein, 1973) and the approaches in semiotics (Benveniste, 1971; Peirce, Collected Papers [C.P.]), these studies point out that talk, as well as bodily conduct and the engagement with material elements of the setting, participate in the practical accomplishment of social activities. When we coordinate our talk with gestures and bodily conduct, we accomplish actions. These actions are situated (Suchman, 1987)—always realized, moment by moment, with respect to the environment in which they are lodged, while they constitute the local context. In exploring these methodological positioning in the context of science and technology studies (STS), I transcribe videotaped instances of the work in fMRI laboratories to focus on the choreography between hands, eyes, and ears in the ongoing social interaction between practitioners of fMRI. These aspects of conduct characterize practitioners’ involvement with the materiality of the laboratory setting and are enacted as a part of doing cognitive neuroscience; they indicate how the imaged body is known through multimodal action in the laboratory. Thus, to talk about the body and embodiment in neuroscience, I turn my STS gaze toward gestures, details of talk, movements of hands, and nodding of heads in the work of science. Rooting the approach in the videotaped records of such actions in the laboratory, I examine how the body is experienced and how it generates meaning as it interacts with the imaging technology. Importantly, I do not use videotaped record to document and represent fMRI practice, but I treat it as an analytical resource (e.g., Heath, 1997: 190; Heath & Hindmarsh, 2002: 104) to discuss how fMRI practitioners engage their bodies with computer screens in learning about imaged brains. Examining Videotaped Work of Science During my study, I videotaped fMRI scientists as they interacted with each other during work sessions and apprenticeship practices. As the videotapes indicate how fMRI practitioners gesture in front of and touch digital screens while working and communicating with their colleagues, they shaped the study, affording access to the interface between the body and technology. Despite the interest in the visual, video recordings are still not widely applied by STS scholars as a methodological tool to study scientific practices. However, if we accept that scientists accomplish their work

In the fMRI Laboratory

9

through tactile interaction with technology and visible semiotic comportment, we must examine such acts. As we go beyond the linguistic aspects of communication (such as talk and writing), our intuition and memory are not reliable sources with which to document the complexities that characterize multimodal interaction. Furthermore, people are often unable to provide accurate accounts of their own conduct, especially when their use of multimodal communication is in question. Gesture, for example, is very dynamic and largely unnoticed. As such, its articulation in the environment of practice cannot be either fully reported in an interview or accurately remembered by an observer. A scientist who was involved in the work when a gesture took place can just have tacit knowledge of how that was done, and an ethnographer who saw the gesture is only at a loss when trying to represent its temporal unfolding and coordination with other elements of the semiotic action. The problem is not only the gesture taken in isolation but also its fine embeddedness in the complexities of the moment of practice. The gesture is contingent upon the local and spatial organization of the setting, and is produced in relation to the ongoing talk and the actions of the coparticipants. To access multimodal semiotic aspects of working hands in the laboratory, video recordings of everyday interactions are crucial. With all their insufficiencies and their inevitably incomplete output, video recordings are currently our best mode to record the dynamicity of the setting in which work and multimodal interaction take place (Goodwin, 2000a). In video recording the scientists, I was particularly interested in the scientists’ communication with each other, their use of instruments, their ad hoc creation of drawings, and their engagement with computer screens. To capture—as much as is possible—the richness that characterizes such acts, I typically positioned myself with the camera behind the practitioners, facing toward the computer they were looking at (as exemplified by figure 1.1). Sometimes I placed additional recording devices in front of the practitioners (e.g., the small recorder seen to the right of the computer in figure 1.1). In such a way, I documented how scientists orient toward each other, how they manipulate things placed on the desk in front of them, how they construct new objects (such as the ad hoc drawings), and how they work on the computers (e.g., how they use mouse and keyboard commands). It cannot be denied that the presence of an ethnographer with

10

Chapter 1

a video camera distorts some of the “naturalness” of the recorded scene. In my experience, however, practitioners busy with the tasks at hand tend to pay little attention to the prolonged presence of the video camera behind them. By scrutinizing the audiovisual record on multiple occasions, slowing it down, and transcribing it, I observed how fMRI practitioners accomplish minute yet vital details of their everyday practice. I analyzed how they engage brain atlases to inscribe landmarks on the scans, how they change the appearance of such scans, and how they point and gesture to enact features of the scans. To warrant my analytic claims, I paid particular attention to how practitioners themselves deal with the specific actions of others (Garfinkel, 2002). I looked at whether practitioners orient toward certain events and whether they treat such events as relevant components of the activities they are engaged in. If, for example, my video record shows that a practitioner treats a shoulder movement of the other scientist as indicating what has happened during a previous scanning session, then I considered such movement to be a communicative act rather than a motion of the body without semiotic import. The videotapes also provided me with an opportunity to share my insights with other researchers and discuss my analysis with fMRI practitioners. During my study, I did conduct interviews with practitioners, but these interviews were mostly done in the beginning of the study to supply general knowledge about the practices. When I conducted interviews in the later stages, they were often organized as discussions of previously video-recorded activities. These occasions (articulated around the videotaped record) not only solidified rapport but also improved the study, as the practitioners, while often unaware of the interactional details through which they organize their conduct, helped clarify specialized vocabulary, use of technology, and understanding of local procedures (Heath & Hindmarsh, 2002: 103). Once I identified the excerpts that were most informative of the practical methods scientists use in making fMRI visuals intelligible, I transcribed them, specifying features of talk-in-interaction (Schegloff, 1987). This included elements such as the length of silences and pauses, onset and overlaps in talk, as well as the aspects of speech delivery and intonation.

In the fMRI Laboratory

11

Such transcripts are useful in deciphering how practitioners produce action with regard to the conduct of others and how they deal with meanings accomplished through the ongoing interaction. Because of my interest in the semiotic and “non-symbolic practical, instrumental routines of the hands” (Haviland, 2005: 213; Streeck, 2009) developed through everyday activities,4 I also annotated occurrences of multimodal semiotic modalities, such as gesture, gaze, facial expression, and body orientation (Goodwin, 2000a). In doing so, my aim was not to describe any one semiotic mean (gesture, for example) in isolation. Rather, I wanted to capture the coordination of multimodal semiotic means. In doing so, I looked at the exact moments of their production, their embeddedness in the instrumental and collaborative sequences of action, their coordination with technological objects, and their situatedness in the laboratory space. Following Goodwin’s (2000a) technique of transcribing visual phenomena, I turned still photographs (retrieved from the video) into line drawings (see figure 1.1b). Using software programs, I delineated the contours of scientists’ bodies and relevant elements of the setting by working directly on the photographs (as seen by comparing figure 1.1a with figure 1.1b). These renderings were enriched with arrows and other signs to indicate relationships between rendered acts (see figure 1.2). In providing such transcripts, my goal was to let the reader see as much as I saw while indicating elements of the practice that the scientists were treating as relevant in their work and interaction. For example, the whiteboard visible behind the computer screen in figure 1.1a is not rendered in figure 1.1b. Even though use of the whiteboard is a crucial component of work and learning in an fMRI laboratory, during the entire sequence of work from which the still image (translated into the line drawing) was taken, the practitioners did not draw on, refer to, or attend to the whiteboard. Because the goal of the transcription of the embodied activity is not only to preserve as much complexity of the video record as possible but also to communicate relevant events as clearly and vividly as possible (Goodwin, 2000a: 161), the whiteboard was not shown in figure 1.1b. Video records, however, neither fully capture nor provide direct access to the meaningful activities in the laboratory. To understand the relevant

12

Chapter 1

patterns in the analyzed data5 as part of scientific practice, I interpreted the transcribed excerpt in light of the knowledge derived from my longterm ethnographic work (e.g., Cicourel, 1987; Lynch, 1993). Ethnographic Study of fMRI The first videotape from this study dates back to summer 2002, when some of the leading scientists and administrators at the University of California, San Diego (UCSD) gathered to inaugurate the new Center for fMRI. UCSD and the nearby Salk Institute are renowned for their research in medicine, neuroscience, and cognitive science, and the $13.5 million center was announced as “the largest brain imaging facility dedicated to research in the Western United States.”6 The speakers at the opening ceremony, among them Edward W. Holmes, chancellor for Health Sciences and dean of the School of Medicine at UCSD, and Roderic Pettigrew, director of the National Institute of Biomedical Imaging and Bioengineering, frequently mentioned the partnership between institutions, research fields, and individuals. Edward W. Holmes, for example, said: With this dedication we announce the availability of this powerful imaging facility that will serve a wide variety of investigators: neurologists, psychiatrists, cognitive scientists, radiologists, engineers, biologists, and chemists.

The initial goal of my ethnographic study was to observe scientific work at the newly opened facility, focusing on how fMRI technology features in collaborations among scientists from multiple fields, and how this technology both generates and is shaped by broader societal phenomena.7 Captivated by the scanner—a monumental and expensive machine— and the large-scale collaboration that it requires, I wanted to see how the distributed work of science, organized around this technology, develops and what it entails. As the project progressed, however, my focus gradually shifted from the fMRI center, where scientists conduct their scanning sessions, to individual laboratories, where the data collected in the scanner go through an extensive process of interpretation before they are turned into the colorful brain visuals usually seen in scientific journals or mass media outlets. Following scientists in their daily work (Latour, 1987), I most often found myself in

In the fMRI Laboratory

13

the laboratory, where practitioners spend long hours working on experimental data. Whereas a scanning session generally takes only a couple of hours, the data analysis may span months, typically involving a much larger group of collaborators. Similarly, graduate students are able to obtain proficiency in operating the scanning machine after a couple of hours of training, but they tend to dedicate their Ph.D. studies to mastering fMRI data analysis. To make sense of what I was observing in the laboratories, I took advantage of my dual positioning in the field: a semiotician trained in cognitive science. During the study, I was a doctoral student in the Department of Cognitive Science at UCSD and hence a member of the research field—at once an insider and an outsider (Mol, 2002). Given this position, the everyday activities of a doctoral student—attending talks, taking classes, presenting and discussing research results—were not easily discernible from my ethnographic work. As I was taking part in the activities of the community, I was observing its dynamics with the goal of generating what Geertz (1973)—borrowing from Gilbert Ryle—calls thick descriptions.8 Being embedded in the community allowed me to spend long periods of time with its members and to capture some of the details that constitute the everyday practices of fMRI. The three laboratories that I studied focus their research on human cognitive processes through the employment of fMRI. Two of these laboratories were located at the UCSD campus and the other at the Salk Institute. One of the two UCSD laboratories researches brain development in the patient population, whereas the other two laboratories specialize in the study of visual processes in healthy adults. My ethnographic observations encompassed the work of principal investigators, researchers, and students in cognitive science, psychology, and neuroscience at the graduate and undergraduate levels and lasted for 3 years. During the study, in addition to the usual participation in the cognitive science community, I attended working sessions and laboratory training activities, carried out interviews, and gathered documents that ranged from e-mail correspondences and architectural plans to scientific reports. The shift of focus from the fMRI center to the laboratory had consequences for the shaping of this study. Once in the laboratory, my attention was absorbed by the ordinary methods (Garfinkel, 2002) that scientists use in their work with fMRI data. Particularly, I was fascinated by the manual

14

Chapter 1

engagement and interaction with technology that such work entails. By observing scientists at work, I had to notice that their encounters with the massive and expensive scanning machines were mediated by their gestures in front of brain visuals and their commands typed on ordinary personal computers. Intrigued by the space between scientists and their computers as the locus of the everyday action and interaction in the fMRI laboratory, I set out to describe aspects of its dynamics. A Return to Laboratory Studies? By attending to everyday features of scientific practice, my approach is a return to the expository style and interest of early laboratory studies (KnorrCetina, 1981; Latour & Woolgar, 1979; Lynch, 1985a, 1993; also Collins, 1985; Pinch, 1986; Traweek, 1988). Like the laboratory studies originating almost three decades ago, of importance are epistemological issues grounded in local research practices. The goal is to examine “the methodical way in which observations are experienced and organized so that sense can be made of them” (Latour & Woolgar, 1979: 37). However, different from earlier studies, I turn attention to the intricacies of multimodal semiotic interaction. At their outset, laboratory studies were responding to a significant absence of knowledge about how scientists actually work. Bruno Latour and Steve Woolgar (1979:17), for example, noticed that an important effort had been dedicated to the study of science, yet the majority of such projects were primarily devoted to examinations of science on a larger scale. To fill this gap, early laboratory studies urged for detailed participant observations (aided by interviews and discourse analysis methods) to be conducted in places where scientists actually do their research. While engaging ideas from Wittgenstein’s (1973) Philosophical Investigations, Kuhn’s (1962) The Structure of Scientific Revolutions, European semiotics of the narratological stem (e.g., Greimas, 1987), ethnomethodology (Garfinkel, 2002), and pragmatics (e.g., Austin, 1962), these empirical relativist studies generated accounts of how scientists prepare experiments, collect data, and discuss and write scientific reports. The interest in how scientific activities actually take place allowed laboratory studies to disturb assumptions about science as a unique sphere of human cognition and to sharpen the awareness toward the local and contextual aspects of science (see also, e.g., Fujimura,

In the fMRI Laboratory

15

1996; Galison, 1987, 1997; Mindell, 2002, 2008; Pickering, 1995; Shapin & Shaffer, 1985). Today, however, although important laboratory studies continue to be accomplished (e.g., Doing, 2004, 2009; Merz & Knorr-Cetina, 1997; Mody, 2001; Roth, 2005; Sims, 2005), the excitement and productivity that characterize the early ethnographies has waned (for a discussion, see Amsterdamska, 2008; Doing, 2008). Even though there is no doubt that the original studies provided a deep grounding for the contemporary STS, with the maturing of the field they have been frequently judged as naïve and too burdensome to conduct.9 Since the 1990s, as the field gained institutional strength, the emphasis on the practical and local has been largely replaced by research more readily focused on scientific texts, larger communities, and societal phenomena. The attention has been turned to policy and governance in public and political institutions, emphasizing global and normative aspects in the practice of science and in extending its analysis to include scientists’ relationships to media, courts, advertising, and funding (e.g., Epstein, 1996; Haraway, 1997; Jasanoff, 1995; Reardon, 2005). This move from the local to larger social structures, though unquestionably productive, has, however, left significant portions of scientific practices still underexplored. Studies of brain imaging are one example. Important work has been done investigating the intersection between popular culture and neuroscience, using cultural criticism to frame the issues of how visual representations are used to explain biological processes, and how they, while generating powerful ideas about our health and identity, are embedded in larger social, political, and economic configurations (e.g., Beaulieu, 2002, 2004; Dumit, 2004; Joyce, 2005, 2008; Prasad, 2005a, 2005b).10 Yet, when these studies target the practice of science, they remain somewhat removed from the details of individual episodes of real-time work with brain scans.11 Their readers, therefore, do not learn much about the material status that digital brains may conserve or acquire during specific instances of laboratory work and interaction. To confront these issues we must reconsider laboratory studies. In fact, despite the prevalence of interest in the large-scale phenomena that characterize the current moment in STS, recent scholarship shows signs of interest in the scientific laboratory. In his study of modern physics, Park Doing (2009), for example, has pointed out the need to investigate

16

Chapter 1

what is left unanswered by the early laboratory studies—that we still do not know much about the relationship between laboratory work and the status of the enduring facts that the laboratory has produced. Even though Doing’s main preoccupation is to reengage laboratory studies and the nowestablished interest in the realm that goes beyond the laboratory, his agenda importantly disturbs the general sense that early laboratory studies, though being foundational for the field, should be considered an accomplished and closed chapter of STS: These days, few sessions at professional meetings, only a handful of journal articles, and even fewer new books are dedicated to the project of ethnographically exploring fact making in the laboratory. After all, why repeat a job that has already been done? Indeed, the job was apparently done so well that there are not even that many laboratory studies in total, despite their subsequent importance to the field. In spite of this unfolding of history, however, questions must be asked of laboratory studies in STS. Did the early lab studies really accomplish what they were purported to have accomplished? Did they, as Knorr Cetina said, show the “‘make’ and accomplished character or technical effects”? And, importantly, are what present studies there are now doing all that they can do? (Doing, 2008: 280–281)

Similarly, in the domain of the history of science, Robert E. Kohler, for example, has recently pointed out a noticeable absence of historical accounts of laboratories. Kohler observes that after a productive start in the 1980s, laboratory history is neglected today. He suggests bringing back the early trends in the “microhistory of laboratory practices” in the form of “macrosocial history of the laboratory” (Kohler, 2008: 1). Handling Digital Brains joins this trend. Yet, rather than broadening the perspective, it invites an even more detailed look at real-time work and interaction in the laboratory. It does so by focusing on what has been acknowledged by original laboratory studies but never thoroughly dealt with: multimodal aspects of scientific practice. Without aspiring to provide an exhaustive treatment of the brain mapping field, my interest is in showing actual instances of brain mapping practice to shed light on the embodied and experiential character of real-time work with digital technology. This is not to say that the laboratory, policies, and media do not coproduce each other; what goes on in the laboratory is always in respect to the broader social structures (and vice versa) (e.g., Latour, 1983). At stake, however, are kinds of entities and agencies that would remain invisible if we were not to look at multimodal semiotic interactions in the

In the fMRI Laboratory

17

laboratory. Paying attention to the dynamic interface between scientists— their gesturing bodies—and the world of instruments and visual displays identifies the objects of scientific practice (our brains and our bodies) as constantly changing, distributed phenomena that dwell at once in multiple spaces. Digital Scientific Visuals as Fields for Interaction Proponents of laboratory studies have argued for the importance of embodied aspects of scientific work, pointing out the centrality of the knowledge that the ethnographer and scientists share (Lynch, 1993). This is particularly the case for those scholars who base their approach in ethnomethodology and phenomenological traditions, such as Michael Lynch, Karin Knorr-Cetina, and Klaus Amann. Knorr-Cetina and Amann, for example, argue that visual imaging (rather than literary inscriptions; Latour & Woolgar, 1979) is central in laboratory work: Images are objects on which work is performed in the laboratory; like other materials handled in the stream of laboratory activities, they are processed. The analysis contained in the data is not written in the image’s face. It is brought to the fore by means of image analysis techniques that look behind the surface of the features displayed. Participants look at the display as one would look through a window that opens to a whole new environment of processes and events. (Knorr-Cetina & Amann, 1990: 262)

The question that remains to be answered is, how? How do scientists engage these visuals in real time? What are the techniques and practical details of such activities? And, specifically, how is this engagement accomplished with respect to the digital nature of the matter they are dealing with? Once we refer to fMRI brain scans as digital and visual at the same time, they are not only “windows” to be “looked through” but also multimodal sites where work is importantly accomplished. In activities such as apprenticeship learning and data analysis, scientists handle brain scans. These practical engagements with a highly malleable substance indicate scientific visuals as fields for interaction. fMRI visuals are malleable fields because they are digital, but also because they can be altered in interaction through the involvement of gesturing hands. In this regard, the fields for interaction are also multimodal: To make sense of the visuals practitioners engage their

18

Chapter 1

eyes (and their thinking brains), but also their hands, ears, and the entire bodies. This active participation and embodied engagement of the scientists suggests that the visibility of digital visuals is relative to the circumstances of their practical and multimodal engagement. The brain visuals, however, cannot be fully characterized as socially constructed. In the laboratory, the imaged bodies perform resistances by placing conditions on their scans. This, though, does not mean that the scans function as transparent conduits that unproblematically reveal the imaged bodies. Instead, the scans, by virtue of being at once visual and highly malleable objects, are involved in articulating digital brains. In other words, fMRI scans function as the center of action with which (not only on which) the work is performed. They allow the practitioners to deal with their experimental data in a manner that is somewhat analogous to our engagement with physical objects. Thus, when handled, fMRI visuals concern at once the material world, the digital reality, and the embodied, culturally shaped, and socially performed actions. They are the places where the scientists’ interactional and experiential bodies are intertwined with the objects of their inquiry. To face fMRI brain scans as sites for interaction that are at once visual and digital, I rely on the interpretative semiotics of Charles Sanders Peirce (1839–1914).12 Semiotics has been an important influence for STS (e.g., Akirich & Latour, 1992; Haraway, 1991, 1992; Hayles, 1993; Latour, 1987, 1993; Latour & Woolgar, 1986; for discussion, see Høstaker, 2005; Lenoir, 1994). Yet, this influence has been primarily grounded in the structural semiotics of Ferdinand de Saussure (1983) and his follower Algirdas Julien Greimas (1987). When referring to Saussure’s semiotics, scholars interested in the material practices of science have suggested ways to overcome the relativism and distance from the material reality and the body characteristic of structural semiotics. For example, Latour (1993, 1999) has spoken about closing the gap between subjects and objects, nature and culture, and Haraway (1991) has proposed the feminist standpoint theory of partial perspective. This trend has, however, overlooked that, in contrast with the Saussurian model of the sign that brackets the referent (excluding from the domain of interest any reference to objects in the world), Peirce’s conception includes what the sign stands for as its necessary part. Peirce’s conception of the

In the fMRI Laboratory

19

sign, with its concern for the materiality of the world, proves particularly valuable when multimodal sign systems and the digitality of the visuals are of interest. As the digitality is seen as a space for embodied engagement, I show how gestural actions and manual handling of the digital matter constitute thinking. Peirce’s semiotic approach—pragmatic and antipsychological—is again of importance as it addresses the issues of the mind without reducing it to an individual’s brain. Instead, the mind is a semiotic process that is dialogic and dynamic. For Peirce, thinking is the operation of signs that regards a communicative agent and comprises language as well as tools and instruments. By embracing Peirce’s semiotics, my taking on of laboratory studies is thus not a fateful return to the original approach, but a proposal of how to deal with the epistemological preoccupations typical of the early approaches while analyzing multimodal action and interaction that characterize local research practices in the computerized age. Inspired by Peirce’s semiotics and turning the gaze toward digital screens (Manovich, 2001) means reconsidering how we understand ourselves and the world in which we live. When cognitive neuroscientists handle digital brains, they also think with their hands while engaging objects of their practice as hybrid phenomena enacted at the junction between the world of technology and the world of corporeal action. At the Opening Ceremony The organization of the chapters of this book follows the movement of fMRI experimental data as they are collected in the scanning facilities (chapter 3), undergo transformations in the laboratory (chapters 3–6), and finally leave the laboratory to be published in professional journals (chapter 7). At the same time, I discuss the status of digital visuals in science (chapter 2), arguing that their boundaries need to be reconceptualized once multimodal aspects of real-time scientific action and interaction are taken into account. The grounding of the discussion in concepts from Peirce’s semiotics and the methodological approaches of ethnomethodology and conversation analysis are my way to join a recent trend in the study of scientific practice exemplified by the work of Annemarie Mol. As

20

Chapter 1

Mol points out, a praxiographic approach no longer captures “a gaze that tries to see objects but instead follows objects while they are being enacted in practice. So, the emphasis shifts. Instead of the observer’s eyes, the practitioner’s hands become the focus point of theorizing” (Mol, 2002: 152). I look at hands in fMRI through the lens of their real-time, multimodal engagements in the laboratory. Once such a grounding has been established, the text describes scientific apprenticeship and the problem of the body and embodiment from the angle of multimodal interaction (chapters 3–5). This perspective leads me to investigate how scientists conceptualize the object of their practice when they access it in terms of digital data (chapters 4 and 6). While highlighting the digital character of fMRI brain scans, I, nevertheless, reassert their visual status (chapter 7). The fecundity of an approach attentive to multimodal enactments shows not only in its capacity to ground an exploration of digital visuals and the objects with which scientists deal but also in its aptitude to allow a critical dialogue between the ethnographer and the field of her study. By focusing on how scientists use their hands to see, and thus understand, what is displayed on the digital screen, Handling Digital Brains reflects upon the problem of the human mind and embodiment, the questions of central interest to cognitive neuroscience (chapter 5). Whereas fMRI technology presupposes that our cognitive processes can be reduced to the workings of an individual’s brain, I argue for a view of cognition that is distributed between people and rooted not only in mental processes and computational inferences but also in a culturally shaped and socially enacted world (e.g., Cole, 2003; Engeström & Middleton, 1996; Hayles, 1999; Hutchins, 1995; Lave, 1988; Middleton & Edwards, 1990; Mukerji, 2009; Neisser, 1982; Norman, 1988; Rogoff & Lave, 1984; Suchman, 1987). The book develops this line of thinking by turning attention toward the body, digital screens, and the processes that are often performative and dynamic in character. As recorded during the aforementioned opening ceremony of the UCSD fMRI brain imaging center, Edward W. Holmes remarked: This state-of-the-art resource will accelerate the pace of discovery in the studies of the brain and its function. Armed with this technology we can address the fundamental questions about what is arguably the most fascinating organ in the body— our minds.

In the fMRI Laboratory

21

In a similar tone, Roderic Pettigrew added: We can now visualize the mind in imaging the brain. With this kind of resolution we are now able to attach the intangibles to tangibles to see the mind in the brain. Memory, forgetfulness, truth, lies, happiness, sadness can be visualized and thereby studied.

The event was conceived as the beginning of a new era for research at UCSD, as the presence of fMRI technology framed the future of scientific endeavors. The speakers talked about the ways in which fMRI technology rearticulates the possibilities for research by allowing scientists to acquire new capabilities for vision. “Armed with the technology,” scientists will be able to see the mind as a process of the human brain. I listened to these statements with a somewhat divided stance. Like the speakers, I saw the promise in the scientific study of human cognition. On the other hand, I felt somewhat uneasy. I could not help but think about the ways in which this technological framing, while generating new possibilities for research, constrains how we understand ourselves as human beings. If “memory, forgetfulness, truth, lies, happiness, sadness” can be studied by analyzing recordings of physiologic changes in the human brain, where does this leave the rest of our bodies, engaged in the sociocultural world we live in? In other words, could our conception of technology and the results it generates preclude us from understanding the mind as a distributed and situated phenomenon where (just like for Jane and Paul at the beginning of this chapter) our thinking emerges through the interaction of our brains with our hands, eyes, ears, and tongues as we engage in the everyday activities, gesturing, seeing, listening, speaking, and interacting with each other and the world of our practice? To deal with these questions, and thus take part in conversation with the field of science under scrutiny, I invite the reader to pay close attention to the role of multimodal interaction in the work of fMRI brain imaging. The practical and semiotic engagement with digital brains in the laboratory problematizes assumptions behind fMRI technology and its associated practices; it suggests that human meaning-making and learning importantly concern our hands.

2 fMRI Brain Visuals as Fields for Interaction

fMRI brain visuals are signs in a very straightforward sense: Cognitive neuroscientists observe the human brain and its processes by consulting its fMRI renderings. Alan Gross (2008: 281) has suggested that the character of fMRI brain visuals should be understood in terms of indexical signs. In his proposal, Gross refers to Peirce’s1 famous distinction between icon, index, and symbol,2 articulated with respect to the relationship between the sign and its object (Peirce, C.P.: 4.531). For Peirce, whereas symbol, most closely related to the Saussurian language-like sign, is a conventional sign denoting its object with respect to a rule, iconic and indexical signs are characterized by their materiality and embodiment. Index is the sign that is physically or causally connected to its referent and thus always bound to specific circumstances of its instantiation. Examples are a pointing finger and footprints in the sand. Icon, on the other hand, is a sign that shares characteristics with the object, perceived as having some similarity with it. Usual examples of icons are a realistic painting and a wax statue. The indexical character of fMRI visuals is evident in how they are generated. Just as a photograph has a causal relationship with its subject,3 there is a causal relationship between the brain and its fMRI rendering. However, fMRI visuals are also iconic. The claim for the iconic character of brain visuals, though, should not be equated with a naïve idea of similarity: fMRI visuals are not iconic signs because they look like the brain and its processes. Rather, fMRI visuals are iconic as they are understood through an active visual inspection and embodied engagement. In other words, the iconicity of fMRI visuals comes to the fore when they are considered from the perspective of real-time, practical engagement.

24

Chapter 2

This chapter looks at a published fMRI figure to show that fMRI visuals are not iconic signs in terms of the naïve idea of similarity, but that they generate meaning by relying on a variety of semiotic structures that function as their “infrastructure for seeing.” The published fMRI figure does not directly “reveal” to a passive eye the brain and its processes; instead, it relies on a variety of signs that indicate what the figure shows as they call upon the viewers’ cultural knowledge and experiential engagement. One should, however, ask what functions as the infrastructure for seeing when scientists engage fMRI visuals during their everyday laboratory work. The question of iconicity and semiotic infrastructure is important for at least two reasons. First, it problematizes the productivity of the dichotomy between the visual and the digital. Second, it calls for a reconceptualization of scientific visuals and their boundaries. fMRI scans, when considered from the perspective of their everyday, real-time engagement, are neither only visual nor only digital; they are at the same time visual and digital. This engagement with brain visuals can be tackled in terms of written as well as gesturally enacted signs. Because they do not generate meaning in the absence of their infrastructure for seeing, such infrastructure is their constitutive element. Digital scientific visuals are, thus, fields for interaction as they have to be understood with respect to how they are worked with and experienced. In other words, their character is not necessarily representational, but it concerns the participation of their readers/ writers. In making this argument, the chapter relies on the interpretative semiotics of Peirce and his follower—Umberto Eco. Whereas Peirce speaks to social studies of science and technology through his own writing in the philosophy of science (e.g., Rescher, 1978), I want to highlight some of the features of Peirce’s semeiotic and pragmati(ci)sm not originally aimed at studies of science and less commonly referred to in STS. This overview, however, is not intended as an exposition of the theory that underpins the practice (expounded in the chapters that follow). Rather, it is to clarify some of the concerns that sustain the practice-oriented analysis that constitutes the core of this book. At the same time, in providing empirical examples and analyzing videotaped material of laboratory work and interaction, the goal is to generate a sense of how the next step in engaging Peirce’s semiotics in STS can be taken.

fMRI Brain Visuals as Fields for Interaction

25

Iconicity and fMRI Brain Visuals In the 1960s and 1970s, semioticians aimed to dismantle iconicity in terms of the naïve idea of similarity, characteristic of our intuitive understanding of visual images (Barthes, 1964; Eco, 1976; Volli, 1972). In accordance with the structuralist tradition, the problem of similarity has been treated in relationship to cultural conventions and codes. Semioticians, when analyzing cultural codes of realistic drawings, cinematic images, and magazine advertisements, wanted, if only partially, to subsume such signs under the umbrella of arbitrariness. This radical position of the early years has been revisited on several occasions. In his most recent book on semiotics, Kant and Platypus, Umberto Eco (1999) sets out to somewhat reconsider his original, and what he calls “iconoclast,” position. Eco laments that since the peak of the debate, many have been influenced by Peircian semiotics, yet this influence primarily concerned the notion of unlimited semiosis, leaving the theorizing of iconism largely unexplored (Eco, 1999: 342). To deal with this neglect, Eco directs attention toward Peirce’s suggestion that iconic signs generate “effects of similarity.” For example, even though fMRI brain visuals should not be equated with what they stand for—the brain and its processes—they generate a sense of resemblance with their referents. How should this idea of resemblance be understood? Although Eco, with Peirce, maintains that the interpretation of the iconic sign contains a perceptual basis,4 the anchoring of the sign in the material world, however, does not mean that an iconic sign should be equated with the iconic nature of perception. Perceiving a brain and seeing its fMRI rendering, for example, are two different phenomena. To deal with the immediate impression of likeness that iconic signs generate, Eco talks about “surrogates for perceptual stimuli.” The idea is that even if under certain conditions a sign generates effects of similarity, we have to acknowledge that these impressions are relative to the surrogates manufactured to generate the effects. To provide an example, Eco talks about a visit to a perfume factory. Experiencing the manufacturing of a perfume highlights the difference between a perceptual iconism and an impression achieved by the way of surrogate stimuli: Anyone who has ever visited a perfume factory will have come up against a curious olfactory experience. We can easily recognize (on the level of perceptual experience) the difference between the scent of violets and that of lavender. But when we want

26

Chapter 2

to produce industrial quantities of essences of violets or lavender (which must produce the same sensation, albeit a little enhanced, stimulated by these plants), the visitor to the factory is assailed by intolerable stenches and foul odors. This means that in order to produce the impression of the scent of violets or lavender, one must mix chemical substances that are most disagreeable to the olfactory sense (even though the result is pleasant). I am not sure nature works like this, but what seems evident is that it is one thing to receive the sensation (fundamental iconism) of the scent of violets and another thing to produce the same impression. This second operation requires the application of various techniques with a view to producing surrogate stimuli. (Eco, 1999: 352)

To figure out the methods that generate the impressions of similarity, Eco talks about the observer whose positioning makes the constructed character of the iconic sign obvious. A visitor to the perfume factory has a different olfactory experience than that of a customer buying a perfume in a department store. Similarly, we can experience a painting as veridical only if we stand at a certain distance from it; if we move too close, the illusion of reality disappears. This means that the surrogate stimuli partly depend on the way in which we engage with them: in the perfume factory versus in the department store, too close to the painting versus at a certain distance from it (Eco, 1999: 353). Thus, instead of discussing the problem of iconicity only in terms of the relationship between the sign and its referent, Eco’s comment indicates that we should consider the acts of perception as essential elements in the functioning of the sign. To discuss the apparent tension between the digital and visual character of fMRI brain scans, I want to explore this direction in understanding the iconic sign. However, instead of using the positioning of the viewer to prove that the iconicity is achieved through construction (as is the case in Eco’s example), I consider the iconicity in terms of the user/designer’s interaction with the visuals. Scientists use fMRI as a way of identifying specific regions on the human cortex that process types of information. Yet, when they indicate the specialized areas with colorful patches on the cortical surface (see, e.g., figure 2.1 [plate 1] as well as figure 7.1 [plate 2]), they do not intend to show how physical brains appear to sight, but how the brain areas and the information they process are related to or distinctive from each other. To understand and work with such renderings, though, fMRI practitioners actively exploit their visual character. Generated through a series of measurements, fMRI scans depict the otherwise invisible cognitive processes

fMRI Brain Visuals as Fields for Interaction

27

in terms of visuospatial features that, combined with the digital character of brain scans, are engaged in ways that parallel how we treat objects and processes in our everyday world.5 This mode of engagement with visuospatial signs is exactly where their iconic character comes into play. The Case of the Published fMRI Figure Let’s start the discussion of iconicity by analyzing the published fMRI brain visual reproduced in figure 2.1 (plate 1). The figure comes from an

Figure 2.1 (plate 1) Published fMRI figure from the article by Martin Sereno, Sabrina Pitzalis, and Antigona Martinez entitled “Mapping of Contralateral Space in Retinotopic Coordinates by a Parietal Cortical Area in Humans” (Science 2001;294:1350–1354).

28

Chapter 2

article by Martin Sereno, Sabrina Pitzalis, and Antigona Martinez entitled “Mapping of Contralateral Space in Retinotopic Coordinates by a Parietal Cortical Area in Humans,” which appeared in Science in 2001 (Sereno et al., 2001; fig. 2).6 The analysis of the figure serves in dissipating the claim that fMRI scans refer to their objects by natural and immediate likeness. In other words, to claim that fMRI brain scans are iconic signs does not have to imply that such renderings need to be understood in terms of naïve iconicity, as their effects of similarity are not necessarily based in a relationship of simple resemblance (or isomorphism) with what they stand for. First, the figure implies the choices that have been made: it excludes some material while representing other. In this sense, the analysis of the figure highlights elements of the discursive universe that characterizes the field of cognitive neuroscience. The way in which the figure articulates what needs to be seen implies expectations in the field of cognitive neuroscience, while at the same time it reveals innovation and the authors’ resistance to dominant forms of representation. Second, the figure is a supervisual: it appeals to our senses and embeddedness in the world, showing what our “naked” eyes cannot see. Scientists, by coordinating their seeing with technology, observe the “wrinkled” cortex as a “flat” and “cut” sheet on which temporal processes are depicted as spatial phenomena (as explained by Paul in his interaction with Jane, reported in the previous chapter). This seeing of what cannot be seen relies on the human aptitude to think by exploiting our visual capacity and our skill of handling objects in the world. The Cerebral Cortex as a Map The article by Sereno and colleagues (from which figure 2.1 was taken) is an example of the brain mapping technique used by scientists to project the results of measured brain activation onto the spatial renderings of the brain. A brief look at the graphical and textual components of figure 2.1 highlights the historically marked, sociocultural elements at play. The figure implicitly refers to arguments around the issues of localization of function, retinotopic mapping of the visual cortex, and the theoretical differences between claims to the existence of cortical maps versus neuronal modules.

fMRI Brain Visuals as Fields for Interaction

29

The concept of human brain mapping and the idea of localization of function primarily refer to the cerebral cortex, the outer structure of the brain (leaving the rest of the brain in the background). The functional role of the cortex was given early attention by the 18th-century mystic Emanuel Swedenborg (1688–1772), who attributed to it sensory, motor, and cognitive functions. In his search for the biological site of the soul, Swedenborg put forward the idea of a somatotopic organization of the motor cortex, where the motor cortex is structured in a map-like fashion containing an array of areas specialized in controlling the movement of different parts of the body (he localized control of the foot in the dorsal cortex, the trunk in an intermediate site, and the face and head in the ventral cortex) (Gross, 1998: 127). Even though Swedenborg’s proposal may not have had any effect on the development of neuroscience (Gross, 1997), the idea of localizing specific brain function in the cerebral cortex persisted. This privileging of the brain cortex, whose understanding involved making a map and localizing its functions, was further developed by phrenologists Josef Gall (1758–1828) and Johann Spurzheim (1776–1832). Despite the fact that the phrenological enterprise, whose goal was to identify an individual’s mental faculties through the measures of her or his skull, is not considered scientifically valid, Gall and Spurzheim’s proposal of a correlation of function with cortical locations is still respected. For example, John Allman (a contemporary neuroscientist well known for his studies of primate cognition) notes that: “The phrenological maps are pure fantasy without any basis in experimental or clinical observations. However, the phrenologists can be credited with the general idea that functions are localized in particular places in the brain” (Allman, 1999: 31). Research on function localization continued its development in the work of some of the early modern experimental neurobiologists, such as Pierre Flourens (1794–1867), Paul Broca (1824–1880), and John Hughlings Jackson (1835– 1911) (see, e.g., Star, 1989), reaching new heights in contemporary fMRI research. This preoccupation with the brain cortex and the interest in localization of brain function is clearly exemplified by figure 2.1. The figure is a result of a computational transformation, which fMRI practitioners call surface reconstruction, where the data represented by a series of “raw” anatomic scans are merged into one single visual. Through the process of making the cortex visible, however, the rest of the brain is selected out so that the

30

Chapter 2

cortical renderings can be employed as a substrate on which functional data (standing for brain processes) are projected. These depictions of the brain function concern the topographic predictions, or so-called retinotopic maps, used to describe the organization of the visual cortex.7 The proposal of topographic organization of the visual cortex has a noteworthy history. While its origins can be traced back to the work of the Arab visual scientist ibn al-Haytham (965–1039) (Gross, 1998: 76), the first to experimentally reveal this organization were “lesions studies,” or studies of damage to delimited areas of the brain. The lesions studies in animals, as well as naturally occurring lesions in humans, were used to show that the visual field, and hence the retina, is represented within the cortex in a very orderly fashion, where adjacent locations in the visual field are represented in adjacent locations in the cortex. Particularly well known are studies of soldiers who suffered head injuries in the Russo-Japanese War (the work of Japanese ophthalmologist Tatsuji Inouye) and in World War I (the research of British neurologist Sir Gordon Holmes) (Kauffmann Jokl & Hiyama, 2007). Today, the enterprise of mapping the visual cortex with fMRI is regarded as one of the most promising research areas in the field of cognitive neuroscience. According to the present-day knowledge in neuroscience, the human visual cortex, located in the posterior part of each hemisphere, consists of multiple areas. Once the information arrives from the eyes, via visual pathways, to visual centers of the brain, it is passed from multiple areas located in the early visual cortex to the areas of the higher-order visual cortex. The early visual areas tend to encode more elementary features, such as lines, whereas higher-order visual areas contain neuronal groups that encode more complex features, such as edges, curves, and composition of features. Scientists know that the early visual areas preserve the topography of the visual field, but, as Sereno and colleagues’ article indicates, they are interested in finding out if the higher-order visual areas are retinotopically organized as well. As reported in the article, the process of identifying retinotopic maps on the cortical surface involves the presentation of a patterned stimuli moving through the field of view of a subject being scanned. Due to the temporal match between the stimulus and the neuronal response, the scientists identify which parts of the visual cortex process stimuli in specific points of the visual scene. Scientists use such reproductions of spatial (not

fMRI Brain Visuals as Fields for Interaction

31

pictorial) relations in the brain cortex to assess the location and borders of a specific brain area.8 These kinds of areas are represented in plate 1 as multicolored patches on the gray surface of the cortical anatomy. When describing the functional organization of a specific region of the human cortex, fMRI practitioners refer to each of the brain activations or their clustering in terms of maps and modules (Op de Beeck, Haushofer, & Kanwisher, 2008). Thus, in addition to seeing the entire cortex of the human brain as a map,9 scientists also use the term map to denote the organization of specific functional areas. For example, scientists who study the retinotopic organization of the visual cortex talk about map-like organizations on the visual cortex. The two labels—maps and modules—evoke an important debate in the contemporary study of the human mind. The proponents of modules argue for the existence of areas that selectively respond to the specific stimuli, where such regions are discontinuous, showing a clear difference in the neuronal response across their boundaries. On the other hand, the maplike organization, like the one depicted in figure 2.1, indicates a gradual shift in the peak of neuronal activation, considered to be a part of a largerscale cortical map. The idea of the neuronal module in neuroscience is often associated with what may be defined as a more complex and theoretically committed concept of module in cognitive science as used by the proponents of the argument for modularity of mind. Philosopher Jerry Fodor (1983), drawing on the linguistic theory of Noam Chomsky, argued that the mind is composed of domain-specific and genetically specified functional units. The idea, resembling some aspects of phrenology (Uttal, 2001), was criticized by the proponents of connectionism (e.g., Elman et al., 1996) and other more recent cognitive science trends that argue for the concept of network in understanding of the brain functioning. These trends propose that mental phenomena should be seen as emergent effects of larger networks (associated with multiple brain locations). By pointing out the existence of a map-like organization, rather than a module, figure 2.1 indicates the positioning of its authors in the debate: They mark themselves as interested in connections between localized areas and thus farther removed from the claims for the modularity of mind. In this sense, the fMRI visual (as a depiction of cerebral cortex that indicates the geographic organization of brain processes) not only shows its sociocultural articulation but also connotes alignments and takes positions.

32

Chapter 2

How the fMRI Figure Indicates Brain Activations Compared with other neuroimaging techniques such as electroencephalography (EEG) and magnetoencephalography (MEG), which generate more precise information about the temporal dynamics of electrophysiologic processes occurring in the millisecond range, the metabolic imaging techniques such as fMRI10 provide a rougher picture of the temporal dynamics but are extremely powerful in localizing brain structures that are active during a cognitive task (Pulvermüller, 1999). Cognitive neuroscientists see the promise of fMRI technique in its capacity to noninvasively generate high-resolution brain visuals used to identify where in the human brain specific cognitive processes take place. In this sense, the scans are conceived as maps, intended to point out location and relationship among brain activations. Though the idea of visual cortex as a map already has a long and intricate history (as pointed out in the previous section), the way in which fMRI figures indicate the existence and location of cognitive processes reveals their complex sociocultural character. With its colors, legends, textual and graphical labels, all enrolled to indicate where a brain activation is located, an fMRI brain visual does not simply resemble the brain that was scanned. One of the immediately noticeable elements of figure 2.1 is the labels located across the brain scans. To indicate general patterns in the locations of brain activations and relationships among them, fMRI researchers have to deal with the individuality of every human brain (just like with our faces, there is a significant variation in the anatomy of our brains). To confront the individuality, as they point out the generality of the research findings, fMRI figures are labeled. These labels, used not only by the “untrained eye,” but also by the “expert reader,” indicate where on the human cortex the relevant activations are located. There are multiple labeling systems adopted by the fMRI community (Brett, Johnsrude, & Owen, 2002), none of which is theory free. Each of the labeling systems, while enjoying different levels of popularity, implies the positioning of its users in the field of cognitive neuroscience. By choosing one labeling system over another, fMRI practitioners, for example, indicate their position in the maps versus modules debate. Those scientists who, like Sereno and his colleagues, talk about brain activations in terms of maps tend to be interested in lower-level cognition

fMRI Brain Visuals as Fields for Interaction

33

(such as visual and auditory processing, instead of linguistic and conceptual issues) and cross-species comparison. As exemplified by figure 2.1, to convey the shape and the exact extent of the activation area, they often represent data sets collected from single persons (avoiding averaging across multiple sets) with a goal of preserving the fine-grained elements of individual data. Typically, these scientists use as their main labeling technique a type of system based on the anatomy of the human brain. The Sereno et al. article suggests that in addition to the well-known retinotopic areas in the early human visual cortex, there are also retinotopic maps in the higher-order visual areas. The article reports the finding of a map, whose function is to represent the angle of a remembered target, located at the border between the visual and somatosensory cortices. To function as evidence for such a claim,11 figure 2.1 marks the location of brain activations with respect to the well-known anatomic formations such as the sulci, or grooves, on the brain cortex. This labeling system is visible in the several layers of graphical signs inscribed over the brain visuals. The figure uses text labels, dotted circles, and legends for colors and scale to situate the brain activations with respect to the positions of the important brain sulci.12 The anatomic structures are not marked to teach the viewer about their location, but to position the activation sites. In accordance with the enterprise of function localization, the gaze needs to be directed toward the activations on the cortical sheet, which are circled by the white dots and situated in relationship to the sulci indexed by the text labels. To distinguish and locate the brain activation on the cortical map, the labels and other graphical signs are, furthermore, coordinated with colors that can be sorted into three distinct groups: the colors associated with the background, the structural representations of the cortex, and the maps of remembered targets (see plate 1). The background, whose role is only to contrast what is of interest and has no intrinsic importance on its own, is black. This black “background” erases the cortical representation of the left hemisphere and the rest of the body of the person being imaged.13 In contrast, the structural scans that represent what is static are in tones of gray. These gray renderings make visible the anatomy (or structure) of the brain on which the maps of the brain processes (its function) are overlaid. The figure also provides a legend to indicate what the gray tones stand for: the lighter gray signifies the existence of a gyrus

34

Chapter 2

(or a convolution of the brain), and the darker gray indicates the presence of a sulcus, signifying the three-dimensionality of the space represented as two-dimensional.14 In contrast, the portions of the cortex that are considered to be of primary interest are indicated with brightly colored patches of red, blue, and green. The bright colors show what part of the human brain is active when “processing contralateral remembered targets.” In other words, the difference in colors, derived through the calculation of the temporal match between the stimuli and the neuronal response, codes the phases of visual processing understood as a response to changes in the visual scene. As the choice of color is not standardized, the caption and legends provide an explanation of what different colors stand for.15 Similar to the use of “false colors” in Earth-satellite photography and astronomy (Ihde, 1998: 92; Lynch, 1991), fMRI visuals that indicate brain activations with bright colors are marked to guide the gaze and indicate, rather than represent, the world in a truthful manner. In sum, the way in which fMRI figures designate brain activations and their location entails a series of choices. Figure 2.1 shows cortical maps by situating their renderings with respect to the anatomic landmarks. This operation is generated by using an array of semiotic structures—textual and graphical signs—which mark what is relevant and how it should be read. The way the figure guides the viewer’s gaze over its territory is another element that complicates the relationship between the fMRI scan and the brain functions. The fMRI Figure as a Supervisual The idea of naïve iconicity is further negated by the “super” character of fMRI brain visuals. Figure 2.1 is a supervisual in at least two senses: it represents temporal changes (function) in terms of spatial phenomena (indicated in different colors, as explained above), and it displays the anatomy of the brain in ways that exceed what an unaided human eye can see. By representing the activations as patches on the cortical surface, figure 2.1 shows as visible and spatial phenomena the invisible and temporal events. These visuospatial signs, inscribed on the renderings of the brain function as translations of temporal events, are another reminder that

fMRI Brain Visuals as Fields for Interaction

35

fMRI visuals cannot be simply judged in terms of resemblance with their referents. Figure 2.1 is also a superimage because of the way it shows its material structure (not only its brain processes). Even though in the case of brain anatomy fMRI visuals depict what is spatial and potentially visible (beneath the skin and skull), in many ways they show what is not accessible to the human eye outside the digital realm. The frequently used digital transformation that generates so-called flattened maps (as discussed in chapter 1) illustrates this point. The technique produces the renderings of the brain cortex where the characteristic brain fissures (or sulci) and convolutions (or gyri) are all depicted as positioned on the same plane so that twodimensional (2D) representations allow for an improved visual inspection of experimental data. By generating visuals that resemble less and less what they stand for, scientists are able to enhance the production of knowledge through sight. As the labels written in capital letters indicate, figure 2.1 displays the cortical sheet as if it were cut into pieces. The figure shows the right hemisphere superior parietal cortex from five different views: lateral, superior, posterior, medial-posterior, and medial. This cancellation of just one physically situated point of view is another way to provide the viewer with an enhanced gaze that sees the three-dimensional (3D) structure in an omniscient manner. When Sereno and colleagues talk about “inflating,” “cutting,” “spreading out,” and “flattening” the cortical surface, they refer to the fMRI computer program designed by the first author of the article. The computer program performs digital transformations of experimental data understood in terms of physical actions. According to the authors, although the representation of a 3D folded cortex preserves a more “natural” appearance, this type of visual rendering does not show what is buried in the fissures on the surface of the brain. Because of the high percentage of concealed cortical structures, “distance measured in 3D space between two points on the cortical surface will substantially underestimate the true distance along the cortical sheet, particularly in cases where the points lie on different banks of a sulcus” (Fischl, Sereno, Tootell, & Dale, 1999: 273). Hence, the technologically enhanced “unfolded” visual, even though further removed from the actual appearance of the biological brain, allows a more precise understanding of the cortex, as this “cyborg view” (the coordination of

36

Chapter 2

digital technology with the human eye) enables the viewer to acquire knowledge through a “superseeing.” By looking at figure 2.1, the viewer learns about human cognition as she sees the invisible brain processes displayed over the cortical map whose renderings are cut and flattened by means of a digital manipulation of fMRI data. While showing this “invisible reality,” the figure evokes the digital processes, given as concrete actions, that have been accomplished to enable its showing. Multivoicedness of the fMRI Figure A strong social shaping of fMRI visuals is exemplified by the multiplicity of human voices (Bakhtin, 1981) that figure 2.1 inscribes. The labels, numbers, and legends in figure 2.1 indicate the voices of a larger scientific community (or, even broader societal forces) as well as the voices of the figure’s authors. The way these voices intertwine is complex. When inscribed over fMRI visuals, the semiotic structures reveal a propensity toward standardization—characteristic of the field of cognitive neuroscience—while implying local resistances and negotiations introduced in the figure by its authors. In other words, the visual organization of the figure not only indicates an array of players involved in its fashioning but also shows their antithetical positions and their, often just momentary, reconciliations. As pointed out in the previous section, the flattened renderings displayed in figure 2.1 are generated to enhance visibility. Somewhat ironically, to be readable by a wider audience interested in the localization of function but tormented by the individual variations, these supervisuals have to refer to other types of representations. Figure 2.1 provides two kinds of brain visuals: In addition to the renderings produced with the program for data analysis designed in Sereno’s laboratory (the supervisuals), the bottom row of the figure displays the more commonly seen renderings of the brain structure. These structural representations of the brain slices—coronal, sagittal, and axial—refer the supervisuals to the standard as they, by way of the yellow cross, indicate the center of the parietal maps on the supervisuals. The linking of the two types of visuals not only explains the meaning of one set with respect to the other set but also indicates a tie between the widely accepted and the local.

fMRI Brain Visuals as Fields for Interaction

37

As the caption of figure 2.1 explains, the supervisuals are obtained from data collected from a single experimental subject. The authors’ aim is to describe a specific map on the human cortex that, they believe, can be most precisely viewed when the cortex of one person is shown. By explicitly pointing out that the figure represents the cortex of a single brain, the caption acknowledges its somewhat exceptional status. The more common procedure, shaped by the overreaching goal to find out how the human brain in general (rather than an individual brain) processes information, is to statistically analyze the data of multiple individuals and then overlay them on the renderings of brain anatomies merged into one single structure. When working with experimental data collected by scanning multiple individuals, fMRI researchers commonly use the well-known spatial normalization procedure specified in Talairach and Tournoux’s stereotaxic atlas (Talairach et al., 1967; Talairach & Tournoux, 1988). The procedure consists of a scaling method according to which each brain representation can be proportionally transformed to match roughly another brain representation in overall size and shape. Once the data have been scaled, researchers compare them with the “standard brain” or “Talairach brain” in the atlas. The Talairach brain is composed of photographed and labeled brain sections—axial, sagittal, and coronal brain slices—from one hemisphere of a 60-year-old French woman, indicating how the drive to achieve generality often erases particularity. The brain activations in “normalized” data are often reported in terms of stereotaxic coordinates where every point in the brain is labeled with respect to the same, well-known geography.16 Figure 2.1, however, does not obediently follow this procedure. As already mentioned, to localize brain activation the figure uses an alternative labeling system based on the anatomy of the human brain: The map of remembered angle is identified by specifying the position of the brain sulci and relating the activation to them. This type of labeling exhibits resistance to the widely used standardization system as it reflects the interest, training, and positioning of the authors in the field of cognitive neuroscience. From the choice of the labeling system, the viewer can read that the authors aim to show the geography of the human cortex with more accuracy, especially with regard to the visual cortex, where the position and shape of sulci are less variable across individuals. The labeling further indicates that the authors are

38

Chapter 2

interested in a comparison between the results obtained from the human brain and the brains of other species, as their labeling system is known to be advantageous in cross-species research. In contrast, the authors find the Talairach approach (based on 3D stereotaxic coordinates, rather than on position relative to the 2D cortical sheet) to be deficient. This is particularly the case when the location of interest is sited near a deep fissure on the cortex, as a small change in coordinate could generate a significant error, corresponding with a large change in distance across the cortical surface.17 Nevertheless, figure 2.1 translates the results of the study into the Talairach coordinate system, consenting to the more general and expected procedures. In addition to relating the found activations to sulci on the cortex, the authors specify the location of the activations in the Talairach coordinate system—the format accepted and used by the cognitive neuroscience community at large. In fact, the caption of the figure indicates how to locate the activation maps on the supervisuals through Talairach coordinates, specifying that the center of the map corresponds with three coordinates: x = 32, y = –68, z = 46 (referring to left-right, posterior-anterior, and ventral-dorsal dimensions, respectively). In this way, the figure, while implying a tension and possibly an attempt to destabilize current practices, inscribes a dialogue between the group of researchers and the standardization procedures endorsed by the larger scientific community.18 The Model Reader and Reading Brain Visuals Figure 2.1 connotes a series of decisions about the ways in which brain activations are conceptualized (e.g., maps versus modules) and how different labeling methods (e.g., macroanatomy versus stereotaxic coordinates) are conceived and negotiated. By enlisting colors, legends, and other textual and graphical strategies to indicate what needs to be seen, the appearance of the figure also shows how fMRI data were handled (e.g., flattened, cut, and color enhanced). This articulation of the fMRI figure (1) disapproves the possibility of treating fMRI brain visuals in terms of naïve iconicity and (2) indicates the central role of the reader of such visuals. Though not necessarily implying conventionality (in other words, the figure is not simply Peirce’s symbolic sign), the figure needs a knowledgeable eye to be comprehended. In other words, when students of cognitive

fMRI Brain Visuals as Fields for Interaction

39

neuroscience first encounter fMRI data, they have to learn how to “see” what the data represent to align their readings with the model reader (Eco, 1979, 1990) of such data. According to Umberto Eco, every text is incomplete, demanding cooperative acts of interpretation to actualize its meanings. At the same time, the text is not open to just any kind of interpretation but demands specific readings. The text anticipates and directs its interpretations by providing indices that guide the reader’s inferences, steering the interpretations toward the preestablished courses of reading. The text aims to organize its readings so that its meaning can be understood in the most suitable way. This textual strategy is the text’s model reader. As our reading of figure 2.1 indicates, the fMRI visual inscribes its model reader as it provides, at the level of expression, indices for its expected reading. To show where in the brain cognitive processes occur, a published fMRI figure comes to its readers as a composite field made up of multiple semiotic layers that point out how the brain scan should be read. Figure 2.1 couples the brain scan with the text of the article, the caption, and the graphical marks laid over the scan to allow its reader to see what the scan shows. These semiotic elements function as its infrastructure for seeing. In Eco’s proposal of the model reader, the text is considered to be independent from the intentions of its empirical author and the effects that it may have on its empirical readers.19 The position, reflecting larger trends in textual analysis and semiotics (especially its structuralist tradition), intentionally avoids individual empirical readings to focus on the internal coherence of the text, as the attention is directed toward the ways in which the text itself inscribes its instructions for reading. The analysis of the discursive strategies present in figure 2.1 implicitly assumed Eco’s idea of the model reader as we discussed the cultural knowledge that the fMRI published figure demands. We took for granted an ideal reader who is familiar with the location of the superior temporal sulcus, who knows what Talairach coordinates refer to, and is informed about procedures through which digital brains are handled in the laboratory. This reader, as an abstract strategy of the text, is able to identify the connotative (Barthes, 1972) dimensions of meaning that the labels, the color legends, and the multiple graphical styles of the brain visuals imply. She uses an array of semiotic layers to understand the aims, general values, and the debates that characterize the research field in which the published figure

40

(a)

Chapter 2

(b)

Figure 2.2 (a) Functional and (b) structural fMRI brain visuals.

participates while perceiving the effects of similarity that it generates. In indicating this competency, my goal was not to force the “optimal” readings on the text. Instead, I wanted to point out that this competency evokes the researchers’ work in the laboratory. Take, for example, figure 2.2. After experimental data have been recorded in scanning facilities, the visuals look like the two corresponding brain scans in figure 2.2. The difference between this figure and figure 2.1 (the presence of colors, graphical and textual signs inscribed over the brain renderings, and their composite character) suggests the laboratory work that has taken place between the moment in which a brain scan has been collected and the moment in which the corresponding scientific results have been published. To produce publishable results, the scanning sessions must be followed by months of tight coordination between digital technology and human labor. That coordination, directed at generating the infrastructure for seeing, inscribes its model reader. Yet again, what functions as the infrastructure for seeing during the everyday work in the laboratory? How is such infrastructure articulated? To uncover the infrastructure for seeing not only in terms of the signs inscribed on the paper but also as ephemeral semiotic acts performed in the laboratory, we must go beyond the idea of model reader to take into account the interactional and phenomenological aspects of empirical

fMRI Brain Visuals as Fields for Interaction

41

readings. We have to consider how scientists inscribe the model readers in the fMRI visuals that they prepare for publication. These interactions not only show how the semiotic layers that participate in generating the meaning of fMRI visuals are enacted in practice but also illustrate the iconic character of such signs. While the recall of practical dealings in the idea of supervisuals already evokes the iconic quality of fMRI renderings, their iconicity irrupts with all its force once the attention is directed toward the real-time interaction in the laboratory (what can be seen and directly experienced). In making this turn from the textual analysis to the description of multimodal practices, the effects of similarity show themselves in how fMRI visuals partake in the materiality of the embodied practice. Their “likeness” concerns how they afford action in the lived world of their readers/writers. fMRI Brain Visuals as Diagrams Peirce has explained that iconic signs, in addition to being images, can also be diagrams and metaphors (e.g., Peirce, C.P.: 2.277). Images are the iconic signs that have the same simple quality as their objects, diagrams are the signs whose parts have analogous relations to those of their objects, and metaphors are the icons that show the representative character of a sign by indicating a parallelism in something else. Thus, a portrait would be an example of the image; a map would be an example of the diagram; and a knife could be treated as a metaphoric sign that stands for a gun because both knives and guns can be used for killing. Since we see fMRI brain renderings as a part of our visualized world, we are prone to slip into understanding them as purely image-like. It is, however, more productive to think about fMRI visuals in terms of diagrams as it takes into account how practitioners deal with them. Scholars agree that the meaning of the diagram in Peirce’s philosophy is much broader than our everyday use of the word diagram (Shin, 2002).20 A diagram, thus, consists of representational elements and the rules that allow for the manipulation of such elements.21 According to Peirce, a good diagrammatic system should be “mainly an Icon” so that the parts of the diagram are related to each other in the same way that the represented elements are related to each other (Peirce, C.P.: 4.531). Peirce’s aim was to

42

Chapter 2

show that diagrammatic signs allow experimentation and generate insight as they can be used to draw new conclusions about the relations existing in the world. To explain the importance of this account for the case of fMRI visuals, consider a passage from Peirce’s 1906 “Prolegomena to an Apology for Pragmatism.” The essay opens with the writer’s explicit invitation to the reader: “Come on, my Reader, and let us construct a diagram to illustrate the general course of thought; I mean a System of diagrammatization by means of which any course of thought can be represented with exactitude” (Peirce, C.P.: 4.530). This initial claim, stating that the diagram can be used to render the dynamics of the thought, is followed by an adversarial question: “But why do that, when the thought itself is present to us?” The writer explains that this question, inquiring into the function of signs, has been posed to him frequently. Among those who raised the point were “superior intelligences,” including “an eminent and glorious General.” This general is the writer’s interlocutor in the imaginary dialogue that follows: Recluse that I am, I was not ready with the counter-question, which should have run, “General, you make use of maps during a campaign, I believe. But why should you do so, when the country they represent is right there?” Thereupon, had he replied that he found details in the maps that were so far from being “right there,” that they were within the enemy’s lines, I ought to have pressed the question, “Am I right, then, in understanding that, if you were thoroughly and perfectly familiar with the country, as for example, if it lay just about the scenes of your childhood, no map of it would then be of the smallest use to you in laying out your detailed plans?” To that he could only have rejoined, “No, I do not say that, since I might probably desire the maps to stick pins into, so as to mark each anticipated day’s change in the situations of the two armies.” To that again, my sur-rejoinder should have been, “Well, General, that precisely corresponds to the advantages of a diagram of the course of a discussion. Indeed, just there, where you have so clearly pointed it out, lies the advantage of diagrams in general. Namely, if I may try to state the matter after you, one can make exact experiments upon uniform diagrams; and when one does so, one must keep a bright lookout for unintended and unexpected changes thereby brought about in the relations of different significant parts of the diagram to one another. Such operations upon diagrams, whether external or imaginary, take the place of the experiments upon real things that one performs in chemical and physical research. Chemists have ere now, I need not say, described experimentation as the putting of questions to Nature. Just so, experiments upon diagrams are questions put to the Nature of the relations concerned. (Peirce, C.P.: 4.530)

fMRI Brain Visuals as Fields for Interaction

43

In Peirce’s view, diagrammatic signs show what cannot be observed otherwise while consenting for engagement. This engagement has the potential for generating insights regarding the relationship between the represented elements. In Peirce’s story, the general uses the map to stick pins into so that he can observe the deployment of forces in a battle. Because the pins on the map are related to each other in the same way that the activities of the two armies are related to each other, the general can comprehend the relationships of activities that take place over the territory and thus adapt his actions to the anticipated changes. Peirce sees the experimentation on the map as an act analogous to the experimentation in the real world: Like chemists, who in their laboratories learn about the chemical reactions in the real word, the general, by observing the map and sticking pins into it, learns about the relationship between the activities of the real-world armies. fMRI visuals, as maps of the human cortex, are designed to depict relations among the processes that take place on the brain cortex. The diagrams as iconic signs, in addition to representing, allow for observation of the relationships between represented elements so that such relationships can be experimented with. fMRI researchers learn about the workings of human cognition by observing where on the brain cortex clusters of neurons with a specific function are located and what is their relationship to the brain processes situated in other areas. When engaged in the intricacies of laboratory work, cognitive neuroscientists, like the general who uses the pins on the map to understand the relationships between the two armies, use fMRI visuals to conceptualize the relation between the brain areas and their processes. The engagement with the diagram can be accomplished in the imagination but also, akin to the rearrangement of the pins on the general’s map, by direct involvement. Visual and Digital as Mutually Codependent The argument for the iconic character of fMRI brain scans has consequences for the understanding of the apparent dichotomy between the numerical and visual character of brain scans. When investigating social aspects of MRI, fMRI, and PET, researchers have been busy discussing this distinction (e.g., Beaulieu, 2002; Dumit, 2004, Joyce, 2005, 2008). Anne Beaulieu (2002), when interviewing neuroscientists, found out that they

44

Chapter 2

highlight the potential of brain imaging measurement to render spatial components and anatomic referents while, at the same time, they downplay the visual form this information takes to emphasize the quantitative information it represents. Beaulieu understands this negation of the importance of visual knowledge in brain mapping research as related to the way evidence is evaluated in modern Western science. She argues that because visual evidence has been regarded as appealing first to the senses, as opposed to reason, and hence is seen as lacking a solid relationship to the truth, visual evidence is judged as not having a particularly high position in the hierarchy of types of scientific evidence. The interviewees claim that those most interested in the visual aspects of brain mapping techniques are usually clinicians, not scientists, suggesting a hierarchy in which the visual is associated with the lower echelon of applied research. Kelly Joyce (2005, 2008), who studied the use of MRI in clinical settings, agrees with the claim of Beaulieu’s interviewees. In introducing the history of MRI, Joyce describes how Paul Lauterbur, an American chemist credited as the first person to use MRI to generate visuals of human anatomy, talked about those renderings in terms of maps, rather than images and pictures, defining them as a “mathematical representation of spatial information” ( Joyce, 2008: 32). Joyce, in contrast, points out that clinical practitioners prototypically talk about pictures of the human body, as their language reflects the saturation with the visual and visible that characterizes our contemporary life: Today, language that highlights the relation of the image to pictures of the anatomical body are often used in clinical practice, while language that calls attention to maps and spatiality is less common. . . . This linguistic difference occurs in part because of the broader recognition of the centrality of images to contemporary life as visualizing technologies such as cameras, computers, video games, and pictureproducing cell phones become more common. ( Joyce, 2008: 32)

Discussions of the tension between the visual and the numerical, and the decision to talk about “pictures” and “images” when referring to MRI, fMRI, and PET visuals, are important. On the one hand, they document how practitioners rationalize and talk about their work; on the other hand, they highlight the pervasiveness of the current focus on scientific texts, larger communities, and societal phenomena in social studies of science and technology. Yet, once we turn our gaze to the real-time practical work in neuroscience, and we adopt the understanding of brain visuals in terms

fMRI Brain Visuals as Fields for Interaction

45

of iconic signs, this dichotomy disappears. Rather than associating the visual character of fMRI visuals with transparency while also coupling their digitality with mediation, interpretation, and choice, the analysis of laboratory work shows visual and digital as mutually codependent. Because of their diagrammatic character, fMRI brain scans are at the same time visual and digital. fMRI Brain Visuals as a Field for Interaction The claim that scientific visuals, such as fMRI brain scans, are diagrams (rather than images) has consequences for the definition of their boundaries. The visibility of diagrammatic signs concerns the eyes as well as the hands. These signs, rather than being well-defined and self-standing representational objects, are fields for interaction—they acquire their meaning through work and interaction in the cognitive neuroscience laboratory.22 Because the digital character of the matter shapes laboratory practices, fMRI researchers deal with their experimental data in an engaged manner. First, the researchers work with fMRI visuals by placing their hands on the keyboards to generate observable effects in the displayed data. Also, they often use their hands to coordinate computer screens with maps, charts, atlases, and laboratory instruments. Finally, they gesture to highlight the features of the brain map or to enact what is still invisible in it. Similar to the general’s pins and the acts of placing them on the map, these gestures mark or further perform what needs to be seen on the brain scans. In this regard, the researchers’ engagement parallels the variety of semiotic forms involved in making the published brain figures meaningful. The marks on the paper and the fine orchestrations of semiotic bodies and technologies in the shared environment of practice are and evoke practical actions. The researchers (just like the model readers of their publications) understand fMRI visuals in terms of what was, can, and should be done with them.23 Practitioners’ gestures, their touching, and their modification of scientific visuals (directly enacted or evoked) are the constitutive element of these visuals. The centrality of an embodied engagement, however, concerns not only the digital but also the visual character of fMRI brain scans. When scientists

46

Chapter 2

work with fMRI scans, their seeing is accomplished through a coordination of eyes with the action of the hands, ears, and the workings of an array of fMRI technologies. Consequently, the viewing of fMRI visuals is about an active, distributed involvement where the contribution of each of its constituents is indispensable. In fact, the relationship with the digital screens, which suggests an understanding that in many respects is analogous to the physical engagement, would not be possible (at least not to this extent) if the data the scientists were dealing were not given in the visual format.24 This understanding of scientific visuals as fields for interaction is parallel to a variety of projects, ranging from art to architecture, where the authors continue their relationship with the objects of their creation beyond the moment after which, traditionally, those objects would have been considered self-standing. To comprehend the character of digital scientific visuals, social scientists have to proceed in a manner similar to artists who observe their work on a display or architects who stay informed on the specifics of a completed building. One example is the photographic opus of the artist JR (http://www.jr-art.net/). JR exhibits his portraits in public spaces, copying, magnifying, and fixing these portraits with wallpaper paste to the sides of buildings, to then photograph the exhibited portraits as they are lived. In such a way, the artist expands the borders of the work of art to include the interaction with and around it. Thus, those activities, traditionally considered to be located outside the external fringes of the photograph, are now a part of the work of art. Another example of this trend can be seen in the practices of the architects James Timberlake and Steven Kieran. As Timberlake pointed out in an interview with Deven Golden (2008), the two architects, in their effort to lower the energy footprint of their buildings, continue to monitor the performance of their projects even after the building has been completed. Using microprocessors to send relevant data from the building to their office, Timberlake and Kieran remain informed on the energy consumption, sustaining an ongoing rapport with the building and its life. We, like artists and architects who include the practical activities in the objects of their creation, need to take into account how scientists participate in generating the meaning of the visuals through their work, interaction, and the inscription of signs that evoke practical engagement. Although this shift from a representational object to a process of its enactment entails a turn toward the agent, it does not, however, imply

fMRI Brain Visuals as Fields for Interaction

47

either an argument for the psychological and individualistic analysis of meaning or a return to the Author. Instead, the attention to the details of multimodal interaction between scientists and brain visuals indicates a process of distribution and delegation. In the fMRI laboratory, the work of science is accomplished by bringing together human actors, technology, and the multiplicity of semiotic means. The details of this process bring forth an alternative idea of meaning-making where the individualistic mind is an effect of the ability to engage our experiential bodies in the lived world. What matters then are the efforts in documenting the coordination across multiple embodied and social agents, technology, and multimodal semiotic acts.

3 fMRI Brain Imaging and the Experience of Sound

In cognitive neuroscience laboratories fMRI visuals are engaged in a multimodal manner. This multimodal engagement regards the bodily conduct involved in working and interacting with digital screens. fMRI practitioners do not only inspect brain scans by passively posing their gazes on the surfaces of the visuals; they also modify the aspects of the visuals by working on computer keyboards, gesturing in front of digital displays, touching the visuals, listening to the sound of the scanning machine, and moving their semiotic bodies while coordinating with each other. Seeing in the laboratory—intended as recognizing meaningful patterns in experimental data—thus refers not only to the sight but also to the touch and hearing. This chapter opens the discussion of the multimodal character of fMRI in the laboratories of cognitive neuroscience by focusing on just one of its modalities: sound. According to Michel Foucault’s (1963) The Birth of the Clinic, the standard of modern medical rationality is what could be seen with the eye. The work of anatomist and physiologist Xavier Bichat (1771–1802) and his contemporaries, Foucault suggests, marks the shift between the 18thcentury study of brain pathology, characterized by its “language of fantasy,” and the discourse of modern medicine. In the history of science, these 18th-century anatomists are remembered for their technique of opening human skulls to observe the brain. The knowledge of the brain was from that point onward linked to perception: [M]edical rationality plunges into the marvelous density of perception, offering the grain of things as the first face of truth, with their colours, their spots, their hardness, their adherence. The breath of the experiment seems to be identified with the domain of the careful gaze, and of an empirical vigilance receptive only to the evidence of visible contents. The eye becomes the depository and source of clarity;

50

Chapter 3

it has the power to bring a truth to light that it receives only to the extent that it has brought it to light; as it opens, the eye first opens the truth: a flexion that marks the transition from the world of classical clarity—from the ‘enlightenment’—to the nineteenth century. (Foucault, 1963: xiii)

Today, we still inhabit this era of constant visibility where with changes in technology (fMRI being the primary example), “the artisanal skill of the brain-breaker” (Foucault, 1963: xiii) has been largely replaced with noninvasive techniques of observing the human brain. fMRI is presented as a technology that allows scientists to learn about the brain anatomy and its processes by looking at the visuals of the living brain displayed on computer screens. Because fMRI practitioners, rather than touching the brain, are centered on digital screens, one may expect the heightening of their visual experience at the expense of the embodied engagement. The eye would take the exclusive primacy in observing and reasoning about the brain, while the role of the hands, ears, and the interacting body would disappear from the scene. The details of work and interaction in the fMRI laboratory disprove this conjecture. Though the visual aspect of fMRI is critical, practitioners deal with the digital brain through a multimodal engagement. With this chapter, I begin to describe the actual moments of practice in fMRI laboratories as I combine interview data with observations from an instance of real-time laboratory work and learning. To generate a sense of the scene, I transcribe the interaction between two fMRI practitioners involved in a data analysis session that also serves as a learning process for one of the practitioners. Cyrus Mody and David Kaiser, in their effort to highlight the need for science studies to more decisively embrace the ideas of pedagogy (see also, e.g., Kaiser, 2005; Nersessian, Kurz-Milcke, Newstetter, & Davies, 2003), state: “By bringing training in focus, we can see that, even in ostensibly nonpedagogical settings, teaching and research activities are mutually reliant. The exigencies of one activity strongly inform the practice and content of the other” (Mody & Kaiser, 2008: 378). The pedagogical dimension of scientific practice (whose discussion continues through chapters 4 and 5) is a way to access the tacit knowledge of competent practitioners (e.g., Collins, 1974; Collins, de Vrjes, & Bijker, 1997; Polanyi, 1958, 1959) while it brings to light its multimodal dimension. The chapter highlights the nature of the experience that the fMRI practitioner goes through to acquire professional knowledge. To become an

fMRI Brain Imaging and the Experience of Sound

51

fMRI practitioner, the student of cognitive neuroscience engages in handson learning that has a distinctly acoustic character. I introduce this process with a brief explanation of how the fMRI scanner works to then describe what being scanned involves. This knowledge is not only important to the reader in understanding the process but also essential for the course of apprenticeship. I, thus, report on an experience of being in the fMRI scanner, which at the same time illustrates the initial stage of professional learning. The details of these activities show how sound, while signaling to the experimental subject what is going on during the scanning session, is used later to help practitioners interpret the experimental data. The ability to refer to the shared experience of sound is a resource for pinning down the definition of visual data in the acoustic experience. fMRI practitioners learn how to categorize brain scans through an encounter with fMRI technology that relies heavily on the acoustic modality. Bringing the acoustic character of fMRI to the fore puts into contrast the practical reasoning that takes place in the fMRI laboratory with how fMRI is used to study (and conceptualize) the human mind. An apprenticeship in the fMRI laboratory involves knowledge acquisition and problemsolving that concern the entire lived body immersed in the world of technologies and scientific communities. Learning by Scanning Your Brain The acquisition of professional knowledge in fMRI follows Peirce’s dictum that “Thinking in general terms is not enough. It is necessary that something should be DONE” (Peirce, C.P.: 4.233). Successful fMRI apprentices have to be versed in neuroscience and interested in issues of cognitive psychology; they also need to be able to operate the laboratory’s computer stations, use statistical packages for data analysis, and read brain atlases and online databases. Though an important part of this knowledge is assimilated through traditional classroom learning in the domains of neuroscience, psychology, biology, and computer science, a vital component of the professional knowledge needs to be acquired through a hands-on apprenticeship, where the learners enter in coordination with their colleagues and the technology. The newcomers to an fMRI laboratory spend long hours with knowledgeable old-timers1 (Lave & Wenger, 1991),

52

Chapter 3

observing how the old-timers engage with fMRI technology as they take part in the work activity. By tuning in with the old-timers and the material world of the laboratory, the newcomers negotiate their participation in the community of practice (Lave & Wenger, 1991) to acquire gradually the capacity to see, hear, and feel as fMRI researchers do. Newcomers often sit with senior colleagues in front of computers to participate in work sessions dedicated to the analysis of experimental data (see, e.g., figure 1.1). The primary goal of these sessions is to get work done, not to teach. Before the apprentices engage in such training, they often serve as fMRI experimental subjects. This kind of arrangement is especially frequent when the apprentice is a graduate student: It provides the laboratory with a reliable experimental subject while allowing the graduate student to learn about scientific practice through an embodied experience. Finding a “good” experimental subject is usually not a simple task. The process of recruiting subjects is made even more complex by the frequent need to repeatedly scan the same person’s brain across a series of experimental sessions. When graduate students start their fMRI apprenticeship, it is assumed that they (in contrast with undergraduate students) will remain in the community for an extensive period of time. This kind of permanence makes them experimental subjects on whom the rest of the laboratory can rely. While gaining trust through their participation in the laboratory enterprise, apprentices negotiate their coparticipation in the community of practice. The custom of initiating an fMRI apprenticeship with a scanning experience, however, has an additional advantage: by playing the role of experimental subjects, the apprentices learn about the scanning machines through first-hand experience. A significant part of fMRI knowledge can be acquired in the form of traditional classroom learning. The newcomers can consult literature on fMRI procedures to read about such things as the acoustic noise of the scanning machine and find out that, for example, the zone within which the magnetic force of the scanner may dangerously attract ferromagnetic objects is called the Gaussian field limits. Yet, when at the fMRI center, apprentices can also walk across the Gaussian lines inscribed on the floor and listen to the sound that the scanning machine generates. As they use their bodies to learn about the experimental data,

fMRI Brain Imaging and the Experience of Sound

53

fMRI apprentices not only train their eyes but also use other senses to acquire the tacit knowledge of an fMRI researcher. In the Scanner To illustrate the initial stages of fMRI apprenticeship I start with a description of a scanning session. A narration of an experimental subject— Sara—illustrates how it feels to be scanned. Sara’s personal account is prefaced by an introduction based in the ethnographic knowledge of the procedure that combines what I learned from observing the scanning procedures and the conversations that I had with practitioners, what I read in the manuals written for the use of the scientists who work at the fMRI center, and what I recall from my experience of being an fMRI experimental subject. Somewhat like fMRI apprentices, in the early stages of my ethnographic project I gained knowledge and negotiated my participation in the community of practice by playing the role of an experimental subject. The UCSD fMRI center has three scanning machines, two of which are specific for the scanning of the human body. The machines are located in separate rooms divided in two areas. In one area—the “magnet room”— there is the scanning machine (in the practitioners’ jargon, the “magnet”), and in the other area there is a computer console. When a human body is placed in a strong magnetic field, the protons in its hydrogen nuclei tend to align with the magnetic field. The static magnetic field of the scanning machine is provided by a large magnet with a cylindrical bore. Inside the bore of the main magnet is the gradient coil, which, during the scanning, pulses on and off to produce a linear gradient in the homogenous magnetic field so that the signals generated in different parts of the body can be distinguished. The experimental subject laying on the bed, which moves slowly into the magnet bore, needs to be placed so that her head is located at the midpoint of the gradient coil. To scan her brain, practitioners position her head inside another coil located adjacent to the head. During a scan, this coil (called the radiofrequency coil) excites the protons in hydrogen nuclei to pick up their signal. The experimental phase of the scanning procedure can range in time from 30 to 50 minutes, during which the experimental and control

54

Chapter 3

conditions alternate. When experimental stimuli are presented, the subject is usually required to use a button-press response box (or a joystick, in some cases) to signal her answers to the experimental task. When in the scanner, the subject is expected to behave in a well-prescribed way, so that the idiosyncrasies of her conduct can be filtered out from the data. To define how the human brain functions, the interference between the experimental data and the particularities of the subject’s behavior have to be reduced to a minimum. Before the practitioner helps place the experimental subject in the bore of the magnet, she provides the subject with instructions. The practitioner trains the subject on how to perform the experimental task while asking her to keep her head extremely still, as movements of the human body generate disturbance in the data.2 Typically, the practitioner applies a thermal plastic mask, vacuum pillows, and a bite-bar to help immobilize the subject’s head. Finally, she explains to the subject that she will exit the room, but remain in close touch through the intercom. Even though the subject will be left alone lying in the bore of the machine, her actions will be directed by the technologically mediated, yet careful guidance of the MRI practitioner. During the scanning, the practitioner operates the magnet through a computer-controlled interface. Located on the other side of the observation window, she may glance in at the subject and the machine periodically, while the computer on which she works displays the brain renderings as the scanning proceeds. The subject’s experience of being in the scanner, however, is more about sounds, movements, and temperature than about visual information. She feels cold, immobile, and overwhelmed with loud sounds, as she tries to perform well on the experimental task. Once inside the coil, the subject hears a slow, deep banging that lasts for a minute, then for 5 to 8 minutes a series of rapidly pulsed high and short signals. These signals are followed by a pause, and then a different tone and rhythm accompanies the experimental task. Sara, the experimental subject with whom I spoke, was involved in an unusual set of fMRI experiments. Despite the fact that fMRI technology relies on complete immobility of the person whose brain is being imaged, the researchers wanted to investigate the cognitive response to a reaching action. Not satisfied with the shallow definition of human embodiment (where the brain metonymically stands for the entire body), the researchers’ goal was to use fMRI technology to study the coordination

fMRI Brain Imaging and the Experience of Sound

55

between the thinking brain and the reaching hand. In other words, they tried to modify somewhat the prototypical scanning procedures while treating human cognition as an embodied process. In doing so, the experimenters had to deal with the resistances exhibited by the technology (which requires an imperative stillness of the subject during the scanning procedure). The tension between the technological resistances and the experimenters’ impetus to defeat these limitations is reflected in the narrated experience of the experimental subject. Sara’s story brings to the fore the complexities of the study of embodiment and embodied experience in the fMRI scanner. When asked about the experience of being in the scanner, Sara recalls: Well, so you are in there for so long (.)3 I’ve been scanned, I’ve been in two scanning experiments and they were both a few hours of being in the scanner and (.) being given a task and a variant on the task and another version of the task and (.), in both cases (.) they were sort of touch-related. Like, I was supposed to in one experiment (.) touch the top of like a Lego board on which some little bricks have been placed, and go like this ((performs a light touching with three fingers of her left hand))4 and feel if (.) the, you know, how many (.) little bricks there were. I cannot quite remember how I was supposed to, I think I was supposed to indicate it with my other hand, and I had a number of buttons that I can push to say how many bricks that I had touched ((performs a touching with her right hand as if carefully playing a piano)). And in the other experiment, uhm ((gestures in front of her face)), I was presented with some lights, and I was supposed to like reach toward the lights ((her head is pushed back onto the armchair as she performs a movement with her hand placed next to her upper body)). But, I guess, the whole thing was sort of set up like this ((performs three careful finger movements while her hand is placed very close to her upper body, as her body sinks even lower into the armchair)) and I was supposed to do this reaching movement, but I could not move my head?5 . . . But it seemed so (.) strange like being in there and being totally immobile. So, you know, you get in there, and they give you ear plugs, and they pack your head and your whole body with as much foam as possible, so you don’t move, and they put a blanket over you. (.) And then probably the most, I don’t know, kind of dehumanizing part of that is that they made a bite specially for my mouth, like crafted, you know, some kind of (.) synthetic plastic thing that I had to bite, and then it got hard, and then positioned right here ((gestures with two hands around her head)) so you know you cannot move your head. So it seemed just so strange, like I am supposed to not actually, you know, grasp the thing or not like really move my arm, but just give the slightest possible movement ((performs again the finger movement with her left hand just in front of her upper body)), as if to show that I am reaching for something. . . .

56

Chapter 3

So it feels totally artificial, and I am like, I know that I am thinking about of like trying to move very slowly because this is supposed to be very important so that I don’t move my head. All the data seemed (.) and like all the instructions seemed to be kind of subservient to this imperative to keep you head still because otherwise nothing is gonna be good about the data, I guess, and it’s not workable. And after the first time that I was in the scanner, the researcher told me that I kept my head very still, and I was so great so that’s why she invited me to do this again and gave me like forty dollars—because I was very good in keeping my head still ((laughs)), so the data was very usable for her. (.) But yeah it’s kind of cold in there and you are so restrained that it’s very hard not to get drowsy. Like I was saying at the beginning, it’s kind of an exhausting experience to try to stay mentally alert when you can’t move at all and you are not really supposed to think at all and you are just doing this repetitive boring, boring tasks, and occasionally have ah -. And, you know, the sound of the scanner ARRRRR ARRRRR ARRRRR when they are taking the structural before they actually start the experiment.

Differently from the canonical scanning procedures, during which experimental subjects observe a series of visual stimuli displayed on a screen in front of them, both of the experiments in which Sara participated asked the experimental subject to perform a hand movement. The first experiment required the subject to touch Lego bricks attached on a board, and the second experiment called for a reaching action toward flashing lights. As she explains the experimental procedure, Sara performs a series of gestures, moving her arm and hand to exemplify a very restrained hand action, enacted just in front of her chest and combined with an uncomfortable positioning of the head and neck. This positioning highlights the awkwardness of the hand movement that takes place while the rest of the body is kept immobile. To develop her description further, Sara mentions the instruments used by fMRI practitioners to minimize the movements of subjects’ bodies, highlighting the bite-bar, specifically designed for individual subjects and used to suppress unconscious movements of the head. During her participation in the experiments, Sara noticed that fMRI practitioners particularly value the ability of the subjects to remain immobile during the scanning procedure. As more thoroughly discussed in the chapter that follows, a subject’s immobility is considered to be an absolute precondition for collecting “good data.” In fact, Sara explains her involvement in the second experiment with her capacity to stay still during the previous one.

fMRI Brain Imaging and the Experience of Sound

57

As she recalls her experience, Sara highlights the contradiction between the requirements to stay extremely still while, at the same time, being asked to perform a movement. She also recalls that the two experiments felt long (lasting for a couple of hours), that she was cold, that it was hard to stay alert and not get drowsy, and that she felt exhausted. While providing her account of the experience, Sara mentions ear plugs and talks about the loud sound. During fMRI experiments, in addition to all movementrestraining devices, experimental subjects wear ear plugs (as well as headphones), used to isolate the subjects from the noise that the scanning machinery produces. To further her narration of the discomfort, Sara evokes the deafening sound of the scanner: “ARRRRR ARRRRR ARRRRR.” Her repetitious enactment depicts a monotonous and intense acoustic experience. The sound that the subject experiences is due to the vibration produced by the interaction between the static magnetic field and the timedependent currents in gradient wires. Scientists are well aware that this high-decibel acoustic noise can impede effective research. In addition to potentially generating discomfort while hampering subjects’ attention, the sound of the scanner may also interfere with experimental stimuli, particularly if such stimuli are presented acoustically (as is often the case in research on language comprehension). Scientists, thus, pay particular attention to mask “background noise,” providing their subjects with ear plugs and headphones. However, these acoustic and experientially overwhelming features of the scanning process (recognized by scientists for their negative effects) can function, if only for brief moments, as tools in laboratory work and learning. Sound as a Pedagogical Component of fMRI Practice As mentioned by Sara, an fMRI scanning is accompanied by a high-decibel sound. Such a sound may be a severe impediment for successful research. Yet, despite difficulties related to the sound, the following excerpt from an fMRI apprenticeship illustrates how a reenactment of the machine’s sound can play an important role in the achievement of specialized knowledge. Somewhat in contrast with the common association of fMRI technique with image production, the reenacted voice of the machine participates in generating understanding of fMRI experimental data.

58

Chapter 3

The experience of being in the scanner, similar to the one described by Sara, frequently functions as an early step in the fMRI apprenticeship. In laboratories that I studied, the new fMRI practitioners go through the experience of being scanned before they start to learn how to analyze fMRI data. During these scanning sessions, senior colleagues collect the experimental data as they instruct the newcomers on what constitutes the fMRI scanning procedure. After the scanning is completed and the data have been sent to the laboratory computers, the practitioners need to carry on data analysis. The process of data analysis is long and complex, and its organization varies from laboratory to laboratory.6 In what follows, I focus on an interaction between two practitioners just starting to work on their experimental data in the laboratory. The interaction takes place between a new member of the laboratory, Nick (N), who is a first-year graduate student, and an oldtimer, Oliver (O), who is an advanced graduate student. Oliver is known in the campus community as a promising graduate student. He introduces Nick to laboratory practices not so much as a fellow graduate student but as a knowledgeable and welcoming laboratory member. Nick, on the other hand, is eager to show that he is already competent in the principles of cognitive neuroscience while being a quick learner. As we join the scene, the two practitioners are seated in front of a computer, and the only other person in the room is the ethnographer who videotapes the interaction (seated behind the two practitioners). Prior to this encounter, Nick has acquired his first laboratory experience by playing the role of an experimental subject. During the scanning session, Oliver and the principal investigator of the laboratory, Peter (P), were present. As they collected the experimental data, the two practitioners also instructed Nick on the particularities of the procedure. Once in the laboratory, Oliver and Nick continue with the apprenticeship by engaging in the analysis of data acquired during the scanning session. However, before they even start to deal with the data, Oliver explains to Nick how the scans are organized in directories. Notably, the explanation involves the acoustic experience. During the interaction, Nick and Oliver refer to three types of fMRI scans: localizer, structural, and functional. What practitioners usually call structurals (or also referred to with the expression MP-rages) are highresolution anatomic scans of the entire brain structure. Rapid dynamic

fMRI Brain Imaging and the Experience of Sound

59

imaging, or functionals, on the other hand, are the scans taken as subjects are presented with experimental stimuli.7 These scans are collected at a high rate to detect the blood volume changes in the brain (which are assumed to be the sign of cognitive activity).8 But before these highresolution scans are collected, just when the experimental subject is placed in the magnet and aligned so that his or her head is at the center of the magnet, the members of the laboratory collect a so-called localizer scan—a single acquisition of one series of slices at the orientation and position that will be used to consequently collect the fMRI data. The scan gives the location of the brain as the main cortical landmarks are identified to function as a reference for the rest of the scans. As mentioned in the introductory chapter, the excerpts from the interaction are transcribed according to the transcription style of conversation analysis (Sacks, 1995). To indicate the intricate ways in which interlocutors coordinate with each other, the transcription adopts the following conventions ( Jefferson, 2004; Sacks, Schegloff, & Jefferson, 1974): =

Equals sign indicates no interval between the end of a prior and the

start of a new piece of talk. (0.0)

Numbers in parentheses indicate elapsed time in tenths of

seconds. (.)

A dot in parentheses indicates a brief interval within or between

utterances. ()

Parentheses indicate that the transcriber is not sure about the words

contained therein. (( )) Double parentheses contain the transcriber’s descriptions. °°° Degree signs are used to indicate that the talk they encompass is spoken noticeably quieter than the surrounding talk. // The double oblique indicates the point at which a current speaker’s talk is overlapped by the talk of another. : The colon indicates that the prior syllable is prolonged. ___

Underscoring indicates stressing.

Punctuation markers are used to indicate “the usual” intonation: .

Dot is used for falling intonation.

?

Question mark is used for rising intonation.

,

Comma is used for rising and falling intonation.

60

Chapter 3

EXCERPT 1 1

O:

2

N:

3

O:

4

Okay there are three studies (0.2) tha:t (0.1) were in this directory.//There was, //Three? Yeah. (There) = cause there is localizer. Do you remember at the very //beginning

5

N:

6

O:

7

N:

8

O:

//At the beginning of functionals? No at the beginning of the //MP-rages //Oh okay You remember you’ve heard sort of clicking ch-ch-ch ((imitates the sound))

9

and then you got just a quick ah ba:uwm, ba:uwm, ba:uwm, ((imitates the sound))

10

and then it was quiet

11

and Peter said this was the warm-up and here is the real one?

12

That’s the localizer

In the beginning of the excerpt, the old-timer—Oliver—directs Nick’s attention to how the collected scans are organized in a computer directory. Oliver emphasizes that there are three studies (line 1). By pointing out this organization, he highlights how laboratory members differentiate fMRI scans. Oliver, thus, not only teaches Nick how to understand the organization of the data but also how to categorize the fMRI visuals. As soon as Oliver says that there are three parts of the study (what he calls “three studies”), Nick signals his trouble in comprehension: “Three?” (line 2). By querying the number of the studies in the directory, Nick demonstrates uncertainty about the meaning of Oliver’s utterance. To align the knowledge between the two practitioners, the old-timer affirms and adds “Yeah. (There)=cause there is localizer” (line 3). Oliver’s assumption that Nick knows that the other two scans are the structural and functional reflects the distinction between the general knowledge that can be acquired through traditional learning and knowledge that regards practices in the laboratory. Because the use of localizer scans varies from laboratory to

fMRI Brain Imaging and the Experience of Sound

61

laboratory, its appearance in the data set, indicating local procedures and routines of the laboratory members, needs to be explained during the apprenticeship. Oliver will, in fact, explain that the localizer scan used to be much more important in the past (until the principal investigator— Peter—figured out the way to retrieve the same information from other scans so that the procedure could be automated) and at the current moment is primarily considered to be the back-up information. By listening to Oliver’s account, Nick acquires cultural knowledge of the laboratory, as knowing about the localizer scan pertains to what it means to be a laboratory member. This knowledge, nevertheless, is embodied and experiential. In line 4, Oliver further specifies the meaning of localizer scan by invoking Nick’s experience in the scanner. The senior practitioner asks his colleague: “Do you remember at the very beginning” (line 4). The utterance opens the narration of the experience, marking its temporal sequence: “at the very beginning” (line 4), “and then” (line 9), “and then” (line 10). The narration is immediately joined by Nick who asks: “at the beginning of functionals?” Nick’s intervention is corrected by Oliver: “No at the beginning of the MP-rages” (line 6). In response to his colleague, who this time applies the term functionals, Oliver uses another specialized term, MP-rages (which stands for “magnetization-prepared rapid gradient echo”), and denotes a protocol used to collect structural scans. Nick affirms (line 7), demonstrating that he can speak like an fMRI practitioner and the laboratory member. Importantly, as the two engage in defining what a localizer scan means (lines 8–9), their interaction indicates how the participation in the community of practice involves shared embodied sensibilities that may concern the nonlinguistic sounds and silences experienced during the scanning session. The definition of the localizer scan is not simply provided by linguistic labels but also enacted through a performance of the scanner’s sound. In lines 8 through 9, Oliver continues the narration by articulating the phenomenological features of the imaging procedure as he imitates the sound of the scanner. By reenacting the acoustic noise that accompanies the scanning process, the practitioner’s voice is aimed at arousing the recall of the experience. Similar to Sara, who imitates the sound of the machine to depict the experience of being in the scanner, Oliver performs the sound to instruct the newcomer how to understand the term localizer. Whereas

62

Chapter 3

Sara’s enactment of the noise generates a more general sense of a loud and uncomfortable noise—“ARRRRR ARRRRR ARRRRR”—not directed at articulating one specific kind of sound, the senior practitioner performs the sound very precisely, marking its distinct phases: It starts with “sort of clicking ch-ch-ch” (line 8), “and then you got just a quick ah ba:uwm, ba:uwm, ba:uwm”(line 9). The enacted definition of the fMRI scan, directed at differentiating one kind of sound from the other, goes beyond propositional knowledge: What Nick needs to learn is not said, but shown. Engrossed in the multimodal act, Nick learns to categorize brain scans while developing the sensibility of the fMRI researcher. He learns to listen to the machine and hear its sound as a researcher rather than as an experimental subject. This sensibility, in part, concerns the machine’s silences. As soon as the enactment of the scanner’s sound ceases, Oliver proceeds with his narration by saying “and then it was quiet” (line 10). Although it is easy to mistake the sound sequences as evidence of the data collection, the attainment of the MRI signal is not directly dependent on sound. It is, in fact, the quiet time after the pulsing of sound during which the protons decay that significant data are registered for measurement. By talking about the absence of sound, the old-timer marks this silence as meaningful. As the apprenticeship proceeds, the coparticipants find themselves immersed in a multiplicity of spaces and voices. In the following line (line 11), Oliver builds the recall by uttering: “and Peter said this was the warm-up and here is the real one?” Peter is the laboratory principal investigator who, as we learn from Oliver’s account, was present during the scanning session, providing a “voiceover” while Nick was being scanned. Oliver’s enactment of the scanning experience, thus, nests the scanning space into the laboratory space: What happened during the scanning session is recalled to play a part of the pedagogical effort in the laboratory. These nested and multipurpose spaces (dedicated to experimentation and pedagogy) are, furthermore, populated by a multiplicity of voices— the human and the machine—mixing together the direct with reported speech. By listening with Nick to Oliver’s explanation of what the localizer scan is, we hear the voice of the machine and the voice of the laboratory director. The director’s voice travels through the intercom to the bore of the scanner as it indicates how Nick should read the sound or its absence:

fMRI Brain Imaging and the Experience of Sound

63

There are moments of silence after the bursts of sound that have to be understood as the end of the “warm up” and the beginning of the “real one.” The director’s voice and the voice of the machine—its noise and its quiet times—are coordinated. The two voices participate in Nick’s learning as he refines the meaning of the professional categories in the context of their use. While being scanned, Nick is instructed to behave like an experimental subject (as happened in Sara’s case) as well as an fMRI practitioner. As he tries to be still to allow his colleagues to collect “good” data, what may have been a cacophony of sounds to Sara becomes a sequence of well-defined and informative tones to Nick. As the apprenticeship initiated at the fMRI center continues in the laboratory, Oliver performs the sound of the machine and voices the principal investigator. His voice, configured by its passage through the scientific culture, bears marks of many voices. Oliver’s narration, however, is not only a transposition of the voices from the scanner to the new space but also an active performance of the learning environment inhabited by a multiplicity of sounds. The enactment of those sounds—generated by humans and machines—provides the opportunity for further refinement of what localizer, MP-rages, functionals, and other professional categories mean. Through the newcomer’s tuning in with his colleagues and the fMRI technology, he learns to see by being trained how to hear and listen as an fMRI practitioner. Learning to Hear Like an fMRI Researcher Whereas understanding scientific data is importantly a matter of social consensus as newcomers need to learn how the data should be comprehended, the objects of knowledge that laboratory activities produce are built through mundane processes of work and embodied interaction. The local management of such engagements invokes previous dealings and cumulative practical know-how, receiving its rhetorical force by reference to the “usual” procedures of laboratory members. Although these procedures are nested in larger, historically evolving, sociotechnical networks, their specifications are bound to the locally instantiated assemblages of instruments, embodied techniques, and everyday discourses in the laboratory (see, e.g., Lynch, 1993). In the fMRI laboratory, such processes concern the corporeal engagement with digital screens. The interaction between

64

Chapter 3

Nick and Oliver indicates how, as the practitioners acquire the knowledge of the community by coparticipating in activities with their colleagues, they engage their bodies in the process. In this respect, the social as well as material and phenomenological worlds are intrinsic elements of conceptualization and inference production in the fMRI laboratory. To capture the meaning of localizer scan, the old-timer neither provides a detailed linguistic and technical explanation of what such a scan is, nor does he point to a representation that could stand for it. Instead, the practitioner generates the definition in terms of a story that re-creates the experience of being in the scanner. By voicing the director and reenacting the sounds that the scanning apparatus generates, Oliver organizes the messy world into categories relevant for the work of the profession (Goodwin, 1994). In other words, to enact objects of knowledge and rationality in the fMRI laboratory, the experienced practitioner articulates the category of fMRI visuals in terms of an acoustic event. When in line 12 he concludes, “That’s the localizer,” a salient trait in defining the localizer scan is the sound of the fMRI machine. What was an undesired aspect of the scanning session is, thus, reintroduced into data analysis to assume an explanatory function: The confounding factor becomes a significant element in the embodied procedure of sense-making. fMRI has been associated with imaging for most of the technique’s history. Sound, however, is a phenomenologically pervasive, one might even say intrusive, part of this modality, emphatically present in the production of fMRI data. Studies of MRI have emphasized the interpretation of recorded visual and numerical data, with little discussion of the acoustic experience of the fMRI data production as experienced by the practitioners and the experimental subjects. This approach parallels the general trend where much effort has been devoted to research on the role and use of visual images in scientific practice while relegating sound to a peripheral, if not largely forgotten, factor (Cartwright, 2008). Yet, sound is an aspect that defines the subjects’ experience with the technology, often unexpected before their arrival to the scanning center. The description of how sound functions as a pedagogical agent in the laboratory calls attention to sound as a significant quality of experience for practitioners who work with fMRI visuals. To display his competency, the newcomer has to coordinate not only with the voice of his colleague but also with the voice of the machine. He is taught how to talk the

fMRI Brain Imaging and the Experience of Sound

65

language of the community of practice by acquiring specialized terminology, but also by learning how to hear as an fMRI practitioner. In the current account of fMRI visuals, this focus on the sound is the first step in indicating how scientists understand and deal with brain imaging data in an embodied manner that, while distributed across people and technologies, involves not only visual but also acoustic modality. In this sense, the complexities of the laboratory knowledge are not only about Foucault’s “careful gaze, and of an empirical vigilance receptive only to the evidence of visible contents” (Foucault, 1973: xiii). Whereas fMRI technology, with its possibilities and constraints, reduces the imaged body to an immobile eye, in the laboratory “the marvelous density of perception” (Foucault, 1973: xiii) regards coordination across people and technology and an embodied experience that calls for the skilled ear.

4 fMRI Brain Visuals and Semiotic Bodies

To continue to unpack the claim that digital scientific visuals are fields for interaction, this chapter focuses on semiotic bodies. The discussion is grounded in two excerpts from an interaction where the fMRI practitioners are involved in reading brain visuals as a part of work and apprenticeship activity. Whereas the previous chapter described sound, this chapter focuses on bodily conduct. It indicates how practitioners engage their gesturing hands, nodding heads, moving necks, hunching shoulders, and bending torsos to make sense of their experimental data. With respect to earlier approaches in the scientific study of the human mind, fMRI is considered to be an “embodied” method.1 Yet, the body captured by fMRI is confined to one of its organs: the brain. Ironically, to investigate the mind in an embodied way, fMRI technology and its accompanying theories keep the rest of the body out of the picture. As indicated by the account of the experimental subject Sara (reported in the previous chapter), to achieve “usable” fMRI data, cognitive neuroscientists have to restrain the subject’s body. At the same time, to analyze the experimental data, they engage their sensing and interacting bodies. This chapter describes how an apprentice acquires a “sense of the data” by being engaged interactionally and taught to feel her own body involved in data analysis. The description, by virtue of the contrast, suggests that the conception of embodiment, assumed and further propagated through the use of fMRI technology, needs to be recast. The body in the cognitive neuroscience laboratory is social and multimodal. By turning attention to the practitioners’ bodies, the chapter also talks about the body of the experimental subject—the imaged body. It points out that the imaged body is neither fully represented by the brain visuals (the practitioners do not look at the computer display as one would look

68

Chapter 4

through a window) nor is it constructed (in opposition to its biological underpinnings). Instead, it is an enacted body. Though the imaged body constrains the reading of the visuals, it is brought into the laboratory space through the scientist’s semiotic actions coordinated with the digital screens and inscribed with cultural meanings. I look at how these acts of coordination engage with the individual of the scanned body in the laboratory. As I venture into a discussion of the relationship between fMRI visuals, scientists, and the imaged body, I foreground the tension between what the use of fMRI shows and how such knowledge is generated. With this I do not mean to downplay the important shift toward the body that fMRI applications exemplify. I do, however, want to complicate the conception of the body and embodiment to challenge the assumptions about the body that fMRI technology implies and enforces. By equating the body with one of its organs—the brain—fMRI presents the thinking body as a unified, biological, and individual thing. On the contrary, the interactional character of digital visuals suggests that the body in the laboratory needs to be understood as a relational achievement. The object of scientific practice is a dynamic phenomenon that simultaneously exists in and dynamically coordinates multiple spaces, while not being reducible to any. Movement Artifact and fMRI Practice This chapter discusses two excerpts from a laboratory apprenticeship dedicated to the detection of research artifacts in experimental data. According to Michael Lynch (1985a), research artifacts are considered to be undesirable elements of experimental data that emerge due to the way the data are obtained rather than due to the world scientists aim to study: The term, artifact, has a specific usage in laboratory research which needs to be distinguished from a more general use of the term in other areas of discourse. Research artifacts comprise an indefinite variety of substantive and methodological features which appear in laboratory accounts of naturalistic phenomena, especially when the phenomena depend on specialized techniques or instruments for their observability. Artifacts are described in such accounts as particular “intrusions” or “distortions,” in the observability of the “natural” features of the world which derive from the instrumental conditions of their perception. The danger that artifacts are

fMRI Brain Visuals and Semiotic Bodies

69

said to present for research is that their presence can go undiscovered and be taken as evidentiary features of purportedly natural phenomena. . . . The possibility of artifacts is an almost inevitable accompaniment of research which relies upon specialized techniques and machinery for making initially “invisible” theoretic entities visible in documentary formats. (Lynch, 1985a: 81–82)

Therefore, the elimination and purification of data from artifacts is one of the central aspects of laboratory practice (see also, e.g., Latour & Woolgar, 1979). The current discussion centers on what fMRI practitioners call a movement artifact. As pointed out by Sara’s account (chapter 3), fMRI practitioners consider the movement of the experimental subject to be highly disruptive in obtaining “good data.” Upon being asked what is crucial for collecting good experimental data, one of the senior researchers replied “a good subject”; that is, “one who doesn’t move during the scanning session.” This statement, while stressing the harmful character of the movements, also attributes the central role to the experimental subject. In other words, fMRI researchers consider the artifact associated with the subject’s movement to be one of the greatest obstacles to successful data collection and analysis. Practitioners, thus, carefully search for signs of movement in their data so that they can be removed. If the movement artifacts are pervasive across a data set, the practitioners may decide that the entire set is not “usable.” The status of the movement artifact in fMRI indicates that movement, as a feature of the living body, is considered to be an impediment that disrupts the visibility of the “natural” characteristics of the data. The moving body must become invisible (along with the rest of the instruments and laboratory work), so that the “embodied” mind (i.e., the brain) can be studied. In this sense, traces of movement are conceptualized as related to “the instrumental conditions which organize the perception” rather than being seen as features of the examined body. It is, then, somewhat paradoxical that whereas the body-in-movement of the experimental subject needs to be erased from the data, the scientists use their own bodies-in-movement to read fMRI visuals. The employment of scientists’ bodies in the process indicates how fMRI visuals function in the laboratory and how scientists conceptualize in practice the body that they image.

70

Chapter 4

Moving in the Imaging Spaces As indicated by the fMRI center’s safety manual, at the scanning facility movement is prescribed. The primary hazard associated with MRI scanning is metal objects, which can become projectiles if the magnet’s strong force attracts them. For this reason, in addition to walls that separate the space where the magnet is located, on the floor of the scanning rooms red lines mark the high magnetic field, or “5 gauss line.” The red lines (also mapped out on charts hanging on the walls) indicate how movements across the space should be structured: Once the red lines are crossed, the occupants of the space need to be free of metallic objects (for this reason, people with pacemakers, metallic implants, neurostimulators, etc., are excluded from fMRI studies). What is more, the occupants have to minimize the rapid movements of their heads to avoid dizziness, vertigo, nausea, or a metallic taste that the magnetic field may cause. In addition to being free of ferromagnetic objects and avoiding rapid movements when in the vicinity of the magnet, the experimental subjects are also asked not to move during the scanning session. Because even small movements of the tongue or jaw can cause artifacts in the data by altering the local magnetic field around scanned areas, fMRI practitioners try to control the head movements of their subjects by using a variety of headrestraint systems, such as thermal plastic masks, vacuum pillows, and bite-bars. Once the scanning starts, the movements of the person being scanned are usually too small to be detected on the computer monitor (where practitioners supervise what happens inside the scanner). These invisible movements, however, frequently reappear in the laboratory where fMRI researchers work with the brain scans. When traces of the subject’s movements are detected, they have to be removed from the data. To perform this operation of removal, practitioners actively engage their bodies. The locales where fMRI research is performed—the scanning facilities and the laboratory—are places where multiple semiotic systems interact: Signs on the floor indicate the presence of a magnetic field, and visuals are intended to denote the human brain and its processes. These signs make visible what is otherwise invisible by inscribing it in the stable semiotic structures: The attractive force of the magnet is represented by the red lines on the floor, and the brain processes are shown through colors and shapes

fMRI Brain Visuals and Semiotic Bodies

71

inscribed on the visual renderings of the brain’s anatomy. Yet, there are also signs that are only materialized in the ephemeral movements of the bodies that interact with brain visuals. This chapter is an invitation to focus on the semiotic movements of the practitioners’ bodies that make the fMRI brain visuals visible. Charles Goodwin (2000a) has pointed out that despite the important treatment that scientific visuals have had in the domain of social studies of science, not much attention has been paid to how scientists organize what they see through their multimodal interaction, and how they, when engaging visuals, attend to each other’s semiotic bodies. Research in science studies has investigated the images produced by scientists, and the way in which they visually and mathematically structure the world that is the focus of their inquiry, without however looking in much detail at how scientists attend to each other as living, meaningful bodies, or structure what they are seeing through the organization of talk-in-interaction. (Goodwin, 2000a: 164)

By describing the details of “motion correction” activities, I analyze how scientists involve and treat each other’s visible bodies to make the features of fMRI visuals understandable. The analysis illustrates how the studied body emerges through multimodal interaction in the laboratory not as a unified and stable object (as the discourses of essentialism, naturalism, and biologism, with their universalists assertions, suggest), but as a phenomenon that concerns an intertwining of lived and digital bodies, instruments, spaces, moments in time, and historically shaped cultural discourses. Correcting for Motion by Enacting Motion Like the activity reported in the preceding chapter, the apprenticeship takes place as a part of an analysis session where practitioners work with previously collected data. However, the apprenticeship activity described in this chapter is situated in a laboratory that studies patients with brain damage, which means that the experimental subject is a clinical patient (rather than the practitioner herself). Over the course of the experiment, a series of brain scans are recorded, where each scan stands for a slice of the brain. During the preparation of the experimental data for statistical examination, researchers assess whether the brain slice scans can be aligned. The assessment is achieved by viewing

72

Chapter 4

Figure 4.1 The computer screen displays brain slices over the time-course of the experiment.

slices (in the axial, sagittal, and coronal views2) shown on the computer screen over the time course of the experiment (figure 4.1). The researchers use mouse commands to alternate the views of individual scans on the screen to check if the visuals of the brain slices align with each other. When they detect nonalignment, the researchers try to correct it to eliminate visible defects in the visuals (such as blurry splotches). This activity of “cleaning the data” from motion artifacts is a routine procedure in the laboratory. To explain the nonalignment between the brain visuals, practitioners try to figure out what movement caused it. Though they understand the subject’s brain and its function as natural phenomena to be examined, they consider the potentially unintentional movements of the subject’s body to be the cause of an intrusion in the visibility of fMRI scans. To remove these distortions from the data set, they seek to understand the character of the aberrant movements that the experimental subject performed inside the magnet. To do so, the practitioners can only infer the

fMRI Brain Visuals and Semiotic Bodies

73

movement they believe caused the nonalignment. One reason for this is that they did not actually see the movements of the experimental subject while she was lying in the scanner. Furthermore, the experimental subject of this particular scanning session was not one of the practitioners who could know the movement through a direct experience. To counter this insufficiency, the practitioners resort to three resources: the series of fMRI brain visuals (rather than any single visual), multimodal semiotic means, and what is known in the laboratory. Their collaborative data analysis shows how these resources are dynamically coordinated so that they can see the movement of the subject in the brain visuals. As the practitioners identify the movement in the misaligned consecutive visuals, they attempt to understand it by publicly enacting it and feeling its effect in their own bodies. The movement is not simply seen during the experiment or in the visuals but is enacted through a coordination of bodies and technology inscribed with cultural knowledge of the laboratory. Learning to See the Subject’s Movement Excerpt 1 reports on 20 seconds of interaction between two fMRI practitioners: a graduate student who is an old-timer in the laboratory (Olga) and an undergraduate student who is new to the laboratory (Nina). Olga is an easy-going but ambitious and hard-working Ph.D. candidate in psychology who has worked in the laboratory for several years. Because of her familiarity with the laboratory procedures, Olga has been asked by the director of the laboratory to guide the newcomer in acquiring laboratory skills so that she can get involved with the research projects that are currently in course. Nina, who is majoring in cognitive science, just started her internship in the laboratory with the goal of gaining practical knowledge in experimental methods. During the observed interaction, Olga is introducing Nina to the data analysis procedure. In figure 4.2, the person on the left is Olga, and Nina is on the right. As Olga guides Nina through an actual data analysis process, the teaching session is also an instance of work practice. The practitioners are seated in front of a computer in one of the laboratory’s rooms where they, engaged in the work, do not seem to pay much attention to the other events that simultaneously take place in the room (laboratory members entering and exiting the room and the ethnographer videotaping the scene). The

74

Chapter 4

atmosphere between the two practitioners is friendly but strictly workoriented. Olga is patient and clear in her explanations, and Nina—softspoken and sometimes shy—shows her willingness to learn. The excerpts from the interaction come from an early stage of analysis where the experimental data are being prepared for statistical examination. Each rendering displayed on the computer screen stands for a brain slice at a particular moment of the experiment. Nina’s immediate task is to master the motion correction procedure by checking for alignment between each visual in the series. The process of motion correction, where practitioners identify the visuals that “show movement,” is done by handling the digital data. Nina, the newcomer, controls the computer, using the mouse to engage the computer display. While she does so, her actions are closely coordinated with the old-timer’s. In excerpt 1, Olga notices a misalignment among visuals and directs the newcomer’s attention toward the computer screen. Olga then performs a gestural reenactment of the noticed misalignment, to which Nina responds, quickly picking up the technique. After searching through the data, Nina also spots a misaligned visual. She points toward the screen and then gestures by employing her hands, upper body, shoulders, and neck. EXCERPT 1 1

O:

That’s definite I can see her in this plane= ((Points to the computer screen))

Figure 4.2

fMRI Brain Visuals and Semiotic Bodies

2

75

going from here to here. ((Sweeps her arm in a downward motion and halts at a certain point))

Figure 4.3 3

Figure 4.4

Aaaa ((Disapproves of what she sees))

4

N:

(Aaaa, this is a good one) ((Points to the sagittal view of the brain slice on the screen))

5

O:

Yeah. Slice, edit ((Provides instructions for pressing buttons while manipulating computer screen))

6

N:

(Oh, here we are)

7

O:

Is she moving any more in thirty? ((Moves closer to the screen))

8

N:

She’s moving= ((Hunches))

Figure 4.5

Figure 4.6

76

9

Chapter 4

Like this one ((Points to the axial view of the brain slice on the screen))

Figure 4.7

10

Is going down. Hhhhh ((Laughs)) ((Swings with right hand (palm down) downward))

Figure 4.8

fMRI Brain Visuals and Semiotic Bodies

77

Gesturing as Pedagogical Tool The pervasive use of the embodied semiotic means and the involvement of the scientists’ bodies in the data analysis characterize the pedagogy in the fMRI laboratory. In excerpt 1, the old-timer engages in the training procedure by gesturing extensively. In line 1, as she notices that there is a misalignment between two consecutive brain visuals, Olga extends her left hand toward the screen (figure 4.2) to draw Nina’s attention to the misaligned visuals while confidently saying: “That’s definite I can see her in this plane.” Thereafter (line 2), Olga sweeps her gesturing arm in a downward motion (figure 4.3) and halts at a certain point (figure 4.4) while briefly looking at Nina to check on her comprehension. Coordinated with the digital data, Olga’s gesture is a performance of the experimental subject’s movement. The role of Olga’s gesture is twofold: The gesture not only allows Olga to position herself in the activity as an experienced member of the laboratory, but it also provides scaffolding (Vygotsky, 1978) for the newcomer’s examination of the brain visuals. Similar to the semiotic infrastructures that complement the published fMRI figure (discussed in chapter 2), Olga’s gesture does not simply reproduce what is already present on the computer screen, but, like the figure caption and graphical signs layered over the published brain visuals, the gesture organizes seeing. The horizontal rather than vertical position of the gesturing hand suggests that the semiotic act, rather than merely standing for the brain scans, also enacts the head of the experimental subject moving in the scanner. Therefore, the gesture edits the brain scans, indicating what the apprentice is expected to learn to see. This generates immediacy. Nina learns not only by looking at the digital visuals but also by coordinating them with the bodily conduct of her colleague, and understanding the visuals in terms of a familiar scene: a human head that moves (as Olga’s gestural production of a three-dimensional horizontal movement is similar to the head moving in the scanner). Rather than just imagining an abstract movement, the two practitioners see now the subject’s movement in the laboratory. Thus, Olga’s pedagogy concerns the visuals that function as Peirce’s iconic signs. She employs the visuals to organize the apprenticeship as an activity where the practitioners can attend to the involvement of each other’s lived and meaningful bodies,

78

Chapter 4

and, thus, understand their experimental data in terms of the everyday world. Learning by Multimodal Embodied Experience The excerpt also shows how the newcomer learns to see by engaging her own body in the process of artifact correction. Moving her body with respect to what the old-timer points out and what she observes on the computer screen is an experiential and semiotic act. It allows the newcomer to feel the experimental data in her own body while displaying her mastery of the procedure. Nina uses mouse commands to manipulate the digital visuals as she coordinates them with her embodied semiotic engagement. Her active involvement starts to be visible in line 4. After Olga’s gestures (performed in lines 1 and 2), the two practitioners, while carefully observing the changes on the computer display, comment about what they see (lines 3–6). In line 3, the old-timer’s “Aaaa” turns their attention toward the brain scan, followed, in line 4, by the newcomer’s similar vocalization combined with the utterance, “This is a good one.” Whereas the action provides the old-timer with a chance to monitor the apprentice’s understanding (displayed through her gestures and talk), it allows Nina to indicate her ability to identify the misalignment. In fact, as the interaction proceeds, Nina reports on another find: “Oh, here we are” (line 6). At this point (line 7), the old-timer moves toward the computer screen as if she is not able to see what her colleague is indicating, and asks: “Is she moving any more in thirty?”3 In response, Nina takes the floor. She illustrates her full command of what she sees by performing the subject’s movement in terms of a complex action that coordinates the brain visual with a series of embodied semiotic enactments (lines 8–10). In line 8, while her hands are engaged with the keyboard, Nina hunches (figure 4.6). In line 9, she points toward the computer screen (figure 4.7). Finally, in line 10, she executes a hand gesture where she swings with the right hand downward (figure 4.8). In line 8, after saying “She’s moving,” Nina abruptly bends her shoulders and neck toward the screen (figures 4.5 and 4.6). The hunching movement is accompanied by a chuckle, as she displays her negative judgment toward what she points. By physically enacting the process of change attributed to the brain visuals, Nina employs her torso to create the

fMRI Brain Visuals and Semiotic Bodies

79

hypothesized movement of the experimental subject. As Nina performs the hunch, Olga turns her gaze from the computer screen toward her (figure 4.6), treating the hunch as a response to the question: “Is she moving any more in thirty?” Through the hunching gesture, Nina not only responds to her colleague but also learns about the trouble in the experimental data. In addition to understanding the nonalignment by comparing the visual features on the computer screen (or by solely observing the old-timer’s action), the newcomer learns through an embodied experience. The hunching movement generates first-person understanding for its performer, allowing the apprentice to experience a movement similar to the one that the experimental subject may have experienced in the scanner. Analogous to Nick’s recall of the scanner’s sound (discussed in chapter 3), Nina learns about the research artifact by feeling a feature of the experimental data in her body. Her use of the embodied semiotic modalities corroborates the claim that such a use is neither peculiar to the role of teacher nor is it only done for the sake of demonstration. Instead, the multimodal semiotic means are here employed by the newcomer to learn and reason about the examined phenomena by feeling their aspects in her own body. Nina’s semiotic action is further elaborated as she points to an exemplary brain visual on the screen (“like this one,” line 9) and performs another gesture (lines 9–10). She first points to the visual (figure 4.7) and then enacts the movement by sweeping her gesturing hand downward (figure 4.8). The articulation of the gesture is rather similar to Olga’s gesture enacted at the outset of the excerpt (lines 1 and 2). Yet, by mirroring the old-timer’s action, Nina demonstrates her competence in seeing what her colleague didn’t notice at first. She performs the gestures of a laboratory member, displaying her progressive acquisition of the ability to proceed on her own (Wittgenstein, 1973). The enactment of the subject’s conduct that demonstrates her capacity to see like a professional researcher allows the apprentice to comprehend the research artifact by experiencing her body in movement. The Body in fMRI Visuals Science-studies scholars (e.g., Knorr-Cetina, 1981; Latour & Woolgar, 1979; Lynch, 1985a; Star, 1989) have discussed how final accounts of scientific work (i.e., in the form of scientific articles, conference talks, or newspaper

80

Chapter 4

reports) standardize, while averaging and filtering out local contingency or individual differences: “One of the mandates of science is to create generalizable results, which are meant to be universal, and this mandate is often conflated with the deletion of local contingency” (Star, 1989: 93). Method sections of fMRI articles list repeatable and well-defined steps of the experimental procedure, as they—with the exception of single-subject studies—report on experimental participants in terms of their number, gender, and social group, thus making individual bodies utterly invisible. Similarly, once published, fMRI brain visuals show general truths about the human brain rather than the features of the individual participants whose brains were imaged. An example is the published fMRI figure discussed in chapter 2. The figure, even though it reports on brain activations in one individual, translates the result into coordinates usually used to average across multisubject data, as such widely known coordinates provide an indication of how the results can be generalized. In contrast with the impersonal style of scientific publications and the use of fMRI data for general claims about the human brain and cognition, the analysis performed in the fMRI laboratory highlights the individual and the particular. During Olga and Nina’s interaction, the experimental subject, because of her movement, becomes visible. Rather than simply noticing the difference between the two brain scans, Olga “sees” the experimental subject moving: “I see her in this plane” (line 1). When in lines 4 and 6 Nina says “Aaaa, this is a good one” and “Oh, here we are,” Olga follows by reintroducing the personal pronoun, as she asks Nina to elaborate: “Is she moving any more in thirty” (line 7). Nina answers by saying: “She is moving” (line 8). This resurfacing of the experimental subject is recognized during the data analysis so that it can then be canceled. The perceived consequences of “her” undesired behavior have to be removed so that the general character of the data can be regained. Similarly, if the data do not show any particular problems, practitioners tend to treat them as anonymous and general: As long as the subject’s body and behavior are seen as yielding themselves to the prescribed procedure, the experimental subject is promptly translated into abstraction. Such anonymous and general data are what the scientists look for, as their quest is to describe the functioning of the brain, not the idiosyncrasies regarding an individual brain. As a result, the reported interactional sequence illustrates the threshold—the

fMRI Brain Visuals and Semiotic Bodies

81

tenuous moment of the passage between the two stages—where the brain visuals still belong to a single individual whose individuality is being noticed so that it can be cleansed from the data. This moment of passage is an opportunity to examine how the individuality of the experimental subject, whom the researchers could have encountered only at the scanning facility, is rendered visible in the laboratory. As illustrated by the excerpt, Nina and Olga talk about the experimental subject by referring to the brain visuals on the computer screen. These visuals, however, do not simply represent the movements of the subject, but also involve the bodies of the scientists and the imaging technology. Nina and Olga go about their work, make sense of their data, and learn how to remove the aberrant visuals from the data set by manipulating the computer screen with a mouse to spot the misalignment between the series of brain visuals, and by experiencing the movement in their own bodies as they engage their hands, arms, necks, heads, and chests to coordinate their reasoning and communicate about their findings. As the practitioners understand the morphology of its movements by coordinating their trained and experiential bodies with the manipulation of the computer screen, the imaged body places conditions on the visuals and limits what goes on in the laboratory. The movements that the practitioners’ bodies enact are shaped by the movements that they expect to have happened in the scanner. The movements of the subject’s body, while disturbing the experimental data, articulate the movements of the practitioners.4 Even though the practitioners openly disapprove of what the subject did in the scanner, to understand and prepare their data for analysis, they must enact the imaged body’s movements. This, however, does not mean that the imaged body shows itself as a somehow “natural” thing. During the scanning session, the body is controlled by the requirements of the scanning procedure while being constrained by the theoretical expectations in cognitive neuroscience. Only certain features of the lived body, relevant for the research activity, are transposed into the visuals, and among the features that can be accessed through the visuals, only some are desirable. This resurfacing of the subject and the embodied involvement of the researchers speak about fMRI visuals and how they gain meaning. Though the subject’s body constrains the visuals, it is not fully shown in them, but it needs the researchers’ embodied engagement. In other words, the visuals,

82

Chapter 4

while selecting, allow the individual body (if only for a moment) into the laboratory; they do so, nevertheless, not by functioning as transparent conduits, but by requesting from the researchers to engage with them in an embodied manner. The experimental subject reappears in the laboratory as the digital visuals bring together the interacting bodies (the body of the subject and the practitioners’ bodies); as the visuals intertwine the brains, hands, eyes, shoulders, and necks of the lived and digital bodies, the imaged body (rather than being “constructed”) is enacted as multiple. The Body Multiple as Multimodal Interactional Achievement in Laboratory Practice Understanding fMRI visuals as fields for interaction leads us to reconsider how we talk about the body: What kind of body is at stake in the fMRI laboratory? By following the interaction between Olga and Nina, we encounter the body that does not simply belong to the experimental subject, as that body can be accessed only through the fMRI visuals and the practitioners’ enactments, and that does not exclusively appertain to the practitioners, either. When Olga and Nina engage their semiotic bodies to make the scientific visuals intelligible, their semiotic bodies are articulated by what they observe as they work with the computer and what is known in the laboratory. To deal with the problem of motion while performing it in a pedagogical manner, the practitioners enact and feel the hypothetical movement of the experimental subject and compare it with movements of the “general human body.” The movement that wasn’t observed when it took place now not only refers to the movement of the experimental subject while it articulates the practitioners’ movements and is responsive to the specific appearance of fMRI visuals, but is also shaped by the expectations of the community of practice. Consequently, as Nina and Olga feel the features of the experimental data in their bodies, they enact multiple and nonpresent bodies, imagining their felt past, following their movements, and enacting their conduct in the shared environment of practice. Their movements are not simply a translation of the subject’s behavior into the practitioners’ behavior. Rather, the enacted bodies, as the object of the scientific scrutiny, emerge as multiparty and dynamic phenomena of interaction.

fMRI Brain Visuals and Semiotic Bodies

83

Corporeal feminism has discussed the body as borderline, indeterminable, split, plural, and dynamic (e.g., Grosz, 1987, 1994). To foreground the need to enrich this highly theoretical proposals for rethinking the body with an empirical strategy, Helen Marshall suggested “doing phenomenology of the ordinary experience of the body” (Marshall, 1996: 256): The methodological point . . . is that when we try to name our bodily experience, we are always involved in a dialogue. So when researchers try to describe women’s experiences in particular situations, we need to ask about who else is there—factually or in imagination. . . . Thus, a useful question for both researchers and theorists to ask may be ‘how many bodies are here, and of what kinds?’ or ‘which players (in the immediate situation and the broader social context) are involved in the creation and recreation of this body. (Marshall, 1996: 262)

Annemarie Mol’s (2002) Body Multiple proposes a related line of argument, combining it with developments in the literature on the social aspects of science and technology. Through an ethnographic study conducted in a Dutch university hospital on the diagnosis and treatment of atherosclerosis, Mol shows that once we take the perspective of practices (what she calls a praxiographic approach), the disease figures as multiple. Based on observations of medical examinations, consultations, and operations, in addition to interviews with personnel and patients, Mol concludes that as atherosclerosis is being enacted in one moment and place in time to the next, it takes on a variety of shapes, figuring as a slightly different phenomenon in any specific site: If practices are foregrounded there is no longer a single passive object in the middle, waiting to be seen from the point of view of seemingly endless series of perspectives. Instead, objects come into being—and disappear—with the practices in which they are manipulated. And since the object of manipulation tends to differ from one practice to another, reality multiplies. The body, the patient, the disease, the doctor, the technician, the technology: all of these are more than one. More than singular. (Mol, 2002: 5)

As we attend to the details of multimodal interaction in the fMRI laboratory, we see parallels. Olga and Nina’s interactional engagement concerns multiple and dynamic realities. These realities emerge as we follow their work with computer screens, as we focus on their attending to each other’s semiotic bodies, and as we take in account their sensing of their own bodies in action. With the proceeding of the apprenticeship session Nina continues to try to spot motion artifacts in the fMRI visuals while Olga checks on her

84

Chapter 4

comprehension of the procedure. After Nina provides another successful reply to Olga’s inquiry, Olga further elaborates on the subject’s movement. In this interactional sequence, the practitioners’ recourse to multimodal semiotic means, their conceptual import, and the skillful coordination between the bodies and the setting in which the activity is rooted show how the body is enacted as multiple not only across different sites and practices (as Mol explains) but all in one breath. EXCERPT 2 1

N:

Up? ((Stretches up))

2

O:

Yep again ((Nods))

3

And then actually it looks like there is more involvement,

4

I am thinking there is more involvement=

5

in the front or in the back, can you tell? ((Taps the front and then the back of her head with a pencil))

Figure 4.9

6 7

N:

Figure 4.10

((Works with the computer)) Back? ((O keeps touching the back of her head)

fMRI Brain Visuals and Semiotic Bodies

8

O:

Yeah right so you can see the voxels= ((Points with the pencil toward the screen and moves the pencil up and down))

Figure 4.11

9

in the back moving a lot more than the voxels in the front ((Turns the pencil in the vertical position and moves it now up and down. At the same time she nods with her head. N approves by nodding))

Figure 4.12

10

85

so she is (.) this is an interesting movement because most people would=

86

11

Chapter 4

if you think if you think laying in the scanner= ((Pushes back against the back of the chair))

Figure 4.13

12

most people would nod= ((Nods with her head))

Figure 4.14

13

before they

fMRI Brain Visuals and Semiotic Bodies

14

((Movement of her neck against the back of the chair. N executes in parallel the same type of movement))

Figure 4.15

15

cause your neck doesn’t= ((Points to the back of her neck))

Figure 4.16

87

88

Chapter 4

16

really stretch quite that way= ((Moves up and stretches her neck))

Figure 4.17

17

right?

18 N:

(No)

19 O:

So I don't know what she is doing ((Performs a gesture of surprise))

20

but she is doing this. Hhhhh ((Laughs)) for some reason

Like in the previously discussed excerpt (excerpt 1), the two practitioners partake in the work and learning activity by coordinating the movements of their bodies with what has to be seen on the computer screen. While Nina performs a movement in line 1, Olga continues to guide her, looking at the visuals from lines 2 to 9. Lines 10–17 are particularly interesting as they indicate the multiple character of the bodies engaged in work and learning. We see a performance and elaboration of the movement in the hypothetical space of action where the practitioners’ bodies assume multiple roles of individual, general, and imagined bodies. The elaboration of the subject’s movement (that the practitioners originally noticed in line 1) starts in line 10 with Olga’s remark: “so she is (.) This is an interesting movement because most people would. . . .” The subject’s movement is labeled as an “interesting” one inasmuch as it differs from the movements that “most people would” do. To underline the unusual nature of the subject’s movement, Olga asks Nina to imagine

fMRI Brain Visuals and Semiotic Bodies

89

herself in the scanner. Olga articulates this imaginary space of movement by performing the movement that “most people would” do. As she expects the normative movement to be shaped by the material conditions of the scanner, where the subject’s head is constrained by its contact with the table on which the subject lays, Olga enacts the normative movement by pushing abruptly against the back of the chair and saying “if you think, if you think laying in the scanner.” While her body in the vertical position maps to a human body in the horizontal position, the chair on which Olga is sitting is transformed into a sign for the scanner’s table on which the subject is lying. Once in the position (line 12), Olga nods to perform the behavior of the general body lying in the scanner, letting her body stand for the body of “most people.” Then (line 14), the practitioner contrasts this expected movement with the subject’s actual movement by moving her neck against the back of the chair. She indicates that the subject’s movements are different from the movements constrained by the physical structure of the scanning environment, and she labels them as impossible and incomprehensible (lines 15–16 and 19–20). This explanation is understood by Nina through her own embodied performance (line 14): The newcomer coordinates with her colleague to perform herself the unexpected movement. Next (line 15), Olga provides a clarification for why the subject’s movement is incomprehensible. She points to the back of her neck explaining that the movement is an “interesting one . . . cause your neck doesn’t really stretch quite that way” (lines 15–16). When stating that the neck doesn’t “really stretch quite that way,” she briefly moves up and stretches her neck to enact the movement that the subject has performed in the scanner (line 16). In other words, the practitioner moves in an “interesting” way, providing an embodied demonstration of how the neck cannot move. When in line 17 she asks the apprentice for a confirmation (“Right?”), Nina promptly agrees (line 18). The subject’s movement is judged impossible while its morphology is felt in the bodies of the two practitioners. By performing and feeling the awkwardness of the movement, as a means of knowing it, Nina learns that “your neck doesn’t really stretch quite that way.” The multimodal engagement of the two practitioners shows the body as an emergent and multiparty achievement that goes beyond the immediate here and now. Rather than pertaining to a single biological body, the

90

Chapter 4

multiplicity of the bodies—the body of the experimental subject and the “general” human body—are experienced through a first-person perspective of the two practitioners immersed in the contingencies of their work environment. The activity continues with Olga’s gesture of surprise and her comment accompanied by her laughter: “So I don’t know what she is doing, but she is doing this, for some reason” (lines 19 and 20). Even though the misalignment between the fMRI visuals attests to the existence of movement, this movement cannot be explained with respect to what is known about the subject’s usual movements. In other words, the subject’s movement, indicated by the visuals, is regarded to be unnatural, as it disrupts what is known and expected in the laboratory. In line 12 Olga explains what “most people would” do, and in line 13 she points out the rarity of the movement performed by the subject (“before they”) as she exhibits the accumulated knowledge of the laboratory. Even though she performs and feels the subject’s movement (line 14), Olga invokes the general principle that the “neck doesn’t really stretch quite that way” (lines 15 and 16). The feeling of the movement is thus framed by the knowledge that goes beyond the specific instance of data analysis: The two practitioners shape the examined body as “culturally inscribed and historically colonized territory” (Foucault, 1963, 1975, 1976). At the same time, to make sense not only of the impossible movement of the scanned body, but also of how the normative body moves, Nina and Olga engage their lived bodies. As a part of the apprenticeship and data analysis session in the fMRI laboratory their embodied semiotic actions, organized by what is known in the laboratory, indicate their involvement in enacting multiple bodies all at once. Bodies in fMRI To approach the problem of the body, I followed the reappearance of the experimental subject’s movements in terms of the research artifact—the disturbance in the visibility of experimental data. The issue of the movement artifact enlightens two contrasting tropes: the absence of the body (beyond the brain) in the fMRI evidence and its centrality in practical problem-solving and learning. The presence and absence of the body in the context of fMRI is curious. During scanning sessions, the subject’s body

fMRI Brain Visuals and Semiotic Bodies

91

is kept as still as possible. Likewise, once the results of scientific practice move outside the laboratory (in the form of scientific articles, conference talks, or newspaper reports), those bodies (beyond the part that is the focus of cognitive neuroscience—the brain) are rendered entirely transparent. Yet, in the laboratory—after the experimental subjects have been scanned and before the experimental results have been published—practitioners make the fMRI visuals intelligible by articulating the scanned body through enactments of its movements. To identify the undesirable features of the data (and thus erase them from the data), the practitioners (1) manipulate the visuals on the computer screen; (2) learn about the movements by referring to the knowledge of the community of practice (taught by Olga); and (3) feel their own bodies in movement. The apprentice acquires fMRI skills by working on the computer, by being trained how the body of a laboratory member and a cognitive neuroscientist should feel and move, and by feeling her own semiotic and experiential body. Her body plays a central role in learning about and discussing the visually displayed digital renderings of a scanned body. Importantly, the body that populates the laboratory is not a unified, biological entity. Turning attention to moment-to-moment interactional dynamics and understanding fMRI brain visuals as fields for interaction suggest that the bodies should be rethought as multiparty and dynamic. The bodies in the imaging laboratory (the scanned bodies and the bodies that make sense of fMRI data) are at the same time individual and collective, material and discursive. The enacted body is a corporality that is here and now but is also an imaginary and relational body that occupies multiple spaces at the same time, as it emerges through the specificities of multimodal interaction.

5 The Semiotic Mind in the fMRI Laboratory

Reconceptualizing fMRI brain visuals as fields for interaction has consequences for the understanding of thinking in the laboratory. The fMRI researchers’ skill to see certain concrete and spatially represented biological phenomena is not confined to an individual brain but is a gradual achievement that involves a fine coordination between digital screens, researchers, and their hands, and an array of graphical inscriptions. Far from being only an intellectual operation, the newcomer learns to see by looking at brain visuals but also by touching the digital screen and coordinating it with charts, maps, sketches, and other paper-based visuospatial inscriptions. The details of these actions indicate that this is true not only for newcomers but also for advanced practitioners: The involvement of their semiotic and working hands with computer screens and graphical inscriptions is an important aspect of how experienced practitioners think in the laboratory. Even though the practitioners that I observed use input devices such as a mouse and keyboard to direct changes displayed on computer screens, they frequently place their hands directly on the screens. In fact, computers in fMRI laboratories regularly show the prints of fingers and hands that have touched them. In this chapter, I take advantage of these apparently redundant actions to discuss their roles in meaning-making. When practitioners actually touch the surface of the screen as part of their laboratory work, they often use their hands to point (e.g., Kita, 2003) and to coordinate the screen with other graphical inscriptions. These handlings organize the visual field by momentarily inscribing forms and enacting new relationships among its elements. In this sense, they are features of the semiotic mind.

94

Chapter 5

Studying the Organization of Visual Brain Areas Here we join an apprenticeship between a postdoctoral student, the oldtimer in the laboratory, Octavia (O), and the newcomer, Nick (N), a firstyear graduate student in neuroscience spending a semester in the laboratory, and already encountered in chapter 3. Nick is a driven, fast learner who is not shy about asking clarification questions. Octavia is the laboratory’s former Ph.D. student, highly regarded by the laboratory’s director and her other colleagues. As someone who has already published important results in the field, Octavia is busy, while cheerful and kind. The atmosphere during the 4 hours of apprenticeship session is friendly but fast-paced and intellectually demanding. While the practitioners focus their efforts to understand how an external scene affects its rendering in the human brain, I want to think about seeing as an embodied process of understanding that is relative to a positioning of the perceiver in the environment of practice. This chapter follows moments of laboratory work where the analyzed data pertain to specific experimental questions. Whereas the initial stages in data analysis (such as the identification of motion artifacts, described in the previous chapter) are common procedure across fMRI laboratories, later moments in data analysis bring to light differences in the work of each laboratory.1 Thus, to master data analysis procedures, it is indispensable for practitioners to go through an apprenticeship, embedding themselves in the procedures of the relevant social group. The laboratory described in this chapter studies visual areas by applying the method of retinotopic mapping. As the general topography and location of the early visual areas relative to one another are believed to be, by and large, consistent across individuals, the researchers use retinotopic mapping to identify the configuration of the visual areas on the fMRI rendering of the subject’s cortex. Once the organization of the early visual areas is identified, that information is used to analyze data from the main experiment. If during the main experiment the researchers investigate how the early visual areas respond to the specific experimental task, they first need to map out the location and borders of these areas.2 As already mentioned in chapter 2 (when discussing the published fMRI brain figure), the idea behind the concept of retinotopy is that there is an orderly mapping between locations in each retinotopically organized brain

The Semiotic Mind in the fMRI Laboratory

95

area and the locations in the visual field. In other words, the retinotopically organized visual areas are considered to be point-to-point copies of the topography of the retina, where what is of interest are the topographic correspondences generated through a translation of information between the eye and the brain. Nearly all visual information reaches the cortex via the primary visual area (V1). V1 is located in the posterior occipital lobe within each hemisphere and provides a precise retinotopic mapping of the visual fields. In the left-hemisphere V1, the right half of the visual field is represented, covering 180° of the field circle. V1 projects in a topographically wellordered fashion to V2 (second visual area), to then connects to numerous visual areas: V3, V3A, VP, V4, V5 or MT, V7, V8, and so forth. Once the projections of visual stimuli in V1 is established, the other retinotopically organized visual areas can be determined with respect to it as the successive areas are considered to be mirror images of each other. As a part of retinotopic mapping, Octavia and Nick are involved in identifying the organization of a “phase map.” The phase map is a static fMRI visual that shows the temporal relationship of the data to certain stimuli, where colors stand for the time at which neuronal activity occurred. When scientists want to obtain such a visual (or a “brain map”), they present experimental subjects with dynamic visual stimuli intended to provoke “waves of activation” in their brains, such that variation in the temporal phase of the activation can be represented by changes of hue in a color map. Using the change in color, practitioners identify borders between visual areas on the human cortical rendering (as shown in figure 5.1).The work in the laboratory exposes the complexities of this procedure. In addition to the change in color, Octavia and Nick engage a variety of multimodal semiotic fields. The fine coordination of those fields is pivotal in generating their seeing of the fMRI visuals. During the observed events, Octavia and Nick are seated in front of a laboratory computer that displays phase maps, while I stand behind them (figure 5.2). The description of their interaction is organized around four excerpts that span from the moment in which Octavia states that identifying visual areas on an fMRI scan is a complex task requiring specialized knowledge, to the moment in which Nick affirms that he can see the organization of the visual areas on the brain map. This progressive acquisition of the skill involves practical handling and

96

Chapter 5

Figure 5.1 Phase map captured at a later stage of the practice after the practitioners had already inscribed borders of early visual areas on it.

Figure 5.2 The two practitioners as they participate in the data analysis session.

The Semiotic Mind in the fMRI Laboratory

97

adjusting of the semiotic bodies to what is observed in the shared, lived world. Look to See The first excerpt marks the beginning of a journey through which the newcomer acquires the skills that will eventually allow him to see the systematic structure of the phase map. The excerpt illustrates how the practitioners themselves talk about the complex character of professional vision (Goodwin, 1994). Octavia directs Nick’s attention toward a brain visual on the computer screen (the “map”) to emphasize the prolonged effort needed to acquire the capacity to see the organization of such visuals. EXCERPT 1 1

O:

2

So if you look here what can you actually see ((Moves closer to the screen)) and it takes quite a bit of training to start and actually see the maps in this (.) noisy data. ((Moves her hands back and forth in front of her, with palms toward her and fingers spread))

Figure 5.3

3

N:

((Turns toward Octavia and nods))

98

Chapter 5

When Octavia says: “So if you look here what can you actually see” (line 1), her utterance has a hypothetical form, inquiring about what can be seen and, thus, functioning as an invitation to “look,” which should eventually lead to “seeing.” The old-timer is aware that Nick cannot yet see the correspondences between the stimuli and the neuronal response that should be identified in the fMRI data (the use of the pronoun “you” in line 1 is generic). The first condition for “seeing,” intended as perceiving and comprehending the structure of the brain visuals, is careful looking. In line 2, Octavia explicitly expresses the idea that the seeing of the fMRI visuals does not happen immediately, “and it takes quite a bit of training to start and actually see the maps in this (.) noisy data” (line 2). The expression “actually see,” already used in line 1, refers to the understanding of the data configuration, as opposed to just perceiving colorful images on the computer screen. By juxtaposing the “noisy data” and the map-like structure, Octavia suggests that the organization is somehow embedded in the noisy data. However, it takes a lot of careful “looking” to “see” and understand the meaning of the data. In other words, the structure of the data is not simply there to be directly perceived but is gradually achieved through intense acts of looking. This is introduced when Octavia says that to see and identify “maps in the noisy data” (lines 2–3) requires training (line 2). In line 3, to emphasize the active nature of seeing, Octavia incorporates gesture in her exposition (figure 5.3). She empathetically moves her hands back and forth between her body and the computer screen, with palms turned toward her and fingers spread. The gesture animates the space between the practitioners and the brain visual, performing a dynamic linkage between the two. Octavia’s intense gesturing captures Nick’s attention, who, by turning his gaze from the computer screen toward her, and nodding to acknowledge his consent with what his colleague is saying, exhibits his readiness to learn to see as an fMRI researcher. As the unfolding of the apprenticeship session indicates, this skill calls for the coordination of the computer screen with a range of semiotic means that include paper-based visuospatial inscriptions.

The Semiotic Mind in the fMRI Laboratory

99

Mutable and Local Inscriptions Important work on the role of inscriptions in scientific cognition has been done by Bruno Latour (e.g., Latour, 1986; 1990). Latour argues that the shift in thinking that characterizes modern culture does not have to do with changes in the human mind but with our practices of mobilizing resources without transforming them. In his article “Visualization and Cognition: Thinking with Eyes and Hands” (Latour, 1986), Latour argues that what is specific to our modern scientific culture has to do with the inscriptions generated through writing and imaging procedures. Latour refers to these inscriptions as immutable mobiles—the resources that can be displaced without transformation. To discuss how the inscriptions are gradually simplified as they are translated into each other, Latour talks about cascades of representations. The immutable mobiles, arranged in cascades of representations, constitute lines of force and networks of power. They are the way to assemble allies, and thus form centers that dominate the rest of the world (Latour, 1990: 23–24). Latour, thus, understands thinking in terms of displacing inscriptions without altering them: “Thinking is tantamount to acquiring the ability to move as fast as possible while conserving as much of the pattern as possible” (Latour, 1990: 51). The proposal, which frees Latour from mentalist and materialist explanations, puts cognition and visualization in direct relationship to power, as the manipulation of inscriptions equals power. Yet, despite Latour’s urge to not ignore “the precise practice and craftsmanship of knowing” (Latour, 1990: 21),3 the argument remains somewhat removed from the lived bodies and their practical methods that bring inscriptions together to generate meaning. This has consequences for Latour’s characterization of inscriptions as immutable mobiles. Though Latour points out that “‘[t]hinking is hand-work,’ as Heidegger said, but what is in the hands are inscriptions” (Latour, 1990: 46), his effort directed toward pursuing this argument is limited.4 In other words, Latour’s main focus is the inscriptions as immutable mobiles rather than the specificities of their interaction with the hands. Once we, however, start to attend to the engagement of the hands, the inscriptions show themselves as not necessarily mobile, while they are frequently mutable. The work of hands often concerns inscriptions that have to be altered through local practices. This alteration commonly

100

Chapter 5

regards their coordination with other semiotic means. As seen in the fMRI laboratory, when coordinated with gesturing hands, graphical inscriptions can be quickly transformed but often do not traverse laboratory walls. In the previous chapter I focused on how gestures enact semiotic content by performing what needs to be understood about the brain visuals. For example, in motion correction activity, the practitioners use their bodies to enact the experimental subject’s movement that they deem to be the cause of the distortion in the data. In this chapter, conversely, I turn attention to the gestures that, rather than enact features of the data (as the enacting gestures do), point to those features. Because they gloss their own production to render something else visible (Hindmarsh & Heath, 2000), pointing gestures often participate in generating content that has to be deciphered directly in the fMRI visuals. Starting from these gestures, we see the mutability of fMRI visuals. fMRI visuals are obviously dynamic and mutable because of their digital character. However, they are also often rendered additionally dynamic when coordinated with pointing gestures (whose actions are sometimes recorded in the laboratory by the prints of fingers and hands on the computer screens), and linked with other inscriptions. The paper-based inscriptions, which range from anatomic maps to short-lived sketches, and which the practitioners coordinate with the digital visuals, often take over a role similar to that of the enacting gestures: They enact the content that then has to be seen in the fMRI visuals. These inscriptions show their mutability when produced (scientists often draw those inscriptions while working and communicating with each other) and when coordinated with the hands engaged in practical actions and pointing. In the laboratory, we see how scientists move the inscriptions around (placing them on the desk or next to the computer screen, for example) to immerse them in new semiotic configurations. They also inscribe (through the use of gestures and writing tools) new forms on the surfaces of those inscriptions. Because of these actions, the paper-based inscriptions are dynamically defined by the specifics of the situation in which they are lodged (Lynch, 1994; Suchman, 1987) and are involved in generating visibility of brain visuals as an emergent phenomenon. Practitioners can make sense of brain visuals by

The Semiotic Mind in the fMRI Laboratory

101

juxtaposing the static of the paper with the dynamic of the hands (and digital screens) and see what is not visible in any semiotic field taken in isolation. Thus, when immersed in the world of the lived bodies and their practical methods, the paper-based inscriptions participate in researchers’ thinking in action. As Latour points out, we should take into account “writing and imaging craftsmanship” (Latour, 1990: 21) to understand how human thinking works. In doing so we must not forget these local and mutable inscriptions. Mutability of the Old-Timer’s Chart In the fMRI laboratory, mutable and short-lived graphical inscriptions are frequently encountered in practitioners’ accounts. There they are acknowledged for their participation in the acts of thinking with hands and eyes. “Octavia’s chart” is a case in point. The chart, reproduced in figure 5.4, is a paper-based inscription that distills laboratory knowledge. Octavia made the chart to explain the organization of brain visuals to Nick. The chart can be divided into two parts: the diagram of the visual field (figure 5.4a) and the retinotopy space (figure 5.4b). The diagram of the visual field stands for what experimental subjects see, and the retinotopy space indicates the cortical organization of the subjects’ visual areas. As such, the chart has an important role in making the experimental data visible. During her interaction with Nick, Octavia coordinates the chart with brain visuals to relate the regions drawn on the chart to the structures that should be seen on the computer screen. These acts indicate how the chart—while it extracts, classifies, and codes—gains its full meaning (even though only momentarily) by entering into dynamic arrangements with hands and digital brains. To find out more about how the practitioners understand the chart and its function in laboratory learning, I engaged in a brief written exchange with Octavia. Octavia’s response to my question about the history of the chart indicates that the chart was constructed through a somewhat ad hoc procedure and is intended to have a shorter life-span. These features render the chart valuable as the practitioners attempt to make sense of the fMRI visuals:

102

(a)

(b) Figure 5.4 Octavia’s chart of the visual field (a) and retinotopy space (b).

Chapter 5

The Semiotic Mind in the fMRI Laboratory

103

The chart was my idea. I was taught how to interpret the retinotopic maps by someone else in the laboratory (who is no longer here). He basically just showed me the fMRI data on the computer screen (as we saw) and explained it all verbally. I thought it would be easier to understand visually so I put together the chart just to help myself understand. The first version was really messy with many erasures as I was just figuring it all out. So I recopied it in a more orderly fashion (the version that you have) [figure 5.4]. Since then, I have used it several times to teach others. People have found it useful and have asked me for copies.

Octavia starts off by stating her authorship of the chart: “The chart was my idea.” At the same time, her narration brings to the fore the multiplicity of voices that the chart, as a cultural artifact (Vygotsky, 1962, 1978), contains. When she was a newcomer to the laboratory, Octavia was taught how to analyze retinotopic maps by another colleague. In this sense, the chart, while being Octavia’s invention (she translated the oral explanation into a visuospatial form), is a distillation of prior experiences and the laboratory’s traditions. Conversely, laboratory members adopted Octavia’s chart to speak for them, as they consider it to be a valid rendering of how the fMRI visuals should be approached.6 As Octavia pointed out, “People have found it useful and have asked me for copies.” Yet, inscriptions like Octavia’s chart are not only repositories of information but also interactive fields for problem solving. When responding to my question, Octavia indicates that the chart was not initially constructed to disseminate knowledge to others but to be useful for its author’s sense-making. The chart was made to help Octavia comprehend the task: “I put together the chart just to help myself understand.” Octavia emphasizes that making the chart gave her the chance to instantiate the information visually. Talking about a colleague who introduced her to the laboratory procedures, Octavia explains, “He basically just showed me the fMRI data on the computer screen . . . and explained it all verbally. I thought it would be easier to understand visually.” Thus, according to the practitioner, this re-representing of the information made the task cognitively easier to manage (Clark & Thornton, 1997; Kirsh, 1995). As Octavia drew the visual field in the form of a gridded parallelogram (figure 5.4a), her drawing not only materializes an element of the explanation that otherwise would need to be imagined (i.e., the visual field, used as something that does not exist in fMRI renderings but is evoked for its understanding) but also, by schematizing it, generates clarity through selection.

104

Chapter 5

The chart and its construction, by making an abstract entity a concrete one and thus highlighting its relevant features, participate in Octavia’s sense-making. In her account, Octavia also makes clear that the construction of the chart was not a translation of some internal structure (i.e., the knowledge previously acquired) into an external one but a loop-like process, where the structuring of the chart was modified with respect to the feedback acquired from the drawing itself. This can be seen when she says, “The first version was really messy with many erasures as I was just figuring it all out. So I recopied it in a more orderly fashion. . . .” The chart came to be what it is through a procedure of gradual refinement that played a role in the practitioner’s comprehension. Octavia’s understanding of the organization of the visual areas was formed and solidified through the process of the chart’s fabrication. Interestingly, whereas figure 5.4 reproduces the chart that was drawn during the apprenticeship practice, at the end of the training session Octavia showed me an already-made chart that she keeps in her office. During her interaction with Nick, Octavia, rather than reusing the alreadyexisting chart, initiates her explanation with a blank piece of paper. She considers the mutability and the short life-span of the chart to be its virtue, as it allows her colleague to participate in the gradual construction of the chart. As Octavia designs the chart, Nick can signal potential difficulties in understanding. Octavia can then modify the components of the chart as well as the way in which she draws it. While this allows the old-timer to adapt her explanation to the newcomer’s participation, the newcomer can have an experience similar to the one the old-timer had when she originally constructed the chart. In fact, the final form of the chart (displayed in figure 5.4) inscribed Nick’s coparticipation. There are, for example, the crossed-out lines on the representations of the visual areas (figure 5.4a), and there are two, almost identical renderings of the visual field (figure 5.4a). When Octavia drew the original arrows on the retinotopy space, Nick indicated that he was not sure how the waves of activity propagate across the second visual area. To clarify her explanation, Octavia crossed out the arrows on the representations of the visual areas and generated an additional, more elaborate sketch of the visual field (the left-hand side of figure 5.4a). The marks that indicate a momentary misalignment between the two practitioners, while

The Semiotic Mind in the fMRI Laboratory

105

also signaling its resolution, make the mutable aspect of the inscription an effective element of learning in the laboratory. Though the mutability of the chart does not provide the teacher with an indisputable power, it promotes an engaged learning experience. This malleability of the graphical inscription is heightened through multimodal semiotic interaction. When the two practitioners draw new lines to complete the chart, they also point to its parts and enact shapes and processes that need to be seen in it. The gesturing is accompanied by verbal explanations that introduce the newcomer to the professional vocabulary of the laboratory. The graphical inscription, generated to function as a structure that organizes the messy world of brain scans into a meaningful domain, is, thus, not simply a location of knowledge; it is a situated field that develops through interaction. This interaction is about reasoning and knowledge acquisition in the laboratory.7 Correlating the Chart, Hands, and fMRI Map to Make the Unknown Known Equipped with the chart, the practitioners can begin to map the territory of the fMRI brain scan displayed on the computer screen. Here, Octavia and Nick are dealing with two different kinds of visuals—what they call the “chart” and the “brain map.” The juxtaposition of the visuals is a critical yet delicate task. It allows practitioners to see what does not exist in either the chart or the map taken in isolation but is relative to the configuration realized through interaction. Excerpt 2 reports on a moment of practice where the practitioners’ immediate goal is to determine the location of the visual area V1 (which is labeled on the chart). To transform the brain scan into what is for both practitioners a known domain, the structure of the knowledge existing in the chart has to be coordinated with the brain scan. This is largely accomplished through manual handling and gesturing. While Octavia moves the chart around, preparing it for an alignment with the map, her hands are concurrently involved in gesturing. The gestures are a method of linking elements from one semiotic field to the other, merging the structure of the chart with the brain map. The conjoined work of the two hands (each being differently engaged in the process) with the visual fields partakes in the production of intelligibility.

106

Chapter 5

EXCERPT 2 4

O:

So probably this is the center ((Touches the screen with her index finger))

5

right here ((Rotates the sheet of paper with the drawings moving it clockwise and points to what represents the fovea on the chart))

6

And when we look at this map it looks something like that ((Picks up the paper and holds it next to the computer screen))

7

So V1 is gonna be in the center ((Briefly traces the borders of the V1 representation of the retinotopy space on the paper with her index and middle finger))

8

it’s gonna be this pie shape it’s probably covering approximately this area ((Carefully places her index and middle finger on the “center” of the phase map on the computer screen and traces the imaginary borders of the V1 representation; repeats the movement six times))

Figure 5.5

9

Okay?

The Semiotic Mind in the fMRI Laboratory

107

Octavia’s talk and gesture show uncertainty about the organization of the brain scan. When Octavia refers to the center of the phase map (line 4), she uses the word probably (“so probably this is the center”). In contrast, her indexing of the center on the chart (line 5) is accompanied by an expression that displays certainty (“right here”). Similarly, in lines 7–8, when she indicates the position of the visual area V1 on the brain scan, she uses the words probably and approximately. Her utterances, expressed in the future tense (lines 7 and 8), target a normative character of seeing: Octavia points out what is expected and should be seen in the data rather than what can be distinguished at this particular moment. By framing the seeing of the map as unsure, Octavia leaves room for the explanation that follows, inviting her colleague to coparticipate in the gradual achievement of visibility. To make the coordination between the two semiotic fields more effective, Octavia first rotates the chart (still placed on the table; line 5) to prepare the alignment and achieve a more straightforward tightening of the relationship between the two. In the following line (line 6), while saying, “And when we look at this map it looks something like that,” she picks up the chart and skillfully places it next to the computer screen (figure 5.5). She keeps the chart in the same position throughout the interaction reported in the excerpt. The physical act of placing the chart closer to the screen and holding it there invites Nick to see the two visuals as comparable. In this sense, the work with the visual inscriptions and their spatial organization participates in the production of seeing. Next, a particular element of the chart (i.e., the representation of the visual area V1) is transposed from the chart onto the brain map. This step in the production of knowledge is again importantly played out by the gesture. In line 7, Octavia places her right hand onto the chart and briefly traces with her index and middle finger the borders of V1. In line 8, she lifts her hand from the chart, moves it toward the brain scan, and carefully places it onto the scan where she traces a similar form with her fingers. The action is not simply an execution of two indexical gestures (one over the chart, and the other over the phase map) but a performance of a complex, multimodal act that blends8 one semiotic field with the other. As the borders of V1 are already drawn on the chart, the gesturing hand, via the touching of the chart (line 7), selects the existing form and position of V1, quickly traces it on the chart, and then carries it over onto the phase

108

Chapter 5

map. In other words, the gesturing over the chart is performed not so much to establish the new structure (as is the case when Octavia gestures over the phase map) but to pick up the existing one and transfer it over to the phase map. Even though the old-timer claims that the map-like structure exists in the brain scan (see excerpt 1), she performs an action where the lines (i.e., the borders of V1) are not simply deduced by observing the change in color on the brain map. Instead, she inscribes them on the brain map by superimposing the form transferred from the chart (line 8). Thus, the relevant features of the brain visual emerge through the juxtaposition of multiple semiotic fields brought together through the multimodal interaction. This production of visibility cannot be reduced to any one of the elements involved. First, the inscription of V1 on the chart is not the exact same size and shape as its counterpart on the brain map (where the location of its borders also depends on the change in color not visible on the chart). What is seen does not exist on the brain map taken in isolation either (the borders of V1 are not simply available to a naïve eye). Moreover, the gesture and talk do not reveal the content of the brain map; because they are indexically linked to the two visuals, they acquire their meaning only with respect to this relationship. For example, the explicit indexical forms such as “this” in line 4, “here” in line 5, “this” and “that” in line 6, and two instances of “this” in line 8 gain meaning by being embedded in the course of action. To see the organization of the brain map, practitioners have to engage it as a temporally unfolding field of action that coordinates the chart, the old-timer’s talk, and her gesture all at once. Just by looking at the fMRI scan, the newcomer cannot yet infer the position of the V1 borders. However, by paying attention to the actions of his colleague, he can observe the solution to the problem, as well as the process of its production, before his eyes. He sees the lines, and hence the V1 borders, by observing Octavia’s gestures on the computer screen and listening to her linguistic explanation. For Octavia, similarly, acts of mapping are not complete until she has performed the action. She knows that she is going to inscribe an area on the brain scan of approximately pie-wedge shape, of approximately this location, and this size. Octavia knows this because she has an internalized knowledge of what is represented on her chart. But until she does the

The Semiotic Mind in the fMRI Laboratory

109

inscribing action on the brain scan, she probably does not see the borders of the visual areas on the map. The act of linking the two visuals to achieve categorization of the one in terms of the other suggests, once again, that important functions of the mind do not exclusively reside in the brain. It has often been pointed out that the process of integration across different domains is a characteristic of human cognition (e.g., Fauconnier & Turner, 2003; Koestler, 1964; Mithen, 1998). Here, by paying attention to the sequence of gestural moves, their interaction with the digital screen and the accompanying graphical inscriptions, we see this process importantly instantiated in the publicly shared environment of practice. Just as thinking out loud can be both communication and thinking, multimodal interaction can simultaneously serve communicative and cognitive functions. Point to Think One way to make brain scans intelligible is to gesture and engage other inscriptions, conjoining them. The other way is to directly act by taking advantage of the digitality that characterizes fMRI data. By clicking on the computer mouse, using keyboards to type commands, and gesturing over the screen, practitioners engage their hands in listening to their data as they manipulate their visual features. The laboratory events show how these actions allow not only the newcomer but also the experienced practitioner to decipher the organization of the brain scan. Octavia’s capacity to see the structure in the experimental data could not be adequately explained if her hands, engaged with the digital screen, were not taken in account. During the interaction reported in the following two excerpts, the borders of the visual areas had already been traced (see figure 5.1). The software that the laboratory members use allows practitioners to draw black lines directly on the fMRI visual. To identify these lines, the laboratory members use the color coding displayed on the screen. By understanding the response to the visual stimuli in terms of waves of activation that change in color over time, they employ the direction of the color change as an indication of the existence of the borders between visual areas. The newcomer, however, expresses his uncertainty about the location of the borders as the highlighted areas correspond only approximately with

110

Chapter 5

what is seen on the chart. Whereas on the chart the areas are represented as the same or similar in size, on the brain map some areas look much larger than others as their shape frequently appears significantly distorted. To explain the location of the borders, Octavia first points to the horizontal array of colors situated above the brain scan (figure 5.4), and is designed to indicate the phase of the neuronal activity for voxels being active as a response to experimental stimuli.9 However, as the phase map may not reveal some of the colors while the other colors may cover a very large portion of the map, Octavia takes advantage of the data’s digitality by “rotating the color map.” When practitioners talk about “the rotation of the color map,” they refer to a computational process that changes the correspondence between colors and the time of neuronal response so that, for example, a response phase that was represented in the original color map by red may become yellow in the rotated color map. Octavia exploits this feature of the software to find an alternative view of the data. Yet, the color coding of the brain scan does not automatically generate seeing for practitioners. These complexities in the production of intelligibility do not only concern the newcomer but the old-timer as well. As Octavia continues to analyze the data, the activity is filled with hesitations, prolongation of actions, and brief suspensions. In her effort to coordinate the inscriptions, Octavia’s actions are tentative, hypothetical, and creative, or what Peirce calls “abductive.”10 The old-timer cannot simply apply already attained propositional knowledge to the situation at hand. To make the brain map visible, Octavia has to form an explanatory hypothesis for what she observes on the computer screen. In doing so she skillfully turns her lived body to the experimental data. The newcomer sees her coordinating the colors on the screen with her work on the computer and involving her multimodal semiotic body in trying to decipher what the data say. While she uses her pointing hands to indicate the existence of the borders on the brain map, the old-timer organizes her own seeing. This orchestrated act, while it disproves the assumption that thinking is a process exclusively localized in specific areas of an individual brain (as implied and further suggested by the use of fMRI in cognitive neuroscience), makes prominent an fMRI seeing and problem solving as actions importantly realized in aligning the practitioners’ bodies with the (technologically articulated) world.

The Semiotic Mind in the fMRI Laboratory

111

EXCERPT 3 10 O: 11 N

So now it’s going from //re::d (.) red to pink to blue (3.5) //re::d

12 O:

and maybe it goes out to red to pink again ((Points to different colors on the map and then briefly places her hand on the desk)) (0.1)

13

(and up there where it breaks down) ((Points again onto the upper portion of the map and then briefly places her hand on the desk)) (1.5)

14

((Mumbles while going over the color scheme)) ((Points back and forth between different colors on the map))

15

((She raises her voice, and her speech becomes clearer when she pronounces the name of the second and subsequent visual areas)) V2 V3 (1.0) V4 ((Puts her hand down)) (3.0)

16

So okay so so one theory is that hahaha ((Silently laughs)) okay V1 (.) so pink to blue is V2 (2.0) ventral then blue to pink (1.0) is V3 ventral and then pink back to blue is V4 ((Points to different colors on the map))

17

((Puts her hand down, and turns toward the newcomer))

18

That’s my best guess based on the data

19 N:

((Tries unsuccessfully to take the floor))

20 O:

Even though it’s very unclear

21 N:

So V1, V2v, V3, and V4 ((Points with the pencil on the map))

The excerpt is composed of Octavia’s three attempts to recognize the color change on the phase map. She starts by identifying the sequence of colors in line 10, placing her index finger on the map and pronouncing the word red with a prolonged “e” sound. The temporal organization of the action allows the practitioner to look at the horizontal color array (located above the brain map) and notice that, to identify the color sequence, the red should be followed by pink and blue colors. The prolonging of the sound “e” in red keeps the idea of “red” as an active, intersubjective phenomenon; it situates the starting point of the color identification in the environment and thus allows for coordination. As Octavia pronounces the word red, Nick joins the enumeration of the colors by contemporarily uttering the word red (line 11); he shows that he can see what

112

Chapter 5

his colleague is indicating. However, Nick’s co-participation in naming the colors ceases shortly thereafter; indicating to Octavia his difficulty in seeing the configuration of the map. After identifying the change from red to pink to blue, Octavia pauses for 3.5 seconds (line 10). While pausing, she keeps her index finger, placed on the brain map, in the steady position. The pointing finger allows for inference to be largely produced as a pattern-matching activity. While proceeding to indicate colors in the higher visual area (line 12), Octavia continues to index the map, premising her listing of the colors with “maybe.” At the same time, her talk becomes quieter, as the moments of silence pervade the activity (e.g., lines 10, 12, 13). The difficulties that characterize Octavia’s effort parallel the larger trend in the field of brain mapping, where the existence and position of higher visual areas is regarded to be more controversial than is the case for the earlier visual areas (whose borders are considered to be an accepted scientific fact). To, thus, identify the borders, the practitioner needs to possess distinct skills while she relies on hypothetical thinking and creativity. The second attempt to point out the change in colors starts in line 14. At this time the old-timer’s voice is so soft that she seems to be mumbling to herself, using speech as a self-regulatory process rather than talking to the newcomer. While mumbling, Octavia points back and forth between different colors on the map, continuing to engage her gesture and talk in the problem solving. In line 15, however, her mumbling turns into clear speech, and her pointing becomes linear. As her semiotic conduct indicates that she finally sees the sequence of color that she knows needs to be identified, the practitioner starts to list the visual areas from V2 up: “V2 V3 (1.0) V4.” Thereafter, she takes her hand away from the computer to mark the momentary accomplishment of the task. In line 16, Octavia engages in the action of listing colors on the map for the third time. By premising her pointing with the phrase “one theory is that” followed by a sotto voce laugh, Octavia expresses the uncertain character of her quest to identify the areas on the brain map. After that, she turns toward the newcomer to check if he saw what she saw and what she had indicated to him (line 17). She then, once again, adds that the identification of the areas is still tentative. This can be seen in line 18, where Octavia explains, “That’s my best guess based on the data,” as she asserts again in line 20, “even though it’s very unclear.”

The Semiotic Mind in the fMRI Laboratory

113

The practitioner’s three attempts highlight the multimodal, yet provisional character of the production of visibility. To transform the messy data into workable evidence, Octavia actively explores the possibilities of the digital materiality and skillfully coordinates multiple semiotic means. This process of “altering” the scientific inscription (i.e., the brain visual), while being attentive to what she sees on the screen and what she knows, allows the practitioner to assemble gradually a support for the proposed layout of the visual areas. She changes the colors of the map while coordinating the prosody and rhythm of her speech with her touching of the phase map. The fMRI visuals have to be made intelligible, with skill and effort, to the newcomer as well as the expert practitioner. In fact, in her attempts to make the organization of the map legible to her colleague, Octavia engages the gesturing hands and talk in her own acts of seeing. Her embodied conduct challenges the very boundary between communicating and problem solving, which, as implied by the fMRI technique, are commonly seen as clearly distinct (as thinking and the mind are understood in terms of the internal processes of a single individual). In line 18, Octavia says, “That’s my best guess based on the data.” The statement indicates her awareness that understanding brain scans is a complex and somewhat tentative process. Whereas practitioners claim that the data “contain” visual areas, in the laboratory they need to suggest “theories” (line 16). These theories importantly concern the fine-tuning between the practitioners and digital technology. Following the old-timer’s interactional efforts, the newcomer starts to distinguish the structure of the data. After two unsuccessful attempts to take the floor (lines 16and 19), Nick steps in and enumerates the areas on the map (line 21). He demonstrates that he sees what the old-timer claims exists on the map, as he is able to continue with the process. This capacity to see as a laboratory member is confirmed in the excerpt that follows. I See What You Are Saying In a further attempt to make the borders between areas clearer, Octavia once again “rotates the color map.” When the new configuration of colors appears on the computer screen, she expresses her appreciation in the form of an aesthetic judgment: “Actually this rainbow looks nicer now, doesn’t it?” After saying that the color configuration looks nice, Octavia briefly

114

Chapter 5

chuckles and then describes it in the following manner: “the V1 (goes from) blue to purple to red.” Nick, however, disagrees at first, stating his preference for the previously explained color configuration: “I like the blue to green the other combination.” To demonstrate that the current configuration of colors has its advantages, Octavia points out the orderliness of the new arrangement. Nick finally acquires the capacity to see. EXCERPT 4 22 O:

Again so you have now it’s yellow orange right? ((Points with her index finger and traces the lines over the borders of the phase map))

23 N:

Mhm

24 O:

Mmmm ((Hesitates))

25

Then it goes out to purple and then back to orange and then out to purple again ((O points with her index finger and traces the lines))

26 N:

I actually I can see that now Hhhh ((Chuckles))

27 O:

You kind of see some of that intermediate (stuff where) it goes from orange to red to purple right? ((Points on the computer screen)

28 N:

(I guess that is better)

29 O:

Yeah okay. Right. ((Takes her hand away from the computer screen, claps over the table and quietly laughs))

30

If you (would) believe me (it would be) very nice haha ((Laughs))

31 N:

No I see it I see what you are saying I see what you are saying

32 O:

Haha ((Laughs)) Okay.

After Octavia has indicated the change in color (lines 22, 25, and 27), in line 31 Nick assures his colleague that, in fact, he can see the form and location of the visual areas she is arguing for: “I actually I can see that now.” Similarly, in line 28 Nick acknowledges that this configuration is more informative: “I guess that is better.” In lines 29 and 32, Octavia express her contentment with her colleague, telling him that he should believe her in order to see (line 30). She jokingly combines the concept of believing and seeing: “If you (would) believe me

The Semiotic Mind in the fMRI Laboratory

115

(it would be) very nice.” Her utterance highlights the idea of seeing as a process situated in the interaction with others (i.e., the newcomer should believe her). Because Octavia’s voice speaks for the laboratory as well as the larger neuroscience community, believing what she says situates seeing in the historically shaped cultural sphere. Yet, that knowledge is also generated as a situated achievement where the practitioners’ recourse to the world is accomplished through the employment of the pointing and working hands. When replying, “No I see it I see what you are saying I see what you are saying” (line 31), Nick first states that he sees, to then repeat his utterance twice more by saying that he sees what the old-timer is saying. The expression indicates that his seeing is shaped by Octavia’s saying. Nick’s expression, “I see what you are saying,” can be read as not simply saying that he understands what is being said but that he understands through seeing. Cognitive neuroscientists use the fMRI technique to infer the nature of mental processes by identifying areas where specific brain activities take place. By localizing cognition in the regions of the brain, scientists draw inferences based on what can be seen. To establish scientific facts about the human mind, fMRI practitioners also work with visual renderings of their data. If that data were given in numeric format, it would be extremely hard, if not impossible, to identify their patterns. Therefore, we can say that fMRI is based on the metaphor of “seeing is believing” (Kutas et al., manuscript in preparation). At the same time, believing is not simply looking, but seeing. To understand the organization of the brain visuals by fine-tuning and adjusting his body to the digital screen, the newcomer learns to see through the old-timer’s “saying.” Attending to his colleague’s body that, through her multimodal semiotic engagements, coordinates digital screens with inscriptions (making them highly mutable) allow the practitioner to acquire the skill to proceed with the task in a contextually appropriate manner. Multimodality and the Semiotic Mind I opened this book with an invitation to think about digital scientific visuals in terms of iconic signs. Once digital visuals are understood as

116

Chapter 5

inseparable from their infrastructure for seeing, the reader/writer—as embodied and practical actor—enters the scene. As visuals open their borders to include the reader/writer, the practices of reading and seeing extend beyond the boundaries of individual brains. This explosion of the mind into the world is importantly semiotic. In the conception of the mind11 that Peirce advocates, the mind is understood as a process accomplished through the use of instruments and communication media. Peirce writes: [T]he psychologists undertake to locate various mental powers in the brain; and above all consider it as quite certain that the faculty of language resides in a certain lobe; but I believe it comes decidedly nearer the truth (though not really true) that language resides in the tongue. In my opinion it is much more true that the thoughts of a living writer are in any printed copy of his book than that they are in his brain. (Peirce, C.P.: 7.364)

Or similarly: A psychologist cuts out a lobe of my brain (nihil animale a me alienum puto) and then, when I find I cannot express myself, he says, “You see, your faculty of language was localized in that lobe.” No doubt it was; and so, if he had filched my inkstand, I should not have been able to continue my discussion until I had got another. Yea, the very thoughts would not come to me. So my faculty of discussion is equally localized in my inkstand. (Peirce, C.P.: 7.366)

Peirce argues that thinking itself emerges through acts of writing. Thus, having material technologies and writing skills is the condition for having certain thoughts. The fact that some thoughts cannot be entertained and developed without being written (they may require a detailed visual representing, as in Octavia’s example where she points out that drawing the chart would allow her to understand the structure of the visual brain areas) suggests that some thoughts cannot even exist if not written or visually rendered: human thinking and semiotic actions are codependent.12 In the fMRI laboratory, practitioners achieve seeing by coordinating a multiplicity of semiotic forms. Parallel to Peirce’s explanation, their seeing is not only relative to processes that take place inside their brains but also instantiated through a practical engagement with signs that are enacted in the laboratory setting. Seeing, thus, comprises drawing charts, providing linguistic accounts, reorganizing the space of interaction, manipulating digital screens, and extensive gesturing. This means that the

The Semiotic Mind in the fMRI Laboratory

117

fMRI practitioners’ seeing is best understood as the temporary upshot of interconnecting relations, semiotically enacted, and developed as part of the work in the laboratory. Octavia, when providing her account of the chart’s history, indicated that she “put together the chart just to help [herself] understand.” Through the process of generating the inscription, Octavia learned about the organization of the cortical areas. Similarly, when indicating the change in color on the phase map, she timed her utterances and used gestures to help herself sort out the patterns of color on the brain scan (and thus locate the boundaries of the visual areas). The practitioners also attended to each other’s gesturing hands and how they rearrange objects in front of the computer screen to figure out how elements from the chart and the brain map are associated. All these actions constitute thinking for the practitioners. They indicate that what are traditionally considered to be internal processes of an individual are intersubjective phenomena, enacted through the coordination of semiotic means, and, thus, available for public scrutiny. Although Octavia’s account of making of the chart clearly resonates with Peirce’s theorizing about the semiotic mind, Peirce, however, does not explicitly discuss thinking that heavily relies on gesture. Whereas he says that the faculty of language can not be confined to the brain but concerns the other parts of the body as well—i.e. the tongue (as exemplified by the reported quote), Peirce does not deal with those communicative elements that are evident in what people are seen to do but not directly articulated through their narrative accounts. Yet, we can certainly extend to the video-recorded conduct Peirce’s proposal that thinking comprises external and internal signs. The interactional analysis of the video suggests that multimodal and embodied engagements are instances of the semiotic mind. This model of the mind resonates with the contemporary positions in distributed cognition (Hutchins, 1995).13 To characterize cognition, Edwin Hutchins uses classical cognitive science terminology, defining culturally and socially distributed processes by adopting the principal metaphor of cognitive science—cognition as computation—previously used only when internal mental processes were described. Hutchins describes cognition in terms of computation as the “propagation of representational state across a variety of media” (Hutchins, 1995: 49):

118

Chapter 5

I propose a broader notion of cognition because I want to preserve a concept of cognition as computation, and I want the sort of computation that cognition is to be as applicable to events that involve the interaction of humans with artefacts and with other humans as it is to events that are entirely internal to individual persons. . . . The actual implementation of many interesting computations is achieved by other than symbolic means. For our purposes, ‘computation’ will be taken, in a broad sense, to refer to the propagation of representational state across representational media. This definition encompasses what we think of as prototypical computations (such as arithmetic operations), as well as a range of other phenomena which I contend are fundamentally computational but which are not covered by narrow view of computation. (Hutchins, 1995: 118)

Developing his argument from the computational metaphor, Hutchins points out that internal cognitive processes are always part of a larger system where even internal representations are still cultural, as they derive from a process generated by a community of practice (Hutchins, 1995: 130). The interaction between Octavia and Nick follows Hutchins’ account of distributed thinking. However, whereas Hutchins’ computational treatment of cognition, paralleled by Latour’s idea of immutable mobiles,14 sees the units of the system as representational, the interactional events in the fMRI laboratory indicate that thinking, while distributed, is not always representational in character. When Octavia’s hands touch the computer screen, they are often indexical signs. As they point, they perform actions (rather than being representations that stand for something). Her working hands direct changes on the computer screen, and her gesturing hands organize the configuration of the brain scan for seeing. As such, they are actions that select and highlight what is of relevance and what should be noticed. To do so they also link multiple semiotic domains, functioning as ways of coordinating brain scans with other inscriptions articulated in the laboratory. Like brain scans, when considered in the interactional engagements, laboratory inscriptions are not only representational. Octavia’s chart, for example, rather than being a vessel that holds information to be further transmitted, is designed as a malleable field, sensitive to the specifics of the situational dynamics and participation framework in which it is used. Because its meaning is realized in concatenation with the other inscriptions and accomplished by way of the working and semiotic hands, a purely representational characterization of the chart would be restrictive.

The Semiotic Mind in the fMRI Laboratory

119

The practitioner’s handling of the chart and its coordination with the other malleable fields generates new configurations that are not reducible to any of the elements that participates in interaction.15 To better comprehend these interactional phenomena Peirce’s account of semiosis proves useful. As is well known, Peirce’s semiotics suggests that meaning is generated through a concatenation of signs where every sign is interpreted by the subsequent one (e.g., Peirce, C.P.: 5.285). Every thought, according to Peirce, is a sign that refers to another sign, in a postponement ad infinitum that never encounters an external thing, if not as a known object or a sign itself. This implies that the meaning of a thought cannot be identified in any precise instance of the interpretative flow but it instead depends on the concatenation of interpretants in the process of semiosis. The meaning [L]ies not in what is actually thought, but in what this thought may be connected with in representation by subsequent thoughts; so that the meaning of a thought is altogether something virtual. . . . At no one instant in my state of mind is there cognition or representation, but in the relation of my states of mind at different instants there is. (Peirce, C.P.: 5.289)

In this sense, as pointed out by Serson: [S]emiosis is the continuous temporal transformation of signs which are never isolated, discrete or atomic units. Signs have no individuality within the process of semiosis. Accordingly, the meaning of a sign cannot be exhausted by any number of actual relations. Semiosis means, therefore, a continuous and temporal transformation or evolution of signs that have no individuality (it is only for the purposes of analysis that signs are ‘individuals’ linked in a chain). Semiosis is best described as a process of fusion, fluidity, ‘merging of part into part.’ (Serson, 1997: 123)

This process is distinct from the proposal of immutable mobiles and the adoption of the computational metaphor when characterizing distributed cognition. Because Peirce’s semiosis is about the mutual co-construction of signs, it leaves open the analytic space for understanding processes of embodied and multimodal semiotic interaction. In other words, Peirce’s semiosis is akin to thinking that is importantly instantiated through gesturing and working hands that enter in coordination with the multitude of inscriptions, instruments, digital technologies, and social actors. This work of interacting hands suggests a semiotic mind that is action based. As indicated by the practice of retinotopic mapping, scientists use

120

Chapter 5

their hands to think with the world—what Alacˇ and Hutchins (2004) call “action as cognition.” Accordingly, seeing is tantamount to a process that is importantly situated in the life world as it is enacted through embodied semiotic acts. In the fMRI laboratory, understanding the brain visuals does involve manipulation of representational signs, but it cannot be fully explained without considering how the practitioners enact volatile semiotic arrangements as they touch digital screens.

6 Materiality of Digital Brains

Rather than dealing with biological matter, fMRI practitioners spend long hours in front of computer screens working with “digital brains.” This allows them to engage with their experimental data in an embodied manner. Because fMRI scans function as Peirce’s iconic signs, they generate effects of similarity (Eco, 1999) not because they simply look like the brains, but because they afford certain actions that are in some ways analogous to the engagement with physical objects. Through this embodied engagement, the scans participate in what practitioners experience as the objects of their practice. This means that when practitioners account for features in experimental data, they deal with phenomenal objects that are simultaneously digital and physical, three-dimensional and two-dimensional, material and mathematical. In other words, the practitioners understand the objects of their practice as hybrid phenomena enacted at the junction between the digital world of technology and the physical world of corporeal action. To capture the laboratory existence of these hybrid phenomenal objects, this chapter focuses on the coordination between interaction and manual work in an fMRI laboratory. I describe how fMRI practitioners coordinate the hands that perform intricate gestural enactments with the hands that are busy using computer input devices. By recalling Michael Lynch’s (1991) discussion of optical and digital topical contextures (see also chapter 1), I examine how the digital brains conserve or acquire their material status during specific instances of work and interaction. In this account, the digital is not only what happens on the other side of the screen but also concerns the bodies in front of the screen. Moreover, it is not only about abstract procedures but also about digits and hands.

122

Chapter 6

Cross-Laboratory Interaction This chapter tackles the interaction between two laboratories: laboratory I (visited in chapters 1 and 2) and laboratory II (featured in chapters 3 and 5). Both laboratories study human visual perception to localize visual processes in specific brain regions and to determine the ways in which visual stimuli are processed there. Dissatisfaction with commercial “off-the-shelf” data analysis tools led each laboratory to design its own software. Although the two software suites are different, they both generate “inflated” and “flattened” cortical maps, and thus allow for more precise identification and location of brain activation (see also chapters 1 and 2). Though the use of such software programs requires specific competencies, the members of the two laboratories emphasize the higher degree of creativity and control over experimental data that such software allows. During my ethnographic study, a Ph.D. student from Paul’s laboratory (i.e., laboratory I), Jane (already encountered at the opening of this book), collaborated with the laboratory visited in the previous chapter (i.e., laboratory II) on a project regarding her doctoral dissertation. Thanks to this collaboration, the two laboratories had a long-awaited opportunity to compare closely the two software programs. The design of this kind of software and the related methodological improvements are the way for laboratories to become obligatory points of passage in the research field (Latour, 1988). The comparison between the two software programs, while allowing for some competition between the laboratories, strengthens the possibility of the laboratories’ impact on the field, largely dominated by commercial software. To accomplish the comparison, Octavia—the bright and patient postdoctoral student encountered in the previous chapter— introduced Jane to the software designed in laboratory II. Jane and Octavia acted as “linkages” between the two laboratories. Whereas contact between the laboratory members is usually confined to scientific conferences and other professional meetings (the members may run into each other when collecting their experimental data at the fMRI center or when serving on various university committees), Jane and Octavia—two young female researchers—crossed the laboratory boundaries to encounter the members of the other laboratory in their work environment. This chapter focuses on two short videotaped excerpts from those encounters. Both excerpts come from an early stage of fMRI data analysis

Materiality of Digital Brains

123

in laboratory I, where what practitioners call functional images (lowresolution visuals that represent cognitive processes) and structural images (high-resolution visuals that reveal the anatomy of the brain; see chapter 2, figure 2.2a, b) have to be aligned with each other. This instance of collaborative work and interaction took place between Octavia, visiting from laboratory II, and the principal investigator of laboratory I, Paul.1 Paul is the charismatic teacher and very accomplished scientist we met in chapter 1 where he discussed with Jane the laboratory procedure of “flattening the cortex.” To carry out the software comparison, Octavia and Paul used the software from laboratory I to process experimental data previously analyzed with laboratory II’s software. The processing of the same data set with the two software programs was of interest to the practitioners inasmuch as it promised to highlight the differences and potential advantages of one software program over the other. The process was of interest to me inasmuch as it promised to recover the ways in which practitioners conceptualize their data as they engage in collaborative achievement of seeing. Seeing the Shearing Artifact During the moments of collaborative work reported here, Octavia and Paul are seated in front of a computer in laboratory I. A small functional brain visual is displayed at the center of the screen (figure 6.1, upper left side). The use of mouse commands allows Paul to alternate the display of functional and structural images (figure 6.1, upper left panel and right-hand panels, respectively). While rapidly switching between the visuals, Paul notices an artifact. The artifact in question is referred to as shearing. In this respect, the visual showing the characteristic distortion is, in brain mappers’ jargon, said to be sheared (from what is called shear strain in classical mechanics), and the activities dedicated to reducing it are called shearing correction.2 The artifact Paul detected concerns the functional image. To picture the distortion, compare the two functional images on the left-hand side of figure 6.1. The upper left panel features a visual with an artifact, whereas the visual in the lower left panel shows how a corrected visual that stands for another layer of the brain may look. In each field a white cross is used to indicate the corresponding spot in the series of brain visuals. The figure

124

Chapter 6

Functional

Structural

Sheared

Corrected Figure 6.1 Functional and structural visuals as they appear on the computer screen

illustrates that the cross in the upper left field is not located on the edge of the brain visual, as is the case for the rest of the visuals. The discrepancy indicates the characteristic type of distortion. Importantly, the distortion in the single functional image displayed on the computer screen indicates that the functional image representing the entire volume of the brain is characterized by it. The practitioners believe that this type of distortion, rather than pertaining to the experimental subject’s brain or to her or his actions during the scanning session (as seen in chapter 4), has to do with the workings of the scanning technology. To deal with the artifact, practitioners direct a group of computational processes to align the functional image to the structural image of the same experimental subject (which is not considered to be distorted). We also see how Paul and Octavia, even though they do not attribute the cause of this artifact to the scanned body (as it was the case in the motion correction activity), work with the problem in an embodied manner. As already discussed in chapter 2, the defining mark of the fMRI culture is its interest in the anatomic specialization of brain regions for processing

Materiality of Digital Brains

125

different types of information. To create brain maps, fMRI researchers project measures of cognitive behavior on the spatial renderings of the human brain. Laboratory I and laboratory II study the visual cortex mapped into complex regions whose different parts are considered to be dedicated to processing of specific visual information. Despite being one of the beststudied parts of the human brain, there are still controversies and disagreements over the existence, general layout, and extent of some of the areas of the visual cortex. Thus, properly aligning the functional data with the corresponding anatomic scans is imperative. Importantly, the shearing artifact that Paul notices was not detected when the other laboratory analyzed the data. Thus, Paul has the opportunity to demonstrate to Octavia how to identify a visual that can be classified as sheared and to show how its correction is achieved by using the software designed in his laboratory.3 As the excerpts from the interaction suggest, the two aspects of the activity are intertwined: The correction itself functions as a part of the classification process, and the classification, oriented toward the practical actions, depends on such processes for its own articulation. It is, therefore, important to decipher what those practical actions entail when the “stuff” that the scientists are dealing with is digital. As Lynch proposes in a study of digital image processing in astronomy, in this context I also will “emphasize implications of digitality that do not turn on the correctness of the visual psychology or epistemology developed in its terms . . . [and] will argue that digitality is an embodied relation with its own forms of practical efficacy” (Lynch, 1991: 62). Certainly, the practitioners’ actions concern a web of relations that go beyond the digital screen. The fact that the shearing artifact was noticed in Paul’s laboratory, whereas it had been overlooked in Octavia’s laboratory, has to do with a chain of historical events and disciplinary-specific practices. Those practices, heavily invested in enhanced ways of looking at the human body (e.g., Cartwright, 1995), shape and are shaped by the design, development, and use of instruments and technology. For example, it is significant that laboratory II’s “predecessor” laboratory had conducted research in human psychology rather than physiology and that the software developed there was originally designed for function localization rather than brain mapping. Software designed for function localization enables the user to compute parts of the process automatically, rather than allowing her or him to engage with some of the components of the data

126

Chapter 6

manipulation process. Frequently, the user looks only at what neuroscientists call regions of interest rather than inspecting scans of the whole brain. It is to be expected that this type of software and the associated techniques for viewing and analyzing fMRI data can lead researchers to overlook certain kinds of artifacts. However, local practices with digital visuals are not simply “shaped by” and do not merely “reflect” the dynamics of the encompassing sociotechnical network. Rather, they are directly involved in the articulation of scientific evidence. Paul proudly reported that the described sessions of collaborative data analysis led laboratory II to modify its software so that the inadequacies in the brain scans can be “more easily spotted.” In other words, the moments of local interaction between the two scientists produced consequences for the configurations of technology and migration of techniques from one laboratory to the other, thus generating potential effects on the future appearance of fMRI visuals. Production of Dynamic Phenomenal Objects The first excerpt opens with Paul using mouse commands to manipulate the computer screen so that serially organized scans can be compared. When he notices that the data were not corrected for shearing in laboratory II, he sets out to display the distortion for others to see. While doing so he ratifies the rational character of the procedures developed and used in his laboratory. In addition to reporting the practitioners’ talk and bodily conduct, the transcript details the effects of the computer screen manipulation. To comprehend how the practitioners understand the object of their practice, of interest is how they temporally and spatially coordinate gestures and talk with the manipulation of the digital screen. In other words, here, in addition to paying close attention to how practitioners’ gesturing hands touch the screen, I attend to the ways these gestures are closely synchronized with the use of the computer mouse. EXCERPT 1 As in the previous three chapters, the practitioners’ utterances follow transcription conventions from Jefferson (2004). In this chapter, to indicate the temporal coordination of the practitioners’ talk with their gestures and the changes on the computer screen, two additional lines are included in

Materiality of Digital Brains

127

the transcripts. The action line, charted above the talk, follows transcription convention from Schegloff (1984): o

Indicates onset movement that ends up as gesture.

a

Indicates acme of gesture, or point of maximum extension.

h

Indicates previously noted occurrence held.

t

Indicates thrust or peak of energy animating gesture.

r

Indicates beginning or retraction of limb involved in gesture.

hm

Indicates that the limb involved in gesture reaches “home position,”

or position from which it departed for gesture. p

Indicates point.

….

Dots indicate extension in time of previously marked action. In the transcript below the talk line, the oblique signs indicate the

temporal change of the digital display: /// ///////

Black oblique signs indicate the display of functional scans. Gray fitted oblique signs indicate the display of structural scans.

Paul is referred to by PA, and Octavia is referred to by OC. 1

PA So (you) usually (0.1) at this point (0.5) / / / / / / / / / / / / / / / / / //////// / / ////// //

2

I usually do a little bit of shearing (0.1) // / / / / / /////////////////// / / / / / / / / / /////////

3

to help get the::(.) / / / / ////// / / ////// / ((The following gesture is a movement of both hands placed as if holding a round object. As the left hand moves up, the right hand moves down. Lexical Affiliate is ‘sheared’.))

4

o…………... you can see //////////

128

5

Chapter 6

…………....a...h………………………………. it’s sort of (0.5)((Paul looks toward Octavia) ///////////////////////////////////

Figure 6.2

6

Figure 6.3

.…………… OC (Interesting) //////////

7 PA

.………………………………………………………. ((Paul turns back toward the screen)) it’s sort of= / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / ///

Figure 6.4

Materiality of Digital Brains

8

129

……... OC =Yeah //////// /

9 PA

……………………........................... it’s actually sh- it’s actually sheared / / ////////////// / / / / / / /////////// / / / / / / /

((The following gesture is a movement of the left hand placed onto the screen with the thumb and index finger an inch or so apart moving along the border of the brain visual. Lexical Affiliate is ‘going up this way’.)) 10

o….p….a…………….h.. So- it’s going up this way ////////////////////////////////// / / /

Figure 6.5

11

Figure 6.6

...r.hm OC Right ((Octavia moves her upper body back, and then returns to the initial position)) / ////// /

Paul starts with the instructions for seeing by saying: “So (you) usually (0.1) at this point (0.5)=I usually do a little bit of shearing (0.1).” By saying “a little bit of shearing” he refers to the procedure of shearing correction. His usage of “usually” and “at this point” evokes and manages the sequential order of the laboratory procedures, while the deictic expression “at this point” also relies upon co-present recipients to “follow” the unfolding sequence in the midst of its scenic details. Notice the interchangeable use

130

Chapter 6

of “I” and “you” in “so (you) usually (0.1) at this point (0.5)” (line 1), and “I usually do a little bit of shearing (0.1)” (line 2), suggesting a reflexive coupling between “what is usually done,” “what should be done,” and “what is being done in the current moment of practice.” Similarly, Paul’s usage of “you” in line 4—“you can see”—does more than represent Paul’s belief that Octavia can already see the distortion. Reminiscent of Octavia’s use of “you can see” in the previous chapter (where she sets up the scene for Nick’s learning to see), Paul’s “you” prepares the terrain for a collaborative management of the visual scene. In and through the techniques used in Paul’s laboratory, Octavia will progressively be subsumed under the generic “you” that can see what he says. One of the significant aspects of Paul’s activity is the rapid alternation of the brain visuals on the computer screen. Over the course of the experiment, series of brain visuals are recorded in which each visual stands for a brain slice. Paul uses mouse commands to alternate rapidly the visuals in a series comparing functional with corresponding structural images. The transcript indicates the display of functional images with a series of obliques while representing structural scans with gray rectangular shapes. When Paul detects a particular nonalignment among visuals, he refers to the shearing artifact. Similar to the motion artifact discussed in chapter 4, the shearing artifact is thus relative to the series of visuals. Likewise, identifying the artifact—ostensively governed by methodological prescriptions— requires a skill that involves a fine coordination of bodies with the digital screen. The excerpt begins with an active search through the series of visuals (lines 1–3). When an appropriate functional image is identified (line 3), Paul utters “you can see it’s sort of” (lines 4–5). Paul then places both of his hands in the space located between the two practitioners and the computer screen, and uses his hands to perform a gestural enactment. First he encloses a portion of the void space as if holding a round object (figure 6.2) and then moves his left hand up while his right hand moves down (figure 6.3). Throughout the activity his two hands conserve the shape in which they were placed at the beginning of the enactment. In line 5, Paul turns toward his addressee, and in line 6 Octavia signals her understanding and the coparticipation in the activity. But while turning toward Octavia, Paul’s hand remains in the position assumed during the previous gesture (figure 6.4). The steady hand position is

Materiality of Digital Brains

131

significant, as it links the enactment carried out in line 5 with its lexical affiliate—“sheared”—which is pronounced only in line 9. As described by Emanuel Schegloff, “the gesture—both its onset and its acme or thrust— precedes the lexical component it depicts” (Schegloff, 1984: 275). Thus, by “projecting” aspects of possible later productions, the gesture makes them available to analysis by a recipient before their actual occurrence (Schegloff, 1984: 267). To understand the organization of the discursive action, however, one needs to also take into account the tight temporal coordination between Paul’s talk and gesture with his alteration of the brain visuals on the computer screen. In line 7, while keeping his left hand in the “frozen” gesture, Paul reinitiates his search through the scan series. He utters the lexical affiliate of the “shearing gesture” only when an appropriate functional image is displayed on the screen (line 9). At this point, he remobilizes his gesturing hand to enact another shearing (line 10). The second gesture is located in the immediate vicinity of the computer screen. Therefore, holding the gesture through lines 5–9 also links the gestural enactment in line 5 with its subsequent elaboration in line 10. The action nicely illustrates how the practitioners, in addition to highlighting and enacting what should be seen on the computer screen (as discussed in chapters 4 and 5), also coordinate the multimodal semiotic enactment with the practical activity at hand. Paul’s linking of the already executed and the future enactment starts with an indexical component (line 10). When he directs his left hand toward the computer screen, Paul briefly points with his index finger to the upper left portion of the brain visual (figure 6.5). After pointing to the visual, he performs the second “shearing gesture.” He places his thumb on the screen (Paul’s pointing is again reminiscent of Octavia’s semiotic actions discussed in the previous chapter), and, while holding his thumb and index finger an inch or so apart, he carefully moves his hand across the screen (figure 6.6). Importantly, during the execution of the enacting gesture, Paul keeps his right hand on the computer mouse to alter the visuals on the screen and orchestrate the entire performance (see, e.g., Whalen, Whalen, & Henderson, 2002). His talk, gestures, and his engagement with the computer function together as techniques for managing perception while they indicate a production of phenomenal objects that can be physically interacted with.

132

Chapter 6

In this respect, the two enacting gestures executed in lines 5 and 10 are particularly important. The gestures, together with talk, gaze, and body orientation, in the context of laboratory practice, superimpose the field of the digital screen and the one inhabited by material bodies. This coordination suggests that the practitioners deal with complex phenomenal objects. Whereas the first gesture (line 5) is performed at a relative distance, but in front of the brain scan, the gesture performed in line 10 (figure 6.5) makes the shearing visible through its physical coupling with the computer screen. The gesture marks the salient feature in the visuals by touching and moving across the computer screen. However, its enactment does more than reveal features of the visual taken in isolation. The gesture, implicated by Paul’s coordination and comparison of the visuals, participates in the enactment of an effect that emerges through its coordination with the work performed on the digital material. Paul gestures to touch the screen just as he replaces the functional scan by the structural one. As the gesture moves across the screen, he alternates the display again, and the functional rendering appears. Through the activity, his gesturing hand links the appearance of the cross (a sign that marks the corresponding spot in the series of brain visuals; see figure 6.1) in the structural rendering with the corresponding sign in the functional rendering. The action allows the two practitioners to experience a phenomenon of motion. Paul’s rapid alteration of the digital visuals, which generates the appearance of motion (so that discrepancies among the visuals can be understood in terms of a unified whole), is coupled with the utterance: “it’s going up this way” (line 10). The utterance is an expression of what proponents of cognitive semantics call fictive motion (e.g., Talmy, 2000); it organizes perception in terms of motion and change, even though the described entity— the fMRI visual—is static. Furthermore, the apparent motion created through the screen manipulation and fictive motion generated through talk are complemented by Paul’s enactment of motion. His gesturing hands in line 5 do not directly “show” a static, distorted round object. Instead, they enact the process of transformation of a round object. Likewise, in line 10 the gestures do more than index a feature in the scan; they participate in the interpretive act of the process of change.4 In other words, as

Materiality of Digital Brains

133

the digitality of brain visuals regards their dynamicity—reflected not only in the character of the fMRI visuals but also in the ways in which they are dealt with—the practitioners experience an object that changes through time. The dynamic quality of the act provides an account of how the distortion came about: Paul generates an explanation of the distortion in terms of a three-dimensional object that is vertically sheared. The power of this explanation does not reside in its representational character but in its capacity to suggest practical engagement. The practitioners I studied know that shearing is a consequence of an abstract computational transformation. Even so, their semiotic enactments indicate concrete, physical transformations. In other words, their practical accounts of the experimental data are given in terms of objects that can be seen, experienced, and dealt with, rather than in terms of causal and abstract explanations. The dynamic embodied account of how the distortion came about implies that the process may be reversed by applying vertical shearing in the opposite direction. The enactment not only performs what happened in the past but also evokes future practical actions. The next section will illustrate in detail the action of applying shearing in the opposite direction. In fact, this is a process through which the practitioners come to understand the work that needs to be done to correct the distortion. The enactments that project future practical actions bring to mind some aspects of Heidegger’s distinction between things “present-at-hand” (Vorhandene) and things “ready-to-hand” (Zuhandene). The two practitioners, by articulating the interface between their bodies and the digital screens, do not represent the sheared features as objective properties that can be passively contemplated. Rather, they enact those features as manipulable things that “subordinate themselves to the manifold assignments of the ‘in-order-to’” (Heidegger, 1962: 98). As lines 5 and 10 indicate, the relevant feature of the fMRI scans is first made manifest through the activity that mimics a direct engagement. The sheared feature becomes visible as a thing encountered through circumspection—a purposeful way of acting without the necessity of having a purpose in mind (see Dreyfus, 1991: 73). However, rather than allowing the fMRI visuals to fade away through such an interaction,5 the enactment of the sheared object generates visibility of the brain scans.

134

Chapter 6

Making the Work Visible Once the artifact is made publicly available through enactment of the “inorder-to” structures that show themselves only in practice, Paul explicates the purification of the experimental data in a step-by-step fashion. This explanation shows how the features of fMRI visuals become perceivable in terms of the work, structured by the disciplinary expectations and the routine practices that have developed for the accomplishment of the task, while being shaped and made publicly available through the interaction between the digital screens and gesture. EXCERPT 2 ((The following gesture consists of two short clockwise motions articulated around wrist. The hand, positioned next to the computer screen, is formed as if holding a round object)). 12 PA

o…..……………t….t….. You rotate (into the into)  

Figure 6.7

Materiality of Digital Brains

13

14

15

16

135

h.……………….. so that you know (.) ((The following gesture first indexes a left portion of the brain visual and then indicates up.)) o..………………p……………………….p (so that) the expanded part is sort of= ((The following gesture is performed as if drawing two axes. Lexical Affiliate is ‘axes’.)) o..a…………a…... in one of the axes ((The following gesture is a grasping motion. Lexical Affiliate is ‘squish’.)) o………..t……. (.) and you squish  

Figure 6.8

136

17

Chapter 6

((The following gesture is a counterclockwise motion articulated around wrist. Lexical Affiliate is ‘rotate back’.)) o……………….t…………... and then (you) rotate back  

Figure 6.9

18

((The following gesture is an abrupt opening of the hand. Lexical Affiliate is “un-squish.”)) o.t…………r….hm (.)and un-squish ((turns toward Octavia))  

Figure 6.10

Materiality of Digital Brains

19

so: rotate 30 degrees, shrink vertically, rotate back, stretch vertically ((Turns towards Octavia and smiles.))

20

OC: Uh-huh. 

137

The organization of the activity described here is shaped by the way in which the software program designed in laboratory I works. Because of the program’s limitations, the series of fMRI visuals cannot be automatically corrected. Instead, a sequence of transformations is required to achieve the correction announced by the gestures in lines 5 and 10 (excerpt 1). Figure 6.11 shows a detail of the computer screen when the shearing artifact was being corrected. The correction requires the user to interact with the scroll bars via mouse commands to direct mathematical transformations on the digital data. The transformations of the digital data are indicated in the visual format; in response to the manipulations of the command menu, the brain scans appear modified. Once the shearing artifact is detected in the brain visuals, four consecutive commands need to be selected. First, the scroll bar labeled “ROTATE BRAIN (deg)” needs to be “moved” toward the right. When this is done, the vertical scroll bar labeled “SCALE BRAIN (percent)” has to be moved down. The third command is the movement of the scroll bar “ROTATE BRAIN (deg)” toward the left. And finally, the horizontal scroll bar “SCALE BRAIN (percent)” has to be moved toward the right. The functions of the commands are illustrated in figure 6.12. To make the distortion more easily perceivable, the figure represents the brain scan in a schematized rectangular, rather than round, form. However, like the brain scan on the screen, the figure is a partial, two-dimensional rendering of the experimental data. Despite the appearance of the two-dimensional (2D) renderings on the computer screen, the practitioners need to work with three-dimensional (3D) data. They need to perform the changes on something that is more akin to a series of 2D visuals than to a single 2D visual. The procedure requires significant expertise. The practitioners have to infer the shape of the 3D object before and after it is digitally modified while looking at a 2D fMRI visual. The design of the computer interface allows them to do so with the scroll bars and labels, which evoke a sense of physically

138

Chapter 6

Figure 6.11 Detail of the computer screen during the shearing correction. (Proportions have been modified to enhance the visibility of the menu.)

manipulating 3D objects; for example, with the command “rotate brain.” Note that these features only evoke objects and actions; the environment in which the interaction between the user and the experimental data takes place is a 2D digital space. Similar to the computer commands and labels are Paul’s linguistic expressions—“rotate” (line 12), “squish” (line 16), “rotate back” (line 17), “un-squish” (line 18), “shrink vertically” and “stretch vertically” (line

Materiality of Digital Brains

139

Figure 6.12 Schematized representation of the four stages of shearing correction.

19)—which evoke a sense of physically manipulating objects. But to engage 3D data in an immediate manner, the practitioners have another resource at hand: the space of gestures and embodied semiotic interaction. In line 12, Paul places his right hand on the computer screen and rotates it clockwise (figure 6.7). Next, in lines 14–15, he explains that the rotation has to position the brain scan so that its expanded part is located in a vertical position (“on one of the axes”). He points briefly toward the brain scan and then upward (line 14). The enactment is followed by a gesture of an ephemeral representation of the two Cartesian axes (line 15). This position is crucial for the next step of the transformation (“squishing”), in which a vertical force is enacted. The gestures in lines 16–18 mime the acts of “squishing” (figure 6.8), “rotating back” (figure 6.9), and “un-squishing” (figure 6.10). After performing the four-step transformation, Paul turns toward Octavia to check that she has been following the action. Paul’s bodily movement and Octavia’s affirmation (“Uh-huh,” line 20) mark the completion of the meaning-making unit. I recorded an analogous activity in the laboratory 2 months earlier, during another instance of data analysis. Paul and two graduate students

140

Chapter 6

(one of whom was Jane) were seated in front of the computer screen when Paul noticed a distortion in the fMRI visual. While orienting toward the computer screen, Paul described how the distortion should be corrected (figures 6. 13 and 6.14):

EXCERPT 3 ((The following gesture is a counterclockwise motion of the right hand articulated around wrist. The hand, positioned in the immediate vicinity of the screen, is formed as if holding a round object. Lexical affiliate is ‘rotate 30 degrees this way’.)) o……….t…………………………….

21 PA

22

(.)If you rotate 30 degrees this way ((The following gesture is a motion articulate with two hands as if holding an object and slightly pushing it from both sides. Lexical Affiliate is ‘squash’.)) o ………..t……... and then squash  

Figure 6.13

Materiality of Digital Brains

23

141

((The following gesture is a clockwise motion articulated with both hands as if holding a round object. Lexical Affiliate is ‘un-rotate’.)) o………….t………… and then un-rotate 

Figure 6.14

24

((The following gesture is an abrupt opening of the right hand. Lexical Affiliate is un-squash.)) o…..t…………. and un-squash 

The semiotic action performed here is quite similar to the one in the previous excerpt, except for the fact that Paul in that instance persistently gestures with one rather than both hands and starts the enactment with a clockwise rotation. The rituality of the performance reveals the tight connection between semiotic enactments and the world of instruments and other technologies. The semiotic enactment is contingent on the design of the computer screen and the “usual” laboratory activities. At the same time, the ritual action highlights the pervasiveness of the performance of physical engagement in the management of perception. Paul reaches toward the computer screen and gestures as if he were holding an object and moving it toward the left/right (lines 21 and 23). Similarly, he enacts the action of “squashing” as if he were squeezing a 3D object (line 22). By enacting the physical manipulation on the fMRI data, the gesture participates in the production of visibility by making the encounter

142

Chapter 6

with the phenomenal object possible. Whereas in lines 5 and 10 (excerpt 1) the two shearing gestures participate in the enactment of phenomenal objects, so that the future work on such objects can be grasped, here the gestures are involved in the performance of the exact steps in such work. Thus, the interaction renders the future work not only graspable but also directly perceivable and publicly shared. Blurring the Boundary between the Digital World and the World of Practice Though the practitioners, to access their data, deal with the digital materiality that generates the effects of manipulability, not everything is available on the computer screen. Paul and Octavia closely look at what is displayed on the screen, yet, Paul’s command selection has consequences not only for the brain slice that they see on the screen but also for the data standing for the whole brain. How do the practitioners deal with what is invisible on the screen? One could speculate that the problem of invisibility could be solved by designing a visual display that would reveal at once all the richness of the data—“it is only a matter of time and technology.” Regarding this point, Paul remarked that if in fact all the layers of the 3D data were displayed (while discussing this point he evoked the image of a “translucent cabbage”), their richness would overwhelm the viewer, who would not be able to deal with the scene. Instead, including the gestural action in the process presents the viewer with a form that evokes the 3D character of the data while still canceling the potentially overwhelming detail.6 As already pointed out in the previous two chapters, when talking about gestural action one should not assume that a single semiotic modality carries meaning on its own. The gesture is rather poor when considered in isolation. The gesturing hand is shaped as if holding a 3D object, but without its interaction with the other semiotic resources, the enacted object remains generic: there is nothing intrinsic to the gesture that defines the object as a brain. Similarly, the brain scans are completely static; they simply exhibit assumed positions before and after their computational manipulation. The linguistic labels displayed on the screen and referenced in Paul’s utterances describe motion, but nothing is moving on the screen. The richness of the process, instead, comes from the interaction.

Materiality of Digital Brains

143

In this regard, the specific location and dynamic qualities of the gesture are significant. Just like Octavia’s gestures described in the previous chapter, the unfolding of Paul’s gestures takes place in close proximity to the computer screen. Paul’s hand, accurately positioned around the visual contours in the brain-slice visual, touches the digital screen while it enacts “rotation” and “squishes” and “squeezes” a round object. Moreover, the sense of time and movement is generated through the gesturing hand and the manipulation of the screen. The process is accompanied by Paul’s verbal labels of the movements as “rotation,” “squishing,” and “squeezing.” Bringing together gesture, talk, and the structure in the environment (what Goodwin calls environmentally coupled gestures; Goodwin, 1995, 2000b) is not simply a cumulative process. The interaction entails acts of selection and delegation. Although the practitioners know that the shearing correction needs to be accomplished through mathematical manipulation, the abstract computational processes, which are not accessible through direct inspection but actually performed on the digital data, are largely delegated to the machine. The practitioners consider the numerical data and mathematical processes to be central and omnipresent in their practice (see Beaulieu, 2002), but they do not access these processes directly while working with “digital brains.” Rather, those invisible layers are indexed by Paul’s gestural enactments, tightly coordinated with the digital visuals and conceived as performances of practical actions. The enactments, which depend on the visuals, generate a sense of physical engagement, even though the visuals cannot be physically manipulated. Nothing like the ordinary sense of the words rotation, squishing, and squashing is performed on the data. This type of activity suggests that the practitioners deal with hybrid forms. Instead of referring to numerical data, digital visuals, or physical objects, they engage with multifaceted phenomena that change through time.7 In other words, the experience allows the practitioners to grasp the nature of the distortion and its correction by encountering something that is simultaneously material and digital, concrete and abstract, human and machine, 3D and 2D, action and object, present and future.8 The enactment of such hybrid objects are part of practical problemsolving in Paul’s laboratory, as well as a component of his “powerful seeing.” Not only does Paul’s reading of the visuals prevail through moments of local interaction; it also participates in propagating the

144

Chapter 6

embodied techniques and the methods of organizing the tools and instruments engaged by his research group. As Paul pointed out, the interaction with Octavia left traces in practices for managing scientific facts beyond the walls of his laboratory. But would this have been possible without Octavia’s willingness to cross the laboratory boundaries and participate in the act of collaborative seeing? Hybridization of the Digital and Physical as Practical Engagement with Experimental Data It has often been pointed out that the involvement with the digital screens transposes the user into an alternative realm—a realm of fiction. To engage with the digital data in an immediate manner, the user needs to “project her/himself on the other side of the screen or pass through the screen to enter the virtual world of fiction” (e.g., Lister et al., 2003; Morse, 1998; see also Turkle, 1995).9 By analogy, it could be claimed that the fMRI practitioners’ semiotic accounts of enacted hybrid objects are imaginary, fictive, or metaphoric renderings instantiated in the public space of action. Several recent studies have dealt with the problem of imagination as a public process accomplished through the coordination of participants’ conduct and the material world of their practice (Murphy, 2004, 2005; Nishizaka, 2000, 2003; Suchman, 2000). In the examples they provide, the participants treat what is imagined and enacted in the interaction space—a loading dock (Murphy, 2004, 2005); submarines’ routes in a computer game (Nishizaka, 2003); or a highway (Suchman, 2000)—as something that has potential for future existence: the loading dock could be constructed, the submarines could be visualized as taking an alternative route on the computer screen, the highway could be lowered and the earth could be removed. In the shearing example, however, the public performance is not conceived as imaginary, nor is it seen as something that could exist outside its local enactment. In fact, we may not even be able to imagine or “see in front of our mind’s eye”10 something that is simultaneously 2D and 3D, abstract and concrete, digital and material. At the same time, and despite the hybrid character of their publicly enacted objects, the practitioners treat them as real and ordinary. While they are tacitly “asked” to partially delegate their work to the machine, the production and understanding of multifaceted phenomenal objects

Materiality of Digital Brains

145

provide the practitioners with a sense of direct engagement with their experimental data. The negotiation and hybridization between the digital and the physical is a practical way to understand and deal with the objects of fMRI practice. Working with fMRI Objects as Hybrid Phenomena The description of the practitioners’ work and interaction shows the laboratory of cognitive neuroscience as a place where an fMRI seeing requires the coordination of multiple semiotic domains with working hands. Paul’s embodied accounts of shearing categorize and make the features of scans visible while they also instruct Octavia on how to spot the artifact by enacting what should be done, can be done, and usually is done in Paul’s laboratory. Such acts of understanding that arise out of experiences with the manipulation of objects and everyday practical dealings explore the border between the virtual and the physical, 2D and 3D, abstract and concrete, past, present and future; they generate emergent properties and distributions of agency. Octavia and Paul talk about, see, and experience a round object that changes and is being changed through time while they interact with menus on a computer screen to display 2D brain scans and to command invisible mathematical transformations on them. Understanding the artifact and its correction by enacting a hybrid object suggests that features of fMRI data (such as the shearing artifact) are not things or well-defined locations represented by the visual, but phenomena that reveal themselves in the linkage between the material world of practice and a variety of visual and embodied semiotic fields. This means that fMRI brain scans and their features cannot be properly described in terms of what they stand for, as the conception of fMRI data as images would imply. Instead, the understanding of fMRI visuals is relative to how they are engaged. By being tied to a local context, digital scientific visuals generate effects of similarity because they can be dealt with in a way that shares some characteristics with how physical objects are manipulated. The materiality of the scientific data—their digital character—allows the practitioners to understand what they are working with as something that is mathematical, while it, at the same time, moves and needs to be rotated, squished, and squashed.

7 Publishing fMRI Visuals

Throughout this book I have been suggesting that fMRI visuals are fields scientists actively engage with, rather than pictures and images they simply look at. By foregrounding the human hands as essential elements of scientific visuals, I have described how, during data analysis sessions, fMRI visuals do not stand for or indicate something but participate in enacting hybrid forms that practitioners experience as objects of their practice. This, however, does not mean that the visual character and the cultural shaping of fMRI evidence are of lesser relevance. In fact, the multimodal approach needs its visual component to be complete, and the visual, despite its embodied and situated character, is undeniably cultural. To highlight the visual and cultural character of fMRI evidence, in this chapter I turn to the process of publishing an fMRI research article. An experimental research article is the end product of laboratory work; it distills the laboratory effort while silently inscribing practices of negotiation with the larger scientific community. These negotiations are clearly manifested in the publishing process. Because the peer review of a scientific article indicates the interface between laboratory work and the community of experts, the process is critical to a comprehensive understanding of scientific practice. Despite the important work that has been done on this aspect of science,1 publishing has still received less attention than it deserves, mainly due to its impenetrability. As discussed in chapter 2, Anne Beaulieu (2002) has shown that brain imaging practitioners undermine the visual character of their evidence. My descriptions of how work is accomplished in the laboratory suggest that fMRI practitioners take advantage of the visual character of their experimental data.2 In this chapter, I show how this visual element permeates the encounter with the community of experts and shapes the

148

Chapter 7

culturally inflected peer-review process. By describing the creative effort of the practitioners, I indicate how reading about fMRI research results regards our senses and how fMRI visuals, at the same time, contain the collectivity. The Visual Character of fMRI Results While conducting my ethnographic research, I was given access to documentation that testifies to production of scientific writing. The writing in question is an experimental research article that was published in Science (Sereno et al., 2001). As I look at its process of submission, revision, and publishing, focusing on the status of brain visuals and their function in the article, I point out the importance of the multimodal character of the experimental data for the scientists, and describe how knowledge, relative to the visuals, is culturally negotiated. The research article under examination was already partially discussed in chapter 2 where the second figure from the article was analyzed (see figure 2.1). The article (Sereno et al., 2001) suggests that, in addition to the well-known retinotopic areas in the early human visual cortex, there are retinotopic maps in the higher-order visual areas as well. The article reports findings of a map, located at the border between the visual and somatosensory cortices, that represents the angle of a remembered target.3 The meaning of the text, and especially the results of the study, are in a significant manner generated through the visuals that feature in the article. The introductory part of the article lays out the historical background, giving the overview of other studies and describing methods that had the goal of defining maps on the cortical surface.4 The text also points out a lack of prior evidence about the specific topic of the article. Next, the focus shifts to a description of the participants and the task of the study (illustrated in figure 2.1). The role of this part of the text is to portray the precisely ordered steps of procedure and the technical equipment used.5 Similar to the introduction, the final section of the article is mainly text-oriented; it is dedicated to placing results in the broader picture, connecting it with larger scientific knowledge, predicting future research, and attempting to suggest links between this study and other domains of cognitive neuroscience (i.e., vision and language).

Publishing fMRI Visuals

149

Yet, importantly, the main part of the text, where the results of the study are presented, is heavily based on the visual material.6 The central section of the article opens by directly indicating figure 2.1 (i.e., Sereno et al. fig. 2) while being mainly dedicated to the discussion of figure 7.1 (plate 2; i.e., Sereno et al. fig. 3). In chapter 2, by analyzing figure 2.1 as a text (rather than a field of action involved in everyday work and interaction of fMRI practitioners), I drew attention to how the constitutive elements of the figure indicate the model reader (Eco, 1979, 1990) as the figure was shaped by the common procedures and expectations of the community of practice. In this chapter, while focusing on the third figure reproduced in the same article (i.e., fig. 3; here figure 7.1), I describe how the negotiations between the authors and the larger community take place. Differently from what Beaulieu’s (2002) informants claim, this negotiation process points out the central role that the visual modality plays in the production of scientific evidence. Review Process Publication of an experimental research article is preceded by a stage where the idea and the design of the experiment are assembled, a long period of meticulous work in the laboratory, and the activity of writing. In addition, a scientific text is also the result of the various revisions based on judgments of journal reviewers before its actual publication. Whereas the former stages tend to leave meager traces in the article’s texture (e.g., KnorrCetina, 1981; Lynch, 1985), the latter phase is usually completely invisible (except for rare occasions when contributions of anonymous reviewers are mentioned in the article’s acknowledgments section). By bringing this often “black-boxed” dimension of the manuscript’s production into the picture, I illustrate the role that the visual modality and its cultural shaping play in the practice of scientific fact production in fMRI. One month after the first submission of the research paper to the journal, the principal investigator (PI) and his colleagues received a letter from the editor informing the authors of a negative decision regarding the publication of the manuscript. The letter, however, also states that it may be possible for the authors to submit a revised version of their manuscript after completing additional experiments and analyses following the referees’ suggestions. The paper was resubmitted after approximately 7 months

150

Chapter 7

and was finally accepted for publication 1 month after the resubmission, to be published during the following winter, about a year after the first submission. But what happened between the first letter from the editor and the letter that informed the authors of the paper’s approval for publication? To tackle this question I analyze the PI’s second letter to the editor, accompanying the resubmission of the manuscript. In the letter, the PI states that, following the reviewers’ comments on the previous version of the paper, new data were collected and new task variations were deployed. It is explained that the paper has been rewritten, and, most importantly for my analysis, all figures have been redone. The letter reveals that the reviewers were dissatisfied with one of the tasks that the participants performed while in the scanner. They also expressed dissatisfaction with the way in which statistical significance was visually rendered. In addition, one reviewer wondered if the spatial maps were the same across different conditions and different scanning days. An accidental reversal in the color scale7 was noticed as well. And, importantly, the reviewers expressed their concern about the activations of the earlier visual areas. These early brain activations are shown in the figure that accompanied the original manuscript submission (figure 7.1a) but are not present in the figure that is a part of the resubmitted, published article (figure 7.1b; see plate 2). The PI’s response is divided into five sections: “Quantitative Measures of Response Magnitude,” “Reliability of Results,” “Task Adequacy,” “Unexplained Activity in Earlier Areas,” and “Small Changes.” Four parts directly deal with the figures. Only the section on the addition of the new experimental task can be seen as not directly dependent on the visuals. Inclusion of a New Task The inclusion of a new experimental task is one of the substantial modifications that appeared in the published article. One of the reviewers (reviewer 3) was dissatisfied with the basic delayed saccade task. In this task, the participant fixates on the center of the image while a peripheral target (a single dot) appears at 15° of eccentricity. Shortly after, a ring of blinking distractors (multiple dots) appears on the screen. Next, the participant has to move her eyes to the remembered location and back. The task seemed too predictable to the reviewer.

(b)

Figure 7.1 (plate 2) Sereno et al. (2001) illustration: (a) originally submitted fig. 3; (b) revised, published fig. 3.

(a)

Publishing fMRI Visuals 151

152

Chapter 7

To respond to the reviewer’s criticism, the researchers added a task where the eccentricity of the next target is unpredictable (i.e., the target can appear in the range of 1° to 15° of eccentricity), and they reported new data, which are very similar to the data from the original task. This modification in the experimental design is apparent in the published figure (figure 7.1b). In comparison with the original version (figure 7.1a), the published figure contains a representation of the extra condition (lower left part of the figure 7.1b). The modification, even though not directly dependent on the visual, is reflected in it. Yet, all the other modifications, which I next discuss, were achieved by directly modifying the original figure. New Annotations In addition to the representation of the new experimental condition, the published figure (figure 7.1b, plate 2) contains another brain rendering not present in the original figure: the representation of statistical significance in the upper right-hand side of the figure. As already noted in chapter 2, the authors of the article proposed an innovative way of visually rendering brain activation, using color to represent at the same time the statistical significance and the existence of the map (see, e.g., the left-hand side of figure 7.1b). One of the reviewers (reviewer 2), however, was unsatisfied with the way in which the color saturation represents the statistical significance: “Reviewer 2 suggested that the representation of the statistical significance was imprecise and difficult to quantify. . . .” (second letter to the editor; emphasis added). The reviewer’s comment shows that colors are perceived as an inadequate way to render the reliability of the measure of levels of neural activity in the cerebral cortex. According to the reviewer, the quality of representation should be strengthened by allowing for precision and ease of quantification. To answer this critique, the authors inserted a significance value (i.e., F-statistic) representation (figure 7.1b, upper right side). If comparing the original submission and the figure in the published version of the article, the data shown in the figure on the left-hand side of the top row (where color is used to represent both the significance [color saturation] and the existence of a map [different colors]) are re-represented on the right side of the figure so that the rendering of the brain contains only a significance

Publishing fMRI Visuals

153

value. The new plot allows the viewer to read by comparing the color plots as already established by the scientific community. The PI writes: “This makes it easy to verify that color saturation is indeed an effective way to portray significance while at the same time demonstrating the existence of the map” (emphasis added). The PI’s response is an example of scientific negotiation between more traditional, widely accepted forms of experimental data presentation (i.e., p values) and the newer ways of visualization characteristic of the practice of retinotopic mapping and largely used by his group. The mediation is achieved by the inclusion of the widely accepted form next to the original plots. The reviewer was not completely satisfied arguing that the colors do not allow for an adequate quantification. The authors provided the additional notation to enable verification and hence aid in the understanding of the figure where color is used to indicate polar angle. The new figure also shows other annotations that have been added to the single-voxel time-course plot (in the upper left-hand corner of the figure). The time-course plot functions as a value judgment on the brain visuals indicating the quality of the fMRI signal (i.e., the technique for generating retinotopic maps produces activation of a portion of a visual field map; that is, parts of the parietal cortex map that represent how different portions of the visual field respond at different points in time). The authors emphasized the cleanliness and the visibility of the data, usually not present when traditional methods are applied (i.e., when practitioners do not use software that records responses to phase-encoded retinal stimulation, as the one designed in the PI’s laboratory). They explain that the plot points out that the phase-encoded paradigm (used by the researchers) produces less “noisy” data, where different phases of activity in nearby voxels can be distinguished: “Here, we were able to show under the more stringent conditions of a phase-encoded mapping paradigm that the parietal cortex map is in fact quite visible, even at the single voxel level.” To attain further specifications, the authors also added the annotation “2%” to the y-coordinate and the annotation “64s” to the x-coordinate of the time-course plot, rendering the value judgment expressed by the plot quantitatively precise. Another new feature of the published figure is a graphical representation of the overlay situated in the lower right-hand side (figure 7.1b). The representation is intended to indicate how different maps were overlaid on

154

Chapter 7

top of each other. This overlaying is part of the answer to the criticism that the spatial maps obtained from the experiment were not identical across different experimental conditions and different scanning days. To reduce this difficulty in comparing visuals, scale bars with significance values and the overlay were added. The authors also pointed out that the overlay of the same maps acquired across different sessions is more precise in comparison with what is usually presented in imaging papers published in the same journal. Finally, figure 7.1b does not contain folded and inflated brain visuals. This omission is a consequence of one reviewer’s complaint that the multiple renderings of the same data set in the original figure were confusing. This modification also shaped the appearance of figure 2.1 (analyzed in chapter 2). Because the authors had to omit folded and inflated brain renderings from figure 7.1b, the visual located in the upper left-hand side of figure 7.1b, converted into a larger-scale view in inflated and folded forms, is represented in figure 2.1. Thus, figure 2.1 preserves the lessconventional type of rendering, which, according to the authors, provides a richer knowledge about the organization of the cortical map, while a “less confusing” and simpler rendering of the data is given in figure 7.1b— the central focus of the article. Activity in Earlier Visual Areas The most striking feature of the new figure is the different color distribution on the fMRI visuals (see plate 2). This aspect of the figure directly touches upon the problem of localization of function. In the second letter to editor, the first author writes: In our previous submission, we rendered figures with the contrast on the color saturation function set too low—especially for the printed page. In the new version we have increased the contrast so that the viewer is not distracted by regions of low significance. Many of the complaints of Reviewer 2 and 3 centered on regions of activity beyond the two that we focus on—which were, the putative human LIP and a more posterior region overlapping V3A. The higher contrast rendering make it easier to distinguish these two regions of high significance and interest.

If the two renderings of the brain in figure 7.1 are compared, it is easy to notice that, whereas the original submission (figure 7.1a) contains color patches all over the cortical surface, the resubmitted, published figure

Publishing fMRI Visuals

155

(figure 7.1b) contains color only at a few well-defined locations. As the passage states, this change is achieved by raising the threshold level. In this way, the figure displays only the most statistically significant values, while the others are veiled away. This remedy allows for more efficient management of the viewer’s gaze over the visual: “the viewer is not distracted by regions of low significance.” The change in the threshold rising is monotonic, which means that there is never a case where less significant data become represented as brighter than the more significant data. It is, however, relevant to notice that the scientific claim and its presentation are made more apt by working directly on the visual renderings of the experimental data, while keeping the quantitative data constant. This manipulation of the visuals has to do with the process of changing the criteria for what is relevant, and what is not; for what has to be shown and explained, and what does not. The rising of the threshold and the modification of the brain visual also speak about expectations held by the viewers familiar with the fMRI culture. To ground the mind in the brain, cognitive neuroscience is accustomed to assigning specific cognitive processes to particular areas of the brain: to find the neuroanatomic substrates of mental activity, cognitive processes are correlated with the well-defined locations in the brain. The reviewers expected to see specific and unique “spots” on the renderings of the brain surface that would unequivocally and clearly prove the point that the higher-level visual areas contain retinotopical mappings. The other, smaller effects, representing attention effects in earlier visual areas— shown in the original version of the figure—are statistically significant, and their existence is scientifically proven (e.g., Gandhi, Heeger, & Boynton, 1999). However, as the PI pointed out during a conversation, “people like less spots”; the existence of different “spots” all around the brain visuals “complicated the story” and, therefore, the spots were erased by raising the threshold. Hence, what the published figure shows are only restricted and well-delimited brain areas that responded in the most statistically significant way to the given stimuli. Seeing as Negotiation Even though the negotiation practices that took place during the manuscript’s revision illustrate complex interrelationships between visual and

156

Chapter 7

quantitatively oriented conventions for data presentation, they cannot be characterized by an irresolvable tension between visuospatial and purely quantitative elements. While the reviewers were oriented toward quantitative aspects of evidence, their comments indicate that their knowledge of these results came substantially from the manuscript’s figures. The reviewers questioned elements (such as the appearance of multiple regions of cortical activity) that were apparent in the figures, not in the text. Once the figures, the central target in the manuscript’s modification process, were adjusted, the paper was accepted for publication. As this book has described, fMRI practitioners work with numerical data visually displayed on computer screens. Though they deal with a large amount of quantitative information, the digital format allows them to think in an embodied manner. Whereas the digital quality of fMRI data is pivotal in allowing practitioners to engage with a large amount of numerical measures (see chapter 6), their engagement with the visual format (as another embodied aspect of their data) is also critical. As suggested by the review process, the visual quality of fMRI data is not irrelevant even when the data enter in contact with the larger community. To explain their reliance on the visual data, my informants evoke a cluster of well-known theories in cognitive science. Beginning with the pioneering work of Larkin and Simon (1987), students of diagrammatic reasoning (e.g., Glasgow, Narayanan, & Chandrasekaran, 1995) have pointed out that the way in which a problem is represented—whether, for example, it explicitly preserves the information about the topological and geometric relations among the components of the problem—is crucial for problem solving (for reviews, see Anderson, Meyer, & Olivier, 2002; Glasgow et al., 1995). Similarly, researchers have pointed out the role of re-coding in cognitive task performance (Kirsh, 1995). In re-coding, an abstract mathematical problem is represented in a form that exploits the specific computational powers of our visual system. The idea has also been called representational re-description (Karmiloff-Smith, 1992) and trading spaces (Clark & Thornton, 1997). Moreover, proponents of cognitive semantics (e.g., Fauconnier & Turner, 2003; Lakoff, 1987) have explained that understanding abstract entities and processes is often based on biological capacities and experiences of functioning in a physical and social environment. One concept from the research domain is the idea of conceptual scaling (Fauconnier & Turner, 2003). Through conceptual scaling, what is invisible or not visual

Publishing fMRI Visuals

157

(as in the case of quantitative data) is transformed into what we can directly experience (i.e., the visuals). A concept is, thus, on human scale with respect to the ease with which it is produced, manipulated, or remembered. The proponents of this idea state that because we use vision as our primary sense, it is more apt for us to analyze, manipulate, and describe complex and abstract entities as visual and spatial. Yet, explaining that visualization deeply shapes scientific activity in terms of the processes related to individual organisms does not entirely do justice to the reality that emerges from the revision processes. Whereas the theories my informants rely upon point out some deep-seated truths, they miss other features of everyday practice. As the description of the publishing process brings to the fore the centrality of the visual (as part of embodied reasoning) in the production of scientific evidence, it also illustrates the inevitable embeddedness of laboratory findings in a particular social, cultural, and historical context, as well as its dependence on the audience. One of the symptomatic points that came up through the revision process was the reviewers’ dissatisfaction with the appearance of “regions of activity beyond the two that the manuscript focused on.” The reviewers’ remarks implied inclinations and expectations to see specific locations of activation, as is customary in fMRI literature. As this negotiation of the “number of cortical activation spots” suggests, the visual is articulated through the filters of social and cultural norms. “Seeing” becomes “clearer,” “less confusing,” and “easier to verify” when it follows the parameters of the scientific culture of which it is part. This means that fMRI visuals cannot escape their relationship with the collectivity. Even if we focus on the production of scientific visuals in the local moments of laboratory practice, we must recognize that these visuals are inevitably defined by their relationship to the larger community and their audience. This shaping of the published visuals by the broader context, nevertheless, does not imply that the authors of these visuals passively follow the impositions of the larger community. Although fMRI practitioners responded to cultural expectations, the publishing process and the form of the published visuals testify their keen participation in the articulation of fMRI data. As they generated fMRI evidence, the practitioners clearly sought to transform the system of expectations and habitual procedures (see also chapter 2) to propose new modes of data presentation. To satisfy

158

Chapter 7

the reviewers, while maintaining the form of the data that they favor, the authors generated a composite figure (as seen in chapter 2, figure 2.1) to be coordinated with the figure discussed by the reviewers (figure 7.1b). Even though the modified figure (figure 7.1b) lost some of its features (seen when comparing figure 7.1a and figure 7.1b), the composite figure shows the resistances exercised by the practitioners. While engaging with the collectivity, the practitioners’ stance, thus, points out the centrality of the visual and embodied way of knowing. In parallel to the arguments of this book, the practitioners strived toward multimodality in fMRI. Handling fMRI Visuals The exploration of multimodality in the practice of publishing a scientific article raises further questions: Can we talk about the multimodality of fMRI visuals once they leave the laboratory space? Once the visuals are published, do they become representations to be passively observed? This chapter has indicated that even when fMRI visuals leave the laboratory, they are still interpreted, negotiated, refashioned, and fought over in ways that point to the importance of (culturally articulated) corporeal experiences. But, whereas these acts of embodied interpretations continue to characterize scientific visuals as they move into classrooms and are read as published data, I expect that with their confinement to paper their multimodality becomes less visible. This, however, is not to legitimate our overlooking of the incarnate aspects of those realities. No matter how much the published images exemplify larger cultural trends, the lived world of hands is where they come from. We must learn how to witness this dimension of scientific visuals as the power of the visuals in fMRI practice is not only about their capacity to indicate the locations of brain activation, but it is also intimately tied to their digitality. It resides in the work where digital screens and computer keyboards intertwine with practitioners’ bodies. In other words, in the age of computers, scientific visuals are importantly multimodal.

8 Conclusion

This book has showed how, by turning attention to the skillful (but habitually overlooked) orchestration of working hands, multimodal semiotic acts, and digital technology, we can get a deeper sense for how science is actually done. Digital video records, augmented with extensive fieldwork, despite its limitations, have provided a way to describe how practical thinking is accomplished, how professional skills are acquired, and how objects of enquiry are experientially understood. The intricacies that constitute such events remain invisible when practice is narrated to interviewers or when scientists report their results to larger audiences. By attending to how fMRI practitioners modify digital displays, how they place their hands onto computer screens, and how they expressively feel what they claim to see on the screen, I have shown how digital scientific visuals gradually become visible to practitioners. As I zoomed in on apprenticeship practices, I saw how newcomers, in front of computers and in coordination with their colleagues, engage their bodies to make sense of experimental data. This approach was also a way to discuss how scientists experientially understand objects of their inquiry. By coordinating multiple semiotic spaces, scientists enact composite and dynamic entities. Rather than simply dealing with what the visuals stand for (somehow untouched by those visuals and the acts of dealing with them) or by exclusively working with what is present on the computer screen, practitioners understand their objects of practice as hybrids that dynamically coordinate those spaces in new, emergent formats. In this sense, though my approach has tended toward the reduction to the essential (the everyday details of science), the inclination was toward complexity. By unpacking aspects of action and interaction in the laboratory, I have observed how practitioners study the

160

Chapter 8

digital brain and the human bodies by semiotically enacting phenomena that blend the computer displays, the practitioners’ own bodies, the mathematical processes executed by the machine, and the physical manipulation of 3D objects. The story that the book has told about fMRI practice, however, is not a comprehensive one. Even though I have carefully grounded my claims in the larger ethnography of the field site, I have worked with only fragments of video-recorded elements of practice. I chose these fragments—as part of the very ordinary events that habitually occur in the laboratory—to explore how brain visuals are dealt with in laboratory work and interaction. In other words, in my use of video, the goal was not to document and represent. I do not even claim that I have wholly described what takes place at any single site where I conducted my ethnographic work. For instance, there are processes in the brain that are definitely central for practical thinking, but I do not analyze those processes in this book. Even though I did discuss how brain processes are studied, I did not provide access to those processes. Instead, I have used videotaped moments of practice as resources for examining engagement with technology, social action, and interaction in the laboratory. My aim was to show how scientists work with digital screens and to suggest how such acts can be studied. In fact, this book has positioned itself in parallel to what it describes. I have shown that fMRI practitioners treat brain visuals as fields for interaction (rather than representations). Just like them, my aim was to propose a study as a field for interaction (rather than generate a representation). Similar to practitioners who engage with the visual to understand how we think, I have been interested in the visual (as well as the audible and tactile) aspects of scientists’ conduct. By scrutinizing how fMRI practitioners go about everyday problems in the laboratory, I have pointed out how they engage with the visually presented data in a multimodal manner. Thanks to the affordance of digital technology, I have repeatedly inspected my records, transcribed the chosen moments, and worked with the stills to delineate the contours of the bodies to make them visible. The approach has left open a possibility for dialogue with the field of study. In this regard, the focus on how the body and digital technology feature in scientific practice not only enhances our understanding of how the body is understood in science but is also a way to engage in a conversation with the field of study.

Conclusion

161

Talking with fMRI Practitioners The attention to multimodal interaction in the real-time description of fMRI practice invites a reflection on theoretical positions in cognitive neuroscience. My goal was not only to discuss how science is actually done but also to provide a way to engage with some of its presuppositions. In this sense, the approach differs from studies of human–computer interaction (HCI) and the classical STS laboratory studies. Even though I have looked at how practitioners interact with computers, my interest has not been in optimizing the process and suggesting design solutions. At the same time, while I have focused primarily on social and interactional aspects of science, I also engaged with current debates in cognitive neuroscience, though I did that from a perspective that is usually not taken in the field. This commitment to cognitive neuroscience from a social science perspective is important not only because it generates an interesting ethnographic experience but also because it engages a field that is progressively entering the territory traditionally occupied by social science. After the early period, during which MRI was closely associated with clinical applications (documented in Joyce, 2008), the turn of the 21st century (and the end of the “decade of the brain”) saw fMRI playing the central role in cognitive neuroscience. My story has tackled this historic period in which scientists extensively used fMRI to understand the cognitive function of the brain. As I finish writing the book, the technique is acquiring relevance in social neuroscience and neuroeconomics, new fields geared toward indicating how social behavior and decision making can be reduced to their neural substrate (accessible through the neuroimaging techniques) (e.g., Cacioppo, Visser, & Pickett, 2005; Gallese, Fadiga, Fogassi, & Rizzolatti, 1996; Kenning & Plassmann, 2005; Rizzolatti, Fadiga, Gallese, & Fogassi, 1996). What does this mean? It clearly shows that researchers are using fMRI to address questions that were traditionally in the hands of social scientists. Should we ignore this move? Should we unanimously accept it?1 I believe that instead, it is time for social science to state its positions on the topic. Though discourse and interaction studies have traditionally engaged with the scientific renderings of human cognition (see, e.g., Coulter, 1989; Suchman, 1987; te Molder & Potter, 2005; van Dijk, 2006), I am convinced that such an engagement is particularly urgent now, when

162

Chapter 8

the proponents of cognitive neuroscience aim to include social interaction in their accounts and models. In this regard, this book has moved to unpack two theoretical assumptions that have dominated the examined historical period in cognitive neuroscience: the idea that human cognition is internal, and the assumption that embodiment concerns a single person and, in particular, her or his brain processes. Handling Digital Brains I opened the book by suggesting that in the laboratory, digital visuals of the brain are part of scientific problem-solving: Rather than understanding them as self-standing mirrors of the world, practitioners think with fMRI visuals. This move has consequences. On one hand, this means that scientific visuals do not represent knowledge and problem solving, but are a part of such processes. On the other hand, the claim is that thinking and problem solving are instantiated beyond the brain of a single individual. To develop an understanding of the mind as a process accomplished through the use of instruments and communication media (as advocated by Peirce), I have turned to its empirical grounding. My goal has been to describe the semiotic mind by focusing on the interacting bodies in everyday scientific practice. How can we reconsider and further extend the concept of the semiotic mind to include the empirical details of multimodal interaction in laboratory practice? I have shown how gesturing and working hands not only populate scientific laboratories but also participate in thinking with digital screens. To comprehend what they observe on the computer screen, fMRI practitioners use their working and thinking hands in conjunction with visually rendered experimental data. Visuals are, thus, not only visual but also multimodal. As part of the semiotic mind, they concern the eyes as well as hands, ears, and moving bodies. The Body in Accomplishment of Science Cognitive neuroscientists use fMRI technique to argue for importance of embodiment in understanding the human mind. The proposed concept of the body, however, is relative to a single part of the body—the brain. First, fMRI technique, while providing possibilities for research, imposes

Conclusion

163

constraints. During scanning sessions, experimental subjects engage with well-designed cognitive tasks while lying still in the bore of the fMRI machine. In other words, to show the mind in the brain, fMRI technology has to keep the rest of the body immobile (as discussed in chapter 3). Published fMRI visuals reflect these technological constraints and procedural arrangements. Second, the technique is framed by the theoretical models in cognitive science that typically do not address social aspects of cognition and communication (e.g., Goodwin, 2003; Sinha & Jensen de López, 2000; Violi, 2008). The attempts to organize the theoretical landscape of embodied cognition (e.g., Wilson, 2002; Ziemke, 2003) reveal this lack. For example, what Zimke (2003) calls (1) structural coupling between agent and environment, (2) historical embodiment as the result of a history of structural coupling, (3) physical embodiment, (4) organismoid embodiment, that is, organismlike bodily form (e.g., humanoid robots), and (5) organismic embodiment of autopoietic, living systems all pertain to human and non-human as well as, in some cases, nonbiological systems. As such, they do not necessarily refer to social practices as their foundational feature. When, on the other hand, embodied views do address the social, their primary concern is the conceptual structures and higher-level knowledge representations realized in the modality-specific areas of the human brain (e.g., Barsalou et al., 2003; Lakoff & Johnson, 1980, 1999). Barsalou and colleagues, for example, define embodied theories of cognition in the following terms: Embodied theories of cognition depart from traditional theories in their assumptions about knowledge representation. In traditional theories, knowledge consists of amodal symbols that redescribe sensory, motor, and introspective states. . . . Conversely, embodied theories represent knowledge as partial simulations of sensory, motor, and introspective states. . . . When an event is experienced originally, the underlying sensory, motor, and introspective states are partially stored. Later, when knowledge of the event becomes relevant in memory, language, and thought, these original states are partially simulated. Thus, remembering an event arises from partially simulating the sensory, motor, and introspective states active at the time. (Barsalou et al., 2003: 44)

As this quote suggests, the embodiment is framed by referring to knowledge representations that simulate some of the sensory, motor, and introspective states that characterized the original experience. Analogously, to describe the mechanisms of embodiment in social cognition, the authors

164

Chapter 8

evoke situated conceptualizations—“packages of inferences, each tailored to current goals and constraints” (Barsalou et al., 2003: 70) —explained in terms of multimodal simulations. They claim that multimodal simulations constitute the core of social knowledge inasmuch as “embodied states represent themselves in these constructs” (Barsalou et al., 2003: 73). For example, [K]nowledge about anger resides in simulations of what anger looks like, how one acts, and how one feels introspectively. On this view, simulations of perception, action, and introspection directly constitute the conceptual content of social knowledge. Knowledge is not a redescription of these states in an amodal language, but is the ability to partially re-enact them. (Barsalou et al., 2003: 73)

Importantly, explanations of social embodiment in terms of internal knowledge representations and their realization in the brain, if not opened to investigations of the real-world meaning-making activities, imply (1) a divide between the body (with the exception of the brain), cognitive (be it embodied or not), and affective states; (2) a reduction of the social world to the stimuli of and their representations in the knowledge system; and (3) a subordination of the body to cognition (i.e., the body is in the function of cognition). In contrast, the practice of fMRI reveals an often cohabiting and sometimes competing, but rather different kind of embodiment. This book has shown how fMRI practitioners, while conducting scanning sessions and analyzing their experimental data, engage in problem solving in a way that is different than the actions executed in the fMRI scanner and cannot be reduced to the internal representations that Barsalou, for example, talks about (even if such representations are conceived as “multimodal simulations”). When working with fMRI visuals, the practitioners extensively use their hands to gesture, handle objects present in the environment, and manipulate digital screens, as fMRI technology (whose constraints erase the rest of the scanned body) requires an embodied engagement. These embodied actions indicate the living body as both material and social phenomenon; a phenomenon that realizes its multiplicity through an active rapport with the world. The upshot of this reversal in perspective indicates that not only are the bodies of scientists comprehended in its coordination with other bodies, technology, and sociocultural elements of the practice, but also the bodies the scientists study are understood this way, too (see, e.g., chapter 4). The

Conclusion

165

object of the scientific study—the human body—is not simply represented to be readily encountered as a self-standing entity, nor could it be fully characterized as socially constructed. Rather, the studied body is enacted through a coordination of multiple bodies and technology. It gains its visibility with respect to a coordination of the multimodal elements of the ongoing action, situated within a sociocultural and historically shaped environment whose properties (varieties of inscriptions, instruments, and other objects) are resources that participate in the scientists’ sense-making. At the same time, the observed body acts upon the scientists, prescribing back to them actions that they need to perform to make sense of what they see on the computer screens. This means that as a part of a multimodal engagement, the body, while being instantiated through its materially and its participation in the specific practice, is not only distributed but also constantly reshaped with the changing contingencies that characterize the situation at hand. By directing attention to the nonverbal communication and various resources that practitioners bring to bear in the accomplishment of action, I have described how the objects of scientific study are being constituted through the researchers’ embodied engagement as multiparty, situated phenomena that indicate their resistances. To make this argument, I have evoked several contemporary theoretical positions in cognitive science. For example, I have talked about distributed cognition (Hutchins, 1995), diagrammatic reasoning (e.g., Anderson, Meyer & Olivier, 2001; Glasgow, Narayanan & Chandrasekaran, 1995), and cognitive semantics and blending theory (Coulson, 2000; Fauconnier & Turner, 2003). I have evoked these theories—disciplinarily closer to the positioning of the practitioners that I studied—not only to explain what the practitioners do but also to develop my argument. Even though this pulling from a variety of sources can be seen as a dangerous crossbreeding, I understand it as a productive way to further ensure the possibility of a dialogue. Openings It may be rightly asked whether the detail of multimodal laboratory conduct sizes up to the questions that have been commonly posed by scholars interested in the social aspects of science and technology. As remarked in the opening chapter, contemporary STS has progressively

166

Chapter 8

broadened to observe interconnections between research institutions, industry, courts, mass media, and funding agencies. This has meant that early methods in laboratory studies have been largely relegated to oblivion. In other words, laboratory studies, though considered to be foundational for the field, have been commonly perceived as a past chapter in the history of the discipline. This book has urged for a reconsideration of this perception, proposing a laboratory study with an emphasis on the lived body in interaction. By adopting techniques and analytic orientation of multimodal conversation analysis and ethomethodology and combining them with insights from Peirce’s semiotics, I have brought the approach of laboratory studies closer to positions advanced by feminist and phenomenological strands in STS. I have provided empirical detail to show not only how the body is “multiple” across settings (Mol, 2002) but also how the body is articulated as multiple in a single moment of scientific practice. In interactional engagement, the body shows itself as belonging to more than one actor while it simultaneously cohabitates several spaces (physical and conceptual). This complexity of the body is a feature of practical problem-solving and conceptualization in the laboratory. Practitioners engage the body as multiple to reason with the digital renderings of the studied body. Yet, my approach has not dealt with popular representations of scientific visuals, their media appearance, nor has it tackled the problem of the translation of the visuals as they move from one location to the other. My story did not include journalists, politicians, health care practitioners, and lawyers. My account ends once fMRI visuals, by way of their publication, leave the laboratory. My analysis does not take into consideration the popular beliefs that go beyond the cognitive neuroscience community and does not look at science dissemination and reception. Does this necessarily mean that I ignored the general public? Can my story say anything to the issues associated with domains such as mass media or jurisprudence? Can it make a difference? Does it have a longer reach? I am convinced that it can; that an interaction-oriented approach can generate larger effects. How? As skillfully pointed out by Joyce (2008) and Dumit (2004), since the late 1990s there has been an overflow of assumptions regarding the power and objective status of brain imaging evidence. It is often the case that lawyers ask courts to take into account fMRI evidence that their clients were subjected to, in order to talk about their

Conclusion

167

clients’ past, mental health, and so forth. In such cases, it is not uncommon to hear them present fMRI visuals as objective truth, contrasting it with the “opinions of some psychiatrists.”2 Once we start to indicate how the practice that involves such technologies actually takes place, these myths (Barthes, 1972), if not swept away, can be significantly moderated. In this sense, rather than combating the myths a posteriori, the descriptions of laboratory practice could help render them innocuous before they are solidified as pervasive realities. By generating openings, the descriptions can help unmask the assumed (or, what can turn into future assumptions), as they allow for different ways of seeing. These openings point to the remarkable creativity of scientific work. To exemplify this, the book has indicated how individual scientists intentionally labor to transform the system of expectations and habitual procedures (see chapter 2) and how by proposing new modes of presenting the visuals, they work toward multimodality (see chapter 7). Thus, I have talked about productivity and frustrations, the beauty of scientific work and the struggle in generating the most useful data. At the same time, I did not intend to praise or glorify scientists. Instead, my goal has been to consider carefully the how of scientific work and the theories that ground it (while being grounded by such work). It is still too early to depict the laboratory as an already known and less interesting field site. This position, often difficult to occupy, helps dissipate two dichotomies: the dichotomy between facts and socially constructed objects, and the dichotomy between the intentions of the author and the readings of the text. First, the objects of scientific practice are not best described in terms of either self-standing realities (that somehow turn into facts) or socially constructed objects. Instead, the multimodal interaction and coordination across multiple social actors and technologies shows them as enactments (see chapters 4 and 5). In this sense, scientists deal with what is at the same time material and social yet does not stably rest in any single body or location. Second, the dichotomy between the empirical author and the text (with its readers) progressively dissolves. As indicated by the analysis of fMRI visuals, a scientific object cannot be adequately understood without considering the practice of its creation and maintenance. To comprehend how fMRI visuals signify, we have to take into account how they are engaged with (in this respect, I have adopted Peirce’s idea of icon to claim that fMRI visuals are iconic signs, as

168

Chapter 8

they are understood through an active visual inspection and embodied engagement). This, however, does not mean that we should understand such an object by referring back to the intentions of its author who, as a rational agent, dominates his or her creation. To deal with the problem, Umberto Eco has talked about limits of interpretation, referring to the author only as a textual strategy (see chapter 2). Once the multimodality of the object is taken in account, however, we must consider empirical actors. Yet, this does not need to be a return to the author as an individual intentionality. Instead, the fMRI laboratory indicates situated and distributed bodies that multimodally engage with technology as part of everyday practice. Despite the constraints that the paper format presents, this book has attempted to reveal traces of the practices through which published scientific visuals have been articulated. Certainly, the general public and the readers of the visuals will live them, making them their own. To provide further power to those readings, however, this book has described some of the ways these visuals were lived through their production.

Notes

Chapter 1 1. For a discussion of a somewhat different idea of cyborg visuality in MRI, see Prasad (2005b). 2. President George H. W. Bush, as part of a larger effort involving the Library of Congress and the National Institute of Mental Health of the National Institutes of Health, proclaimed the 1990s to be the “decade of the brain.” 3. One of the main proponents of this research trend, interested in the involvement of gesture in practical activities and thus attentive to the context in which gestures are deployed, is Charles Goodwin. Goodwin has studied the involvement of gestural action in situations as diverse as courtroom procedures (e.g., 1994), playground activities (e.g., Goodwin, Harness Goodwin, & Yaeger-Dror, 2002), interaction with an aphasic man (e.g., 2000b, 2003a, 2003b), and in scientific activities: archeological excavations (e.g., 2003a, an oceanographic research vessel (1995), and a team of geochemists (1997). 4. This approach is different from the studies of gesture that apply methods akin to the experimental approaches in linguistics and psychology (McNeill, 1995, 2000). 5. See Lynch’s (1991) argument drawing upon the phenomenology of Aaron Gurwitch and Maurice Merlealu-Ponty. 6. UCSD News, October 30, 2000. 7. My doctoral advisor was Ed Hutchins, and my thinking was influenced by his work on distributed systems (Hutchins, 1995). 8. “[A] stratified hierarchy of meaningful structures in terms of which twitches, winks, fake-winks, parodies, rehearsals of parodies are produced, perceived, and interpreted, and without which they would not (not even the zero-form twitches, which, as a cultural category, are as much nonwinks as winks are nontwitches) in fact exist, no matter what anyone did or didn’t do with his eyelids” (Geertz, 1973: 5–6).

170

Notes

9. In Scientific Practice and Ordinary Action (Lynch, 1993) (following his original laboratory study Art and Artifact in Laboratory Science; Lynch, 1985a), Michael Lynch, reflecting on the current situation, observes: Rather than undertake the difficult, time-consuming, and epistemologically suspect tasks of ethnography, many sociologists of science have preferred to take refuge in offices and libraries. There they can act as if they are observing “science in action” while engaging in more respectable academic pursuits: sifting through historical archives and secondary sources, composing scholarly syntheses of the diverse literatures in the sociology of science and related areas, and performing close textual analyses. (Lynch, 1993: 105)

10. For example, in his book on positron emission tomography (PET), Joseph Dumit (2004) has used cultural studies and historical and ethnographic methods to study how visual brain representations are generated by scientists, how they are presented in the media, perceived as facts by the general audience, and put to persuasive use in courtrooms, doctors’ offices, and before Congress. In doing so, Dumit points out how “PET brain images of mind and personhood . . . make claims on us because they portray kinds of brains” (Dumit, 2004: 4, 5). Similarly, Kelly Joyce, in her book on MRI in clinical medicine, examines “the cultural, political, and economic factors that contribute to the making of MRI anatomical images as a cultural icon” ( Joyce, 2008: 2). Particularly prominent is Joyce’s analysis of how narratives from popular culture and media accounts obscure the decisions, values, human relations, institutional context, and economic structure that stand behind the transparency inscribed into MRI representations. 11. Kelly Joyce (2008) and Joseph Dumit (2004), for example, provide the reader with a sense of how activities around scans are organized, describing general procedures and stages in data acquisition, analysis, and publication, while interweaving such accounts with excerpts of discourse taken from their interviews with scientists, physicians, and technologists. Though successfully generating overviews of MRI and positron emission tomography (PET) and engaging with questions about objectivity, standardization, and generalization, these accounts, however, do not provide moment-by-moment descriptions of the activities through which scientists engage scanning technologies, digital computers, and each other to accomplish their everyday activities. 12. My interest in Peirce’s semiotics was acquired through my undergraduate and doctoral adviser, Umberto Eco. Chapter 2 1. For Peirce, signs are relations composed of other relations and linked in series. In this sense, Peirce spoke of the sign as being triadic while calling the process of continuous development of signs in other signs “semiosis.” The triadic sign in Peirce’s semiotics is composed of a representamen, interpretant, and an object. A representamen, or sign vehicle, broadly corresponds with the better known Saussurian

Notes

171

signifier and can be understood as the form of the sign, and the interpretant can be defined as the sense of the sign roughly analogous to the Saussurian signified. 2. Based on the triadic idea of the sign, Peirce generated multiple typologies of signs (in his 1903 account of semiotics he suggested 10 classes of signs, but announced 66 classes of signs in his final typology), the distinction between icon, index, and symbol being the best known. 3. Roland Barthes (1981) talked about a “guarantee of the co-presence.” 4. To deal with primary iconicity, or the iconic character of perception, Eco follows Gibson’s (1966) suggestion of conformity where the idea of likeness depends on the concordance between the perceptual system and the invariants of the informative stimulus (Eco, 1999:105). 5. I will back up this claim with examples from laboratory practice in chapter 6. 6. This figure is chosen because the publishing process of the Sereno et al. article will be examined in chapter 7. 7. The retinotopic maps are of central interest as two of the laboratories in which this study took place research human visual cortex. 8. The practice of retinotopic mapping will be described in more detail in chapter 5. 9. As already mentioned, even though scientists often use the word image when talking about fMRI brain visuals, they define such visuals as maps. The idea of the map, usually associated with a graphical representation of a geographic space, is a depiction of the relationships between elements within the rendered space. As remarked by Roger Tootell and his colleagues (neuroscientists who use fMRI to study visual cortex), fMRI practitioners see themselves as geographers: “In daily life, one would normally begin exploring a new geographical region only after first consulting the most accurate available map of that region. Analogously, to make progress exploring human visual cortex, we first need to have an accurate map of the areas which exist there” (Tootell et al., 1998: 175). 10. Another metabolic imaging technique is PET. 11. The rest of the figures and the way in which they function in coordination with the text of the article will be discussed in chapter 7. 12. The dotted circles and yellow crosses indicate the brain activations as the zones of central interest. Text labels, on the other hand, guide the viewer’s comprehension of what is represented as they indicate how the activations are spatially related to the sulci. The function of the larger labels, written in capital letters and positioned above each brain visual, is to categorize and provide names for the parts of the

172

Notes

human brain’s right hemisphere—lateral, superior, posterior, medial-posterior, and medial—represented in the figure. Differently, the function of the smaller labels, written in lowercase letters and inscribed over the brain visuals, is to name the landmark structures on the cortical geography. Such labels are linked with small arrows or curved over the territory to coordinate the name with the representation of the sulci. 13. For a similar discussion on ultrasound imaging, see Hartouni (1991). 14. This point will be discussed in the following section. 15. The caption published with figure 2.1 indicates that red stands for the neuronal activations that correlate with the appearance of the stimuli in the upper left visual field, blue stands for the left horizontal visual field, and green stands for the lower left visual field. 16. When using the Talairach and Tournoux coordinate system, the researchers identify three axes. They are the anterior and posterior commissure—two regions of the brain that are believed to be invariant across individuals—and the interhemispheric fissure (the structure that separates the two hemispheres), used to align the specific brain representation to the Talairach axes. When applying the stereotaxic coordinates as the labeling system, scientists mark the location of an activation by identifying the distance (in millimeters) from the intersection of all axes at the midpoint of the anterior commissure to name any cortical location. 17. This positioning toward the different labeling system is also visible in the spatial organization of the figure as well as its caption. The figure is organized so that the superimages are placed on the top and in the center. Similarly, the figure caption is almost entirely dedicated to the superimages, mentioning the brains marked with the stereotaxic coordinates only in its last line. 18. I further discuss this conflict and reconciliation in chapter 7, when I describe the publishing process of the article in which figure 2.1 appears. 19. To this original position, Eco’s latest writings add a phenomenological bend (e.g., Eco, 2004), as is detectable in his perfume factory analogy. 20. As Sun-Joo Shin explains: “It is obvious that Peirce’s diagram is not restricted to a figure (such as those in Euclidean proofs), but is rather closer to a unit of a system equipped with representational import and its own transformation rules” (Shin, 2002: 21). 21. Peirce, as a founder of modern symbolic logic, is famous for inventing, in addition to his symbolic logical system, a nonsymbolic representational system: existential graphs. With existential graphs, Peirce tried to generate transformation rules for diagrams in the same way that there are rules of algebra specifying which transformations of symbols are permitted and which are not.

Notes

173

22. The claim parallels Peirce’s pragmatist view, according to which diagrams, to be understood, have to be immersed in the situation at hand (e.g., Peirce, C.P.: 3.419). 23. This idea will be specifically developed in chapter 6. 24. The centrality of the visual character of fMRI brain visuals will be discussed in chapter 7. Chapter 3 1. Throughout the book I use the terms newcomer and old-timer, adopted from Jean Lave and Etienne Wenger’s discussion of communities of practice (where people learn together by participating in a common endeavor), to refer to recent laboratory members and those who joined the community in a more distant time. 2. This is discussed more extensively in the chapter that follows. 3. Following the tradition of Conversation Analysis (Sacks, 1995; Sacks, Schegloff, & Jefferson, 1974), dot inside parentheses is used to indicate a micropause. 4. (()) Double parentheses contain the transcriber’s comments and extralinguistic information (e.g., about gesture, bodily movements, and actions). 5. The question mark is not a punctuation sign but an indicator of the rising intonation of Sara’s voice. 6. This issue is specifically addressed in chapter 5. 7. For examples of such scans, see figure 2.2. 8. Functional scans are susceptible to distortion or research artifacts, which are seen by practitioners as problems often related to the hardware or to the subject’s movement. I discuss the practical ways in which fMRI researchers deal with artifacts in chapters 4 and 6. Chapter 4 1. As explored in the introductory chapter. 2. The axial sections are vertical sections made from the front to the back of the brain. The sagittal sections are vertical sections ordered from the center of the brain out to the side. The coronal or horizontal sections are displayed from the top to the bottom of the brain. 3. The following chapter discusses how the old-timer learns to see through the use of multimodal semiotic means. 4. To understand the complexities of this reading activity, Peirce’s conception of the sign again proves useful. As I pointed out in the introductory chapter, Peirce’s

174

Notes

semiotics is distinguished by an understanding of the sign that includes what the sign refers to as its necessary component. Peirce’s sign is not only about a relationship between expression and content but also contains the referent as a part of the sign. Peirce differentiates between the Immediate Object and the Dynamical Object: “[W]e have to distinguish the Immediate Object, which is the Object as the Sign itself represents it, and whose Being is thus dependent upon the Representation of it in the Sign, from the Dynamical Object, which is the Reality which by some means contrives to determine the Sign to its Representation” (Peirce, C.P.: 4.536). The Immediate Object is the object that is a component of the sign, shaped by the sign. The Dynamical Object, on the other hand, shapes the object of the sign but can only be accessed through the sign. In other words, the worldly objects, while only accessible through the sign, are part of the sign that they constrain, placing limits on its successful signification. Chapter 5 1. The variations in fMRI analysis often relate to the ways in which data are processed to achieve generalized results. According to one of my informants, typically there are two statistical stages. First, practitioners examine each subject’s data set, performing statistical analysis so that the data across various experimental conditions can be generalized. Then, the data from all the subjects are translated into the same coordinate system, allowing the second statistical analysis across the group to be carried out. However, none of the three laboratories that I studied uses this “typical” procedure consistently. How the statistical analysis is performed depends on the investigated issue, and is often, via the educational history of the laboratory’s principal investigator, related to the disciplinary divisions inside the field of cognitive neuroscience. The laboratory discussed in the previous chapter studies patient population and thus modifies its method of procedure depending on the analyzed data set. If the data set was obtained from experimental subjects with no neurologic problems, the researchers often analyze separately only a few subjects’ data and then use a group analysis to report the results. If, however, the data set derives from patients with brain damage, the researchers analyze the data from every subject separately because the brain structures are not seen as being in the same place as in the “normal brain.” To explain this phenomenon, the researchers say that the “brains do not normalize easily.” 2. While the approach permits the scientists to report with more accuracy the regional boundaries of the specific brain areas, it also allows them to leave out the second step of averaging and spatial normalization (avoiding the problem of inaccurate overlap of the data from multiple subjects due to the intersubject variability). 3. Latour’s intention is to avoid the use of large-scale entities to explain science and technology (Latour, 1990: 56–57).

Notes

175

4. In Pandora’s Hope, Latour provides a nice example of a pedologist working in the field, whereby taking a piece of dirt and moving it from one hand to the other, the pedologist allows the dirt to move from thing to sign (Latour, 1999: 49). 5. Latour does talk about recombination, reshuffling, and superimposition, but these terms miss the fluid and multimodal merges that characterize embodied engagement with digital screens. In his thinking about immutable mobiles, Latour’s theorizing is influenced by de Saussure’s approach; when describing laboratory work of a pedologist, he explains that “At the card table, with so many trumps in hand, every scientist becomes a structuralist” (Latour, 1999: 38). 6. In a similar manner, the laboratory’s manual—the document that contains information regarding the laboratory’s procedures and is intended for their future transmission—explicitly refers to the chart: Look at Octavia’s chart to define visual areas. This is best for expanding rings and rotating wedges. Define the visual areas using the expanding rings and then delineate V1, V2, V3, etc. using the horizontal meridians and data from the rotating wedges. You may want to stop short of V7 or V8 because they are not well defined.

7. The reiterative multimodal engagements with the paper-based graphical inscription are not qualitatively different from the actions that take place when digital visuals are engaged. Yet, when the digitality of the visuals is added to the picture, the process of thinking with hands and eyes is even more prominent. 8. Fauconnier and Turner (2003). 9. The colors inscribed on the fMRI visuals do not mimic the world that they stand for. As already discussed in chapter 2, these colors are not what you would see if you were able to open a skull and look at the brain of a person involved in a cognitive task. Rather, the colors, arbitrarily selected, are a way to represent the changes in neuronal processes through time. In other words, fMRI practitioners engage with temporal and invisible events as they are translated into spatial and visual renderings. Far from being irrelevant for the practice of science, the function of the colors aids comprehension. 10. In his discussion of pragmaticism, Peirce writes: Abduction is the process of forming an explanatory hypothesis. It is the only logical operation which introduces any new idea; for induction does nothing but determine a value, and deduction merely evolves the necessary consequences of a pure hypothesis. Deduction proves that something must be; Induction shows that something actually is operative; Abduction merely suggests that something may be. Its only justification is that from its suggestion deduction can draw a prediction which can be tested by induction, and that, if we are ever to learn anything or to understand phenomena at all, it must be by abduction that this is to be brought about. No reason whatsoever can be given for it, as far as I can discover; and it needs no reason, since it merely offers suggestions. (Peirce, CP 5.171)

176

Notes

A man must be downright crazy to deny that science has made many true discoveries. But every single item of scientific theory which stands established today has been due to Abduction (Pierce, CP 5.172) However man may have acquired his faculty of divining the ways of Nature, it has certainly not been by a self-controlled and critical logic. Even now he cannot give any exact reason for his best guesses. It appears to me that the clearest statement we can make of the logical situation—the freest from all questionable admixture—is to say that man has a certain Insight, not strong enough to be oftener right than wrong, but strong enough not to be overwhelmingly more often wrong than right, into the Thirdnesses, the general elements, of Nature. An Insight, I call it, because it is to be referred to the same general class of operations to which Perceptive Judgments belong. (Peirce, CP: 5.173)

For a discussion of Peirce’s abduction and semiotics, see Eco, 1986, section I.11. 11. Peirce’s conception of the mind is not limited to the psychological mind (see Peirce, C.P.: 4.550.1). 12. See Peter Skagestad (1996) who reported the quote (also cited in Ransdell, 2003). For an exciting exploration of Peirce’s semiotics for the understanding of human thought and its evolution, see Terrence Deacon (1997). 13. Distributed cognition in the studies of science and technology has, for example, been discussed by Ronald Giere and Barton Moffatt (2003), Karin Knorr-Cetina (1999), Bruno Latour (1999), Chandra Murkeji (2009), and Nancy Nersessian (Nersessian et al., 2003). 14. In his discussion of immutable mobiles, Latour (1986), in fact, cites Hutchins (1980), as well as Cole and Scribner (1974); and Lave, Murtaugh, and de la Rocha (1984) function as allies in strengthening his argument. 15. During Olga and Nina’s interaction, described in the previous chapter, we followed semiotic enactments that clearly contain representational elements. Nevertheless, they also have experiential qualities and participate in the dynamic, compositional arrangements that cannot be referred back to stable objects and locations in the world. This hybrid character of research objects semiotically enacted in the fMRI laboratories is further discussed in the chapter that follows. Chapter 6 1. Paul was also one of the main designers of the software used in his laboratory. 2. To picture the distortion, one can consult its schematized rendering in the upper portion of figure 6.1. 3. Kelly Joyce pointed out that in order to see artifacts in MRI images, radiologists need to “train their sight over time ‘through’ . . . interaction with other physicians, texts, and machines” ( Joyce, 2005: 449).

Notes

177

4. Even though not explicitly discussed, a similar phenomenon was seen in chapter 4 where the practitioners discussed the movement of the experimental subject while manipulating static scans on the computer screen and enacting motion in the shared space of interaction. 5. The concept of “ready-to-hand” is frequently referred to in the literature on interaction with technology. See, for example, Dourish (2001: 109) and Suchman (1987: 53). 6. This certainly does not deny that an expert has no need to resort to such public resources. But it does point out the public grounding of expertise. 7. The importance of hybrid semantic structures in human understanding has been pointed out by conceptual integration theory (Coulson, 2000; Fauconnier & Turner, 2003). Rather than accounting for stable knowledge structures represented in longterm memory, the theory identifies systematic projections of language, imagery, and inferential form to model the dynamic evolution of speakers’ online representations (Coulson & Oakley, 2000). The semiotic enactments of shearing artifact share some features with the process of conceptual integration. The hybrid semiotic construct is at the same time about the brain visual that indexes the experimental data and the embodied multimodal performances that make such data and their manipulations publicly available. Mutual reference and integration provides access to the problem solution in terms of ordinary actions. The practitioners deal with the experimental data and their correction by performing round objects that share some visual features with the digital image and are being rotated, squished, and squashed, even though neither the digital data nor the fMRI images can be subjected to those operations. The process of hybridization is largely generated in the social space of action. Rather than being exclusively mental phenomena internal to single individuals, as it is largely the case in the usual applications of the conceptual integration theory, important elements of the production of visibility involve integrations between multiple semiotic fields generated through the use of gesture, digital visuals, and body orientation as features of the practical problem-solving. For a related account, in which only one of the semantic spaces is a material phenomenon, see Hutchins (2005). 8. Ochs et al. (1996) describe somewhat similar phenomenon—a blending between the scientists and the objects of their study—as referentially ambiguous entities. 9. For an example of the popular rendering of the idea, see the 1982 Walt Disney Productions science fiction film Tron directed by Steven Lisberger. 10. Nishizaka (2003) contains a comprehensive discussion and critique of the idea of mental image.

178

Notes

Chapter 7 1. Karin Knorr-Cetina’s laboratory study (1981) describes how a scientific article, while reporting on laboratory research, filters what happens in the laboratory. Charles Bazerman (1988) studies the genre of the experimental research article as a cultural form and traces its emergence, evolution, and stabilization. 2. For a discussion of the centrality of the visual in MRI, see also Joyce (2005, 2008). 3. This is accomplished by asking participants to move their gaze over the visual field to the location where a previously shown, and now remembered, target (a small white square on a black background) was presented. 4. Latour (1987) talks about gathering resources and allies to make dissent impossible. 5. For a discussion on the distinction between the normative sequence of a project’s course as described in the methods section of a paper and the actual performance of actions by members in the project, see Lynch (1985a: 57). 6. If we consider only the central part of the text, where the results are presented, we find that almost one half of sentences directly refer to figures. 7. Red, instead of being used to represent the upper left visual field, is used to represent the lower left visual field; likewise, green, instead of being used for the lower left visual field, is used for the upper left visual field. Chapter 8 1. This, to my surprise, has often been done in social science writings when referring to the mirror neurons proposal (e.g., Gallese et al., 1996; Rizzolatti et al., 1996). 2. See, for example, the case of John Allen Muhhamad, reported by BBC News on November 10, 2009.

References

Akrich, Madeleine, and Bruno Latour. 1992. A Summary of a Convenient Vocabulary for the Semiotics of Human and Nonhuman Assemblies. In Shaping Technology/ Building Society: Studies in Sociotechnical Change, ed. Wiebe E. Bijker and John Law. Cambridge, MA: MIT Press, 259–264. Alacˇ, Morana, and Edwin Hutchins. 2004. I See What You Are Saying: Action as Cognition in fMRI Brain Mapping Practice. Journal of Cognition and Culture 4 (3):629–661. Allman, John. 1999. Evolving Brains. New York: Scientific American Library. Amsterdamska, Olga. 2008. Practices, People, and Places. In The Handbook of Science and Technology Studies. 3rd ed., ed. Edward J. Hackett, Olga Amsterdamska, Lynch Michael, and Judy Waycman. Cambridge, MA: MIT Press, 205–210. Anderson, Michael, Bernd Meyer, and Patrick Olivier, eds. 2001. Diagrammatic Representation and Reasoning. London: Springer-Verlag. Austin, John L. 1962. How to Do Things with Words. Oxford: Clarendon. Bakhtin, Mikhail. 1981. The Dialogical Imagination. Austin: University of Texas Press. Barad, Karen. 2003. Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter. Signs: Journal of Women in Culture and Society 28: 801–831. Barad, Karen. 2007. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Durham, NC: Duke University Press. Barthes, Roland. 1971. Mythologies. London: Paladin. Barthes, Roland. 1964. Rhétorique de l’image. Communications 4:40–51. Barthes, Roland. 19801. Camera Lucida: Reflections on Photograpraphy. Paris: Gallimard, and Seuil.

180

References

Barsalou, Lawrence, Niedenthal, Paula, Barbey, Aron, and Jennifer Ruppert. 2003. Social Embodiment. In The Psychology of Learning and Motivation, Volume 43, ed. Brian Ross. San Diego: Academic Press, 43–92. Bazerman, Charles. 1988. Shaping Written Knowledge: The Genre and Activity of the Experimental Article in Science. Madison: University of Wisconsin Press. Beaulieu, Anne. 2002. Images are Not the (Only) Truth: Brain Mapping, Visual Knowledge, and Iconoclasm. Science, Technology & Human Values 27:53–86. Beaulieu, Anne. 2004. From Brain Bank to Database: The Informational Turn in the Study of the Brain. Studies in History and Philosophy of Biological and Biomedical Sciences 35:367–390. Benveniste, Emile. 1971. Problems in General Linguistics. Coral Gables, FL: University of Miami Press. Brett, Matthew, Ingrid Johnsrude, and Adrian Owen. 2002. The Problem of Functional Localization in the Human Brain. Neuroscience 3:243–249. Butler, Judith. 1993. Bodies that Matter: On the Discursive Limits of “Sex”. London: Routledge. Cacioppo, John T., Visser, Penny S., and Cynthia L. Pickett, eds. 2005. Social Neuroscience: People Thinking about Thinking People. Cambridge, MA: MIT Press. Cartwright, Lisa. 1995. Screening the Body: Tracing Medicine’s Visual Culture. Minneapolis: University of Minnesota Press. Cartwright, Lisa. 2008. Moral Spectatorship: Technologies of Voice and Affect in Postwar Representations of the Child. Durham, NC: Duke University Press. Cicourel, Aaron. 1987. Interpretation of Communicative Context: Examples from Medical Encounters. Social Psychology Quarterly 50:217–226. Clark, Andy, and Chris Thorton. 1997. Trading Spaces: Computation, Representation, and the Limits of Uninformed Learning. Behavioral and Brain Sciences 20:57–90. Cole, Michael. 2003. Culture and Cognitive Science. Outlines (Copenhagen) 5 (1). Cole, Michael, and Sylvia Scribner, eds. 1974. Culture and Thought: A Psychological Introduction. New York: Wiley. Collins, H. M. 1974. TEA Set—Tacit Knowledge and Scientific Networks. Science Studies 4 (2):165–185. Collins, H. M. 1985. Changing Order: Replication and Induction in Scientific Practice. London: Sage.

References

181

Collins, H. M., G. H. de Vrijes, and W. E. Bijker. 1997. Ways of Going On: An Analysis of Skill Applied to Medical Practice. Science, Technology & Human Values 22 (3):267–285. Coulson, Seana. 2000. Semantic Leaps: Frame-shifting and Conceptual Blending in Meaning Construction. New York: Cambridge University Press. Coulson, Seana, and Todd Oakley. 2000. Blending Basics. Cognitive Linguistics 11 (3–4):175–196. Coulter, Jeff. 1989. Mind in Action. Oxford: Polity. Damasio, Antonio. 1994. Descartes’ Error: Emotion, Reason, and the Human Brain. New York: Avon. Daston, Lorraine, and Peter Galison. 2007. Objectivity. Cambridge, MA: MIT Press. Deacon, Terrence. 1997. Symbolic Species: The Coevolution of Language and the Brain. New York: Norton. de Saussure, Ferdinand. 1983. Course in General Linguistics. La Salle, IL: Open Court. Doing, Park. 2004. “Lab Hands” and the “Scarlet O”: Epistemic Politics and (Scientific) Labor. Social Studies of Science 34 (3):299–323. Doing, Park. 2008. Give Me a Laboratory and I Will Raise a Discipline. In The Handbook of Science and Technology Studies. 3rd ed., ed. Edward J. Hackett, Olga Amsterdamska, Lynch Michael, and Judy Waycman. Cambridge, MA: MIT Press, 277–295. Doing, Park. 2009. Velvet Revolution at the Synchrotron: Biology, Physics, and Change in Science. Cambridge, MA: MIT Press. Dourish, Paul. 2001. Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press. Dreyfus, Hubert L. 1991. Being-in-the-world: A Commentary on Heidegger’s Being and Time, Division I. Cambridge, MA: MIT Press. Dumit, Joseph. 2004. Picturing Personhood: Brain Scans and Biomedical Identity. Princeton, NJ: Princeton University Press. Eco, Umberto. 1976. A Theory of Semiotics. Bloomington: Indiana University Press. Eco, Umberto. 1979. Lector in fabula. Milano: Bompiani. Eco, Umberto. 1986. Semiotics and the Philosophy of Language. Bloomington: Indiana University Press. Eco, Umberto. 1990. The Limits of Interpretation (Advances in Semiotics). Bloomington: Indiana University Press.

182

References

Eco, Umberto. 1999. Kant and the Platypus: Essays on Language and Cognition. New York: Harcourt Brace. Eco, Umberto. 2004. Les Sémaphores sous la Pluie. In On Literature. Orlando, FL: Harcourt, 180–200. Elman, Jeffrey, Elizabeth Bates, Mark Johnson, Annette Karmiloff-Smith, Domenico Parisi, and Kim Plunkett. 1996. Rethinking Innateness: A Connectionist Perspective on Development. Cambridge, MA: MIT Press. Engeström, Yrjo, and David Middleton, eds. 1996. Cognition and Communication at Work. New York: Cambridge University Press. Epstein, Steve. 1996. Impure Science: AIDS, Activism, and the Politics of Knowledge. Berkeley: University of California Press. Fabbrichesi Leo, Rosella. 2004. Peirce and Wittgenstein on Common Sense. Cognitio 6. Fauconnier, Gilles, and Mark Turner. 2003. The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities. New York: Basic Books. Fodor, Jerry. 1983. The Modularity of Mind: An Essay in Faculty Psychology. Cambridge, MA: MIT Press. Foucault, Michel. 1973. The Birth of the Clinic. New York: Pantheon. Foucault, Michel. 1977. Discipline and Punish. New York: Pantheon. Foucault, Michel. 1988–1990. History of Sexuality, 3 volumes: Introduction, The Uses of Pleasure, and Care of the Self. New York: Vintage Books. Fujimura, Joan. 1996. Crafting Science: A Sociohistory of the Quest for the Genetics of Cancer. Cambridge, MA: Harvard University Press. Galison, Peter. 1987. How Experiments End. Chicago, IL: University of Chicago Press. Galison, Peter. 1997. Image and Logic: Material Culture of Microphysics. Chicago, IL: University of Chicago Press. Gallese, Vittorio, Luciano Fadiga, Leonardo Fogassi, and Giacomo Rizzolatti. 1996. Action Recognition in the Premotor Cortex. Brain 119 (2):593–609. Gandhi, Sunil, David Heeger, and Geoffrey Boynton. 1999. Spatial Attention Affects Brain Activity in Human Primary Visual Cortex. Proceedings of the National Academy of Sciences of the United States of America 96:3312–3319. Garfinkel, Harold. 2002. Ethnomethodology’s Program. New York: Rowman & Littlefield.

References

183

Geertz, Clifford. 1973. Thick Description: Toward and Interpretative Theory of Culture. In The Interpretation of Culture: Selected Essays. New York: Basic Books, 3–30. Gibson, James. 1966. The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin. Giere, Ronald, and Barton Moffatt. 2003. Distributed Cognition: Where the Cognitive and the Social Merge. Social Studies of Science 33:301–310. Glasgow, Janice, N. Hari Narayanan, and B. Chandrasekaran, eds. 1995. Diagrammatic Reasoning: Cognitive and Computational Perspectives. Cambridge, MA: MIT Press. Golden, Deven. 2008. James Timberlake. Bomb 104:75–81. Gombrich, Ernst. 1960. Art and Illusion: A Study in the Psychology of Pictorial Representation. New York: Pantheon. Goodwin, Charles. (3):606–633.

1994.

Professional

Vision.

American

Anthropologist

96

Goodwin, Charles. 1995. Seeing in Depth. Social Studies of Science 25:237–274. Goodwin, Charles. 1997. The Blackness of Black: Color Categories as Situated Practice. In Discourse, Tools and Reasoning: Essays on Situated Cognition, ed. L. Resnick, R. Säljö, C. Pontecorvo and B. Burge. New York: Springer-Verlag, 111–140. Goodwin, Charles. 2000a. Practices of Seeing, Visual Analysis: An Ethnomethodological Approach. In Handbook of Visual Analysis, ed. T. van Leeuwen and C. Jewitt. London: Sage, 157–182. Goodwin, Charles. 2000b. Action and Embodiment within Situated Human Interaction. Journal of Pragmatics 32(10):1489–1522. Goodwin, Charles. 2000c. Practices of Color Classification. Mind, Culture, and Activity 7 (1–2):19–36. Goodwin, Charles. 2003a. The Body in Action. In Discourse, the Body and Identity, ed. J. Coupland and R. Gwyn. New York: Palgrave/Macmillan, 19–42. Goodwin, Charles. 2003b. Conversational Frameworks for the Accomplishment of Meaning in Aphasia. In Conversation and Brain Damage, ed. C. Goodwin. Oxford: Oxford University Press, 90–116. Goodwin, Charles, Harness Goodwin, Marjorie, and Malcah Yaeger-Dror. 2002. Multi-modality in Girls’ Game Disputes. Journal of Pragmatics 24(10–11): 1621–1649. Greimas, Algirdas J. 1987. On Meaning: Selected Writings in Semiotic Theory. Minneapolis: University of Minnesota Press.

184

References

Gross, Alan. 2008. The Brains in Brain: The Coevolution of Localization and its Images. Journal of the History of the Neurosciences 17 (3):380–392. Gross, Charles. 1998. Brain, Vision and Memory: Tales of the History of Neuroscience. Cambridge, MA: MIT Press. Gross, Charles. 1997. Emanuel Swedenborg: A Neuroscientist Before His Time. Neuroscientist 3:142–147. Grosz, Elisabeth. 1987. Notes Towards a Corporeal Feminism. Australian Feminist Studies 5:1–16. Grosz, Elisabeth. 1994. Volatile Bodies: Towards a Corporeal Feminism. London: Routledge. Haraway, Donna. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist Studies 14:575–599. Haraway, Donna. 1991. A Cyborg Manifesto: Science, Technology, and SocialistFeminism in the Late Twentieth Century. In Simians, Cyborgs and Women: The Reinvention of Nature. New York: Routledge, 149–181. Haraway, Donna. 1992. The Promises of Monsters: A Regenerative Politics for Inappropriate/d Others. In Cultural Studies, ed. Lawrence Grosberg, Cary Nelson, and Paula Treichler. New York and London: Routledge, 295–337. Haraway, Donna. 1997. Modest_Witness@Second_Millennium.FemaleMan©Meets _OncoMouse™: Feminism and Technoscience. New York: Routledge. Hartouni, Valeri. 1991. Containing Women: Reproductive Discourse in the 1980s. In Technoculture, ed. Constance Penley and Andrew Ross. Minneapolis: University of Minnesota Press, 27–56. Haviland, John. 2005. Gesture as Cultural and Linguistic Practice. In Linguistic Anthropology, ed. Anita Sujoldzic. Oxford: Eolss Publishers, 197–221. Hayles, Katherine. 1993. Constrained Constructivism: Locating Scientific Inquiry in the Theater of Representation. In Realism and Representation: Essays on the Problem of Realism in Relation to Science, Literature, and Culture, ed. George Levine. Madison: University of Wisconsin Press, 27–43. Hayles, Katherine. 1999. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. Chicago, IL: University of Chicago Press. Heidegger, Martin. 1962. Being and Time. New York: Harper. Heidegger, Martin. 1977. The Question Concerning Technology. New York: Harper and Row. Heidegger, Martin. 1982. The Basic Problems of Phenomenology. Bloomington: Indiana University Press.

References

185

Heath, Christian. 1997. The Analysis of Activities in Face to Face Interaction Using Video. In Qualitative Resaerch: Theory, Method and Practice, ed. David Silverman. London: Sage, 183–200. Heath, Christian, and Jon Hindmarsh. 2002. Analyzing Interaction: Video Ethnography and Situated Conduct. In Qualitative Research in Action, ed. Tim May. London: Sage, 99–121. Hindmarsh, Jon, and Christian Heath. 2000. Embodied Reference: A Study of Deixis in Workplace Interaction. Journal of Pragmatics 32:1855–1878. Høstaker, Roar. 2005. Latour—Semiotics and Science Studies. Science Studies 18 (2):5–25. Hutchins, Edwin. 1980. Culture and Inference: A Trobrian Case Study. Cambridge, MA: Harvard University Press. Hutchins, Edwin. 1995. Cognition in the Wild. Cambridge, MA: MIT Press. Hutchins, Edwin. 2005. Material Anchors for Conceptual Blends. Journal of Pragmatics 37 (10):1555–1577. Ihde, Don. 1990 Technology and the Lifeworld: From Garden to Earth. Bloomington: Indiana University Press. Ihde, Don. 1993. Philosophy of Technology: An Introduction. New York: Paragon. Ihde, Don. 1998. Expanding Hermeneutics: Visualism in Science. Evanston, IL: Northwestern University Press. Ihde, Don. 2002. Bodies in Technology. Minneapolis: University of Minnesota Press. Irigaray, Luce. 1985. This Sex which is not One. Ithaca, NY: Cornell University Press. Jasanoff, Sheila. 1995. Science at the Bar: Law, Science, and Technology in America. Cambridge, MA: Harvard University Press. Jefferson, Gail. 2004. Glossary of Transcript Symbols with an Introduction. In Conversation Analsysis: Studies from the First Generation, ed. G.H. Lerner. New York: John Benjamins, 13–31. Joyce, Kelly. 2005. Appealing Images: Magnetic Resonance Imaging and the Production of Authoritative Knowledge. Social Studies of Science 35 (3):437–462. Joyce, Kelly. 2008. Magnetic Appeal: MRI and the Myth of Transparency. Ithaca, NY: Cornell University Press. Kauffmann Jokl, Danny, and Fusako Hiyama. 2007. Tatsuji Inouye—Topographer of the Visual Cortex: Exemplar of the Germany-Japan Ophthalmic Legacy of the Meiji Era. Neuro-Ophthalmology (Aeolus Press) 31 (3):33–43.

186

References

Kaiser, David, ed. 2005. Pedagogy and the Practice of Science: Historical and Contemporary Perspective. Cambridge, MA: MIT Press. Karmiloff-Smith, Annette. 1992. Beyond Modularity. Cambridge, MA: MIT Press. Kenning, Peter, and Hilke Plassmann. 2005. NeuroEconomics: An Overview from an Economic Perspective. Brain Research Bulletin 67 (15):343–354. Kirsh, David. 1995. The Intelligent Use of Space. Artificial Intelligence 73:31–68. Kita, Sotaro, ed. 2003. Pointing: Where Language, Culture, and Cognition Meet. Mahwah, NJ: Lawrence Erlbaum. Knorr-Cetina, Karin. 1981. The Manufacture of Knowledge: An Essay on the Constructivist and Contextual Nature of Science. Oxford: Pergamon Press. Knorr-Cetina, Karin. 1999. Epistemic Cultures: How the Sciences Make Knowledge. Cambridge, MA: Harvard University Press. Knorr-Cetina, Karin, and Klaus Amann. 1990. Image Dissection in Natural Scientific Inquiry. Science, Technology & Human Values 15:259–283. Koestler, Arthur. 1964. The Act of Creation. New York: Macmillan. Kohler, Robert E. 2008. Lab History: Reflections. Isis 99:761–768. Korte, Adolf. 1915. Kinematoskopische Untersuchungen. [Kinematoscopic Investigations] Zeitschrift für Psychologie 72:194–296. Koschmann, Timothy, Le Baron, Curtis, Goodwin, Charles, Zemel, Alan, and Gary Dunnington. 2007. Formulating the Triangle of Doom. Gesture 7:97–117. Kuhn, Thomas. 1962. The Structure of Scientific Revolutions. Chicago, IL, and London: University of Chicago Press. Kutas, Marta, Federmeier, Kara, Schul, Rina, and Jonathan King. (manuscript in preparation) The Inference Problems Involved in “Brain Mapping”. Lacan, Jacques. 1977. Écrits, A Selection. New York: Norton. Lacan, Jacques. 1982. Feminine Sexuality: Jacques Lacan and the Ecole Freudienn, ed. Juliet Mitchell and Jacqueline Rose. London: Macmillan. Lakoff, George. 1987. Woman, Fire, and Dangerous Things. Chicago, IL: University of Chicago Press. Lakoff, George, and Mark Johnson. 1980. Metaphors we Live By. Chicago: University of Chicago Press. Lakoff, George, and Mark Johnson. 1999. Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. New York: Basic Books. Larkin, Jill, and Herbert Simon. 1987. Why a Diagram Is (Sometimes) Worth Ten Thousand Words. Cognitive Science 11:65–99.

References

187

Latour, Bruno. 1983. Give Me a Laboratory and I Will Raise the World. In Science Observed, ed. Karin Knorr-Cetina and Michael Mulkay. London: Sage, 141–170. Latour, Bruno. 1986. Visualization and Cognition: Thinking with Eyes and Hands. Knowledge and Society 6:1–40. Latour, Bruno. 1987. Science in Action. Cambridge, MA: Harvard University Press. Latour, Bruno. 1990. Visualization and Cognition: Drawing Things Together. In Representation in Scientific Activity, ed. Michael Lynch and Steve Woolgar. Cambridge, MA: MIT Press, 19–68. Latour, Bruno. 1993. Pasteur on Lactic Acid Yeast: A Partial Semiotics Analysis. Configurations 1.1:129–146. Latour, Bruno. 1995. The “Pedofil” of Boa Vista—A Photo-Philosophical Montage. Common Knowledge 4 (1):145–187. Latour, Bruno. 1999. Pandora’s Hope: Essays on the Reality of Science Studies. Cambridge, MA: Harvard University Press. Latour, Bruno, and Steve Woolgar. 1979. Laboratory Life: The Social Construction of Scientific Facts. London and Beverly Hills, CA: Sage. Lave, Jean. 1988. Mind, Mathematics and Culture in Everyday Life. Cambridge: Cambridge University Press. Lave, Jean, and Etienne Wenger. 1991. Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press. Lave, Jean, Michael Murtaugh, and Olivia de la Rocha. 1984. The Dialectic of Arithmetic in Grocery Shopping. In Everyday Cognition: Its Development in Social Context, ed. Barbara Rogoff and Jean Lave. Cambridge, MA: Harvard University Press, 67–94. LeBaron, Curtis. 2007. Formulating the Triangle of Doom. Gesture 7 (1):97–118. Lenoir, Timothy. 1994. Was That Last Turn a Right Turn? The Semiotic Turn and A.J. Greimas. Configurations 2:119–136. Lynch, Michael. 1985a. Art and Artifact in Laboratory Science: A Study of Shop Work and Shop Talk in a Research Laboratory. London: Routledge & Kegan Paul. Lynch, Michael. 1985b. Discipline and the Material Form of Images: An Analysis of Scientific Visibility. Social Studies of Science 15 (1):37–66. Lynch, Michael. 1990. The Externalized Retina: Selection and Mathematization in the Visual Documentation of Objects in the Life Sciences. In Representation in Scientific Practice, ed. Michael Lynch and Steve Woolgar. Cambridge, MA: MIT Press, 153–186.

188

References

Lynch, Michael. 1991. Laboratory Space and the Technological Complex: An Investigation of Topical Contextures. Science in Context 4 (1):81–109. Lynch, Michael. 1993. Scientific Practice and Ordinary Action: Ethnomethodology and Social Studies of Science. New York: Cambridge University Press. Lynch, Michael. 1994. Representation Is Overrated: Some Critical Remarks about the Concept of Representation in Science Studies. Configurations 2 (1): 137–149. Lynch, Michael, and Steve Woolgar, eds. 1990. Representation in Scientific Practice. Cambridge, MA: MIT Press. Manovich, Lev. 2001. The Language of New Media. Cambridge, MA: MIT Press. Marshall, Helen. 1996. Our Bodies, Ourselves: Why We Should Add Old Fashioned Empirical Phenomenology to the New Theories of the Bodies. Women’s Studies International Forum 19 (5):253–265. McNeill, David. 1995. Hand and Mind: What Gestures Reveal about Thought. Chicago, IL: University of Chicago Press. McNeill, D., ed. 2000. Language and Gesture. Cambridge: Cambridge University Press. Merleau-Ponty, Maurice. 1962. Phenomenology of Perception. New York: Humanities Press. Merleau-Ponty, Maurice. 1968. The Visible and the Invisible. Evanston, IL: Northwestern University Press. Merz, Martina, and Karin Knorr-Cetina. 1997. Deconstruction in a Thinking Science: Theoretical Physics at Work. Social Studies of Science 27 (1):73–111. Middleton, David, and Derek Edwards, eds. 1990. Collective Remembering. London: Sage. Mindell, David. 2002. Between Human and Machine: Feedback, Control, and Computing Before Cybernetics. Baltimore, MD: John Hopkins University Press. Mindell, David. 2008. Digital Apollo: Human and Machine in Spaceflight. Cambridge, MA: MIT Press. Mithen, Steven. 1998. Creativity in Human Evolution and Prehistory. London and New York: Routledge. Mody, Cyrus. 2001. A Little Dirt Never Hurt Anyone: Knowledge-Making and Contamination in Materials Science. Social Studies of Science 31 (1):7–36. Mody, Cyrus, and David Kaiser. 2008. Scientific Training and the Creation of Scientific Knowledge. In The Handbook of Science and Technology Studies. 3rd ed., ed.

References

189

Edward J. Hackett, Olga Amsterdamska, Lynch Michael, and Judy Waycman. Cambridge, MA: MIT Press, 377–402. Mol, Annemarie. 2002. The Body Multiple: Ontology in Medical Practice. Durham, NC: Duke University Press. Mondada, Lorenza. 2007. Multimodal Resources for Turn-Taking: Pointing and the Emergence of Possible Next Speaker. Discourse Studies 9 (2): 194–225. Mukerji, Chandra. 2009. Impossible Engineering: Technology and Territoriality on the Canal du Midi. Princeton, NJ: Princeton University Press. Murphy, Keith M. 2004. Imagination as Joint Activity: The Case of Architectural Interaction. Mind, Culture, and Activity 11 (4):267–278. Murphy, Keith M. 2005. Collaborative Imagining: The Interactive Use of Gestures, Talk, and Graphic Representation in Architectural Practice. Semiotica 156 (1–4): 113–145. Neisser, Ulric. 1982. Memory Observed: Remembering in Natural Contexts. Oxford: Freeman. Nersessian, Nancy, Elke Kurz-Milcke, Wendy Newstetter, and Jim Davies. 2003. Research Laboratories as Evolving Distributed Cognitive Systems. In Proceedings of the Twenty-Fifth Annual Conference of the Cognitive Science Society, ed. Robert Alterman and David Kirsh. Hillsdale, NJ: Lawrence Erlbaum, 857–862. Nishizaka, Aug. 2000. Seeing What One Sees: Perception, Emotion, and Activity. Mind, Culture, and Activity 7 (1–2):105–123. Nishizaka, Aug. 2003. Imagination in Action. Theory & Psychology 13 (2):177–207. Norman, Donald. 1988. The Psychology of Everyday Things. New York: Basic Books. Ochs, Elinor, Patrick Gonzales, and Sally Jacoby. 1996. “When I Come Down I’m in a Domain State”: Talk, Gesture, and Graphic Representation in the Interpretative Activity of Physicists. In Interaction and Grammar, ed. Elinor Ochs, Emanuel Schegloff, and Sandra Thomson. Cambridge: Cambridge University Press, 328– 369. Op de Beeck, Hans, Johannes Haushofer, and Nancy Kanwisher. 2008. Interpreting fMRI Data: Maps, Modules and Dimensions. Nature Reviews. Neuroscience 9 (2):123–135. Peirce, C. S. 1934–1958. Collected Papers. Cambridge, MA: Harvard University Press. Pickering, Andrew. 1995. The Mangle of Practice: Time, Agency and Science. Chicago, IL: University of Chicago Press.

190

References

Pinch, Trevor. 1986. Confronting Nature: The Sociology of Solar Neutrino Detection. Dordrecht: D. Reidel. Polanyi, Michael. 1958. Personal Knowledge: Toward a Post-critical Philosophy. Chicago, IL: University of Chicago Press. Polanyi, Michael. 1959. The Study of Man. Chicago, IL: University of Chicago Press. Prasad, Amid. 2005a. Scientific Culture in the “Other” Theatre of “Modern Science”: An Analysis of the Culture of Magnetic Resonance Imaging Research in India. Social Studies of Science 35 (6):463–489. Prasad, Amid. 2005b. Making Images/Making Bodies: Visibilizing and Disciplining through Magnetic Resonance Imaging. Science, Technology & Human Values 30 (2):291–316. Pulvermüller, Friedemann. 1999. Words in the Brain’s Language. Behavioral and Brain Sciences 22:253–336. Ransdell, Joseph. 2003. The Relevance of Peircean Semiotic to Computational Intelligence Augmentation. SEED 3:5–36. Reardon, Jenny. 2005. Race to the Finish: Identity and Governance in an Age of Genomics. Princeton, NJ: Princeton University Press. Rescher, Nicholas. 1978. Peirce’s Philosophy of Science: Critical Studies in his Theory of Induction and Scientific Method. Notre Dame, IN: Notre Dame University Press. Rheinberger, Hans-Jörg. 1998. Experimental Systems—Graphematic Spaces. In Inscribing Science: Scientific Texts and the Materiality of Communication, ed. Timothy Lenoir and Hans Ulrich Gumbrecht. Stanford, CA: Stanford University Press, 285–303. Rizzolati, Giacomo, Luciano Fadiga, Vittorio Gallese, and Leonardo Fogassi. 1996. Premotor Cortex and the Recognition of Motor Actions. Brain Research. Cognitive Brain Research 3:131–141. Rogoff, Barbara, and Jean Lave, eds. 1984. Everyday Cognition: Its Development in Social Context. Cambridge, MA: Harvard University Press. Roth, Wolf Michael. 2005. Making Classifications (at) Work: Ordering Practices in Science. Social Studies of Science 35 (4):581–621. Rudwick, Martin. 1976. The Emergence of a Visual Language for Geological Science: 1760-1840. History of Science 14:148–195. Sacks, Harvey. 1995. Lectures on Conversation. Oxford: Blackwell Publishing.

References

191

Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson. 1974. A Simplest Systematics for the Organization of Turn-Taking for Conversation. Language 50: 696–735. Schegloff, Emanuel A. 1984. On Some Gestures Relation to Talk. In Structures of Social Action: Studies in Conversation Analysis, ed. J. Maxwell Atkinson. Cambridge: Cambridge University Press. Schegloff, Emanuel A. 1987. Between Macro and Micro: Contexts and Other Connections. In The Micro-Macro Link, ed. Jeffrey Alexander, Bernhard Giesen, Richard Munch, and Neil Smelser. Berkeley and Los Angeles: University of California Press, 207–234. Shapin, Steven, and Simon Schaffer. 1985. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Princeton, NJ: Princeton University Press. Sereno, Martin, Sabrina Pitzalis, and Antigona Martinez. 2001. Mapping of Contralateral Space in Retinotopic Coordinates by a Parietal Cortical Area in Humans. Science 294:1350–1354. Serson, Breno. 1997 On Peirce’s Pure Grammar as a General Theory of Cognition: From the Thought-Sign of 1968 to the Semeiotic Theory of Assertion. Semiotica 113 (1/2):107–158. Shin, Sun-Joo. 2002. The Iconic Logic of Peirce’s Graphs. Cambridge, MA: MIT Press. Sims, Ben. 2005. Material and Social Order in Laboratory Work. Social Studies of Science 35 (3):333–366. Sinha, Chris, and Kristine Jensen de López. 2000. Language, Culture and the Embodiment of Spatial Cognition. Cognitive Linguistics 11:17–41. Skagestad, Peter. 1996. The Mind’s Machines: The Turing Machine, the Memex, and the Personal Computer. Semiotica 111 (3/4):217–243. Sobchack, Vivian. 2004. Carnal Thoughts: Embodiment and Moving Image Culture. Berkeley: University of California Press. Star, Susan Lee. 1983. Simplification in Scientific Work: An Example from Neuroscience Research. Social Studies of Science 13:205–228. Star, Susan Leigh. 1989. Regions of the Mind: Brain Research and the Quest for Scientific Certainty. Stanford, CA: Stanford University Press. Streeck, Jürgen. 2009. Gesturecraft: The Manu-Facture of Meaning. Amsterdam and Philadelphia: John Benjamins. Suchman, Lucy. 1987. Plans and Situated Actions: The Problem of Human-Machine Communication. New York: Cambridge University Press.

192

References

Suchman, Lucy. 2000. Embodied Practices of Engineering Work. Mind, Culture, and Activity 7 (1–2):4–18. Suchman, Lucy. 2007. Human-Machine Reconfigurations: Plans and Situated Actions, 2nd Expanded Edition. New York: Cambridge University Press. Talairach, Jean, and Pierre Tournoux. 1988. Co-Planar Stereotaxic Atlas of the Human Brain. New York: Thieme Medical Publishers. Talairach, J., G. Szikla, P. Tournoux, A. Prossalentis, M. Bordas-Ferrer, and L. Covello. 1967. Atlas d’Anatomie Stéréotaxique du Telencephale. Paris: Masson. Talmy, Leonard. 2000. Toward a Cognitive Semantics. Cambridge, MA: MIT Press. te Molder, Hewig, and Jonathan Potter. 2005. ed. Conversation and Cognition. Cambridge: Cambridge University Press. Tootell, Roger, Nouchine Hadjikhani, Wim Vanduffel, Janine Mendola, Martin Sereno, and Dale Anders. 1998. Functional Analysis of Primary Visual Cortex (V1) in Humans. Proceedings of the National Academy of Sciences of the United States of America 95:811–871. Traweek, Sharon. 1988. Beamtimes and Lifetimes: The World of High Energy Physics. Cambridge, MA: Harvard University Press. Uttal, William. 2001. The New Phrenology. Cambridge, MA: MIT Press. van Dijk, Teun A. ed. 2006. Discourse, Interaction and Cognition. Discourse Studies 8(1). Violi, Patrizia. 2008. Beyond the Body: Towards a Full Embodied Semiosis. In Body, Language and Mind, ed. Frank, Roslyn, Dirven, René, Ziemke, Tom, and Enrique Bernárdez. Berlin, New York: Mouton de Gruyter, 53–76. Volli, Ugo. 1972. Some Possible Developments of the Concept of Iconism. Versus 3:14–29. Vygotsky, Lev. 1962. Thought and Language. Cambridge, MA: MIT Press. Vygotsky, Lev. 1978. Mind and Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press. Wertheimer, Max. 1912. Experimentelle Studien über das Sehen von Bewegung. [Experimental Studies on Seeing Motion] Zeitschrift fur Psychologie mit Zeitschrift fur Angewandte Psychologie 61:161–265. Whalen, Jack, Marilyn Whalen, and Kathryn Henderson. 2002. Improvisational Choreography in Teleservice Work. British Journal of Sociology 53 (2):239–258. Wilson, Elizabeth. 2004. Psychosomatic: Feminism and the Neurological Body. Durham, NC: Duke University Press.

References

193

Wilson, Margaret. 2002. Six Views of Embodied Cognition. Psychological Bulletin and Review 9(4): 625–638. Wittgenstein, Ludwig. 1973. Philosophical Investigations. Englewood Cliffs, NJ: Prentice- Hall. Ziemke, Tom. 2003. What’s That Thing Called Embodiment? In Proceedings of the 25th Annual Meeting in the Cognitive Science Society, ed. Robert Alterman and David Kirsh. Hillsdale, NJ: Lawrence Erlbaum, 1134–1310.

Index

Abduction, 110, 112, 175–176n10. See also Creativity Aesthetic judgment, 113–114 Agency, 38, 40–41, 46–47, 103, 116, 144–145, 167–168 Aman, Klaus, 4, 17 Apprenticeship, 50, 51–53, 58, 62–63, 67, 71, 78, 94, 98, 104–105. See also Experiential knowing; Learning newcomers and old-timers, 51–52, 78–79, 110, 113, 173n1 Architecture, 46 Art, 46 Artifact, research, 68–69, 173n8 movement, 69, 74, 90 shearing, 123–124, 137–138, 144 Audience, 157–158, 168 Author, 47, 167–168 Bakhtin, Mikhail, 36 Barsalou, Lawrence, 163–164 Barthes, Roland, 39, 167, 171n3 Beaulieu, Anne, 15, 43–44, 143, 147, 149 Body, 6–7, 8, 50, 67, 69, 71, 77, 78, 81–82, 83–84, 88–91, 121, 160 as a fluid site of potential, 6 multiple, 68, 83–84, 88–91, 166 as a semiotic mean, 78 Brain cortex, 29 as a map, 28, 29, 43 Brain mapping, 28, 125

Cartwright, Lisa, 4, 7, 64, 125 Cascades of representations, 99 Categorization, 64, 125, 145 Cognitive neuroscience, 5–6, 80, 161–162 and social science, 160–162 Cognitive science, theories in, 156, 165 Cognitive semantics, 156, 165 Complexity, 159. See also Conceptual integration/blending theory; Fictive motion Conceptual integration/blending theory, 107, 109, 165, 177n Conceptual scaling, 156–157 Connectionism, 31 Connotation, 39–40 Constant visibility, 50, 64–65 Conversation analysis, 7, 10–11, 126–127 transcription, 59, 126–127 Coordination of interaction and manual work, 121, 126, 131–132, 141, 145 Coordination of semiotic fields, 105–108, 110, 118–119 Cortical maps, 30–31, 33, 125 Creativity, 110, 112, 122, 148, 152, 157–158, 167. See also Abduction Cultural artifact, 103 Cyborg visuality/seeing, 2, 35–36, 169n1

196

Index

Decade of the brain, 5, 161 Diagrammatic reasoning, 156, 165 Digital screens, 142, 144 interaction with, 3–4, 5, 14, 18, 45–47, 68, 81–82, 93, 107–109, 115,

134, 145, 147, 158, 160. See also fMRI visuals; Scientific visuals fMRI culture, 124, 125–126, 147, 154–155, 157 fMRI laboratory, 12–14, 174n

130, 133 Distributed cognition, 20, 117–118, 165, 176n13 systems, 12, 47, 65, 118, 144–145, 169n7 Doing, Park, 15–16 Dourish, Paul, 177n5 Dreyfus, Herbert, 133 Dumit, Joseph, 166–167, 170n10–11

fMRI scanning, 53–55, 70 fMRI seeing, 2–3, 49, 73, 78, 93, 95, 97–98, 105–106, 110, 113, 115, 116–117, 120, 123, 133–134, 145 as collaborative, 123, 130 as cultural, 157 as multimodal, 49, 71, 78, 95, 98, 105, 113, 115, 116–117, 120 as practical action, 3, 36, 43, 45–46, 109, 120, 131, 133, 145 as superseeing, 35–36 fMRI software, 122–123, 125–126 fMRI visuals, 1–2, 6, 17–18, 23, 40, 69, 71, 81–82, 91, 93, 100, 105–106, 120, 142, 156, 167–168 boundaries of, 45–47 color, 33–34, 109–110, 152–153, 154–155, 175n9 diagrammatic character of, 42–43, 45, 162 digitality of, 2–3, 4, 5, 15, 17–18, 24, 26–27, 35–36, 45, 67, 109–110, 121, 125, 130, 133, 139, 142, 145, 156, 160, 175n7 functional image, 40, 58–59, 123 as iconic signs, 26–27, 28, 41, 43, 115–116, 121, 145, 167–168 as images, 44–45, 145, 147, 171n9 as indexical signs, 23, 147 labels, 32–33, 37–38, 171–172n12 localizer scan, 58–59 as maps, 171n9 quantitative character of, 43–44, 152, 155–156 structural image, 33, 40, 58–59, 123 visual character of, 26–27, 45–46, 142, 147–149, 150, 152, 155–157 Foucault, Michel, 6, 49–50, 65, 90

Eco, Umberto, 24, 25, 39, 121, 149, 170n12, 172n4 Effects of similarity, 25, 28, 121, 145 Embodiment in cognitive neuroscience and fMRI, 5–6, 7, 54–55, 56–57, 67–68, 69, 162–163 in feminist studies of science and technology, 6–7, 83 in postphenomenological studies of science and technology, 6–7 Environmentally coupled gestures, 143 Ethnography, 12–14, 15, 17, 20, 53, 148, 160–162 Ethnomethodology, 7, 10, 13, 20 Experiential knowing, 51, 61–65, 78–79, 82, 145 Experimental subject, 1, 52, 53–55, 56–57, 63, 67, 69, 80–82, 89, 90–91. See also Imaged body Fauconnier, Gilles, 109, 165, 177n7 Feeling, 79, 82. See also Experiential knowing Fictive motion, 132 Fields for interaction, scientific visuals as, 17–18, 24, 46, 67, 81–82, 93, 103, 105, 110, 118–120, 130, 132–133,

Index

Functional magnetic resonance imaging (fMRI), 1–2, 32, 90–91 vs. electroencephalography (EEG), 32 vs. magnetoencephalography (MEG), 32 Garfinkel, Harold, 7, 10, 13, 20 Gesture, 3–4, 8, 9, 77, 79, 100, 126, 131, 142–143 enacting, 77, 79, 100, 130–132, 139, 141 pointing, 93, 100, 107, 110, 112, 118 Goodwin, Charles, 7, 9, 11, 64, 71, 97, 143, 169n3 Gross, Alan, 23 Grosz, Elizabeth, 6, 83 Haraway, Dona, 7, 15, 18 Heath, Christian, 7, 8, 10, 100 Heidegger, Martin, 6, 99, 133 Hindmarsh, Jon, 7, 8, 10, 100 Human-technology coupling, 2, 7, 9, 18, 35–36, 47, 62–65, 144 Hutchins, Edwin, 20, 117–118, 120, 165, 177n7 Hybrid phenomenal/enacted objects, 121, 131–133, 142, 143–145, 147. See also Objects of scientific inquiry engaged as physical objects, 121, 131, 133, 139, 141, 144–145 Icon, sign, 23–24, 41, 77 diagram, 41–42, 172n20, 173n22 image, 41 metaphor, 41 Iconicity, 24, 25–26, 41, 43, 171n4 Ihde, Don, 7 Imaged body, 67–68, 80–82, 165. See also Experimental subject Imagination, 144 Immutable mobiles, 99–101, 118, 176n Infrastructure for seeing/Semiotic infrastructure, 24, 34, 39–41, 152–154

197

Interaction between fMRI laboratories, 122–123, 126 Interpretative semiotics, 18, 23, 119, 173–174n4 Jefferson, Gayle, 7, 126 Joyce, Kelly, 15, 44, 166–167, 170n10– 11, 176n3, 178n2 JR, 46 Knorr-Cetina, Karin, 4, 14–17, 80, 176n13, 178n1 Kohler, Robert E., 16 Laboratory studies, 14–17, 19, 166, 170n9 Lacan, Jacques, 6 Latour, Bruno, 4, 12, 14, 16–18, 80, 99–101, 118, 122, 174n3, 175n4–5, 178n4 Lave, Jean, 51, 173n1 Learning and studies of science and technology, 50 Learning to see, 39, 105, 113–115 Learning, 52, 57, 78–79, 91, 105. See also Apprenticeship Localization of function, 29, 33, 124–125, 154–155 Local resistances, 36–37, 158 Local scientific practice and their relationship with the larger social forms, 15–16, 63, 80–81, 99–100, 115, 126, 157–158, 165–167 Lynch, Michael, 5, 12, 17, 63, 68–69, 79, 100, 121, 125, 169n5, 170n9, 178n5 Magnetic resonance imaging (MRI), 1 Map, 41–44, 98, 171n Marshall, Helen, 6, 83 Materiality, 19, 121, 125, 143, 145 Merleau-Ponty, Maurice, 6

198

Index

Mind, thinking, 93, 99–101, 103, 116 in cognitive science and fMRI, 5–6 computational/representational, 6, 118 in practice, 5, 7, 19, 20, 93, 99, 119

170n12, 172n21, 173n22, 173– 174n4, 175–176n10, 176n11 Phase map, 95 Phrenology, 29, 31 Power, 99, 105, 143–144

Mindell, David, 5, 15 Model reader, 39–41, 149 Modularity of the Mind, 31 Mol, Annemarie, 7, 13, 19–20, 83–84, 166 Movement, 69–70, 72–73, 78–79, 90 Multimodal conversation analysis, 7–8 Multimodal interaction, 9, 11, 16, 41, 47, 67, 71, 77, 78–79, 83–84, 91, 93, 99–100, 107–108, 110, 119, 139, 142, 162 transcription of, 11 Multimodal simulations, 164 Multimodality, 2, 5, 7–8, 17–18, 49, 53, 54, 64–65, 71, 73, 115, 117, 158, 160, 168 Multivoicedness, 36, 62–63, 103 Mutable and local scientific inscriptions, 99–101, 103–105, 118

Prasad, Amit, 15, 169n1 Present-at-hand (Vorhandene) vs. ready-to-hand (Zuhandene), 133, 177n5 Professional vision, 97 Publishing, 147–150

Negotiation, 36, 38, 147, 153–157 Neuroetics, 161 Neuronal modules, 31 Objects of scientific inquiry, 17, 19, 121, 126, 131, 145, 147, 164, 167 as dynamic and multiple, 16–17, 18, 19, 71, 83–84, 165 as enacted, 81–82, 84, 91, 165, 167 as hybrid, 19, 121, 145, 147 as multimodal, 64, 162 Obligatory points of passage, 122 Ochs, Elinor, 7, 177n8 Opticism vs. digitality, 5, 24, 43–45, 121 Pedagogy, 50, 62, 64, 77, 82 Peirce, Charles Sanders, 18, 23, 24, 25, 41–43, 51, 116, 119, 121, 167,

Reader, 38–41 Re-coding, 103, 156 Reference, 18–19, 26, 81–82, 173–174n4 Immediate Object, 173–174n4 Dynamical Object, 173–174n4 Referentially ambiguous entities, 177n8 Representation, 46, 99, 118, 120, 133, 145, 147, 158, 162, 164, 165, 176n Representational re-description, 156 Retinotopic mapping, 30–32, 94–95 Sacks, Harvey, 7, 59 Scaffolding, 77 Schegloff, Emanuel, 7, 10–11, 126–127, 131 Scientific community, 15–16, 36, 44, 82, 90–91, 115, 147, 149 Scientific visuals, 4, 17, 68, 71, 81–82, 99–101, 158 Selection, 34, 103, 118, 143 Semiosis, 119, 170n1 Semiotic mind, the, 19, 21, 93, 103–105, 107–110, 112, 116, 119, 162, 175n7 Semiotics and STS, 18 Sign, 23 index, 23 Peirce’s typology, 171n2 symbol, 23 triadic, 170–171n1, 173–174n4

Index

Situated action, 8, 115, 118, 165 Skill, 51, 64–65, 91, 95–98, 113, 130 Social constructionism, 6, 18, 68, 82, 165, 167 Social neuroscience, 161 Sound, 49, 51, 57, 61, 63–65 Spatial normalization, 37–38 Standardization, 36–38, 79–80 standard brain/Talairach brain, 37, 172n16 Star, Susan Leigh, 79–80 Streeck, Jürgen, 7, 11 Studies of science and technology and societal phenomena, 15–16, 44 Suchman, Lucy, 7, 8, 20, 100, 144, 161, 177n5 Supervisual, 34–36 Surrogates for perceptual stimuli, 25–26 Talairach and Tournoux’s stereotaxic atlas, 37 Talmy, Leonard, 132 Text, 15, 28, 39, 41, 167–168 Timberlake, James and Steven Kieran, 46 Trading spaces, 103, 156 Turner, Mark, 109, 165, 177n7 Video, as analytical resource, 8–12, 117, 160 Violi, Patrizia, 163 Visual cortex, 30, 94–95, 122 Visuospatial inscriptions, 93, 100–101, 118 Vygotsky, Lev, 77, 103 Wenger, Etienne, 51, 173n1 Woolgar, Steve, 4, 14, 17, 18, 80

199

Plate 1 (figure 2.1) Published fMRI figure from the article by Martin Sereno, Sabrina Pitzalis, and Antigona Martinez entitled “Mapping of Contralateral Space in Retinotopic Coordinates by a Parietal Cortical Area in Humans” (Science 2001;294:1350–1354).

(b)

Plate 2 (figure 7.1) Sereno et al. (2001) illustration: (a) originally submitted fig. 3; (b) revised, published fig. 3.

(a)