Advances in Natural Multimodal Dialogue Systems
Text, Speech and Language Technology VOLUME 30
Series Editors Nancy Ide, Vassar College, New York Jean Véronis, Université de Provence and CNRS, France Editorial Board Harald Baayen, Max Planck Institute for Psycholinguistics, The Netherlands Kenneth W. Church, AT & T Bell Labs, New Jersey, USA Judith Klavans, Columbia University, New York, USA David T. Barnard, University of Regina, Canada Dan Tufis, Romanian Academy of Sciences, Romania Joaquim Llisterri, Universitat Autonòma de Barcelona, Spain Stig Johansson, University of Oslo, Norway Joseph Mariani, LIMSI-CNRS, France
The titles published in this series are listed at the end of this volume.
Advances in Natural Multimodal Dialogue Systems Edited by
Jan C.J. van Kuppevelt Waalre, The Netherlands
Laila Dybkj ær University of Southern Denmark, Odense, Denmark
and
Niels Ole Bernsen University of Southern Denmark, Odense, Denmark
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN-10 ISBN-13 ISBN-10 ISBN-13 ISBN-10 ISBN-13
1-4020-3934-4 (PB) 978-1-4020-3934-8 (PB) 1-4020-3932-8 (HB) 978-1-4020-3032-4 (HB) 1-4020-3933-6 (e-book) 978-1-4020-3933-1 (e-book)
Published by Springer, P.O. Box 17, 3300 AA Dordrecht, The Netherlands. www.springer.com
Printed on acid-free paper
All Rights Reserved © 2005 Springer No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written permission from the Publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Printed in the Netherlands
Contents
Preface
xi
1 Natural and Multimodal Interactivity Engineering - Directions and Needs Niels Ole Bernsen and Laila Dybkjaer 1. Introduction 2. Chapter Presentations 3. NMIE Contributions by the Included Chapters 4. Multimodality and Natural Interactivity References
1 1 2 7 16 19
Part I Making Dialogues More Natural: Empirical Work and Applied Theory 2 Social Dialogue with Embodied Conversational Agents Timothy Bickmore and Justine Cassell 1. Introduction 2. Embodied Conversational Agents 3. Social Dialogue 4. Related Work 5. Social Dialogue in REA 6. A Study Comparing ECA Social Dialogue with Audio-Only Social Dialogue 7. Conclusion References 3 A First Experiment in Engagement for Human-Robot Interaction in Hosting Activities Candace L. Sidner and Myroslava Dzikovska 1. Introduction 2. Hosting Activities 3. What is Engagement? 4. First Experiment in Hosting: A Pointing Robot 5. Making Progress on Hosting Behaviours 6. Engagement for Human-Human Interaction 7. Computational Modelling of Human-Human Hosting and Engagement 8. A Next Generation Mel
v
23 23 25 29 32 36 40 48 49 55 55 56 57 59 62 63 70 73
vi
Advances in Natural Multimodal Dialogue Systems 9. Summary References
Part II
74 74
Annotation and Analysis of Multimodal Data: Speech and Gesture
4 FORM Craig H. Martell 1. Introduction 2. Structure of FORM 3. Annotation Graphs 4. Annotation Example 5. Preliminary Inter-Annotator Agreement Results 6. Conclusion: Applications to HLT and HCI? Appendix: Other Tools, Schemes and Methods of Gesture Analysis References 5 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 2. Study 3. Data Analysis 4. Results 5. Discussion 6. Related Work 7. Future Work 8. Conclusions Appendix: Questionnaire MYST III - EXILE References 6 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 2. Breakdown and Repair 3. Analysing Communicative Co-ordination 4. Discussion References 7 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 2. Methodology and Experimental Set-Up 3. Results: Presentation and Discussion 4. Conclusion References
79 79 80 85 86 88 90 91 95 97 97 99 101 103 106 106 108 108 110 111 113 113 117 125 126 127 131 131 134 141 153 154
Contents 8 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 2. Related Work 3. Multimodal Multisensor Domain 4. Results 5. Single Multimodal Sensor Domain 6. Integration References
vii 159 159 160 162 166 167 175 178
Part III Animated Talking Heads and Evaluation 9 The Psychology and Technology of Talking Heads: Applications in Language Learning Dominic W. Massaro 1. Introduction 2. Facial Animation and Visible Speech Synthesis 3. Speech Science 4. Language Learning 5. Research on the Educational Impact of Animated Tutors 6. Summary References 10 Effective Interaction with Talking Animated Agents in Dialogue Systems Björn Granström and David House 1. Introduction 2. The KTH Talking Head 3. Effectiveness in Intelligibility and Information Presentation 4. Effectiveness in Interaction 5. Experimental Applications 6. The Effective Agent as a Language Tutor 7. Experiments and 3D Recordings for the Expressive Agent References 11 Controlling the Gaze of Conversational Agents Dirk Heylen, Ivo van Es, Anton Nijholt and Betsy van Dijk 1. Introduction 2. Functions of Gaze 3. The Experiment 4. Discussion 5. Conclusion References
183 183 184 191 194 197 210 211 215 215 217 219 223 231 235 237 239 245 245 248 252 258 260 260
viii
Advances in Natural Multimodal Dialogue Systems
Part IV Architectures and Technologies for Advanced and Adaptive Multimodal Dialogue Systems 12 MIND: A Context-Based Multimodal Interpretation Framework in Conversational Systems Joyce Y. Chai, Shimei Pan and Michelle X. Zhou 1. Introduction 2. Related Work 3. MIND Overview 4. Example Scenario 5. Semantics-Based Representation 6. Context-Based Multimodal Interpretation 7. Discussion References 13 A General Purpose Architecture for Intelligent Tutoring Systems Brady Clark, Oliver Lemon, Alexander Gruenstein, Elizabeth Owen Bratt, John Fry, Stanley Peters, Heather Pon-Barry, Karl Schultz, Zack Thomsen-Gray and Pucktada Treeratpituk 1. Introduction 2. An Intelligent Tutoring System for Damage Control 3. An Architecture for Multimodal Dialogue Systems 4. Activity Models 5. Dialogue Management Architecture 6. Benefits of ACI for Intelligent Tutoring Systems 7. Conclusion References 14 MIAMM – A Multimodal Dialogue System using Haptics Norbert Reithinger, Dirk Fedeler, Ashwani Kumar, Christoph Lauer, Elsa Pecourt and Laurent Romary 1. Introduction 2. Haptic Interaction in a Multimodal Dialogue System 3. Visual Haptic Interaction – Concepts in MIAMM 4. Dialogue Management 5. The Multimodal Interface Language (MMIL) 6. Conclusion References 15 Adaptive Human-Computer Dialogue Sorin Dusan and James Flanagan 1. Introduction 2. Overview of Language Acquisition 3. Dialogue Systems 4. Language Knowledge Representation 5. Dialogue Adaptation 6. Experiments
265 265 267 267 268 270 277 282 283 287
287 288 294 295 298 301 302 303 307 308 309 313 319 326 331 331 333 333 334 337 340 341 347
Contents 7. Conclusion References
ix 351 353
16 Machine Learning Approaches to Human Dialogue Modelling Yorick Wilks, Nick Webb, Andrea Setzer, Mark Hepple and Roberta Catizone 1. Introduction 2. Modality Independent Dialogue Management 3. Learning to Annotate Utterances 4. Future work: Data Driven Dialogue Discovery 5. Discussion References
355 357 362 366 367 368
Index
371
355
Preface
The chapters in this book jointly contribute to what we shall call the field of natural and multimodal interactive systems engineering. This is not yet a well-established field of research and commercial development but, rather, an emerging one in all respects. It brings together, in a process that, arguably, was bound to happen, contributors from many different, and often far more established, fields of research and industrial development. To mention but a few, these include speech technology, computer graphics and computer vision. The field’s rapid expansion seems driven by a shared vision of the potential of new interactive modalities of information representation and exchange for radically transforming the world of computer systems, networks, devices, applications, etc. from the GUI (graphical user interface) paradigm into something which will enable a far deeper and much more intuitive and natural integration of computer systems into people’s work and lives. Jointly, the chapters present a broad and detailed picture of where natural and multimodal interactive systems engineering stands today. The book is based on selected presentations made at the International Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems held in Copenhagen, Denmark, in 2002 and sponsored by the European CLASS project. CLASS was initiated on the request of the European Commission with the purpose of supporting and stimulating collaboration among Human Language Technology (HLT) projects as well as between HLT projects and relevant projects outside Europe. The purpose of the workshop was to bring together researchers from academia and industry to discuss innovative approaches and challenges in natural and multimodal interactive systems engineering. The Copenhagen 2002 CLASS workshop was not just a very worthwhile event in an emerging field due to the general quality of the papers presented. It was also largely representative of the state of the art in the field. Given the increasing interest in natural interactivity and multimodality and the excellent quality of the work presented, it was felt to be timely to publish a book reflecting recent developments. Sixteen high-quality papers from the workshop were selected for publication. Content-wise, the chapters in this book illustrate most aspects of natural and multimodal interactive systems engineering: applicable
xi
xii
Advances in Natural Multimodal Dialogue Systems
theory, empirical work, data annotation and analysis, enabling technologies, advanced systems, re-usability of components and tools, evaluation, and future visions. The selected papers have all been reviewed, revised, extended, and improved after the workshop. We are convinced that people who work in natural interactive and multimodal dialogue systems engineering – from graduate students and Ph.D. students to experienced researchers and developers, and no matter exactly which community they come from originally - may find this collection of papers interesting and useful to their own work. We would like to express our sincere gratitude to all those who helped us in preparing this book. Especially we would like to thank all reviewers for their valuable and extensive comments and criticism which have helped improve the quality of the individual chapters as well as the entire book. THE EDITORS
Chapter 1 NATURAL AND MULTIMODAL INTERACTIVITY ENGINEERING - DIRECTIONS AND NEEDS Niels Ole Bernsen and Laila Dybkjær Natural Interactive Systems Laboratory University of Southern Denmark Campusvej 55, 5230 Odense M, Denmark
{nob, laila}@nis.sdu.dk Abstract
This introductory chapter discusses the field of natural and multimodal interactivity engineering and presents the following 15 chapters in this context. A brief presentation of each chapter is given, their contributions to specific natural and multimodal interactivity engineering needs are discussed, and the concepts of multimodality and natural interactivity are explained along with an overview of the modalities investigated in the 15 chapters.
Keywords:
Natural and multimodal interactivity engineering.
1.
Introduction
Chapters 2 through 16 of this book present original contributions to the emerging field of natural and multimodal interactivity engineering (henceforth NMIE). A prominent characteristic of NMIE is that the field is not yet an established field of research and commercial development but, rather, an emerging one in all respects, including applicable theory, experimental results, platforms and development environments, standards (guidelines, de facto standards, official standards), evaluation paradigms, coherence, ultimate scope, enabling technologies for software engineering, general topology of the field itself, ”killer applications”, etc. The NMIE field is vast and brings together practitioners from very many different, and often far more established, fields of research and industrial development, such as signal processing, speech technology, computer graphics, computer vision, human-computer interaction, virtual and augmented reality, non-speech sound, haptic devices, telecommunications, computer games, etc. 1 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 1–19. © 2005 Springer. Printed in the Netherlands.
2
Advances in Natural Multimodal Dialogue Systems
Table 1.1. Needs for progress in natural and multimodal interactivity engineering (NMIE). General
Specific
Understand issues, problems, solutions
Applicable theory: for any aspect of NMIE. Empirical work and analysis: controlled experiments, behavioural studies, simulations, scenario studies, task analysis on roles of, and collaboration among, specific modalities to achieve various benefits. Annotation and analysis: new quality data resources, coding schemes, coding tools, and standards. Future visions: visions, roadmaps, etc., general and per sub-area.
Build systems
Enabling technologies: new basic technologies needed. More advanced systems: new, more complex, versatile, and capable system aspects. Make it easy: Re-usable platforms, components, toolkits, architectures, interface languages, standards, etc.
Evaluate
Evaluate all aspects: of components, systems, technologies, processes, etc.
It may be noted that the fact that a field of research has been established over decades in its own right is fully compatible with many if not most of its practitioners being novices in NMIE. It follows that NMIE community formation is an ongoing challenge for all. Broadly speaking, the emergence of a new systems field, such as NMIE, takes understanding of issues, problems and solutions, knowledge and skills for building (or developing) systems and enabling technologies, and evaluation of any aspect of the process and its results. In the particular case of NMIE, these goals or needs could be made more specific as shown in Table 1.1. Below, we start with a brief presentation of each of the following 15 chapters (Section 2). Taking Table 1.1 as point of departure, Section 3 provides an overview of, and discusses the individual chapters’ contributions to specific NMIE needs. Section 4 explains multimodality and natural interactivity and discusses the modalities investigated in the chapters of this book.
2.
Chapter Presentations
We have found it appropriate to structure the 15 chapters into four parts under four headlines related to how the chapters contribute to the specific NMIE needs listed in Table 1.1. Each chapter has a main emphasis on issues that contribute to NMIE needs captured by the headline of the part of the book to which it belongs. The division can of course not be a sharp one. Several chap-
Natural and Multimodal Interactivity Engineering - Directions and Needs
3
ters include discussions of issues that would make them fit into several parts of the book. Part one focuses on making dialogues more natural and has its main emphasis on experimental work and the application of theory. Part two concerns annotation and analysis of multimodal data, in particular, the modalities of speech and gesture. Part three addresses animated talking heads and related evaluation issues. Part four covers issues in building advanced multimodal dialogue systems, including architectures and technologies.
2.1
Making Dialogues More Natural: Empirical Work and Applied Theory
Two chapters have been categorised under this headline. They both aim at making interaction with a virtual or physical agent more natural. Experimental work is central in both chapters and so is the application of theory. The chapter by Bickmore and Cassell (Chapter 2) presents an empirical study of the social dialogue of an embodied conversational real-estate agent during interaction with users. The study is a comparative one. In one setting, users could see the agent while interacting with it. In the second setting, users could talk to the agent but not see it. The paper presents several interesting findings on social dialogue and interaction with embodied conversational agents. Sidner and Dzikovska (Chapter 3) present empirical results from humanrobot interaction in hosting activities. Their focus is on engagement, i.e. on the establishment and maintenance of a connection between interlocutors and the ending of it when desired. The stationary robot can point, make beat gestures, and move its eyes while conducting dialogue and tutoring on the use of a gas turbine engine shown to the user on a screen. The authors also discuss human-human hosting and engagement. Pursuing and building on theory of human-human engagement and using input from experiments, their idea is to continue to add capabilities to the robot which will make it become better to show engagement.
2.2
Annotation and Analysis of Multimodal Data: Speech and Gesture
The five chapters in this part have a common emphasis on data analysis. While one chapter focuses on annotation of already collected data, the other four chapters all describe experiments with data collection. In all cases, the data are subsequently analysed in order to, e.g., provide new knowledge on conversation or show whether a hypothesis was true or not. Martell (Chapter 4) presents the FORM annotation scheme and illustrates its use. FORM enables annotators to mark up the kinematic information of
4
Advances in Natural Multimodal Dialogue Systems
gestures in videos. Although designed for gesture markup, FORM is also designed to be extensible to markup of speech and other conversational information. The goal is to establish an extensible corpus of annotated videos which can be used for research in various aspects of conversational interaction. In an appendix, Martell provides a brief overview of other tools, schemes and methods for gesture analysis. Corradini and Cohen (Chapter 5) report on a Wizard-of-Oz study in which it was investigated how people use gesture and speech during interaction with a video game when they do not have access to standard input devices. The subjects’ interaction with the game was recorded, transcribed, further coded, and analysed. The primary analysis of the data concerns the users’ use of speech-only, gesture-only and speech and gesture in combination. Moreover, subjective data were collected by asking subjects after the experiment about their modality preferences during interaction. Although subjects’ answers and their actual behaviour in the experiment did not always match, the study indicates a preference for multimodal interaction. The chapter by Healey et al. (Chapter 6) addresses the analysis of humanhuman interaction. The motivation is the lack of support for the design of systems for human-human interaction. They discuss two psycholinguistic approaches and propose a third approach based on the detection and resolution of communication problems. This approach is useful for measuring the effectiveness of human-human interaction across tasks and modalities. A coding protocol for identification of repair phenomena across different modalities is presented followed by evaluation results from testing the protocol on a small corpus of repair sequences. The presented approach has the potential to help in judging the effectiveness of multimodal communication. Carbonell and Kieffer (Chapter 7) report on an experimental study which investigated if oral messages facilitate visual search tasks on a crowded display. Using the mouse, subjects were asked to search and select visual targets in complex scenes presented on the screen. Before the presentation of each scene, the subject would either see the target alone, receive an oral description of the target and spatial information about its position in the scene, or get a combination of the visual and oral target descriptions. Analysis of the data collected suggests that appropriate oral messages can improve search accuracy as well as selection time. However, both objectively and subjectively, multimodal messages were most effective. Darrell et al. (Chapter 8) discuss the problem of knowing who is speaking during multi-speaker interaction with a computer. They present two methods based on geometric and statistical source separation approaches, respectively. These methods are used for audiovisual segmentation of multiple speakers and have been used in experiments. One setup in a conference room used several stereo cameras and a ceiling-mounted microphone array grid. In this case a
Natural and Multimodal Interactivity Engineering - Directions and Needs
5
geometric method was used to identify the speaker. In a second setup, involving use of a handheld device or a kiosk, a single camera and a single omnidirectional microphone were used and a statistical method applied for speaker identification. Data analysis showed that each approach was valuable in the intended domain. However, the authors propose that a combination of the two methods would be of benefit and initial integration efforts are discussed.
2.3
Animated Talking Heads and Evaluation
Three chapters address animated talking heads. These chapters all include experimental data collection and data analysis, and present evaluations of various aspects of talking head technology and its usability. The chapter by Massaro (Chapter 9) concerns computer-assisted speech and language tutors for the deaf, hard of hearing and autistic children. The author presents an animated talking head for visible speech synthesis. The skin of the head can be made transparent so that one can see the tongue and the palate. The technology has been used in a language training program in which the agent guides the user through a number of exercises in vocabulary and grammar. The aim is to improve speech articulation and develop linguistic and phonological awareness in the users. The reported experiments show positive learning results. Granström and House (Chapter 10) describe work on animated talking heads, focusing on the increased intelligibility and efficiency provided by the addition of the talking face which uses text-to-speech synthesis. Results from various studies are presented. The studies include intelligibility tests and perceptual evaluation experiments. Among other things, facial cues to convey, e.g., feedback, turn-taking, and prosodic functions like prominence have been investigated. A number of applications of the talking head are described, including a language tutor. Heylen et al. (Chapter 11) report on how different eye gaze behaviours of a cartoon-like talking face affect the interaction with users. Three versions of the talking face were included in an experiment in which users had to make concert reservations by interacting with the talking face through typed input. One version of the face was aimed to be a close approximation to human-like gaze behaviour. In a second version gaze shifts were kept minimal, and in a third version gaze shifts were random. Evaluation of data from the experiment clearly showed a better performance of, and preference for, the human-like version.
6
2.4
Advances in Natural Multimodal Dialogue Systems
Architectures and Technologies for Advanced and Adaptive Multimodal Dialogue Systems
The last part of this book comprises five chapters which all have a strong focus on aspects of developing advanced multimodal dialogue systems. Several of the chapters present architectures which may be reused across applications, while others emphasise learning and adaptation. The chapter by Chai et al. (Chapter 12) addresses the difficult task of interpreting multimodal user input. The authors propose to use a fine-grained semantic model that characterises the meaning of user input and the overall conversation, and an integrated interpretation approach drawing on context knowledge, such as conversation histories and domain knowledge. These two approaches are discussed in detail and are included in the multimodal interpretation framework presented. The framework is illustrated by a real-estate application in which it has been integrated. Clark et al. (Chapter 13) discuss the application of a general-purpose architecture in support of multimodal interaction with complex devices and applications. The architecture includes speech recognition, natural language understanding, text-to-speech synthesis, an architecture for conversational intelligence, and use of the Open Agent Architecture. The architecture takes advantage of reusability and has been deployed in a number of dialogue systems. The authors report on its deployment in an intelligent tutoring system for shipboard damage control. Details about the tutoring system and the architecture are presented. Reithinger et al. (Chapter 14) present a multimodal system for access to multimedia databases on small handheld devices. Interaction in three languages is supported. Emphasis is on haptic interaction via active buttons combined with spoken input and visual and acoustic output. The overall architecture of the system is explained and so is the format for data exchange between modules. Also, the dialogue manager is described, including its architecture and multimodal fusion issues. Dusan and Flanagan (Chapter 15) address the difficult issue of ensuring sufficient vocabulary coverage in a spoken dialogue system. To overcome the problem that there may always be a need for additional words or word forms, the authors propose a method for adapting the vocabulary of a spoken dialogue system at run-time. Adaptation is done by the user by adding new concepts to existing pre-programmed concept classes and by providing semantic information about the new concepts. Multiple input modalities are available for doing the adaptation. Positive results from preliminary experiments with the method are reported. Wilks et al. (Chapter 16) discuss machine learning approaches to the modelling of human-computer interaction. They first describe a dialogue manager
Natural and Multimodal Interactivity Engineering - Directions and Needs
7
built for multimodal dialogue handling. The dialogue manager uses a kind of stereotypical dialogue patterns, called Dialogue Action Frames, for representation. The authors then describe an analysis module which learns to assign dialogue acts and semantic contents from corpora. The idea is to enable automatic derivation of Dialogue Action Frames, so that the dialog manager will be able to use Dialogue Action Frames that are automatically leaned from corpora.
3.
NMIE Contributions by the Included Chapters
Using the righthand column entries of Table 1.1, Table 1.2 indicates how the 15 chapters in this book contribute to the NMIE field. A preliminary conclusion based on Table 1.2 is that, for an emerging field which is only beginning to be exploited commercially, the NMIE research being done world-wide today is already pushing the frontiers in many of the directions needed. In the following, we discuss the chapter contributions to each of the lefthand entries in Table 1.2.
3.1
Applicable Theory
It may be characteristic of the NMIE field at present that our sample of papers only includes a single contribution of a primarily theoretical nature, i.e. Healey et al., which applies a psycholinguistic model of dialogue to help identify a subset of communication problems in order to judge the effectiveness of multimodal communication. Human-machine communication problems, their nature and identification by human or machine, has recently begun to attract the attention of more than a few NMIE researchers, and it has become quite clear that we need far better understanding of miscommunication in natural and multimodal interaction than we have at present. However, the relative absence of theoretical papers is not characteristic in the sense that the field does not make use of, or even need, applicable theory. On the contrary, a large number of chapters actually do apply existing theory in some form, ranging from empirical generalisations to full-fledged theory of many different kinds. For instance, Bickmore and Cassell test generalisations on the effects on communication of involving embodied conversational agents; Carbonell and Kieffer apply modality theory; Chai et al. apply theories of human-human dialogue to the development of a fined-grained, semanticsbased multimodal dialogue interpretation framework; Massaro applies theories of human learning; and Sidner and Dzikovska draw on conversation and collaboration theory.
8
Advances in Natural Multimodal Dialogue Systems
Table 1.2. NMIE needs addressed by the chapters in this book. Specific to NMIE Applicable theory Empirical work analysis
Contributions and
Annotation and analysis Future visions Enabling technologies
More advanced systems
Make it easy
Evaluate
3.2
No new theory except 6 but plenty of applied theory, e.g. 2, 3. Effects on communication of animated conversational agents. 2, 10, 11 Spoken input in support of visual search. 7 Gesture and speech for video game playing. 5 Multi-speaker speech recognition. 8 Gaze behaviour for more likeable animated interface agents. 2, 11 Audio-visual speech output. 9, 10 Animated talking heads for language learning. 9, 10 Tutoring. 3, 9, 10, 13 Hosting robots. 3 Coding scheme for conversational interaction research. 4 Standard for internal representation of NMIE data codings. 4 Many papers with visions concerning new challenges in their research. Interactive robotics: robots controlled multimodally, tutoring and hosting robots. 3 Multi-speaker speech recognition. 8 Audio-visual speech synthesis for talking heads. 9, 10 Machine learning of language and dialogue acts assignment. 15, 16 Multilinguality. 14 Ubiquitous (mobile) application. 14 On-line observation-based user modelling for adaptivity. 12, 14 Complex natural interactive dialogue management. 12, 13, 14, 16 Machine learning for more advanced dialogue systems. 15, 16 Platform for natural interactivity. 6 Re-usable components (many papers). Architectures for multimodal systems and dialogue management. 3, 12, 13, 14, 16, 16 Multimodal interface language. 14 XML for data exchange. 10, 14 Effects on communication of animated conversational agents. 2, 10, 11 Evaluations of talking heads. 9, 10, 11 Evaluation of audio-visual speech synthesis for learning. 9, 10
Empirical Work and Analysis
Novel theory tends to be preceded by empirical exploration and generalisation. The NMIE field is replete with empirical studies of human-human and human-computer natural and multimodal interaction [Dehn and van Mulken,
Natural and Multimodal Interactivity Engineering - Directions and Needs
9
2000]. By their nature, empirical studies are closer to the process of engineering than is theory development. We build NMIE research systems not only from theory but, perhaps to a far greater extent, from hunches, contextual assumptions, extrapolations from previous experience and untried transfer from different application scenarios, user groups, environments, etc., or even Wizard of Oz studies, which are in themselves a form of empirical study, see, e.g., the chapter by Corradini and Cohen and [Bernsen et al., 1998]. Having built a prototype system, we are keen to find out how far those hunches, etc. got us. Since empirical testing, evaluation, and assessment are integral parts of software and systems engineering, all we have to do is to include ”assumptions testing” in the empirical evaluation of the implemented system which we would be doing anyway. The drawback of empirical studies is that they usually do not generalise much due to the multitude of independent variables involved. This point is comprehensively argued and illustrated for the general case of multimodal and natural interactive systems which include speech in [Bernsen, 2002]. Still, as we tend to work on the basis of only slightly fortified hunches anyway, the results could often serve to inspire fellow researchers to follow them up. Thus, best-practice empirical studies are of major importance in guiding NMIE progress. The empirical chapters in this book illustrate well the points made above. One cluster of findings demonstrate the potential of audio-visual speech output by animated talking heads for child language learning (Massaro) and, more generally, for improving intelligibility and efficiency of human-machine communication, including the substitution of facial animation for the, still-missing, prosody in current speech synthesis systems (Granström and House). In counter-point, so to speak, Darrell et al. convincingly demonstrate the advantage of audio-visual input for tackling an important next step in speech technology, i.e. the recognition of multi-speaker spoken input. Jointly, the three chapters do a magnificent job of justifying the need for natural and multimodal (audiovisual) interaction independently of any psychological or social-psychological argument in favour of employing animated conversational agents. A key question seems to be: for which purpose(s), other than harvesting the benefits of using audio-visual speech input/output described above, do we need to accompany spoken human-computer dialogue with more or less elaborate animated conversational interface agents [Dehn and van Mulken, 2000]? By contrast with spoken output, animated interface agents occupy valuable screen real estate, do not necessarily add information of importance to the users of large classes of applications, and may distract the user from the task at hand. Whilst a concise and comprehensive answer to this question is still pending, Bickmore and Cassell go a long way towards explaining that the introduction of life-like animated interface agents into human-computer spoken dialogue is
10
Advances in Natural Multimodal Dialogue Systems
a tough and demanding proposition. As soon as an agent appears on the display, users tend to switch expectations from talking to a machine to talking to a human. By comparison, the finding of Heylen et al. that users tend to appreciate an animated cartoon agent more if it shows a minimum of human-like gaze behaviour might speak in favour of preferring cartoon-style agents over life-like animated agents because the former do not run the risk of facing our full expectations to human conversational behaviour. Sidner and Dzikovska do not involve a virtual agent but, rather, a robot in the dialogue with the user, so they do not have the problem of an agent occupying part of the screen. But they have still have the behavioural problems of the robot to look into just as, by close analogy, do the people who work with virtual agents. The experiments by Sidner and Dzikovska show that there is still a long way to go before we fully understand and can model the subtle details of human behaviour in dialogue. On the multimodal side of the natural interactivity/multimodality semidivide, several papers address issues of modality collaboration, i.e., how the use of modality combinations could facilitate, or even enable, human-computer interaction tasks that could not be done easily, if at all, using unimodal interaction. Carbonell and Kieffer report on how combined speech and graphics output can facilitate display search, and Corradini and Cohen show how the optional use of different input modalities can improve interaction in a particular virtual environment.
3.3
Annotation and Analysis
It is perhaps not surprising that we are not very capable of predicting what people will do, or how they will behave, when interacting with computer systems using new modality combinations and possibly also new interactive devices. More surprising, however, is the fact that we are often just as ignorant when trying to predict natural interactive behaviours which we have the opportunity to observe every day in ourselves and others, such as: which kinds of gestures, if any, do people perform when they are listening to someone else speaking? This example illustrates that, to understand the ways in which people communicate with one another as well as the ways in which people communicate with the far more limited, current NMI systems, we need extensive studies of behavioural data. The study of data on natural and multimodal interaction is becoming a major research area full of potential for new discoveries. A number of chapters make use of, or refer to, data resources for NMIE, but none of them take a more general view on data resource issues. One chapter addresses NMIE needs for new coding schemes. Martell presents a kinematically-based gesture annotation scheme for capturing the kinematic information in gestures from videos of speakers. Linking the urgent issue of
Natural and Multimodal Interactivity Engineering - Directions and Needs
11
new, more powerful coding tools with the equally important issue of standardisation, Martell proposes a standard for the internal representation of NMIE codings.
3.4
Future Visions
None of the chapters have a particular focus on future visions for the NMIE field. However, many authors touch on future visions, e.g., their descriptions of future work and what they would like to achieve. This includes the important driving role of re-usable platforms, tools, and components for making rapid progress. Moreover, there are several hints at the future importance to NMIE of two generic enabling technologies which are needed to extend spoken dialogue systems to full natural interactive systems. These technologies are (i) computer vision for processing camera input, and (ii) computer animation systems. It is only recently that the computer vision community has begun to address issues of natural interactive and multimodal human-system communication, and there is a long way to go before computer vision can parallel speech recognition as a major input medium for NMIE. The chapters by Massaro and Granström and House illustrate current NMIE efforts to extend natural and multimodal interaction beyond traditional information systems to new major application areas, such as training and education which has been around for a while already, notably in the US-dominated paradigm of tutoring systems using animated interface agents, but also to edutainment and entertainment. While the GUI, including the current WWW, might be said to have the edutainment potential of a schoolbook or newspaper, NMIE systems have the much more powerful edutainment potential of brilliant teachers, comedians, and exiting human-human games.
3.5
Enabling Technologies
An enabling technology is often developed over a long time by some separate community, such as by the speech recognition community from the 1950s to the late 1980s. Having matured to the point at which practical applications become possible, the technology transforms into an omnipresent tool for system developers, as is the case with speech recognition technology today. NMIE needs a large number of enabling technologies and these are currently at very different stages of maturity. Several enabling technologies, some of which are at an early stage and some of which are finding their way into applications, are presented in this book in the context of their application to NMIE problems, including robot interaction and agent technology, multi-speaker interaction and recognition, machine learning, and talking face technology. Sidner and Dzikovska focus on robot interaction in the general domain of ”hosting”, i.e., where a virtual or physical agent provides guidance, education,
12
Advances in Natural Multimodal Dialogue Systems
or entertainment based on collaborative goals negotiation and subsequent action. A great deal of work remains to be done before robot interaction becomes natural in any approximate sense of the term. For instance, the robot’s spoken dialogue capabilities must be strongly improved and so must its embodied appearance and global communicative behaviours. In fact, Sidner and Dzikovska make some of the same conclusions as Bickmore and Cassell, namely that agents need to become far more human-like in all or most respects before they are really appreciated by humans. Darrell et al. address the problem in multi-speaker interaction of knowing who is addressing the computer when. Their approach is to use a combination of microphones and computer vision to find out who is talking. Developers of spoken dialogue applications must cope with problems resulting from vocabulary and grammar limitations and from difficulties in enabling much of the flexibility and functionality inherent in human-human communication. Despite having carried out systematic testing, the developer often finds that words are missing when a new user addresses the application. Dusan and Flanagan propose machine learning as a way to overcome part of this problem. Using machine learning, the system can learn new words and grammars taught to it by the user in a well-defined way. Wilks et al. address machine learning or transformation-based learning - in the context of assigning dialogue acts as part of an approach to improved dialogue modelling. In another part of their approach, Wilks et al. consider the use of dialogue action frames, i.e., a set of stereotypical dialogue patterns which perhaps may be learned from corpus data, as a means for flexibly switching back and forth between topics during dialogue. Granström and House and Massaro describe the gain in intelligibility that can be obtained by combining speech synthesis with a talking face. There is still much work to do both on synthesis and face articulation. For most languages, speech synthesis is still not very natural to listen to and if one wants to develop a particular voice to fit a certain animated character, this is not immediately possible with today’s technology. With respect to face articulation, faces need to become much more natural in terms of, e.g., gaze, eyebrow movements, lip and mouth movements, and head movements, as this seems to influence users’ perception of the interaction, cf. the chapters by Granström and House and Heylen et al.
3.6
More Advanced Systems
Enabling technologies for NMIE are often component technologies, and their description, including state of the art, current research challenges, and unsolved problems, can normally be made in a relatively systematic and focused manner. It is far more difficult to systematically describe the complexity
Natural and Multimodal Interactivity Engineering - Directions and Needs
13
of the constant push in research and industry towards exploring and exploiting new NMIE application types and new application domains, addressing new user populations, increasing the capabilities of systems in familiar domains of application, exploring known technologies with new kinds of devices, etc. In general, the picture is one of pushing present boundaries in all or most directions. During the past few years, a core trend in NMIE has been to combine different modalities in order to build more complex, versatile and capable systems, getting closer to natural interactivity than is possible with only a single modality. This trend is reflected in several chapters. Part of the NMIE paradigm is that systems must be available whenever and wherever convenient and useful, making ubiquitous computing an important application domain. Mobile devices, such as mobile phones, PDAs, and portable computers of any (portable) size have become popular and are rapidly gaining functionality. However, the interface and interaction affordances of small devices require careful consideration. Reithinger et al. present some of those considerations in the context of providing access to large amounts of data about music. It can be difficult for users to know how to interact with new NMIE applications. Although not always very successful in practice, the classical GUI system has the opportunity to present its affordances in static graphics (including text) before the user chooses how to interact. A speech-only system, by contrast, cannot do that because of the dynamic and transitory nature of acoustic modalities. NMIE systems, in other words, pose radically new demands on how to support the user prior to, and during, interaction. Addressing this problem, several chapters mention user modelling or repositories of user preferences built on the basis of interactions with a system, cf. the chapters by Chai et al. and Reithinger et al. Machine learning, although another example of less-than-expected pace of development during the past 10 years, has great potential for increasing interaction support. In an advanced application of machine learning, Dusan and Flanagan propose to increase the system’s vocabulary and grammar by letting users teach the system new words and their meaning and use. Wilks et al. use machine learning as part of an approach to more advanced dialogue modelling. Increasingly advanced systems require increasingly complex dialogue management, cf. the chapters by Chai et al., Clark et al., and Wilks et al. Lifelikeness of animated interface agents and conversational dialogue are among the key challenges in achieving the NMIE vision. Multilinguality of systems is an important NMIE goal. Multilingual applications are addressed by Reithinger et al. In their case, the application is running on a handheld device. Multi-speaker input speech is mentioned by Darrell et al. For good reason, recognition of multi-speaker input has become a lively research topic. We
14
Advances in Natural Multimodal Dialogue Systems
need solutions in order to, e.g., build meeting minute-takers, separate the focal speaker’s input from that of other speakers, exploit the huge potential of spoken multi-user applications, etc.
3.7
Make It Easy
Due to the complexity of multimodal natural interaction, it is becoming dramatically important to be able to build systems as easily as possible. It seems likely that no single research lab or development team in industry, even including giants such as Microsoft, is able to master all of the enabling technologies required for broad-scale NMIE progress. To advance efficiently, everybody needs access to those system components, and their built-in know-how, which are not in development focus. This implies strong attention to issues, such as re-usable platforms, components and architectures, development toolkits, interface languages, data formats, and standardisation. Clark et al. have used the Open Agent Architecture (OAA, http://www.ai.sri.com/∼oaa/), a framework for integrating heterogeneous software agents in a distributed environment. What OAA and other architectural frameworks, such as CORBA (http://www.corba.org/), aim to do is provide a means for modularisation, synchronous and asynchronous communication, well-defined inter-module communication via some interface language, such as IDL (CORBA) or ICL (OAA), and the possibility of implementation in a distributed environment. XML (Extensible Markup Language) is a simple, flexible text format derived from SGML (ISO 8879) which has become popular as, among other things, a message exchange format, cf. Reithinger et al. and Granström and House. Using XML for wrapping inter-module messages is one way to overcome the problem of different programming languages used for implementing different modules. Some chapters express a need for reusable components. Many of the applications described include off-the-shelf software, including components developed in other projects. This is particularly true for mature enabling technologies, such as speech recognition and synthesis components. As regards multimodal dialogue management, there is an expressed need for reuse in, e.g., the chapter by Clark et al. who discuss a reusable dialogue management architecture in support of multimodal interaction. In conclusion, there are architectures, platforms, and software components available which facilitate the building of new NMIE applications, and standards are underway for certain aspects. There is still much work to be done on standardisation, new and better platforms, and improvement of component software. In addition, we need, in particular, more and better toolkits in support of system development and a better understanding of those components which
Natural and Multimodal Interactivity Engineering - Directions and Needs
15
cannot be bought off-the-shelf and which are typically difficult to reuse, such as dialogue managers. Advancements such as these are likely to require significant corpus work. Corpora with tools and annotation schemes as described by Martell are exactly what is needed in this context.
3.8
Evaluate
Software systems and components evaluation is a broad area, ranging from technical evaluation over usability evaluation to customer evaluation. Customer evaluation has never been a key issue in research but has, rather, tended to be left to the marketing departments of companies. Technical evaluation and usability evaluation, including evaluation of functionality from both perspectives, are, on the other hand, considered important research issues. The chapters show a clear trend towards focusing on usability evaluation and comparative performance evaluation. It is hardly surprising that performance evaluation and usability issues are considered key topics today. We know little about what happens when we move towards increasingly multimodal and natural interactive systems, both as regards how these new systems will perform compared to alternative solutions and how the systems will be received and perceived by their users. We only know that a technically optimal system is not sufficient to guarantee user satisfaction. Comparative performance evaluation objectively compares users’ performance on different systems with respect to, e.g., how well they understand speech-only versus speech combined with a talking face or with an embodied animated agent, cf. Granström and House, Massaro, and Bickmore and Cassell. The usability issues evaluated all relate to users’ perception of a particular system and include parameters, such as life-likeness, credibility, reliability, efficiency, personality, ease of use, and understanding quality, cf. Heylen et al. and Bickmore and Cassell. Two chapters address how the intelligibility of what is being said can be increased through visual articulation, cf. Granström and House and Massaro. Granström and House have used a talking head in several applications, including tourist information, real estate (apartment) search, aid for the hearing impaired, education, and infotainment. Evaluation shows a significant gain in intelligibility for the hearing impaired. Eyebrow and head movement enhance perception of emphasis and syllable prominence. Over-articulation may be useful as well when there are special needs for intelligibility. The findings of Massaro support these promising conclusions. His focus is on applications for the hard-of-hearing, children with autism, and child language learning more generally. Granström and House also address the increase in efficiency of communication/interaction produced by using an animated talking head. Probably,
16
Advances in Natural Multimodal Dialogue Systems
naturalness is a key point here. This is suggested by Heylen et al. who made controlled experiments on the effects of different eye gaze behaviours of a cartoon-like talking face on the quality of human-agent dialogues. The most human-like agent gaze behaviour led to higher appreciation of the agent and more efficient task performance. Bickmore and Cassell evaluate the effects on communication of an embodied conversational real-estate agent versus an over-the-phone version of the same system, cf. also [Cassell et al., 2000]. In each condition, two variations of the system was available. One would be fully task-oriented while the second version would include some small-talk options. In general, users liked the system better in the phone condition. In the phone condition, subjects appreciated the small-talk whereas, in the embodied condition, subjects wanted to get down to business. The implication is that agent embodiment has strong effects on the interlocutors. Users tend to compare their animated interlocutors with humans rather than machines. To work with users, animated agents need considerably more naturalness and personally attractive features communicated non-verbally. This imposes a tall research agenda on both speech and non-verbal output, requiring conversational abilities both verbally and nonverbally. Jointly, the chapters on evaluation demonstrate a broad need for performance evaluation, comparative as well as non-comparative, that can inform us on the possible benefits and shortcomings of new natural interactive and multimodal systems. The chapters show a similar need for usability evaluation that can help us find out how users perceive these new systems, and a need for finding ways in which usability and user satisfaction might be correlated with technical aspects in order for the former to be derived from the latter.
4.
Multimodality and Natural Interactivity
Conceptually, NMIE combines natural interactive and multimodal systems and components engineering. While both concepts, natural interactivity and multimodality, have a long history, it would seem that they continue to sit somewhat uneasily side by side in the minds of most of us. Multimodality is the idea of being able to choose any input/output modality or combination of input/output modalities for optimising interaction with the application at hand, such as speech input for many heads-up, hands-occupied applications, speech and haptic input/output for applications for the blind, etc. A modality is a particular way of representing input or output information in some physical medium, such as something touchable, light, sound, or the chemistry for producing olfaction and gustation [Bernsen, 2002], see also the chapter by Carbonell and Kieffer. The physical medium of the speech modalities, for instance, is sound or acoustics but this medium obviously enables the trans-
Natural and Multimodal Interactivity Engineering - Directions and Needs
17
mission of information in many acoustic modalities other than speech, such as earcons, music, etc. The term multimodality thus refers to any possible combination of elementary or unimodal modalities. Compared to multimodality, the notion of natural interactivity appears to be the more focused of the two. This is because natural interactivity comes with a focused vision of the future of interaction with computer systems as well as a relatively well-defined set of modalities required for the vision to become reality. The natural interactivity vision is that of humans communicating with computer systems in the same ways in which humans communicate with one another. Thus, natural interactivity specifically emphasises human-system communication involving the following input/output modalities used in situated human-human communication: speech, gesture, gaze, facial expression, head and body posture, and object manipulation as integral part of the communication (or dialogue). As the objects being manipulated may themselves represent information, such as text and graphics input/output objects, natural interaction subsumes the GUI paradigm. Technologically, the natural interactivity vision is being pursued vigorously by, among others, the emerging research community in talking faces and embodied conversational agents as illustrated in the chapters by Bickmore and Cassell, Granström and House, Heylen et al., and Massaro. An embodied conversational agent may be either virtual or a robot, cf. the chapter by Sidner and Dzikovska. A weakness in our current understanding of natural interactivity is that it is not quite clear where to draw the boundary between the natural interactivity modalities and all those other modalities and modality combinations which could potentially be of benefit to human-system interaction. For instance, isn’t pushing a button on the mouse or otherwise, although never used in humanhuman communication for the simple reason that humans do not have communicative buttons on them, as natural as speaking? If it is, then, perhaps, all or most research on useful multimodal input/output modality combinations is also research into natural interactivity even if the modalities addressed are not being used in human-human communication? In addition to illustrating the need for more and better NMIE theory, the point just made may explain the uneasy conceptual relationship among the two paradigms of natural interactivity and multimodality. In any case, we have decided to combine the paradigms and address them together as natural and multimodal interactivity engineering. Finally, by NMI ’engineering’ we primarily refer to software engineering. It follows that the expression ’natural and multimodal interactivity engineering’ primarily represents the idea of creating a specialised branch of software engineering for the field addressed in this book. It is important to add, however, that NMIE enabling technologies are being developed in fields whose practitioners do not tend to regard themselves as doing software engineering, such as signal processing. For instance, the recently launched European Network of
18
Advances in Natural Multimodal Dialogue Systems
Excellence SIMILAR (http://www.similar.cc) addresses signal processing for natural and multimodal interaction.
4.1
Modalities Investigated in This Book
We argued above that multimodality includes all possible modalities for the representation and exchange of information among humans and between humans and computer systems, and that natural interactivity includes a rather vaguely defined, large sub-set of those modalities. Within this wide space of unimodal modalities and modality combinations, it may be useful to look at the modalities actually addressed in the following chapters. These are summarised by chapter in Table 1.3. Table 1.3. Modalities addressed in the included chapters (listed by chapter number plus first listed author). Chapter
Input
Output
2. Bickmore
speech, gesture (via camera) vs. speech-only speech, mouse clicks, (new version includes face and gesture input via camera)
embodied conversational agent + images vs. speech-only + images robot pointing and beat gestures, speech, gaze N/A video game
6. Healey 7. Carbonell 8. Darrell
gesture speech, gesture, object manipulation/manipulative gesture N/A gesture (mouse) speech, camera-based graphics
9. Massaro
mouse, speech
10. Granström
N/A
11. Heylen
typed text
12. Chai 13. Clark 14. Reithinger
speech, text, gesture (pointing) speech, gesture (pointing) speech, haptic buttons
15. Dusan
speech, keyboard, mouse, penbased drawing and pointing, camera speech and possibly other modalities
3. Sidner
4. Martell 5. Corradini
16. Wilks
N/A speech, graphics N/A audio-visual speech synthesis, talking head, images, text audio-visual speech synthesis, talking head talking head, gaze speech, graphics speech, text, graphics music, speech, text, graphics, tactile rythm speech, graphics, text display
speech and possibly other modalities. Focus is on dialogue modelling so input/output modalities are not discussed in detail
Natural and Multimodal Interactivity Engineering - Directions and Needs
19
Combined speech input/output which, in fact, means spoken dialogue almost throughout, is addressed in about half of the chapters (Bickmore and Cassell, Chai et al., Clark et al., Corradini and Cohen, Dusan and Flanagan, Reithinger et al., and Sidner and Dzikovska). Almost two thirds of the chapters address gesture input in some form (Bickmore and Cassell, Chai et al., Clark et al., Corradini and Cohen, Darrell et al., Dusan and Flanagan, Martell, Reithinger et al., and Sidner and Dzikovska). Five chapters address output modalities involving talking heads, embodied animated agents, or robots (Bickmore and Cassell, Granström and House, Heylen et al., Massaro, and Sidner and Dzikovska). Three chapters (Darrell et al., Bickmore and Cassell, and Sidner and Dzikovska) address computer vision. Dusan and Flanagan also mention that their system has camera-based input. Facial expression of emotion is addressed by Granström and House. Despite its richness and key role in natural interactivity, input or output speech prosody is hardly discussed. Granström and House discuss graphical ways of replacing missing output speech prosody by facial expression means. In general, the input and output modalities and their combinations discussed would appear representative of the state-of-the-art in NMIE. The authors make quite clear how far we are from mastering the very large number of potentially useful unimodal ”compounds” theoretically, in input recognition, in output generation, as well as in understanding and generation.
References Bernsen, N. O. (2002). Multimodality in Language and Speech Systems - From Theory to Design Support Tool. In Granström, B., House, D., and Karlsson, I., editors, Multimodality in Language and Speech Systems, pages 93–148. Dordrecht: Kluwer Academic Publishers. Bernsen, N. O., Dybkjær, H., and Dybkjær, L. (1998). Designing Interactive Speech Systems. From First Ideas to User Testing. Springer Verlag. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., and Yan, H. (2000). Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Embodied Conversational Agents, pages 29–63. Cambridge, MA: MIT Press. Dehn, D. and van Mulken, S. (2000). The Impact of Animated Interface Agents: A Review of Empirical Research. International Journal of Human-Computer Studies, 52:1–22.
PART I
MAKING DIALOGUES MORE NATURAL: EMPIRICAL WORK AND APPLIED THEORY
Chapter 2 SOCIAL DIALOGUE WITH EMBODIED CONVERSATIONAL AGENTS Timothy Bickmore Northeastern University, USA
[email protected]
Justine Cassell Northwestern University, USA
[email protected]
Abstract
The functions of social dialogue between people in the context of performing a task is discussed, as well as approaches to modelling such dialogue in embodied conversational agents. A study of an agent’s use of social dialogue is presented, comparing embodied interactions with similar interactions conducted over the phone, assessing the impact these media have on a wide range of behavioural, task and subjective measures. Results indicate that subjects’ perceptions of the agent are sensitive to both interaction style (social vs. task-only dialogue) and medium.
Keywords:
Embodied conversational agent, social dialogue, trust.
1.
Introduction
Human-human dialogue does not just comprise statements about the task at hand, about the joint and separate goals of the interlocutors, and about their plans. In human-human conversation participants often engage in talk that, on the surface, does not seem to move the dialogue forward at all. However, this talk – about the weather, current events, and many other topics without significant overt relationship to the task at hand – may, in fact, be essential to how humans obtain information about one another’s goals and plans and decide whether collaborative work is worth engaging in at all. For example, realtors use small talk to gather information to form stereotypes (a collection 23 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 23–54. © 2005 Springer. Printed in the Netherlands.
24
Advances in Natural Multimodal Dialogue Systems
of frequently co-occurring characteristics) of their clients – people who drive minivans are more likely to have children, and therefore to be searching for larger homes in neighbourhoods with good schools. Realtors – and salespeople in general – also use small talk to increase intimacy with their clients, to establish their own expertise, and to manage how and when they present information to the client [Prus, 1989]. Nonverbal behaviour plays an especially important role in such social dialogue, as evidenced by the fact that most important business meetings are still conducted face-to-face rather than on the phone. This intuition is backed up by empirical research; several studies have found that the additional nonverbal cues provided by video-mediated communication do not effect performance in task-oriented interactions, but in interactions of a more social nature, such as getting acquainted or negotiation, video is superior [Whittaker and O’Conaill, 1997]. These studies have found that for social tasks, interactions were more personalized, less argumentative and more polite when conducted via video-mediated communication, that participants believed video-mediated (and face-to-face) communication was superior, and that groups conversing using video-mediated communication tended to like each other more, compared to audio-only interactions. Together, these findings indicate that if we are to develop computer agents capable of performing as well as humans on tasks such as real estate sales then, in addition to task goals such reliable and efficient information delivery, they must have the appropriate social competencies designed into them. Further, since these competencies include the use of nonverbal behaviour for conveying communicative and social cues, then our agents must have the capability of producing and recognizing nonverbal cues in simulations of face-to-face interactions. We call agents with such capabilities “Embodied Conversational Agents” or “ECAs.” The current chapter extends previous work which demonstrated that social dialogue can have a significant impact on a user’s trust of an ECA [Bickmore and Cassell, 2001], by investigating whether these results hold in the absence of nonverbal cues. We present the results of a study designed to determine whether the psychological effects of social dialogue – namely to increase trust and associated positive evaluations – vary when the nonverbal cues provided by the embodied conversational agent are removed. In addition to varying medium (voice only vs. embodied) and dialogue style (social dialogue vs. taskonly) we also assessed and examined effects due to the user’s personality along the introversion/extroversion dimension, since extroversion is one indicator of a person’s comfort level with face-to-face interaction.
Social Dialogue with Embodied Conversational Agents
2.
25
Embodied Conversational Agents
Embodied conversation agents are animated anthropomorphic interface agents that are able to engage a user in real-time, multimodal dialogue, using speech, gesture, gaze, posture, intonation, and other verbal and nonverbal behaviours to emulate the experience of human face-to-face interaction [Cassell et al., 2000c]. The nonverbal channels are important not only for conveying information (redundantly or complementarily with respect to the speech channel), but also for regulating the flow of the conversation. The nonverbal channel is especially crucial for social dialogue, since it can be used to provide such social cues as attentiveness, positive affect, and liking and attraction, and to mark shifts into and out of social activities [Argyle, 1988].
2.1
Functions versus Behaviours
Embodiment provides the possibility for a wide range of behaviours that, when executed in tight synchronization with language, carry out a communicative function. It is important to understand that particular behaviours, such as the raising of the eyebrows, can be employed in a variety of circumstances to produce different communicative effects, and that the same communicative function may be realized through different sets of behaviours. It is therefore clear that any system dealing with conversational modelling has to handle function separately from surface-form or run the risk of being inflexible and insensitive to the natural phases of the conversation. Here we briefly describe some of the fundamental communication categories and their functional sub-parts, along with examples of nonverbal behaviour that contribute to their successful implementation. Table 2.1 shows examples of mappings from communicative function to particular behaviours and is based on previous research on typical North American nonverbal displays, mainly [Chovil, 1991; Duncan, 1974; Kendon, 1980].
Conversation initiation and termination Humans partake in an elaborate ritual when engaging and disengaging in conversations [Kendon, 1980]. For example, people will show their readiness to engage in a conversation by turning towards the potential interlocutors, gazing at them and then exchanging signs of mutual recognition typically involving a smile, eyebrow movement and tossing the head or waving of the arm. Following this initial synchronization stage, or distance salutation, the two people approach each other, sealing their commitment to the conversation through a close salutation such as a handshake accompanied by a ritualistic verbal exchange. The greeting phase ends when the two participants re-orient their bodies, moving away from a face-on orientation to stand at an angle. Terminating a conversation similarly moves through stages, starting with non-verbal cues, such as orientation shifts
26
Advances in Natural Multimodal Dialogue Systems
Table 2.1. Some examples of conversational functions and their behaviour realization [Cassell et al., 2000b]. Communicative Functions Initiation and termination Reacting Inviting Contact Distance Salutation Close Salutation Break Away Farewell Turn-Taking Give Turn Wanting Turn Take Turn Feedback Request Feedback Give Feedback
Communicative Behaviour Short Glance Sustained Glance, Smile Looking, Head Toss/Nod, Raise Eyebrows, Wave, Smile Looking, Head Nod, Embrace or Handshake, Smile Glance Around Looking, Head Nod, Wave Looking, Raise Eyebrows (followed by silence) Raise Hands into gesture space Glance Away, Start talking Looking, Raise Eyebrows Looking, Head Nod
or glances away and cumulating in the verbal exchange of farewells and the breaking of mutual gaze.
Conversational turn-taking and interruption Interlocutors do not normally talk at the same time, thus imposing a turn-taking sequence on the conversation. The protocols involved in floor management – determining whose turn it is and when the turn should be given to the listener – involve many factors including gaze and intonation [Duncan, 1974]. In addition, listeners can interrupt a speaker not only with voice, but also by gesturing to indicate that they want the turn. Content elaboration and emphasis Gestures can convey information about the content of the conversation in ways for which the hands are uniquely suited. For example, the two hands can better indicate simultaneity and spatial relationships than the voice or other channels. Probably the most commonly thought of use of the body in conversation is the pointing (deictic) gesture, possibly accounting for the fact that it is also the most commonly implemented for the bodies of animated interface agents. In fact, however, most conversations don’t involve many deictic gestures [McNeill, 1992] unless the interlocutors are discussing a shared task that is currently present. Other conversational gestures also convey semantic and pragmatic information. Beat gestures are small, rhythmic baton like movements of the hands that do not change in form with the content of the accompanying speech. They serve a pragmatic func-
Social Dialogue with Embodied Conversational Agents
27
tion, conveying information about what is “new” in the speaker’s discourse. Iconic and metaphoric gestures convey some features of the action or event being described. They can be redundant or complementary relative to the speech channel, and thus can convey additional information or provide robustness or emphasis with respect to what is being said. Whereas iconics convey information about spatial relationships or concepts, metaphorics represent concepts which have no physical form, such as a sweeping gesture accompanying “the property title is free and clear.”
Feedback and error correction During conversation, speakers can nonverbally request feedback from listeners through gaze and raised eyebrows and listeners can provide feedback through head nods and paraverbals (“uh-huh”, “mmm”, etc.) if the speaker is understood, or a confused facial expression or lack of positive feedback if not. The listener can also ask clarifying questions if they did not hear or understand something the speaker said.
2.2
Interactional versus Propositional Behaviours
The mapping from form (behaviour) to conversational function relies on a fundamental division of conversational goals: contributions to a conversation can be propositional and interactional. Propositional information corresponds to the content of the conversation. This includes meaningful speech as well as hand gestures and intonation used to complement or elaborate upon the speech content (gestures that indicate the size in the sentence “it was this big” or rising intonation that indicates a question with the sentence “you went to the store”). Interactional information consists of the cues that regulate conversational process and includes a range of nonverbal behaviours (quick head nods to indicate that one is following) as well as regulatory speech (“huh?”, “Uhhuh”). This theoretical stance allows us to examine the role of embodiment not just in task- but also process-related behaviours such as social dialogue [Cassell et al., 2000b].
2.3
REA
Our platform for conducting research into embodied conversational agents is the REA system, developed in the Gesture and Narrative Language Group at the MIT Media Lab [Cassell et al., 2000a]. REA is an embodied, multi-modal real-time conversational interface agent which implements the conversational protocols described above in order to make interactions as natural as face-toface conversation with another person. In the current task domain, REA acts as a real estate salesperson, answering user questions about properties in her database and showing users around the virtual houses (Figure 2.1).
28
Advances in Natural Multimodal Dialogue Systems
Figure 2.1.
User interacting with REA.
REA has a fully articulated graphical body, can sense the user passively through cameras and audio input, and is capable of speech with intonation, facial display, and gestural output. The system currently consists of a large projection screen on which REA is displayed and which the user stands in front of. Two cameras mounted on top of the projection screen track the user’s head and hand positions in space. Users wear a microphone for capturing speech input. A single SGI Octane computer runs the graphics and conversation engine of REA, while several other computers manage the speech recognition and generation and image processing. REA is able to conduct a conversation describing the features of the task domain while also responding to the users’ verbal and non-verbal input. When the user makes cues typically associated with turn taking behaviour such as gesturing, REA allows herself to be interrupted, and then takes the turn again when she is able. She is able to initiate conversational error correction when she misunderstands what the user says, and can generate combined voice, facial expression and gestural output. REA’s responses are generated by an incremental natural language generation engine based on [Stone and Doran, 1997] that has been extended to synthesize redundant and complementary gestures synchronized with speech output [Cassell et al., 2000b]. A simple discourse
Social Dialogue with Embodied Conversational Agents
29
model is used for determining which speech acts users are engaging in, and resolving and generating anaphoric references.
3.
Social Dialogue
Social dialogue is talk in which interpersonal goals are foregrounded and task goals – if existent – are backgrounded. One of the most familiar contexts in which social dialogue occurs is in human social encounters between individuals who have never met or are unfamiliar with each other. In these situations conversation is usually initiated by “small talk” in which “light” conversation is made about neutral topics (e.g., weather, aspects of the interlocutor’s physical environment) or in which personal experiences, preferences, and opinions are shared [Laver, 1981]. Even in business or sales meetings, it is customary (at least in American culture) to begin with some amount of small talk before “getting down to business”.
3.1
The Functions of Social Dialogue
The purpose of small talk is primarily to build rapport and trust among the interlocutors, provide time for them to “size each other up”, establish an interactional style, and to allow them to establish their reputations [Dunbar, 1996]. Although small talk is most noticeable at the margins of conversational encounters, it can be used at various points in the interaction to continue to build rapport and trust [Cheepen, 1988], and in real estate sales, a good agent will continue to focus on building rapport throughout the relationship with a buyer [Garros, 1999]. Small talk has received sporadic treatment in the linguistics literature, starting with the seminal work of Malinowski who defined “phatic communion” as “a type of speech in which ties of union are created by a mere exchange of words”. Small talk is the language used in free, aimless social intercourse, which occurs when people are relaxing or when they are accompanying “some manual work by gossip quite unconnected with what they are doing” [Malinowski, 1923]. Jacobson also included a “phatic function” in his well-known conduit model of communication, that function being focused on the regulation of the conduit itself (as opposed to the message, sender, or receiver) [Jakobson, 1960]. More recent work has further characterized small talk by describing the contexts in which it occurs, topics typically used, and even grammars which define its surface form in certain domains [Cheepen, 1988; Laver, 1975; Schneider, 1988]. In addition, degree of “phaticity” has been proposed as a persistent goal which governs the degree of politeness in all utterances a speaker makes, including task-oriented ones [Coupland et al., 1992].
30
3.2
Advances in Natural Multimodal Dialogue Systems
The Relationship between Social Dialogue and Trust
Figure 2.2 outlines the relationship between small talk and trust. REA’s dialogue planner represents the relationship between her and the user using a multi-dimensional model of interpersonal relationship based on [Svennevig, 1999]: familiarity describes the way in which relationships develop through the reciprocal exchange of information, beginning with relatively nonintimate topics and gradually progressing to more personal and private topics. The growth of a relationship can be represented in both the breadth (number of topics) and depth (public to private) of information disclosed [Altman and Taylor, 1973]. solidarity is defined as “like-mindedness” or having similar behaviour dispositions (e.g., similar political membership, family, religions, profession, gender, etc.), and is very similar to the notion of social distance used by Brown and Levinson in their theory of politeness [Brown and Levinson, 1978]. There is a correlation between frequency of contact and solidarity, but it is not necessarily a causal relation [Brown and Levinson, 1978; Brown and Gilman, 1972]. affect represents the degree of liking the interactants have for each other, and there is evidence that this is an independent relational attribute from the above three [Brown and Gilman, 1989]. The mechanisms by which small talk are hypothesized to effect trust include facework, coordination, building common ground, and reciprocal appreciation.
Facework The notion of “face” is “the positive social value a person effectively claims for himself by the social role others assume he has taken during a particular contact” [Goffman, 1967]. Interactants maintain face by having their social role accepted and acknowledged. Events which are incompatible with their line are “face threats” and are mitigated by various corrective measures if they are not to lose face. Small talk avoids face threat (and therefore maintains solidarity) by keeping conversation at a safe level of depth. Coordination The process of interacting with a user in a fluid and natural manner may increase the user’s liking of the agent, and user’s positive affect, since the simple act of coordination with another appears to be deeply gratifying. “Friends are a major source of joy, partly because of the enjoyable things they do together, and the reason that they are enjoyable is perhaps the coordination.” [Argyle, 1990]. Small talk increases coordination between the two participants by allowing them to synchronize short units of talk and nonverbal acknowledgement (and therefore leads to increased liking and positive affect).
Social Dialogue with Embodied Conversational Agents
31
Figure 2.2. How small talk effects trust [Cassell and Bickmore, 2003].
Building common ground Information which is known by all interactants to be shared (mutual knowledge) is said to be in the “common ground” [Clark, 1996]. The principle way for information to move into the common ground is via face-to-face communication, since all interactants can observe the recognition and acknowledgment that the information is in fact mutually shared. One strategy for effecting changes to the familiarity dimension of the relationship model is for speakers to disclose personal information about themselves – moving it into the common ground – and induce the listener to do the same. Another strategy is to talk about topics that are obviously in the common ground – such as the weather, physical surroundings, and other topics available in the immediate context of utterance. Small talk establishes common ground (and therefore increases familiarity) by discussing topics that are clearly in the context of utterance. Reciprocal appreciation In small talk, demonstrating appreciation for and agreement with the contributions of one’s interlocutor is obligatory. Performing this aspect of the small talk ritual increases solidarity by showing mutual agreement on the topics discussed.
32
3.3
Advances in Natural Multimodal Dialogue Systems
Nonverbal Behaviour in Social Dialogue
According to Argyle, nonverbal behaviour is used to express emotions, to communicate interpersonal attitudes, to accompany and support speech, for self presentation, and to engage in rituals such as greetings [Argyle, 1988]. Of these, coverbal and emotional display behaviours have received the most attention in the literature on embodied conversational agents and facial and character animation in general, e.g. [Cassell et al., 2000c]. Next to these, the most important use of nonverbal behaviour in social dialogue is the display of interpersonal attitude [Argyle, 1988]. The display of positive or negative attitude can greatly influence whether we approach someone or not and our initial perceptions of them if we do. The most consistent finding in this area is that the use of nonverbal “immediacy behaviours” – close conversational distance, direct body and facial orientation, forward lean, increased and direct gaze, smiling, pleasant facial expressions and facial animation in general, nodding, frequent gesturing and postural openness – projects liking for the other and engagement in the interaction, and is correlated with increased solidarity [Argyle, 1988; Richmond and McCroskey, 1995]. Other nonverbal aspects of “warmth” include kinesic behaviours such as head tilts, bodily relaxation, lack of random movement, open body positions, and postural mirroring and vocalic behaviours such as more variation in pitch, amplitude, duration and tempo, reinforcing interjections such as “uh-huh” and “mm-hmmm”, greater fluency, warmth, pleasantness, expressiveness, and clarity and smoother turn-taking [Andersen and Guerrero, 1998]. In summary, nonverbal behaviour plays an important role in all face-to-face interaction – both conveying redundant and complementary propositional information (with respect to speech) and regulating the structure of the interaction. In social dialogue, however, it provides the additional, and crucial, function, of conveying attitudinal information about the nature of the relationship between the interactants.
4. 4.1
Related Work Related Work on Embodied Conversational Agents
Work on the development of ECAs, as a distinct field of development, is best summarized in [Cassell et al., 2000c]. The current study is based on the REA ECA (see Figure 2.1), a simulated real estate agent, who uses vision-based gesture recognition, speech recognition, discourse planning, sentence and gesture planning, speech synthesis and animation of a 3D body [Cassell et al., 1999]. Some of the other major systems developed to date are Steve [Rickel and Johnson, 1998], the DFKI Persona [André et al., 1996], Olga [Beskow and
Social Dialogue with Embodied Conversational Agents
33
McGlashan, 1997], and pedagogical agents developed by Lester et al. [1999]. Sidner and Dzikovska [2005] report progress on a robotic ECA that performs hosting activities, with a special emphasis on “engagement” – an interactional behaviour whose purpose is to establish and maintain the connection between interlocutors during a conversation. These systems vary in their linguistic generativity, input modalities, and task domains, but all aim to engage the user in natural, embodied conversation. Little work has been done on modelling social dialogue with ECAs. The August system is an ECA kiosk designed to give information about local restaurants and other facilities. In an experiment to characterize the kinds of things that people would say to such an agent, over 10,000 utterances from over 2,500 users were collected. It was found that most people tried to socialize with the agent, with approximately 1/3 of all recorded utterances classified as social in nature [Gustafson et al., 1999].
4.2
Related Studies on Embodied Conversational Agents
Koda and Maes [1996] and Takeuchi and Naito [1995] studied interfaces with static or animated faces, and found that users rated them to be more engaging and entertaining than functionally equivalent interfaces without a face. Kiesler and Sproull [1997] found that users were more likely to be cooperative with an interface agent when it had a human face (vs. a dog or cartoon dog). André, Rist and Muller found that users rated their animated presentation agent (“PPP Persona”) as more entertaining and helpful than an equivalent interface without the agent [André et al., 1998]. However, there was no difference in actual performance (comprehension and recall of presented material) in interfaces with the agent vs. interfaces without it. In another study involving this agent, van Mulken, André and Muller found that when the quality of advice provided by an agent was high, subjects actually reported trusting a text-based agent more than either their ECA or a video-based agent (when the quality of advice was low there were no significant differences in trust ratings between agents) [van Mulken et al., 1999]. In a user study of the Gandalf system [Cassell et al., 1999], users rated the smoothness of the interaction and the agent’s language skills significantly higher under test conditions in which Gandalf utilized limited conversational behaviour (gaze, turn-taking and beat gesture) than when these behaviours were disabled. In terms of social behaviours, Sproull et al. [1997] showed that subjects rated a female embodied interface significantly lower in sociability and gave it a significantly more negative social evaluation compared to a text-only interface. Subjects also reported being less relaxed and assured when interacting with the embodied interface than when interacting with the text interface.
34
Advances in Natural Multimodal Dialogue Systems
Finally, they gave themselves significantly higher scores on social desirability scales, but disclosed less (wrote significantly less and skipped more questions in response to queries by the interface) when interacting with an embodied interface vs. a text-only interface. Men were found to disclose more in the embodied condition and women disclosed more in the text-only condition. Most of these evaluations have tried to address whether embodiment of a system is useful at all, by including or not including an animated figure. In their survey of user studies on embodied agents, Dehn and van Mulken conclude that there is no “persona effect”, that is a general advantage of an interface with an animated agent over one without an animated agent [Dehn and van Mulken, 2000]. However, they believe that lack of evidence and inconsistencies in the studies performed to date may be attributable to methodological shortcomings and variations in the kinds of animations used, the kinds of comparisons made (control conditions), the specific measures used for the dependent variables, and the task and context of the interaction.
4.3
Related Studies on Mediated Communication
Several studies have shown that people speak differently to a computer than another person, even though there are typically no differences in task outcomes in these evaluations. Hauptmann and Rudnicky [1988] performed one of the first studies in this area. They asked subjects to carry out a simple informationgathering task through a (simulated) natural language speech interface, and compared this with speech to a co-present human in the same task. They found that speech to the simulated computer system was telegraphic and formal, approximating a command language. In particular, when speaking to what they believed to be a computer, subject’s utterances used a small vocabulary, often sounding like system commands, with very few task-unrelated utterances, and fewer filled pauses and other disfluencies. These results were extended in research conducted by Oviatt [Oviatt, 1995; Oviatt and Adams, 2000; Oviatt, 1998], in which she found that speech to a computer system was characterized by a low rate of disfluencies relative to speech to a co-present human. She also noted that visual feedback has an effect on disfluency: telephone calls have a higher rate of disfluency than co-present dialogue. From these results, it seems that people speak more carefully and less naturally when interacting with a computer. Boyle et al. [1994] compared pairs of subjects working on a map-based task who were visible to each other with pairs of subjects who were co-present but could not see each other. Although no performance difference was found between the two conditions, when subjects could not see one another, they compensated by giving more verbal feedback and using longer utterances. Their conversation was found to be less smooth than that between mutually visible
Social Dialogue with Embodied Conversational Agents
35
partners, indicated by more interruptions, and less efficient, as more turns were required to complete the task. The researchers concluded that visual feedback improves the smoothness and efficiency of the interaction, but that we have devices to compensate for this when visibility is restricted. Daly-Jones et al. [1998] also failed to find any difference in performance between video-mediated and audio-mediated conversations, although they did find differences in the quality of the interactions (e.g., more explicit questions in audio-only condition). Whittaker and O’Conaill [1997] survey the results of several studies which compared video-mediated communication with audio-only communication and concluded that the visual channel does not significantly impact performance outcomes in task-oriented collaborations, although it does affect social and affective dimensions of communication. Comparing video-mediated communication to face-to-face and audio-only conversations, they also found that speakers used more formal turn-taking techniques in the video condition even though users reported that they perceived many benefits to video conferencing relative to the audio-only mode. In a series of studies on the effects of different media and activities on trust, Zheng, Veinott et al. have demonstrated that social interaction, even if carried out online, significantly increases people’s trust in each other [Zheng et al., 2002]. Similarly, Bos et al. [2002] demonstrated that richer media – such as face-to-face, video-, and audio-mediated communication – leads to higher trust levels than media with lower bandwidth such as text chat. Finally, a number of studies have been done comparing face-to-face conversations with conversations on the phone [Rutter, 1987]. These studies find that, in general, there is more cooperation and trust in face-to-face interaction. One study found that audio-only communication encouraged negotiators to behave impersonally, to ignore the subtleties of self-presentation, and to concentrate primarily on pursuing victory for their side. Other studies found similar gains in cooperation among subjects playing prisoner’s dilemma face-to-face compared to playing it over the phone. Face-to-face interactions are also less formal and more spontaneous than conversations on the phone. One study found that face-to-face discussions were more protracted and wide-ranging while subjects communicating via audio-only kept much more to the specific issues on the agenda (the study also found that when the topics were more wide-ranging, changes in attitude among the participants was more likely to occur). Although several studies found increases in favourable impressions of interactants in face-to-face conversation relative to audio-only, these effects have not been consistently validated.
36
4.4
Advances in Natural Multimodal Dialogue Systems
Trait-Based Variation in User Responses
Several studies have shown that users react differently to social agents based on their own personality and other dispositional traits. For example, Reeves and Nass have shown that users like agents that match their own personality (on the introversion/ extraversion dimension) more than those which do not, regardless of whether the personality is portrayed through text or speech [Nass and Gong, 2000; Reeves and Nass, 1996]. Resnick and Lammers showed that in order to change user behaviour via corrective error messages, the messages should have different degrees of “humanness” depending on whether the user has high or low self-esteem (“computer-ese” messages should be used with low self-esteem users, while “human-like” messages should be used with highesteem users) [Resnick and Lammers, 1985]. Rickenberg and Reeves showed that different types of animated agents affected the anxiety level of users differentially as a function of whether users tended towards internal or external locus of control [Rickenberg and Reeves, 2000]. In our earlier study on the effects of social dialogue on trust in ECA interactions, we found that social dialogue significantly increased trust for extraverts, while it made no significant difference for introverts [Cassell and Bickmore, 2003]. In light of the studies summarized here, the question that remains is whether these effects continue to hold if the nonverbal cues provided by the ECA are removed.
5.
Social Dialogue in REA
For the purpose of trust elicitation and small talk, we have constructed a new kind of discourse planner that can interleave small talk and task talk during the initial buyer interview, based on the relational model outlined above. An overview of the planner is provided here; details of its implementation can be found in Cassell and Bickmore [2003].
5.1
Planning Model
Given that many of the goals in a relational conversational strategy are nondiscrete (e.g., minimize face threat), and that trade-offs among multiple goals have to be achieved at any given time, we have moved away from static world discourse planning, and are using an activation network-based approach based on Maes’ Do the Right Thing architecture [Maes, 1989]. This architecture provides the capability to transition smoothly from deliberative, planned behaviour to opportunistic, reactive behaviour, and is able to pursue multiple, non-discrete goals. In our implementation each node in the network represents a conversational move that REA can make.
Social Dialogue with Embodied Conversational Agents
37
Thus, during task talk, REA may ask questions about users’ buying preferences, such as the number of bedrooms they need. During small talk, REA can talk about the weather, events and objects in her shared physical context with the user (e.g., the lab setting), or she can tell stories about the lab, herself, or real estate. REA’s conversational moves are planned in order to minimize the face threat to the user, and maximize trust, while pursuing her task goals in the most efficient manner possible. That is, REA attempts to determine the face threat of her next conversational move, assesses the solidarity and familiarity which she currently holds with the user, and judges which topics will seem most relevant and least intrusive to users. As a function of these factors, REA chooses whether or not to engage in small talk, and what kind of small talk to choose. The selection of which move should be pursued by REA at any given time is thus a non-discrete function of the following factors: Closeness REA continually assesses her “interpersonal” closeness with the user, which is a composite representing depth of familiarity and solidarity, modelled as a scalar quantity. Each conversational topic has a predefined, pre-requisite closeness that must be achieved before REA can introduce the topic. Given this, the system can plan to perform small talk in order to “grease the tracks” for task talk, especially about sensitive topics like finance. Topic REA keeps track of the current and past conversational topics. Conversational moves which stay within topic (maintain topic coherence) are given preference over those which do not. In addition, REA can plan to execute a sequence of moves which gradually transition the topic from its current state to one that REA wants to talk about (e.g., from talk about the weather, to talk about Boston weather, to talk about Boston real estate). Relevance REA maintains a list of topics that she thinks the user knows about, and the discourse planner prefers moves which involve topics in this list. The list is initialized to things that anyone talking to REA would know about – such as the weather outside, Cambridge, MIT, or the laboratory that REA lives in. Task goals REA has a list of prioritized goals to find out about the user’s housing needs in the initial interview. Conversational moves which directly work towards satisfying these goals (such as asking interview questions) are preferred. Logical preconditions Conversational moves have logical preconditions (e.g., it makes no sense for REA to ask users what their major is
38
Advances in Natural Multimodal Dialogue Systems
until she has established that they are students), and are not selected for execution until all of their preconditions are satisfied. One advantage of the activation network approach is that by simply adjusting a few gains we can make REA more or less coherent, more or less polite (attentive to closeness constraints), more or less task-oriented, or more or less deliberative (vs. reactive) in her linguistic behaviour. In the current implementation, the dialogue is entirely REA-initiated, and user responses are recognized via a speaker-independent, grammar-based, continuous speech recognizer (currently IBM ViaVoice). The active grammar fragment is specified by the current conversational move, and for responses to many REA small talk moves the content of the user’s speech is ignored; only the fact that the person responded at all is enough to advance the dialogue. At each step in the conversation in which REA has the floor (as tracked by a conversational state machine in REA’s Reaction Module [Cassell et al., 2000a], the discourse planner is consulted for the next conversational move to initiate. At this point, activation values are incrementally propagated through the network (following [Maes, 1989]) until a move is selected whose preconditions are satisfied and whose activation value is over a specified threshold. Within this framework, REA decides to do small talk whenever closeness with the user needs to be increased (e.g., before a task query can be asked), or the topic needs to be moved little-by-little to a desired topic and small talk contributions exist which can facilitate this. The activation energy from the user relevance condition described above leads to REA starting small talk with topics that are known to be in the shared environment with the user (e.g., talk about the weather or the lab).
5.2
Interactional Behaviour during Social Dialogue
Shifts between small talk moves and task moves are marked by conventional contextualization cues – discourse markers and beat gestures. Discourse markers include “so” on the first small talk to task talk transition, “anyway” on resumption of task talk from small talk, and “you know” on transition to small talk from task talk [Clark, 1996]. Prior to producing lengthy utterances, REA gazes away briefly before she starts her turn, partly as a turn-taking and floor-holding move and partly to mask the processing delays in generating long utterances. Finally, REA smiles as soon as she detects that users have started their speaking turns (using audio thresholding of the user’s speech).
5.3
Example Interaction
An interview between REA and a user typically proceeds as shown in the following dialogue. (User responses are only shown in positions in which they
Social Dialogue with Embodied Conversational Agents
39
effect the selection of subsequent moves; REA contributions are her actual utterances.) 1 That microphone is terrible, I hate using those things. 2 Sorry about my voice, this is some engineer’s idea of natural sounding. 3 Are you one of our sponsors? User: Yes 4 Were you at our last sponsor meetings? 5 I got so exhausted at the last sponsor meeting I think I was starting to lose my voice by the end. 6 So, where would you like to live? 7 How many bedrooms do you need? 8 Do you need access to the subway? 9 Is one bath enough? 10 You know, Boston is certainly more expensive than it used to be. 11 Anyway, what can you afford? 12 What kind of down payment can you make? 13 Let me see what I have available. Dialogue 1. “Small Talk” REA.
In this example, REA opens with small talk moves regarding things in her shared physical environment with the user (1-2). She then proceeds to small talk related to sponsors (after establishing that the user is a sponsor). After a few turns, enough closeness has been established (simply by doing small talk) that REA can move into task talk (6-9). However, before bringing up the topic of finance – a topic that is potentially very face threatening for the user – REA decides that additional closeness needs to be established, and moves back into small talk (10). This small talk move not only increases closeness but shifts the topic to finance, enabling REA to then bring up the issue of how much the user is able to afford (11-12). If REA’s adherence to closeness preconditions is reduced, by decreasing the contributions of these preconditions to the activation of joint projects, this results in her engaging in less small talk and being more task goal oriented. If everything else is held constant (relative to the prior example) the following dialogue is produced.
40
Advances in Natural Multimodal Dialogue Systems
1 So, where would you like to live? 2 What can you afford? 3 What kind of down payment can you make? 4 How many bedrooms do you need? 5 Do you need access to the subway? 6 Is one bath enough? 7 Let me see what I have available. Dialogue 2. “Task-only REA”.
In this example, REA does not perform any small talk and sequences the task questions in strictly decreasing order of priority.
6.
A Study Comparing ECA Social Dialogue with Audio-Only Social Dialogue
The dialogue model presented above produces a reasonable facsimile of the social dialogue observed in service encounters such as real estate sales. But, does small talk produced by an ECA in a sales encounter actually build trust and solidarity with users? And, does nonverbal behaviour play the same critical role in human-ECA social dialogue as it appears to play in human-human social interactions? In order to answer these questions, we conducted an empirical study in which subjects were interviewed by REA about their housing needs, shown two “virtual” apartments, and then asked to submit a bid on one of them. For the purpose of the experiment, REA was controlled by a human wizard and followed scripts identical to the output of the planner (but faster, and not dependent on automatic speech recognition or computational vision). Users interacted with one of two versions of REA which were identical except that one had only task-oriented dialogue (TASK condition) while the other also included the social dialogue designed to avoid face threat, and increase trust (SOCIAL condition). A second manipulation involved varying whether subjects interacted with the fully embodied REA – appearing in front of the virtual apartments as a life-sized character (EMBODIED condition) – or viewed only the virtual apartments while talking with REA over a telephone. Together these variables provided a 2x2 experimental design: SOCIAL vs. TASK and EMBODIED vs. PHONE. Our hypotheses follow from the literature on small talk and on trust among humans. We expected subjects in the SOCIAL condition to trust REA more, feel closer to REA, like her more, and feel that they understood each other more
Social Dialogue with Embodied Conversational Agents
41
than in the TASK condition. We also expected users to think the interaction was more natural, lifelike, and comfortable in the SOCIAL condition. Finally, we expected users to be willing to pay REA more for an apartment in the SOCIAL condition, given the hypothesized increase in trust. We also expected all of these SOCIAL effects to be amplified in the EMBODIED condition relative to the PHONE-only condition.
6.1
Experimental Methods
This was a multivariate, multiple-factor, between-subjects experimental design, involving 58 subjects (69% male and 31% female).
6.1.1 Apparatus. One wall of the experiment room was a rear-projection screen. In the EMBODIED condition REA appeared life-sized on the screen, in front of the 3D virtual apartments she showed, and her synthetic voice was played through two speakers on the floor in front of the screen. In the PHONE condition only the 3D virtual apartments were displayed and subjects interacted with REA over an ordinary telephone placed on a table in front of the screen. For the purpose of this experiment, REA was controlled via a wizard-of-oz setup on another computer positioned behind the projection screen. The interaction script included verbal and nonverbal behaviour specifications for REA (e.g., gesture and gaze commands as well as speech), and embedded commands describing when different rooms in the virtual apartments should be shown. Three pieces of information obtained from the user during the interview were entered into the control system by the wizard: the city the subject wanted to live in; the number of bedrooms s/he wanted; and how much s/he was willing to spend. The first apartment shown was in the specified city, but had twice as many bedrooms as the subject requested and cost twice as much as s/he could afford (they were also told the price was “firm”). The second apartment shown was in the specified city, had the exact number of bedrooms requested, but cost 50% more than the subject could afford (but this time, the subject was told that the price was “negotiable”). The scripts were comprised of a linear sequence of utterances (statements and questions) that would be made by REA in a given interaction: there was no branching or variability in content beyond the three pieces of information described above. This helped ensure that all subjects received the same intervention regardless of what they said in response to any given question by REA. Subject-initiated utterances were responded to with either backchannel feedback (e.g., “Really?”) for statements or “I don’t know” for questions, followed by an immediate return to the script. The scripts for the TASK and SOCIAL conditions were identical, except that the SOCIAL script had additional small talk utterances added to it, as
42
Advances in Natural Multimodal Dialogue Systems
described in [Bickmore and Cassell, 2001]. The part of the script governing the dialogue from the showing of the second apartment through the end of the interaction was identical in both conditions. Procedure. Subjects were told that they would be interacting with REA, who played the role of a real estate agent and could show them apartments she had for rent. They were told that they were to play the role of someone looking for an apartment in the Boston area. In both conditions subjects were told that they could talk to REA “just like you would to another person”.
6.1.2 Measures. Subjective evaluations of REA – including how friendly, credible, lifelike, warm, competent, reliable, efficient, informed, knowledgeable and intelligent she was – were measured by single items on nine-point Likert scales. Evaluations of the interaction – including how tedious, involving, enjoyable, natural, satisfying, fun, engaging, comfortable and successful it was – were also measured on nine-point Likert scales. Evaluation of how well subjects felt they knew REA, how well she knew and understood them and how close they felt to her were measured in the same manner. All scales were adapted from previous research on user responses to personality types in embodied conversational agents [Moon and Nass, 1996]. Liking of REA was an index composed of three items – how likeable and pleasant REA was and how much subjects liked her – measured items on ninepoint Likert scales (Cronbach’s alpha =.87). Amount Willing to Pay was computed as follows. During the interview, REA asked subjects how much they were able to pay for an apartment; subjects’ responses were entered as $X per month. REA then offered the second apartment for $Y (where Y = 1.5 X), and mentioned that the price was negotiable. On the questionnaire, subjects were asked how much they would be willing to pay for the second apartment, and this was encoded as Z. The task measure used was (Z - X) / (Y - X), which varies from 0% if the user did not budge from their original requested price, to 100% if they offered the full asking price. Trust was measured by a standardized trust scale [Wheeless and Grotz, 1977] (alpha =.93). Although trust is sometimes measured behaviourally using a Prisoner’s Dilemma game [Zheng et al., 2002], we felt that our experimental protocol was already too long and that game-playing did not fit well into the real estate scenario. Given literature on the relationship between user personality and preference for computer behaviour, we were concerned that subjects might respond differentially based on predisposition. Thus, we also included composite measures for introversion and extroversion on the questionnaire. Extrovertedness was an index composed of seven Wiggins [Wiggins, 1979] extrovert adjective items: Cheerful, Enthusiastic, Extroverted, Jovial, Outgo-
Social Dialogue with Embodied Conversational Agents
43
ing, and Perky. It was used for assessment of the subject’s personality (alpha =.87). Introvertedness was an index composed of seven Wiggins [Wiggins, 1979] introvert adjective items: Bashful, Introverted, Inward, Shy, Undemonstrative, Unrevealing, and Unsparkling. It was used for assessment of the subject’s personality (alpha =.84). Note that these personality scales were administered on the post-test questionnaire. For the purposes of this experiment, therefore, subjects who scored over the mean on introversion-extroversion were said to be extroverts, while those who scored under the mean were said to be introverts.
6.1.3 Behavioural measures. Rates of speech disfluency (as defined in [Oviatt, 1995]) and utterance length were coded from the video data. Observation of the videotaped data made it clear that some subjects took the initiative in the conversation, while others allowed REA to lead. Unfortunately, REA is not yet able to deal with user-initiated talk, and so user initiative often led to REA interrupting the speaker. To assess the effect of this phenomenon, we therefore divided subjects into PASSIVE (below the mean on number of user-initiated utterances) and ACTIVE (above the mean on number of userinitiated utterances). To our surprise, these measures turned out to be independent of introversion/extroversion (Pearson r=0.042), and to not be predicted by these latter variables.
6.2
Results
Full factorial single measure ANOVAs were run, with SOCIALITY (Task vs. Social), PERSONALITY OF SUBJECT (Introvert vs. Extrovert), MEDIUM (Phone vs. Embodied) and INITIATION (Active vs. Passive) as independent variables.
6.2.1 Subjective assessments of REA. In looking at the questionnaire data, our first impression is that subjects seemed to feel more comfortable interacting with REA over the phone than face-to-face. Thus, subjects in the phone condition felt that they knew REA better (F=5.02; p<.05), liked her more (F=4.70; p<.05), felt closer to her (F=13.37; p<.001), felt more comfortable with the interaction (F=3.59; p<.07), and thought REA was more friendly (F=8.65;p <.005), warm (F=6.72; p<.05), informed (F=5.73; p<.05), and knowledgeable (F=3.86; p<.06) than those in the embodied condition. However, in the remainder of the results section, as we look more closely at different users, different kinds of dialogue styles, and users’ actual behaviour, a more complicated picture emerges. Subjects felt that REA knew them (F=3.95; p<.06) and understood them (F=7.13; p<.05) better when she used task-only dialogue face-to-face; these trends were reversed for phone-based interactions. Task-only dialogue was more fun (F=3.36; p<.08) and less tedious (F=8.77;
44
Advances in Natural Multimodal Dialogue Systems
Figure 2.3.
Ratings of TEDIOUS.
Figure 2.4. ing to pay.
Amount subjects were will-
p<.005; see Figure 2.3) when embodied, while social dialogue was more fun and less tedious on the phone. That is, subjects preferred to interact, and felt better understood, face-to-face when it was a question of simply “getting down to business,” and preferred to interact, and felt better understood, by phone when the dialogue included social chit-chat. These results may be telling us that REA’s nonverbal behaviour inadvertently projected an unfriendly, introverted personality that was especially inappropriate for social dialogue. REA’s model of non-verbal behaviour, at the time of this experiment, was limited to those behaviours linked to the discourse context. Thus, REA’s smiles were limited to those related to the ends of turns, and she did not have a model of immediacy or other nonverbal cues for liking and warmth typical of social interaction [Argyle, 1988]. According to Whittaker and O’Connail [1993], non-verbal information is especially crucial in interactions involving affective cues, such as negotiation or relational dialogue, and less important in purely problem-solving tasks. This interpretation of the results is backed up by comments such as this response from a subject in the face-to-face social condition: The only problem was how she would respond. She would pause then just say “OK”, or “Yes”. Also when she looked to the side and then back before saying something was a little bit unnatural.
This may explain why subjects preferred task interactions face-to-face, while on the phone REA’s social dialogue had its intended effect of making subjects feel that they knew REA better, that she understood them better, and that the experience was more fun and less tedious. In our earlier study, looking only at an embodied interface, we reported that extroverts trusted the system more when it engaged in small talk, while introverts were not affected by the use of small talk [Bickmore and Cassell, 2001]. In the current study, these results were re-confirmed, but only in the embodied
Social Dialogue with Embodied Conversational Agents
45
interaction; that is, a three-way interaction between SOCIALITY, PERSONALITY and MEDIUM (F=3.96; p<.06) indicated that extroverts trusted REA more when she used social dialogue in embodied interactions, but there was essentially no effect of user’s personality and social dialogue on trust in phone interactions. Further analysis of the data indicates that this result derives from the substantial difference between introverts and extroverts in the face-to-face task-only condition. Introverts trusted REA significantly more in the faceto-face task-only condition than in the other conditions (p<.03), while extroverts trusted her significantly less in this condition than in the other conditions (p.<01). In light of these new observations, our earlier results indicating that social dialogue leads to increased trust (for extroverts at least) needs to be revised. This further analysis indicates that the effects we observed may be due to the attraction of a computer displaying similar personality characteristics, rather than the process of trust-building. That is, in the face-to-face, task-only condition, both verbal and nonverbal channels appear to have inadvertently indicated that REA was an introvert (also supported by the comments that REA’s gaze-away behaviour was too frequent, an indication of introversion [Wilson, 1977]), and in this condition we find the introverts trusting more, and extroverts trusting less. In all other conditions, the personality cues are either conflicting (a mismatch between verbal and nonverbal behaviour has been demonstrated to be disconcerting to users [Nass and Gong, 2000]) or only one channel of cues is available (i.e. on the phone), yielding trust ratings that are close to the overall mean. There was, nevertheless, a preference by extroverts for social dialogue as demonstrated by the fact that, overall, extroverts liked REA more when she used social dialogue, while introverts liked her more when she only talked about the task (F=8.09; p<.01). Passive subjects felt more comfortable interacting with REA than active subjects did, regardless of whether the interaction was face-to-face or on the phone, or whether REA used social dialogue or not. Passive subjects said that they enjoyed the interaction more (F=4.47; p<.05), felt it was more successful (F=6.04; p<.05) and liked REA more (F=3.24; p<.08), and that REA was more intelligent (F=3.40; p<.08), and knew them better (F=3.42; p<.08) than active subjects. These differences may be explained by the fixed-initiative dialogue model used in the WOZ script. REA’s interaction was designed for passive users – there was very little capability in the interaction script to respond to unanticipated user questions or statements – and user initiation attempts were typically met with uncooperative system responses or interruptions. But, given the choice between phone and face-to-face, passive users preferred to interact with REA face-to-face: they rated her as more friendly (F=3.56; p<.07) and informed (F=6.30; p<.05) in this condition. Passive users also found the phone
46
Advances in Natural Multimodal Dialogue Systems
to be more tedious, while active users also found the phone to be less tedious (F=5.15; p<.05). Active users may have found the face-to-face condition particularly frustrating since processing delays may have led to the perception that the floor was open (inviting an initiation attempt), when in fact the wizard had already instructed REA to produce her next utterance. However, when interacting on the phone, active users differed from passive users in that active users felt she was more reliable when using social dialogue and passive users felt she was more reliable when using task-only dialogue. When interacting face-to-face with REA, there was no such distinction between active and passive users (F=4.67; p<.05).
6.2.2 Effects on task measure. One of the most tantalizing results obtained is that extroverts were willing to pay more for the same apartment in the embodied condition, while introverts were willing to pay more over the phone (F=3.41; p<.08), as shown in Figure 2.4. While potentially very significant, this finding is a little difficult to explain, especially given that trust did not seem to play a role in the evaluation. Perhaps, since we asked our subjects to play the role of someone looking for an apartment, and given that the apartments displayed were cartoon renditions, the subjects may not have felt personally invested in the outcome, and thus may have been more likely to be persuaded by associative factors like the perceived liking and credibility of REA. In fact, trust has been shown to not play a role in persuasion when “peripheral route” decisions are made, which is the case when the outcome is not of personal significance [Petty and Wegener, 1998]. Further, extroverts are not only more sociable, but more impulsive than introverts [Wilson, 1977], and impulse buying is governed primarily by novelty [Onkvisit and Shaw, 1994]. Extroverts did rate face-to-face interaction as more engaging than phone-based interaction (though not at a level of statistical significance), while introverts rated phone-based interactions as more engaging, providing some support for this explanation. It is also possible that this measure tells us more about subjects’ assessment of the house than of the realtor. In future experiments we may ask more directly whether the subject perceived the realtor to be asking a fair price. Perception of fairness of a price may be more linked to trust than is actual price demanded for a property. 6.2.3 Gender effects. Women felt that REA was more efficient (F=5.61; p<.05) and reliable (F=4.99; p<.05) in the embodied condition than when interacting with her over the phone, while men felt that she was more efficient and reliable by phone. Of course, REA has a female body and a female voice and so in order to have a clearer picture of the meaning of these results, a similar study would need to be carried out with a male realtor.
47
Social Dialogue with Embodied Conversational Agents Table 2.2. Speech disfluencies per 100 words.
Disfluencies
Embodied
Phone
Overall
4.83
6.73
5.83
Table 2.3. Speech disfluencies per 100 Words for different types of human-human and simulated human-computer interactions (adapted from Oviatt [Oviatt, 1995]). Human-human speech Two-person telephone call Two-person face-to-face dialogue
8.83 5.5
Human-computer speech Unconstrained computer interaction Structured computer interaction
1.80 0.83
6.2.4 Effects on behavioural measures. Although subjects’ beliefs about REA and about the interaction are important, it is at least equally important to look at how subjects act, independent of their conscious beliefs. In this context we examined subjects’ disfluencies when speaking with REA. Remember that disfluency can be a measure of naturalness – human-human conversation demonstrates more disfluency than does human-computer communication [Oviatt, 1995]. The rates of speech disfluencies (per 100 words) are shown in Table 2.2. Comparing these to results from previous studies (see Table 2.3) indicates that interactions with REA were more similar to humanhuman conversation than to human-computer interaction. When asked if he was interacting with a computer or a person, one subject replied “A computerperson I guess. It was a lot like a human.” There were no significant differences in utterance length (MLU) across any of the conditions. Strikingly, the behavioural measures indicate that, with respect to speech disfluency rates, talking to REA is more like talking to a person than talking to a computer. Once again, there were significant effects of MEDIUM, SOCIALITY and PERSONALITY on disfluency rate (F=7.09; p<.05), such that disfluency rates were higher in TASK than SOCIAL, higher overall for INTROVERTs than EXTROVERTs, higher for EXTROVERTs on the PHONE, and higher for INTROVERTs in EMBODIED condition. These effects on disfluency rates are consistent with our conclusion that REA’s nonverbal behaviours inadvertently projected an introverted and asocial persona, and with the secondary hypoth-
48
Advances in Natural Multimodal Dialogue Systems
esis that the primary driver on disfluency is cognitive load, once the length of the utterance is controlled for [Oviatt, 1995]. Given our results, this hypothesis would indicate that social dialogue requires lower cognitive load than task-oriented dialogue, that conversation requires a higher cognitive load on introverts than extraverts, that talking on the phone is more demanding than talking face-to-face for extraverts, and that talking face-to-face is more demanding than talking on the phone for introverts, all of which seem reasonable.
7.
Conclusion
The complex results of this study give us hope for the future of embodied conversational agents, but also a clear roadmap for future research. In terms of their behaviour with REA, users demonstrated that they treat conversation with her more like human-human conversation than like human-computer conversation. Their verbal disfluencies are the mark of unplanned speech, of a conversational style. However, in terms of their assessment of her abilities, this did not mean that users saw REA through rose-colored glasses. They were clear about the necessity not only to embody the interaction, but to design every aspect of the embodiment in the service of the same interaction. That is, face-to-face conversations with ECAs must demonstrate the same quick timing of nonverbal behaviours as humans (not an easy task, given the state of the technologies we employ). In addition, the persona and nonverbal behaviour of an ECA must be carefully designed to match the task, a conversational style, and user expectations. Relative to other studies on social dialogue and interactions with ECAs, this study has taught us a great deal about how to build ECAs capable of social dialogue and the kinds of applications in which this is important to do. As demonstrated in the study of the August ECA, we found that people will readily conduct social dialogue with an ECA in situations in which there is no time pressure to complete a task or in which it is a normal part of the script for similar human-human interactions, and many people actually prefer this style of interaction. Consequently, we are in the process of developing and evaluating an ECA in the area of coaching for health behaviour change [Bickmore, 2002], an area in which social dialogue – and relationship-building behaviours in general – are known to significantly effect task outcomes. Relative to prior findings in human-human interaction that social dialogue builds trust, our inability to find this effect across all users is likely due to shortcomings in our model, especially in the area of appropriate nonverbal behaviour and lack of uptake of subjects’ conversational contributions. Given these shortcomings, similarity attraction effects appear to have overwhelmed the social-dialogue-trust effects, and we observed introverts liking and trusting REA more when she behaved consistently introverted and extraverts liking and
Social Dialogue with Embodied Conversational Agents
49
trusting her more when she behaved consistently extraverted. Our conclusion from this is that, adding social dialogue to embodied conversational agents will require a model of social nonverbal behaviour consistent with verbal conversational strategies, before the social dialogue fulfils its trust-enhancing role. As computers begin to resemble humans, the bar of user expectations is raised: people expect that REA will hold up her end of the conversation, including dealing with interruptions by active users. We have begun to demonstrate the feasibility of embodied interfaces. Now it is time to design ECAs that people wish to spend time with, and that are able to use their bodies for conversational tasks for which human face-to-face interaction is unparalleled, such as social dialogue, initial business meetings, and negotiation.
Acknowledgements Thanks to Ian Gouldstone, Jennifer Smith and Elisabeth Sylvan for help in conducting the experiment and analyzing data, and to the rest of the Gesture and Narrative Language Group for their help and support.
References Altman, I. and Taylor, D. (1973). Social Penetration: The Development of Interpersonal Relationships. New York: Holt, Rinhart & Winston. Andersen, P. and Guerrero, L. (1998). The Bright Side of Relational Communication: Interpersonal Warmth as a Social Emotion. In Andersen, P. and Guerrero, L., editors, Handbook of Communication and Emotion, pages 303–329. New York: Academic Press. André, E., Muller, J., and Rist, T. (1996). The PPP Persona: A Multipurpose Animated Presentation Agent. In Proceedings of Advanced Visual Interfaces, pages 245–247, Gubbio, Italy. André, E., Rist, T., and Muller, J. (1998). Integrating Reactive and Scripted Behaviors in a Life-Like Presentation Agent. In Proceedings of the Second International Conference on Autonomous Agents, pages 261–268, Minneapolis, Minnesota, USA. Argyle, M. (1988). Bodily Communication. New York: Methuen & Co. Ltd. Argyle, M. (1990). The Biological Basis of Rapport. Psychological Inquiry, 1:297–300. Beskow, J. and McGlashan, S. (1997). Olga: A Conversational Agent with Gestures. In André, E., editor, Proceedings of the IJCAI 1997 Workshop on Animated Interface Agents: Making Them Intelligent, Nagoya, Japan. San Francisco: Morgan-Kaufmann Publishers. Bickmore, T. (2002). When Etiquette Really Matters: Relational Agents and Behavior Change. In Proceedings of AAAI Fall Symposium on Etiquette for Human-Computer Work, pages 9–10, Falmouth, MA.
50
Advances in Natural Multimodal Dialogue Systems
Bickmore, T. and Cassell, J. (2001). Relational Agents: A Model and Implementation of Building User Trust. In Proceedings of CHI 2001, pages 396– 403, Seattle, WA. Bos, N., Olson, J. S., Gergle, D., Olson, G. M., and Wright, Z. (2002). Effects of Four Computer-Mediated Communications Channels on Trust Development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 135–140, Minneapolis, Minnesota, USA. Boyle, E., Anderson, A., and Newlands, A. (1994). The Effects of Visibility in a Cooperative Problem Solving Task. Language and Speech, 37(1):1–20. Brown, P. and Levinson, S. (1978). Universals in Language Usage: Politeness Phenomena. In Goody, E., editor, Questions and Politeness: Strategies in Social Interaction, pages 56–289. Cambridge: Cambridge University Press. Brown, R. and Gilman, A. (1972). The Pronouns of Power and Solidarity. In Giglioli, P., editor, Language and Social Context, pages 252–282. Harmondsworth: Penguin. Brown, R. and Gilman, A. (1989). Politeness Theory and Shakespeare’s Four Major Tragedies. Language in Society, 18:159–212. Cassell, J. and Bickmore, T. (2003). Negotiated Collusion: Modeling Social Language and its Relationship Effects in Intelligent Agents. User Modeling and Adaptive Interfaces, 13(1-2):89–132. Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., and Yan, H. (1999). Embodiment in Conversational Interfaces: Rea. In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems, pages 520–527, Pittsburgh, PA. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., and Yan, H. (2000a). Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Embodied Conversational Agents, pages 29–63. Cambridge, MA: MIT Press. Cassell, J., Bickmore, T., H, Vilhjálmsson, and Yan, H. (2000b). More Than Just a Pretty Face: Affordances of Embodiment. In Proceedings of the 5th International Conference on Intelligent User Interfaces, pages 52–59, New Orleans, Louisiana. Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. (2000c). Embodied Conversational Agents. Cambridge, MA: MIT Press. Cheepen, C. (1988). The Predictability of Informal Conversation. New York: Pinter. Chovil, N. (1991). Discourse-Oriented Facial Displays in Conversation. Research on Language and Social Interaction, 25(1991/1992):163–194. Clark, H. H. (1996). Using Language. Cambridge: Cambridge University Press. Coupland, J., Coupland, N., and Robinson, J. D. (1992). How Are You? Negotiating Phatic Communion. Language in Society, 21:207–230.
Social Dialogue with Embodied Conversational Agents
51
Daly-Jones, O., Monk, A. F., and Watts, L. A. (1998). Some Advantages of Video Conferencing over High-Quality Audio Conferencing: Fluency and Awareness of Attentional Focus. International Journal of Human-Computer Studies, 49(1):21–58. Dehn, D. M. and van Mulken, S. (2000). The Impact of Animated Interface Agents: A Review of Empirical Research. International Journal of HumanComputer Studies, 52:1–22. Dunbar, R. (1996). Grooming, Gossip, and the Evolution of Language. Cambridge, MA: Harvard University Press. Duncan, S. (1974). On the Structure of Speaker-Auditor Interaction during Speaking Turns. Language in Society, 3:161–180. Garros, D. (1999). Real Estate Agent, Home and Hearth Realty. Personal communication. Goffman, I. (1967). On Face-Work. In Interaction Ritual: Essays on Face-toFace Behavior, pages 5–46. New York: Pantheon. Gustafson, J., Lindberg, N., and Lundeberg, M. (1999). The August Spoken Dialogue System. In Proceedings of the Eurospeech 1999 Conference, pages 1151–1154, Budapest, Hungary. Hauptmann, A. G. and Rudnicky, A. I. (1988). Talking to Computers: An Empirical Investigation. International Journal of Man-Machine Studies, 8(6): 583–604. Jakobson, R. (1960). Concluding Statement: Linguistics and Poetics. In Sebeok, T., editor, Style in Language, pages 351–377. New York: Wiley. Kendon, A. (1980). Conducting Interaction: Patterns of Behavior in Focused Encounters, volume 7. Cambridge: Cambridge University Press. Kiesler, S. and Sproull, L. (1997). Social Human-Computer Interaction. In Friedman, B., editor, Human Values and the Design of Computer Technology, pages 191–199. Stanford, CA: CSLI Publications. Koda, T. and Maes, P. (1996). Agents with Faces: The Effect of Personification. In Proceedings of the Fifth IEEE International Workshop on Robot and Human Communication (RO-MAN 1996), pages 189–194, Tsukuba, Japan. Laver, J. (1975). Communicative Functions of Phatic Communion. In Kendon, A., Harris, R., and Key, M., editors, The Organization of Behavior in Faceto-Face Interaction, pages 215–238. The Hague: Mouton. Laver, J. (1981). Linguistic Routines and Politeness in Greeting and Parting. In Coulmas, F., editor, Conversational Routine, pages 289–304. The Hague: Mouton. Lester, J., Stone, B., and Stelling, G. (1999). Lifelike Pedagogical Agents for Mixed-Initiative Problem Solving in Constuctivist Learning Environments. User Modeling and User-Adapted Interaction, 9(1-2):1–44. Maes, P. (1989). How to Do the Right Thing. Connection Science Journal, 1(3):291–323.
52
Advances in Natural Multimodal Dialogue Systems
Malinowski, B. (1923). The Problem of Meaning in Primitive Languages. In Ogden, C. K. and Richards, I. A., editors, The Meaning of Meaning, pages 296–346. Routledge & Kegan Paul. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Cambridge: Cambridge University Press. Moon, Y. and Nass, C. I. (1996). How “Real” are Computer Personalities? Psychological Responses to Personality Types in Human-Computer Interaction. Communication Research, 23(6):651–674. Nass, C. and Gong, L. (2000). Speech Interfaces from an Evolutionary Perspective. Communications of the ACM, 43(9):36–43. Onkvisit, S. and Shaw, J. J. (1994). Consumer Behavior: Strategy and Analysis. New York: Macmillan College Publishing Company. Oviatt, S. (1995). Predicting Spoken Disfluencies during Human-Computer Interaction. Computer Speech and Language, 9:19–35. Oviatt, S. and Adams, B. (2000). Designing and Evaluating Conversational Interfaces with Animated Characters. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 319– 345. Cambridge, MA: MIT Press. Oviatt, S. L. (1998). User-Centered Modeling for Spoken Language and Multimodal Interfaces. In Maybury, M. T. and Wahlster, W., editors, Readings in Intelligent User Interfaces, pages 620–630. San Francisco, CA: Morgan Kaufmann Publishers, Inc. Petty, R. and Wegener, D. (1998). Attitude Change: Multiple Roles for Persuasion Variables. In Gilbert, D., Fiske, S., and Lindzey, G., editors, The Handbook of Social Psychology, pages 323–390. New York: McGraw-Hill. Prus, R. (1989). Making Sales: Influence as Interpersonal Accomplishment. Mewbury Park, CA: Sage. Reeves, B. and Nass, C. (1996). The Media Equation. Cambridge: Cambridge University Press. Resnick, P. V. and Lammers, H. B. (1985). The Influence of Self-Esteem on Cognitive Responses to Machine-Like versus Human-Like Computer Feedback. The Journal of Social Psychology, 125(6):761–769. Richmond, V. and McCroskey, J. (1995). Immediacy - Nonverbal Behavior in Interpersonal Relations. Boston: Allyn & Bacon. Rickel, J. and Johnson, W. L. (1998). Task-Oriented Dialogs with Animated Agents in Virtual Reality. In Proceedings of the First Workshop on Embodied Conversational Characters, pages 39–46. Rickenberg, R. and Reeves, B. (2000). The Effects of Animated Characters on Anxiety, Task Performance, and Evaluations of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 49–56, The Hague, Amsterdam.
Social Dialogue with Embodied Conversational Agents
53
Rutter, D. R. (1987). Communicating by Telephone. New York: Pergamon Press. Schneider, K. P. (1988). Small Talk: Analysing Phatic Discourse. Marburg: Hitzeroth. Sidner, C. and Dzikovska, M. (2005). A First Experiment in Engagement for Human-Robot Interaction in Hosting Activities. In van Kuppevelt, J., Dybkjær, L., and Bernsen, N. O., editors, Advances in Natural Multimodal Dialogue Systems. Springer. This volume. Sproull, L., Subramani, M., Kiesler, S., Walker, J., and Waters, K. (1997). When the Interface is a Face. In Friedman, B., editor, Human Values and the Design of Computer Technology, pages 163–190. Stanford, CA: CSLI Publications. Stone, M. and Doran, C. (1997). Sentence Planning as Description Using TreeAdjoining Grammar. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics (ACL/EACL 1997), pages 198–205, Madrid, Spain. Svennevig, J. (1999). Getting Acquainted in Conversation. Philadelphia: John Benjamins. Takeuchi, A. and Naito, T. (1995). Situated Facial Displays: Towards Social Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 450–455, Denver, Colorado. van Mulken, S., André, E., and Muller, J. (1999). An Empirical Study on the Trustworthiness of Lifelike Interface Agents. In Bullinger, H.-J. and Ziegler, J., editors, Human-Computer Interaction (Proceedings of HCI-International 1999), pages 152–156. Mahwah, NJ: Lawrence Erlbaum Associates. Wheeless, L. and Grotz, J. (1977). The Measurement of Trust and Its Relationship to Self-Disclosure. Human Communication Research, 3(3):250–257. Whittaker, S. and O’Conaill, B. (1993). An Evaluation of Video Mediated Communication. In Proceedings of Human Factors and Computing Systems (INTERACT/CHI 1993), pages 73–74, Amsterdam, The Netherlands. Whittaker, S. and O’Conaill, B. (1997). The Role of Vision in Face-to-Face and Mediated Communication. In Finn, K., Sellen, A., and Wilbur, S., editors, Video-Mediated Communication, pages 23–49. Lawrence Erlbaum Associates, Inc. Wiggins, J. (1979). A Psychological Taxonomy of Trait-Descriptive Terms. Journal of Personality and Social Psychology, 37(3):395–412. Wilson, G. (1977). Introversion/Extraversion. In Blass, T., editor, Personality Variables in Social Behavior, pages 179–218. New York: John Wiley & Sons.
54
Advances in Natural Multimodal Dialogue Systems
Zheng, J., Veinott, E. S., Bos, N., Olson, J. S., and Olson, G. M. (2002). Trust without Touch: Jumpstarting Long-Distance Trust with Initial Social Activities. In Proceedings of the International Conference for Human-Computer Interaction (CHI), pages 141–146.
Chapter 3 A FIRST EXPERIMENT IN ENGAGEMENT FOR HUMAN-ROBOT INTERACTION IN HOSTING ACTIVITIES∗ Candace L. Sidner Mitsubishi Electric Research Labs 201 Broadway Cambridge, MA 02139, USA
[email protected]
Myroslava Dzikovska Department of Computer Science University of Rochester Rochester, NY 14627, USA
[email protected]
Abstract
To participate in collaborations with people, robots must not only see and talk with people but also make use of the conventions of conversation and of the means to be connected to their human counterparts. This chapter reports on initial research on engagement in human-human interaction and applications to stationary robots interacting with humans in hosting activities.
Keywords:
Human-robot interaction, engagement, gestures, looking, conversation, collaboration, hosting.
1.
Introduction
As part of our ongoing research on collaborative interface agents, we have begun to explore engagement in human interaction. Engagement is the process ∗ Portions
of this chapter are reprinted with permission from C. Sidner and M. Dzikovska, “Human-Robot Interaction: Engagement between Humans and Robots for Hosting Activities,” The Fourth IEEE International Conference on Multi-modal Interfaces, October, 2002, pages 123-128. @ 2002 IEEE.
55 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 55–76. © 2005 Springer. Printed in the Netherlands.
56
Advances in Natural Multimodal Dialogue Systems
by which two, or more, participants establish, maintain and end their perceived connection during interactions they jointly undertake. This process includes: establishment of the initial contact with another participant, negotiation of a collaboration to undertake activities of mutual interest, determination of the ongoing intent of the other participant to continue in the interaction, evaluation of one’s own intentions in staying involved, and determination of when to end the interaction. To understand the engagement process we are studying human-to-human engagement in interactions. Our study provides an understanding of the capabilities required for human-robot interaction. At the same time, experimentation with human-robot interaction provides a valid means to test theories about engagement as well as to produce useful technological results. In this chapter we report on our initial experiments in programming a stationary robot to have initial engagement abilities.
2.
Hosting Activities
This research on engagement is framed around the activity of hosting. Hosting activities are a class of collaborative activity in which an agent provides guidance in the form of information, entertainment, education or other services in the user’s environment (which may be an artificial or the natural world) and may also request that the user undertake actions to support the fulfilment of those services. Hosting activities are situated or embedded activities, because they depend on the surrounding environment as well as the participants involved. They are social activities because, when undertaken by humans, they depend upon the social roles that people play to determine the choice of the next actions, timing of those actions, and negotiation about the choice of actions. In this research, the host of the environment are agents serving as guides, either on-screen animated ones or physical robots. Tutoring applications require hosting activities; this chapter reports on experience with a robot host who is acting as a tutor. Some portions of tutoring, such as testing a student’s knowledge, trouble shooting concepts that a student fails to grasp, and keeping track of what a student knows go beyond the informational services of hosting activities. Thus hosting is part of tutoring but not vice versa. Another common hosting activity is hosting a user in a room with a collection of artefacts. In such an environment, the ability of the host to interact with the physical world and visitors becomes essential, and justifies the creation of physical agents. Room hosting is a core activity in tour guiding in museums, and other indoor and outdoor spaces; see [Burgard et al., 1998] for a robot that can guide a museum tour. Sales activities include hosting as part of their mission in order to make customers aware of types of products and features, locations, personnel, and the like. In many activities, hosting may be
Experiment in Engagement for Human-Robot Interaction
57
intermingled with other tasks, e.g. with selling items in retail sales or student performance evaluation tasks in tutoring. Hosting activities are collaborative because neither party completely determines the goals to be undertaken nor the means of reaching the goal; these actions must be shared between the parties. While the visitor’s interests in the room may seem paramount in determining shared goals, the host’s (private) knowledge of the environment also constrains the goals that can be achieved. Typically the goals undertaken will need to be negotiated between visitor and host. Even in tutoring, where the tutor-host’s plans for how to tutor the student may seem to drive the interaction, the tutor and student negotiate on the problems they will undertake in their encounter. This work hypothesizes that by creating computer agents which function more like human hosts, the human participants will focus on the hosting activity and be less distracted by the agent interface. For example, the agent will gaze at the human partner and at domain objects in ways that appropriately indicate the agent’s attention to each. For example, if a human agent directs her gaze at a partner rather than an object, one can infer that the human is interested in her partner. When a human partner gazes away from the robot or objects of discussion, the robot must be able to assess whether the human has lost interest, and if so, determine how to re-establish or end the engagement between the two participants. Assuring that a robot behaves in ways with which people are familiar in human-to-human interactions increases the likelihood that the interaction will not break down due to the robot’s misusing or misunderstanding the cues of engagement.
3.
What is Engagement?
Engagement is fundamentally a collaborative process [Grosz and Sidner, 1990; Grosz and Kraus, 1996], although it also requires significant private planning on the part of each participant in the engagement. Engagement is collaborative principally because the people who are interacting intend to connect together and cannot do so alone. However, they may be less aware of the actions involved in accomplishing the joint engagement goals, e.g. gaze, head and hand gestures, unlike the conscious actions in other types of collaboration. Engagement, like other types of collaborations, consists of establishing the collaborative goal (the goal to be connected), maintaining the connection, and then ending the engagement. The collaboration process may include negotiation of the goal because a potential collaborator might not decide to become engaged right away or at all. In addition, participants might have to negotiate the means to achieve their goals [Sidner, 1994a; Sidner, 1994b]. For engagement in an interaction, participants negotiate the means for achieving engagement through the various ways they maintain engagement, and repair
58
Advances in Natural Multimodal Dialogue Systems
engagement when it appears to be failing. Described this way, engagement is similar to other collaborative activities. Engagement is an activity that contributes centrally to collaboration on other activities in the world and the conversations that support them. In fact, conversation is impossible without engagement. This claim does not imply that engagement is just a part of conversation. Rather engagement is a collaborative process that occurs in its own right, simply to establish connection between people, a natural social phenomenon of human existence. It is entirely possible to engage another without a single word being said and to maintain the engagement process with no conversation. That is not to say that engagement is possible without any communication; it is not. A person who engages another without language must rely effectively on some form of gestural communication to establish the engagement joint goal and to maintain the engagement. Gesture is also a significant feature of face-to-face interaction when conversations take place [McNeill, 1992]. Being engaged with another can also be the sole purpose of an interaction. The use of just a few words and gestures can establish and maintain connection with another when no other intended goals are relevant. For example, an exchange of hellos, a brief exchange of eye contact and a set of good-byes can accomplish an interaction just to be engaged. In such interactions, one can reasonably claim that the only purpose is to be connected. The current work focuses on interactions, ones that include conversations, where the participants wish to accomplish action in the world rather than just the relational connection that engagement can provide. Much of the engagement process can be accomplished by linguistic means only. Evidence for this statement derives from telephone conversations where participants are engaged with each other and have only the words they say, and prosodic effects (pitch, timing, duration, voice quality and the like) to indicate their desire for establishing, continuing and ending their connection to each other. However, in face-to-face interaction, people look at one another as they talk, and make use of gestures to indicate their interest in the other’s communication, and to indicate that they wish to continue while at the same time using other gestures to access other information in the environment. The engagement process must always balance the need to convey ongoing engagement with the conversational partner (or signal its demise) with the need to look at objects in the environment and perform actions called for by the collaboration (in addition to ones that are independent of it), as well as to interpret those gestures from the partner who has the same requirement to balance these needs.
Experiment in Engagement for Human-Robot Interaction
4.
59
First Experiment in Hosting: A Pointing Robot
In order to experiment with engagement in hosting activities, this effort began with a well-delimited problem: appropriate pointing and beat gestures for a stationary robot, called Mel, while conducting a conversation. Mel’s behaviour is a direct product of extensive research on animated pedagogical agents [Johnson et al., 2000]. It shares with those agents concerns about conversational signals and pointing. Unlike those efforts, Mel has greater dialogue capability, and its conversational signalling, including deixis, comes from combining the CollagenTM [Rich et al., 2001; Rich and Sidner, 1998] and Rea architectures [Cassell et al., 2001a]. Furthermore, while on-screen embodied agents [Cassell et al., 2000b] can point to things in an on-screen environment, on-screen agents cannot effectively point in a 3D space. So it seemed appropriate to explore the effects of deictic behaviour with a robot. To build a robot host, the effort relied significantly on the PACO agent [Rickel et al., 2002] built using CollagenTM for tutoring a user on the operation of a gas turbine engine. The PACO agent tutors a student on the procedures needed to control two engines by their various buttons and dials. The robot served as the tutor in this application and took on the task of speaking all the output and pointing to the portions of the display, tasks normally done by an on-screen agent in the PACO system. The student’s operation of the display, through a combination of speech input and mouse clicks, remained unchanged. Understanding of the student’s speech was accomplished with the IBM ViaVoiceTM speech recognizer, the IBM JSAPI1 to parse and interpret utterances, and the CollagenTM middleware to provide dialogue interpretation and next moves in the conversation, to manage the tutoring goals and to provide a student model for tutoring. The PACO screen for gas turbine engine tutoring is shown in Figure 3.1. The agent is represented in a small window, where text, a cursor hand and an iconic face appear. The face changes to indicate six states: the agent is speaking, is listening to the user, is waiting for the user to reply, is thinking, is acting on the interface, and has failed due to a system crash. The cursor hand is used to point out objects in the display. The robotic agent, Mel, is a stationary robot created at Mitsubishi Electric Research Labs, and consists of 5 servomotors to control the movement of the robot’s head, mouth and two appendages. The robot takes the appearance of a penguin. Mel can open and close his beak, move his head in up-down, and left-right combinations, and flap his "wings" up and down. He also has a laser light on his beak, and a speaker provides audio output for him. See Figure 3.2 for Mel pointing to a button on the gas turbine control panel.
1 See
the ViaVoice SDK, at http://www4.ibm.com/software/speech/dev/sdk java.html.
60
Advances in Natural Multimodal Dialogue Systems
Figure 3.1. The PACO agent for gas turbine engine tutoring.
For gas turbine tutoring, Mel sits in front of a large (2 feet x 3 feet) horizontal flat-screen display on which the gas turbine display panel is projected. To conduct a conversation with the student, Mel addresses the student faceon, and beats with his wings at appropriate points in his turn in the conversation. He uses the PACO system to teach the student procedures on the display. When he wishes to point to a button or dial on the display panel, he points with his beak. When he finishes pointing, he addresses the student face-on again. While Mel’s motor operations are extremely limited, they offer enough movement to undertake beat gestures, which indicate new and old information in utterances [Cassell et al., 2001b]. The head movement is also sufficient to point effectively at objects, so that students can readily see the objects on the panel. The architecture of a CollagenTM agent and an application using Mel is shown in Figure 3.3. Specifics of the CollagenTM internal organization and the means by which CollagenTM is connected to applications are beyond the scope of this chapter; see [Rich and Sidner, 1998; Rich et al., 2001] for more information. Basically, the application is connected to the CollagenTM system through the application adapter. The adapter translates between the semantic events CollagenTM understands and the events/function calls understood by the
Experiment in Engagement for Human-Robot Interaction
61
Figure 3.2. Mel pointing to the gas turbine control panel.
application. The agent controls the application by sending events to perform to the application, and the adapter sends performed events to CollagenTM when a user performs actions on the application. CollagenTM is notified of the propositions uttered by the agent via uttered events. They also go to the AgentHome window, which is a graphical component responsible in CollagenTM for showing the agent’s words on screen as well as generating speech in a speechenabled system. The shaded area highlights the components that were added to the standard CollagenTM middleware. With these additions, utterance events go through the Mel annotator and the BEAT system [Cassell et al., 2001b] in order to generate gestures as well as the utterances that Collagen already produces. More details on the architecture and Mel’s function with it can be found in [Sidner and Dzikovska, 2002]. In tutoring, the CollagenTM architecture is instantiated by means of a detailed set of recipes for the tutoring domain that must be specified in the Planning and Discourse module. Recipes are the means by which the hosting environment is specified for the robot. The recipes do not specify dialogue actions, but instead must detail the actions needed to operate the gas turbine panel and the actions needed to tutor students to use the panel. Additional rules in the
62
Advances in Natural Multimodal Dialogue Systems
Agent help the robot tutor decide what to say or do next, by choosing from a list of next moves created by the Planning and Discourse module. The recipes and rules apply to the domain that is being tutored only, and do not affect the engagement mechanisms that determine the robot’s wing and head gestures. The engagement mechanisms are usable in any tutoring activity. However, the tutoring domain recipes do provide a piece of information for the engagement mechanisms, namely what items in the display need to be pointed out. In this way, the current architecture separates linguistic and gesture functions. Thus, like a person, the robot could convey engagement with its linguistic behaviour but also convey the desire to disengage with its gestures.
Figure 3.3. Architecture of Mel.
5.
Making Progress on Hosting Behaviours
Mel is quite effective at pointing in a display so that it can be readily followed by people. Mel’s beak is a large enough pointer to operate in the way
Experiment in Engagement for Human-Robot Interaction
63
that a finger does. Pointing within a very small margin of error (which is assured by careful calibration before Mel begins talking) makes it possible to locate the appropriate buttons and dials on the screen. Mel returns his “gaze” 2 to the student after pointing, which is a signal that he is staying engaged with the user. These two behaviours are a first step in creating engagement. They make it evident to the student that Mel is interacting with the student and that his looks away are not intended to disengage but rather to accomplish a part of the task at hand. However, human engagement is far richer than what this robot can currently do [Sidner et al., 2003]. Most significantly engagement is a twopart activity: engaging behaviours must be produced, and at the same time, the engaging agent must interpret engaging behaviours from its conversational collaborator. Two of the most basic aspects of engagement are beginning and ending it. The means by which one begins and ends an interaction with Mel are minimal. While Mel responds to a student greeting to start the conversation, he does not have any means or goals to decide when and how to begin the conversation itself. Mel also does not know how to end a conversation or when it is appropriate to do so. Furthermore, Mel has only two weak ways of checking on his partner’s signals of engagement during their interaction: to ask "okay?" and await a response from the user after every explanation he offers, and to await (including indefinitely) a user response (utterance or action) after each time he instructs the user to act. In human-to-human interactions, engagement activities range over far more linguistic and gestural behaviours. In more recent system building efforts using Mel, [Sidner et al., 2005], Mel produces a much wider range of interactions to look at and interpret some looking acts from the human participant, to gesture at objects, and to begin and end conversations. All these capabilities result from more sophisticated subsystems for Mel, such as vision algorithms for detecting human faces and sound location algorithms, as well as careful study of human-human scenarios and video data [Sidner et al., 2003] for determining the types of engagement strategies that humans use effectively in hosting situations. To understand more about the variety of behaviours to signal engagement, in the next section, properties of engagement in human-human interaction are discussed in detail, without regard to what and how robots might do this.
6.
Engagement for Human-Human Interaction
Engagement concerns the establishment of connection, the maintenance of it and the ending of the engagement when one or more of the participants in
2 Mel’s
eyes do not move, so to look at the person, the whole head must turn.
64
Advances in Natural Multimodal Dialogue Systems
an interaction desire it. First let us consider the problem of establishing a connection. A typical beginning of an interaction is familiar to everyone. One approaches another person with whom one would like to interact, looks towards them, attempts to catch their eye. Thereafter additional actions take place, called opening ritual greetings [Luger, 1983]. The two people smile and offer greetings. Verbal greetings can be accompanied by physical behaviours, such as hand shakes, hugs or bowing (which is common in Asian cultures). The greetee responds in kind coordination with the greeter. However, this beginning is not the only possible pattern. The greetee could have responded with a request for delay after part of the process (as in "oh hi, give me a minute," and then attending to another matter). Such a response puts the start of the interaction on hold, and the greeter has the option not to continue, just as the greetee does. A third option is illustrated in the scenario presented in Figure 3.4, part 1. This scenario, constructed for purposes of illustrating engagement, presents an interaction between a host and a visitor in a laboratory hosting setting. It illustrates a less than successful attempt at engagement. The visitor does not immediately engage with the host, who uses a greeting and an offer to provide a tour as means of (1) engaging the visitor and (2) proposing a joint activity in the hosting world. Neither the engagement nor the joint activity are accepted by the visitor. The visitor accomplishes this non-acceptance by ignoring the uptake of the engagement activity, which also quashes the tour offer. While there are many variations on the examples above, the point of the initial part of the interaction is to engage, that is, establish connection between the participants. It is a collaborative process, where each offers signals of engagement, which are used to begin the rest of the greeting process. It is collaborative because each person must do their part and intend their part to establish the connection. Failure to intend to participate denies the collaboration and failure to do one’s part causes the greeting to fail, although with failed greetings, the participants may try again. Once an interaction is established, there are a large number of indications of engagement. The list below is illustrative of ways in which engagement is signalled either directly or in concert with some other activity in the interaction. Note that not all of these signals are essential to interaction because people are capable of undertaking instructions and having conversations by phone and asynchronous connections such as email. However, without these signals, the collaboration is a slow enterprise and likely to breakdown because some actions may be interpreted as intending to disengage. Understanding what each signal conveys about engagement allows us to better understand human communication as well as build artificial agents that engage properly. The signals include:
Experiment in Engagement for Human-Robot Interaction
65
Part 1
Host: Hello, I’m the room host. Would you like me to show you around? Visitor: Part 2 Visitor: What is this? Host: That’s a camera that allows a computer to see as well as a person to track people as they move around a room. Visitor: What does it see? Host: Come over here and look at this monitor <points>. It will show you what the camera is seeing and what it identifies at each moment. Part 3 Visitor: Uh-huh. What are the boxes around the heads? Host: The program identifies the most interesting things in the room – faces. That shows it is finding a face. Visitor: Oh, I see. Well, what else is there? Part 4 Host: Let’s take a look at the multi-level screen over there <points> . Visitor: Host: Is there something else you want to see? Visitor: No, I think I’ve seen enough. Bye. Host: Ok. Bye. Figure 3.4.
Scenario for Room Hosting.
talking about and performing the task; turn taking and grounding [Clark, 1996]; timing (i.e. the pace of uptake of a turn); use of gaze at the speaker, gaze away for taking turns [Duncan, 1974; Cassell et al., 2000a], to track the speaker’s gestures with objects; use of gaze by speaker or non-speaker to check on the attention of other; hand gestures for pointing, iconic description, beat gestures, etc., see [Cassel, 2000; Johnson et al., 2000], and in the hosting setting, gestures associated with domain objects;
66
Advances in Natural Multimodal Dialogue Systems
head gestures (e.g. nods, shakes, sideways turns); body stance (i.e. facing towards the other, turning away, standing up when previously sitting and sitting down); facial gestures (not explored in this work but see [Pelachaud et al., 1996]); non-linguistic auditory responses (e.g. snorts, laughs); social relational activities (e.g. telling jokes, role playing, supportive rejoinders). Among these signals, the use of gaze and looks is one of the most direct means of conveying engagement or lack of it. The principle of conversational tracking governs looking in interactions: a participant in a collaborative conversation tracks the other participant’s face during the conversation in balance with the requirement to look away in order to: (1) participate in actions relevant to the collaboration, or (2) multi-task with activities unrelated to the current collaboration, such as scanning the surrounding environment for interest or danger, avoiding collisions, or performing personal activities [Sidner et al., 2005]. The robot reported here is successful in certain simple looking behaviours. It turns its head to point at objects relevant to the collaboration and then turns again to look at the human tutor for its own turn as well as the human’s. However, the robot cannot perform in the inverse capacity. It does not recognize the looks or gazes of its human partner and hence cannot tell whether the human is looking at objects relevant to the collaboration or looking around out of lack of interest and intentions to disengage. By contrast the host in the scenario in Figure 3.4, part 4 notices when the visitor fails to look at the relevant objects for the interaction and the host takes this as evidence of disengagement, which the host pursues. Taking part in the conversation and the collaboration which a conversation serves are clear evidence of engagement. Failing to perform the actions of a collaboration, however, are evidence of the intention to disengage. In the scenario in Figure 3.4, the visitor follows the host to the appropriate place in part 3. Failure to do so would indicate the intention to disengage from the collaborative task (of answering the visitor’s question about what the camera sees). When the visitor fails to follow the host’s pointing in part 4, this failure to uptake a proposed action that is necessary to their collaborative task is also a sign of disengagement. Of course, when the partner does not take up the task, but has some argument for something else to do, the partner is still engaged. The timing of a conversational turn, taking the turn, and offering backchannels are behaviours that are part of acknowledging the contribution of a conversational partner. During a conversation, the participant acting as the hearer
Experiment in Engagement for Human-Robot Interaction
67
for the conversation grounds the contributions of the other participant by indicating that he or she has heard and understood what was said [Clark, 1996]. Taking a turn makes it possible to use the first part of the turn to do so, and uptake of the turn in pace with the other speaker indicates that all is going smoothly. Both taking a turn and maintaining the pace of the turn are evidence of engagement. When a participant does not take a turn or delays the turn while attending to something outside the collaboration, that participant conveys that they no longer are attending to the interaction. Loss of attention is the most basic indicator that a participant both intends to disengage and has begun to do so. Backchannels are the means by which the hearer grounds the speaker’s turn when a full turn is not taken. Backchannels (using such phrases as “mmhm, yes, uh-huh” during pauses in the other participant’s speaking) permit the hearer to indicate they are following what is said. Most backchannels occur following a speaker utterance which uses prosody and duration to indicate that the backchannel is expected. Failure to provide this expected result is also indication that disengagement is intended. Body stance is a means of expressing the primary focus of one’s attention. Attending to another makes it possible to employ the principle of conversational tracking. Facing one’s conversational partner with one’s physical stance thus supports the ability to track the partner. However, many physical tasks that are part of a collaboration may preclude a frontal stance to the other participant. When washing the dishes, one must face the sink and dishes. When looking under the hood of a car for evidence of a problem, both participants must face the car and lean in under the hood. Using spoken linguistic utterances and more occasional looking indicates one’s ongoing engagement. Just how participants decide to balance looking in these types of circumstances is not well understood. Social cues can serve as signals of engagement. Social relational activities that maintain or establish social relationships among the dialogue participants are evidence of engagement. Social activities such as telling jokes, supportive rejoinders to what another has said, role playing, as well as auditory responses (such as snorts of approval/disapproval and laughing) may serve to accomplish a collaboration task by indicating some trouble with the task. However, they also reinforce social relationships among the participants. Bickmore [2003] notes that social dialogue (even without the performance of accompanying physical actions) increases the trust between dialogue participants, a claim also supported in [Bickmore and Cassell, 2001; Katagiri et al., 2001]. Through videotaped sessions of hosts and visitors, we have observed aspects of social relationships. The hosts and visitors in the videotaped sessions tell humorous stories, offer rejoinders or replies that go beyond conveying that the informa-
68
Advances in Natural Multimodal Dialogue Systems
tion just offered was understood, and even take on role playing with the host and the objects being exhibited. Figure 3.5 contains a portion of a transcript of one hosting session. In that session, the visitor and the host spontaneously play the part of two children using the special restaurant table that the host was demonstrating. The reader should note that their play is coordinated and interactive and is not discussed before it occurs. The role-playing begins at 10 in the figure and ends at 17. This segment of the transcript is preceded by the host P having shown the visitor C how restaurant customers order food in an imaginary restaurant using an actual electronic table, and having explained how wait staff might use the new electronic table to assist customers. Note that utterances by P and C are labelled with their letter, a colon, and italics, while other material describes their body actions. Social relational activities such as the role playing in Figure 3.5 allow participants to demonstrate that they are socially connected to one another in a demonstrable way. They are more than just looking at each other and nodding to one another, especially to accomplish their domain goals. They actively seek ways to indicate to the other that they have some social relation to each other. Failure of a participant to take part in social activities indicates trouble in the interaction; the participant may be simply wishing to change the relationship or indicate how she or he feels about it. However, the failure to participate may also indicate that the participant intends to disengage from the interaction. All collaborations have an end, either because the participants give up on the goal, cf. [Cohen and Levesque, 1990], or because the collaboration succeeds in achieving the desired goals. When collaboration on a domain task ends, participants can elect to negotiate an additional task collaboration or refrain from doing so. When they refrain, they then undertake to terminate their collaboration and end the engagement. Their means to do so are presumably as varied as the rituals to begin engagement, but common patterns comprise pre-closing, expressing appreciation, saying goodbye, with an optional handshake, or other gesture, and then moving away from one another. Preclosings [Schegloff and Sacks, 1973] convey that the end is coming. Expressing appreciation is part of a socially determined custom in the US (and many other cultures) when someone has performed a service for an individual. In the hosting data, the visitor expresses appreciation, with acknowledgement of the host. Where the host has had some role in persuading the visitor to participate, the host may express appreciation as part of the preclosing. Moving away is a strong cue that the disengagement has taken place. However, disengagement can be less smooth. In Figure 3.4, part 4, the host’s offer in the hosting activities is not accepted, not by a verbal denial, but by lack of response, an indication of disengagement. The host, who could have chosen to re-state his offer (with some persuasive comments), instead takes a simpler
Experiment in Engagement for Human-Robot Interaction
69
54: P turns head/eyes to C, raises hands up. C’s head down, eyes on table. 55: P moves away from C and table, raises hands and shakes them; moves totally away, fully upright . 56: P: Uh and show you how the system all works C looks at P and nods. 58: P sits down. P: ah 00: P: ah another aspect that we’re P rotates each hand in coordination. C looks at P. 01: P: worried about P shakes hands. 02: P: you know C nods. 04: P: sort of a you know this would fit very nicely in a sort of theme restaurant P looks at C; looks down. 05: C: MM-hm C looks at P, nods at "MM-hm." P: where you have lots of 06: P draws hands back to chest while looking at C. C: MM-hm P: kids C nods, looking at P. 07: P: I have kids. If you brought them to a P has hands out and open, looks down then at C. C still nods, looking at P. 09: P: restaurant like this P brings hands back to chest. C smiles and looks at P. 10: P looks down; at “oh oh” lunges out with arm and together points to table and looks at table. P: they would go oh oh 11: C: one of these, one of these, one of these C points at each phrase above and looks at table. P laughs. 13: P: I want ice cream <point>, I want cake <point> C: yes yes <simultaneous with “cake”> C points at “cake” looks at P, then brushes hair back. P looking at table. 15: P: pizza <points> P looking at table. C: Yes yes French fries <point> C looks at table as starts to point. 16: P: one of everything P pulls hands back ,looks at C. C: yes C looks at P. 17: P: and if the system just ordered stuff right then and there P looks at C, hands out and shakes, shakes again after “there.” C looking at P; brushes hair. C: Right right (said after “there”) 20: P: you’d be in big trouble || P looking at C and shakes hands again in same way as before. C looking at P, nods at ||. 23: C: But your kids would be ecstatic C looking at P. P looks at C, puts hands in lap Figure 3.5.
A Playtime Example.
70
Advances in Natural Multimodal Dialogue Systems
negotiation tack and asks what the visitor would like to see. This aspect of the interaction illustrates the private assessment and planning which individual participants undertake in engagement. Essentially, it addresses the private question: what will keep us engaged? With the question directed to the visitor, the host also intends to re-engage the visitor in the interaction, which is minimally successful. The visitor responds but uses the response to indicate that the interaction is drawing to a close. The closing ritual presented in Figure 3.4 is, in fact, abrupt, given the overall interaction that has preceded it because the visitor does not follow the American cultural convention of expressing appreciation or offering a simple thanks for the activities performed by the host.
7.
Computational Modelling of Human-Human Hosting and Engagement
A more solid basis of study of human hosting is needed. To that end, ongoing analysis of several videotaped interactions between human hosts and visitors in a natural hosting situation provides details about the use of looking in engagement in hosting [Sidner et al., 2003]. Human-to-human hosting is common enough that many people have participated in such an activity in museums, outdoor tours, or retail settings. Gathering data using videotaping is somewhat intrusive on the typical hosting encounter, but not so much so that people are aware of the taping at all times. Their behaviour is a reliable indicator of typical hosting interactions. Hosting involves more than just engagement, because the host and the visitor have to perform a joint task which is related to the hosting activity. Therefore, engagement has to be investigated in a task-oriented setting where the collaboration on a domain task is ongoing. In each videotaped session, the host was a lab researcher, while the visitor was a guest invited by the first author to visit and see the work going on in the lab. The host demonstrated new technology in a research lab to the visitor for between 28 and 50 minutes, with the variation determined by the host and the equipment available. Careful study of this data has provided new insights about the use of one’s looks that has lead to the principle of conversational tracking. Engagement is a collaboration that often happens together with collaboration on a domain task. In effect, at every moment in the hosting interactions, there are two collaborations happening, one for the participants to accomplish hosting (for example, to tour a lab, which is the domain task), and the other for the participants to stay engaged with each other. While the first collaboration provides evidence for the ongoing process of the second, it is not enough in and of itself. Engagement, as has been shown in this article, depends on many gestural actions as well as conversational comments. Furthermore, the initiation of engagement generally takes place before the domain task is explored,
Experiment in Engagement for Human-Robot Interaction
71
and engagement happens when there are no domain tasks being undertaken. Filling out this story is one of our ongoing research tasks. Collaboration on engagement transpires before, during and after collaboration on a domain task. So what theoretical models will help explain this multicollaboration process? One might want to argue that more complex machinery is needed than that so far suggested in conversational models of collaboration, cf. [Grosz and Sidner, 1990; Grosz and Kraus, 1996; Lochbaum, 1998]. However, we believe it is possible to account for engagement within this framework and to use the framework to develop a working computer agent, in particular a robot. In the conversational models of collaboration, collaboration on a domain task or tasks proceeds from group activity and is accompanied by conversation. The conversation reflects the tasks being undertaken through the structure of the segments of the conversation and the intentions conveyed by participants in those segments. Tasks, or as the theory dubs them, goals are modelled by a set of recipes that specify how actions are performed in the domain to achieve the goal. Actions in the recipe can be performed by either participant, and the participants are presumed to mutually believe or come to mutually believe the recipes of the collaboration. Participants also come to believe individual intentions to perform actions in the recipe. The theory assumes that each participant uses the actions and recipes (1) to recognize how actions by the other participant contribute to the goal and (2) to plan his or her own acts. Conversational collaboration theory does not specifically consider the nature of collaboration for engagement as part of conversation, but the theory and model are specified in a generic way that should also apply to engagement as a collaboration. To apply this theory to engagement, our challenge is to specify the set of rules and recipes that participants in hosting believe will achieve the goals of starting, maintaining and ending engagement. Furthermore, to express this theory computationally, a computational participant (such as a robot) must be able to recognize actions that use those recipes. Clearly recipes for the opening and closing of a conversation as a means of starting and ending engagement can be expressed in terms of actions on the part of the participants in a domain task collaboration. What remains to be discovered is the sets of actions and action groups that form the process of maintaining engagement during a domain task collaboration. In particular, turns in the conversation about the domain task as well as certain gaze, body stance and pointing gestures form the class of engagement actions. The exact composition of that class is as yet unclear. What is clear is that the robot has a two-part task: to engage with the visitor and to track engagement behaviours from the visitor. Finally, social relational behaviours play a part in both the domain collaboration and the engagement process. How does one account for the social relational behaviours discussed above in collaboration theory? While social
72
Advances in Natural Multimodal Dialogue Systems
relational behaviours also tell participants that their counterparts are engaged, they are enacted in the context of the domain task collaboration, and hence must function with the mechanisms for that purpose. Intermixing relational connection, a social goal, and domain collaboration are feasible in collaboration theory models. In particular, the goal of making a relational connection can be accomplished via actions that contribute to the goal of the domain task collaboration. However, each collaborator must ascertain through presumably complex reasoning that the actions (and associated recipes) will serve their social goals as well as contribute to the domain goals. Hence they must choose actions that contribute to social goals as well as domain goals. Then they must also ascertain that the social goals are compatible with ongoing engagement collaboration. Furthermore, they must undertake these goals jointly. The remarkable aspect of the preceding playtime example is that the participants do not explicitly agree to demonstrate how kids will act in the restaurant. Rather the host, who has previously demonstrated other aspects of eating in the electronic restaurant, relates the problem of children in a restaurant and begins to demonstrate the matter when the visitor jumps in and participates jointly. The host accepts this participation by simply continuing his part in it. It appears that they are jointly participating in the hosting goal, but at the same time they are also participating jointly in a social interaction. The details that describe and explain how hosting agents and visitors accomplish this second collaboration are an important goal of ongoing research. Not all social behaviours can be interpreted in the context of the domain task. Sometimes participants interrupt their collaborations to tell a story that is either not pertinent to the collaboration or while pertinent, is somehow out of order. These stories are interruptions of the current domain task collaboration and are understood as having some conversational purpose. As interruptions, they signal that engagement is happening as expected as long as the conversational details of the interruption operate to signal engagement. It is not interruptions in general that signal disengagement or a desire to move to disengage; it is failure to take up the interruption that signals disengagement possibilities.
7.1
Open Questions
The discussion above raises a number of questions that must be addressed in ongoing work. First, in the video data, the host and visitor often look away from each other at non-turn taking times, especially when they are displaying or using demo objects. They also look up or towards the other’s face in the midst of demo activities. The conversational collaboration model does not account for the kind of fine detail required to explain gaze changes, nor do the standard models of turn taking. How are we to account for these gaze changes as part of engagement? What drives collaborators to gaze away and back when
Experiment in Engagement for Human-Robot Interaction
73
undertaking actions with objects so that they and their collaborators remain engaged? Second, in the data, participants do not always explicitly acknowledge or accept what another participant has uttered. Sometimes they use laughs, snorts or expressions of surprise (such as “wow”) to indicate that they have heard and understood and even confirm what another has said. These verbal expressions are appropriate because they express appreciation of a joke, a humorous story or outcome of a demo. We are interested in the range and character of these phenomena, as well as how they are generated and interpreted. Third, this chapter argues that much of engagement can be modelled using the computational collaboration theory model of Grosz and Sidner [1990], Grosz and Kraus [1996], and Lochbaum [1998]. However, a fuller computational picture is needed to explain how participants decide to signal engagement as continuing and how to recognize these signals. One such picture has been explored in [Sidner et al., 2005].
8.
A Next Generation Mel
While pursuing theory of human-human engagement, we continue to add new capabilities for Mel that are founded on human communication. To accomplish this, the next generation Mel combines hosting conversations with other research at MERL on face tracking and sound location [Sidner et al., 2005]. This combination makes it possible to locate visitors and then greet them in ways similar to human experience. These vision and sound algorithms, as well as others under development, permit Mel to make use of nodding and changes in looking, which are important indicators in conversation for turn taking as well as expressions of attention. Mel’s architecture continues to evolve to the point that Mel has both a “brain,” performing CollagenTM related functions and a “body,” fusing sensory data to feed to the brain and controlling Mel’s motions. Building a robot that can detect faces, track them and notice when the face disengages for a brief or extended period of time demonstrates more engagement behaviour than has been possible before. One challenge for a robot host is to experiment with techniques for dealing with unexpected speech input. People, it is said, say that darndest things. While the CollagenTM middleware continues to be used for modelling conversation, the struggle continues to find reasonable behaviours for unexpected visitor utterances. For example, when demonstrating a device that requires filling a cup with water, a visitor may make a mistake and spill water on the floor and say “Oops, I spilled water on the floor.” To understand this utterance, the speech recognizer must correctly process the words, and the sentence semantics must give it a meaningful description, after which the conversation engine must determine the purpose of the meaning description. Finally Mel must re-
74
Advances in Natural Multimodal Dialogue Systems
spond to it. If this sort of utterance was not predicted to occur (and there will be many such utterances), the best response that Mel can currently produce is “I do not understand the purpose of your utterance. Please find a human to help me.” Even that response is only possible if Mel has understood the visitor utterance up to its purposive intent. Failures at speech recognition or sentence interpretation will produce even less informative error messages. These difficulties result from the limits of current speech understanding technology and continue to limit the naturalness of interaction with Mel. Research on improving speech recognition and sentence interpretation is therefore essential to the next generation Mel to improve interaction with human subjects.
9.
Summary
Hosting activities are a natural and common activity among humans, and one that can be accommodated by human-robot interaction. Making the human-machine experience a natural one requires understanding the nature of engagement and applying the same types of human engagement behaviour to robot participation in hosting activities, as well as other activities. Engagement is a collaborative activity that is accomplished through both linguistic and gestural means. The experiments described in this chapter with a stationary robot that can converse and point provide an initial example of an engaged conversationalist. Through study of human-human hosting activities, new models of engagement for human-robot hosting interaction will provide us with a more detailed means of interacting between humans and robots.
Acknowledgements The authors wish to acknowledge the work of Paul Dietz in creating Mel, Neal Lesh, Charles Rich, and the late Jeff Rickel on Collagen and PACO.
References Bickmore, T. (2003). Relational Agents: Effecting Change through HumanComputer Relationships. PhD thesis, Media Arts and Sciences, Massachusetts Institute of Technology. Bickmore, T. and Cassell, J. (2001). Relational Agents: A Model and Implementation of Building User Trust. In Proceedings of the International Conference for Human-Computer Interaction (CHI), pages 396–403. New York: ACM Press. Burgard, W., Cremes, A. B., Fox, D., Haehnel, D., Lakemeyer, G., Schulz, D., Steiner, W., and Thrun, S. (1998). The Interactive Museum Tour Guide Robot. In Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI), pages 11–18. Menlo Park, CA: AAAI Press.
Experiment in Engagement for Human-Robot Interaction
75
Cassel, J. (2000). Nudge Nudge Wink Wink: Elements of Face-to-Face Conversation for Embodied Conversational Agents. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 1–27. Cambridge, MA: MIT Press. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., and Yan, H. (2000a). Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 29–63. Cambridge, MA: MIT Press. Cassell, J., Nakano, Y. I., Bickmore, T. W., Sidner, C. L., and Rich, C. (2001a). Non-Verbal Cues for Discourse Structure. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL), pages 106–115. Menlo Park, CA: Morgan Kaufman Publishers. Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. (2000b). Embodied Conversational Agents. Cambridge, MA: MIT Press. Cassell, J., Vilhjálmsson, H., and Bickmore, T. W. (2001b). BEAT: The Behavior Expression Animation Toolkit. In Proceedings of SIGGRAPH 2001, pages 477–486. New York: ACM Press. Clark, H. H. (1996). Using Language. Cambridge: Cambridge University Press. Cohen, P. and Levesque, H. (1990). Persistence, Intention and Commitment. In Cohen, P., Morgan, J., and Pollack, M. E., editors, Intentions in Communication, pages 33–70. Cambridge, MA: MIT Press. Duncan, S. (1974). Some Signals and Rules for Taking Speaking Turns in Conversation. In Weitz, S., editor, Nonverbal Communication. New York: Oxford University Press. Grosz, B. J. and Kraus, S. (1996). Collaborative Plans for Complex Group Action. Artificial Intelligence, 86(2):269–357. Grosz, B. J. and Sidner, C. L. (1990). Plans for Discourse. In Cohen, P., Morgan, J., and Pollack, M., editors, Intentions in Communication, pages 417– 444. Cambridge: MIT Press. Johnson, W. L., Rickel, J. W., and Lester, J. C. (2000). Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education, 11:47–78. Katagiri, Y., Takahashi, T., and Takeuchi, Y. (2001). Social Persuasion in Human-Agent Interaction. In Proceedings of the Second IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pages 64–69. Menlo Park, CA: Morgan Kaufman Publishers. Lochbaum, K. E. (1998). A Collaborative Planning Model of Intentional Structure. Computational Linguistics, 24(4):525–572. Luger, H. H (1983). Some Aspects of Ritual Communication. Journal of Pragmatics, 7(3):695–711.
76
Advances in Natural Multimodal Dialogue Systems
McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press. Pelachaud, C., Badler, N., and Steedman, M. (1996). Generating Facial Expressions for Speech. Cognitive Science, 20(1):1–46. Rich, C. and Sidner, C. L. (1998). COLLAGEN: A Collaboration Manager for Software Interface Agents. User Modeling and User-Adapted Interaction, 8(3/4):315–350. Rich, C., Sidner, C. L., and Lesh, N. (2001). COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction. AI Magazine, Special Issue on Intelligent User Interfaces, 22(4):15–25. Rickel, J., Lesh, N., Rich, C., Sidner, C. L., and Gertner, A. (2002). Collaborative Discourse Theory as a Foundation for Tutorial Dialogue. In Proceedings of Intelligent Tutoring Systems. New York: ACM Press. Schegloff, E. and Sacks, H. (1973). Opening Up Closing. Semiotica, 7(4):289– 327. Sidner, C. L. (1994a). An Artificial Discourse Language for Collaborative Negotiation. In Proceedings of the Twelfth National Conference on Artificial Intelligence, volume 1, pages 814–819. Cambridge, MA: MIT Press. Sidner, C. L. (1994b). Negotiation in Collaborative Activity: A Discourse Analysis. Knowledge-Based Systems, 7(4):265–267. Sidner, C. L. and Dzikovska, M. (2002). Hosting Activities: Experience with and Future Directions for a Robot Agent Host. In Proceedings of the 2002 Conference on Intelligent User Interfaces, pages 143–150. New York: ACM Press. Sidner, C. L., Kidd, C., Lee, C., Lesh, N., and Rich, C. (2005). Explorations in Engagement for Humans and Robots. Artificial Intelligence. Forthcoming. Sidner, C. L., Lee, C., and Lesh, N. (2003). Engagement by Looking: Behaviors for Robots when Collaborating with People. In Kruijff-Korbayova, I. and Kosny, C., editors, DiaBruck: The Proceedings of the Seventh Workshop on the Semantics and Pragmatics of Dialogue, pages 123–130, University of Saarland, Germany.
PART II
ANNOTATION AND ANALYSIS OF MULTIMODAL DATA: SPEECH AND GESTURE
Chapter 4 FORM An Extensible, Kinematically-Based Gesture Annotation Scheme Craig H. Martell Department of Computer Science and The MOVES Institute∗ Naval Postgraduate School, Monterey, CA, USA [email protected]
Abstract
Annotated corpora have played a critical role in speech and natural language research; and, there is an increasing interest in corpora-based research in sign language and gesture as well. We present a non-semantic, geometrically-based annotation scheme, FORM, which allows an annotator to capture the kinematic information in a gesture just from videos of speakers. In addition, FORM stores this gestural information in Annotation Graph format—allowing for easy integration of gesture information with other types of communication information, e.g., discourse structure, parts of speech, intonation information, etc.1
Keywords:
Gesture, annotation, corpora, corpus-based methods, multimodal communication.
1.
Introduction
FORM2 is an annotation scheme designed both to describe the kinematic information in a gesture, as well as to be extensible in order to add speech and other conversational information.
∗ Much
of this work was done at the University of Pennsylvania and at The RAND Corporation as well. presentation is a modified version of [Martell, 2002]. 2 The author wishes to sincerely thank Adam Kendon for his input on the FORM project. He has provided not only suggestions as to the direction of the project, but also his unpublished work on a kinematicallybased gesture annotation scheme was the FORM project’s starting point [Kendon, 2000]. 1 This
79 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 79–95. © 2005 Springer. Printed in the Netherlands.
80
Advances in Natural Multimodal Dialogue Systems
Our goal is to build an extensible corpus of annotated videos in order to allow for general research on the relationship among the many different aspects of conversational interaction. Additionally, further tools and algorithms to add additional annotations and evaluate inter-annotator agreement will be developed. The end result of this work will be a corpus of annotated conversational interaction, which can be: extended to include new types of information concerning the same conversations; as new tag-sets and coding schemes are developed— discourse-structure or facial-expression, for example—new annotations could easily be added; used to test scientific hypotheses concerning the relationship of the paralinguistic aspects of communication to speech and to meaning; used to develop statistical algorithms to automatically analyze and generate these paralinguistic aspects of communication (e.g., for HumanComputer Interface research).
2.
Structure of FORM
FORM3 is designed as a series of tracks representing different aspects of the gestural space. Generally, each independently moved part of the body has two tracks, one track for Location/Shape/Orientation, and one for Movement. When a part of the body is held without movement, a Location object describes its position and spans the amount of time the position is held. When a part of the body is in motion, Location objects of zero duration are placed at the beginning and end of the movement. Location objects of zero duration are also used to indicate the Location information at critical points in certain complex gestures An object in a movement track spans the time period for which the body part in question is in motion. It is often the case that one part of the body will remain static while others move. For example, a single hand shape may be held throughout a gesture in which the upper arm moves. FORM’s multi-track system allows such disparate parts of single gestures to be recorded separately and efficiently and to be viewed easily once recorded. Once all tracks are filled with the appropriate information, it is easy to see the structure of a gesture broken down into its anatomical components. At the highest level of FORM are groups. Groups can contain subgroups. Within each group or subgroup are tracks. Each track contains a list of attributes concerning a particular part of the arm or body. At the lowest level 3 The author wishes to acknowledge Jesse Friedman and Paul Howard in this section.
here is from their Code Book section of http://www.ldc.upenn.edu/Projects/FORM/.
Most of what is written
81
FORM
(under each attribute), all possible values are listed. Described below are the tracks for the Location of the Right or Left UpperArm. Right/Left Arm Upper Arm (from the shoulder to the elbow). Location Upper arm lift (from side of the body) no lift 0-45 approx. 45 45-90 approx. 90 90-135 approx. 135 135-180 approx. 180 Relative elbow position: The upper arm lift attribute defines a circle on which the elbow can lie. The relative elbow position attribute indicates where on that circle the elbow lies. Combined, these two attributes provide full information about the location of the elbow and reveal total location information (in relation to the shoulder) of the upper arm. extremely inward inward front front-outward outward (in frontal plane) behind far behind Figure 4.1 - Figure 4.4 are example stills with the appropriate values of the above two attributes given. The next three attributes individually indicate the direction in which the biceps muscle is pointed in one spatial dimension. Taken together, these three attributes give the orientation of the upper arm.
82
Advances in Natural Multimodal Dialogue Systems
Figure 4.1.
Upper arm lift: approx. 90; Relative elbow position: outward.
Figure 4.2.
Upper arm lift: approx. 45; Relative elbow position: front.
Figure 4.3.
Upper arm lift: 0-45; Relative elbow position: behind.
Biceps: Inward/Outward none inward (see Figure 4.5) outward (see Figure 4.6)
83
FORM
Figure 4.4. Upper arm lift: no lift; Relative elbow position: outward.
Biceps: Upward/Downward none upward (see Figure 4.7) downward (see Figure 4.8) Biceps: Forward/Backward none forward (see Figure 4.9) backward (see Figure 4.10)
Figure 4.5. inward.
Obscured: This is an binary attribute which allows the annotator to indicate if the attributes and values chosen were “guesses” necessitated by visual occlusion. This attribute is present in each of FORM’s tracks. Again, we have only presented the Location tracks for the Right or Left Arm UpperArm group. The full “Code Book” can be found at
84
Advances in Natural Multimodal Dialogue Systems
Figure 4.6. outward.
Figure 4.7. upward.
Figure 4.8.
downward.
http://www.ldc.upenn.edu/Projects/FORM/. Listed there are all the Group, Subgroup, Track, Attribute and Value possibilities.
85
FORM
Figure 4.9.
Figure 4.10.
3.
forward.
backward.
Annotation Graphs
In order to allow for maximum extensibility, FORM uses annotation graphs (AGs) as its logical representation4 . As described in [Bird and Liberman, 1999], annotation graphs are a formal framework for “representing linguistic annotations of time series data.” AGs do this by extracting away from the physical-storage layer, as well as from application-specific formatting, to provide a “logical layer for annotation systems.” An annotation graph is a collection arcs and nodes which share a common time line, that of a video tape, for example. Each node represents a time stamp and each arc represents some linguistic event spanning the time between the nodes. In FORM, the arcs are labelled with both attributes and values, so that the arc given by the 4-tuple (1,5,Wrist Movement,Side-to-side) represents that there was side-to-side wrist movement between time stamp 1 and time stamp 5.
4 Cf.
[Martell, 2002] for a more complete discussion of FORM’s use of AGs.
86
Advances in Natural Multimodal Dialogue Systems
The advantage of using annotation graphs as the logical representation is that it is easy to combine heterogeneous data—as long as they share a common time line. So, if we have a dataset consisting of gesture-arcs, as above, we can easily extend this dataset by adding more arcs representing discourse structure, for example, simply by adding other arcs which have discourse-structure attributes and values. Again, this allows different researchers to use the same linguistic data for many different purposes, while, at the same time, allowing others to explore the correlations between the different phenomena being studied.
4.
Annotation Example
To gain a better understanding of the process of FORM annotation, we present here a small visual example. The four stills of Figure 4.11 are from a video sequence of Brian MacWhinney teaching a research methods course at Carnegie Mellon University5 . We show these four key frames, here, for illustrative purposes only. The character of the gesture is gleaned from viewing the continuous movement in the video. However, these key frames would be used to set the time stamp and locations of the beginning and end of the movement and in-between points that are important to capturing its shape. The arcs in the annotation graph described below (Figure 4.13) capture the information for the movement in between the key frames.
Figure 4.11. Snapshots of Brian MacWhinney on January 24, 2000.
The FORM annotation, then, of the video, from time stamp 1:13.34 (1 minute 13.34 seconds) to time stamp 1:14.01 is shown in Figure 4.12. This is the view on the data that a particular tool, Anvil [Kipp, 2001], presents to the annotator6 . As described above, FORM uses annotation graphs as its logical 5 These
data were chosen because they are part of the TalkBank collection (http://www.talkbank.org). TalkBank was responsible for funding a large part of this project. 6 Anvil is described in further detail in the appendix.
87
FORM
Figure 4.12.
FORM annotation of Jan24.mov, using Anvil as the annotation tool.
representation of the data; so regardless of choice of annotation tool, FORM’s internal view is the annotation graph given in Figure 4.13.
Figure 4.13.
FORM/Annotation Graph representation of example gesture.
88
Advances in Natural Multimodal Dialogue Systems
Again, FORM uses vectors of attribute:value pairs to capture the gestural information of each section of the arms and hands. In Figure 4.13, then, the arc labelled HandandWrist.Movement from 1:13.34 to 1:13.57 encodes the kinematics of Brian’s moving his right hand or wrist during this time period, and the arc from 1:13.24 to 1:13.67 encodes a change in his right hand’s shape.7
5.
Preliminary Inter-Annotator Agreement Results
Preliminary results from FORM show that with sufficient training, agreement among the annotators can be very high. Table 4.1 shows preliminary interannotator agreement results from a FORM pilot study.8 The results are for two trained annotators for approximately 1.5 minutes of Jan24-09.mov, the video from Figure 4.11. For this clip, the two annotators agreed that there were at least these 4 gesture excursions. One annotator found 2 additional excursions. Precision refers to the decimal precision of the time stamps given for the beginning and end of gestural components. The SAME value means that all time-stamps were given the same value. This was done in order to judge agreement with having to judge the exact beginning and end of an excursion factored out. Exact vs. No-Value percentage refers to whether both the attributes and values matched exactly or whether just the attributes matched exactly. This distinction is included because a gesture excursion is defined as all movement between two rest positions of the arms and hands. For an excursion, the annotators have to judge both which parts of the arms and hands are salient to the movement (e.g., upper-arm lift and rotation, forearm change in orientation and hand/wrist position) as well as what values to assign (e.g., the upper-arm lifted 15-degrees and rotated 45-degrees). So, the No-Value% column captures the degree to which the annotators agree just on the structure of the movement, while Exact% measures agreement on both structure and values. The degree to which inter-annotator agreement varies among these gestures might suggest difficulty in reaching consensus. However, the results on intraannotator agreement studies demonstrate that a single annotator shows similar variance when doing the same video-clip at different times. Table 4.2 gives the intra-annotator results for one annotator annotating the first 2 gesture excursions of Jan24-09.mov. For both sets of data, the pattern is the same: the less precise the time-stamps, the better the results; No-Value% is significantly higher than Exact%. 7 For the example given in Figure 4.11, Brian is only moving his right hand. Accordingly, the Right. which normally would have been prefixed to the arc-labels has been left off. 8 Essentially, all the arcs for each annotator are thrown into a bag. Then all the bags are combined and the intersection is extracted. This intersection constitutes the overlap in annotation, i.e., where the annotators agreed. The percentage of the intersection to the whole is then calculated to get the scores presented.
89
FORM Table 4.1. Inter-Annotator Agreement on Jan24-09.mov. Gesture Excursion
Precision
Exact%
No-Value%
1
2 1 0 SAME
3.41 10.07 29.44 56.92
4.35 12.8 41.38 86.15
2
2 1 0 SAME
37.5 60 75.56 73.24
52.5 77.5 94.81 95.77
3
2 1 0 SAME
0 19.25 62.5 67.61
0 27.81 86.11 95.77
4
2 1 0 SAME
10.2 25.68 57.77 68.29
12.06 31.72 77.67 95.12
Table 4.2. Intra-Annotator Agreement on Jan24-09.mov. Gesture Excursion
Precision
Exact%
No-Value%
1
0 1 0 SAME
5.98 20.52 58.03 85.52
7.56 25.21 74.64 96.55
2
2 1 0 SAME
0 25.81 89.06 90.91
0 28.39 95.31 93.94
It is also important to note that Gesture Excursion 1 is far more complex than Gesture Excursion 2. And, in both simple and complex gestures, interannotator agreement is approaching intra-annotator agreement. Notice, also, that for Excursion 2, inner-annotator agreement is actually better than intraannotator agreement for the first two rows. This is a result of the difficulty for even the same person over time to precisely pin down the beginning and end of a gesture excursion. Although the preliminary results are very encouraging,
90
Advances in Natural Multimodal Dialogue Systems
all of the above suggests that further research concerning training and how to judge similarity of gestures is necessary. Visual information may need very different similarity criteria. Also, it is not clear as of the time of writing how these results might generalize. In particular, the relationship between interand intra-annotator agreement needs to be further explored. In addition, comparison studies with other methods of judging agreement are necessary. For example, how does FORM’s method compare with Cronbach’s alpha evaluated at discrete time-slices9 ? And, what would be the result adding a kappa-score analysis to the bag-of-arcs technique?
6.
Conclusion: Applications to HLT and HCI?
We are augmenting FORM to include richer paralinguistic information (Head/Torso Movement, Transcription/Syntactic Information, and Intonation/Pitch Information). This will create a corpus that allows for research that heretofore we have been unable to do. It will facilitate experiments that we predict will be useful for speech recognition and other Human-Language Technologies (HLT). As an example of similar research, consider the work of Francis Quek et al. [2001]. They have been able to demonstrate that gestural information is useful in helping with automatic detection of discourse transition. However, their results are limited by the amount of kinematic information they can gather with their video-capture system. Further, we believe an augmented-FORM corpus will contain much more specific data and will allow for more fine-grained analyses than is currently feasible. Additionally, knowing the relationships among the different facets of human conversation will allow for more informed research in Human-Computer Interaction (HCI). If one of the goals of HCI is to have better immersive-training, then it will be imperative that we understand the subtle connections among the paralinguistic aspects of interaction. A virtual human, for example, would be much better if it were able to understand, and act in accordance with, all of our communicative modalities. Having an extensible corpus such as we describe in this chapter is a first step that will allow many researchers, across many disciplines, to explore these and other useful ideas.
9I
am indebted to an anonymous reviewer for this suggestion.
91
FORM
Appendix: Other Tools, Schemes and Methods of Gesture Analysis FORM has been designed to be simultaneously useful for both: (1) capturing the kinematics of gesture; and (2) developing a corpus of annotated videos useful for computational analysis and synthesis. Prior research along each of these dimensions that has contributed to, or has motivated, FORM. In this section we briefly review this prior work.
A.1
Non-computational Gesture Analysis
Two important figures in linguistic and psychological (read: non-computational) analysis of gesture are David McNeil and Adam Kendon. Each has developed annotation schemes and systems to analyze and annotate the gestures of speakers in video. However, their respective levels of analysis are quite different.
A.1.1 David McNeill: Hand and Mind. David McNeil [1992] uses as scheme which divides the gesture space into four basic types: Beat gestures; Iconic gestures; Metaphoric gestures; and Deictic gestures. These categories are not meant to be mutually exclusive, although McNeill [1992] has been blamed for making it appear so. According to the McNeill Lab web site: A misconception has arisen about the nature of the gesture categories described in Hand and Mind, to wit, that they are mutually exclusive bins into which gestures should be dumped. In fact, pretty much any gesture is going to involve more than one category. Take a classic upward path gesture of the sort that many subjects produce when they describe the event of the cat climbing up the pipe in our cartoon stimulus. This gesture involves an iconic path-for-path mapping, but is also deictic. . . . Even "simple" beats are often made in a particular location which the speaker has given further structure (e.g. by setting up an entity there and repeatedly referring to it in that spatial location). Metaphoric gestures are de facto iconic gestures. . . . The notion of a type, therefore, should be considered as a continuum–with a given gesture having more or less iconicity, metaphoricity, etc.10 10 http://mcneilllab.uchicago.edu/topics/type.html,
as of 12/15/2003.
92
Advances in Natural Multimodal Dialogue Systems
This work has been very influential, and has been the basis for at least one major computational project (see the BEAT toolkit, below). However, this level of analysis only serves to categorize the gesture. It provides no useful computational information for either automatic gesture analysis or for the automatic generation of gestures in computational agents.
A.1.2 Adam Kendon: The Kinetics of Gesture. Adam Kendon’s approach, best articulated in “An Agenda for Gesture Studies” [Kendon, 1996], is to annotate and analyze at a more fine-grained, level. His goal is to develop a “kinetics” of gesture, analogous to the “phonetics” of speech. As such, he develops in [Kendon, 2000] a scheme which captures how joints are bent, how the different aspects of the arm move, and even how these different dimensions of gesture align with speech. This system describes positions and changes in position of the speakers arms, hands, head and torso. Unfortunately, from our perspective, the annotation scheme was designed to be written on paper or to be used with a word processor. As such, there is not a sufficient way to do finegrained time alignment of gesture to speech. FORM’s original motivation was to computerize this scheme so that fine-grained time alignment was possible. Kendon’s work is the fundamental starting point for FORM.
A.2
Computer-based Annotation Tools and Systems
A.2.1 CHILDES/CLAN. The CHILDES/CLAN system [MacWhinney, 1996] is a suite of tools for studying conversational interactions in general. The suite allows for, among other things, the coding and analyzing of transcripts and for linking those transcripts to digitized audio and video. CLAN supports both CHAT and CA (Conversational Analysis) notation, with the alignment of text to the digitized media at the phrase level. The CHILDES/CLAN system has the major advantage of being one of the first of its kind. The CHILDES database of transcripts of parent-child interactions has dramatically pushed forward both the theory and science of linguistics and language-acquisition. Additionally, it appears possible—in the future—to integrate FORM data with that developed by CHILDES/CLAN into a unified data set. This is due to the open-ended nature and extensibility of both systems. However, from the perspective of actually annotating videos with fine-grained, time-aligned gesture data, CLAN presents a problem. It is possible to describe the gesture that occurred during an utterance, but, given that time alignment is only at the phrasal level, we are unable to finely associate the parts of the gesture with other aspects of conversational interaction. A.2.2 SignStream. SignStream [Neidle et al., 2001] allows users to annotate video and audio language data in multiple parallel fields that display the temporal alignment and relations among events. It has been used most
93
FORM
extensively for analysis of signed languages. It allows for annotation of manual and non-manual (head, face, body) information; type of message (e.g. Whquestion); parts of speech; and spoken-language translations of sentences. Although SignStream would work with the FORM annotation scheme, and there has been some attempt at integrating the two projects, its interface is too comprehensive. Anvil, described below, allows an annotator to quickly see the relationship among all the aspects of left arm, right arm, head and torso movement.
A.2.3 Anvil. Anvil [Kipp, 2001] is a Java-based tool which permits multi-layered annotation of video with gesture, posture, and discourse information. The tags used can be freely specified, and can easily be hierarchically arranged. See Figure 4.12 as an example. Anvil is the tool of choice for the work done in the FORM Lab. It works well for creating multi-tiered, hierarchical, time-aligned annotations. In the beginning of FORM, we toyed with the idea of building FORMTool, our own annotation tool. However, we soon realized that we were just duplicating the benefits of Anvil. Additionally, the extensible nature of Anvil will allow for the development of an Annotation Graph plug-in, so our data can be directly exported to AG format. Currently, we save the data in Anvil XML format and convert to AG format. This future plug-in will avoid this step.
A.3
Systems for Computational Analysis and Generation
A.3.1 VISLab: Francis Quek. The VISLab project11 is a large-scale, low-level-of-analysis research project developed and led by Francis Quek at Wright State University. It has achieved significant results in understanding the relationship of speech to gesture. See [Quek et al., 2001], for example. The long-term intent of the project is to create a large-scale dataset of videos annotated with information about gesture, speech and gaze. This project is in the same spirit as the FORM project, and there has been significant collaboration. There are plans for the VISLab to store the gesture aspects of their data in the FORM format. There are, however, major differences between the two projects. Firstly, FORM aims at developing a mid-level representation that humans can use to annotate gestures and that machines can use to analyze and generate gestures. The VISLab system’s level of representation is much lower. They use multiple cameras to extract 3D information about position, velocity, acceleration, etc. concerning a gesture. They are doing the physics of gesture, where FORM is looking at something closer to the phonetics of gesture. Secondly, the VISLab system requires a complex set up
11 http://vislab.cs.wright.edu/
94
Advances in Natural Multimodal Dialogue Systems
of multiple, precisely-positioned cameras and proper placement of the subjects in order to gather their data. FORM allows any researcher with a notebook PC and a video camera to generate useful data. Thus, FORM can be used in the “field,” where the VISLab system requires a laboratory setting.
A.3.2 BEAT Toolkit: Justine Cassell et al. The other important, large-scale project is the Behaviour Expression Animation Toolkit (BEAT) [Cassell et al., 2001]. It was developed at the MIT Media Lab in the Gesture and Narrative Language Research Group. This work is advanced and is, by far, the most influential to date. It allows for the easy generation of synchronized speech and gesture in computer-animated characters. The animator simply types in the sentence that he/she wishes the character to say, and the BEAT Toolkit generates marked-up text which can serve as the input to any of a number animation systems. The system is extensible to many different communicative behaviours and domains. The output generated for a given input string is domain specific, and the training data for that domain must be provided. The main purpose of BEAT is to appropriately schedule gestures (and other non-verbal behaviours) so they are synchronized with the speech. The BEAT toolkit is the first of a new generation (the beat generation) of animation tool that extracts actual linguistic and contextual information from text in order to suggest correlated gestures, eye gaze, and other nonverbal behaviours, and to synchronize those behaviours to one another. For those animators who wish to maintain the most control over output, BEAT can be seen as a kind of “snap-to-grid” for communicative actions: if animators input text, and a set of eye, face, head and hand behaviours for phrases, the system will correctly align the behaviours to one another, and send the timings to an animation system. For animators who wish to concentrate on higher level concerns such as personality, or lower level concerns such as motion characteristics, BEAT takes care of the middle level of animation: choosing how nonverbal behaviours can best convey the message of typed text, and scheduling them.12
FORM’s relationship to BEAT is more one of potential partners than as competitors. BEAT is most concerned with the automatic generations of the timings, and the higher and lower levels, as aforementioned, are left to the animator. In particular, the lower level of specifying motion characteristics is where FORM is most concerned. We see FORM as potentially a more robust way to specify the gestures for which BEAT schedules the timings. The typology of gestures that BEAT uses is based on the work of McNeill13 . As such, it sees gestures through the eyes of his ontology. It is, then, left up to the animator to specify exactly how a beat or a deictic, for example, is to be animated. 12 [Cassell 13 Cf.
et al., 2001, page 8]. discussion of McNeill, above.
FORM
95
We believe the data generated by the FORM annotation system will allow for a more robust output from BEAT, which would further alleviate the work of animators.
References Bird, S. and Liberman, M. (1999). A Formal Framework for Linguistic Annotation. Technical Report MS-CIS-99-01, Department of Computer and Information Sciences, University of Pennsylvania, Philadelphia, Pennsylvania. http://citeseer.nj.nec.com/article/bird99formal.html. Cassell, J., Vilhjálmsson, H. H., and Bickmore, T. (2001). BEAT: The Behavior Expression Animation Toolkit. In Fiume, E., editor, Proceedings of SIGGRAPH, pages 477–486. ACM Press / ACM SIGGRAPH. http://citeseer.ist.psu.edu/cassell01beat.html. Kendon, A. (1996). An Agenda for Gesture Studies. Semiotic Review of Books, 7(3):8–12. Kendon, A. (2000). Suggestions for a Descriptive Notation for Manual Gestures. Unpublished. Kipp, M. (2001). Anvil - A Generic Annotation Tool for Multimodal Dialogue. In Proceedings of Eurospeech 2001, pages 1367–1370, Aalborg, Denmark. MacWhinney, B. (1996). The CHILDES System. American Journal of SpeechLanguage Pathology, 5:5–14. Martell, C. (2002). FORM: An Extensible, Kinematically-based Gesture Annotation Scheme. In Proceedings of International Language Resources and Evaluation Conference (LREC), pages 183–187. European Language Resources Association. http://www.ldc.upenn.edu/Projects/FORM. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago, USA. Neidle, C., Sclaroff, S., and Athitsos, V. (2001). SignStream: A Tool for Linguistic and Computer Vision Research on Visual-Gestural Language Data. In Behavior Research Methods, Instruments, and Computers, volume 33:3, pages 311–320. Psychonomic Society Publications. http://www.bu.edu/asllrp/SignStream/. Quek, F., Bryll, R., McNeill, D., and Harper, M. (2001). Gestural Origo and Loci-Transitions in Natural Discourse Segmentation. Technical Report VISLab-01-12, Department of Computer Science and Engineering, Wright State University. http://vislab.cs.vt.edu/Publications/2001/QueBMH01.html.
Chapter 5 ON THE RELATIONSHIPS AMONG SPEECH, GESTURES, AND OBJECT MANIPULATION IN VIRTUAL ENVIRONMENTS: INITIAL EVIDENCE Andrea Corradini and Philip R. Cohen Center for Human-Computer Communication Department of Computer Science and Engineering Oregon Health & Science University, Portland, OR, USA
{andrea, pcohen}@cse.ogi.edu Abstract
This chapter reports on a study whose goal was to investigate how people make use of gestures and spoken utterances while playing a videogame without the support of standard input devices. We deploy a Wizard of Oz technique to collect audio- video- and body movement-related data on people’s free use of gesture and speech input. Data was collected from ten subjects for up to 60 minutes of game interaction each. We provide information on preferential mode use, as well as the predictability of gesture based on the objects in the scene. The long-term goal of this on-going study is to collect natural and reliable data from different input modalities, which could provide training data for the design and development of a robust multimodal recognizer.
Keywords:
Multimodal speech-gesture data analysis and collection, interaction in immersive environment, object affordance and manipulation, Wizard of Oz study.
1.
Introduction
Human-computer interaction in virtual environments has long been based on gesture, with the user’s hand(s) being tracked acoustically, magnetically, or via computer vision. In order to execute operations in virtual environments, users often are equipped with datagloves, whose hand-shapes are captured digitally, or a tracked device that is equipped with multiple buttons. For a number of reasons, these systems have frequently been difficult to use. First, although the user’s hand/arm motions are commonly called “gesture,” the movements to 97 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 97–112. © 2005 Springer. Printed in the Netherlands.
98
Advances in Natural Multimodal Dialogue Systems
be recognized are typically chosen by the developer. Thus, rather than recognize people’s naturally occurring movements, such systems require users to learn how to move “properly.” Secondly, gestural devices have many buttons and modes, making it difficult for a naïve subject to remember precisely which button in a given mode accomplishes which function. Third, the 3D interaction paradigm usually derives from the 2D-based direct manipulation style, in which one selects an object and then operates upon it. Some systems have modelled the virtual environment interface even more strongly upon the WIMP (windows, icons, menus, pointing device) graphical user interface, providing users with menus that need to be manipulated in the 3D world. Unfortunately, it turns out to be very difficult to select objects and menu entries in 3D environments. Various researchers have attempted to overcome these awkward interfaces in different ways. For example, Hinckley et al. [1994] gave users real-world models to manipulate, causing analogous actions to take place in the virtual environment. Stoakley et al. [1995] provided a miniature copy of the virtual world in the user’s virtual hand, thereby allowing smaller movements in hand to have analogous results on the world itself. Fisher et al. [1986] developed an early multimodal 3D interface for simulated Space Station operations, incorporating limited speech recognition, as well as hand gestures using a VPL dataglove. Weimer and Ganapathy [1989] developed a prototype virtual environment interface that incorporated a VPL dataglove, and a simple speech recognizer. Although only three gestures, all by the user’s thumb, were recognized, and the speech system offered just a 40 word vocabulary, the authors remarked upon the apparent improvement in the interface once voice was added. Based in part on this prior research, we hypothesize that multimodal interaction in virtual environments can ease the users’ burden by distributing the communicative tasks to the most appropriate modalities. By employing speech for its strengths, such as asking questions, invoking actions, etc., while using gesture to point at locations and objects, trace paths, and manipulate objects, users can more easily engage in virtual environment interaction. In order to build such multimodal systems, we need to understand how, if at all, people would speak and/or gesture in virtual environments if given the choice. What would users do on their own, without being limited to the researchers’ preferred gestures and language? Would gestures and language be predictable, and if so from what? Can the recognition of gesture and/or speech in virtual environments be improved by recognizing or understanding the input in another modality, as we find in 2D map-systems [Cohen et al., 1997; Oviatt, 1999]. Regarding predictability of speech and gesture, we hypothesize that without instruction, people will manipulate manufactured objects in VE in the ways they were designed to be manipulated – using their affordances [Gibson, 1977;
On the Relationships among Speech, Gestures, and Object Manipulation
99
Norman, 1988]. Given data indicating a user’s viewpoint on the object, and the degrees of freedom afforded by the object, a system should be able to predict how the user’s hand/arm will move. If the user can also speak, will they employ the same gestures during multimodal interaction as they employ using gesture alone? In order to answer these questions, and to provide a first set of data for training recognizers and statistically-based multimodal integration systems, we conducted a Wizard-of-Oz study of multimodal interaction with a simple, though realistically rendered, computer game.
2.
Study
Ten volunteer subjects (nine adults, one 12 year-old child), interacted with the MystTM III game played on a 2GHz Dell computer with Nvidia GeForce 3 graphics card. None of the subjects had played Myst before; three had played video games regularly. Myst is a semi-immersive 2.5-dimensional game in which a user moves around a complex world, containing both indoor and outdoor scenes. The user views the world through a (moderate) fish-eye viewport, which s/he can rotate 360 degrees, as well as tilt to see above and below. In Myst III, the user’s task is (partially) to travel around an island, rotating a series of beacons so that they shine on one another in a certain sequence, etc. Thus, the game involves navigation, manipulation of objects (doors, a knife switch, beacons, push-buttons, etc.) and search. The subjects wore a set of four 3D magnetic trackers attached to their head and dominant arm. They were told that they could interact with the game as they wished, and that the system could understand their speech and gestures.
2.1
Wizard of Oz Study
The classic method for studying recognition-based systems before the appropriate recognizers have been trained is to employ a Wizard of Oz paradigm [Oviatt et al., 1994]. In this kind of study, an unseen assistant plays the role of the computer, processing the user’s input and responding, as the system should. Importantly, the response time should be rapid enough to support satisfactory interactive performance. In the present study, subjects were told that they would be playing the Myst III computer game, to which they could speak and gesture freely. The user chose where s/he wanted to stand. Subjects “played” the game standing in front of a 50” diagonal flat-panel plasma display in wide-view mode. They could and did speak and/or gesture without constraint. Unbeknownst to the subject, a researcher observed the subjects’ inputs, and controlled the game on a local computer, whose audio and video output was sent to the subject. The “wizard’s” responses resulted in scene navigation and object state changes, with a response time that aver-
100
Advances in Natural Multimodal Dialogue Systems
Figure 5.1. Example of experimental setup, utterance, gesture, and block figure reproducing the subject’s motions.
aged less than 0.5 seconds. Since Myst III (and its predecessors) assumes the user is employing a mouse, it is designed to minimize actual gesturing, allowing only mouse-selection. Although occasionally the Wizard made errors, subjects received no explicitly marked recognition errors. A research assistant was present in the room with the subject, and would upon request give the subject hints about how to play the game, though not about what to say or how to gesture.
2.2
Equipment
To acquire the gesture data, the six-degree-of-freedom Flock of Bird (FOB) magnetic tracking device from Ascension Technology Corporation [2002] was used. We attached four sensors to the subject; one on the top head, one on the upper arm to register the position and orientation of the humerus, one on the lower arm for the wrist position and lower arm orientation, and finally one on the top of the hand. The last three sensors are aligned with each other anytime the subject stretched his or her arm to the side of the body, keeping the palm of the hand facing and parallel to the ground (see Figure 5.2). The data from the FOB are delivered via serial lines, one for each sensor. The four data streams can be processed in real time by a single SGI Octane machine employing the Virtual Reality Peripheral Network (VRPN) package [Taylor, 2001], which provides time-stamps of the data and distrib-
On the Relationships among Speech, Gestures, and Object Manipulation
101
Figure 5.2. Arrangement of trackers on subject’s body.
utes it to customer processes. Because the FOB devices uses a magnetic field that is affected by metallic objects, and the laboratory is constructed of steelreinforced concrete, the data from the sensors is often distorted. As a result, the data is processed with a median filter to partially eliminate noise. A block figure plays back the sensor data, providing both a check on accuracy and distortion. Accuracy of the magnetic tracker in our environment was approximately 0.5 cm.
3.
Data Analysis
The subject’s body motions were captured by the FOB, while the video recorded the subject’s view, and vocal interaction (see Figure 5.1). The speech and gesture on the video were transcribed, an example of which follows:
TRANSCRIPT FROM THE GAME Bold = speech # = location of gesture when not overlapping speech (. . . ) = hesitation/pause XXX = undecipherable [ ] = speech-gestural stroke overlapped event Indications such as “08-01-42-25/44-11” = VCR time-code
102
Advances in Natural Multimodal Dialogue Systems 08-01-42-25/44-11: # go across the bridge [hand held palm open to point toward the bridge then hand used as cursor along bridge] 08-01-47-00: keep going no gesture 08-01-50-03/57-27: [grab] this thing (. . . ) just [grab it] and pull it down and see what happen # reach for rim of telescope with hand, [close fist and pull hand from up to down], [one more time], [one more time] 08-01-59-29/07-04: # can I pull this thing ? (. . . ) ah ahaa # reach for rim of telescope, [close fist, pull hand from left to right circularly], [one more time] 08-02-10-00/15-16: ok [look at] look at this purple and see if there is anything to see [move hand toward the purple ball as to push at it] 08-02-15-16/18-28: # no (...) [back] move hand toward lens of telescope, [close fist to grab at rim of telescope and pull hand back toward the body], [move open hand again back toward the body] 08-02-19-04/26-25: ok # [turn it] again # reach for rim of telescope, [close fist and pull hand from left to right circularly as to rotate rim of telescope], [one more time], [one more time]
The transcript includes both speech to the system, as well as self-talk, but not requests for hints asked of the research assistant. Transcription took approximately two person-months.
3.1
Coding
The following categories were coded: For events that required explicit interaction with the system beyond causing the scene to rotate around, subjects were coded as using gesture-only, speech-only, or multimodal interaction. Numerous subcategories were coded, but this chapter only reports on the subset of gesture and multimodal interaction for which the user employed gesture “manipulatively,” when interacting with an object. Interrater reliability for secondscoring of 18% of the multimodal data was 98%. For ONLY GESTURE: Consistent manipulative gesture: gesture used with the objective of changing the state of an object in the game — e.g., turning a wheel or
On the Relationships among Speech, Gestures, and Object Manipulation
103
pressing a button. Such gestures are consistent if the movement matches the way the object operates. Manipulative gesture, NOT consistent: see above but the movement does not match the way the object operates. For ONLY SPEECH: Speech manipulative: involving any change of state of an object in the scene — e.g., standing in front of the wheel and saying “turn the wheel”, or “press the button” when in the elevator. For SPEECH AND GESTURE TOGETHER Speech manipulative + consistent manipulative gesture: e.g. saying “turn the wheel” AND mimicking the gesture of turning a wheel. Speech manipulative + gesture NOT manipulative: like above but the gesture does not match the way the object functions.
4.
Results
Of the 3956 “interactive events,” we totalled the use of gesture-only, speechonly, or multimodal interaction (see Figure 5.3). Subjects were classified as “habitual users” of a mode of communication if they employed that mode during at least 60% of the available times. As can be seen, 90% of the subjects were habitual users of speech (including multimodal interaction), and 60% of the subjects were habitual users of multimodal interaction. Subjects were classified as “habitual users” of the consistent manipulation of objects strategy if they gesturally manipulated the object at least 10 times in a given communication mode (using gesture-only or multimodally) and 60% of those times, their body motion was consistent with the affordances of that object (i.e., the ways humans would normally employ it). Figure 5.4 provides Myst III images that show some of those objects, whose affordances the reader can readily determine. We examined the two cases separately: use of gesture-only and use of gesture within a multimodal event. Six subjects used gesture within their multimodal interaction manipulatively, rather than deictically, in order to change the state of an object. Of those six, five gestured in a fashion consistent with the object’s affordances, indicating that subjects using multimodal interaction manipulated digital “artefacts” according to the actions afforded by their design (Wilcoxon, p<0.03, Z= -1.89, one-tailed). The one subject who did not do so used a speak-and-point strategy (“turn the wheel” <point>). As for gesture alone, four subjects changed the state of an object with manipulative gesture used in a fashion consistent with the object affordances,
104
Advances in Natural Multimodal Dialogue Systems
Figure 5.3.
Figure 5.4.
Subjects’ use of modalities.
Other objects that can be manipulated in Myst III.
On the Relationships among Speech, Gestures, and Object Manipulation
105
whereas no subjects using manipulative gestures inconsistent with the object’s affordances (Wilcoxon, p=0.023, Z= -2.000 one-tailed).
4.1
Results from Questionnaire
The subjects filled out a post-test questionnaire (see Appendix) indicating, on scales of 1(very low) -10 (very high), their: 1. 2. 3. 4. 5.
Experience with adventure games Immersive experience Involvement in the game Ranking of the interface System response latency
(x = 2.3, σ (x = 4.2, σ (x = 4.1, σ (x = 6.1, σ (x = 5.7, σ
= 1.8) = 2.5) = 2.6) = 1.7) = 2.7)
In other words, the game itself was not particularly exciting, and subjects were only moderately involved in it. However, the speech/gesture interface was slightly better than middling ratings, while the latency inherent in a Wizard-ofOz system did not make the interaction annoyingly slow.
4.2
Comments from Subjects
Subjects were asked, “Do you think there was a ‘preferred’ channel between speech or gesture for the interaction? Or did you feel both channels were equally effective? Please explain”. In response, we received the following comments, which often did not match the subjects’ performance: “I feel that speech was easier for commands, since it is more economical in energy than gesture, and more precise. I grew tired of trying to motion precisely with my arms, and the time it took to turn. I would prefer to use speech over gesture, at least with the way gesture seemed to be implemented here.” Used speech-only 49%, gesture-only 45% of the time. “I am confident that anything I was doing, I could do with either channel. Speech seemed more natural for some situations, and gestures for others, though.” A habitual user of speech-only interaction, using gesture 20% of the time, speechonly 74%. “Preferred (channel) seemed to be speech. The movement felt like I was doing exercises.” A habitual user of speech (100%). “I found voice commands worked better for me than the hand gestures.” A habitual user of multimodal interaction (62%), with no gesture-only interaction. “I think the preferred channel is a gesture, with speech used to make clarifications.” A habitual user of multimodal interaction (83%). “In the first few minutes, I realized that gesturing worked best for me. I always talk to my computer or my car while operating them, so that was more unconscious....I find it very involving to talk to the game...”. A habitual user of multimodal interaction (75%).
106
Advances in Natural Multimodal Dialogue Systems “I was under the distinct impression that gesture did nothing and the system was a speech-recognition one.” A habitual user of multimodal interaction (69%). “I liked hands but used speech when confusing.” A habitual user of multimodal interaction (63%).
5.
Discussion
Results show that if given the opportunity, most subjects (60%) would use multimodal interaction more than 60% of the time for interacting with the game; an additional 30% would use speech-only interaction 60% of the time or more, but only 10% would even use gesture-only half the time. Overall, subjects were found to employ gesture alone 14% of the time, speech alone 41% of the time, and used speech and gesture for 45% of their interactions. These latter results are somewhat skewed by subjects who employed only speech, since in order to navigate in the scene, they issued many more navigation commands (e.g., “go left”) than would be necessary if multimodal interaction were employed. Subjects’ opinions about which modalities were important were often belied by their actions. Some thought that gesture was the key modality, but used multimodal interaction habitually, while others thought speech was the essential modality, but also used multimodal interaction frequently. It would appear that having both available would suit just about all the subjects. When given the opportunity to use whatever gestures they wanted, most subjects who attempted to manipulate an artefact generally did so according to the affordances of those objects – subjects turned wheels, pulled down knife switches, pushed in doorbells, etc. Thus, a future virtual environment control program that detects that a manufactured object is in the scene should be able to predict how a person’s arm would be shaped and would move in order to manipulate that object properly. The gesture recognizer could then adapt to the scene itself, giving such gestures higher weight. As novices to these kinds of games, the subjects found Myst III to be a modestly immersive experience with a moderate degree of involvement. Unlike true 3D games, there is no avatar in the scene that represents the user, and thus one would expect a lesser degree of immersion than a full 3D virtual reality environment. Also, given that the subjects were not supplied with explicit objectives in playing the game, it is not surprising that their degree of involvement with the game was moderate. A number of subjects very much liked the speech-gesture interface, while most gave it mixed reviews.
6.
Related Work
Most of the existing research on gestures has been performed by cognitive scientists who are interested in how people gesture, and the reasons why people
On the Relationships among Speech, Gestures, and Object Manipulation
107
gesture [McNeill, 1993; Krauss, 1998; Goldin-Meadow, 1999; Kendon, 2000]. Various taxonomies of gesture have been offered like e.g., McNeill’s description of iconic, deictic, emblematic, and beat gestures [McNeill, 1993], which usefully inform scientists who build gesture-based systems and avatars [Cassell and Stone, 1999; Wilson et al., 1997]. Specific claims can also be useful to technologists. McNeill [1993] argues that speech and gesture derive from an internal knowledge representation (termed a “catchment”) that encodes both semantic and visual information. Our results tend to confirm this claim, in that the visual representation depicts the object’s affordances, which then determines the manipulative gesture. The present corpus could also be used to confirm Kendon’s [2000] claim that the stroke phase of gestures tends to cooccur with phonologically prominent words. Quek et al. [2001] have provided a case-study of cross-modal cues for discourse segmentation during natural conversation by observing a subject describing her living space to an interlocutor. A comparative analysis of both video and audio data was performed to extract the segmentation cues while expert transcribed the gestures. Then both the gestural and spoken data was correlated for 32 seconds of video. A strong correlation between handedness and high-level semantic content of the discourse was found, and baseline data on the kinds of gestures used was provided. Whereas this style of observational research is needed as a foundation, it needs to be combined with quantitative observational and experimental work in order to be useful to building computer systems. Wilson et al. [1997] employed the McNeill theory to motivate their research to distinguish bi-phasic gestures (e.g., beats) from more meaningful tri-phasic gestures that have preparatory, stroke, and retraction phases. Early work by Hauptmann [1989] employed a Wizard of Oz paradigm and investigated the use of multimodal interaction for simple 3D tasks. It was found that people prefer to use combined speech and gesture interaction over either modality alone, and given the opportunity to do so in a factorially designed experiment, chose to use both 70% of the time (vs. 13% gesture only, and 16% speech only). Their factorial study has the advantage that all subjects were exposed to all ways of communicating, whereas our more ecologically realistic study allowed users to develop their own ways of interacting. On the other hand, it also allowed users to become functionally fixed into their first successful way of operating. Perhaps the most relevant work to ours is that of Sowa and Wachsmuth [2002] who employed a WOZ paradigm to collect subjects’ gestures as they attempt to describe a limited set of objects to a listener within a virtual construction domain. It was found that subjects’ hand shapes corresponded to features of the objects themselves. Based on this data, a prototype system was built that decomposes each gesture into spatial primitives and identifies
108
Advances in Natural Multimodal Dialogue Systems
their interrelationships. The object recognition engine then employs a graphmatching technique to compare the structure of the objects and that of the gesture. This latter work differs from the results presented here in that the game we studied included the task of manipulating the object rather than describing it. Thus, we find people attempt to manipulate artefacts in the ways they were built to be manipulated, whereas Sowa and Wachsmuth found people used their hands in describing objects to indicate their shape. Clearly, the subjects’ goals and intentions make a difference to the kinds of gestures they use, whether in a multimodal or unimodal context.
7.
Future Work
The next steps in this research are to analyze the syntax and semantics of the utterances in conjunction with the form and meaning of the gesture. To date, we have employed unification as the primary information fusion operation for multimodal integration [Cohen et al., 1997; Johnston et al., 1997]. We hypothesize that by using a feature-structured action representation that offers a “manner” attribute [Badler et al., 2000], whose value can itself be an action, and by representing the meanings of gestures as such actions, unification can again serve as the mechanism for information fusion across modalities. The analyses of utterance and gesture will be used to test this hypothesis. Given the identification of the same kind of gesture by the subjects to manipulate the same object, a gesture recognizer can then be trained [Corradini, 2002] and used in a multimodal architecture. We also will investigate to what extent mutual disambiguation of modalities [Oviatt, 1999] can be used to overcome recognition errors in a complete multimodal virtual environment system [Kaiser et al., 2003].
8.
Conclusions
This chapter has provided initial empirical results based on a Wizard of Oz paradigm that indicate people will prefer to use multimodal interaction in virtual environments than speech or gesture alone. Furthermore, when they attempt to manipulate virtual representations of artefacts, they do so according to the objects’ affordances – the ways the objects were designed to be manipulated. We therefore conclude that virtual environment systems should be capable of multimodal interaction that would employ subjects’ natural gestures, and that such systems should in fact employ information about the objects currently in the user’s view to predict the kinds of gestures that will ensue. In the future, information about the users’ gaze could be employed to restrict still further the kinds of gestures that might be employed.
On the Relationships among Speech, Gestures, and Object Manipulation
109
Acknowledgements This research has been supported by the Office of Naval Research, under Grants N00014-99-1-0377, N00014-99-1-0380 and N00014-02-1-0038. We are thankful to Rachel F. Coulston for help in running the study, and to our volunteer subjects.
110
Advances in Natural Multimodal Dialogue Systems
Appendix: Questionnaire MYST III - EXILE PRE-SESSION: 1 Have you ever played MYST III: Exile - the sequel to the MYST and RIVEN series? 2 Have you ever played adventure or puzzle-like games? 3 How would you rank your experience in playing the game on a scale from 1 to 10? (1 = no experience at all - 10 = very experienced player) 4 Would you say that your current state of fitness is: as usual, you are sick, or both? Please explain. POST-SESSION: 5 Did you like the game? Please explain. 6 Would you like to play again? Please explain. 7 What did you find difficult when playing? Please explain. 8 How would you rank the immersive effect of the game on a scale from 1 to 10? (1 = not immersive at all – 10 = amazingly immersive) 9 How would you rank your involvement in playing the game on a scale from 1 to 10? (1 = I played because I was requested to – 10 = I fully wanted to get to the end of the game)? 10 How would you rank the interface on a scale from 1 to 10? (1 = extremely bad, it never did what I wanted to – 10 = amazingly good) 11 How would you rank the latency response of the interface on a scale from 1 to 10? (1 = when I entered a command the interface reacted too slowly – 10 = I did not even realize there was a latency between my commands and the response of the interface) 12 Would you have preferred to play using a mouse/joystick/trackball/keyboard instead the way you played? Please explain. 13 Did you consciously decide when to use speech or gesture for playing? Please explain. 14 Did you have some hints concerning if and how to use gesture or speech or did you just do what it come naturally to you? Please explain. 15 Do you think there was a “preferred” channel between speech or gesture for the interaction? Or did you feel both channels were equally effective? Please explain. 16 Were the sensors attached to your body cumbersome? Did you feel restricted in your movements? Please explain. 17 Do you have any suggestions/criticism concerning the experiment?
On the Relationships among Speech, Gestures, and Object Manipulation
111
References Ascension Technology (2002). Flock of Birds. http://www.ascensiontech.com/. Badler, N., Bindinganavale, R., Allbeck, J., Schuler, W., Zao, L., and Palmer, M. (2000). Paramaterized Action Representation for Virtual Human Agents. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 256–286. MIT Press. Cassell, J. and Stone, M. (1999). Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems. In Proceedings of the AAAI 1999 Fall Symposium on Psychological Models of Communication in Collaborative Systems, pages 34–42. North Falmouth, MA: AAAI Press. Cohen, P. R., Johnston, M., McGee, D. R., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. (1997). QuickSet: Multimodal Interaction for Distributed Applications. In Proceedings of the Fifth Annual ACM International Multimedia Conference (Multimedia 1997), pages 31–40. Seattle, WA: ACM Press. Corradini, A. (2002). Real-Time Gesture Recognition by Means of Hybrid Recognizers. In Sowa, T. and Wachsmuth, I., editors, Gesture and Sign Language in Human-Computer Interaction, pages 34–46. Berlin: Springer Verlag. Fisher, S. S., McGreevy, M., Humphries, J., and Robinett, W. (1986). Virtual Environment Display System. In Proceedings of the ACM Workshop on Interactive 3D Graphics, pages 77–87. Chapel Hill, NC: ACM Press. Gibson, J. J. (1977). The Theory of Affordances. In Shaw, R. and Bransford, J., editors, Perceiving, Acting, and Knowing, pages 67–82. Hillsdale, NJ: Lawrence Erlbaum Associates. Goldin-Meadow, S. (1999). The Role of Gesture in Communication and Thinking. Trends in Cognitive Science, 3(11):419–429. Hauptmann, A. G. (1989). Speech and Gestures for Graphic Image Manipulation. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 241–245. New York: ACM Press. Hinckley, K., Pausch, R., Goble, J. C., and Kassell, N. F. (1994). Passive RealWorld Interface Props for Neurosurgical Visualization. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 452– 458. Johnston, M., Cohen, P. R., McGee, D., Oviatt, S., Pittman, J. A., and Smith, I. (1997). Unification-Based Multimodal Integration. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL), pages 281–288, Madrid, Spain. ACL Press. Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P. R., and Feiner, S. (2003). Mutual Disambiguation of 3D Multimodal In-
112
Advances in Natural Multimodal Dialogue Systems
teraction in Augmented and Virtual Reality. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI-PUI), pages 12–19, Vancouver, B.C., Canada. Kendon, A. (2000). Language and Gesture: Unity or Duality? In McNeill, D., editor, Language and Gesture, pages 47–63. Cambridge, UK: Cambridge University Press. Krauss, R. M. (1998). Why do we Gesture When we Speak? Current Directions in Psychological Science, 7:54–59. McNeill, D. (1993). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press. Norman, D. A. (1988). The Design of Everyday Things. New York, NY: Currency/Doubleday. Oviatt, S. L. (1999). Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 576–583. New York: ACM Press. Oviatt, S. L., Cohen, P. R., and Wang, M. Q. (1994). Toward Interface Design for Human Language Technology: Modality and Structure as Determinants of Linguistic Complexity. Speech Communication, 15(3-4):283–300. Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K. E., and Ansari, R. (2001). Gesture and Speech Multimodal Conversational Interaction. Technical report, Electrical Engineering and Computer Science Department, University of Illinois, Chicago. Sowa, T. and Wachsmuth, I. (2002). Interpretation of Shape-Related Iconic Gestures in Virtual Environments. In Sowa, T. and Wachsmuth, I., editors, Gesture and Sign Language in Human-Computer Interaction, pages 21–33. Berlin: Springer Verlag. Stoakley, R., Conway, M. J., and Pausch, R. (1995). Virtual Reality on a WIM: Interactive Worlds in Miniature. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 265–272. ACM Press. Taylor, R. M. (2001). VRPN: A Device-Independent, Network-Transparent VR Peripheral System. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pages 55–61. New York: ACM Press. Weimer, D. and Ganapathy, S. K. (1989). A Synthetic Visual Environment with Hand Gesturing and Voice Input. In Bice, K. and Lewis, C., editors, Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 235–240. New York: ACM Press. Wilson, A. D., Bobick, A. F., and Cassell, J. (1997). Temporal Classification of Natural Gesture and Application to Video Coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 948– 954. New York: IEEE Press.
Chapter 6 ANALYSING MULTIMODAL COMMUNICATION Repair-Based Measures of Human Communicative Coordination Patrick G. T. Healey, Marcus Colman and Mike Thirlwell Interaction, Media, and Communication Research Group Department of Computer Science Queen Mary, University of London, UK
{ph, marcus, miket}@dcs.qmul.ac.uk Abstract
There are few techniques available to inform the design of systems to support human-human interaction. Psycholinguistic models have the potential to fill this gap however existing approaches have some conceptual and practical limitations. This chapter presents a technique, based on the conversation analytic model of breakdown and repair, for modality and task independent analysis of communicative exchanges. The rationale for the approach is presented and a protocol for coding repair is described. The potential of this approach for analysing multimodal interactions is discussed.
Keywords:
Communication, conversation analysis, repair, evaluation.
1.
Introduction
Communication is central to many human activities, even for tasks and technologies that are ostensibly individual, see e.g., [Suchman, 1987; Heath and Luff, 2000; Hutchins, 1995; Nardi and Miller, 1991]. It underpins the co-ordination of routine activities and people’s responses to unexpected contingencies. Ethnomethodological studies of workplace interaction have documented how technologies not directly designed to support communication are nonetheless recruited in unanticipated ways to facilitate communicative coordination, e.g., [Heath and Luff, 2000; Hughes et al., 1992; Bowers et al., 1995]. Technologies such as text-based messaging and chat tools, integrated conferencing systems and large-scale collaborative virtual environments are 113 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 113–129. © 2005 Springer. Printed in the Netherlands.
114
Advances in Natural Multimodal Dialogue Systems
explicitly designed to support human communication. However, current approaches to the design and evaluation of these technologies focus on the analysis of the human-system interaction. There are few techniques that directly address the analysis of human-human interaction. Existing conceptual and empirical analyses of human communication should have much to contribute to design and evaluation. However, there are few established techniques either for identifying the communication related requirements of a given activity or for evaluating the impact different technologies have on the effectiveness of communication. Despite the elegance of the empirical observations, workplace studies have a problematic relationship to design, see e.g., [Hughes et al., 1992; Button and Dourish, 1996]. They require intensive effort and special analytical skills. They are also retrospective in character and do not provide a basis for systematic comparisons across different technologies or situations. The models and techniques developed in the psycholinguistic tradition are ostensibly more promising. They explicitly aim to quantify communicative phenomena and develop explanations that support comparative and predictive generalisations. The earliest applications of this approach used simple structural measures of interaction such as turn-taking, interruptions, backchannels, and gaze, see e.g., [O’Conaill et al., 1993; O’Malley et al., 1996]. Although these measures support direct comparison between some aspects of, for example, video-mediated interaction and face-to-face exchanges, the coding categories are coarse and do not take account of communicative function. For example, the category of ‘interruptions’ is sometimes formulated as the occurrence of overlapping speech where there has been no signal that a speaker is relinquishing the floor. This conflates accidental overlap, disruptive or competitive interventions and cooperative interventions such as collaborative completions. As a result the distribution of e.g., numbers of turns, lengths of turn, and interruptions are consistent with a number of possible interpretations, cf. [Anderson et al., 1997]. The finding that, for example, video-mediated communication leads participants to use more turns and words than they would when face-to-face are thus difficult to interpret. It might indicate either that video adds something to the interaction or that participants are working to compensate for some limitations it introduces. This chapter begins by reviewing the strengths and limitations of two psycholinguistic approaches to the analysis of human-human interaction: Dialogue Games analysis and the Collaborative Model of Dialogue. We argue that both approaches have important limitations as techniques for analysing multi-modal dialogue. An alternative approach is presented that focuses on breakdown and repair: the detection and resolution of communication problems. We propose that this can provide more robust, comparative measures of the effectiveness of human-human interaction across tasks and modalities. In
Analysing Multimodal Communication
115
order to turn this approach into a practical tool it is first necessary to develop a coding protocol that supports reliable identification of repair phenomena across different modalities. We present a protocol and test it against a corpus of attested examples of conversational repair. We then go on to sketch some of its potential applications to the analysis of multi-modal interactions.
1.1
Dialogue Games Analysis
Problems with the interpretation of simple structural analyses have been addressed to some extent by techniques which analyse communicative function directly. For example, Anderson et al. [1997] compared the performance of subjects carrying out a map drawing task under different media conditions. This task has a reliable dialogue coding system developed for it which characterises the functional structure of the dialogues, analysing each utterance as a move (e.g., instruct, explain, check, clarify, query-w, align) and structured sequences of moves as dialogue games [Kowtko et al., 1991]. Anderson et al. [1997] have shown how this approach can be sued to compare the dialogue moves used to carry out the task either face-to-face, using audio only or using video-mediated communication. Unlike the structural analyses, this provides a means of isolating the communicative functions that are preserved or impeded by different media. Although it supports systematic comparisons between media, functional analysis also has limitations. The coding system was designed to exhaustively classify utterance function in a particular collaborative task, the map task. As a result the move types are tailored to the transactional character of the task. Although it has been shown to generalise to some other information exchange tasks, it is unclear whether the coding scheme is adequate for qualitatively different kinds of exchange such as competitive negotiation. A practical limitation is that because the coding system is exhaustive, every utterance is analysed and coded. This is labour intensive and has the consequence that the sensitivity of the coding system is traded-off against its coverage, see [Carletta et al., 1996]. An additional concern is that the functional analysis doesn’t discriminate between incidental exchanges and those necessary for the activity at hand. All exchanges are coded and all contributions are given equal weight in the analysis.
1.2
The Collaborative Model of Dialogue
A second approach to modelling communication that also improves on simple structural analyses is the collaborative model of dialogue (CM) developed by Clark and co-workers [Clark, 1996; Clark and Wilkes-Gibbs, 1986; Clark and Schaefer, 1989]. This focuses on the process through which people build up their common ground during interaction. The basic principle is that the par-
116
Advances in Natural Multimodal Dialogue Systems
ties to an interaction only consider an utterance, or other communicative act, to be successful where some positive evidence of its acceptance or ‘grounding’ has been obtained. The grounding principle is modified according to the types of evidence and the degree of effort required to secure mutual-understanding in a given case. Two of the most important qualifications are that; a) interlocutors always seek to reduce the joint, as opposed to individual, effort necessary to successfully ground a communicative act and b) interlocutors will attempt to ground a contribution only up to a criterion level (the grounding criterion) which is adjusted according to circumstances. This framework has been used to characterise the properties of different communicative media and modalities. Clark and Brennan [1991] identify a set of 8 constraints on grounding that derive from the signal characteristics of different communicative media (Copresence, Visibility, Audibility, Cotemporality, Simultaneity, Sequentiality, Reviewability and Revisability). These constraints alter the ease with which evidence of grounding can be provided and change the costs associated with different grounding techniques (Formulation costs, Production costs, Reception costs, Understanding Costs, Start-up costs, Delay costs, Asynchrony costs, Speaker Change costs, Fault costs, Repair costs). During interaction, individuals must make a trade-off between the different types of action they can undertake in order to ground a particular contribution and their cost in a particular medium. For example, where turn taking costs are high individuals may invest more in the construction of each utterance and less in attempts at concurrent feedback. The CM has also been applied to the analysis of system feedback. The CM distinguishes a number of levels at which an action is considered complete. For example, an utterance may be perceived but not understood, or it may be understood but the action it proposes is not undertaken. These action levels are ordered according to the principle that feedback which indicates completion at a higher level presupposes completion at a lower level: if I comply with your request then this is evidence that I have also heard and understood it. Consequently, the current state of the common ground with respect to communicative action can vary depending on the degree of grounding that has been secured. The maintenance of context in interaction with a system can be supported by giving feedback which signals the level of grounding that a particular action has achieved [Brennan, 1988]. The level of feedback given can be modulated by the risks associated with possible misunderstandings. Although it generalises to a variety of tasks and communication modalities there are some important limitations to these applications of the CM model. One problem is that it provides only a limited analysis of situations in which contributions fail to secure acceptance. Where this occurs there is a general expectation that some repair will ensue, for example through reformulation of the contribution in a way that is acceptable to the addressee(s). A num-
Analysing Multimodal Communication
117
ber of possible types of reformulation e.g., alternative descriptions, instalment descriptions and trial references, are distinguished but the pattern of choice amongst these types, and their relationship to the success of repairs, is not addressed in the model. Although it is explicitly acknowledged that processes of conversational repair play a critical role in sustaining the mutual-intelligibility of interactions [Brennan, 1988; Clark, 1996], no specific mechanism has so far been developed for dealing with this. A more practical limitation on the application of the CM to the analysis of interaction is its relative underspecification. As Clark and Brennan [1991] note the media constraints and costs invoked to explain particular patterns of communication in different media are heuristic. The list of costs and constraints are neither exhaustive nor exclusive and there is no systematic means of quantifying them or calculating the possible trade-offs involved. In order to do this some quantification of the communicative effort invested in an interaction would be required. Without a means of comparing the grounding criteria being employed in different cases, i.e. of estimating the grounding criterion, the rationale behind particular patterns of communicative response cannot be determined. For example, a pattern of small, instalment, contributions may indicate a situation in which turn costs are low or a situation in which the grounding criterion is high.
2.
Breakdown and Repair
The position developed in this chapter is that the problems with functional and CM analyses of communication identified above can be addressed by focusing on the analysis of breakdown and repair in communication. The distinguishing feature of this approach is that it is concerned only with those parts of an interaction in which communicative trouble is encountered. The approaches described above depend on positive characterisations of communicative success; on analysing how things go right. The present approach, by contrast, focuses on analysing how things go wrong, and how people respond to this. A detailed framework for characterising these situations has been developed in the conversation analytic (CA) tradition [Schegloff et al., 1977; Schegloff, 1987; Schegloff, 1992]. Before describing the potential application of this framework to the analysis of mediated interactions, it is important to set out the basic CA repair framework and to clarify the concept of communicative problem it invokes. Two basic aspects to the CA model are of particular relevance to the present chapter; the structure of repairs and how specific they are. Structurally, the CA repair framework distinguishes between three things; who initiates a repair, where in the turn taking structure it occurs, and the subsequent trajectory of the repair to completion. For example, self-initiation occurs where the speaker
118
Advances in Natural Multimodal Dialogue Systems
identifies a problem with one of their own turns in a conversation. Otherinitiation occurs in situations in which someone signals a problem with another participant’s turn. The point at which the problem is addressed, the ‘repair’, is also classified in terms of ‘self’ and ‘other’. Self-repair occurs where the person who produced the problematic utterance also addresses the problem, regardless of whether they signalled that it was problematic. Other-repair occurs where someone addresses a putative problem with someone else’s turn. Four positions are distinguished in which a problem can be signalled or addressed. First position repair occurs in the turn in which a problem occurs, second position repair takes place in the next turn that occurs as a response to the problem turn. Third position repair-initiation occurs in the next turn that occurs as a response to the second position and so on [Schegloff, 1992; Schegloff, 1987]. Repair initiations are also distinguished according to the specificity with which they localise a problem. Schegloff et al. [1977] proposed ordering of other-initiation types according to their power. The most general kind of initiation is a “huh?” or “what?” which signals that the utterer has a problem but gives almost no clues about its precise character. This is followed, in order of increasing specificity, by a ‘wh’ question, such as “who?” or “what”. In this case the nature of the signalled problem is clearer and could potentially be associated with a particular sub-part of the problematic turn. More specific still are a partial repeat plus a ‘wh’ question or just a partial repeat. This type of reprise clarification provides information about the specific element(s) of the original turn that caused the problem. The strongest form of repair initiation in Schegloff et al.’s ordering is to offer a full paraphrase or reformulation prefaced by “you mean . . . ”. In this situation the recipient of the turn has successfully recognised and parsed what was said but wishes to test a possible interpretation of it with the original speaker. Schegloff et al. [1977] propose that there is a preference for using the strongest or most specific type of initiation available in any given case. This is supported by the observation that weaker initiations are interrupted by stronger initiations and, where several initiations occur in sequence, they increase in specificity.
2.1
A Repair-Based Analysis
The CA repair framework can be adapted to the comparative analysis of technologically mediated communication. As noted, this framework focuses only on those junctures where something goes wrong in an interaction. The CA approach to breakdown and repair does not involve appeal to a model of what is ‘actually’ being communicated or a theory of error. For example, Schegloff [1992] states that: “adequacy of understanding and intersubjectivity is assessed not against some general criterion of meaning or efficacy (such as convergent paraphrase) and not
Analysing Multimodal Communication
119
by ’external’ analysts, but by the parties themselves vis-à-vis the exigencies of the circumstances in which they find themselves” [Schegloff, 1992, page 1338].
The question of whether a breakdown in communication occurs because of some objectively verifiable misunderstanding is explicitly suspended [Garfinkel, 1967]. The focus instead is on analysing the situations which the participants treat as problematic, independently of whether there is any ‘real’ problem. The analysis is not concerned with whether a turn was, from an observer’s viewpoint, correct or accurate but only with whether the participants treat it as intelligible. This gives rise to some potentially counter-intuitive analyses. For example, a request for clarification that does not signal a problem with the intelligibility of a preceding utterance is not a repair in this sense. In addition a complaint about audibility when using a computer conferencing system does not count as a communication problem unless there is evidence that the complaint itself was not understood. This orientation has two potential practical benefits. Firstly, the analysis of communicative coordination is separated from analysis of the task domain. It is not necessary for the analyst to develop a theory of what task people are engaged with or how it is carried out. The patterns of repair type and trajectory can be analysed independently of the transaction involved. In contrast to some functional schemes of analysis such as dialogue games, the analysis should thus be applicable to a variety of different kinds of communicative interaction. The second potential benefit is that it promises to improve the sensitivity of the analysis by concentrating attention on those exchanges in which communicative problems occur. This claim trades on the assumption that, relative to other kinds of exchange, the frequency with which problems are signalled and addressed is moderated by their perceived importance to the coherence of the interaction. This assumption is based on the observation in the CA literature that turns which initiate repair are avoided if possible, especially those that signal problems with another participant’s contribution [Schegloff et al., 1977]. A focus on repair should thus help to filter out those exchanges which participants consider incidental to their purposes from those which they consider essential.
2.2
Adapting the Repair Model
In order to exploit the CA model in the comparative analysis of interaction it is necessary to develop a coding protocol that operationalises and specifies criteria for identifying the instances of repair phenomena. The CA repair model was developed in a tradition in which statistical generalisations are often specifically eschewed, see e.g., [Schegloff, 1992; Schegloff, 1993]. For example, Schegloff [1993] argues that the attempt to contrast categories of ‘sociability’ of participants by reference to “laughter per minute” and “backchannels
120
Advances in Natural Multimodal Dialogue Systems
per minute” is flawed [Schegloff, 1993, page 104]. One concern is that ‘per minute’ is not a valid unit of conversation. Laughter and other phenomena are only relevant at certain points or ‘environments’ in a conversation. Frequent laughter outside these conversational contexts is more typically treated as inappropriate than as a sign of sociability. Schegloff [1993] also highlights the difficulty in defining or operationalising conversational phenomena. For example, the form of a contribution such as “Yeah” does not in itself determine if it is used as a backchannel or to signal a change of speaker, cf. [Levinson, 1983]. These concerns are less problematic, we believe, for developing a systematised analysis of repair. Firstly, Schegloff notes that the environments of occurrence for repair initiations are unrestricted: “In fact, because nothing can be excluded in principle from the class ‘repairable’ [Schegloff et al., 1977, page 363], such repair initiation by the recipient of some talk appears to be the only type of turn in conversation with an unrestricted privilege of occurrence; it can in principle occur after any turn at talk. In that respect, then, its “environments of relevant possible occurrence” are well defined.” [Schegloff, 1993, page 115].
Prima facie this applies as much to the producer of the talk as it does to the recipient and therefore can be extended to include both ‘self’ and ‘other’ initiation. Secondly, the form and trajectory of repairs have been more extensively studied, and are better understood, than many other conversational phenomena. Consequently, this provides a promising domain for the definition of criteria that support the reliable identification of instances of repair, se [Schegloff, 1993, page 115].
2.3
The Repair Protocol
The Repair Protocol presented here draws on the procedures for signalling and addressing communicative problems described by CA and adapts them to the purposes of developing a tool that supports reliable identification of the occurrence of repair phenomena in communication under different conditions and media. The complete repair protocol is shown in Figure 6.1.1 It has been designed to be useable by people who have no specific knowledge of the body of conversation analytic research that it draws on. It is constructed so that an analyst takes each contribution to an interaction and tests it against a number of questions. The questions employ a yes/no format and are formulated to be as simple as possible. Because a given turn may contain more than one repair related event the protocol is applied iteratively to a turn until no further repairs are 1 Key:
P1= Position; P2 = Position 2; P3 = Position 3; SI = Self-initiated; OI = Other-initiated; SR = Self-repair; OI = Other-repair; NTRI = Next Turn Repair Initiator.
NO
P2, SI, OR, (Formulation)
YES
NO
P1, SI, SR (Formulation )
Is this revision completed by another participant?
YES
P1, SI, SR Transition Space
Does the revision occur before completion (or a possible completion) of the contribution?
YES
YES
NO
Repair is P3, SI, SR
NO
Figure 6.1.
The Repair Protocol.
Repeat until no new instances are detected
Request/Prompt is P2 NTRI
Repair is P3, OI, SR
YES
YES
End
P2 OI, OR
YES
NO
End
P2 NTRI
YES
P2, NTRI, (Incomplete)
NO
Does the other participant acknowledge or accept the proposal to repeat or revise their contribution?
NO
Does the initiator also provide a proposed revision?
YES
Is this contribution introduced to propose repetition or revision of another participant’s contribution?
Is this contribution introduced to edit, amend, or reprise a previous contribution by the initiator?
NO
Is this contribution introduced to accept or confirm another participant’s interpretation of one of the initiator’s previous contributions?
NO
Was this edit, amendment, or reprise requested or intentionally prompted by another participant?
P1, SI, SR, (Articulation)
NO
Is the edit, amendment, or reprise introduced to change the meaning of the contribution?
YES
Does the initiator edit, amend, or reprise part of their contribution before another participant responds to it?
Analysing Multimodal Communication
121
122
Advances in Natural Multimodal Dialogue Systems
detected. The status of a contribution is often determined only by the response it receives. As a result, the protocol also operates partly retrospectively. For example, some position two repair initiations are only classified as such after the response to them has been analysed. Although built on empirical studies of verbal interactions, the protocol is designed to be modality neutral. It aims to capture repair phenomena across a variety of modalities including, for example, graphical and gestural interaction. As a result it refers to initiators rather than speakers and, following Clark and Schaefer [1989] contributions, and modifications to contributions, rather than turns. More importantly, it has also been designed to avoid reliance on clearly identifiable sequences of turns. This is important for situations such as text and whiteboard based interaction where turn sequence is, in principle, more flexible than in verbal interaction. The protocol does require, however, that analysts can identify what, if any, preceding parts of the interaction a contribution may be addressed to. It is an assumption of this approach that, for example, gestural and graphical analogues of the verbal repair phenomena can be identified. There are several other departures from the existing literature on repair that should be noted. Firstly, there is no attempt to isolate 4th position repairs because they are too rare to be useful for making quantitative comparisons. Secondly the protocol does not currently distinguish between third turn and third position repairs [Schegloff, 1987]. This is an issue we hope to address in the future. Thirdly it does not capture embedded repairs where a correction or adjustment is made without any explicit signalling of a problem [Jefferson, 1982]. From a practical perspective this phenomenon is difficult to capture but it is also unclear whether it reflects breakdown of the kind that is of interest here.
2.3.1 Example application of the repair protocol. To illustrate the application of the protocol we take an established example of position two other initiation and other repair from the Conversation analytic literature (turns are labelled T1-T5). T1: Lori: But you know single beds are awfully thin to sleep on. T2: Sam: What? T3: Lori: Single beds. They’reT4: Ellen: You mean narrow? T5: Lori: They’re awfully narrow, yeah. (from Schegloff et al. [Schegloff et al., 1977, page 378]) Starting at the top of Figure 6.1 we apply the first test to T1. In this case there is no evidence of changes (edit/amend/reprise) before Sam responds so the answer is ‘NO’ and we proceed down the right hand branch of the proto-
Analysing Multimodal Communication
123
col. The next box asks whether T1 is introduced to accept or confirm another participants’ interpretation of one of the initiator’s previous contributions. The answer to this, and the next two questions, is ‘NO’ and we exit the first iteration through the protocol. We then move onto T2. The answer to the first question is again ’NO’ and, following the right branch of the protocol, the answer to the second question is ’NO’. The answer for the next question: "Is this contribution introduced to propose repetition or revision of another participant’s contribution?" is ‘YES’ since the ‘wh’ question signifies a request for some sort of clarification. The next test is then: "Does the initiator also provide a proposed revision?" Here the answer is ‘NO’ and, since Lori (initiator) does subsequently respond to this request then T2 is provisionally classified as a position 2 (P2) next turn repair initiator (NTRI). We then proceed to T3. Again, going through the protocol in sequence leads to the second question on the right hand branch, "Is this contribution designed to edit, amend or reprise a previous contribution by the initiator?" In this case, the phrase "single beds" is a reprise of part of the earlier contribution by this speaker, so the answer is ‘YES’. As this reprise was prompted by another speaker, the answer to the following box is also ‘YES’. This labels the contribution as P3 OI SR, and confirms that the prompt was P2 NTRI. Note that it is only after T3 has been analysed that the classification of T2 is confirmed. This illustrates the sense in which the protocol is partly ‘backward-looking’. T4 is produced by a new participant, Ellen. Working through the protocol we encounter the question: "Is this contribution introduced to propose repetition or revision of another participant’s contribution?" The answer is ‘YES’ but, unlike T2, T4 also offers a candidate reformulation of Lori’s T1. As a result the answer to the next question: "Does the initiator also provide a proposed revision?" is also ‘YES’. T4 is therefore categorised as P2 OI OR (Position 2, other-initiated, other-repair). Note that although T4 occurs several turns after T1 it is still classified as ‘position two’ because it is oriented to T1 and not to any of the intervening turns. T5 confirms the previous interpretation (Ellen’s) of Lori’s earlier contribution (T1) and this leads to an ‘END’ box on the right hand side of the repair protocol.
2.3.2 Validity and reliability. The reliability and validity of the protocol have been initially assessed by analysing a corpus of 65 example repair sequences containing 76 instances of repair initiations/repair drawn from Jefferson [1982], Schegloff [1982, 1987, 1992, 2000] and Schegloff et al. [1977]. The examples were chosen to cover as many phenomena as possible. They were extracted and stripped of the markers of their original coding. Two of the authors, who were not involved in preparing the corpus, then independently applied a draft version of the protocol to these examples. Because the examples
124
Advances in Natural Multimodal Dialogue Systems
Table 6.1. Validity By Category. Target Phenomenon
Number of Instances
Percent Identified
P1SI SR P1 TS R P1 Fail P2 R P2 RI P2 OI Fail P3 SI SR P3 OI SR
8 2 2 11 28 3 16 6
88% 75% 50% 87% 71% 67% 75% 67%
Overall
76
75%
Table 6.2. Reliability Broken Down By Category. Target Phenomenon
Instances Judge 1
Instances Judge 2
Percent Agreement
Position 1 Position 2 Position 3
83 64 39
75 66 45
63% 72% 62%
Self Other
122 64
120 66
63% 73%
Repair Initiation
145 41
153 33
63% 50%
drawn from the literature contain more repair phenomena than those they are used to illustrate, this provides a way to estimate both validity (by comparison with original analyses) and reliability by comparison between judges. Overall validity was good with on average 75% of the original examples being assigned to the same category as the original paper. Table 6.1 breaks down the results by category. Note that for some phenomena the number of instances is very low and confidence in the effectiveness of the protocol analysis for those categories is consequently reduced. Inter-judge reliability was calculated as percentage agreement in each of the categories: Position, Self-Other, Repair and Initiation. This was lower at 64% however it should be noted that this way of calculating reliability cannot account for levels of agreement on contributions that did not involve any repair (it is unclear how to count ‘unrepaired’ events). As a result the effective level of reliability is higher than these figures suggest. The scores for each category are given in Table 6.2. Examination of a confusion matrix showed that levels of
Analysing Multimodal Communication
125
agreement were reduced primarily by situations in which one judge identified something that another did not. Only in relatively few situations were the same repair events assigned to different categories. The primary source of confusion was between classifications of events as position 1 formulation problems and position 1 articulation problems. Although there is a large body of research which has applied the CA analyses to a variety of examples, to our knowledge this is the first time validity and reliability of repair categorisation has been assessed. These initial results suggest that this framework can be used to carry out systematic, replicable, analyses of corpora of interactions.
3.
Analysing Communicative Co-ordination
Repair per se is not necessarily an indicator of lower communicative coherence. It could, for example, reflect greater efforts to understand exactly what is being said or reflect a more complex exchange. Instead of making global comparisons of the frequency of repair, the present proposal is to use the structure and distribution of specific types of breakdown and repair to provide indices of communicative co-ordination. A systematic test of this application of the protocol is the subject of ongoing work. This section illustrates the potential of this approach to analyse multi-modal communication and suggests some of the potential measures of communicative co-ordination it could provide. The simplest index that this approach can provide is a measure of the difficulty of producing a contribution in a particular medium. In conversation a significant proportion of utterances show problems with articulation. Analogous problems arise in text chat where typos are frequently a problem, and in drawing where problems can arise in drawing complex shapes or fine detail. In the protocol problems of this kind are classified as Articulation problems and their frequency of occurrence provides a basic index of the difficulty of externalising a contribution in a particular medium. Because the protocol captures only those ‘typos’ or ‘disfluencies’ that the initiator of a contribution chooses to correct, it reflects the participant’s estimate of the impact of the articulation problem on the effectiveness of the interaction. It can also be used to index the grounding criteria (see above) that an individual employs. For example, if we hold task and media constant the frequency of Articulation repairs should be proportional to the effort participants are investing in making their contributions clear. A second index that the protocol can provide is a measure of the effect of a medium on the difficulty of formulating a contribution. In the protocol this is captured by the position 1, self-initiated, self-repairs (P1,SI,SR). These are repairs in which the initiator makes modifications that, unlike typos, alter the possible meaning of their contribution during production. For example a
126
Advances in Natural Multimodal Dialogue Systems
referring expression may be rephrased by replacing “he” with “she”, or part of a drawing may be erased and redrawn before being presented as complete. It might be expected that media which produce a persistent representation of a contribution, e.g., text chat or email, should, all things being equal, lead to more Formulation repairs than those that produce a transient representations, for example speech. Alternatively, if we hold the medium constant then the frequency of Formulation repairs should vary as a function of the cognitive load the task places on participants. Perhaps the most interesting potential measures are those which promise to directly index the communicative load associated with different media and tasks. One way in which this could be addressed is by assessing the frequency of, for example, position two and position three repair initiations under different task and media conditions. All things being equal, if a particular medium alters the intelligibility of interaction in some situation then this should be reflected in the frequency of repair initiations. More subtle distinctions can also be made. Arguably, self-initiated, self-repair in position three is indicative of high communicative co-ordination since it depends on sensitivity to a recipient’s interpretation of one of the initiator’s preceding utterances. Measures like this could provide for characterisation of the relative communicative ‘transparency’ of different media. A further interactional measure can be derived from the analysis of the specificity of the problems encountered. The ability of interlocutors to efficiently localise and deal with a problem provides an index of their communicative coherence. One possibility provided directly by the CA framework is to exploit the ranking of initiation types according to their power to locate a ‘repairable’ as discussed above. However, the notions of paraphrase and ‘wh’ question do not generalise to non-verbal interaction. If it is assumed that, all things being equal, more severe problems will require more extensive repairs then analysis can focus on the amount of the preceding material that is replaced. For verbal exchanges this could be indexed by the proportion of words altered or amended in the repair. For graphical exchanges it can be indexed by the proportion of a drawing or sketch that is revised.
4.
Discussion
This chapter has proposed that a repair-based analysis can provide useful operationalisations of several aspects of communicative co-ordination. We have shown that repairs can be systematically identified and described some ways in which these phenomena can be used to assess the relative effectiveness of human-human interaction. The discussion of specific measures and their application to multi-modal interaction is however speculative. The present claim is that a repair-based approach can provide a more effective analysis of com-
Analysing Multimodal Communication
127
municative coordination than existing applications of psycholinguistic techniques. One important gap in the current protocol is that it doesn’t directly capture problems with the handover of turns. Informal observations of remote multi-modal communication have suggested that this is an important class of communication problem, particularly where there are more than two participants. An interesting corollary of the present analysis is that communicative coherence should be enhanced by situations or technologies that make visible as much of the structure of each individual’s contribution as possible. This follows from the observation that repairs are initiated and effected by manipulating the structure of preceding contributions. To give a concrete example, in current implementations of text chat it is difficult to signal a problem with preceding turns. It is difficult both to identify the preceding turn itself and to identify what elements of the turn were problematic. Users usually have to repeat the problematic material together with some identifier in order to effect a repair initiation. Shared whiteboards, by contrast, support much simpler devices. For example, users can circle or underline problematic contributions directly. The original contribution is thus more easily edited on a whiteboard than in text chat. Media or environments that allow users to manipulate the structure of each others contributions should, on this view, provide more effective support for co-ordinating understanding.
Acknowledgements This work was partly supported by AvayaLabs, US under the “Mixed Mode Communication Project” in the Department of Computer Science at Queen Mary University of London. This chapter supersedes an earlier, unpublished, version presented at the 1999 AAAI fall symposium “Psychological Models of Communication in Collaborative Systems". We gratefully acknowledge Anjum Khan, Ioannis Spyradakis and Sylvia Wilbur for their contributions to the development of this work.
References Anderson, A. H., O’Malley, C., Doherty-Sneddon, G., Langton, S., Newlands, A., Mullin, J., Fleming, A. M., and van der Velden, J. (1997). The impact of VMC on Collaborative Problem Solving: An Analysis of Task Performance, Communicative Process and User Satisfaction. In Finn, K. E., Sellen, A., and Wilbur, S. B., editors, Video-Mediated Communication, pages 133–156. Mahwah, New Jersey: Lawrence Erlbaum Associates. Bowers, J., Button, G., and Sharrock, W. (1995). Workflow from Within and Without: Technology and Co-operative Work on the Print Industry Shop
128
Advances in Natural Multimodal Dialogue Systems
Floor. In Proceedings of the fourth European conference on ComputerSupported Co-operative Work, pages 51–66. New York: ACM Press. Brennan, S. (1988). The Grounding Problem in Conversations with and through Computers. In Fussell, S. R. and Kreuz, R. J., editors, Social and Cognitive Approaches to Interpersonal Communication, pages 201–225. Mahwah, New Jersey: Lawrence Erlbaum Associates. Button, D. and Dourish, P. (1996). Technomethodology: Paradoxes and Possibilities. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 19–26. New York: ACM Press. Carletta, J., Isard, A., Isard, S., Kowtko, J., Doherty-Sneddon, G., and Anderson, A. (1996). The Reliability of a Dialogue Structure Coding Scheme. Computational Linguistics, 23(1):13–31. Clark, H. H. (1996). Using Language. Cambridge: Cambridge University Press. Clark, H. H. and Brennan, S. (1991). Grounding in Communication. In Resnick, L., Levine, J., and Teasley, S., editors, Perspectives on Socially Shared Cognition, pages 127–150. American Psychological Association. Clark, H. H. and Schaefer, E. F. (1989). Contributing to Discourse. Cognitive Science, 13:259–294. Clark, H. H. and Wilkes-Gibbs, D. (1986). Referring as a Collaborative Process. Cognition, 22:1–39. Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliff: Prentice Hall. Heath, C. and Luff, P. (2000). Technology in Action. Cambridge University Press, Cambridge. Hughes, J., Randall, D., and Shapiro, D. (1992). Faltering from Ethnography to Design. In Proceedings of the Conference on Computer Supported Cooperative Work (CSCW), pages 115–122. Hutchins, E. (1995). How a Cockpit Remembers its Speeds. Cognitive Science, 19:265–288. Jefferson, G. (1982). On Exposed and Embedded Correction in Conversation. In Button, G. and Lee, J. R., editors, Talk and Social Organisation, pages 86–100. Clevedon: Multilingual Matters. Kowtko, J. C., Isard, S. D., and Doherty, G. M (1991). Conversational Games within Dialogue. In Caenepeel, M., Delin, J. L., Oversteegen, L., Redeker, G., and Sanders, J., editors, Proceedings of the DANDI Workshop on Discourse, HCRC, Edinburgh, UK. Levinson, S. C. (1983). Pragmatics. Cambridge: Cambridge University Press. Nardi, B. and Miller, J. (1991). Twinkling Lights and Nested Loops: Distributed Problem Solving and Spreadsheet Development. International Journal of Man-Machine Studies, 34:161–184.
Analysing Multimodal Communication
129
O’Conaill, B., Whittaker, S., and Wilbur, S. (1993). Conversations over Video Conferences: An Evaluation of the Spoken Aspects of Video-Mediated Communication. Human-Computer Interaction, 8:389–428. O’Malley, C., Langton, S., Anderson, A., Doherty-Sneddon, G., and Bruce, V. (1996). Comparison of Face-to-Face and Video-Mediated Interaction. Interacting with Computers, 8(2):177–192. Schegloff, E. A. (1982). Recycled Turn Beginnings: A Precise Repair Mechanism in Conversation’s Turn-Taking Organisation. In Button, G. and Lee, J. R., editors, Talk and Social Organisation, pages 70–85. Clevedon: Multilingual Matters. Schegloff, E. A. (1987). Some Sources of Misunderstanding in Talk-in-Interaction. Linguistics, 25:201–218. Schegloff, E. A. (1992). Repair after Next Turn: The Last Structurally Provided Defense of Intersubjectivity in Conversation. American Journal of Sociology, 97(5):1295–1345. Schegloff, E. A. (1993). Reflections on Quantification in the Study of Conversation. Research on Language and Social Interaction, 26(1):99–128. Schegloff, E. A. (2000). When ’Others’ Initiate Repair. Applied Linguistics, 21(2):205–243. Schegloff, E. A., Jefferson, G., and Sacks, H. (1977). The Preference for SelfCorrection in the Organization of Repair in Conversation. Language, 53(2): 361–382. Suchman, L. (1987). Plans and Situated Actions: The Problem of HumanMachine Communication. Cambridge: Cambridge University Press.
Chapter 7 DO ORAL MESSAGES HELP VISUAL SEARCH? Noëlle Carbonell and Suzanne Kieffer LORIA, 615 rue du Jardin Botanique 54600 Villers-lès-Nancy, France
{Noelle.Carbonell, Suzanne.Kieffer}@loria.fr Abstract
A preliminary experimental study is presented, that aims at eliciting the contribution of oral messages to facilitating visual search tasks on crowded displays. Results of quantitative and qualitative analyses suggest that appropriate verbal messages can improve both target selection time and accuracy. In particular, multimodal messages comprising a visual presentation of the isolated target together with absolute spatial oral information on its location in the displayed scene seem most effective. These messages also got top-ranking ratings from most subjects.
Keywords:
Multimodal interaction, multimedia presentations, visual search, spatial oral messages, speech and graphics, usability experimental study.
1.
Context and Motivation
1.1
Multimodality: State of the Art
Numerous forms of speech-based input multimodality have been proposed, implemented and tested. Combinations of speech with gestural modalities have been studied extensively, especially combinations of speech with modalities exploiting new input media, such as touch screens, pens, data gloves, haptic devices. Both usability and implementation issues have been considered; see, among others, [Oviatt et al., 1997]1 or [Robbe et al., 2000]2 for the first category of issues, [Nigay and Coutaz, 1993; Stock et al., 1997] for the second category.
1 On 2 On
speech and pen. speech and finger gestures on a touch screen.
131 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 131–157. © 2005 Springer. Printed in the Netherlands.
132
Advances in Natural Multimodal Dialogue Systems
Contrastingly, speech combined with text and graphics has only motivated a few studies. As an output modality, speech is mostly used either as a substitute for standard visual presentation modes (cf. phone services) or for supplementing deficiencies in visual exchange channels. Recent research efforts have been focusing on two main application domains: Providing blind or partially sighted users with easy computer access, e.g. [Grabowski and Barner, 1998; Yu et al., 2001], and Implementing appropriate interaction facilities in contexts of use where access to standard screen displays is difficult or even impossible. This is the case, for instance, in contexts where only small displays are available (e.g. PDAs and wearable computers), or in situations where the user’s gaze is involved in other activities (e.g. while driving a car); see, for instance, [Baber, 2001] concerning the first class of situations, and [de Vries and Johnson, 1997] concerning the second one. However, there is not yet, at least to our knowledge, a substantial amount of scientific work on the integration of speech into the output modalities of standard user interfaces, that is interfaces intended for standard categories of users using standard application software in standard environments and contexts of use; graphical user interfaces (or GUIs) implementing direct manipulation design principles [Shneiderman, 1983] are still prevailing. Published research on output forms of multimodality including speech amounts to usability studies of the role of speech comments in multimedia presentations, such as [Faraday and Sutcliffe, 1997], and contributions to the automatic generation of multimedia presentations, cf. [André and Rist, 1993; Maybury, 1993; Maybury, 2001]. This lack of interest for output forms of multimodality on the part of the research community may result, at least partly, from the fact that, although multimedia and multimodality refer to different concepts, these terms are often used as synonyms, especially when applied to system outputs. See [Maybury, 2001] for precise definitions of both terms.
1.2
Motivation and Objectives
The definition of the information content and semantics of multimedia presentations is commonly viewed as the responsibility of experts in the specific application domain considered. It is seldom viewed as lying within the scope of research on human-computer interaction. Therefore, the frequent assimilation of multimodality to multimedia may explain why the design of appropriate multimodal system responses has raised but little interest in the user interface research community, especially from an ergonomic angle, save for studies focused on specific categories of users or specific contexts of use. However, if standard users are offered speech facilities together with other input modalities,
Do Oral Messages Help Visual Search?
133
it is mandatory that the system responses are not limited to visual messages. Communication situations where one interlocutor can speak and the other cannot, are rather unusual. Research is then needed on the usability and software issues concerning the generation of appropriate multimodal system responses in standard human-computer interaction environments, and for standard user categories, including the general public. The main objective of the preliminary experimental study presented here is to contribute to scientific advances in this research area, by addressing one of the major usability issues relating to the generation of effective oral system messages, namely: How to design oral messages that facilitate the visual exploration of crowded displays? In particular, how to design messages that effectively help users to locate specific graphical objects on such displays? Perceptual and cognitive processes involved in visual search vary according to whether the goal of the search is familiar or not to the user. Familiarity here refers to the user’s visual familiarity with the graphical object they are looking for. Resorting to linguistic deictics and visual enhancements of graphical targets is a solution which seems “natural”. However, it is no more effective than the sole visual enhancement of the target. Another approach is to implement a human-like animated embodiment of the system, to visualise it on the screen and to endow it with a pointing device; see, for instance the PPP 3 persona which impersonates a car dealer, and uses a pointing stick for attracting the user’s attention on the assets of the currently displayed car [André, 1997]. However, the contribution of personae to the usability and effectiveness of human-computer interaction is still unclear, cf. [van Mulken et al., 1999]. Further testing is required in order to determine the usefulness of animated graphical system embodiments in graphical user interfaces. These reasons explain why we chose to focus first on assessing whether oral messages including spatial information actually facilitate the visual exploration of complex displays, especially the localization of graphical targets, in the context of standard human-computer interaction and environment. We selected visual search as the experimental task for the following reasons. It is one of the few human visual activities, besides reading, that have motivated a significant amount of psychological research; see, for instance, [Henderson and Hollingworth, 1998], [Findlay and Gilchrist, 1998] or [Chelazzi, 1999]. The 3 PPP
means ‘Personalized Plan Based Presenter’. See also Candace Sidner’s robot, a penguin which can point at locations on large horizontal displays with its beak [Sidner and Dzikovska, 2002].
134
Advances in Natural Multimodal Dialogue Systems
design of numerous computer applications may benefit from a better knowledge of this activity, especially applications for the general public such as: Online help to current interactive application software. For instance, novice users interacting with present graphical interfaces are often confused by the increasing number of toolbars and icons displayed concurrently; Map reading environments (cf. geographical applications), and navigation systems in vehicles; Data mining in visualisations of very large data sets; see, for instance, the complex hyperbolic graph visualisations proposed in [Lamping et al., 1995], or the treemap representations in [Fekete and Plaisant, 2002], and [Shneiderman, 1996; Card and Mackinlay, 1997; Card et al., 1999] for a general overview of visualisation techniques and their use. The methodology and experimental set-up of the experiment presented here are described in the next section, together with the underlying scientific hypotheses. Then, quantitative and qualitative results are presented and discussed in Section 3. Future research directions stemming from these results are described in the general conclusion, cf. Section 4.
2. 2.1
Methodology and Experimental Set-Up Overall Experimental Protocol
To assess the potential contribution of oral spatial information to facilitating visual search, we designed a preliminary experiment with: target presentation mode as independent variable; target search+selection time and target selection accuracy, as dependent variables. Eighteen subjects were to locate and select visual targets in thirty six complex displayed scenes, using the mouse. They were requested to carry out target localization and selection as fast as they could. Colour displays only were used. Each scene display was preceded by one out of three possible presentations of the corresponding target: Display of the isolated target at the centre of the screen during three seconds (visual presentation or VP); Oral characterization of the target plus spatial information on its position in the scene (oral presentation, OP);
135
Do Oral Messages Help Visual Search? Table 7.1. Overall task set-up. Group
VP
OP
MP
Group
VP
OP
MP
G1 G2 G3
P1 P3 P2
P2 P1 P3
P3 P2 P1
G4 G5 G6
P2 P1 P3
P1 P3 P2
P3 P2 P1
Gi : group of 3 subjects (3 × 6 = 18 subjects). Pi : set of 12 visual scenes (3 × 12 = 36 scenes).
Simultaneous display of the visual and oral presentations of the target (i.e. multimodal presentation, MP). These three sets of thirty six dual stimuli4 defined three experimental situations, namely the VP, OP and MP conditions. In the MP condition, the visual and oral presentations used in the VP and OP conditions respectively were presented simultaneously. Subjects were randomly split up into three groups, so that each subject processed twelve pairs of stimuli per condition, and each pair of stimuli was processed by six subjects. In order to neutralize possible task learning effects, the processing order was counterbalanced inside each group of subjects as follows: VP-PO-MP (three subjects), and OP-VP-MP (three subjects). All subjects performed the MP condition last. The size of groups in usability studies or cognitive ergonomics experimental studies seldom exceeds six subjects, most likely because analysing the behaviours of subjects performing realistic tasks in realistic environments is indeed a costly undertaking; for instance, Ahlberg, Williamson and Shneiderman [1992] report an experimental evaluation of three different user interfaces (meant for exploring information spaces) which also involved eighteen subjects split up into three groups of six subjects each, one group per user interface. The overall set-up is summed up in table 7.1. Experimental design choices were mainly motivated by the intent to assess the soundness of the three following working hypotheses which use the VP condition as the reference situation: A. Multimodal presentations of targets will reduce selection times and error rates in comparison with visual presentations. B. Oral presentations of targets will also improve accuracy, compared to visual presentations. is 108 pairs of stimuli (36 scenes × 3 presentation modes), each pair consisting in a presentation (visual, oral or multimodal) of a target followed by the display of the scene including this target.
4 That
136
Advances in Natural Multimodal Dialogue Systems
C. The type of spatial information included in oral target presentations will influence selection times and error rates. In particular, absolute and relative spatial indications will prove more effective than references to a priori knowledge (see subsection 2.3 for illustrations of these concepts), and absolute spatial information will be more effective than relative spatial indications. These hypotheses are based on common sense reasoning, in the absence, at least to our knowledge, of earlier published results and models stemming from experiments comparable to ours, that is involving similar tasks and interaction environments. Targets were unique graphical objects in the scenes including them. However, since they were presented out of context in the VP presentation mode, they might be easily confounded with other graphical objects in the scene, while unambiguous linguistic designations of targets would prevent detection errors due to such confusions from occurring. It follows then that accuracy would be higher in the OP and MP conditions than in the VP condition. As regards selection times, we assumed that subjects would use the spatial information included in oral messages, and that this information would enable them to focus visual search for the current target on a rather limited area in the scene. Scenes being complex and displayed on a rather large screen5 , we should then observe sensibly shorter selection times in the OP and MP conditions than in the VP condition. Concerning hypothesis C, absolute spatial information implies a one-step target localization process, while relative spatial information induces a twostep visual search; the latter is then less effective than the former as regards selection time, cognitive workload and accuracy. As for references to a priori knowledge, they involve more complex cognitive processes than the two previous types of spatial information; in addition, cultural knowledge varies greatly among users. In the remainder of the section, further information is given on: the criteria used for selecting visual scenes and targets (2.2); the structure and information content of oral messages (2.3); subjects’ profiles (2.4); the experimental set-up (2.5); the methodology adopted for analysing subjects’ results (2.6). 5 Scenes
included several scores of graphical objects and were displayed on a 21 inches screen.
Do Oral Messages Help Visual Search?
2.2
137
Scene Selection Criteria
Most visual scenes were taken from currently available Web pages in order to provide subjects with realistic attractive task environments. They were classified according to criteria stemming from Bernsen’s taxonomy of output modalities [Bernsen, 1994], our aim being to investigate the possible influence of the type of visual scenes displayed on target selection times and accuracy. Our classification was derived from the graphical categories in Bernsen’s taxonomy as follows. We focused on static graphical displays exclusively6 , on the ground that the localization and selection of moving targets in animated visual presentations is a much more complex activity than the selection of still targets in static visual presentations. Issues relating to the exploration of visual animated scenes will be addressed at a later stage in our research. We established two main classes of static presentations: Class 1 comprises displays of structured or unstructured collections of symbolic or arbitrary graphical objects, such as maps, flags, graphs, geometrical forms (cf. classes 9, 11, 21, 25 in Bernsen’s taxonomy); Class 2 includes displays of realistic objects or scenes, namely photographs or naturalistic drawings figuring complex real objects (e.g. monuments) or everyday life environments, such as views of rooms, town or country landscapes, . . . (cf. class 10 in Bernsen’s taxonomy). Half of the thirty six visual scenes belonged to class 1, and the other half to class 2. Class 1 and class 2 scenes in each of the three subsets described in subsection 2.1 (cf. table 7.1) were randomly ordered. Targets were objects or component parts of complex objects (cf. the complex real objects in class 2). They were chosen according to the following criteria. An acceptable target was a unique definite graphical object that could be designated unequivocally by a short simple verbal phrase. Although all targets were unique, some of them could be easily confused (visually) with other objects in the scene. In order to avoid task learning effects, target visual properties7 and position were varied from one scene to another.
2.3
Message Structure and Content
All messages included a noun phrase meant to designate the target unequivocally. For instance, “the pear” refers to the target unequivocally in the realistic scene reproduced in Figure 7.1. For any target, we chose the shortest simplest 6 Cf.
the five types of static graphical presentations in Bernsen’s taxonomy, namely classes 9, 10, 11, 21, 25. 7 These properties mainly include: colour, shape, orientation and size (within the limits of the fixed size target presentation box).
138
Advances in Natural Multimodal Dialogue Systems
Figure 7.1. pear.”
Basket with fruit (class 2 picture). Oral message: “On the left of the apple, the
noun phrase that could characterize it, with respect to other elements in the scene, without ambiguity and redundancy. For instance, one of the scenes represented geometrical figures and included several squares. The target we chose was the smallest square and the only pink one. We referred to one of these features, the colour, to appropriately reduce the scope of the substantive “square” in the oral message; “the pink square” is a noun phrase that refers to the target without ambiguity or redundancy. On the other hand, the use of the substantive “pear” is sufficient for referring unambiguously to the target in the scene reproduced in Figure 7.1. We experimented three types of spatial information in the verbal phrases referring to target locations, using an ad hoc classification inspired by the taxonomy presented in [Frank, 1998]: Absolute spatial information (ASI), such as “on the left/right” or “at the top/bottom”; Relative spatial information (RSI), for instance “on the left of the apple” (cf. Figure 7.1); Implicit spatial information (ISI), that is spatial information that can be easily inferred from common a priori knowledge and the visual context; for instance, it is easy to locate and identify the Mexican flag among twenty other national flags from the simple message “The Mexican flag.”, if the scene represents a planisphere, and national flags are placed inside the boundaries of the matching countries (cf. Figure 7.2).
Do Oral Messages Help Visual Search?
139
Figure 7.2. National flags (class 1). Oral message: “The Mexican flag.”
Messages included one or two spatial phrases, according to scene complexity; phrase pairs comprised one or two types of spatial information, namely ASI+RSI or ASI+ISI. Careful attention was paid to the choice of spatial prepositions [Burhans et al., 1995]. In order to make the assessment of hypothesis C possible, all messages had the same syntactical structure, so that information content was the only factor pertaining to the design of messages that could influence localization times and selection errors. The following structure, which emphasizes spatial information, was adopted for most messages, some ISI messages including no spatial phrase: [Spatial phrase] + Noun phrase (designation) This structure was preferred to the usual one (Noun phrase+Spatial phrase), based on the assumption that information in messages would be more effective if it was presented in the same order as it would be interpreted and used by the user.
2.4
Subjects’ Profiles
As this study involved a restricted number of subjects (18) and was a first attempt at validating hypotheses A, B and C, we defined strong constraints on subjects’ profiles in order to reduce inter-individual diversity, especially regarding task performance, and limit the number of factors that might influence
140
Advances in Natural Multimodal Dialogue Systems
subjects’ performances. To achieve homogeneity, we selected 18 undergraduate or graduate students in computer science with normal eyesight8 and ages ranging from 22 to 29. Thus, all participants were expert mouse users with alike quick motor reactions, familiar with visual search tasks, and capable of performing the chosen experimental tasks accurately and rapidly.
2.5
Experimental Set-Up
First, the experimenter presented the overall experimental set-up. Then, after a short training (6 target selections in the VP situation), each subject processed 12 scenes per situation, in the order VP+OP+MP or OP+VP+MP. Before each change of condition (i.e. before each change in target presentation), the experimenter explained the new specific set-up to the subject. For each visual scene: The target was first presented, during three seconds: – either visually in a fixed-size box in the centre of the screen; – or orally (together with a blank screen); – or orally and visually, simultaneously, that is resorting to synergetic multimodality [Coutaz and Caelen, 1991] redundantly [Coutaz et al., 1995]. Then, a button appeared in the centre of the screen together with a written message requesting the subject to click on the button for launching the display of the scene. Therefore, at the beginning of each target selection step, the mouse was positioned in the centre of the screen, making it possible to compare subjects’ selection times. The next target was presented as soon as the subject had clicked on any object in the current displayed scene. At the end of the session, subjects had to fill in a questionnaire requesting them to rate the difficulty of each task using a four degree scale (ranging from “very easy” to “very difficult”). The session ended up in a debriefing interview.
2.6
Analysis Methodology
Quantitative results comprise: average localization + selection times; and error (i.e. wrong target selections) counts or percentages. 8 Save
for one subject who was slightly colour-blind.
Do Oral Messages Help Visual Search?
141
computed over all subjects and scenes, as well as per class of scenes and per type of oral message. We tested the statistical significance of these quantitative results whenever a sufficient number of samples was available. Qualitative analyses of subjects’ performances, especially comparisons between the numbers of target selection errors in the VP, OP and MP conditions, provided useful information for defining further research directions. In order to elicit the possible factors at the origin of selection errors, scenes and targets were characterized using the following criteria: Scene characterization: – complexity (according to the number of displayed objects); and, – for class 1 scenes only, visual structure (e.g. random layout of objects; tree, crown or matrix structures; . . . ). Target characterization: – position on the screen (centre, left, . . . ); – visual saliency; – familiarity versus unfamiliarity (oddness); – ambiguity (i.e. the number of possible visual confusions with other graphical objects in the scene). Quantitative and qualitative results are presented and discussed in the next section.
3. 3.1
Results: Presentation and Discussion Quantitative Results
Results were computed over 34 scenes; two scenes (both in class 1) had to be excluded from quantitative and qualitative analyses by reason of technical incidents. Therefore, the corpus of experimental data we actually used comprised the results of 612 visual search tasks, that is 204 tasks per condition (VP, OP, or MP), each scene having been processed in each condition by 6 subjects. Statistical tests, mainly t-tests, were performed on these performance data (successes/failures and execution times), especially on the three sets of data collected in the three conditions.
142
Advances in Natural Multimodal Dialogue Systems
Table 7.2. Subjects’ performances per target presentation mode. Target presentation mode
Number of errors
Average selection time (sec.)
Standard deviation (sec.)
VP OP MP
31 14 8
2.83 3.92 2.70
1.70 3.50 1.93
Target presentation mode VP versus OP VP versus MP OP versus MP
Number of errors t=-2.70 t=-3.94 t=-1.31
p=0.007 p<0.0001 p=0.189
Average selection time (sec.) t=+3.79 t=-0.70 t=-4.20
p=0.0002 p=0.4852 p<0.0001
Upper half: best results are in bold type, and lowest ones are underscored. Lower half: significant statistical results are in bold type.
3.1.1
Global analysis.
Description. Concerning the order of conditions (VP then OP, versus OP then VP), comparisons between subjects’ performances in each of the two groups yielded no significant inter-group differences. These results indicate that no perceptible task learning effect occurred in the course of the experiment. The absence of any such effect is not surprising, due to the low number of tasks in each condition (i.e., 12), and the brief duration of the overall session (about 10 min.). Therefore, learning effects may be rightly excluded from the factors to be considered in the interpretation of the quantitative and qualitative results of our analyses, even as regards the MP condition although it was performed last by all subjects. As for selection accuracy, oral messages proved much more effective than visual target presentations, as shown by comparisons between the VP and OP conditions (cf. table 7.2). The total number of errors in the OP condition decreased by 55%. However, selection was slower in the absence of prior visualizations of isolated targets, that is, in comparison with the VP and MP conditions. Average selection time in the OP condition increased by over 38% compared to the VP condition. These differences are statistically significant. Table 7.2 also shows that multimodal presentations of targets reduced selection times compared to oral presentations, and error rates compared to visual presentations, both results being statistically significant. These results are in keeping with those observed for selection errors. To conclude, results presented in 7.2 confirm hypothesis B whereas hypothesis A is confirmed for selection error numbers only.
Do Oral Messages Help Visual Search?
143
Interpretation and discussion. As regards accuracy, spatial indications and target verbal designations included in oral messages may have reduced the frequency of visual confusions, the former by reducing the scope of visual search for the target, the latter by preventing confusions between the target and graphical objects of similar visual appearance in the scene. Significantly longer average selection time in the OP condition, together with a much higher standard deviation, may be explained by the fact that subjects were unfamiliar with the visual search tasks they had to carry out during the OP condition; this situation being rather unusual compared to the VP and MP situations which occur frequently in everyday life. Therefore, the higher variability of selection times in the OP condition may be assumed to reflect the high inter-individual diversity of cognitive abilities and learning processes. However, the longer selection times observed in the OP condition may be explained more satisfactorily by the intrinsic differences between the tasks subjects had to perform in the OP and VP conditions. Visual search of a visually known graphical target is a much less complex task than searching for a graphical object that matches a given verbal specification, even if only a part of the scene needs to be explored thanks to spatial verbal information. Bieger and Glock [1986] observed similar effects on the performances of subjects in an experiment that aimed at comparing the efficiency of text and graphics for presenting various types of information in instructional material. In particular, concerning spatial information9 , Bieger and Glock found that subjects who were given textual presentations of spatial information made fewer task execution errors than those who were given graphical presentations of the same information; on the other hand, the latter completed the prescribed tasks faster. Nevertheless, further comparisons between this study and ours would be meaningless, due to important design differences between the two experimental protocols: in Bieger’s and Glock’s study, the tasks subjects had to carry out were procedural assembly tasks (versus target selection in our experiment), and the modalities considered were text and graphics (versus speech and graphics). Concerning subjects’ performances in the MP condition, comparisons with their performances in the two other conditions suggest that, in the MP condition, they succeeded in making the most of the information provided by each modality, thus compensating for the weaknesses of each unimodal presentation mode. A likely interpretation is that they achieved: easy disambiguation of possible confusions between targets and other objects of similar appearance in the scene, thanks to spatial and denominative verbal information; and 9 Namely
the final position and orientation of elements in the workspace (assembly tasks).
144
Advances in Natural Multimodal Dialogue Systems
rapid identification of targets thanks to visual information, matching based on visual characteristics being faster than identification based on abstract properties or stereotyped mental representations, which involves complex decision-making processes. Surprisingly enough, spatial verbal information which, according to hypothesis A, should have accelerated target localization, did not actually contribute to reducing selection times significantly, or so it seems. This finding, together with the longer selection times observed in the OP condition may be explained within the framework of current visual perception models which assume that eye-movements are less influenced by cognitive (top-down) processes than by visual stimuli (bottom-up processes) during visual exploration tasks [Henderson and Hollingworth, 1998]. On the other hand, visual exploration in search of a graphical object matching a verbal specification is a task that combines complex, hence slow, cognitive processes with perceptual activities. These differences may explain why longer selection times were observed in the OP condition than in the VP and MP conditions, and why spatial oral information did not noticeably reduce selection times in the MP condition compared to the VP condition. This interpretation of the noticeable differences observed between average selection times in the three conditions, fits in with Rasmussen’s cognitive model [Rasmussen, 1986], provided that the search of a visually known target is assimilated to a skilled or automatic activity, and the search of a visually unknown target from a verbal specification of its characteristics as a problem-solving activity. The model then predicts that selection time of the known target will be shorter than that of the unknown target, based on the assumption that automatic or skilled responses to stimuli amount to the activation of precompiled schemata or the compilation and activation of pre-defined schemata respectively, while problem-solving involves more complex cognitive processes. The application of this model to the MP condition suggests the tentative conclusion that subjects used both sources of information optimally within an overall opportunistic strategy favouring visual search over more complex matching processes, such matching processes being activated only in cases when visual information was insufficient or ambiguous. However, although Rasmussen’s model constitutes an appropriate framework for predicting subjects’ performances, it is too general to provide any meaningful insight into the strategies underlying these performances and their implementation, that is, how multimodal stimuli are processed, how the results of modality specific processes are integrated and unified to produce meaningful coherent interpretations and appropriate actions or motor responses. Subjects’ performances are indeed compatible with cognitive models of multimodal input processing that postulate the existence of high level interactions between unimodal perceptual and interpretative processes rather than
Do Oral Messages Help Visual Search?
145
early low-level interactions; see, for instance, the model proposed in [Engelkamp, 1992]. However, further research is needed in order to determine which type of interaction (competition, synergism or complementarity) would best account for subjects’ results within the framework of our experimental protocol. For instance, reducing progressively the duration of visual target presentations in the MP condition appears as a promising, but difficult to implement, experimental paradigm for increasing our knowledge of the cognitive processes involved in the processing of multimodal inputs [Massaro, 2002]. Future work. To achieve significant advances in the investigation of multimodal perception and interpretation processes, experimental approaches have to overcome major difficulties. In particular, homogeneous sets of visual search tasks are needed in order to make it possible to achieve meaningful reliable intra- and inter-subject comparisons, as the same scene cannot be processed in the three conditions by the same subject. The “difficulty” of the visual search tasks proposed to our subjects varied greatly from one scene to another. For instance, in the VP condition, all subjects (6) failed to select the correct target for one scene, and only 8 scene+target pairs occasioned 26 out of the 31 errors observed in this condition. We are currently refining our visual and semiotic characterizations of scenes and targets in order to be capable of defining and generating sets of really equivalent scenes. The number of participants and scenes should also be increased considerably, so that a greater number of more sophisticated inter-related hypotheses can be tested simultaneously, and their soundness evaluated accurately, thanks to the possible application of appropriate statistical techniques. Ergonomic recommendations. To conclude, the quantitative results presented in this section contribute to validating hypotheses A and B. However, further research is needed to determine whether the significantly longer selection times observed in condition OP are mainly due either to the differences between the tasks subjects performed in the OP condition and those they carried out in the VP and MP conditions, or to the fact that subjects were unfamiliar with the tasks they carried out in the OP condition. These results also suggest useful recommendations for improving user interface design. In order to facilitate visual search tasks on crowded displays without resorting to standard visual enhancement techniques, two novel forms of user support may provide useful alternatives: a. If accuracy only is sought for, an oral message comprising an unambiguous verbal designation of the graphical target and spatial information on its location in the display will prove to be sufficient. b. If both accuracy and rapidity are sought for, a multimodal message will be more appropriate, that is a message comprising a context-free visual
146
Advances in Natural Multimodal Dialogue Systems
presentation of the target together with an oral message including the same information as mentioned in recommendation a. However, further experimental research is needed to confirm these recommendations beyond doubt, in-as-much as they have been inferred from a relatively small sample of experimental data and measurements. In addition, oral and multimodal messages should be compared, in terms of effectiveness and comfort, with other forms of user support, such as target visual enhancement through colour, animation, zooming, etc. Until a sufficient amount of experimental data has been collected, recommendations a. and b. should be considered as tentative. Analyses of results per class of scenes and type of messages, i.e., type of spatial information are presented next. These analyses make it possible to refine and enrich our initial working hypotheses.
3.1.2
Detailed analysis.
Results per class of scenes. Subjects’ results, grouped per scene class and target presentation mode, are presented in table 7.3. Error percentages were computed over: 96 samples per condition for class 1 (due to the exclusion of two class 1 scenes, as explained at the beginning of section 3); and 108 samples per condition for class 2. No statistical analysis was performed on these results by reason of the rather small number of samples in each set. Multimodal messages proved to be most effective, in comparison with visual and oral presentations, especially for scenes representing symbolic or arbitrary objects. For scenes in class 1, comparisons between the three conditions indicate that errors were reduced in the MP condition by 12.6% (compared to the VP condition) and 5.3% (compared to the OP condition), while average selection times decreased by 0.24 and 1.27 seconds respectively. In short, concerning class 1 scenes, 86% of the selection errors observed in the VP condition did not occur in the MP condition, and selection times were one third longer in the OP condition than in the MP condition. As for realistic scenes, the average selection time in the MP condition is similar to the VP one (2.40 sec. versus 2.43 sec.), and markedly inferior to the OP one (3.58 sec.), whereas the number of selection errors is similar to the OP one, and inferior to the VP one (by 10.2%). These results confirm the main assumption proposed in the previous subsection for interpreting global results, namely that, in the MP condition, subjects took advantage of both the visual and oral information available, especially for
147
Do Oral Messages Help Visual Search? Table 7.3. Results per target presentation mode and class of scenes. Target presentation mode
Error percentage
Average selection time (sec.)
Standard deviation (sec.)
VP-C1 VP-C2 OP-C1 OP-C2 MP-C1 MP-C2
14.6 15.7 7.3 6.5 2 5.5
3.27 2.43 4.30 3.58 3.03 2.40
1.94 1.39 4.09 2.87 2.36 1.36
Percentages were computed over the total number of samples (target selection tasks) per condition.
processing class 1 scenes. For instance, we observed that some subjects lacked the a priori knowledge required for taking advantage of the implicit information conveyed by ISI messages; visual information can prove to be most helpful for achieving successful target identification in such cases. On the other hand, class 1 scenes which consisted in collections of symbolic or arbitrary graphical objects (e.g. flags or geometric figures) favoured visual confusions between targets and similar objects in the displayed collections; verbal designations and spatial information undoubtedly helped subjects to solve possible visual “ambiguities”. The fact that average selection times were consistently longer for class 1 scenes than for class 2 scenes can be explained as follows. If the target is a familiar object (such as a pan) in a familiar realistic scene (a kitchen, for instance), visual exploration of the scene is facilitated by a priori knowledge of the standard structure of such environments and the likely locations of the target object therein. Such knowledge is not available in the case of unrealistic class 1 scenes; the structure of such a scene and the possible locations of the target in it cannot be foreseen using a priori knowledge, so that a more careful search, or even an exhaustive exploration, of the scene is necessary for succeeding in locating the target. This assumption may also explain why multimodal target presentations proved to be most effective for scenes belonging to class 1: both oral and visual information contributed to compensate for the lack of pragmatic a priori knowledge.
Five categories of verbal messages were Results per type of messages. experimented (cf. Subsection 2.3). Messages were classified according to the type of spatial information they comprised: absolute (ASI), relative (RSI), implicit (ISI), plus absolute-relative (ASI+RSI) and absolute-implicit (ASI+ISI).
148
Advances in Natural Multimodal Dialogue Systems
Subjects’ results, grouped according to these categories, are presented in table 7.4. Subjects’ performances in the VP condition are also reported, although messages were excluded from target presentations in this condition; they serve as references in the assessment of the effectiveness and efficiency of the user support provided by oral messages in the two other conditions (OP and MP). Percentages were computed, for each condition, over 48 samples (ASI), 72 samples (RSI), 24 samples (ISI), 36 samples (ASI+RSI), and 12 samples (ASI+ISI)10 . For each category of messages, comparisons between results achieved by subjects in the VP, OP and MP situations suggest that absolute and/or relative spatial information improved selection accuracy markedly (cf. the ASI, RSI and ASI+RSI types of messages). However, the usefulness of ISI messages seems questionable, at least in the OP condition. Their effectiveness in the MP condition denotes the complexity of the interpretation processes at work in the interpretation of multimodal stimuli. Average selection times for scenes associated with RSI and ASI messages were much longer in the OP condition (4.03 and 6.12 respectively) than in the other conditions (i.e. 2.86 and 1.84 for the VP condition, 3.12 and 2.1 for the MP condition). For RSI messages, this effect may be due to the complexity of the visual search strategy induced by relative spatial information when the target is unknown visually. In such cases, the search strategy is likely to include two successive steps: first, localization of the reference object, then exploration of its vicinity in search of the target [Gramopadhye and Madhani, 2001]. Each step being longer than the single step search for a visually known target, the resulting global selection time is necessarily much longer than in the other conditions where the target is visually known. This interpretation may also explain why RSI messages did not affect selection times in the MP condition noticeably. The target being in the vicinity of the reference object and having been viewed previously, it can be recognized through peripheral vision, so that one eye fixation only is required for locating both the reference object and the target, cf. [van Diepen et al., 1998]. However, it is also possible that, in the MP condition, subjects tended to adopt a simpler search strategy based exclusively on the available visual information (hence comprising a single visual search step) whenever the oral message induced a complex slow selection strategy. This second interpretation has the advantage of explaining why both RSI and ISI messages exerted no perceptible influence on selection times in the MP condition.
is 192 samples instead of 6 × 34 = 204 samples (cf. the two scenes that were excluded). Two additional scenes (6 x 2 samples) were not taken into account because the corresponding messages did not include any spatial information, the visual saliency of the target making such information superfluous.
10 That
149
Do Oral Messages Help Visual Search? Table 7.4. Results per target presentation mode and type of verbal message. VP
Condition
(reference)
Scenes grouped per message type
Percentage of errors
Average selection time (sec.)
Standard deviation (sec.)
ASI RSI ISI ASI+RSI ASI+ISI
10.4 26.4 4.17 13.89 8.33
2.87 2.86 1.84 3.57 3.54
1.19 2.14 0.57 1.99 0.98
2.91 4.03 6.12 3.82 5.19
3.41 5.94 3.78 3.78 3.37
2.42 3.12 2.1 2.82 2.98
1.41 2.43 1.06 1.84 2.53
OP ASI RSI ISI ASI+RSI ASI+ISI
0 8.33 16.67 5.56 16.67 MP
ASI RSI ISI ASI+RSI ASI+ISI
Condition
4.17 8.33 0 0 0
Condition
Best results in the OP condition are in bold type, and lowest ones are underscored. Percentages were computed over the total number of samples per message type. (Results in the VP condition are reported, VP being used here as the reference condition).
As for ISI messages, their exploitation involves complex cognitive processes which may have slowed down target selection in the OP condition. The poor results observed in the OP condition might also suggest that the usefulness of ISI messages is intrinsically limited. To sum up, whereas the inclusion of any category of verbal spatial information in multimodal target presentations seems worthwhile, absolute spatial information should be preferred over other information types in the design of spatial information messages, in order to effectively facilitate visual target localization and improve both selection times and accuracy. However, these observations, which refine hypothesis C (cf. Subsection 2.1), should be viewed as working hypotheses rather than reliable, or even tentative, conclusions. Their appropriateness has to be assessed through careful systematic experimentation on a large scale. Analysis of eye-movements (by means of an eye-tracker) would provide invaluable information on visual search strategies, especially
150
Advances in Natural Multimodal Dialogue Systems
on the exact influence of the information content of oral messages on these strategies.
3.2
Qualitative Analyses: Selection Errors
Qualitative analyses were focused on the subjects’ errors exclusively, with a view to: getting a better understanding of the contribution of verbal messages to assisting users in visual search tasks; and obtaining useful knowledge for improving message design. Analyses use the detailed characterizations of scenes and messages listed in Subsection 2.6, as well as the subjects’ subjective ratings of the difficulty of the prescribed visual search tasks (cf. the post-session questionnaires mentioned in Subsection 2.5). Scenes were filtered so that, in each condition, only the scenes which had occasioned more than one error were considered, on the basis of the following assumption: for a given scene in a given condition, the reasons for the failure of one single subject are more likely to originate from the subject’s capabilities than from the scene characteristics or the message information content.
3.2.1 Visual condition. The main plausible factors at the origin of selection errors observed in the VP condition are presented next. Percentages represent: the number of errors which a given characteristic of the scene may explain, by itself or in conjunction with other factors; computed over the total number of filtered errors (i.e. 26). Factors are listed in decreasing order of the percentages of errors they contribute to account for: Concerning targets: lack of salience (85%), eccentric position in the scene (69%), possible confusions with other objects (69%), unfamiliarity (50%); Concerning scenes: crowded (69%), unstructured (46%), figuring geometric forms (42%). This analysis of subjects’ errors in the VP condition will be used as a reference in the next subsection which is focused on errors in the OP and MP conditions.
Do Oral Messages Help Visual Search?
151
3.2.2 Oral and multimodal conditions. Five scenes in the OP condition and only two scenes in the MP condition occasioned more than one error, against eight in the VP condition. In addition, 24 errors in the VP condition were “corrected” in the OP condition, so that seven out of the eight scenes occasioning more than one error in the VP condition yielded error-free results in the OP condition. These comparisons bring out the usefulness of oral messages for improving target selection accuracy. However, four scenes yielding error-free results in the VP and MP conditions occasioned ten out of the twelve11 filtered errors observed in the OP condition12 . Therefore, it is likely that the main factor at the origin of these errors is the poor quality of the information content of the oral messages paired off with these scenes. The analysis of these four messages, together with the information provided by questionnaires and debriefings, support this conclusion. Four errors were motivated by an ISI message which referred to knowledge unfamiliar to the majority of subjects. A too complex ASI+ISI message (structure and length) referring to knowledge some subjects were unfamiliar with may account for two other errors. As for the two remaining pairs of errors, they may be reliably ascribed to the use, in both verbal target designations, of technical substantives the exact meanings of which were unfamiliar to some subjects. The fact that none of these errors occurred in the MP condition, together with the fact that two scenes only occasioned the four13 filtered errors observed in this condition, illustrates the advantages of combining visual and verbal information in target presentations. In the MP condition, two errors occurred on a “difficult” scene which occasioned six errors in the VP condition (crowded scene, and non salient unfamiliar target easy to confound with other objects), and two errors in the OP condition (use of technical vocabulary). The other two errors were occasioned by a scene which was processed successfully by all subjects in the OP condition, but occasioned three errors in the VP condition. This may hint that the processing of multimodal incoming information is controlled or influenced by visual perception strategies rather than by high level cognitive processes. In short, the qualitative analysis of errors confirms the usefulness of oral messages for improving the accuracy of visual target identification, provided that: messages are short, their syntactical structure straightforward, the vocabulary familiar to users; and above all, their information content is appropriate. 11 The
total number of errors in the OP condition amounted to 14. patterns for these images were as follows: 4, 2, 2, 2. 13 Out of 8. 12 Error
152
Advances in Natural Multimodal Dialogue Systems
Table 7.5. Subjects’ judgments on task difficulty and preferences. Condition
“Very easy”
“Easy”
“Difficult”
“Very difficult”
VP OP MP
22% 22% 72%
28% 61% 17%
39% 17% 11%
11% 0% 0%
Top-ranking preference
VP
OP
MP
17%
17%
66%
Percentages were computed over the total number of subjects. Highest values in each line are in bold type.
3.2.3 Subjects’ subjective judgements. Subjects expressed positive judgments on the contribution of oral messages to facilitating visual search tasks in the post-experiment questionnaires. The MP condition achieved the highest rate of subjective satisfaction as shown in table 7.5. The majority of subjects (72%) rated the execution of visual search tasks in the MP condition as “very easy”, whereas a minority of 22% only applied this rating to the OP and VP conditions. The VP condition got the lowest ratings. In addition, the MP condition, compared to the VP and OP conditions, was judged most efficient, in terms of rapidity, by fourteen subjects. Thus, the majority of subjects considered implicitly that oral messages had helped them to achieve the prescribed search tasks, especially when verbal information was associated with a visual presentation of the target. Finally, the majority of subjects (66%) preferred the MP condition to the others. Nevertheless, three subjects complained of the content or wording of some oral messages. Criticisms concern, for one subject, messages without spatial information (two messages), for the second one, the use of a somewhat technical word outside his vocabulary (one message), and for the last one, the length and complexity of RSI messages. In addition, a fourth subject considered oral messages were not useful in the MP condition. To conclude, the MP condition came first in the subjective judgments of most subjects, as regards both the utility and usability of oral messages. These results are encouraging in view of the numerous potential interactive applications involving visual search tasks. However, voluntary participants in experimental evaluations of novel artefacts or techniques are prone to judge them positively (experimental bias), especially if the evaluation consists in a single session like ours. Their judgments may evolve under the influence of experience. Further usability studies are needed, in particular for assessing how future users will appraise the support provided by oral messages after extensive practice in real contexts of use.
Do Oral Messages Help Visual Search?
4.
153
Conclusion
A preliminary experimental study has been presented, which aims at eliciting the contribution of oral messages to facilitating visual search tasks on crowded visual displays. Results of quantitative and qualitative analyses suggest that appropriate verbal messages can improve both search accuracy and selection times. In particular, multimodal messages including absolute spatial oral information on the target location in the visual scene, together with a visual presentation of the isolated target, are most effective. This type of messages also got the highest subjective satisfaction ratings from most subjects. Subjects’ acceptance, even in an experimental environment, is a valuable asset. Numerous potential applications exist. Facilitating visual search could indeed improve the efficiency and usability of present human-computer interaction sensibly, as direct manipulation of GUIs is the prevalent user interface design paradigm for the moment. However, these results are only tentative, by reason of the relatively small number of subjects involved in the experiment (18), the limited number of scenes they had to process (12 per condition), and the coarseness of the measures used which were restricted to search accuracy and selection times (including search, identification, and selection of the target). In addition, qualitative analyses (focused on subjects’ errors) suggest the possible influence, on subjects’ results, of factors that were not systematically taken into account in the design of our experimental protocol. Our current short term research directions are based on these observations. We are currently planning a series of experimental studies focused on indepth investigation of the possible influence, on search accuracy and selection times, of the visual characteristics of scenes and targets, namely scene structure or target position. In particular, structure seems a factor capable of facilitating the exploration of class 1 scenes significantly, see [Cribbin and Chen, 2001a], whereas the efficiency of relative spatial information can be influenced by the proximity of the target to the salient object [Gramopadhye and Madhani, 2001]. Each of these experiments will be designed along the same lines as the one presented here, and implemented using a similar experimental protocol. However, it will address a few specific related issues and involve a large number of participants, in order to make it possible to refine and enrich the results of the initial study presented here. In addition, to achieve sound meaningful interpretations of future quantitative experimental results, we shall compare them with qualitative eye-tracking data obtained from a carefully selected sample of participants. We believe that a better understanding of visual search strategies, of their inter-individual diversity [Cribbin and Chen, 2001b], their sensitivity to the visual characteristics
154
Advances in Natural Multimodal Dialogue Systems
of the scenes displayed, and their evolution under the influence of oral support, will prove useful for improving the design of oral user support in visual search tasks. In particular, such knowledge could help designers to tailor the information content and wording of oral messages to individual search strategies and visual scene characteristics. A further step may be to compare, in terms of effectiveness and usability, oral support to visual search with various visual enhancements of targets.
References Ahlberg, C., Williamson, C., and Shneiderman, B. (1992). Dynamic Queries for Information Exploration: An Implementation and Evaluation. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 619–626, New York. ACM Press. André, E. (1997). WIP and PPP: A Comparison of two Multimedia Presentation Systems in Terms of the Standard Reference Model. Computer Standards and Interfaces, 18(6-7):555–564. André, E. and Rist, T. (1993). The Design of Illustrated Documents as a Planning Task. In Maybury, M. T., editor, Intelligent Multimedia Interfaces, pages 96–116. Menlo Park, CA: AAAI/MIT Press. Baber, C. (2001). Computing in a Multimodal World. In Proceedings of the First International Conference on Universal Access in Human-Computer Interaction (UAHCI), pages 232–236. Mahwah, NJ: Lawrence Erlbaum. Bernsen, N. O. (1994). Foundations of Multimodal Representations, a Taxonomy of Representational Modalities. Interacting with Computers, 6:347– 371. Bieger, G. R. and Glock, M. D. (1986). Comprehending Spatial and Contextual Information in Picture-Text Instructions. Journal of Experimental Education, 54:181–188. Burhans, D. T., Chopra, R., and Srihari, R. K. (1995). Domain Specific Understanding of Spatial Expressions. In Proceedings of the IJCAI Workshop on the Representation and Processing of Spatial Expressions, pages 33–40, Montréal, Canada. Card, S. K. and Mackinlay, J. (1997). The Structure of the Information Visualization Design Space. In Proceedings of IEEE Symposium on Information Visualization (InfoVis), pages 92–99, Phoenix, AZ. IEEE Computer Society Press. Card, S. K., Mackinlay, J., and Shneiderman, B. (1999). Readings in Information Visualization – Using Vision to Think. San Francisco (CA): Morgan Kaufmann Publishers. Chelazzi, L. (1999). Serial Attention Mechanisms in Visual Search: A Critical Look at the Evidence. Psychological Research, 62(2-3):195–219.
Do Oral Messages Help Visual Search?
155
Coutaz, J. and Caelen, J. (1991). A Taxonomy for Multimedia and Multimodal User Interfaces. In Proceedings of the First ERCIM Workshop on Multimodal HCI, pages 143–148, INESC, Lisbon. Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., and Young, R. (1995). Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE Properties. In Proceedings of INTERACT, pages 115–120, Lillehammer, Norway. Cribbin, T. and Chen, C. (2001a). A Study of Navigation Strategies in SpatialSemantic Visualizations. In Proceedings of the Ninth International Conference on Human-Computer Interaction (HCI International), pages 948–952. Mahwah, NJ: Lawrence Erlbaum. Cribbin, T. and Chen, C. (2001b). Exploring Cognitive Issues in Visual Information Retrieval. In Proceedings of INTERACT, pages 166–173, Tokyo, Japan. Amsterdam: IOS Press. de Vries, G. and Johnson, G.I. (1997). Spoken Help for a Car Stereo: An Exploratory Study. Behaviour and Information Technology, 16(2):79–87. Engelkamp, J. (1992). Modality and Modularity of the Mind. In Actes du 5ème Colloque de l’ARC ’Percevoir, Raisonner, Agir – Articulation de Modèles Cognitifs’, pages 321–343, Nancy. Faraday, P. and Sutcliffe, A. (1997). Designing Effective Multimedia Presentations. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 272–278. New York: ACM Press & Addison Wesley. Fekete, J.-D. and Plaisant, C. (2002). Interactive Information Visualization of a Million Items. In Proceedings of IEEE Symposium on Information Visualization (InfoVis), pages 117–124. Boston, MA: IEEE Press. Findlay, J. M. and Gilchrist, D. (1998). Eye Guidance and Visual Search. In Underwood, G., editor, Eye Guidance in Reading and Scene Perception, chapter 13, pages 295–312. Amsterdam: Elsevier. Frank, A. (1998). Formal Models for Cognition - Taxonomy of Spatial Location Description and Frames of Reference. In Freksa, C., Habel, C., and Wender, K. F., editors, Spatial Cognition - An Interdisciplinary Approach to Representing and Processing Spatial Knowledge, volume 1, pages 293–312. Berlin: Springer Verlag. Grabowski, N. A. and Barner, K. E. (1998). Data Visualisation Methods for the Blind Using Force Feedback and Sonification. In Proceedings of the SPIE Conference on Telemanipulator and Telepresence Technologies, volume 3524, pages 131–139. Bellingham, WA: International Society for Optical Engineering (SPIE). Gramopadhye, A. K. and Madhani, K. (2001). Visual Search and Visual Lobe Size. In Arcelli, C., editor, Proceedings of Fourth International Conference on Visual Form, Lecture Notes in Computer Science, volume 2059, pages 525–531. Berlin Heidelberg: Springer Verlag.
156
Advances in Natural Multimodal Dialogue Systems
Henderson, J. M. and Hollingworth, A. (1998). Eye Movements during Scene Viewing: an Overview. In Underwood, G., editor, Eye Guidance in Reading and Scene Perception, chapter 12, pages 269–293. Amsterdam: Elsevier. Lamping, J., Rao, R., and Pirolli, P. (1995). A Focus + Context Technique Based on Hyperbolic Geometry for Visualizing Large Hierarchies. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 401–408. New York: ACM Press & Addison Wesley. Massaro, D. (2002). Personal communication. Maybury, M. T. (2001). Universal Multimedia Information Access. In Proceedings of the First International Conference on Universal Access in HumanComputer Interaction (UAHCI), pages 382–386. Mahwah, NJ: Lawrence Erlbaum. Maybury, M.T. (1993). Planning Multimedia Explanations Using Communicative Acts. In Maybury, M.T., editor, Intelligent Multimedia Interfaces, pages 59–74. Menlo Park, CA: AAAI/MIT Press. Nigay, L. and Coutaz, J. (1993). A Design Space for Multimodal Systems: Concurrent Processing and Data Fusion. In Proceedings of INTERCHI, pages 172–178. New York: ACM Press & Addison Wesley. Oviatt, S., DeAngeli, A., and Kuhn, K. (1997). Integration and Synchronisation of Input Modes during Multimodal Human-Computer Interaction. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 415–422. New York: ACM Press & Addison Wesley. Rasmussen, J. (1986). Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. Amsterdam: North-Holland. Robbe, S., Carbonell, N., and Dauchy, P. (2000). Expression Constraints in Multimodal Human-Computer Interaction. In Proceedings of the International Conference on Intelligent User Interfaces (IUI), pages 225–229. New York: ACM Press. Shneiderman, B. (1983). Direct Manipulation: A Step Beyond Programming Languages. IEEE Computer, 16:57–69. Shneiderman, B. (1996). The Eyes Have It: A task by Data Type Taxonomy for Information Visualizations. In Proceedings of IEEE Workshop on Visual Languages (VL), pages 336–343, Boulder, CO. Sidner, C. L. and Dzikovska, M. (2002). Hosting Activities: Experience with and Future Directions for a Robot Agent Host. In Proceedings of the International Conference on Intelligent User Interfaces (IUI), pages 143–150. New York: ACM Press. Stock, O., Strappavara, C., and Zancanaro, M. (1997). Explorations in an Environment for Natural Language Multimodal Information Access. In Maybury, M. T., editor, Intelligent Multimedia Information Retrieval, pages 381– 398. Menlo Park, CA: AAAI/MIT Press.
Do Oral Messages Help Visual Search?
157
van Diepen, M. J., Wampers, M., and d’Ydewall, G. (1998). Functional Division of the Visual Field: Moving Masks and Moving Windows. In Underwood, G., editor, Eye Guidance in Reading and Scene Perception, chapter 15, pages 337–355. Amsterdam: Elsevier. van Mulken, S., André, E., and Muller, J. (1999). An Empirical Study on the Trustworthiness of Life-Like Interface Agents. In Proceedings of the Eighth International Conference on Human-Computer Interaction (HCI International), pages 152–156. Mahwah, NJ: Lawrence Erlbaum. Yu, W., Ramloll, R., and Brewster, S. A. (2001). Haptic Graphs for Blind Computer Users. Haptic Human-Computer Interaction, 2058:41–51.
Chapter 8 GEOMETRIC AND STATISTICAL APPROACHES TO AUDIOVISUAL SEGMENTATION Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa Computer Science and Artificial Intelligence Laboratory M.I.T., Cambridge, MA 02139, USA
{trevor, fisher, kwilson, siracusa}@csail.mit.edu Abstract
Multimodal approaches are proposed for segmenting multiple speakers using geometric or statistical techniques. When multiple microphones and cameras are available, 3-D audiovisual tracking is used for source segmentation and array processing. With just a single camera and microphone, an information theoretic criteria separates speakers in a video sequence and associates relevant portions of the audio signal. Results are shown for each approach, and an initial integration effort is discussed.
Keywords:
Source separation, vision tracking, microphone array, mutual information.
1.
Introduction
Conversational dialog systems have become practically useful in many application domains, including travel reservations, traffic information, and database access. However most existing conversational speech systems require tethered interaction, and work primarily for a single user. Users must wear an attached microphone or speak into a telephone handset, and must do so one at a time. This limits the range of use of dialog systems, since in many applications users might expect to freely approach and interact with a device. Worse, they may wish to arrive as a group, and talk among themselves while interacting with the system. To date it has been difficult for speech recognition systems to handle such conditions and correctly recognize the utterances intended for the device. Given only a single sensing modality, and perhaps only a single sensor, disambiguating the audio from multiple speakers can be a challenge. But with 159 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 159–180. © 2005 Springer. Printed in the Netherlands.
160
Advances in Natural Multimodal Dialogue Systems
multiple modalities, and possibly multiple sets of sensors, segmentation can become feasible. In this chapter we present two methods for audiovisual segmentation of multiple speakers, based on geometric and statistical source separation approaches. We have explored two configurations which are of practical interest. The first is based on a “smart environment” or “smart room” enabled with multiple stereo cameras and a ceiling mounted large-aperture microphone array grid. In this configuration users can move arbitrarily through the room or environment while focused audiovisual streams are generated from their appearance and utterances. In the second configuration we presume a single omnidirectional microphone and single camera is available. This is akin to what one might find on a PDA or cell phone, or low-cost PC videoconferencing installation. In a multisensor environment, we use a geometric approach, and use multiview image correspondence and tracking methods combined with acoustic beamforming techniques. A multimodal approach can track sources even in acoustically reverberant environments with dynamic illumination, conditions that are challenging for audio or video processing alone. When only a single multimodal sensor pair (audio and video) is available, we use a statistical approach, jointly modelling audio and video variation to identify cross-modal correspondences. We show how this approach can detect which user is speaking when several are facing a device. This allows the segregation of users’ utterances from each other’s speech, and from background noise events. We first review related work, and then present our method for geometric source separation and vision-guided microphone array processing. We then describe our single-camera/microphone method for audiovisual correspondence using joint statistical processing. We show results with each of these techniques, and describe our initial integration effort.
2.
Related Work
Humans routinely perform tasks in which ambiguous auditory and visual data are combined in order to support accurate perception. In contrast, automated approaches for statistical processing of multimodal data sources lag far behind. This is primarily due to the fact that few methods adequately model the complexity of the audio/visual relationship. Classical approaches to multimodal fusion at a signal processing level often either assume a statistical relationship which is too simple (e.g. jointly Gaussian) or defer fusion to the decision level when many of the joint (and useful) properties have been lost. While such pragmatic choices may lead to simple statistical measures, they do so at the cost of modelling capacity.
Geometric and Statistical Approaches to Audiovisual Segmentation
161
An information theoretic approach motivates fusion at the measurement level without regard to specific parametric densities. The idea of using information theoretic principles in an adaptive framework is not new (e.g. see [Deco and Obradovic, 1996] for an overview) with many approaches suggested over the last 30 years. A critical distinction in most information theoretic approaches lies in how densities are modelled (either explicitly or implicitly), how entropy (and by extension mutual information) is approximated or estimated, and the types of mappings which are used (e.g. linear vs. nonlinear). Approaches which use a Gaussian assumption include [Plumbley and Fallside, 1988; Plumbley, 1991] and [Becker, 1992]. Additionally, Becker [1992] applies the method to fusion of artificially generated random dot stereograms. There has been substantial progress on feature-level integration of speech and vision. For example, Meier et al. [1999], Wolff et al. [1994] and others have built visual speech reading systems that can improve speech recognition results dramatically. Our system, described below, is designed to be able to detect and disambiguate cases where audio and video signals are coming from different sources. Other audio/visual work which is closely related to ours is that of Hershey and Movellan [1999], which examined the per-pixel correlation relative to an audio track, detecting which pixels have related variation. Again, an inherent assumption of this method was that the joint statistics were Gaussian. Slaney and Covell [2000] looked at optimizing temporal alignment between audio and video tracks using canonical correlations (equivalent to mutual information in the jointly Gaussian case), but did not address the problem of detecting whether two signals came from the same person. Several authors have explored geometric approaches to audiovisual segmentation using array processing techniques. Microphone arrays are a special case of the more general problem of sensor arrays, which have been studied extensively in the context of applications such as radar and sonar [van Veen and Buckley, 1988]. The Huge Microphone Array project [Silverman et al., 1998] is investigating the use of very large arrays containing hundreds of microphones. Their work concentrates on audio-only solutions to array processing. Another related project is Wang and Brandstein’s audio-guided active camera [Wang and Brandstein, 1999], which uses audio localization to steer a camera on a pan/tilt base. A number of projects [Bub et al., 1995; Casey et al., 1995; Collobert et al., 1996] have used vision to steer a microphone array, but because they use a single camera to steer a far-field array, they cannot obtain or make use of full 3-D position information; they can only select sound coming from a certain direction. We are exploring both a geometric and statistical approach to audiovisual segmentation. In the next section, we describe our geometric approach based on microphone and camera arrays. Following that, we present our statistical
162
Advances in Natural Multimodal Dialogue Systems
approach using an information theoretic measure to relate single channel audio and video signals.
3.
Multimodal Multisensor Domain
The association between sound and location makes a microphone array a powerful tool for audiovisual segmentation. In combination with additional sensors and contextual information from the environment, a microphone array can effectively amplify and separate sounds of interest from complex background noise. To focus a microphone array, the location of the speaker(s) of interest must be known. A number of techniques exist for localizing sound sources using only acoustic cues [Viberg and Krim, 1997], but the performance of these localization techniques tends to degrade significantly in the presence of reverberation and/or multiple sound sources. Unfortunately, most common office and meeting room environments are highly reverberant, with reflective wall and table surfaces, and will normally contain multiple speakers. However, in a multimodal setting we can take advantage of other sensors in the environment to perform localization of multiple speakers despite reverberation. We use a set of cameras to track the position of speakers in the environment, and report the relative geometry of speakers, cameras, and microphones. The vision modality is not affected by acoustic reverberation, but its accuracy will depend on the calibration and segmentation procedures. In practice we use video information to restrict the range of possible acoustic source locations to a region small enough to allow for acoustic localization techniques to operate without severe problems with reverberation and multiple speakers.
3.1
Microphone Array Processing Overview
Many problems can be addressed through array processing. The two array processing problems that are relevant to our system are beamforming and source localization. Beamforming is a type of spatial filtering in which the signals from individual array elements are filtered and added together to produce an output that amplifies signals coming from selected regions of space and attenuates sounds from other regions of space. In the simplest form of beamforming, delay-andsum beamforming, each channel’s filter is a pure delay. The delay for each channel is chosen such that signals from a chosen “target location” are aligned in the array output. Signals from other locations will tend to be combined incoherently. Source localization is a complementary problem to beamforming whose goal is to estimate the location of a signal source. One way to do this is to beamform to all candidate locations and to pick the location that yields the
Geometric and Statistical Approaches to Audiovisual Segmentation
163
Figure 8.1. Array power response as a function of position (two speakers). This plot shows the array output power as the array’s focus is scanned through a plane centred on one speaker while another speaker is nearby. The central speaker is easily discernible in the plot, but the peak corresponding to the weaker speaker is difficult to distinguish among the sidelobe peaks. Using vision-based person tracking cues can disambiguate this case.
strongest response. This method works well, but the amount of computation required to do a full search of a room is prohibitively large. Another method for source localization consists of estimating relative delays among channels and using these delays to calculate the location of the source. Delay-estimation techniques are computationally efficient but tend to perform poorly in the presence of multiple sources and/or reverberation. For microphone arrays that are small in size compared to the distance to the sources of interest, incoming wavefronts are approximately planar. Because of this, only source direction can be determined; source distance remains ambiguous. When the array is large compared to the source distance, the sphericity of the incoming wavefronts is detectable, and both direction and distance can be determined. These effects of array size apply both to localization and to beamforming, so if sources at different distances in the same direction must be
164
Advances in Natural Multimodal Dialogue Systems
Figure 8.2. The test environment. This figure shows a schematic view of the environment with stereo cameras represented by black triangles and microphones represented by empty circles. The single microphone used for comparison in the experiments is shown filled in.
separated, a large array must be used. As a result, with large arrays the signalto-noise ratio (for a given source) at different sensors will vary with source location. Because of this, signals with better signal-to-noise ratios should be weighted more heavily in the output of the array. Our formulation of the steering algorithm presented in Section 8.3.3 takes this into account.
3.2
Person Tracking Overview
Tracking people in known environments has recently become an active area of research in computer vision. Several person-tracking systems have been developed to detect the number of people present as well as their 3D position over time. These systems use a combination of foreground/background classification, clustering of novel points, and trajectory estimation over time in one or more camera views [Darrell et al., 2000; Krumm et al., 2000]. Colour-based approaches to background modelling have difficulty with illumination variation due to changing lighting and/or video projection. To overcome this problem, several researchers have supported the use of background models based on stereo range data [Darrell et al., 2000; Ivanov et al., 2000]. Unfortunately, most of these systems are based on computationally intense, exhaustive stereo disparity search. We have developed a system that can perform dense, fast, range-based tracking with modest computational complexity. We apply ordered disparity search techniques to prune most of the disparity search computation during foreground detection and disparity estimation, yielding a fast, illumination-insen-
Geometric and Statistical Approaches to Audiovisual Segmentation
165
sitive 3D tracking system. Details of our system are presented in [Darrell et al., 2001], and work on integrating audio more directly into the tracking is presented in [Checka et al., 2003]. Our system reports the 3-D position of people moving about an environment equipped with an array of stereo cameras.
3.3
Vision-Guided Acoustic Volume Selection
We perform both audio localization and beamforming with a large, ceilingmounted microphone array. Localization uses information from both audio and video, while beamforming uses only the audio data and the results of the localization processing. A large array allows us to select a volume of 3-D space, rather than simply form a 2-D beam of enhanced response as anticipated by the standard array localization algorithms. However, the usual assumption of constant target signal-to-noise ratio (SNR) across the array does not hold when the array geometry is large (when the array width is on the same scale as the target distance.) Our system uses the location estimate from the vision tracker as the initial guess from which to begin a gradient ascent search for a local maximum in beam output power. In our system, beam power is defined as the integral over a half-second window of the square of the output amplitude. The vision tracker is accurate to within less than one meter. Gradient ascent to the nearest local maximum can therefore be expected to converge to the location of the speaker of interest when no other speakers are very close by. For small microphone arrays, the relative SNRs of the individual channels do not vary significantly as a function of source location. This is, however, not true for larger microphone arrays. For our array, which is roughly 4 meters across, we must take into account the fact that some elements will have better signals than others. Specifically, if we assume that we have signals x1 and x2 which are versions of the unit-variance desired signal, s, that have been contaminated by unit-variance uncorrelated noise, we can analyze the problem as follows: x1 = a1 s + n1 x2 = a2 s + n2 In this model, the signal-to-noise ratios of x1 and x2 will be a21 and a22 , respectively. Their optimal linear combination will be of the form y = bx1 + x2 . Because of the uncorrelated noise assumption, the SNR of this combination will be (ba1 + a2 )2 b2 + 1 By taking the derivative of this expression with respect to b and setting the result equal to zero, one finds that the optimal value of b is: SN R(y) =
166
Advances in Natural Multimodal Dialogue Systems
Table 8.1. Audio-video localization performance.
Distant microphone Video only Audio only (dominant speaker) Audio-Video
a1 b= = a2
SNR (dB) −7.7 −6.1 −4.2 0.1
SN R(x1 ) SN R(x2 )
Individual elements’ signals should be scaled by a constant proportional to the square root of their SNRs. We use the location estimate to weight individual channels assuming a 1/r attenuation due to the spherical spreading of the source: an = 1/rn .
4.
Results
Our test environment, depicted in Figure 8.2, is a conference room equipped with 15 omnidirectional microphones spread across the ceiling and 2 stereo cameras on adjacent walls. The audio and video subsystems were calibrated independently, and for our experiments, we performed a joint calibration by finding the least-squares bestfit alignment between the two coordinate systems. Figure 8.1 is an example of what happens when multiple speakers are present in the room. Audio-only gradient ascent could easily find one of the undesirable local maxima. Because our vision-based tracker is accurate to within one meter, we can safely assume that we will find the correct local maximum even in the presence of interferers. To validate our localization and source separation techniques, we ran an experiment in which two speakers spoke simultaneously while one of them moved through the room. We tracked the moving speaker with the stereo tracker and processed the corresponding audio stream using three different localization techniques. For each, we used a reference signal collected with a close-talking microphone to calculate a time-averaged SNR (Table 8.1). For performance comparison, we use the signal from a single distant microphone near the centre of the room. This provides no spatial selectivity, but for our scenario it tends to receive the desired speech more strongly than the interfering speech. The SNR for the single microphone case is negative because of a combination of the interfering speaker and diffuse noise from the room’s ventilation system.
Geometric and Statistical Approaches to Audiovisual Segmentation
167
Table 8.2. Percent correct word recognition rates for male(female) speakers. Performance was computed based on 20 sentence queries from five male and two female speakers. The close-talking microphone was clipped to the lapel of the speaker. The microphone array is as described above. The distant microphone is one array element from near the centre of the room. Interferer Close-talking microphone Microphone array Distant microphone
None 95(96) 82(43) 73(35)
-24 dB 95(96) 80(41) 74(28)
-12 dB 95(95) 64(13) 51(7)
To evaluate the microphone array’s effects on recognition rates for automated speech recognition (ASR), we connected our system to the MIT Spoken Language Systems (SLS) Group’s JUPITER weather information system [Zue et al., 2000]. We had two speakers issue several weather-related queries from different locations in the room. As collected, the data contains quiet but audible noise from the ventilation system in the room. To evaluate the results under noisier conditions, additional noise was added to these signals. The results are shown in Table 8.2. As can be seen in the table, the beamformed signal from the microphone array was in all cases superior to the single distant microphone, but not as good as a close-talking microphone. The -12 dB interferer significantly degraded the performance of the array. We are currently working on adaptive null-steering algorithms that should improve performance in the presence of stronger interferers such as this.
5.
Single Multimodal Sensor Domain
In the single (multimodal) sensor domain, geometry is less useful, and array processing impossible; in this case we instead exploit audiovisual joint statistics to localize speakers. We adopt the paradigm of looking at a single camera view, and seeing what information from a single microphone can tell us about that view (and vice-versa.) We propose an independent cause model to capture the relationship between generated signals in each individual modality. Using principles from information theory and nonparametric statistics, we show how an approach for learning maximally informative joint subspaces can find cross-modal correspondences. We analyze the graphical model of multimodal generation and show under what conditions related subcomponents of each signal have high mutual information. Nonparametric statistical density models can be used to measure the degree of mutual information in complex phenomena [Fisher and Principe, 1998]. We apply these models to audio/visual data. This technique simultaneously learns projections of images in the video sequence and projections of sequences of periodograms taken from the audio sequence. The projections are computed
168
Advances in Natural Multimodal Dialogue Systems
adaptively such that the video and audio projections have maximum mutual information (MI). We first review the basic method for audio-visual fusion and information theoretic adaptive methods, see [Fisher and Darrell, 2002] for full details. We present our probabilistic model for cross-modal signal generation and show how audiovisual correspondences can be found by identifying components with maximal mutual information. In an experiment comparing the audio and video of every combination of a group of eight users, our technique was able to perfectly match the corresponding audio and video for each user. Finally, we show a new result on a monocular speaker segmentation task where we segment the audio between several speakers seen by the camera. These results are based purely on the instantaneous cross-modal mutual information between the projections of the two signals, and do not rely on any prior experience or model of user’s speech or appearance.
5.1
Probabilistic Models of Audio-Visual Fusion
We consider multimodal scenes which can be modelled probabilistically with one joint audiovisual source and distinct background interference sources for each modality. Each observation is a combination of information from the joint source, and information from the background interferer for that channel. In contrast with the array processing case, we explicitly model visual appearance variation, not just 3D geometry. We use a graphical model (Figure 8.3) to represent this relationship. In the diagrams, B represents the joint source, while A and C represent single modality background interference. Our purpose here is to analyze under which conditions our methodology should uncover the underlying cause of our observations. Figure 8.3a shows an independent cause model for our typical case, where {A, B, C} are unobserved random variables representing the causes of our (high-dimensional) observations in each modality {X a , X v }. In general there may be more causes and more measurements, but this simple case can be used to illustrate our algorithm. An important aspect is that the measurements have dependence on only one common cause. The joint statistical model consistent with the graph of figure 8.3a is P (A, B, C, X a , X v ) = P (A)P (B)P (C)P (X a |A, B)P (X v |B, C). Given the independent cause model, a simple application of Bayes’ rule (or the equivalent graphical manipulation) yields the graph of figure 8.3b which is consistent with P (A, B, C, X a , X v ) = P (X a )P (C)P (A, B|X a )P (X v |B, C), which shows that information about X a contained in X v is conveyed through the joint statistics of A and B. The consequence being that, in general, we
Geometric and Statistical Approaches to Audiovisual Segmentation
A
169
A Xa
B
Xa B
Xv C
Xv C
(a)
(b) a XA
A
a XB
a XB
B
v XB
B v XB v XC
C
(c)
a XB
B
v XB
(d)
Figure 8.3. Graphs illustrating the various statistical models exploited by the algorithm: (a) the independent cause model - X a and X v are independent of each other conditioned on {A, B, C}, (b) information about X a contained in X v is conveyed through joint statistics of A and B, (c) the graph implied by the existence of a separating function, and (d) two equivalent Markov chains which can be extracted from the graphs if the separating functions can be found.
cannot disambiguate the influences that A and B have on the measurements. A similar graph is obtained by conditioning on X v . Suppose decompositions of the measurement X a and X v exist such that the following joint densities can be written: P (A, B, C, X a , X v ) = a a v |A)P (XB |B)P (XB |B)P (XCv |C) P (A)P (B)P (C)P (XA a , X a ] and X v = [X v , X v ]. An example for our specific where X a = [XA B B C application would be segmenting the video image (or filtering the audio signal). In this case, we get the graph of Figure 8.3c and from that graph we can extract the Markov chain which contains elements related only to B. Figure 8.3d shows equivalent graphs of the extracted Markov chain. As a consequence, there is no influence due to A or C. Of course, we are still left with the formidable task of finding a decomposition, but given the decomposition it can be shown, using the data processing inequality [Cover and Thomas, 1991], that the following inequality holds: a v a , XB ) ≤ I(XB , B) I(XB a v v I(XB , XB ) ≤ I(XB , B)
(8.1) (8.2)
a and X v (e.q. More importantly, these inequalities hold for functions of XB B a a v v Y = f (X ; ha ) and Y = f (X ; hv )). Consequently, by maximizing the
170
Advances in Natural Multimodal Dialogue Systems
mutual information between I(Y a ; Y v ), we must necessarily increase the mutual information between Y a and B and Y v and B. The implication is that fusion in such a manner discovers the underlying cause of the observations; that is, the joint density of p(Y a , Y v ) is strongly related to B. Furthermore, with an approximation, we can optimize this criterion without estimating the separating function directly. In the event that a perfect decomposition does not exist, it can be shown that the method will approach a “good” solution in the Kullback-Leibler sense. From the perspective of information theory, estimating separate projections of the audio video measurements which have high mutual information makes intuitive sense as such features will be predictive of each other. The advantage is that the form of those statistics is not subject to the strong parametric assumptions (e.g. joint Gaussianity) which we wish to avoid. We can find these projections using a technique that maximizes the mutual information between the projections of the two spaces. Following [Fisher et al., 2000], we use a nonparametric model of joint density for which an analytic gradient of the mutual information with respect to projection parameters is available. In principle the method may be applied to any function of the measurements, Y = f (X; h), which is differentiable in the parameters h (e.g. as shown in [Fisher et al., 2000]). We consider a linear fusion model which results in a significant computational savings at a minimal cost to the representational power (largely due to the nonparametric density modelling of the output):
v y1v · · · yN a y1a · · · yN
=
hTv 0T
0T hTa
xv1 · · · xvN xa1 · · · xaN
(8.3)
where xvi ∈ Nv and xai ∈ Na are lexicographic samples of images and periodograms, respectively, from an A/V sequence. The linear projection defined by hTv ∈ Mv ×Nv and hTa ∈ Ma ×Na maps A/V samples to low dimensional features yiv ∈ Mv and yia ∈ Ma . Treating xi and yi as samples from a random variable our goal is to choose hv and ha to maximize the mutual information, I (Y a ; Y v )), of the derived measurements. Mutual information indicates the amount of information that one random variable conveys on average about another. The usual difficulty of MI as a criterion for adaptation is that it is an integral function of probability densities. Furthermore, in general we are not given the densities themselves, but samples from which they must be inferred. We use a second-order entropy approximation with a nonparametric density estimator such that the gradient terms with respect to the projection coefficients can be computed exactly by evaluating a finite number of functions at a finite number of sample locations in the output space as shown in [Fisher and Principe, 1997; Fisher and Principe, 1998].
Geometric and Statistical Approaches to Audiovisual Segmentation
171
Figure 8.4. Video sequence contains one speaker and monitor which is flickering: (a) one image from the sequence, (b) pixel-wise image of standard deviations taken over the entire sequence, (c) image of the learned projection, hv , (d) image of hv for incorrect audio.
This method requires that the projection be differentiable, which it is in our case. Additionally some form of capacity control is necessary as the method results in a system of underdetermined equations. To address this problem, we impose an L2 penalty on the projection coefficients of ha and hv [Fisher and Darrell, 2002]. Furthermore, we impose the criterion that if we consider the projection hv as a filter, it has low output energy when convolved with images in the sequence (on average). This constraint is the same as that proposed by Mahalanobis et al. [1987] for designing optimized correlators, the difference being that in their case the projection output was designed explicitly while in our case it is derived from the MI optimization in the output space. For the full details of this method, see [Fisher et al., 2000; Fisher and Darrell, 2002].
5.2
Single Microphone and Camera Experiments
Our motivating scenario for this application is a group of users interacting with an anonymous handheld device or kiosk using spoken commands. Given a received audio signal, we would like to verify whether the person speaking the command is in the field of view of the camera on the device, and if so to localize which person is speaking.
172
Advances in Natural Multimodal Dialogue Systems
Simple techniques which check only for the presence of a face (or moving face) would fail when two people were looking at their individual devices and one spoke a command. Since interaction may be anonymous, we presume no prior model of the voice or appearance of users is available to perform the verification and localization. We collected audio-video data from eight subjects. In all cases the video data was collected at 29.97 frames per second at a resolution of 360x240. The audio signal was collected at 48KHz, but only 10Khz of frequency content was used. All subjects were asked to utter the phrase “How’s the weather in Taipei?”. This typically yielded 2–2.5 seconds of data. Video frames were processed as is, while the audio signal was transformed to a series of periodograms. The window length of the periodogram was 2/29.97 seconds (i.e. spanning the width of two video frames). Upon estimating projections, the mutual information between the projected audio and video data samples is used as the measure of consistency. All values for mutual information are in terms of the maximum possible value, which is the value obtained (in the limit) if the two variables are uniformly distributed and perfectly predict one another. In all cases, we assume that there is not significant head movement on the part of the speaker during the utterance of the sentence. While this assumption might be violated in practice, one might account for head movement using a tracking algorithm, in which case the algorithm as described would process the images after tracking. Figure 8.4a shows a single video frame from one sequence of data. In the figure, there is a single speaker and a video monitor. Throughout the sequence the video monitor exhibits significant flicker. Figure 8.4b shows an image of the pixel-wise standard deviations of the image sequence. As can be seen, the energy associated with changes due to monitor flicker is greater than that due to the speaker. Figure 8.6a shows the associated periodogram sequence where the horizontal axis is time and the vertical axis is frequency (0-10 Khz). Figure 8.4c shows the coefficients of the learned projection when fused with the audio signal. As can be seen the projection highlights the region about the speaker’s lips. Figure 8.5 shows results from another sequence in which there are two people. The person on the left was asked to utter the test phrase, while the person on the right moved their lips, but did not speak. This sequence is interesting in that a simple face detector would not be sufficient to disambiguate the audio and video stream. Figure 8.5b shows the pixel variance as before. There are significant changes about both subjects lips. Figure 8.5c shows the coefficients of the learned projection when the video is fused with the audio and again the region about the correct speaker’s lips is highlighted. In addition to localizing the audio source in the image sequence, we can also check for consistency between the audio and video. Such a test is useful in the case that the person to which a system is visually attending is not the
Geometric and Statistical Approaches to Audiovisual Segmentation
173
Figure 8.5. Video sequence containing one speaker (person on left) and one person who is randomly moving their mouth/head (but not speaking): (a) one image from the sequence, (b) pixel-wise image of standard deviations taken over the entire sequence, (c) image of the learned projection, hv , (d) image of hv for incorrect audio.
person who actually spoke. Having learned a projection which optimizes MI in the output feature space, we can then estimate the resulting MI and use that estimate to quantify the audio/video consistency. Using the sequences of figure 8.4 and 8.5, we compared the fusion result when using a separately recorded audio sequence from another speaker. The periodogram of the alternate audio sequence is shown in figure 8.6b. Figures 8.4d and 8.5d show the resulting hv when the alternate audio sequence is used. In the case that the alternate audio was used, we see that coefficients related to the video monitor increase significantly in 8.4d while energy is distributed throughout the image of 8.5d. For figure 8.4, the estimate of mutual information was 0.68 relative to the maximum possible value for the correct audio sequence. In contrast when compared to the periodogram of 8.6b, the value drops to 0.08 of maximum. For the sequence of figure 8.5, the estimate of mutual information for the correct sequence was 0.61 relative to maximum, while it drops to 0.27 when the alternate audio is used. Data was collected from six additional subjects for this experiment, and each video sequence was compared to each audio sequence. (No attempt was made to temporally align the mismatched audio sequences at a fine scale, but
174
Advances in Natural Multimodal Dialogue Systems
Figure 8.6. Gray scale magnitude of audio periodograms. Frequency increases from bottom to top, while time is from left to right. (a) audio signal for image sequence of figure 8.4. (b) alternate audio signal recorded from different subject.
Table 8.3. Summary of results over eight video sequences. The columns indicate which audio sequence was used while the rows indicate which video sequence was used. In all cases the correct audio/video pair had the highest relative MI score. v1 v2 v3 v4 v5 v6 v7 v8
a1 0.68 0.20 0.05 0.12 0.17 0.20 0.18 0.13
a2 0.19 0.61 0.27 0.24 0.05 0.05 0.15 0.05
a3 0.12 0.10 0.55 0.32 0.05 0.05 0.07 0.10
a4 0.05 0.11 0.05 0.55 0.05 0.13 0.05 0.05
a5 0.19 0.05 0.05 0.22 0.55 0.14 0.05 0.31
a6 0.11 0.05 0.05 0.05 0.05 0.58 0.05 0.16
a7 0.12 0.18 0.05 0.05 0.20 0.05 0.64 0.12
a8 0.05 0.32 0.05 0.10 0.09 0.07 0.26 0.69
they were coarsely aligned). Table 8.3 summarizes the results. The previous sequences correspond to subjects 1 and 2 in the table. In every case the matching audio/video pairs exhibited the highest mutual information after estimating the projections. Finally, we tested how this method can segregate speech of multiple users in a single field of view. In concert with a face detection module, it is possible to detect which user is speaking and whether they are facing the camera. The audiovisual mutual information method is able to match the visual speech motion with the acoustic signal, and ignore confounding motions of the other user’s head or other motions in the scene. Figure 8.7 shows the results of tracking two users speaking in turns in front of a single camera and microphone, and detecting which is most likely to be speaking based on the measured audiovisual consistency.
Geometric and Statistical Approaches to Audiovisual Segmentation
175
Figure 8.7. Top row presents four frames from a video sequence with two speakers in front of a single camera and microphone. Audiovisual consistency is measured using a mutual information criteria. In the first two frames the left person is speaking, while in the last two the right person is speaking. The consistency measure shown in the bottom row for each frame correctly detects who is speaking.
6.
Integration
We have shown separately how geometric and statistical approaches can be used to solve audiovisual segmentation tasks and enable untethered conversational interaction. The geometric approach used 3-D tracking and array processing and ignored appearance variation. The statistical approach used a mutual information analysis of appearance and spectral variation, and ignored 3-D geometry. While each approach is already valuable in the intended domain, it is clear that they would benefit from combination. Our initial integration effort, described in detail in [Siracusa et al., 2003], combines geometric and statistical insights in a system for determining who is speaking in its environment. It uses multiple audio and video cues and allows for robust front-end processing in a multi-person conversational interface. In the system, data from a stereo camera and small, two-element linear microphone array are processed to yield measurements of audio direction of arrival (DOA) and of audio-visual synchrony. Using this simple array allows us to use a standard PC audio card for input, but it is only able to recover DOA in contrast to the full 3-D position information discussed in Section 3. In order to allow for real-time processing on a standard personal computer we restrict the complexity of our statistical models. A head tracker obtains stabilized regions of interest for each person in the scene and determines their positions with respect to the camera. A simple measure of audio-visual synchrony is then calculated for each of these regions in a similar style to Hershey and Movellan [1999]. We combine the two cues (DOA and audio-visual synchrony) in simple hypothesis testing framework which yields better performance than using any single cue alone.
176
Advances in Natural Multimodal Dialogue Systems
When multiple people interact with each other or with a conversational kiosk there are no guarantees about where they are located. When people are well-separated, audio DOA is sufficient to determine who is speaking, but when two or more people are standing close together or are in unknown locations outside the field of view of the camera, audio DOA may be insufficient. Measuring audio-visual synchrony in such situations can disambiguate who is speaking by giving some indication of whose lips are in synchrony with the audio. In the case when the speaker is not visible there should be no audio-visual synchrony for any of the visible users thus giving a hint that the speaker is off camera. The usefulness of our approach is shown in results for two kiosk-interaction scenarios. Each scenario involved two or three individuals conversing with each other or the audio-visual rig. Two individuals were tracked and remained visible to the camera at all times. The first scenario involved only two people who remained sufficiently separated throughout the recorded sequence. The second scenario involved three people with the third person not visible in the camera and located fairly close to the second person, thus producing the ambiguous situation we wished to address. Each sequence consisted of over 2000 frames. Three versions of our system were tested on each sequence. The first version used only the DOA information. The second version ignored DOA and used audio-visual synchrony as its measurement, while the third version combined both. Table 8.4. Sequence accuracy results–Percentage of the frames in which the system correctly determined who was speaking (% Accuracy). Sequence Two Person Three Person
DOA Only 89 % 80 %
A/V Sync. Only 76% 77 %
Combined 89 % 89 %
A summary of our results is presented in Table 8.4. Combining both measurements did not improve our systems performance on the two person sequence. This shows that audio DOA is a sufficient measurement for speaker localization when the sources are sufficiently separated. However, in the three person case, adding a measure of audio-visual synchrony helps disambiguate who is speaking. In this sequence the combined system reduced the error by nearly a factor of two. Figure 8.8 shows results for two frames of the three person sequence evaluated using the DOA only system (top) and the combined system (bottom). Using DOA only, the system cannot disambiguate between the right person speaking and an unknown individual who is nearby. Adding an audiovisual synchrony measure clarifies the situation. It is clear that when the right per-
Geometric and Statistical Approaches to Audiovisual Segmentation
177
Figure 8.8. Results for two frames of the three person video sequence. In frame 1875, the person on the right is speaking. In frame 1936, someone off camera is speaking. (Top) Uses only DOA. The first row shows segmented head regions of interest for each tracked person. The second row shows audio DOA information in a plan-view plot of the estimated configuration. The position of each person is marked with an ’x’. The third row shows posterior probabilities for who is speaking. (Bottom) Uses DOA and audio-video synchrony. A visualization of the audio-video synchrony measurement is shown in the second row.
178
Advances in Natural Multimodal Dialogue Systems
son is speaking there is a strong relationship between his mouth and the audio. Furthermore, when an off camera source is speaking their is no audio-visual synchrony for those who are visible. This work is an initial step toward producing robust conversational interfaces that demonstrates that combining geometric and statistical approaches can improve audiovisual segmentation. We are currently exploring different real-time integration techniques involving efficient implementations of the more sophisticated statistical models described in Section 5.
References Becker, S. (1992). An Information-Theoretic Unsupervised Learning Algorithm for Neural Networks. PhD thesis, University of Toronto. Bub, U., Hunke, M., and Waibel, A. (1995). Knowing Who to Listen to in Speech Recognition: Visually Guided Beamforming. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pages 848–851. Casey, M., Gardner, W., and Basu, S. (1995). Vision Steered Beam-forming and Transaural Rendering for the Artificial Life Interactive Video Environment (ALIVE). In Proceedings of the 99th Convention of the Audio Engineering Society (AES). Preprint 4052. Checka, N., Wilson, K., Rangarajan, V., and Darrell, T. (2003). A Probabilistic Framework for Multi-modal Multi-person Tracking. In Proceedings of Workshop on Multi-Object Tracking. http://www.ai.mit.edu/projects/vip/papers/checka-et-al-womot.pdf. Collobert, M., Feraud, R., LeTourneur, G., Bernier, O., Viallet, J. E., Mahieux, Y., and Collobert, D. (1996). LISTEN: A System for Locating and Tracking Individual Speakers. In Proceedings of Second International Conference on Face and Gesture Recognition, pages 283–288. Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. New York: John Wiley & Sons, Inc. Darrell, T., Demirdjian, D., Checka, N., and Felzenszwalb, P. (2001). PlanView Trajectory Estimation with Dense Stereo Background Models. In Proceedings of International Conference on Computer Vision, volume 2, pages 628–635. Darrell, T., Gordon, G. G., Harville, M., and Woodfill, J. (2000). Integrated Person Tracking Using Stereo, Color, and Pattern Detection. International Journal of Computer Vision, 37(2):199–207. Deco, G. and Obradovic, D. (1996). An Information Theoretic Approach to Neural Computing. New York: Springer Verlag. Fisher, J. W. III and Darrell, T. (2002). Probabalistic Models and Informative Subspaces for Audiovisual Correspondence. In Heyden, A., Sparr, G.,
Geometric and Statistical Approaches to Audiovisual Segmentation
179
Nielsen, M., and Johansen, P., editors, Proceedings of the Seventh European Conference on Computer Vision (ECCV), volume 3, pages 592–603. Springer Lecture Notes in Computer Science 2352. Fisher, J. W. III, Darrell, T., Freeman, W. T., and Viola, P. (2000). Learning Joint Statistical Models for Audio-Visual Fusion and Segregation. In Leen, T. K., Dietterich, T. G., and Tresp, V., editors, Advances in Neural Information Processing Systems (NIPS), volume 13, pages 1–7. MIT Press. Fisher, J. W. III and Principe, J. C. (1997). Entropy Manipulation of Arbitrary Nonlinear Mappings. In Principe, J. C., editor, Proceedings of IEEE Workshop on Neural Networks for Signal Processing VII, pages 14–23. Fisher, J. W. III and Principe, J. C. (1998). A Methodology for Information Theoretic Feature Extraction. In Stuberud, A., editor, Proceedings of the IEEE International Joint Conference on Neural Networks, volume 3, pages 1712–1716. Hershey, J. and Movellan, J. (1999). Using Audio-Visual Synchrony to Locate Sounds. In Solla, S. A., Leen, T. K., and Müller, K.-R., editors, Advances in Neural Information Processing Systems (NIPS), volume 12, pages 813–819. MIT Press. Ivanov, Y. A., Bobick, A. F., and Liu, J. (2000). Fast Lighting Independent Background Subtraction. International Journal of Computer Vision, 37(2): 199–207. Krumm, J., Harris, S., Meyers, B., Brummit, B., Hale, M., and Shafer, S. (2000). Multi-Camera Multi-Person Tracking for Easyliving. In Proceedings of the Third IEEE Workshop on Visual Surveillance, pages 3–10. Mahalanobis, A., Kumar, B., and Casasent, D. (1987). Minimum Average Correlation Energy Filters. Applied Optics, 26(17):3633–3640. Meier, U., Stiefelhagen, R., Yang, J., and Waibel, A. (1999). Towards Unrestricted Lipreading. In Proceedings of the Second International Conference on Multimodal Interfaces (ICMI), Hong Kong. Plumbley, M. (1991). On Information Theory and Unsupervised Neural Networks. Technical Report CUED/F-INFENG/TR. 78, Cambridge University Engineering Department, UK. Plumbley, M. and Fallside, S. (1988). An Information-Theoretic Approach to Unsupervised Connectionist Models. In Touretzky, D., Hinton, G., and Sejnowski, T., editors, Proceedings of the 1988 Connectionists Models Summer School, pages 239–245. San Mateo, CA: Morgan Kaufman. Silverman, H. F., Patterson, W. R., and Flanagan, J. L. (1998). The Huge Microphone Array. IEEE Concurrency, pages 36–46. Siracusa, M., Morency, L.-P., Wilson, K., Fisher, J., and Darrell, T. (2003). A Multi-Modal Approach for Determining Speaker Location and Focus. In Proceedings of the Fifth International Conference on Multimodal Interfaces
180
Advances in Natural Multimodal Dialogue Systems
(ICMI), pages 77–80, Vancouver, Canada. http://www.ai.mit.edu/projects/vip/papers/Siracusa icmi2003.pdf. Slaney, M. and Covell, M. (2000). FaceSync: A linear Operator for Measuring Synchronization of Video Facial Images and Audio Tracks. In Leen, T. K., Dietterich, T. G., and Tresp, V., editors, Advances in Neural Information Processing Systems (NIPS), volume 13, pages 814–820. MIT Press. van Veen, B. D. and Buckley, K. M. (1988). Beamforming: A Versatile Approach to Spatial Filtering. IEEE Acoustics, Speech, and Signal Processing (ASSP) Magazine, 5(2):4–24. Viberg, M. and Krim, H. (1997). Two Decades of Statistical Array Processing. In Proceedings of the 31st Asilomar Conference on Signals, Systems, and Computers, volume 1, pages 775–777. Wang, C. and Brandstein, M. (1999). Multi-Source Face Tracking with Audio and Visual Data. In Proceedings of the IEEE International Workshop on Multimedia Signal Processing, pages 169–174, Copenhagen, Denmark. Wolff, G., Prasad, K. V., Stork, D. G., and Hennecke, M. (1994). Lipreading by Neural Networks: Visual Preprocessing, Learning and Sensory Integration. In Cowan, J., Tesauro, G., and Alspector, J., editors, Proceedings of Neural Information Processing Systems (NIPS-6), pages 1027–1034. Zue, V., Glass, J., Polifroni, J., Pao, C., Hazen, T., and Hetherington, L. (2000). Jupiter: A Telephone-Based Conversational Interface for Weather Information. IEEE Transactions on Speech and Audio Processing, 8(1):85–96.
PART III
ANIMATED TALKING HEADS AND EVALUATION
Chapter 9 THE PSYCHOLOGY AND TECHNOLOGY OF TALKING HEADS: APPLICATIONS IN LANGUAGE LEARNING Dominic W. Massaro Perceptual Science Laboratory Department of Psychology, University of California Santa Cruz, CA 95064, USA [email protected]
Abstract
Given the value of visible speech, our persistent goal has been to develop, evaluate, and apply animated agents to produce accurate visible speech. The goal of our recent research has been to increase the number of agents and to improve the accuracy of visible speech. Perceptual tests indicted positive results of this work. Given this technology and the framework of the fuzzy logical model of perception (FLMP), we have developed computer-assisted speech and language tutors for deaf, hard of hearing, and autistic children. Baldi1 , as the conversational agent, guides students through a variety of exercises designed to teach vocabulary and grammar, to improve speech articulation, and to develop linguistic and phonological awareness. The results indicate that the psychology and technology of Baldi holds great promise in language learning and speech therapy.
Keywords:
Facial animation, visible speech, language learning, speech perception, vocabulary tutor, autism, children with language challenges.
1.
Introduction
The face presents visual information during speech that is critically important for effective communication. While the auditory signal alone is adequate for communication, visual information from movements of the lips, tongue and jaws enhance intelligibility of the acoustic stimulus (particularly in noisy en-
1 Baldi R
is a registered trademark of Dominic W. Massaro.
183 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 183–214. © 2005 Springer. Printed in the Netherlands.
184
Advances in Natural Multimodal Dialogue Systems
vironments). Moreover, speech is enriched by the facial expressions, emotions and gestures produced by a speaker [Massaro, 1998]. The visual components of speech offer a lifeline to those with severe or profound hearing loss. Even for individuals who hear well, these visible aspects of speech are especially important in noisy environments. For individuals with severe or profound hearing loss, understanding visible speech can make the difference in effectively communicating orally with others or a life of relative isolation from oral society [Trychin, 1997]. Our persistent goal has been to develop, evaluate, and apply animated agents to produce accurate visible speech. These agents have a tremendous potential to benefit virtually all individuals, but especially those with hearing problems (> 28,000,000 in the USA), including the millions of people who acquire agerelated hearing loss every year (http://www.nidcd.nih.gov/health/hb.htm), and for whom visible speech takes on increasing importance. One of many applications of animated characters allows the training of individuals with hearing loss to "read" visible speech, and thus facilitate face-to-face oral communication in all situations (educational, social, work-related, etc). These enhanced characters can also function effectively as language tutors, reading tutors, or personal agents in human machine interaction. For the past ten years, my colleagues and I have been improving the accuracy of visible speech produced by an animated talking face - Baldi (Figure 9.1; [Massaro, 1998, chapters 12-14]). Baldi has been used effectively to teach vocabulary to profoundly deaf children at Tucker-Maxon Oral School in a project funded by an NSF Challenge Grant [Barker, 2003; Massaro, 2000]. The same pedagogy and technology has been employed for language learning with autistic children [Massaro et al., 2003]. While Baldi’s visible speech and tongue model probably represent the best of the state of the art in real-time visible speech synthesis by a talking face, experiments have shown that Baldi’s visible speech is not as effective as human faces. Preliminary observations strongly suggest that the specific segmental and prosodic characteristics are not defined optimally. One of our goals, therefore, is to significantly improve the communicative effectiveness of synthetic visual speech.
2.
Facial Animation and Visible Speech Synthesis
Visible speech synthesis is a sub-field of the general areas of speech synthesis and computer facial animation (Chapter 12 in [Massaro, 1998] organizes the representative work that has been done in this area). The goal of the visible speech synthesis in the Perceptual Science Laboratory (PSL) has been to develop a polygon (wireframe) model with realistic motions (but not to duplicate the musculature of the face to control this mask). We call this technique terminal analogue synthesis because its goal is to simply use the final speech
The Psychology and Technology of Talking Heads
185
Figure 9.1. Baldi, a computer-animated talking head, in normal and wireframe presentations, and a close-up of the tongue.
product to control the facial articulation of speech (rather than illustrate the physiological mechanisms that produce it). This method of rendering visible speech synthesis has also proven most successful with audible speech synthesis. One advantage of the terminal analogue synthesis is that calculations of the changing surface shapes in the polygon models can be carried out much faster than those for muscle and tissue simulations. For example, our software can generate a talking face in real time on a commodity PC, whereas muscle and tissue simulations are usually too computationally intensive to perform in real time [Massaro, 1998]. More recently, image synthesis, which joins together images of a real speaker, has been gaining in popularity because of the realism that it provides. These systems also are not capable of real-time synthesis because of their computational intensity. Finally, performance-based synthesis, e.g., [Guenter et al., 1998], does not have the flexibility of saying anything at any time in real time, as does our text-to-speech system. Our own current software [Cohen and Massaro, 1993; Cohen et al., 1996; Cohen et al., 1998; Massaro, 1998] is a descendant of Parke’s software and his particular 3-D talking head [Parke, 1975]. Our modifications over the last 8 years have included increased resolution of the model, additional and modified control parameters, three generations of a tongue (which was lacking in Parke’s model), a new visual speech synthesis coarticulatory control strategy, controls for paralinguistic information and affect in the face, alignment with natural speech, text-to-speech synthesis, and bimodal (auditory/visual) synthesis. Most of our current parameters move vertices (and the polygons formed from these vertices) on the face by geometric functions such as rotation (e.g. jaw rotation) or translation of the vertices in one or more dimensions (e.g., lower and upper lip height, mouth widening). Other parameters work
186
Advances in Natural Multimodal Dialogue Systems
by scaling and interpolating different face subareas. Many of the face shape parameters–such as cheek, neck, or forehead shape, and also some affect parameters such as smiling–use interpolation. We have used phonemes as the basic unit of speech synthesis. In this scheme, any utterance can be represented as a string of successive phonemes, and each phoneme is represented as a set of target values for the control parameters such as jaw rotation, mouth width, etc. Because speech production is a continuous process involving movements of different articulators (e.g., tongue, lips, jaw) having mass and inertia, phoneme utterances are influenced by the context in which they occur by a process called coarticulation. In our visual speech synthesis algorithm [Cohen and Massaro, 1993] [Massaro, 1998, chapter 12], coarticulation is based on a model of speech production using rules that describe the relative dominance of the characteristics of the speech segments. In our model, each segment is specified by a target value for each facial control parameter. For each control parameter of a speech segment, there are also temporal dominance functions dictating the influence of that segment over the control parameter. These dominance functions determine independently for each control parameter how much weight its target value carries against those of neighbouring segments, which will in turn determine the final control values. Our animated face can be aligned with either the output of a speech synthesizer or natural auditory speech [Massaro et al., 2005]. We have also developed the phoneme set and the corresponding target and coarticulation values to allow synthesis of several other languages. These include Spanish (Baldero), Italian (Baldini, [Cosi et al., 2002]), Mandarin (Bao), Arabic (Badr, [Ouni et al., 2003]), French (Balduin) and German (Balthasar). Baldi, can be seen at: http://mambo.ucsc.edu. Baldi’s synthetic tongue is constructed of a polygon surface defined by sagittal and coronal b-spline curves (see Figure 9.1). The control points of these b-spline curves are controlled singly and in pairs by speech articulation control parameters. There are now 9 sagittal and 3 * 7 coronal parameters that are modified to mimic natural tongue movements. The tongue, teeth, and palate interactions during speaking require an algorithm to prevent the tongue from going into rather than colliding with the teeth and palate. To ensure this, we have developed a fast collision detection method to instantiate the appropriate interactions. Two sets of observations of real talkers have been used to inform the appropriate movements of the tongue. These include 1) three dimensional ultrasound measurements of upper tongue surfaces and 2) EPG data collected from a natural talker using a plastic palate insert that incorporates a grid of about a hundred electrodes that detect contact between the tongue and palate at a fast rate (e.g. a full set of measurements 100 times per second). These measurements were made in collaboration with Maureen Stone at John Hop-
The Psychology and Technology of Talking Heads
187
Figure 9.2. High-resolution texture (left) and a high-resolution polygon mesh (right) obtained from a Cyberware laser scan.
kins University. Minimization and optimization routines are used to create animated tongue movements that mimic the observed tongue movements [Cohen et al., 1998].
2.1
Recent Progress in Visible Speech Synthesis
Important goals for the application of talking heads are to have a large gallery of possible agents and to have highly intelligible and realistic synthetic visible speech. Our development of visible speech synthesis is based on facial animation of a single canonical face, called Baldi (see Figure 9.1; [Massaro, 1998]. Although the synthesis, parameter control, coarticulation scheme, and rendering engine are specific to Baldi, we have developed software to reshape our canonical face to match various target facial models. To achieve realistic and accurate synthesis, we use measurements of facial, lip, and tongue movements during speech production to optimize both the static and dynamic accuracy of the visible speech. This optimization process is called minimization because we seek to minimize the error between the empirical observations of real human speech and the speech produced by our synthetic talker [Cohen et al., 1998; Cohen et al., 2001; Cohen et al., 2002].
2.2
Improving the Static Model
A Cyberware 3D laser scanning system is used to enroll new citizens in our gallery of talking heads. To illustrate this procedure, we describe how a Cyberware laser scan of DWM was made, how Baldi’s generic morphology was mapped into the form of DWM, how this head was trained on real data, and how the quality of its speech was evaluated. A laser scan of a new target head produces a very high polygon count representation. Figure 9.2 shows a high-
188
Advances in Natural Multimodal Dialogue Systems
Figure 9.3. Laser scan of high resolution head and canonical Baldi low resolution head with alignment points.
resolution texture mapped Cyberware scan of DWM and the accompanying high-resolution mesh. Rather than trying to animate this high-resolution head (which is impossible to do in real-time with current hardware), our software uses these data to reshape our canonical head to take on the shape of the new target head. In this approach, a human operator marks corresponding facial landmarks on both the laser scan head and the generic Baldi head (Figure 9.3). Our canonical head is then warped until it assumes as closely as possible the shape of the target head, with the additional constraint that the landmarks of the canonical face move to positions corresponding to those on the target face. This morphing algorithm is based on the work of Kent, Carlson, and Parent [1992]. In this approach, all the triangles making up the source and target models are projected on a unit sphere centred at the origin. The models must be convex or star shaped so that there is at least one point within the model from where all vertices of all triangles are visible. This can be confirmed by a separate vertex visibility test procedure that checks for this requirement. If a model is non-convex or non-star shaped, then it may be necessary to ignore or modify these sections of the model. In order to meet this requirement, portions of the ears, eyes, and lips are handled separately from the rest of Baldi’s head. For the main portion of the head, we first translate all vertices so that the centre point of the model coincides with the coordinate system origin. We then move the vertices so that they are at a unit distance from the origin. At this point, the vertices of the triangles making up the model are on the surface of the unit sphere. This is done to both Baldi’s source head and the Cyberware laser scan target head. The landmarks are then connected into a mesh of their
The Psychology and Technology of Talking Heads
189
own. As these landmarks are moved into their new positions, the non-landmark points contained in triangles defined by the landmark points are moved to keep their relative positions within the landmark triangles. Then, for each of these source vertices we determine the location on the target model to which a given source vertex projects. This gives us a homeomorphic mapping (1 to 1 and onto) between source and target datasets, and we can thereby determine the morph coordinate of each source vertex as a barycentric coordinate of the target triangle to which it maps. This mapping guides the final morph between the source and target datasets. A different technique is used to interpolate polygon patches, which were earlier culled out of the target model on account of being non-convex. These patches are instead stretched to fit the new boundaries of the culled regions in the morphed head. Because this technique does not capture as much of the target shape’s detail as our main method of interpolation, we try to minimize the size of the patches that are culled in this manner. To output the final topology the program then reconnects all the source polygonal patches and outputs them in a single topology file. The source connectivity is not disturbed and is the same as the original source connectivity.
2.3
Improving the Dynamic Model
To improve the intelligibility of our talking heads, we have developed software for using dynamic 3D optical measurements (Optotrak) of points on a real face while talking. In one study, we recorded a large speech database with 19 markers affixed to the face of DWM at important locations [Cohen et al., 2002]. Fitting of these dynamic data occurred in several stages. To begin, we assigned points on the surface of the synthetic model that best correspond to the Optotrak measurement points. In the training, the Optotrak data were adjusted in rotation, translation, and scale to best match the corresponding points marked on the synthetic face. The data collected for the training consisted of 100 CID sentences recorded by DWM speaking in a fairly natural manner. In the first stage fit, for each time frame (30 fps) we automatically and iteratively adjusted 11 facial control parameters of the face to get the best fit (the least sum of squared distances) between the Optotrak measurements and the corresponding point locations on the synthetic face. In the second stage fit, the goal was to tune the segment definitions (parameter targets, dominance function strengths, attack and decay rates, and peak strength time offsets) used in our coarticulation algorithm [Cohen and Massaro, 1993] to get the best fit with the parameter tracks obtained in the first stage fit. We first used Viterbi alignment on the acoustic speech data of each sentence to obtain the phoneme durations used to synthesize each
190
Advances in Natural Multimodal Dialogue Systems
Figure 9.4. Proportion words correct as a function of initial consonant of all words in the test sentences for auditory alone, synthetic and real face conditions.
sentence. Given the phonemes and durations, we used our standard parametric phoneme synthesis and coarticulation algorithm to synthesize the parameter tracks for all 100 CID sentences. These were compared with the parameter tracks obtained from the first stage fit, the error computed, and the parameters adjusted until the best fit was achieved.
2.4
Perceptual Evaluation
We carried out a perceptual recognition experiment with human subjects to evaluate how well this improved synthetic talker conveyed speech information relative to the real talker. To do this we presented the 100 CID sentences in three conditions: auditory alone, auditory + synthetic talker, and auditory + real talker. In all cases there was white (speech band) noise added to the audio channel. Each of the 100 CID sentences was presented in each of the three modalities for a total of 300 trials. Each trial began with the presentation of the sentence, and subjects then typed in as many words as they could recognize. Students in an introductory psychology course served as subjects. Figure 9.4 shows the proportion of correct words reported as a function of the initial consonant under the three presentation conditions. There was a significant advantage of having the visible speech, and the advantage of the synthetic head was equivalent to the original video of the real face. Overall, the proportion of correctly reported words for the three conditions was 0.22 auditory, 0.43 synthetic face, and 0.42 with the real face. The results of the current evaluation study, using the stage 1 best fitting parameters is encouraging. In studies to follow, we’ll be comparing performance
The Psychology and Technology of Talking Heads
191
with visual TTS synthesis based on the segment definitions from the stage 2 fits, both for single segments, context sensitive segments, and also using concatenation of diphone sized chunks from the stage 1 fits. In addition, we will be using a higher resolution canonical head with many additional polygons and an improved texture map.
3.
Speech Science
Speech science evolved as the study of a unimodal phenomenon. Speech was viewed as a solely auditory event, as captured by the seminal speech-chain illustration shown in [Denes and Pinson, 1963]. This view is no longer viable as witnessed by a burgeoning record of research findings. Speech as a multimodal phenomenon is supported by experiments indicating that our perception and understanding are influenced by a speaker’s face and accompanying gestures, as well as the actual sound of the speech. Many communication environments involve a noisy auditory channel, which degrades speech perception and recognition. Visible speech from the talker’s face improves intelligibility in these situations. Visible speech also is an important communication channel for individuals with hearing loss and others with specific deficits in processing auditory information. The number of words understood from a degraded auditory message can often be doubled by pairing the message with visible speech from the talker’s face [Jesse et al., 2001]. The combination of auditory and visual speech has been called super-additive because their combination can lead to accuracy that is much greater than accuracy on either modality alone. Furthermore, the strong influence of visible speech is not limited to situations with degraded auditory input. A perceiver’s recognition of an auditory-visual syllable reflects the contribution of both sound and sight. For example, if the ambiguous auditory sentence, My bab pop me poo brive, is paired with the visible sentence, My gag kok me koo grive, the perceiver is likely to hear, My dad taught me to drive. Two ambiguous sources of information are combined to create a meaningful interpretation [Massaro, 1998].
3.1
Value of Multimodal Speech
There are several reasons why the use of auditory and visual information together in face to face interactions is so successful. These include a) robustness of visual speech, b) complementarity of auditory and visual speech, and c) optimal integration of these two sources of information. Speech reading, or the ability to obtain speech information from the face, is robust in that perceivers are fairly good at speech reading even when they are not looking directly at the talker’s lips. Furthermore, accuracy is not dramatically reduced when the facial image is blurred (because of poor vision, for example), when the face
192
Advances in Natural Multimodal Dialogue Systems
is viewed from above, below, or in profile, or when there is a large distance between the talker and the viewer [Massaro, 1998, Chapter 14]. Complementarity of auditory and visual information simply means that one of the sources is strong when the other is weak. A distinction between two segments robustly conveyed in one modality is relatively ambiguous in the other modality. For example, the place difference between /ba/ and /da/ is easy to see but relatively difficult to hear. On the other hand, the voicing difference between /ba/ and /pa/ is relatively easy to hear but very difficult to discriminate visually. Two complementary sources of information make their combined use much more informative than would be the case if the two sources were noncomplementary, or redundant [Massaro, 1998, pages 424–427]. The final reason is that perceivers combine or integrate the auditory and visual sources of information in an optimally efficient manner. It might seem obvious, given the joint influence of audible and visible speech, that these two sources of information are combined or integrated. Integration is not the only process that can account for an advantage of two sources of information relative to just one, however. There are many possible ways to treat two sources of information: use only the most informative source; use only the auditory source if it is identified and if it isn’t use the visible source; average the two sources together; or integrate them in such a fashion in which both sources are used but that the least ambiguous source has the most influence. Research has shown that perceivers do in fact integrate the information available from each modality to perform as efficiently as possible. We now describe a model that predicts this optimally efficient process of combination [Massaro, 1998].
3.2
Fuzzy Logical Model of Perception
The fuzzy logical model of perception (FLMP), shown in Figure 9.5, assumes necessarily successive but overlapping stages of processing. The perceiver of speech is viewed as having multiple sources of information supporting the identification and interpretation of the language input. The model assumes that 1) each source of information is evaluated to give the continuous degree to which that source supports various alternatives, 2) the sources of information are evaluated independently of one another, 3) the sources are integrated to provide an overall degree of support for each alternative, and 4) perceptual identification and interpretation follows the relative degree of support among the alternatives [Massaro et al., 2001; Massaro, 2002]. The paradigm that we have developed permits us to determine how visible speech is processed and integrated with other sources of information. The results also inform us about which of the many potentially functional cues are actually used by human observers [Massaro, 1987, Chapter 1]. The systematic variation of properties of the speech signal combined with the quantitative test
193
The Psychology and Technology of Talking Heads Ai Vj
Evaluation ai
vj
Integration sk
Decision
Rk
Figure 9.5. Schematic representation of the three processes involved in perceptual recognition. The three processes are shown to proceed left to right in time to illustrate their necessarily successive but overlapping processing. These processes make use of prototypes stored in long-term memory. The sources of information are represented by uppercase letters. Auditory information is represented by Ai and visual information by Vj . The evaluation process transforms these sources of information into psychological values (indicated by lowercase letters ai and vj ) These sources are then integrated to give an overall degree of support, sk , for each speech alternative k. The decision operation maps the outputs of integration into some response alternative, Rk . The response can take the form of a discrete decision or a rating of the degree to which the alternative is likely.
of models of speech perception enables the investigator to test the psychological validity of different cues. This paradigm has already proven to be effective in the study of audible, visible, and bimodal speech perception [Massaro, 1987; Massaro, 1998]. Thus, our research strategy not only addresses how different sources of information are evaluated and integrated, but can uncover what sources of information are actually used. We believe that the research paradigm confronts both the important psychophysical question of the nature of information and the process question of how the information is transformed and mapped into behaviour. Many independent tests point to the viability of the FLMP as a general description of pattern recognition. The FLMP is centred around a universal law of how people integrate multiple sources of information. This law and its relationship to other laws is developed in detail in [Massaro, 1998]. The FLMP is also valuable because it motivates our approach to language learning. Baldi can display a midsagital view, or the skin on the face can be made transparent to reveal the internal articulators. The orientation of the face can be changed to display different viewpoints while speaking, such as a side view, or a view from the back of the head [Massaro, 1999; Massaro, 2000]. The auditory and visual speech can also be independently controlled and manipulated, permitting customized enhancements of the informative characteristics of speech. These features offer novel approaches in language training, permitting one to pedagogically illustrate appropriate articulations that are usually
194
Advances in Natural Multimodal Dialogue Systems
hidden by the face. This technology has the potential to help individuals with language delays and deficits, and we have been utilizing Baldi to carry out language tutoring with hard of hearing children and children with autism.
4.
Language Learning
As with most issues in social science, there is no consensus on the best way to teach or to learn language. There is agreement, however, about the importance of time on task; learning and retention are positively correlated with the time spent learning. Our technology offers a platform for unlimited instruction, which can be initiated when and wherever the child and/or supervisor chooses. Baldi and the accompanying lessons are perpetual. Take, for example, children with autism, who have irregular sleep patterns. A child could conceivably wake in the middle of the night and participate in language learning with Baldi as his or her friendly guide. Several advantages of utilizing a computer-animated agent as a language tutor are clear, including the popularity of computers and embodied conversational agents with children with autism. A second advantage is the availability of the program. Instruction is always available to the child, 24 hours a day 365 days a year. Furthermore, instruction occurs in a one-on-one learning environment for the students. We have found that the students enjoy working with Baldi because he offers extreme patience, he doesn’t become angry, tired, or bored, and he is in effect a perpetual teaching machine.
4.1
Vocabulary Learning
Vocabulary knowledge is critically important for understanding the world and for language competence in both spoken language and in reading. There is empirical evidence that very young children more easily form conceptual categories when category labels are available than when they are not [Waxman, 2002]. There is also evidence that there is a sudden increase in the rate at which new words are learned once the child knows about 150 words. Grammatical skill also emerges at this time. Even children experiencing language delays because of specific language impairment benefit once this level of word knowledge is obtained. It follows that increasing the pervasiveness and effectiveness of vocabulary learning offers a huge opportunity for improving conceptual knowledge and language competence for all individuals, whether or not they are disadvantaged because of sensory limitations, learning disabilities, or social condition. Finally, it is well-known that vocabulary knowledge is positively correlated with both listening and reading comprehension [Anderson and Freebody, 1981]. Our Language Tutor, Baldi, encompasses and instantiates the developments in the pedagogy of how language is learned, remembered and used. Educa-
The Psychology and Technology of Talking Heads
195
tion research has shown that children can be taught new word meanings by using direct instruction, e.g., [McKeown et al., 1985; Stahl, 1986]. It has also been convincing demonstrated that direct teaching of vocabulary by computer software is possible, and that an interactive multimedia environment is ideally suited for this learning [Berninger and Richards, 2002; Wood, 2001]. As cogently observed by Wood [2001], “Products that emphasize multimodal learning, often by combining many of the features discussed above, perhaps make the greatest contribution to dynamic vocabulary learning. Multimodal features not only help keep children actively engaged in their own learning, but also accommodate a range of learning styles by offering several entry points: When children can see new words in context, hear them pronounced, type them into a journal, and cut and paste an accompanying illustration (or create their own), the potential for learning can be dramatically increased.” Following this logic, many aspects of our lessons enhance and reinforce learning. For example, the existing program and planned modifications make it possible for the student to 1) Observe the words being spoken by a realistic talking interlocutor (Baldi), 2) See the word as written as well as spoken, 3) See visual images of referents of the words or view an animation of a meaningful scene, 4) Click on or point to the referent, 5) Hear himself or herself say the word, 6) Spell the word by typing, 7) Observe the word used in context, and 8) Incorporate the word into his or her own speech act. Other benefits of our program include the ability to seamlessly meld spoken and written language, provide a semblance of a game-playing experience while actually learning, and to lead the child along a growth path that always bridges his or her current “zone of proximal development.”
4.2
Description of Language Wizard and Tutor
The Language Tutor and Wizard is a user-friendly application that allows the presentation and composition of lessons with minimal computer experience [Bosseler and Massaro, 2003; Barker, 2003].2 The lessons encompass and instantiate the developments in the pedagogy of how language is learned, remembered and used. Figure 9.6 shows a view of the screen from the Presentation exercise in a prototypical lesson. In this lesson, the students learn to identify vegetables. In this exercise, text corresponding to each item is presented when the item is tutored. An outlined region around the zucchini designates the selected object. Emoticon “stickers” (not shown) can also be used as feedback for the responses. 2 The
development of this application was carried out in collaboration with the Center for Spoken Language Understanding at the Oregon Health Sciences University and the Tucker Maxon Oral School, both in Portland, Oregon.
196
Advances in Natural Multimodal Dialogue Systems
Figure 9.6. A prototypical lesson illustrating the format of the Language Tutor. Each lesson contains Baldi, the vocabulary items and written text and captioning (optional, not shown), and emoticons (not shown). In this application the students learn to identify vegetables. For example, Baldi says "this is a zucchini" in the Presentation exercise.
All of the exercises required the children to respond to spoken directives such as “click on the little chair”, or “find the red fox”. These images were associated with the corresponding spoken vocabulary words, see [Bosseler and Massaro, 2003] for vocabulary examples). The items became highlighted whenever the mouse passed over that region. The student selected his or her response by clicking the mouse on one of the designated areas. The Language Wizard consists of 8 different exercises. These exercises are pre-test, presentation, recognition, reading, spelling, imitation, elicitation, and post-test. The Wizard is equipped with easily changeable default settings that determine what Baldi says and how he says it, the oral feedback and emoticons given for responses, the number of attempts permitted for the student in each exercise, and the number of times each item is presented. The program automatically creates and writes all student performance information to a log file stored in the student’s directory.
The Psychology and Technology of Talking Heads
5.
197
Research on the Educational Impact of Animated Tutors
Research has shown that this pedagogical and technological program is highly effective for both children with hearing loss and children with autism. Processing information presented via the visual modality reinforces learning [Courchesne et al., 1994] and is consistent with the TEEACH [Schopler et al., 1995] suggestion for the use of visually presented material. These children tend to have major difficulties in acquiring language, and they serve as particularly challenging tests for the effectiveness of our pedagogy. We now describe some recent research carried out to evaluate our animated tutor to teach both children with hearing loss [Barker, 2003; Massaro et al., 2003] and children with autism [Bosseler and Massaro, 2003].
5.1
Improving the Vocabulary of Hard of Hearing Children
It is well-known that hard of hearing children have significant deficits in vocabulary knowledge. In many cases, the children do not have names for specific things and concepts. These children often communicate with phrases such as “the window in the front of the car,” “the big shelf where the sink is,” or “the step by the street” rather than “windshield,” “counter,” or “curb” [Barker, 2003, citing Pat Stone]. The Language Tutor has been in use at the Tucker Maxon Oral School in Portland, Oregon, and Barker [2003] evaluated its effectiveness. Students were given cameras to photograph objects at home and surroundings. The pictures of these objects were then incorporated as items in the lessons. A given lesson had between 10 and 15 items. Students worked on the items about 10 minutes a day until they reached 100% on the post-test. They then moved on to another lesson. About one month after each successful (100%) post-test, they were retested on the same items. Ten girls and nine boys the “upper school” and the “lower school” participated in the applications. There were six hard of hearing children and one hearing child between 8 and 10 years of age in the lower school. Ten hard of hearing and two hearing children, between 11 and 14 years of age, participated from the upper school. Similar results were found for both age groups. Students knew about onehalf of the items without any learning, they successfully learned the other half of the items, and retained about one-half of the newly learned items when retested 30 days later. These results demonstrate the effectiveness of the language Tutor for learning and retaining new vocabulary. The results of the Barker evaluation [Barker, 2003] indicated that hard of hearing children learned a significant number of new words, and retained about half of them a month after training ended. No control groups were used in that evaluation, however, and it is possible the children were learning the words
198
Advances in Natural Multimodal Dialogue Systems
outside of the Language Tutor environment. Furthermore, the time course of learning with the Language Tutor was not evaluated. It is of interest how quickly words can be learned with the Language Tutor to give some idea of how this learning environment would compare to a real teacher. Finally, both identification and production of the words should be assessed given that only identification was measured previously.
5.2
Testing the Validity of the Vocabulary Tutor
To address these issues, Massaro and Light [2004a] carried out an experiment based on a within student multiple baseline design [Baer et al., 1968] where certain words were continuously being tested while other words were being tested and trained. Although the student’s instructors and speech therapists agreed not to teach or use these words during our investigation, it is still possible that the words could be learned outside of the Language Tutor environment. The single student multiple baseline design monitors this possibility by providing a continuous measure of the knowledge of words that are not being trained. Thus, any significant differences in performance on the trained words and untrained words can be attributed to the Language Tutor training program itself rather than some other factor. Eight hard of hearing children, 2 male ages 6 and 7, 6 female ages 9 and 10, were recruited from The Jackson Hearing Center in Los Altos, California and were given parental consent to participate. The male students were in grade 1 and the female students in grade 4 respectively and all students needed help with their vocabulary building skills as suggested by their regular day teachers. One child had a cochlear implant and the seven other children had hearing aids in both ears except for one child with an aid in just a single ear. Using the Language Wizard, the experimenter developed a set of lessons with a collection of vocabulary items that was individually tailored for each student. Each collection of items was comprised of 24 items, broken down into 3 categories of 8 items each. Three lessons with 8 items each were made for each child. Images of the vocabulary items were presented on the screen next to Baldi as he spoke, as illustrated in Figure 9.6. Some of the exercises required the child to respond to Baldi’s instructions such as “click on the cabbage”, or “show me the yam”, by clicking on the highlighted area or by moving the computer mouse over the appropriate image until an item was highlighted and then clicking on it. Two other exercises asked the child to recognize the written word and to type the word, respectively. The production exercises asked the child to repeat after Baldi once he named the highlighted image or to name the highlighted image on their own, followed by Baldi’s naming of the image. Figure 9.7 gives the results of identification and production for one of the eight students. The results were highly consistent across the eight students.
199
The Psychology and Technology of Talking Heads S tu d e n t 8 Pr e tr ain in g
T r ain in g
Po s ttr ain in g
Pr o p o r tio n C o r r e ct
1 0.8 0.6 0.4 0.2
S et 1 0
p r o p o r tio n co r r e ct
1 0.8 0.6 identif ic ation 0.4
produc tion
0.2
S et 2
0
1 0 .8 0 .6 0 .4 0 .2
S et 3 0 1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18
S e s s ion
Figure 9.7. Proportion of correctly identified (solid black diamonds) and correctly produced (empty white squares) items across the testing sessions for student 1. The training on a set of words occurred between the two vertical bars. The figure illustrates that once training was implemented identification performance increased dramatically, and remained accurate without further training (from [Massaro and Light, 2004a]).
As expected, identification accuracy (mean = .72) was always higher than production accuracy (mean = .64). This result is not unexpected because a stu-
200
Advances in Natural Multimodal Dialogue Systems
dent could know the name of an item without being able to pronounce it correctly. There was little knowledge of the test items shown without training, even though these items were repeatedly tested for many days. Once training began on a set of items, however, performance improved fairly quickly until asymptotic knowledge was obtained. This knowledge did not degrade after training on these words ended and training on other words took place. In addition, a reassessment test given about 4 weeks after completion of the experiment revealed that the students retained the items that were learned. The average number of trials required to reach criterion was 5, 4.3, and 3.4 for mastering the first, second, and third sets of categories. Given that the word lists were randomized across participants, differences in the difficulty of the word sets was probably not responsible for this difference. Learning vocabulary actually involves three different things to be learned. Stimulus learning involves recognizing the stimulus, response leaning requires acquiring the appropriate response, and stimulus-response learning requires an association of the stimulus and the response. The testing of the items actually gave the students experience that could contribute to both stimulus learning and response learning. Thus when the items were finally trained, the students only had to master the stimulus-response association. This would give an advantage of learning the second and third sets of words relative to the first and second, once training was initiated.
5.3
Improving the Vocabulary of Autistic Children
Autism is a spectrum disorder characterized by a variety of characteristics, which usually include perceptual, cognitive, and social differences. Among the defining characteristics of autism, the limited ability to produce and comprehend spoken language is the most common factor leading to diagnosis [American Psychiatric Association, 1994]. The language and communicative deficits extend across a broad range of expression [Tager-Flusberg, 1999]. Individual variations occur in the degree to which these children develop the fundamental lexical, semantic, syntactic, phonological, and pragmatic components of language including those who fail to develop one or more of these elements of language comprehension and production. Approximately one-half of the autistic population fails to develop any form of functional language [Tager-Flusberg, 2000]. Within the population that does develop language, the onset and rate at which the children pass through linguistic milestones are often delayed compared to non-autistic children (e.g. no single words by age 2 years, no communicative phrases by age 3) [American Psychiatric Association, 1994]. The ability to label objects is often severely delayed in this population as well as the deviant use and knowledge of verbs and adjectives. Van Lancker et al. [1991] investigated the abilities of autistic
The Psychology and Technology of Talking Heads
201
and schizophrenic children to identify concrete nouns, non-emotional adjectives, and emotional adjectives. The results showed that the performance of children with autism was below controls in all three areas. Despite the prevalence of language delays in autistic individuals, formalized research has been limited, partly due to the social challenges inherent in this population [Tager-Flusberg, 2000]. Intervention programs for children with autism typically emphasize developing speech and communication skills (e.g. TEEACH, Applied Behavioural Analysis). These programs most often focus on the fundamental lexical, semantic, syntactic, phonological, and pragmatic components of language. The behavioural difficulties speech therapists and instructors encounter, such as lack of cooperation, aggression, and lack of motivation to communicate, create difficult situations that are not optimal for learning. Thus, creating motivational environments necessary to develop these language skills introduces many inherent obstacles [Tager-Flusberg, 2000]. In this study [Bosseler and Massaro, 2003], the Tutors were constructed and run on a 600 MHz PC with 128 MB RAM hard drive running Microsoft Windows NT 4 with a Gforce 256 AGP-V6800 DDR graphics board. The tutorials were presented on a Graphic Series view Sonic 20” monitor. All students wore a Plantronics PC Headset model SR1. Students completed 2 sessions a week, a minimum of 2 lessons per session, and an average of 3, and sometimes as many as 8. The sessions lasted between 10 and 40 minutes. A total of 559 different vocabulary items were selected from the curriculum of both schools for a total of over 84 unique vocabulary lessons. A series of observations by the experimenter during the course of each lesson led to many changes in the program, including the use of headsets, isolating the student from the rest of the class and removal of negative verbal feedback from Baldi (such as, “No, (student) that’s not right”. The students appeared to enjoy working with Baldi. We documented the children saying such things as “Hi Baldi” and “I love you Baldi”. The stickers generated for correct (happy face) and incorrect (sad face) responses proved to be an effective way to provide feedback for the children, although some students displayed frustration when he or she received more than one sad face. The children responded to the happy faces by saying such things like “Look, I got them all right”, or laughing when a happy face appeared. We also observed the students providing verbal praise to themselves such as “Good job”, or prompting the experimenter to say “Good job” after every response. For the autistic children, several hundred vocabulary tutors were constructed, consisting of various vocabulary items selected from the curriculum of two schools. The children were administered the tutorial lessons until 100% accuracy was attained on the post-test exercise. Once 100% accuracy was attained on the final post-test module, the child did not see these lessons again until reassessment approximately 30 days later.
202
Advances in Natural Multimodal Dialogue Systems
Number of Vocabulary Items
Evaluation of Language Wizard/Player 100 90 80 70 60 50 40 30 20 10 0
49
0
7
49
42
0 39
39
39
Assessment
Final Post-test
30 days Later
Incorrect Learned known
Condition Figure 9.8. The mean observed proportion of correct identifications for the initial assessment, final posttest and reassessment for each of the seven students. reassessment. The results reveal these seven students were able to accurately identify significantly more words during the reassessment than the initial assessment (from [Bosseler and Massaro, 2003]).
Figure 9.8 shows that the children learned many new words, grammatical constructions, and concepts, proving that the language tutors are a valuable learning environment for these children. In order to assess how well the children would retain the vocabulary items that were learned during the tutorial lesson, we administered the assessment test to the student at least 30 days following the final post-test. As can be seen in Figure 9.8, the students were able to recall 85% of the newly-learned vocabulary items at least 30 days following training.
5.4
Validity and Generalization
Although all of the children demonstrated learning from initial assessment to final reassessment, the children might have been learning the words outside of our program, for example, from speech therapists, at home, or in their school curriculum. Furthermore, we questioned whether the vocabulary knowledge would generalize to new pictorial instances of the words. To address these issues we conducted a second experiment. Collaborating with the children’s instructors and speech therapists, we gathered an assortment of vocabulary words that the children supposedly did not know. We used these words in the Horner
The Psychology and Technology of Talking Heads
203
and Baer [1978] single subject multiple probe design. We randomly separated the words to be trained into three sets, established individual pre-training performance for each set of vocabulary items, and trained on the first set of words while probing performance for both the trained and untrained sets of words. Once the student was able to attain 100% identification accuracy during a training session, a generalization probe to new instances of the vocabulary images was initiated. If the child did not meet the criterion, he or she was trained on these new images. Generalization training continued until the criterion was met, at which time training began on the next set of words. Probe tests continued on the original learned set of words and images until the end of the study. We continued this procedure until the student completed training on all three sets of words. Our goal was to observe a significant increase in identification accuracy during the post-training sessions relative to the pre-training sessions. Figure 9.9 displays the proportion of correct responses for a typical student during the probe sessions conducted at pre-training and post-training for each of the three word sets. The vertical lines in each of the three panels indicates the last pre-training session before the onset of training. Some of the words were clearly known prior to training, and were even learned to some degree without training. As can be seen in the figure, however, training was necessary for substantial learning to occur. In addition, the children were able to generalize accurate identification to four instances of untrained images. In summary, the goal of these investigations was to evaluate the potential of using a computer-animated talking tutor for children with language delays. The results showed a significant gain in vocabulary. We also found that the students were able to recall much of the new vocabulary when reassessed 30 days after learning. Follow-up research showed that the learning is indeed occurring from the computer program and vocabulary knowledge can transfer to novel images. It should be emphasized that the present studies used a within-subject design (with fewer participants) than the field’s general acceptance of between-subject designs. There are several limitations with between-subject designs, however. First, different groups might differ on a pretest so that differences in the pretestposttest differences might not be a valid dependent measure [Loftus, 1978]. Second, even if subjects do not differ in the pretest, they might differ in their learning ability. Thus group differences might reflect other differences rather than the learning conditions. These potential limitations are less problematic in the within-subject design used in the current research. Furthermore, the multiple baseline design enforces a within-subject comparison in which the effectiveness of the independent variable can be evaluated directly. In our case, a child was continuously tested on items that had not yet been learned, were currently being learned, or had already been learned. The direct comparisons
204
Advances in Natural Multimodal Dialogue Systems
Figure 9.9. Proportion correct during the Pretraining, Posttraining, and Generalization for one of the six students. The vertical lines separate the Pretraining and Posttraining conditions. Generalization results are given by the open squares (from [Bosseler and Massaro, 2003]).
among these conditions showed conclusively that the training program was responsible for the learning. An obvious question is how effective our training program is relative to a live teacher. Given the fairly fast rate of learning for both the hard of hearing and autistic children, the potential advantage of a live teacher cannot be very large. Even if learning with the live teacher is significantly faster, our program is still valuable because it is low cost and always available,
The Psychology and Technology of Talking Heads
5.5
205
Value of the Face for Autistic Children
We believe that the children in our investigation profited from having the face and that seeing and hearing spoken language can better guide language learning than either modality alone. A direct test of this hypothesis would involve comparing learning with and without the face. The purpose of this investigation was to evaluate whether pairing an animated tutor, Baldi, with audible speech in vocabulary training facilitates learning and retention relative to presenting the audible speech alone [Massaro and Bosseler, 2003]. If children with autism do not extract meaningful information from the face, then we would expect to see no difference in learning between the two conditions. This evaluation was carried out using two within-subject experimental conditions: training with the face and the voice and training with the voice only. We evaluated whether the face would increase the rate of learning for both receptive and verbal production measures. To accomplish our goal we compared the two conditions according to an alternating treatment design [Baer et al., 1968; Horner and Baer, 1978] in which each student received each of the two learning conditions concurrently, the order of presentation of the two conditions counterbalanced across days. This alternating treatment design permitted us to assess the individual performance of word identification and production, eliminating inter-subject variability and permitted a direct observation of the two treatment conditions [Baer et al., 1968; Horner and Baer, 1978]. To determine the effect of training on retention, an additional testing block was carried once training was terminated. Figure 9.10 gives the identification results for one of the five students for the pre-training, training Post-test, and post-training blocks. The left and right vertical lines in figure separate these three conditions. As can be seen in the figure, this student showed a very large advantage of having the face during learning. Across all 5 students, there was faster learning and more retention with the face than without the face. In summary, the studies using the Language Wizard/Tutor show that children with language challenges were able to learn a significant amount of new vocabulary. By implementing experimental controls in our evaluation, we were able to conclude that the Language Wizard/Tutor was responsible for this learning. Finally, by systematically varying whether Baldi’s face was present during the language tutoring, we learned that the presence of the face contributed significantly to the language learning.
5.6
Training Speech Production
Baldi can actually provide more information than a natural face. Baldi has a tongue, hard palate and three-dimensional teeth and his internal articulatory movements have been trained with electropala