Advances in Natural Multimodal Dialogue Systems
Text, Speech and Language Technology VOLUME 30
Series Editors Nancy Ide, Vassar College, New York Jean Véronis, Université de Provence and CNRS, France Editorial Board Harald Baayen, Max Planck Institute for Psycholinguistics, The Netherlands Kenneth W. Church, AT & T Bell Labs, New Jersey, USA Judith Klavans, Columbia University, New York, USA David T. Barnard, University of Regina, Canada Dan Tufis, Romanian Academy of Sciences, Romania Joaquim Llisterri, Universitat Autonòma de Barcelona, Spain Stig Johansson, University of Oslo, Norway Joseph Mariani, LIMSI-CNRS, France
The titles published in this series are listed at the end of this volume.
Advances in Natural Multimodal Dialogue Systems Edited by
Jan C.J. van Kuppevelt Waalre, The Netherlands
Laila Dybkj ær University of Southern Denmark, Odense, Denmark
and
Niels Ole Bernsen University of Southern Denmark, Odense, Denmark
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN-10 ISBN-13 ISBN-10 ISBN-13 ISBN-10 ISBN-13
1-4020-3934-4 (PB) 978-1-4020-3934-8 (PB) 1-4020-3932-8 (HB) 978-1-4020-3032-4 (HB) 1-4020-3933-6 (e-book) 978-1-4020-3933-1 (e-book)
Published by Springer, P.O. Box 17, 3300 AA Dordrecht, The Netherlands. www.springer.com
Printed on acid-free paper
All Rights Reserved © 2005 Springer No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written permission from the Publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Printed in the Netherlands
Contents
Preface
xi
1 Natural and Multimodal Interactivity Engineering - Directions and Needs Niels Ole Bernsen and Laila Dybkjaer 1. Introduction 2. Chapter Presentations 3. NMIE Contributions by the Included Chapters 4. Multimodality and Natural Interactivity References
1 1 2 7 16 19
Part I Making Dialogues More Natural: Empirical Work and Applied Theory 2 Social Dialogue with Embodied Conversational Agents Timothy Bickmore and Justine Cassell 1. Introduction 2. Embodied Conversational Agents 3. Social Dialogue 4. Related Work 5. Social Dialogue in REA 6. A Study Comparing ECA Social Dialogue with Audio-Only Social Dialogue 7. Conclusion References 3 A First Experiment in Engagement for Human-Robot Interaction in Hosting Activities Candace L. Sidner and Myroslava Dzikovska 1. Introduction 2. Hosting Activities 3. What is Engagement? 4. First Experiment in Hosting: A Pointing Robot 5. Making Progress on Hosting Behaviours 6. Engagement for Human-Human Interaction 7. Computational Modelling of Human-Human Hosting and Engagement 8. A Next Generation Mel
v
23 23 25 29 32 36 40 48 49 55 55 56 57 59 62 63 70 73
vi
Advances in Natural Multimodal Dialogue Systems 9. Summary References
Part II
74 74
Annotation and Analysis of Multimodal Data: Speech and Gesture
4 FORM Craig H. Martell 1. Introduction 2. Structure of FORM 3. Annotation Graphs 4. Annotation Example 5. Preliminary Inter-Annotator Agreement Results 6. Conclusion: Applications to HLT and HCI? Appendix: Other Tools, Schemes and Methods of Gesture Analysis References 5 On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Environments: Initial Evidence Andrea Corradini and Philip R. Cohen 1. Introduction 2. Study 3. Data Analysis 4. Results 5. Discussion 6. Related Work 7. Future Work 8. Conclusions Appendix: Questionnaire MYST III - EXILE References 6 Analysing Multimodal Communication Patrick G. T. Healey, Marcus Colman and Mike Thirlwell 1. Introduction 2. Breakdown and Repair 3. Analysing Communicative Co-ordination 4. Discussion References 7 Do Oral Messages Help Visual Search? Noëlle Carbonell and Suzanne Kieffer 1. Context and Motivation 2. Methodology and Experimental Set-Up 3. Results: Presentation and Discussion 4. Conclusion References
79 79 80 85 86 88 90 91 95 97 97 99 101 103 106 106 108 108 110 111 113 113 117 125 126 127 131 131 134 141 153 154
Contents 8 Geometric and Statistical Approaches to Audiovisual Segmentation Trevor Darrell, John W. Fisher III, Kevin W. Wilson, and Michael R. Siracusa 1. Introduction 2. Related Work 3. Multimodal Multisensor Domain 4. Results 5. Single Multimodal Sensor Domain 6. Integration References
vii 159 159 160 162 166 167 175 178
Part III Animated Talking Heads and Evaluation 9 The Psychology and Technology of Talking Heads: Applications in Language Learning Dominic W. Massaro 1. Introduction 2. Facial Animation and Visible Speech Synthesis 3. Speech Science 4. Language Learning 5. Research on the Educational Impact of Animated Tutors 6. Summary References 10 Effective Interaction with Talking Animated Agents in Dialogue Systems Björn Granström and David House 1. Introduction 2. The KTH Talking Head 3. Effectiveness in Intelligibility and Information Presentation 4. Effectiveness in Interaction 5. Experimental Applications 6. The Effective Agent as a Language Tutor 7. Experiments and 3D Recordings for the Expressive Agent References 11 Controlling the Gaze of Conversational Agents Dirk Heylen, Ivo van Es, Anton Nijholt and Betsy van Dijk 1. Introduction 2. Functions of Gaze 3. The Experiment 4. Discussion 5. Conclusion References
183 183 184 191 194 197 210 211 215 215 217 219 223 231 235 237 239 245 245 248 252 258 260 260
viii
Advances in Natural Multimodal Dialogue Systems
Part IV Architectures and Technologies for Advanced and Adaptive Multimodal Dialogue Systems 12 MIND: A Context-Based Multimodal Interpretation Framework in Conversational Systems Joyce Y. Chai, Shimei Pan and Michelle X. Zhou 1. Introduction 2. Related Work 3. MIND Overview 4. Example Scenario 5. Semantics-Based Representation 6. Context-Based Multimodal Interpretation 7. Discussion References 13 A General Purpose Architecture for Intelligent Tutoring Systems Brady Clark, Oliver Lemon, Alexander Gruenstein, Elizabeth Owen Bratt, John Fry, Stanley Peters, Heather Pon-Barry, Karl Schultz, Zack Thomsen-Gray and Pucktada Treeratpituk 1. Introduction 2. An Intelligent Tutoring System for Damage Control 3. An Architecture for Multimodal Dialogue Systems 4. Activity Models 5. Dialogue Management Architecture 6. Benefits of ACI for Intelligent Tutoring Systems 7. Conclusion References 14 MIAMM – A Multimodal Dialogue System using Haptics Norbert Reithinger, Dirk Fedeler, Ashwani Kumar, Christoph Lauer, Elsa Pecourt and Laurent Romary 1. Introduction 2. Haptic Interaction in a Multimodal Dialogue System 3. Visual Haptic Interaction – Concepts in MIAMM 4. Dialogue Management 5. The Multimodal Interface Language (MMIL) 6. Conclusion References 15 Adaptive Human-Computer Dialogue Sorin Dusan and James Flanagan 1. Introduction 2. Overview of Language Acquisition 3. Dialogue Systems 4. Language Knowledge Representation 5. Dialogue Adaptation 6. Experiments
265 265 267 267 268 270 277 282 283 287
287 288 294 295 298 301 302 303 307 308 309 313 319 326 331 331 333 333 334 337 340 341 347
Contents 7. Conclusion References
ix 351 353
16 Machine Learning Approaches to Human Dialogue Modelling Yorick Wilks, Nick Webb, Andrea Setzer, Mark Hepple and Roberta Catizone 1. Introduction 2. Modality Independent Dialogue Management 3. Learning to Annotate Utterances 4. Future work: Data Driven Dialogue Discovery 5. Discussion References
355 357 362 366 367 368
Index
371
355
Preface
The chapters in this book jointly contribute to what we shall call the field of natural and multimodal interactive systems engineering. This is not yet a well-established field of research and commercial development but, rather, an emerging one in all respects. It brings together, in a process that, arguably, was bound to happen, contributors from many different, and often far more established, fields of research and industrial development. To mention but a few, these include speech technology, computer graphics and computer vision. The field’s rapid expansion seems driven by a shared vision of the potential of new interactive modalities of information representation and exchange for radically transforming the world of computer systems, networks, devices, applications, etc. from the GUI (graphical user interface) paradigm into something which will enable a far deeper and much more intuitive and natural integration of computer systems into people’s work and lives. Jointly, the chapters present a broad and detailed picture of where natural and multimodal interactive systems engineering stands today. The book is based on selected presentations made at the International Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems held in Copenhagen, Denmark, in 2002 and sponsored by the European CLASS project. CLASS was initiated on the request of the European Commission with the purpose of supporting and stimulating collaboration among Human Language Technology (HLT) projects as well as between HLT projects and relevant projects outside Europe. The purpose of the workshop was to bring together researchers from academia and industry to discuss innovative approaches and challenges in natural and multimodal interactive systems engineering. The Copenhagen 2002 CLASS workshop was not just a very worthwhile event in an emerging field due to the general quality of the papers presented. It was also largely representative of the state of the art in the field. Given the increasing interest in natural interactivity and multimodality and the excellent quality of the work presented, it was felt to be timely to publish a book reflecting recent developments. Sixteen high-quality papers from the workshop were selected for publication. Content-wise, the chapters in this book illustrate most aspects of natural and multimodal interactive systems engineering: applicable
xi
xii
Advances in Natural Multimodal Dialogue Systems
theory, empirical work, data annotation and analysis, enabling technologies, advanced systems, re-usability of components and tools, evaluation, and future visions. The selected papers have all been reviewed, revised, extended, and improved after the workshop. We are convinced that people who work in natural interactive and multimodal dialogue systems engineering – from graduate students and Ph.D. students to experienced researchers and developers, and no matter exactly which community they come from originally - may find this collection of papers interesting and useful to their own work. We would like to express our sincere gratitude to all those who helped us in preparing this book. Especially we would like to thank all reviewers for their valuable and extensive comments and criticism which have helped improve the quality of the individual chapters as well as the entire book. THE EDITORS
Chapter 1 NATURAL AND MULTIMODAL INTERACTIVITY ENGINEERING - DIRECTIONS AND NEEDS Niels Ole Bernsen and Laila Dybkjær Natural Interactive Systems Laboratory University of Southern Denmark Campusvej 55, 5230 Odense M, Denmark
{nob, laila}@nis.sdu.dk Abstract
This introductory chapter discusses the field of natural and multimodal interactivity engineering and presents the following 15 chapters in this context. A brief presentation of each chapter is given, their contributions to specific natural and multimodal interactivity engineering needs are discussed, and the concepts of multimodality and natural interactivity are explained along with an overview of the modalities investigated in the 15 chapters.
Keywords:
Natural and multimodal interactivity engineering.
1.
Introduction
Chapters 2 through 16 of this book present original contributions to the emerging field of natural and multimodal interactivity engineering (henceforth NMIE). A prominent characteristic of NMIE is that the field is not yet an established field of research and commercial development but, rather, an emerging one in all respects, including applicable theory, experimental results, platforms and development environments, standards (guidelines, de facto standards, official standards), evaluation paradigms, coherence, ultimate scope, enabling technologies for software engineering, general topology of the field itself, ”killer applications”, etc. The NMIE field is vast and brings together practitioners from very many different, and often far more established, fields of research and industrial development, such as signal processing, speech technology, computer graphics, computer vision, human-computer interaction, virtual and augmented reality, non-speech sound, haptic devices, telecommunications, computer games, etc. 1 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 1–19. © 2005 Springer. Printed in the Netherlands.
2
Advances in Natural Multimodal Dialogue Systems
Table 1.1. Needs for progress in natural and multimodal interactivity engineering (NMIE). General
Specific
Understand issues, problems, solutions
Applicable theory: for any aspect of NMIE. Empirical work and analysis: controlled experiments, behavioural studies, simulations, scenario studies, task analysis on roles of, and collaboration among, specific modalities to achieve various benefits. Annotation and analysis: new quality data resources, coding schemes, coding tools, and standards. Future visions: visions, roadmaps, etc., general and per sub-area.
Build systems
Enabling technologies: new basic technologies needed. More advanced systems: new, more complex, versatile, and capable system aspects. Make it easy: Re-usable platforms, components, toolkits, architectures, interface languages, standards, etc.
Evaluate
Evaluate all aspects: of components, systems, technologies, processes, etc.
It may be noted that the fact that a field of research has been established over decades in its own right is fully compatible with many if not most of its practitioners being novices in NMIE. It follows that NMIE community formation is an ongoing challenge for all. Broadly speaking, the emergence of a new systems field, such as NMIE, takes understanding of issues, problems and solutions, knowledge and skills for building (or developing) systems and enabling technologies, and evaluation of any aspect of the process and its results. In the particular case of NMIE, these goals or needs could be made more specific as shown in Table 1.1. Below, we start with a brief presentation of each of the following 15 chapters (Section 2). Taking Table 1.1 as point of departure, Section 3 provides an overview of, and discusses the individual chapters’ contributions to specific NMIE needs. Section 4 explains multimodality and natural interactivity and discusses the modalities investigated in the chapters of this book.
2.
Chapter Presentations
We have found it appropriate to structure the 15 chapters into four parts under four headlines related to how the chapters contribute to the specific NMIE needs listed in Table 1.1. Each chapter has a main emphasis on issues that contribute to NMIE needs captured by the headline of the part of the book to which it belongs. The division can of course not be a sharp one. Several chap-
Natural and Multimodal Interactivity Engineering - Directions and Needs
3
ters include discussions of issues that would make them fit into several parts of the book. Part one focuses on making dialogues more natural and has its main emphasis on experimental work and the application of theory. Part two concerns annotation and analysis of multimodal data, in particular, the modalities of speech and gesture. Part three addresses animated talking heads and related evaluation issues. Part four covers issues in building advanced multimodal dialogue systems, including architectures and technologies.
2.1
Making Dialogues More Natural: Empirical Work and Applied Theory
Two chapters have been categorised under this headline. They both aim at making interaction with a virtual or physical agent more natural. Experimental work is central in both chapters and so is the application of theory. The chapter by Bickmore and Cassell (Chapter 2) presents an empirical study of the social dialogue of an embodied conversational real-estate agent during interaction with users. The study is a comparative one. In one setting, users could see the agent while interacting with it. In the second setting, users could talk to the agent but not see it. The paper presents several interesting findings on social dialogue and interaction with embodied conversational agents. Sidner and Dzikovska (Chapter 3) present empirical results from humanrobot interaction in hosting activities. Their focus is on engagement, i.e. on the establishment and maintenance of a connection between interlocutors and the ending of it when desired. The stationary robot can point, make beat gestures, and move its eyes while conducting dialogue and tutoring on the use of a gas turbine engine shown to the user on a screen. The authors also discuss human-human hosting and engagement. Pursuing and building on theory of human-human engagement and using input from experiments, their idea is to continue to add capabilities to the robot which will make it become better to show engagement.
2.2
Annotation and Analysis of Multimodal Data: Speech and Gesture
The five chapters in this part have a common emphasis on data analysis. While one chapter focuses on annotation of already collected data, the other four chapters all describe experiments with data collection. In all cases, the data are subsequently analysed in order to, e.g., provide new knowledge on conversation or show whether a hypothesis was true or not. Martell (Chapter 4) presents the FORM annotation scheme and illustrates its use. FORM enables annotators to mark up the kinematic information of
4
Advances in Natural Multimodal Dialogue Systems
gestures in videos. Although designed for gesture markup, FORM is also designed to be extensible to markup of speech and other conversational information. The goal is to establish an extensible corpus of annotated videos which can be used for research in various aspects of conversational interaction. In an appendix, Martell provides a brief overview of other tools, schemes and methods for gesture analysis. Corradini and Cohen (Chapter 5) report on a Wizard-of-Oz study in which it was investigated how people use gesture and speech during interaction with a video game when they do not have access to standard input devices. The subjects’ interaction with the game was recorded, transcribed, further coded, and analysed. The primary analysis of the data concerns the users’ use of speech-only, gesture-only and speech and gesture in combination. Moreover, subjective data were collected by asking subjects after the experiment about their modality preferences during interaction. Although subjects’ answers and their actual behaviour in the experiment did not always match, the study indicates a preference for multimodal interaction. The chapter by Healey et al. (Chapter 6) addresses the analysis of humanhuman interaction. The motivation is the lack of support for the design of systems for human-human interaction. They discuss two psycholinguistic approaches and propose a third approach based on the detection and resolution of communication problems. This approach is useful for measuring the effectiveness of human-human interaction across tasks and modalities. A coding protocol for identification of repair phenomena across different modalities is presented followed by evaluation results from testing the protocol on a small corpus of repair sequences. The presented approach has the potential to help in judging the effectiveness of multimodal communication. Carbonell and Kieffer (Chapter 7) report on an experimental study which investigated if oral messages facilitate visual search tasks on a crowded display. Using the mouse, subjects were asked to search and select visual targets in complex scenes presented on the screen. Before the presentation of each scene, the subject would either see the target alone, receive an oral description of the target and spatial information about its position in the scene, or get a combination of the visual and oral target descriptions. Analysis of the data collected suggests that appropriate oral messages can improve search accuracy as well as selection time. However, both objectively and subjectively, multimodal messages were most effective. Darrell et al. (Chapter 8) discuss the problem of knowing who is speaking during multi-speaker interaction with a computer. They present two methods based on geometric and statistical source separation approaches, respectively. These methods are used for audiovisual segmentation of multiple speakers and have been used in experiments. One setup in a conference room used several stereo cameras and a ceiling-mounted microphone array grid. In this case a
Natural and Multimodal Interactivity Engineering - Directions and Needs
5
geometric method was used to identify the speaker. In a second setup, involving use of a handheld device or a kiosk, a single camera and a single omnidirectional microphone were used and a statistical method applied for speaker identification. Data analysis showed that each approach was valuable in the intended domain. However, the authors propose that a combination of the two methods would be of benefit and initial integration efforts are discussed.
2.3
Animated Talking Heads and Evaluation
Three chapters address animated talking heads. These chapters all include experimental data collection and data analysis, and present evaluations of various aspects of talking head technology and its usability. The chapter by Massaro (Chapter 9) concerns computer-assisted speech and language tutors for the deaf, hard of hearing and autistic children. The author presents an animated talking head for visible speech synthesis. The skin of the head can be made transparent so that one can see the tongue and the palate. The technology has been used in a language training program in which the agent guides the user through a number of exercises in vocabulary and grammar. The aim is to improve speech articulation and develop linguistic and phonological awareness in the users. The reported experiments show positive learning results. Granström and House (Chapter 10) describe work on animated talking heads, focusing on the increased intelligibility and efficiency provided by the addition of the talking face which uses text-to-speech synthesis. Results from various studies are presented. The studies include intelligibility tests and perceptual evaluation experiments. Among other things, facial cues to convey, e.g., feedback, turn-taking, and prosodic functions like prominence have been investigated. A number of applications of the talking head are described, including a language tutor. Heylen et al. (Chapter 11) report on how different eye gaze behaviours of a cartoon-like talking face affect the interaction with users. Three versions of the talking face were included in an experiment in which users had to make concert reservations by interacting with the talking face through typed input. One version of the face was aimed to be a close approximation to human-like gaze behaviour. In a second version gaze shifts were kept minimal, and in a third version gaze shifts were random. Evaluation of data from the experiment clearly showed a better performance of, and preference for, the human-like version.
6
2.4
Advances in Natural Multimodal Dialogue Systems
Architectures and Technologies for Advanced and Adaptive Multimodal Dialogue Systems
The last part of this book comprises five chapters which all have a strong focus on aspects of developing advanced multimodal dialogue systems. Several of the chapters present architectures which may be reused across applications, while others emphasise learning and adaptation. The chapter by Chai et al. (Chapter 12) addresses the difficult task of interpreting multimodal user input. The authors propose to use a fine-grained semantic model that characterises the meaning of user input and the overall conversation, and an integrated interpretation approach drawing on context knowledge, such as conversation histories and domain knowledge. These two approaches are discussed in detail and are included in the multimodal interpretation framework presented. The framework is illustrated by a real-estate application in which it has been integrated. Clark et al. (Chapter 13) discuss the application of a general-purpose architecture in support of multimodal interaction with complex devices and applications. The architecture includes speech recognition, natural language understanding, text-to-speech synthesis, an architecture for conversational intelligence, and use of the Open Agent Architecture. The architecture takes advantage of reusability and has been deployed in a number of dialogue systems. The authors report on its deployment in an intelligent tutoring system for shipboard damage control. Details about the tutoring system and the architecture are presented. Reithinger et al. (Chapter 14) present a multimodal system for access to multimedia databases on small handheld devices. Interaction in three languages is supported. Emphasis is on haptic interaction via active buttons combined with spoken input and visual and acoustic output. The overall architecture of the system is explained and so is the format for data exchange between modules. Also, the dialogue manager is described, including its architecture and multimodal fusion issues. Dusan and Flanagan (Chapter 15) address the difficult issue of ensuring sufficient vocabulary coverage in a spoken dialogue system. To overcome the problem that there may always be a need for additional words or word forms, the authors propose a method for adapting the vocabulary of a spoken dialogue system at run-time. Adaptation is done by the user by adding new concepts to existing pre-programmed concept classes and by providing semantic information about the new concepts. Multiple input modalities are available for doing the adaptation. Positive results from preliminary experiments with the method are reported. Wilks et al. (Chapter 16) discuss machine learning approaches to the modelling of human-computer interaction. They first describe a dialogue manager
Natural and Multimodal Interactivity Engineering - Directions and Needs
7
built for multimodal dialogue handling. The dialogue manager uses a kind of stereotypical dialogue patterns, called Dialogue Action Frames, for representation. The authors then describe an analysis module which learns to assign dialogue acts and semantic contents from corpora. The idea is to enable automatic derivation of Dialogue Action Frames, so that the dialog manager will be able to use Dialogue Action Frames that are automatically leaned from corpora.
3.
NMIE Contributions by the Included Chapters
Using the righthand column entries of Table 1.1, Table 1.2 indicates how the 15 chapters in this book contribute to the NMIE field. A preliminary conclusion based on Table 1.2 is that, for an emerging field which is only beginning to be exploited commercially, the NMIE research being done world-wide today is already pushing the frontiers in many of the directions needed. In the following, we discuss the chapter contributions to each of the lefthand entries in Table 1.2.
3.1
Applicable Theory
It may be characteristic of the NMIE field at present that our sample of papers only includes a single contribution of a primarily theoretical nature, i.e. Healey et al., which applies a psycholinguistic model of dialogue to help identify a subset of communication problems in order to judge the effectiveness of multimodal communication. Human-machine communication problems, their nature and identification by human or machine, has recently begun to attract the attention of more than a few NMIE researchers, and it has become quite clear that we need far better understanding of miscommunication in natural and multimodal interaction than we have at present. However, the relative absence of theoretical papers is not characteristic in the sense that the field does not make use of, or even need, applicable theory. On the contrary, a large number of chapters actually do apply existing theory in some form, ranging from empirical generalisations to full-fledged theory of many different kinds. For instance, Bickmore and Cassell test generalisations on the effects on communication of involving embodied conversational agents; Carbonell and Kieffer apply modality theory; Chai et al. apply theories of human-human dialogue to the development of a fined-grained, semanticsbased multimodal dialogue interpretation framework; Massaro applies theories of human learning; and Sidner and Dzikovska draw on conversation and collaboration theory.
8
Advances in Natural Multimodal Dialogue Systems
Table 1.2. NMIE needs addressed by the chapters in this book. Specific to NMIE Applicable theory Empirical work analysis
Contributions and
Annotation and analysis Future visions Enabling technologies
More advanced systems
Make it easy
Evaluate
3.2
No new theory except 6 but plenty of applied theory, e.g. 2, 3. Effects on communication of animated conversational agents. 2, 10, 11 Spoken input in support of visual search. 7 Gesture and speech for video game playing. 5 Multi-speaker speech recognition. 8 Gaze behaviour for more likeable animated interface agents. 2, 11 Audio-visual speech output. 9, 10 Animated talking heads for language learning. 9, 10 Tutoring. 3, 9, 10, 13 Hosting robots. 3 Coding scheme for conversational interaction research. 4 Standard for internal representation of NMIE data codings. 4 Many papers with visions concerning new challenges in their research. Interactive robotics: robots controlled multimodally, tutoring and hosting robots. 3 Multi-speaker speech recognition. 8 Audio-visual speech synthesis for talking heads. 9, 10 Machine learning of language and dialogue acts assignment. 15, 16 Multilinguality. 14 Ubiquitous (mobile) application. 14 On-line observation-based user modelling for adaptivity. 12, 14 Complex natural interactive dialogue management. 12, 13, 14, 16 Machine learning for more advanced dialogue systems. 15, 16 Platform for natural interactivity. 6 Re-usable components (many papers). Architectures for multimodal systems and dialogue management. 3, 12, 13, 14, 16, 16 Multimodal interface language. 14 XML for data exchange. 10, 14 Effects on communication of animated conversational agents. 2, 10, 11 Evaluations of talking heads. 9, 10, 11 Evaluation of audio-visual speech synthesis for learning. 9, 10
Empirical Work and Analysis
Novel theory tends to be preceded by empirical exploration and generalisation. The NMIE field is replete with empirical studies of human-human and human-computer natural and multimodal interaction [Dehn and van Mulken,
Natural and Multimodal Interactivity Engineering - Directions and Needs
9
2000]. By their nature, empirical studies are closer to the process of engineering than is theory development. We build NMIE research systems not only from theory but, perhaps to a far greater extent, from hunches, contextual assumptions, extrapolations from previous experience and untried transfer from different application scenarios, user groups, environments, etc., or even Wizard of Oz studies, which are in themselves a form of empirical study, see, e.g., the chapter by Corradini and Cohen and [Bernsen et al., 1998]. Having built a prototype system, we are keen to find out how far those hunches, etc. got us. Since empirical testing, evaluation, and assessment are integral parts of software and systems engineering, all we have to do is to include ”assumptions testing” in the empirical evaluation of the implemented system which we would be doing anyway. The drawback of empirical studies is that they usually do not generalise much due to the multitude of independent variables involved. This point is comprehensively argued and illustrated for the general case of multimodal and natural interactive systems which include speech in [Bernsen, 2002]. Still, as we tend to work on the basis of only slightly fortified hunches anyway, the results could often serve to inspire fellow researchers to follow them up. Thus, best-practice empirical studies are of major importance in guiding NMIE progress. The empirical chapters in this book illustrate well the points made above. One cluster of findings demonstrate the potential of audio-visual speech output by animated talking heads for child language learning (Massaro) and, more generally, for improving intelligibility and efficiency of human-machine communication, including the substitution of facial animation for the, still-missing, prosody in current speech synthesis systems (Granström and House). In counter-point, so to speak, Darrell et al. convincingly demonstrate the advantage of audio-visual input for tackling an important next step in speech technology, i.e. the recognition of multi-speaker spoken input. Jointly, the three chapters do a magnificent job of justifying the need for natural and multimodal (audiovisual) interaction independently of any psychological or social-psychological argument in favour of employing animated conversational agents. A key question seems to be: for which purpose(s), other than harvesting the benefits of using audio-visual speech input/output described above, do we need to accompany spoken human-computer dialogue with more or less elaborate animated conversational interface agents [Dehn and van Mulken, 2000]? By contrast with spoken output, animated interface agents occupy valuable screen real estate, do not necessarily add information of importance to the users of large classes of applications, and may distract the user from the task at hand. Whilst a concise and comprehensive answer to this question is still pending, Bickmore and Cassell go a long way towards explaining that the introduction of life-like animated interface agents into human-computer spoken dialogue is
10
Advances in Natural Multimodal Dialogue Systems
a tough and demanding proposition. As soon as an agent appears on the display, users tend to switch expectations from talking to a machine to talking to a human. By comparison, the finding of Heylen et al. that users tend to appreciate an animated cartoon agent more if it shows a minimum of human-like gaze behaviour might speak in favour of preferring cartoon-style agents over life-like animated agents because the former do not run the risk of facing our full expectations to human conversational behaviour. Sidner and Dzikovska do not involve a virtual agent but, rather, a robot in the dialogue with the user, so they do not have the problem of an agent occupying part of the screen. But they have still have the behavioural problems of the robot to look into just as, by close analogy, do the people who work with virtual agents. The experiments by Sidner and Dzikovska show that there is still a long way to go before we fully understand and can model the subtle details of human behaviour in dialogue. On the multimodal side of the natural interactivity/multimodality semidivide, several papers address issues of modality collaboration, i.e., how the use of modality combinations could facilitate, or even enable, human-computer interaction tasks that could not be done easily, if at all, using unimodal interaction. Carbonell and Kieffer report on how combined speech and graphics output can facilitate display search, and Corradini and Cohen show how the optional use of different input modalities can improve interaction in a particular virtual environment.
3.3
Annotation and Analysis
It is perhaps not surprising that we are not very capable of predicting what people will do, or how they will behave, when interacting with computer systems using new modality combinations and possibly also new interactive devices. More surprising, however, is the fact that we are often just as ignorant when trying to predict natural interactive behaviours which we have the opportunity to observe every day in ourselves and others, such as: which kinds of gestures, if any, do people perform when they are listening to someone else speaking? This example illustrates that, to understand the ways in which people communicate with one another as well as the ways in which people communicate with the far more limited, current NMI systems, we need extensive studies of behavioural data. The study of data on natural and multimodal interaction is becoming a major research area full of potential for new discoveries. A number of chapters make use of, or refer to, data resources for NMIE, but none of them take a more general view on data resource issues. One chapter addresses NMIE needs for new coding schemes. Martell presents a kinematically-based gesture annotation scheme for capturing the kinematic information in gestures from videos of speakers. Linking the urgent issue of
Natural and Multimodal Interactivity Engineering - Directions and Needs
11
new, more powerful coding tools with the equally important issue of standardisation, Martell proposes a standard for the internal representation of NMIE codings.
3.4
Future Visions
None of the chapters have a particular focus on future visions for the NMIE field. However, many authors touch on future visions, e.g., their descriptions of future work and what they would like to achieve. This includes the important driving role of re-usable platforms, tools, and components for making rapid progress. Moreover, there are several hints at the future importance to NMIE of two generic enabling technologies which are needed to extend spoken dialogue systems to full natural interactive systems. These technologies are (i) computer vision for processing camera input, and (ii) computer animation systems. It is only recently that the computer vision community has begun to address issues of natural interactive and multimodal human-system communication, and there is a long way to go before computer vision can parallel speech recognition as a major input medium for NMIE. The chapters by Massaro and Granström and House illustrate current NMIE efforts to extend natural and multimodal interaction beyond traditional information systems to new major application areas, such as training and education which has been around for a while already, notably in the US-dominated paradigm of tutoring systems using animated interface agents, but also to edutainment and entertainment. While the GUI, including the current WWW, might be said to have the edutainment potential of a schoolbook or newspaper, NMIE systems have the much more powerful edutainment potential of brilliant teachers, comedians, and exiting human-human games.
3.5
Enabling Technologies
An enabling technology is often developed over a long time by some separate community, such as by the speech recognition community from the 1950s to the late 1980s. Having matured to the point at which practical applications become possible, the technology transforms into an omnipresent tool for system developers, as is the case with speech recognition technology today. NMIE needs a large number of enabling technologies and these are currently at very different stages of maturity. Several enabling technologies, some of which are at an early stage and some of which are finding their way into applications, are presented in this book in the context of their application to NMIE problems, including robot interaction and agent technology, multi-speaker interaction and recognition, machine learning, and talking face technology. Sidner and Dzikovska focus on robot interaction in the general domain of ”hosting”, i.e., where a virtual or physical agent provides guidance, education,
12
Advances in Natural Multimodal Dialogue Systems
or entertainment based on collaborative goals negotiation and subsequent action. A great deal of work remains to be done before robot interaction becomes natural in any approximate sense of the term. For instance, the robot’s spoken dialogue capabilities must be strongly improved and so must its embodied appearance and global communicative behaviours. In fact, Sidner and Dzikovska make some of the same conclusions as Bickmore and Cassell, namely that agents need to become far more human-like in all or most respects before they are really appreciated by humans. Darrell et al. address the problem in multi-speaker interaction of knowing who is addressing the computer when. Their approach is to use a combination of microphones and computer vision to find out who is talking. Developers of spoken dialogue applications must cope with problems resulting from vocabulary and grammar limitations and from difficulties in enabling much of the flexibility and functionality inherent in human-human communication. Despite having carried out systematic testing, the developer often finds that words are missing when a new user addresses the application. Dusan and Flanagan propose machine learning as a way to overcome part of this problem. Using machine learning, the system can learn new words and grammars taught to it by the user in a well-defined way. Wilks et al. address machine learning or transformation-based learning - in the context of assigning dialogue acts as part of an approach to improved dialogue modelling. In another part of their approach, Wilks et al. consider the use of dialogue action frames, i.e., a set of stereotypical dialogue patterns which perhaps may be learned from corpus data, as a means for flexibly switching back and forth between topics during dialogue. Granström and House and Massaro describe the gain in intelligibility that can be obtained by combining speech synthesis with a talking face. There is still much work to do both on synthesis and face articulation. For most languages, speech synthesis is still not very natural to listen to and if one wants to develop a particular voice to fit a certain animated character, this is not immediately possible with today’s technology. With respect to face articulation, faces need to become much more natural in terms of, e.g., gaze, eyebrow movements, lip and mouth movements, and head movements, as this seems to influence users’ perception of the interaction, cf. the chapters by Granström and House and Heylen et al.
3.6
More Advanced Systems
Enabling technologies for NMIE are often component technologies, and their description, including state of the art, current research challenges, and unsolved problems, can normally be made in a relatively systematic and focused manner. It is far more difficult to systematically describe the complexity
Natural and Multimodal Interactivity Engineering - Directions and Needs
13
of the constant push in research and industry towards exploring and exploiting new NMIE application types and new application domains, addressing new user populations, increasing the capabilities of systems in familiar domains of application, exploring known technologies with new kinds of devices, etc. In general, the picture is one of pushing present boundaries in all or most directions. During the past few years, a core trend in NMIE has been to combine different modalities in order to build more complex, versatile and capable systems, getting closer to natural interactivity than is possible with only a single modality. This trend is reflected in several chapters. Part of the NMIE paradigm is that systems must be available whenever and wherever convenient and useful, making ubiquitous computing an important application domain. Mobile devices, such as mobile phones, PDAs, and portable computers of any (portable) size have become popular and are rapidly gaining functionality. However, the interface and interaction affordances of small devices require careful consideration. Reithinger et al. present some of those considerations in the context of providing access to large amounts of data about music. It can be difficult for users to know how to interact with new NMIE applications. Although not always very successful in practice, the classical GUI system has the opportunity to present its affordances in static graphics (including text) before the user chooses how to interact. A speech-only system, by contrast, cannot do that because of the dynamic and transitory nature of acoustic modalities. NMIE systems, in other words, pose radically new demands on how to support the user prior to, and during, interaction. Addressing this problem, several chapters mention user modelling or repositories of user preferences built on the basis of interactions with a system, cf. the chapters by Chai et al. and Reithinger et al. Machine learning, although another example of less-than-expected pace of development during the past 10 years, has great potential for increasing interaction support. In an advanced application of machine learning, Dusan and Flanagan propose to increase the system’s vocabulary and grammar by letting users teach the system new words and their meaning and use. Wilks et al. use machine learning as part of an approach to more advanced dialogue modelling. Increasingly advanced systems require increasingly complex dialogue management, cf. the chapters by Chai et al., Clark et al., and Wilks et al. Lifelikeness of animated interface agents and conversational dialogue are among the key challenges in achieving the NMIE vision. Multilinguality of systems is an important NMIE goal. Multilingual applications are addressed by Reithinger et al. In their case, the application is running on a handheld device. Multi-speaker input speech is mentioned by Darrell et al. For good reason, recognition of multi-speaker input has become a lively research topic. We
14
Advances in Natural Multimodal Dialogue Systems
need solutions in order to, e.g., build meeting minute-takers, separate the focal speaker’s input from that of other speakers, exploit the huge potential of spoken multi-user applications, etc.
3.7
Make It Easy
Due to the complexity of multimodal natural interaction, it is becoming dramatically important to be able to build systems as easily as possible. It seems likely that no single research lab or development team in industry, even including giants such as Microsoft, is able to master all of the enabling technologies required for broad-scale NMIE progress. To advance efficiently, everybody needs access to those system components, and their built-in know-how, which are not in development focus. This implies strong attention to issues, such as re-usable platforms, components and architectures, development toolkits, interface languages, data formats, and standardisation. Clark et al. have used the Open Agent Architecture (OAA, http://www.ai.sri.com/∼oaa/), a framework for integrating heterogeneous software agents in a distributed environment. What OAA and other architectural frameworks, such as CORBA (http://www.corba.org/), aim to do is provide a means for modularisation, synchronous and asynchronous communication, well-defined inter-module communication via some interface language, such as IDL (CORBA) or ICL (OAA), and the possibility of implementation in a distributed environment. XML (Extensible Markup Language) is a simple, flexible text format derived from SGML (ISO 8879) which has become popular as, among other things, a message exchange format, cf. Reithinger et al. and Granström and House. Using XML for wrapping inter-module messages is one way to overcome the problem of different programming languages used for implementing different modules. Some chapters express a need for reusable components. Many of the applications described include off-the-shelf software, including components developed in other projects. This is particularly true for mature enabling technologies, such as speech recognition and synthesis components. As regards multimodal dialogue management, there is an expressed need for reuse in, e.g., the chapter by Clark et al. who discuss a reusable dialogue management architecture in support of multimodal interaction. In conclusion, there are architectures, platforms, and software components available which facilitate the building of new NMIE applications, and standards are underway for certain aspects. There is still much work to be done on standardisation, new and better platforms, and improvement of component software. In addition, we need, in particular, more and better toolkits in support of system development and a better understanding of those components which
Natural and Multimodal Interactivity Engineering - Directions and Needs
15
cannot be bought off-the-shelf and which are typically difficult to reuse, such as dialogue managers. Advancements such as these are likely to require significant corpus work. Corpora with tools and annotation schemes as described by Martell are exactly what is needed in this context.
3.8
Evaluate
Software systems and components evaluation is a broad area, ranging from technical evaluation over usability evaluation to customer evaluation. Customer evaluation has never been a key issue in research but has, rather, tended to be left to the marketing departments of companies. Technical evaluation and usability evaluation, including evaluation of functionality from both perspectives, are, on the other hand, considered important research issues. The chapters show a clear trend towards focusing on usability evaluation and comparative performance evaluation. It is hardly surprising that performance evaluation and usability issues are considered key topics today. We know little about what happens when we move towards increasingly multimodal and natural interactive systems, both as regards how these new systems will perform compared to alternative solutions and how the systems will be received and perceived by their users. We only know that a technically optimal system is not sufficient to guarantee user satisfaction. Comparative performance evaluation objectively compares users’ performance on different systems with respect to, e.g., how well they understand speech-only versus speech combined with a talking face or with an embodied animated agent, cf. Granström and House, Massaro, and Bickmore and Cassell. The usability issues evaluated all relate to users’ perception of a particular system and include parameters, such as life-likeness, credibility, reliability, efficiency, personality, ease of use, and understanding quality, cf. Heylen et al. and Bickmore and Cassell. Two chapters address how the intelligibility of what is being said can be increased through visual articulation, cf. Granström and House and Massaro. Granström and House have used a talking head in several applications, including tourist information, real estate (apartment) search, aid for the hearing impaired, education, and infotainment. Evaluation shows a significant gain in intelligibility for the hearing impaired. Eyebrow and head movement enhance perception of emphasis and syllable prominence. Over-articulation may be useful as well when there are special needs for intelligibility. The findings of Massaro support these promising conclusions. His focus is on applications for the hard-of-hearing, children with autism, and child language learning more generally. Granström and House also address the increase in efficiency of communication/interaction produced by using an animated talking head. Probably,
16
Advances in Natural Multimodal Dialogue Systems
naturalness is a key point here. This is suggested by Heylen et al. who made controlled experiments on the effects of different eye gaze behaviours of a cartoon-like talking face on the quality of human-agent dialogues. The most human-like agent gaze behaviour led to higher appreciation of the agent and more efficient task performance. Bickmore and Cassell evaluate the effects on communication of an embodied conversational real-estate agent versus an over-the-phone version of the same system, cf. also [Cassell et al., 2000]. In each condition, two variations of the system was available. One would be fully task-oriented while the second version would include some small-talk options. In general, users liked the system better in the phone condition. In the phone condition, subjects appreciated the small-talk whereas, in the embodied condition, subjects wanted to get down to business. The implication is that agent embodiment has strong effects on the interlocutors. Users tend to compare their animated interlocutors with humans rather than machines. To work with users, animated agents need considerably more naturalness and personally attractive features communicated non-verbally. This imposes a tall research agenda on both speech and non-verbal output, requiring conversational abilities both verbally and nonverbally. Jointly, the chapters on evaluation demonstrate a broad need for performance evaluation, comparative as well as non-comparative, that can inform us on the possible benefits and shortcomings of new natural interactive and multimodal systems. The chapters show a similar need for usability evaluation that can help us find out how users perceive these new systems, and a need for finding ways in which usability and user satisfaction might be correlated with technical aspects in order for the former to be derived from the latter.
4.
Multimodality and Natural Interactivity
Conceptually, NMIE combines natural interactive and multimodal systems and components engineering. While both concepts, natural interactivity and multimodality, have a long history, it would seem that they continue to sit somewhat uneasily side by side in the minds of most of us. Multimodality is the idea of being able to choose any input/output modality or combination of input/output modalities for optimising interaction with the application at hand, such as speech input for many heads-up, hands-occupied applications, speech and haptic input/output for applications for the blind, etc. A modality is a particular way of representing input or output information in some physical medium, such as something touchable, light, sound, or the chemistry for producing olfaction and gustation [Bernsen, 2002], see also the chapter by Carbonell and Kieffer. The physical medium of the speech modalities, for instance, is sound or acoustics but this medium obviously enables the trans-
Natural and Multimodal Interactivity Engineering - Directions and Needs
17
mission of information in many acoustic modalities other than speech, such as earcons, music, etc. The term multimodality thus refers to any possible combination of elementary or unimodal modalities. Compared to multimodality, the notion of natural interactivity appears to be the more focused of the two. This is because natural interactivity comes with a focused vision of the future of interaction with computer systems as well as a relatively well-defined set of modalities required for the vision to become reality. The natural interactivity vision is that of humans communicating with computer systems in the same ways in which humans communicate with one another. Thus, natural interactivity specifically emphasises human-system communication involving the following input/output modalities used in situated human-human communication: speech, gesture, gaze, facial expression, head and body posture, and object manipulation as integral part of the communication (or dialogue). As the objects being manipulated may themselves represent information, such as text and graphics input/output objects, natural interaction subsumes the GUI paradigm. Technologically, the natural interactivity vision is being pursued vigorously by, among others, the emerging research community in talking faces and embodied conversational agents as illustrated in the chapters by Bickmore and Cassell, Granström and House, Heylen et al., and Massaro. An embodied conversational agent may be either virtual or a robot, cf. the chapter by Sidner and Dzikovska. A weakness in our current understanding of natural interactivity is that it is not quite clear where to draw the boundary between the natural interactivity modalities and all those other modalities and modality combinations which could potentially be of benefit to human-system interaction. For instance, isn’t pushing a button on the mouse or otherwise, although never used in humanhuman communication for the simple reason that humans do not have communicative buttons on them, as natural as speaking? If it is, then, perhaps, all or most research on useful multimodal input/output modality combinations is also research into natural interactivity even if the modalities addressed are not being used in human-human communication? In addition to illustrating the need for more and better NMIE theory, the point just made may explain the uneasy conceptual relationship among the two paradigms of natural interactivity and multimodality. In any case, we have decided to combine the paradigms and address them together as natural and multimodal interactivity engineering. Finally, by NMI ’engineering’ we primarily refer to software engineering. It follows that the expression ’natural and multimodal interactivity engineering’ primarily represents the idea of creating a specialised branch of software engineering for the field addressed in this book. It is important to add, however, that NMIE enabling technologies are being developed in fields whose practitioners do not tend to regard themselves as doing software engineering, such as signal processing. For instance, the recently launched European Network of
18
Advances in Natural Multimodal Dialogue Systems
Excellence SIMILAR (http://www.similar.cc) addresses signal processing for natural and multimodal interaction.
4.1
Modalities Investigated in This Book
We argued above that multimodality includes all possible modalities for the representation and exchange of information among humans and between humans and computer systems, and that natural interactivity includes a rather vaguely defined, large sub-set of those modalities. Within this wide space of unimodal modalities and modality combinations, it may be useful to look at the modalities actually addressed in the following chapters. These are summarised by chapter in Table 1.3. Table 1.3. Modalities addressed in the included chapters (listed by chapter number plus first listed author). Chapter
Input
Output
2. Bickmore
speech, gesture (via camera) vs. speech-only speech, mouse clicks, (new version includes face and gesture input via camera)
embodied conversational agent + images vs. speech-only + images robot pointing and beat gestures, speech, gaze N/A video game
6. Healey 7. Carbonell 8. Darrell
gesture speech, gesture, object manipulation/manipulative gesture N/A gesture (mouse) speech, camera-based graphics
9. Massaro
mouse, speech
10. Granström
N/A
11. Heylen
typed text
12. Chai 13. Clark 14. Reithinger
speech, text, gesture (pointing) speech, gesture (pointing) speech, haptic buttons
15. Dusan
speech, keyboard, mouse, penbased drawing and pointing, camera speech and possibly other modalities
3. Sidner
4. Martell 5. Corradini
16. Wilks
N/A speech, graphics N/A audio-visual speech synthesis, talking head, images, text audio-visual speech synthesis, talking head talking head, gaze speech, graphics speech, text, graphics music, speech, text, graphics, tactile rythm speech, graphics, text display
speech and possibly other modalities. Focus is on dialogue modelling so input/output modalities are not discussed in detail
Natural and Multimodal Interactivity Engineering - Directions and Needs
19
Combined speech input/output which, in fact, means spoken dialogue almost throughout, is addressed in about half of the chapters (Bickmore and Cassell, Chai et al., Clark et al., Corradini and Cohen, Dusan and Flanagan, Reithinger et al., and Sidner and Dzikovska). Almost two thirds of the chapters address gesture input in some form (Bickmore and Cassell, Chai et al., Clark et al., Corradini and Cohen, Darrell et al., Dusan and Flanagan, Martell, Reithinger et al., and Sidner and Dzikovska). Five chapters address output modalities involving talking heads, embodied animated agents, or robots (Bickmore and Cassell, Granström and House, Heylen et al., Massaro, and Sidner and Dzikovska). Three chapters (Darrell et al., Bickmore and Cassell, and Sidner and Dzikovska) address computer vision. Dusan and Flanagan also mention that their system has camera-based input. Facial expression of emotion is addressed by Granström and House. Despite its richness and key role in natural interactivity, input or output speech prosody is hardly discussed. Granström and House discuss graphical ways of replacing missing output speech prosody by facial expression means. In general, the input and output modalities and their combinations discussed would appear representative of the state-of-the-art in NMIE. The authors make quite clear how far we are from mastering the very large number of potentially useful unimodal ”compounds” theoretically, in input recognition, in output generation, as well as in understanding and generation.
References Bernsen, N. O. (2002). Multimodality in Language and Speech Systems - From Theory to Design Support Tool. In Granström, B., House, D., and Karlsson, I., editors, Multimodality in Language and Speech Systems, pages 93–148. Dordrecht: Kluwer Academic Publishers. Bernsen, N. O., Dybkjær, H., and Dybkjær, L. (1998). Designing Interactive Speech Systems. From First Ideas to User Testing. Springer Verlag. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., and Yan, H. (2000). Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Embodied Conversational Agents, pages 29–63. Cambridge, MA: MIT Press. Dehn, D. and van Mulken, S. (2000). The Impact of Animated Interface Agents: A Review of Empirical Research. International Journal of Human-Computer Studies, 52:1–22.
PART I
MAKING DIALOGUES MORE NATURAL: EMPIRICAL WORK AND APPLIED THEORY
Chapter 2 SOCIAL DIALOGUE WITH EMBODIED CONVERSATIONAL AGENTS Timothy Bickmore Northeastern University, USA
[email protected]
Justine Cassell Northwestern University, USA
[email protected]
Abstract
The functions of social dialogue between people in the context of performing a task is discussed, as well as approaches to modelling such dialogue in embodied conversational agents. A study of an agent’s use of social dialogue is presented, comparing embodied interactions with similar interactions conducted over the phone, assessing the impact these media have on a wide range of behavioural, task and subjective measures. Results indicate that subjects’ perceptions of the agent are sensitive to both interaction style (social vs. task-only dialogue) and medium.
Keywords:
Embodied conversational agent, social dialogue, trust.
1.
Introduction
Human-human dialogue does not just comprise statements about the task at hand, about the joint and separate goals of the interlocutors, and about their plans. In human-human conversation participants often engage in talk that, on the surface, does not seem to move the dialogue forward at all. However, this talk – about the weather, current events, and many other topics without significant overt relationship to the task at hand – may, in fact, be essential to how humans obtain information about one another’s goals and plans and decide whether collaborative work is worth engaging in at all. For example, realtors use small talk to gather information to form stereotypes (a collection 23 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 23–54. © 2005 Springer. Printed in the Netherlands.
24
Advances in Natural Multimodal Dialogue Systems
of frequently co-occurring characteristics) of their clients – people who drive minivans are more likely to have children, and therefore to be searching for larger homes in neighbourhoods with good schools. Realtors – and salespeople in general – also use small talk to increase intimacy with their clients, to establish their own expertise, and to manage how and when they present information to the client [Prus, 1989]. Nonverbal behaviour plays an especially important role in such social dialogue, as evidenced by the fact that most important business meetings are still conducted face-to-face rather than on the phone. This intuition is backed up by empirical research; several studies have found that the additional nonverbal cues provided by video-mediated communication do not effect performance in task-oriented interactions, but in interactions of a more social nature, such as getting acquainted or negotiation, video is superior [Whittaker and O’Conaill, 1997]. These studies have found that for social tasks, interactions were more personalized, less argumentative and more polite when conducted via video-mediated communication, that participants believed video-mediated (and face-to-face) communication was superior, and that groups conversing using video-mediated communication tended to like each other more, compared to audio-only interactions. Together, these findings indicate that if we are to develop computer agents capable of performing as well as humans on tasks such as real estate sales then, in addition to task goals such reliable and efficient information delivery, they must have the appropriate social competencies designed into them. Further, since these competencies include the use of nonverbal behaviour for conveying communicative and social cues, then our agents must have the capability of producing and recognizing nonverbal cues in simulations of face-to-face interactions. We call agents with such capabilities “Embodied Conversational Agents” or “ECAs.” The current chapter extends previous work which demonstrated that social dialogue can have a significant impact on a user’s trust of an ECA [Bickmore and Cassell, 2001], by investigating whether these results hold in the absence of nonverbal cues. We present the results of a study designed to determine whether the psychological effects of social dialogue – namely to increase trust and associated positive evaluations – vary when the nonverbal cues provided by the embodied conversational agent are removed. In addition to varying medium (voice only vs. embodied) and dialogue style (social dialogue vs. taskonly) we also assessed and examined effects due to the user’s personality along the introversion/extroversion dimension, since extroversion is one indicator of a person’s comfort level with face-to-face interaction.
Social Dialogue with Embodied Conversational Agents
2.
25
Embodied Conversational Agents
Embodied conversation agents are animated anthropomorphic interface agents that are able to engage a user in real-time, multimodal dialogue, using speech, gesture, gaze, posture, intonation, and other verbal and nonverbal behaviours to emulate the experience of human face-to-face interaction [Cassell et al., 2000c]. The nonverbal channels are important not only for conveying information (redundantly or complementarily with respect to the speech channel), but also for regulating the flow of the conversation. The nonverbal channel is especially crucial for social dialogue, since it can be used to provide such social cues as attentiveness, positive affect, and liking and attraction, and to mark shifts into and out of social activities [Argyle, 1988].
2.1
Functions versus Behaviours
Embodiment provides the possibility for a wide range of behaviours that, when executed in tight synchronization with language, carry out a communicative function. It is important to understand that particular behaviours, such as the raising of the eyebrows, can be employed in a variety of circumstances to produce different communicative effects, and that the same communicative function may be realized through different sets of behaviours. It is therefore clear that any system dealing with conversational modelling has to handle function separately from surface-form or run the risk of being inflexible and insensitive to the natural phases of the conversation. Here we briefly describe some of the fundamental communication categories and their functional sub-parts, along with examples of nonverbal behaviour that contribute to their successful implementation. Table 2.1 shows examples of mappings from communicative function to particular behaviours and is based on previous research on typical North American nonverbal displays, mainly [Chovil, 1991; Duncan, 1974; Kendon, 1980].
Conversation initiation and termination Humans partake in an elaborate ritual when engaging and disengaging in conversations [Kendon, 1980]. For example, people will show their readiness to engage in a conversation by turning towards the potential interlocutors, gazing at them and then exchanging signs of mutual recognition typically involving a smile, eyebrow movement and tossing the head or waving of the arm. Following this initial synchronization stage, or distance salutation, the two people approach each other, sealing their commitment to the conversation through a close salutation such as a handshake accompanied by a ritualistic verbal exchange. The greeting phase ends when the two participants re-orient their bodies, moving away from a face-on orientation to stand at an angle. Terminating a conversation similarly moves through stages, starting with non-verbal cues, such as orientation shifts
26
Advances in Natural Multimodal Dialogue Systems
Table 2.1. Some examples of conversational functions and their behaviour realization [Cassell et al., 2000b]. Communicative Functions Initiation and termination Reacting Inviting Contact Distance Salutation Close Salutation Break Away Farewell Turn-Taking Give Turn Wanting Turn Take Turn Feedback Request Feedback Give Feedback
Communicative Behaviour Short Glance Sustained Glance, Smile Looking, Head Toss/Nod, Raise Eyebrows, Wave, Smile Looking, Head Nod, Embrace or Handshake, Smile Glance Around Looking, Head Nod, Wave Looking, Raise Eyebrows (followed by silence) Raise Hands into gesture space Glance Away, Start talking Looking, Raise Eyebrows Looking, Head Nod
or glances away and cumulating in the verbal exchange of farewells and the breaking of mutual gaze.
Conversational turn-taking and interruption Interlocutors do not normally talk at the same time, thus imposing a turn-taking sequence on the conversation. The protocols involved in floor management – determining whose turn it is and when the turn should be given to the listener – involve many factors including gaze and intonation [Duncan, 1974]. In addition, listeners can interrupt a speaker not only with voice, but also by gesturing to indicate that they want the turn. Content elaboration and emphasis Gestures can convey information about the content of the conversation in ways for which the hands are uniquely suited. For example, the two hands can better indicate simultaneity and spatial relationships than the voice or other channels. Probably the most commonly thought of use of the body in conversation is the pointing (deictic) gesture, possibly accounting for the fact that it is also the most commonly implemented for the bodies of animated interface agents. In fact, however, most conversations don’t involve many deictic gestures [McNeill, 1992] unless the interlocutors are discussing a shared task that is currently present. Other conversational gestures also convey semantic and pragmatic information. Beat gestures are small, rhythmic baton like movements of the hands that do not change in form with the content of the accompanying speech. They serve a pragmatic func-
Social Dialogue with Embodied Conversational Agents
27
tion, conveying information about what is “new” in the speaker’s discourse. Iconic and metaphoric gestures convey some features of the action or event being described. They can be redundant or complementary relative to the speech channel, and thus can convey additional information or provide robustness or emphasis with respect to what is being said. Whereas iconics convey information about spatial relationships or concepts, metaphorics represent concepts which have no physical form, such as a sweeping gesture accompanying “the property title is free and clear.”
Feedback and error correction During conversation, speakers can nonverbally request feedback from listeners through gaze and raised eyebrows and listeners can provide feedback through head nods and paraverbals (“uh-huh”, “mmm”, etc.) if the speaker is understood, or a confused facial expression or lack of positive feedback if not. The listener can also ask clarifying questions if they did not hear or understand something the speaker said.
2.2
Interactional versus Propositional Behaviours
The mapping from form (behaviour) to conversational function relies on a fundamental division of conversational goals: contributions to a conversation can be propositional and interactional. Propositional information corresponds to the content of the conversation. This includes meaningful speech as well as hand gestures and intonation used to complement or elaborate upon the speech content (gestures that indicate the size in the sentence “it was this big” or rising intonation that indicates a question with the sentence “you went to the store”). Interactional information consists of the cues that regulate conversational process and includes a range of nonverbal behaviours (quick head nods to indicate that one is following) as well as regulatory speech (“huh?”, “Uhhuh”). This theoretical stance allows us to examine the role of embodiment not just in task- but also process-related behaviours such as social dialogue [Cassell et al., 2000b].
2.3
REA
Our platform for conducting research into embodied conversational agents is the REA system, developed in the Gesture and Narrative Language Group at the MIT Media Lab [Cassell et al., 2000a]. REA is an embodied, multi-modal real-time conversational interface agent which implements the conversational protocols described above in order to make interactions as natural as face-toface conversation with another person. In the current task domain, REA acts as a real estate salesperson, answering user questions about properties in her database and showing users around the virtual houses (Figure 2.1).
28
Advances in Natural Multimodal Dialogue Systems
Figure 2.1.
User interacting with REA.
REA has a fully articulated graphical body, can sense the user passively through cameras and audio input, and is capable of speech with intonation, facial display, and gestural output. The system currently consists of a large projection screen on which REA is displayed and which the user stands in front of. Two cameras mounted on top of the projection screen track the user’s head and hand positions in space. Users wear a microphone for capturing speech input. A single SGI Octane computer runs the graphics and conversation engine of REA, while several other computers manage the speech recognition and generation and image processing. REA is able to conduct a conversation describing the features of the task domain while also responding to the users’ verbal and non-verbal input. When the user makes cues typically associated with turn taking behaviour such as gesturing, REA allows herself to be interrupted, and then takes the turn again when she is able. She is able to initiate conversational error correction when she misunderstands what the user says, and can generate combined voice, facial expression and gestural output. REA’s responses are generated by an incremental natural language generation engine based on [Stone and Doran, 1997] that has been extended to synthesize redundant and complementary gestures synchronized with speech output [Cassell et al., 2000b]. A simple discourse
Social Dialogue with Embodied Conversational Agents
29
model is used for determining which speech acts users are engaging in, and resolving and generating anaphoric references.
3.
Social Dialogue
Social dialogue is talk in which interpersonal goals are foregrounded and task goals – if existent – are backgrounded. One of the most familiar contexts in which social dialogue occurs is in human social encounters between individuals who have never met or are unfamiliar with each other. In these situations conversation is usually initiated by “small talk” in which “light” conversation is made about neutral topics (e.g., weather, aspects of the interlocutor’s physical environment) or in which personal experiences, preferences, and opinions are shared [Laver, 1981]. Even in business or sales meetings, it is customary (at least in American culture) to begin with some amount of small talk before “getting down to business”.
3.1
The Functions of Social Dialogue
The purpose of small talk is primarily to build rapport and trust among the interlocutors, provide time for them to “size each other up”, establish an interactional style, and to allow them to establish their reputations [Dunbar, 1996]. Although small talk is most noticeable at the margins of conversational encounters, it can be used at various points in the interaction to continue to build rapport and trust [Cheepen, 1988], and in real estate sales, a good agent will continue to focus on building rapport throughout the relationship with a buyer [Garros, 1999]. Small talk has received sporadic treatment in the linguistics literature, starting with the seminal work of Malinowski who defined “phatic communion” as “a type of speech in which ties of union are created by a mere exchange of words”. Small talk is the language used in free, aimless social intercourse, which occurs when people are relaxing or when they are accompanying “some manual work by gossip quite unconnected with what they are doing” [Malinowski, 1923]. Jacobson also included a “phatic function” in his well-known conduit model of communication, that function being focused on the regulation of the conduit itself (as opposed to the message, sender, or receiver) [Jakobson, 1960]. More recent work has further characterized small talk by describing the contexts in which it occurs, topics typically used, and even grammars which define its surface form in certain domains [Cheepen, 1988; Laver, 1975; Schneider, 1988]. In addition, degree of “phaticity” has been proposed as a persistent goal which governs the degree of politeness in all utterances a speaker makes, including task-oriented ones [Coupland et al., 1992].
30
3.2
Advances in Natural Multimodal Dialogue Systems
The Relationship between Social Dialogue and Trust
Figure 2.2 outlines the relationship between small talk and trust. REA’s dialogue planner represents the relationship between her and the user using a multi-dimensional model of interpersonal relationship based on [Svennevig, 1999]: familiarity describes the way in which relationships develop through the reciprocal exchange of information, beginning with relatively nonintimate topics and gradually progressing to more personal and private topics. The growth of a relationship can be represented in both the breadth (number of topics) and depth (public to private) of information disclosed [Altman and Taylor, 1973]. solidarity is defined as “like-mindedness” or having similar behaviour dispositions (e.g., similar political membership, family, religions, profession, gender, etc.), and is very similar to the notion of social distance used by Brown and Levinson in their theory of politeness [Brown and Levinson, 1978]. There is a correlation between frequency of contact and solidarity, but it is not necessarily a causal relation [Brown and Levinson, 1978; Brown and Gilman, 1972]. affect represents the degree of liking the interactants have for each other, and there is evidence that this is an independent relational attribute from the above three [Brown and Gilman, 1989]. The mechanisms by which small talk are hypothesized to effect trust include facework, coordination, building common ground, and reciprocal appreciation.
Facework The notion of “face” is “the positive social value a person effectively claims for himself by the social role others assume he has taken during a particular contact” [Goffman, 1967]. Interactants maintain face by having their social role accepted and acknowledged. Events which are incompatible with their line are “face threats” and are mitigated by various corrective measures if they are not to lose face. Small talk avoids face threat (and therefore maintains solidarity) by keeping conversation at a safe level of depth. Coordination The process of interacting with a user in a fluid and natural manner may increase the user’s liking of the agent, and user’s positive affect, since the simple act of coordination with another appears to be deeply gratifying. “Friends are a major source of joy, partly because of the enjoyable things they do together, and the reason that they are enjoyable is perhaps the coordination.” [Argyle, 1990]. Small talk increases coordination between the two participants by allowing them to synchronize short units of talk and nonverbal acknowledgement (and therefore leads to increased liking and positive affect).
Social Dialogue with Embodied Conversational Agents
31
Figure 2.2. How small talk effects trust [Cassell and Bickmore, 2003].
Building common ground Information which is known by all interactants to be shared (mutual knowledge) is said to be in the “common ground” [Clark, 1996]. The principle way for information to move into the common ground is via face-to-face communication, since all interactants can observe the recognition and acknowledgment that the information is in fact mutually shared. One strategy for effecting changes to the familiarity dimension of the relationship model is for speakers to disclose personal information about themselves – moving it into the common ground – and induce the listener to do the same. Another strategy is to talk about topics that are obviously in the common ground – such as the weather, physical surroundings, and other topics available in the immediate context of utterance. Small talk establishes common ground (and therefore increases familiarity) by discussing topics that are clearly in the context of utterance. Reciprocal appreciation In small talk, demonstrating appreciation for and agreement with the contributions of one’s interlocutor is obligatory. Performing this aspect of the small talk ritual increases solidarity by showing mutual agreement on the topics discussed.
32
3.3
Advances in Natural Multimodal Dialogue Systems
Nonverbal Behaviour in Social Dialogue
According to Argyle, nonverbal behaviour is used to express emotions, to communicate interpersonal attitudes, to accompany and support speech, for self presentation, and to engage in rituals such as greetings [Argyle, 1988]. Of these, coverbal and emotional display behaviours have received the most attention in the literature on embodied conversational agents and facial and character animation in general, e.g. [Cassell et al., 2000c]. Next to these, the most important use of nonverbal behaviour in social dialogue is the display of interpersonal attitude [Argyle, 1988]. The display of positive or negative attitude can greatly influence whether we approach someone or not and our initial perceptions of them if we do. The most consistent finding in this area is that the use of nonverbal “immediacy behaviours” – close conversational distance, direct body and facial orientation, forward lean, increased and direct gaze, smiling, pleasant facial expressions and facial animation in general, nodding, frequent gesturing and postural openness – projects liking for the other and engagement in the interaction, and is correlated with increased solidarity [Argyle, 1988; Richmond and McCroskey, 1995]. Other nonverbal aspects of “warmth” include kinesic behaviours such as head tilts, bodily relaxation, lack of random movement, open body positions, and postural mirroring and vocalic behaviours such as more variation in pitch, amplitude, duration and tempo, reinforcing interjections such as “uh-huh” and “mm-hmmm”, greater fluency, warmth, pleasantness, expressiveness, and clarity and smoother turn-taking [Andersen and Guerrero, 1998]. In summary, nonverbal behaviour plays an important role in all face-to-face interaction – both conveying redundant and complementary propositional information (with respect to speech) and regulating the structure of the interaction. In social dialogue, however, it provides the additional, and crucial, function, of conveying attitudinal information about the nature of the relationship between the interactants.
4. 4.1
Related Work Related Work on Embodied Conversational Agents
Work on the development of ECAs, as a distinct field of development, is best summarized in [Cassell et al., 2000c]. The current study is based on the REA ECA (see Figure 2.1), a simulated real estate agent, who uses vision-based gesture recognition, speech recognition, discourse planning, sentence and gesture planning, speech synthesis and animation of a 3D body [Cassell et al., 1999]. Some of the other major systems developed to date are Steve [Rickel and Johnson, 1998], the DFKI Persona [André et al., 1996], Olga [Beskow and
Social Dialogue with Embodied Conversational Agents
33
McGlashan, 1997], and pedagogical agents developed by Lester et al. [1999]. Sidner and Dzikovska [2005] report progress on a robotic ECA that performs hosting activities, with a special emphasis on “engagement” – an interactional behaviour whose purpose is to establish and maintain the connection between interlocutors during a conversation. These systems vary in their linguistic generativity, input modalities, and task domains, but all aim to engage the user in natural, embodied conversation. Little work has been done on modelling social dialogue with ECAs. The August system is an ECA kiosk designed to give information about local restaurants and other facilities. In an experiment to characterize the kinds of things that people would say to such an agent, over 10,000 utterances from over 2,500 users were collected. It was found that most people tried to socialize with the agent, with approximately 1/3 of all recorded utterances classified as social in nature [Gustafson et al., 1999].
4.2
Related Studies on Embodied Conversational Agents
Koda and Maes [1996] and Takeuchi and Naito [1995] studied interfaces with static or animated faces, and found that users rated them to be more engaging and entertaining than functionally equivalent interfaces without a face. Kiesler and Sproull [1997] found that users were more likely to be cooperative with an interface agent when it had a human face (vs. a dog or cartoon dog). André, Rist and Muller found that users rated their animated presentation agent (“PPP Persona”) as more entertaining and helpful than an equivalent interface without the agent [André et al., 1998]. However, there was no difference in actual performance (comprehension and recall of presented material) in interfaces with the agent vs. interfaces without it. In another study involving this agent, van Mulken, André and Muller found that when the quality of advice provided by an agent was high, subjects actually reported trusting a text-based agent more than either their ECA or a video-based agent (when the quality of advice was low there were no significant differences in trust ratings between agents) [van Mulken et al., 1999]. In a user study of the Gandalf system [Cassell et al., 1999], users rated the smoothness of the interaction and the agent’s language skills significantly higher under test conditions in which Gandalf utilized limited conversational behaviour (gaze, turn-taking and beat gesture) than when these behaviours were disabled. In terms of social behaviours, Sproull et al. [1997] showed that subjects rated a female embodied interface significantly lower in sociability and gave it a significantly more negative social evaluation compared to a text-only interface. Subjects also reported being less relaxed and assured when interacting with the embodied interface than when interacting with the text interface.
34
Advances in Natural Multimodal Dialogue Systems
Finally, they gave themselves significantly higher scores on social desirability scales, but disclosed less (wrote significantly less and skipped more questions in response to queries by the interface) when interacting with an embodied interface vs. a text-only interface. Men were found to disclose more in the embodied condition and women disclosed more in the text-only condition. Most of these evaluations have tried to address whether embodiment of a system is useful at all, by including or not including an animated figure. In their survey of user studies on embodied agents, Dehn and van Mulken conclude that there is no “persona effect”, that is a general advantage of an interface with an animated agent over one without an animated agent [Dehn and van Mulken, 2000]. However, they believe that lack of evidence and inconsistencies in the studies performed to date may be attributable to methodological shortcomings and variations in the kinds of animations used, the kinds of comparisons made (control conditions), the specific measures used for the dependent variables, and the task and context of the interaction.
4.3
Related Studies on Mediated Communication
Several studies have shown that people speak differently to a computer than another person, even though there are typically no differences in task outcomes in these evaluations. Hauptmann and Rudnicky [1988] performed one of the first studies in this area. They asked subjects to carry out a simple informationgathering task through a (simulated) natural language speech interface, and compared this with speech to a co-present human in the same task. They found that speech to the simulated computer system was telegraphic and formal, approximating a command language. In particular, when speaking to what they believed to be a computer, subject’s utterances used a small vocabulary, often sounding like system commands, with very few task-unrelated utterances, and fewer filled pauses and other disfluencies. These results were extended in research conducted by Oviatt [Oviatt, 1995; Oviatt and Adams, 2000; Oviatt, 1998], in which she found that speech to a computer system was characterized by a low rate of disfluencies relative to speech to a co-present human. She also noted that visual feedback has an effect on disfluency: telephone calls have a higher rate of disfluency than co-present dialogue. From these results, it seems that people speak more carefully and less naturally when interacting with a computer. Boyle et al. [1994] compared pairs of subjects working on a map-based task who were visible to each other with pairs of subjects who were co-present but could not see each other. Although no performance difference was found between the two conditions, when subjects could not see one another, they compensated by giving more verbal feedback and using longer utterances. Their conversation was found to be less smooth than that between mutually visible
Social Dialogue with Embodied Conversational Agents
35
partners, indicated by more interruptions, and less efficient, as more turns were required to complete the task. The researchers concluded that visual feedback improves the smoothness and efficiency of the interaction, but that we have devices to compensate for this when visibility is restricted. Daly-Jones et al. [1998] also failed to find any difference in performance between video-mediated and audio-mediated conversations, although they did find differences in the quality of the interactions (e.g., more explicit questions in audio-only condition). Whittaker and O’Conaill [1997] survey the results of several studies which compared video-mediated communication with audio-only communication and concluded that the visual channel does not significantly impact performance outcomes in task-oriented collaborations, although it does affect social and affective dimensions of communication. Comparing video-mediated communication to face-to-face and audio-only conversations, they also found that speakers used more formal turn-taking techniques in the video condition even though users reported that they perceived many benefits to video conferencing relative to the audio-only mode. In a series of studies on the effects of different media and activities on trust, Zheng, Veinott et al. have demonstrated that social interaction, even if carried out online, significantly increases people’s trust in each other [Zheng et al., 2002]. Similarly, Bos et al. [2002] demonstrated that richer media – such as face-to-face, video-, and audio-mediated communication – leads to higher trust levels than media with lower bandwidth such as text chat. Finally, a number of studies have been done comparing face-to-face conversations with conversations on the phone [Rutter, 1987]. These studies find that, in general, there is more cooperation and trust in face-to-face interaction. One study found that audio-only communication encouraged negotiators to behave impersonally, to ignore the subtleties of self-presentation, and to concentrate primarily on pursuing victory for their side. Other studies found similar gains in cooperation among subjects playing prisoner’s dilemma face-to-face compared to playing it over the phone. Face-to-face interactions are also less formal and more spontaneous than conversations on the phone. One study found that face-to-face discussions were more protracted and wide-ranging while subjects communicating via audio-only kept much more to the specific issues on the agenda (the study also found that when the topics were more wide-ranging, changes in attitude among the participants was more likely to occur). Although several studies found increases in favourable impressions of interactants in face-to-face conversation relative to audio-only, these effects have not been consistently validated.
36
4.4
Advances in Natural Multimodal Dialogue Systems
Trait-Based Variation in User Responses
Several studies have shown that users react differently to social agents based on their own personality and other dispositional traits. For example, Reeves and Nass have shown that users like agents that match their own personality (on the introversion/ extraversion dimension) more than those which do not, regardless of whether the personality is portrayed through text or speech [Nass and Gong, 2000; Reeves and Nass, 1996]. Resnick and Lammers showed that in order to change user behaviour via corrective error messages, the messages should have different degrees of “humanness” depending on whether the user has high or low self-esteem (“computer-ese” messages should be used with low self-esteem users, while “human-like” messages should be used with highesteem users) [Resnick and Lammers, 1985]. Rickenberg and Reeves showed that different types of animated agents affected the anxiety level of users differentially as a function of whether users tended towards internal or external locus of control [Rickenberg and Reeves, 2000]. In our earlier study on the effects of social dialogue on trust in ECA interactions, we found that social dialogue significantly increased trust for extraverts, while it made no significant difference for introverts [Cassell and Bickmore, 2003]. In light of the studies summarized here, the question that remains is whether these effects continue to hold if the nonverbal cues provided by the ECA are removed.
5.
Social Dialogue in REA
For the purpose of trust elicitation and small talk, we have constructed a new kind of discourse planner that can interleave small talk and task talk during the initial buyer interview, based on the relational model outlined above. An overview of the planner is provided here; details of its implementation can be found in Cassell and Bickmore [2003].
5.1
Planning Model
Given that many of the goals in a relational conversational strategy are nondiscrete (e.g., minimize face threat), and that trade-offs among multiple goals have to be achieved at any given time, we have moved away from static world discourse planning, and are using an activation network-based approach based on Maes’ Do the Right Thing architecture [Maes, 1989]. This architecture provides the capability to transition smoothly from deliberative, planned behaviour to opportunistic, reactive behaviour, and is able to pursue multiple, non-discrete goals. In our implementation each node in the network represents a conversational move that REA can make.
Social Dialogue with Embodied Conversational Agents
37
Thus, during task talk, REA may ask questions about users’ buying preferences, such as the number of bedrooms they need. During small talk, REA can talk about the weather, events and objects in her shared physical context with the user (e.g., the lab setting), or she can tell stories about the lab, herself, or real estate. REA’s conversational moves are planned in order to minimize the face threat to the user, and maximize trust, while pursuing her task goals in the most efficient manner possible. That is, REA attempts to determine the face threat of her next conversational move, assesses the solidarity and familiarity which she currently holds with the user, and judges which topics will seem most relevant and least intrusive to users. As a function of these factors, REA chooses whether or not to engage in small talk, and what kind of small talk to choose. The selection of which move should be pursued by REA at any given time is thus a non-discrete function of the following factors: Closeness REA continually assesses her “interpersonal” closeness with the user, which is a composite representing depth of familiarity and solidarity, modelled as a scalar quantity. Each conversational topic has a predefined, pre-requisite closeness that must be achieved before REA can introduce the topic. Given this, the system can plan to perform small talk in order to “grease the tracks” for task talk, especially about sensitive topics like finance. Topic REA keeps track of the current and past conversational topics. Conversational moves which stay within topic (maintain topic coherence) are given preference over those which do not. In addition, REA can plan to execute a sequence of moves which gradually transition the topic from its current state to one that REA wants to talk about (e.g., from talk about the weather, to talk about Boston weather, to talk about Boston real estate). Relevance REA maintains a list of topics that she thinks the user knows about, and the discourse planner prefers moves which involve topics in this list. The list is initialized to things that anyone talking to REA would know about – such as the weather outside, Cambridge, MIT, or the laboratory that REA lives in. Task goals REA has a list of prioritized goals to find out about the user’s housing needs in the initial interview. Conversational moves which directly work towards satisfying these goals (such as asking interview questions) are preferred. Logical preconditions Conversational moves have logical preconditions (e.g., it makes no sense for REA to ask users what their major is
38
Advances in Natural Multimodal Dialogue Systems
until she has established that they are students), and are not selected for execution until all of their preconditions are satisfied. One advantage of the activation network approach is that by simply adjusting a few gains we can make REA more or less coherent, more or less polite (attentive to closeness constraints), more or less task-oriented, or more or less deliberative (vs. reactive) in her linguistic behaviour. In the current implementation, the dialogue is entirely REA-initiated, and user responses are recognized via a speaker-independent, grammar-based, continuous speech recognizer (currently IBM ViaVoice). The active grammar fragment is specified by the current conversational move, and for responses to many REA small talk moves the content of the user’s speech is ignored; only the fact that the person responded at all is enough to advance the dialogue. At each step in the conversation in which REA has the floor (as tracked by a conversational state machine in REA’s Reaction Module [Cassell et al., 2000a], the discourse planner is consulted for the next conversational move to initiate. At this point, activation values are incrementally propagated through the network (following [Maes, 1989]) until a move is selected whose preconditions are satisfied and whose activation value is over a specified threshold. Within this framework, REA decides to do small talk whenever closeness with the user needs to be increased (e.g., before a task query can be asked), or the topic needs to be moved little-by-little to a desired topic and small talk contributions exist which can facilitate this. The activation energy from the user relevance condition described above leads to REA starting small talk with topics that are known to be in the shared environment with the user (e.g., talk about the weather or the lab).
5.2
Interactional Behaviour during Social Dialogue
Shifts between small talk moves and task moves are marked by conventional contextualization cues – discourse markers and beat gestures. Discourse markers include “so” on the first small talk to task talk transition, “anyway” on resumption of task talk from small talk, and “you know” on transition to small talk from task talk [Clark, 1996]. Prior to producing lengthy utterances, REA gazes away briefly before she starts her turn, partly as a turn-taking and floor-holding move and partly to mask the processing delays in generating long utterances. Finally, REA smiles as soon as she detects that users have started their speaking turns (using audio thresholding of the user’s speech).
5.3
Example Interaction
An interview between REA and a user typically proceeds as shown in the following dialogue. (User responses are only shown in positions in which they
Social Dialogue with Embodied Conversational Agents
39
effect the selection of subsequent moves; REA contributions are her actual utterances.) 1 That microphone is terrible, I hate using those things. 2 Sorry about my voice, this is some engineer’s idea of natural sounding. 3 Are you one of our sponsors? User: Yes 4 Were you at our last sponsor meetings? 5 I got so exhausted at the last sponsor meeting I think I was starting to lose my voice by the end. 6 So, where would you like to live? 7 How many bedrooms do you need? 8 Do you need access to the subway? 9 Is one bath enough? 10 You know, Boston is certainly more expensive than it used to be. 11 Anyway, what can you afford? 12 What kind of down payment can you make? 13 Let me see what I have available. Dialogue 1. “Small Talk” REA.
In this example, REA opens with small talk moves regarding things in her shared physical environment with the user (1-2). She then proceeds to small talk related to sponsors (after establishing that the user is a sponsor). After a few turns, enough closeness has been established (simply by doing small talk) that REA can move into task talk (6-9). However, before bringing up the topic of finance – a topic that is potentially very face threatening for the user – REA decides that additional closeness needs to be established, and moves back into small talk (10). This small talk move not only increases closeness but shifts the topic to finance, enabling REA to then bring up the issue of how much the user is able to afford (11-12). If REA’s adherence to closeness preconditions is reduced, by decreasing the contributions of these preconditions to the activation of joint projects, this results in her engaging in less small talk and being more task goal oriented. If everything else is held constant (relative to the prior example) the following dialogue is produced.
40
Advances in Natural Multimodal Dialogue Systems
1 So, where would you like to live? 2 What can you afford? 3 What kind of down payment can you make? 4 How many bedrooms do you need? 5 Do you need access to the subway? 6 Is one bath enough? 7 Let me see what I have available. Dialogue 2. “Task-only REA”.
In this example, REA does not perform any small talk and sequences the task questions in strictly decreasing order of priority.
6.
A Study Comparing ECA Social Dialogue with Audio-Only Social Dialogue
The dialogue model presented above produces a reasonable facsimile of the social dialogue observed in service encounters such as real estate sales. But, does small talk produced by an ECA in a sales encounter actually build trust and solidarity with users? And, does nonverbal behaviour play the same critical role in human-ECA social dialogue as it appears to play in human-human social interactions? In order to answer these questions, we conducted an empirical study in which subjects were interviewed by REA about their housing needs, shown two “virtual” apartments, and then asked to submit a bid on one of them. For the purpose of the experiment, REA was controlled by a human wizard and followed scripts identical to the output of the planner (but faster, and not dependent on automatic speech recognition or computational vision). Users interacted with one of two versions of REA which were identical except that one had only task-oriented dialogue (TASK condition) while the other also included the social dialogue designed to avoid face threat, and increase trust (SOCIAL condition). A second manipulation involved varying whether subjects interacted with the fully embodied REA – appearing in front of the virtual apartments as a life-sized character (EMBODIED condition) – or viewed only the virtual apartments while talking with REA over a telephone. Together these variables provided a 2x2 experimental design: SOCIAL vs. TASK and EMBODIED vs. PHONE. Our hypotheses follow from the literature on small talk and on trust among humans. We expected subjects in the SOCIAL condition to trust REA more, feel closer to REA, like her more, and feel that they understood each other more
Social Dialogue with Embodied Conversational Agents
41
than in the TASK condition. We also expected users to think the interaction was more natural, lifelike, and comfortable in the SOCIAL condition. Finally, we expected users to be willing to pay REA more for an apartment in the SOCIAL condition, given the hypothesized increase in trust. We also expected all of these SOCIAL effects to be amplified in the EMBODIED condition relative to the PHONE-only condition.
6.1
Experimental Methods
This was a multivariate, multiple-factor, between-subjects experimental design, involving 58 subjects (69% male and 31% female).
6.1.1 Apparatus. One wall of the experiment room was a rear-projection screen. In the EMBODIED condition REA appeared life-sized on the screen, in front of the 3D virtual apartments she showed, and her synthetic voice was played through two speakers on the floor in front of the screen. In the PHONE condition only the 3D virtual apartments were displayed and subjects interacted with REA over an ordinary telephone placed on a table in front of the screen. For the purpose of this experiment, REA was controlled via a wizard-of-oz setup on another computer positioned behind the projection screen. The interaction script included verbal and nonverbal behaviour specifications for REA (e.g., gesture and gaze commands as well as speech), and embedded commands describing when different rooms in the virtual apartments should be shown. Three pieces of information obtained from the user during the interview were entered into the control system by the wizard: the city the subject wanted to live in; the number of bedrooms s/he wanted; and how much s/he was willing to spend. The first apartment shown was in the specified city, but had twice as many bedrooms as the subject requested and cost twice as much as s/he could afford (they were also told the price was “firm”). The second apartment shown was in the specified city, had the exact number of bedrooms requested, but cost 50% more than the subject could afford (but this time, the subject was told that the price was “negotiable”). The scripts were comprised of a linear sequence of utterances (statements and questions) that would be made by REA in a given interaction: there was no branching or variability in content beyond the three pieces of information described above. This helped ensure that all subjects received the same intervention regardless of what they said in response to any given question by REA. Subject-initiated utterances were responded to with either backchannel feedback (e.g., “Really?”) for statements or “I don’t know” for questions, followed by an immediate return to the script. The scripts for the TASK and SOCIAL conditions were identical, except that the SOCIAL script had additional small talk utterances added to it, as
42
Advances in Natural Multimodal Dialogue Systems
described in [Bickmore and Cassell, 2001]. The part of the script governing the dialogue from the showing of the second apartment through the end of the interaction was identical in both conditions. Procedure. Subjects were told that they would be interacting with REA, who played the role of a real estate agent and could show them apartments she had for rent. They were told that they were to play the role of someone looking for an apartment in the Boston area. In both conditions subjects were told that they could talk to REA “just like you would to another person”.
6.1.2 Measures. Subjective evaluations of REA – including how friendly, credible, lifelike, warm, competent, reliable, efficient, informed, knowledgeable and intelligent she was – were measured by single items on nine-point Likert scales. Evaluations of the interaction – including how tedious, involving, enjoyable, natural, satisfying, fun, engaging, comfortable and successful it was – were also measured on nine-point Likert scales. Evaluation of how well subjects felt they knew REA, how well she knew and understood them and how close they felt to her were measured in the same manner. All scales were adapted from previous research on user responses to personality types in embodied conversational agents [Moon and Nass, 1996]. Liking of REA was an index composed of three items – how likeable and pleasant REA was and how much subjects liked her – measured items on ninepoint Likert scales (Cronbach’s alpha =.87). Amount Willing to Pay was computed as follows. During the interview, REA asked subjects how much they were able to pay for an apartment; subjects’ responses were entered as $X per month. REA then offered the second apartment for $Y (where Y = 1.5 X), and mentioned that the price was negotiable. On the questionnaire, subjects were asked how much they would be willing to pay for the second apartment, and this was encoded as Z. The task measure used was (Z - X) / (Y - X), which varies from 0% if the user did not budge from their original requested price, to 100% if they offered the full asking price. Trust was measured by a standardized trust scale [Wheeless and Grotz, 1977] (alpha =.93). Although trust is sometimes measured behaviourally using a Prisoner’s Dilemma game [Zheng et al., 2002], we felt that our experimental protocol was already too long and that game-playing did not fit well into the real estate scenario. Given literature on the relationship between user personality and preference for computer behaviour, we were concerned that subjects might respond differentially based on predisposition. Thus, we also included composite measures for introversion and extroversion on the questionnaire. Extrovertedness was an index composed of seven Wiggins [Wiggins, 1979] extrovert adjective items: Cheerful, Enthusiastic, Extroverted, Jovial, Outgo-
Social Dialogue with Embodied Conversational Agents
43
ing, and Perky. It was used for assessment of the subject’s personality (alpha =.87). Introvertedness was an index composed of seven Wiggins [Wiggins, 1979] introvert adjective items: Bashful, Introverted, Inward, Shy, Undemonstrative, Unrevealing, and Unsparkling. It was used for assessment of the subject’s personality (alpha =.84). Note that these personality scales were administered on the post-test questionnaire. For the purposes of this experiment, therefore, subjects who scored over the mean on introversion-extroversion were said to be extroverts, while those who scored under the mean were said to be introverts.
6.1.3 Behavioural measures. Rates of speech disfluency (as defined in [Oviatt, 1995]) and utterance length were coded from the video data. Observation of the videotaped data made it clear that some subjects took the initiative in the conversation, while others allowed REA to lead. Unfortunately, REA is not yet able to deal with user-initiated talk, and so user initiative often led to REA interrupting the speaker. To assess the effect of this phenomenon, we therefore divided subjects into PASSIVE (below the mean on number of user-initiated utterances) and ACTIVE (above the mean on number of userinitiated utterances). To our surprise, these measures turned out to be independent of introversion/extroversion (Pearson r=0.042), and to not be predicted by these latter variables.
6.2
Results
Full factorial single measure ANOVAs were run, with SOCIALITY (Task vs. Social), PERSONALITY OF SUBJECT (Introvert vs. Extrovert), MEDIUM (Phone vs. Embodied) and INITIATION (Active vs. Passive) as independent variables.
6.2.1 Subjective assessments of REA. In looking at the questionnaire data, our first impression is that subjects seemed to feel more comfortable interacting with REA over the phone than face-to-face. Thus, subjects in the phone condition felt that they knew REA better (F=5.02; p<.05), liked her more (F=4.70; p<.05), felt closer to her (F=13.37; p<.001), felt more comfortable with the interaction (F=3.59; p<.07), and thought REA was more friendly (F=8.65;p <.005), warm (F=6.72; p<.05), informed (F=5.73; p<.05), and knowledgeable (F=3.86; p<.06) than those in the embodied condition. However, in the remainder of the results section, as we look more closely at different users, different kinds of dialogue styles, and users’ actual behaviour, a more complicated picture emerges. Subjects felt that REA knew them (F=3.95; p<.06) and understood them (F=7.13; p<.05) better when she used task-only dialogue face-to-face; these trends were reversed for phone-based interactions. Task-only dialogue was more fun (F=3.36; p<.08) and less tedious (F=8.77;
44
Advances in Natural Multimodal Dialogue Systems
Figure 2.3.
Ratings of TEDIOUS.
Figure 2.4. ing to pay.
Amount subjects were will-
p<.005; see Figure 2.3) when embodied, while social dialogue was more fun and less tedious on the phone. That is, subjects preferred to interact, and felt better understood, face-to-face when it was a question of simply “getting down to business,” and preferred to interact, and felt better understood, by phone when the dialogue included social chit-chat. These results may be telling us that REA’s nonverbal behaviour inadvertently projected an unfriendly, introverted personality that was especially inappropriate for social dialogue. REA’s model of non-verbal behaviour, at the time of this experiment, was limited to those behaviours linked to the discourse context. Thus, REA’s smiles were limited to those related to the ends of turns, and she did not have a model of immediacy or other nonverbal cues for liking and warmth typical of social interaction [Argyle, 1988]. According to Whittaker and O’Connail [1993], non-verbal information is especially crucial in interactions involving affective cues, such as negotiation or relational dialogue, and less important in purely problem-solving tasks. This interpretation of the results is backed up by comments such as this response from a subject in the face-to-face social condition: The only problem was how she would respond. She would pause then just say “OK”, or “Yes”. Also when she looked to the side and then back before saying something was a little bit unnatural.
This may explain why subjects preferred task interactions face-to-face, while on the phone REA’s social dialogue had its intended effect of making subjects feel that they knew REA better, that she understood them better, and that the experience was more fun and less tedious. In our earlier study, looking only at an embodied interface, we reported that extroverts trusted the system more when it engaged in small talk, while introverts were not affected by the use of small talk [Bickmore and Cassell, 2001]. In the current study, these results were re-confirmed, but only in the embodied
Social Dialogue with Embodied Conversational Agents
45
interaction; that is, a three-way interaction between SOCIALITY, PERSONALITY and MEDIUM (F=3.96; p<.06) indicated that extroverts trusted REA more when she used social dialogue in embodied interactions, but there was essentially no effect of user’s personality and social dialogue on trust in phone interactions. Further analysis of the data indicates that this result derives from the substantial difference between introverts and extroverts in the face-to-face task-only condition. Introverts trusted REA significantly more in the faceto-face task-only condition than in the other conditions (p<.03), while extroverts trusted her significantly less in this condition than in the other conditions (p.<01). In light of these new observations, our earlier results indicating that social dialogue leads to increased trust (for extroverts at least) needs to be revised. This further analysis indicates that the effects we observed may be due to the attraction of a computer displaying similar personality characteristics, rather than the process of trust-building. That is, in the face-to-face, task-only condition, both verbal and nonverbal channels appear to have inadvertently indicated that REA was an introvert (also supported by the comments that REA’s gaze-away behaviour was too frequent, an indication of introversion [Wilson, 1977]), and in this condition we find the introverts trusting more, and extroverts trusting less. In all other conditions, the personality cues are either conflicting (a mismatch between verbal and nonverbal behaviour has been demonstrated to be disconcerting to users [Nass and Gong, 2000]) or only one channel of cues is available (i.e. on the phone), yielding trust ratings that are close to the overall mean. There was, nevertheless, a preference by extroverts for social dialogue as demonstrated by the fact that, overall, extroverts liked REA more when she used social dialogue, while introverts liked her more when she only talked about the task (F=8.09; p<.01). Passive subjects felt more comfortable interacting with REA than active subjects did, regardless of whether the interaction was face-to-face or on the phone, or whether REA used social dialogue or not. Passive subjects said that they enjoyed the interaction more (F=4.47; p<.05), felt it was more successful (F=6.04; p<.05) and liked REA more (F=3.24; p<.08), and that REA was more intelligent (F=3.40; p<.08), and knew them better (F=3.42; p<.08) than active subjects. These differences may be explained by the fixed-initiative dialogue model used in the WOZ script. REA’s interaction was designed for passive users – there was very little capability in the interaction script to respond to unanticipated user questions or statements – and user initiation attempts were typically met with uncooperative system responses or interruptions. But, given the choice between phone and face-to-face, passive users preferred to interact with REA face-to-face: they rated her as more friendly (F=3.56; p<.07) and informed (F=6.30; p<.05) in this condition. Passive users also found the phone
46
Advances in Natural Multimodal Dialogue Systems
to be more tedious, while active users also found the phone to be less tedious (F=5.15; p<.05). Active users may have found the face-to-face condition particularly frustrating since processing delays may have led to the perception that the floor was open (inviting an initiation attempt), when in fact the wizard had already instructed REA to produce her next utterance. However, when interacting on the phone, active users differed from passive users in that active users felt she was more reliable when using social dialogue and passive users felt she was more reliable when using task-only dialogue. When interacting face-to-face with REA, there was no such distinction between active and passive users (F=4.67; p<.05).
6.2.2 Effects on task measure. One of the most tantalizing results obtained is that extroverts were willing to pay more for the same apartment in the embodied condition, while introverts were willing to pay more over the phone (F=3.41; p<.08), as shown in Figure 2.4. While potentially very significant, this finding is a little difficult to explain, especially given that trust did not seem to play a role in the evaluation. Perhaps, since we asked our subjects to play the role of someone looking for an apartment, and given that the apartments displayed were cartoon renditions, the subjects may not have felt personally invested in the outcome, and thus may have been more likely to be persuaded by associative factors like the perceived liking and credibility of REA. In fact, trust has been shown to not play a role in persuasion when “peripheral route” decisions are made, which is the case when the outcome is not of personal significance [Petty and Wegener, 1998]. Further, extroverts are not only more sociable, but more impulsive than introverts [Wilson, 1977], and impulse buying is governed primarily by novelty [Onkvisit and Shaw, 1994]. Extroverts did rate face-to-face interaction as more engaging than phone-based interaction (though not at a level of statistical significance), while introverts rated phone-based interactions as more engaging, providing some support for this explanation. It is also possible that this measure tells us more about subjects’ assessment of the house than of the realtor. In future experiments we may ask more directly whether the subject perceived the realtor to be asking a fair price. Perception of fairness of a price may be more linked to trust than is actual price demanded for a property. 6.2.3 Gender effects. Women felt that REA was more efficient (F=5.61; p<.05) and reliable (F=4.99; p<.05) in the embodied condition than when interacting with her over the phone, while men felt that she was more efficient and reliable by phone. Of course, REA has a female body and a female voice and so in order to have a clearer picture of the meaning of these results, a similar study would need to be carried out with a male realtor.
47
Social Dialogue with Embodied Conversational Agents Table 2.2. Speech disfluencies per 100 words.
Disfluencies
Embodied
Phone
Overall
4.83
6.73
5.83
Table 2.3. Speech disfluencies per 100 Words for different types of human-human and simulated human-computer interactions (adapted from Oviatt [Oviatt, 1995]). Human-human speech Two-person telephone call Two-person face-to-face dialogue
8.83 5.5
Human-computer speech Unconstrained computer interaction Structured computer interaction
1.80 0.83
6.2.4 Effects on behavioural measures. Although subjects’ beliefs about REA and about the interaction are important, it is at least equally important to look at how subjects act, independent of their conscious beliefs. In this context we examined subjects’ disfluencies when speaking with REA. Remember that disfluency can be a measure of naturalness – human-human conversation demonstrates more disfluency than does human-computer communication [Oviatt, 1995]. The rates of speech disfluencies (per 100 words) are shown in Table 2.2. Comparing these to results from previous studies (see Table 2.3) indicates that interactions with REA were more similar to humanhuman conversation than to human-computer interaction. When asked if he was interacting with a computer or a person, one subject replied “A computerperson I guess. It was a lot like a human.” There were no significant differences in utterance length (MLU) across any of the conditions. Strikingly, the behavioural measures indicate that, with respect to speech disfluency rates, talking to REA is more like talking to a person than talking to a computer. Once again, there were significant effects of MEDIUM, SOCIALITY and PERSONALITY on disfluency rate (F=7.09; p<.05), such that disfluency rates were higher in TASK than SOCIAL, higher overall for INTROVERTs than EXTROVERTs, higher for EXTROVERTs on the PHONE, and higher for INTROVERTs in EMBODIED condition. These effects on disfluency rates are consistent with our conclusion that REA’s nonverbal behaviours inadvertently projected an introverted and asocial persona, and with the secondary hypoth-
48
Advances in Natural Multimodal Dialogue Systems
esis that the primary driver on disfluency is cognitive load, once the length of the utterance is controlled for [Oviatt, 1995]. Given our results, this hypothesis would indicate that social dialogue requires lower cognitive load than task-oriented dialogue, that conversation requires a higher cognitive load on introverts than extraverts, that talking on the phone is more demanding than talking face-to-face for extraverts, and that talking face-to-face is more demanding than talking on the phone for introverts, all of which seem reasonable.
7.
Conclusion
The complex results of this study give us hope for the future of embodied conversational agents, but also a clear roadmap for future research. In terms of their behaviour with REA, users demonstrated that they treat conversation with her more like human-human conversation than like human-computer conversation. Their verbal disfluencies are the mark of unplanned speech, of a conversational style. However, in terms of their assessment of her abilities, this did not mean that users saw REA through rose-colored glasses. They were clear about the necessity not only to embody the interaction, but to design every aspect of the embodiment in the service of the same interaction. That is, face-to-face conversations with ECAs must demonstrate the same quick timing of nonverbal behaviours as humans (not an easy task, given the state of the technologies we employ). In addition, the persona and nonverbal behaviour of an ECA must be carefully designed to match the task, a conversational style, and user expectations. Relative to other studies on social dialogue and interactions with ECAs, this study has taught us a great deal about how to build ECAs capable of social dialogue and the kinds of applications in which this is important to do. As demonstrated in the study of the August ECA, we found that people will readily conduct social dialogue with an ECA in situations in which there is no time pressure to complete a task or in which it is a normal part of the script for similar human-human interactions, and many people actually prefer this style of interaction. Consequently, we are in the process of developing and evaluating an ECA in the area of coaching for health behaviour change [Bickmore, 2002], an area in which social dialogue – and relationship-building behaviours in general – are known to significantly effect task outcomes. Relative to prior findings in human-human interaction that social dialogue builds trust, our inability to find this effect across all users is likely due to shortcomings in our model, especially in the area of appropriate nonverbal behaviour and lack of uptake of subjects’ conversational contributions. Given these shortcomings, similarity attraction effects appear to have overwhelmed the social-dialogue-trust effects, and we observed introverts liking and trusting REA more when she behaved consistently introverted and extraverts liking and
Social Dialogue with Embodied Conversational Agents
49
trusting her more when she behaved consistently extraverted. Our conclusion from this is that, adding social dialogue to embodied conversational agents will require a model of social nonverbal behaviour consistent with verbal conversational strategies, before the social dialogue fulfils its trust-enhancing role. As computers begin to resemble humans, the bar of user expectations is raised: people expect that REA will hold up her end of the conversation, including dealing with interruptions by active users. We have begun to demonstrate the feasibility of embodied interfaces. Now it is time to design ECAs that people wish to spend time with, and that are able to use their bodies for conversational tasks for which human face-to-face interaction is unparalleled, such as social dialogue, initial business meetings, and negotiation.
Acknowledgements Thanks to Ian Gouldstone, Jennifer Smith and Elisabeth Sylvan for help in conducting the experiment and analyzing data, and to the rest of the Gesture and Narrative Language Group for their help and support.
References Altman, I. and Taylor, D. (1973). Social Penetration: The Development of Interpersonal Relationships. New York: Holt, Rinhart & Winston. Andersen, P. and Guerrero, L. (1998). The Bright Side of Relational Communication: Interpersonal Warmth as a Social Emotion. In Andersen, P. and Guerrero, L., editors, Handbook of Communication and Emotion, pages 303–329. New York: Academic Press. André, E., Muller, J., and Rist, T. (1996). The PPP Persona: A Multipurpose Animated Presentation Agent. In Proceedings of Advanced Visual Interfaces, pages 245–247, Gubbio, Italy. André, E., Rist, T., and Muller, J. (1998). Integrating Reactive and Scripted Behaviors in a Life-Like Presentation Agent. In Proceedings of the Second International Conference on Autonomous Agents, pages 261–268, Minneapolis, Minnesota, USA. Argyle, M. (1988). Bodily Communication. New York: Methuen & Co. Ltd. Argyle, M. (1990). The Biological Basis of Rapport. Psychological Inquiry, 1:297–300. Beskow, J. and McGlashan, S. (1997). Olga: A Conversational Agent with Gestures. In André, E., editor, Proceedings of the IJCAI 1997 Workshop on Animated Interface Agents: Making Them Intelligent, Nagoya, Japan. San Francisco: Morgan-Kaufmann Publishers. Bickmore, T. (2002). When Etiquette Really Matters: Relational Agents and Behavior Change. In Proceedings of AAAI Fall Symposium on Etiquette for Human-Computer Work, pages 9–10, Falmouth, MA.
50
Advances in Natural Multimodal Dialogue Systems
Bickmore, T. and Cassell, J. (2001). Relational Agents: A Model and Implementation of Building User Trust. In Proceedings of CHI 2001, pages 396– 403, Seattle, WA. Bos, N., Olson, J. S., Gergle, D., Olson, G. M., and Wright, Z. (2002). Effects of Four Computer-Mediated Communications Channels on Trust Development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 135–140, Minneapolis, Minnesota, USA. Boyle, E., Anderson, A., and Newlands, A. (1994). The Effects of Visibility in a Cooperative Problem Solving Task. Language and Speech, 37(1):1–20. Brown, P. and Levinson, S. (1978). Universals in Language Usage: Politeness Phenomena. In Goody, E., editor, Questions and Politeness: Strategies in Social Interaction, pages 56–289. Cambridge: Cambridge University Press. Brown, R. and Gilman, A. (1972). The Pronouns of Power and Solidarity. In Giglioli, P., editor, Language and Social Context, pages 252–282. Harmondsworth: Penguin. Brown, R. and Gilman, A. (1989). Politeness Theory and Shakespeare’s Four Major Tragedies. Language in Society, 18:159–212. Cassell, J. and Bickmore, T. (2003). Negotiated Collusion: Modeling Social Language and its Relationship Effects in Intelligent Agents. User Modeling and Adaptive Interfaces, 13(1-2):89–132. Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., and Yan, H. (1999). Embodiment in Conversational Interfaces: Rea. In Proceedings of ACM SIGCHI Conference on Human Factors in Computing Systems, pages 520–527, Pittsburgh, PA. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., and Yan, H. (2000a). Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Embodied Conversational Agents, pages 29–63. Cambridge, MA: MIT Press. Cassell, J., Bickmore, T., H, Vilhjálmsson, and Yan, H. (2000b). More Than Just a Pretty Face: Affordances of Embodiment. In Proceedings of the 5th International Conference on Intelligent User Interfaces, pages 52–59, New Orleans, Louisiana. Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. (2000c). Embodied Conversational Agents. Cambridge, MA: MIT Press. Cheepen, C. (1988). The Predictability of Informal Conversation. New York: Pinter. Chovil, N. (1991). Discourse-Oriented Facial Displays in Conversation. Research on Language and Social Interaction, 25(1991/1992):163–194. Clark, H. H. (1996). Using Language. Cambridge: Cambridge University Press. Coupland, J., Coupland, N., and Robinson, J. D. (1992). How Are You? Negotiating Phatic Communion. Language in Society, 21:207–230.
Social Dialogue with Embodied Conversational Agents
51
Daly-Jones, O., Monk, A. F., and Watts, L. A. (1998). Some Advantages of Video Conferencing over High-Quality Audio Conferencing: Fluency and Awareness of Attentional Focus. International Journal of Human-Computer Studies, 49(1):21–58. Dehn, D. M. and van Mulken, S. (2000). The Impact of Animated Interface Agents: A Review of Empirical Research. International Journal of HumanComputer Studies, 52:1–22. Dunbar, R. (1996). Grooming, Gossip, and the Evolution of Language. Cambridge, MA: Harvard University Press. Duncan, S. (1974). On the Structure of Speaker-Auditor Interaction during Speaking Turns. Language in Society, 3:161–180. Garros, D. (1999). Real Estate Agent, Home and Hearth Realty. Personal communication. Goffman, I. (1967). On Face-Work. In Interaction Ritual: Essays on Face-toFace Behavior, pages 5–46. New York: Pantheon. Gustafson, J., Lindberg, N., and Lundeberg, M. (1999). The August Spoken Dialogue System. In Proceedings of the Eurospeech 1999 Conference, pages 1151–1154, Budapest, Hungary. Hauptmann, A. G. and Rudnicky, A. I. (1988). Talking to Computers: An Empirical Investigation. International Journal of Man-Machine Studies, 8(6): 583–604. Jakobson, R. (1960). Concluding Statement: Linguistics and Poetics. In Sebeok, T., editor, Style in Language, pages 351–377. New York: Wiley. Kendon, A. (1980). Conducting Interaction: Patterns of Behavior in Focused Encounters, volume 7. Cambridge: Cambridge University Press. Kiesler, S. and Sproull, L. (1997). Social Human-Computer Interaction. In Friedman, B., editor, Human Values and the Design of Computer Technology, pages 191–199. Stanford, CA: CSLI Publications. Koda, T. and Maes, P. (1996). Agents with Faces: The Effect of Personification. In Proceedings of the Fifth IEEE International Workshop on Robot and Human Communication (RO-MAN 1996), pages 189–194, Tsukuba, Japan. Laver, J. (1975). Communicative Functions of Phatic Communion. In Kendon, A., Harris, R., and Key, M., editors, The Organization of Behavior in Faceto-Face Interaction, pages 215–238. The Hague: Mouton. Laver, J. (1981). Linguistic Routines and Politeness in Greeting and Parting. In Coulmas, F., editor, Conversational Routine, pages 289–304. The Hague: Mouton. Lester, J., Stone, B., and Stelling, G. (1999). Lifelike Pedagogical Agents for Mixed-Initiative Problem Solving in Constuctivist Learning Environments. User Modeling and User-Adapted Interaction, 9(1-2):1–44. Maes, P. (1989). How to Do the Right Thing. Connection Science Journal, 1(3):291–323.
52
Advances in Natural Multimodal Dialogue Systems
Malinowski, B. (1923). The Problem of Meaning in Primitive Languages. In Ogden, C. K. and Richards, I. A., editors, The Meaning of Meaning, pages 296–346. Routledge & Kegan Paul. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Cambridge: Cambridge University Press. Moon, Y. and Nass, C. I. (1996). How “Real” are Computer Personalities? Psychological Responses to Personality Types in Human-Computer Interaction. Communication Research, 23(6):651–674. Nass, C. and Gong, L. (2000). Speech Interfaces from an Evolutionary Perspective. Communications of the ACM, 43(9):36–43. Onkvisit, S. and Shaw, J. J. (1994). Consumer Behavior: Strategy and Analysis. New York: Macmillan College Publishing Company. Oviatt, S. (1995). Predicting Spoken Disfluencies during Human-Computer Interaction. Computer Speech and Language, 9:19–35. Oviatt, S. and Adams, B. (2000). Designing and Evaluating Conversational Interfaces with Animated Characters. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 319– 345. Cambridge, MA: MIT Press. Oviatt, S. L. (1998). User-Centered Modeling for Spoken Language and Multimodal Interfaces. In Maybury, M. T. and Wahlster, W., editors, Readings in Intelligent User Interfaces, pages 620–630. San Francisco, CA: Morgan Kaufmann Publishers, Inc. Petty, R. and Wegener, D. (1998). Attitude Change: Multiple Roles for Persuasion Variables. In Gilbert, D., Fiske, S., and Lindzey, G., editors, The Handbook of Social Psychology, pages 323–390. New York: McGraw-Hill. Prus, R. (1989). Making Sales: Influence as Interpersonal Accomplishment. Mewbury Park, CA: Sage. Reeves, B. and Nass, C. (1996). The Media Equation. Cambridge: Cambridge University Press. Resnick, P. V. and Lammers, H. B. (1985). The Influence of Self-Esteem on Cognitive Responses to Machine-Like versus Human-Like Computer Feedback. The Journal of Social Psychology, 125(6):761–769. Richmond, V. and McCroskey, J. (1995). Immediacy - Nonverbal Behavior in Interpersonal Relations. Boston: Allyn & Bacon. Rickel, J. and Johnson, W. L. (1998). Task-Oriented Dialogs with Animated Agents in Virtual Reality. In Proceedings of the First Workshop on Embodied Conversational Characters, pages 39–46. Rickenberg, R. and Reeves, B. (2000). The Effects of Animated Characters on Anxiety, Task Performance, and Evaluations of User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 49–56, The Hague, Amsterdam.
Social Dialogue with Embodied Conversational Agents
53
Rutter, D. R. (1987). Communicating by Telephone. New York: Pergamon Press. Schneider, K. P. (1988). Small Talk: Analysing Phatic Discourse. Marburg: Hitzeroth. Sidner, C. and Dzikovska, M. (2005). A First Experiment in Engagement for Human-Robot Interaction in Hosting Activities. In van Kuppevelt, J., Dybkjær, L., and Bernsen, N. O., editors, Advances in Natural Multimodal Dialogue Systems. Springer. This volume. Sproull, L., Subramani, M., Kiesler, S., Walker, J., and Waters, K. (1997). When the Interface is a Face. In Friedman, B., editor, Human Values and the Design of Computer Technology, pages 163–190. Stanford, CA: CSLI Publications. Stone, M. and Doran, C. (1997). Sentence Planning as Description Using TreeAdjoining Grammar. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics (ACL/EACL 1997), pages 198–205, Madrid, Spain. Svennevig, J. (1999). Getting Acquainted in Conversation. Philadelphia: John Benjamins. Takeuchi, A. and Naito, T. (1995). Situated Facial Displays: Towards Social Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 450–455, Denver, Colorado. van Mulken, S., André, E., and Muller, J. (1999). An Empirical Study on the Trustworthiness of Lifelike Interface Agents. In Bullinger, H.-J. and Ziegler, J., editors, Human-Computer Interaction (Proceedings of HCI-International 1999), pages 152–156. Mahwah, NJ: Lawrence Erlbaum Associates. Wheeless, L. and Grotz, J. (1977). The Measurement of Trust and Its Relationship to Self-Disclosure. Human Communication Research, 3(3):250–257. Whittaker, S. and O’Conaill, B. (1993). An Evaluation of Video Mediated Communication. In Proceedings of Human Factors and Computing Systems (INTERACT/CHI 1993), pages 73–74, Amsterdam, The Netherlands. Whittaker, S. and O’Conaill, B. (1997). The Role of Vision in Face-to-Face and Mediated Communication. In Finn, K., Sellen, A., and Wilbur, S., editors, Video-Mediated Communication, pages 23–49. Lawrence Erlbaum Associates, Inc. Wiggins, J. (1979). A Psychological Taxonomy of Trait-Descriptive Terms. Journal of Personality and Social Psychology, 37(3):395–412. Wilson, G. (1977). Introversion/Extraversion. In Blass, T., editor, Personality Variables in Social Behavior, pages 179–218. New York: John Wiley & Sons.
54
Advances in Natural Multimodal Dialogue Systems
Zheng, J., Veinott, E. S., Bos, N., Olson, J. S., and Olson, G. M. (2002). Trust without Touch: Jumpstarting Long-Distance Trust with Initial Social Activities. In Proceedings of the International Conference for Human-Computer Interaction (CHI), pages 141–146.
Chapter 3 A FIRST EXPERIMENT IN ENGAGEMENT FOR HUMAN-ROBOT INTERACTION IN HOSTING ACTIVITIES∗ Candace L. Sidner Mitsubishi Electric Research Labs 201 Broadway Cambridge, MA 02139, USA
[email protected]
Myroslava Dzikovska Department of Computer Science University of Rochester Rochester, NY 14627, USA
[email protected]
Abstract
To participate in collaborations with people, robots must not only see and talk with people but also make use of the conventions of conversation and of the means to be connected to their human counterparts. This chapter reports on initial research on engagement in human-human interaction and applications to stationary robots interacting with humans in hosting activities.
Keywords:
Human-robot interaction, engagement, gestures, looking, conversation, collaboration, hosting.
1.
Introduction
As part of our ongoing research on collaborative interface agents, we have begun to explore engagement in human interaction. Engagement is the process ∗ Portions
of this chapter are reprinted with permission from C. Sidner and M. Dzikovska, “Human-Robot Interaction: Engagement between Humans and Robots for Hosting Activities,” The Fourth IEEE International Conference on Multi-modal Interfaces, October, 2002, pages 123-128. @ 2002 IEEE.
55 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 55–76. © 2005 Springer. Printed in the Netherlands.
56
Advances in Natural Multimodal Dialogue Systems
by which two, or more, participants establish, maintain and end their perceived connection during interactions they jointly undertake. This process includes: establishment of the initial contact with another participant, negotiation of a collaboration to undertake activities of mutual interest, determination of the ongoing intent of the other participant to continue in the interaction, evaluation of one’s own intentions in staying involved, and determination of when to end the interaction. To understand the engagement process we are studying human-to-human engagement in interactions. Our study provides an understanding of the capabilities required for human-robot interaction. At the same time, experimentation with human-robot interaction provides a valid means to test theories about engagement as well as to produce useful technological results. In this chapter we report on our initial experiments in programming a stationary robot to have initial engagement abilities.
2.
Hosting Activities
This research on engagement is framed around the activity of hosting. Hosting activities are a class of collaborative activity in which an agent provides guidance in the form of information, entertainment, education or other services in the user’s environment (which may be an artificial or the natural world) and may also request that the user undertake actions to support the fulfilment of those services. Hosting activities are situated or embedded activities, because they depend on the surrounding environment as well as the participants involved. They are social activities because, when undertaken by humans, they depend upon the social roles that people play to determine the choice of the next actions, timing of those actions, and negotiation about the choice of actions. In this research, the host of the environment are agents serving as guides, either on-screen animated ones or physical robots. Tutoring applications require hosting activities; this chapter reports on experience with a robot host who is acting as a tutor. Some portions of tutoring, such as testing a student’s knowledge, trouble shooting concepts that a student fails to grasp, and keeping track of what a student knows go beyond the informational services of hosting activities. Thus hosting is part of tutoring but not vice versa. Another common hosting activity is hosting a user in a room with a collection of artefacts. In such an environment, the ability of the host to interact with the physical world and visitors becomes essential, and justifies the creation of physical agents. Room hosting is a core activity in tour guiding in museums, and other indoor and outdoor spaces; see [Burgard et al., 1998] for a robot that can guide a museum tour. Sales activities include hosting as part of their mission in order to make customers aware of types of products and features, locations, personnel, and the like. In many activities, hosting may be
Experiment in Engagement for Human-Robot Interaction
57
intermingled with other tasks, e.g. with selling items in retail sales or student performance evaluation tasks in tutoring. Hosting activities are collaborative because neither party completely determines the goals to be undertaken nor the means of reaching the goal; these actions must be shared between the parties. While the visitor’s interests in the room may seem paramount in determining shared goals, the host’s (private) knowledge of the environment also constrains the goals that can be achieved. Typically the goals undertaken will need to be negotiated between visitor and host. Even in tutoring, where the tutor-host’s plans for how to tutor the student may seem to drive the interaction, the tutor and student negotiate on the problems they will undertake in their encounter. This work hypothesizes that by creating computer agents which function more like human hosts, the human participants will focus on the hosting activity and be less distracted by the agent interface. For example, the agent will gaze at the human partner and at domain objects in ways that appropriately indicate the agent’s attention to each. For example, if a human agent directs her gaze at a partner rather than an object, one can infer that the human is interested in her partner. When a human partner gazes away from the robot or objects of discussion, the robot must be able to assess whether the human has lost interest, and if so, determine how to re-establish or end the engagement between the two participants. Assuring that a robot behaves in ways with which people are familiar in human-to-human interactions increases the likelihood that the interaction will not break down due to the robot’s misusing or misunderstanding the cues of engagement.
3.
What is Engagement?
Engagement is fundamentally a collaborative process [Grosz and Sidner, 1990; Grosz and Kraus, 1996], although it also requires significant private planning on the part of each participant in the engagement. Engagement is collaborative principally because the people who are interacting intend to connect together and cannot do so alone. However, they may be less aware of the actions involved in accomplishing the joint engagement goals, e.g. gaze, head and hand gestures, unlike the conscious actions in other types of collaboration. Engagement, like other types of collaborations, consists of establishing the collaborative goal (the goal to be connected), maintaining the connection, and then ending the engagement. The collaboration process may include negotiation of the goal because a potential collaborator might not decide to become engaged right away or at all. In addition, participants might have to negotiate the means to achieve their goals [Sidner, 1994a; Sidner, 1994b]. For engagement in an interaction, participants negotiate the means for achieving engagement through the various ways they maintain engagement, and repair
58
Advances in Natural Multimodal Dialogue Systems
engagement when it appears to be failing. Described this way, engagement is similar to other collaborative activities. Engagement is an activity that contributes centrally to collaboration on other activities in the world and the conversations that support them. In fact, conversation is impossible without engagement. This claim does not imply that engagement is just a part of conversation. Rather engagement is a collaborative process that occurs in its own right, simply to establish connection between people, a natural social phenomenon of human existence. It is entirely possible to engage another without a single word being said and to maintain the engagement process with no conversation. That is not to say that engagement is possible without any communication; it is not. A person who engages another without language must rely effectively on some form of gestural communication to establish the engagement joint goal and to maintain the engagement. Gesture is also a significant feature of face-to-face interaction when conversations take place [McNeill, 1992]. Being engaged with another can also be the sole purpose of an interaction. The use of just a few words and gestures can establish and maintain connection with another when no other intended goals are relevant. For example, an exchange of hellos, a brief exchange of eye contact and a set of good-byes can accomplish an interaction just to be engaged. In such interactions, one can reasonably claim that the only purpose is to be connected. The current work focuses on interactions, ones that include conversations, where the participants wish to accomplish action in the world rather than just the relational connection that engagement can provide. Much of the engagement process can be accomplished by linguistic means only. Evidence for this statement derives from telephone conversations where participants are engaged with each other and have only the words they say, and prosodic effects (pitch, timing, duration, voice quality and the like) to indicate their desire for establishing, continuing and ending their connection to each other. However, in face-to-face interaction, people look at one another as they talk, and make use of gestures to indicate their interest in the other’s communication, and to indicate that they wish to continue while at the same time using other gestures to access other information in the environment. The engagement process must always balance the need to convey ongoing engagement with the conversational partner (or signal its demise) with the need to look at objects in the environment and perform actions called for by the collaboration (in addition to ones that are independent of it), as well as to interpret those gestures from the partner who has the same requirement to balance these needs.
Experiment in Engagement for Human-Robot Interaction
4.
59
First Experiment in Hosting: A Pointing Robot
In order to experiment with engagement in hosting activities, this effort began with a well-delimited problem: appropriate pointing and beat gestures for a stationary robot, called Mel, while conducting a conversation. Mel’s behaviour is a direct product of extensive research on animated pedagogical agents [Johnson et al., 2000]. It shares with those agents concerns about conversational signals and pointing. Unlike those efforts, Mel has greater dialogue capability, and its conversational signalling, including deixis, comes from combining the CollagenTM [Rich et al., 2001; Rich and Sidner, 1998] and Rea architectures [Cassell et al., 2001a]. Furthermore, while on-screen embodied agents [Cassell et al., 2000b] can point to things in an on-screen environment, on-screen agents cannot effectively point in a 3D space. So it seemed appropriate to explore the effects of deictic behaviour with a robot. To build a robot host, the effort relied significantly on the PACO agent [Rickel et al., 2002] built using CollagenTM for tutoring a user on the operation of a gas turbine engine. The PACO agent tutors a student on the procedures needed to control two engines by their various buttons and dials. The robot served as the tutor in this application and took on the task of speaking all the output and pointing to the portions of the display, tasks normally done by an on-screen agent in the PACO system. The student’s operation of the display, through a combination of speech input and mouse clicks, remained unchanged. Understanding of the student’s speech was accomplished with the IBM ViaVoiceTM speech recognizer, the IBM JSAPI1 to parse and interpret utterances, and the CollagenTM middleware to provide dialogue interpretation and next moves in the conversation, to manage the tutoring goals and to provide a student model for tutoring. The PACO screen for gas turbine engine tutoring is shown in Figure 3.1. The agent is represented in a small window, where text, a cursor hand and an iconic face appear. The face changes to indicate six states: the agent is speaking, is listening to the user, is waiting for the user to reply, is thinking, is acting on the interface, and has failed due to a system crash. The cursor hand is used to point out objects in the display. The robotic agent, Mel, is a stationary robot created at Mitsubishi Electric Research Labs, and consists of 5 servomotors to control the movement of the robot’s head, mouth and two appendages. The robot takes the appearance of a penguin. Mel can open and close his beak, move his head in up-down, and left-right combinations, and flap his "wings" up and down. He also has a laser light on his beak, and a speaker provides audio output for him. See Figure 3.2 for Mel pointing to a button on the gas turbine control panel.
1 See
the ViaVoice SDK, at http://www4.ibm.com/software/speech/dev/sdk java.html.
60
Advances in Natural Multimodal Dialogue Systems
Figure 3.1. The PACO agent for gas turbine engine tutoring.
For gas turbine tutoring, Mel sits in front of a large (2 feet x 3 feet) horizontal flat-screen display on which the gas turbine display panel is projected. To conduct a conversation with the student, Mel addresses the student faceon, and beats with his wings at appropriate points in his turn in the conversation. He uses the PACO system to teach the student procedures on the display. When he wishes to point to a button or dial on the display panel, he points with his beak. When he finishes pointing, he addresses the student face-on again. While Mel’s motor operations are extremely limited, they offer enough movement to undertake beat gestures, which indicate new and old information in utterances [Cassell et al., 2001b]. The head movement is also sufficient to point effectively at objects, so that students can readily see the objects on the panel. The architecture of a CollagenTM agent and an application using Mel is shown in Figure 3.3. Specifics of the CollagenTM internal organization and the means by which CollagenTM is connected to applications are beyond the scope of this chapter; see [Rich and Sidner, 1998; Rich et al., 2001] for more information. Basically, the application is connected to the CollagenTM system through the application adapter. The adapter translates between the semantic events CollagenTM understands and the events/function calls understood by the
Experiment in Engagement for Human-Robot Interaction
61
Figure 3.2. Mel pointing to the gas turbine control panel.
application. The agent controls the application by sending events to perform to the application, and the adapter sends performed events to CollagenTM when a user performs actions on the application. CollagenTM is notified of the propositions uttered by the agent via uttered events. They also go to the AgentHome window, which is a graphical component responsible in CollagenTM for showing the agent’s words on screen as well as generating speech in a speechenabled system. The shaded area highlights the components that were added to the standard CollagenTM middleware. With these additions, utterance events go through the Mel annotator and the BEAT system [Cassell et al., 2001b] in order to generate gestures as well as the utterances that Collagen already produces. More details on the architecture and Mel’s function with it can be found in [Sidner and Dzikovska, 2002]. In tutoring, the CollagenTM architecture is instantiated by means of a detailed set of recipes for the tutoring domain that must be specified in the Planning and Discourse module. Recipes are the means by which the hosting environment is specified for the robot. The recipes do not specify dialogue actions, but instead must detail the actions needed to operate the gas turbine panel and the actions needed to tutor students to use the panel. Additional rules in the
62
Advances in Natural Multimodal Dialogue Systems
Agent help the robot tutor decide what to say or do next, by choosing from a list of next moves created by the Planning and Discourse module. The recipes and rules apply to the domain that is being tutored only, and do not affect the engagement mechanisms that determine the robot’s wing and head gestures. The engagement mechanisms are usable in any tutoring activity. However, the tutoring domain recipes do provide a piece of information for the engagement mechanisms, namely what items in the display need to be pointed out. In this way, the current architecture separates linguistic and gesture functions. Thus, like a person, the robot could convey engagement with its linguistic behaviour but also convey the desire to disengage with its gestures.
Figure 3.3. Architecture of Mel.
5.
Making Progress on Hosting Behaviours
Mel is quite effective at pointing in a display so that it can be readily followed by people. Mel’s beak is a large enough pointer to operate in the way
Experiment in Engagement for Human-Robot Interaction
63
that a finger does. Pointing within a very small margin of error (which is assured by careful calibration before Mel begins talking) makes it possible to locate the appropriate buttons and dials on the screen. Mel returns his “gaze” 2 to the student after pointing, which is a signal that he is staying engaged with the user. These two behaviours are a first step in creating engagement. They make it evident to the student that Mel is interacting with the student and that his looks away are not intended to disengage but rather to accomplish a part of the task at hand. However, human engagement is far richer than what this robot can currently do [Sidner et al., 2003]. Most significantly engagement is a twopart activity: engaging behaviours must be produced, and at the same time, the engaging agent must interpret engaging behaviours from its conversational collaborator. Two of the most basic aspects of engagement are beginning and ending it. The means by which one begins and ends an interaction with Mel are minimal. While Mel responds to a student greeting to start the conversation, he does not have any means or goals to decide when and how to begin the conversation itself. Mel also does not know how to end a conversation or when it is appropriate to do so. Furthermore, Mel has only two weak ways of checking on his partner’s signals of engagement during their interaction: to ask "okay?" and await a response from the user after every explanation he offers, and to await (including indefinitely) a user response (utterance or action) after each time he instructs the user to act. In human-to-human interactions, engagement activities range over far more linguistic and gestural behaviours. In more recent system building efforts using Mel, [Sidner et al., 2005], Mel produces a much wider range of interactions to look at and interpret some looking acts from the human participant, to gesture at objects, and to begin and end conversations. All these capabilities result from more sophisticated subsystems for Mel, such as vision algorithms for detecting human faces and sound location algorithms, as well as careful study of human-human scenarios and video data [Sidner et al., 2003] for determining the types of engagement strategies that humans use effectively in hosting situations. To understand more about the variety of behaviours to signal engagement, in the next section, properties of engagement in human-human interaction are discussed in detail, without regard to what and how robots might do this.
6.
Engagement for Human-Human Interaction
Engagement concerns the establishment of connection, the maintenance of it and the ending of the engagement when one or more of the participants in
2 Mel’s
eyes do not move, so to look at the person, the whole head must turn.
64
Advances in Natural Multimodal Dialogue Systems
an interaction desire it. First let us consider the problem of establishing a connection. A typical beginning of an interaction is familiar to everyone. One approaches another person with whom one would like to interact, looks towards them, attempts to catch their eye. Thereafter additional actions take place, called opening ritual greetings [Luger, 1983]. The two people smile and offer greetings. Verbal greetings can be accompanied by physical behaviours, such as hand shakes, hugs or bowing (which is common in Asian cultures). The greetee responds in kind coordination with the greeter. However, this beginning is not the only possible pattern. The greetee could have responded with a request for delay after part of the process (as in "oh hi, give me a minute," and then attending to another matter). Such a response puts the start of the interaction on hold, and the greeter has the option not to continue, just as the greetee does. A third option is illustrated in the scenario presented in Figure 3.4, part 1. This scenario, constructed for purposes of illustrating engagement, presents an interaction between a host and a visitor in a laboratory hosting setting. It illustrates a less than successful attempt at engagement. The visitor does not immediately engage with the host, who uses a greeting and an offer to provide a tour as means of (1) engaging the visitor and (2) proposing a joint activity in the hosting world. Neither the engagement nor the joint activity are accepted by the visitor. The visitor accomplishes this non-acceptance by ignoring the uptake of the engagement activity, which also quashes the tour offer. While there are many variations on the examples above, the point of the initial part of the interaction is to engage, that is, establish connection between the participants. It is a collaborative process, where each offers signals of engagement, which are used to begin the rest of the greeting process. It is collaborative because each person must do their part and intend their part to establish the connection. Failure to intend to participate denies the collaboration and failure to do one’s part causes the greeting to fail, although with failed greetings, the participants may try again. Once an interaction is established, there are a large number of indications of engagement. The list below is illustrative of ways in which engagement is signalled either directly or in concert with some other activity in the interaction. Note that not all of these signals are essential to interaction because people are capable of undertaking instructions and having conversations by phone and asynchronous connections such as email. However, without these signals, the collaboration is a slow enterprise and likely to breakdown because some actions may be interpreted as intending to disengage. Understanding what each signal conveys about engagement allows us to better understand human communication as well as build artificial agents that engage properly. The signals include:
Experiment in Engagement for Human-Robot Interaction
65
Part 1
Host: Hello, I’m the room host. Would you like me to show you around? Visitor: Part 2 Visitor: What is this? Host: That’s a camera that allows a computer to see as well as a person to track people as they move around a room. Visitor: What does it see? Host: Come over here and look at this monitor <points>. It will show you what the camera is seeing and what it identifies at each moment. Part 3 Visitor: Uh-huh. What are the boxes around the heads? Host: The program identifies the most interesting things in the room – faces. That shows it is finding a face. Visitor: Oh, I see. Well, what else is there? Part 4 Host: Let’s take a look at the multi-level screen over there <points> . Visitor: Host: Is there something else you want to see? Visitor: No, I think I’ve seen enough. Bye. Host: Ok. Bye. Figure 3.4.
Scenario for Room Hosting.
talking about and performing the task; turn taking and grounding [Clark, 1996]; timing (i.e. the pace of uptake of a turn); use of gaze at the speaker, gaze away for taking turns [Duncan, 1974; Cassell et al., 2000a], to track the speaker’s gestures with objects; use of gaze by speaker or non-speaker to check on the attention of other; hand gestures for pointing, iconic description, beat gestures, etc., see [Cassel, 2000; Johnson et al., 2000], and in the hosting setting, gestures associated with domain objects;
66
Advances in Natural Multimodal Dialogue Systems
head gestures (e.g. nods, shakes, sideways turns); body stance (i.e. facing towards the other, turning away, standing up when previously sitting and sitting down); facial gestures (not explored in this work but see [Pelachaud et al., 1996]); non-linguistic auditory responses (e.g. snorts, laughs); social relational activities (e.g. telling jokes, role playing, supportive rejoinders). Among these signals, the use of gaze and looks is one of the most direct means of conveying engagement or lack of it. The principle of conversational tracking governs looking in interactions: a participant in a collaborative conversation tracks the other participant’s face during the conversation in balance with the requirement to look away in order to: (1) participate in actions relevant to the collaboration, or (2) multi-task with activities unrelated to the current collaboration, such as scanning the surrounding environment for interest or danger, avoiding collisions, or performing personal activities [Sidner et al., 2005]. The robot reported here is successful in certain simple looking behaviours. It turns its head to point at objects relevant to the collaboration and then turns again to look at the human tutor for its own turn as well as the human’s. However, the robot cannot perform in the inverse capacity. It does not recognize the looks or gazes of its human partner and hence cannot tell whether the human is looking at objects relevant to the collaboration or looking around out of lack of interest and intentions to disengage. By contrast the host in the scenario in Figure 3.4, part 4 notices when the visitor fails to look at the relevant objects for the interaction and the host takes this as evidence of disengagement, which the host pursues. Taking part in the conversation and the collaboration which a conversation serves are clear evidence of engagement. Failing to perform the actions of a collaboration, however, are evidence of the intention to disengage. In the scenario in Figure 3.4, the visitor follows the host to the appropriate place in part 3. Failure to do so would indicate the intention to disengage from the collaborative task (of answering the visitor’s question about what the camera sees). When the visitor fails to follow the host’s pointing in part 4, this failure to uptake a proposed action that is necessary to their collaborative task is also a sign of disengagement. Of course, when the partner does not take up the task, but has some argument for something else to do, the partner is still engaged. The timing of a conversational turn, taking the turn, and offering backchannels are behaviours that are part of acknowledging the contribution of a conversational partner. During a conversation, the participant acting as the hearer
Experiment in Engagement for Human-Robot Interaction
67
for the conversation grounds the contributions of the other participant by indicating that he or she has heard and understood what was said [Clark, 1996]. Taking a turn makes it possible to use the first part of the turn to do so, and uptake of the turn in pace with the other speaker indicates that all is going smoothly. Both taking a turn and maintaining the pace of the turn are evidence of engagement. When a participant does not take a turn or delays the turn while attending to something outside the collaboration, that participant conveys that they no longer are attending to the interaction. Loss of attention is the most basic indicator that a participant both intends to disengage and has begun to do so. Backchannels are the means by which the hearer grounds the speaker’s turn when a full turn is not taken. Backchannels (using such phrases as “mmhm, yes, uh-huh” during pauses in the other participant’s speaking) permit the hearer to indicate they are following what is said. Most backchannels occur following a speaker utterance which uses prosody and duration to indicate that the backchannel is expected. Failure to provide this expected result is also indication that disengagement is intended. Body stance is a means of expressing the primary focus of one’s attention. Attending to another makes it possible to employ the principle of conversational tracking. Facing one’s conversational partner with one’s physical stance thus supports the ability to track the partner. However, many physical tasks that are part of a collaboration may preclude a frontal stance to the other participant. When washing the dishes, one must face the sink and dishes. When looking under the hood of a car for evidence of a problem, both participants must face the car and lean in under the hood. Using spoken linguistic utterances and more occasional looking indicates one’s ongoing engagement. Just how participants decide to balance looking in these types of circumstances is not well understood. Social cues can serve as signals of engagement. Social relational activities that maintain or establish social relationships among the dialogue participants are evidence of engagement. Social activities such as telling jokes, supportive rejoinders to what another has said, role playing, as well as auditory responses (such as snorts of approval/disapproval and laughing) may serve to accomplish a collaboration task by indicating some trouble with the task. However, they also reinforce social relationships among the participants. Bickmore [2003] notes that social dialogue (even without the performance of accompanying physical actions) increases the trust between dialogue participants, a claim also supported in [Bickmore and Cassell, 2001; Katagiri et al., 2001]. Through videotaped sessions of hosts and visitors, we have observed aspects of social relationships. The hosts and visitors in the videotaped sessions tell humorous stories, offer rejoinders or replies that go beyond conveying that the informa-
68
Advances in Natural Multimodal Dialogue Systems
tion just offered was understood, and even take on role playing with the host and the objects being exhibited. Figure 3.5 contains a portion of a transcript of one hosting session. In that session, the visitor and the host spontaneously play the part of two children using the special restaurant table that the host was demonstrating. The reader should note that their play is coordinated and interactive and is not discussed before it occurs. The role-playing begins at 10 in the figure and ends at 17. This segment of the transcript is preceded by the host P having shown the visitor C how restaurant customers order food in an imaginary restaurant using an actual electronic table, and having explained how wait staff might use the new electronic table to assist customers. Note that utterances by P and C are labelled with their letter, a colon, and italics, while other material describes their body actions. Social relational activities such as the role playing in Figure 3.5 allow participants to demonstrate that they are socially connected to one another in a demonstrable way. They are more than just looking at each other and nodding to one another, especially to accomplish their domain goals. They actively seek ways to indicate to the other that they have some social relation to each other. Failure of a participant to take part in social activities indicates trouble in the interaction; the participant may be simply wishing to change the relationship or indicate how she or he feels about it. However, the failure to participate may also indicate that the participant intends to disengage from the interaction. All collaborations have an end, either because the participants give up on the goal, cf. [Cohen and Levesque, 1990], or because the collaboration succeeds in achieving the desired goals. When collaboration on a domain task ends, participants can elect to negotiate an additional task collaboration or refrain from doing so. When they refrain, they then undertake to terminate their collaboration and end the engagement. Their means to do so are presumably as varied as the rituals to begin engagement, but common patterns comprise pre-closing, expressing appreciation, saying goodbye, with an optional handshake, or other gesture, and then moving away from one another. Preclosings [Schegloff and Sacks, 1973] convey that the end is coming. Expressing appreciation is part of a socially determined custom in the US (and many other cultures) when someone has performed a service for an individual. In the hosting data, the visitor expresses appreciation, with acknowledgement of the host. Where the host has had some role in persuading the visitor to participate, the host may express appreciation as part of the preclosing. Moving away is a strong cue that the disengagement has taken place. However, disengagement can be less smooth. In Figure 3.4, part 4, the host’s offer in the hosting activities is not accepted, not by a verbal denial, but by lack of response, an indication of disengagement. The host, who could have chosen to re-state his offer (with some persuasive comments), instead takes a simpler
Experiment in Engagement for Human-Robot Interaction
69
54: P turns head/eyes to C, raises hands up. C’s head down, eyes on table. 55: P moves away from C and table, raises hands and shakes them; moves totally away, fully upright . 56: P: Uh and show you how the system all works C looks at P and nods. 58: P sits down. P: ah 00: P: ah another aspect that we’re P rotates each hand in coordination. C looks at P. 01: P: worried about P shakes hands. 02: P: you know C nods. 04: P: sort of a you know this would fit very nicely in a sort of theme restaurant P looks at C; looks down. 05: C: MM-hm C looks at P, nods at "MM-hm." P: where you have lots of 06: P draws hands back to chest while looking at C. C: MM-hm P: kids C nods, looking at P. 07: P: I have kids. If you brought them to a P has hands out and open, looks down then at C. C still nods, looking at P. 09: P: restaurant like this P brings hands back to chest. C smiles and looks at P. 10: P looks down; at “oh oh” lunges out with arm and together points to table and looks at table. P: they would go oh oh 11: C: one of these, one of these, one of these C points at each phrase above and looks at table. P laughs. 13: P: I want ice cream <point>, I want cake <point> C: yes yes <simultaneous with “cake”> C points at “cake” looks at P, then brushes hair back. P looking at table. 15: P: pizza <points> P looking at table. C: Yes yes French fries <point> C looks at table as starts to point. 16: P: one of everything P pulls hands back ,looks at C. C: yes C looks at P. 17: P: and if the system just ordered stuff right then and there P looks at C, hands out and shakes, shakes again after “there.” C looking at P; brushes hair. C: Right right (said after “there”) 20: P: you’d be in big trouble || P looking at C and shakes hands again in same way as before. C looking at P, nods at ||. 23: C: But your kids would be ecstatic C looking at P. P looks at C, puts hands in lap Figure 3.5.
A Playtime Example.
70
Advances in Natural Multimodal Dialogue Systems
negotiation tack and asks what the visitor would like to see. This aspect of the interaction illustrates the private assessment and planning which individual participants undertake in engagement. Essentially, it addresses the private question: what will keep us engaged? With the question directed to the visitor, the host also intends to re-engage the visitor in the interaction, which is minimally successful. The visitor responds but uses the response to indicate that the interaction is drawing to a close. The closing ritual presented in Figure 3.4 is, in fact, abrupt, given the overall interaction that has preceded it because the visitor does not follow the American cultural convention of expressing appreciation or offering a simple thanks for the activities performed by the host.
7.
Computational Modelling of Human-Human Hosting and Engagement
A more solid basis of study of human hosting is needed. To that end, ongoing analysis of several videotaped interactions between human hosts and visitors in a natural hosting situation provides details about the use of looking in engagement in hosting [Sidner et al., 2003]. Human-to-human hosting is common enough that many people have participated in such an activity in museums, outdoor tours, or retail settings. Gathering data using videotaping is somewhat intrusive on the typical hosting encounter, but not so much so that people are aware of the taping at all times. Their behaviour is a reliable indicator of typical hosting interactions. Hosting involves more than just engagement, because the host and the visitor have to perform a joint task which is related to the hosting activity. Therefore, engagement has to be investigated in a task-oriented setting where the collaboration on a domain task is ongoing. In each videotaped session, the host was a lab researcher, while the visitor was a guest invited by the first author to visit and see the work going on in the lab. The host demonstrated new technology in a research lab to the visitor for between 28 and 50 minutes, with the variation determined by the host and the equipment available. Careful study of this data has provided new insights about the use of one’s looks that has lead to the principle of conversational tracking. Engagement is a collaboration that often happens together with collaboration on a domain task. In effect, at every moment in the hosting interactions, there are two collaborations happening, one for the participants to accomplish hosting (for example, to tour a lab, which is the domain task), and the other for the participants to stay engaged with each other. While the first collaboration provides evidence for the ongoing process of the second, it is not enough in and of itself. Engagement, as has been shown in this article, depends on many gestural actions as well as conversational comments. Furthermore, the initiation of engagement generally takes place before the domain task is explored,
Experiment in Engagement for Human-Robot Interaction
71
and engagement happens when there are no domain tasks being undertaken. Filling out this story is one of our ongoing research tasks. Collaboration on engagement transpires before, during and after collaboration on a domain task. So what theoretical models will help explain this multicollaboration process? One might want to argue that more complex machinery is needed than that so far suggested in conversational models of collaboration, cf. [Grosz and Sidner, 1990; Grosz and Kraus, 1996; Lochbaum, 1998]. However, we believe it is possible to account for engagement within this framework and to use the framework to develop a working computer agent, in particular a robot. In the conversational models of collaboration, collaboration on a domain task or tasks proceeds from group activity and is accompanied by conversation. The conversation reflects the tasks being undertaken through the structure of the segments of the conversation and the intentions conveyed by participants in those segments. Tasks, or as the theory dubs them, goals are modelled by a set of recipes that specify how actions are performed in the domain to achieve the goal. Actions in the recipe can be performed by either participant, and the participants are presumed to mutually believe or come to mutually believe the recipes of the collaboration. Participants also come to believe individual intentions to perform actions in the recipe. The theory assumes that each participant uses the actions and recipes (1) to recognize how actions by the other participant contribute to the goal and (2) to plan his or her own acts. Conversational collaboration theory does not specifically consider the nature of collaboration for engagement as part of conversation, but the theory and model are specified in a generic way that should also apply to engagement as a collaboration. To apply this theory to engagement, our challenge is to specify the set of rules and recipes that participants in hosting believe will achieve the goals of starting, maintaining and ending engagement. Furthermore, to express this theory computationally, a computational participant (such as a robot) must be able to recognize actions that use those recipes. Clearly recipes for the opening and closing of a conversation as a means of starting and ending engagement can be expressed in terms of actions on the part of the participants in a domain task collaboration. What remains to be discovered is the sets of actions and action groups that form the process of maintaining engagement during a domain task collaboration. In particular, turns in the conversation about the domain task as well as certain gaze, body stance and pointing gestures form the class of engagement actions. The exact composition of that class is as yet unclear. What is clear is that the robot has a two-part task: to engage with the visitor and to track engagement behaviours from the visitor. Finally, social relational behaviours play a part in both the domain collaboration and the engagement process. How does one account for the social relational behaviours discussed above in collaboration theory? While social
72
Advances in Natural Multimodal Dialogue Systems
relational behaviours also tell participants that their counterparts are engaged, they are enacted in the context of the domain task collaboration, and hence must function with the mechanisms for that purpose. Intermixing relational connection, a social goal, and domain collaboration are feasible in collaboration theory models. In particular, the goal of making a relational connection can be accomplished via actions that contribute to the goal of the domain task collaboration. However, each collaborator must ascertain through presumably complex reasoning that the actions (and associated recipes) will serve their social goals as well as contribute to the domain goals. Hence they must choose actions that contribute to social goals as well as domain goals. Then they must also ascertain that the social goals are compatible with ongoing engagement collaboration. Furthermore, they must undertake these goals jointly. The remarkable aspect of the preceding playtime example is that the participants do not explicitly agree to demonstrate how kids will act in the restaurant. Rather the host, who has previously demonstrated other aspects of eating in the electronic restaurant, relates the problem of children in a restaurant and begins to demonstrate the matter when the visitor jumps in and participates jointly. The host accepts this participation by simply continuing his part in it. It appears that they are jointly participating in the hosting goal, but at the same time they are also participating jointly in a social interaction. The details that describe and explain how hosting agents and visitors accomplish this second collaboration are an important goal of ongoing research. Not all social behaviours can be interpreted in the context of the domain task. Sometimes participants interrupt their collaborations to tell a story that is either not pertinent to the collaboration or while pertinent, is somehow out of order. These stories are interruptions of the current domain task collaboration and are understood as having some conversational purpose. As interruptions, they signal that engagement is happening as expected as long as the conversational details of the interruption operate to signal engagement. It is not interruptions in general that signal disengagement or a desire to move to disengage; it is failure to take up the interruption that signals disengagement possibilities.
7.1
Open Questions
The discussion above raises a number of questions that must be addressed in ongoing work. First, in the video data, the host and visitor often look away from each other at non-turn taking times, especially when they are displaying or using demo objects. They also look up or towards the other’s face in the midst of demo activities. The conversational collaboration model does not account for the kind of fine detail required to explain gaze changes, nor do the standard models of turn taking. How are we to account for these gaze changes as part of engagement? What drives collaborators to gaze away and back when
Experiment in Engagement for Human-Robot Interaction
73
undertaking actions with objects so that they and their collaborators remain engaged? Second, in the data, participants do not always explicitly acknowledge or accept what another participant has uttered. Sometimes they use laughs, snorts or expressions of surprise (such as “wow”) to indicate that they have heard and understood and even confirm what another has said. These verbal expressions are appropriate because they express appreciation of a joke, a humorous story or outcome of a demo. We are interested in the range and character of these phenomena, as well as how they are generated and interpreted. Third, this chapter argues that much of engagement can be modelled using the computational collaboration theory model of Grosz and Sidner [1990], Grosz and Kraus [1996], and Lochbaum [1998]. However, a fuller computational picture is needed to explain how participants decide to signal engagement as continuing and how to recognize these signals. One such picture has been explored in [Sidner et al., 2005].
8.
A Next Generation Mel
While pursuing theory of human-human engagement, we continue to add new capabilities for Mel that are founded on human communication. To accomplish this, the next generation Mel combines hosting conversations with other research at MERL on face tracking and sound location [Sidner et al., 2005]. This combination makes it possible to locate visitors and then greet them in ways similar to human experience. These vision and sound algorithms, as well as others under development, permit Mel to make use of nodding and changes in looking, which are important indicators in conversation for turn taking as well as expressions of attention. Mel’s architecture continues to evolve to the point that Mel has both a “brain,” performing CollagenTM related functions and a “body,” fusing sensory data to feed to the brain and controlling Mel’s motions. Building a robot that can detect faces, track them and notice when the face disengages for a brief or extended period of time demonstrates more engagement behaviour than has been possible before. One challenge for a robot host is to experiment with techniques for dealing with unexpected speech input. People, it is said, say that darndest things. While the CollagenTM middleware continues to be used for modelling conversation, the struggle continues to find reasonable behaviours for unexpected visitor utterances. For example, when demonstrating a device that requires filling a cup with water, a visitor may make a mistake and spill water on the floor and say “Oops, I spilled water on the floor.” To understand this utterance, the speech recognizer must correctly process the words, and the sentence semantics must give it a meaningful description, after which the conversation engine must determine the purpose of the meaning description. Finally Mel must re-
74
Advances in Natural Multimodal Dialogue Systems
spond to it. If this sort of utterance was not predicted to occur (and there will be many such utterances), the best response that Mel can currently produce is “I do not understand the purpose of your utterance. Please find a human to help me.” Even that response is only possible if Mel has understood the visitor utterance up to its purposive intent. Failures at speech recognition or sentence interpretation will produce even less informative error messages. These difficulties result from the limits of current speech understanding technology and continue to limit the naturalness of interaction with Mel. Research on improving speech recognition and sentence interpretation is therefore essential to the next generation Mel to improve interaction with human subjects.
9.
Summary
Hosting activities are a natural and common activity among humans, and one that can be accommodated by human-robot interaction. Making the human-machine experience a natural one requires understanding the nature of engagement and applying the same types of human engagement behaviour to robot participation in hosting activities, as well as other activities. Engagement is a collaborative activity that is accomplished through both linguistic and gestural means. The experiments described in this chapter with a stationary robot that can converse and point provide an initial example of an engaged conversationalist. Through study of human-human hosting activities, new models of engagement for human-robot hosting interaction will provide us with a more detailed means of interacting between humans and robots.
Acknowledgements The authors wish to acknowledge the work of Paul Dietz in creating Mel, Neal Lesh, Charles Rich, and the late Jeff Rickel on Collagen and PACO.
References Bickmore, T. (2003). Relational Agents: Effecting Change through HumanComputer Relationships. PhD thesis, Media Arts and Sciences, Massachusetts Institute of Technology. Bickmore, T. and Cassell, J. (2001). Relational Agents: A Model and Implementation of Building User Trust. In Proceedings of the International Conference for Human-Computer Interaction (CHI), pages 396–403. New York: ACM Press. Burgard, W., Cremes, A. B., Fox, D., Haehnel, D., Lakemeyer, G., Schulz, D., Steiner, W., and Thrun, S. (1998). The Interactive Museum Tour Guide Robot. In Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI), pages 11–18. Menlo Park, CA: AAAI Press.
Experiment in Engagement for Human-Robot Interaction
75
Cassel, J. (2000). Nudge Nudge Wink Wink: Elements of Face-to-Face Conversation for Embodied Conversational Agents. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 1–27. Cambridge, MA: MIT Press. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., and Yan, H. (2000a). Human Conversation as a System Framework: Designing Embodied Conversational Agents. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 29–63. Cambridge, MA: MIT Press. Cassell, J., Nakano, Y. I., Bickmore, T. W., Sidner, C. L., and Rich, C. (2001a). Non-Verbal Cues for Discourse Structure. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL), pages 106–115. Menlo Park, CA: Morgan Kaufman Publishers. Cassell, J., Sullivan, J., Prevost, S., and Churchill, E. (2000b). Embodied Conversational Agents. Cambridge, MA: MIT Press. Cassell, J., Vilhjálmsson, H., and Bickmore, T. W. (2001b). BEAT: The Behavior Expression Animation Toolkit. In Proceedings of SIGGRAPH 2001, pages 477–486. New York: ACM Press. Clark, H. H. (1996). Using Language. Cambridge: Cambridge University Press. Cohen, P. and Levesque, H. (1990). Persistence, Intention and Commitment. In Cohen, P., Morgan, J., and Pollack, M. E., editors, Intentions in Communication, pages 33–70. Cambridge, MA: MIT Press. Duncan, S. (1974). Some Signals and Rules for Taking Speaking Turns in Conversation. In Weitz, S., editor, Nonverbal Communication. New York: Oxford University Press. Grosz, B. J. and Kraus, S. (1996). Collaborative Plans for Complex Group Action. Artificial Intelligence, 86(2):269–357. Grosz, B. J. and Sidner, C. L. (1990). Plans for Discourse. In Cohen, P., Morgan, J., and Pollack, M., editors, Intentions in Communication, pages 417– 444. Cambridge: MIT Press. Johnson, W. L., Rickel, J. W., and Lester, J. C. (2000). Animated Pedagogical Agents: Face-to-Face Interaction in Interactive Learning Environments. International Journal of Artificial Intelligence in Education, 11:47–78. Katagiri, Y., Takahashi, T., and Takeuchi, Y. (2001). Social Persuasion in Human-Agent Interaction. In Proceedings of the Second IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, pages 64–69. Menlo Park, CA: Morgan Kaufman Publishers. Lochbaum, K. E. (1998). A Collaborative Planning Model of Intentional Structure. Computational Linguistics, 24(4):525–572. Luger, H. H (1983). Some Aspects of Ritual Communication. Journal of Pragmatics, 7(3):695–711.
76
Advances in Natural Multimodal Dialogue Systems
McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press. Pelachaud, C., Badler, N., and Steedman, M. (1996). Generating Facial Expressions for Speech. Cognitive Science, 20(1):1–46. Rich, C. and Sidner, C. L. (1998). COLLAGEN: A Collaboration Manager for Software Interface Agents. User Modeling and User-Adapted Interaction, 8(3/4):315–350. Rich, C., Sidner, C. L., and Lesh, N. (2001). COLLAGEN: Applying Collaborative Discourse Theory to Human-Computer Interaction. AI Magazine, Special Issue on Intelligent User Interfaces, 22(4):15–25. Rickel, J., Lesh, N., Rich, C., Sidner, C. L., and Gertner, A. (2002). Collaborative Discourse Theory as a Foundation for Tutorial Dialogue. In Proceedings of Intelligent Tutoring Systems. New York: ACM Press. Schegloff, E. and Sacks, H. (1973). Opening Up Closing. Semiotica, 7(4):289– 327. Sidner, C. L. (1994a). An Artificial Discourse Language for Collaborative Negotiation. In Proceedings of the Twelfth National Conference on Artificial Intelligence, volume 1, pages 814–819. Cambridge, MA: MIT Press. Sidner, C. L. (1994b). Negotiation in Collaborative Activity: A Discourse Analysis. Knowledge-Based Systems, 7(4):265–267. Sidner, C. L. and Dzikovska, M. (2002). Hosting Activities: Experience with and Future Directions for a Robot Agent Host. In Proceedings of the 2002 Conference on Intelligent User Interfaces, pages 143–150. New York: ACM Press. Sidner, C. L., Kidd, C., Lee, C., Lesh, N., and Rich, C. (2005). Explorations in Engagement for Humans and Robots. Artificial Intelligence. Forthcoming. Sidner, C. L., Lee, C., and Lesh, N. (2003). Engagement by Looking: Behaviors for Robots when Collaborating with People. In Kruijff-Korbayova, I. and Kosny, C., editors, DiaBruck: The Proceedings of the Seventh Workshop on the Semantics and Pragmatics of Dialogue, pages 123–130, University of Saarland, Germany.
PART II
ANNOTATION AND ANALYSIS OF MULTIMODAL DATA: SPEECH AND GESTURE
Chapter 4 FORM An Extensible, Kinematically-Based Gesture Annotation Scheme Craig H. Martell Department of Computer Science and The MOVES Institute∗ Naval Postgraduate School, Monterey, CA, USA [email protected]
Abstract
Annotated corpora have played a critical role in speech and natural language research; and, there is an increasing interest in corpora-based research in sign language and gesture as well. We present a non-semantic, geometrically-based annotation scheme, FORM, which allows an annotator to capture the kinematic information in a gesture just from videos of speakers. In addition, FORM stores this gestural information in Annotation Graph format—allowing for easy integration of gesture information with other types of communication information, e.g., discourse structure, parts of speech, intonation information, etc.1
Keywords:
Gesture, annotation, corpora, corpus-based methods, multimodal communication.
1.
Introduction
FORM2 is an annotation scheme designed both to describe the kinematic information in a gesture, as well as to be extensible in order to add speech and other conversational information.
∗ Much
of this work was done at the University of Pennsylvania and at The RAND Corporation as well. presentation is a modified version of [Martell, 2002]. 2 The author wishes to sincerely thank Adam Kendon for his input on the FORM project. He has provided not only suggestions as to the direction of the project, but also his unpublished work on a kinematicallybased gesture annotation scheme was the FORM project’s starting point [Kendon, 2000]. 1 This
79 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 79–95. © 2005 Springer. Printed in the Netherlands.
80
Advances in Natural Multimodal Dialogue Systems
Our goal is to build an extensible corpus of annotated videos in order to allow for general research on the relationship among the many different aspects of conversational interaction. Additionally, further tools and algorithms to add additional annotations and evaluate inter-annotator agreement will be developed. The end result of this work will be a corpus of annotated conversational interaction, which can be: extended to include new types of information concerning the same conversations; as new tag-sets and coding schemes are developed— discourse-structure or facial-expression, for example—new annotations could easily be added; used to test scientific hypotheses concerning the relationship of the paralinguistic aspects of communication to speech and to meaning; used to develop statistical algorithms to automatically analyze and generate these paralinguistic aspects of communication (e.g., for HumanComputer Interface research).
2.
Structure of FORM
FORM3 is designed as a series of tracks representing different aspects of the gestural space. Generally, each independently moved part of the body has two tracks, one track for Location/Shape/Orientation, and one for Movement. When a part of the body is held without movement, a Location object describes its position and spans the amount of time the position is held. When a part of the body is in motion, Location objects of zero duration are placed at the beginning and end of the movement. Location objects of zero duration are also used to indicate the Location information at critical points in certain complex gestures An object in a movement track spans the time period for which the body part in question is in motion. It is often the case that one part of the body will remain static while others move. For example, a single hand shape may be held throughout a gesture in which the upper arm moves. FORM’s multi-track system allows such disparate parts of single gestures to be recorded separately and efficiently and to be viewed easily once recorded. Once all tracks are filled with the appropriate information, it is easy to see the structure of a gesture broken down into its anatomical components. At the highest level of FORM are groups. Groups can contain subgroups. Within each group or subgroup are tracks. Each track contains a list of attributes concerning a particular part of the arm or body. At the lowest level 3 The author wishes to acknowledge Jesse Friedman and Paul Howard in this section.
here is from their Code Book section of http://www.ldc.upenn.edu/Projects/FORM/.
Most of what is written
81
FORM
(under each attribute), all possible values are listed. Described below are the tracks for the Location of the Right or Left UpperArm. Right/Left Arm Upper Arm (from the shoulder to the elbow). Location Upper arm lift (from side of the body) no lift 0-45 approx. 45 45-90 approx. 90 90-135 approx. 135 135-180 approx. 180 Relative elbow position: The upper arm lift attribute defines a circle on which the elbow can lie. The relative elbow position attribute indicates where on that circle the elbow lies. Combined, these two attributes provide full information about the location of the elbow and reveal total location information (in relation to the shoulder) of the upper arm. extremely inward inward front front-outward outward (in frontal plane) behind far behind Figure 4.1 - Figure 4.4 are example stills with the appropriate values of the above two attributes given. The next three attributes individually indicate the direction in which the biceps muscle is pointed in one spatial dimension. Taken together, these three attributes give the orientation of the upper arm.
82
Advances in Natural Multimodal Dialogue Systems
Figure 4.1.
Upper arm lift: approx. 90; Relative elbow position: outward.
Figure 4.2.
Upper arm lift: approx. 45; Relative elbow position: front.
Figure 4.3.
Upper arm lift: 0-45; Relative elbow position: behind.
Biceps: Inward/Outward none inward (see Figure 4.5) outward (see Figure 4.6)
83
FORM
Figure 4.4. Upper arm lift: no lift; Relative elbow position: outward.
Biceps: Upward/Downward none upward (see Figure 4.7) downward (see Figure 4.8) Biceps: Forward/Backward none forward (see Figure 4.9) backward (see Figure 4.10)
Figure 4.5. inward.
Obscured: This is an binary attribute which allows the annotator to indicate if the attributes and values chosen were “guesses” necessitated by visual occlusion. This attribute is present in each of FORM’s tracks. Again, we have only presented the Location tracks for the Right or Left Arm UpperArm group. The full “Code Book” can be found at
84
Advances in Natural Multimodal Dialogue Systems
Figure 4.6. outward.
Figure 4.7. upward.
Figure 4.8.
downward.
http://www.ldc.upenn.edu/Projects/FORM/. Listed there are all the Group, Subgroup, Track, Attribute and Value possibilities.
85
FORM
Figure 4.9.
Figure 4.10.
3.
forward.
backward.
Annotation Graphs
In order to allow for maximum extensibility, FORM uses annotation graphs (AGs) as its logical representation4 . As described in [Bird and Liberman, 1999], annotation graphs are a formal framework for “representing linguistic annotations of time series data.” AGs do this by extracting away from the physical-storage layer, as well as from application-specific formatting, to provide a “logical layer for annotation systems.” An annotation graph is a collection arcs and nodes which share a common time line, that of a video tape, for example. Each node represents a time stamp and each arc represents some linguistic event spanning the time between the nodes. In FORM, the arcs are labelled with both attributes and values, so that the arc given by the 4-tuple (1,5,Wrist Movement,Side-to-side) represents that there was side-to-side wrist movement between time stamp 1 and time stamp 5.
4 Cf.
[Martell, 2002] for a more complete discussion of FORM’s use of AGs.
86
Advances in Natural Multimodal Dialogue Systems
The advantage of using annotation graphs as the logical representation is that it is easy to combine heterogeneous data—as long as they share a common time line. So, if we have a dataset consisting of gesture-arcs, as above, we can easily extend this dataset by adding more arcs representing discourse structure, for example, simply by adding other arcs which have discourse-structure attributes and values. Again, this allows different researchers to use the same linguistic data for many different purposes, while, at the same time, allowing others to explore the correlations between the different phenomena being studied.
4.
Annotation Example
To gain a better understanding of the process of FORM annotation, we present here a small visual example. The four stills of Figure 4.11 are from a video sequence of Brian MacWhinney teaching a research methods course at Carnegie Mellon University5 . We show these four key frames, here, for illustrative purposes only. The character of the gesture is gleaned from viewing the continuous movement in the video. However, these key frames would be used to set the time stamp and locations of the beginning and end of the movement and in-between points that are important to capturing its shape. The arcs in the annotation graph described below (Figure 4.13) capture the information for the movement in between the key frames.
Figure 4.11. Snapshots of Brian MacWhinney on January 24, 2000.
The FORM annotation, then, of the video, from time stamp 1:13.34 (1 minute 13.34 seconds) to time stamp 1:14.01 is shown in Figure 4.12. This is the view on the data that a particular tool, Anvil [Kipp, 2001], presents to the annotator6 . As described above, FORM uses annotation graphs as its logical 5 These
data were chosen because they are part of the TalkBank collection (http://www.talkbank.org). TalkBank was responsible for funding a large part of this project. 6 Anvil is described in further detail in the appendix.
87
FORM
Figure 4.12.
FORM annotation of Jan24.mov, using Anvil as the annotation tool.
representation of the data; so regardless of choice of annotation tool, FORM’s internal view is the annotation graph given in Figure 4.13.
Figure 4.13.
FORM/Annotation Graph representation of example gesture.
88
Advances in Natural Multimodal Dialogue Systems
Again, FORM uses vectors of attribute:value pairs to capture the gestural information of each section of the arms and hands. In Figure 4.13, then, the arc labelled HandandWrist.Movement from 1:13.34 to 1:13.57 encodes the kinematics of Brian’s moving his right hand or wrist during this time period, and the arc from 1:13.24 to 1:13.67 encodes a change in his right hand’s shape.7
5.
Preliminary Inter-Annotator Agreement Results
Preliminary results from FORM show that with sufficient training, agreement among the annotators can be very high. Table 4.1 shows preliminary interannotator agreement results from a FORM pilot study.8 The results are for two trained annotators for approximately 1.5 minutes of Jan24-09.mov, the video from Figure 4.11. For this clip, the two annotators agreed that there were at least these 4 gesture excursions. One annotator found 2 additional excursions. Precision refers to the decimal precision of the time stamps given for the beginning and end of gestural components. The SAME value means that all time-stamps were given the same value. This was done in order to judge agreement with having to judge the exact beginning and end of an excursion factored out. Exact vs. No-Value percentage refers to whether both the attributes and values matched exactly or whether just the attributes matched exactly. This distinction is included because a gesture excursion is defined as all movement between two rest positions of the arms and hands. For an excursion, the annotators have to judge both which parts of the arms and hands are salient to the movement (e.g., upper-arm lift and rotation, forearm change in orientation and hand/wrist position) as well as what values to assign (e.g., the upper-arm lifted 15-degrees and rotated 45-degrees). So, the No-Value% column captures the degree to which the annotators agree just on the structure of the movement, while Exact% measures agreement on both structure and values. The degree to which inter-annotator agreement varies among these gestures might suggest difficulty in reaching consensus. However, the results on intraannotator agreement studies demonstrate that a single annotator shows similar variance when doing the same video-clip at different times. Table 4.2 gives the intra-annotator results for one annotator annotating the first 2 gesture excursions of Jan24-09.mov. For both sets of data, the pattern is the same: the less precise the time-stamps, the better the results; No-Value% is significantly higher than Exact%. 7 For the example given in Figure 4.11, Brian is only moving his right hand. Accordingly, the Right. which normally would have been prefixed to the arc-labels has been left off. 8 Essentially, all the arcs for each annotator are thrown into a bag. Then all the bags are combined and the intersection is extracted. This intersection constitutes the overlap in annotation, i.e., where the annotators agreed. The percentage of the intersection to the whole is then calculated to get the scores presented.
89
FORM Table 4.1. Inter-Annotator Agreement on Jan24-09.mov. Gesture Excursion
Precision
Exact%
No-Value%
1
2 1 0 SAME
3.41 10.07 29.44 56.92
4.35 12.8 41.38 86.15
2
2 1 0 SAME
37.5 60 75.56 73.24
52.5 77.5 94.81 95.77
3
2 1 0 SAME
0 19.25 62.5 67.61
0 27.81 86.11 95.77
4
2 1 0 SAME
10.2 25.68 57.77 68.29
12.06 31.72 77.67 95.12
Table 4.2. Intra-Annotator Agreement on Jan24-09.mov. Gesture Excursion
Precision
Exact%
No-Value%
1
0 1 0 SAME
5.98 20.52 58.03 85.52
7.56 25.21 74.64 96.55
2
2 1 0 SAME
0 25.81 89.06 90.91
0 28.39 95.31 93.94
It is also important to note that Gesture Excursion 1 is far more complex than Gesture Excursion 2. And, in both simple and complex gestures, interannotator agreement is approaching intra-annotator agreement. Notice, also, that for Excursion 2, inner-annotator agreement is actually better than intraannotator agreement for the first two rows. This is a result of the difficulty for even the same person over time to precisely pin down the beginning and end of a gesture excursion. Although the preliminary results are very encouraging,
90
Advances in Natural Multimodal Dialogue Systems
all of the above suggests that further research concerning training and how to judge similarity of gestures is necessary. Visual information may need very different similarity criteria. Also, it is not clear as of the time of writing how these results might generalize. In particular, the relationship between interand intra-annotator agreement needs to be further explored. In addition, comparison studies with other methods of judging agreement are necessary. For example, how does FORM’s method compare with Cronbach’s alpha evaluated at discrete time-slices9 ? And, what would be the result adding a kappa-score analysis to the bag-of-arcs technique?
6.
Conclusion: Applications to HLT and HCI?
We are augmenting FORM to include richer paralinguistic information (Head/Torso Movement, Transcription/Syntactic Information, and Intonation/Pitch Information). This will create a corpus that allows for research that heretofore we have been unable to do. It will facilitate experiments that we predict will be useful for speech recognition and other Human-Language Technologies (HLT). As an example of similar research, consider the work of Francis Quek et al. [2001]. They have been able to demonstrate that gestural information is useful in helping with automatic detection of discourse transition. However, their results are limited by the amount of kinematic information they can gather with their video-capture system. Further, we believe an augmented-FORM corpus will contain much more specific data and will allow for more fine-grained analyses than is currently feasible. Additionally, knowing the relationships among the different facets of human conversation will allow for more informed research in Human-Computer Interaction (HCI). If one of the goals of HCI is to have better immersive-training, then it will be imperative that we understand the subtle connections among the paralinguistic aspects of interaction. A virtual human, for example, would be much better if it were able to understand, and act in accordance with, all of our communicative modalities. Having an extensible corpus such as we describe in this chapter is a first step that will allow many researchers, across many disciplines, to explore these and other useful ideas.
9I
am indebted to an anonymous reviewer for this suggestion.
91
FORM
Appendix: Other Tools, Schemes and Methods of Gesture Analysis FORM has been designed to be simultaneously useful for both: (1) capturing the kinematics of gesture; and (2) developing a corpus of annotated videos useful for computational analysis and synthesis. Prior research along each of these dimensions that has contributed to, or has motivated, FORM. In this section we briefly review this prior work.
A.1
Non-computational Gesture Analysis
Two important figures in linguistic and psychological (read: non-computational) analysis of gesture are David McNeil and Adam Kendon. Each has developed annotation schemes and systems to analyze and annotate the gestures of speakers in video. However, their respective levels of analysis are quite different.
A.1.1 David McNeill: Hand and Mind. David McNeil [1992] uses as scheme which divides the gesture space into four basic types: Beat gestures; Iconic gestures; Metaphoric gestures; and Deictic gestures. These categories are not meant to be mutually exclusive, although McNeill [1992] has been blamed for making it appear so. According to the McNeill Lab web site: A misconception has arisen about the nature of the gesture categories described in Hand and Mind, to wit, that they are mutually exclusive bins into which gestures should be dumped. In fact, pretty much any gesture is going to involve more than one category. Take a classic upward path gesture of the sort that many subjects produce when they describe the event of the cat climbing up the pipe in our cartoon stimulus. This gesture involves an iconic path-for-path mapping, but is also deictic. . . . Even "simple" beats are often made in a particular location which the speaker has given further structure (e.g. by setting up an entity there and repeatedly referring to it in that spatial location). Metaphoric gestures are de facto iconic gestures. . . . The notion of a type, therefore, should be considered as a continuum–with a given gesture having more or less iconicity, metaphoricity, etc.10 10 http://mcneilllab.uchicago.edu/topics/type.html,
as of 12/15/2003.
92
Advances in Natural Multimodal Dialogue Systems
This work has been very influential, and has been the basis for at least one major computational project (see the BEAT toolkit, below). However, this level of analysis only serves to categorize the gesture. It provides no useful computational information for either automatic gesture analysis or for the automatic generation of gestures in computational agents.
A.1.2 Adam Kendon: The Kinetics of Gesture. Adam Kendon’s approach, best articulated in “An Agenda for Gesture Studies” [Kendon, 1996], is to annotate and analyze at a more fine-grained, level. His goal is to develop a “kinetics” of gesture, analogous to the “phonetics” of speech. As such, he develops in [Kendon, 2000] a scheme which captures how joints are bent, how the different aspects of the arm move, and even how these different dimensions of gesture align with speech. This system describes positions and changes in position of the speakers arms, hands, head and torso. Unfortunately, from our perspective, the annotation scheme was designed to be written on paper or to be used with a word processor. As such, there is not a sufficient way to do finegrained time alignment of gesture to speech. FORM’s original motivation was to computerize this scheme so that fine-grained time alignment was possible. Kendon’s work is the fundamental starting point for FORM.
A.2
Computer-based Annotation Tools and Systems
A.2.1 CHILDES/CLAN. The CHILDES/CLAN system [MacWhinney, 1996] is a suite of tools for studying conversational interactions in general. The suite allows for, among other things, the coding and analyzing of transcripts and for linking those transcripts to digitized audio and video. CLAN supports both CHAT and CA (Conversational Analysis) notation, with the alignment of text to the digitized media at the phrase level. The CHILDES/CLAN system has the major advantage of being one of the first of its kind. The CHILDES database of transcripts of parent-child interactions has dramatically pushed forward both the theory and science of linguistics and language-acquisition. Additionally, it appears possible—in the future—to integrate FORM data with that developed by CHILDES/CLAN into a unified data set. This is due to the open-ended nature and extensibility of both systems. However, from the perspective of actually annotating videos with fine-grained, time-aligned gesture data, CLAN presents a problem. It is possible to describe the gesture that occurred during an utterance, but, given that time alignment is only at the phrasal level, we are unable to finely associate the parts of the gesture with other aspects of conversational interaction. A.2.2 SignStream. SignStream [Neidle et al., 2001] allows users to annotate video and audio language data in multiple parallel fields that display the temporal alignment and relations among events. It has been used most
93
FORM
extensively for analysis of signed languages. It allows for annotation of manual and non-manual (head, face, body) information; type of message (e.g. Whquestion); parts of speech; and spoken-language translations of sentences. Although SignStream would work with the FORM annotation scheme, and there has been some attempt at integrating the two projects, its interface is too comprehensive. Anvil, described below, allows an annotator to quickly see the relationship among all the aspects of left arm, right arm, head and torso movement.
A.2.3 Anvil. Anvil [Kipp, 2001] is a Java-based tool which permits multi-layered annotation of video with gesture, posture, and discourse information. The tags used can be freely specified, and can easily be hierarchically arranged. See Figure 4.12 as an example. Anvil is the tool of choice for the work done in the FORM Lab. It works well for creating multi-tiered, hierarchical, time-aligned annotations. In the beginning of FORM, we toyed with the idea of building FORMTool, our own annotation tool. However, we soon realized that we were just duplicating the benefits of Anvil. Additionally, the extensible nature of Anvil will allow for the development of an Annotation Graph plug-in, so our data can be directly exported to AG format. Currently, we save the data in Anvil XML format and convert to AG format. This future plug-in will avoid this step.
A.3
Systems for Computational Analysis and Generation
A.3.1 VISLab: Francis Quek. The VISLab project11 is a large-scale, low-level-of-analysis research project developed and led by Francis Quek at Wright State University. It has achieved significant results in understanding the relationship of speech to gesture. See [Quek et al., 2001], for example. The long-term intent of the project is to create a large-scale dataset of videos annotated with information about gesture, speech and gaze. This project is in the same spirit as the FORM project, and there has been significant collaboration. There are plans for the VISLab to store the gesture aspects of their data in the FORM format. There are, however, major differences between the two projects. Firstly, FORM aims at developing a mid-level representation that humans can use to annotate gestures and that machines can use to analyze and generate gestures. The VISLab system’s level of representation is much lower. They use multiple cameras to extract 3D information about position, velocity, acceleration, etc. concerning a gesture. They are doing the physics of gesture, where FORM is looking at something closer to the phonetics of gesture. Secondly, the VISLab system requires a complex set up
11 http://vislab.cs.wright.edu/
94
Advances in Natural Multimodal Dialogue Systems
of multiple, precisely-positioned cameras and proper placement of the subjects in order to gather their data. FORM allows any researcher with a notebook PC and a video camera to generate useful data. Thus, FORM can be used in the “field,” where the VISLab system requires a laboratory setting.
A.3.2 BEAT Toolkit: Justine Cassell et al. The other important, large-scale project is the Behaviour Expression Animation Toolkit (BEAT) [Cassell et al., 2001]. It was developed at the MIT Media Lab in the Gesture and Narrative Language Research Group. This work is advanced and is, by far, the most influential to date. It allows for the easy generation of synchronized speech and gesture in computer-animated characters. The animator simply types in the sentence that he/she wishes the character to say, and the BEAT Toolkit generates marked-up text which can serve as the input to any of a number animation systems. The system is extensible to many different communicative behaviours and domains. The output generated for a given input string is domain specific, and the training data for that domain must be provided. The main purpose of BEAT is to appropriately schedule gestures (and other non-verbal behaviours) so they are synchronized with the speech. The BEAT toolkit is the first of a new generation (the beat generation) of animation tool that extracts actual linguistic and contextual information from text in order to suggest correlated gestures, eye gaze, and other nonverbal behaviours, and to synchronize those behaviours to one another. For those animators who wish to maintain the most control over output, BEAT can be seen as a kind of “snap-to-grid” for communicative actions: if animators input text, and a set of eye, face, head and hand behaviours for phrases, the system will correctly align the behaviours to one another, and send the timings to an animation system. For animators who wish to concentrate on higher level concerns such as personality, or lower level concerns such as motion characteristics, BEAT takes care of the middle level of animation: choosing how nonverbal behaviours can best convey the message of typed text, and scheduling them.12
FORM’s relationship to BEAT is more one of potential partners than as competitors. BEAT is most concerned with the automatic generations of the timings, and the higher and lower levels, as aforementioned, are left to the animator. In particular, the lower level of specifying motion characteristics is where FORM is most concerned. We see FORM as potentially a more robust way to specify the gestures for which BEAT schedules the timings. The typology of gestures that BEAT uses is based on the work of McNeill13 . As such, it sees gestures through the eyes of his ontology. It is, then, left up to the animator to specify exactly how a beat or a deictic, for example, is to be animated. 12 [Cassell 13 Cf.
et al., 2001, page 8]. discussion of McNeill, above.
FORM
95
We believe the data generated by the FORM annotation system will allow for a more robust output from BEAT, which would further alleviate the work of animators.
References Bird, S. and Liberman, M. (1999). A Formal Framework for Linguistic Annotation. Technical Report MS-CIS-99-01, Department of Computer and Information Sciences, University of Pennsylvania, Philadelphia, Pennsylvania. http://citeseer.nj.nec.com/article/bird99formal.html. Cassell, J., Vilhjálmsson, H. H., and Bickmore, T. (2001). BEAT: The Behavior Expression Animation Toolkit. In Fiume, E., editor, Proceedings of SIGGRAPH, pages 477–486. ACM Press / ACM SIGGRAPH. http://citeseer.ist.psu.edu/cassell01beat.html. Kendon, A. (1996). An Agenda for Gesture Studies. Semiotic Review of Books, 7(3):8–12. Kendon, A. (2000). Suggestions for a Descriptive Notation for Manual Gestures. Unpublished. Kipp, M. (2001). Anvil - A Generic Annotation Tool for Multimodal Dialogue. In Proceedings of Eurospeech 2001, pages 1367–1370, Aalborg, Denmark. MacWhinney, B. (1996). The CHILDES System. American Journal of SpeechLanguage Pathology, 5:5–14. Martell, C. (2002). FORM: An Extensible, Kinematically-based Gesture Annotation Scheme. In Proceedings of International Language Resources and Evaluation Conference (LREC), pages 183–187. European Language Resources Association. http://www.ldc.upenn.edu/Projects/FORM. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago, USA. Neidle, C., Sclaroff, S., and Athitsos, V. (2001). SignStream: A Tool for Linguistic and Computer Vision Research on Visual-Gestural Language Data. In Behavior Research Methods, Instruments, and Computers, volume 33:3, pages 311–320. Psychonomic Society Publications. http://www.bu.edu/asllrp/SignStream/. Quek, F., Bryll, R., McNeill, D., and Harper, M. (2001). Gestural Origo and Loci-Transitions in Natural Discourse Segmentation. Technical Report VISLab-01-12, Department of Computer Science and Engineering, Wright State University. http://vislab.cs.vt.edu/Publications/2001/QueBMH01.html.
Chapter 5 ON THE RELATIONSHIPS AMONG SPEECH, GESTURES, AND OBJECT MANIPULATION IN VIRTUAL ENVIRONMENTS: INITIAL EVIDENCE Andrea Corradini and Philip R. Cohen Center for Human-Computer Communication Department of Computer Science and Engineering Oregon Health & Science University, Portland, OR, USA
{andrea, pcohen}@cse.ogi.edu Abstract
This chapter reports on a study whose goal was to investigate how people make use of gestures and spoken utterances while playing a videogame without the support of standard input devices. We deploy a Wizard of Oz technique to collect audio- video- and body movement-related data on people’s free use of gesture and speech input. Data was collected from ten subjects for up to 60 minutes of game interaction each. We provide information on preferential mode use, as well as the predictability of gesture based on the objects in the scene. The long-term goal of this on-going study is to collect natural and reliable data from different input modalities, which could provide training data for the design and development of a robust multimodal recognizer.
Keywords:
Multimodal speech-gesture data analysis and collection, interaction in immersive environment, object affordance and manipulation, Wizard of Oz study.
1.
Introduction
Human-computer interaction in virtual environments has long been based on gesture, with the user’s hand(s) being tracked acoustically, magnetically, or via computer vision. In order to execute operations in virtual environments, users often are equipped with datagloves, whose hand-shapes are captured digitally, or a tracked device that is equipped with multiple buttons. For a number of reasons, these systems have frequently been difficult to use. First, although the user’s hand/arm motions are commonly called “gesture,” the movements to 97 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 97–112. © 2005 Springer. Printed in the Netherlands.
98
Advances in Natural Multimodal Dialogue Systems
be recognized are typically chosen by the developer. Thus, rather than recognize people’s naturally occurring movements, such systems require users to learn how to move “properly.” Secondly, gestural devices have many buttons and modes, making it difficult for a naïve subject to remember precisely which button in a given mode accomplishes which function. Third, the 3D interaction paradigm usually derives from the 2D-based direct manipulation style, in which one selects an object and then operates upon it. Some systems have modelled the virtual environment interface even more strongly upon the WIMP (windows, icons, menus, pointing device) graphical user interface, providing users with menus that need to be manipulated in the 3D world. Unfortunately, it turns out to be very difficult to select objects and menu entries in 3D environments. Various researchers have attempted to overcome these awkward interfaces in different ways. For example, Hinckley et al. [1994] gave users real-world models to manipulate, causing analogous actions to take place in the virtual environment. Stoakley et al. [1995] provided a miniature copy of the virtual world in the user’s virtual hand, thereby allowing smaller movements in hand to have analogous results on the world itself. Fisher et al. [1986] developed an early multimodal 3D interface for simulated Space Station operations, incorporating limited speech recognition, as well as hand gestures using a VPL dataglove. Weimer and Ganapathy [1989] developed a prototype virtual environment interface that incorporated a VPL dataglove, and a simple speech recognizer. Although only three gestures, all by the user’s thumb, were recognized, and the speech system offered just a 40 word vocabulary, the authors remarked upon the apparent improvement in the interface once voice was added. Based in part on this prior research, we hypothesize that multimodal interaction in virtual environments can ease the users’ burden by distributing the communicative tasks to the most appropriate modalities. By employing speech for its strengths, such as asking questions, invoking actions, etc., while using gesture to point at locations and objects, trace paths, and manipulate objects, users can more easily engage in virtual environment interaction. In order to build such multimodal systems, we need to understand how, if at all, people would speak and/or gesture in virtual environments if given the choice. What would users do on their own, without being limited to the researchers’ preferred gestures and language? Would gestures and language be predictable, and if so from what? Can the recognition of gesture and/or speech in virtual environments be improved by recognizing or understanding the input in another modality, as we find in 2D map-systems [Cohen et al., 1997; Oviatt, 1999]. Regarding predictability of speech and gesture, we hypothesize that without instruction, people will manipulate manufactured objects in VE in the ways they were designed to be manipulated – using their affordances [Gibson, 1977;
On the Relationships among Speech, Gestures, and Object Manipulation
99
Norman, 1988]. Given data indicating a user’s viewpoint on the object, and the degrees of freedom afforded by the object, a system should be able to predict how the user’s hand/arm will move. If the user can also speak, will they employ the same gestures during multimodal interaction as they employ using gesture alone? In order to answer these questions, and to provide a first set of data for training recognizers and statistically-based multimodal integration systems, we conducted a Wizard-of-Oz study of multimodal interaction with a simple, though realistically rendered, computer game.
2.
Study
Ten volunteer subjects (nine adults, one 12 year-old child), interacted with the MystTM III game played on a 2GHz Dell computer with Nvidia GeForce 3 graphics card. None of the subjects had played Myst before; three had played video games regularly. Myst is a semi-immersive 2.5-dimensional game in which a user moves around a complex world, containing both indoor and outdoor scenes. The user views the world through a (moderate) fish-eye viewport, which s/he can rotate 360 degrees, as well as tilt to see above and below. In Myst III, the user’s task is (partially) to travel around an island, rotating a series of beacons so that they shine on one another in a certain sequence, etc. Thus, the game involves navigation, manipulation of objects (doors, a knife switch, beacons, push-buttons, etc.) and search. The subjects wore a set of four 3D magnetic trackers attached to their head and dominant arm. They were told that they could interact with the game as they wished, and that the system could understand their speech and gestures.
2.1
Wizard of Oz Study
The classic method for studying recognition-based systems before the appropriate recognizers have been trained is to employ a Wizard of Oz paradigm [Oviatt et al., 1994]. In this kind of study, an unseen assistant plays the role of the computer, processing the user’s input and responding, as the system should. Importantly, the response time should be rapid enough to support satisfactory interactive performance. In the present study, subjects were told that they would be playing the Myst III computer game, to which they could speak and gesture freely. The user chose where s/he wanted to stand. Subjects “played” the game standing in front of a 50” diagonal flat-panel plasma display in wide-view mode. They could and did speak and/or gesture without constraint. Unbeknownst to the subject, a researcher observed the subjects’ inputs, and controlled the game on a local computer, whose audio and video output was sent to the subject. The “wizard’s” responses resulted in scene navigation and object state changes, with a response time that aver-
100
Advances in Natural Multimodal Dialogue Systems
Figure 5.1. Example of experimental setup, utterance, gesture, and block figure reproducing the subject’s motions.
aged less than 0.5 seconds. Since Myst III (and its predecessors) assumes the user is employing a mouse, it is designed to minimize actual gesturing, allowing only mouse-selection. Although occasionally the Wizard made errors, subjects received no explicitly marked recognition errors. A research assistant was present in the room with the subject, and would upon request give the subject hints about how to play the game, though not about what to say or how to gesture.
2.2
Equipment
To acquire the gesture data, the six-degree-of-freedom Flock of Bird (FOB) magnetic tracking device from Ascension Technology Corporation [2002] was used. We attached four sensors to the subject; one on the top head, one on the upper arm to register the position and orientation of the humerus, one on the lower arm for the wrist position and lower arm orientation, and finally one on the top of the hand. The last three sensors are aligned with each other anytime the subject stretched his or her arm to the side of the body, keeping the palm of the hand facing and parallel to the ground (see Figure 5.2). The data from the FOB are delivered via serial lines, one for each sensor. The four data streams can be processed in real time by a single SGI Octane machine employing the Virtual Reality Peripheral Network (VRPN) package [Taylor, 2001], which provides time-stamps of the data and distrib-
On the Relationships among Speech, Gestures, and Object Manipulation
101
Figure 5.2. Arrangement of trackers on subject’s body.
utes it to customer processes. Because the FOB devices uses a magnetic field that is affected by metallic objects, and the laboratory is constructed of steelreinforced concrete, the data from the sensors is often distorted. As a result, the data is processed with a median filter to partially eliminate noise. A block figure plays back the sensor data, providing both a check on accuracy and distortion. Accuracy of the magnetic tracker in our environment was approximately 0.5 cm.
3.
Data Analysis
The subject’s body motions were captured by the FOB, while the video recorded the subject’s view, and vocal interaction (see Figure 5.1). The speech and gesture on the video were transcribed, an example of which follows:
TRANSCRIPT FROM THE GAME Bold = speech # = location of gesture when not overlapping speech (. . . ) = hesitation/pause XXX = undecipherable [ ] = speech-gestural stroke overlapped event Indications such as “08-01-42-25/44-11” = VCR time-code
102
Advances in Natural Multimodal Dialogue Systems 08-01-42-25/44-11: # go across the bridge [hand held palm open to point toward the bridge then hand used as cursor along bridge] 08-01-47-00: keep going no gesture 08-01-50-03/57-27: [grab] this thing (. . . ) just [grab it] and pull it down and see what happen # reach for rim of telescope with hand, [close fist and pull hand from up to down], [one more time], [one more time] 08-01-59-29/07-04: # can I pull this thing ? (. . . ) ah ahaa # reach for rim of telescope, [close fist, pull hand from left to right circularly], [one more time] 08-02-10-00/15-16: ok [look at] look at this purple and see if there is anything to see [move hand toward the purple ball as to push at it] 08-02-15-16/18-28: # no (...) [back] move hand toward lens of telescope, [close fist to grab at rim of telescope and pull hand back toward the body], [move open hand again back toward the body] 08-02-19-04/26-25: ok # [turn it] again # reach for rim of telescope, [close fist and pull hand from left to right circularly as to rotate rim of telescope], [one more time], [one more time]
The transcript includes both speech to the system, as well as self-talk, but not requests for hints asked of the research assistant. Transcription took approximately two person-months.
3.1
Coding
The following categories were coded: For events that required explicit interaction with the system beyond causing the scene to rotate around, subjects were coded as using gesture-only, speech-only, or multimodal interaction. Numerous subcategories were coded, but this chapter only reports on the subset of gesture and multimodal interaction for which the user employed gesture “manipulatively,” when interacting with an object. Interrater reliability for secondscoring of 18% of the multimodal data was 98%. For ONLY GESTURE: Consistent manipulative gesture: gesture used with the objective of changing the state of an object in the game — e.g., turning a wheel or
On the Relationships among Speech, Gestures, and Object Manipulation
103
pressing a button. Such gestures are consistent if the movement matches the way the object operates. Manipulative gesture, NOT consistent: see above but the movement does not match the way the object operates. For ONLY SPEECH: Speech manipulative: involving any change of state of an object in the scene — e.g., standing in front of the wheel and saying “turn the wheel”, or “press the button” when in the elevator. For SPEECH AND GESTURE TOGETHER Speech manipulative + consistent manipulative gesture: e.g. saying “turn the wheel” AND mimicking the gesture of turning a wheel. Speech manipulative + gesture NOT manipulative: like above but the gesture does not match the way the object functions.
4.
Results
Of the 3956 “interactive events,” we totalled the use of gesture-only, speechonly, or multimodal interaction (see Figure 5.3). Subjects were classified as “habitual users” of a mode of communication if they employed that mode during at least 60% of the available times. As can be seen, 90% of the subjects were habitual users of speech (including multimodal interaction), and 60% of the subjects were habitual users of multimodal interaction. Subjects were classified as “habitual users” of the consistent manipulation of objects strategy if they gesturally manipulated the object at least 10 times in a given communication mode (using gesture-only or multimodally) and 60% of those times, their body motion was consistent with the affordances of that object (i.e., the ways humans would normally employ it). Figure 5.4 provides Myst III images that show some of those objects, whose affordances the reader can readily determine. We examined the two cases separately: use of gesture-only and use of gesture within a multimodal event. Six subjects used gesture within their multimodal interaction manipulatively, rather than deictically, in order to change the state of an object. Of those six, five gestured in a fashion consistent with the object’s affordances, indicating that subjects using multimodal interaction manipulated digital “artefacts” according to the actions afforded by their design (Wilcoxon, p<0.03, Z= -1.89, one-tailed). The one subject who did not do so used a speak-and-point strategy (“turn the wheel” <point>). As for gesture alone, four subjects changed the state of an object with manipulative gesture used in a fashion consistent with the object affordances,
104
Advances in Natural Multimodal Dialogue Systems
Figure 5.3.
Figure 5.4.
Subjects’ use of modalities.
Other objects that can be manipulated in Myst III.
On the Relationships among Speech, Gestures, and Object Manipulation
105
whereas no subjects using manipulative gestures inconsistent with the object’s affordances (Wilcoxon, p=0.023, Z= -2.000 one-tailed).
4.1
Results from Questionnaire
The subjects filled out a post-test questionnaire (see Appendix) indicating, on scales of 1(very low) -10 (very high), their: 1. 2. 3. 4. 5.
Experience with adventure games Immersive experience Involvement in the game Ranking of the interface System response latency
(x = 2.3, σ (x = 4.2, σ (x = 4.1, σ (x = 6.1, σ (x = 5.7, σ
= 1.8) = 2.5) = 2.6) = 1.7) = 2.7)
In other words, the game itself was not particularly exciting, and subjects were only moderately involved in it. However, the speech/gesture interface was slightly better than middling ratings, while the latency inherent in a Wizard-ofOz system did not make the interaction annoyingly slow.
4.2
Comments from Subjects
Subjects were asked, “Do you think there was a ‘preferred’ channel between speech or gesture for the interaction? Or did you feel both channels were equally effective? Please explain”. In response, we received the following comments, which often did not match the subjects’ performance: “I feel that speech was easier for commands, since it is more economical in energy than gesture, and more precise. I grew tired of trying to motion precisely with my arms, and the time it took to turn. I would prefer to use speech over gesture, at least with the way gesture seemed to be implemented here.” Used speech-only 49%, gesture-only 45% of the time. “I am confident that anything I was doing, I could do with either channel. Speech seemed more natural for some situations, and gestures for others, though.” A habitual user of speech-only interaction, using gesture 20% of the time, speechonly 74%. “Preferred (channel) seemed to be speech. The movement felt like I was doing exercises.” A habitual user of speech (100%). “I found voice commands worked better for me than the hand gestures.” A habitual user of multimodal interaction (62%), with no gesture-only interaction. “I think the preferred channel is a gesture, with speech used to make clarifications.” A habitual user of multimodal interaction (83%). “In the first few minutes, I realized that gesturing worked best for me. I always talk to my computer or my car while operating them, so that was more unconscious....I find it very involving to talk to the game...”. A habitual user of multimodal interaction (75%).
106
Advances in Natural Multimodal Dialogue Systems “I was under the distinct impression that gesture did nothing and the system was a speech-recognition one.” A habitual user of multimodal interaction (69%). “I liked hands but used speech when confusing.” A habitual user of multimodal interaction (63%).
5.
Discussion
Results show that if given the opportunity, most subjects (60%) would use multimodal interaction more than 60% of the time for interacting with the game; an additional 30% would use speech-only interaction 60% of the time or more, but only 10% would even use gesture-only half the time. Overall, subjects were found to employ gesture alone 14% of the time, speech alone 41% of the time, and used speech and gesture for 45% of their interactions. These latter results are somewhat skewed by subjects who employed only speech, since in order to navigate in the scene, they issued many more navigation commands (e.g., “go left”) than would be necessary if multimodal interaction were employed. Subjects’ opinions about which modalities were important were often belied by their actions. Some thought that gesture was the key modality, but used multimodal interaction habitually, while others thought speech was the essential modality, but also used multimodal interaction frequently. It would appear that having both available would suit just about all the subjects. When given the opportunity to use whatever gestures they wanted, most subjects who attempted to manipulate an artefact generally did so according to the affordances of those objects – subjects turned wheels, pulled down knife switches, pushed in doorbells, etc. Thus, a future virtual environment control program that detects that a manufactured object is in the scene should be able to predict how a person’s arm would be shaped and would move in order to manipulate that object properly. The gesture recognizer could then adapt to the scene itself, giving such gestures higher weight. As novices to these kinds of games, the subjects found Myst III to be a modestly immersive experience with a moderate degree of involvement. Unlike true 3D games, there is no avatar in the scene that represents the user, and thus one would expect a lesser degree of immersion than a full 3D virtual reality environment. Also, given that the subjects were not supplied with explicit objectives in playing the game, it is not surprising that their degree of involvement with the game was moderate. A number of subjects very much liked the speech-gesture interface, while most gave it mixed reviews.
6.
Related Work
Most of the existing research on gestures has been performed by cognitive scientists who are interested in how people gesture, and the reasons why people
On the Relationships among Speech, Gestures, and Object Manipulation
107
gesture [McNeill, 1993; Krauss, 1998; Goldin-Meadow, 1999; Kendon, 2000]. Various taxonomies of gesture have been offered like e.g., McNeill’s description of iconic, deictic, emblematic, and beat gestures [McNeill, 1993], which usefully inform scientists who build gesture-based systems and avatars [Cassell and Stone, 1999; Wilson et al., 1997]. Specific claims can also be useful to technologists. McNeill [1993] argues that speech and gesture derive from an internal knowledge representation (termed a “catchment”) that encodes both semantic and visual information. Our results tend to confirm this claim, in that the visual representation depicts the object’s affordances, which then determines the manipulative gesture. The present corpus could also be used to confirm Kendon’s [2000] claim that the stroke phase of gestures tends to cooccur with phonologically prominent words. Quek et al. [2001] have provided a case-study of cross-modal cues for discourse segmentation during natural conversation by observing a subject describing her living space to an interlocutor. A comparative analysis of both video and audio data was performed to extract the segmentation cues while expert transcribed the gestures. Then both the gestural and spoken data was correlated for 32 seconds of video. A strong correlation between handedness and high-level semantic content of the discourse was found, and baseline data on the kinds of gestures used was provided. Whereas this style of observational research is needed as a foundation, it needs to be combined with quantitative observational and experimental work in order to be useful to building computer systems. Wilson et al. [1997] employed the McNeill theory to motivate their research to distinguish bi-phasic gestures (e.g., beats) from more meaningful tri-phasic gestures that have preparatory, stroke, and retraction phases. Early work by Hauptmann [1989] employed a Wizard of Oz paradigm and investigated the use of multimodal interaction for simple 3D tasks. It was found that people prefer to use combined speech and gesture interaction over either modality alone, and given the opportunity to do so in a factorially designed experiment, chose to use both 70% of the time (vs. 13% gesture only, and 16% speech only). Their factorial study has the advantage that all subjects were exposed to all ways of communicating, whereas our more ecologically realistic study allowed users to develop their own ways of interacting. On the other hand, it also allowed users to become functionally fixed into their first successful way of operating. Perhaps the most relevant work to ours is that of Sowa and Wachsmuth [2002] who employed a WOZ paradigm to collect subjects’ gestures as they attempt to describe a limited set of objects to a listener within a virtual construction domain. It was found that subjects’ hand shapes corresponded to features of the objects themselves. Based on this data, a prototype system was built that decomposes each gesture into spatial primitives and identifies
108
Advances in Natural Multimodal Dialogue Systems
their interrelationships. The object recognition engine then employs a graphmatching technique to compare the structure of the objects and that of the gesture. This latter work differs from the results presented here in that the game we studied included the task of manipulating the object rather than describing it. Thus, we find people attempt to manipulate artefacts in the ways they were built to be manipulated, whereas Sowa and Wachsmuth found people used their hands in describing objects to indicate their shape. Clearly, the subjects’ goals and intentions make a difference to the kinds of gestures they use, whether in a multimodal or unimodal context.
7.
Future Work
The next steps in this research are to analyze the syntax and semantics of the utterances in conjunction with the form and meaning of the gesture. To date, we have employed unification as the primary information fusion operation for multimodal integration [Cohen et al., 1997; Johnston et al., 1997]. We hypothesize that by using a feature-structured action representation that offers a “manner” attribute [Badler et al., 2000], whose value can itself be an action, and by representing the meanings of gestures as such actions, unification can again serve as the mechanism for information fusion across modalities. The analyses of utterance and gesture will be used to test this hypothesis. Given the identification of the same kind of gesture by the subjects to manipulate the same object, a gesture recognizer can then be trained [Corradini, 2002] and used in a multimodal architecture. We also will investigate to what extent mutual disambiguation of modalities [Oviatt, 1999] can be used to overcome recognition errors in a complete multimodal virtual environment system [Kaiser et al., 2003].
8.
Conclusions
This chapter has provided initial empirical results based on a Wizard of Oz paradigm that indicate people will prefer to use multimodal interaction in virtual environments than speech or gesture alone. Furthermore, when they attempt to manipulate virtual representations of artefacts, they do so according to the objects’ affordances – the ways the objects were designed to be manipulated. We therefore conclude that virtual environment systems should be capable of multimodal interaction that would employ subjects’ natural gestures, and that such systems should in fact employ information about the objects currently in the user’s view to predict the kinds of gestures that will ensue. In the future, information about the users’ gaze could be employed to restrict still further the kinds of gestures that might be employed.
On the Relationships among Speech, Gestures, and Object Manipulation
109
Acknowledgements This research has been supported by the Office of Naval Research, under Grants N00014-99-1-0377, N00014-99-1-0380 and N00014-02-1-0038. We are thankful to Rachel F. Coulston for help in running the study, and to our volunteer subjects.
110
Advances in Natural Multimodal Dialogue Systems
Appendix: Questionnaire MYST III - EXILE PRE-SESSION: 1 Have you ever played MYST III: Exile - the sequel to the MYST and RIVEN series? 2 Have you ever played adventure or puzzle-like games? 3 How would you rank your experience in playing the game on a scale from 1 to 10? (1 = no experience at all - 10 = very experienced player) 4 Would you say that your current state of fitness is: as usual, you are sick, or both? Please explain. POST-SESSION: 5 Did you like the game? Please explain. 6 Would you like to play again? Please explain. 7 What did you find difficult when playing? Please explain. 8 How would you rank the immersive effect of the game on a scale from 1 to 10? (1 = not immersive at all – 10 = amazingly immersive) 9 How would you rank your involvement in playing the game on a scale from 1 to 10? (1 = I played because I was requested to – 10 = I fully wanted to get to the end of the game)? 10 How would you rank the interface on a scale from 1 to 10? (1 = extremely bad, it never did what I wanted to – 10 = amazingly good) 11 How would you rank the latency response of the interface on a scale from 1 to 10? (1 = when I entered a command the interface reacted too slowly – 10 = I did not even realize there was a latency between my commands and the response of the interface) 12 Would you have preferred to play using a mouse/joystick/trackball/keyboard instead the way you played? Please explain. 13 Did you consciously decide when to use speech or gesture for playing? Please explain. 14 Did you have some hints concerning if and how to use gesture or speech or did you just do what it come naturally to you? Please explain. 15 Do you think there was a “preferred” channel between speech or gesture for the interaction? Or did you feel both channels were equally effective? Please explain. 16 Were the sensors attached to your body cumbersome? Did you feel restricted in your movements? Please explain. 17 Do you have any suggestions/criticism concerning the experiment?
On the Relationships among Speech, Gestures, and Object Manipulation
111
References Ascension Technology (2002). Flock of Birds. http://www.ascensiontech.com/. Badler, N., Bindinganavale, R., Allbeck, J., Schuler, W., Zao, L., and Palmer, M. (2000). Paramaterized Action Representation for Virtual Human Agents. In Cassell, J., Sullivan, J., Prevost, S., and Churchill, E., editors, Embodied Conversational Agents, pages 256–286. MIT Press. Cassell, J. and Stone, M. (1999). Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems. In Proceedings of the AAAI 1999 Fall Symposium on Psychological Models of Communication in Collaborative Systems, pages 34–42. North Falmouth, MA: AAAI Press. Cohen, P. R., Johnston, M., McGee, D. R., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. (1997). QuickSet: Multimodal Interaction for Distributed Applications. In Proceedings of the Fifth Annual ACM International Multimedia Conference (Multimedia 1997), pages 31–40. Seattle, WA: ACM Press. Corradini, A. (2002). Real-Time Gesture Recognition by Means of Hybrid Recognizers. In Sowa, T. and Wachsmuth, I., editors, Gesture and Sign Language in Human-Computer Interaction, pages 34–46. Berlin: Springer Verlag. Fisher, S. S., McGreevy, M., Humphries, J., and Robinett, W. (1986). Virtual Environment Display System. In Proceedings of the ACM Workshop on Interactive 3D Graphics, pages 77–87. Chapel Hill, NC: ACM Press. Gibson, J. J. (1977). The Theory of Affordances. In Shaw, R. and Bransford, J., editors, Perceiving, Acting, and Knowing, pages 67–82. Hillsdale, NJ: Lawrence Erlbaum Associates. Goldin-Meadow, S. (1999). The Role of Gesture in Communication and Thinking. Trends in Cognitive Science, 3(11):419–429. Hauptmann, A. G. (1989). Speech and Gestures for Graphic Image Manipulation. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 241–245. New York: ACM Press. Hinckley, K., Pausch, R., Goble, J. C., and Kassell, N. F. (1994). Passive RealWorld Interface Props for Neurosurgical Visualization. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 452– 458. Johnston, M., Cohen, P. R., McGee, D., Oviatt, S., Pittman, J. A., and Smith, I. (1997). Unification-Based Multimodal Integration. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL), pages 281–288, Madrid, Spain. ACL Press. Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P. R., and Feiner, S. (2003). Mutual Disambiguation of 3D Multimodal In-
112
Advances in Natural Multimodal Dialogue Systems
teraction in Augmented and Virtual Reality. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI-PUI), pages 12–19, Vancouver, B.C., Canada. Kendon, A. (2000). Language and Gesture: Unity or Duality? In McNeill, D., editor, Language and Gesture, pages 47–63. Cambridge, UK: Cambridge University Press. Krauss, R. M. (1998). Why do we Gesture When we Speak? Current Directions in Psychological Science, 7:54–59. McNeill, D. (1993). Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press. Norman, D. A. (1988). The Design of Everyday Things. New York, NY: Currency/Doubleday. Oviatt, S. L. (1999). Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 576–583. New York: ACM Press. Oviatt, S. L., Cohen, P. R., and Wang, M. Q. (1994). Toward Interface Design for Human Language Technology: Modality and Structure as Determinants of Linguistic Complexity. Speech Communication, 15(3-4):283–300. Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K. E., and Ansari, R. (2001). Gesture and Speech Multimodal Conversational Interaction. Technical report, Electrical Engineering and Computer Science Department, University of Illinois, Chicago. Sowa, T. and Wachsmuth, I. (2002). Interpretation of Shape-Related Iconic Gestures in Virtual Environments. In Sowa, T. and Wachsmuth, I., editors, Gesture and Sign Language in Human-Computer Interaction, pages 21–33. Berlin: Springer Verlag. Stoakley, R., Conway, M. J., and Pausch, R. (1995). Virtual Reality on a WIM: Interactive Worlds in Miniature. In Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 265–272. ACM Press. Taylor, R. M. (2001). VRPN: A Device-Independent, Network-Transparent VR Peripheral System. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, pages 55–61. New York: ACM Press. Weimer, D. and Ganapathy, S. K. (1989). A Synthetic Visual Environment with Hand Gesturing and Voice Input. In Bice, K. and Lewis, C., editors, Proceedings of Conference on Human Factors in Computing Systems (CHI), pages 235–240. New York: ACM Press. Wilson, A. D., Bobick, A. F., and Cassell, J. (1997). Temporal Classification of Natural Gesture and Application to Video Coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 948– 954. New York: IEEE Press.
Chapter 6 ANALYSING MULTIMODAL COMMUNICATION Repair-Based Measures of Human Communicative Coordination Patrick G. T. Healey, Marcus Colman and Mike Thirlwell Interaction, Media, and Communication Research Group Department of Computer Science Queen Mary, University of London, UK
{ph, marcus, miket}@dcs.qmul.ac.uk Abstract
There are few techniques available to inform the design of systems to support human-human interaction. Psycholinguistic models have the potential to fill this gap however existing approaches have some conceptual and practical limitations. This chapter presents a technique, based on the conversation analytic model of breakdown and repair, for modality and task independent analysis of communicative exchanges. The rationale for the approach is presented and a protocol for coding repair is described. The potential of this approach for analysing multimodal interactions is discussed.
Keywords:
Communication, conversation analysis, repair, evaluation.
1.
Introduction
Communication is central to many human activities, even for tasks and technologies that are ostensibly individual, see e.g., [Suchman, 1987; Heath and Luff, 2000; Hutchins, 1995; Nardi and Miller, 1991]. It underpins the co-ordination of routine activities and people’s responses to unexpected contingencies. Ethnomethodological studies of workplace interaction have documented how technologies not directly designed to support communication are nonetheless recruited in unanticipated ways to facilitate communicative coordination, e.g., [Heath and Luff, 2000; Hughes et al., 1992; Bowers et al., 1995]. Technologies such as text-based messaging and chat tools, integrated conferencing systems and large-scale collaborative virtual environments are 113 J.C.J. van Kuppevelt et al. (eds.), Advances in Natural Multimodal Dialogue Systems, 113–129. © 2005 Springer. Printed in the Netherlands.
114
Advances in Natural Multimodal Dialogue Systems
explicitly designed to support human communication. However, current approaches to the design and evaluation of these technologies focus on the analysis of the human-system interaction. There are few techniques that directly address the analysis of human-human interaction. Existing conceptual and empirical analyses of human communication should have much to contribute to design and evaluation. However, there are few established techniques either for identifying the communication related requirements of a given activity or for evaluating the impact different technologies have on the effectiveness of communication. Despite the elegance of the empirical observations, workplace studies have a problematic relationship to design, see e.g., [Hughes et al., 1992; Button and Dourish, 1996]. They require intensive effort and special analytical skills. They are also retrospective in character and do not provide a basis for systematic comparisons across different technologies or situations. The models and techniques developed in the psycholinguistic tradition are ostensibly more promising. They explicitly aim to quantify communicative phenomena and develop explanations that support comparative and predictive generalisations. The earliest applications of this approach used simple structural measures of interaction such as turn-taking, interruptions, backchannels, and gaze, see e.g., [O’Conaill et al., 1993; O’Malley et al., 1996]. Although these measures support direct comparison between some aspects of, for example, video-mediated interaction and face-to-face exchanges, the coding categories are coarse and do not take account of communicative function. For example, the category of ‘interruptions’ is sometimes formulated as the occurrence of overlapping speech where there has been no signal that a speaker is relinquishing the floor. This conflates accidental overlap, disruptive or competitive interventions and cooperative interventions such as collaborative completions. As a result the distribution of e.g., numbers of turns, lengths of turn, and interruptions are consistent with a number of possible interpretations, cf. [Anderson et al., 1997]. The finding that, for example, video-mediated communication leads participants to use more turns and words than they would when face-to-face are thus difficult to interpret. It might indicate either that video adds something to the interaction or that participants are working to compensate for some limitations it introduces. This chapter begins by reviewing the strengths and limitations of two psycholinguistic approaches to the analysis of human-human interaction: Dialogue Games analysis and the Collaborative Model of Dialogue. We argue that both approaches have important limitations as techniques for analysing multi-modal dialogue. An alternative approach is presented that focuses on breakdown and repair: the detection and resolution of communication problems. We propose that this can provide more robust, comparative measures of the effectiveness of human-human interaction across tasks and modalities. In
Analysing Multimodal Communication
115
order to turn this approach into a practical tool it is first necessary to develop a coding protocol that supports reliable identification of repair phenomena across different modalities. We present a protocol and test it against a corpus of attested examples of conversational repair. We then go on to sketch some of its potential applications to the analysis of multi-modal interactions.
1.1
Dialogue Games Analysis
Problems with the interpretation of simple structural analyses have been addressed to some extent by techniques which analyse communicative function directly. For example, Anderson et al. [1997] compared the performance of subjects carrying out a map drawing task under different media conditions. This task has a reliable dialogue coding system developed for it which characterises the functional structure of the dialogues, analysing each utterance as a move (e.g., instruct, explain, check, clarify, query-w, align) and structured sequences of moves as dialogue games [Kowtko et al., 1991]. Anderson et al. [1997] have shown how this approach can be sued to compare the dialogue moves used to carry out the task either face-to-face, using audio only or using video-mediated communication. Unlike the structural analyses, this provides a means of isolating the communicative functions that are preserved or impeded by different media. Although it supports systematic comparisons between media, functional analysis also has limitations. The coding system was designed to exhaustively classify utterance function in a particular collaborative task, the map task. As a result the move types are tailored to the transactional character of the task. Although it has been shown to generalise to some other information exchange tasks, it is unclear whether the coding scheme is adequate for qualitatively different kinds of exchange such as competitive negotiation. A practical limitation is that because the coding system is exhaustive, every utterance is analysed and coded. This is labour intensive and has the consequence that the sensitivity of the coding system is traded-off against its coverage, see [Carletta et al., 1996]. An additional concern is that the functional analysis doesn’t discriminate between incidental exchanges and those necessary for the activity at hand. All exchanges are coded and all contributions are given equal weight in the analysis.
1.2
The Collaborative Model of Dialogue
A second approach to modelling communication that also improves on simple structural analyses is the collaborative model of dialogue (CM) developed by Clark and co-workers [Clark, 1996; Clark and Wilkes-Gibbs, 1986; Clark and Schaefer, 1989]. This focuses on the process through which people build up their common ground during interaction. The basic principle is that the par-
116
Advances in Natural Multimodal Dialogue Systems
ties to an interaction only consider an utterance, or other communicative act, to be successful where some positive evidence of its acceptance or ‘grounding’ has been obtained. The grounding principle is modified according to the types of evidence and the degree of effort required to secure mutual-understanding in a given case. Two of the most important qualifications are that; a) interlocutors always seek to reduce the joint, as opposed to individual, effort necessary to successfully ground a communicative act and b) interlocutors will attempt to ground a contribution only up to a criterion level (the grounding criterion) which is adjusted according to circumstances. This framework has been used to characterise the properties of different communicative media and modalities. Clark and Brennan [1991] identify a set of 8 constraints on grounding that derive from the signal characteristics of different communicative media (Copresence, Visibility, Audibility, Cotemporality, Simultaneity, Sequentiality, Reviewability and Revisability). These constraints alter the ease with which evidence of grounding can be provided and change the costs associated with different grounding techniques (Formulation costs, Production costs, Reception costs, Understanding Costs, Start-up costs, Delay costs, Asynchrony costs, Speaker Change costs, Fault costs, Repair costs). During interaction, individuals must make a trade-off between the different types of action they can undertake in order to ground a particular contribution and their cost in a particular medium. For example, where turn taking costs are high individuals may invest more in the construction of each utterance and less in attempts at concurrent feedback. The CM has also been applied to the analysis of system feedback. The CM distinguishes a number of levels at which an action is considered complete. For example, an utterance may be perceived but not understood, or it may be understood but the action it proposes is not undertaken. These action levels are ordered according to the principle that feedback which indicates completion at a higher level presupposes completion at a lower level: if I comply with your request then this is evidence that I have also heard and understood it. Consequently, the current state of the common ground with respect to communicative action can vary depending on the degree of grounding that has been secured. The maintenance of context in interaction with a system can be supported by giving feedback which signals the level of grounding that a particular action has achieved [Brennan, 1988]. The level of feedback given can be modulated by the risks associated with possible misunderstandings. Although it generalises to a variety of tasks and communication modalities there are some important limitations to these applications of the CM model. One problem is that it provides only a limited analysis of situations in which contributions fail to secure acceptance. Where this occurs there is a general expectation that some repair will ensue, for example through reformulation of the contribution in a way that is acceptable to the addressee(s). A num-
Analysing Multimodal Communication
117
ber of possible types of reformulation e.g., alternative descriptions, instalment descriptions and trial references, are distinguished but the pattern of choice amongst these types, and their relationship to the success of repairs, is not addressed in the model. Although it is explicitly acknowledged that processes of conversational repair play a critical role in sustaining the mutual-intelligibility of interactions [Brennan, 1988; Clark, 1996], no specific mechanism has so far been developed for dealing with this. A more practical limitation on the application of the CM to the analysis of interaction is its relative underspecification. As Clark and Brennan [1991] note the media constraints and costs invoked to explain particular patterns of communication in different media are heuristic. The list of costs and constraints are neither exhaustive nor exclusive and there is no systematic means of quantifying them or calculating the possible trade-offs involved. In order to do this some quantification of the communicative effort invested in an interaction would be required. Without a means of comparing the grounding criteria being employed in different cases, i.e. of estimating the grounding criterion, the rationale behind particular patterns of communicative response cannot be determined. For example, a pattern of small, instalment, contributions may indicate a situation in which turn costs are low or a situation in which the grounding criterion is high.
2.
Breakdown and Repair
The position developed in this chapter is that the problems with functional and CM analyses of communication identified above can be addressed by focusing on the analysis of breakdown and repair in communication. The distinguishing feature of this approach is that it is concerned only with those parts of an interaction in which communicative trouble is encountered. The approaches described above depend on positive characterisations of communicative success; on analysing how things go right. The present approach, by contrast, focuses on analysing how things go wrong, and how people respond to this. A detailed framework for characterising these situations has been developed in the conversation analytic (CA) tradition [Schegloff et al., 1977; Schegloff, 1987; Schegloff, 1992]. Before describing the potential application of this framework to the analysis of mediated interactions, it is important to set out the basic CA repair framework and to clarify the concept of communicative problem it invokes. Two basic aspects to the CA model are of particular relevance to the present chapter; the structure of repairs and how specific they are. Structurally, the CA repair framework distinguishes between three things; who initiates a repair, where in the turn taking structure it occurs, and the subsequent trajectory of the repair to completion. For example, self-initiation occurs where the speaker
118
Advances in Natural Multimodal Dialogue Systems
identifies a problem with one of their own turns in a conversation. Otherinitiation occurs in situations in which someone signals a problem with another participant’s turn. The point at which the problem is addressed, the ‘repair’, is also classified in terms of ‘self’ and ‘other’. Self-repair occurs where the person who produced the problematic utterance also addresses the problem, regardless of whether they signalled that it was problematic. Other-repair occurs where someone addresses a putative problem with someone else’s turn. Four positions are distinguished in which a problem can be signalled or addressed. First position repair occurs in the turn in which a problem occurs, second position repair takes place in the next turn that occurs as a response to the problem turn. Third position repair-initiation occurs in the next turn that occurs as a response to the second position and so on [Schegloff, 1992; Schegloff, 1987]. Repair initiations are also distinguished according to the specificity with which they localise a problem. Schegloff et al. [1977] proposed ordering of other-initiation types according to their power. The most general kind of initiation is a “huh?” or “what?” which signals that the utterer has a problem but gives almost no clues about its precise character. This is followed, in order of increasing specificity, by a ‘wh’ question, such as “who?” or “what”. In this case the nature of the signalled problem is clearer and could potentially be associated with a particular sub-part of the problematic turn. More specific still are a partial repeat plus a ‘wh’ question or just a partial repeat. This type of reprise clarification provides information about the specific element(s) of the original turn that caused the problem