Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
5610
Julie A. Jacko (Ed.)
Human-Computer Interaction New Trends 13th International Conference, HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings, Part I
13
Volume Editor Julie A. Jacko University of Minnesota Institute of Health Informatics MMC 912, 420 Delaware Street S.E., Minneapolis, MN 55455, USA E-mail:
[email protected]
Library of Congress Control Number: 2009929048 CR Subject Classification (1998): H.5, I.3, I.7.5, I.5, I.2.10 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13
0302-9743 3-642-02573-0 Springer Berlin Heidelberg New York 978-3-642-02573-0 Springer Berlin Heidelberg New York
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12705719 06/3180 543210
Foreword
The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human–Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in the knowledge and effective use of computers in a variety of application areas. This volume, edited by Julie A. Jacko, contains papers in the thematic area of Human–Computer Interaction, addressing the following major topics: • • • • •
Novel Techniques for Measuring and Monitoring Evaluation Methods, Techniques and Tools User Studies User Interface Design Development Approaches, Methods and Tools
The remaining volumes of the HCI International 2009 proceedings are: • • • • •
Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 3, LNCS 5612, Human–Computer Interaction––Ambient, Ubiquitous and Intelligent Interaction (Part III), edited by Julie A. Jacko Volume 4, LNCS 5613, Human–Computer Interaction - Interacting in Various Application Domains (Part IV), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis
VI
Foreword
• • • • • • • • • • •
Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 11, LNCS 5620, Digital Human Modeling, edited by Vincent G. Duffy Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen Volume 17, LNAI 5639, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris
I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.
Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland
Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA
Foreword
Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea
Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany
Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA
Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China
Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA
Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK
VII
VIII
Foreword
Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa
Matthew J.W. Thomas, Australia Mark Young, UK
Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA
Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK
Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA
Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA
Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK
Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA
Foreword
Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA
Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China
Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria
Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan
Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA
Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA
IX
X
Foreword
Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK
Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany
Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China
Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China
Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan
Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan
In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.
Foreword
XI
I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis
HCI International 2011
The 14th International Conference on Human–Computer Interaction, HCI International 2011, will be held jointly with the affiliated conferences in the summer of 2011. It will cover a broad spectrum of themes related to human–computer interaction, including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/
General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email:
[email protected]
Table of Contents
Part I: Novel Techniques for Measuring and Monitoring Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kiyohiko Abe, Shoichi Ohi, and Minoru Ohyama
3
A Usability Study of WebMaps with Eye Tracking Tool: The Effects of Iconic Representation of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ¨ Ozge Ala¸cam and Mustafa Dalcı
12
Feature Extraction and Selection for Inferring User Engagement in an HCI Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stylianos Asteriadis, Kostas Karpouzis, and Stefanos Kollias
22
Informative or Misleading? Heatmaps Deconstructed . . . . . . . . . . . . . . . . . Agnieszka (Aga) Bojko
30
Toward EEG Sensing of Imagined Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan
40
Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Gao, Armando Barreto, and Malek Adjouadi
49
Usability Evaluation by Monitoring Physiological and Other Data Simultaneously with a Time-Resolution of Only a Few Seconds . . . . . . . . K´ aroly Hercegfi, M´ arton P´ aszti, Sarolta T´ ov¨ olgyi, and Lajos Izs´ o
59
Study of Human Anxiety on the Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . Santosh Kumar Kalwar and Kari Heikkinen
69
The Research on Adaptive Process for Emotion Recognition by Using Time-Dependent Parameters of Autonomic Nervous Response . . . . . . . . . Jonghwa Kim, Mincheol Whang, and Jincheol Woo
77
Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu-Jin Kim, Jin Ah Bae, and Byeong Ho Jeon
85
Toward Constructing an Electroencephalogram Measurement Method for Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masaki Kimura, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto
95
XVI
Table of Contents
Automated Analysis of Eye-Tracking Data for the Evaluation of Driver Information Systems According to ISO/TS 15007-2:2001 . . . . . . . . . . . . . . Christian Lange, Martin Wohlfarter, and Heiner Bubb Brain Response to Good and Bad Design . . . . . . . . . . . . . . . . . . . . . . . . . . . Haeinn Lee, Jungtae Lee, and Ssanghee Seo An Analysis of Eye Movements during Browsing Multiple Search Results Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuko Matsuda, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Development of Estimation System for Concentrate Situation Using Acceleration Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masashi Okubo and Aya Fujimura Psychophysiology as a Tool for HCI Research: Promises and Pitfalls . . . . Byungho Park Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Genaro Rebolledo-Mendez, Ian Dunwell, Erika A. Mart´ınez-Mir´ on, Mar´ıa Dolores Vargas-Cerd´ an, Sara de Freitas, Fotis Liarokapis, and Alma R. Garc´ıa-Gaona Effect of Body Movement on Music Expressivity in Jazz Performances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mamiko Sakata, Sayaka Wakamiya, Naoki Odaka, and Kozaburo Hachimura
105 111
121
131 141
149
159
A Method to Monitor Operator Overloading . . . . . . . . . . . . . . . . . . . . . . . . . Dvijesh Shastri, Ioannis Pavlidis, and Avinash Wesley
169
Decoding Attentional Orientation from EEG Spectra . . . . . . . . . . . . . . . . . Ramesh Srinivasan, Samuel Thorpe, Siyi Deng, Tom Lappas, and Michael D’Zmura
176
On the Possibility about Performance Estimation Just before Beginning a Voluntary Motion Using Movement Related Cortical Potential . . . . . . . Satoshi Suzuki, Takemi Matsui, Yusuke Sakaguchi, Kazuhiro Ando, Nobuyuki Nishiuchi, Toshimasa Yamazaki, and Shin’ichi Fukuzumi
184
Part II: Evaluation Methods, Techniques and Tools A Usability Evaluation Method Applying AHP and Treemap Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toshiyuki Asahi, Teruya Ikegami, and Shin’ichi Fukuzumi
195
Table of Contents
Evaluation of User-Interfaces for Mobile Application Development Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florence Balagtas-Fernandez and Heinrich Hussmann User-Centered Design and Evaluation – The Big Picture . . . . . . . . . . . . . . Victoria Bellotti, Shin’ichi Fukuzumi, Toshiyuki Asahi, and Shunsuke Suzuki
XVII
204 214
Web-Based System Development for Usability Evaluation of Ubiquitous Computing Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jong Kyu Choi, Han Joon Kim, Beom Suk Jin, and Yonggu Ji
224
Evaluating Mobile Usability: The Role of Fidelity in Full-Scale Laboratory Simulations with Mobile ICT for Hospitals . . . . . . . . . . . . . . . . Yngve Dahl, Ole Andreas Alsos, and Dag Svanæs
232
A Multidimensional Approach for the Evaluation of Mobile Application User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jos´e Eust´ aquio Rangel de Queiroz and Danilo de Sousa Ferreira
242
Development of Quantitative Usability Evaluation Method . . . . . . . . . . . . Shin’ichi Fukuzumi, Teruya Ikegami, and Hidehiko Okada
252
Reference Model for Quality Assurance of Speech Applications . . . . . . . . . Cornelia Hipp and Matthias Peissner
259
Toward Cognitive Modeling for Predicting Usability . . . . . . . . . . . . . . . . . . Bonnie E. John and Shunsuke Suzuki
267
Webjig: An Automated User Data Collection System for Website Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mikio Kiura, Masao Ohira, and Ken-ichi Matsumoto ADiEU: Toward Domain-Based Evaluation of Spoken Dialog Systems . . . Jan Kleindienst, Jan Cuˇr´ın, and Martin Labsk´ y Interpretation of User Evaluation for Emotional Speech Synthesis System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ho-Joon Lee and Jong C. Park Multi-level Validation of the ISOmetrics Questionnaire Based on Qualitative and Quantitative Data Obtained from a Conventional Usability Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan-Paul Leuteritz, Harald Widlroither, and Michael Kl¨ uh What Do Users Really Do? Experience Sampling in the 21st Century . . . Gavin S. Lew Evaluating Usability-Supporting Architecture Patterns: Reactions from Usability Professionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edgardo Luzcando, Davide Bolchini, and Anthony Faiola
277 287
295
304 314
320
XVIII
Table of Contents
Heuristic Evaluations of Bioinformatics Tools: A Development Case . . . . Barbara Mirel and Zach Wright A Prototype to Validate ErgoCoIn: A Web Site Ergonomic Inspection Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcelo Morandini, Walter de Abreu Cybis, and Dominique L. Scapin
329
339
Mobile Phone Usability Questionnaire (MPUQ) and Automated Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Young Sam Ryu
349
Estimating Productivity: Composite Operators for Keystroke Level Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jeff Sauro
352
Paper to Electronic Questionnaires: Effects on Structured Questionnaire Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Trujillo
362
Website Designer as an Evaluator: A Formative Evaluation Method for Website Interface Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chao-Yang Yang
372
Part III: User Studies Building on the Usability Study: Two Explorations on How to Better Understand an Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anshu Agarwal and Madhu Prabaker Measuring User Performance for Different Interfaces Using a Word Processor Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tanya R. Beelders, Pieter J. Blignaut, Theo McDonald, and Engela H. Dednam
385
395
Evaluating User Effectiveness in Exploratory Search with TouchGraph Google Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kemal Efe and Sabriye Ozerturk
405
What Do Users Want to See? A Content Preparation Study for Consumer Electronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yinni Guo, Robert W. Proctor, and Gavriel Salvendy
413
“I Love My iPhone... But There Are Certain Things That ‘Niggle’ Me” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Haywood and Gemma Boguslawski
421
Table of Contents
Acceptance of Future Technologies Using Personal Data: A Focus Group with Young Internet Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fabian Hermann, Doris Janssen, Daniel Schipke, and Andreas Schuller Analysis of Breakdowns in Menu-Based Interaction Based on Information Scent Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yukio Horiguchi, Hiroaki Nakanishi, Tetsuo Sawaragi, and Yuji Kuroda E-Shopping Behavior and User-Web Interaction for Developing a Useful Green Website . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei-Hui Huang, Ying-Lien Lee, and Sheue-Ling Hwang Interaction Comparison among Media Internet Genre . . . . . . . . . . . . . . . . . Sang Hee Kweon, Eun Joung Cho, and Ae Jin Cho Comparing the Usability of the Icons and Functions between IE6.0 and IE7.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chiuhsiang Joe Lin, Min-Chih Hsieh, Hui-Chi Yu, Ping-Jung Tsai, and Wei-Jung Shiang
XIX
431
438
446
455
465
Goods-Finding and Orientation in the Elderly on 3D Virtual Store Interface: The Impact of Classification and Landmarks . . . . . . . . . . . . . . . . Cheng-Li Liu, Shiaw-Tsyr Uang, and Chen-Hao Chang
474
Effects of Gender Difference on Emergency Operation Interface Design in Semiconductor Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hunszu Liu
484
Evaluating a Personal Communication Tool: Sidebar . . . . . . . . . . . . . . . . . Malena Mesarina, Jhilmil Jain, Craig Sayers, Tyler Close, and John Recker
490
“You’ve Got IMs!” How People Manage Concurrent Instant Messages . . . Shailendra Rao, Judy Chen, Robin Jeffries, and Richard Boardman
500
Investigating Children Preferences of a User Interface Design . . . . . . . . . . Jamaliah Taslim, Wan Adilah Wan Adnan, and Noor Azyanti Abu Bakar
510
Usability Evaluation of Graphic Design for Ilmu’s Interface . . . . . . . . . . . . Tengku Siti Meriam Tengku Wook and Siti Salwa Salim
514
Are We Trapped by Majority Influences in Electronic Word-of-Mouth? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu Tong and Yinqing Zhong
520
XX
Table of Contents
Leveraging a User Research Framework to Guide Research Investments: Windows Vista Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gayna Williams A Usability Evaluation of Public Icon Interface . . . . . . . . . . . . . . . . . . . . . . Sungyoung Yoon, Jonghoon Seo, Joonyoung Yoon, Seungchul Shin, and Tack-Don Han
530 540
Part IV: User Interface Design Little Design Up-Front: A Design Science Approach to Integrating Usability into Agile Requirements Engineering . . . . . . . . . . . . . . . . . . . . . . . Sisira Adikari, Craig McDonald, and John Campbell
549
Aesthetics in Human-Computer Interaction: Views and Reviews . . . . . . . Salah Uddin Ahmed, Abdullah Al Mahmud, and Kristin Bergaust
559
Providing an Efficient Way to Make Desktop Icons Visible . . . . . . . . . . . . Toshiya Akasaka and Yusaku Okada
569
An Integration of Task and Use-Case Meta-models . . . . . . . . . . . . . . . . . . . R´emi Bastide
579
Model-Based Specification and Validation of User Interface Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Birgit Bomsdorf and Daniel Sinnig
587
A Position Paper on ’Living Laboratories’: Rethinking Ecological Designs and Experimentation in Human-Computer Interaction . . . . . . . . . Ed H. Chi
597
Embodied Interaction or Context-Aware Computing? An Integrated Approach to Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Johan Eliasson, Teresa Cerratto Pargman, and Robert Ramberg
606
Supporting Multidisciplinary Teams and Early Design Stages Using Storyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mieke Haesen, Jan Meskens, Kris Luyten, and Karin Coninx
616
Agent-Based Architecture for Interactive System Design: Current Approaches, Perspectives and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . Christophe Kolski, Peter Forbrig, Bertrand David, Patrick Girard, Chi Dung Tran, and Houcine Ezzedine BunBunMovie: Scenario Visualizing System Based on 3-D Character . . . Tomoya Matsuo and Takashi Yoshino Augmented Collaborative Card-Based Creative Activity with Digital Pens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motoki Miura, Taro Sugihara, and Susumu Kunifuji
624
634
644
Table of Contents
Usability-Engineering-Requirements as a Basis for the Integration with Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karsten Nebe and Volker Paelke Design Creation Based on KANSEI in Toshiba . . . . . . . . . . . . . . . . . . . . . . Yosoko Nishizawa and Kanya Hiroi High-Fidelity Prototyping of Interactive Systems Can Be Formal Too . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Philippe Palanque, Jean-Fran¸cois Ladry, David Navarre, and Eric Barboni
XXI
652 660
667
RUCID: Rapid Usable Consistent Interaction Design Patterns-Based Mobile Phone UI Design Library, Process and Tool . . . . . . . . . . . . . . . . . . . Avinash Raj and Vihari Komaragiri
677
The Appropriation of Information and Communication Technology: A Cross-Cultural Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jose Rojas and Matthew Chalmers
687
UISK: Supporting Model-Driven and Sketch-Driven Paperless Prototyping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vin´ıcius Costa Villas Bˆ oas Segura and Simone Diniz Junqueira Barbosa
697
Beyond the User Interface: Towards User-Centred Design of Online Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Sikorski
706
Designing for Change: Engineering Adaptable and Adaptive User Interaction by Focusing on User Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruno S. da Silva, Ariane M. Bueno, and Simone D.J. Barbosa
715
Productive Love: A New Proposal for Designing Affective Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ramon Solves Pujol and Hiroyuki Umemuro
725
Insight into Kansei Color Combinations in Interactive User Interface Designing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . K.G.D. Tharangie, Shuichi Matsuzaki, Ashu Marasinghe, and Koichi Yamada Learn as Babies Learn: A Conceptual Model of Designing Optimum Learnability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Douglas Xiaoyong Wang Time-Oriented Interface Design: Picking the Right Time and Method for Information Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Keita Watanabe, Kei Sugawara, Shota Matsuda, and Michiaki Yasumura
735
745
752
XXII
Table of Contents
Enabling Interactive Access to Web Tables . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Yang, Wenchang Xu, and Yuanchun Shi
760
Integration of Creativity into Website Design . . . . . . . . . . . . . . . . . . . . . . . . Liang Zeng, Robert W. Proctor, and Gavriel Salvendy
769
Part V: Development Approaches, Methods and Tools YVision: A General Purpose Software Composition Framework . . . . . . . . Ant˜ ao Almada, Gon¸calo Lopes, Andr´e Almeida, Jo˜ ao Fraz˜ ao, and Nuno Cardoso Collaborative Development and New Devices for Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hans-J¨ org Bullinger and Gunnar Brink
779
789
Orchestration Modeling of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . Bertrand David and Ren´e Chalon
796
An Exploration of Perspective Changes within MBD . . . . . . . . . . . . . . . . . Anke Dittmar and Peter Forbrig
806
Rapid Development of Scoped User Interfaces . . . . . . . . . . . . . . . . . . . . . . . Denis Dub´e, Jacob Beard, and Hans Vangheluwe
816
PaMGIS: A Framework for Pattern-Based Modeling and Generation of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J¨ urgen Engel and Christian M¨ artin
826
People-Oriented Programming: From Agent-Oriented Analysis to the Design of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steve Goschnick
836
Visualization of Software and Systems as Support Mechanism for Integrated Software Project Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Liggesmeyer, Jens Heidrich, J¨ urgen M¨ unch, Robert Kalckl¨ osch, Henning Barthel, and Dirk Zeckzer Collage: A Declarative Programming Model for Compositional Development of Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruce Lucas, Rahul Akolkar, and Charlie Wiecha Hypernetwork Model to Represent Similarity Details Applied to Musical Instrument Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuya Maeshiro, Midori Maeshiro, Katsunori Shimohara, and Shin-ichi Nakayama
846
856
866
Table of Contents
Open Collaborative Development: Trends, Tools, and Tactics . . . . . . . . . . Kathrin M. Moeslein, Angelika C. Bullinger, and Jens Soeldner
XXIII
874
Investigating the Run Time Behavior of Distributed Applications by Using Tiny Java Virtual Machines with Wireless Communications . . . . . . Tsuyoshi Miyazaki, Takayuki Suzuki, and Fujio Yamamoto
882
OntoDesk: Ontology-Based Persistent System-Wide Undo on the Desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . David Nemeskey, Buntarou Shizuki, and Jiro Tanaka
890
Peer-to-Peer File Sharing Communication Detection System with Traffic Mining and Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Satoshi Togawa, Kazuhide Kanenishi, and Yoneo Yano
900
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
911
Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images Kiyohiko Abe1, Shoichi Ohi2, and Minoru Ohyama3 1
College of Engineering, Kanto Gakuin University, 1-50-1 Mutsuura-higashi, Kanazawa-ku, Yokohama, Kanagawa 236-8501, Japan 2 School of Engineering, Tokyo Denki University, 2-2 Kandanishiki-cho, Chiyoda-ku, Tokyo 101-8457, Japan 3 School of Information Environment, Tokyo Denki University, 2-1200 Muzaigakuendai, Inzai-shi, Chiba 270-1382, Japan
[email protected],
[email protected],
[email protected]
Abstract. We propose a new eye blink detection method that uses NTSC video cameras. This method utilizes split-interlaced images of the eye. These split images are odd- and even-field images in the NTSC format and are generated from NTSC frames (interlaced images). The proposed method yields a time resolution that is double that in the NTSC format; that is, the detailed temporal change that occurs during the process of eye blinking can be measured. To verify the accuracy of the proposed method, experiments are performed using a high-speed digital video camera. Furthermore, results obtained using the NTSC camera were compared with those obtained using the high-speed digital video camera. We also report experimental results for comparing measurements made by the NTSC camera and the high-speed digital video camera. Keywords: Eye Blink, Interlaced Image, Natural Light, Image Analysis, HighSpeed Camera.
1 Introduction The blinking of the eye is related to factors such as human cognition, fatigue, and depressed consciousness; many studies have investigated eye blinking in relation to these factors. Most conventional methods for the measurement of the eye blink analyze eye images (images of the eye and its surrounding skin) captured by a video camera [1], [2], [3]. The NTSC video cameras that are commonly used are capable of detecting eye blinks; however, it is difficult for these cameras to measure the detailed temporal change occurring during the process of eye blinking, because eye blinks occur relatively fast (within a few hundred milliseconds). Therefore, a high-speed camera is required for an accurate measurement of the eye blink [3]. NTSC video cameras capture moving images at 60 fields/s and these field images are mixed with images that have a frame rate of 30 frames/s (fps) to field interlaced images. In this paper, we propose a new method for measuring the eye blink that uses J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 3–11, 2009. © Springer-Verlag Berlin Heidelberg 2009
4
K. Abe, S. Ohi, and M. Ohyama
NTSC video cameras. This method utilizes split-interlaced images of the eye captured by an NTSC video camera. These split images are odd- and even-field images in the NTSC format and are generated from NTSC frames (interlaced images). The proposed method yields a time resolution that is twice that in the NTSC format. Therefore, the detailed temporal change that occurs during the process of eye blinking can be measured. To verify the accuracy of the proposed method, we performed experiments using a high-speed digital video camera. Thereafter, we compared results obtained using the NTSC cameras with those obtained using the high-speed digital video camera. This paper also presents experiments that evaluate the proposed automatic method for measuring eye blinks.
2 Open-Eye Area Extraction Method by Image Analysis In general, eye blinks are estimated by measuring the open-eye area [2] or on basis of characteristics of specific moving points between the upper and lower eyelids [3]. Many of these methods utilize image analysis. It is possible to measure the wave pattern of eye blinks if the entire process of an eye blink is captured [3]. Furthermore, the type of eye blink and/or its velocity can be estimated on the basis of this wave pattern. However, it is difficult to measure the wave patterns of eye blinks by using video cameras that are commonly used for measuring eye blinks because the resulting eye images include high noise content owing to the change in light conditions. We have developed a new method for measuring the wave pattern of an eye blink. This method can be used with common indoor lighting sources such as fluorescent lights, and it can measure the wave pattern automatically. Hence, our proposed measurement method can be used under a variety of experimental conditions. In this method, the wave pattern is obtained by counting the number of pixels in the openeye area of the image as captured by a video camera. This image is enlarged for capturing the detailed eye image. We have proposed an algorithm for extracting the open-eye area in a previous study [4]. It utilizes color information of eye images. We have adapted the algorithm to our proposed method for elucidating the wave pattern of eye blink measurement. This algorithm has been developed for our eye-gaze input system, in which it compensates and traces head-movement [5]. Furthermore, the algorithm has been used under common indoor sources of light for a prolonged period. Hereafter, we describe in detail our image-processing algorithm for extracting the open-eye area. 2.1 Binarization Using Color Information on Image Many methods have been developed for the purpose of skin-color extraction; these methods are primarily focused on facial image processing, including those that utilize color information on a facial image. They mostly determine threshold skin-color values statistically or empirically [6]. We have developed an automatic algorithm for estimating thresholds of skin-color. Our algorithm can extract the open-eye area from the eye image on the basis of the skin-color.
Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images
5
Using our algorithm, skin-color threshold is determined by the histogram of the color-difference signal ratio of each pixel—Cr/Cb—that is calculated from the YCbCr image transformed from the RGB image. The histogram of the Cr/Cb value has 2 peaks indicating skin area and open-eye area. The Cr/Cb value indicated by the minimum value between the 2 peaks is designated as the threshold for open-eye area extraction. 2.2
Binarization by Pattern Matching Method
The method described in Subsection 2.1 can extract the open-eye area almost completely. However, the results of this extraction sometimes leave deficits around the corner of eye, because the Cr/Cb value around the corner of eye is similar to the value on skin in certain subjects. To resolve this problem, we have developed a method for open-eye extraction without deficits by combining 2 extraction results. One of them is a binarized image using color information, as described in Section 2.1. The other extraction result is a binarized image using light intensity information, which includes in the extraction result the area around the corner of the eye. Binarization using light intensity information utilizes the threshold estimated by a pattern matching method, which determines the matching point by using the color information of the binarized image as reference data. Hence, the threshold level is estimated automatically. The original image and the extracted open-eye area image are shown in Fig. 1(a) and Fig. 1(b).
(a)
(b)
Fig. 1. Original eye image (a) and extracted open-eye area (b)
3 Measurement Method of Wave Patterns of Eye Blinks Using Split-Interlaced Images Commonly used NTSC video cameras output interlaced images. One interlaced image has 2 field images, which are designated as odd or even fields. If an NTSC camera captures a fast movement such as an eye blink, there is a great divergence between the captured odd- and even-field images. Therefore, the area around eyelids on the captured image has comb-like noise. This phenomenon occurs because of mixing of 2 field images of the fast movement of eyelids. An example of interlaced images during eye blinking is shown in Fig. 2. To describe this phenomenon most clearly, Fig. 2 has been captured at low resolution (145 × 80 pixels).
6
K. Abe, S. Ohi, and M. Ohyama
If one interlaced image is split by scanning even- and odd-numbered lines separately, 2 field images are generated. Thus, the time resolution of the motion images doubles, but the amount of information in the vertical direction decreases by half. These field images are captured at 60 fields/sec, and the NTSC interlaced moving images are captured at 30 fps; therefore, this method yields a time resolution that is double that available in the NTSC format. The duration of a conscious blink is a few hundred milliseconds; therefore, it is difficult to measure accurately the wave pattern of an eye blink by using NTSC cameras. However, the detailed wave pattern of an eye blink can be measured by using our proposed method. The split-interlaced images are shown in Fig. 3. The 2 eye images shown in Fig. 3 are enlarged in a vertical direction and were generated from the interlaced image shown in Fig. 2. Our proposed method measures the wave patterns of eye blinks from these images.
Fig. 2. Blinking eye image (interlaced)
Fig. 3. Split-interlaced image generated from Fig. 2
4 Evaluation Experiment for Proposed Method Either 4 or 5 subjects participated in experiments to evaluate our proposed method, as described in Subsections 4.1 and 4.2, respectively. The experimental setup includes an NTSC DV camera (for home use), a high-speed digital video camera, and a personal computer (PC). The PC analyzes sequenced eye images captured by the video cameras. The DV camera captures interlaced images at 30 fps, and the high-speed digital video camera captures non-interlaced images at 300 fps. In the experiments performed using these video cameras, the wave pattern of eye blinks is measured from sequenced eye images. The experimental setup is shown in Fig. 4.
Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images PC
7
Display
User NTSC or High-speed digital video camera
Fig. 4. Hardware configuration of experimental system
4.1 Experiment for Eye Blink Measurement Using NTSC Camera In this experiment, sequenced eye images were captured using the DV camera at 30 fps in NTSC format. In addition, split-interlaced images are generated from these interlaced NTSC images. These split-interlaced images have a time resolution of 60 fields/s. The wave pattern of eye blinks is measured by the interlaced NTSC images and split-interlaced images. The binarization threshold for open-eye area extraction is determined automatically from the first field image of the experimental moving images. This threshold is estimated by the method described in Section 2. A typical result from this experiment is shown in Fig. 5.
Pixels of open-eye area
1.1 1 0.9 0.8
60fps
0.7
30fps
0.6 0.5 0.4 1
6
11 16 21 Sampling point (1/60 sec.)
26
Fig. 5. Wave patterns of eye blinks measured by DV (30 fps and 60 fps)
In Fig. 5, the longitudinal axis and the abscissa axis indicate pixels of open-eye area and sampling point (interval: 1/60 sec), respectively. To compare the 2 wave patterns of eye blinks, these plots are normalized using the pixels of open-eye area at the first field image. The bottoms of the plots indicate the eye-closed condition. Our proposed algorithm classifies the area of eyelid outline and cilia into the open-eye area; therefore, the pixels at the bottom of the plots are not reduced to zero. From Fig. 5, it is evident that sequenced images at 60 fields/s can be used to estimate the detailed wave pattern of an eye blink. During the eye blink, there is a great difference in the 2 plots of pixels of the open-eye area; however, this difference is not dependent on individual subjects.
8
K. Abe, S. Ohi, and M. Ohyama
Results of the wave pattern of eye blink measurements for 5 subjects are shown in Fig. 6, where the longitudinal axis and the abscissa axis show pixels of open-eye area and sampling point, respectively. These plots also are normalized in a manner similar to those in Fig. 5. From Fig. 6, it is evident that there are great differences in the results for each subject. 1.1
Pixels of open-eye area
1 0.9 0.8 0.7 0.6 0.5 0.4
Subject A Subject D
0.3
Subject B Subject E
Subject C
0.2 1
11 21 31 41 Sampling point (1/60 sec.)
51
Fig. 6. Wave patterns of eye blinks of 5 subjects measured by DV (60 fps)
4.2 Experiment for Eye Blink Measurement Using High-Speed Video Camera To verify the accuracy of the proposed method that utilizes split-interlaced images, experiments were conducted with 4 subjects; this experiment and the one described in Subsection 4.1 were conducted separately. Subjects A and E (listed in Fig. 6) were enrolled in this experiment continuously, in which sequenced images at 3 different frame rates (30, 60, and 150 fps) were generated from moving images captured by the high-speed digital video camera. These sequenced images were then analyzed to measure the wave pattern of eye blinks. The results of eye blink measurements performed using the sequenced images at 3 different frame rates and those taken at 300 fps are compared. Typical examples of measurement results are shown in Fig. 7, Fig. 8, and Fig. 9, which display results at 30, 60, and 150 fps, respectively. From Fig. 7 and Fig. 8, it is evident that the degree of accuracy of measurement at 60 fps is higher than that at 30 fps. The minimum of the wave pattern (bottom of the curve) is quite characteristic of when an eye blink occurs. Results at 60 fps show that the bottom of the plot is measured with a high degree of accuracy. Therefore, sequenced images at this frame rate are suitable for measurement of eyelid movement velocity. Moreover, our proposed method using split-interlaced images (described in Section 3) utilizes 2 field images generated from one interlaced image; that is, the
Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images
9
spatial information of these field images is decreased by half. We have confirmed that this decrease in spatial information does not affect measurement accuracy via an experiment using sequenced images at 60 fps. The sequenced images at 60 fps were generated from moving images captured by a high-speed digital video camera. In this experiment, we generated half-sized eye images by extracting scanned odd-numbered lines from sequenced images at 60 fps. We estimated wave patterns of eye blinks using these half-sized images. Our results show that the measured open-eye area decreases by half, which is in agreement with the results shown in Fig.8.
Pixels of open-eye area
46000 300 fps
44000
30 fps
42000 40000 38000 36000 34000 32000 1
11
21 31 41 51 61 71 Sampling point (1/300 sec.)
81
Fig. 7. Wave pattern of eye blinks measured by high-speed video camera (30 fps)
Pixels of open-eye area
46000
300 fps
44000
60 fps
42000 40000 38000 36000 34000 32000 1
11
21 31 41 51 61 71 Sampling point (1/300 sec.)
81
Fig. 8. Wave pattern of eye blinks measured by high-speed video camera (60 fps)
10
K. Abe, S. Ohi, and M. Ohyama
Pixels of open-eye area
46000
300 fps
44000
150 fps
42000 40000 38000 36000 34000 32000 1
11
21 31 41 51 61 71 Sampling point (1/300 sec.)
81
Fig. 9. Wave pattern of eye blinks measured by high-speed video camera (150 fps)
4.3 Discussion On the basis of Fig.5, it is evident that by using split-interlaced images, the time resolution of measurement is doubled than that of the results obtained in previous studies. These split images are odd- and even-numbered field images in the NTSC format that are generated from NTSC frames. This method can also be utilized for any subject under common indoor lighting sources, such as fluorescent lights. We have shown the wave patterns of eye blinks for 5 subjects in Fig. 6. From results shown in Fig. 7, Fig. 8, and Fig. 9, it is evident that the degree of accuracy of measurement increases with increasing frame rate. A closer estimate of eye blinking velocity can be achieved if the wave pattern of an eye blink were to be measured with higher accuracy. In other words, the type of eye blink can be classified with a high degree of accuracy. In addition, our proposed method can measure the wave patterns of eye blinks efficiently even by using half-sized eye images. As shown by our experimental results presented earlier, we have verified the reliability of our proposed method described in Section 3. Thus, detailed wave patterns of eye blinks can be measured by using our proposed method.
5 Conclusions We present a new automatic method for measuring eye blinks. Our method utilizes split-interlaced images of the eye captured by an NTSC video camera. These split images are odd- and even-numbered field images in the NTSC format and are generated from NTSC moving images. By using this method, the time resolution for measurement increases to 60 fps, which is double that of conventional methods. Besides the function of automatic measurement of eye blinks, our method can be used under common indoor lighting sources, such as fluorescent lights. In evaluation experiments, we measured eye blinks of all subjects without problems.
Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images
11
To verify the accuracy of our proposed method, we performed experiments using a high-speed digital video camera. On comparison of the results obtained using NTSC cameras with those obtained using a high-speed digital video camera, it is evident that the degree of accuracy of measurement increases with increased resolution time. Additionally, a decrease in area of the split-interlaced image has no adverse effect on the results of eye blink measurements. We confirmed that our proposed method is capable of measuring the wave pattern of eye blinks with high accuracy by using an NTSC video camera. In the future, we plan to develop a new method for classifying types of eye blinks using our proposed measurement method reported above. That new method will be capable of profiling eye blinks according to velocity of open-eye area changes. We also plan to apply this new method to more general ergonomic measurements.
References 1. Grauman, K., Betke, M., Gips, J., Bradski, G.R.: Communication via Eye Blinks - Detection and Duration Analysis in Real Time. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1010–1017, Lihue, HI (2001) 2. Morris, T., Blenkhorn, P., Zaidi, F.: Blink Detection for Real-Time Eye Tracking. J. Network and Computer Applications 25(2), 129–143 (2002) 3. Ohzeki, K., Ryo, B.: Video Analysis for Detecting Eye Blinking Using a High-Speed Camera. In: Proc. of Fortieth Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, pp. 1081–1085 (2006) 4. Abe, K., Ohyama, M., Ohi, S.: Eye-Gaze Input System with Multi-Indicators Based on Image Analysis under Natural Light. J. The Institute of Image Information and Television Engineers 58(11), 1656–1664 (2004) (in Japanese) 5. Abe, K., Ohi, S., Ohyama, M.: An Eye-Gaze Input System Using Information on Eye Movement History. In: Proc. on 12th International Conference on Human-Computer Interaction, HCI International 2007, Beijing, vol. 6, pp. 721–729 (2007) 6. Garcia, C., Tziritas, G.: Face Detection Using Quantized Skin Color Regions Merging and Wavelet Packet Analysis. IEEE Trans. on Multimedia 1(3), 264–277 (1999)
A Usability Study of WebMaps with Eye Tracking Tool: The Effects of Iconic Representation of Information Özge Alaçam and Mustafa Dalcı Human Computer Interaction Research and Application Laboratory, Computer Center, Middle East Technical University, 06531 Ankara/Turkey {ozge,mdalci}@metu.edu.tr
Abstract. In this study, we aim to conduct usability tests on different WebMap sites with eye movement analysis. Overall task performance, the effects of iconic representation of information, and the efficiency of pop-up usage were evaluated. The eye tracking technology is used for this study in order to follow the position of the users’ eye-gaze. The results show that there are remarkable differences in task performance between WebMaps. Addition, they also differ in the use of iconic representations according to results of users’ evaluation. It is also found that efficiency of pop-up windows’ usage has an effect on task performance. Keywords: Web mapping, usability, eye tracking, cognitive processes, iconic representations, and the efficiency of pop-ups.
1 Introduction Web mapping sites became widespread in many professional areas since they provide opportunities such as designing and sharing maps on the World Wide Web. Beside to their role on professional area, it also became very considerable part of our daily life since it makes the navigation easier [13]. Addition to large number of web mapping sites’ users which access these sites with their desktop and laptop PCs, improvements in technology make the internet available in nearly everywhere providing a chance to connect with mobile devices (mobile phones, smart phones, PDAs) and multiply the number of web mapping sites’ users. By the increasing number of web mapping sites’ users, researchers started to conduct usability studies of these sites and to investigate the effects of usability [2, 8, 13, 15]. The term usability is defined by ISO 9241 [9] as “the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments” [9, 13]. Another definition from Nielson, one of the pioneers in the usability field, states usability as a quality evaluation that assesses how easy user interfaces are to use. According to his definition, usability is composed of five quality components [16]; these components are learnability, efficiency (task completion time), memorability, errors, satisfaction. Addition to these parameters obtained from usability study, usage of the eye tracking tools adds a different aspect to the usability field for the reason that it provides objective and quantitative evidence to investigate user’s cognitive processes such as visual and attentional J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 12–21, 2009. © Springer-Verlag Berlin Heidelberg 2009
A Usability Study of WebMaps with Eye Tracking Tool
13
processes [5]. Usage of eye tracking on usability field started at 1950’s [6]. However due to the difficulties in the analysis of huge data obtained from the eye tracking tools, it lost its popularity in 1970’s. With the improvements of the eye tracking technologies, eye tracking tools gain their impacts on the usability field again [10] and nowadays they are accepted as a tool to improve computer interface. In one of the studies about WebMap usability conducted by Nivala [13], severity of the usability problems were investigated. In our study, we aim to make additional analysis to find the reason of these usability problems and make them more clear by analyzing eye movements of the users. The focus of this study is to analyze the effects of the iconic representation of the information and to investigate whether the pop-ups are used efficiently by the user. The eye tracking tool is used for this study in order to follow the position of the users’ eye movements, which helps to measure the attended location on the map. It is known that eye movements provide information about cognitive processes such as perception, thinking, decision making and memory [1, 3, 4, 12, 14]. Evaluation of eye movement provided us the opportunity to focus on the iconic representations, efficiency of pop-up windows and their effects in map comprehension in different WebMaps.
2 Method and Materials 26 subjects (12 female, 14 male) either university students or graduate in a range of ages between 18 and 32 participated to this study. In order to get information about their prior knowledge about WebMap usage and to get the user’s idea about the comprehensibility of the icons and preferences about the WebMaps, a questionnaire was carried out. Each subject evaluated two different WebMaps for different places in US in random order. Six tasks shown in Table 1 were used in the experiment. Users were told that they could give up the task or the experiment whenever they wanted to. Tasks given to the users include; to find given address, to find definite places which are represented with icons (s.t airport, metro station, hospital) and to show the route to specific locations. The experiments are conducted at the Human-Computer Interaction Research and Application Laboratory at Middle East Technical University. Eye movements of users were collected by Tobii 1750 Eye Tracker and analyzed with Tobii Studio. Table 1. Task Description Task No Task Description Instruction Welcome to X City/State. You are planning to look at the city map to specify the locations that you want to visit before starting your trip 1 Point the nearest highway intersection to X International Airport You want to go from X International Airport to X University. Could you describe 2 how to arrive to that location? Find the address of the hospital nearest to X Park 3 Now, you are in X Location. Show the nearest metro/railway station to this place 4 You are searching for the library nearest to X place. Find and point it on the map. 5 Show the intersection point of the given address with X street. 6
14
Ö. Alaçam and M. Dalcı
In the Nivela et al.’s study [13], there is an evaluation of four different web mapping sites. These are Google Maps, MSN Maps & Directions, MapQuest and Multimap. However, since the MSN Maps and Directions, Multimap are based on Microsoft Virtual Earth, we replaced these sites with Live Search Maps that is also based on MS Virtual Earth. Since these are well-known and all have zooming and panning options on their 2D map applications, they are very good candidates for usability testing. Although their common properties mentioned above, they differed in terms of usage of icon representation and pop-up window properties. We conducted the usability testing of Google Maps, Live Search Map, MapQuest and Yahoo Map and investigated the effect of iconic representation of information and pop-up windows by analyzing eye movements. We use the term “The iconic representation of information” as to state the relationship between their semantics and appearance. Addition to evaluation of task completion performance (s.t. task completion score and time), eye tracking data such as fixation length, fixation count, observation length was collected.
3 Results Results are presented under three categories; task performance, analysis of the iconic representations and analysis of pop-up windows. 3.1 Task Performance Users are grouped into two categories according to their WebMap usage experience; experienced users (14 users) for high-level usage frequency and inexperienced users (12 users) for low level usage frequency. One way ANOVA test was conducted to compare mean fixation length on task completion time for experience level. Result shows that user’s experience level has a significant effect on task completion time, F(1,52)=5,30, p>.05. One of the evaluation criteria of comparing the usability of WebMaps is users’ task completion scores. Task completion score was evaluated under three categories; accomplished tasks, unaccomplished tasks and partially accomplished tasks that the users thought that they accomplished a task when they actually did not. Table 2 provides the percentage of users, who accomplished, partially accomplished and did not accomplish each task and also overall score was calculated for each WebMap site. Fig. 1 shows the overall completion score for each map. Results of one way ANOVA shows that task completion score of Google Map is significantly different than MapQuest and Yahoo Map, F(3,48)= 8.629 p<.05. It is also worth to note that significance value of difference between Live and Yahoo is .05. In addition to analysis of task completion score, mean fixation length for each task was analyzed individually (see Fig. 2 for comparison). Only fixation length on accomplished and partially accomplished tasks was counted. The results show that, for the first task, there is no significant difference in fixation length according to map type. The fixation length during task two, a significant difference was found between Live Search Map and MapQuest, Google Map and MapQuest, Yahoo Map and MapQuest F (3,43)= 12.538, p<.05. For the third task, Google Map is significantly
A Usability Study of WebMaps with Eye Tracking Tool
15
different than MapQuest, F (3,33)=3.768, p<.05. For the fourth task, Google Map is significantly different than MapQuest and Live Search Map, F(3,23)= 5.398 p<.05. The fixation length on the fifth task, Google Map is significantly different than MapQuest, Yahoo Map and Live Search Map, F(3,35)=12.058, p<.05. For the task six, only difference in Live Search Map and MapQuest is significant, F(3,41)=2.444, p<.05. Statistical analysis of mean fixation count for each task also shows that there are significant differences between the pairs given above. Table 2. Percentage of users’ task completion scores
Task2
Task3
Task4
Task5
Task6
Overall
tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished
Live Search Map 86.7 13.3 0.0 93.3 0.0 6.7 46.7 26.7 26.7 46.7 0.0 53.3 53.3 13.3 33.3 100.0 0.0 0.0 71.1 8.9 20.0
Map Quest 100.0 0.0 0.0 50.0 35.7 14.2 64.3 14.2 21.4 14.2 7.1 78.6 57.1 0.0 42.8 57.1 0.0 42.9 57.1 9.5 33.3
120,0 Number of participants (%)
Task1
Google Map 100.0 0.0 0.0 100.0 0.0 0.0 35.7 21.4 42.8 78.5 14.2 7.1 78.5 21.4 0.0 100.0 0.0 0.0 82.1 9.5 8.3
100,0 80,0 60,0 40,0 20,0 0,0 Google Map tasks accomplished
Live Search
Map Quest
tasks partially accomplished
Yahoo Map tasks unaccomplished
Fig. 1. The percentage of users according to task completion score
Yahoo Map 100.0 0.0 0.0 63.6 0.0 36.2 36.4 27.2 36.4 9.1 36.4 54.5 45.5 18.1 36.4 45.5 18.2 36.4 50.0 16.7 33.3
16
Ö. Alaçam and M. Dalcı
600 500 400 300 200 100 0 görev1
görev2
Live Search Map
görev3 Google Map
görev4 Yahoo Map
görev5
görev6
MapQuest
Fig. 2. Fixation Length (sec.) on each task according to WebMaps
3.2 Analysis of the Iconic Representations In order to investigate the efficiency of iconic representations in these WebMaps, the observation length on icons was counted frame by frame in specific tasks. In order to analyze icon usage, first, third and fourth tasks were selected since these tasks contain specific places which can be represented by icons (such that airport icon for first task, hospital icon for task three, metro/railway icon for task 4, and pointers that appears after users’ search for all three tasks). Other icons displayed on the map during these tasks were also investigated. Fixation length on each icon and the time which they are displayed on the map were counted and then the percentage of iconic representation looking time was calculated. One-way ANOVA results shows that there is no significant difference in observation length on icons for each WebMap, F(3,48)=1,859, p>.05. In addition, no correlation between the icon usage looking time percentage and completion score was found. (Spearman’s rho=.37, p>.05) Icons are divided into two categories as, task related icons and task unrelated icons. However, since every map could not have a representation for each icon, they were investigated individually, without conducting statistical analysis. Since MapQuest had only pointer icons, these are not compared with the specific icons in other maps. Table 3 provides the percentage of looking time for each icon in each WebMap. Observation length on Airport in Yahoo Map, is 13,3% of the time that it appears on the map. However, since the other WebMaps (Google Maps, MapQuest and Live Search Map) do not have icon for airport, there can be no comparison analysis. The pointers which represent the searched location have approximately same looking time for all maps. For the metro and railway icons, Google icon has biggest looking time percentage, and then it is followed by Live Search Map and Yahoo Map respectively. Since the MapQuest and Live Search Maps do not have hospital icons, looking time of icons on only Yahoo Map and Google Map were investigated. Even tough Google and Yahoo Maps contain an icon for “Hospital”, the users are expected to zoom close
A Usability Study of WebMaps with Eye Tracking Tool
17
enough for that icon to be visible. However none of the users were inclined to zoom to that distance while performing their tasks. The eye movement analysis of users which perform the experiment on Live Map Search shows an interesting outcome; some task unrelated icons (s.t. park, hotel or sponsored link icons) have been remarkably fixated by the users. Table 3. The percentage of looking time for each icon Icon Type Google Map Task Related Icons
Unrelated Icons
Live Search
Map Quest
Yahoo Map
Airport
Na*
Na
Na
13,3
Pointers
10,6
10,4
10,9
10,5
Metro/railway
13,5
10,8
Na
7,2
University
3,5
Na
Na
Na
Hospital
0,0
Na
Na
0,0
Park
1,4
Na
Na
Na
Hotel
Na
5,5
Na
Na
Sponsored link
Na
1,7
Na
Na
*Not Applicable.
Task based investigation has been made regarding iconic representations. Correlations between the icon looking time percentage, completion score and observation length for Task 1,3 and 4 were examined individually. For Task 1, no correlation between these parameters was found. The analysis conducted for the third task, correlations have been found between the icon looking time percentage and completion score (Pearson’s r=.523; p<.05) and between icon looking time and observation length (Pearson’s r=-.368; p<.05). For Task four, a correlation has been found between icon looking time and observation length (Pearson’s r=-.289; p<.05). On the other hand, no correlation has been found between the icon looking time percentage and completion score (Pearson’s r=.165; p>.05). Addition to eye movement analysis, user evaluation questionnaire was carried out. The users were asked to predict the meaning of icons and rate their comprehensibility in a scale from 1 to 10 (the results are given in Table 4). This gives us the opportunity to evaluate the efficiency of the relationship between their semantics and appearances and show whether there is an ambiguity. The comprehensibility ratings for only correct predictions were counted. It can be claimed that there is a consistency between its appearance and semantic when both the correct predictions and comprehensibility rate of a icon is high. Although the comprehensibility evaluated by the users is high for the hospital icon in Google, only 6.3 % of the users predict the meaning of the icon correctly. The comparison of the results for metro/railway icons indicates that iconic representations in Google Maps and Yahoo Maps are comprehended more easily by the users than in Live Search Maps. None of the users predicts the meaning of the sponsored link and hotel icon in Live Search Map. When we look at the Table 3 for looking time percentage of these icons, it can be concluded that the users notice them without attaching a meaning.
18
Ö. Alaçam and M. Dalcı Table 4. Users’ ratings on iconic representations Icon
% of correct definition by users
Comprehensibility
+
6,3
9,0
Hospital (Yahoo Map)
87,5
8,5
Airport (Yahoo Map) Metro/Railway (GoogleMap) Metro/Railway (Yahoo Map) Metro/Railway (Live Search Map)
100
10
81,3 / 93,8
6,6 / 7,9
81,3
6,6
62,5
6,2
Hotel (Live Search Map)
0
Na
Sponsored Links (Live Search Map)
0
Na
Hospital (Google Map)
Moreover, users were asked to specify their icon preference and rate the usabilty of the maps which they used during the experiment. We can group the icons into three main categories; these are Pictorial, Textual and Numerical/Alphabetical icons. Textual icons like (s.t. ) are mostly prefered icons (37.5%). 31,3 % of the users prefer pictorial icons (s.t. ), to be used in maps. Another 31,3 % of the users prefer numerical/alphabetical icons (s.t. ). Additionally, users’ usabilty ratings of Web Maps are parallel with task completion score given in Table 2. Their ratings are 8.3 for Google Map 5.1 for live Search Map, 4.6 for Yahoo Map and 3,4 for MapQuest. 3.3 Analysis of Pop-Up Windows Additional analysis was conducted to investigate the usage of pop-up windows. It can be suggested that the aim of the pop-up is to direct users focus on something else where they are provided with additional information either regarding their task or not. Moreover, these pop-ups can facilitate additional search on locations that the task requires. Therefore pop-up windows are very important parts of the web maps and they frequently appear during map usage. In order to investigate the efficiency of popup usage in these WebMaps, the sections that contain pop-up windows on map area were extracted for each task. Fixation length and fixation count on pop-up windows and display time of it on map area was counted. The analysis of looking time of popup windows when they appear on the map gives us an idea about whether the users prefer to use them. The results of this analysis showed that Google Maps use the popup windows more efficiently, since % 64.8 of the fixations on the map are on the popup area. It is followed by MapQuest, Live Search Map and Yahoo Map respectively (see Fig. 3). One way ANOVA results also indicates that there is a significant difference between Google Map and Live Search Map in pop-up usage percentage. Google
A Usability Study of WebMaps with Eye Tracking Tool
19
Observation Length (sec)
200,00 180,00 160,00 140,00 120,00 100,00 80,00 60,00 40,00 20,00 0,00
Google Map
Live Search Map
MapQuest
popup
48,96
40,05
63,16
25,79
map
75,52
96,78
115,19
77,38
map
Yahoo Map
popup
Fig. 3. Observation Length on Pop-up / Map for each WebMap
Map is also significantly different than Yahoo Map. Addition, there is also difference between Yahoo Map and MapQuest, F(3,51)=5.939, p<.05. Additionally, correlations have been found between fixation count on pop-ups and overall completion score (Spearman’s rho=.396; p<.05) and between fixation length on pop-ups and overall completion score (Spearman’s rho=.423; p<.05).
4 Conclusion By analyzing eye movements which are indicators of cognitive processes, we examined the effects of iconic representations, pop-up window usage, and their roles on the usability of these mapping sites. Since comprehension of maps contains very complicated cognitive process, decreasing the cognitive load by making the icons more comprehensible and the pop-ups more usable will cause effectiveness and efficiency in the usage of web mapping sites. Task performance evaluation shows that there is a significant difference between these Webmaps. When the tasks are getting complicated (ex. to find an address of specific location near to another location), differences between these WebMaps in terms of task completion time and score has become apparent. Besides the differences in overall display organization, in a micro level framework, we examined the effect of the iconic representation of information and efficiency of the usage of popup windows. The analysis indicates that there is gab between user’s ratings on icons and their looking time percentages. Even the icons that are correctly predicted and given high comprehensibility level by the users have a low looking time. Additionally, it is also worth to be noted that icon’s looking time percentage is low for even task related icons. This means users have difficulty to detect them when they are performing their tasks although the icons are highly related with tasks. To make them more visible can help the users notice them, and increase their task completion performance. Moreover, analysis on the efficiency of pop-up windows, which are other
20
Ö. Alaçam and M. Dalcı
widely used elements of WebMaps, shows that users’ pop-up usage differs significantly according to WebMap type and this parameter is positively correlated with task completion score. In addition, as an expected outcome, experience level has a significant role on WebMap usage performance.
5 Further Studies However, icons for local traffic signs (ex. Highway numbers) were highly fixated during tasks; however these findings have been disregarded due to the user’s lack of familiarity towards to locations. Follow up studies to investigate these icons can be conducted with native users of the location. In addition, to evaluate the interaction between some particular areas of web site (s.t. search bars, menus, map area, information area which shows the results of the searched location) gives additional information about the efficiency and effectiveness of the usability of the WebMaps. Acknowledgements. We thank to the Computer Center / Middle East Technical University, and TUBİTAK (for support under grant SOBAG 104K098) for providing eye tracking system. We also thank to Yasemin Saatiçioğlu Oran and our colleagues in METU Computer Center for their valuable support.
References 1. Barkowsky, T.: Mental Processing of Geographic Knowledge. In: Montello, D.R. (ed.) COSIT 2001. LNCS, vol. 2205, pp. 371–386. Springer, Heidelberg (2001) 2. Davies, C., Wood, L., Fountain, L.: User-centred GI: hearing the voice of the customer. In: AGI 2005: People, Places and Partnerships, Annual Conference of the Association for Geographic Information, London, UK, November 8–10, 2005, Association for Geographic Information, London (2005) 3. Downs, R.G., Liben, L.S.: The Development of Expertise in Geography: A CognitiveDevelopmental Approach to Geographic Education. Annals of the Association of American Geographers 81(2), 304–327 (2005) 4. Downs, R.G., Liben, L.S., Daggs, D.G.: The Development of Expertise in Geography: A Cognitive-On Education and Geographers. The Role of Cognitive Developmental Theory in Geographic Education 78(4), 680–700 (2005) 5. Duchowski, A.T.: A Breadth-First Survey of Eye Tracking Applications. Behavior Research Methods, Instruments, & Computers (BRMIC) 34, 455–470 (2002) 6. Fitts, P.M., Jones, J.E., Milton, J.L.: Eye movements of aircraft pilots during instrument landing approaches. Aeronautical Engineering Review 9(2), 24–29 (1950) 7. Global Mapping International, http://www.gmi.org/mapping/websites.htm 8. Haklay, M., Zafiri, A.: Usability Engineering for GIS: Learning from a Screenshot. The Cartographic Journal 45(2), 87–97 (2008) 9. ISO. ISO/DIS 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability, International Organization for Standardization (1998) 10. Jacob, R.J.K., Karn, K.S.: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises (Section commentary). In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eyes: Cognitive and Applied Aspects of Eye Movements, Elsevier Science, Oxford (2003)
A Usability Study of WebMaps with Eye Tracking Tool
21
11. Montello, D.R.: Cognitive Map-Design Research in the Twentieth Century: Theoretical and Empirical Approaches. Cartography and Geographic Information Science 29(3), 283– 304 (2002) 12. Murata, A., Furukawa, N.: Relationships among display features, eye movement caharcteristics and reaction time in visual search. Human Factors 47(3) (2005) 13. Nivala, A.M., Brewster, S., Sarjakoski, L.T.: Usability Evaluation of Web Mapping Sites. The Cartographic Journal 45(2), 129–138 (2008) 14. Richardson, D.C., Dale, R., Spivey, M.J.: Eye movements in language and cognition: A brief introduction. In: Gonzalez- Marquez, M., Coulson, S., Mittelberg, I., Spivey, M.J. (eds.) Methods in cognitive linguistics, John Benjamins, Amsterdam (in press) 15. Skarlatidou, A., Haklay, M.: Public Web Mapping: Preliminary Usability Evaluation. In: Proceedings of GIS Research UK Conference, Nottingham, April 5-7 (2006) 16. Usability 101: Introduction to Usability (August 25, 2003), http://www.useit.com/alertbox/20030825.html (retrieved 16:22, September 21,2008)
Feature Extraction and Selection for Inferring User Engagement in an HCI Environment Stylianos Asteriadis, Kostas Karpouzis, and Stefanos Kollias National Technical University of Athens, School of Electrical and Computer Engineering, Image, Video and Multimedia Systems Laboratory, GR-157 80 Zographou, Greece
[email protected], {kkarpou,stefanos}@cs.ntua.gr
Abstract. In this paper, we present our work towards estimating the engagement of a person to the displayed information of a computer monitor. Deciding whether a user is attentive or not, and frustrated or not, helps adapting the displayed information of a computer in special environments, such as e-learning. The aim of the current work is the development of a method that can work userindependently, without necessitating special lighting conditions and with only requirements in terms of hardware, a computer and a web-camera. Keywords: User engagement, Head Pose, Eye Gaze, Facial Feature tracking.
1 Introduction While a lot of work has been done regarding the issue of user attention estimation in multi-user environments, such as meetings [7], very few articles have been published for estimating attention (engagement) in an HCI environment [6]. In this work, we propose a method for inferring user attention based on the extraction of features deriving from facial analysis. Such systems have mainly been proposed for estimating drivers' attention. For example, in [8], a monocular system is used and color precicates are employed to detect and track the driver's facial features. Facial geometry and eye closure are calculated and driver's attention is extracted with the use of three finite state automata. In [3], the authors use solely eyes closure to determine whether a driver is attentive or not. To this aim, they use a gaussian model to describe driver's attention and, eye closure for certain amounts of time would denote inattention or fatigue. In [10], the position in space of each facial feature is detected using a stereo vision system. Based on these positions, a least square algorithm estimates the head orientation and further analysis follows for the detection of the eye gaze. The combination of the above gives a good estimate of the direction at which a user is looking, however, it requires initialization. In our work, we combine head pose with eye gaze, as well as other biometrics, in a monocular environment, not necessitating any particular lighting conditions or calibration. There is not much work in bibliography combining the two features in an unconstrained environment. Typical work is the one J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 22–29, 2009. © Springer-Verlag Berlin Heidelberg 2009
Feature Extraction and Selection for Inferring User Engagement
23
reported in [12], where facial symmetry along with Gabor filters are used for estimating head pose and eye gaze respectively. A look-up table is then built for corresponding the resulting eye gaze and head pose with the final focus of attention estimate. Here, we propose a work that can be summarized as follows: Face detection [11], followed by facial feature detection is the first step of our method, while tracking follows. Based on facial features' motion, a series of biometric measurements are extracted and their appropriateness is evaluated for inferring the level of frustration or attentiveness of a user in a Human-Computer-Interaction scenario. Our algorithm is able to recover and re-initialize in cases of occlusion or tracking failure.
2 Facial Points Detection and Tracking The method reported in [1] is used for face and eye centre and mouth corner (here, enhanced with upper and lower lip points) localization. For the detection of the eye corners (left, right, upper and lower eyelids) a technique similar to that described in [13] is used. In the current work, the point between the nostrils and two points on each eyebrow have also been used, as will be discussed later. For nostrils detection, an area around a segment of the perpendicular to the inter-ocular line was extended starting from the middle of the eyes. The darkest row of this area is considered as the vertical position of the nostrils and the middle point of this row is further used for our experiments. In a similar manner, two points on each eyebrow are extracted, as the darkest points in a neighborhood above the eye corners. The above are illustrated in Fig. 1, where the luminance values of two search areas have been projected on the vertical axis. The minimum of the projections corresponds to the features in search. Tracking is done using a three-Pyramid Lukas-Kanade algorithm. Geometrical face models, and prototypes of natural human motion are employed for recovering from erroneous tracking (see subsection 3.1).
Fig. 1. Eyebrow and nose detection search regions
3 Feature Extraction The features extracted in our method are the following: Head pose, eye gaze, eyebrow movements, head horizontal and vertical speed components, mouth horizontal and vertical opening, relative movements of the user back and forwards.
24
S. Asteriadis, K. Karpouzis, and S. Kollias
3.1 Head Pose Estimation Head rotation is calculated by examining the translation of the point in the middle of the inter-ocular line with regards to its position when the user was rotated frontally (see Fig. 2), thus providing the Head Pose Vector p = [px py], where px and py are the horizontal and vertical components of the eye middle point’s translation, respectively, normalized with the inter-ocular distance, as calculated at start-up, to cater for scale variations. The fraction of the inter-ocular distance with the vertical distance between the eyes and the mouth is monitored, and if it is restricted within certain limits with regards to its value at a frontal position, no rotation is decided. As tracking, many times, fails, thus giving false estimates of head pose (as well as the other features), a series of rules were integrated: After large rotations, some features are occluded and cannot be further recovered. In this case, when the user comes back to a frontal position, after nt1 frames, the pose length reduces in length but is above a certain threshold, as one of the eyes is not well tracked and the eye center is not at the same neighborhood as at start-up. In this case, the algorithm can re-initialize. The above can be modeled as in equations (1),(2):
p(n) < thr1 ⋅ p(n − nt1 )
(
)
var p n−nt 2:n < thr2
(1) (2)
where ║*║ denotes a vector length metric. Equations (1)-(2) are interpreted as follows: If the Head Pose Vector length at the current frame n is smaller than a fraction thr1 of its value at frame n-nt1, and its variance for the last nt2 frames (with nt2
Fig. 2. Pose changes during a video of a person in front of a monitor
Feature Extraction and Selection for Inferring User Engagement
25
smaller than thr2, the algorithm re-initializes. In our experiments, we used nt1=10, nt2=7, thr1=0.7, thr2=0.05. If the above conditions are met, but the user has not turned frontally, face detection fails and, thus, frontal rotation is not decided. In general conditions, however, the algorithm re-initializes by re-detecting the face, facial features and re-starting to track. Further constraints that are taken into account are related with the displacement of features in subsequent frames. By assuming an orthographic projection in the interval between two subsequent frames, it is expected that features are shifted in a uniform way. Finding such outliers and re-calculating the mean shift with the rest of the features helps positioning erroneous points to the position that would agree with the rest of the features’ shift. As experiments showed, the above refinement is achieved after 7-10 iterations per frame.
Fig. 3. Gaze changes during a video of a person not moving his head in front of a monitor
3.2 Eye Gaze Estimation Eye gaze is extracted by monitoring the eye centre movements with regards to a coordinate system defined by the positions of the eye corners and eyelids at each frame (see Fig. 3). The resulting displacement provides with the Eye Gaze Vector gy], where gx and gy is the horizontal and vertical component respectively.
g = [gx
3.3 Extraction of Further Features The vertical movements of the eyebrows with regards to the upper eyelids are also extracted, and the horizontal and vertical components of the speed (in pixels per frame) of the head movements are calculated. Furthermore, mouth opening is calculated, with reference to the initial distance between the mouth corners and the lips distance at start-up. Finally, the inter-ocular distance changes are monitored, and
26
S. Asteriadis, K. Karpouzis, and S. Kollias
calculated as fractions of the eye centres’ distance with regards to the first frame of each initialization of the system. In this way, when changes in inter-ocular distance are not due to head rotations, qualitative measurements of user movement back and forth are achieved.
4 Feature Selection The experiments were conducted on a database consisting of children with learning difficulties, between the age of 8 and 10. The recorded videos were 720×576 pixels and the frame-rate 25fps. A total of about 10000 and 12250 frames were used for the case of attention/non-attention and frustration/non-frustration problems respectively. The videos were annotated by experts. One of the difficulties of the dataset was that the positive instances (frustration, non-attentiveness) were very few in comparison to the negative ones, and this limited the training session prototypes. In order to evaluate the appropriateness of each of the above features, the Fisher's exact test [4] was used. To this aim, the 3-bin histogram of each of the features was calculated and the resulting distribution for positive instances was compared against the distribution of the same feature throughout all videos, regardless of the state. In our case, we chose Fisher's exact test and not any other method (e.g. chi-square method) because it is ideal for small scale data. Indeed, in many occasions (for example, horizontal speed of the head when the user is frustrated), there are only a few instances where there are low or high values at the correspondent histogram bins. Fisher's exact test is ideal in cases of such small samples. For the event of non-attentiveness, among all features, tests showed that for the features of Head Pose, Eye Gaze, Inter-Ocular Distance Changes and Head Speed, the null hypothesis (that observed and expected distributions do not differ) should be rejected with higher confidence than for the rest of the features. For the event of frustration, Head Pose, Horizontal and Vertical Head speed and Eye Gaze do not follow the null hypothesis as much as the rest of the features do, as it was expected. Figure 4 justifies the rejection of some of the features due to high p-values, while Figures 5 and 6 illustrate examples of data for each class.
Fig. 4. p-values for feature selection in attention/non-attention and frustration/non-frustration scenarios
Feature Extraction and Selection for Inferring User Engagement
27
Fig. 5. Features used for attention/non-attention classification
Fig. 6. Features used for frustration/non-frustration classification
5 Experimental Results For testing the accuracy of our system, a Sugeno-type fuzzy [9] inference system was built for each case. The idea behind using fuzzy systems has to do with the fact that behavioral states do not necessarily belong to certain classes but, rather, they are fuzzy concepts. For example, frustration or distraction can be given confidence values, and outputs of fuzzy systems are ideal in this case. Prior to training, our data were clustered using the sub-cluster algorithm described in [2]. This algorithm, instead of using a grid partition of the data, clusters them and, thus, leads to fuzzy systems deprived of the curse of dimensionality. The number of clusters created by the algorithm determines the optimum number of the fuzzy rules. After defining the fuzzy inference system architecture, its parameters (membership function centers and widths), were acquired by applying a least squares and back-propagation gradient descent method [5]. Tables 1 and 2 summarize the results of the overall accuracy of our system in estimating the behavior of a user in the cases attention/non-attention and frustration/non-frustration experiments using different sets of features with low p-value as inputs and 1 or 0 the target states (attention/non-attention or frustration/non-frustration). Testing was done by adopting a leave-one-out protocol.
28
S. Asteriadis, K. Karpouzis, and S. Kollias
From tables 1 and 2, it can be seen that, in the case of frustration, although eye gaze has low p-value (see Fig. 3), excluding it from experiments does not deteriorate the results but they are marginally higher. This is due to the fact that, in cases of frustration, in our dataset, eye gaze vector length was strongly correlated with head pose vector length. Similarly, although Head Speed (horizontal and vertical) has low p-values in the attention tests, results have shown that, excluding these parameters from our decision systems, would improve the results. More careful observation of our data and the corresponding annotation gave the following explanation: Head speed is only large at the beginning of those time segments when a person is turning his/her head away from the camera. At those time segments, head pose vector has small values but head speed is high. However, such movements can also be met during attention time-stamps, as it is very frequent for a reader to make small rapid movements without changing his head pose a lot. For the above reason, it was decided to exclude head speed from our experiments in the case of attention estimation. The database we used was acquired under normal lighting conditions, with very challenging subjects: Children with learning difficulties. Testing our system on such a dataset is challenging, not only because of its nature, but also due to the fact that annotation is subjective. However, the results obtained are extremely promising. Table 1. Neuro-Fuzzy System decision accuracy for two different sets of low p-value features for detecting User Attention Features Head Pose, Eye gaze, Distance changes, Head speed Head Pose, Eye gaze, Distance changes
Overall success rates 84.00% 88.00%
Table 2. Neuro-Fuzzy System decision accuracy for two different sets of low p-value features for detecting User Frustration Features Head Pose, Horizontal and Vertical Head speed Head Pose, Horizontal and Vertical Head speed, Eye Gaze
Overall success rates 82.00% 80.63%
6 Conclusions and Future Work We presented a method for the automatic estimation of the behavior of a person in front of an HCI environment. Our system is un-intrusive, thus, leaving space for spontaneous behavior, it does not depend on controlled conditions in terms of lighting, and this constitutes it ideal for different settings. Furthermore, since the system does not require any a-priory knowledge of the user or the camera, it does not need any kind of training or calibration beforehand. Future extensions of our work shall include the creation of a common framework for discriminating among a set of states simultaneously. To this aim, we will build a database suitable for our research and work on developing a facial feature tracker, highly specialized for such applications.
Feature Extraction and Selection for Inferring User Engagement
29
Acknowledgments This work has been funded by the FP6 IP CALLAS (Conveying Affectiveness in Leading-edge Living Adaptive Systems), Contract number IST-34800 and by the IST Project ’FEELIX’, (under contract FP6 IST-045169).
References 1. Asteriadis, S., Nikolaidis, N., Pitas, I., Pardàs, M.: Detection of facial characteristics based on edge information. In: 2nd International Conference on Computer Vision Theory and Applications (VISAPP), Barcelona, Spain, pp. 247–252 (2007) 2. Chiu, S.L.: Fuzzy Model Identification Based on Cluster Estimation. Journal of Intelligent and Fuzzy Systems 2(3), 267–278 (1994) 3. D’Orazio, T., Leo, M., Guaragnella, C., Distante, A.: A visual approach for driver inattention detection. Pattern Recognition 40(8), 2341–2355 (2007) 4. Fisher, R.A.: Statistical Methods for Research Workers. Hafner Publishing (1970) 5. Jang, J.S.R.: ANFIS: Adaptive-Network-Based Fuzzy Inference System. IEEE Transactions on Systems, Man, and Cybern. 23, 665–684 (1993) 6. Matsumoto, Y., Ogasawara, T., Zelinsky, A.: Behavior recognition based on head pose and gaze direction measurement. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Takamatsu, Japan, pp. 2127–2132 (2000) 7. Otsuka, K., Takemae, Y., Yamato, J.: A probabilistic inference of multiparty-conversation structure based on markov-switching models of gaze patterns, head directions, and utterances. In: ICMI, pp. 191–198 (2005) 8. Smith, P., Member, S., Shah, M., Lobo, N.D.V.: Determining driver visual attention with one camera. IEEE Trans. on Intelligent Transportation Systems 4, 205–218 (2003) 9. Takagi, T., Sugeno, M.: Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. Syst. Man Cybern. 15(1), 116–132 (1985) 10. Victor, T., Blomberg, O., Zelinsky, A.: Automating driver visual behavior measurement. In: 9th Vision in Vehicles Conference, Australia (2001) 11. Viola, P.A., Jones, M.J.: Rapid object detection using a boosted cascade of simple features. In: Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511–518 (2001) 12. Weidenbacher, U., Layher, G., Bayerl, P., Neumann, H.: Detection of head pose and gaze direction for human-computer interaction. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS, vol. 4021, pp. 9–19. Springer, Heidelberg (2006) 13. Zhou, Z.H., Geng, X.: Projection functions for eye detection. Pattern Recognition 37(5), 1049–1056 (2004)
Informative or Misleading? Heatmaps Deconstructed Agnieszka (Aga) Bojko User Centric, Inc. 2 Trans Am Plaza Dr, Ste 100, Oakbrook Terrace, IL 60181 USA
[email protected]
Abstract. Eye tracking heatmaps have become very popular and easy to create over the last few years. They are very compelling and can be effective in summarizing and communicating data. However, heatmaps are often used incorrectly and for the wrong reasons. In addition, many do not include all the information that is necessary for proper interpretation. This paper describes several types of heatmaps as representations of different aspects of visual attention, and provides guidance on when to use and how to interpret heatmaps. It explains how heatmaps are created and how their appearance can be modified by manipulating different display settings. Guidelines for proper use of heatmaps are also proposed. Keywords: Heatmaps, attention maps, eye tracking.
1 Introduction Heatmaps are two-dimensional graphical representations of data where the values of a variable are shown as colors. Heatmaps are compelling for two reasons. First, the intuitive nature of the color scale as it relates to temperature minimizes the amount of learning necessary to understand it. From experience, we know that yellow is warmer than green, orange is warmer than yellow, and red is hot. It is not difficult to then figure out that the amount of heat is proportional to the level of the represented variable. Second, heatmaps show the data directly over the stimulus. Because the data could not be any closer to the elements to which they pertain, little mental effort is required to read a heatmap. Heatmaps can be of great value to papers, reports, and presentations because they summarize large quantities of data that would be much more difficult to grasp if presented numerically. Heatmaps help us quickly see “the big picture” including any patterns or trends that may exist in the data. In the user experience field, heatmaps can represent various types of data, such as usage (e.g., clicks, key presses), accuracy, or visual attention. This paper focuses exclusively on attention heatmaps, which have become popular over the past few years due to the increased usage of eye tracking technology. Attention heatmaps can be easily generated with the help of an eye tracking software application such as Tobii’s ClearView, Tobii Studio, SMI’s BeGaze, NYAN, or J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 30–39, 2009. © Springer-Verlag Berlin Heidelberg 2009
Informative or Misleading? Heatmaps Deconstructed
31
EyeTools. Anyone with an eye tracker and the right software can create a heatmap. No knowledge of eye movements or of how heatmaps are created is required. As a result, heatmaps are often generated unnecessarily or are misinterpreted by those who do not understand what the visualizations are really showing or, perhaps even more importantly, not showing. Heatmaps can be deceptive because they look so intuitive that we often do not realize how much we actually do not understand. This paper describes different heatmap types and their limitations, as well as settings used to manipulate the appearance of heatmaps. It also discusses when heatmaps should and should not be used. Proposed guidelines for using heatmaps correctly conclude this work.
2 Types of Attention Heatmaps Heatmaps are often shown with little, if any, description of what it is they are representing. The assumption is that they are showing “attention” or “eye movements” but knowing that is certainly not enough to be able to truly understand a heatmap. There are different aspects of eye movements that heatmaps can represent. Examples include fixation count, absolute or relative gaze duration, and percentage/proportion of participants who fixated on each area of the stimulus. Choosing the right heatmap to present depends on the study objectives and the eye movement measures that address these objectives. For example, if search efficiency was of interest to the researchers, one of the measures collected and analyzed might be the number of fixations prior to acquiring the target [1]. Therefore, assuming that the analysis would benefit from data visualization, a fixation count heatmap should be presented. A fixation count heatmap would also be appropriate if the study goal was to determine the amount of interest generated by various elements of the stimulus during a free-view task (i.e., task with no specific task instructions) [1]. However, if noticeability of a particular element was of interest, the percentage of participants who fixated on the element could be used as a measure (in addition to, for example, time to first fixation), which would warrant a participant percentage heatmap. Because each heatmap type has different limitations that impact its interpretation, it is not only important to be aware of these limitations, but also to know the types of all heatmaps included in papers, reports, and presentations. 2.1 Fixation Count Heatmap A visual fixation can be loosely defined as a relatively stationary eye position focused on a particular location of the stimulus (a more precise definition is discussed in section 3.1). Fixations are important events that provide insight into human cognition because during each fixation we extract visual information that we process [2]. A fixation count heatmap (see Fig. 1) shows the accumulated number of fixations across participants. Each fixation made by each participant adds a value to the color map at the location of the fixation [3]. This value is the same for each fixation regardless of its duration, so a 100 ms fixation is represented in the same way as a 900 ms fixation. Thus, when looking at a fixation count heatmap, we cannot assume that areas of the same color received similar total gaze time.
32
A. (Aga) Bojko
Fig. 1. Fixation count heatmap (n = 13; 0 – 12 s; free-view task)
Fixation count heatmaps can also be biased towards individuals who show high interest in elements that others do not. For example, two elements can be the same color but one attracted ten fixations from one participant only, while the other attracted attention of ten participants, one fixation from each. Therefore, we cannot assume that areas that appear similar in terms of “heat” are equivalent in terms of the number of participants who looked at them. Another limitation of fixation count heatmaps is the fact that they can be skewed towards individuals who had a longer exposure to the stimulus and thus an opportunity to produce more fixations. For example, if participant A spent twice as much time on the stimulus than participant B, participant A’s data would impact the heatmap twice as much as participant B’s data. This is something to always keep in mind when viewing heatmaps created based on unequal exposure times. 2.2 Absolute Gaze Duration Heatmap An absolute gaze duration heatmap (see Fig. 2) shows the accumulated time participants spent looking at the different areas of the stimulus. Each fixation made by each participant adds a value to the color map that is proportional to its duration [3]. For example, a 900 ms fixation will be nine times higher in color value than a 100 ms fixation. Because fixation duration is an indicator of cognitive processing [4], a heatmap that is scaled by fixation duration not only shows which areas were attended to but also represents the level of cognitive processing that the areas required.
Fig. 2. Absolute gaze duration heatmap (n = 13; 0 – 12 s; free-view task)
Informative or Misleading? Heatmaps Deconstructed
33
An absolute gaze duration heatmap can be misleading because it displays different phenomena in the exact same way. For example, this type of heatmap will make one 900 ms fixation look the same as nine 100 ms fixations. A 900 ms fixation on an element indicates that one person looked at it for a while, while nine 100 ms fixations could mean, for example, that one person made nine brief fixations to the element or nine people made one brief fixation each. In addition, similar to the fixation count heatmaps, absolute gaze duration heatmaps can be biased towards individuals who spent more time looking at the stimulus. To eliminate any bias due to unequal exposure times, the gaze duration data can be normalized to create relative gaze duration heatmaps. 2.3 Relative Gaze Duration Heatmap A relative gaze duration heatmap shows the accumulated time each participant spent fixating at the different areas of the stimulus relative to the total time the participant spent looking at the stimulus [3]. In other words, if participant A spent 6 seconds on a web page including 2 seconds on the navigation and participant B spent 60 seconds on the same web page including 20 seconds on the navigation, this type of heatmap will make their data, as it relates to the navigation, the same weight. Similar to the absolute gaze duration heatmap, this heatmap will also show a high gaze time by an individual (proportional to his/her total viewing time) the same way as several short gaze times by a number of individuals (proportional to their total viewing times). If the exposure time is equal across participants (e.g., all participants saw the page for 12 s), which is the case in the examples presented in this paper, the relative gaze duration heatmap and the absolute duration heatmap will be identical. 2.4 Participant Percentage Heatmap A participant percentage heatmap (see Fig. 3) shows the percentage of participants who fixated on the different areas of the stimulus. Each participant who looked at any given location adds a value to the color map. This value is the same for each participant regardless of the number of fixations he or she made or the fixation durations. Thus, an area that was briefly fixated once by each of the participants will be presented in the same color as an area that was fixated multiple times by each of the participants and the fixations were much longer.
Fig. 3. Participant percentage heatmap (n = 13; 0 – 12 s; free-view task)
34
A. (Aga) Bojko
3 Display Settings for Creating Heatmaps The appearance of heatmaps can be modified by manipulating various display settings. While adjusting the settings cannot change the relative distribution of attention, certain areas can be made to appear “hotter” or “colder.” This can be done, for example, by changing the fixation criteria used for the analysis, showing raw data instead of fixation data, changing the upper threshold definition of the color scale, or modifying the time segment for the presented data. Since heatmap display settings can have a great impact on the appearance of heatmaps, they must be properly selected and communicated to ensure accurate interpretation. 3.1 Changing Fixation Criteria There are several algorithms that can be used to define a fixation. A common algorithm used in commercial eye tracking software is based on duration and dispersion threshold identification. To define a fixation using this algorithm, two parameters need to be specified: minimum fixation duration (e.g., 80 ms) and maximum dispersion threshold (e.g., 0.5 degree of visual angle) [5]. A fixation defined by 80 ms and 0.5° will encompass all consecutive eye movements that occurred within 0.5° from each other for at least 80 ms. Unless noted otherwise, the heatmaps in this paper were created using these settings. Manipulating the duration and dispersion thresholds will change the number of fixations in the data. For example, increasing the minimum fixation duration from 80 ms to 200 ms will decrease the number of fixations because it will exclude all the fixations between 80 ms and 200 ms. Increasing the maximum dispersion threshold from 0.5 degree to 1 degrees of visual angle will also decrease the number of fixations because some of the fixations that are closer together will be combined. Conversely, reducing the minimum fixation duration and maximum dispersion threshold will increase the number of fixations. More fixations in the data will increase the amount of “heat” in the heatmap, as shown in Figure 4.
Fig. 4. Fixation count heatmaps based on the same data (n = 13; 0 – 12 s; free-view task). On the left: minimum fixation duration = 200 ms; on the right: minimum fixation duration = 80 ms.
The lack of explicit fixation definition is an issue that does not pertain specifically to heatmaps but to entire papers and reports. Many user experience studies that analyze fixation data never mention how these fixations were defined. However, this information is very important for two reasons. First, the results of different studies are
Informative or Misleading? Heatmaps Deconstructed
35
not comparable unless it is clear that fixations were defined in the same way. Second, if the definition is not provided, it cannot be verified whether or not the fixation criteria were appropriate for the stimuli used in the study. For example, fixation duration threshold for image viewing should be higher than for reading because image viewing tends to produce longer fixations than reading due to the fact that more information is being processed in a single fixation [6]. The fixation definition used to create heatmaps should match the definition used for data analysis. Changing the fixation duration to obtain a visualization of a particular intensity is not a good practice. 3.2 Displaying Raw Data Instead of Fixation Data One step further from decreasing the fixation duration and dispersion thresholds is presenting raw data instead of fixation data. Raw data consists of meaningful eye movements (raw fixation points) and “noise” – eye movements that have little meaning in most user experience research. The noise includes eye movements that take place during saccades (rapid eye movements between fixations) as well as drifts, tremors, and flicks that occur during fixations [5]. Adding all the noise intensifies the heatmap, increasing the area covered in red (see Fig. 5).
Fig. 5. Fixation count heatmaps based on the same data (n = 13; 0 – 12 s; free-view task). On the left: fixation data; on the right: raw data.
As a general rule, fixation data rather than raw data should be used when creating visualizations unless the stimulus has moving elements and the heatmap has to show smooth pursuit eye movements. We can assume that fixation data was used if the paper or report specifies how the researchers defined a fixation. 3.3 Changing the Definition of the Color Scale Upper Threshold Another way to manipulate the amount of heat in heatmaps is by changing the definition of the upper threshold on the scale, which is usually indicated by the red color. If the requirements for an area to be red are lowered, the amount of red in the heatmap will increase. Lowering of the upper threshold of the scale can be achieved by decreasing the minimum number of fixations in fixation count heatmaps (see Fig. 6) or by decreasing the minimum gaze length in absolute gaze duration heatmaps.
36
A. (Aga) Bojko
There is no set process for choosing the right upper threshold. The rule of thumb is to make sure that the heatmap properly captures the range of values that are of interest to the study. Threshold selection can be compared to setting the maximum value on the Y axis in a graph, where the Y axis indicates values of the independent variable. If the Y axis is too short, the data points that exceed the maximum on the axis are cut off. As a result, we only know that these data points are higher than the maximum but we do not know what they are exactly and how they differ from one another. On the other hand, if the Y axis is too high, the graph data will appear compressed and the differences between the data points will look smaller. Similarly, if a heatmap’s upper threshold is set too low, many areas will be covered in red with no differentiation between the amount of attention each attracted. Conversely, setting a heatmap’s upper threshold too high will limit the range of colors (e.g., constricting it to yellow or orange as the maximum) and no areas will be covered in red. Regardless of what criteria have been selected, they should be explicitly communicated in the figure legend or caption (e.g., “red = 10+ fixations” or “red color indicates areas that accumulated 10 s or more of gaze time”). It is also useful to put this value in context by providing the average number of fixations each participant made on the stimulus or the average time each participant spent looking at it. The heatmaps presented in this paper were created based on data with the average of 42 fixations on the page per participant and consistent exposure time of 12 seconds. If heatmaps are generated for different experimental conditions or participant groups, their upper threshold definition should be identical, so the heatmaps can be compared. If the display settings are not the same, the differences between the heatmaps may be due to factors other than the data itself.
Fig. 6. Fixation count heatmaps based on the same data (n = 13; 0 – 12s; free-view task). On the left: red = 10+ fixations; on the right: red = 3+ fixations.
3.4 Modifying the Time Segment Sometimes it may be appropriate to present data based on a shorter time segment than the total time during which a stimulus was shown to the participants. This will obviously decrease the amount of data, thus reducing the size of red areas in all heatmap types mentioned in this paper except for the relative gaze duration heatmap (see Fig. 7). Therefore, if a heatmap presents data from a time segment that is shorter than the total viewing time, this needs to be specified in the figure legend or caption.
Informative or Misleading? Heatmaps Deconstructed
37
In addition to the time segment, it should also be clear what participants were trying to do when the data presented in the heatmap was being collected. All too often heatmaps are shown without any context of the task. Eye movements are very taskdependent [7] and participants trying to log in to a website will produce very different attention distribution than if they were trying to find a product. Even if there was no specific task, it should still be noted that the data was collected in a free-view situation, which is how the heatmaps included in this paper were obtained.
Fig. 7. Fixation count heatmaps of the same free-view task (n = 13). On the left: data from 0 to 6 seconds; on the right: data from 0 to 12 seconds.
4 Heatmap Usage 4.1 Common Mistakes There are a few common mistakes when it comes to using heatmaps. The biggest of them is a belief that heatmaps are appropriate for just about any user experience study and for any research question. Sometimes creating heatmaps even becomes the objective of the study (e.g., “we just wanted to see where people look”). This popular “let’s-track-and-see-what-happens” approach has a limited value because of its lack of focus (i.e., the study design does not target specific questions) and frequent lack of proper data analysis. Researchers often draw conclusions or make recommendations based on the results of those studies, which is inappropriate. For example, we cannot say that the reason why an element did not get much attention was because of its suboptimal placement or insufficient size unless we have tested other conditions (i.e., alternative placements or sizes) and compared the data using appropriate statistics. Even if different conditions were tested, sometimes conclusions regarding differences between conditions are made just by looking at the heatmaps. However, heatmaps do not lend themselves to any systematic comparison. Without any data analysis, it is impossible to tell if there are real differences between heatmaps, even if the heatmaps appear to be different. 4.2 Proper Usage Examples Heatmaps should be used in a purposeful way and only when they add value. They can serve as illustrations of participants’ viewing behavior and distribution of attention.
38
A. (Aga) Bojko
While they can communicate data, they cannot explain it or help analyze it. Therefore, heatmaps can rarely stand on their own. To maximize their usefulness and reduce ambiguity, heatmaps should accompany a quantitative analysis. One of our studies investigated a new standardized label template for prescription drug labels [8]. The goal was to determine the impact of the template on pharmacists’ drug selection speed and accuracy as compared to the existing label designs. The eye tracking measures included the number of fixations prior to target selection as an indicator of search efficiency, average fixation duration as a measure of information processing difficulty, and pupil diameter as a measure of cognitive workload. The results were presented in the form of statistical analyses but no heatmaps were included in the report. The study was a quantitative assessment of the effectiveness of the new labels, and heatmaps showing attention distribution were simply of no value. In another study, we evaluated a new homepage design for a professional organization against the original homepage [9]. Our objective was to identify which design was better and why based on a series of tasks during which participants attempted to locate the correct entry point on the homepage. Measures of search efficiency such as the number of fixations and the number of eye visits to the target prior to target selection were analyzed. The analysis was supplemented with heatmaps to show the distribution of attention on the page and to help account for any inefficiencies that occurred. For example, several tasks were more efficient using the new design which had a more centralized navigation. The heatmaps of the original design showed scattered fixations covering multiple navigation areas, while heatmaps of the new design revealed fixations focused mostly around the targets.
5 Guidelines for Using Heatmaps Even though heatmaps are very compelling and seemingly easy to understand, they should be used with caution and according to the following guidelines, summarized based on the discussion from the previous sections of this paper: A. B. C. D.
E. F. G. H.
I. J.
Generate heatmaps only if they add value to the research. Use heatmaps for data visualization instead of data analysis. Use heatmaps to support quantitative analysis rather than on their own. Understand the different heatmap types and only use the ones that represent measures which address your study objectives (e.g., when analyzing gaze time, use a gaze duration heatmap). Specify the type of data the heatmap is representing (e.g., fixation count or absolute fixation duration). Know the limitations of each heatmap type to avoid incorrect interpretation. When creating heatmaps, use fixation data rather than raw data. Provide fixation definition and keep it consistent for analyses and visualizations within a study (e.g., min fixation duration = 100 ms and max dispersion threshold = 0.5°). Provide the definition for the upper threshold of the heatmap color scale (e.g., red = 10+ fixations). Put the upper threshold value in context (e.g., average number of fixations on the stimulus per participant).
Informative or Misleading? Heatmaps Deconstructed
39
K. Specify the time segment based on which the heatmap was created (e.g., the first 10 seconds of exposure). L. Provide task context for each heatmap (e.g., data obtained from participants during the checkout task). M. Use the same heatmap settings (e.g., upper threshold and time segment) for conditions that you are comparing. N. If a paper or report does not provide important information about its heatmaps (e.g., type, fixation definition, upper threshold definition, time segment), ask the authors for clarification before making any assumptions.
6 Conclusion Blinded by the attractiveness and apparent intuitiveness of heatmaps, we often do not realize how much information in addition to the visualization itself is necessary to fully understand a heatmap and properly interpret the data it represents. In other words, the biggest danger involved in creating and reading heatmaps is that we are often unaware of what we do not know, and thus we do not look or ask for the missing information. This paper has exposed some of these gaps in our meta-knowledge in the hope to encourage more critical thinking about the usage of heatmaps.
References 1. Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises. In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, pp. 573–605. Elsevier Science, Amsterdam (2003) 2. Liversedge, S.P., Findlay, J.M.: Saccadic Eye Movements and Cognition. Trends in Cognitive Sciences 4, 6–14 (2000) 3. Tobii.: Tobii Studio 1.X User Manual (2008) 4. Duchowski, A.: Eye Tracking Methodology: Theory and Practice. Springer, Heidelberg (2003) 5. Salvucci, D.D., Goldberg, J.H.: Identifying Fixations and Saccades in Eye-Tracking Protocols. In: Proceedings of Eye Tracking Research and Applications Symposium (2000) 6. Castelhano, M.S., Rayner, K.: Eye Movements During Reading, Visual Search, Scene Perception: An Overview. In: Rayner, K., Shem, D., Bai, X., Yan, G. (eds.) Cognitive and Cultural Influences on Eye Movements, pp. 3–33. Psychology Press (2008) 7. Yarbus, A.L.: Eye Movements and Vision. Plenum Press (1967) 8. Bojko, A.: Measuring the Effects of Drug Label Design and Similarity on Pharmacists’ Performance. In: Tullis, T., Albert, W. (eds.) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, pp. 271–280. Morgan Kaufmann, San Francisco (2008) 9. Bojko, A.: Using Eye Tracking to Compare Web Page Designs: A Case Study. Journal of Usability Studies 1, 112–120 (2006)
Toward EEG Sensing of Imagined Speech Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan Department of Cognitive Sciences, UC Irvine, SSPB 3219, Irvine, CA 92697-5100 {mdzmura,sdeng,tlappas,sthorpe,r.srinivasan}@uci.edu
Abstract. Might EEG measured while one imagines words or sentences provide enough information for one to identify what is being thought? Analysis of EEG data from an experiment in which two syllables are spoken in imagination in one of three rhythms shows that information is present in EEG alpha, beta and theta bands. Envelopes are used to compute filters matched to a particular experimental condition; the filters' action on data from a particular trial lets one determine the experimental condition used for that trial with appreciably greater-than-chance performance. Informative spectral features within bands lead us to current work with EEG spectrograms. Keywords: EEG, imagined speech, covert speech, classification.
1 Introduction Dewan reported in 1967 [1] that he and several others could transmit letters of the alphabet using EEG recording; their ability to control voluntarily the amplitude of their alpha waves let them send letters using Morse code. Farwell and Donchin [2] used a second method for transmitting linguistic information by EEG; the P300 response to targets let them determine which of a sequence of displayed letters a person had in mind. Finally, Suppes and colleagues [3,4] reported that they were able to classify EEG responses to heard sentences and that there was information available through EEG concerning imagined speech. The work reported here verifies this latter result and develops it further, with the aim of using EEG brain waves to communicate imagined speech. Classification experiments show that brain-wave signatures of imagined speech lets one distinguish linguistic content, with varying degree of success. These signatures include differences in alpha-, beta- and theta-band activity. Our near-term goal is to use such signatures to design filters that let one distinguish linguistic elements in real time. One application is in further experiments which provide feedback to the thinker as to how recognizable a particular thought is, with the aim of training the thinker to produce brain waves which are more discernible. The focus within is on the results of a single experiment in which subjects produce in their imagination one of two syllables in one of three different rhythms. Analysis of the data using envelopes within alpha, beta and theta frequency bands, spectral features within single bands, and spectrograms in current work show that EEG provides considerable information on imagined speech in this experimental setting. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 40–48, 2009. © Springer-Verlag Berlin Heidelberg 2009
Toward EEG Sensing of Imagined Speech
41
2 Methods Four subjects participated in an experiment with six conditions determined factorially through combination of two syllables and three rhythms (see Figure 1). A single experimental session comprised 20 trials for each of the six conditions; conditions were presented in block-randomized order. Each subject participated in six such sessions for a total of 120 trials per condition. EEG was recorded using a 128 Channel Sensor Net (Electrical Geodesics) in combination with an amplifier and acquisition software (Advanced Neuro Technology). The EEG was sampled at 1024Hz and on-line average referenced. Subjects were instructed to keep their eyes open in the dimly-lit recording room and to avoid eye and other movement during the six seconds, following the cue, during which speech was imagined without any vocalization whatsoever.
Fig. 1. Timelines for the six conditions, labeled at left, in the imagined speech experiment. Durations are indicated in eighths of a second. The syllable /ba/ was imagined in one of three different rhythms in conditions 1-3; the syllable /ku/ was imagined in conditions 4-6. The condition for each trial was cued during an initial period of duration 4.5 sec (12/8 sec = 1.5 sec; 3 x 1.5 sec = 4.5 sec). During this initial period, subjects heard through Stax electrostatic earphones either a spoken "ba" or a spoken "ku" followed by a train of clicks (arrows) indicating the rhythm to be reproduced. Subjects were instructed to speak in their imagination the cued syllable, illustrated for condition 1 by {ba}, using the cued rhythm and tempo. Desired imagined syllable onset times reproduce the cued rhythm and are indicated by asterisks.
3 Analysis and Results Our aims were to classify single trials offline according to condition and discern condition signatures needed to create online filters. EEG waveform envelopes provide encouraging results when used to classify trials according to experimental condition.
42
M. D’Zmura et al.
These envelopes were computed for each electrode in the theta (3-8Hz), alpha (813Hz) and beta (13-18Hz) frequency bands and used to construct matched filters. These filters work well to classify trials according to condition, yet further results with spectra lead us to explore a finer-grained analysis using spectrograms. 3.1 Matched-Filter Classification Using Envelopes We used waveform envelopes to compute matched filters, one for each of the six conditions (see Figure 2). The inner product of each matched filter with a particular trial's envelope provides six numbers. The matched filter which gives rise to the largest inner product is the best match, and the condition to which the matched filter corresponds is declared the best guess as to the experimental condition for that trial. Offline preprocessing steps include • segment the EEG data to provide time-varying waveforms for each condition, electrode and trial; • remove from further consideration 18 electrodes most sensitive to electromyographic artifact: those with the lowest positions about the head and spanning locations close to the eyes, low on the temple, at or below the ear, and at or below the external occipital protuberance; • remove the mean and linear trend from each segmented waveform; • low-pass filter the detrended, segmented, waveforms to remove 60Hz line noise; • use thresholds to identify and remove from further consideration filtered waveforms likely contaminated with electromyographic artifact. Alpha-, beta- and theta-band activity were computed for each electrode and trial using band-pass elliptic filters. These band-pass filtered waveforms (we[t] and bottom right plot in Fig. 2) were Hilbert-transformed to provide corresponding envelopes (ve[t] and middle right plot). The envelopes serve as input to the matched-filter classification and as data used to construct the matched filters. An electrode's average envelopes, found by averaging across trials for each of the six conditions, serve as matched filters with one further step: for each electrode, pseudoinvert the six conditions' average envelopes to provide six filters (Fec[t] and top right plot). The inversion of the average envelopes provides filters which return an inner product of one for the corresponding condition's average envelope and an inner product of zero for all other condition's average envelopes. Information may be integrated across electrodes by summing the measures pe,c across electrodes and determining the maximum of the resulting six numbers. Information may be integrated across bands by weighing individual response measures for the three bands by band reliability estimates or by a voting procedure. Classification results shown in Table 1 show that the beta band (13-18Hz) is, for all but one subject, the most informative frequency band; theta (3-8Hz) and alpha (813Hz) are comparable for all subjects but S1. The per-condition classification performance is shown for subject S2 in Fig. 3. Rows refer to a trial's actual condition, while columns refer to the condition of the matched filter for which the maximal response was obtained. Darker values indicate higher percentages of trials. These dark values describe diagonals, which indicates a match between a trial's actual condition and the condition with matched filter providing the greatest response.
Toward EEG Sensing of Imagined Speech
43
Fig. 2. Preprocessed and band-pass-filtered waveform we[t], recorded by electrode e, is Hilberttransformed to provide envelope ve[t]. Shown at bottom right is the preprocessed waveform from an electrode with mid-parietal location from subject S1 for a trial during which {ba} is spoken in imagination at times 1.5, 3.0 and 4.5 sec (condition 1). Shown at middle right is the corresponding envelope. The inner product <, > of this envelope with each matched filter Fe,c[t], one filter per condition per electrode, provides six numbers pe,c. These six numbers measure how well the particular trial's envelope matches the filter for each condition. The maximum of these six numbers is used to determine the most likely condition c˜ e . Shown at top right is a matched filter for condition c = 1.
The distributions of classification performance across the scalp are similar across the four subjects; those for subject S4 are shown in Figure 4. The most informative electrodes lie largely near the top of the head (vertex) where electromyographic artifacts have their least influence. These distributions of information differ significantly from the distributions of envelope amplitude, which follow well-known patterns for alpha activity (parietal) and theta activity (frontal).
44
M. D’Zmura et al.
Table 1. Classification performance using matched filters for envelopes in three frequency bands. The fraction of correctly-classified trials (720 trials per subject, identified in the left column) is indicated. The chance performance level in this classification among six conditions is 1/6 (0.17).
S1 S2 S3 S4
alpha 0.38 0.63 0.44 0.64
beta 0.80 0.87 0.68 0.62
theta 0.63 0.59 0.46 0.59
Fig. 3. Classification matrices for subject S2. Values along the diagonals indicate correct classification, while non-white values off the diagonal indicate errors. The black gray-value in the middle panel (beta-band) for actual condition 1 and matched filter condition 1 (top left square) represents 91% of the trials; lighter shades indicate smaller values (through white indicating 0%). Perfect performance would be indicated by all off-diagonal entries set to white.
Fig. 4. Distributions of classification performance across the scalp for alpha (α), beta (β) and theta (θ) bands for subject S4. Darker values indicate the positions of electrodes providing better classification performance.
3.2 Spectral Features
Analysis of trials’ power spectral densities shows that differences in power within single frequency bands can provide information concerning trial condition. Spectral
Toward EEG Sensing of Imagined Speech
45
differences within a single band are completely invisible to the previous analysis, which grouped together activity at all frequencies within a single band. Data for subjects S1 and S2 show that, for condition 3, there is a peak in the power spectrum in the range 5-6Hz, relative to baseline power measured at 3.5-4.5Hz. This peak is largely absent for condition 1. This difference between the average power spectra for conditions 3 and 1 is localized to electrodes with front central locations on the scalp (see Fig. 5, leftmost panels). This spectral difference alone, within the theta band, provides ~75% correct two-way classification between these two conditions. The peak and the localized spatial distribution do not exist for subjects S3 and S4 (see Fig. 5, rightmost panels).
Fig. 5. Difference between condition 3 and condition 1 theta sub-band spectral power localized primarily to electrodes with front, center locations for subjects S1 and S2; this pattern is not evident for S3 and S4
3.3 Spectrographic Analysis
The spectral data suggest several things. The first is that one would do well to represent EEG data spectrographically: record power as a fine-grained function of both frequency and time. The second and third are more cautionary. There is a tremendous amount of trial-by-trial variability in EEG recordings. Adding channels, like further frequency bands of narrow range as in spectrograms, may have the effect of masking signals more effectively. Furthermore, fine subdivision of the frequency domain may reveal individual differences (as in Fig. 5), so that averages across subjects become less meaningful. These caveats notwithstanding, our current work focuses on spectrographic means of identifying imagined speech.
4 Discussion The syllables /ba/ and /ku/ have little semantic content in and of themselves, so that differences in EEG records underlying classification performance are unlikely to reflect semantic contributions to imagined speech production. Furthermore, each experimental trial starts with a period during which the cue to the condition is presented; cue recognition and response planning occur during the cue period and are likely absent during the following period in which EEG recording is made of imagined speech. The EEG recordings analyzed here concern most likely immediate aspects of imagined speech production based on readout from working memory.
46
M. D’Zmura et al.
EEG electrodes record not only signals from brain cortex but also electromyographic activity. Common sources of the latter are muscles responsible for eye and head movements and other muscles near recording electrodes, like those in the temples and neck. Electrodes atop or close to such muscles are most sensitive to electromyographic activity. The most flagrant such offenders were excluded from analysis in the work reported here. Furthermore, recordings for which EEG signals (detrended and filtered to excluded 60Hz line noise) exceeded 30 μV in absolute value were excluded from consideration. This threshold eliminated many (but not all) recordings contaminated with artifact. The most informative electrodes are those positioned on the scalp near the top of the head (Fig. 3), and this suggests that the information is due more to cortical than electromyographic activity. Subjects were instructed to think the imagined speech without any vocal or subvocal activity: without moving any muscles involved in producing overt speech. This contrasts with what is the most successful method yet for recognizing speech produced without any acoustic signal: surface electromyographic (SEMG) recordings from face and throat muscles. Substantial progress has been made in automatic speech recognition from surface EMG recordings of vocal tract articulators made during both spoken speech [5-10] and subvocalized speech [11-17]. Further non-acoustic methods for extracting imagined speech information potentially include magnetoencephalography (MEG) and invasive methods, like electrocorticography (ECoG), which involve implanted electrodes. While there are no published positive results yet for classification of imagined speech using MEG, its high temporal resolution and moderate spatial resolution (comparable to high-density EEG) suggest its value, as does success in classifying heard speech [18-21]. MEG is most sensitive to cortical signals from surfaces perpendicular to the scalp (in sulci) while EEG is most sensitive to signals from parallel surfaces (gyri) [22], so that the two methods are complementary. While the use of electrodes implanted in the brain is limited to small clinical populations for which these electrodes are indicated in combination with neurosurgery, it can provide very useful information concerning imagined speech [23]. Functional magnetic resonance imaging (fMRI) studies have contributed significantly to our neuroscientific knowledge of language perception and production [24-27]. Yet the poor temporal resolution of fMRI and of functional near-infrared imaging (fNIR) make these unlikely candidates for speech recognition. EEG is a non-invasive, non-injurious method for probing cortical activity which has a high temporal resolution, moderate spatial resolution (~2cm), a relatively low cost and which is increasingly portable. We feel that its greatest contribution as a braincomputer interface is likely to be found in circumstances where a human subject is trained to produce activity discernible by EEG [28]. Such training can reduce the variability associated with imagined speech production and also help one emphasize aspects of imagined speech which are most easily discerned by EEG. Our next step is to close the loop by providing real-time feedback concerning imagined speech signals.
Acknowledgements We thank David Poeppel for suggesting the value of an experiment like this. This work was supported by ARO 54228-LS-MUR.
Toward EEG Sensing of Imagined Speech
47
References 1. Dewan, E.M.: Occipital alpha rhythm eye position and lens accommodation. Nature 214, 975–977 (1967) 2. Farwell, L.A., Donchin, D.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogy and Clinical Neurophysiology 70, 510–523 (1988) 3. Suppes, P., Lu, Z.-L., Han, B.: Brain wave recognition of words. Proceedings of the National Academy of Science USA 94, 14965–14969 (1997) 4. Suppes, P., Han, B., Lu, Z.-L.: Brain wave recognition of sentences. Proceedings of the National Academy of Science USA 95, 15861–15866 (1998) 5. Morse, M.S., O’Brien, E.M.: Research summary of a scheme to ascertain the availability of speech information in the myoelectric signals of neck and head muscles using surface electrodes. Computers in Biology and Medicine 16, 399–410 (1986) 6. Chan, A.D.C., Englehart, K., Hudgins, B., Lovely, D.F.: Hidden Markov model classification of myoelectric signals in speech. IEEE Engineering in Medicine and Biology 21, 143– 146 (2002a) 7. Chan, A.D.C., Englehart, K., Hudgins, B., Lovely, D.F.: Multiexpert automatic speech recognition using acoustic and myoelectric signals. IEEE Transactions on Biomedical Engineering 53, 676–685 (2002b) 8. Bu, N., Tsuji, T., Arita, J., Ohga, M.: Phoneme classification for speech synthesizer using differential EMG signals between muscles. In: Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pp. 5962–5966 (2005) 9. Jou, S.-C., Maier-Hein, L., Schultz, T., Waibel, A.: Articulatory feature classification using surface electromyography. In: Acoustics, Speech and Signal Processing, ICASSP 2006 Proceedings, pp. I–605–I–608 (2006) 10. Jou, S.-C., Schultz, T., Walliczek, M., Kraft, F., Waibel, A.: Towards continuous speech recognition using surface electromyography. In: Interspeech 2006 – ICSLP, pp. 573–576 (2006) 11. Jorgensen, C., Lee, D.D., Agabon, S.: Sub-auditory speech recognition based on EMG signals. In: Proceedings of the International Joint Conference on Neural Networks, Portland Oregon (July 2003) 12. Betts, B.J., Jorgensen, C.: Small vocabulary recognition using surface electromyography in an acoustically harsh environment. NASA/TM-2005-213471 (2005) 13. Maier-Hein, L.: Speech recognition using surface electromyography. Diplomarbeit, Universität Karlsruhe (2005) 14. Maier-Hein, L., Metze, F., Schultz, T., Waibel, A.: Session independent non-audible speech recognition using surface electromyography. In: 2005 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 331–336 (2005) 15. Jorgensen, C., Binsted, K.: Web browser control using EMG based sub vocal speech recognition. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS 2005), pp. 294–301 (2005) 16. Binsted, K., Jorgensen, C.: Sub-auditory speech recognition. In: HICSS (2006) 17. Walliczek, M., Kraft, F., Jou, S.-C., Schultz, T., Waibel, A.: Sub-word unit based nonaudible speech recognition using surface electromyography. In: Interspeech 2006 - ICSLP, pp. 1487–1490 (2006) 18. Numminen, J., Curio, G.: Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex. Neuroscience Letters 272(1), 29–32 (1999)
48
M. D’Zmura et al.
19. Ahissar, E., Nagarajan, S., Ahissar, M., Protopapas, A., Mahncke, H., Merzenich, M.M.: Speech comprehension is correlated with temporal response patterns recorded from auditory cortex. Proceedings of the National Academy of Sciences 98, 13367–13372 (2001) 20. Houde, J.F., Nagarajan, S.S., Sekihara, K., Merzenich, M.M.: Modulation of the auditory cortex during speech: an MEG study. Journal of Cognitive Neuroscience 15, 1125–1138 (2002) 21. Luo, H., Poeppel, D.: Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54, 1001–1010 (2007) 22. Nunez, P.L., Srinivasan, R.: Electric Fields of the Brain: The Neurophysics of EEG, 2nd edn. Oxford University Press, New York (2006) 23. Barras, C.: Brain implant helps stroke victim speak again. New Scientist (July 2008) 24. Indefrey, P., Levelt, W.J.M.: The spatial and temporal signatures of word production components. Cognition 92, 101–144 (2004) 25. Hickok, G., Poeppel, D.: Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92, 67–99 (2004) 26. Hickok, G., Poeppel, D.: The cortical organization of speech processing. Nature Reviews Neuroscience 8, 393–402 (2007) 27. Poeppel, D., Idsardi, W.M., van Wassenhove, V.: Speech perception at the interface of neurobiology and linguistics. Philosophical Transactions of the Royal Society B 363, 1071–1086 (2007) 28. Fetz, E.: Volitional control of neural activity: implications for brain-computer interfaces. Journal of Physiology 579, 571–579 (2007)
Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User Ying Gao, Armando Barreto, and Malek Adjouadi Department of Electrical and Computer Engineering Florida International University Miami, FL 33174 USA {ygao002,barretoa,adjouadi}@fiu.edu
Abstract. The pupil diameter (PD) has been found to respond to cognitive and emotional processes. However, the pupillary light reflex (PLR), is known to be the dominant factor in determining pupil size. In this paper, we attempt to minimize the PLR-driven component in the measured PD signal, through an ∞ Adaptive Interference Canceller (AIC), with the H time-varying (HITV) adaptive algorithm, so that the output of the AIC, the Modified Pupil Diameter (MPD), can be used as an indication of the pupillary affective response (PAR) after some post-processing. The results of this study confirm that the AIC with the HITV adaptive algorithm is able to minimize the PD changes caused by PLR to an acceptable level, to facilitate the affective assessment of a computer user through the resulting MPD signal. Keywords: Pupil diameter (PD), Pupillary light reflex (PLR), Pupillary affec∞ tive response (PAR), Adaptive Interference Canceller (AIC), H time-varying (HITV) adaptive algorithm.
1 Introduction Affective Computing, defined as “computing that relates to, arises from, or deliberately influences emotions” by Picard in 1997 [1] requires computers to have the ability to remain aware of their users’ affective states. Therefore, the affective assessment of the computer user has been considered as one of the key challenges to overcome for improvement of the relationship between human and computers. This challenge has been addressed, among other approaches, through the processing of a variety of physiological signals from the computer users, such as the Galvanic Skin Response (GSR), the Blood Volume Pulse (BVP), etc. We are currently studying the possibility of analyzing pupil diameter (PD) variations in the computer user for the detection of affective changes. The human pupil is the variable-size opening in the iris of the eye, through which light passes to the retina. The size of pupil is known to be controlled by two opposing sets of muscles in the iris, the sphincter and dilator pupillae, which are governed by the sympathetic and parasympathetic divisions of the Autonomic Nervous System (ANS) [2]. Traditionally, light intensity has been considered as the major determinant of the pupillary response, through the Pupillary Light Reflex (PLR). However, other J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 49–58, 2009. © Springer-Verlag Berlin Heidelberg 2009
50
Y. Gao, A. Barreto, and M. Adjouadi
psychological factors controlling pupil size, such as emotional processes have recently been investigated. For example, in 2003, Partala and Surakka found, using auditory emotional stimulation, that the pupil size variation can also be seen as an indication of affective processing during human-computer interaction [3]. Therefore, in this paper, we focus our study on monitoring and processing the pupil diameter (PD) signal for the affective assessment of a computer user. To achieve the required separation of the PLR-driven component from PD changes due to Pupillary Affective Responses (PAR) we propose an Adaptive Interference Canceller (AIC), which is known to be able to remove an unwanted interference component z(k) that pollutes a measured signal s(k) using an independent measurement of the interference n(k) [4] (See Figure 1). The core portion of this AIC system is an Adaptive Transversal Filter (ATF), which performs the adaptive algorithm to implement the function of interference canceling. To ensure a successful noise removal from the AIC, it is important to select an adaptive algorithm that possesses good robustness properties. The H∞ adaptive algorithm, introduced in robust control theory, is an attempt at addressing this need, with features that are useful in safeguarding against the worst-case of model uncertainties and makes no assumption on the (statistical) nature of the signals [5].
s (k )
d (k )
d (k )
z (k )
n( k ) d (k )
n( k )
r ( k ) = n( k )
r (k )
∧
∧
y (k ) = z (k )
Σ
s ( k ) = e( k )
u (k ) = 0
Fig. 1. Adaptive Interference Canceller (AIC) Block Diagram
Thus, we propose to use an H∞ –based adaptive technique, namely the H∞ timevarying (HITV) algorithm, in the Adaptive Interference Canceller for the removal of the PLR-driven component from the pupil diameter variation. Our intent is to use the output of the AIC, the Modified Pupil Diameter (MPD), further refined by additional processing (median evaluation of a sliding window), to indicate the occurrence of affective changes (e.g., onset of stress) in the computer user.
2 Methods 2.1 AIC Overview A basic AIC block diagram is illustrated in Figure 1, above. The signal of interest, polluted with an uncorrelated noise signal, is transmitted over the top channel, and constitutes the primary input to the AIC. The bottom channel receives a signal, which
Monitoring and Processing of the PD Signal
51
is uncorrelated with the signal of interest, but correlated with the interference, constituting the reference input. The output of AIC is expected to provide a signal where the correlated interference has been removed. As described in the previous paragraph, the key equations describing the AIC are: Primary Input: Reference Input:
Output:
d (k ) = s (k ) + z (k )
(1)
r ( k ) = n( k ) + u ( k )
(2)
e(k ) = d (k ) − y (k ) = d (k ) − zˆ (k ) = sˆ(k )
(3)
where, 9 9 9 9
s(k) is the signal of interest (PD- driven by PAR); z(k) is the interference in the primary sensor (PD- driven by PLR); n(k) is the actual source of the interference (Illumination changes); u(k) is the measurement noise. (In this study, u(k) is assumed to be zero);
In the AIC system, the core element is an adaptive transversal filter (ATF), where the reference signal r(k) is processed to produce an output y (k ) = zˆ (k ) that is an approximation of z(k). The state space model of the ATF is given by [5]: w(k + 1) = w(k ) + Δw(k )
(4)
d (k ) = r (k ) w(k ) + v(k )
(5)
z (k ) = r (k ) w(k ) + υ (k )
(6)
T
T
v(k ) = s(k ) + υ (k )
(7)
In these equations, 9 9 9 9 9 9
w(k) = system state vector, is the ATF coefficient vector of size m × 1 (m is the order of ATF); d(k) = measurement sequence, is the observed pupil diameter(PD) signal. z(k) = sequence to be estimated Δw(k ) = process noise vector, represents the time variation of the ATF weights w(k). v(k) = measurement noise vector, includes s(k) (PD- driven by affective changes) and model uncertainties υ (k ) .
r (k ) = [rk , r( k −1) ,......, r( k − m +1) ]T is the interference vector of size m ×1
As shown in Figure 1, for this study we define the primary input of the AIC as the recorded pupil diameter signal, which is composed by the signal of interest s(k) (PD- driven by PAR) and the interference z(k) (PD- driven by PLR). Although, the reference input comprises both the actual source of the interference n(k) (actual illumination changes) and the measurement noise u(k), under the assumption that the measurement noise is negligible, an independent measurement of illumination in the neighborhood of the eye of the subject (IL) is used as the reference input. We expect
52
Y. Gao, A. Barreto, and M. Adjouadi
the adaptive transversal filter (ATF) to emulate the transformation of the illumination variations to pupil diameter changes, which would convert the noise n(k) into a closeenough replication of the PLR-driven components of PD (the output y(k)). Therefore, the error, e(k) (Modified Pupil Diameter), would be the estimation of the desired signal s(k), i.e., the PD variations due exclusively to affective changes (PAR). 2.2 H∞ Time-Varying Adaptive Algorithm
In this study, the adaptive algorithm we applied to the Adaptive Interference Canceller system is the H∞ time-varying (HITV) adaptive algorithm, which aims to remove the noise from the recorded signal by adaptively adjusting the impulse response of the ATF. The robustness of this HITV algorithm is derived from its minimization of the maximum energy gain from the disturbances to the estimation errors with the following solutions [6]: P% −1 (k ) = P −1 (k ) − ε g−2 r(k )rT (k )
(8)
~
g (k ) =
P(k )r(k ) ~
1 + rT (k ) P(k )r(k)
(9)
wˆ (k + 1) = wˆ (k ) + g (k )( y (k ) − rT (k ) wˆ (k ))
(10)
P(k + 1) = [ P -1 (k ) + (1- ε g-2 )r (k )r T (k )]-1 + γ 0
(11)
P (0) = η I = Π 0 , ϒ 0 = ρ I
(12)
Here, g ( k ) is the gain factor; ε g , η and ρ are positive constants. Note that ρ reflects a priori knowledge of how rapidly the state vector w(k ) varies with time, and η reflects the a priori knowledge of how reliable the initial estimate available for the state vector w(0) is.
3 Experiments 3.1 Subjects
We have collected data from twenty-two volunteer students who have completed the protocol described below. All the subjects reported to have normal color vision and had experience using computers. This paper presents results achieved through the processing methods proposed on a subset of the subject pool recorded to date. 3.2 Task
In order to observe the response of pupil diameter changes to affective stimuli, the “Stroop Color-Word Interference Test”, implemented as a flash program [7], was used to elicit mild mental stress in the participating subjects during the experiment.
Monitoring and Processing of the PD Signal
Fig. 2. Sample of Stroop Test Interface
53
Fig. 3. Stimuli schedule of the Stroop Test
In the test, a word presented to the subject designates a color that may or not match the font of the word. The subjects are instructed automatically by the program to select (out of 5 possible choices) the screen button that indicates the font color of the word presented, by clicking on it (An example, showing the appearance of the test interface is shown in Figure 2). The stimuli schedule of the test in the experiment is shown in the Figure 3. The complete protocol is composed of three consecutive sections. In each section, there were four segments. 9 9 9 9
‘IS’ – the Introductory Segment to let the subject get used to the task environment; ’C’ – the Congruent segment of the Stroop Test, in which the subject was asked to click the on-screen button naming the font color of a word that correctly spelled the font color being displayed; ‘IC’ – the Incongruent segment of the Stroop Test in which the subject was asked to click the on-screen button naming the font color of a word that spelled the name of a different color; ‘RS’ – is a Resting Segment to let the subject relax for some time.
The incongruent Stroop segments (IC) were expected to elicit mild mental stress in the subject, according to previous research found in the psychophysiological literature [8]. In contrast, the congruent Stroop segments (C) were expected to allow the subject to continue in a relaxed state. The binary numbers shown in Figure 3 represent the demultiplexed output of the stimulus generator, which is used to insert the corresponding values (1, 2, 3) in the event channel of the PD data file, and the corresponding time marks in the illumination measurement data, recorded simultaneously. 3.3 Instruments
In this study, a desk-mounted eye tracking system (TOBII T60) was used to measure the pupil diameter signals from both eyes of the subjects at 60 samples/sec. The average ((L+R)/2) was recorded as the “Measured PD” signal, which corresponds to d(k), in Figure 1. Simultaneously, the illumination intensity level present in the area around the eyes of the subjects was recorded by a system built for that purpose. This system is composed by a BS500B0F photo-diode (Sharp), placed on the forehead of the subject and connected to an amplification circuit to provide an analog output voltage that is proportional (~0.0043 v/Lux) to the illumination intensity level [9]. The “Measured IL” signal, r(k), shown in Figure 1 was finally provided by sampling the analog
54
Y. Gao, A. Barreto, and M. Adjouadi
output of the luminance meter at the frequency of 360 Hz with a Data Acquisition (DAQ) System (PCI-DAS6023 board from Measurement Computing Co.) 3.4 Procedure
In our experiments, participants were asked to remain seated in front of the TOBII screen, interacting with the “Stroop Test” for about 30 minutes, while wearing a head band with the photo-diode. During that time, all the normal lights in the room were kept on, but an additional level of illumination provided by a desk lamp placed above the eye level of the subject was switched ON and OFF, alternatively, at intervals not previously known by the subject, using a dimmer. This was done to repeatedly introduce passages of high and low illumination in the experiment, which would trigger the pupillary light reflex.
4 Results Before its application to the adaptive interference canceller, the recorded pupil diameter signal is pre-processed by a blink-removal algorithm implemented in MATLAB, which is able to: 1. Detect the PD data interruptions due to eye blinks (identified as a value of “4” in the validity code provided by the TOBII system); 2. Compensate the missing data by linear interpolation. 3. Filter out the blink responses through a low pass, 512th order FIR filter designed for a cutoff frequency 0.13Hz. Figure 4 illustrates the stages of the blink-removal process on data collected from subject 13. MEASURED PD SIGNAL 6 4 2 0 -2
0
0.5
1
1.5
2
2.5
3 4
x 10 VALIDITY CODES FROM BOTH EYES 6 4 2 0 -2
0
0.5
1
1.5
2
2.5
3 4
x 10 PD SIGNAL AFTER BLINK COMPENSATION 6 4 2 0 -2
0
0.5
1
1.5
2
2.5
3 4
x 10 PD SIGNAL AFTER LOW PASS FILTER 6 4 2 0 -2
C1 0
0.5
IC1 1
C2 1.5
IC2 2
C3 2.5
IC3 3 4
x 10
Fig. 4. PD Signal Before & After Blink-Removal
Monitoring and Processing of the PD Signal
55
The PD signal obtained after blink-removal is applied to the AIC system as the primary input signal d(k), and the reference input r(k) is the simultaneously measured illumination intensity level signal, which is down-sampled from 360Hz to 60 Hz to share the same sampling rate with the PD signal. A MATLAB program was created to apply the HITV adaptive algorithms for the ATF with 120 weights and the parameter settings of η = 0.001 , ε g = 2.0 as well as a time-varying parameter ρ , changed according to the IL value to enable the AIC system to have a quicker response when there is a sudden increase in IL. The output of the AIC, the MPD, is shown in the bottom plot of Figure 5. This signal is further processed as described below to become a useful indicator of pupillary affective response in the subject. AIC PRIMARY INPUT: PD SIGNAL 6 4 2
0
0.5
1
1.5
2
2.5
3 4
x 10 AIC REFERENCE INPUT: IL SIGNAL 2 1.5 1
0
0.5
1
1.5
2
2.5
3 4
x 10 -6
1
AIC TIME-VARING PARAMETER: RHU
x 10
0.5 0
0
0.5
1
1.5
2
2.5
3 4
x 10 AIC OUTPUT: MPD SIGNAL 2 0 -2
C1 0
0.5
IC1 1
C2 1.5
IC2 2
C3 2.5
IC3 3 4
x 10
Fig. 5. AIC Implementation with HITV Algorithm
The affective state of “Stress” is expected to cause a dilation of the pupil [2]. Therefore, the negative portions of the MPD signal, are zeroed to isolate significant MPD increases, which indicate the emergence of stress in the subject. The result of applying this non-negative restriction to the MPD signal is shown in the bottom panel of Figure 6.
56
Y. Gao, A. Barreto, and M. Adjouadi
A sliding window with a width of 1200 samples is applied throughout the nonnegative MPD signal to calculate the median value within each window. The effect of this process on both the original PD signal and the non-negative MPD signal is compared in Figure 7. In this figure, it is clear that the significant increases that have been isolated in the processed MPD signal correlate closely with the occurrence of IC segments, regardless of the presence of higher illumination passages during segments IC2 and C3. It should also be noted that the same post-processing operated on the PD signal obtained directly from the eye gaze tracking system does not set apart the IC segments as clearly. Sliding Window Median Analysis of PD
AIC Output: MPD Signal 2
6
1
5
0
4
-1
0
0.5
1
1.5
2
2.5
3
3
0
0.5
1
1.5
2
2.5
3
4
4
x 10
x 10
Non Negative MPD Signal
Sliding Window Median Analysis of MPD
2
1
1
.5
0
0 C1
-1
0
0.5
IC1 1
C2 1.5
IC2
C3
2
IC3
2.5
C1 .5
3
0
0.5
IC1 1
C2
IC2
1.5
C3 2
2.5
4
x 10
Fig. 6. MPD Signal Non Negative Processing for Stress Indication
Fig. 7. Comparison of Sliding Window Median Analysis on PD & MPD
Signal Processing Results of Subject 11 3.5 3 2.5 MPD 2 Non Negtive1.5 MPD 1 Sliding 0.5 Window Median MPD 0 C1 -0.5
0
0.5
IC1 1
3 4
x 10
GSR
IC3
C2 1.5
IC2 2
C3 2.5
IC3 3
3.5 4
x 10
Fig. 8. Signal Processing Results of Subject 11
Monitoring and Processing of the PD Signal
57
A similar result is observed for data collected from Subject 11, as shown in Figure 8, where the 3 top signals (GSR, which was recorded simultaneously, MPD and Nonnegative MPD) have been shifted up to make the display clear. In this case, also, the most significant increases of the bottom trace, after the initial adaptation that precedes segment C1, occur during the incongruent segments (IC1, IC2 and IC3). This figure shows the appearance of Skin Conductivity Responses (SCRs) and overall elevation of the GSR signal during the incongruent segments. Furthermore, the bottom plot (the result of the proposed post-processing) also shows significant increases during IC1, IC2 and IC3, with minimal output on the other segments after the initial adaptation.
5 Conclusions This study implemented an H∞ time-varying adaptive algorithm in an Adaptive Interference Canceller to discount the influence of Pupillary Light Reflex from a measured PD signal, so that the output of the AIC, the MPD, with the application of a non negative constraint and sliding window median analysis, can be used as an indicator of Pupillary Affective Responses due to, for example, subject stress. This indicates that this approach might be useful for the affective assessment of a computer user, even in the presence of illumination changes. A comparison of this result with the outcome obtained by applying the same post-processing directly to the recorded pupil diameter signal, points out the advantage of the adaptive implementation, which output relatively more distinctive increases when the incongruent Stroop segments occurred. Data from other subjects who participated in our experiment have revealed similar results. These outcomes, therefore, encourage the continued exploration of adaptive processing algorithms applied to pupil diameter signals, as a non-invasive mechanism to achieve affective assessment of computer users in ordinary environments. Acknowledgements. This work was sponsored by NSF grants CNS-0520811, HRD0833093 and CNS-0426125. Ms. Ying Gao is the recipient of a Dissertation Year Fellowship from Florida International University.
References 1. Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997) 2. Beatty, J., Lucero-Wagoner, B.: The Pupillary System. In: Cacioppo, Tassinary, Berntson (eds.) Handbook of Psychophysiology, 2nd edn., pp. 142–162. Cambridge University Press, Cambridge (2000) 3. Partala, T., Surakka, V.: Pupil size variation as an indication of affective processing. Int. J. of Human-Computer Studies 59, 185–198 (2003) 4. Widrow, B., Stearns, S.D.: Adaptive signal processing, pp. 337–339. Prentice-Hall, Englewood Cliffs (1985) 5. Hassibj, B., Kailath, T.: H∞ adaptive filtering. In: Proc. Int. Conf. Acoustics, Speech, Signal Processing, pp. 945–952 (1995) 6. Puthusserypady, S.: H∞ Adaptive filters for eye blink artifact minimization from electroencephalogram. IEEE Signal Processing Letters 12, 816–819 (2005)
58
Y. Gao, A. Barreto, and M. Adjouadi
7. Zhai, J., Barreto, A.: Significance of Pupil Diameter Measurements for the Assessment of Affective State in Computer Users. Biomedical Sciences Instrumentation 42, 495–500 (2006) 8. Renaud, P., Blondin, J.-P.: The stress of Stroop performance: physiological and emotional responses to color-word interference, task pacing, and pacing speed. Int. Journal of Psychophysiology. 27, 87–97 (1997) 9. Todoroki, A., Hana, N.: Luminance change method for cure monitoring of GFRP. Key Engineering Materials, 321–323, 1316–1321 (2006)
Usability Evaluation by Monitoring Physiological and Other Data Simultaneously with a Time-Resolution of Only a Few Seconds Károly Hercegfi, Márton Pászti, Sarolta Tóvölgyi, and Lajos Izsó Budapest University of Technology and Economics, Department of Ergonomics and Psychology, Egry J. u. 1, 1111 Budapest, Hungary {hercegfi,tim,ebediyen,izsolajos}@erg.bme.hu
Abstract. This paper outlines the INTERFACE methodology developed by researchers of our department. It is based on the simultaneous assessment of Heart Period Variability (HPV), Skin Conductance (SC), and other data. The objective and significance of this paper are (1) showing its capability of identifying quality attributes of software elements with a time-resolution of only a few seconds and (2) presenting its practical applicability in the evaluation phase of a real software development process. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics carried out a contract-based applied research project for the GeneraliProvidencia Insurance Co. Ltd. The Company was in the process of further developing the software used in its customer centers, and our Department contracted to assess the user interface. Both analytical and empirical usability evaluation methods were applied. In this paper, we highlight the new experiences of the INTERFACE testing methodology. Keywords: Usability testing and evaluation, empirical methods, case study, Heart Period Variability (HPV), Skin Conductance (SC).
1 The Frame of the Project During the last decades, the Department of Ergonomics and Psychology at the Budapest University of Technology and Economics developed mutually useful relationships with some industrial partners successfully accomplishing various Research & Development (R&D) projects 4. The Genarali-Providencia Insurance Co. Ltd. as a member of the Generali Group is owned by the Generali PPF Holding that has 9 million customers in Central and Eastern Europe. From these 9 million customers, 1.2 million are given by the Genarali-Providencia Insurance Co. Ltd. Two kinds of different core systems are used by the Genarali-Providencia Insurance Co. Ltd. to keep their data: one for the life insurance (it is called SYNPAC), and another (it is called VIAS) for the non-life insurance, e.g. liability insurance, Casco insurance, etc. The management started to establish a new system that can visualize the data of both core systems. To carry it out, the IT department of the company started to develop a new frame system called Genesys. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 59–68, 2009. © Springer-Verlag Berlin Heidelberg 2009
60
K. Hercegfi et al.
This software is used by approximately 1500 users. From these 1500 users, 300 users use it daily. The main part of the users work in personal customer centers. One smaller part of the users work as call center operators. It is a smaller group of the users, however, the time pressure in their job underlines the importance of the usability factors. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics latched on to the software development to carry out the usability assessment. The main goal of our project was to collect problems of the user interface (UI). From these misses we made a list that we gave it to the developers. Developing solutions for the found problems can be a next project. Other goal of the Company was supporting a scientifically interesting research project transferring results of foundational researches to really applicable methodology in real software development process.
2 Applied Methods The preparation of the project started in November 2007. The contract was signed in March 2008. We got in touch with the developers at the beginning of April 2008. We started our project with studying the software and collecting data and facts. While we were studying it, the system was updated, so sometimes it was difficult to be up to date. In the 2nd part of April we started to observe the users of the software, and we also made interviews with them. We performed the observation and interviews at 3 different locations: in the biggest customer service of the firm, in another type, smaller customer service, and in a call center. We obtained some logfile data to establish objective initial data for the subsequent assessment. We applied analytical methods (usability inspection methods) in May and June 2008. The backbone of the analytical evaluation was a Guideline Review, supported by Cognitive Walkthrough elements. Also GOMS model-based analysis was carried out. The main part of the assessment was a carefully planned, deep empirical series of experiments applying the INTERFACE methodology described in the following section. The series of experiment were carried out in July 2008; the analysis of the collected records was performed in August 2008.
3 Description of Our Main Methodology: The INTERFACE Figure 1 shows the conceptual arrangement of the INTERFACE (INTegrated Evaluation and Research Facilities for Assessing Computer-users' Efficiency) workstation.
Usability Evaluation by Monitoring Physiological and Other Data
61
observable behavior current screen content
data collecting and
keystrokes and mouse clicks
processing frame system
physiological signals by ISAX
Fig. 1. Conceptual arrangement of the INTERFACE user interface testing workstation
The advantage of the methodology applied in our study lies in its capability of recording continuous on-line data characterizing the user’s current mental effort derived from Heart Period Variability (HPV) and the user’s emotional state indicated by Skin Conductance (SC) parameters simultaneously and synchronized with other characteristics of Human-Computer Interaction (HCI). This way, a very detailed picture can be obtained which serves as a reliable basis for the deeper understanding and interpretation of psychological mechanisms underlying HCI. Elementary steps of HCI, like the different mental actions of users followed by a series of keystrokes and mouse-clicks, are the basic and usually critical components of using software. These steps can be modeled and analyzed by experts, but empirical studies of real users’ interactions often highlight new HCI issues or give more objective results than expert analyses. One of the key aspects of the empirical methods is measuring mental effort as it is laid down e.g. in the earlier international standard of software product evaluation (ISO/IEC 9126:1991). Hence we need methods capable of monitoring users’ current mental effort during these elementary steps. To attain the above, a complex methodology was developed earlier at the Budapest University of Technology and Economics, by Prof. Lajos Izsó and his team [3, 4, 5]. This study presents an improved methodology and a new case study. The INTERFACE simultaneously investigates the following: • Users' observable actions and behavior − keystroke and mouse events; − video record of the current screen content; − video records of users’ behavior: (1) mimics, (2) posture and gestures. • Psycho-physiological parameters − Power spectrum of Heart Period Variability (HPV), regarded as an objective measure of current mental effort – we apply this signal successfully since more than 15 years [1, 2, 3, 4]; − Skin Conductance (SC) parameters, indicating mainly the emotional reactions – recently integrated into our system.
62
K. Hercegfi et al.
A number of studies [1, 2, 3, 5, 7, 8, 9, 10] have shown that an increase in mental effort causes a decrease in the mid-frequency (MF) peak of the HPV power spectrum. The main advantage of the assessment method of the spectral components integrated into our system over the previously existing HPV-based methods is that the MF component of HPV shows changes in mental effort in the time range of several seconds (as opposed to the earlier methods with a resolution of tens of seconds at the best). This feature was achieved by an appropriate windowing data processing technique, and application of an all-pole auto-regressive model with built-in recursive Akaike's Final Prediction Error criteria and a modified Burg’s algorithm. We watch the Alternating Current (AC) component of the Skin Conductance (SC) responses focusing mainly on the emotional aspects of the HCI, in addition to our well-tried approach of mental effort. An interesting series of experiments analyzing SC responses is finished by one of our colleagues [6]. It is a good example of the promising way to use data mining techniques in empirical usability studies. Participant, operator of the Computer used Camera to Display Standard call center with standard by the record the with IP headset regularly used in the participant – facial the phone call center. The Skin according to expressions. software of the Conductance (SC) electrodes can the standard Motorized face currently call tested center tracking, be seen on the left hand, the workstation of zoomable ECG electrodes are on the torso the call center
The ISAX equipment to record the physiological signals
Camera to record the body posture
Computer of the experimenter – during the session, online curves of the physiological signals, video images of the cameras and the editor window for the comments can be seen
Fig. 2. The experimental arrangement applied during the sessions of the INTERFACE usability testing installed on a standard workstation of the call center
Usability Evaluation by Monitoring Physiological and Other Data
63
4 Applying INTERFACE in the Current Series of Experiments The empirical sequence of experiments applying the previously introduced INTERFACE assessment was performed with the call center operators in the middle of July 2008. Due to the impersonality caused by the phone calls, on one hand the simulation was more authentic than it could be in the personal customer centers. On the other hand, because of the specialty of the call center, we could count on concentrated, quick problem-solving. What more, the usability problems are the most critical in the call centers due to the time pressure. The tasks occurred in the call center are usually not solved only by means of the Genesys, but these other software are not being examined in this project. However,
The screen just seen by the user Two cameras: Facial expression and body posture
Keyboard and mouse actions
Experimenter’s comments
Upper (blue) curve: AC component of the Skin Conductance (SC). The higher deviation means more emotional event.
Signals derived from the ECG, related to mental effort. Red RR curve in the middle: periods between the subsequent heart beats in ms. Last (green) profile curve: the MidFrequency (MF) power of the variability of the RR curve. Its low values mean significant mental effort; peaks mean relief, relaxation.
Fig. 3. The INTERFACE Viewer screen with a record of the empirical test of the Genesys software. As it can clearly be seen, currently the user makes significant mental effort – it is shown by the facial expression and gesture, and the low value of the last green profile curve of the Mid-Frequency (MF) power of the Heart Rate Variability (HRV) at the cross-hair.
64
K. Hercegfi et al.
we had to give controlled, simulated tasks to the users in order that they can solve them in the Genesys system. These simulated tasks helped us to be able to compare the 12 sessions. We used the version of the Genesys from the test server that is substantially equal to the version actually used in real. 12 real operators were involved as participants, each of whom we recorded a onehour-long session with. The quantity of data gained from these sessions is really significant according to the depth of the enquiry. Due to the real life situation the users participating the series of experiments, were more or less disturbed by their colleagues’ calls or talks. Since these are the employees’ real conditions, they have got used to them, and so they could work on typical, real workstations of their workplace. These advantages gave reason not to hold this usability testing in laboratories. Nevertheless, we chose a workstation that was located in the corner of the operators’ room, in order not to disturb the others. Behind this workstation, the team leader’s glass wall can be found, and so our staff could sit and make simulated phone calls from behind this pane. As mentioned before, during the recorded sessions, the users were given tasks from the real life, with quasi-real data, names, problems, questions, etc.; the main difference was that the customers were not real customers, but from our staff. Three ECG electrodes were put on the users’ torso and one on the left hand (in case of a right-handed person) for measuring Skin Conductance. After that the users put on their headsets and adjusted their seats. Figure 2 shows the experimental arrangement applied during the sessions of the INTERFACE usability testing installed on a standard workstation of the call center. Figure 3 shows the INTERFACE Viewer screen with a record of the empirical test of the Genesys software. At the beginning of the session, we asked the users to relax for two minutes. We told them the aim and the details of the assessment. We always emphasized that we are not willing to assess they themselves, but the Genesys software by means of their help. Relaxation was followed by two-minute mental effort: mental arithmetic. The result of the counting was not important; we only wanted to generate mental effort. These periods were planned for “calibrating” the physiological curves. After that, the hard work started. The first customer (from our staff) rang the phone and asked some questions. To answer these questions the operator had to use the software under testing. Later 4 more similar calls came. Each call contained 2 to 4 questions. One part of the questions were just to “warm up”, others were really difficult. The subtasks were based on the interviews, observations and expert analyses performed earlier. The last part of the INTERFACE session was an interview.
5 Validation As it was mentioned, the periods of relaxation and mental arithmetic were planned for “calibrating” the physiological curves. The curves shown in the upper part of the Figure 4 were recorded during session #11, the ones shown at the bottom were recorded during session #10.
Usability Evaluation by Monitoring Physiological and Other Data
relaxation
65
mental arithm.
relief
relaxation
mental arithm.
relief
Fig. 4. The typical pattern of the relaxation and mental arithmetic in cases of two participants
In both cases, the three curves are the blue curves of the AC of Skin Conductance (SC), the red RR curves (Heart Periods), and the green profile curves of the MidFrequency (MF) power of Heart Rate Variability (HPV). The blue curve of AC of SC is relatively smooth during both the relaxation and the mental arithmetic. During these sections, there are not any emotional peaks, and these two participants can be characterized as “stabile” type according to the typology of physiology. However, the beginnings and the ends of the sections are followed by peaks. During relaxation, the MF component of the HPV increases, then the red RR curve has zigzags, and the green profile curve is relatively high. (In case of perfect relaxation, the profile curve should be consistently high. However, this is not expected in this experimental situation. The curve can be considered as high, especially in comparison with the next section.) During mental arithmetic, the red RR curve gets smoother, and the green profile curve is significantly low. After the “calibration” tasks, the participants really relieve. During this short period of relief, the participants get more relaxed than during the conscious, intended relaxation: the green curves have their highest peaks here.
66
K. Hercegfi et al.
Mean of MF power of HPV [ms2]
These “calibration” tasks prove a validation of our method. The values of the MF power of the HPV were significantly higher during relaxation than during mental arithmetic. A non-parametric statistical method, the Wilcoxon Signed Ranks Test proves the difference (sig. 0.037 – Figure 5).
100,0
80,0
60,0
40,0
20,0
0,0 relaxation
mental arithm.
Fig. 5. Validation of measuring Mid-Frequency (MF) power of Heart Rate Variability (HPV) as an indicator of mental effort: the MF power of HPV was significantly higher during relaxation than during mental arithmetic (sig.0.037)
It is a significant difference, in spite of the non-perfect relaxation. However, the mental arithmetic task works better: the significance of the difference between the values of MF power of HPV during mental arithmetic and in general, during the whole software usage section is better: the Wilcoxon test results sig. 0.002. The values of the deviation of the AC component of Skin Conductance (SC) do not differ during the relaxation and the mental arithmetic significantly. As it was described earlier, this is the expected result. However, the deviations of the AC of SC during the relaxation and the mental arithmetic are significantly lower than in general, during the whole software usage section: the Wilcoxon tests result sig. 0.009 and sig. 0.017. After these results, we can say that the low value of the curve of the MF power of HPV really means mental effort, and the high deviation of AC of SC probably means higher emotions. Than, in the section of software usage, we look for moments with relatively high (and unwanted) mental efforts and high (and unwanted and not positive) emotions. This method gives us the key to find the problems of the UI.
6 A Sample UI Problem Identified by the INTERFACE Methodology Commercial sensitivities prevent publication of the most of the details of the particular software problems found. However, Figure 6 gives an illustration.
Usability Evaluation by Monitoring Physiological and Other Data
67
Fig. 6. The 11th participant during the first task of the second call. The mental effort can clearly be seen: it is shown by the facial expression and gesture, and the low value of the last green profile curve of the Mid-Frequency (MF) power of the Heart Rate Variability (HRV) at the cross-hair. In this case, the problem was caused by a bad design solution to ensure choosing a time period for a list view. This problem can also be found by analytical methods – but the INTERFACE highlighted this problem and gave objective evidence for it.
7 Conclusion Based on the results presented here as well as in related papers, it can be stated that the INTERFACE methodology in its present form is capable of identifying the relative weak points of the HCI. By this methodology and the related workstation, it was possible to study events occurring during the HCI in such detail and objectivity that would not have been possible using other methods presently known to us. The sophisticated Heart Period Variance (HPV) profile function integrated into the INTERFACE system is a powerful tool for monitoring events in such a narrow time frame that it can practically be considered as a time-continuous recording of relevant elementary events. Measuring the Skin Conductance (SC) is a new opportunity to modulate the results. Acknowledgements. The authors would like to thank Dr. Eszter Láng for the earlier developments, the Generali-Providencia Insurrance Co. Ltd. for the support of the deep research, and the participants of the series of experiments for their valuable contribution.
68
K. Hercegfi et al.
References 1. Chen, D., Vertegaal, R.: Using Mental Load for Managing Interruptions in Physiologically Attentive User Interfaces. In: Proc. CHI 2004, pp. 1513–1516. ACM Press, New York (2004) 2. Hercegfi, K., Kiss, O.E., Bali, K., Izsó, L.: INTERFACE: Assessment of HumanComputer Interaction by Monitoring Physiological and Other Data with a Time-Resolution of Only a Few Seconds. In: Proc. ECIS 2006, ECIS Standing Comm., pp. 2288–2299 (2006) 3. Izsó, L.: Developing Evaluation Methodologies for Human-computer Interaction. Delft University Press, Delft, The Netherlands (2001) 4. Izsó, L., Hercegfi, K.: HCI Group of the Department of Ergonomics and Psychology at the Budapest University of Technology and Economics. In: Ext. Abstracts CHI 2004, pp. 1077–1078. ACM Press, New York (2004) 5. Izsó, L., Láng, E.: Heart Period Variability as Mental Effort Monitor in Human Computer Interaction. Behaviour Information Technology 19(4), 297–306 (2000) 6. Laufer, L., Németh, B.: Predicting User Action from Skin Conductance. In: Proc. IUI 2008, pp. 357–360. ACM Press, New York (2008) 7. Lin, T., Imamiya, A.: Evaluating Usability Based on Multimodal Information: An Empirical Study. In: Proc. ICMI 2006, pp. 364–371. ACM Press, New York (2006) 8. Mulder, G., Mulder-Hajonides van der Meulen, W.R.E.H.: Mental Load and the Measurement of Heart Rate Variability. Ergonomics 16, 69–83 (1973) 9. Orsilia, R., Virtanen, M., Luukkaala, T., Tarvainen, M., Karjalainen, P., Viik, J., Savinainen, M., Nygard, C.-H.: Perceived Mental Stress and Reactions in Heart Rate Variability – A Pilot Study Among Employees of an Electronics Company. International Journal of Occupational Safety and Ergonomics (JOSE) 14(3), 275–283 (2008) 10. Rowe, D.W., Sibert, J., Irwin, D.: Heart Rate Variability: Indicator of User State as an Aid to Human-Computer Interaction. In: Proc. CHI 1998, pp. 18–23. ACM Press, New York (1998)
Study of Human Anxiety on the Internet Santosh Kumar Kalwar and Kari Heikkinen Lappeenranata University of Technology, Department of Information Technology, Lappeenranata, Finland
[email protected],
[email protected]
Abstract. In this paper a conceptualization of human anxiety on the Internet is introduced; it is built on the understanding of human behavior with regard to technology. The objective of this paper is to conceptualize the human anxiety. An integral part of understanding is an inter-disciplinary (psychology science, cognitive science, behavioral science and communication technology) literature review, of which and overall summary is presented. The understanding is conceptualized by designing, implementing and evaluating through a developed user study model. In this paper the preliminary result of utilizing the developed user study found seven particular anxiety areas which need further studies. Keywords: Human, study, anxiety, internet.
1 Introduction Twenty-seven years ago, Internet was like a small dog at the bottom of the application pile, fighting for recognition, today, vast numbers of people are using Web Services through the Internet. One of the fascinating things about the Internet is that it comes with outstanding tools for usage in regards to recreation, education, society and much more. Current estimated statistics have shown that more than 63 million humans use the Internet in homes and the importance of the Internet in daily activities is growing. There is little which cannot be accomplished from the comfort zone of home; paying bills, research work, buying, selling, updating, uploading, downloading, shopping and, most importantly, communicating with your family and friends half way across the world. Since the establishment of the WWW, the number of Internet users has grown from an estimated 16 million in 1995 to more than 500 million in 2002 – explosive growth to say the least [1]. The rather recent but enormous and rapidly progressing emergence of the Internet has not gone unnoticed by scholars working in many different fields of research [1]. The Internet has become both an object of study and a tool of research. A basic definition of the term host in the case of a computer network is a computer connected to the Internet. The number of hosts connected to the Internet has grown rapidly. In 1983, when the Transmission Control Protocol/Internet Protocol (TCP/IP) standard was first adopted there were only about 250 hosts connected to the Internet. By late 1993, figure had grown to two million. During 2005, the number of hosts on the Internet had increased to 317.6 million. In recent years, the number has J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 69–76, 2009. © Springer-Verlag Berlin Heidelberg 2009
70
S.K. Kalwar and K. Heikkinen
skyrocketed to 541 million [2]. The fact and digital data from various sources confirms that number of humans accessing the Internet is growing at a rapid pace. As the Internet evolves in terms of number of human online, it feels as if it is evolving as social community. Worldwide, the Internet population is growing at a rapid pace. The number of people getting access to information, learning, and going online is booming like never before. It should be remembered that as late 1988, only a few countries were connected to the Internet. According to the 2004 CIA World Fact book, over 50 countries have at least one million humans using the Internet [3]. Humans once found it difficult and expensive to communicate during the times of voice telephones but with the rapid technological development communication has improved drastically. The distance has been shortened from family, friends and from seeking information which can be seen as the replacement of very important daily interactions. Boundaries of time, distance and identity are broken by the transfer of simple applications like e-mail to the complex world of virtual communities. Together with the positive growth, its negative effects are growing too. According to the U.S. Department of Justice, the Internet is an anonymous and effective way for many predators to find and groom children for illegal activities [4]. The fear of using the internet is further amplified by social disintegration, psychological and cognitive implications. 1.1 Objective and Scope The main research question is, “how can we address the challenges such as Internet addiction, psychology and human computer interaction it is currently facing now?” To understand the objectives, some of the questions were researched in details. The paper aims to find answers to the following hypotheses: • • • • •
Do users shows increased or reduced anxiety level when using the Internet? What kinds of behaviors are shown when using the Internet? What is the role of the content? Finding types of the anxiety behaviors? How human process information at the internet interface?
The scope of the work includes design, implement and evaluate methods used in understanding the behavior of humans using the Internet technology. It has wide range of scope from the field of psychological perspective to cognitive science, behavioral science and communication technology. The scope is complex. However, To study human and their tasks and how to relate information to design style, human behavior theories, standards, procedures or guidelines in order to build an appropriate model of interaction with the help of some existing methods is investigated. 1.2 Internet Anxiety A lot has been written in past about the negative use of the Internet anxiety, Internet addiction and full dependence on the internet is welling up [5] [6] [7]. The service disruption because of network faults, software bugs, administrator mistakes and version upgrade could seem less tolerable. Millions of human around the world use Internet to search, inform, find, communicate, work and play. Internet should not be
Study of Human Anxiety on the Internet
71
only viewed as negative such as addiction and pathological nor should it be vilified. One must be aware of negative consequences of overuse of the Internet by understanding the behavior of themselves and from others. Four types of Internet anxiety was identified by Presno. C,, using qualitative study method [8]. These are: Internet terminology anxiety: anxiety produced by an introduction to a host of new vocabulary words and acronyms. Net search anxiety: anxiety produced by searching for information in a maze-like cyberspace. Internet time delay anxiety: anxiety produced by busy signals, time delays, and more and more people clogging the Internet. General fear of Internet failure: a generalized anxiety produced by fear that one will be unable to negotiate the Internet, or complete required work on the Internet. Additional three areas of the Internet anxiety from the qualitative study were found in this research. Experience anxiety: an anxiety produced by lack of concentration or focus. Usage anxiety: a generalized anxiety produced by excessive usage of the Internet. Environment and attraction anxiety: anxiety produced by content on the Internet. For example: interactive game, pornography and larger number of colorful applications. Table 1. Types of Anxiety recorded for Subject I and Subject II Types of anxiety Terminology Search Time delay General fear Experience Usage Environment & attraction
p1
p2
p3
x x x x x
x x x x x
p4 x x x x x x
p5
p1 x
x x
p2 x
p3 x x
x x
x
x x x
x x x
x x
x x x x
p4
p5
x x x
x x x x x
x
2 Inter-disciplinary Literature Review Sociability is important as well as usability of applications in the Internet. While usability is concerned with making sure that the application, software and system is consistent, predictable, and easy and satisfying to use, sociability and the social aspect of building and maintaining an online community focuses on processes and styles of social aspect in interaction that support human behavior on the internet to some extent. Research has shown certain social groups to be under-represented on the Internet [9] [10] not simply because of a lack of access, but more because of cognitive, motivational and affective factors [11]. Psychology therefore has an important role in advancing the understanding of why humans choose to use or not to choose the usage of the Internet [12]. There will always be an argument to model psychology with technology or technology with psychology however; combining psychology with technology will give rise to new technology called psycho technology. To understand the Internet technology in broader ways, interaction between human and the technology through the Human Computer Interaction becomes essential. Brain Computer
72
S.K. Kalwar and K. Heikkinen
Interface techniques are only studied in this research work, no practical implementation was carried out due to the limitation of resources. In BCI, skill developed by a human involves proper control of electro physiological signals which are easily adapted and modulated by the brain for better feedback. In understanding human anxiety, BCI techniques could be used for testing and analyzing different activities on the Internet. The ethnographic study of the Internet can be divided into two categories. First, user-based and second, content based. User-based analysis is about the investigation, examination and the study of humans using the Internet. Whereas, content based analysis is mainly focused on text. Humans are capable of providing reasons to support their points of view if asked a question such as “What color is my shirt?” and are capable of knowing without explicit Deduction or reasoning answers to questions like, “If I were you, I would hate myself”, whereas computers cannot function without specific programming instructions.
3 Methods, Design and Implementation The user study model used both qualitative and quantitative research methods. The qualitative research was conducted using interviews and observational analysis. The quantitative research was conducted in three iterations by using questionnaires and surveys. The Questions were analyzed with the help of different types of questionnaires being constructed. Intentionally, Participants were given very general types of question to answer. Two types of Questionnaires were used: Using the Internet and Using the Pen and Paper method. Task calculation was carried out by dividing the task into subtask. Two types of subject were categorized based on skill level (novice, intermediate and expert human): Subject I and Subject II. Usability Test recording form was used for each participant. This form was used to record both the verbal and non-verbal behaviors of humans using the Internet. The task was chosen which was based on the
Fig. 1. The user study model implemented to test behavior of humans using the Internet
Study of Human Anxiety on the Internet
73
Internet. The task contained three modules which were divided into task 1, task 2 and task 3 based on the level of difficulties. It was found from the task analysis that humans using the Internet took less than a second to complete all of these tasks. There is a division of the participant’s overt behavior into two major general categories: verbal and nonverbal. Verbal behavior includes anything a participant says and Nonverbal behaviors include various activities that the participants actually do. In non verbal behavior mostly facial expression such as smiling, looks of surprise, furrowing brow or showing body gestures such as leaning close to the screen, rubbing the head is shown by the participants these are facial expression, eye-tracking, pupil diameter, skin conductance among others.
4 Results and Discussion The goal of the study was to determine “how can we address the challenges such as Internet addiction, psychology and human computer interaction it is currently facing now?” In order to evaluate the main research question, the main research question was broken down into hypothesis. There were five different hypothesis formulated in the beginning of the research. Now, Let us try to discuss these hypotheses to see our method, design, evaluation and analysis of the research was supported (fully supported, partially supported and not supported) or not. H1: Do users shows increased or reduced anxiety level when using the Internet? Hypothesis 1 was fully supported. Human shows the sign of increased level of anxiety when using the Internet. It appears that, with any given task on the Internet number of anxiety increase in human. More number of participants said “yes” to five or more items from the QS 2, which indicates problematic Internet Usage. Using HADS and PHQ-9 it was observed that, one participant seemed to have a Case of higher depression scale. Therefore, in this particular case users showed increased anxiety level when using the Internet. H2: What kinds of behaviors are shown when using the Internet? Hypothesis 2 was fully supported. Literature review revealed that there are two types of behavior shown by human using the Internet: Verbal and Non verbal. During Observation of behavior for the participants, Most of the times humans were laughing, smiling, drumming their fingers on the table and looking aimlessly around. These behaviors pattern was verbal and non-verbal. These types of the behavior patterns were observed among narrowly selected group of the participants. H3: What is the role of the content? Hypothesis 3 was partially supported. Role of content could determine the predicted or unpredicted human behavior on the Internet. Such as addiction, anxiety and stress of using the Internet, since humans were successfully able to complete the task with ease, it could be predicted that-any sort of given task for humans is very easy to do on the Internet. Therefore, the role of content has principal impact on how human behaves on the Internet.
74
S.K. Kalwar and K. Heikkinen
H4: Finding types of the anxiety behaviors? Hypothesis 4 was fully supported. It was found that in this particular case there are seven main area of anxiety in the humans: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Using Observation methodology and Comparing two types of subjects: Subject I and Subject II, it concludes that- all the participants showed these above cited anxiety. H5: How human process information at the internet interface? Hypothesis 5 was partially supported. When human interacts with the internet interface, it appears that, everything that human senses such as sight, hearing, touch, smell and taste are processed as the information in the mind. This information could result in behavior such as verbal and non-verbal. Even if behavior initially disappears, it may partially return as undamaged parts of the brain reorganize their linkages. The human behavior in totality of processing information includes internal cognitive processes which can result in observable behavior. Processing of the information at the internet interface has the realistic approach such as thinking of mental processes as several railroad lines that all feed to same terminal. Two schools of thought have emerged which confirms with the hypothesis that, there are two types of the behavior while using the Internet: Verbal and Non-Verbal behavior. In more general terms human using the Internet can use the content available on the Internet in two different ways: Positive or Negative. The gestures or types of human behavior shown could lead to anxiety. Seven major types of anxiety were studied and validated: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, and general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Two types of behavior (verbal and non-verbal) were formulated from relevant literature study, empirical analysis and evaluation. The study of Brain Computer Interface concludes that signal could be send in human brain physically to control and observe behavior of humans, however the BCI techniques were not used in the study and without using BCI techniques, the study conducted discovered sample of humans showing increased level of anxiety when using the Internet. The task completion behaviors of humans were calculated. By the end of this discussion session, We can reach to the conclusion that, to reduce Internet anxiety, addiction and depression scale on the Internet, it is important to have many multicultural experiences, and control over own self behavior to accumulate successful experiences of behavior.
5 Conclusions and Future Work Taking the results and discussion to the logical conclusion it appears to the authors knowledge that, “Internet has lulled humans with the sense of dependency to greater extent”. Five types of hypothetical questions were answered in this study. These questions were: Do users shows increased or reduced anxiety level when using the Internet? What kinds of behaviors are shown when using the Internet? What is the role of the content? Finding types of the anxiety behaviors? , And How human process information at the internet interface? Seven major types of anxiety were studied
Study of Human Anxiety on the Internet
75
and validated: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, and general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Two types of behavior (verbal and non-verbal) were formulated from relevant literature study, empirical analysis and evaluation. The study of Brain Computer Interface concludes that signal could be send in human brain physically to control and observe behavior of humans, however the BCI techniques were not used in the study and without using BCI techniques, the study conducted discovered sample of humans showing increased level of anxiety when using the Internet. From the book by Sir Tony Hoare, The first passage in Communicating Sequential Processes reads, “Forget for a while about computers and computer programming, and think instead about objects in the world around us, which act and interact with us and with each other in accordance with some characteristic pattern of behavior”. The same idea is followed in the study of human anxiety on the Internet. • Large sample size, different demography structure and discovery of perfect user study model are needed, for larger impact and generalization. • In contrast to several findings of negative effect in the Internet addiction, anxiety and depression group, some positive effects could be determined in future by building the framework for future learning through Imagination, Investigation and Innovation. • In depth analysis and comparisons of the human brain and the network open System Interconnection (OSI) model could be performed. Despite the above limitations, undoubtedly the Internet has provided a collection of applications that is having a profound effect on mankind. Like the wheel, the plow, and steam power before it, it is a proving a truly differentiating tool in our world, changing the very ways in which we interact with each other. Progress is relatively easy to recognize if we follow technology exploration. A more challenge is to find technology we want to change ourselves and for our civilization. Understanding the human factors for the design and development of the technology, system and services to ensure successful and perfect application environment is major concern. The forms of anxiety identified suggest areas for future Internet development and research.
References 1. ISOC.org (2008), http://www.isoc.org/internet/history/brief.shtml 2. ISC.org. Millions of host on the internet (2008), https://www.isc.org/ 3. Central Intelligence Agency. The DigiWorld in the global economy. In: DigiWorld 2008 (2008), https://www.cia.gov/library/publications/ the-world-factbook/geos/xx.html; https://www.cia.gov/library/publications/ the-world-factbook/fields/2184.html 4. Golden, S.M.J.: Protecting children in the internet age (2008), http://www.senate.state.ny.us/sws/ Protecting%20Children%20in%20the%20Internet%20Age.pdf
76
S.K. Kalwar and K. Heikkinen
5. Kraut, R.E. (2008), http://www.cs.cmu.edu/~kraut/RKraut.site.files/articles/ Bessiere06-Internet-SocialResource-DepressionL.pdf 6. Chou, C.: Incidence and correlates of internet anxiety among high school teachers in taiwan. Computers in Human Behaviour 19, 731–749 (2003) 7. Skinner, B.F. (2008), http://www.bfskinner.org/aboutfoundation.html; http://www.bfskinner.org/f/Science_and_Human_Behavior.pdf 8. Presno, C.: Taking the byte out of internet anxiety: Instructional techniques that reduce computer/internet anxiety in the classroom. J. Educ. Comput. Res. 18, 147–161 (1998) 9. Jackson, L.A.: Social psychology and the digital divide. In: The 1999 Conference of the Society for Experimental Social (1999) 10. Sax, L.J., Ceja, M., Teranishi, R.T.: Technological preparedness among entering freshmen: the role of race, class and gender. Journal of Educational Computing Research 24, 363– 383 11. Jackson, L.A., Ervin, K.S., Gardner, P.D., Schmitt, N.: Gender and the internet: Women communicating and men searching. Sex Roles 44(5), 363–379 (2001) 12. Jackson, L.A., Ervin, K.S., Gardner, P.D., Schmitt, N.: The racial digital divide:Motivational, affective and cognitive correlates of internet use. Journal of Applied Social Psychology 31, 2019–2046 (2001)
The Research on Adaptive Process for Emotion Recognition by Using Time-Dependent Parameters of Autonomic Nervous Response Jonghwa Kim1, Mincheol Whang2, and Jincheol Woo1 1
Dept. of Computer Science, Sangmyung University, 7 Hongji-dong, Jongno Gu, Seoul, Korea {rmx2003,mcun}@naver.com 2 Dept. of Digital Media Technology, Sangmyung University, 7 Hongji-dong, Jongno Gu, Seoul, Korea
[email protected]
Abstract. This study is to propose new method, called by TDP (time dependenet parameter) anlaysis, of physiological signal processing for emoiton recognition. TDP consised of delay, activation, half recovery and full recovery. TDP was determined from running average and normalization of physiological signals for finding tonic and phasic reponse according to emotion at entire time range from stimulating emotion to recovery. As the results of this study, TDP analysis and adaptive TDP analysis enhanced accuracy of emotion recognition in the comparison with tonic analysis. Speciallly, TDP analysis enhanced the accuracy while adaptive TDP analysis reduced the individual difference of the accuracy. Keywords: Physiological signal, GSR, ECG, PPG, Skin temperature, emotion recognition, accuracy.
1 Introduction Human emotion has been tried to be recognized by physiological measurements based on assumption that emotion was expressed by them[1]. Indentifying physiological signals to subjective emotion has been main issue of emotion recognition research. However, since the signals were vulnerable to be affected by noise, emotion has had recognition error. One of methods avoiding noise has been running average at narrow time interval of incoming signal. This signal process has been effective to treat noise in autonomic response having relatively low frequency of physiological signals [2-4]. Physiological signals have been analyzed into tonic level and phasic response which were important variables according to emotion response[5]. Data treatment should be considered to discriminate level and reaction from the signal. Physiological signal has contained information of both spontaneous and non-spontaneous response. Phasic response could provide non-spontaneous overall oscillation and specific continuous level according to stimulus[5]. Recently, normalization of stimulus state from reference state has been enabled to count both tonic level and phasic response for emotion recognition [6-8]. However, since response before and after stimulus according to time variation was not examined enough, non-specific phase response of physiological signals has less clarified to cause low accuracy of emotion recognition. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 77–84, 2009. © Springer-Verlag Berlin Heidelberg 2009
78
J. Kim, M. Whang, and J. Woo
Physiological response has been characterized by individual difference. Same emotion could be recognized by different regulation of physiological response. Therefore, the strategy or rule of emotion recognition should consider individual characteristics. Some findings showed that algorithm of emotion recognition set physiological variation automatically based on verification of subjective emotion and that this process enhances the accuracy of emotion recognition[9]. Therefore, emotion could be well recognized by considerations of noise reduction, discrimination between tonic level and phasic response of physiological signals, and individualization. Considering these issues, this study is to suggest new analysis methods of physiological signals, called by TDP (time dependent parameter) and to attempt to show effectiveness of emotion recognition when it is used.
2 Method 2.1 Research Purpose This study is to propose new analysis methods for physiological response and to prove that this method was effective for emotion recognition. This research was proceeded to compare the accuracies of emotion recognition from different methods which were tonic analysis, TDP analysis, and adaptive TDP analysis. 2.2 Definition of TDP TDP (Time dependant parameter) of physiological response was defined in this study as shown in Fig. 1. The delay was moment difference between stimulation and the activation. The activation meant time at peak form the beginning. The half recovery indicated the time at half peak and the full recovery did the time back to base state. The full recovery could be inferred from half recovery when it could not be measured. In this study, ECG (Electrocardiogram), RSP (Respiration), PPG (photoplethysmogram), GSR (Galvnic skin resistance) and SKT(Skin temperature) were processed to construct TDP curve as shown in Fig. 1.
Fig. 1. TDP (time dependent parameter) of physiological measurement
The Research on Adaptive Process for Emotion Recognition
79
2.3 Emotion Induction Emotion was tried to be induced by image pictures. The images were chosen by previous study [8]. 100 university students (33 females and 67 males) participated, and were not visually handicapped. Participants were asked to score subjective emotion after watching the images. Significant images of emotion induction were categorized into 2 dimensional emotion model[10]. 6 images were selected for evoking unpleasantness-arousal emotion and 10 for pleasantness-relaxation emotion as shown Fig. 2. The pleasantness-relaxation was called as the positive emotion, and the unpleasantness-arousal was called as the negative emotion in this study.
Fig. 2. The images evoking emotions
2.4 Experiment 4 university students (average 26.5 years old) participated in the experiments and were healthy with no problem of vision. 24 prepared images were presented to participants for inducing emotion. PPG, GSR, RSP and SKT were measured during presenting images. Experiment procedure was shown as fig. 3. Participant experienced first non-image state as a reference state for 30 seconds followed by presenting the image for 10 seconds. Then, non-image state called by neutral state was presented for 30 seconds. A procedure consisted of presenting 4 images and a participant experienced 6 times a procedure. The experiment took 190 seconds for a procedure and total experimental time was 1440 seconds for a participant.
Fig. 3. Experimental procedure
80
J. Kim, M. Whang, and J. Woo
3 Analysis 3.1 Data Acquisition Collected were 6 sets consisting of 4 physiological signals and subjective score of emotion. For purpose of analysis, data unit was set at 70 seconds including neutral state for 30 seconds, stimulation for 10 seconds and another neutral state for 30 seconds. Total 74 data (6 set * 4 pictures * 4 participants) were prepared for tonic analysis, TDP analysis, and adaptive TDP analysis. 3.2 Running Average for Noise Reduction and Normalization The running average has been effective to reduce noise [3]. In this research, the time interval for running averaging was determined by response rate of each signal. Time intervals of physiological signals were tried to be set for noise reduction. The determination was done by visual scanning for confirming signal stability. The time interval of GSR and SKT was set at 0.5 seconds while one of RSP was at 3 seconds. PPG was analyzed to convert HR by frequency analysis. Therefore, time intervals for PPG and HR were set 2 seconds. Running average was performed by sliding window method at the pre-determined time interval on all the physiological signals. Then, the stimulus state of physiological signal was normalized from the neutral state. This process was able to observe activation level (tonic level) of physiological signal. Normalization was determined by equation 1 and performed at every 0.5 seconds. Normalized state = (Stimulus state –Neutral state) / Neutral state
(1)
3.3 TDP Rule for Emotion Recognition TDP rule for emotion recognition was determined from previous study [8]. Visual stimulus from the prepared images induced their emotion as shown Fig 2. Then, physiological signals were analyzed into TDP and their threshold values of emotion recognition were set for constructing rule of emotion recognition as shown Table 1. Table 1 showed mean and standard deviation of physiological response time of emotion based on TDP definition. The range was defined by mean plus and minus standard deviation for recognizing respective neutral, positive and negative emotion. 3.4 Adaptive TDP Rule for Individualization Since physiological response on same emotion was individually different, TDP rule for emotion recognition needed to be developed for individualization. The process of adaptive TDP rule was shown as Fig. 4. First, emotion was recognized by nonadaptive TDP rule. Second, difference between measured subjective emotion and emotion by physiological signals was calculated. Then, if difference existed, TDP rule was adaptively set by individual input of subjective emotion. Otherwise, emotion recognition was performed. Through these processes, it became individual and accurate adaptively for a particular person.
The Research on Adaptive Process for Emotion Recognition
81
Table 1. The TDP rule (unit: seconds) t1(Delay) t2(Activation) t3(Half Recovery) Neg. Neu. Pos. Neg. Neu. Pos. Neg. Neu. Pos. 1.55 1.92 4.08 5.79 1.57 3.08 HR ±0.92 ±1.00 ±2.91 ±2.31 ±0.17 ±1.40 15.85 9.61 20.00 5.60 5.83 16.00 SKT ±10.00 ±0.40 ±9.21 ±5.09 ±0.74 ±6.57 3.67 1.00 RR ±1.80 ±0.05 2.65 1.56 3.69 0.79 4.51 GSR ±1.16 ±0.40 ±1.95 ±11.06 ±6.55 PPG 2.25 2.43 3.90 4.37 3.45 8.00 1.75 6.03 amp ±2.74 ±0.31 ±2.91 ±1.97 ±1.48 ±4.93 ±12.06 ±6.03
t4(Recovery) Neg. Neu. Pos. 2.85 1.44 1.92 ±1.20 ±0.24 ±1.68 -
-
-
3.33 ±2.50
1.00 ±0.05
-
-
-
-
-
-
Neg.: Negative, Neu.: Neutral, Pos.: Positive
Fig. 4. The process of adaptive TDP rule for individualization
4 Result Results showed the accuracies of emotion recognition from three different methods such as tonic analysis, TDP analysis and adaptive TDP analysis as shown table 2-5. The accuracy was determined by match rate between subjective emotion and emotion determined by physiological signals. Tonic response rule was determined from previous research of autonomic response pattern for emotion[5, 11]. If negative emotion was evoked, electrodermal and cardiovascular response were increased and thermal response was decreased and vice versa [11]. As shown in Table 2, the accuracy of emotion recognition was about 62% in negative emotion recognition and about 50% in positive emotion. In the same condition, the accuracy was enhanced up to 60% when TDP analysis was used as shown in Table 3. It was about 70% in recognition of both positive and negative. Therefore, the accuracy of emotion recognition by TDP analysis could observe responses more than one by tonic analysis. There were findings for individualization of emotion recognition as shown Table 5. Adaptive TDP rule made the accuracy enhanced little more. Interestingly, the participants having the accuracy lower than 70% increased up to 70% or more. Therefore, adaptive TDP analysis could be effective to increase accuracy of a particular person who was low. Figure 5 showed overall accuracies from three analyses. TDP analysis showed improvement of accuracy but adaptive TDP did not much. Since accuracy improved in individual showing low, overall accuracy did not contribute improvement.
82
J. Kim, M. Whang, and J. Woo Table 2. Accuracy of emotion recognition from tonic analysis
participants
Negative emotion Emotion by Subjective physiological emotion signals
Positive emotion Emotion by Subjective physiological emotion signals
Accuracy
A
8
5
10
6
61.1%
B
6
3
9
4
46.7%
C
7
4
8
4
53.3%
D
8
6
9
4
58.8%
Sum
29
18
36
18
Accuracy
62.1%
55.0%
50.0%
Table 3. Accuracy of emotion recognition by TDP rule
Participants
Negative emotion Emotion by Subjective physiological emotion signals
Positive emotion Emotion by Subjective physiological emotion signals
Accuracy
A
8
7
10
7
77.8%
B
6
3
9
6
60.0%
C
7
4
8
7
73.3%
D
8
6
9
5
64.7%
Sum
29
20
36
25
Accuracy
69.0%
69.0%
69.4%
Table 4. Accuracy of emotion recognition by adaptive TDP rule
Participants
Negative emotion Emotion by Subjective physiological emotion signals
Positive emotion Emotion by Subjective physiological emotion signals
Accuracy
A
8
6
10
7
72.2%
B
6
4
9
6
66.7%
C
7
4
8
7
73.3%
D
8
6
9
6
70.6%
Sum
29
20
36
26
Accuracy
69.0%
72.2%
70.1%
The Research on Adaptive Process for Emotion Recognition
83
Fig. 5. Accuracy comparison of three analyses such as tonic, TDP and adaptive TDP analysis
5 Conclusion TDP and adaptive TDP were newly proposed to analyze physiological response for emotion recognition. The methods were successful of enhance its accuracy. The adaptive TDP was effective to count individual difference of physiological response. Comparing accuracies of emotion recognition among tonic analysis, TDP analysis, and adaptive TDP analysis, this study concluded the followings. First of all, TDP analysis enhanced the accuracy more than tonic analysis. Accuracy by tonic analysis was an average of 55%, and one of TDP analysis was of 69%. Second, the accuracy of adaptive TDP analysis reduced the individual difference. In the case of analysis by using TDP rule, the accuracy of individual difference was between 60% and 77% while in the case of adaptive TDP rule, the accuracy of individual difference was between 66.7% and 73.3%. Results showed adaptive TDP could enhance accuracy that was relatively low but work less for a participant whose accuracy was already high. Therefore, TDP and adaptive TDP method may be useful of emotion recognition and observe detail significant response of physiological response.
References 1. Christine, L., titia, L., Fatma, N.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J. Appl. Signal Process., 1672–1687 (2004) 2. Allanson, J., Fairclough, S.H.: A research agenda for physiological computing. Interacting with Computers 16, 857–878 (2004) 3. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: First steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS, vol. 3068, pp. 36–48. Springer, Heidelberg (2004)
84
J. Kim, M. Whang, and J. Woo
4. Mandryk, R.L., Atkins, M.S.: A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of HumanComputer Studies 65, 329–347 (2007) 5. Boucsein, W.: Electrodermal Activity. Plenum Press, New York (1992) 6. Whang, M.: The emotional computer adaptive to human emotion. Phillips Research: Probing Experience 8, 209–219 (2008) 7. Whang, M., Lim, J., Boucsein, W.: Preparing computers for affective communication: a psychophysiological concept and preliminary results. The Journal of the Human Factors and Ergonomics 45, 623–634 (2003) 8. Kim, J., Whang, M., Kim, J., Woo, J.: The study on emotion recognition by timedependent parameters of autonomic nervous response. Korean Journal of the science of emotion & Sensibility 11, 637–644 (2008) 9. Fredrickson, B.L., Losada, M.F.: Positive Affect and the Complex Dynamics of Human Flourishing. American Psychologist 60, 678–686 (2005) 10. Russell, J.A.: A circumplex model of affect. Journal of personality and social psychology 39, 1161–1178 (1980) 11. Whang, M., Chang, G., Kim, S.: Research on Emotion Evaluation using Autonomic Response. Korean Journal of the science of emotion & Sensibility 7, 51–56 (2004)
Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking Yu-Jin Kim, Jin Ah Bae, and Byeong Ho Jeon Dept. of Media Image Art & Technology, Kongju National University, Sinkwandong, Gongju, South Korea {yujinkim,jinabae,bhjeon}@kongju.ac.kr
Abstract. In this paper, we used eye tracking methodologies to investigate students’ visual perceptions of lectures using 3D real-time virtual studio technology. For measuring learning performance, we also gave the students multiple-choice paper quizzes at the end of the lectures. Three virtual lectures were created with different types of lecture materials (text-centered, image-centered, and lecturercentered) and 3D virtual sets (classroom, cyberspace, and lecture-theme space). Through analyzing students’ eye movements in viewing still and moving scenes of the virtual lectures, we found that layouts and movements of design elements on lecture screens significantly influenced students’ scanpaths and areas of interest (AOIs). Lecture material types affected learning performance while 3D virtual sets had no effect due to students’ inattention to the virtual background areas. We discuss effective ways to develop virtual lectures and design lecture screens for better presentation of lecture content and higher learning performance. Keywords: Virtual lectures, virtual studios, eye tracking, visual perception, learning performance, user-centered screen design.
1 Introduction Over the last two decades, advances in digital technologies brought about the emergence of new teaching pedagogies such as Computer-Assisted Instruction (CAI), Computer-Supported Collaborative Learning (CSCL), Web-Based Instruction (WBI), and others. Besides these pedagogies using computer and internet technologies, there has been growing interest in virtual studio technology, which composites images of performers shot in front of a chroma-key blue background and visual images created through real-time 3D computer graphics [1]. The virtual studio technology increases face-to-face interaction between a lecturer and students and helps the lecturer interact with various types of lecture materials in dynamic virtual environments [2]. In academia, however, little attention has been paid to the methods to create courses using virtual studios for delivering learner-centered lecture materials and enhancing learning performance. To fill this research niche, this study aims to analyze how students look at the screens of virtual lectures [3] using eye tracking technology. Eye tracking provides insight into students’ visual perceptions and cognitive processes by measuring their eye movements. We research how lecture material types and 3D virtual sets affect learning performance. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 85–94, 2009. © Springer-Verlag Berlin Heidelberg 2009
86
Y.-J. Kim, J.A. Bae, and B.H. Jeon
2 Related Work 2.1 Lectures Using Virtual Studio Technology Virtual education, by the use of cyberspace, eliminates the spatial and temporal limitations by removing the need for the lecturer and students to be present at an instructional site at a designated time [4]. Recently, the proliferation of information and communication technologies (ICTs) has spawned a boom in virtual education by creating a variety of virtual lecture content production techniques such as Flash, Web 3D, 3D real-time virtual studio, and others. Among these production technologies, several researchers have explored the effectiveness of the virtual studio application to lectures. In 2003, Morozov, Debelov, and Zhmulevskaya claimed that the application of virtual studio systems enhances the efficiency of instructional processes by providing lecturers more possibilities for diverse forms of education [5]. According to Brown and Cruickshank (2003), a virtual lecture can be more effective than classroom lecture by achieving significant savings in the costs of lecture delivery and student support, though a very large investment of time and effort is required by the lecturer [6]. Along with the previously mentioned research interests in the educational effects of the application of new technology, researchers have studied the factors affecting the delivery of lecture content in virtual education in order to increase the motivation and effects of learning. These factors include screen design [7, 8], content design [9], content suggestion style, and lecture material [10]. 2.2 Eye Tracking in Visual Information Processing Eye movements are driven both by properties of the visual world and processes in a person’s mind [11]. Therefore, many scholars have used eye tracking techniques in order to explore the relations between eye movements and visual information processing in diverse research fields. In particular, researchers in HCI have tracked eye movements to understand visual and display-based information processing, as well as to discover the factors that may impact the usability of system interfaces [12]. In fact, eye movement data provide more detailed and specific information about a user’s cognitive process in many different kinds of displays [13, 14]. They also help researchers determine the roots of some usability problems, and then come up with effective solutions. Meanwhile, the recent trend of eye-tracking studies in HCI shows that eye-tracking methods have been rapidly employed in conducting usability testing of websites. Laarni et al. (2003) tracked the eye movements of users while they were selecting and reading online news items on a small computer [15]. Cowen (2004) asked the subjects to perform two tasks on a website and measured their total fixation duration, the number of total fixations, average fixation duration, and distribution of fixations on the screen [13]. Through these measurements, he suggested effective ways to design webpages in terms of user interface. In 2007, Cutrell and Guan also measured web surfers' eye movements to observe their web search strategies [16]. Like the aforementioned studies, many researchers have analyzed the interface of and interaction with web content, mainly focusing on a sequence of still scenes of web pages.
Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking
87
However, there have been increasing interests in employing eye-tracking methods in moving visual images along with the proliferation of more dynamic web content, which is developed by adopting new types of web authoring tools and media, such as Flash, Web 3D, Virtual Studio, and others. 2.3 Eye Tracking Metrics The main measurements used in eye-tracking research are “fixations,” which are moments when eyes are relatively stationary, taking in or encoding information, and “saccades,” which are quick eye movements occurring between fixations [12]. From these basic measurements, a multitude of metrics are also derived [17]: (1) “gaze duration,” the cumulative duration and average spatial location of a series of consecutive fixations within an area of interest; (2) “area of interest (AOI),” the area of a display or visual environment that is of interest to the research or design team and, thus, defined by them (not by the participant); and (3) “scanpath,” a special arrangement of a sequence of fixations.
3 Hypotheses The focus of our study is the investigation of students’ visual perceptions in virtual lectures. In order to identify the patterns of students’ visual perceptions, we tracked their eye movements according to different layouts and movements of design elements on lecture screens. We also studied the effects of lecture material types and 3D virtual sets on students’ learning performance. In line with these purposes, we investigated the following research hypotheses: Hypotheses 1a. Students’ eye movements (AOIs, scanpaths) are affected by the layouts of design elements on lecture screens. Hypotheses 1b. Students’ eye movements (AOIs, scanpaths) are affected by the movements of design elements on lecture screens. Hypotheses 2a. Students’ learning performance varies according to lecture material types. Hypotheses 2b. Students’ learning performance is different according to 3D virtual set types.
4 Experiment 4.1 Virtual Lecture Prototypes In order to test the aforementioned hypotheses, we designed and conducted an experiment to analyze students’ eye movements and learning performance in the context of virtual studio-based lectures. As experimental material, three virtual lecture prototypes were created under the theme of “Media and Culture” with varying layouts and movements of lecture screen design elements (lecturer, lecture board, 3D virtual sets, text, images, and movies on the lecture board). The three prototypes (see Figure 1) were also produced with different types of lecture materials (text-centered, image-centered,
88
Y.-J. Kim, J.A. Bae, and B.H. Jeon
and lecturer-centered) and 3D virtual sets (classroom, lecture-theme space, and cyberspace): (1) Prototype 1 - text-centered lecture materials in classroom background sets; (2) Prototype 2 - lecturer-centered lecture materials in cyberspace background sets; and (3) Prototype 3 - image-centered lecture materials in lecture-theme space background sets. Figure 1 shows screen shots of these three prototypes. Three screen shots located in the first row display their 3D background sets. The other three shots in the second row illustrate how the same lecture content about the phenomena of mass communication is delivered by different types of lecture materials.
Fig. 1. Screen shots of three prototypes (left: Prototype 1, middle: Prototype 2, and right: Prototype 3)
After modeling and animating the above 3D virtual sets using 3D Studio Max 9.0., we combined these 3D background sources with the live-action footage of lecturers by a real-time chroma key technique of the "VS2000" system. In addition, we created animation buttons for controlling lecture screen design elements using the VS scripts of the "HotAction" program. 4.2 Participants Forty participants were selected for the experiment, but ten of them had problem with the calibration of the eye-tracking system. Calibration was a fine-tuning process for the experiment, so 30 participants successfully completed the experiment. The 30 subjects were divided into three groups of ten who participated in the experiment composed of three kinds of virtual lectures. The subjects were freshmen and sophomores of K University and had almost no previous knowledge of the content of the experiment lectures. They were between the ages of 18 and 23, and the gender ratio was 1:1. Twenty-four of the students had experienced distance lectures, and eight of them had been exposed to VR lectures. 4.3 Apparatus This experiment used the hardware "Eyegaze Development System" developed by LC Technologies, Inc. This device tracks the x-y coordinates of the participant’s
Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking
89
gazepoint on the computer screen automatically using the pupil-center-corneal reflection (PCCR) method, which directs infrared light into the eye. This system generates raw eyegaze point location data at a camera field rate of 60 Hz [18] and calculates three kinds of gazepoints: (1) a moving point that a participant looks at for only 1/60 second, then looking at other points; (2) a fixating point that a participant looks around in; and (3) a fixation completed point that a participant maintains the gaze in. This study used the software "EyeTrack v1.0" and “EyeTrackMovie v1.0” for analyzing eye movements in viewing still and moving scenes, respectively. These programs were developed by the Human Computer Interaction Laboratory (HCIL) of KAIST for the visual monitoring and analysis of eye-tracking results that were expressed in coordinates [19]. “EyeTrack” and “EyeTrackMovie” provide “Replay” and “Analysis.” "Fixation Mark" and "Color Variation" options enable the monitoring of eye-tracking data with a variety of graphic effects in chronological order. For an efficient analysis of eye movement patterns, it also provides five analysis options: (1) "shadow," which shows the area that subjects focused on and illustrates the rest in shades with gradation effects; (2) "frequent area," which marks a clear boundary between the area that the subjects focused on and the rest, highlighting the former; (3) "hotspot," which shows the amount of gaze (red meaning more gaze and green meaning less gaze); (4) "selected area," which shows the amount of gaze in a certain area and its duration in numbers; and (4) "priority order," which shows the duration of a gaze over time in circles. All participants’ eye movements could be monitored at once due to the function of the programs which can analyze multiple data simultaneously. The patterns of AOIs and scanpaths, described in Section 2.3, were discovered from these eye movements. 4.4 Experimental Design and Procedure The experiment was performed in four stages, and it took about an hour for a participant to complete the experiment: (1) questionnaires on demographics, media education level, and cyber learning experiences; (2) still scene eye-tracking with the captured scenes according to distinguishable types of screen layouts (Prototype 1: 6 scenes, Prototype 2: 9 scenes; and Prototype 3: 7 scenes); (3) moving scene eyetracking with three prototype movies; and (4) a quiz with multiple-choice questions. Eye movements were tracked twice for still and moving scenes of virtual lectures. Compared with the eye-tracking of moving scenes, the eye-tracking of still scenes allowed more detailed and accurate analysis of eye movements for the different screen layout types. On the other hand, the eye-tracking of the moving scenes enabled an additional analysis of the effects of the movements of screen layout elements on eye movements.
5 Results 5.1 AOIs According to the Layouts and Movements of Lecture Screen Design Elements Through analyzing students’ eye movements in viewing still and moving scenes of the virtual lectures, we found that layouts and the movement of lecture screen design
90
Y.-J. Kim, J.A. Bae, and B.H. Jeon
elements (the lecturer, lecture board, and 3D background set) significantly influenced students’ scanpaths and areas of interest (AOIs) [H1a and H1b]. In the case of AOIs, parts that students paid close attention to were similar in both still and moving scene tests. By pointing out the design elements that received the most attention on the sill scenes, it was found that students generally gazed into a lecturer’s face in all of the still scenes that feature a lecturer (Prototype 1: 4 scenes, Prototype 2: 5 scenes, and Prototype 3: 5 scenes) regardless of the lecturer’s size and position on the screen. Figure 2 shows the “hotspot” option analysis results of the three scenes (left: Prototype 3, middle: Prototype 2, and right: Prototype 3), where different sizes of a lecturer appear in different positions. We calculated the distribution of fixations on the face of the lecturer in the left scene of Figure 2 using the “selected area” option and found that 52.5% of the total fixations were on the lecturer’s face.
Fig. 2. “Hotspot” option analysis results (Note: The amounts of fixations gradually increased from green to red, and the reddest parts are marked in circle)
In addition, we found that the students’ points-of-regard are likely to stay not just on the lecture’s face, but also on the face of people illustrated in the lecture material images (see Figure 3). Another finding was that the student’s attention to the people whose profile or back views were shown to decrease compared with front view images.
Fig. 3. “Selected area” option analysis results (1. Front view (28%), 2. Profile view (11.9%), and 3. Back view (12.5%) of people in the image)
Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking
91
Fig. 4. Comparing visual attention to text areas (3. Body (40.1%) > 2. Subtitle (29.5%) > 1. Title (7.3%))
Along with the aforementioned visual attention to people’s faces in the still scenes, we also found a tendency in the moving scenes that students’ points-of regard are mostly directed at the lecturer’s face as well as the faces of people illustrated in the lecture material images. In addition, the analysis results also suggest that students’ eye movements and text sizes in lecture boards were inversely proportional. There were less eye movements and shorter gazes on larger texts. In fact, visual attention was generally highest in the body, subtitles, and title (see Figure 4). In other words, considering that titles take up more space than subtitles, but receive less visual attention, it would be effective to deliver new and important messages using subtitles. 5.2 Scanpaths According to the Layouts and Movements of Lecture Screen Design Elements In order to measure the student’s scanpaths and scanpath duration, we selected a student whose eye movement pattern can represent the visual attention of the ten students participating in the same still scene experiment and then analyzed his/her eye movement over time using the “fixation map” and “priority order” options. From the results, it turned out that the farther from the center the objects are positioned, the
Fig. 5. Scanpath tracking (left: from left to right, right: from top to bottom)
92
Y.-J. Kim, J.A. Bae, and B.H. Jeon
more likely they are to be out of the scanpath or stared at later. This analysis also verified the commonly accepted idea that people in Korean-speaking culture read words and sentences from left to right and from top to bottom (see Figure 5). On the other hand, students’ scanpaths showed different patterns in still and moving scene tests in terms of the starting point of the eye’s gaze. In the still scene test, the starting points of the students’ gazes were related to screen design element types, as well as their positions. For example, the majority of students firstly gazed into the lecturer’s face, regardless of its position, and then moved their gazes to other elements located around the center of the screens. In the moving scene test, the movement of screen design elements, rather than their type, dominantly influenced the starting points, as illustrated in Figure 6. We also found that students moved their gazes to empty parts where the next lecture content was expected to show up. Properly timed anticipation could enable the students to better perceive animated lecture content by preparing them for what will come next in the lecture.
Fig. 6. Starting points of students' gazes according to the lecture screen layout type
5.3 Students’ Learning Performance According to Lecture Materials and 3D Virtual Sets The average quiz scores for the three groups who respectively watched text-centered, image-centered, and lecturer-centered virtual lectures (12.9, 9.7, and 10.2, respectively) were significant at the alpha level of 0.05 (F=4.694). These results support H 2a, which suggests the possibility of a relationship between students’ learning performance and lecture material types. Students could easily understand and memorize lecture content in text-centered lecture material which presented the lecturer’s
Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking
93
explanations in clear print on the screen. While lecture material types affected learning performance, 3D virtual sets had no effect on learning performance due to students’ lack of attention to the background areas. Consequently, H 2b, which suggests a strong relationship between students’ learning performance and 3D virtual set type, was not supported. In fact, the effects of the virtual sets on the learning process are very small because students’ visual attention is usually drawn to the background only when they found some noticeable images or figures in the set.
6 Conclusions This study investigated students’ visual perceptions of virtual lectures by analyzing their AOIs and scanpaths in the still and moving scenes of the lectures. It also explored how to develop lecture materials and 3D background sets of virtual lectures for improving learning performance by testing the students’ comprehension with multiple-choice questions. Our research revealed that the following issues on virtual lecture production should be carefully considered for better presentation of lecture content: (1) the lecturer’s size and positions on lecture screens and people’s postures illustrated in lecture materials; (2) text’s sizes and positions in lecture materials; (3) movements of screen design elements (the lecturer, lecture board, animating parts on the lecture board); and (4) balanced layouts between the lecturer, lecture materials, and virtual background sets. Educators and material designers should also consider that text-centered lecture materials were the most effective for higher learning performance than imagecentered and lecturer-centered ones. Even though statistical significance was not reached between 3D background sets and learning performance, an effective background set design suited to the lecture objectives and content is necessary for enhancing students’ learning interest and motivation. Finally, it is hoped that this study will assist lecturers in understanding students’ visual information processing in virtual lectures, in designing virtual lecture content more effectively, and in improving the instructional effects of virtual lectures.
Acknowledgment We thank Kun-pyo Lee and Jung-mi Park for their comments and help in conducting our eye-tracking experiment.
References 1. Fukaya, T., Fujikake, H., Yamanouchi, Y., Mitsumine, H., Yagi, N., Inoue, S., Kikuchi, H.: An Effective Interaction Tool for Performance in the Virtual Studio-Invisible Light Projection System. NHK Science & Technical Research Laboratories, Japan (2003) 2. Dolgovesov, B.S., Morozov, B.B., Shevtsov, M.Y.: The System for Interactive Virtual Teaching Based on “Focus” Virtual Studio. In: International Conference Graphicon 2003 in Moscow, Russia (2003)
94
Y.-J. Kim, J.A. Bae, and B.H. Jeon
3. In this paper, a virtual lecture means instructional content that is created using virtual studio techniques for virtual education 4. Kuroda, K., Shanawez, H.D.: Strategies for Promoting Virtual Higher Education: General Considerations on Africa and Asia. Africa and Asian Studies 2(4), 565–575 (2003) 5. Morozov, B.S., Develov, B.B., Zhmulevskaya, M.Y.: The System for Interactive Virtual Teaching Based on “Focus” Virtual Studio. In: International Conference Graphicon, Moscow, Russia (2003), http://www.graphicon.ru/ 6. Brown, S., Cruickshank, I.: The Virtual Studio. International Journal of Art & Design Education 22(3), 281–288 (2003) 7. Kim, M.R.: Strategies on Screen Design of Learner-Centered Web-based Instructional Systems. Journal of Educational Technology 16(4), 51–65 (2008) (in Korean) 8. Shon, M., Chung, H.H.: An Analysis on the Learning Hindrance Factors in BlendedLearning Environment. Journal of Educational Information and Media 13(2), 251–276 (2007) (in Korean) 9. Ryu, I.: Factors Influencing the Effectiveness of Web-Based Distance Learning. Management Education Review 6(2), 7–27 (2003) (in Korean) 10. Kang, M.H., Gu, M.H., Moon, H.N., Jung, S.Y., Chung, J.Y., Kim, J.S.: Examining the Effects of Tutor Delivery Modes on Cognitive Presence and Learning Outcomes in Online Lectures. Journal of Educational Information and Media 13(4), 155–181 (2007) (in Korean) 11. Richardson, D.C., Spivey, M.J.: Eye-Tracking: Characteristics and Methods, 1-9. In: Wnek, G., Bowlin, G.: Encyclopedia of Biomaterials and Biomedical Engineering. Informa HealthCare (2004) 12. Poole, A., Ball, L.J.: Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future Prospects. In: Ghaoui, C. (ed.) Encyclopedia of Human Computer Interaction, Idea group (2004) 13. Cowen, L.: An eye movement analysis of web-page usability. Masters by Research in the Design and Evaluation of Advanced Interactive Systems (2001) 14. Lohse, G.L.: Consumer Eye Movement Patterns on Yellow Pages Advertising. Journal of Advertising 26(1), 61–73 (1997) 15. Laarni, J., Isotalus, P., Kojo, I., Kärkkäinen, L.: Reading News from a Pocket Computer: An Eye-Movement Study. In: Harris, C.D., Duffy, V., Smith, M.J., Stephanidis, C. (eds.) Human-Centered Computing: Cognitive, Social and Ergonomic Aspects. The Proceedings of HCI International 2003, Lawrence Erlbaum, Mahwah (2003) 16. Cutrell, E., Guan, Z.: What Are You Looking for?: an Eye-tracking Study of Information Usage in Web Search. In: Proceedings of the SIGCHI conference on Human factors in computing systems, San Jose, California, USA (2007) 17. Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human-computer Interaction and Usability Research: Ready to Deliver the Promises (Section commentary). In: Hyönä, J., Radach, R., Deubel, H. (eds.) The Mind’s Eyes: Cognitive and Applied Aspects of Eye Movements, Elsevier Science, Oxford (2003) 18. Eyegaze Development System Information, http://www.eyegaze.com 19. Park, J., Lee, K.: Eyetrack - Developing Eyegaze Analysis Visualization Software for Designers’ Use. In: KEER 2007, Sapporo, Japan, p. 10 (2007)
Toward Constructing an Electroencephalogram Measurement Method for Usability Evaluation Masaki Kimura*, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology {masaki-k,hideta-u,masao,matumoto}@is.nasit.jp
Abstract. This paper describes our pilot study toward constructing an electroencephalogram (EEG) measurement method for usability evaluation. The measurement method consists of two steps: (1) measuring EEGs of subjects for several tens of seconds after events or tasks that are targets to evaluate, and (2) analyzing how much components of the alpha and/or beta rhythm are contained in the measured EEGs. However, there only exists an empirical rule on measurement time length of EEGs for usability evaluation. In this paper, we conduct an experiment to reveal the optimal time length of EEGs for usability evaluation by analyzing changes of EEGs over time. From the results of the experiments, we have found that the time length suitable for usability evaluation was more than 0~56.32 seconds.
1 Introduction Existing usability evaluation methods include interview, think-aloud protocols [1], and questionnaires [2, 3, 4]. These methods are widely used for usability evaluation because they require no measurement apparatuses and allow usability experts to measure usability of software systems in a relatively easy way. However, they also have some shortcomings for usability evaluation. For instance, usability evaluation using the above methods often requires a huge amount of time to analyze and evaluate collected data. Also, the results of the analysis are sometimes hard to replicate because the collected data is based on qualitative and/or subjective evaluations from system users or subjects participating in usability testing. In order to complement the limitations of these methods, many studies try to quantitatively and objectively develop evaluation methods for measuring the mental or psychological state of users from biological information. In this paper, we focus on the electroencephalogram (EEG) of users after using software as a quantitative measurement method of usability and aim to construct an EEG measurement method for usability evaluation. Alpha rhythm and beta rhythm, which are the frequency component of EEG, change according to mental condition [4], and EEG measurement methods have features that do not disturb subjects using a computer. In general, the EEG measurement method is used as follows: * Graduate School of Information Science, Nara Institute of Science and Technology 8916-5,Takayama, Ikoma, Nara, Japan. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 95–104, 2009. © Springer-Verlag Berlin Heidelberg 2009
96
M. Kimura et al.
(1) experimenters (or usability experts) measure a subject’s EEG for several tens of seconds after the subject finishes target tasks for usability evaluation, and (2) experimenters analyze how much the measured EEG contains the alpha and beta rhythm components, which respectively indicate a comfortable or uncomfortable state of the subject’s mind. The results of the evaluation should change according to the time length to perform the EEG. After the tasks, subjects’ EEGs return to the usual condition as time passes. Hence, the time length of the analysis is too long, and the ratio of the EEG changed by the tasks decreases. Conversely, if the time length is too short, evaluation accuracy decreases because the ratio of the noise to the entire the EEG increases. However, the measurement time at EEG analysis has been decided by experiential standards. So we need to analyze EEGs by time series for usability evaluation. In this paper, we try to obtain the proper time length of EEG data.
2 Related Research In this paper, we quantitatively evaluate a psychological state of computer users during a system in-use, using alpha and beta rhythms composed of frequency components of brain waves. Power spectrums of alpha and beta rhythms, which are obtained by discrete Fourier transform, the ratio of alpha and beta rhythms in all brain waves, and beta/alpha, which is the ratio of alpha and beta rhythms, are often used as common indicators for observing the psychological state of human beings. Matsunaga et al. developed a brain wave measurement system for evaluating satisfaction of human beings and validated the hypothesis that people feel comfortable if the amount of information processing in the brain is small, while people feel uncomfortable if the amount of information processing in the brain is large [5]. In this paper, we also use the ratio of alpha and beta rhythms in brain waves and beta/alpha as indicators. These indicators are often used for studies using brain waves. So our experimental results are easy to compare with implications and insights from previous work.
3 Experiment 3.1 Overview Using Microsoft Excel 2003 and Excel 2007, which are the most popular spreadsheet software, participants performed eight kinds of tasks operating spreadsheets, and we measured participants’ EEGs after each task. Excel 2003 and Excel 2007 are different versions of software released in 2003 and 2007, respectively. Both versions have almost the same functions, but have a different look and feel in their graphical user interfaces (GUIs). Excel 2007 has a new GUI called “ribbon,” which is designed to improve task performance and user experience. However, the newly designed ribbon interface introduced users to a lot of changes to menus, tool bars, and working windows. So, even if users would like to use a familiar function such as “Save As,” they need to select a menu or button with different names and/or positions between the two versions. Furthermore, even if names and positions of menus are not changed, the design of working windows displayed after selecting a menu is often different from Excel 2003. In this way, not only new users but also existing users of Excel 2003 need to learn how to operate the new interface of Excel 2007.
Toward Constructing an EEG Measurement Method for Usability Eval.
97
Table 1. Subjects’ usage frequency of Excel 2003 and Excel 2007
never several times per year several times per month several times per week
Excel 2003 0 2 3 5
Excel 2007 6 1 2 1
Using Excel 2003 and Excel 2007, we investigate the relationship between the experiences of software and the attributes of brain waves without the effect of functional differences. In our experiment, we also analyzed the relationship between the results of subjective evaluation by questionnaire and the attributes of brain waves. 3.2 Participants Ten master’s students from the graduate school of information science participated in the experiment. Table 1 shows participants’ usage frequency of Excel 2003 and Excel 2007. All participants had experience in using Excel 2003 and understood basic operations and functions, but half of the participants had never used Excel 2007. 3.3 Task Participants performed eight tasks (four types of tasks for each version of Excel) operating the spreadsheets given in advance. Table 2 shows a list of tasks used in the experiment. All tasks can perform both Excel 2003 and Excel 2007. The content (a grade report) of the data file that is used in this experiment is the same in all tasks. Participants can continue the tasks until the task time exceeds five minutes. We counterbalanced the order of the tasks to minimize learning and fatigue effects. The following are the details of each task. Same Place Task In this task, the participant selects a particular menu that has the same name in the same position in different versions of Excel. The task is completed when the participant selects the objective menu. Different Place Task The participant selects a menu that has a different name in a different position in the two versions of Excel. This task is completed when the participant selects the objective menu. Same Interface Task The Same Interface Task consists of functions that have the same interfaces of modal dialog in different Excel versions. In the task, menu name and position were given to the participant before the task. Different Interface Task In this task, functions that have different dialog interfaces in Excel 2003 and Excel 2007 were used. As with the Same Interface Task, menu name and position were given to the participant before the task.
98
M. Kimura et al. Table 2. A task list used in the experiment
Task type Same place Different place
Same interface
Task name Open Clip Art Pane Filter Setting Display of version information Record of macros Format Cells Page Orientation
Different interface
Conditional Formatting Insert Bar Chart
Description Open clip art pane to select clip art from a list. Set options for data filtering. Display the version information of Excel. Change date formats from Mar-01 to 03/01. Change a page orientation to landscape and set margins. Change a page orientation to landscape and set margins. Indicate cells that have less than “C” or “Absence” as red font. Insert stacked bar chart of student's scores with chart/axis titles.
3.4 Environment The Emotional Spectrum Analysis System ESA-16 was employed to record EEGs of participants. After the task, we recorded participants’ EEGs for two minutes at 200Hz sampling frequency in the eyes-closed, resting condition. Electrode locations are based on the International 10-20 System, shown in Figure 1. We adapted referential derivation to observe the EEG, and used the right earlobe (A2) as a reference electrode. As a ground electrode, the center of the forehead (Fpz) was employed and the center of the parietal (Pz) was used as an exploring electrode to minimize the electromyogram (EMG) artifact. We also recorded electrocardiogram (ECG) from both arms. In addition to this, we used a headrest and elastic net bandage to secure electrodes placed on the head. Before the first task, each participant adjusted the height of chair and position of the mouse/keyboard. 3.5 Procedure The procedure of the experiment was as follows. 1. Preparation: Authors informed participant about experiment and EEG measurement. 2. Environment setting: Put the electrodes on the participant at the points described in Section 3.4, then set up the EEG analyzer. 3. Practice tasks: The participant performed two practice tasks to understand the procedure of EEG measurement. These tasks were excluded from analysis. 4. Perform a task: The participant performed a main task described in Table 2. 5. EEG measurement: After each task, the EEG of the participant was measured. 6. Perform eight tasks: The participant performed tasks repeatedly until finishing eight tasks and EEG measurements. 7. Questionnaire: After the tasks, the participant filled out the questionnaire described in Section 3.6.
Toward Constructing an EEG Measurement Method for Usability Eval.
99
3.6 Questionnaire After the eight tasks, participants answered the questionnaire sheet to investigate subjective satisfactions for each version of Excel and usage frequency of each function that was used in the tasks. The questionnaire was created by the authors based on the Questionnaire for User Interaction Satisfaction (QUIS). Each question about usage frequency consists of a four-point scale (from “Never” to “Few times per week”) and seven-point scale (from “Strongly disagree” to “Strongly agree”) for subjective satisfaction. Figure 2 shows a part of questionnaire sheet that used in the experiment.
Fp1 F7
T3
Fpz
Fp1
Fz
F3
C3
F4
F8
T4
C4
Cz
A2
A1 T5
P3
P4
Pz O1
T6
O2
Fig. 1. Electrode Locations in the International 10-20 System
Brain wave time[sec]
Fig. 2. Questionnaire Sheet
Method 1. Power spectral analysis at even intervals
5.12
Method 2. Power spectral analysis using different lengths of time window
5.12
5.12
5.12
10.24 15.36
Fig. 3. Two Kinds of Analysis Methods for Electroencephalogram
4 Analysis for EEG We applied power spectral analysis to EEG data we collected at a sampling frequency of 200Hz. To have a clear understanding of how frequency components of brain waves changed over time in the setting of our experiment and how analysis results varied according to lengths of the analysis window, we used two analysis methods for the EEG data as follows. Figure 3 illustrates the difference between the analysis methods.
100
M. Kimura et al.
Method 1. Power spectral analysis at even intervals This analysis aims to observe how brain waves change over time. We analyzed EEG data with a time interval of 5.12 seconds by cutting out the entire EEG data every 5.12 seconds so as not to overlap analysis data. The EEG data with 19 intervals (from 0 to 92.16 second) was analyzed. Method 2. Power spectral analysis using different lengths of time window This analysis aims to observe how analysis results differ according to lengths of analysis window. We analyzed EEG data by increasing time length of the analysis window by 5.12 seconds, without changing the start position of our analysis. The time length was increased by 5.12 seconds (min.) to 97.28 seconds (max.) (i.e., 0~5.12 sec., 0~10.24 sec , …, and 0~97.28 sec.). Next, the target data was filtered to reduce the artifacts from eye blinking, myoelectric activity and so on. We used a high-pass filter (HPF, 3Hz cutoff frequency, +6dB/oct attenuation factor), a low-pass filter (LPF, 60Hz cutoff frequency, -6dB/oct attenuation factor), and a band-elimination filter (BEF, 60Hz central frequency, 47.5Hz~72.5Hz stopband, second order). The band-elimination filter was used to remove the influence of an alternating-current power supply. After the EEG data was multiplied by the Hamming window and processed with the fast Fourier transform (FFT), we obtained the power spectrum from the EEG data. From the obtained power spectrum, we calculated the respective proportions of alpha rhythm and beta rhythm to all brain waves, and also calculated beta/alpha, which divided the proportion of alpha rhythm into the proportion of beta rhythm. In accordance with the classification of the international 10-20 system, we set the frequency components of alpha rhythm and beta rhythm to 8~13Hz and 13~30Hz respectively. We also set the range of all brain waves to 3~30Hz. Since the proportions of alpha and beta rhythms to all brain waves are widely used for observing various activities in the brain, we also decided to use them as indexes for measuring the physiological state of subjects after the tasks. However, because the proportions and intensity of alpha and beta rhythms vary from individual to individual, comparisons of brain waves with the absolute value would be inappropriate. In this paper, we normalized EEG data by an average value of each subject’s power spectrum and compared it with each data.
5 Results 5.1 Results of the Power Spectral Analysis at Even Intervals Figures 4, 5, and 6 respectively show the mean and the standard deviation of the alpha rhythm, beta rhythm and beta/alpha in the power spectral analysis at even intervals. In each graph, the left x-axis, the right x-axis, and the y-axis are the mean, standard deviation, and time respectively. Figure 4 indicates that the mean of alpha rhythms for Excel 2003 were larger than that for Excel 2007 after 56.32 seconds and that the difference of the alpha rhythms between Excel 2003 and 2007 was largest at 81.92 seconds. The standard deviation was always comparatively small and lowest at 56.32 seconds. Figure 5 shows the mean of the beta rhythms for Excel 2007 was larger than that for Excel 2003 after 46.08 seconds, and the difference of the beta rhythms between Excel 2003 and 2007
Toward Constructing an EEG Measurement Method for Usability Eval.
101
was greatest at 87.04 seconds. The standard deviation was higher on the whole than the standard deviation of alpha rhythms. The lowest standard deviation was observed at 40.96 seconds. Figure 6 presents that the mean of beta/alpha in Excel 2007 was larger than that in Excel 2003 after 46.08 seconds and that the difference of beta/alpha between Excel 2003 and 2007 was at 81.92 seconds. The standard deviation is larger than the results of alpha rhythms and beta rhythms and smallest at 40.96 seconds. 5.2 Results of the Power Spectral Analysis Using Different Length of Time Window
0.30
1.05
0.25
1.00
0.20
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.00
0.40
1.15
0.35
1.10
0.30
1.05
0.25
1.00
0.20
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28
1.20
Time length[sec]
Start time [sec]
1.10
0.30
1.05
0.25
1.00
0.20
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.00
1.20
0.40
1.15
0.35
1.10
0.30
1.05
0.25
1.00
0.20
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.00
Start time [sec]
Time length[sec]
0.35
1.10
0.30
1.05
0.25
1.00
0.20
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.00
Standard deviation
0.40
1.15
Average of normalized β/ α
Fig. 8. Beta Rhythm in Method 2
1.20
0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16
Average of normalized β/ α
Fig. 5. Beta Rhythm in Method 1
Fig. 6. Beta/Alpha in Method 1
Standard deviation
0.35
5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28
0.40
1.15
Standard deviation
1.20
Average of normalized β rhythm
Fig. 7. Alpha Rhythm in Method 2
0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16
Average of normalized β rhythm
Fig. 4. Alpha Rhythm in Method 1
Start time [sec]
Standard deviation
0.35
1.20
0.40
1.15
0.35
1.10
0.30
1.05
0.25
1.00
0.20
0.95
0.15
0.90
0.10
0.85
0.05
0.80
0.00
Time length[sec]
Fig. 9. Beta/Alpha in Method 2
Standard deviation
Excel 2007 Excel 2007 SD
5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28
1.10
Excel 2003 Excel 2003 SD
Standard deviation
1.15
Average of normalized α rhythm
0.40
1.20
0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16
Average of normalized α rhythm
Figures 7, 8, and 9 respectively show the mean and the standard deviation of the alpha rhythm, beta rhythm and beta/alpha in the power spectral analysis using different lengths of time window. In each graph, the left x-axis, the right x-axis, and the y-axis are the mean, standard deviation, and time respectively.
102
M. Kimura et al.
Figure 7 shows that the mean of alpha rhythms in Excel 2003 was larger than that in Excel 2007 in the case of using time windows over 40.96 seconds. As lengths of time window became longer, the alpha rhythms in Excel 2003 tended to increase and the standard deviation decreased. Figure 8 presents that the mean of the beta rhythms in Excel 2007 was larger than that in Excel 2003 in the case of using time windows over 56.32 seconds. As lengths of time window became longer, the standard deviation tended to decrease as well as the results of the alpha rhythms. Figure 9 indicates that the mean of beta/alpha in Excel 2007 was larger than that in Excel 2003 in the case of using time windows over 40.96 seconds. As lengths of time window became longer, the standard deviation tended to decrease as well as the results of the alpha and beta rhythms. Table 3. A Result of Questionnaire Usage
Understand Productivity
frequency Excel 2003 Average 3.3 SD 0.82 Excel 2007 Average 1.8 SD 1.14 p < 0.05 yes
5.0 1.33 3.5 2.17 no
Simple Interface Easy Satisfacto use to use tion 5.4 4.8 4.9 5.3 5.0 1.26 1.32 1.45 1.34 1.15 3.5 4.0 3.1 3.4 3.3 1.72 1.63 2.13 1.90 1.95 no yes yes yes yes
5.3 Results of Questionnaire Table 3 shows the mean, standard deviation and the result of the two-sample t-test of each questionnaire item. In the table, there were significant differences in “Productivity,” “Interface,” “Easy to use” and “Satisfaction” between Excel 2003 and Excel 2007. Our subjects gave Excel 2007 lower scores than Excel 2003.
6 Discussions From the results of the power spectral analysis at even intervals, we could confirm that all three indexes tended to be relatively stable after 56.32 seconds. In fact, the alpha rhythms in Excel 2003 were larger than those in Excel 2007. The beta rhythms and beta/alpha in Excel 2007 were larger those in Excel 2003. The standard deviations of all three indexes also indicated lowest or relatively lower from 40.96 seconds to 56.32 seconds. These results would imply that EEG data from 56.32 seconds to 61.44 seconds is stable among all subjects, is less influenced by artifacts, and appropriately reflects the influence of the difference between Excel 2003 and 2007. However, for all indexes, the difference of the influence between two versions of Excel and the standard deviations constantly fluctuated. We consider that this might occur due to individual differences of subjects’ brain waves, fatigue from the long experiment duration, myogenic potential by attitude variation, and so forth. Therefore, it is necessary not to use a time window with short length, but to use one with long length, in order to conduct accurate usability evaluation.
Toward Constructing an EEG Measurement Method for Usability Eval.
103
From the results of the power spectral analysis using different lengths of time window, we could observe that the alpha rhythms in Excel 2003 were stably higher in the case of using time windows over 40.96 seconds, the beta rhythms in Excel 2007 were stably higher in the case of using time windows over 56.32 seconds, and beta/alpha in Excel 2007 was stably higher in the case of using time windows over 40.96 seconds. For all indexes, the standard deviations tended to become small as the time window became longer. This might be because the rate of EEG influenced by artifacts became small as the time window became longer. These results suggest us that we could analyze EEG data with no influence of artifacts by using a time window over 56.32 seconds. The results of our analysis showed that the alpha rhythms in Excel 2003 were larger than those in Excel 2007, and the beta rhythms and beta/alpha in Excel 2007 were larger than those in Excel 2003. It is clarified by previous studies on EEG measurement methods that the amount of alpha rhythms is decreased and the amount of beta rhythms and the value of beta/alpha are increased when a subject’s mental work load is high. The results of the questionnaire showed our subjects preferred Excel 2007 to Excel 2003. These results of questionnaire and our analysis agree with previous work. Since the results of our analysis and questionnaire also supported the results of the past studies on EEG, we can conclude EEG measurement would be useful for evaluating software usability.
7 Conclusion and Future Work In this paper, we have conducted an experiment to have a clear understanding of the appropriate timing and lengths of time window in order to analyze EEG data for accurate usability evaluation. From the results of the experiments, we could obtain the following insights. • A short length of time window (e.g. 5.12 seconds) is not suitable for usability evaluation because the frequency components of brain waves constantly fluctuate. • The accuracy of usability evaluation can be improved by using length of time windows of 56.32 seconds. In this experiment, we could not observe normal state of EEG data but EEG influenced by the tasks. In the near future, we need to conduct another experiment to observe it.
References 1. Ericsson, K.A., Simon, H.A.: Protocol analysis: Verbal reports as data. MIT Press, Cambridge (1993) 2. Osgood, C.E., Suci, G.J., Tannenbaum, P.H.: The measurement of meaning. University of Illinois Press, Urbana (1957) 3. Chin, J.P., Norman, K.L., Shneiderman, B.: Subjective user evaluation of CF PASCAL programming tools. Technical Report (CAR-TR-304) (1987)
104
M. Kimura et al.
4. Hart, S.G., StaveLand, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Handcock, P.A., Meshkati, N. (eds.) Human Mental Workload, pp. 139–183. Elsevier, Amsterdam (1988) 5. Matsunaga, H., Nakazawa, H.: A Study on Human-Oriented Manufacturing System (HOMS) – Development of Satisfaction Measurement System (SMS) and Evaluation of Element Technologies of HOMS using SMS. In: Int. Conf. Manufacturing Milestones toward 21st Century, pp. 217–222 (1997)
Automated Analysis of Eye-Tracking Data for the Evaluation of Driver Information Systems According to ISO/TS 15007-2:2001 Christian Lange, Martin Wohlfarter, and Heiner Bubb Lehrstuhl für Ergonomie, Technische Universiät München Bolzmannstrasse 15, 85747 Garching {lange,m.wohlfarter,bubb}lfe.mw.tum.de
Abstract. First of all, the most important content of the ISO/TS 15007-2:2001 standard for performing eye tracking experiments will be described. The following text includes a detailed description of how eye/gaze experiments using the Dikablis eye tracking system are conducted according to the above mentioned standard and how recorded statistical evaluations can be automated and visualized. Keywords: ISO/TS 15007-2:2001, Eye tracking, Driver Assistance Systems, Driver Information Systems.
1 Introduction Adhering to the guidelines of the European Statement of Principles ESoP, the need for good and less distracting design of driver information systems will grow enormously. The future will yield the driver information systems which will hinder or distract the driver from the driving task being performed as least as possible. The advantage of the standardized experimental conduction lies in the improvements delivered by progressive experimental comparability, error reduced and faster processing. The duration of the analysis of uplifted data is therefore enormously reduced. Subsequently, the standard conduction of gaze experiments using the Dikablis Toolkit can be proved to comply with the ISO/TS 15007-2:2001 standard.
2 Toolkits for Standardized Experimentation and Testing The Dikablis software package supports the standards required by eye tracking experimentation. In figure 1 the interference between Dikablis and the ISO/TS 150072:2001 standard regarding test planning, calculations and analysis of eye tracking data is illustrated. Dikablis will be deployed in order to evaluate the tracking of the gaze direction and eye and head movements of the test subject. The Dikablis Toolkit consists of a Recording Software, an Analysis Software for post-processing of the eye-tracking data and manual data analysis and D-LAB for completely automated data analysis. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 105–110, 2009. © Springer-Verlag Berlin Heidelberg 2009
106
C. Lange, M. Wohlfarter, and H. Bubb
Fig. 1. Workflow schema for standardized testing and experimentation of Dikablis and ISO/TS 15007-2:2001 interferences/differences
With the Recording Software the gaze behavior of the test subject can precisely be recorded. The test subject wears the Dikablis Head Unit, on which two cameras are installed. One camera is directed towards the test subject’s eye and is used to recognize the subject’s gaze behavior (pupil movements). The other camera is directed straight on in front of the subject and monitors the subject’s environment. By processing both of these pieces of video data, the gaze direction of the test subject can almost precisely be determined. Offline re-calibration and post-processing of the eye-detection can be made with the Dikablis Analysis Software using the recorded data even after the recording software has finished recording. These possibilities to adjust and post-process guarantee clean and always analyzable test results under any circumstances. The D-LAB module contains a package for conducting experiments, from test planning and defining of special Areas of Interest (AOI) visible within the gaze region, to the automatic calculation of glance durations and graphical presentation of these results. Successively, the proceedings to standardized testing and experimentation will sbe described subsequently. Herein the capital and lower case letters are always related to figure 1’s demonstrated procedural (workflow) plan and the relationships between the small angled boxes located within the figure.
Automated Analysis of Eye-Tracking Data
107
2.1 Construction of an Experimental Plan The compliancy to test partitioning presented by ISO/TS 15007-2:2001 into „experimental condition“, „task“, and „subtask“ can be defined into one test plan with DLAB (see figure 2 left). This way a test can be represented in the form of intertwined and nested intervals in which: • „experimental condition“ an entire Experiment is unfolded (Ex. driving on country roads); • “task” defines the interaction between a determined system presented within the Experiment (ex. Operation of Navigation Systems); • A specification of a “task” as “subtask” will be considered. (Ex. Operation of a navigation system via a touch screen display). D-LAB offers the possibility to additionally define „subsubtasks“ as the fourth layer, ex. in order to mark the appearance of critical events presented within the experiment (ex. sharp breaking situations) or automatic analysis of the display screen (ex. an individual input screen within the navigation system, for example inputting a destination address).
Fig. 2. Left: constructing a test sequence; Right: Automatically created block shifting diagram/interface for online test segment sequence marking
2.2 Record Gaze Behavior and Mark the Beginning and End Points of Each Trial Interval Using one of the D-Lab integrated applications one can automatically create a shifting block diagram/interface using the trial definitions, which can be seen on the right side of figure 2. Each shifting block represents a test segment. By pressing a block, a network event is created, which marks the start or ending of a task interval directly in the Recording Software. The functionality of the Dikablis recording software is described
108
C. Lange, M. Wohlfarter, and H. Bubb
in written detail in Lange et al., 2006a und Lange et al., 2006b. These events mark the beginning respectively the end of a trial segment and are synchronously saved to the calculated gaze data. The events can also be started from another data recorder such as a driving simulator. 2.3 Validating Gaze Data and Trial Intervals The Dikablis analysis software is firm in validating the gaze data after a trial and in optimizing. In preparation, the gaze data if necessary is calibrated after the trial so as to optimize offline pupil recognition (see Lange et al., 2006a and Lange et al. 2006b). Further processing in D-LAB follows after the optional jointing is successful. The first move consists of testing whether all trial intervals would be marked correctly. The marked trial intervals located under the gaze player window synchronized to the playtime line shown on the user interface layer (see figure 3). D-LAB also offers a testing function which automatically identifies inconsistencies within the trial segment. The segment can be manually adjusted or changed, tasks can be added or deleted. Figure 3 shows the D-LAB interface for management of a trial interval.
Fig. 3. Validation and post-processing of a trial segment in D-LAB
2.4 Definition of Areas of Interest The areas of interest used in the calculation of glance durations in relevant gaze regions (ex. the navigation system display, the street view, the rear view mirror, the left angle mirror, the dashboard, etc.) as required by the ISO/TS 15007-2:2001 are defined by using D-LAB’s functionality of marking specific AOIs. Thus in D-LAB a random number of AOIs can be defined in the form of a polyline and labeled with names. Figure 4 demonstrates the Head-Up Display and Dashboard as defined AOIs.
Automated Analysis of Eye-Tracking Data
109
Fig. 4. Defining selected AOIs within the gaze region such as the Head-Up Display and the Dashboard
2.5 Automatic Calculation of Glance Metrics to the Defined AOIs In order to automatically calculate glance duration and glance frequency, the pupil of the test subject as well as his/her head position in relation to the test environment (relative to the defined AOIs) must be recognized. The determination of the head position is carried out with the help of the so called marker(s) (see the quadratic black and white object in figure 4), which describe the environment reference. These markers are found using image processing in the photo of the field camera and used in the head position calculation of the test subject. For every defined and established AOI, using the D-LAB function „Calculate Gaze Durations“ it is calculated when the test subject glanced at an AOI and when their glance was away from a certain AOI. The result of this calculation is displayed analogously to the graphical representation of the trial intervals located under the gaze film player window synchronous to the playtime line. Hence, the operator can impose calculation corrections at any time simply by testing and manually correcting if necessary. The gaze specific values from the ISO/TS 15007-2 standard can be calculated using the automated determination of glance durations via the defined AOIs. Thus the operator can input the values which need be calculated for the following gaze experiment in the form of an „Analysis Series“. Therefore the specific values pertaining to specific tasks and AOIs to be calculated must be defined. The following values are available to choose from: • • • • • • • •
Total Glance Time Glance Frequency Time off road-scene-ahead Total glance time as a percentage Fixation probabilities Link value probabilities Maximum Glance Duration Mean Glance Duration
110
C. Lange, M. Wohlfarter, and H. Bubb
D-Lab hereupon calculates the inputted values and saves the results in the form of text file. These automated value calculations can be conducted for all defined AOIs, all terms of the experiment plan as well as for all gaze metrics. The text file is built concurrently so that it can be opened for further statistical analysis using the SPSS statistics program without indirection. 2.6 Graphical Representation of Calculated Metrics Next to the internal calculation of values, D-LAB additionally offers a trial result(s) graphical representation capability. For this purpose several graphical diagrams are displayed which support the interpretation of the result(s). There are two examples in figure 5 that exemplify this. To the top one can find the progressive course of glance duration on defined AOIs for a single subtask presented by four independent test subjects. To the bottom the course of the users mean glance fixation duration in a critical situation is shown in order to allow conclusions on the basis of mental strain.
Fig. 5. Top: Glance duration on the defined AOIs on all trial experiments. Bottom: User glance fixation duration in critical situations.
References 1. ISO/TS 15007-2:2001: Road vehicles - Measurement of driver visual behavior with respect to transport information and control systems - Part 2: Equipment and procedures 2. Lange, C., Yoo, J.-W., Wohlfarter, M., Bubb, H.: Dikablis - Operation mode and evaluation of the human-machine interaction. In: Spring Conference of Ergonomics Society of Korea, Seoul, May 12 (2006) 3. Lange, C., Wohlfarter, M., Bubb, H.: Dikablis - engineering and application area. In: Proceedings IEA 2006 16th World Congress on Ergonomics, Maastricht the Netherlands (2006)
Brain Response to Good and Bad Design Haeinn Lee1, Jungtae Lee2, and Ssanghee Seo2 1
107A Kiehle Visual Arts Center, 720 Fourth Avenue South St. Cloud, MN 56301-4498, USA 2 6409-1 Dept of Computer Science & Engineering, Pusan National University Jangjeon-dong, Geumjeong-gu, Busan, 609-735, Republic of Korea
Abstract. This paper is about the decision of whether good or bad design is the result of the human brain process. Our research team has used the technique of functional MRI and Electroencephalogram (EEG) to address the question of how the brain answers while subjects viewed different designs. Classifying the good or bad designs, subjects chose a mouse button to decide their perception of good or bad design and we analyzed their patterns of EEG rhythms and fMRI. The results of fMRI showed that the perceptions of different feelings of designs are associated with the frontal lobe and the occipital lobe. After analyzing the EEG by the Event-related brain potentials (ERP) method, we also found that the amplitude of ERP components in perception of bad design is greater and latency is shorter than that of good design. Therefore, the human brain responds sooner and stronger in perception of bad feeling. Keywords: Human Behaviors, EEG, fMRI, ERP, Interaction and Interface Design, Usability Test, Brainwork, Visual Brain.
1 Introduction With advances in computer video display technologies such as EEG, fMRI, PET, and SPECT, brain research has become a contemporary issue. Most people thought that the brain is just black box and is not an interesting area to explore, but because of developed new technologies such as EEG, PET, and fMRI, research of the brain revealed how it is structured and worked. [1] The weight of the human brain is about 1400g and contains between 1 billion and 100 billion neuron, and is divided into the brain stem, limbic system, and cortex. The brain stem is the major route by which the forebrain sends information to and receives information from the spinal cord and peripheral nerves. The limbic system is a system of nerves in the brain involving several different areas concerned with basic emotions such as fear and anger, and basic needs such as the need to eat and to have sex. [1] The Cortex forms 90 percent of the brain, and plays a key role in memory, attention, perception, awareness, thought, language, and consciousness. Also this part of the brain is divided into four sections: the occipital lobe, the temporal lobe, the parietal lobe, and the frontal lobe. Functions, such as vision, hearing, and speech, are distributed in these selected regions. Brain research has become a contemporary issue and J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 111–120, 2009. © Springer-Verlag Berlin Heidelberg 2009
112
H. Lee, J. Lee, and S. Seo
has explored diverse aspects of many different major areas such as medical science, engineering, psychology, biotechnology, linguistics, economics, music, etc. This research has provided new insight into underlying disease mechanisms and is beginning to suggest new treatment. Also using the Neuro-Lingustic Programming (NLP) method, researchers understand human behavior and cure the human mind to lead a better life. [2] The brain controls body activities, feels the five senses, and shapes thoughts, hopes, dreams, and imaginations. In short, the brain is what makes people human. [1] Recently, there are lots of brain researches relating to linguistics and music activities. For example, a person hears two different sentences; “He takes coffee with cream” and “He takes coffee with dog” and examined their brain movements. There are certain distinctions between the two sentences. The human brain has a strong reaction when people hear the latter sentence that is not normal. Likewise, there are certain differences of brain movements when people hear natural and unnatural melodies from music. [3,4,5] It can be linked to the art and design areas, so that research can be about how the human brain acts when people do artistic activities, and how people feel when they look at artistic works or designs. If people look at the smile of the Mona Lisa, they usually get a good feeling. On the other hand, if people look at a photo that shows a scene with a terrible car accident, they get a bad feeling. According to this fact, if our researchers can clarify and measure how the brain works when people get a good or bad feeling, we can come up with an objective evaluation of which part of design makes people get a good feeling or bad feeling. Also we can determine the main aspects of a design with good value. We have used the technique of functional MRI and EEG to address the question of how the brain answers while subjects viewed different designs. Twenty-five subjects participated in this research and viewed fifty design images randomly. Classifying the good or bad designs, subjects chose a mouse button to decide their perception of good or bad design. We analyzed their patterns of EEG rhythms and fMRI to find the characteristics of brain activity based on good and bad designs.
2 Brain Structure and Movement 2.1 Brain Structure As I mentioned before, the brain is divided into four sections: the occipital lobe, the temporal lobe, the parietal lobe, and the frontal lobe, and functions such as vision, hearing, and speech, are distributed in these selected regions. The occipital lobe is located in the back of the brain and plays a role in processing visual information. (Fig.1) The parietal lobe plays a role in sensory processes, particularly determining spatial sense and navigation, attention, and language. The frontal lobe has a role in controlling movement and in the planning and coordinating of behavior. The temporal lobe is involved in auditory processing and is home to the primary auditory cortex. [1].
Brain Response to Good and Bad Design
113
Fig. 1. Brain Classifications [1]
2.2 Brain Movement and Technology Functional magnetic resonance imaging (fMRI). People use the MRI, which provides a high quality, three-dimension image of organs and structures inside the body to examine a body structure. But to examine the brain activity people need to use fMRI and this is most popular neuroimaging technique today. This technique compares brain activity under resting and active conditions, and it combines the high spatial resolution, which is a correlate for neuronal activity. This technique allows for more detailed maps of brain areas underlying human mental activities in health and disease. Our team has used the technique of fMRI to find out which part of brain is more activated when subjects viewed good or bad design. [1]. Electroencephalogram (EEG). Many of the recent advances in understanding the brain are due to the development of techniques that allow scientists to directly monitor neurons throughout the body. Electroencephalogram is the recording of electrical activity along the scalp produced by the firing of neurons within the brain. In this method, electrodes placed in specific parts of the brain, which vary depending on which sensory system is being tested-make recordings that are then processed by a computer. [1]. EEG has several strong sides as a tool of exploring brain activity; for example, its time resolution is very high (on the level of a single millisecond). Other methods looking at brain activity, such as PET and fMRI have time resolution between seconds and minutes and it measures the brain’s electrical activity directly. EEG can be used simultaneously with fMRI so that high-temporal-resolution data can be recorded at the same time as high-spatial-resolution data. [6]. Our team has used the technique of EEG (Electroencephalogram) to address the question of how the brain answers while subjects viewed images. Most important thing to use EEG is which part of brain we need to place electrodes to measure the brain waves. Electrode location and names are specified by the 10-20 system for most research applications. This system is an internationally recognized method to describe and apply the location of electrodes in the context of an EEG test. [6] (Fig.2).
114
H. Lee, J. Lee, and S. Seo
Fig. 2. Electrodes location to check the EEG (10-20 System) [8]
Event-related potential (ERP). During the experiment, we used event-related brain potentials (ERP), which is a method in checking an electroencephalogram in certain moment when subjects received the event like viewed designs. ERP is any measured brain response that is directly the result of a thought or perception. It can be reliably measured using the EEG, a procedure that measures electrical activity of the brain through the skull and scalp. [7]. There are two important components in ERP waveform, which are P300 and N400. The N400 ERP component is described as a negative voltage deflection occurring approximately 400ms after stimulus onset, where as P300 component describes as a positive voltage deflection 300ms after stimulus onset. The presence, magnitude, topography and time of this signal are often used as metrics of cognitive function in decision making processes. While the neural substrates of this ERP still remain hazy, the reproducibility of this signal makes it a common choice for related researches. [6].
3 Brain Response to Good and Bad Design Our team has used the technique of functional MRI and EEG to address the question of how the brain answers while subjects viewed different designs. Twenty-five subjects participated in this research and viewed fifty design images randomly. First, as I am a professional graphic designer, I subjectively chose twenty-five designs that evoked good feelings and twenty-five that evoked bad feelings. (Fig.3) Because the author’s judgment of good or bad designs was purely subjective, the individual subjects were asked to categorize each design as good or bad. As they viewed each image, they either clicked the right mouse button to classify the image as good, or the left mouse button to classify the image as bad. As they did this, we analyzed their patterns of EEG rhythms and fMRI.
Brain Response to Good and Bad Design
Fig. 3. Design image examples for experiment
Fig. 4. The fMRI response of good design
115
116
H. Lee, J. Lee, and S. Seo
3.1 The Results of the fMRI Based on Good and Bad Design Our team used fMRI machine (ISOL fORTE) in the Brain Science Research Center at Korea Advanced Institute of Science and Technology (KAIST) and shooting variable is as follows: TR=3000ms, TE=35ms, Number of Slice=25, FOV=24cm, Image natrix=64x64, Slice Thickness=5mm. The scenario of the experiment is as follows: 1. Shows “+” on a screen for 3seconds. 2. Shows good and bad design randomly for 3seconds. 3. Shows “+” on a screen for 3seconds again. We kept repeating this process to show all fifty designs. At the same time, subjects chose a mouse button to decide their perception of good or bad designs. Fig 4 presents the part of brain that showed activity when one subject saw a good design. The top left brain image shows the midsagittal section, the right image is showing the coronal section, and the bottom image shows a horizontal section. The left part of the midsagittal is the location of the occipital lobe and the right part is the location of the frontal lobe. The top part of horizontal section is the right part of midsagittal section and bottom part is the left part of midsagittal section.
Fig. 5. The fMRI response of bad design
Brain Response to Good and Bad Design
117
The bottom table of fig4 shows the coordinates of where the brain was activated. Using these coordinates we can indicate the broadmann area in the activated brain areas. After observing the broadmann area, the most activated areas were points 19,7,11,and 6. These areas are the occipital lobe, the parietal lobe, the prefrontal lobe, and the frontal lobe. The decisions to judge images either good or bad are made in the occipital lobe and the frontal lobe. The broadmann 7 area, which controls visualmotor coordination, is located in the parietal lobe, but it is close to the occipital lobe. Fig 5 presents the part of brain that showed activity when one subject saw a bad design. When subjects judged the design as bad, their brain was activated in broadmann area 18, 19,7,11,6, and 20 and these areas are location of the occipital lobe, parietal lobe, prefrontal lobe, frontal lobe, and temporal lobe. The broadmann 20 area, which controls the high-level visual processing and recognition, is located in the temporal lobe. Similarly, The occipital lobe, frontal lobe, and parietal lobe were activated while subjects looked a bad design. We noticed that there were differences in specific activated areas, for example, the brain action when subjects had a bad feeling was much stronger than when subjects had a good feeling. Also, there was little difference in brain action between the left and right brain when the subject perceived the design as good, but left-brain action was strong when subjects perceived the design as bad. 3.2 The Results of the EEG Based on Good and Bad Design Based on the fMRI result, our researchers decided to attach electrodes in the area of the occipital and the frontal lobe, which is Fp1, Fp2, O1, and O2 from 10-20 system, and examine the EEG in the areas. (See Fig.2.) Twenty-five subjects participated this research and the scenario of the experiment is as follows: 1.Shows “+” on a computer screen for 1minute to be a steady state of brain. 2. Shows good and bad design randomly for 3seconds. 3. Subjects choose a mouse button to decide their perception of good or bad designs. 4. Shows “+” on a computer screen for 1minute to be a steady state of brain again. We kept repeating this process to finish all fifty designs. During the experiment, we used event-related brain potentials (ERP), to check EEG in certain moment when subjects received the different feelings. Fig6 image shows the amplitude of the occipital lobe when the subjects perceived the design as good or bad. The latency of P300 component from channel O1 and O2 when subjects had a bad feeling appeared faster than when subjects had a good feeling. Also the amplitude of the perception of bad feeling is higher than the amplitude of the perception of good feeling. Fig 7 shows the result of difference of latency based on subjects’ perception of good or bad design. From this figure, the x-axis represents the result of the latency (G) of a bad design subtracts from the latency (B) of a bad design, and the y-axis is the subjects’ number. Most points placed in plus area which means the latency (G) of a good design is longer than the latency (B) of a bad design. Therefore, the human brain responds sooner and stronger action when subjects perceived the design as bad. We also examined the latency of the frontal lobe when the subjects perceived the design as good or bad, and the result is similar with the occipital lobe. As I mentioned before, the human brain responds sooner to in perception of bad feeling. However, these results are the mean value of subjects, there are people responded sooner in perception of good feeling.
118
H. Lee, J. Lee, and S. Seo
Fig. 6. Comparison of the occipital lobe characteristic
Fig. 7. Comparison of the occipital lobe latency
4 Conclusion With advances in computer video display technologies such as EEG, fMRI, PET, and SPECT, brain research has become a contemporary issue. Based on the issue, many different major areas such as Medical Science, Engineering, Psychology, Linguistics,
Brain Response to Good and Bad Design
119
Biotechnology, Economics and Music explored diverse aspects of brain research. Hence, there should be lots of possibilities to study brain research relative to Art and Design. When one makes the decision that something is good or bad his/her decision is a result of the human brain process. It is obvious that the brain works differently depending on a good feeling or a bad feeling. In other words, the brain will answer different ways when people look at good design or bad design. Although there would be a difference in determining design values, it depends on the individual. In the case of a masterpiece, most people admit its artistic value. According to this fact, if our researchers clarify how the brain works when people look at good and bad design, we come up with an objective evaluation of how people judge the design value. Based on the result, we also determine what can be the main aspects of a design with good value. We have used the technique of functional MRI and EEG to address the question of how the brain answers while subjects viewed different designs. The results of fMRI show that the perceptions of different feeling for designs are associated with the frontal lobe and the occipital Lobe. The occipital lobe is placed in back of the brain part and has a visual cortex so controls with the organ of vision. The frontal lobe is placed in front of the brain part and takes care of complicated functions such as thinking, planning, and deciding. Based on the fMRI result, our researchers decided to attach electrodes in the area of the occipital and the frontal lobe and examine the EEG in the areas. During the experiment, we used the ERP, which is a method in checking the EEG in certain moment when subjects received the event like viewed designs. After analyzed the EEG by the ERP method, we also found that the amplitude of ERP components in perception of bad design was higher and latency was shorter than that of good design. From all examinations, we found there is a significant distinction between an individual because of a characteristic of brain. Some people have strong action in leftbrain and some are not. But in general, the human brain responds sooner and stronger to action in perception of bad feeling. Thus, we assume it can be an objective fact to determine good value of design when brain responds slower and exhibit a long latency. Considering this issue, we also can determine what aspect that makes good value of design and apply as a solution to create a better design. In the future, the research includes not only pure design but also can be applied into the human interface design. One of the most important issues in the human interface design is a usability test to grasp a person’s individual feeling and measure the emotional satisfaction. The idea of this paper will show as a great method of usability test to classifies the good or bad interface design as well as link with ease of use. Acknowledgments. I want to give a special thank you to my research team and family: my father, mother, and brother for their constant and unconditional love, encouragement, and support. I am deeply indebted to them and dedicate this study to them.
References 1. Society for Neuroscience: Brain Facts, A primer on the Brain and Nervous System (2005), http://www.sfn.org 2. Mark, F., Barry, W., Michael, A.: Neuroscience: Exploring the Brain, 3rd edn. Lippincott Williams & Wikins (2006)
120
H. Lee, J. Lee, and S. Seo
3. Lee, J.Y.: Neurophysiology and Brain-imaging study of Music-music& language, music & emotion. Nangman Music Magazine 18(3) (2006) 4. West, W.C., Rourke, T., Holcomb, P.J.: Event-Related Brain Potentials and Language Comprehension: A Cognitive Neuroscience Approach to the Study of Intellectual Functioning, Tufts University (1998) 5. Lu, H., Wang, M., Yu, H.: EEG Model and Location in Brain when Enjoying Music. In: Proceedings of the 2005 IEEE Engineering in Medicine and Biology, Shanghai, China, pp. 2695–2698 (2005) 6. Wikipedia, encyclopedia: http://en.wikipedia.org/wiki/EEG 7. Coles, Michael, G.H., Rugg, M.D.: Event-related brain potentials: an introduction, Electrophysiology of Mind, Oxford scholarship Online Monographs, pp. 1–27 (1996) 8. Kim, D.S.: Electroencephalogram, Korea Medical (2001)
An Analysis of Eye Movements during Browsing Multiple Search Results Pages Yuko Matsuda, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology 8916-5,Takayama, Ikoma, Nara, Japan {yuko-m,hideta-u,masao,matumoto}@is.nasit.jp
Abstract. In general, most search engines display a certain number of search results on a search results page at one time, separating the entire search results into multiple search results pages. Therefore, lower ranked results (e.g., 11thranked result) may be displayed on the top area of the next (second) page and might be more likely to be browsed by users, rather than results displayed on the bottom of the previous (first) results page. To better understand users' activities in web search, it is necessary to analyze the effect of display positions of search results while browsing multiple search results pages. In this paper, we present the results of our analysis of users' eye movements. We have conducted an experiment to measure eye movements during web search and analyzed how long users spend to view each search result. From the analysis results, we have found that search results displayed on the top of the latter page were viewed for a longer time than those displayed on the bottom of the former page. Keywords: Eye tracking, Web search, User activity, Search results page.
1 Introduction Web search engines are designed to search for useful information on the World Wide Web. Each search engine uses different algorithms to rank web pages, but their interfaces are similar to each other; that is, users type some words into a query box and receive a rank-ordered list of web pages that is relevant to the words. Better understanding user activities would provide us insights into improvements of interactions during web search and the usefulness of the interfaces for search engines. Up to now, many studies have reported analysis results of users’ web search activities based on eye tracking, which is well-known as an effective means to understand what users are looking for during web search. One of the studies [2] showed that users tend to spend most time browsing a few, top results on a search results page while they spend less time browsing the bottom of the page. The study concluded users strongly rely on the correctness of a ranked order presented by a search engine. As with the above study, most previous studies also have focused on user interactions on the first page of the entire search results. However, many users browse multiple results pages in using a search engine [3]. Neglecting this fact might lead to an incomplete understanding of user activities in web search. When a number of search J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 121–130, 2009. © Springer-Verlag Berlin Heidelberg 2009
122
Y. Matsuda et al.
results exist, most search engines separate the entire results into multiple search results pages and display one results page at a time (e.g., 10 search results on a page. The numbers of results displayed on one page depends on user preference.) In this case, lower ranked results (e.g., 11th result) may be displayed on the top area of the next page and might be more likely to be browsed by users, rather than results displayed on the bottom of the previous (first) results page. Therefore, it is important to analyze and understand user activities in searching multiple search results pages in order to provide users with a means or new interface that enables them to more naturally browse search results in a ranked order recommended by a search engine. In this paper, we have conducted an experiment to observe effects of search result positions on user activities in web search. We measured eye movements during web search tasks and analyzed them to understand how long users spend to browse each result.
2 Related Work There have been many studies on user activities in web search. A popular approach to analyzing user activities in web search is to use data of browsing histories or access logs [6][7][8]. Although using history data helps us understand a user’s access paths or interests in a specific web page, it cannot be used for analyzing a user’s attention to search results during web search and influences of displayed positions of search results on web search activities. Another approach is to use eye tracking instruments to capture user activities based on eye movements. Cutrell, et al. [1] used eye tracking to analyze the influence of the length of a site summary (a snippet text) presented in web search results on a user’s search activities. The results of their experiments indicated that a long site summary has an effect of decreasing search time and increasing a user’s search correctness in informational search tasks where users are trying to find a specific web site or homepage. In contrast to informational search tasks, in navigational search tasks where users are trying to find web pages that include some kind of information, it has an effect on the increase of search time and the decrease of user’s search correctness. Guan et al. [2] also measured eye movements and then analyzed the influence of positions of target results that users are looking for. They found that users took longer to search the target results and were less successful in finding the targets when the search targets were placed in low positions on a search results page. Although the study provides useful insights into the design of a new interface for web search engines, it only focused on the first search results page. So it still fails to capture user activities in browsing multiple search results pages. Lorigo et al. [9] analyzed differences of task types in web search. They reported that users performing informational search tasks took longer to complete those tasks than navigational ones and spent more time to stay at pages linked by search results. However, the study also did not focus on time consumption for each search result and user interactions with multiple search results pages. In this paper, using multiple search results pages and two types of tasks (i.e., informational and navigational tasks), we measure and analyze users’ eye movements for each set of search results to provide a new understanding of user activities in web search.
An Analysis of Eye Movements during Browsing Multiple Search Results Pages
123
3 Experiment 3.1 Overview To observe how users look at each set of search results during browsing multiple search results pages, we analyzed total time of eye movements on each search result. In the experiment, participants were asked to search for appropriate web pages (target results) from the search results pages of Google to find particular information with predetermined words. The information and words were specified by experimenters. The experimenters measured eye movements of the participants during the tasks. To analyze user’s eye movements helps us understand how users browse search results. In the experiment, WebTracer [10] was used as an eye tracking system. WebTracer allows us to collect and analyze data of a user’s eye movements and operations (e.g., mouse and keyboard operations) during web search. After the tasks, participants answered a questionnaire about their usual search activities and were interviewed about observed interactions in the tasks. Participants of the experiment were 21 undergraduate students studying information science. All participants used web search in daily life and used Google as their main search engine. 3.2 Apparatus In the experiment, the following equipment was used. • Display: 21-inch LCD monitor (Viewable screen size: H30 x W40cm, Resolution 1,024 x 768 pixels) • Distance from subject’s face to display: approx. 50cm • Device for measurement of sight-line: NAC, EMR-NC (View angle: 0.28 degrees, resolution on the screen: approx. 2.4mm) • Recording and playing of sight-line data: WebTracer 3.3 Task The tasks performed by participants were (1) to search for appropriate web pages linked by search results from search results pages, (2) to find particular information specified by the experimenters and (3) to bookmark the target pages.
Fig. 1. Example of Rearrangement Search Results
124
Y. Matsuda et al.
The time limit of each task was ten minutes whether participants could complete the task or not. In the experiment, participants needed to use predetermined words and they were prohibited from changing search words during the tasks. Since the purpose of this experiment was to observe user’s activities in using multiple web search results pages, participants were only permitted to move to web pages linked by Google’s search results. The order of the tasks was counterbalanced to consider the learning effect. The tasks themselves were based on the test collection provided by NTCIR (NTCIR4 WEB) [11][12]. In this experiment, the two types of tasks were selected as follows. • Informational Task: required participants to find specific information (e.g., web pages including information on university entrance exams). The task was completed by finding three web pages linked by target results and bookmarking them. • Navigational Task: required participants to find specific web pages (e.g., official web page of the university). The task was completed by finding a web page linked by a target result and bookmarking it. Each participant performed ten tasks (five tasks for each task type). 3.4 Design of Web Search Results Pages To prevent the bias effect of the numbers of target results and their positions, we modified the results pages that were saved in a local computer when we searched with Google. The participants performed the search tasks with the modified search results pages. The previous study showed users search about 2.35 pages [3]. Therefore, we prepared three search results pages and allocated target results randomly on the search results pages. Advertisements and information irrelevant for web search were removed. We used Google’s default setting in which 10 search results are displayed at a time. In addition, Google’s original (unmodified) search results pages were used for fourth or later search results pages. Note that each search result and display position of search results followed Google’s page rank. To prevent participants from finishing their search only on the first page, we allocated target results on the second or third page. Figure 1 is an example of a way of inserting target search results.
Fig. 2. Inserted Positions of Target Search Results
An Analysis of Eye Movements during Browsing Multiple Search Results Pages
125
For the design of search results pages, we prepared four rearrangement patterns of search results (Figure 2). In the Informational Task, we displayed target results in top (I-1), middle (I-2), bottom (I-3) and even (I-4) in second and third search results page. In Navigational Tasks, we displayed target results in the top (N-1, N-3) and bottom (N-2, N-4) on the second and third search results pages. Since participants may notice the experimenters’ intention (i.e., target results were displayed only after the second page), we insert a dummy task (original search results of Google) into each task. 3.5 Experimental Procedure 1. Explanation of the experiment and preparation: Experimenters explained the experiment and the eye tracking system to participants. 2. Configuration of the eye tracking system: We configured devices for measurement of sight line and checked sight line (calibration). 3. Task for practice: To understand the flow of the experiment, participants practiced a task. The task was an Informational Task. Original search results pages of Google were used. 4. Performing tasks: Experimenters explained each search task and participants started to search. This was repeated until all the tasks were finished. 5. Questionnaire: At the end of the experiment, participants were asked to answer a questionnaire about their daily use of web search engines. 6. Interview: Participants were also asked to answer an interview about observed characteristic activities during the tasks.
4 Results 4.1 Eye Movement Figure 3 shows the example of eye movements gathered in the experiment. The vertical axis shows the position of search results and the horizontal axis shows the time of appeared sight line. In the Figure, horizontal line describes eye movements on search results and the circle shows user click to the search result. The figure showed this user searched the results from top to bottom. The total time of eye movements on the clicked result was longer than other results. 0
Eye Movements Mouse Click
)k nar 10 (lt us eR hrc ae 20 S 30 0:03
0:37
1:13
1:57 2:14 2:41 3:41 Search Time(min:sec)
4:01
4:16
Fig. 3. Eye Movement and Clicked Search Result during the Task
5:05
126
Y. Matsuda et al. Table 1. Classification of Search Completion Pages
Group
Last Search Result
Last Search Results Page
Number of Task
G0
~5
Less than 1
Informational 6
Navigational 5
G1
6~15
1
18
21
G2
16~25
2
33
38
G3
26~
Over 3
27
20
4.2 Analytic Procedure To calculate the total time of eye movements, we separated the tasks by search completion pages. The search completion page is calculated from the position of the lowest search result looked at by each subject. Table 1 shows the classification of search completion pages by group. In this paper, we would like to analyze users’ activities searching multiple search results pages. Hence, we analyze the tasks which finish at the second page (G2) and after the third page (G3). We use the length of time to analyze the eye movements. Even if a users’ gaze appears at a certain search result, it does not necessarily mean that the reviewer has interest that line. Hence, we have to distinguish a focus (i.e., interest) from users' eye movements. In this paper, we defined focus as the eye remaining on a certain search result more than 100ms. To increase correctness of the analysis, we also remove eye movements that stay a long time on a particular position. When the user reads a search result intensively, the time of eye movements to the results greatly increases. However, this increase is not the effect of display position but the effect of the content of the result itself, that is, title of the web page, snippet (description of the web page), and URL. To distinguish a user’s intensive reading, the average reading time of clicked search results was adopted. Basically, users read the results before the clicking, to decide to move to the web page. Hence, it is reasonable to remove the eye movements to search results that stay more than the average time of clicked search results. In this experiment, the average time in the Informational Task was 3.18 seconds, and 2.44 seconds for the Navigational Task. 4.3 Analysis Result Figures 4 and 5 describe the mean time of eye movements on each search results classified as G2 and G3, respectively. The vertical axis shows the mean time of eye movements and the horizontal axis shows the rank of results. In both groups, users tend to view search results longer in informational tasks than navigational tasks. To evaluate effects of search result positions on user activities in web search, we calculated the total time of eye movements on top search results and bottom search results. Table 2 shows the average time of eye movements for the three results that were ranked high and displayed at bottom of the page (HB), and for the three results that were ranked low and displayed at top of the page (LT) in G2. Table 3 shows the average time of eye movements for the three HB results and that for the three LT results in G3. The table describes that the mean time of eye movements to LT is
An Analysis of Eye Movements during Browsing Multiple Search Results Pages
yeE fo e im T
)c 3 es 2.5 ( 2 st ne 1.5 1 em vo 0.5 M 0
Informational Task
1
3
5
7
9
11
13 15 17 19 Search Result (rank)
21
127
Navigational Task
23
25
27
29
Fig. 4. Mean Time of Eye Movements on each Search Result in Informational Task (square) and Navigational Task (triangle) of G2
yeE fo em iT
c)e 3 s( 2.5 st 2 ne 1.5 m ev 0.51 oM 0
Informational Task
1
3
5
7
9
11
13 15 17 19 Search Result (rank)
21
Navigational Task
23
25
27
29
Fig. 5. Mean Time of Eye Movements on each Search Result in Informational Task (square) and Navigational Task (triangle) of G3 Table 2. Mean Time of Eye Movements for High-Rank Results Displayed in Bottom Area 3 and Lower-Rank Results Displayed in Top Area 3 in G2
Group
Search Results
LT_1 HB_1 LT_2 HB_2
1~3 8~10 11~13 18~20
Mean Time of Eye Movements (sec) Informational Navigational 1.67 1.24 1.46 1.04 1.51 0.96 0.66 0.79
Table 3. Mean Time of Eye Movements for High-Rank Results Displayed in Bottom Area 3 and Lower-Rank Results Displayed in Top Area 3 in G3
Group
Search Results
LT_1 HB_1 LT_2 HB_2 LT_3 HB_3
1~3 8~10 11~13 18~20 21~23 28~30
Mean Time of Eye Movements (sec) Informational 1.38 1.56 1.43 1.25 1.48 0.75
Navigational 0.95 0.88 0.84 0.83 0.95 0.61
almost the same as HB, or longer than HB in some cases (e.g. between HB_2 and LT_3 at G3). This result indicated users did not view the search results in proportion with page rank.
128
Y. Matsuda et al.
In particular, we focused on the top-three search results (LT) of each page (see Figures 4 and 5). In Figure 4 (eye movements of G2), the mean time of eye movements to the first result of each page (ranks 1 and 11) was shorter than the second and third results (ranks 2, 3, 12, and 13) at both task types. Also at G3 (Figure 5), eye movements to the first result of the page were shorter than the second and third search results.
5 Discussion 5.1 Effect of Task Differences The result of the experiment describes that users tend to view search results longer in informational tasks than navigational tasks. In the Navigational task, users read the snippet of the search result, then decide to click the search result or not. On the other hand, in the Navigational Task, users read the title and URL of the result instead of the snippet to decide to click the result. Reading the title and URL of the result requires less time than reading the snippet, therefore the length of the eye movements in Informational Tasks is longer than Navigational Tasks. The result suggests that when users browse multiple web search results pages, they adopt different reading patterns for each task. 5.2 Effect of Position within a Result Page/Screen The result of the experiment describes that time of eye movements on LT is longer than HB. The result shows that users are impressed not only by the rank but also the position of the search results within the results page. That is, the time length of eye movements on the search results is influenced by the position within a results page. The detailed analysis showed the time of eye movements on second and third results in each page are longer than the first result. This result suggests users’ eye movements are attracted by the position on the screen. In the experiment, users viewed the middle area of the screen more than the top area. The second and third results are displayed on the middle of the screen when the user went to the search results page. On the other hand, the first result is displayed on the top of the screen. The first result is turned out by the scroll, hence, the user interest moved to the second/third results. This assumption was verified from interviews of the subjects. 5.3 Design Implications Using the results of the experiment, we propose a design encouraging users to browse the search results based on the rank. To increase the time of the eye movements of the users, the results displayed on the bottom of the page should be emphasized to get more attention from the users. In the Navigational Task, users concentrated their eye movements on the title of the web page or URL since the result page was a goal of the task. Hence, a thumbnail and/or attribute (e.g. the official page or blog) of the Web page are useful information for the users searching a specific web site.
An Analysis of Eye Movements during Browsing Multiple Search Results Pages
129
6 Conclusion In this paper, we experimentally analyzed the effect of the result position on the results pages. In the experiment, we measured users’ eye movements during web search tasks to analyze how long users spend on each result of the results pages. As a result, we found the results displayed on the bottom of the page were viewed for a shorter time than the results displayed on the top of the next page. Also, we found the tendency that the second/third results of each page were viewed longer than the first result of the results page. As a future work, we will analyze the effect of the display position on the screen. Acknowledgments. We would like to thank all the participants. In this paper, we used a part of the data of the NTCIR-4 WEB task that is sponsored by the National Institute of Informatics, as organizers of the NTCIR-4 WEB task project.
References 1. Cutrell, E., Guan, Z.: What Are You Looking For?: An Eye-tracking Study of Information Usage in Web Search. In: CHI 2007: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 407–416 (2007) 2. Guan, Z., Cutrell, E.: An Eye Tracking Study of the Effect of Target Rank on Web Search. In: CHI 2007: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 417–420 (2007) 3. Jansen, B.J., Spink, A., Saracevic, T.: Real life, real users, and real needs: a study and analysis of user queries on the web. Inf. Process. Manage. 36(2), 207–227 (2000) 4. Broder, A.: A taxonomy of web search. SIGIR Forum 36(2), 3–10 (2002) 5. Rose, D.E., Levinson, D.: Understanding User Goals in Web Search. In: WWW 2004: Proceedings of the 13th international conference on World Wide Web, pp. 13–19 (2004) 6. Murata, T., Saito, K.: Extraction and Visualization of Web Users’ Interests Using SiteKeyword Graphs. Journal of Japan Society for Fuzzy Theory and Intelligent Informatics 18(5), 701–715 (2006) (in Japanese) 7. Clarke, C.L.A., Pan, B., Agichtein, E., Dumais, S., White, R.W.: The Influence of Caption Features on Clickthrough Patterns in Web Search. In: SIGIR 2007: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 135–142 (2007) 8. Otsuka, S., Toyoda, M., Kitsuregawa, M.: A Study for Analysis of Web Access Logs with Web Communities. Transactions of Information Processing Society of Japan 44(18), 32– 44 (2003) (in Japanese) 9. Lorigo, L., Pan, B., Hembrooke, H., Joachims, T., Granka, L., Gay, G.: The influence of task and gender on search and evaluation behavior using Google. Inf. Process. Manage. 42(4), 1123–1131 (2006) 10. Sakai, M., Nakamichi, N., Shima, K., Nakamura, M., Matsumoto, K.: WebTracer: A New Web Usability Evaluation Environment Using Gazing Point Information. Transactions of Information Processing Society of Japan 44(11), 2575–2586 (2003) (in Japanese)
130
Y. Matsuda et al.
11. Eguchi, K., Oyama, K., Aizawa, A., Ishikawa, H.: Overview of the Informational Retrieval Task at NTCIR-4 WEB. In: Proceedings of the Fourth NTCIR Workshop on Research in Information Access Technologies Information Retrieval, Question Answering and Summarization (2004) 12. Oyama, K., Eguchi, K., Ishikawa, H., Aizawa, A.: Overview of the NTCIR-4 WEB Navigational Retrieval Task 1. In: Proceedings of the Fourth NTCIR Workshop on Research in Information Access Technologies Information Retrieval, Question Answering and Summarization (2004)
Development of Estimation System for Concentrate Situation Using Acceleration Sensor Masashi Okubo and Aya Fujimura Doshisha University, 1-3 Miyakodani, Tatara, Kyotanabe, Kyoto, 610-0321, Japan
[email protected],
[email protected]
Abstract. Recently, to discipline to increase powers of concentration is popular. One of the reason, it is difficult to concentrate something in these days because of a flood of information. However we discipline our concentration by using the how-to books and the portable games, we cannot evaluate the training effect on the practical life. In this paper, we propose an evaluation system for user’s powers of concentration in which the method for the estimate user’s sitting situation is utilized. This system is constructed by two kinds of method, one is the method which estimates the sitting situation and the other is the evaluation method for user’s powers of concentration situation. These methods use user’s motion that is obtained from the acceleration sensor that is fixed on the chair. And we prepare the three kinds of Graphical User Interface (GUI) which presents the concentration situation to the user. Keywords: Powers of concentration, GUI, Sensory evaluation and Selfmanagement.
1 Introduction The modern society can be described as information society because of a flood of miscellaneous information. It is getting difficult to handle with these information obtained by Internet and e-mail as well as newspapers and televisions with full understanding. We have a difficulty to deliberate and concentrate something in our life environments. On the other hand, since the launch of game software aiming at brain activation, commonly known as ‘Notore’, which means brain training, various games to train brains, how-to books for improving our concentration are becoming popular among wide range of age groups. One of main factors of this phenomenon is attributed to middle-aged and senior people’s demands for prevention of brain aging and wide-range age groups’ desires for obtaining abilities to deal with a large number of information in a short time. Since these trainings are mostly performed by recitations and simple calculations, users can improve the ability as though they were playing some kind of games. Moreover the results of the trainings can be described as ‘age of brain’ so that they can easily check their abilities. This is one of the reasons that ‘brain training’ software are widely accepted among various age groups. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 131–140, 2009. © Springer-Verlag Berlin Heidelberg 2009
132
M. Okubo and A. Fujimura
However, even our thinking ability and concentration powers can be improved in such games and trainings, we cannot evaluate the effects in the practical life. Currently, the method using biosignals including breathing, heartbeat, and brain waves…etc. is proposed in order to measure person’s powers of concentration [1]. For example, it is found that a brain wave called Frontal Midline theta is generated when solving a problem and concentrating on abstract thinking. Also it has been researched that transition of learner’s skin potential level meets his interest in lectures. However, this sort of method put a heavy burden on a measured person as well as it is costly and time-consuming. Therefore in this study, assuming the situation as if users are working with a computer, studying and doing some paper works in sitting situation, we propose an evaluation system for user’s powers of concentration by processing information obtained by motion sensor fixed on a chair and presenting the concentration situation to the user. Along with development of the actual system, we also prepare the three kinds of Graphical User Interface (GUI) and perform the sensory evaluation experiments for the proposed system and the interface evaluation using these GUI.
2 Hardware Configuration of Estimation System in Sitting Situation Hardware configuration of proposed system is showed as Fig. 1. The motion sensor sends acceleration data to Personal Computer (PC) through Bluetooth communication after measuring a movement of a chair. User’s sitting or standing up situation and his power of concentration are estimated on PC according to the received data, and the estimation result is presented to the user. Monitor
User Bluetooth communication Acceleration sensor
PC
Chair
Fig. 1. System configuration
In the proposed system, the user’s sitting situation on a revolving chair that is generally used in households and offices as shown in Fig. 2 is estimated. Since the revolving chair reflects the various users’ movements, we can estimate the user’s movements by measuring the acceleration of the movements and swings using the motion sensor.
Development of Estimation System for Concentrate Situation
133
We use a remote controller of Nintendo Wii games as the motion sensor. This remote controller uses Bluetooth for connecting to the main unit so that we can easily connect to PC. Since Wii game is available at a relatively low cost, it has already been popular among many households. Fig.2 shows that we fix the remote controller on the upper side of the backrest of the chair because the inclination of the backrest and the revolution of seating face appear prominently there and yet it doesn’t disturb the user. The three dimensional motion sensor is installed near A button located around the center of the remote controller surface. Fig.2 also shows the coordinate axes with putting it lengthwise: X axis is horizontal direction, Y axis is vertical direction and Z axis is depth direction [2].
Y X Z
Fig. 2. Coordinate of 3-axis accelerometers
3 Software Configuration of Estimation System in Sitting Posture 3.1 Estimation of Time of Sitting and Leaving This proposed system estimates if the user is sitting down or standing up and user’s powers of concentration in sitting situation. At first we propose the method for estimating if the user is sitting down or standing up. There are the following two kinds of methods for estimating user’s situation. The first one is a method for estimation by momentary change of acceleration at the time of sitting down and at the time of standing up. The second one is a method for estimation by the change of acceleration along with time. In the proposed system, we use these two methods to ensure the reliability. Estimation by Momentary Change of Acceleration Speed. We sample the acceleration around 100Hz from Wii remote controller via Bluetooth. We instruct several subjects to sit down, work and stand up the chairs with the remote controller fixed on and perform the preparatory experiment to measure the acceleration of these movements. As a result, we find that the accelerations significantly change at the time of sitting down and standing up in the all axes directions. However, we also find that relatively greater acceleration is measured in X-axis and Z-axis at the time of sitting down. Therefore we estimate whether sitting down or standing up by Y-axis acceleration which shows less change. Moreover we focus on momentary change of
134
M. Okubo and A. Fujimura
acceleration at the time of sitting and standing up, respectively. Fig. 3 shows an example of transition of Y-axis acceleration at the time of sitting down and standing up. It shows that the acceleration shifts to a negative direction after shifting greatly to a positive direction. This occurs due to the rise of the seat front as a reaction when the subject is being seated and the seat front sinks down once. On contrary, the acceleration shifts to a positive direction when standing up after shifting greatly to a negative direction. This occurs due that the seat front sinks down as a reaction when the subject is standing up and the seat front rises, which means that the Y-axis acceleration shifts from a positive direction to a negative direction when sitting down and shifts from a negative direction to a positive direction when standing up. Moreover, these shifts of the acceleration such as positive to negative by sitting down and negative to positive by standing up occur within 50ms. Using these shift patterns of the acceleration, we estimate the subjects’ sitting down and standing up situations. Acceleration G(×9.8m/s2)
Time(ms) (a) When sitting down
Time(ms) (b) When standing up
Fig. 3. Transition of the Y-axis acceleration by (a) sitting down and (b)standing up
Now we perform the preparatory experiment for setting the threshold value against the acceleration in order to distinguish the movements among sitting down, standing up and working in sitting situation. Among the nine subjects, maximum value of 0.27G is measured by two subjects and minimum value of -0.23G is measured by another subject. Therefore we set the threshold value of the acceleration to estimate sitting down and standing up as 0.3G in the positive direction and -0.3G in the negative direction. We summarize the estimation method for sitting down and standing up using the momentary acceleration change. At first if more than 0.3G acceleration is measured, and if less than -0.3G is measured within 50ms during standing up, we estimate the subject is sitting down. Also if less than -0.3G acceleration is measured during sitting down and if more than 0.3G acceleration within 50ms is measured during sitting down, we estimate the subject is standing up. Estimation Using the Acceleration Changes with Time Course. The acceleration measured at the time of sitting down and standing up varies with each individual. Sometimes it is difficult to estimate if the acceleration occurred by sitting down and standing up or movements generated during sitting down. According to the maximum and minimum value of Y-axis acceleration when sitting down and standing up, the
Development of Estimation System for Concentrate Situation
135
acceleration of one subject shows less change when sitting down; at 0.11G and -0.19G. If the acceleration is more than 0.1G and less than 0.3G, and if it is less than -0.1G and more than -0.3G when sitting down and standing up, the estimation will be inaccurate simply by using the momentary acceleration. Therefore, we also use the estimation method using the power spectrum sequentially obtained by sampling the acceleration in a certain period of time. At first we seek the power spectrum by executing Fourier transform of 256 pcs sample data among about 2.5s acceleration in the three axes. The frequency component 0.01Hz being regarded as noise is eliminated. We can find that the power spectrum is barely measured in the all frequency domains when absence but is measured in many frequency domains during sitting. Then we add up the value of power spectrum except the spectrum at 0.01Hz frequency. We will judge between absence and sitting by sum of the power spectrum obtained as above. Fig. 4 shows the transition of sum of 3-axis acceleration power spectrum when absence, sitting, and working by using the acceleration data obtained by preparatory experiment Sitting Not available down
Working
Sum total of diff of acceleration
Rotation of the seat
Inclining the rear
Standing up
of
X-axis Y-axis Z-axis
Time(s)
Fig. 4. Transition of sum of 3-axis acceleration power spectrum
Comparing between absence and working in Fig.4, we find that the sum of Y axis accelerations power spectrum does not change very much comparing to that of other axes. On the other hand, the sum of X and Z axis acceleration power spectrum changes greatly depending on the user’s motion. For example, the sum of X axis acceleration power spectrum increases by the user’s motion like rotating the seat front and the sum of Z axis acceleration power spectrum increases by the user’s motion like inclining the backrest of the seat. We summarize the estimation method for sitting down and standing up using the power spectrum. In case more than -0.3 G and less than -0.1G acceleration is measured when absence and the sum of power spectrum exceeds 50 afterwards, we estimate for “sitting down”, otherwise we estimate for “absence”. On contrary, in case more than -0.3G and within -0.1G acceleration, or more than 0.1G and under 0.3G acceleration are measured and the sum of power spectrum is between 5s and 10s, we estimate for “standing up”, otherwise we estimate for “working”.
136
M. Okubo and A. Fujimura
3.2 Relationship of Sitting Situation and Power Spectrum Experiment Objective. Generally, it is considered that a person’s motion strongly tends to lessen when concentrating. That is, when concentrating, it is highly possible that the sum of X-axis and Z-axis acceleration power spectrum obtained by the motion sensor fixed on the chair (hereinafter called the sum of the acceleration power spectrum) become smaller. We perform the preparatory experiments in order to examine this assumption. In the experiments the subjects are instructed to type on the computer for quantitative evaluation for concentration. Experiment ・ Evaluation Method. In this experiment we examine the relationship between the sum of the acceleration power spectrum and the typing speed per unit of time while three subjects engage in typing on the computer for 30 minutes. The subjects are instructed to sit down the chair for 30 minutes and engage in either typing or taking a rest. Except the first 1 minute and last 1minute in each of 30-minutes data, we divide the remaining 28-minutes data into 1 minute to find the cross-correlation functions. Evaluation result. The transition of the sum of power spectrum and typing speed, and the examples of the cross-correlation function is as shown in Fig.5. According to Fig. 5 (a), we find that the sum of power spectrum increases when the typing speed becomes 0, when taking a rest. Also Fig.5 (b) shows the fact that approximately -0.7 negative correlation is obtained due to the result as mentioned above. Fig.5(c) shows the decrease of the sum of power spectrum when increasing the typing speed. Fig.5 (d) shows that approximately -0.7 negative correlation is obtained likewise. The average and standard deviation of the minimum of cross-correlation function of the three subjects are as shown in Fig.6. The average of over -0.4 negative correlation function can be seen in Fig.6. Thus we understand that the strong negative crosscorrelation exist in the sum of the power spectrum of user’s motion and the typing speed per unit of time. Also, the validity of estimation that the subjects are concentrating when the sum of the power spectrum become smaller is clarified. Power spectrum Time(s) (b)
60
Time(s) (a)
30
60
Time(s) (c)
Power spectrum
0
Typing speed Power spectrum of acc.
Time(s) (d)
Fig. 5. Cross-correlation functions (b),(d) between power spectrum of user’s motion and typing speed(a),(c)
Development of Estimation System for Concentrate Situation
Subject A
Subject B
137
Subject C
Fig. 6. Average and standard deviation of the minimum of cross-correlation function
3.3 Interface Set-Up Recently, cars installed instantaneous fuel consumption meters are popular in the market. Drivers are likely to loosen gas pedals because of this meter by knowing his situation at the moment and by reviewing his own behavior. That is also the aim of this proposed system. The situations of users who are sitting are described with avatar so that the users can monitor their own situation objectively. The four types of avatars are prepared: “NOT AVAILABLE”,“AVAILABE”, “WORKING”, “PLAYING” and two patterns such as “male”(upper) and “female”(lower) are created. If the system is used as a self-management tool, it becomes important not only to present the user’s situation whether concentrating or not concentrating, but also to present the powers of the concentration. Therefore, as well as displaying the avatar, we create a line graph that present the powers of concentration in the long term and three patterns of GUI using a level meter to describe momentary powers of concentration. Fig.7 (a) shows the GUI displaying the avatar only. In the Fig. 7 (b), GUI that presents transition of the powers of the concentration is added to the avatar. As we assume that the less the sum of the power spectrum decreases, the more the degree of the concentration increases, we set the upper limit of the sum of the power spectrum at 500. Then we reduce the sum of the power spectrum from 500 so that the line goes up when the Graph of power of concentration while long term
Line graph of power of concentrations while short term.
Level meter (a) GUI (avatar only)
(b) GUI (with line graph)
Fig. 7. Graphical User Interface
(c) GUI (with level meter)
138
M. Okubo and A. Fujimura
degree of the concentration becomes high. The graph in the lower right of GUI presents the powers of the concentration in the previous 30s. Also the graph in the upper side of GUI presents the powers of the concentration for a last few minutes for users to check the record. In Fig. 7 (c), the level meter which displays the momentary change of the powers of concentration nonlinearly is added. Even if the sum of the power spectrum is small when concentrating, up down of the scale of the meter is visually confirmed by decreasing the sum per one scale.
4 System Sensory Evaluation Experiment 4.1 Experiment Objective We examine the usefulness and usability of the proposed system and understandably of the powers of concentration presented in three types of GUI. To be more specific, we confirm the operation capability of the estimation system for powers of concentration and perform the sensory evaluation experiments for aforementioned three types of GUI. 4.2 Experiment Conditions Methods In the experiment, ten subjects ( five males and five females in their twenties) are instructed to sit down the chairs with Wii remote controller fixed on, and to engage in typing on computers which the proposed system are running (Fig.8). They are asked to type letters printed on papers in word processing program for five minutes. The GUI is displayed in the upper left of the screen so that it will not disturb their works and the subjects can check it during the works. GUI
Fig. 8. Experimental scene
The experiments are conducted under the four kinds of conditions as follows: (1) without the proposed system, (2) with GUI using only avatar, (3) with GUI using avatar and line graph, and (4) with GUI having avatar and level meter. Taking into account of the order effect, the order of the experiment conditions changes every subject. After the experiment, we give the subjects evaluation questionnaires by five grades. 4.3 Result of Experiment The average and standard deviation obtained by scoring the answers of questionnaire on understandability of the powers of concentration and the interface is shown in
Development of Estimation System for Concentrate Situation
139
Fig.9. These results show that comparing to when having no system and having GUI using avatar only; the subjects finds easier to understand their degree of the concentration when having GUI with the line graph and level meter. Also we conduct a sign test for all combinations in all questionnaires. About the understandability of the powers of the concentration, significant differences at 5% significant level are seen among the combination between having no system and having the line graph, and between having no system and having the level meter. This result also shows that GUI using the line graph and level meter is easier to understand the powers of concentration than having no system. 4.4 Discussion The GUI with the line graphs and the one with level meters were preferred alike as mentioned previously. The subjects who prefer the line graphs evaluated the ability to check the archival record and the others who prefer the level meter evaluated the ability to understand the powers of concentration. However, some said that they were preoccupied by the system and couldn’t concentrate. So the system which users can choose their favorite GUI and save the archival record to look back the day is desirable. Simplicity of understanding for power of concentration
P<0.05
Good
Neutral
No good No system Only avatar Line graph Level meter
Fig. 9. Result of questionnaire
5 Conclusion This study aims that users conduct self-management by presenting their situations while they are using the systems. We develop the systems which estimate user’s powers of the concentration and sitting down and standing up situation, and also present the powers of concentrations using interface to the users. We conduct further experiments on the sensory evaluation to evaluate the operation effectiveness of the systems and interface. As a result, interface using the line graphs which presents the transition of the long-term powers of concentration and interface using the level meter which presents transition of the momentary powers of concentration are preferred equally by the users.
140
M. Okubo and A. Fujimura
The proposed system is developed in order to review users’ own behaviors by understanding their own situations. Although we consider that this system enables to facilitate the concentration, long hours of concentration turns out to be stress and have a bad influence on mental and health. Therefore, we think it is possible to build up the system to facilitate taking appropriate rests during long hours of concentration. We will validate its effectiveness in the long-term experiments along with improvement of the existing system and interface.
References 1. Tamura, H.: Human Interface. Ohmsha, 44–68 (1998) (in Japanese) 2. http://www.wiili.org/index.php/Wiimote
Psychophysiology as a Tool for HCI Research: Promises and Pitfalls Byungho Park Graduate School of Information and Media Management Korea Advanced Institute of Technology (KAIST) 207-43 Cheongryangri 2-dong, Dongdaemun-gu, Seoul, 130-722, Korea
[email protected]
Abstract. Psychophysiology, an area of psychology that measures individual's physiological responses to refer to one's psychological state, can provide a set of useful measures HCI researchers can take advantage of. However, there are limitations to the method itself and room for misinterpretation. This paper introduces psychophysiology, and also shows how research methods psychophysiology offer can be used for HCI research, advantages and disadvantages of using research tools from psychophysiology.
1 Introduction A student essay written and posted online by MIT in 1998 [14] reads "...from studying how human physiology and psychology, we can design better interfaces for people to interact with computers. Work in this domain is only beginning (indeed the number of papers written on this topic has increased in the past few years), and there is much that we don't yet know about the way the human mind works that would allow more perfect user interfaces to be built. (p.4)" Yes, ten years has passed. Though it may have not been as magnificent as this (presumably) young student predicted, there were small yet solid advances made here and there in the area of psychology research. There were also technological advances made for physiology research. Today, as Kim et al. [7] put, HCI researchers are calling for study of human interaction, cognition and physiology to apply such knowledge in system development and implementations. However, it seems that measuring and analyzing physiology data is still not commonly found in HCI research today. Part of the reason may be entrance barrier (understanding of the relationship between human psychology AND physiology, learning to collect physiological data, high cost of physiology equipment, etc.). Another reason may be lack of effort to introduce and share knowledge on using physiology for HCI research. This paper is an attempt to introduce the advantages (and also warn about the possible mistakes made) of applying psychophysiology for HCI research. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 141–148, 2009. © Springer-Verlag Berlin Heidelberg 2009
142
B. Park
2 Psychophysiology and HCI 2.1 Psychophysiology as an Academic Discipline Psychophysiology as an academic discipline that studies the interrelationships between the physiological and psychological aspects of human behavior [15]. A typical study in this area observes human’s physiological responses to understand human’s psychological status and/or changes. Due to its interdisciplinary nature, research from various disciplines including, but not limited to, psychology, cognitive science, medicine, anatomy and neuroscience are incorporated. The type of physiological responses used for psychophysiology studies include blood flow pattern in the brain (functional magnetic resonance imaging; fMRI), heart rate variability, sweat production (skin conductance; SC), respiration, Electroencephalography (EEG; commonly known as ‘brain wave’), muscle contraction (Electromyography; EMG), eye movement, and much more. 2.2 Psychophysiology Providing Tools for HCI Research A wide variety of techniques ranging from qualitative to quantitative are used in HCI studies. The qualitative ones such as in-depth interviews and focus group interviews can provide great depth of insight. However, they are open to misinterpretation and requires a great deal of experience to avoid this. The quantitative ones can be further divided to self-reports and observational ones. Former includes questionnaire and survey, and latter includes response time, counting number of errors (or attempts), and so on. The advantage of using quantitative methods is that the data produced can be used for inferential statistical analysis, providing insight beyond descriptive statistics. However, many quantitative techniques, especially the self-report types, are also open to serious errors. The data may be less-than-accurate because the subject cannot remember certain things clearly, or even motivated to lie (e.g., “have you voted during the last election?”). The research tools psychophysiology offers are quantitative and observational. What makes these tools interesting is that they provide a way to tap into human mind. A large body of literature in psychophysiology provides a fairly solid ground for interpreting the data. In other words, it can provide an alternative way to find how much attention is being paid to the task or how difficult the interface is to use – both which subjects might provide inaccurate answers to when asked using a self-report questionnaire, depending on circumstances.
3 Useful Psychophysiological Research Techniques As mentioned above, a large variety of physiological responses are subject for study in psychophysiology. This section will introduce a few of them that HCI researchers are more familiar with, or may find more useful than others.
Psychophysiology as a Tool for HCI Research: Promises and Pitfalls
143
3.1 Sweat Production (SC) and Stress It is known that people sweat under stress. This is also true when computer users are stressed out from the difficulty of the task given. In order to find how easy or difficult an interface is to use, HCI researchers may measure the amount of sweat production, or skin conductance (SC) of the subject [11, 16]. Deep down, skin conductance is directly related to the activation of sympathetic system of the central nervous system. This is good because it means skin conductance is independent from the activation of the parasympathetic system, which can cause misinterpretation of the activity of many human organs or systems which are under influence of both central nervous systems (for example, human heart may beat faster because the sympathetic system is activated, or because the parasympathetic system is deactivated, or both). However, there are downsides for using skin conductance. One of them is the sensor (electrode) placement. Sweat glands in humans are concentrated in palms and soles. Many computer-related tasks and their interfaces require users’ hands. Collecting skin conductance data from the palm may restrict subjects from using both hands freely. Furthermore, in certain tasks that require both hands (such as typing text with a keyboard), using palm is out of question. Alternatively, sole of the foot may be used, but this is inconvenient since subjects are required to remove their socks and have their foot kept lifted through the whole session to prevent electrodes touching the floor. Using skin conductance as an index of task (or interface) difficulty is it’s slow response speed. Rather than constantly producing sweat, human sweat glands ‘spout out’ sweat. It takes about six seconds for sweat glands to respond to an arousing event, though exact timing varies by individual. This makes it hard for researchers to pinpoint the exact timing of a particular event that causes stress in the human subject. For this reason, comparing the total amount sweat or average skin conductance level over time within a subject during different tasks (which takes longer than 10 seconds to complete) is recommended rather than attempting to identify the exact moment that causes stress [15]. 3.2 Heart Rate (HR) Variability and Attention Human heart is affected by both sympathetic and parasympathetic systems of the central nervous system. This means that its activity is affected by both arousal and paying attention to external stimuli. Research shows that the speed of heart beat slows down when one is paying attention to a stimulus presented [17]. This can be explained from an evolutionary psychology perspective: when an unknown change is noticed in the environment, the body automatically responds to it by calming down (or slowing down) the activities of the internal organs until proper assessment of the situation is made. If the change turns out to be life-threatening, then the famous ‘fight or flight’ reaction, which includes intensive activity in sweat glands and quick acceleration of heart rate follows. If the change does not pose any harm to human, then the heart rate gets faster, but no faster than the normal speed.
144
B. Park
It is important to acknowledge that deceleration of heart rate is an index of external attention. Measurement of external attention is useful for HCI research since it serves as an index of whether the subject is paying attention to the menu of the computer screen, or whether that gentle chime has actually got the subject’s attention. The other type of attention is called internal attention, which happens when one puts mental effort to solve math questions [9]. When this happens, heart rate acceleration is observed. Hence, it is important to see the context of the settings heart rate data has been collected and determine whether the heart rate getting fast is a heart rate acceleration which is a byproduct of internal attention, or simply a heart rate deceleration from not paying attention. Also to put into consideration is the subject’s arousal level – excitation leads to arousal in the sympathetic system, which triggers heart to beat faster. When analyzing heart rate data, context matters a lot. There are two ways to measure heart rate. One is to measure the electrical pulse produced by the human heart every time it contracts and pumps blood out to the whole body. This measurement is called electrocardiogram, or ECG. The other way is to measure the blood flowing in and out to the tip of the finger and/or toe, which is called photoplethysmography (PPG). Photoplethysmography is typically measured by emitting infrared light into the skin. The level of light absorption changes by the amount of blood flowing underneath the skin, and this is used to measure heart rate. ECG monitors the electrical activity of heart, while PPG monitors the mechanical activity of heart. In a perfect world, both has to be able to serve as equally perfect indices of heart activity. But neither is perfect. ECG requires three electrodes, attached on both arms, both legs, or above the chest. Chest placement is rarely used outside of a hospital setting, though it provides the clearest ECG. Attaching electrodes on arms or legs is more practical for HCI research, but the distance from the heart tends to make it more vulnerable to noise (caused by subject’s body movement and internal organ activities) and weak pulse. From my personal experience, arms tend to provide cleaner ECG than legs. PPG needs to have one sensor attached to a finger or a toe. Generally, fingers are better than toes for collecting PPG data because they are closer to the heart. Nevertheless, if the subject’s heart is relatively weak and the hand or foot is in a position that makes it hard for blood to flow in, there may not be enough blood flowing in and out for the infrared light sensor to notice and record the PPG. For these reasons, both are somewhat vulnerable to errors, especially by missing individual heart beats. However, many of these issues can be prevented by preparing the best possible settings for the subject. As long as the study condition is fine, ECG is as reliable as PPG [13]. Sensor placement again can be a problem: ECG requires three locations on the arm or the leg. PPG may require only one but it has to be either a finger or a toe. And it is best to avoid the sensors touching hard surface. As a result, type of data to be collected for heart rate and location for sensor placement depends on the type of task for the subjects. 3.3 Eye Tracking and Attention As the maxim “eye is the window to mind” suggests, it has been long thought that human gaze reflects the top priority of cognitive processes [6]. This is one of the
Psychophysiology as a Tool for HCI Research: Promises and Pitfalls
145
reasons HCI researchers find eye tracking to have great potential to be an important research tool. In the past, electrooculography (EOG) that measures resting potential of the retina was used to track the eye movements. However, EOG can only show to which general direction the eye has moved to; not exactly where the eye gaze if fixed to [6, 15]. With advances in real-time optical data processing, monitoring of the retina by using infra-red light became more sophisticated. Data produced by eye tracking is useful not only since it provides quantitative data that can be subjected for statistical analysis, but also because it can be visualized in ways that are intuitive for audience when presented. One is heat map, which colorcodes each area of the computer screen where usually red is colored to the area that the eye fixation happened longest and dark blue (or no color) represents little or no eye gaze in the area. The other is an animation of which part of the screen the gaze has been moved to (which, some companies call “gaze replay”) in accordance to time. When used properly, both visualization techniques are powerful enough to convince audience with little or no knowledge in statistics [3, 4, 6]. In the past, eye tracking equipments required a head-mounted camera that monitors the retina movement and alto a head-mount that fixes the subject’s head since the equipment was not capable of making adjustments according to head movements. Today, there are equipments in the market that does not require head-mounted cameras. Some of them are capable to make adjustments to minor (and gentle) head movements. However, eye tracking technology available today still has its own limitations. The largest challenge is that it is hard to use on subjects with seriously bad vision. This is not because of the vision itself, but because of the lenses (both glasses or contact lenses). In theory, the equipment should be able to be adjusted for such lenses, but in practice, virtually no machine in the market provides such calibration. However, there are indeed quite a few equipments that automatically adjust to subjects who wear glasses for relatively light myopia. Incapability to track vision to sudden and/or fast head movement is another limitation for contemporary eye tracking technology. 3.4 Facial Muscle Activity (EMG) and Emotion Muscle fibers of human (as well as other animals) contract when they are triggered by electrical current. Electromyography (EMG) is a measure of the electrical activity of muscles, which directly index muscle activities. Facial EMG refers to indexing of facial muscle activities. Psychophysiology researchers have been using facial EMG to monitor emotional changes in human [5, 12]. There are three muscle groups monitored for this purpose: Currugator supercilii, Zygomatic major, and Orbicularis oculi [15]. The Corrugator supercilii muscle group is the “frowning” muscle. It is located between the nose and the forehead and when activated, it draws the eyebrow slightly downward and creates vertical wrinkles between the eyebrows. Corrugator activity is typically used as an index of negative emotion. The Zygomatic major is a muscle group that is located at both ends of the lips. When activated, it pulls the lips from both sides and mostly used when one is smiling. For this reason, it is used as an index of positive emotion. However, it is also
146
B. Park
activated when one is experiencing other types of non-positive emotion (e.g., One screaming “Eeeek!” out of disgust), so the data has to be analyzed with caution. The Orbicularis oculi is another muscle group that is getting attention as an alternative for the Zygomatic major muscle group as an index for positive emotion. The Orbicularis oculi muscle group is located right below the lower eye lids, and it is primarily responsible for eye blinks. It is also responsible for gathering of skin around the eyes when one is smiling, which is called the Duchenne smile. A Duchenne smile involves contraction of both the zygomatic major muscle group and the orbicularis oculi muscle group, and it is seen as an indicator of one experiencing true happiness (a Non-Duchenne smile only involves the zygomatic major muscle group and interpreted as an indicator of social smiling which does not involve happiness; for review, see Ekman, Davidson, and Friesen [2]). One of the challenges using facial EMG is that the muscle groups are small and the electrical activity is very weak. Especially, orbicularis oculi muscle group is so small that researchers are forced to place two mini electrodes (diameter of 4-millimeters) very close (about 5-millimeters), which makes it vulnerable to errors during data collection.
4 Conclusion: No One Measure Is Perfect This paper has reviewed skin conductance, heart rate, eye tracking, and facial EMG as tools for HCI research. There are, of course, other tools such as Electroencephalography (EEG; also known as ‘brain wave’), functional Magnetic Resonance Imaging (fMRI; also known as ‘brain imaging’), and many more. Each has its own unique advantages and disadvantages as a tool for research. Though psychophysiology promises to provide useful research tools for HCI, there are limitations that generally apply to them [8]. The largest one is external validity. As mentioned above, skin conductance, heart rate, facial EMG all requires some kind of sensor to be attached on parts of the subject’s body. And this adds to the awkward feeling to already unnatural setting (e.g., being put into a usability lab with a one-way window and, possibly, video camera rolling) the subject may experience during a session. Eye tracking is no less artificial than other measurements. The least intrusive technology still requires participants keep their head movement minimal and if needed, move gently. The subject has to be reminded of this, and this inevitably causes unnatural tension in the subject which, depending on the type of study, may have fairly large negative impact on the data collected. Another limitation is that the data collected is open to interpretation. Though many people tend to believe psychophysiology provides an absolutely objective data and interpretation. For example, Hornbaek (2005) argues that physiological measures are objective, using "physiological measures of fun in playing computer games (p.92)" as an example. Unfortunately, this view can be wrong – especially in the context of HCI research. Studies in psychology tend to use simple stimuli, which helps avoid unexpected factors to interfere. But in many cases, HCI research has to use real-life equipments and/or interfaces as stimuli that are much more complex than those stimuli used in a
Psychophysiology as a Tool for HCI Research: Promises and Pitfalls
147
typical psychology experiment. So the data collected always have to be interpreted with the context in mind. Could the heart rate have picked up speed because the subject was not paying attention, or because the subject was puzzled by the menu system and had to think about it? Was the subject sweating more because the interface was hard to use or because there was an image of a huge spider shown on the screen? When designing the session, it would be the best to eliminate all factors that may result unwanted interference during data collection, but even after data has been collected, it would be the best to go back and think if there is any room for alternative explanation for certain physiological responses. And researchers should keep in mind that though research tools provided by psychophysiology may offer new insights, none of them is perfect by itself and it is best to be used in combination with other research methods. Through his online column, Jacob Nielsen, a well-known HCI guru and consultant, advices design agencies who are seeking ways to convince clients to pay for usability testing, "using sound methodology is the true sign of professionalism" and they have to "point out usability's astounding return on investment."[10] Including psychophysiology to the set of tools available may also be a good idea for design agencies for improving usability of the final product. Psychophysiology should be able to make good friends with both HCI researchers and HCI practitioners. It’s just the matter of choosing the right tool and applying it the right way to get the right answer to the right question.
References [1] Allanson, J., Wilson, G.M.: Physiological Computing. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI 2002), pp. 912–913 (2002) [2] Ekman, P., Davidson, R.J., Friesen, W.V.: The Duchenne Smile: Emotional Expression and Brain Physiology II. Journal of Personality and Social Psychology 58(2), 342–353 (1990) [3] Hewig, J., Trippe, R.H., Hecht, H., Straube, T., Miltner, W.H.R.: Gender Differences for Specific Body Regions When Looking at Men and Women. Journal of Nonverbal Behavior 32, 67–78 (2008) [4] Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human–Computer Interaction and Usability Research: Ready to Deliver the Promises. In: Hyona, Radach, Deubel (eds.) The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, Oxford, England (2003) [5] Jones, C.M., Dlay, S.S.: The Face as an Interface: The New Paradigm for HCI. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), vol. 1, pp. 774–779 (1999) [6] Just, M.A., Carpenter, P.A.: Eye fixations and cognitive processes. Cognitive Psychology 8, 441–480 (1976) [7] Kim, S., Godbole, A., Huang, R., Panchadhar, R., Smari, W.: Toward an Integrated Human-centered Knowledge-based Collaborative Decision Making System. In: Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, LasVegas, NV, November 2004, pp. 394–401 (2004) [8] Lin, T., Hu, W., Omata, M., Imamiya, A.: Do physiological data relate to traditional usability indexes? In: Proceedings of the 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future (OZCHI), vol. 122, pp. 1–10 (2005)
148
B. Park
[9] Mansell, W., Clark, D.M., Ehlers, A.: Internal versus external attention in social anxiety: An investigation using a novel paradigm. Behaviour research and therapy 41(5), 555–572 (2003) [10] Nielsen, J.: Convincing Clients to Pay for Usability (May 19, 2003), http://www.useit.com/alertbox/20030519.html [11] Pecchinenda, A., Smith, C.A.: The affective significance of skin conductance activity during a difficult problem-solving task. Cognition and Emotion 10(5), 481–504 (1996) [12] Prendingera, H., Mori, J., Ishizuka, M.: Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game. International Journal of Human-Computer Studies 62, 231–245 (2005) [13] Selvaraj, N., Jaryal, A., Santhosh, J., Deepak, K.K., Anand, S.: Assessment of heart rate variability derived from finger-tip photoplethysmography as compared to electrocardiography. Journal of Medical Engineering & Technology 32(6), 479–484 (2008) [14] Spiegel, D.: Human Computer Interaction (October 22, 1998), http://xenia.media.mit.edu/~spiegel/papers/HCI.pdf [15] Stern, R., Ray, W.J., Quigley, K.S.: Psychophysiological Recording, 2nd edn. Oxford University Press, New York (2001) [16] Svebak, S.: The effect of task difficulty and threat of aversive electric shock upon tonic physiological changes. Biological Psychology 14(1-2), 113–128 (1982) [17] Ward, R.D., Marsden, P.H.: Physiological responses to different WEB page designs. International Journal of Human-Computer Studies 59, 199–212 (2003)
Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise Genaro Rebolledo-Mendez1,3, Ian Dunwell1, Erika A. Martínez-Mirón2, María Dolores Vargas-Cerdán3, Sara de Freitas1, Fotis Liarokapis4, and Alma R. García-Gaona3 1
Serious Games Institute, Coventry University, UK 2 CCADET, UNAM, Mexico 3 Facultad de Estadistica e Informatica, Universidad Veracruzana, Mexico 4 Interactive Worlds Applied Research Group, Coventry University, UK {GRebolledoMendez,IDunwell,Sfreitas, F.Liarokapis}@cad.coventry.ac.uk,
[email protected], {dvargas,agarcia}@uv.mx
Abstract. This paper presents the results of a usability evaluation of the NeuroSky’s MindSet (MS). Until recently most Brain Computer Interfaces (BCI) have been designed for clinical and research purposes partly due to their size and complexity. However, a new generation of consumer-oriented BCI has appeared for the video game industry. The MS, a headset with a single electrode, is based on electro-encephalogram readings (EEG) capturing faint electrical signals generated by neural activity. The electrical signal across the electrode is measured to determine levels of attention (based on Alpha waveforms) and then translated into binary data. This paper presents the results of an evaluation to assess the usability of the MS by defining a model of attention to fuse attention signals with user-generated data in a Second Life assessment exercise. The results of this evaluation suggest that the MS provides accurate readings regarding attention, since there is a positive correlation between measured and selfreported attention levels. The results also suggest there are some usability and technical problems with its operation. Future research is presented consisting of the definition a standardized reading methodology and an algorithm to level out the natural fluctuation of users’ attention levels if they are to be used as inputs.
1 Introduction This paper presents a usability evaluation of NeuroSky’s MindSet (MS) device. An aspect of interest was to investigate whether MS readings can be combined with usergenerated data. The amalgamation of physiological and user-generated data would allow the programming of more sophisticated user models. An experimental setting was set up to analyze MS usability in an assessment exercise in Second Life. The assessment [10] is based on a multiple-choice questionnaire in the area of programming for Computer Science undergraduate students. The questionnaire is presented by an Artificial Intelligence-controlled avatar (AI-avatar) who is aware of the levels of attention of the J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 149–158, 2009. © Springer-Verlag Berlin Heidelberg 2009
150
G. Rebolledo-Mendez et al.
person interacting with it. The MS1 also provides a measurement of the user’s meditative state (derived from alpha wave activity). In this paper, however, only the levels of attention are used, given their role and importance in educational settings. The objective of this study is threefold: firstly, the MS general usability is examined. Secondly, an analysis of how well it is possible to fuse information generated as part of normal interactions with brain activity. Thirdly, an analysis of the MS adaptability to different ableusers is provided. The significance of this work lies in that it presents evidence of the usability of a commercially available BCI and its suitability to be incorporated into serious games. The paper is organized in five sections. Section two presents a literature review about Brain Computer Interfaces and their use for learning. Section three describes the Assessment exercise used as test bed and presents the materials, participants and methodology followed during the evaluation. Section four presents the results of the evaluation and, finally, section five provides the conclusions and future research.
2 Brain Computer Interfaces (BCI) Brain Computer Interface (BCI) technology represents a rapidly emerging field of research with applications ranging from prosthetics and control systems [6] through to medical diagnostics. This study only considers BCI technologies that use sensors that measure and interpret brain activity (commonly termed neural bio-recorders [14]) as a source of input. The longest established method of neural bio-recording, developed in 1927 by Berger [3], is the application of electrodes that measure the changes in field potential over time arising from synaptic currents. This forms the basis for EEG. In the last two decades, advances in medical imaging technology have presented a variety of alternative means for bio-recording, such as functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and positron emission tomography (PET). A fundamental difference between bio-recording technologies used for diagnostic imaging, and those used for BCI applications, is a typical requirement for real or quasi-real time performance in order to translate user input into interactive responses. In 2003, a taxonomy by Mason and Birch [8] identified MEG, PET and fMRI as unsuitable for BCI applications, due to the equipment required to perform and analyze the scan in real-time, but more recent attempts to use fMRI as a BCI input device have demonstrated significant future potential in this area [12]. Bio-recording BCIs have become a topic of research interest both as a means for obtaining user input, and studying responses to stimuli. Several studies have already demonstrated the ability of an EEG-based BCI to control a simple pointing device similar to a mouse [9, 12] and advancing these systems to allow users more accurate and responsive control systems is a significant area for research. Of particular interest to this study is the use of BCI technologies in learning-related applications. The recent use of fMRI to decode mental [4] and cognitive [11] states illustrates a definite capability to measure affect through bio-recording, but the intrusiveness of the scanning equipment makes it difficult to utilize the information gained to provide feedback to a user performing typical real-world learning activities. In this study, the effectiveness of one of the first commercially available lightweight EEG devices, NeuroSky’s MS, is considered. Via the application of a single 1
The MB is a developer-only headset. NeuroSky’s newest headset has been designed to address comfort and fitting problems and is available to both developers and consumers.
Assessing NeuroSky’s Usability to Detect Attention Levels
151
electrode and signal-processing unit in a headband arrangement, the MS provides two 100-state outputs operating at 1Hz. These outputs are described by the developers as providing separate measures of ‘attention’ and ‘meditation’, and it is thus assumed these readings are inferred from processing beta and alpha wave activity respectively. Although the MS provides a much coarser picture of brain activity than multielectrode EEG or the other aforementioned technologies, the principle advantage of the MS is its unobtrusive nature, which minimises the aforementioned difficulties in conducting accurate user studies due to the stress or distraction induced by the scanning process. Research into EEG biofeedback as a tool to aid individuals with learning difficulties [5] represents an area for ongoing study, and the future widespread availability of devices similar to the MS to home users presents an interesting opportunity to utilize these technologies in broader applications.
3 An Assessment Exercise in Second Life An assessment exercise was developed to examine the MS. The exercise works in combination with a model of attention [10] built around dynamic variables generated by the learner’s brain (MS inputs) and the learner’s actions in a computer-based learning situation. The combination of physiological (attention) and data variables is not new [7, 1]. Our approach, however, fuses MS readings (providing a more accurate reading of the learner’s attention based on neural activity) with user-generated data. In our model, attention readings are combined with information such as the number of questions answered correctly (or incorrectly), or the time taken to answer each question, to model attention within the assessment exercise. The MS reads attention levels in an arbitrary scale ranging from 0 to 100. There is an initial delay of between 7 and 10 seconds before the first value reaches the computer and newer values of attention are calculated at a rate of 1Hz (one value per second, see Figure 1). A value of -3 indicates no signal is being read and values equal to or greater than 0 indicate increasing levels of attention with a maximum value of 100. Given the dynamic nature of the attention patterns and the potentially large data sets obtained, the model of attention underpinning the assessment exercise is associated to a particular learning episode lasting more than one second. The model of attention not only determines (detects) attention patterns but also provides (reacts) feedback to the learner [10]. The assessment exercise consists of presenting a Second Life2, AI-driven avatar able to pose questions, use a pre-defined set of reactions and have limited conversations with learners in Second Life. The AI-driven avatar was programmed using C# (C-sharp) in combination with the lib second life library3. Lib Second Life is a project aimed at understanding and extending Second Life’s client to allow the programming of features using the C# programming languages. This tool enables the manipulation of avatars’ behaviors so that they respond to other avatars’. To do so, the AI-driven avatar collects user-generated data during the interaction including MS inputs. The current implementation of the AI-driven avatar asks questions in a multiple-choice 2 3
http://secondlife.com/ http://www.libsecondlife.org/
152
G. Rebolledo-Mendez et al.
format, while dynamically collecting information (answers to questions, time taken to respond, and whether users fail to answer). The data generated by the MS is transmitted to the computer via a USB interface and organized via a C# class which communicates with the AI-driven avatar. In this way, the model of attention is updated dynamically and considers input from the MS as well as the learner’s performance behavior while underpinning the AI-driven avatar’s behavior. For the purposes of assessing MS’s usability, the assessment exercise consisted of ten questions in the area of Informatics, specifically for the area of Algorithms. This area was targeted since it has been noted first year students in the Informatics department often struggle with the conception and definition of algorithms, a fundamental part of programming. The assessment exercise asked Fig. 1. Attention readings as read by the NeuroSky nine theoretical questions and presented three possible answers. For example, the avatar would ask ‘How do you call a finite and ordered number of steps to solve a computational problem?’ while offering ‘a) Program, b) Algorithm, c) Programming language’ as possible answers. The assessment exercise also includes the resolution of one practical problem, answered by the learner by hand while still wearing the MS. 3.1 Materials To evaluate the MS’s reliability, two adaptations of the Attention Deficit and Hyperactivity disorder (ADHD) test and a usability questionnaire were defined. The attention tests consisted of seven items based on the DSV4-IV criteria [2]. The items chosen for the attention test were: 1. Difficulty to stay in one position, 2. Difficulty in sustaining attention, 3. Difficulty to keep quiet often interrupting others, 4. Difficulty to follow through on instructions, 5. Difficulty to organize tasks and activities, 6. Difficulty or avoidance of tasks that require sustained mental effort and 7. Difficulty to listen to what is being said by others. Each item was adapted to assess attention both in the class and at interaction time. To answer individual questions, participants were asked to choose the degree which they believed reflected their behavior in a Likert type scale with 5 options. For example, question 1 of the attention questionnaire asked the participant: ‘How often is it difficult for me to remain seated in one position whilst working with algorithms in class/during the interaction?’ with the answers 1) all the time, 2) most of the time, 3) some times, 4) occasionally and 5) never. Note that for both questionnaires the same seven questions were asked but were rephrased considering the class for the pre-test 4
Diagnostic and Statistical Manual of Mental Disorders.
Assessing NeuroSky’s Usability to Detect Attention Levels
153
or the interaction for the post-test. The usability questionnaire consisted of adapting three principles of usability into three questions (a) comfort of the device; (b) easiness to wear; and (c) degree of frustration. To answer the usability questionnaire participants were asked to select the degree to which they felt the MS faired during the interaction via a Likert type scale with 5 options. For example, question 1 of the usability questionnaire asked the student: ‘Was using Neurosky’ 1) Very uncomfortable, 2) Uncomfortable, 3) Neutral, 4) Comfortable, 5) Very comfortable. Note that to report the usability of the MS, other factors were also considered such as battery life, light indicators and data read/write times and intervals. 3.2 Participants and Methodology An evaluation (N=40) to assess the usability of the MS was conducted among firstyear undergraduate students in the Informatics Department at the University of Veracruz, Mexico. The population consisted of 28 males and 12 females, 38 undertaking the first year of their studies and 2 undertaking the third year. 26 students (65%) of the population were 18 years old, 12 students (30%) were 19 years old and 2 students (5%) were 20 years old. The participants interacted with the AI-avatar for an average of 9.48 minutes answering ten questions posed by the avatar within the assessment exercise (see previous section). During the experiment, the following procedure was followed: 1) students were asked to read the consent form, specifying the objectives of the study and prompted to either agree or disagree, 2) students were asked to solve an online pre-test consisting of the adaptation of the attention deficit and hyperactivity disorder (ADHD) questionnaire to assess their attention levels in class, 3) students were instructed on how to use the learning environment, and finally 4) the students were asked to answer an online post-test consisting of the usability questionnaire and the adaptation of the ADHD questionnaire to assess their attention levels during the interaction in the assessment exercise. Individual logs registering the students’ answers and attention levels as read by the MS were kept for analyses. All students agreed to participate in the experiment but in some cases (N=6) the data was discarded since the MS did not produce readings for these participants. See the results section for a description of these problems. Cases with missing data were not considered in the analysis.
4 Results The results of this evaluation are organized to consider the MS’s usability, how well the model fuses user-generated data and attention readings and the MS’s adaptability. 4.1 Usability and Appropriateness of MS for Assessment Exercises The main aspect of interest was MS’s usability considering the responses to three questions (see materials section). This questionnaire considered three aspects to assess the usability of new computer-based devices: Comfort, Ease of Use, and the Degree of Frustration. The answers to the questionnaire are organized around each aspect considered. There was one question associated to every usability aspect.
154
G. Rebolledo-Mendez et al.
Comfort The results showed that for 5% (N=2) the MS was uncomfortable, for 10% (N=4) somewhat uncomfortable, for 35% (N=14) neither comfortable nor uncomfortable, for 25% (N=10) somewhat comfortable and for 25% (N=10) comfortable. Ease of Use The results showed 15% (N=6) students found the MS difficult to wear, 12.5% (N=5) found it somewhat difficult to wear, 37.5% (N=15) thought it was neither easy nor difficult to wear, 12.5% (N=5) found it somewhat easy to wear and 22.5% (N=9) thought it was easy to wear. Degree of Frustration The answers showed 2.5% (N=1) found the experience frustrating, 2.5% (N=1) thought it was somewhat frustrating, 22.5% (N=9) found the experiment neither frustrating nor satisfactory, 25% (N=10) thought it was somewhat satisfactory and 47.5% (N=19) had a satisfactory experience using the MS. There were three aspects that only became apparent once the evaluation was over. The first aspect of interest was in relation to the pace and the way readings were collected. The attention model [1] considered readings in the space of time used by learners to formulate an answer for each question. The pace in which data was collected by the model was 10Hz which produced repeated measurements in some logs. This method of collecting data is inefficient as plotting attention fluctuations considering fixed, regular intervals might be difficult. People interested in programming the MS device should consider that, due to a hardware processing delay, the MS outputs operate at 1Hz, and need to program their algorithms accordingly. The second aspect of interest is in relation to difficulties wearing the device. When connection is lost, there is a delay of 7-10 seconds before a new reading is provided. Designers should consider this as a constant input might not be possible. The third aspect refers to MS’s suitability as an input device for interface control. Developers need to consider that attention levels (and associated patterns) vary considerably between users (see Figure 2), as expected. If developers employ higher levels of attention as triggers for interface or system changes, they should consider some users normally have higher levels of attention without being prompted to put more attention. This normal variability creates the need to research and develop an algorithm to level-out initial differences in attention levels and patterns. On a related topic, MS’s readings vary in a scale from 0 to 100 (see Figure 1): however, it is not yet clear what relationship exists between wave activity and processed output, whether the scale is linear, or whether the granularity of the 100-point scale is appropriate for all users. Finally, there were some usability problems that caused data loss, in particular: 1. In 3 cases the MS did not fit the participant’s head properly leading to adjustments by the participants leading to not constant and unreliable readings. Another problem was people with longer hair having problems wearing the device to allow sensors touch the skin behind the ears at all times. During the experiment, extra time was required to make sure people with longer hair placed the device adequately. 2. In other 3 cases the MS ran out of battery. The battery was checked before each participant interacted with the assessment exercise using NeuroSky software via its associated software. However, despite the precautions taken and after having checked the green light on one of the device’s side, battery life was very short. The
Assessing NeuroSky’s Usability to Detect Attention Levels
155
device does not alert the user when battery levels are low, so it was not clear when batteries needed to be replaced. This was a problem at the beginning of the experiment but later on batteries were replaced on daily basis. 4.2 Adaptability to Different Users One of the characteristics of the MS reader is that it can be worn by different users producing different outputs. This would allow for adaptation of the model [10] in the frame of the assessment exercise. It was expected MS outputs would vary for different users reflecting varying levels of attention. Furthermore, this adaptation would be fast and seamless without the need to train the device for a new user. To throw some light onto the issue of adaptability, it was speculated attention readings would be different for individuals. It was also hypothesized there would be a positive correlation between the readings and the self-assessment attention test (see materials section). To assess variability among participants, a test of normality was done to see the distribution of the participants’ average attention levels. Table 1 shows descriptive statistics of the readings for the population (N=34). The results of a test of normal distribution showed that the data is normally distributed (Shapiro-Wilk = .983, p = .852) suggesting there is not a tendency to replicate particular readings. Figure 2 illustrates the Q-Q plot for this sample suggesting a good distribution of average attention levels during the assessment exercise. Table 1. Descriptive statistics
N 34 34
Student's attention levels Student's self reported attention
Min 14.99 3.0
Max 88.00 5.0
Mean 53.40 4.27
Std. Dev. 16.69 .44548
Another test designed to see whether MS readings adapted to individual participants, was a correlation between the readings and the self-reported attention using the post-test questionnaire. A positive correlation was expected between these two variables. Table 1 shows the descriptive statistics for the two variables. To calculate self-reported attention levels, the mean of the answers to the 7 items of the attention posttest was calculated per participant; lesser values indicate lesser attention levels. The results of a PearObserved Value son’s correlation between the two variables indicated a significant, Fig. 2. Q-Q plot of students’ average attention positive correlation (Pearson’s = levels during the assessment exercise -.391, p = .022). .3
.2
.1
0.0
Dev from Normal
-.1
-.2
-.3
-.4
20
30
40
50
60
70
80
90
156
G. Rebolledo-Mendez et al.
4.3 Fusing User-Generated Information with MS Readings One way to analyze whether the data was fused correctly was to check the logs for missing or incorrect data. The results of this analysis showed that there were six participants (15%, original sample N=40) for which the MS did not produce accurate readings. An analysis of the logs for the remaining participants (N=34) showed the device produced readings throughout the length of the experiment (average time = 9.48 minutes) without having an erroneous datum (attention = -3). The causes for the lack of readings in 6 cases were due to usability problems (see following section). Another way of throwing some light on how well the MS readings and usergenerated data were fused consisted of analyzing the logs to see whether there was a variation on the model’s reactions for the sample. Since the reactions given by the AI avatar could be of six types [1], the frequency was calculated for each reaction type for the entire population with correct NeuroSky readings (N=34), see Table 2. Table 2. Frequencies associated to the model’s reaction types for the population (N=34)
Reaction Type Frequency
6 128
5 172
4 0
3 77
2 13
1 0
It was expected the frequencies for reaction types 4 and 1 would be 0 given the averages of the four binary inputs. Reaction Types 5 was the most common type followed by Reaction Types 6, 3 and 2. Given the 8 possible results of averaging out the four binary inputs [10], it was expected Reaction Type 3 would be the most frequent. However, this was not the case suggesting the model did vary and the reactions type provided were in accordance to the variations in attention, time, and whether answers were correct. Finally, the responses to two questions in the post-test questionnaire gave an indication of students’ subjective perceptions about how well the reaction types were adequate to their attention needs. The first question asked: ‘how frequently the reactions helped you realize there was something wrong with the way you were answering the questions?’ The answers showed 25% (N=10) of students felt the avatar helped them all the time, 20% (N=8) said most of the time, 35% (N=14) mentioned some times, 12.5% (N=5) said rarely and 7.5% (N=3) stated never. The second question asked ‘how appropriate they thought the combined use of MS and avatars was appropriate for computer-based educational purposes?’ Students’ answered with 65% (N=26) saying it was appropriate, 12.5% (N=5) saying it was appropriate most of the time, 15% (N=6) saying it was neither appropriate nor inappropriate, 2.5% (N=1) saying it was somewhat inappropriate and 5% (N=2) saying it was inappropriate.
5 Conclusions and Future Work The reliability of MS readings to assess attention levels and to amalgamate with usergenerated data was evaluated in an assessment exercise in Second Life, N=34. The results showed there is variability in the readings and they correlate with self-reported attention levels suggesting the MS adapts to different users providing accurate readings of attention. The results of analyzing the device’s usability suggest some users
Assessing NeuroSky’s Usability to Detect Attention Levels
157
have problems with wearing the device due to head sizes or hair interference and that the device’s signals to indicate flat batteries are poor. By analyzing individual logs it was possible to determine that, when the device fits properly, the MS provides valid and constant data as expected. Log analyses also helped establish the frequency different reactions types were provided in the exercise in the light of attention variability. The frequencies suggested the model did not lean to the most expected reaction (Type 3) but that it tended to be distributed amongst Reaction Types 5 and 6, providing an indication that user-generated data was fusing adequately with attention readings. When asked about their experience, 35% of the population said the avatar helped them realize there was something wrong with how s/he was answering the questions and 65% indicated using a MS in combination with avatars was appropriate in computer-based educational settings. When asked about comfort, 35% thought the device was neither comfortable not uncomfortable, 37.5% thought it was neither easy nor difficult to wear and 47.5% said they had a satisfactory experience with the device. There were other results that were apparent only after the evaluation. In particular, it was found that: 1) sampling rates need to be considered to organize data in fixed, regular intervals to determine attention. 2) Developers need to be aware there is a delay when readings are lost due to usability issues. 3) Variability imposes new challenges for developers who wish to use levels of attention as input to control or alter interfaces. Work for the future includes the combination of MS readings with other technologies such as using learner’s gaze, body posture and facial expressions to read visual attention. Future work will be carried out to find out the degree of attention variability to program an algorithm capable of leveling-out different patterns of attention. In addition, future work will explore how attention data can be used to develop learner models that help understanding attention and engagement for informing gamebased learning design and user modeling.
Acknowledgements The research team thank the NeuroSky Corporation for providing the device for testing purposes. We also thank the students, lecturers and support staff at the Faculty of Informatics, Universidad Veracruzana, Mexico.
References 1. Amershi, S., Conati, C., McLaren, H.: Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games. In: Workshop in Motivational and Affective Issues in ITS, 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan (2006) 2. Association, A.P.: Diagnostic and Statistical Manual of Mental Disorders. American Psychiatric Press (1994) 3. Berger, H.: On the electroencephalogram of man. In: Gloor, P. (ed.) The fourteen original reports on the human electroencephalogram, Amsterdam (1969) 4. Haynes, J.D., Rees, G.: Decoding mental states from brain activity in humans. Nature Neuroscience 7(7) (2006)
158
G. Rebolledo-Mendez et al.
5. Linden, M., Habib, T., Radojevic, V.: A controlled study of the effects of EEG biofeedback on cognition and behavior of children with attention deficit disorder and learning disabilities. Applied Psychophysiology and Biofeedback 21(1) (1996) 6. Loudin, J.D., et al.: Optoelectronic retinal prosthesis: system design and performance. Journal of Neural Engineering 4, 72–84 (2007) 7. Manske, M., Conati, C.: Modelling Learning in an Educational Game. In: 12th Conference on Artificial Intelligence in Education, IOS Press, Amsterdam (2005) 8. Mason, S.G., Birch, G.E.: A general framework for brain-computer interface design. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11, 70–85 (2003) 9. Poli, R., Cinel, C., Citi, L., Sepulveda, F.: Evolutionary brain computer interfaces. In: Giacobini, M. (ed.) EvoWorkshops 2007. LNCS, vol. 4448, pp. 301–310. Springer, Heidelberg (2007) 10. Rebolledo-Mendez, G., De Freitas, S.: Attention modeling using inputs from a Brain Computer Interface and user-generated data in Second Life. In: The Tenth International Conference on Multimodal Interfaces (ICMI 2008), Crete, Greece (2008) 11. Sona, D., Veeramachaneni, S., Olivetti, E., Avesani, P.: Inferring cognition from fMRI brain images. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D.P. (eds.) ICANN 2007. LNCS, vol. 4669, pp. 869–878. Springer, Heidelberg (2007) 12. Sitaram, R., et al.: fMRI Brain-Computer Interfaces. IEEE Signal Processing Magazine 25(1), 95–106 (2008) 13. Trejo, L.J., Rosipal, R., Matthews, B.: Brain-computer interfaces for 1-D and 2-D cursor control: designs using volitional control of the EEG spectrum or steady-state visual evoked potentials. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14(2), 225–229 (2006) 14. Vaughan, T., et al.: Brain-computer interface technology: a review of the second international meeting. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11(2), 94–109 (2003)
Effect of Body Movement on Music Expressivity in Jazz Performances Mamiko Sakata1, Sayaka Wakamiya2, Naoki Odaka2, and Kozaburo Hachimura3 1
Faculty of Culture and Information Science, Doshisha University, 1-3 Tatara Miyakodani, Kyotanabe City, 610-0394 Japan
[email protected] 2 Graduate School of Human Development and Environment, Kobe University
[email protected],
[email protected] 3 College of Information Science & Engineering, Ritsumeikan University
[email protected]
Abstract. In this study, we tried to examine empirically how body motion contributes to music expressivity, both in terms of intensity and manners, during impromptu jazz performances. Psychological rating experiments showed that music expressivity in jazz performances are assessed in two aspects, namely power and aesthetic quality. In the assessment of musical performances, the music itself basically contributed to how observers evaluated its expressivity. However, it was also shown that body motion had a greater influence on assessing the quality of music in terms of “hard or soft” and “light or heavy.”As a result of the three-dimensional motion analysis using motion capture, we learned that the characteristics of the player’s body motions changed with the playing style and the playing dynamics. The player, therefore, is making music not only by producing the “sound,” but by also showing “body motions” for creating that sound. Keywords: Jazz Performances, Music Expressivity, Body Movement, Motion Capture.
1 Introduction When performing classical music, the musicians share a general, predetermined understanding of the expression of their music and timing and play under the conductor’s direction. In impromptu jazz performances, however, the musicians begin to play their music only after agreeing on minimal requirements, such as the chord progression and structure of the musical composition they are about to perform. This means that each player must understand, on a real-time basis, the music expressivity of the other players and respond to or go along with them. Many studies on music expressivity have been conducted concerning its technical aspects, such as performance methods and their relationship with the impressions aroused by music, but hardly any empirical research has been conducted up to this point concerning body movements as representative of music expressivity; in other J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 159–168, 2009. © Springer-Verlag Berlin Heidelberg 2009
160
M. Sakata et al.
words, the visual expression of body movements. Recently, some studies (those by Davidson [1], Okada [2], and Maruyama [3] and others, or example) have made some reference to the role played by the body in musical performances, but such works are sporadic, and the discussion has really only just begun. This study aims to illuminate “the visual roles of body movements during impromptu performances of jazz music” and empirically show the modes and intensity of body movements that contribute to music expressivity. For this purpose, we employ “Kansei” information processing techniques, motion capturing, feature extraction from motion data and some statistical analyses. “Kansei” is a Japanese word whose meaning is close to “feeling” or “sensibility” in English. Kansei information processing is a method of extracting features related to Kansei conveyed by the media we receive. Conversely, it is also a method of adding or generating some Kansei factors to media produced by computers [4]. We employ motion capturing techniques for obtaining images of human body motions. This technique is now used commonly in movie and CG animation production. Several systems are commercially available nowadays. This study uses motion capturing to analyze jazz performances and quantitatively analyze the roles played by body movements.
2 Study Subjects For study subjects, we prepared materials with the help of professional jazz musicians in order to study the role of body movements in music expressivity. We asked a 10year veteran male alto-saxophone player (24 years old) to play the jazz standard “Summertime.” He played the front theme1, an ad-lib solo in the middle and the back theme for two choruses (with each chorus containing 16 bars). He was asked to play them in three different modes: “ordinary,” “expressionless” and “over expressive.” In this study, these three different modes of expression are defined as “expression dynamics.” In order to retain the characteristic feature of freestyle jazz performance, which is to perceive each other’s music expressivity in real-time, respond to them or follow them, we asked other players to join in the performance of our subject. The back band consisted of a drummer, a wood base player and a guitarist. The drummer was asked to keep the BPM=120 tempo, and all the other players were asked to follow the alto-saxophonist’s performance.
3 Motion Capture System We used an optical motion capture system (Motion Analysis Corporation, EvaRT with Eagle cameras) to measure body movements during a jazz performance. Fig. 1 shows a scene from the motion capturing session in our studio. Reflective markers were attached to the joints on the player’s body, and several high-precision, highspeed video cameras were used to track the motion. In our case, 33+2(on the instrument) markers were put on the player's body (see Fig. 2), and the movement was 1
The pre-composed part of a jazz number is called “theme”. In common jazz performances, musicians play the theme first, then the solo, and then the theme again.
Effect of Body Movement on Music Expressivity in Jazz Performances
161
measured with 10 cameras. The acquired data can be observed as a time-series using the three-dimensional coordinate values (x, y, z) of each marker in each frame (frame rate is 120 fps).
Fig. 1. Motion Capture
Fig. 2. Positions of markers
4 Psychological Rating Experiments In order to determine what kind of impressions are perceived from jazz performances, the object of our study, and how the different modalities, namely the “sound” and the “body,” contribute to musical expressivity, we conducted psychological rating experiments. Thirty-eight observers (20 men and 18 women) participated in this experiment. The mean and standard deviations of age among the 38 observers were 20.5 and 1.43, respectively. All observers had some training in jazz. 4.1 Stimuli for Experiments The motion measured by the motion capture system described above was filmed by a digital camera (SONY) and then edited to produce experimental stimuli. The stimuli were obtained by editing the performances of the front theme and ad-lib solo played with three different expression dynamics, and then further edited using the three modalities of “sound only,” “visual images only” and “sound and visuals.” Table 1 shows the order in which the stimuli were presented (the modalities, dynamics and styles) and the length of each stimulus. 4.2 Procedure We briefed the observers on the experiment, asked them to answer questions concerning their personal attributes and then presented the stimuli, one type at a time, in the order of “visual images only” -> “sound only”-> “sound and visuals.” We provided an interval after showing one type of stimulus twice. In the first showing of the stimuli, the subjects were asked to closely and carefully observe the stimuli. In the second showing, they were asked to fill in the Answer Sheet using the Assessment Words on
162
M. Sakata et al.
a scale from 1 to 7 for each word in the adjective pair, which are shown in Table 22. The videotaped recording was temporarily stopped during the intervals between showing the different types of stimuli. The new stimulus was presented after making sure all subjects finished filling in their Answer Sheet. Table 1. Order of stimuli Order of display 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Modality
Expression dynamics
Style
Duration
visual images only visual images only visual images only visual images only visual images only visual images only sound only sound only sound only sound only sound only sound only sound and visuals sound and visuals sound and visuals sound and visuals sound and visuals sound and visuals
ordinary expressionless over expressive over expressive expressionless ordinary expressionless over expressive ordinary over expressive expressionless ordinary over expressive expressionless over expressive expressionless over expressive ordinary
solo theme theme solo solo theme solo theme theme solo theme solo theme solo solo theme theme solo
64 sec 66 sec 64 sec 62 sec 63 sec 65 sec 63 sec 64 sec 65 sec 62 sec 66 sec 64 sec 65 sec 63 sec 62 sec 66 sec 64 sec 64 sec
Table 2. Assessment Words 1 2 3 4 5 6 7 loose --+--+--+--+--+--+--+-soft --+--+--+--+--+--+--+-powerful --+--+--+--+--+--+--+-clear-cut --+--+--+--+--+--+--+-impressive --+--+--+--+--+--+--+-have presence-+--+--+--+--+--+--+-neat --+--+--+--+--+--+--+-plain --+--+--+--+--+--+--+-happy --+--+--+--+--+--+--+-light --+--+--+--+--+--+--+-unique --+--+--+--+--+--+--+-fantasy-like--+--+--+--+--+--+--+-rich --+--+--+--+--+--+--+-beautiful --+--+--+--+--+--+--+-fast --+--+--+--+--+--+--+-warm --+--+--+--+--+--+--+-subdued --+--+--+--+--+--+--+-inarticulate--+--+--+--+--+--+--+-favorable --+--+--+--+--+--+--+-good --+--+--+--+--+--+--+--
2
tight hard weak unclear unimpressive no presence messy passionate sad heavy ordinary realistic poor ugly slow cold bright articulate unfavorable bad
In defining the Assessment Words used in our experiment, we referred to Iwamiya [5] and added our own 10 pairs of adjectives to create a group of Assessment Words and started our preliminary experiments using these words. After running a factor analysis, we deleted the terms that were not affecting any of the factors, deleted one of the highly-correlated pairs and finally selected 20 pairs of adjectives.
Effect of Body Movement on Music Expressivity in Jazz Performances
163
4.3 Results of Kansei Assessment Experiment The results of the assessments of each stimulus were converted into scores from 1 to 7 using the SD method. We also obtained the average for the adjectives. Extraction of KANSEI information from the stimuli. After conducting a principal component analysis based on the Kansei Assessment Scores obtained, we extracted two principal components with a characteristic value greater than 1. (The cumulative contribution rate was 0.879 up to the second principal component.) Table 3 shows the values of factor loading of each word pair to the two principal components. The shaded areas in the Table indicate the significant image word pair ratings to each principal component with a magnitude larger than 0.8. Table 3. Results of PCA for the rating experiment Assessment Words
PC1
PC2
Table 4. Result of multiple regression analysis Assessment Words
Standardized Coefficients Sound
Adjusted R2
Visuals
loose-tight
.882
.122
soft-hard
.650
.025
powerful-weak
.959
-.141
clear-unclear
.948
.252
powerful-weak
0.926
0.055
0.876
impressive-unimpressive
.964
.108
clear-unclear
0.942
0.081
0.932 0.928
loose-tight
0.851
0.204
0.931
soft-hard
0.593
0.601
0.897
.974
.158
impressive-unimpressive
0.999
-0.031
-.649
.737
0.931
0.077
plain-passionate
.932
.102
have presence-no presence
happy-sad
.749
-.524
neat-messy
0.773
0.222
0.914
light-heavy
.673
-.385
plain-passionate
0.841
0.193
0.959
happy-sad
0.617
0.433
0.847
light-heavy
0.587
0.656
0.959
unique-ordinary
-
-
-
fantasy-like-realistic
-
-
-
rich-poor
1.028
-0.047
0.991
beautiful-ugly
0.943
0.051
0.945
fast-slow
0.670
0.348
0.861
-
-
-
0.657
0.405
0.910 0.969
have presence-no presence neat-messy
unique-ordinary
.974
-.076
fantasy-like-realistic
.859
.036
rich-poor
.789
.589
beautiful-ugly
.308
.942
fast-slow
.743
-.653
warm-cold
.618
.640
subdued-bright
-.701
.706
inarticulate-articulate
-.464
.853
favorable-unfavorable
.490
.846
good-bad
.468
.865
Eigenvalue
11.708
5.877
Variance (%)
58.539 87.925
warm-cold subdued-bright
0.926
inarticulate-articulate
0.687
0.341
favorable-unfavorable
0.915
0.102
0.898
good-bad
0.884
0.145
0.988
From Table 3, one can interpret the PC1 to be the variable concerned with the “power” of a musical performance, and PC2 to be the variable concerned with “aesthetic quality.” We plotted the principal components of PC1 and PC2 on the x- and y-axes and also plotted 18 types of stimuli on a graph (see Fig. 3). On this graph, the presence or power of a performance increases as you move toward the right, while the aesthetic quality increases as you move upward. From the PCA results, it became clear that musical expressivity in jazz performance is perceived from two aspects, namely the “power” and “aesthetic quality.”
164
M. Sakata et al.
Fig. 3. Plot of PCA score for each motion
The Body and the Music in Music Expressivity. To examine which of the two factors, namely “body motions” and “sound,” contributes more to music expressivity, we made a multiple regression analysis for each of the adjectives listed in Table 2, using Kansei Assessment Scores obtained by showing “visual images only” (in other words, body motions only) and “sound only” (in other words, music only) as independent variables. The Kansei Assessment Scores obtained by showing the “sound and visuals” were treated as dependent variables. As a result, we obtained a multiple regression equation (p<0.05) with a high contribution rate for many of the adjectives, as shown in Table 4. Based on the information in Table 4, we see that music contributes more than the body motion (visual images) in the Kansei assessment, as expressed by the words “loose-tight,” “powerful-weak,” “clear-unclear,” “impressive-unimpressive,” “have presence-no presence,” “neat-messy,” “plain-passionate,” “happy-sad,” “rich-poor,” “beautiful-ugly,” “fast-slow,” “subdued-bright,” “inarticulate-articulate,” “favorableunfavorable” and “good-bad.” On the other hand, the body motion (visual images) contributes more than the music in the Kansei assessment, as expressed by words like “soft-hard” and “light-heavy.”
5 Feature Values for Body Motion In the current study, the angles of each body part (back, sides, and knee), velocity (finger tips, elbow, sacral, head, and toe) and the distance moved on the floor (heel movement per unit time) were adopted as “feature values for body motion.” 5.1 Extracting Physical Parameters Angle. This parameter shows how the various body parts change in a time-series manner during musical performances. In our study, we measured the angles of the back, the body’s side and the knee. In Fig. 2, the angle created by marker numbers 5, 18 and 31 shows the angle of the back. The angle created by marker numbers 7, 4 and
Effect of Body Movement on Music Expressivity in Jazz Performances
165
18 is the angle of the side of the body. The angle created by marker numbers 19, 21 and 31 is the angle of the knee. For example, in the case of the back, we set the origin at marker no.18 (x2, y2, z2) and measured the angleθ created between marker no. 5 (x1, y1, z1) and marker no. 31 (x3, y3, z3). Then, using Equation (1) below, we calculated cosθ and returned the obtained cosine radian to a radian using the Arc Cosine. Then we used the Degrees function to convert the angles in degrees into numerical values of the angle. (1) Velocity. This parameter shows the time-series change in the movement of the body parts during a musical performance. In the current study, we measured the speeds of the fingertips of the right hand (no.11), elbow (no.7), sacral (no.18), head (no.2) and the toe (no.27). For each marker, we obtained the Euclidian distance in the frames from the data expressed by the x, y and z coordinates. We then multiplied the Euclidian distance with the frame rate to obtain the time-series data of the velocity. For example, when the x, y and z coordinates of the marker in Frame i are expressed as xi, yi and zi, we can obtain the distance d from Equation (2). (2) Then, by multiplying the d with the motion data frame rate, 120[Frame/sec], we can obtain the velocity |v|. |v| = d * 120
(3)
For the elbow, we used the relative coordinates by using the shoulders as the origin. The sacral were set as the origin to obtain the relative coordinates for the head and toe. This means that the velocity of the elbow, head and toe is expressed by a relative velocity based on the shoulder and the sacral. We obtained the velocity of the fingertips of the right hand and the sacral by using absolute coordinates using an origin determined in the capture area. Floor Travel Distance. This parameter shows how much the player moved on the floor during the performance. We obtained the distance traveled by the left heel (no.32) frame by frame. 5.2 Feature Values for Body Motion By conducting a principal component analysis after obtaining each parameter (raw data) described in Section 5.1 above, we extracted three components with characteristic values greater than 1. (The cumulative contribution rate up to the third principal component was 0.782.) From the values of factor loading shown by the Table 5, we can interpret that PC1 is the component showing the velocity of the upper part of the body, PC2 is the component showing flow travel distance, and PC3 is the component showing the bending of the body.
166
M. Sakata et al. Table 5. Results of PCA for the motion capture data Physical parameters Angles of the back
PC1
PC2
PC3
-.088
.181
.240
.285
.601
Angles of the knee
-.396
.628
-.107
Speed of the hand
.925
.251
-.053
Speed of the elbow
.922
.278
-.093
Speed of the head
.922
.258
-.155
Speed of the sacral
.913
.065
.000
Speed of the toe
.883
-.312
.091
Floor travel distance
.422
-.744
.190
4.572
1.371
1.094
Angles of the body’s side
Eigenvalue Variance (%)
.801
50.798 66.029 78.186
Fig. 4. Plot of PCA score for each motion by motion capture data
In the left graph of Fig. 4, the centers of gravity for PC1 and PC2 are plotted on the x and y axes, and six types of stimuli are plotted. In this figure, the more you move to the right, the greater velocity at which the top part of the body moves. The lower you go on the figure, the more distance covered in floor travel. Looking at each stimulus, the player showed greater floor travel when playing solo than when playing the theme. In terms of expression dynamics, the velocity of the top part of the body increased in this order: “ordinary” -> “expressionless” -> “over expressive.” Likewise, we plotted the centers of gravity for the principal components of PC1 and PC3 on the x and y axes and plotted each stimulus in a graph, which is shown in the right graph of Fig. 4. In this graph, the more you go to the right, the greater velocity of the top part of the body, and the lower you go, the more bending there is of the body. This graph shows that playing solo, rather than playing the theme, resulted in greater body bending. And when playing either solo or the theme, the player’s body bending was greatest during the “ordinary” mode of playing.
Effect of Body Movement on Music Expressivity in Jazz Performances
167
5.3 Relationship between Kansei Assessment and Feature Values for Body Motion We calculated the average and standard deviations for each stimulus for the nine parameters obtained in Section 5.1. In order to examine the relationship between the Kansei Assessment and the Feature Values for Body Motion, we calculated the coefficient of correlation between the principal component score of each performance obtained in Chapter 4 and the characteristic value of body motion of each performance(see table 6). The shaded area shows the combinations that showed a significant correlation of 5%. In the “Power Component,” we found a significant correlation between all Power Components and body motion parameters, except for the parameters “average angle of body’s side,” “average foot travel distance” and “standard deviation of foot travel distance.” For the Aesthetics Components, on the other hand, we could not find any correlation with any of the body motion parameters. Table 6. Correlation matrix Physical parameters Angles of the back Angles of the body’s side Angles of the knee Speed of the hand Speed of the elbow Speed of the head Speed of the sacral Speed of the toe Floor travel distance
Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD
PC1 Power -0.889* 0.880* 0.377 0.879* -0.949* 0.880* 0.894* 0.927* 0.908* 0.954* 0.931* 0.968* 0.908* 0.944* 0.881* 0.859* 0.710 0.678
PC2 Aesthetics 0.130 0.163 0.040 0.254 0.080 0.211 0.244 0.242 0.241 0.226 0.209 0.194 0.247 0.212 0.256 0.270 0.269 0.198
*p<0.05
This means that in musical performance, body motion contributes a large measure to the Power, but not to the Aesthetics.
6 Discussion and Conclusion In this study, we tried to examine empirically how body motion contributes to music expressivity, both in terms of intensity and manners, during impromptu jazz performances. Psychological rating experiments showed that music expressivity in jazz performances are assessed in two aspects, namely power and aesthetic quality. In the Kansei assessment of musical performances, the music itself basically contributed to how observers evaluated its expressivity. However, it was also shown that body motion
168
M. Sakata et al.
had a greater influence on assessing the quality of music in terms of “hard or soft” and “light or heavy.” As a result of the three-dimensional motion analysis using motion capture, we learned that the characteristics of the player’s body motions changed with the playing mode and the playing dynamics. The player, therefore, is making music not only by producing the “sound,” but by also showing “body motions” for creating that sound. It was found that body motions had a great role in creating “power,” but were not much related to the “aesthetics quality.” Naturally, the Kansei emanated from the sound itself is central to music expressivity. However, we have shown empirically that the body motions people make when making music also contribute greatly in music expressivity. This study offers a basic examination of the role of body motions in musical performances; however, many challenges and problems still remain to be explored. Acknowledgments. This work was supported in part by the Grant-in-Aid for Scientific Research, for Young Scientists (B) No. 19700493 of the Ministry of Education, Culture, Sports, Science and Technology, Japan. The authors would like to express their sincere gratitude to Mr. Junya Kondo for his cooperation with our research. Thanks are also due to Mr. Takahiro Yorino, Ms. Hitomi Toyoka and Mr. Naoyuki Okamoto for their kind help in motion capturing experiment.
References 1. Davidson, J.W.: Visual Perception of performance manner in the movements of Solo Musicians. Psychology of Music 21, 103–113 (1993) 2. Okada, A.: The body playing the piano. Shunjusha Publishing Company (2003) (in Japanese) 3. Maruyama, S.: The Embodied Sense of Music: Case Studies on the Rhetorical Function of Bodily Gestures by Highly Practiced Musicians. Cognitive studies: bulletin of the Japanese Cognitive Science Society 14(4), 471–493 (2007) (in Japanese) 4. Sakata, M., Hachimura, K.: KANSEI Information Processing of Human Body Movement. In: Smith, M.J., Salvendy, G. (eds.) HCII 2007. LNCS, vol. 4557, pp. 930–939. Springer, Heidelberg (2007) 5. Iwamiya, S.: Multimodal Communication of Music and Image. Kyushu University Press (2000) (in Japanese) 6. Iwamiya, S.: Design of Sounds. Kyushu University Press (2007) (in Japanese) 7. Nagashima, Y.: Drawing-in Effect on Perception of Beats in Multimedia. The Journal of the Society for Art and Science 3(1), 108–148 (2004)
A Method to Monitor Operator Overloading Dvijesh Shastri, Ioannis Pavlidis, and Avinash Wesley Computational Physiology Lab, Department of Computer Science, University of Houston, Houston, TX, 77204 {dshastri,ipavlidis,awesley}@uh.edu
Abstract. This paper describes research that aims to quantify stress levels of operators who perform multiple tasks. The proposed method is based on the thermal signature of the face. It measures physiological function from a standoff distance and therefore, it can unobtrusively monitor a machine operator. The method was tested on 11 participants. The results show that multi-tasking elevates metabolism in the supraorbital area, which is an indirect indication of increased mental load. This local metabolic change alters heat dissipation and thus, it can be measured through thermal imaging. The methodology could serve as a benchmarking tool in scenarios where an operator’s divided attention may cause harmful outcomes. A classic example is the case of a vehicle driver who talks on the cell phone. This stress measurement method when combined with user performance metrics can delineate optimal operational envelopes. Keywords: Human-Machine Interaction, divided attention, stress, thermal imaging.
1 Introduction In many occasions, machine operators simultaneously perform more than one task. When a combination of overlapping events demands critical decisions and rapid actions, they raise the operator’s alertness. If this persists, it develops to stress that overloads the operator. Stress due to operator’s divided attention may lead to degradation of his/her performance. This work aimed to analyze stress-induced change in facial physiology during divided attention situations. The supraorbital thermal signature emerged as the physiological variable of interest, which can be measured through thermal imaging. In fact, thermal imaging based stress monitoring is an increasingly popular approach [1] [2] [3]. In contrast to probe based stress monitoring methods, it is totally unobtrusive [7] [8]. Although concurrent performance of multiple tasks is part of human life, insufficient research has been done to understand its effect on human emotional states and performance. The purpose of this study is to develop an effective tool to gauge stress load of operators engaging in multi-tasking. The experimental design focused on cellphone communication during driving simulation [9][10][11] and was tested on 11 participants. However, other task pairs could have been chosen, such as reading in the presence of background distractions. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 169–175, 2009. © Springer-Verlag Berlin Heidelberg 2009
170
D. Shastri, I. Pavlidis, and A. Wesley
2 Methodology During dual tasking there is considerable temperature increase in the supraorbital region of a participant. This locally elevated temperature is the result of increased metabolic activity due to activation of the forehead muscle group. The phenomenon is consistent with findings in prior experiments involving Stroop color conflict testing [1]. In that previous work, the stress signal was extracted from the evolution of the mean thermal footprint of the entire supraorbital region (see Fig. 1. a). This approach, however, introduced noise in the extracted signal, partly due to the wide probing area and partly due to sub-optimal tracking performance [4]. In the present work, the tracking region was differentiated from the measurement region. An even bigger area that included sharp contrasts (e.g., skin versus hair) was selected for tracking. This improved tracking performance. However, only a small subset within the tracking region was selected for the thermal measurement. This subset was confined in the area were metabolic changes are more dramatic, to reduce the effect of probing noise (Fig. 1. b).
(a)
(b)
Fig. 1. (a) Tracking and measurements regions coincide in legacy method. (b) Measurement region (in pink) is a subset of tracking region in the current method.
For every participant in the experiment, tracking and measurement regions of interest were selected as described above. The mean temperature of the measurement region of interest was computed for every frame in the thermal clip. Thus, a 1D supraorbital signal was produced from the 2D thermal data. Any residual noise in the supraorbital signal was suppressed by a noise cleaning algorithm based on Fast Fourier Transform (FFT) [5]. The supraorbital signal was split into segments corresponding to the phases of the experiment (resting, initial single task, dual task, latter single task, and cool-off). Each segment was approximated with a linear fit (Fig. 2), which described the local metabolic rate at the time.
3 Experimental Design A high quality Thermal Imaging (TI) system was used for data collection. The centerpiece of the TI system was a ThermoVision SC6000 Mid-Wave Infrared (MWIR) camera (FLIR Systems) (sensitivity = 0.025oC) [12]. The experimental protocol included thermal imaging of the participant’s face while he was resting, engaging in a
A Method to Monitor Operator Overloading
171
driving simulation game (single task), engaging in a driving simulation game and talking on the cell phone (dual task), and relaxing. The dataset featured participants of both genders, different races, and with varying physical characteristics. The participants were placed 6 feet away from the thermal camera (Fig. 3). We used a XBOX-360 game console and the Test Drive: Unlimited game to simulate driving. The participants were asked to follow all traffic signs, drive normally, and not race during the experiment. They were given an opportunity to testdrive before the experiment began to acquaint themselves with the driving simulation. In the first formal phase of the experiment, the participants were asked to rest for 5 min while being imaged. This helped to isolate effects of other stress factors that participants may have carried inside them. This was the baseline phase.
Fig. 2. Supraorbital raw thermal signal (marked in blue color), noise cleaned thermal signal (marked in pink color) and linear fitting (marked in yellow color). Slope values for the linear segments are shown in blue colored text.
Next, the participants were asked to play the driving simulation game. This phase of the experiment also lasted about 5 min. After around 1 min of driving simulation (initial single task phase), the participant received a cell phone call that played a set of prerecorded questions in the following order: Instruction: Please do not hang up until you are told so. Q1: Are the lights ON in the room, yes or no? Q2: Are you a male or female? Q3: Who won the American civil war, the north or the south? Q4: What is 11 + 21? Q5: How many letters ‘e’ are in the word experiment? Q6: I am the son of a mom whose mother in law's son hit. How am I related to the other son? Q7: My grandma's son hit his son. How are the sons related? Q8: A man is injured in 1958 and died in 1956. How is that possible? Q9: What is 27 + 14? Instruction: You may now hang up the phone and pay attention to the game.
172
D. Shastri, I. Pavlidis, and A. Wesley
The question set was a combination of basic, logical, simple math, and ambiguous questions. The order of the questions was designed to build-up pressure on the participants. Additional pressure was achieved by repeating one more time every question that was incorrectly answered. The participants were supposed to drive while talking on the cell phone (dual task phase). At the end of the phone conversation, participants put the phone down and continued driving till the end of the experiment (the latter single task phase). Finally, the participants relaxed for 5 minutes. The purpose of this so-called cool-off segment was to monitor physiological changes after the simulated driving experiment.
Fig. 3. Experimental setup: Participant, imaging equipment, xBOX-360
4 Experimental Results The slopes of the linearly fitted segments, computed according to the method described in Section 2, were used as stress indicators. Fig.4 shows the mean slope values of the various segments for the entire data set (statistically constructed mean participant). The graph clearly indicates that the temperature increase during dual tasking is the highest among all segments (sole exception was participant S-6). Since the temperature increase is correlated to metabolic rate, the results indicate elevated metabolism in the supraorbital region during dual tasking. This is presumably due to strong muscle activation, associated with frowning, a facial expression autonomically associated with mental engagement. Stress during the latter single task phase was stronger than the initial single task phase (Fig. 4). Apparently, this was due to residual effect from the dual task that preceded the latter single task phase. Most of the participants admitted during debriefing that they were thinking about their dual task performance while performing the latter single task.
A Method to Monitor Operator Overloading
173
Fig. 4. Mean slope value of the experimental segments. This stress indicator is the highest during dual tasking.
Interestingly, baseline stress was a bit higher than the initial single task stress. This indicates that either the baseline was poorly designed (just sitting idle can be stressful) or participants carried some residual stress from the informal test-driving phase that preceded it. Participant S-6 is an interesting case. It appears that his stress started decreasing in the middle of dual tasking (Fig. 5). On careful examination of the data, the investigators found that this is the only participant who started perspiring in the middle of the experiment, apparently due to overwhelming stress. Perspiration reduces the thermal signature and the current method wrongly interprets this as lower metabolic rate and thus, lower stress. It is exactly the opposite. A method that identifies the emergence of perspiration and switches measurement metrics is needed to overcome this issue. In all cases, the rate of thermal change of the cool-off segment had an opposite global trend than that of the dual task segment. In most cases, the rate of thermal change of the cool-off segment had also an opposite global trend than that of the initial and latter single task segments. This illustrates that the participants indeed felt relaxed after 5 minutes of intense mental activity but at various degrees. Those participants who were thinking about their performance during the cool-off period exhibited slow recovery. This is an interesting finding, as it illustrates that not only an action but also thoughts about the action (past action in this case) could affect stress levels. Performance of the drivers degraded during the dual task segment, as measured by the point system of the simulator. This was inversely proportional to the mean stress level measured through the supraorbital channel.
174
D. Shastri, I. Pavlidis, and A. Wesley
Fig. 5. During the dual task period, the supraorbital signal (marked in blue color) of S6 showed ascending global trend in the first half and then descending global trend in the second half (marked in green color). The culprit is the onset of perspiration.
5 Conclusion This research brings to the fore a stress quantification method ideally suited to situations where the attention of the machine operator is divided. Unobtrusive quantification of stress and its correlation to operator performance and emotions are of singular importance in man-machine interaction. A feedback system can be developed that alerts the operator about his/her stress status based on the facial thermal signature. Results from a pilot experiment on the effect of cell phone communication during driving are more than encouraging. They open the way for a plethora of other multitasking experiments drawn from daily life. At the technical level, the issue of perspiration, which corresponds to the onset of extreme stress, cannot be handled with the current method. A method that identifies perspiratory patterns and handles thermal computation in a different manner from that point onward is needed in the future.
Acknowledgments This material is based upon work supported by the National Science Foundation under Grant No. #ISS-0812526, entitled “Do Nintendo Surgeons Defy Stress.” Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References 1. Puri, C., Olson, L., Pavlidis, I., Levine, J., Starren, J.: StressCam: Non-contact measurement of users’ emotional states through thermal imaging. In: CHI 2005 Extended Abstracts on Human Factors in Computing Systems, pp. 1725–1728 (2005)
A Method to Monitor Operator Overloading
175
2. Palvidis, I., Eberhardt, N.L., Levine, J.: Human behavior: Seeing through the face of deception. Nature 415, 35 (2002) 3. Palvidis, I., Levine, J.: Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine 21(6), 56–64 (2002) 4. Dowdall, J., Pavlidis, I., Tsiamyrtzis, P.: Coalitional tracking. Computer Vision and Image Understanding 106, 205–219 (2007) 5. Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I., Frank, M.G., Ekman, P.: Imaging facial physiology for the detection of deceit. International Journal of Computer Vision 71(2), 197–214 (2006) 6. Standage, D.I., Trappenberg, T.P., Klein, R.M.: A continuous attractor neural network model of divided visual attention. In: Processing of IEEE international Joint Conference on Neural Networks, vol. 5, pp. 2897–2902 (2005) 7. Yamakoshi, T., Yamakoshi, K., Tanaka, S., Nogawa, M., Shibata, M., Sawada, Y., Rolfe, P., Hirose, Y.: A Preliminary Study on Driver’s Stress Index Using a New Method Based on Differential Skin Temperature Measurement. In: 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 722–725 (2007) 8. Yamaguchi, M., Wakasugi, J., Sakakima, J.: Evaluation of Driver Stress using Biomarker in Motor-vehicle Driving Simulator. In: 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1834–1837 (2006) 9. Yili, L.: A queuing network model of human performance of concurrent spatial and verbal tasks. IEEE Transactions on Systems, Man and Cybernetics 27(2), 195–207 (1997) 10. Kenmochi, A., Takaki, Y., Fukuzumi, S.: Psychological tension estimation during the use of a driving simulator: A finger and ear pulse volume study. In: Proceedings of the 18th Annual International Conference of the IEEE Bridging Disciplines for Biomedicine, vol. 5, pp. 1804–1805 (1996) 11. Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6, 156–166 (2005) 12. FLIR Systems, 70 Castilian Dr., Goleta, California 93117, http://www.flir.com
Decoding Attentional Orientation from EEG Spectra Ramesh Srinivasan, Samuel Thorpe, Siyi Deng, Tom Lappas, and Michael D’Zmura Department of Cognitive Sciences, UC Irvine, SSPA 3151, Irvine, CA 92697-5100 {r.srinivasan,sthorpe,sdeng,tlappas,mdzmura}@uci.edu
Abstract. We have carried out preliminary experiments to determine if EEG spectra can be used to decode the attentional orientation of an observer in threedimensional space. Our task cued the subject to direct attention to speech in one location and ignore simultaneous speech originating from another location. We found that during the period where the subject directs attention to one location in anticipation of the speech signal, EEG spectral features can be used to predict the orientation of attention. We propose to refine this method by training subjects using feedback to improve classification performance. Keywords: EEG, attention, orienting, classification.
1 Introduction This report summarizes our preliminary studies towards developing a BCI to decode intended direction, i.e., the attentional orientation of an individual, from EEG recordings alone. This effort is part of a larger project (ARO 54228-LS-MUR, Silent Communication Among Dispersed Forces) to develop a BCI to decode intended speech and intended direction from EEG signals. The most obvious indicators of attentional orientation are head and eye orientation. However, even when head and eyes are oriented in one direction, an observer can attend to other directions. In experimental psychology, this is referred to as covert attention[1-3]. Covert attention involves all the senses including visual, somatosensory, and auditory information processing [4-5]. Auditory information processing is particularly relevant to covert attention, because the auditory system can identify sources which lie outside the visual scene, including to the side of and behind the observer. Mechanisms which monitor and detect sources throughout the environment selectively are essential to guide head and eye movements. The close relationship between attention and orienting eye and head movements is supported by experimental studies and theoretical models which suggest a motor programming role for attention [6-7]. Most previous EEG and fMRI studies focused on the effects of orienting attention on the response to a sensory stimulus [8-9]. Across all of these studies, the main effect is that larger responses are recorded to attended stimuli. More recently, a more nuanced view has emerged which suggests that distinct brain networks respond to attended and unattended stimuli [10-11]. Our goal is to identify the neural signature of orienting attention in one direction even when there is no stimulus at that location. Thus, we are interested in the J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 176–183, 2009. © Springer-Verlag Berlin Heidelberg 2009
Decoding Attentional Orientation from EEG Spectra
177
top-down instruction to orient attention in one direction (or to one location) rather than the orienting elicited by a salient stimulus (bottom-up). Recently, a number of EEG and fMRI studies have been directed at this question and have demonstrated preparatory neural activity when attention is directed by instruction (cued) to one location on a computer screen [12-15]. The fMRI studies examined retinotopically mapped areas of the visual system and demonstrated preparatory increases in neuronal activity as indexed by the BOLD signal. EEG studies showed increased alpha rhythm in the parietal cortex ipsilateral to the attended visual field. Although these studies have elucidated some of the mechanisms of top-down attentional orienting, they are limited in their usefulness towards developing a BCI that can decode attentional orientation. In general, orienting attention takes place in a larger sensory field, not just a limited sector of visual space within 10-12 degrees of fixation on a computer monitor. In audition, perception of sources takes place in all directions, even behind the subject. Thus, in our experiment observers selectively attend to auditory rather than visual stimuli in order to investigate attentional orientation in a wider span of sensory space. We have carried out experiments directing attention to one of two directions and identify the attended direction by classification of the EEG spectra. Our results suggest that attentional orientation can potentially be decoded from the EEG but that further work is needed to train the observers and improve classification methods.
2 Methods Procedure. Six subjects participated in a speech perception experiment (see Figure 1). The subject was seated in a dimly lit room between two speakers (each at 1 m distance) and instructed to fixate on a point. There were two experimental conditions – attend left or attend right. The subject is given the instruction through both speakers. After a variable ISI (500, 700, 900, 1100, or 1300 ms), two different speech stimuli were presented one through each speaker. The speech stimuli were synthesized (http://cepstral.com) in two distinct male voices, one played through each speaker. The assignment of voices to speakers was independently randomized on each trial. The stimuli were a simplified version of the Coordinate Response Measure corpus [16]. These sentences were structured as “(Arrow, Baron, Eagle or Tiger) go to (Blue, Green, Red, or White) now”. The subjects’ task was to identify the two words played through the attended speaker with a response on a keypad. In the example shown in the figure the correct response is “Baron” and “Blue”. The experiment was designed in this manner to demand that the subject direct attention to the correct speaker before the speech was played; otherwise the subject would miss the first code word. The variable ISI and randomized voice ensured that the observer quickly deployed and maintained attention to the appropriate speaker. In addition an adaptive staircase procedure was used to control subject performance. When the subject responded correctly to the first word, the volume was reduced by 5 % on the attended speaker and increased by 5% on the unattended speaker on the next trial. When the subject responded incorrectly to the first word, the volume was increased by 10% on the attended speaker and decreased by 10% on the unattended speaker. This procedure resulted in subjects performing the task correctly about 70% of the time.
178
R. Srinivasan et al.
Fig. 1. The experimental setup. A. The physical layout of the experiment. Note that each speaker was 45 degrees away from fixation and could not be seen by observer without moving the eyes. B. The time course of each trial in the experiment. In the example shown, the instruction is to attend to the right and after a variable ISI, two distinct sentences are played through each speaker. In the example shown, the subject responds “Baron” and “Blue”, and received feedback indicating a correct response.
At this threshold, the amplitude of the attended speaker was typically 30 dB below the unattended speaker. A single experimental session comprised 200 trials, presented in two 100 trial blocks with a break. Each subject participated in three such sessions, each lasting around an hour. EEG recording. EEG was recorded using a 128 Channel Geodesic Sensor Net (Electrical Geodesics, Inc., Eugene, OR, USA) in combination with an amplifier and acquisition software (Advanced Neuro Technology, Inc, Enschede, NL). The EEG was sampled at 1024 Hz and on-line average referenced. Artifact editing was performed through a combination of automatic editing using an amplitude threshold and a manual editing to check the results. Trials with excessive bad channels (> 15 %) were first discarded and then channels with bad trials were discarded. This typically yielded 80100 usable EEG channels (out of 128) and 500-550 usable trials. Data Analysis. We focused on three intervals within the ISI – 0-500, 200-700, and 400-900 ms. For the later intervals, we discarded trials where the target stimulus had already started, resulting in different numbers of trials for the three intervals – 600, 480, and 360. The data were Fourier transformed using an FFT (Matlab) for each 500 ms interval ( f = 2 Hz) and the power spectrum calculated as the squared magnitude of the Fourier coefficients. We limited further analyses to the interval from 4-22 Hz.
Decoding Attentional Orientation from EEG Spectra
179
This was motivated by the goals of this study to identify robust spectral features that can predict attentional orientation. Below 4 Hz EEG is often contaminated with movement and eye blink artifacts. Above 20 Hz the EEG is often contaminated with EMG artifacts. EEG power at each frequency was log transformed and normalized against the total power from 4-22 Hz. Classification. Classification was performed using a naïve Bayes classifier (Matlab) which assumes independent variables, i.e., a diagonal covariance matrix. The data were divided into three conditions based on instruction and performance: Correct Left, Correct Right, and Incorrect. There were roughly equal numbers of Correct Left
Fig. 2. Examples of classification performance versus the number of variables for the three classification intervals. Overall S0 had the best classification performance and S5 had the worst performance. The other subjects showed intermediate levels of classification performance.
180
R. Srinivasan et al.
and Correct Right trials, and 10-20% fewer Incorrect trials. To facilitate comparisons in classification performance between the analysis intervals, we used a fixed number (determined by the 400-900ms interval) of randomly selected “training” trials (typically 150) to calculate a linear classifier, which was applied to classify another 150 “test” trials. The classification proceeded in two steps. First we evaluated the performance of each individual variable (80-100 channels x 10 frequencies) in classification of the “test” data, using 30 random samples of “training” and “test” trials. Then we evaluated the performance of the best 10, 20, 50, 75, and 100 variables using 50 random samples of “training” and “test trials”.
Fig. 3. Power spectra from the channel that provided the best three-way classification for each subject. For 3 subjects (S0, S3, and S7) the channel shown is located over parietal cortex. For the other subjects (S2, S4 and S5) the channel shown is located over the frontal lobes. Note that spectral differences are present at most frequencies.
Decoding Attentional Orientation from EEG Spectra
181
Fig. 4. Topographic distribution of channel sensitivity to attentional orientation. Each channel classification ability was scored and averaged across frequencies. The results were plotted as a topographic map, where white shows the highest predictive power and black indicates no predictive power.
3 Discussion In a simple cued spatial attention experiment we were able to demonstrate preliminary results indicating that the EEG contains information that can be used to decode the orientation of attention in space. Our approach here has been very direct and overly
182
R. Srinivasan et al.
simplistic. We have made use of spectral features of the EEG and a naïve Bayes classification scheme. Both approaches can be significantly improved. Despite the limitation of these methods, we were able to achieve as high as 75% classification performance in 2-way classification and 60% classification performance in 3-way classification. This indicates that EEG signals clearly contain information about attentional orientation that can be used to decode attentional orientation for BCI applications. The data also suggest that the signatures of attentional orientation can be more robustly decoded at a longer interval following the cue. In our comparisons, 400-900 ms after the cue provided the best classification in each subject. This is consistent with theoretical and experimental studies of the episodic nature of attention, which suggested that following the cue the attention window will take at least 300 ms to open [17]. How long this window remains open and whether more robust classification can be obtained remains to be tested. Estimating the duration for which the neural signatures of attentional orientation are sustained will require a larger amount of data. Our results suggest that electrodes over frontal and parietal cortex have the greatest sensitivity to attentional orientation. This finding is consistent with a large body of experimental research with EEG and fMRI that indicate that large-scale networks spanning parietal and frontal cortex mediate selective attention [18-20]. More surprising was the lack of frequency specificity in our classification results. Previous reports had suggested occipital/parietal alpha rhythms would be sensitive to attentional orientation [15, 21]. However, those studies used visual displays where attention was directed to one of two regions within 4-6 degrees of fixation. Thus, those results were specifically related to attentional orienting within the narrowly defined retinotopically mapped visual space. Our results relate to orienting to two regions of auditory space separated by 90 degrees and out of the field of view while fixating. The results of this paper are preliminary and indicate the potential for obtaining an attentional orientation BCI. An important factor we have not yet considered is training subjects to optimize the BCI. Our current work extends this study through a consideration of the full 360 degrees of auditory space surrounding the subject.
Acknowledgements This work was supported by ARO 54228-LS-MUR.
References 1. Desimone, R., Duncan, J.: Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222 (1995) 2. Egeth, H.E., Yantis, S.: Visual attention: control, representation, and time course. Annual Reviews in Psychology 48, 269–297 3. Kastner, S., Ungerleider, L.G.: Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience 23, 315–341 (2000) 4. Spence, C., Pavani, F., Driver, J.: Crossmodal links between vision and touch in covert endogeneous spatial attention. Journal of Experimental Psychology: Human Perception and Peformance 26, 1298–1319 (2000)
Decoding Attentional Orientation from EEG Spectra
183
5. Spence, C., Driver, J.: Crossmodal attention. Current Opinion in Neurobiology 8, 245–253 (1998) 6. Moore, T., Armstrong, K.M., Fallah: Visuomotor origins of covert spatial attention. Neuron 40, 671–683 7. Sheliga, B.M., Riggio, L., Rizzolatti, G.: Orienting of attention and eye movements. Experimental Brain Research 98, 507–522 (1994) 8. Hillyard, S.A., Anllo-Vento, L.: Event-related brain potentials in the study of visual selective attention. PNAS 95, 781–787 (1998) 9. Kastner, S., Pinsk, M., De Weerd, P., Desimone, R., Ungerleider, L.: Increased activity in human visual cortex during directed attention in the absence of visual stimulation. Neuron 22, 751–761 (1999) 10. Corbetta, M., Shulman, G.L.: Control of goal- and stimulus-driven attention in the brain. Nature Reviews Neuroscience 31, 201–215 (2002) 11. Ding, J., Sperling, G., Srinivasan, R.: SSVEP power modulation by attention depends on the network tagged by the flicker frequency. Cerebral Cortex 16, 1016–1029 (2006) 12. Sylvester, C., Jack, A., Corbetta, M., Shulman, G.: Anticipatory Suppression of Nonattended Locations in Visual Cortex Marks Target Location and Predicts Perception. Journal of Neuroscience 28(26), 6549–6556 (2008) 13. Shulman, G., Ollinger, J., Akbudak, E., Conturo, T., Snyder, A., Petersen, S., Corbetta, M.: Areas Involved in Encoding and Applying Directional Expectations to Moving Objects. The Journal of Neuroscience 19(21), 9480–9496 (1999) 14. Fu, K., Foxe, J., Murray, M., Higgins, B., Javitt, D., Schroeder, C.: Attention-dependent suppression of distracter visual input can be cross-modally cued as indexed by anticipatory parieto–occipital alpha-band oscillations. Cognitive Brain Research 12, 145–152 (2001) 15. Worden, M., Foxe, J., Wang, N., Simpson, G.: Anticipatory Biasing of Visuospatial Attention Indexed by Retinotopically Specific alpha band Electroencephalography Increases over Occipital Cortex. The Journal of Neuroscience 20 (2000) 16. Moore, T.J.: Voice communication jamming research. In: AGARD Conference Proceedings 331: Aural Communication in Aviation, vol. 2, pp. 1–6 (1981) 17. Weichselgartner, E., Sperling, G.: Dynamics of automatic and controlled visual attention. Science 238, 778–780 (1987) 18. Gitelman, D., Nobre, A., Parrish, T., LaBar, K., Kim, Y., Meyer, M., Mesulam, M.: A large-scale distributed network for covert spatial attention, Further anatomical delineation based on stringent behavioural and cognitive controls. Brain 122(6), 1093–1106 (1999) 19. Posner, M., Petersen, S.: The attention system of the human brain. Annual Review Neuroscience 13, 25–42 (1990) 20. Coull, J.T.: Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology. Progress in Neurobiology 55(4), 343– 361 (1998) 21. Foxe, J., Simpson, G., Ahlfors, S.: Parieto-occipital ~10 Hz activity reflects anticipatory state of visual attention mechanisms. NeuroReport 9, 3929–3933 (1998)
On the Possibility about Performance Estimation Just before Beginning a Voluntary Motion Using Movement Related Cortical Potential Satoshi Suzuki1, Takemi Matsui1, Yusuke Sakaguchi1, Kazuhiro Ando1, Nobuyuki Nishiuchi1, Toshimasa Yamazaki2, and Shin’ichi Fukuzumi3 1
Tokyo Metropolitan University, Asahigaoka 6-6, Hino, Tokyo 191-0065, Japan Kyusyu Institute of Technology, Kawazu 680-4, Iizuka, Fukuoka 820-8502, Japan 3 NEC Common Platform Software Research Laboratories, Shibaura 2-11-5, Minatoku, Tokyo 108-8557, Japan
[email protected] 2
Abstract. The present study aimed to investigate this tripartite relationship, regarding MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance. Experiments were conducted 'reaching' task; the subject touches the target appears 300 pixels away from the start point in a vertical direction on the touch sensitive screen with the forefinger. During experiments, EEG, EMG as trigger, image by high-speed camera and the efficiency of task were acquired. As a result, significant differences between the high and poor performance groups were clear on the NS in MRCP acquired from Fz (p < 0.05), Cz (p < 0.05) and Pz (p < 0.05). Furthermore, the difference was confirmed on the duration of ballistic movement. Based on our findings, we attempted to extract MRCP rapidly and automatically without using signal averaging and discuss whether it is possible to estimate accuracy just before the motion is executed. Keywords: Accuracy, ballistic movement, movement-related cortical potential (MRCP), reaching, voluntary motion.
1 Introduction Movement-related cortical potential (MRCP) is a feature of brain waves that appears toward the midline of the head approximately one to two seconds before the onset of voluntary motion [1, 2, 3]. Although the divisions differ slightly between researchers, MRCP is usually separated into the Bereitschafts Potential (BP), the Intermediate Slope (IS) and the Negative Slope (NS). The BP and IS appear at the front and top of the head and have a gradual slope from 2000-500 msec before movement, while the NS starts approximately 500 msec before motion and, in the case of hand movement, has an amplitude of 10 – 15 μV [3]. It can also be observed if the subject imagines the motion in their head without undertaking any real activity [4]. Recently, some studies have demonstrated a relationship between MRCP and eye movements [5], and J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 184–191, 2009. © Springer-Verlag Berlin Heidelberg 2009
On the Possibility about Performance Estimation
185
to use as a method of medical diagnostics [6], although not all details have been fully clarified. Previous work has also shown that the thinking process and information processing within the brain is broken into several steps. Many useful models of this information processing in the head are proposed [7, 8]. These models commonly have 4 steps: perception, cognition, motion planning and motion. Related to this model, MRCP is generally assumed to reflect a preparation or planning stage for beginning the motion [9]. On the other hand, “reaching”, such voluntary goal-directed movement, is known to be one of the most important components of motion by human arms. Mathematical models considering jerk, torque and dynamics of the musculoskeletal system have been used to understand how the motion of reaching is planned and controlled in the brain [10, 11, 12]. Movement efficiency has also been studied and Fitts’ Law proved that reaction time during reaching constantly changes according to the difficulty of the task [13, 14]. During the motion process reaching has two characteristics, the ballistic and corrective movements [15, 16], which influence both the accuracy and duration time of the motion. Observation of these characteristics enables us to comprehend the efficiency of the task performance. There is believed to be a relationship between the ballistic movement and MRCP as the former is a feed-forward movement and MRCP reflects a planning stage for this motion. Building on this, a relationship has also been suggested between MRCP, the ballistic movement and the accuracy of motion (Figure 1). Considering the tripartite relationship among MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance has meaning for ergonomics field. Motion Start Perception Cognition Controlled Processing
Information Processing
Action
MRCP EEG
Motion
BP
IS
NS N-P feedforward
feedback
Ballistic Movement
Collective Movement
Task Performance
Motion Time Accuracy
Fig. 1. The concept of this research. The aim of this study is an investigation the tripartite relationship, regarding MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance.
186
S. Suzuki et al.
The present study aimed to investigate this tripartite relationship, regarding MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance within the field of ergonomics. Based on our findings, we attempted to extract MRCP rapidly and automatically without using signal averaging. We explain and consider the results of this extracting system and discuss whether it is possible to estimate accuracy just before the motion is executed.
2 Methods 2.1 Electroencephalogram MRCP is generally observed at scalp positions Fz, Pz and Cz, on the midline of the frontal and parietal regions of the head. To achieve high spatial resolution around these regions, electroencephalogram (EEG) data were acquired using a 128ch sensor net (Geodegic Sensor Net, Electrical Geodesics Inc., OR, USA) and analyze system (Net Station 4.3, Electrical Geodesics Inc., OR, USA). Electrode impedance was set between 10-50 kΩ. EEG was recorded using a 0.1–50 Hz bandpass (3 dB attenuation). Signals were sampled at 1 kHz and were digitized. 2.2 EMG and Trigger The trajectory of voluntary motion was observed using a high-speed camera (Fastcam512PCI, Fotron Co., Tokyo, Japan) with 125 fps (Figure 2). The trigger signal was generated using the surface EMG on the common digital extensor muscle. Data from the EEG and the high-speed camera were clipped at each trial using triggers. EMG data were acquired using a bio-amp system (Biotop, NEC-Sanei Co., Tokyo, Japan) with a sampling rate of 10 kHz to accurately synchronize and analyze in real time using a Field Programmable Gate Array (FPGA) module (PCI-7831, National Instruments Co., TX, USA). 2.3 Subjects and Task Procedure Experiments were conducted on eight healthy male subjects ranging in age from 22 to 24 years old (average 22.88 +/- 0.83 years). All subjects were right-handed as confirmed by the Edinburgh handedness test [17]. Experiments were conducted in an electromagnetic-shielded room using the following protocols: the subject touches the center of a 17-inch touch-sensitive screen (LCD-AD172F2-T, I/O Data Co., Tokyo, Japan) located 30 cm in front of them with their forefinger. A small cross-shaped target then appears 300 pixels away from the point previously touched in a vertical direction (screen pixel pitch 0.264 × 0.264 mm). The subject moves their forefinger to the center of the displaying target and touches it. The trial is repeated a total of 50 times in two sets. Specific instructions regarding motion speed and accuracy during trial performance were not given to all subjects. The subject’s head was placed on a jaw rest and the right forearm on an armrest to reduce artifacts other than motion of the forefinger.
On the Possibility about Performance Estimation
187
2.4 Analysis MRCP and high-speed camera data were clipped in each trial by a trigger signal generated using the surface EMG on the common digital extensor muscle. EEG data were clipped from 1500 msec before the start of the movement to 500 msec afterward, according to previous studies. The gap from the center of the target to the position actually touched was used as an evaluation index for the accuracy of the voluntary movement. Based on the gap average of task performance, we divided data of performance in each trial into two
EMG System EMG Amp.
PC1 (Full-wave Rectification) Trigger Signal 1
EEG System Shielded room
EEG Electrodes
128ch EEG Amp.
EMG Electrode
PC2 (EEG Analysis)
Motion Analysis System High Speed Camera
PC3 (Motion Analysis)
Lenz
Touch Panel
Trigger Signal 2
Task System PC4 (Task Control)
Fig. 2. Block diagram and data flow. Experiments were conducted on eight healthy male subjects using EEG, high-speed camera, EMG and task systems.
EEG DATA
Ti …..
T1T2T3T4
T50 Trigger
Task performance Average
A
High Accuracy Low Accuracy
EEG DATA
B
A BB
AA B
A B
….. …..
T5 T7T8 T10 ….. Signal T2 T9 ….. Averaging T1 T3T4 T6
MRCPgroupA MRCPgroupB
Comparison
Fig. 3. Analytical process. MRCP and high-speed camera data were clipped in each trial by a trigger signal generated using the surface EMG. Based on the gap average of task performance, the clipped EEG and high-speed camera data from each trial were separated into two groups corresponding to the ‘A’ and ‘B’ groups. The typical waveforms in each group were calculated by averaging and compared with each other.
188
S. Suzuki et al.
groups: a high performance group ‘A’ and a low performance group ‘B’. The clipped EEG and high-speed camera data from each trial were separated into two groups corresponding to the ‘A’ and ‘B’ groups. The typical waveforms in each group were calculated by averaging and compared with each other (Figure 3).
3 Results 3.1 Task Performance Figure 4 shows the distribution of gaps in each trial according to a subject (S1). The distribution resembled a gamma distribution with an average gap of 5 pixels. The trend of the distribution form was seen in all subjects. EEG signals corresponding to trials when the gap was less than 5 pixels were placed in group A and those over 5 pixels were placed in group B, in this subject S1’s case. Typical waveforms corresponding to the two groups were calculated by averaging. High-speed camera data were also divided into two groups. 3.2 MRCP Figures 5 a) and Figure 5 b) show sample of typical MRCP waveforms corresponding to the two groups acquired from Fz and Cz based on the same subject S1 shown in Figure 4. Differences between BPs (observed from 2000 msec before movement) and ISs (observed from 900 to 500 msec before movement) of groups A and B were not obvious. However, differences in NSs (observed from 500 msec before movement) were clearly observed between the two groups, with group A showing a steeper slope than group B. This trend was confirmed from around Fz, Cz, and Pz of all subjects (Figure 6) and the difference in average values between the two groups was found to be significant (Fz: p < 0.05 (p = 0.019), Cz: p < 0.05 (p = 0.050), Pz: p < 0.05 (p = 0.017)). This suggests that there is a relationship between the performance of the arm and the NS slope. Average
20
A group
B group
frequency
15
10
5
0 0
5
10
15
20
Accuracy (pixel)
Fig. 4. Sample data of result of performance (Subject 1). The distribution resembled a gamma distribution with an average gap of 5 pixels, in this subject S1’s case.
On the Possibility about Performance Estimation
A group
B group
A group
Trigger
B group
189
Trigger
-
-1500
-1000
-500
0
500
msec
-1500
5μV
-1000
-500
0
500 msec
5μV
+
+
a) Fz
b) Cz
Fig. 5. Sample data of MRCP (Subject 1). Typical MRCP waveforms corresponding to the two groups acquired from Fz (in Fig. 5 a)) and Cz (in Fig. 5 b))based on the same subject S1 shown in Figure 4. *
15
*
*
Slope (μV/sec)
A group B group
10
(*:p<0.05)
5
0 Fz
Cz
Pz
Fig. 6. Comparison of NS slopes in 2 groups (All subjects). Differences in NS (observed from 500 msec before movement) were clearly observed between the two groups, with group A showing a steeper slope than group B.
3.3 Process of the Reaching Motion Figure 7 shows the movement distance of the forefinger peak. A small difference in ballistic movement between groups A and B was confirmed, but we could not confirm a difference in corrective movement. This is because the experimental task is simple and primitive and the quantity allocated for corrective movement was few so it is not necessary to allocate for corrective movement. Movement distance (mm) (mm)
500
400 400
300
A group B group
200 200 100
00
1 Ballistic Movement
2
Corrective Movement
Fig. 7. Comparison of movement processes in 2 groups (All subjects). A small difference in ballistic movement between groups A and B was confirmed.
190
S. Suzuki et al.
4 Discussion and Conclusion In the present study, we attempted to confirm the tripartite relationship between MRCP, ballistic movement and accuracy of task performance. We could not confirm any difference between high-performance group ‘A’ and low- performance group ‘B’ on BP and IS in MRCP acquired from Cz and Fz. However, significant differences between the groups were clear on the NS in MRCP acquired from Cz and Fz. Furthermore, the difference was confirmed on the duration of ballistic motion. As NS is generally believed to represent a preparation stage for voluntary motion, it appears that the observed difference between groups influences the process of motion and the task performance. These results show the possibility of estimating execution of the performance just before beginning the voluntary motion using MRCP. On the other hand, the motion “reaching” is nearly optimized in terms of smoothness over the entire movement in the field considering with mathematical models. Various optimal criteria are proposed for trajectory planning of this motion in terms of multi-joint arm movements, like human arms. The minimum-jerk criterion [10] plans smooth trajectories in the extrinsic task space, while the minimum-joint-anglejerk criteria, the minimum-torque-change criteria [11], and the minimum-motorcommand-change criteria plan smooth trajectories [12] in the intrinsic body space. These models and criteria are discussed under the assumption that humans plan the trajectory of arm movement before beginning the motion. However, Hogan & Wolpert [18] pointed out that it is difficult to explain the biological relevance of factors such as jerk or torque change in previous models of arm trajectory. They showed that the movement can be achieved by reducing the variance of errors at the end of the movement. The final goal of the “reaching” movement is to minimize the gap at the end of the motion, as suggested by their minimum-variance theory. Although it is not clear how the cerebrum and cerebellum contribute to achieving this movement, this model is an effective way of considering this movement. If we place our own results in context with this model, the concept becomes validated as it is known that humans plan and learn the trajectory of reaching with optimal efficiency just before beginning the motion. Finally in the current study, we attempted to develop a prototype system to derive the NS slope in MRCP from EEG data automatically and in real time. When we observe the P300 and N200, the importance of the shape of the waveform necessitates signal averaging. If the event-related potentials (ERP), such as N200 and P300, was observed, the technique signal averaging is generally used not only to remove the artifact but also to extract more information, i.e. latency, amplitude, the shape of the waveform and so on. However, in the case of MRCP, we only need to observe slopes during the 500 ms just before the motion is executed, namely, just before the trigger. We are therefore attempting to develop a prototype system that obtains a sequentialmemory of the value during 2000 ms of EEG and sequentially calculates the value of the NSs slope in 500 ms. This prototype system is currently being developed using LabVIEW (National Instruments Co., TX, USA) and we already confirmed that the performance can be estimate with 75 % maximum of the time using an index based on NS values. Thus, in the future, our concept and method appears to be promising for use controlling devises safety, such as at car driving.
On the Possibility about Performance Estimation
191
References 1. Kornhuber, H.H., Deecke, L.: Hirnpotentialanderungen bei Willkurbewegungen und passiven Bewegungen des Menschen. Bereitschafts potential und reaffernte potentiale. Pfligers Arch, Gesamte Physiol Menschen Tiere 284, 1–17 (1965) 2. Barrett, G., Shibasaki, H., Neshige, R.A.: Computer-assisted method for averaging movement-related cortical potentials with respect to EMG onset. Electroencephalogr. Clin. Neurophysiol. 60(3), 276–281 (1985) 3. Shibasaki, H., Hallett, M.: What is the Bereitschaftspotential? Clin. Neurophysiol 117(11), 2341–2356 (2006) 4. MacKay, D.M., MacKay, V.: Behind the eye. Basil Blackwell, Malden (1991) 5. Yamamoto, J., Ikeda, A., Satow, T., et al.: Human eye fields in the frontal lobe as studied by epicortical recording of movement-related cortical potentials. Brain 127, 873–887 (2004) 6. Barrett, G., et al.: Cortical potential shifts preceding voluntary movements are nomal in parkinsonism. Electroencephalogr. Clin. Neurophysiol. 63, 340–348 (1986) 7. Card, S.K., Moran, T.P., Newell, A.: The psychology of human-computer interaction. Lawrence Erlbaum Associates, New Jersey (1983) 8. Gopher, D., Sanders, A.F.: S-Oh-R: Oh Stages! Oh Resources! In: Prinz, W., Sanders, A.F. (eds.) Cognition and motor processes, Springer, Heidelberg (1984) 9. Revelle, W.: Indivisual differences in personality and motivation: ’non-cognitive’ determinants of cognitive performance. In: Baddeley, A., Weiskrantz, L. (eds.) Awareness Control, Clarendon Press, Oxford (1991) 10. Flash, T., Hogan, N.: The Coordination of Arm Movements: An Experimentally Confirmed Mathematical Model. Journal of Neuroscience 5, 1688–1703 (1985) 11. Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement - Minimum torque-change model. Biological Cybernetics 61(2), 89–101 (1989) 12. Kawato, M.: Optimization and learning in neural networks for formation and control of coordinated movement. In: Meyer, D., Kornblum, S. (eds.) Attention and performance XIV, pp. 821–849. MIT Press, Cambridge (1993) 13. Fitts, P.M., Peterson, J.R.: Information capacity of discrete motor responses. Journal of Experimental Psychology 67, 103–112 (1964) 14. Accot, J., Zhai, S.: Beyond Fitts’ Law: Models for trajectory-based HCI tasks. In: CHI 1997, pp. 295–302 (1997) 15. Flowers, K.: Ballistic and corrective movement on an aiming task. Neurology 25, 413–421 (1975) 16. Pratt, J., Abrams, R.A.: Practice and component submovements: The roles of programming and feedback in rapid aimed limb movements. Journal of Motor Behavior 28(2), 149–156 (1996) 17. Oldfield, R.C.: The assessent and analysis of handedness: The Edinburgh Inventory. Neuropsychologia 9, 97–113 (1971) 18. Harris, C.M., Wolpert, D.M.: Signal-dependent noise determinations motor planning. Nature 394, 780–784 (1998)
A Usability Evaluation Method Applying AHP and Treemap Techniques Toshiyuki Asahi, Teruya Ikegami, and Shin’ichi Fukuzumi Common Platform Software Research Laboratories, NEC Corporation, 8916-47, Takayama-Cho, Ikoma, Nara 630-0101, Japan
[email protected]
Abstract. This report proposes a visualization technique for checklist-based usability quantification methods. By applying the Treemap method, the hierarchical structure of checklists, weights of check items and evaluation results for target systems can be viewed at a glance. Effective support for usability analysis and its presentation tasks of usability evaluation results are expected. A prototype tool was implemented on a PC and experimental studies assuming actual usability evaluation tasks were conducted. The results indicate that the proposed method improves performance time of some typical tasks. Usability engineers gave higher subjective scores on the usefulness of the proposed method than that of printed table presentation .
Keywords: Usability quantification, checklist, Treemap, visualization, analytic hierarchy process, design tools.
1 Introduction Usability quantification has been considered one of the key technologies for systematizing and spreading user-centered design and has been studied for some time [1]. Several methods such as 1) performance analysis/prediction, 2) subjective questionnaire, 3) checklist-based h