Human-Computer Interaction. New Trends: 13th International Conference, HCI International 2009, San Diego, CA, USA, July 19-24, 2009, Proceedings, Part ... Applications, incl. Internet Web, and HCI)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5610

Julie A. Jacko (Ed.)

Human-Computer Interaction New Trends 13th International Conference, HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings, Part I

13

Volume Editor Julie A. Jacko University of Minnesota Institute of Health Informatics MMC 912, 420 Delaware Street S.E., Minneapolis, MN 55455, USA E-mail: [email protected]

Library of Congress Control Number: 2009929048 CR Subject Classification (1998): H.5, I.3, I.7.5, I.5, I.2.10 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-02573-0 Springer Berlin Heidelberg New York 978-3-642-02573-0 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12705719 06/3180 543210

Foreword

The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human–Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in the knowledge and effective use of computers in a variety of application areas. This volume, edited by Julie A. Jacko, contains papers in the thematic area of Human–Computer Interaction, addressing the following major topics: • • • • •

Novel Techniques for Measuring and Monitoring Evaluation Methods, Techniques and Tools User Studies User Interface Design Development Approaches, Methods and Tools

The remaining volumes of the HCI International 2009 proceedings are: • • • • •

Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 3, LNCS 5612, Human–Computer Interaction––Ambient, Ubiquitous and Intelligent Interaction (Part III), edited by Julie A. Jacko Volume 4, LNCS 5613, Human–Computer Interaction - Interacting in Various Application Domains (Part IV), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis

VI

Foreword

• • • • • • • • • • •

Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 11, LNCS 5620, Digital Human Modeling, edited by Vincent G. Duffy Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen Volume 17, LNAI 5639, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris

I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.

Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland

Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA

Foreword

Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea

Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany

Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA

Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China

Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA

Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK

VII

VIII

Foreword

Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa

Matthew J.W. Thomas, Australia Mark Young, UK

Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA

Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK

Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA

Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA

Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK

Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA

Foreword

Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA

Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China

Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria

Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan

Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA

Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA

IX

X

Foreword

Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK

Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany

Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China

Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China

Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan

Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan

In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.

Foreword

XI

I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis

HCI International 2011

The 14th International Conference on Human–Computer Interaction, HCI International 2011, will be held jointly with the affiliated conferences in the summer of 2011. It will cover a broad spectrum of themes related to human–computer interaction, including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/

General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email: [email protected]

Table of Contents

Part I: Novel Techniques for Measuring and Monitoring Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kiyohiko Abe, Shoichi Ohi, and Minoru Ohyama

3

A Usability Study of WebMaps with Eye Tracking Tool: The Effects of Iconic Representation of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ¨ Ozge Ala¸cam and Mustafa Dalcı

12

Feature Extraction and Selection for Inferring User Engagement in an HCI Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stylianos Asteriadis, Kostas Karpouzis, and Stefanos Kollias

22

Informative or Misleading? Heatmaps Deconstructed . . . . . . . . . . . . . . . . . Agnieszka (Aga) Bojko

30

Toward EEG Sensing of Imagined Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan

40

Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Gao, Armando Barreto, and Malek Adjouadi

49

Usability Evaluation by Monitoring Physiological and Other Data Simultaneously with a Time-Resolution of Only a Few Seconds . . . . . . . . K´ aroly Hercegfi, M´ arton P´ aszti, Sarolta T´ ov¨ olgyi, and Lajos Izs´ o

59

Study of Human Anxiety on the Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . Santosh Kumar Kalwar and Kari Heikkinen

69

The Research on Adaptive Process for Emotion Recognition by Using Time-Dependent Parameters of Autonomic Nervous Response . . . . . . . . . Jonghwa Kim, Mincheol Whang, and Jincheol Woo

77

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu-Jin Kim, Jin Ah Bae, and Byeong Ho Jeon

85

Toward Constructing an Electroencephalogram Measurement Method for Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masaki Kimura, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto

95

XVI

Table of Contents

Automated Analysis of Eye-Tracking Data for the Evaluation of Driver Information Systems According to ISO/TS 15007-2:2001 . . . . . . . . . . . . . . Christian Lange, Martin Wohlfarter, and Heiner Bubb Brain Response to Good and Bad Design . . . . . . . . . . . . . . . . . . . . . . . . . . . Haeinn Lee, Jungtae Lee, and Ssanghee Seo An Analysis of Eye Movements during Browsing Multiple Search Results Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuko Matsuda, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Development of Estimation System for Concentrate Situation Using Acceleration Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masashi Okubo and Aya Fujimura Psychophysiology as a Tool for HCI Research: Promises and Pitfalls . . . . Byungho Park Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Genaro Rebolledo-Mendez, Ian Dunwell, Erika A. Mart´ınez-Mir´ on, Mar´ıa Dolores Vargas-Cerd´ an, Sara de Freitas, Fotis Liarokapis, and Alma R. Garc´ıa-Gaona Effect of Body Movement on Music Expressivity in Jazz Performances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mamiko Sakata, Sayaka Wakamiya, Naoki Odaka, and Kozaburo Hachimura

105 111

121

131 141

149

159

A Method to Monitor Operator Overloading . . . . . . . . . . . . . . . . . . . . . . . . . Dvijesh Shastri, Ioannis Pavlidis, and Avinash Wesley

169

Decoding Attentional Orientation from EEG Spectra . . . . . . . . . . . . . . . . . Ramesh Srinivasan, Samuel Thorpe, Siyi Deng, Tom Lappas, and Michael D’Zmura

176

On the Possibility about Performance Estimation Just before Beginning a Voluntary Motion Using Movement Related Cortical Potential . . . . . . . Satoshi Suzuki, Takemi Matsui, Yusuke Sakaguchi, Kazuhiro Ando, Nobuyuki Nishiuchi, Toshimasa Yamazaki, and Shin’ichi Fukuzumi

184

Part II: Evaluation Methods, Techniques and Tools A Usability Evaluation Method Applying AHP and Treemap Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toshiyuki Asahi, Teruya Ikegami, and Shin’ichi Fukuzumi

195

Table of Contents

Evaluation of User-Interfaces for Mobile Application Development Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florence Balagtas-Fernandez and Heinrich Hussmann User-Centered Design and Evaluation – The Big Picture . . . . . . . . . . . . . . Victoria Bellotti, Shin’ichi Fukuzumi, Toshiyuki Asahi, and Shunsuke Suzuki

XVII

204 214

Web-Based System Development for Usability Evaluation of Ubiquitous Computing Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jong Kyu Choi, Han Joon Kim, Beom Suk Jin, and Yonggu Ji

224

Evaluating Mobile Usability: The Role of Fidelity in Full-Scale Laboratory Simulations with Mobile ICT for Hospitals . . . . . . . . . . . . . . . . Yngve Dahl, Ole Andreas Alsos, and Dag Svanæs

232

A Multidimensional Approach for the Evaluation of Mobile Application User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jos´e Eust´ aquio Rangel de Queiroz and Danilo de Sousa Ferreira

242

Development of Quantitative Usability Evaluation Method . . . . . . . . . . . . Shin’ichi Fukuzumi, Teruya Ikegami, and Hidehiko Okada

252

Reference Model for Quality Assurance of Speech Applications . . . . . . . . . Cornelia Hipp and Matthias Peissner

259

Toward Cognitive Modeling for Predicting Usability . . . . . . . . . . . . . . . . . . Bonnie E. John and Shunsuke Suzuki

267

Webjig: An Automated User Data Collection System for Website Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mikio Kiura, Masao Ohira, and Ken-ichi Matsumoto ADiEU: Toward Domain-Based Evaluation of Spoken Dialog Systems . . . Jan Kleindienst, Jan Cuˇr´ın, and Martin Labsk´ y Interpretation of User Evaluation for Emotional Speech Synthesis System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ho-Joon Lee and Jong C. Park Multi-level Validation of the ISOmetrics Questionnaire Based on Qualitative and Quantitative Data Obtained from a Conventional Usability Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan-Paul Leuteritz, Harald Widlroither, and Michael Kl¨ uh What Do Users Really Do? Experience Sampling in the 21st Century . . . Gavin S. Lew Evaluating Usability-Supporting Architecture Patterns: Reactions from Usability Professionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edgardo Luzcando, Davide Bolchini, and Anthony Faiola

277 287

295

304 314

320

XVIII

Table of Contents

Heuristic Evaluations of Bioinformatics Tools: A Development Case . . . . Barbara Mirel and Zach Wright A Prototype to Validate ErgoCoIn: A Web Site Ergonomic Inspection Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcelo Morandini, Walter de Abreu Cybis, and Dominique L. Scapin

329

339

Mobile Phone Usability Questionnaire (MPUQ) and Automated Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Young Sam Ryu

349

Estimating Productivity: Composite Operators for Keystroke Level Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jeff Sauro

352

Paper to Electronic Questionnaires: Effects on Structured Questionnaire Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Trujillo

362

Website Designer as an Evaluator: A Formative Evaluation Method for Website Interface Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chao-Yang Yang

372

Part III: User Studies Building on the Usability Study: Two Explorations on How to Better Understand an Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anshu Agarwal and Madhu Prabaker Measuring User Performance for Different Interfaces Using a Word Processor Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tanya R. Beelders, Pieter J. Blignaut, Theo McDonald, and Engela H. Dednam

385

395

Evaluating User Effectiveness in Exploratory Search with TouchGraph Google Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kemal Efe and Sabriye Ozerturk

405

What Do Users Want to See? A Content Preparation Study for Consumer Electronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yinni Guo, Robert W. Proctor, and Gavriel Salvendy

413

“I Love My iPhone... But There Are Certain Things That ‘Niggle’ Me” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Haywood and Gemma Boguslawski

421

Table of Contents

Acceptance of Future Technologies Using Personal Data: A Focus Group with Young Internet Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fabian Hermann, Doris Janssen, Daniel Schipke, and Andreas Schuller Analysis of Breakdowns in Menu-Based Interaction Based on Information Scent Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yukio Horiguchi, Hiroaki Nakanishi, Tetsuo Sawaragi, and Yuji Kuroda E-Shopping Behavior and User-Web Interaction for Developing a Useful Green Website . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei-Hui Huang, Ying-Lien Lee, and Sheue-Ling Hwang Interaction Comparison among Media Internet Genre . . . . . . . . . . . . . . . . . Sang Hee Kweon, Eun Joung Cho, and Ae Jin Cho Comparing the Usability of the Icons and Functions between IE6.0 and IE7.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chiuhsiang Joe Lin, Min-Chih Hsieh, Hui-Chi Yu, Ping-Jung Tsai, and Wei-Jung Shiang

XIX

431

438

446

455

465

Goods-Finding and Orientation in the Elderly on 3D Virtual Store Interface: The Impact of Classification and Landmarks . . . . . . . . . . . . . . . . Cheng-Li Liu, Shiaw-Tsyr Uang, and Chen-Hao Chang

474

Effects of Gender Difference on Emergency Operation Interface Design in Semiconductor Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hunszu Liu

484

Evaluating a Personal Communication Tool: Sidebar . . . . . . . . . . . . . . . . . Malena Mesarina, Jhilmil Jain, Craig Sayers, Tyler Close, and John Recker

490

“You’ve Got IMs!” How People Manage Concurrent Instant Messages . . . Shailendra Rao, Judy Chen, Robin Jeffries, and Richard Boardman

500

Investigating Children Preferences of a User Interface Design . . . . . . . . . . Jamaliah Taslim, Wan Adilah Wan Adnan, and Noor Azyanti Abu Bakar

510

Usability Evaluation of Graphic Design for Ilmu’s Interface . . . . . . . . . . . . Tengku Siti Meriam Tengku Wook and Siti Salwa Salim

514

Are We Trapped by Majority Influences in Electronic Word-of-Mouth? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu Tong and Yinqing Zhong

520

XX

Table of Contents

Leveraging a User Research Framework to Guide Research Investments: Windows Vista Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gayna Williams A Usability Evaluation of Public Icon Interface . . . . . . . . . . . . . . . . . . . . . . Sungyoung Yoon, Jonghoon Seo, Joonyoung Yoon, Seungchul Shin, and Tack-Don Han

530 540

Part IV: User Interface Design Little Design Up-Front: A Design Science Approach to Integrating Usability into Agile Requirements Engineering . . . . . . . . . . . . . . . . . . . . . . . Sisira Adikari, Craig McDonald, and John Campbell

549

Aesthetics in Human-Computer Interaction: Views and Reviews . . . . . . . Salah Uddin Ahmed, Abdullah Al Mahmud, and Kristin Bergaust

559

Providing an Efficient Way to Make Desktop Icons Visible . . . . . . . . . . . . Toshiya Akasaka and Yusaku Okada

569

An Integration of Task and Use-Case Meta-models . . . . . . . . . . . . . . . . . . . R´emi Bastide

579

Model-Based Specification and Validation of User Interface Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Birgit Bomsdorf and Daniel Sinnig

587

A Position Paper on ’Living Laboratories’: Rethinking Ecological Designs and Experimentation in Human-Computer Interaction . . . . . . . . . Ed H. Chi

597

Embodied Interaction or Context-Aware Computing? An Integrated Approach to Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Johan Eliasson, Teresa Cerratto Pargman, and Robert Ramberg

606

Supporting Multidisciplinary Teams and Early Design Stages Using Storyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mieke Haesen, Jan Meskens, Kris Luyten, and Karin Coninx

616

Agent-Based Architecture for Interactive System Design: Current Approaches, Perspectives and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . Christophe Kolski, Peter Forbrig, Bertrand David, Patrick Girard, Chi Dung Tran, and Houcine Ezzedine BunBunMovie: Scenario Visualizing System Based on 3-D Character . . . Tomoya Matsuo and Takashi Yoshino Augmented Collaborative Card-Based Creative Activity with Digital Pens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motoki Miura, Taro Sugihara, and Susumu Kunifuji

624

634

644

Table of Contents

Usability-Engineering-Requirements as a Basis for the Integration with Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karsten Nebe and Volker Paelke Design Creation Based on KANSEI in Toshiba . . . . . . . . . . . . . . . . . . . . . . Yosoko Nishizawa and Kanya Hiroi High-Fidelity Prototyping of Interactive Systems Can Be Formal Too . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Philippe Palanque, Jean-Fran¸cois Ladry, David Navarre, and Eric Barboni

XXI

652 660

667

RUCID: Rapid Usable Consistent Interaction Design Patterns-Based Mobile Phone UI Design Library, Process and Tool . . . . . . . . . . . . . . . . . . . Avinash Raj and Vihari Komaragiri

677

The Appropriation of Information and Communication Technology: A Cross-Cultural Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jose Rojas and Matthew Chalmers

687

UISK: Supporting Model-Driven and Sketch-Driven Paperless Prototyping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vin´ıcius Costa Villas Bˆ oas Segura and Simone Diniz Junqueira Barbosa

697

Beyond the User Interface: Towards User-Centred Design of Online Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Sikorski

706

Designing for Change: Engineering Adaptable and Adaptive User Interaction by Focusing on User Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruno S. da Silva, Ariane M. Bueno, and Simone D.J. Barbosa

715

Productive Love: A New Proposal for Designing Affective Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ramon Solves Pujol and Hiroyuki Umemuro

725

Insight into Kansei Color Combinations in Interactive User Interface Designing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . K.G.D. Tharangie, Shuichi Matsuzaki, Ashu Marasinghe, and Koichi Yamada Learn as Babies Learn: A Conceptual Model of Designing Optimum Learnability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Douglas Xiaoyong Wang Time-Oriented Interface Design: Picking the Right Time and Method for Information Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Keita Watanabe, Kei Sugawara, Shota Matsuda, and Michiaki Yasumura

735

745

752

XXII

Table of Contents

Enabling Interactive Access to Web Tables . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Yang, Wenchang Xu, and Yuanchun Shi

760

Integration of Creativity into Website Design . . . . . . . . . . . . . . . . . . . . . . . . Liang Zeng, Robert W. Proctor, and Gavriel Salvendy

769

Part V: Development Approaches, Methods and Tools YVision: A General Purpose Software Composition Framework . . . . . . . . Ant˜ ao Almada, Gon¸calo Lopes, Andr´e Almeida, Jo˜ ao Fraz˜ ao, and Nuno Cardoso Collaborative Development and New Devices for Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hans-J¨ org Bullinger and Gunnar Brink

779

789

Orchestration Modeling of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . Bertrand David and Ren´e Chalon

796

An Exploration of Perspective Changes within MBD . . . . . . . . . . . . . . . . . Anke Dittmar and Peter Forbrig

806

Rapid Development of Scoped User Interfaces . . . . . . . . . . . . . . . . . . . . . . . Denis Dub´e, Jacob Beard, and Hans Vangheluwe

816

PaMGIS: A Framework for Pattern-Based Modeling and Generation of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J¨ urgen Engel and Christian M¨ artin

826

People-Oriented Programming: From Agent-Oriented Analysis to the Design of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steve Goschnick

836

Visualization of Software and Systems as Support Mechanism for Integrated Software Project Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Liggesmeyer, Jens Heidrich, J¨ urgen M¨ unch, Robert Kalckl¨ osch, Henning Barthel, and Dirk Zeckzer Collage: A Declarative Programming Model for Compositional Development of Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruce Lucas, Rahul Akolkar, and Charlie Wiecha Hypernetwork Model to Represent Similarity Details Applied to Musical Instrument Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuya Maeshiro, Midori Maeshiro, Katsunori Shimohara, and Shin-ichi Nakayama

846

856

866

Table of Contents

Open Collaborative Development: Trends, Tools, and Tactics . . . . . . . . . . Kathrin M. Moeslein, Angelika C. Bullinger, and Jens Soeldner

XXIII

874

Investigating the Run Time Behavior of Distributed Applications by Using Tiny Java Virtual Machines with Wireless Communications . . . . . . Tsuyoshi Miyazaki, Takayuki Suzuki, and Fujio Yamamoto

882

OntoDesk: Ontology-Based Persistent System-Wide Undo on the Desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . David Nemeskey, Buntarou Shizuki, and Jiro Tanaka

890

Peer-to-Peer File Sharing Communication Detection System with Traffic Mining and Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Satoshi Togawa, Kazuhide Kanenishi, and Yoneo Yano

900

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

911

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images Kiyohiko Abe1, Shoichi Ohi2, and Minoru Ohyama3 1

College of Engineering, Kanto Gakuin University, 1-50-1 Mutsuura-higashi, Kanazawa-ku, Yokohama, Kanagawa 236-8501, Japan 2 School of Engineering, Tokyo Denki University, 2-2 Kandanishiki-cho, Chiyoda-ku, Tokyo 101-8457, Japan 3 School of Information Environment, Tokyo Denki University, 2-1200 Muzaigakuendai, Inzai-shi, Chiba 270-1382, Japan [email protected], [email protected], [email protected]

Abstract. We propose a new eye blink detection method that uses NTSC video cameras. This method utilizes split-interlaced images of the eye. These split images are odd- and even-field images in the NTSC format and are generated from NTSC frames (interlaced images). The proposed method yields a time resolution that is double that in the NTSC format; that is, the detailed temporal change that occurs during the process of eye blinking can be measured. To verify the accuracy of the proposed method, experiments are performed using a high-speed digital video camera. Furthermore, results obtained using the NTSC camera were compared with those obtained using the high-speed digital video camera. We also report experimental results for comparing measurements made by the NTSC camera and the high-speed digital video camera. Keywords: Eye Blink, Interlaced Image, Natural Light, Image Analysis, HighSpeed Camera.

1 Introduction The blinking of the eye is related to factors such as human cognition, fatigue, and depressed consciousness; many studies have investigated eye blinking in relation to these factors. Most conventional methods for the measurement of the eye blink analyze eye images (images of the eye and its surrounding skin) captured by a video camera [1], [2], [3]. The NTSC video cameras that are commonly used are capable of detecting eye blinks; however, it is difficult for these cameras to measure the detailed temporal change occurring during the process of eye blinking, because eye blinks occur relatively fast (within a few hundred milliseconds). Therefore, a high-speed camera is required for an accurate measurement of the eye blink [3]. NTSC video cameras capture moving images at 60 fields/s and these field images are mixed with images that have a frame rate of 30 frames/s (fps) to field interlaced images. In this paper, we propose a new method for measuring the eye blink that uses J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 3–11, 2009. © Springer-Verlag Berlin Heidelberg 2009

4

K. Abe, S. Ohi, and M. Ohyama

NTSC video cameras. This method utilizes split-interlaced images of the eye captured by an NTSC video camera. These split images are odd- and even-field images in the NTSC format and are generated from NTSC frames (interlaced images). The proposed method yields a time resolution that is twice that in the NTSC format. Therefore, the detailed temporal change that occurs during the process of eye blinking can be measured. To verify the accuracy of the proposed method, we performed experiments using a high-speed digital video camera. Thereafter, we compared results obtained using the NTSC cameras with those obtained using the high-speed digital video camera. This paper also presents experiments that evaluate the proposed automatic method for measuring eye blinks.

2 Open-Eye Area Extraction Method by Image Analysis In general, eye blinks are estimated by measuring the open-eye area [2] or on basis of characteristics of specific moving points between the upper and lower eyelids [3]. Many of these methods utilize image analysis. It is possible to measure the wave pattern of eye blinks if the entire process of an eye blink is captured [3]. Furthermore, the type of eye blink and/or its velocity can be estimated on the basis of this wave pattern. However, it is difficult to measure the wave patterns of eye blinks by using video cameras that are commonly used for measuring eye blinks because the resulting eye images include high noise content owing to the change in light conditions. We have developed a new method for measuring the wave pattern of an eye blink. This method can be used with common indoor lighting sources such as fluorescent lights, and it can measure the wave pattern automatically. Hence, our proposed measurement method can be used under a variety of experimental conditions. In this method, the wave pattern is obtained by counting the number of pixels in the openeye area of the image as captured by a video camera. This image is enlarged for capturing the detailed eye image. We have proposed an algorithm for extracting the open-eye area in a previous study [4]. It utilizes color information of eye images. We have adapted the algorithm to our proposed method for elucidating the wave pattern of eye blink measurement. This algorithm has been developed for our eye-gaze input system, in which it compensates and traces head-movement [5]. Furthermore, the algorithm has been used under common indoor sources of light for a prolonged period. Hereafter, we describe in detail our image-processing algorithm for extracting the open-eye area. 2.1 Binarization Using Color Information on Image Many methods have been developed for the purpose of skin-color extraction; these methods are primarily focused on facial image processing, including those that utilize color information on a facial image. They mostly determine threshold skin-color values statistically or empirically [6]. We have developed an automatic algorithm for estimating thresholds of skin-color. Our algorithm can extract the open-eye area from the eye image on the basis of the skin-color.

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images

5

Using our algorithm, skin-color threshold is determined by the histogram of the color-difference signal ratio of each pixel—Cr/Cb—that is calculated from the YCbCr image transformed from the RGB image. The histogram of the Cr/Cb value has 2 peaks indicating skin area and open-eye area. The Cr/Cb value indicated by the minimum value between the 2 peaks is designated as the threshold for open-eye area extraction. 2.2

Binarization by Pattern Matching Method

The method described in Subsection 2.1 can extract the open-eye area almost completely. However, the results of this extraction sometimes leave deficits around the corner of eye, because the Cr/Cb value around the corner of eye is similar to the value on skin in certain subjects. To resolve this problem, we have developed a method for open-eye extraction without deficits by combining 2 extraction results. One of them is a binarized image using color information, as described in Section 2.1. The other extraction result is a binarized image using light intensity information, which includes in the extraction result the area around the corner of the eye. Binarization using light intensity information utilizes the threshold estimated by a pattern matching method, which determines the matching point by using the color information of the binarized image as reference data. Hence, the threshold level is estimated automatically. The original image and the extracted open-eye area image are shown in Fig. 1(a) and Fig. 1(b).

(a)

(b)

Fig. 1. Original eye image (a) and extracted open-eye area (b)

3 Measurement Method of Wave Patterns of Eye Blinks Using Split-Interlaced Images Commonly used NTSC video cameras output interlaced images. One interlaced image has 2 field images, which are designated as odd or even fields. If an NTSC camera captures a fast movement such as an eye blink, there is a great divergence between the captured odd- and even-field images. Therefore, the area around eyelids on the captured image has comb-like noise. This phenomenon occurs because of mixing of 2 field images of the fast movement of eyelids. An example of interlaced images during eye blinking is shown in Fig. 2. To describe this phenomenon most clearly, Fig. 2 has been captured at low resolution (145 × 80 pixels).

6

K. Abe, S. Ohi, and M. Ohyama

If one interlaced image is split by scanning even- and odd-numbered lines separately, 2 field images are generated. Thus, the time resolution of the motion images doubles, but the amount of information in the vertical direction decreases by half. These field images are captured at 60 fields/sec, and the NTSC interlaced moving images are captured at 30 fps; therefore, this method yields a time resolution that is double that available in the NTSC format. The duration of a conscious blink is a few hundred milliseconds; therefore, it is difficult to measure accurately the wave pattern of an eye blink by using NTSC cameras. However, the detailed wave pattern of an eye blink can be measured by using our proposed method. The split-interlaced images are shown in Fig. 3. The 2 eye images shown in Fig. 3 are enlarged in a vertical direction and were generated from the interlaced image shown in Fig. 2. Our proposed method measures the wave patterns of eye blinks from these images.

Fig. 2. Blinking eye image (interlaced)

Fig. 3. Split-interlaced image generated from Fig. 2

4 Evaluation Experiment for Proposed Method Either 4 or 5 subjects participated in experiments to evaluate our proposed method, as described in Subsections 4.1 and 4.2, respectively. The experimental setup includes an NTSC DV camera (for home use), a high-speed digital video camera, and a personal computer (PC). The PC analyzes sequenced eye images captured by the video cameras. The DV camera captures interlaced images at 30 fps, and the high-speed digital video camera captures non-interlaced images at 300 fps. In the experiments performed using these video cameras, the wave pattern of eye blinks is measured from sequenced eye images. The experimental setup is shown in Fig. 4.

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images PC

7

Display

User NTSC or High-speed digital video camera

Fig. 4. Hardware configuration of experimental system

4.1 Experiment for Eye Blink Measurement Using NTSC Camera In this experiment, sequenced eye images were captured using the DV camera at 30 fps in NTSC format. In addition, split-interlaced images are generated from these interlaced NTSC images. These split-interlaced images have a time resolution of 60 fields/s. The wave pattern of eye blinks is measured by the interlaced NTSC images and split-interlaced images. The binarization threshold for open-eye area extraction is determined automatically from the first field image of the experimental moving images. This threshold is estimated by the method described in Section 2. A typical result from this experiment is shown in Fig. 5.

Pixels of open-eye area

1.1 1 0.9 0.8

60fps

0.7

30fps

0.6 0.5 0.4 1

6

11 16 21 Sampling point (1/60 sec.)

26

Fig. 5. Wave patterns of eye blinks measured by DV (30 fps and 60 fps)

In Fig. 5, the longitudinal axis and the abscissa axis indicate pixels of open-eye area and sampling point (interval: 1/60 sec), respectively. To compare the 2 wave patterns of eye blinks, these plots are normalized using the pixels of open-eye area at the first field image. The bottoms of the plots indicate the eye-closed condition. Our proposed algorithm classifies the area of eyelid outline and cilia into the open-eye area; therefore, the pixels at the bottom of the plots are not reduced to zero. From Fig. 5, it is evident that sequenced images at 60 fields/s can be used to estimate the detailed wave pattern of an eye blink. During the eye blink, there is a great difference in the 2 plots of pixels of the open-eye area; however, this difference is not dependent on individual subjects.

8

K. Abe, S. Ohi, and M. Ohyama

Results of the wave pattern of eye blink measurements for 5 subjects are shown in Fig. 6, where the longitudinal axis and the abscissa axis show pixels of open-eye area and sampling point, respectively. These plots also are normalized in a manner similar to those in Fig. 5. From Fig. 6, it is evident that there are great differences in the results for each subject. 1.1

Pixels of open-eye area

1 0.9 0.8 0.7 0.6 0.5 0.4

Subject A Subject D

0.3

Subject B Subject E

Subject C

0.2 1

11 21 31 41 Sampling point (1/60 sec.)

51

Fig. 6. Wave patterns of eye blinks of 5 subjects measured by DV (60 fps)

4.2 Experiment for Eye Blink Measurement Using High-Speed Video Camera To verify the accuracy of the proposed method that utilizes split-interlaced images, experiments were conducted with 4 subjects; this experiment and the one described in Subsection 4.1 were conducted separately. Subjects A and E (listed in Fig. 6) were enrolled in this experiment continuously, in which sequenced images at 3 different frame rates (30, 60, and 150 fps) were generated from moving images captured by the high-speed digital video camera. These sequenced images were then analyzed to measure the wave pattern of eye blinks. The results of eye blink measurements performed using the sequenced images at 3 different frame rates and those taken at 300 fps are compared. Typical examples of measurement results are shown in Fig. 7, Fig. 8, and Fig. 9, which display results at 30, 60, and 150 fps, respectively. From Fig. 7 and Fig. 8, it is evident that the degree of accuracy of measurement at 60 fps is higher than that at 30 fps. The minimum of the wave pattern (bottom of the curve) is quite characteristic of when an eye blink occurs. Results at 60 fps show that the bottom of the plot is measured with a high degree of accuracy. Therefore, sequenced images at this frame rate are suitable for measurement of eyelid movement velocity. Moreover, our proposed method using split-interlaced images (described in Section 3) utilizes 2 field images generated from one interlaced image; that is, the

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images

9

spatial information of these field images is decreased by half. We have confirmed that this decrease in spatial information does not affect measurement accuracy via an experiment using sequenced images at 60 fps. The sequenced images at 60 fps were generated from moving images captured by a high-speed digital video camera. In this experiment, we generated half-sized eye images by extracting scanned odd-numbered lines from sequenced images at 60 fps. We estimated wave patterns of eye blinks using these half-sized images. Our results show that the measured open-eye area decreases by half, which is in agreement with the results shown in Fig.8.

Pixels of open-eye area

46000 300 fps

44000

30 fps

42000 40000 38000 36000 34000 32000 1

11

21 31 41 51 61 71 Sampling point (1/300 sec.)

81

Fig. 7. Wave pattern of eye blinks measured by high-speed video camera (30 fps)

Pixels of open-eye area

46000

300 fps

44000

60 fps

42000 40000 38000 36000 34000 32000 1

11

21 31 41 51 61 71 Sampling point (1/300 sec.)

81

Fig. 8. Wave pattern of eye blinks measured by high-speed video camera (60 fps)

10

K. Abe, S. Ohi, and M. Ohyama

Pixels of open-eye area

46000

300 fps

44000

150 fps

42000 40000 38000 36000 34000 32000 1

11

21 31 41 51 61 71 Sampling point (1/300 sec.)

81

Fig. 9. Wave pattern of eye blinks measured by high-speed video camera (150 fps)

4.3 Discussion On the basis of Fig.5, it is evident that by using split-interlaced images, the time resolution of measurement is doubled than that of the results obtained in previous studies. These split images are odd- and even-numbered field images in the NTSC format that are generated from NTSC frames. This method can also be utilized for any subject under common indoor lighting sources, such as fluorescent lights. We have shown the wave patterns of eye blinks for 5 subjects in Fig. 6. From results shown in Fig. 7, Fig. 8, and Fig. 9, it is evident that the degree of accuracy of measurement increases with increasing frame rate. A closer estimate of eye blinking velocity can be achieved if the wave pattern of an eye blink were to be measured with higher accuracy. In other words, the type of eye blink can be classified with a high degree of accuracy. In addition, our proposed method can measure the wave patterns of eye blinks efficiently even by using half-sized eye images. As shown by our experimental results presented earlier, we have verified the reliability of our proposed method described in Section 3. Thus, detailed wave patterns of eye blinks can be measured by using our proposed method.

5 Conclusions We present a new automatic method for measuring eye blinks. Our method utilizes split-interlaced images of the eye captured by an NTSC video camera. These split images are odd- and even-numbered field images in the NTSC format and are generated from NTSC moving images. By using this method, the time resolution for measurement increases to 60 fps, which is double that of conventional methods. Besides the function of automatic measurement of eye blinks, our method can be used under common indoor lighting sources, such as fluorescent lights. In evaluation experiments, we measured eye blinks of all subjects without problems.

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images

11

To verify the accuracy of our proposed method, we performed experiments using a high-speed digital video camera. On comparison of the results obtained using NTSC cameras with those obtained using a high-speed digital video camera, it is evident that the degree of accuracy of measurement increases with increased resolution time. Additionally, a decrease in area of the split-interlaced image has no adverse effect on the results of eye blink measurements. We confirmed that our proposed method is capable of measuring the wave pattern of eye blinks with high accuracy by using an NTSC video camera. In the future, we plan to develop a new method for classifying types of eye blinks using our proposed measurement method reported above. That new method will be capable of profiling eye blinks according to velocity of open-eye area changes. We also plan to apply this new method to more general ergonomic measurements.

References 1. Grauman, K., Betke, M., Gips, J., Bradski, G.R.: Communication via Eye Blinks - Detection and Duration Analysis in Real Time. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1010–1017, Lihue, HI (2001) 2. Morris, T., Blenkhorn, P., Zaidi, F.: Blink Detection for Real-Time Eye Tracking. J. Network and Computer Applications 25(2), 129–143 (2002) 3. Ohzeki, K., Ryo, B.: Video Analysis for Detecting Eye Blinking Using a High-Speed Camera. In: Proc. of Fortieth Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, pp. 1081–1085 (2006) 4. Abe, K., Ohyama, M., Ohi, S.: Eye-Gaze Input System with Multi-Indicators Based on Image Analysis under Natural Light. J. The Institute of Image Information and Television Engineers 58(11), 1656–1664 (2004) (in Japanese) 5. Abe, K., Ohi, S., Ohyama, M.: An Eye-Gaze Input System Using Information on Eye Movement History. In: Proc. on 12th International Conference on Human-Computer Interaction, HCI International 2007, Beijing, vol. 6, pp. 721–729 (2007) 6. Garcia, C., Tziritas, G.: Face Detection Using Quantized Skin Color Regions Merging and Wavelet Packet Analysis. IEEE Trans. on Multimedia 1(3), 264–277 (1999)

A Usability Study of WebMaps with Eye Tracking Tool: The Effects of Iconic Representation of Information Özge Alaçam and Mustafa Dalcı Human Computer Interaction Research and Application Laboratory, Computer Center, Middle East Technical University, 06531 Ankara/Turkey {ozge,mdalci}@metu.edu.tr

Abstract. In this study, we aim to conduct usability tests on different WebMap sites with eye movement analysis. Overall task performance, the effects of iconic representation of information, and the efficiency of pop-up usage were evaluated. The eye tracking technology is used for this study in order to follow the position of the users’ eye-gaze. The results show that there are remarkable differences in task performance between WebMaps. Addition, they also differ in the use of iconic representations according to results of users’ evaluation. It is also found that efficiency of pop-up windows’ usage has an effect on task performance. Keywords: Web mapping, usability, eye tracking, cognitive processes, iconic representations, and the efficiency of pop-ups.

1 Introduction Web mapping sites became widespread in many professional areas since they provide opportunities such as designing and sharing maps on the World Wide Web. Beside to their role on professional area, it also became very considerable part of our daily life since it makes the navigation easier [13]. Addition to large number of web mapping sites’ users which access these sites with their desktop and laptop PCs, improvements in technology make the internet available in nearly everywhere providing a chance to connect with mobile devices (mobile phones, smart phones, PDAs) and multiply the number of web mapping sites’ users. By the increasing number of web mapping sites’ users, researchers started to conduct usability studies of these sites and to investigate the effects of usability [2, 8, 13, 15]. The term usability is defined by ISO 9241 [9] as “the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments” [9, 13]. Another definition from Nielson, one of the pioneers in the usability field, states usability as a quality evaluation that assesses how easy user interfaces are to use. According to his definition, usability is composed of five quality components [16]; these components are learnability, efficiency (task completion time), memorability, errors, satisfaction. Addition to these parameters obtained from usability study, usage of the eye tracking tools adds a different aspect to the usability field for the reason that it provides objective and quantitative evidence to investigate user’s cognitive processes such as visual and attentional J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 12–21, 2009. © Springer-Verlag Berlin Heidelberg 2009

A Usability Study of WebMaps with Eye Tracking Tool

13

processes [5]. Usage of eye tracking on usability field started at 1950’s [6]. However due to the difficulties in the analysis of huge data obtained from the eye tracking tools, it lost its popularity in 1970’s. With the improvements of the eye tracking technologies, eye tracking tools gain their impacts on the usability field again [10] and nowadays they are accepted as a tool to improve computer interface. In one of the studies about WebMap usability conducted by Nivala [13], severity of the usability problems were investigated. In our study, we aim to make additional analysis to find the reason of these usability problems and make them more clear by analyzing eye movements of the users. The focus of this study is to analyze the effects of the iconic representation of the information and to investigate whether the pop-ups are used efficiently by the user. The eye tracking tool is used for this study in order to follow the position of the users’ eye movements, which helps to measure the attended location on the map. It is known that eye movements provide information about cognitive processes such as perception, thinking, decision making and memory [1, 3, 4, 12, 14]. Evaluation of eye movement provided us the opportunity to focus on the iconic representations, efficiency of pop-up windows and their effects in map comprehension in different WebMaps.

2 Method and Materials 26 subjects (12 female, 14 male) either university students or graduate in a range of ages between 18 and 32 participated to this study. In order to get information about their prior knowledge about WebMap usage and to get the user’s idea about the comprehensibility of the icons and preferences about the WebMaps, a questionnaire was carried out. Each subject evaluated two different WebMaps for different places in US in random order. Six tasks shown in Table 1 were used in the experiment. Users were told that they could give up the task or the experiment whenever they wanted to. Tasks given to the users include; to find given address, to find definite places which are represented with icons (s.t airport, metro station, hospital) and to show the route to specific locations. The experiments are conducted at the Human-Computer Interaction Research and Application Laboratory at Middle East Technical University. Eye movements of users were collected by Tobii 1750 Eye Tracker and analyzed with Tobii Studio. Table 1. Task Description Task No Task Description Instruction Welcome to X City/State. You are planning to look at the city map to specify the locations that you want to visit before starting your trip 1 Point the nearest highway intersection to X International Airport You want to go from X International Airport to X University. Could you describe 2 how to arrive to that location? Find the address of the hospital nearest to X Park 3 Now, you are in X Location. Show the nearest metro/railway station to this place 4 You are searching for the library nearest to X place. Find and point it on the map. 5 Show the intersection point of the given address with X street. 6

14

Ö. Alaçam and M. Dalcı

In the Nivela et al.’s study [13], there is an evaluation of four different web mapping sites. These are Google Maps, MSN Maps & Directions, MapQuest and Multimap. However, since the MSN Maps and Directions, Multimap are based on Microsoft Virtual Earth, we replaced these sites with Live Search Maps that is also based on MS Virtual Earth. Since these are well-known and all have zooming and panning options on their 2D map applications, they are very good candidates for usability testing. Although their common properties mentioned above, they differed in terms of usage of icon representation and pop-up window properties. We conducted the usability testing of Google Maps, Live Search Map, MapQuest and Yahoo Map and investigated the effect of iconic representation of information and pop-up windows by analyzing eye movements. We use the term “The iconic representation of information” as to state the relationship between their semantics and appearance. Addition to evaluation of task completion performance (s.t. task completion score and time), eye tracking data such as fixation length, fixation count, observation length was collected.

3 Results Results are presented under three categories; task performance, analysis of the iconic representations and analysis of pop-up windows. 3.1 Task Performance Users are grouped into two categories according to their WebMap usage experience; experienced users (14 users) for high-level usage frequency and inexperienced users (12 users) for low level usage frequency. One way ANOVA test was conducted to compare mean fixation length on task completion time for experience level. Result shows that user’s experience level has a significant effect on task completion time, F(1,52)=5,30, p>.05. One of the evaluation criteria of comparing the usability of WebMaps is users’ task completion scores. Task completion score was evaluated under three categories; accomplished tasks, unaccomplished tasks and partially accomplished tasks that the users thought that they accomplished a task when they actually did not. Table 2 provides the percentage of users, who accomplished, partially accomplished and did not accomplish each task and also overall score was calculated for each WebMap site. Fig. 1 shows the overall completion score for each map. Results of one way ANOVA shows that task completion score of Google Map is significantly different than MapQuest and Yahoo Map, F(3,48)= 8.629 p<.05. It is also worth to note that significance value of difference between Live and Yahoo is .05. In addition to analysis of task completion score, mean fixation length for each task was analyzed individually (see Fig. 2 for comparison). Only fixation length on accomplished and partially accomplished tasks was counted. The results show that, for the first task, there is no significant difference in fixation length according to map type. The fixation length during task two, a significant difference was found between Live Search Map and MapQuest, Google Map and MapQuest, Yahoo Map and MapQuest F (3,43)= 12.538, p<.05. For the third task, Google Map is significantly

A Usability Study of WebMaps with Eye Tracking Tool

15

different than MapQuest, F (3,33)=3.768, p<.05. For the fourth task, Google Map is significantly different than MapQuest and Live Search Map, F(3,23)= 5.398 p<.05. The fixation length on the fifth task, Google Map is significantly different than MapQuest, Yahoo Map and Live Search Map, F(3,35)=12.058, p<.05. For the task six, only difference in Live Search Map and MapQuest is significant, F(3,41)=2.444, p<.05. Statistical analysis of mean fixation count for each task also shows that there are significant differences between the pairs given above. Table 2. Percentage of users’ task completion scores

Task2

Task3

Task4

Task5

Task6

Overall

tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished

Live Search Map 86.7 13.3 0.0 93.3 0.0 6.7 46.7 26.7 26.7 46.7 0.0 53.3 53.3 13.3 33.3 100.0 0.0 0.0 71.1 8.9 20.0

Map Quest 100.0 0.0 0.0 50.0 35.7 14.2 64.3 14.2 21.4 14.2 7.1 78.6 57.1 0.0 42.8 57.1 0.0 42.9 57.1 9.5 33.3

120,0 Number of participants (%)

Task1

Google Map 100.0 0.0 0.0 100.0 0.0 0.0 35.7 21.4 42.8 78.5 14.2 7.1 78.5 21.4 0.0 100.0 0.0 0.0 82.1 9.5 8.3

100,0 80,0 60,0 40,0 20,0 0,0 Google Map tasks accomplished

Live Search

Map Quest

tasks partially accomplished

Yahoo Map tasks unaccomplished

Fig. 1. The percentage of users according to task completion score

Yahoo Map 100.0 0.0 0.0 63.6 0.0 36.2 36.4 27.2 36.4 9.1 36.4 54.5 45.5 18.1 36.4 45.5 18.2 36.4 50.0 16.7 33.3

16

Ö. Alaçam and M. Dalcı

600 500 400 300 200 100 0 görev1

görev2

Live Search Map

görev3 Google Map

görev4 Yahoo Map

görev5

görev6

MapQuest

Fig. 2. Fixation Length (sec.) on each task according to WebMaps

3.2 Analysis of the Iconic Representations In order to investigate the efficiency of iconic representations in these WebMaps, the observation length on icons was counted frame by frame in specific tasks. In order to analyze icon usage, first, third and fourth tasks were selected since these tasks contain specific places which can be represented by icons (such that airport icon for first task, hospital icon for task three, metro/railway icon for task 4, and pointers that appears after users’ search for all three tasks). Other icons displayed on the map during these tasks were also investigated. Fixation length on each icon and the time which they are displayed on the map were counted and then the percentage of iconic representation looking time was calculated. One-way ANOVA results shows that there is no significant difference in observation length on icons for each WebMap, F(3,48)=1,859, p>.05. In addition, no correlation between the icon usage looking time percentage and completion score was found. (Spearman’s rho=.37, p>.05) Icons are divided into two categories as, task related icons and task unrelated icons. However, since every map could not have a representation for each icon, they were investigated individually, without conducting statistical analysis. Since MapQuest had only pointer icons, these are not compared with the specific icons in other maps. Table 3 provides the percentage of looking time for each icon in each WebMap. Observation length on Airport in Yahoo Map, is 13,3% of the time that it appears on the map. However, since the other WebMaps (Google Maps, MapQuest and Live Search Map) do not have icon for airport, there can be no comparison analysis. The pointers which represent the searched location have approximately same looking time for all maps. For the metro and railway icons, Google icon has biggest looking time percentage, and then it is followed by Live Search Map and Yahoo Map respectively. Since the MapQuest and Live Search Maps do not have hospital icons, looking time of icons on only Yahoo Map and Google Map were investigated. Even tough Google and Yahoo Maps contain an icon for “Hospital”, the users are expected to zoom close

A Usability Study of WebMaps with Eye Tracking Tool

17

enough for that icon to be visible. However none of the users were inclined to zoom to that distance while performing their tasks. The eye movement analysis of users which perform the experiment on Live Map Search shows an interesting outcome; some task unrelated icons (s.t. park, hotel or sponsored link icons) have been remarkably fixated by the users. Table 3. The percentage of looking time for each icon Icon Type Google Map Task Related Icons

Unrelated Icons

Live Search

Map Quest

Yahoo Map

Airport

Na*

Na

Na

13,3

Pointers

10,6

10,4

10,9

10,5

Metro/railway

13,5

10,8

Na

7,2

University

3,5

Na

Na

Na

Hospital

0,0

Na

Na

0,0

Park

1,4

Na

Na

Na

Hotel

Na

5,5

Na

Na

Sponsored link

Na

1,7

Na

Na

*Not Applicable.

Task based investigation has been made regarding iconic representations. Correlations between the icon looking time percentage, completion score and observation length for Task 1,3 and 4 were examined individually. For Task 1, no correlation between these parameters was found. The analysis conducted for the third task, correlations have been found between the icon looking time percentage and completion score (Pearson’s r=.523; p<.05) and between icon looking time and observation length (Pearson’s r=-.368; p<.05). For Task four, a correlation has been found between icon looking time and observation length (Pearson’s r=-.289; p<.05). On the other hand, no correlation has been found between the icon looking time percentage and completion score (Pearson’s r=.165; p>.05). Addition to eye movement analysis, user evaluation questionnaire was carried out. The users were asked to predict the meaning of icons and rate their comprehensibility in a scale from 1 to 10 (the results are given in Table 4). This gives us the opportunity to evaluate the efficiency of the relationship between their semantics and appearances and show whether there is an ambiguity. The comprehensibility ratings for only correct predictions were counted. It can be claimed that there is a consistency between its appearance and semantic when both the correct predictions and comprehensibility rate of a icon is high. Although the comprehensibility evaluated by the users is high for the hospital icon in Google, only 6.3 % of the users predict the meaning of the icon correctly. The comparison of the results for metro/railway icons indicates that iconic representations in Google Maps and Yahoo Maps are comprehended more easily by the users than in Live Search Maps. None of the users predicts the meaning of the sponsored link and hotel icon in Live Search Map. When we look at the Table 3 for looking time percentage of these icons, it can be concluded that the users notice them without attaching a meaning.

18

Ö. Alaçam and M. Dalcı Table 4. Users’ ratings on iconic representations Icon

% of correct definition by users

Comprehensibility

+

6,3

9,0

Hospital (Yahoo Map)

87,5

8,5

Airport (Yahoo Map) Metro/Railway (GoogleMap) Metro/Railway (Yahoo Map) Metro/Railway (Live Search Map)

100

10

81,3 / 93,8

6,6 / 7,9

81,3

6,6

62,5

6,2

Hotel (Live Search Map)

0

Na

Sponsored Links (Live Search Map)

0

Na

Hospital (Google Map)

Moreover, users were asked to specify their icon preference and rate the usabilty of the maps which they used during the experiment. We can group the icons into three main categories; these are Pictorial, Textual and Numerical/Alphabetical icons. Textual icons like (s.t. ) are mostly prefered icons (37.5%). 31,3 % of the users prefer pictorial icons (s.t. ), to be used in maps. Another 31,3 % of the users prefer numerical/alphabetical icons (s.t. ). Additionally, users’ usabilty ratings of Web Maps are parallel with task completion score given in Table 2. Their ratings are 8.3 for Google Map 5.1 for live Search Map, 4.6 for Yahoo Map and 3,4 for MapQuest. 3.3 Analysis of Pop-Up Windows Additional analysis was conducted to investigate the usage of pop-up windows. It can be suggested that the aim of the pop-up is to direct users focus on something else where they are provided with additional information either regarding their task or not. Moreover, these pop-ups can facilitate additional search on locations that the task requires. Therefore pop-up windows are very important parts of the web maps and they frequently appear during map usage. In order to investigate the efficiency of popup usage in these WebMaps, the sections that contain pop-up windows on map area were extracted for each task. Fixation length and fixation count on pop-up windows and display time of it on map area was counted. The analysis of looking time of popup windows when they appear on the map gives us an idea about whether the users prefer to use them. The results of this analysis showed that Google Maps use the popup windows more efficiently, since % 64.8 of the fixations on the map are on the popup area. It is followed by MapQuest, Live Search Map and Yahoo Map respectively (see Fig. 3). One way ANOVA results also indicates that there is a significant difference between Google Map and Live Search Map in pop-up usage percentage. Google

A Usability Study of WebMaps with Eye Tracking Tool

19

Observation Length (sec)

200,00 180,00 160,00 140,00 120,00 100,00 80,00 60,00 40,00 20,00 0,00

Google Map

Live Search Map

MapQuest

popup

48,96

40,05

63,16

25,79

map

75,52

96,78

115,19

77,38

map

Yahoo Map

popup

Fig. 3. Observation Length on Pop-up / Map for each WebMap

Map is also significantly different than Yahoo Map. Addition, there is also difference between Yahoo Map and MapQuest, F(3,51)=5.939, p<.05. Additionally, correlations have been found between fixation count on pop-ups and overall completion score (Spearman’s rho=.396; p<.05) and between fixation length on pop-ups and overall completion score (Spearman’s rho=.423; p<.05).

4 Conclusion By analyzing eye movements which are indicators of cognitive processes, we examined the effects of iconic representations, pop-up window usage, and their roles on the usability of these mapping sites. Since comprehension of maps contains very complicated cognitive process, decreasing the cognitive load by making the icons more comprehensible and the pop-ups more usable will cause effectiveness and efficiency in the usage of web mapping sites. Task performance evaluation shows that there is a significant difference between these Webmaps. When the tasks are getting complicated (ex. to find an address of specific location near to another location), differences between these WebMaps in terms of task completion time and score has become apparent. Besides the differences in overall display organization, in a micro level framework, we examined the effect of the iconic representation of information and efficiency of the usage of popup windows. The analysis indicates that there is gab between user’s ratings on icons and their looking time percentages. Even the icons that are correctly predicted and given high comprehensibility level by the users have a low looking time. Additionally, it is also worth to be noted that icon’s looking time percentage is low for even task related icons. This means users have difficulty to detect them when they are performing their tasks although the icons are highly related with tasks. To make them more visible can help the users notice them, and increase their task completion performance. Moreover, analysis on the efficiency of pop-up windows, which are other

20

Ö. Alaçam and M. Dalcı

widely used elements of WebMaps, shows that users’ pop-up usage differs significantly according to WebMap type and this parameter is positively correlated with task completion score. In addition, as an expected outcome, experience level has a significant role on WebMap usage performance.

5 Further Studies However, icons for local traffic signs (ex. Highway numbers) were highly fixated during tasks; however these findings have been disregarded due to the user’s lack of familiarity towards to locations. Follow up studies to investigate these icons can be conducted with native users of the location. In addition, to evaluate the interaction between some particular areas of web site (s.t. search bars, menus, map area, information area which shows the results of the searched location) gives additional information about the efficiency and effectiveness of the usability of the WebMaps. Acknowledgements. We thank to the Computer Center / Middle East Technical University, and TUBİTAK (for support under grant SOBAG 104K098) for providing eye tracking system. We also thank to Yasemin Saatiçioğlu Oran and our colleagues in METU Computer Center for their valuable support.

References 1. Barkowsky, T.: Mental Processing of Geographic Knowledge. In: Montello, D.R. (ed.) COSIT 2001. LNCS, vol. 2205, pp. 371–386. Springer, Heidelberg (2001) 2. Davies, C., Wood, L., Fountain, L.: User-centred GI: hearing the voice of the customer. In: AGI 2005: People, Places and Partnerships, Annual Conference of the Association for Geographic Information, London, UK, November 8–10, 2005, Association for Geographic Information, London (2005) 3. Downs, R.G., Liben, L.S.: The Development of Expertise in Geography: A CognitiveDevelopmental Approach to Geographic Education. Annals of the Association of American Geographers 81(2), 304–327 (2005) 4. Downs, R.G., Liben, L.S., Daggs, D.G.: The Development of Expertise in Geography: A Cognitive-On Education and Geographers. The Role of Cognitive Developmental Theory in Geographic Education 78(4), 680–700 (2005) 5. Duchowski, A.T.: A Breadth-First Survey of Eye Tracking Applications. Behavior Research Methods, Instruments, & Computers (BRMIC) 34, 455–470 (2002) 6. Fitts, P.M., Jones, J.E., Milton, J.L.: Eye movements of aircraft pilots during instrument landing approaches. Aeronautical Engineering Review 9(2), 24–29 (1950) 7. Global Mapping International, http://www.gmi.org/mapping/websites.htm 8. Haklay, M., Zafiri, A.: Usability Engineering for GIS: Learning from a Screenshot. The Cartographic Journal 45(2), 87–97 (2008) 9. ISO. ISO/DIS 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability, International Organization for Standardization (1998) 10. Jacob, R.J.K., Karn, K.S.: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises (Section commentary). In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eyes: Cognitive and Applied Aspects of Eye Movements, Elsevier Science, Oxford (2003)

A Usability Study of WebMaps with Eye Tracking Tool

21

11. Montello, D.R.: Cognitive Map-Design Research in the Twentieth Century: Theoretical and Empirical Approaches. Cartography and Geographic Information Science 29(3), 283– 304 (2002) 12. Murata, A., Furukawa, N.: Relationships among display features, eye movement caharcteristics and reaction time in visual search. Human Factors 47(3) (2005) 13. Nivala, A.M., Brewster, S., Sarjakoski, L.T.: Usability Evaluation of Web Mapping Sites. The Cartographic Journal 45(2), 129–138 (2008) 14. Richardson, D.C., Dale, R., Spivey, M.J.: Eye movements in language and cognition: A brief introduction. In: Gonzalez- Marquez, M., Coulson, S., Mittelberg, I., Spivey, M.J. (eds.) Methods in cognitive linguistics, John Benjamins, Amsterdam (in press) 15. Skarlatidou, A., Haklay, M.: Public Web Mapping: Preliminary Usability Evaluation. In: Proceedings of GIS Research UK Conference, Nottingham, April 5-7 (2006) 16. Usability 101: Introduction to Usability (August 25, 2003), http://www.useit.com/alertbox/20030825.html (retrieved 16:22, September 21,2008)

Feature Extraction and Selection for Inferring User Engagement in an HCI Environment Stylianos Asteriadis, Kostas Karpouzis, and Stefanos Kollias National Technical University of Athens, School of Electrical and Computer Engineering, Image, Video and Multimedia Systems Laboratory, GR-157 80 Zographou, Greece [email protected], {kkarpou,stefanos}@cs.ntua.gr

Abstract. In this paper, we present our work towards estimating the engagement of a person to the displayed information of a computer monitor. Deciding whether a user is attentive or not, and frustrated or not, helps adapting the displayed information of a computer in special environments, such as e-learning. The aim of the current work is the development of a method that can work userindependently, without necessitating special lighting conditions and with only requirements in terms of hardware, a computer and a web-camera. Keywords: User engagement, Head Pose, Eye Gaze, Facial Feature tracking.

1 Introduction While a lot of work has been done regarding the issue of user attention estimation in multi-user environments, such as meetings [7], very few articles have been published for estimating attention (engagement) in an HCI environment [6]. In this work, we propose a method for inferring user attention based on the extraction of features deriving from facial analysis. Such systems have mainly been proposed for estimating drivers' attention. For example, in [8], a monocular system is used and color precicates are employed to detect and track the driver's facial features. Facial geometry and eye closure are calculated and driver's attention is extracted with the use of three finite state automata. In [3], the authors use solely eyes closure to determine whether a driver is attentive or not. To this aim, they use a gaussian model to describe driver's attention and, eye closure for certain amounts of time would denote inattention or fatigue. In [10], the position in space of each facial feature is detected using a stereo vision system. Based on these positions, a least square algorithm estimates the head orientation and further analysis follows for the detection of the eye gaze. The combination of the above gives a good estimate of the direction at which a user is looking, however, it requires initialization. In our work, we combine head pose with eye gaze, as well as other biometrics, in a monocular environment, not necessitating any particular lighting conditions or calibration. There is not much work in bibliography combining the two features in an unconstrained environment. Typical work is the one J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 22–29, 2009. © Springer-Verlag Berlin Heidelberg 2009

Feature Extraction and Selection for Inferring User Engagement

23

reported in [12], where facial symmetry along with Gabor filters are used for estimating head pose and eye gaze respectively. A look-up table is then built for corresponding the resulting eye gaze and head pose with the final focus of attention estimate. Here, we propose a work that can be summarized as follows: Face detection [11], followed by facial feature detection is the first step of our method, while tracking follows. Based on facial features' motion, a series of biometric measurements are extracted and their appropriateness is evaluated for inferring the level of frustration or attentiveness of a user in a Human-Computer-Interaction scenario. Our algorithm is able to recover and re-initialize in cases of occlusion or tracking failure.

2 Facial Points Detection and Tracking The method reported in [1] is used for face and eye centre and mouth corner (here, enhanced with upper and lower lip points) localization. For the detection of the eye corners (left, right, upper and lower eyelids) a technique similar to that described in [13] is used. In the current work, the point between the nostrils and two points on each eyebrow have also been used, as will be discussed later. For nostrils detection, an area around a segment of the perpendicular to the inter-ocular line was extended starting from the middle of the eyes. The darkest row of this area is considered as the vertical position of the nostrils and the middle point of this row is further used for our experiments. In a similar manner, two points on each eyebrow are extracted, as the darkest points in a neighborhood above the eye corners. The above are illustrated in Fig. 1, where the luminance values of two search areas have been projected on the vertical axis. The minimum of the projections corresponds to the features in search. Tracking is done using a three-Pyramid Lukas-Kanade algorithm. Geometrical face models, and prototypes of natural human motion are employed for recovering from erroneous tracking (see subsection 3.1).

Fig. 1. Eyebrow and nose detection search regions

3 Feature Extraction The features extracted in our method are the following: Head pose, eye gaze, eyebrow movements, head horizontal and vertical speed components, mouth horizontal and vertical opening, relative movements of the user back and forwards.

24

S. Asteriadis, K. Karpouzis, and S. Kollias

3.1 Head Pose Estimation Head rotation is calculated by examining the translation of the point in the middle of the inter-ocular line with regards to its position when the user was rotated frontally (see Fig. 2), thus providing the Head Pose Vector p = [px py], where px and py are the horizontal and vertical components of the eye middle point’s translation, respectively, normalized with the inter-ocular distance, as calculated at start-up, to cater for scale variations. The fraction of the inter-ocular distance with the vertical distance between the eyes and the mouth is monitored, and if it is restricted within certain limits with regards to its value at a frontal position, no rotation is decided. As tracking, many times, fails, thus giving false estimates of head pose (as well as the other features), a series of rules were integrated: After large rotations, some features are occluded and cannot be further recovered. In this case, when the user comes back to a frontal position, after nt1 frames, the pose length reduces in length but is above a certain threshold, as one of the eyes is not well tracked and the eye center is not at the same neighborhood as at start-up. In this case, the algorithm can re-initialize. The above can be modeled as in equations (1),(2):

p(n) < thr1 ⋅ p(n − nt1 )

(

)

var p n−nt 2:n < thr2

(1) (2)

where ║*║ denotes a vector length metric. Equations (1)-(2) are interpreted as follows: If the Head Pose Vector length at the current frame n is smaller than a fraction thr1 of its value at frame n-nt1, and its variance for the last nt2 frames (with nt2
Fig. 2. Pose changes during a video of a person in front of a monitor

Feature Extraction and Selection for Inferring User Engagement

25

smaller than thr2, the algorithm re-initializes. In our experiments, we used nt1=10, nt2=7, thr1=0.7, thr2=0.05. If the above conditions are met, but the user has not turned frontally, face detection fails and, thus, frontal rotation is not decided. In general conditions, however, the algorithm re-initializes by re-detecting the face, facial features and re-starting to track. Further constraints that are taken into account are related with the displacement of features in subsequent frames. By assuming an orthographic projection in the interval between two subsequent frames, it is expected that features are shifted in a uniform way. Finding such outliers and re-calculating the mean shift with the rest of the features helps positioning erroneous points to the position that would agree with the rest of the features’ shift. As experiments showed, the above refinement is achieved after 7-10 iterations per frame.

Fig. 3. Gaze changes during a video of a person not moving his head in front of a monitor

3.2 Eye Gaze Estimation Eye gaze is extracted by monitoring the eye centre movements with regards to a coordinate system defined by the positions of the eye corners and eyelids at each frame (see Fig. 3). The resulting displacement provides with the Eye Gaze Vector gy], where gx and gy is the horizontal and vertical component respectively.

g = [gx

3.3 Extraction of Further Features The vertical movements of the eyebrows with regards to the upper eyelids are also extracted, and the horizontal and vertical components of the speed (in pixels per frame) of the head movements are calculated. Furthermore, mouth opening is calculated, with reference to the initial distance between the mouth corners and the lips distance at start-up. Finally, the inter-ocular distance changes are monitored, and

26

S. Asteriadis, K. Karpouzis, and S. Kollias

calculated as fractions of the eye centres’ distance with regards to the first frame of each initialization of the system. In this way, when changes in inter-ocular distance are not due to head rotations, qualitative measurements of user movement back and forth are achieved.

4 Feature Selection The experiments were conducted on a database consisting of children with learning difficulties, between the age of 8 and 10. The recorded videos were 720×576 pixels and the frame-rate 25fps. A total of about 10000 and 12250 frames were used for the case of attention/non-attention and frustration/non-frustration problems respectively. The videos were annotated by experts. One of the difficulties of the dataset was that the positive instances (frustration, non-attentiveness) were very few in comparison to the negative ones, and this limited the training session prototypes. In order to evaluate the appropriateness of each of the above features, the Fisher's exact test [4] was used. To this aim, the 3-bin histogram of each of the features was calculated and the resulting distribution for positive instances was compared against the distribution of the same feature throughout all videos, regardless of the state. In our case, we chose Fisher's exact test and not any other method (e.g. chi-square method) because it is ideal for small scale data. Indeed, in many occasions (for example, horizontal speed of the head when the user is frustrated), there are only a few instances where there are low or high values at the correspondent histogram bins. Fisher's exact test is ideal in cases of such small samples. For the event of non-attentiveness, among all features, tests showed that for the features of Head Pose, Eye Gaze, Inter-Ocular Distance Changes and Head Speed, the null hypothesis (that observed and expected distributions do not differ) should be rejected with higher confidence than for the rest of the features. For the event of frustration, Head Pose, Horizontal and Vertical Head speed and Eye Gaze do not follow the null hypothesis as much as the rest of the features do, as it was expected. Figure 4 justifies the rejection of some of the features due to high p-values, while Figures 5 and 6 illustrate examples of data for each class.

Fig. 4. p-values for feature selection in attention/non-attention and frustration/non-frustration scenarios

Feature Extraction and Selection for Inferring User Engagement

27

Fig. 5. Features used for attention/non-attention classification

Fig. 6. Features used for frustration/non-frustration classification

5 Experimental Results For testing the accuracy of our system, a Sugeno-type fuzzy [9] inference system was built for each case. The idea behind using fuzzy systems has to do with the fact that behavioral states do not necessarily belong to certain classes but, rather, they are fuzzy concepts. For example, frustration or distraction can be given confidence values, and outputs of fuzzy systems are ideal in this case. Prior to training, our data were clustered using the sub-cluster algorithm described in [2]. This algorithm, instead of using a grid partition of the data, clusters them and, thus, leads to fuzzy systems deprived of the curse of dimensionality. The number of clusters created by the algorithm determines the optimum number of the fuzzy rules. After defining the fuzzy inference system architecture, its parameters (membership function centers and widths), were acquired by applying a least squares and back-propagation gradient descent method [5]. Tables 1 and 2 summarize the results of the overall accuracy of our system in estimating the behavior of a user in the cases attention/non-attention and frustration/non-frustration experiments using different sets of features with low p-value as inputs and 1 or 0 the target states (attention/non-attention or frustration/non-frustration). Testing was done by adopting a leave-one-out protocol.

28

S. Asteriadis, K. Karpouzis, and S. Kollias

From tables 1 and 2, it can be seen that, in the case of frustration, although eye gaze has low p-value (see Fig. 3), excluding it from experiments does not deteriorate the results but they are marginally higher. This is due to the fact that, in cases of frustration, in our dataset, eye gaze vector length was strongly correlated with head pose vector length. Similarly, although Head Speed (horizontal and vertical) has low p-values in the attention tests, results have shown that, excluding these parameters from our decision systems, would improve the results. More careful observation of our data and the corresponding annotation gave the following explanation: Head speed is only large at the beginning of those time segments when a person is turning his/her head away from the camera. At those time segments, head pose vector has small values but head speed is high. However, such movements can also be met during attention time-stamps, as it is very frequent for a reader to make small rapid movements without changing his head pose a lot. For the above reason, it was decided to exclude head speed from our experiments in the case of attention estimation. The database we used was acquired under normal lighting conditions, with very challenging subjects: Children with learning difficulties. Testing our system on such a dataset is challenging, not only because of its nature, but also due to the fact that annotation is subjective. However, the results obtained are extremely promising. Table 1. Neuro-Fuzzy System decision accuracy for two different sets of low p-value features for detecting User Attention Features Head Pose, Eye gaze, Distance changes, Head speed Head Pose, Eye gaze, Distance changes

Overall success rates 84.00% 88.00%

Table 2. Neuro-Fuzzy System decision accuracy for two different sets of low p-value features for detecting User Frustration Features Head Pose, Horizontal and Vertical Head speed Head Pose, Horizontal and Vertical Head speed, Eye Gaze

Overall success rates 82.00% 80.63%

6 Conclusions and Future Work We presented a method for the automatic estimation of the behavior of a person in front of an HCI environment. Our system is un-intrusive, thus, leaving space for spontaneous behavior, it does not depend on controlled conditions in terms of lighting, and this constitutes it ideal for different settings. Furthermore, since the system does not require any a-priory knowledge of the user or the camera, it does not need any kind of training or calibration beforehand. Future extensions of our work shall include the creation of a common framework for discriminating among a set of states simultaneously. To this aim, we will build a database suitable for our research and work on developing a facial feature tracker, highly specialized for such applications.

Feature Extraction and Selection for Inferring User Engagement

29

Acknowledgments This work has been funded by the FP6 IP CALLAS (Conveying Affectiveness in Leading-edge Living Adaptive Systems), Contract number IST-34800 and by the IST Project ’FEELIX’, (under contract FP6 IST-045169).

References 1. Asteriadis, S., Nikolaidis, N., Pitas, I., Pardàs, M.: Detection of facial characteristics based on edge information. In: 2nd International Conference on Computer Vision Theory and Applications (VISAPP), Barcelona, Spain, pp. 247–252 (2007) 2. Chiu, S.L.: Fuzzy Model Identification Based on Cluster Estimation. Journal of Intelligent and Fuzzy Systems 2(3), 267–278 (1994) 3. D’Orazio, T., Leo, M., Guaragnella, C., Distante, A.: A visual approach for driver inattention detection. Pattern Recognition 40(8), 2341–2355 (2007) 4. Fisher, R.A.: Statistical Methods for Research Workers. Hafner Publishing (1970) 5. Jang, J.S.R.: ANFIS: Adaptive-Network-Based Fuzzy Inference System. IEEE Transactions on Systems, Man, and Cybern. 23, 665–684 (1993) 6. Matsumoto, Y., Ogasawara, T., Zelinsky, A.: Behavior recognition based on head pose and gaze direction measurement. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Takamatsu, Japan, pp. 2127–2132 (2000) 7. Otsuka, K., Takemae, Y., Yamato, J.: A probabilistic inference of multiparty-conversation structure based on markov-switching models of gaze patterns, head directions, and utterances. In: ICMI, pp. 191–198 (2005) 8. Smith, P., Member, S., Shah, M., Lobo, N.D.V.: Determining driver visual attention with one camera. IEEE Trans. on Intelligent Transportation Systems 4, 205–218 (2003) 9. Takagi, T., Sugeno, M.: Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. Syst. Man Cybern. 15(1), 116–132 (1985) 10. Victor, T., Blomberg, O., Zelinsky, A.: Automating driver visual behavior measurement. In: 9th Vision in Vehicles Conference, Australia (2001) 11. Viola, P.A., Jones, M.J.: Rapid object detection using a boosted cascade of simple features. In: Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511–518 (2001) 12. Weidenbacher, U., Layher, G., Bayerl, P., Neumann, H.: Detection of head pose and gaze direction for human-computer interaction. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS, vol. 4021, pp. 9–19. Springer, Heidelberg (2006) 13. Zhou, Z.H., Geng, X.: Projection functions for eye detection. Pattern Recognition 37(5), 1049–1056 (2004)

Informative or Misleading? Heatmaps Deconstructed Agnieszka (Aga) Bojko User Centric, Inc. 2 Trans Am Plaza Dr, Ste 100, Oakbrook Terrace, IL 60181 USA [email protected]

Abstract. Eye tracking heatmaps have become very popular and easy to create over the last few years. They are very compelling and can be effective in summarizing and communicating data. However, heatmaps are often used incorrectly and for the wrong reasons. In addition, many do not include all the information that is necessary for proper interpretation. This paper describes several types of heatmaps as representations of different aspects of visual attention, and provides guidance on when to use and how to interpret heatmaps. It explains how heatmaps are created and how their appearance can be modified by manipulating different display settings. Guidelines for proper use of heatmaps are also proposed. Keywords: Heatmaps, attention maps, eye tracking.

1 Introduction Heatmaps are two-dimensional graphical representations of data where the values of a variable are shown as colors. Heatmaps are compelling for two reasons. First, the intuitive nature of the color scale as it relates to temperature minimizes the amount of learning necessary to understand it. From experience, we know that yellow is warmer than green, orange is warmer than yellow, and red is hot. It is not difficult to then figure out that the amount of heat is proportional to the level of the represented variable. Second, heatmaps show the data directly over the stimulus. Because the data could not be any closer to the elements to which they pertain, little mental effort is required to read a heatmap. Heatmaps can be of great value to papers, reports, and presentations because they summarize large quantities of data that would be much more difficult to grasp if presented numerically. Heatmaps help us quickly see “the big picture” including any patterns or trends that may exist in the data. In the user experience field, heatmaps can represent various types of data, such as usage (e.g., clicks, key presses), accuracy, or visual attention. This paper focuses exclusively on attention heatmaps, which have become popular over the past few years due to the increased usage of eye tracking technology. Attention heatmaps can be easily generated with the help of an eye tracking software application such as Tobii’s ClearView, Tobii Studio, SMI’s BeGaze, NYAN, or J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 30–39, 2009. © Springer-Verlag Berlin Heidelberg 2009

Informative or Misleading? Heatmaps Deconstructed

31

EyeTools. Anyone with an eye tracker and the right software can create a heatmap. No knowledge of eye movements or of how heatmaps are created is required. As a result, heatmaps are often generated unnecessarily or are misinterpreted by those who do not understand what the visualizations are really showing or, perhaps even more importantly, not showing. Heatmaps can be deceptive because they look so intuitive that we often do not realize how much we actually do not understand. This paper describes different heatmap types and their limitations, as well as settings used to manipulate the appearance of heatmaps. It also discusses when heatmaps should and should not be used. Proposed guidelines for using heatmaps correctly conclude this work.

2 Types of Attention Heatmaps Heatmaps are often shown with little, if any, description of what it is they are representing. The assumption is that they are showing “attention” or “eye movements” but knowing that is certainly not enough to be able to truly understand a heatmap. There are different aspects of eye movements that heatmaps can represent. Examples include fixation count, absolute or relative gaze duration, and percentage/proportion of participants who fixated on each area of the stimulus. Choosing the right heatmap to present depends on the study objectives and the eye movement measures that address these objectives. For example, if search efficiency was of interest to the researchers, one of the measures collected and analyzed might be the number of fixations prior to acquiring the target [1]. Therefore, assuming that the analysis would benefit from data visualization, a fixation count heatmap should be presented. A fixation count heatmap would also be appropriate if the study goal was to determine the amount of interest generated by various elements of the stimulus during a free-view task (i.e., task with no specific task instructions) [1]. However, if noticeability of a particular element was of interest, the percentage of participants who fixated on the element could be used as a measure (in addition to, for example, time to first fixation), which would warrant a participant percentage heatmap. Because each heatmap type has different limitations that impact its interpretation, it is not only important to be aware of these limitations, but also to know the types of all heatmaps included in papers, reports, and presentations. 2.1 Fixation Count Heatmap A visual fixation can be loosely defined as a relatively stationary eye position focused on a particular location of the stimulus (a more precise definition is discussed in section 3.1). Fixations are important events that provide insight into human cognition because during each fixation we extract visual information that we process [2]. A fixation count heatmap (see Fig. 1) shows the accumulated number of fixations across participants. Each fixation made by each participant adds a value to the color map at the location of the fixation [3]. This value is the same for each fixation regardless of its duration, so a 100 ms fixation is represented in the same way as a 900 ms fixation. Thus, when looking at a fixation count heatmap, we cannot assume that areas of the same color received similar total gaze time.

32

A. (Aga) Bojko

Fig. 1. Fixation count heatmap (n = 13; 0 – 12 s; free-view task)

Fixation count heatmaps can also be biased towards individuals who show high interest in elements that others do not. For example, two elements can be the same color but one attracted ten fixations from one participant only, while the other attracted attention of ten participants, one fixation from each. Therefore, we cannot assume that areas that appear similar in terms of “heat” are equivalent in terms of the number of participants who looked at them. Another limitation of fixation count heatmaps is the fact that they can be skewed towards individuals who had a longer exposure to the stimulus and thus an opportunity to produce more fixations. For example, if participant A spent twice as much time on the stimulus than participant B, participant A’s data would impact the heatmap twice as much as participant B’s data. This is something to always keep in mind when viewing heatmaps created based on unequal exposure times. 2.2 Absolute Gaze Duration Heatmap An absolute gaze duration heatmap (see Fig. 2) shows the accumulated time participants spent looking at the different areas of the stimulus. Each fixation made by each participant adds a value to the color map that is proportional to its duration [3]. For example, a 900 ms fixation will be nine times higher in color value than a 100 ms fixation. Because fixation duration is an indicator of cognitive processing [4], a heatmap that is scaled by fixation duration not only shows which areas were attended to but also represents the level of cognitive processing that the areas required.

Fig. 2. Absolute gaze duration heatmap (n = 13; 0 – 12 s; free-view task)

Informative or Misleading? Heatmaps Deconstructed

33

An absolute gaze duration heatmap can be misleading because it displays different phenomena in the exact same way. For example, this type of heatmap will make one 900 ms fixation look the same as nine 100 ms fixations. A 900 ms fixation on an element indicates that one person looked at it for a while, while nine 100 ms fixations could mean, for example, that one person made nine brief fixations to the element or nine people made one brief fixation each. In addition, similar to the fixation count heatmaps, absolute gaze duration heatmaps can be biased towards individuals who spent more time looking at the stimulus. To eliminate any bias due to unequal exposure times, the gaze duration data can be normalized to create relative gaze duration heatmaps. 2.3 Relative Gaze Duration Heatmap A relative gaze duration heatmap shows the accumulated time each participant spent fixating at the different areas of the stimulus relative to the total time the participant spent looking at the stimulus [3]. In other words, if participant A spent 6 seconds on a web page including 2 seconds on the navigation and participant B spent 60 seconds on the same web page including 20 seconds on the navigation, this type of heatmap will make their data, as it relates to the navigation, the same weight. Similar to the absolute gaze duration heatmap, this heatmap will also show a high gaze time by an individual (proportional to his/her total viewing time) the same way as several short gaze times by a number of individuals (proportional to their total viewing times). If the exposure time is equal across participants (e.g., all participants saw the page for 12 s), which is the case in the examples presented in this paper, the relative gaze duration heatmap and the absolute duration heatmap will be identical. 2.4 Participant Percentage Heatmap A participant percentage heatmap (see Fig. 3) shows the percentage of participants who fixated on the different areas of the stimulus. Each participant who looked at any given location adds a value to the color map. This value is the same for each participant regardless of the number of fixations he or she made or the fixation durations. Thus, an area that was briefly fixated once by each of the participants will be presented in the same color as an area that was fixated multiple times by each of the participants and the fixations were much longer.

Fig. 3. Participant percentage heatmap (n = 13; 0 – 12 s; free-view task)

34

A. (Aga) Bojko

3 Display Settings for Creating Heatmaps The appearance of heatmaps can be modified by manipulating various display settings. While adjusting the settings cannot change the relative distribution of attention, certain areas can be made to appear “hotter” or “colder.” This can be done, for example, by changing the fixation criteria used for the analysis, showing raw data instead of fixation data, changing the upper threshold definition of the color scale, or modifying the time segment for the presented data. Since heatmap display settings can have a great impact on the appearance of heatmaps, they must be properly selected and communicated to ensure accurate interpretation. 3.1 Changing Fixation Criteria There are several algorithms that can be used to define a fixation. A common algorithm used in commercial eye tracking software is based on duration and dispersion threshold identification. To define a fixation using this algorithm, two parameters need to be specified: minimum fixation duration (e.g., 80 ms) and maximum dispersion threshold (e.g., 0.5 degree of visual angle) [5]. A fixation defined by 80 ms and 0.5° will encompass all consecutive eye movements that occurred within 0.5° from each other for at least 80 ms. Unless noted otherwise, the heatmaps in this paper were created using these settings. Manipulating the duration and dispersion thresholds will change the number of fixations in the data. For example, increasing the minimum fixation duration from 80 ms to 200 ms will decrease the number of fixations because it will exclude all the fixations between 80 ms and 200 ms. Increasing the maximum dispersion threshold from 0.5 degree to 1 degrees of visual angle will also decrease the number of fixations because some of the fixations that are closer together will be combined. Conversely, reducing the minimum fixation duration and maximum dispersion threshold will increase the number of fixations. More fixations in the data will increase the amount of “heat” in the heatmap, as shown in Figure 4.

Fig. 4. Fixation count heatmaps based on the same data (n = 13; 0 – 12 s; free-view task). On the left: minimum fixation duration = 200 ms; on the right: minimum fixation duration = 80 ms.

The lack of explicit fixation definition is an issue that does not pertain specifically to heatmaps but to entire papers and reports. Many user experience studies that analyze fixation data never mention how these fixations were defined. However, this information is very important for two reasons. First, the results of different studies are

Informative or Misleading? Heatmaps Deconstructed

35

not comparable unless it is clear that fixations were defined in the same way. Second, if the definition is not provided, it cannot be verified whether or not the fixation criteria were appropriate for the stimuli used in the study. For example, fixation duration threshold for image viewing should be higher than for reading because image viewing tends to produce longer fixations than reading due to the fact that more information is being processed in a single fixation [6]. The fixation definition used to create heatmaps should match the definition used for data analysis. Changing the fixation duration to obtain a visualization of a particular intensity is not a good practice. 3.2 Displaying Raw Data Instead of Fixation Data One step further from decreasing the fixation duration and dispersion thresholds is presenting raw data instead of fixation data. Raw data consists of meaningful eye movements (raw fixation points) and “noise” – eye movements that have little meaning in most user experience research. The noise includes eye movements that take place during saccades (rapid eye movements between fixations) as well as drifts, tremors, and flicks that occur during fixations [5]. Adding all the noise intensifies the heatmap, increasing the area covered in red (see Fig. 5).

Fig. 5. Fixation count heatmaps based on the same data (n = 13; 0 – 12 s; free-view task). On the left: fixation data; on the right: raw data.

As a general rule, fixation data rather than raw data should be used when creating visualizations unless the stimulus has moving elements and the heatmap has to show smooth pursuit eye movements. We can assume that fixation data was used if the paper or report specifies how the researchers defined a fixation. 3.3 Changing the Definition of the Color Scale Upper Threshold Another way to manipulate the amount of heat in heatmaps is by changing the definition of the upper threshold on the scale, which is usually indicated by the red color. If the requirements for an area to be red are lowered, the amount of red in the heatmap will increase. Lowering of the upper threshold of the scale can be achieved by decreasing the minimum number of fixations in fixation count heatmaps (see Fig. 6) or by decreasing the minimum gaze length in absolute gaze duration heatmaps.

36

A. (Aga) Bojko

There is no set process for choosing the right upper threshold. The rule of thumb is to make sure that the heatmap properly captures the range of values that are of interest to the study. Threshold selection can be compared to setting the maximum value on the Y axis in a graph, where the Y axis indicates values of the independent variable. If the Y axis is too short, the data points that exceed the maximum on the axis are cut off. As a result, we only know that these data points are higher than the maximum but we do not know what they are exactly and how they differ from one another. On the other hand, if the Y axis is too high, the graph data will appear compressed and the differences between the data points will look smaller. Similarly, if a heatmap’s upper threshold is set too low, many areas will be covered in red with no differentiation between the amount of attention each attracted. Conversely, setting a heatmap’s upper threshold too high will limit the range of colors (e.g., constricting it to yellow or orange as the maximum) and no areas will be covered in red. Regardless of what criteria have been selected, they should be explicitly communicated in the figure legend or caption (e.g., “red = 10+ fixations” or “red color indicates areas that accumulated 10 s or more of gaze time”). It is also useful to put this value in context by providing the average number of fixations each participant made on the stimulus or the average time each participant spent looking at it. The heatmaps presented in this paper were created based on data with the average of 42 fixations on the page per participant and consistent exposure time of 12 seconds. If heatmaps are generated for different experimental conditions or participant groups, their upper threshold definition should be identical, so the heatmaps can be compared. If the display settings are not the same, the differences between the heatmaps may be due to factors other than the data itself.

Fig. 6. Fixation count heatmaps based on the same data (n = 13; 0 – 12s; free-view task). On the left: red = 10+ fixations; on the right: red = 3+ fixations.

3.4 Modifying the Time Segment Sometimes it may be appropriate to present data based on a shorter time segment than the total time during which a stimulus was shown to the participants. This will obviously decrease the amount of data, thus reducing the size of red areas in all heatmap types mentioned in this paper except for the relative gaze duration heatmap (see Fig. 7). Therefore, if a heatmap presents data from a time segment that is shorter than the total viewing time, this needs to be specified in the figure legend or caption.

Informative or Misleading? Heatmaps Deconstructed

37

In addition to the time segment, it should also be clear what participants were trying to do when the data presented in the heatmap was being collected. All too often heatmaps are shown without any context of the task. Eye movements are very taskdependent [7] and participants trying to log in to a website will produce very different attention distribution than if they were trying to find a product. Even if there was no specific task, it should still be noted that the data was collected in a free-view situation, which is how the heatmaps included in this paper were obtained.

Fig. 7. Fixation count heatmaps of the same free-view task (n = 13). On the left: data from 0 to 6 seconds; on the right: data from 0 to 12 seconds.

4 Heatmap Usage 4.1 Common Mistakes There are a few common mistakes when it comes to using heatmaps. The biggest of them is a belief that heatmaps are appropriate for just about any user experience study and for any research question. Sometimes creating heatmaps even becomes the objective of the study (e.g., “we just wanted to see where people look”). This popular “let’s-track-and-see-what-happens” approach has a limited value because of its lack of focus (i.e., the study design does not target specific questions) and frequent lack of proper data analysis. Researchers often draw conclusions or make recommendations based on the results of those studies, which is inappropriate. For example, we cannot say that the reason why an element did not get much attention was because of its suboptimal placement or insufficient size unless we have tested other conditions (i.e., alternative placements or sizes) and compared the data using appropriate statistics. Even if different conditions were tested, sometimes conclusions regarding differences between conditions are made just by looking at the heatmaps. However, heatmaps do not lend themselves to any systematic comparison. Without any data analysis, it is impossible to tell if there are real differences between heatmaps, even if the heatmaps appear to be different. 4.2 Proper Usage Examples Heatmaps should be used in a purposeful way and only when they add value. They can serve as illustrations of participants’ viewing behavior and distribution of attention.

38

A. (Aga) Bojko

While they can communicate data, they cannot explain it or help analyze it. Therefore, heatmaps can rarely stand on their own. To maximize their usefulness and reduce ambiguity, heatmaps should accompany a quantitative analysis. One of our studies investigated a new standardized label template for prescription drug labels [8]. The goal was to determine the impact of the template on pharmacists’ drug selection speed and accuracy as compared to the existing label designs. The eye tracking measures included the number of fixations prior to target selection as an indicator of search efficiency, average fixation duration as a measure of information processing difficulty, and pupil diameter as a measure of cognitive workload. The results were presented in the form of statistical analyses but no heatmaps were included in the report. The study was a quantitative assessment of the effectiveness of the new labels, and heatmaps showing attention distribution were simply of no value. In another study, we evaluated a new homepage design for a professional organization against the original homepage [9]. Our objective was to identify which design was better and why based on a series of tasks during which participants attempted to locate the correct entry point on the homepage. Measures of search efficiency such as the number of fixations and the number of eye visits to the target prior to target selection were analyzed. The analysis was supplemented with heatmaps to show the distribution of attention on the page and to help account for any inefficiencies that occurred. For example, several tasks were more efficient using the new design which had a more centralized navigation. The heatmaps of the original design showed scattered fixations covering multiple navigation areas, while heatmaps of the new design revealed fixations focused mostly around the targets.

5 Guidelines for Using Heatmaps Even though heatmaps are very compelling and seemingly easy to understand, they should be used with caution and according to the following guidelines, summarized based on the discussion from the previous sections of this paper: A. B. C. D.

E. F. G. H.

I. J.

Generate heatmaps only if they add value to the research. Use heatmaps for data visualization instead of data analysis. Use heatmaps to support quantitative analysis rather than on their own. Understand the different heatmap types and only use the ones that represent measures which address your study objectives (e.g., when analyzing gaze time, use a gaze duration heatmap). Specify the type of data the heatmap is representing (e.g., fixation count or absolute fixation duration). Know the limitations of each heatmap type to avoid incorrect interpretation. When creating heatmaps, use fixation data rather than raw data. Provide fixation definition and keep it consistent for analyses and visualizations within a study (e.g., min fixation duration = 100 ms and max dispersion threshold = 0.5°). Provide the definition for the upper threshold of the heatmap color scale (e.g., red = 10+ fixations). Put the upper threshold value in context (e.g., average number of fixations on the stimulus per participant).

Informative or Misleading? Heatmaps Deconstructed

39

K. Specify the time segment based on which the heatmap was created (e.g., the first 10 seconds of exposure). L. Provide task context for each heatmap (e.g., data obtained from participants during the checkout task). M. Use the same heatmap settings (e.g., upper threshold and time segment) for conditions that you are comparing. N. If a paper or report does not provide important information about its heatmaps (e.g., type, fixation definition, upper threshold definition, time segment), ask the authors for clarification before making any assumptions.

6 Conclusion Blinded by the attractiveness and apparent intuitiveness of heatmaps, we often do not realize how much information in addition to the visualization itself is necessary to fully understand a heatmap and properly interpret the data it represents. In other words, the biggest danger involved in creating and reading heatmaps is that we are often unaware of what we do not know, and thus we do not look or ask for the missing information. This paper has exposed some of these gaps in our meta-knowledge in the hope to encourage more critical thinking about the usage of heatmaps.

References 1. Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises. In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, pp. 573–605. Elsevier Science, Amsterdam (2003) 2. Liversedge, S.P., Findlay, J.M.: Saccadic Eye Movements and Cognition. Trends in Cognitive Sciences 4, 6–14 (2000) 3. Tobii.: Tobii Studio 1.X User Manual (2008) 4. Duchowski, A.: Eye Tracking Methodology: Theory and Practice. Springer, Heidelberg (2003) 5. Salvucci, D.D., Goldberg, J.H.: Identifying Fixations and Saccades in Eye-Tracking Protocols. In: Proceedings of Eye Tracking Research and Applications Symposium (2000) 6. Castelhano, M.S., Rayner, K.: Eye Movements During Reading, Visual Search, Scene Perception: An Overview. In: Rayner, K., Shem, D., Bai, X., Yan, G. (eds.) Cognitive and Cultural Influences on Eye Movements, pp. 3–33. Psychology Press (2008) 7. Yarbus, A.L.: Eye Movements and Vision. Plenum Press (1967) 8. Bojko, A.: Measuring the Effects of Drug Label Design and Similarity on Pharmacists’ Performance. In: Tullis, T., Albert, W. (eds.) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, pp. 271–280. Morgan Kaufmann, San Francisco (2008) 9. Bojko, A.: Using Eye Tracking to Compare Web Page Designs: A Case Study. Journal of Usability Studies 1, 112–120 (2006)

Toward EEG Sensing of Imagined Speech Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan Department of Cognitive Sciences, UC Irvine, SSPB 3219, Irvine, CA 92697-5100 {mdzmura,sdeng,tlappas,sthorpe,r.srinivasan}@uci.edu

Abstract. Might EEG measured while one imagines words or sentences provide enough information for one to identify what is being thought? Analysis of EEG data from an experiment in which two syllables are spoken in imagination in one of three rhythms shows that information is present in EEG alpha, beta and theta bands. Envelopes are used to compute filters matched to a particular experimental condition; the filters' action on data from a particular trial lets one determine the experimental condition used for that trial with appreciably greater-than-chance performance. Informative spectral features within bands lead us to current work with EEG spectrograms. Keywords: EEG, imagined speech, covert speech, classification.

1 Introduction Dewan reported in 1967 [1] that he and several others could transmit letters of the alphabet using EEG recording; their ability to control voluntarily the amplitude of their alpha waves let them send letters using Morse code. Farwell and Donchin [2] used a second method for transmitting linguistic information by EEG; the P300 response to targets let them determine which of a sequence of displayed letters a person had in mind. Finally, Suppes and colleagues [3,4] reported that they were able to classify EEG responses to heard sentences and that there was information available through EEG concerning imagined speech. The work reported here verifies this latter result and develops it further, with the aim of using EEG brain waves to communicate imagined speech. Classification experiments show that brain-wave signatures of imagined speech lets one distinguish linguistic content, with varying degree of success. These signatures include differences in alpha-, beta- and theta-band activity. Our near-term goal is to use such signatures to design filters that let one distinguish linguistic elements in real time. One application is in further experiments which provide feedback to the thinker as to how recognizable a particular thought is, with the aim of training the thinker to produce brain waves which are more discernible. The focus within is on the results of a single experiment in which subjects produce in their imagination one of two syllables in one of three different rhythms. Analysis of the data using envelopes within alpha, beta and theta frequency bands, spectral features within single bands, and spectrograms in current work show that EEG provides considerable information on imagined speech in this experimental setting. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 40–48, 2009. © Springer-Verlag Berlin Heidelberg 2009

Toward EEG Sensing of Imagined Speech

41

2 Methods Four subjects participated in an experiment with six conditions determined factorially through combination of two syllables and three rhythms (see Figure 1). A single experimental session comprised 20 trials for each of the six conditions; conditions were presented in block-randomized order. Each subject participated in six such sessions for a total of 120 trials per condition. EEG was recorded using a 128 Channel Sensor Net (Electrical Geodesics) in combination with an amplifier and acquisition software (Advanced Neuro Technology). The EEG was sampled at 1024Hz and on-line average referenced. Subjects were instructed to keep their eyes open in the dimly-lit recording room and to avoid eye and other movement during the six seconds, following the cue, during which speech was imagined without any vocalization whatsoever.

Fig. 1. Timelines for the six conditions, labeled at left, in the imagined speech experiment. Durations are indicated in eighths of a second. The syllable /ba/ was imagined in one of three different rhythms in conditions 1-3; the syllable /ku/ was imagined in conditions 4-6. The condition for each trial was cued during an initial period of duration 4.5 sec (12/8 sec = 1.5 sec; 3 x 1.5 sec = 4.5 sec). During this initial period, subjects heard through Stax electrostatic earphones either a spoken "ba" or a spoken "ku" followed by a train of clicks (arrows) indicating the rhythm to be reproduced. Subjects were instructed to speak in their imagination the cued syllable, illustrated for condition 1 by {ba}, using the cued rhythm and tempo. Desired imagined syllable onset times reproduce the cued rhythm and are indicated by asterisks.

3 Analysis and Results Our aims were to classify single trials offline according to condition and discern condition signatures needed to create online filters. EEG waveform envelopes provide encouraging results when used to classify trials according to experimental condition.

42

M. D’Zmura et al.

These envelopes were computed for each electrode in the theta (3-8Hz), alpha (813Hz) and beta (13-18Hz) frequency bands and used to construct matched filters. These filters work well to classify trials according to condition, yet further results with spectra lead us to explore a finer-grained analysis using spectrograms. 3.1 Matched-Filter Classification Using Envelopes We used waveform envelopes to compute matched filters, one for each of the six conditions (see Figure 2). The inner product of each matched filter with a particular trial's envelope provides six numbers. The matched filter which gives rise to the largest inner product is the best match, and the condition to which the matched filter corresponds is declared the best guess as to the experimental condition for that trial. Offline preprocessing steps include • segment the EEG data to provide time-varying waveforms for each condition, electrode and trial; • remove from further consideration 18 electrodes most sensitive to electromyographic artifact: those with the lowest positions about the head and spanning locations close to the eyes, low on the temple, at or below the ear, and at or below the external occipital protuberance; • remove the mean and linear trend from each segmented waveform; • low-pass filter the detrended, segmented, waveforms to remove 60Hz line noise; • use thresholds to identify and remove from further consideration filtered waveforms likely contaminated with electromyographic artifact. Alpha-, beta- and theta-band activity were computed for each electrode and trial using band-pass elliptic filters. These band-pass filtered waveforms (we[t] and bottom right plot in Fig. 2) were Hilbert-transformed to provide corresponding envelopes (ve[t] and middle right plot). The envelopes serve as input to the matched-filter classification and as data used to construct the matched filters. An electrode's average envelopes, found by averaging across trials for each of the six conditions, serve as matched filters with one further step: for each electrode, pseudoinvert the six conditions' average envelopes to provide six filters (Fec[t] and top right plot). The inversion of the average envelopes provides filters which return an inner product of one for the corresponding condition's average envelope and an inner product of zero for all other condition's average envelopes. Information may be integrated across electrodes by summing the measures pe,c across electrodes and determining the maximum of the resulting six numbers. Information may be integrated across bands by weighing individual response measures for the three bands by band reliability estimates or by a voting procedure. Classification results shown in Table 1 show that the beta band (13-18Hz) is, for all but one subject, the most informative frequency band; theta (3-8Hz) and alpha (813Hz) are comparable for all subjects but S1. The per-condition classification performance is shown for subject S2 in Fig. 3. Rows refer to a trial's actual condition, while columns refer to the condition of the matched filter for which the maximal response was obtained. Darker values indicate higher percentages of trials. These dark values describe diagonals, which indicates a match between a trial's actual condition and the condition with matched filter providing the greatest response.

Toward EEG Sensing of Imagined Speech

43

Fig. 2. Preprocessed and band-pass-filtered waveform we[t], recorded by electrode e, is Hilberttransformed to provide envelope ve[t]. Shown at bottom right is the preprocessed waveform from an electrode with mid-parietal location from subject S1 for a trial during which {ba} is spoken in imagination at times 1.5, 3.0 and 4.5 sec (condition 1). Shown at middle right is the corresponding envelope. The inner product <, > of this envelope with each matched filter Fe,c[t], one filter per condition per electrode, provides six numbers pe,c. These six numbers measure how well the particular trial's envelope matches the filter for each condition. The maximum of these six numbers is used to determine the most likely condition c˜ e . Shown at top right is a matched filter for condition c = 1.

The distributions of classification performance across the scalp are similar across the four subjects; those for subject S4 are shown in Figure 4. The most informative electrodes lie largely near the top of the head (vertex) where electromyographic artifacts have their least influence. These distributions of information differ significantly from the distributions of envelope amplitude, which follow well-known patterns for alpha activity (parietal) and theta activity (frontal).

44

M. D’Zmura et al.

Table 1. Classification performance using matched filters for envelopes in three frequency bands. The fraction of correctly-classified trials (720 trials per subject, identified in the left column) is indicated. The chance performance level in this classification among six conditions is 1/6 (0.17).

S1 S2 S3 S4

alpha 0.38 0.63 0.44 0.64

beta 0.80 0.87 0.68 0.62

theta 0.63 0.59 0.46 0.59

Fig. 3. Classification matrices for subject S2. Values along the diagonals indicate correct classification, while non-white values off the diagonal indicate errors. The black gray-value in the middle panel (beta-band) for actual condition 1 and matched filter condition 1 (top left square) represents 91% of the trials; lighter shades indicate smaller values (through white indicating 0%). Perfect performance would be indicated by all off-diagonal entries set to white.

Fig. 4. Distributions of classification performance across the scalp for alpha (α), beta (β) and theta (θ) bands for subject S4. Darker values indicate the positions of electrodes providing better classification performance.

3.2 Spectral Features

Analysis of trials’ power spectral densities shows that differences in power within single frequency bands can provide information concerning trial condition. Spectral

Toward EEG Sensing of Imagined Speech

45

differences within a single band are completely invisible to the previous analysis, which grouped together activity at all frequencies within a single band. Data for subjects S1 and S2 show that, for condition 3, there is a peak in the power spectrum in the range 5-6Hz, relative to baseline power measured at 3.5-4.5Hz. This peak is largely absent for condition 1. This difference between the average power spectra for conditions 3 and 1 is localized to electrodes with front central locations on the scalp (see Fig. 5, leftmost panels). This spectral difference alone, within the theta band, provides ~75% correct two-way classification between these two conditions. The peak and the localized spatial distribution do not exist for subjects S3 and S4 (see Fig. 5, rightmost panels).

Fig. 5. Difference between condition 3 and condition 1 theta sub-band spectral power localized primarily to electrodes with front, center locations for subjects S1 and S2; this pattern is not evident for S3 and S4

3.3 Spectrographic Analysis

The spectral data suggest several things. The first is that one would do well to represent EEG data spectrographically: record power as a fine-grained function of both frequency and time. The second and third are more cautionary. There is a tremendous amount of trial-by-trial variability in EEG recordings. Adding channels, like further frequency bands of narrow range as in spectrograms, may have the effect of masking signals more effectively. Furthermore, fine subdivision of the frequency domain may reveal individual differences (as in Fig. 5), so that averages across subjects become less meaningful. These caveats notwithstanding, our current work focuses on spectrographic means of identifying imagined speech.

4 Discussion The syllables /ba/ and /ku/ have little semantic content in and of themselves, so that differences in EEG records underlying classification performance are unlikely to reflect semantic contributions to imagined speech production. Furthermore, each experimental trial starts with a period during which the cue to the condition is presented; cue recognition and response planning occur during the cue period and are likely absent during the following period in which EEG recording is made of imagined speech. The EEG recordings analyzed here concern most likely immediate aspects of imagined speech production based on readout from working memory.

46

M. D’Zmura et al.

EEG electrodes record not only signals from brain cortex but also electromyographic activity. Common sources of the latter are muscles responsible for eye and head movements and other muscles near recording electrodes, like those in the temples and neck. Electrodes atop or close to such muscles are most sensitive to electromyographic activity. The most flagrant such offenders were excluded from analysis in the work reported here. Furthermore, recordings for which EEG signals (detrended and filtered to excluded 60Hz line noise) exceeded 30 μV in absolute value were excluded from consideration. This threshold eliminated many (but not all) recordings contaminated with artifact. The most informative electrodes are those positioned on the scalp near the top of the head (Fig. 3), and this suggests that the information is due more to cortical than electromyographic activity. Subjects were instructed to think the imagined speech without any vocal or subvocal activity: without moving any muscles involved in producing overt speech. This contrasts with what is the most successful method yet for recognizing speech produced without any acoustic signal: surface electromyographic (SEMG) recordings from face and throat muscles. Substantial progress has been made in automatic speech recognition from surface EMG recordings of vocal tract articulators made during both spoken speech [5-10] and subvocalized speech [11-17]. Further non-acoustic methods for extracting imagined speech information potentially include magnetoencephalography (MEG) and invasive methods, like electrocorticography (ECoG), which involve implanted electrodes. While there are no published positive results yet for classification of imagined speech using MEG, its high temporal resolution and moderate spatial resolution (comparable to high-density EEG) suggest its value, as does success in classifying heard speech [18-21]. MEG is most sensitive to cortical signals from surfaces perpendicular to the scalp (in sulci) while EEG is most sensitive to signals from parallel surfaces (gyri) [22], so that the two methods are complementary. While the use of electrodes implanted in the brain is limited to small clinical populations for which these electrodes are indicated in combination with neurosurgery, it can provide very useful information concerning imagined speech [23]. Functional magnetic resonance imaging (fMRI) studies have contributed significantly to our neuroscientific knowledge of language perception and production [24-27]. Yet the poor temporal resolution of fMRI and of functional near-infrared imaging (fNIR) make these unlikely candidates for speech recognition. EEG is a non-invasive, non-injurious method for probing cortical activity which has a high temporal resolution, moderate spatial resolution (~2cm), a relatively low cost and which is increasingly portable. We feel that its greatest contribution as a braincomputer interface is likely to be found in circumstances where a human subject is trained to produce activity discernible by EEG [28]. Such training can reduce the variability associated with imagined speech production and also help one emphasize aspects of imagined speech which are most easily discerned by EEG. Our next step is to close the loop by providing real-time feedback concerning imagined speech signals.

Acknowledgements We thank David Poeppel for suggesting the value of an experiment like this. This work was supported by ARO 54228-LS-MUR.

Toward EEG Sensing of Imagined Speech

47

References 1. Dewan, E.M.: Occipital alpha rhythm eye position and lens accommodation. Nature 214, 975–977 (1967) 2. Farwell, L.A., Donchin, D.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogy and Clinical Neurophysiology 70, 510–523 (1988) 3. Suppes, P., Lu, Z.-L., Han, B.: Brain wave recognition of words. Proceedings of the National Academy of Science USA 94, 14965–14969 (1997) 4. Suppes, P., Han, B., Lu, Z.-L.: Brain wave recognition of sentences. Proceedings of the National Academy of Science USA 95, 15861–15866 (1998) 5. Morse, M.S., O’Brien, E.M.: Research summary of a scheme to ascertain the availability of speech information in the myoelectric signals of neck and head muscles using surface electrodes. Computers in Biology and Medicine 16, 399–410 (1986) 6. Chan, A.D.C., Englehart, K., Hudgins, B., Lovely, D.F.: Hidden Markov model classification of myoelectric signals in speech. IEEE Engineering in Medicine and Biology 21, 143– 146 (2002a) 7. Chan, A.D.C., Englehart, K., Hudgins, B., Lovely, D.F.: Multiexpert automatic speech recognition using acoustic and myoelectric signals. IEEE Transactions on Biomedical Engineering 53, 676–685 (2002b) 8. Bu, N., Tsuji, T., Arita, J., Ohga, M.: Phoneme classification for speech synthesizer using differential EMG signals between muscles. In: Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pp. 5962–5966 (2005) 9. Jou, S.-C., Maier-Hein, L., Schultz, T., Waibel, A.: Articulatory feature classification using surface electromyography. In: Acoustics, Speech and Signal Processing, ICASSP 2006 Proceedings, pp. I–605–I–608 (2006) 10. Jou, S.-C., Schultz, T., Walliczek, M., Kraft, F., Waibel, A.: Towards continuous speech recognition using surface electromyography. In: Interspeech 2006 – ICSLP, pp. 573–576 (2006) 11. Jorgensen, C., Lee, D.D., Agabon, S.: Sub-auditory speech recognition based on EMG signals. In: Proceedings of the International Joint Conference on Neural Networks, Portland Oregon (July 2003) 12. Betts, B.J., Jorgensen, C.: Small vocabulary recognition using surface electromyography in an acoustically harsh environment. NASA/TM-2005-213471 (2005) 13. Maier-Hein, L.: Speech recognition using surface electromyography. Diplomarbeit, Universität Karlsruhe (2005) 14. Maier-Hein, L., Metze, F., Schultz, T., Waibel, A.: Session independent non-audible speech recognition using surface electromyography. In: 2005 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 331–336 (2005) 15. Jorgensen, C., Binsted, K.: Web browser control using EMG based sub vocal speech recognition. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS 2005), pp. 294–301 (2005) 16. Binsted, K., Jorgensen, C.: Sub-auditory speech recognition. In: HICSS (2006) 17. Walliczek, M., Kraft, F., Jou, S.-C., Schultz, T., Waibel, A.: Sub-word unit based nonaudible speech recognition using surface electromyography. In: Interspeech 2006 - ICSLP, pp. 1487–1490 (2006) 18. Numminen, J., Curio, G.: Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex. Neuroscience Letters 272(1), 29–32 (1999)

48

M. D’Zmura et al.

19. Ahissar, E., Nagarajan, S., Ahissar, M., Protopapas, A., Mahncke, H., Merzenich, M.M.: Speech comprehension is correlated with temporal response patterns recorded from auditory cortex. Proceedings of the National Academy of Sciences 98, 13367–13372 (2001) 20. Houde, J.F., Nagarajan, S.S., Sekihara, K., Merzenich, M.M.: Modulation of the auditory cortex during speech: an MEG study. Journal of Cognitive Neuroscience 15, 1125–1138 (2002) 21. Luo, H., Poeppel, D.: Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54, 1001–1010 (2007) 22. Nunez, P.L., Srinivasan, R.: Electric Fields of the Brain: The Neurophysics of EEG, 2nd edn. Oxford University Press, New York (2006) 23. Barras, C.: Brain implant helps stroke victim speak again. New Scientist (July 2008) 24. Indefrey, P., Levelt, W.J.M.: The spatial and temporal signatures of word production components. Cognition 92, 101–144 (2004) 25. Hickok, G., Poeppel, D.: Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92, 67–99 (2004) 26. Hickok, G., Poeppel, D.: The cortical organization of speech processing. Nature Reviews Neuroscience 8, 393–402 (2007) 27. Poeppel, D., Idsardi, W.M., van Wassenhove, V.: Speech perception at the interface of neurobiology and linguistics. Philosophical Transactions of the Royal Society B 363, 1071–1086 (2007) 28. Fetz, E.: Volitional control of neural activity: implications for brain-computer interfaces. Journal of Physiology 579, 571–579 (2007)

Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User Ying Gao, Armando Barreto, and Malek Adjouadi Department of Electrical and Computer Engineering Florida International University Miami, FL 33174 USA {ygao002,barretoa,adjouadi}@fiu.edu

Abstract. The pupil diameter (PD) has been found to respond to cognitive and emotional processes. However, the pupillary light reflex (PLR), is known to be the dominant factor in determining pupil size. In this paper, we attempt to minimize the PLR-driven component in the measured PD signal, through an ∞ Adaptive Interference Canceller (AIC), with the H time-varying (HITV) adaptive algorithm, so that the output of the AIC, the Modified Pupil Diameter (MPD), can be used as an indication of the pupillary affective response (PAR) after some post-processing. The results of this study confirm that the AIC with the HITV adaptive algorithm is able to minimize the PD changes caused by PLR to an acceptable level, to facilitate the affective assessment of a computer user through the resulting MPD signal. Keywords: Pupil diameter (PD), Pupillary light reflex (PLR), Pupillary affec∞ tive response (PAR), Adaptive Interference Canceller (AIC), H time-varying (HITV) adaptive algorithm.

1 Introduction Affective Computing, defined as “computing that relates to, arises from, or deliberately influences emotions” by Picard in 1997 [1] requires computers to have the ability to remain aware of their users’ affective states. Therefore, the affective assessment of the computer user has been considered as one of the key challenges to overcome for improvement of the relationship between human and computers. This challenge has been addressed, among other approaches, through the processing of a variety of physiological signals from the computer users, such as the Galvanic Skin Response (GSR), the Blood Volume Pulse (BVP), etc. We are currently studying the possibility of analyzing pupil diameter (PD) variations in the computer user for the detection of affective changes. The human pupil is the variable-size opening in the iris of the eye, through which light passes to the retina. The size of pupil is known to be controlled by two opposing sets of muscles in the iris, the sphincter and dilator pupillae, which are governed by the sympathetic and parasympathetic divisions of the Autonomic Nervous System (ANS) [2]. Traditionally, light intensity has been considered as the major determinant of the pupillary response, through the Pupillary Light Reflex (PLR). However, other J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 49–58, 2009. © Springer-Verlag Berlin Heidelberg 2009

50

Y. Gao, A. Barreto, and M. Adjouadi

psychological factors controlling pupil size, such as emotional processes have recently been investigated. For example, in 2003, Partala and Surakka found, using auditory emotional stimulation, that the pupil size variation can also be seen as an indication of affective processing during human-computer interaction [3]. Therefore, in this paper, we focus our study on monitoring and processing the pupil diameter (PD) signal for the affective assessment of a computer user. To achieve the required separation of the PLR-driven component from PD changes due to Pupillary Affective Responses (PAR) we propose an Adaptive Interference Canceller (AIC), which is known to be able to remove an unwanted interference component z(k) that pollutes a measured signal s(k) using an independent measurement of the interference n(k) [4] (See Figure 1). The core portion of this AIC system is an Adaptive Transversal Filter (ATF), which performs the adaptive algorithm to implement the function of interference canceling. To ensure a successful noise removal from the AIC, it is important to select an adaptive algorithm that possesses good robustness properties. The H∞ adaptive algorithm, introduced in robust control theory, is an attempt at addressing this need, with features that are useful in safeguarding against the worst-case of model uncertainties and makes no assumption on the (statistical) nature of the signals [5].

s (k )

d (k )

d (k )

z (k )

n( k ) d (k )

n( k )

r ( k ) = n( k )

r (k )

∧

∧

y (k ) = z (k )

Σ

s ( k ) = e( k )

u (k ) = 0

Fig. 1. Adaptive Interference Canceller (AIC) Block Diagram

Thus, we propose to use an H∞ –based adaptive technique, namely the H∞ timevarying (HITV) algorithm, in the Adaptive Interference Canceller for the removal of the PLR-driven component from the pupil diameter variation. Our intent is to use the output of the AIC, the Modified Pupil Diameter (MPD), further refined by additional processing (median evaluation of a sliding window), to indicate the occurrence of affective changes (e.g., onset of stress) in the computer user.

2 Methods 2.1 AIC Overview A basic AIC block diagram is illustrated in Figure 1, above. The signal of interest, polluted with an uncorrelated noise signal, is transmitted over the top channel, and constitutes the primary input to the AIC. The bottom channel receives a signal, which

Monitoring and Processing of the PD Signal

51

is uncorrelated with the signal of interest, but correlated with the interference, constituting the reference input. The output of AIC is expected to provide a signal where the correlated interference has been removed. As described in the previous paragraph, the key equations describing the AIC are: Primary Input: Reference Input:

Output:

d (k ) = s (k ) + z (k )

(1)

r ( k ) = n( k ) + u ( k )

(2)

e(k ) = d (k ) − y (k ) = d (k ) − zˆ (k ) = sˆ(k )

(3)

where, 9 9 9 9

s(k) is the signal of interest (PD- driven by PAR); z(k) is the interference in the primary sensor (PD- driven by PLR); n(k) is the actual source of the interference (Illumination changes); u(k) is the measurement noise. (In this study, u(k) is assumed to be zero);

In the AIC system, the core element is an adaptive transversal filter (ATF), where the reference signal r(k) is processed to produce an output y (k ) = zˆ (k ) that is an approximation of z(k). The state space model of the ATF is given by [5]: w(k + 1) = w(k ) + Δw(k )

(4)

d (k ) = r (k ) w(k ) + v(k )

(5)

z (k ) = r (k ) w(k ) + υ (k )

(6)

T

T

v(k ) = s(k ) + υ (k )

(7)

In these equations, 9 9 9 9 9 9

w(k) = system state vector, is the ATF coefficient vector of size m × 1 (m is the order of ATF); d(k) = measurement sequence, is the observed pupil diameter(PD) signal. z(k) = sequence to be estimated Δw(k ) = process noise vector, represents the time variation of the ATF weights w(k). v(k) = measurement noise vector, includes s(k) (PD- driven by affective changes) and model uncertainties υ (k ) .

r (k ) = [rk , r( k −1) ,......, r( k − m +1) ]T is the interference vector of size m ×1

As shown in Figure 1, for this study we define the primary input of the AIC as the recorded pupil diameter signal, which is composed by the signal of interest s(k) (PD- driven by PAR) and the interference z(k) (PD- driven by PLR). Although, the reference input comprises both the actual source of the interference n(k) (actual illumination changes) and the measurement noise u(k), under the assumption that the measurement noise is negligible, an independent measurement of illumination in the neighborhood of the eye of the subject (IL) is used as the reference input. We expect

52

Y. Gao, A. Barreto, and M. Adjouadi

the adaptive transversal filter (ATF) to emulate the transformation of the illumination variations to pupil diameter changes, which would convert the noise n(k) into a closeenough replication of the PLR-driven components of PD (the output y(k)). Therefore, the error, e(k) (Modified Pupil Diameter), would be the estimation of the desired signal s(k), i.e., the PD variations due exclusively to affective changes (PAR). 2.2 H∞ Time-Varying Adaptive Algorithm

In this study, the adaptive algorithm we applied to the Adaptive Interference Canceller system is the H∞ time-varying (HITV) adaptive algorithm, which aims to remove the noise from the recorded signal by adaptively adjusting the impulse response of the ATF. The robustness of this HITV algorithm is derived from its minimization of the maximum energy gain from the disturbances to the estimation errors with the following solutions [6]: P% −1 (k ) = P −1 (k ) − ε g−2 r(k )rT (k )

(8)

~

g (k ) =

P(k )r(k ) ~

1 + rT (k ) P(k )r(k)

(9)

wˆ (k + 1) = wˆ (k ) + g (k )( y (k ) − rT (k ) wˆ (k ))

(10)

P(k + 1) = [ P -1 (k ) + (1- ε g-2 )r (k )r T (k )]-1 + γ 0

(11)

P (0) = η I = Π 0 , ϒ 0 = ρ I

(12)

Here, g ( k ) is the gain factor; ε g , η and ρ are positive constants. Note that ρ reflects a priori knowledge of how rapidly the state vector w(k ) varies with time, and η reflects the a priori knowledge of how reliable the initial estimate available for the state vector w(0) is.

3 Experiments 3.1 Subjects

We have collected data from twenty-two volunteer students who have completed the protocol described below. All the subjects reported to have normal color vision and had experience using computers. This paper presents results achieved through the processing methods proposed on a subset of the subject pool recorded to date. 3.2 Task

In order to observe the response of pupil diameter changes to affective stimuli, the “Stroop Color-Word Interference Test”, implemented as a flash program [7], was used to elicit mild mental stress in the participating subjects during the experiment.

Monitoring and Processing of the PD Signal

Fig. 2. Sample of Stroop Test Interface

53

Fig. 3. Stimuli schedule of the Stroop Test

In the test, a word presented to the subject designates a color that may or not match the font of the word. The subjects are instructed automatically by the program to select (out of 5 possible choices) the screen button that indicates the font color of the word presented, by clicking on it (An example, showing the appearance of the test interface is shown in Figure 2). The stimuli schedule of the test in the experiment is shown in the Figure 3. The complete protocol is composed of three consecutive sections. In each section, there were four segments. 9 9 9 9

‘IS’ – the Introductory Segment to let the subject get used to the task environment; ’C’ – the Congruent segment of the Stroop Test, in which the subject was asked to click the on-screen button naming the font color of a word that correctly spelled the font color being displayed; ‘IC’ – the Incongruent segment of the Stroop Test in which the subject was asked to click the on-screen button naming the font color of a word that spelled the name of a different color; ‘RS’ – is a Resting Segment to let the subject relax for some time.

The incongruent Stroop segments (IC) were expected to elicit mild mental stress in the subject, according to previous research found in the psychophysiological literature [8]. In contrast, the congruent Stroop segments (C) were expected to allow the subject to continue in a relaxed state. The binary numbers shown in Figure 3 represent the demultiplexed output of the stimulus generator, which is used to insert the corresponding values (1, 2, 3) in the event channel of the PD data file, and the corresponding time marks in the illumination measurement data, recorded simultaneously. 3.3 Instruments

In this study, a desk-mounted eye tracking system (TOBII T60) was used to measure the pupil diameter signals from both eyes of the subjects at 60 samples/sec. The average ((L+R)/2) was recorded as the “Measured PD” signal, which corresponds to d(k), in Figure 1. Simultaneously, the illumination intensity level present in the area around the eyes of the subjects was recorded by a system built for that purpose. This system is composed by a BS500B0F photo-diode (Sharp), placed on the forehead of the subject and connected to an amplification circuit to provide an analog output voltage that is proportional (~0.0043 v/Lux) to the illumination intensity level [9]. The “Measured IL” signal, r(k), shown in Figure 1 was finally provided by sampling the analog

54

Y. Gao, A. Barreto, and M. Adjouadi

output of the luminance meter at the frequency of 360 Hz with a Data Acquisition (DAQ) System (PCI-DAS6023 board from Measurement Computing Co.) 3.4 Procedure

In our experiments, participants were asked to remain seated in front of the TOBII screen, interacting with the “Stroop Test” for about 30 minutes, while wearing a head band with the photo-diode. During that time, all the normal lights in the room were kept on, but an additional level of illumination provided by a desk lamp placed above the eye level of the subject was switched ON and OFF, alternatively, at intervals not previously known by the subject, using a dimmer. This was done to repeatedly introduce passages of high and low illumination in the experiment, which would trigger the pupillary light reflex.

4 Results Before its application to the adaptive interference canceller, the recorded pupil diameter signal is pre-processed by a blink-removal algorithm implemented in MATLAB, which is able to: 1. Detect the PD data interruptions due to eye blinks (identified as a value of “4” in the validity code provided by the TOBII system); 2. Compensate the missing data by linear interpolation. 3. Filter out the blink responses through a low pass, 512th order FIR filter designed for a cutoff frequency 0.13Hz. Figure 4 illustrates the stages of the blink-removal process on data collected from subject 13. MEASURED PD SIGNAL 6 4 2 0 -2

0

0.5

1

1.5

2

2.5

3 4

x 10 VALIDITY CODES FROM BOTH EYES 6 4 2 0 -2

0

0.5

1

1.5

2

2.5

3 4

x 10 PD SIGNAL AFTER BLINK COMPENSATION 6 4 2 0 -2

0

0.5

1

1.5

2

2.5

3 4

x 10 PD SIGNAL AFTER LOW PASS FILTER 6 4 2 0 -2

C1 0

0.5

IC1 1

C2 1.5

IC2 2

C3 2.5

IC3 3 4

x 10

Fig. 4. PD Signal Before & After Blink-Removal

Monitoring and Processing of the PD Signal

55

The PD signal obtained after blink-removal is applied to the AIC system as the primary input signal d(k), and the reference input r(k) is the simultaneously measured illumination intensity level signal, which is down-sampled from 360Hz to 60 Hz to share the same sampling rate with the PD signal. A MATLAB program was created to apply the HITV adaptive algorithms for the ATF with 120 weights and the parameter settings of η = 0.001 , ε g = 2.0 as well as a time-varying parameter ρ , changed according to the IL value to enable the AIC system to have a quicker response when there is a sudden increase in IL. The output of the AIC, the MPD, is shown in the bottom plot of Figure 5. This signal is further processed as described below to become a useful indicator of pupillary affective response in the subject. AIC PRIMARY INPUT: PD SIGNAL 6 4 2

0

0.5

1

1.5

2

2.5

3 4

x 10 AIC REFERENCE INPUT: IL SIGNAL 2 1.5 1

0

0.5

1

1.5

2

2.5

3 4

x 10 -6

1

AIC TIME-VARING PARAMETER: RHU

x 10

0.5 0

0

0.5

1

1.5

2

2.5

3 4

x 10 AIC OUTPUT: MPD SIGNAL 2 0 -2

C1 0

0.5

IC1 1

C2 1.5

IC2 2

C3 2.5

IC3 3 4

x 10

Fig. 5. AIC Implementation with HITV Algorithm

The affective state of “Stress” is expected to cause a dilation of the pupil [2]. Therefore, the negative portions of the MPD signal, are zeroed to isolate significant MPD increases, which indicate the emergence of stress in the subject. The result of applying this non-negative restriction to the MPD signal is shown in the bottom panel of Figure 6.

56

Y. Gao, A. Barreto, and M. Adjouadi

A sliding window with a width of 1200 samples is applied throughout the nonnegative MPD signal to calculate the median value within each window. The effect of this process on both the original PD signal and the non-negative MPD signal is compared in Figure 7. In this figure, it is clear that the significant increases that have been isolated in the processed MPD signal correlate closely with the occurrence of IC segments, regardless of the presence of higher illumination passages during segments IC2 and C3. It should also be noted that the same post-processing operated on the PD signal obtained directly from the eye gaze tracking system does not set apart the IC segments as clearly. Sliding Window Median Analysis of PD

AIC Output: MPD Signal 2

6

1

5

0

4

-1

0

0.5

1

1.5

2

2.5

3

3

0

0.5

1

1.5

2

2.5

3

4

4

x 10

x 10

Non Negative MPD Signal

Sliding Window Median Analysis of MPD

2

1

1

.5

0

0 C1

-1

0

0.5

IC1 1

C2 1.5

IC2

C3

2

IC3

2.5

C1 .5

3

0

0.5

IC1 1

C2

IC2

1.5

C3 2

2.5

4

x 10

Fig. 6. MPD Signal Non Negative Processing for Stress Indication

Fig. 7. Comparison of Sliding Window Median Analysis on PD & MPD

Signal Processing Results of Subject 11 3.5 3 2.5 MPD 2 Non Negtive1.5 MPD 1 Sliding 0.5 Window Median MPD 0 C1 -0.5

0

0.5

IC1 1

3 4

x 10

GSR

IC3

C2 1.5

IC2 2

C3 2.5

IC3 3

3.5 4

x 10

Fig. 8. Signal Processing Results of Subject 11

Monitoring and Processing of the PD Signal

57

A similar result is observed for data collected from Subject 11, as shown in Figure 8, where the 3 top signals (GSR, which was recorded simultaneously, MPD and Nonnegative MPD) have been shifted up to make the display clear. In this case, also, the most significant increases of the bottom trace, after the initial adaptation that precedes segment C1, occur during the incongruent segments (IC1, IC2 and IC3). This figure shows the appearance of Skin Conductivity Responses (SCRs) and overall elevation of the GSR signal during the incongruent segments. Furthermore, the bottom plot (the result of the proposed post-processing) also shows significant increases during IC1, IC2 and IC3, with minimal output on the other segments after the initial adaptation.

5 Conclusions This study implemented an H∞ time-varying adaptive algorithm in an Adaptive Interference Canceller to discount the influence of Pupillary Light Reflex from a measured PD signal, so that the output of the AIC, the MPD, with the application of a non negative constraint and sliding window median analysis, can be used as an indicator of Pupillary Affective Responses due to, for example, subject stress. This indicates that this approach might be useful for the affective assessment of a computer user, even in the presence of illumination changes. A comparison of this result with the outcome obtained by applying the same post-processing directly to the recorded pupil diameter signal, points out the advantage of the adaptive implementation, which output relatively more distinctive increases when the incongruent Stroop segments occurred. Data from other subjects who participated in our experiment have revealed similar results. These outcomes, therefore, encourage the continued exploration of adaptive processing algorithms applied to pupil diameter signals, as a non-invasive mechanism to achieve affective assessment of computer users in ordinary environments. Acknowledgements. This work was sponsored by NSF grants CNS-0520811, HRD0833093 and CNS-0426125. Ms. Ying Gao is the recipient of a Dissertation Year Fellowship from Florida International University.

References 1. Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997) 2. Beatty, J., Lucero-Wagoner, B.: The Pupillary System. In: Cacioppo, Tassinary, Berntson (eds.) Handbook of Psychophysiology, 2nd edn., pp. 142–162. Cambridge University Press, Cambridge (2000) 3. Partala, T., Surakka, V.: Pupil size variation as an indication of affective processing. Int. J. of Human-Computer Studies 59, 185–198 (2003) 4. Widrow, B., Stearns, S.D.: Adaptive signal processing, pp. 337–339. Prentice-Hall, Englewood Cliffs (1985) 5. Hassibj, B., Kailath, T.: H∞ adaptive filtering. In: Proc. Int. Conf. Acoustics, Speech, Signal Processing, pp. 945–952 (1995) 6. Puthusserypady, S.: H∞ Adaptive filters for eye blink artifact minimization from electroencephalogram. IEEE Signal Processing Letters 12, 816–819 (2005)

58

Y. Gao, A. Barreto, and M. Adjouadi

7. Zhai, J., Barreto, A.: Significance of Pupil Diameter Measurements for the Assessment of Affective State in Computer Users. Biomedical Sciences Instrumentation 42, 495–500 (2006) 8. Renaud, P., Blondin, J.-P.: The stress of Stroop performance: physiological and emotional responses to color-word interference, task pacing, and pacing speed. Int. Journal of Psychophysiology. 27, 87–97 (1997) 9. Todoroki, A., Hana, N.: Luminance change method for cure monitoring of GFRP. Key Engineering Materials, 321–323, 1316–1321 (2006)

Usability Evaluation by Monitoring Physiological and Other Data Simultaneously with a Time-Resolution of Only a Few Seconds Károly Hercegfi, Márton Pászti, Sarolta Tóvölgyi, and Lajos Izsó Budapest University of Technology and Economics, Department of Ergonomics and Psychology, Egry J. u. 1, 1111 Budapest, Hungary {hercegfi,tim,ebediyen,izsolajos}@erg.bme.hu

Abstract. This paper outlines the INTERFACE methodology developed by researchers of our department. It is based on the simultaneous assessment of Heart Period Variability (HPV), Skin Conductance (SC), and other data. The objective and significance of this paper are (1) showing its capability of identifying quality attributes of software elements with a time-resolution of only a few seconds and (2) presenting its practical applicability in the evaluation phase of a real software development process. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics carried out a contract-based applied research project for the GeneraliProvidencia Insurance Co. Ltd. The Company was in the process of further developing the software used in its customer centers, and our Department contracted to assess the user interface. Both analytical and empirical usability evaluation methods were applied. In this paper, we highlight the new experiences of the INTERFACE testing methodology. Keywords: Usability testing and evaluation, empirical methods, case study, Heart Period Variability (HPV), Skin Conductance (SC).

1 The Frame of the Project During the last decades, the Department of Ergonomics and Psychology at the Budapest University of Technology and Economics developed mutually useful relationships with some industrial partners successfully accomplishing various Research & Development (R&D) projects 4. The Genarali-Providencia Insurance Co. Ltd. as a member of the Generali Group is owned by the Generali PPF Holding that has 9 million customers in Central and Eastern Europe. From these 9 million customers, 1.2 million are given by the Genarali-Providencia Insurance Co. Ltd. Two kinds of different core systems are used by the Genarali-Providencia Insurance Co. Ltd. to keep their data: one for the life insurance (it is called SYNPAC), and another (it is called VIAS) for the non-life insurance, e.g. liability insurance, Casco insurance, etc. The management started to establish a new system that can visualize the data of both core systems. To carry it out, the IT department of the company started to develop a new frame system called Genesys. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 59–68, 2009. © Springer-Verlag Berlin Heidelberg 2009

60

K. Hercegfi et al.

This software is used by approximately 1500 users. From these 1500 users, 300 users use it daily. The main part of the users work in personal customer centers. One smaller part of the users work as call center operators. It is a smaller group of the users, however, the time pressure in their job underlines the importance of the usability factors. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics latched on to the software development to carry out the usability assessment. The main goal of our project was to collect problems of the user interface (UI). From these misses we made a list that we gave it to the developers. Developing solutions for the found problems can be a next project. Other goal of the Company was supporting a scientifically interesting research project transferring results of foundational researches to really applicable methodology in real software development process.

2 Applied Methods The preparation of the project started in November 2007. The contract was signed in March 2008. We got in touch with the developers at the beginning of April 2008. We started our project with studying the software and collecting data and facts. While we were studying it, the system was updated, so sometimes it was difficult to be up to date. In the 2nd part of April we started to observe the users of the software, and we also made interviews with them. We performed the observation and interviews at 3 different locations: in the biggest customer service of the firm, in another type, smaller customer service, and in a call center. We obtained some logfile data to establish objective initial data for the subsequent assessment. We applied analytical methods (usability inspection methods) in May and June 2008. The backbone of the analytical evaluation was a Guideline Review, supported by Cognitive Walkthrough elements. Also GOMS model-based analysis was carried out. The main part of the assessment was a carefully planned, deep empirical series of experiments applying the INTERFACE methodology described in the following section. The series of experiment were carried out in July 2008; the analysis of the collected records was performed in August 2008.

3 Description of Our Main Methodology: The INTERFACE Figure 1 shows the conceptual arrangement of the INTERFACE (INTegrated Evaluation and Research Facilities for Assessing Computer-users' Efficiency) workstation.

Usability Evaluation by Monitoring Physiological and Other Data

61

observable behavior current screen content

data collecting and

keystrokes and mouse clicks

processing frame system

physiological signals by ISAX

Fig. 1. Conceptual arrangement of the INTERFACE user interface testing workstation

The advantage of the methodology applied in our study lies in its capability of recording continuous on-line data characterizing the user’s current mental effort derived from Heart Period Variability (HPV) and the user’s emotional state indicated by Skin Conductance (SC) parameters simultaneously and synchronized with other characteristics of Human-Computer Interaction (HCI). This way, a very detailed picture can be obtained which serves as a reliable basis for the deeper understanding and interpretation of psychological mechanisms underlying HCI. Elementary steps of HCI, like the different mental actions of users followed by a series of keystrokes and mouse-clicks, are the basic and usually critical components of using software. These steps can be modeled and analyzed by experts, but empirical studies of real users’ interactions often highlight new HCI issues or give more objective results than expert analyses. One of the key aspects of the empirical methods is measuring mental effort as it is laid down e.g. in the earlier international standard of software product evaluation (ISO/IEC 9126:1991). Hence we need methods capable of monitoring users’ current mental effort during these elementary steps. To attain the above, a complex methodology was developed earlier at the Budapest University of Technology and Economics, by Prof. Lajos Izsó and his team [3, 4, 5]. This study presents an improved methodology and a new case study. The INTERFACE simultaneously investigates the following: • Users' observable actions and behavior − keystroke and mouse events; − video record of the current screen content; − video records of users’ behavior: (1) mimics, (2) posture and gestures. • Psycho-physiological parameters − Power spectrum of Heart Period Variability (HPV), regarded as an objective measure of current mental effort – we apply this signal successfully since more than 15 years [1, 2, 3, 4]; − Skin Conductance (SC) parameters, indicating mainly the emotional reactions – recently integrated into our system.

62

K. Hercegfi et al.

A number of studies [1, 2, 3, 5, 7, 8, 9, 10] have shown that an increase in mental effort causes a decrease in the mid-frequency (MF) peak of the HPV power spectrum. The main advantage of the assessment method of the spectral components integrated into our system over the previously existing HPV-based methods is that the MF component of HPV shows changes in mental effort in the time range of several seconds (as opposed to the earlier methods with a resolution of tens of seconds at the best). This feature was achieved by an appropriate windowing data processing technique, and application of an all-pole auto-regressive model with built-in recursive Akaike's Final Prediction Error criteria and a modified Burg’s algorithm. We watch the Alternating Current (AC) component of the Skin Conductance (SC) responses focusing mainly on the emotional aspects of the HCI, in addition to our well-tried approach of mental effort. An interesting series of experiments analyzing SC responses is finished by one of our colleagues [6]. It is a good example of the promising way to use data mining techniques in empirical usability studies. Participant, operator of the Computer used Camera to Display Standard call center with standard by the record the with IP headset regularly used in the participant – facial the phone call center. The Skin according to expressions. software of the Conductance (SC) electrodes can the standard Motorized face currently call tested center tracking, be seen on the left hand, the workstation of zoomable ECG electrodes are on the torso the call center

The ISAX equipment to record the physiological signals

Camera to record the body posture

Computer of the experimenter – during the session, online curves of the physiological signals, video images of the cameras and the editor window for the comments can be seen

Fig. 2. The experimental arrangement applied during the sessions of the INTERFACE usability testing installed on a standard workstation of the call center

Usability Evaluation by Monitoring Physiological and Other Data

63

4 Applying INTERFACE in the Current Series of Experiments The empirical sequence of experiments applying the previously introduced INTERFACE assessment was performed with the call center operators in the middle of July 2008. Due to the impersonality caused by the phone calls, on one hand the simulation was more authentic than it could be in the personal customer centers. On the other hand, because of the specialty of the call center, we could count on concentrated, quick problem-solving. What more, the usability problems are the most critical in the call centers due to the time pressure. The tasks occurred in the call center are usually not solved only by means of the Genesys, but these other software are not being examined in this project. However,

The screen just seen by the user Two cameras: Facial expression and body posture

Keyboard and mouse actions

Experimenter’s comments

Upper (blue) curve: AC component of the Skin Conductance (SC). The higher deviation means more emotional event.

Signals derived from the ECG, related to mental effort. Red RR curve in the middle: periods between the subsequent heart beats in ms. Last (green) profile curve: the MidFrequency (MF) power of the variability of the RR curve. Its low values mean significant mental effort; peaks mean relief, relaxation.

Fig. 3. The INTERFACE Viewer screen with a record of the empirical test of the Genesys software. As it can clearly be seen, currently the user makes significant mental effort – it is shown by the facial expression and gesture, and the low value of the last green profile curve of the Mid-Frequency (MF) power of the Heart Rate Variability (HRV) at the cross-hair.

64

K. Hercegfi et al.

we had to give controlled, simulated tasks to the users in order that they can solve them in the Genesys system. These simulated tasks helped us to be able to compare the 12 sessions. We used the version of the Genesys from the test server that is substantially equal to the version actually used in real. 12 real operators were involved as participants, each of whom we recorded a onehour-long session with. The quantity of data gained from these sessions is really significant according to the depth of the enquiry. Due to the real life situation the users participating the series of experiments, were more or less disturbed by their colleagues’ calls or talks. Since these are the employees’ real conditions, they have got used to them, and so they could work on typical, real workstations of their workplace. These advantages gave reason not to hold this usability testing in laboratories. Nevertheless, we chose a workstation that was located in the corner of the operators’ room, in order not to disturb the others. Behind this workstation, the team leader’s glass wall can be found, and so our staff could sit and make simulated phone calls from behind this pane. As mentioned before, during the recorded sessions, the users were given tasks from the real life, with quasi-real data, names, problems, questions, etc.; the main difference was that the customers were not real customers, but from our staff. Three ECG electrodes were put on the users’ torso and one on the left hand (in case of a right-handed person) for measuring Skin Conductance. After that the users put on their headsets and adjusted their seats. Figure 2 shows the experimental arrangement applied during the sessions of the INTERFACE usability testing installed on a standard workstation of the call center. Figure 3 shows the INTERFACE Viewer screen with a record of the empirical test of the Genesys software. At the beginning of the session, we asked the users to relax for two minutes. We told them the aim and the details of the assessment. We always emphasized that we are not willing to assess they themselves, but the Genesys software by means of their help. Relaxation was followed by two-minute mental effort: mental arithmetic. The result of the counting was not important; we only wanted to generate mental effort. These periods were planned for “calibrating” the physiological curves. After that, the hard work started. The first customer (from our staff) rang the phone and asked some questions. To answer these questions the operator had to use the software under testing. Later 4 more similar calls came. Each call contained 2 to 4 questions. One part of the questions were just to “warm up”, others were really difficult. The subtasks were based on the interviews, observations and expert analyses performed earlier. The last part of the INTERFACE session was an interview.

5 Validation As it was mentioned, the periods of relaxation and mental arithmetic were planned for “calibrating” the physiological curves. The curves shown in the upper part of the Figure 4 were recorded during session #11, the ones shown at the bottom were recorded during session #10.

Usability Evaluation by Monitoring Physiological and Other Data

relaxation

65

mental arithm.

relief

relaxation

mental arithm.

relief

Fig. 4. The typical pattern of the relaxation and mental arithmetic in cases of two participants

In both cases, the three curves are the blue curves of the AC of Skin Conductance (SC), the red RR curves (Heart Periods), and the green profile curves of the MidFrequency (MF) power of Heart Rate Variability (HPV). The blue curve of AC of SC is relatively smooth during both the relaxation and the mental arithmetic. During these sections, there are not any emotional peaks, and these two participants can be characterized as “stabile” type according to the typology of physiology. However, the beginnings and the ends of the sections are followed by peaks. During relaxation, the MF component of the HPV increases, then the red RR curve has zigzags, and the green profile curve is relatively high. (In case of perfect relaxation, the profile curve should be consistently high. However, this is not expected in this experimental situation. The curve can be considered as high, especially in comparison with the next section.) During mental arithmetic, the red RR curve gets smoother, and the green profile curve is significantly low. After the “calibration” tasks, the participants really relieve. During this short period of relief, the participants get more relaxed than during the conscious, intended relaxation: the green curves have their highest peaks here.

66

K. Hercegfi et al.

Mean of MF power of HPV [ms2]

These “calibration” tasks prove a validation of our method. The values of the MF power of the HPV were significantly higher during relaxation than during mental arithmetic. A non-parametric statistical method, the Wilcoxon Signed Ranks Test proves the difference (sig. 0.037 – Figure 5).

100,0

80,0

60,0

40,0

20,0

0,0 relaxation

mental arithm.

Fig. 5. Validation of measuring Mid-Frequency (MF) power of Heart Rate Variability (HPV) as an indicator of mental effort: the MF power of HPV was significantly higher during relaxation than during mental arithmetic (sig.0.037)

It is a significant difference, in spite of the non-perfect relaxation. However, the mental arithmetic task works better: the significance of the difference between the values of MF power of HPV during mental arithmetic and in general, during the whole software usage section is better: the Wilcoxon test results sig. 0.002. The values of the deviation of the AC component of Skin Conductance (SC) do not differ during the relaxation and the mental arithmetic significantly. As it was described earlier, this is the expected result. However, the deviations of the AC of SC during the relaxation and the mental arithmetic are significantly lower than in general, during the whole software usage section: the Wilcoxon tests result sig. 0.009 and sig. 0.017. After these results, we can say that the low value of the curve of the MF power of HPV really means mental effort, and the high deviation of AC of SC probably means higher emotions. Than, in the section of software usage, we look for moments with relatively high (and unwanted) mental efforts and high (and unwanted and not positive) emotions. This method gives us the key to find the problems of the UI.

6 A Sample UI Problem Identified by the INTERFACE Methodology Commercial sensitivities prevent publication of the most of the details of the particular software problems found. However, Figure 6 gives an illustration.

Usability Evaluation by Monitoring Physiological and Other Data

67

Fig. 6. The 11th participant during the first task of the second call. The mental effort can clearly be seen: it is shown by the facial expression and gesture, and the low value of the last green profile curve of the Mid-Frequency (MF) power of the Heart Rate Variability (HRV) at the cross-hair. In this case, the problem was caused by a bad design solution to ensure choosing a time period for a list view. This problem can also be found by analytical methods – but the INTERFACE highlighted this problem and gave objective evidence for it.

7 Conclusion Based on the results presented here as well as in related papers, it can be stated that the INTERFACE methodology in its present form is capable of identifying the relative weak points of the HCI. By this methodology and the related workstation, it was possible to study events occurring during the HCI in such detail and objectivity that would not have been possible using other methods presently known to us. The sophisticated Heart Period Variance (HPV) profile function integrated into the INTERFACE system is a powerful tool for monitoring events in such a narrow time frame that it can practically be considered as a time-continuous recording of relevant elementary events. Measuring the Skin Conductance (SC) is a new opportunity to modulate the results. Acknowledgements. The authors would like to thank Dr. Eszter Láng for the earlier developments, the Generali-Providencia Insurrance Co. Ltd. for the support of the deep research, and the participants of the series of experiments for their valuable contribution.

68

K. Hercegfi et al.

References 1. Chen, D., Vertegaal, R.: Using Mental Load for Managing Interruptions in Physiologically Attentive User Interfaces. In: Proc. CHI 2004, pp. 1513–1516. ACM Press, New York (2004) 2. Hercegfi, K., Kiss, O.E., Bali, K., Izsó, L.: INTERFACE: Assessment of HumanComputer Interaction by Monitoring Physiological and Other Data with a Time-Resolution of Only a Few Seconds. In: Proc. ECIS 2006, ECIS Standing Comm., pp. 2288–2299 (2006) 3. Izsó, L.: Developing Evaluation Methodologies for Human-computer Interaction. Delft University Press, Delft, The Netherlands (2001) 4. Izsó, L., Hercegfi, K.: HCI Group of the Department of Ergonomics and Psychology at the Budapest University of Technology and Economics. In: Ext. Abstracts CHI 2004, pp. 1077–1078. ACM Press, New York (2004) 5. Izsó, L., Láng, E.: Heart Period Variability as Mental Effort Monitor in Human Computer Interaction. Behaviour Information Technology 19(4), 297–306 (2000) 6. Laufer, L., Németh, B.: Predicting User Action from Skin Conductance. In: Proc. IUI 2008, pp. 357–360. ACM Press, New York (2008) 7. Lin, T., Imamiya, A.: Evaluating Usability Based on Multimodal Information: An Empirical Study. In: Proc. ICMI 2006, pp. 364–371. ACM Press, New York (2006) 8. Mulder, G., Mulder-Hajonides van der Meulen, W.R.E.H.: Mental Load and the Measurement of Heart Rate Variability. Ergonomics 16, 69–83 (1973) 9. Orsilia, R., Virtanen, M., Luukkaala, T., Tarvainen, M., Karjalainen, P., Viik, J., Savinainen, M., Nygard, C.-H.: Perceived Mental Stress and Reactions in Heart Rate Variability – A Pilot Study Among Employees of an Electronics Company. International Journal of Occupational Safety and Ergonomics (JOSE) 14(3), 275–283 (2008) 10. Rowe, D.W., Sibert, J., Irwin, D.: Heart Rate Variability: Indicator of User State as an Aid to Human-Computer Interaction. In: Proc. CHI 1998, pp. 18–23. ACM Press, New York (1998)

Study of Human Anxiety on the Internet Santosh Kumar Kalwar and Kari Heikkinen Lappeenranata University of Technology, Department of Information Technology, Lappeenranata, Finland [email protected], [email protected]

Abstract. In this paper a conceptualization of human anxiety on the Internet is introduced; it is built on the understanding of human behavior with regard to technology. The objective of this paper is to conceptualize the human anxiety. An integral part of understanding is an inter-disciplinary (psychology science, cognitive science, behavioral science and communication technology) literature review, of which and overall summary is presented. The understanding is conceptualized by designing, implementing and evaluating through a developed user study model. In this paper the preliminary result of utilizing the developed user study found seven particular anxiety areas which need further studies. Keywords: Human, study, anxiety, internet.

1 Introduction Twenty-seven years ago, Internet was like a small dog at the bottom of the application pile, fighting for recognition, today, vast numbers of people are using Web Services through the Internet. One of the fascinating things about the Internet is that it comes with outstanding tools for usage in regards to recreation, education, society and much more. Current estimated statistics have shown that more than 63 million humans use the Internet in homes and the importance of the Internet in daily activities is growing. There is little which cannot be accomplished from the comfort zone of home; paying bills, research work, buying, selling, updating, uploading, downloading, shopping and, most importantly, communicating with your family and friends half way across the world. Since the establishment of the WWW, the number of Internet users has grown from an estimated 16 million in 1995 to more than 500 million in 2002 – explosive growth to say the least [1]. The rather recent but enormous and rapidly progressing emergence of the Internet has not gone unnoticed by scholars working in many different fields of research [1]. The Internet has become both an object of study and a tool of research. A basic definition of the term host in the case of a computer network is a computer connected to the Internet. The number of hosts connected to the Internet has grown rapidly. In 1983, when the Transmission Control Protocol/Internet Protocol (TCP/IP) standard was first adopted there were only about 250 hosts connected to the Internet. By late 1993, figure had grown to two million. During 2005, the number of hosts on the Internet had increased to 317.6 million. In recent years, the number has J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 69–76, 2009. © Springer-Verlag Berlin Heidelberg 2009

70

S.K. Kalwar and K. Heikkinen

skyrocketed to 541 million [2]. The fact and digital data from various sources confirms that number of humans accessing the Internet is growing at a rapid pace. As the Internet evolves in terms of number of human online, it feels as if it is evolving as social community. Worldwide, the Internet population is growing at a rapid pace. The number of people getting access to information, learning, and going online is booming like never before. It should be remembered that as late 1988, only a few countries were connected to the Internet. According to the 2004 CIA World Fact book, over 50 countries have at least one million humans using the Internet [3]. Humans once found it difficult and expensive to communicate during the times of voice telephones but with the rapid technological development communication has improved drastically. The distance has been shortened from family, friends and from seeking information which can be seen as the replacement of very important daily interactions. Boundaries of time, distance and identity are broken by the transfer of simple applications like e-mail to the complex world of virtual communities. Together with the positive growth, its negative effects are growing too. According to the U.S. Department of Justice, the Internet is an anonymous and effective way for many predators to find and groom children for illegal activities [4]. The fear of using the internet is further amplified by social disintegration, psychological and cognitive implications. 1.1 Objective and Scope The main research question is, “how can we address the challenges such as Internet addiction, psychology and human computer interaction it is currently facing now?” To understand the objectives, some of the questions were researched in details. The paper aims to find answers to the following hypotheses: • • • • •

Do users shows increased or reduced anxiety level when using the Internet? What kinds of behaviors are shown when using the Internet? What is the role of the content? Finding types of the anxiety behaviors? How human process information at the internet interface?

The scope of the work includes design, implement and evaluate methods used in understanding the behavior of humans using the Internet technology. It has wide range of scope from the field of psychological perspective to cognitive science, behavioral science and communication technology. The scope is complex. However, To study human and their tasks and how to relate information to design style, human behavior theories, standards, procedures or guidelines in order to build an appropriate model of interaction with the help of some existing methods is investigated. 1.2 Internet Anxiety A lot has been written in past about the negative use of the Internet anxiety, Internet addiction and full dependence on the internet is welling up [5] [6] [7]. The service disruption because of network faults, software bugs, administrator mistakes and version upgrade could seem less tolerable. Millions of human around the world use Internet to search, inform, find, communicate, work and play. Internet should not be

Study of Human Anxiety on the Internet

71

only viewed as negative such as addiction and pathological nor should it be vilified. One must be aware of negative consequences of overuse of the Internet by understanding the behavior of themselves and from others. Four types of Internet anxiety was identified by Presno. C,, using qualitative study method [8]. These are: Internet terminology anxiety: anxiety produced by an introduction to a host of new vocabulary words and acronyms. Net search anxiety: anxiety produced by searching for information in a maze-like cyberspace. Internet time delay anxiety: anxiety produced by busy signals, time delays, and more and more people clogging the Internet. General fear of Internet failure: a generalized anxiety produced by fear that one will be unable to negotiate the Internet, or complete required work on the Internet. Additional three areas of the Internet anxiety from the qualitative study were found in this research. Experience anxiety: an anxiety produced by lack of concentration or focus. Usage anxiety: a generalized anxiety produced by excessive usage of the Internet. Environment and attraction anxiety: anxiety produced by content on the Internet. For example: interactive game, pornography and larger number of colorful applications. Table 1. Types of Anxiety recorded for Subject I and Subject II Types of anxiety Terminology Search Time delay General fear Experience Usage Environment & attraction

p1

p2

p3

x x x x x

x x x x x

p4 x x x x x x

p5

p1 x

x x

p2 x

p3 x x

x x

x

x x x

x x x

x x

x x x x

p4

p5

x x x

x x x x x

x

2 Inter-disciplinary Literature Review Sociability is important as well as usability of applications in the Internet. While usability is concerned with making sure that the application, software and system is consistent, predictable, and easy and satisfying to use, sociability and the social aspect of building and maintaining an online community focuses on processes and styles of social aspect in interaction that support human behavior on the internet to some extent. Research has shown certain social groups to be under-represented on the Internet [9] [10] not simply because of a lack of access, but more because of cognitive, motivational and affective factors [11]. Psychology therefore has an important role in advancing the understanding of why humans choose to use or not to choose the usage of the Internet [12]. There will always be an argument to model psychology with technology or technology with psychology however; combining psychology with technology will give rise to new technology called psycho technology. To understand the Internet technology in broader ways, interaction between human and the technology through the Human Computer Interaction becomes essential. Brain Computer

72

S.K. Kalwar and K. Heikkinen

Interface techniques are only studied in this research work, no practical implementation was carried out due to the limitation of resources. In BCI, skill developed by a human involves proper control of electro physiological signals which are easily adapted and modulated by the brain for better feedback. In understanding human anxiety, BCI techniques could be used for testing and analyzing different activities on the Internet. The ethnographic study of the Internet can be divided into two categories. First, user-based and second, content based. User-based analysis is about the investigation, examination and the study of humans using the Internet. Whereas, content based analysis is mainly focused on text. Humans are capable of providing reasons to support their points of view if asked a question such as “What color is my shirt?” and are capable of knowing without explicit Deduction or reasoning answers to questions like, “If I were you, I would hate myself”, whereas computers cannot function without specific programming instructions.

3 Methods, Design and Implementation The user study model used both qualitative and quantitative research methods. The qualitative research was conducted using interviews and observational analysis. The quantitative research was conducted in three iterations by using questionnaires and surveys. The Questions were analyzed with the help of different types of questionnaires being constructed. Intentionally, Participants were given very general types of question to answer. Two types of Questionnaires were used: Using the Internet and Using the Pen and Paper method. Task calculation was carried out by dividing the task into subtask. Two types of subject were categorized based on skill level (novice, intermediate and expert human): Subject I and Subject II. Usability Test recording form was used for each participant. This form was used to record both the verbal and non-verbal behaviors of humans using the Internet. The task was chosen which was based on the

Fig. 1. The user study model implemented to test behavior of humans using the Internet

Study of Human Anxiety on the Internet

73

Internet. The task contained three modules which were divided into task 1, task 2 and task 3 based on the level of difficulties. It was found from the task analysis that humans using the Internet took less than a second to complete all of these tasks. There is a division of the participant’s overt behavior into two major general categories: verbal and nonverbal. Verbal behavior includes anything a participant says and Nonverbal behaviors include various activities that the participants actually do. In non verbal behavior mostly facial expression such as smiling, looks of surprise, furrowing brow or showing body gestures such as leaning close to the screen, rubbing the head is shown by the participants these are facial expression, eye-tracking, pupil diameter, skin conductance among others.

4 Results and Discussion The goal of the study was to determine “how can we address the challenges such as Internet addiction, psychology and human computer interaction it is currently facing now?” In order to evaluate the main research question, the main research question was broken down into hypothesis. There were five different hypothesis formulated in the beginning of the research. Now, Let us try to discuss these hypotheses to see our method, design, evaluation and analysis of the research was supported (fully supported, partially supported and not supported) or not. H1: Do users shows increased or reduced anxiety level when using the Internet? Hypothesis 1 was fully supported. Human shows the sign of increased level of anxiety when using the Internet. It appears that, with any given task on the Internet number of anxiety increase in human. More number of participants said “yes” to five or more items from the QS 2, which indicates problematic Internet Usage. Using HADS and PHQ-9 it was observed that, one participant seemed to have a Case of higher depression scale. Therefore, in this particular case users showed increased anxiety level when using the Internet. H2: What kinds of behaviors are shown when using the Internet? Hypothesis 2 was fully supported. Literature review revealed that there are two types of behavior shown by human using the Internet: Verbal and Non verbal. During Observation of behavior for the participants, Most of the times humans were laughing, smiling, drumming their fingers on the table and looking aimlessly around. These behaviors pattern was verbal and non-verbal. These types of the behavior patterns were observed among narrowly selected group of the participants. H3: What is the role of the content? Hypothesis 3 was partially supported. Role of content could determine the predicted or unpredicted human behavior on the Internet. Such as addiction, anxiety and stress of using the Internet, since humans were successfully able to complete the task with ease, it could be predicted that-any sort of given task for humans is very easy to do on the Internet. Therefore, the role of content has principal impact on how human behaves on the Internet.

74

S.K. Kalwar and K. Heikkinen

H4: Finding types of the anxiety behaviors? Hypothesis 4 was fully supported. It was found that in this particular case there are seven main area of anxiety in the humans: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Using Observation methodology and Comparing two types of subjects: Subject I and Subject II, it concludes that- all the participants showed these above cited anxiety. H5: How human process information at the internet interface? Hypothesis 5 was partially supported. When human interacts with the internet interface, it appears that, everything that human senses such as sight, hearing, touch, smell and taste are processed as the information in the mind. This information could result in behavior such as verbal and non-verbal. Even if behavior initially disappears, it may partially return as undamaged parts of the brain reorganize their linkages. The human behavior in totality of processing information includes internal cognitive processes which can result in observable behavior. Processing of the information at the internet interface has the realistic approach such as thinking of mental processes as several railroad lines that all feed to same terminal. Two schools of thought have emerged which confirms with the hypothesis that, there are two types of the behavior while using the Internet: Verbal and Non-Verbal behavior. In more general terms human using the Internet can use the content available on the Internet in two different ways: Positive or Negative. The gestures or types of human behavior shown could lead to anxiety. Seven major types of anxiety were studied and validated: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, and general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Two types of behavior (verbal and non-verbal) were formulated from relevant literature study, empirical analysis and evaluation. The study of Brain Computer Interface concludes that signal could be send in human brain physically to control and observe behavior of humans, however the BCI techniques were not used in the study and without using BCI techniques, the study conducted discovered sample of humans showing increased level of anxiety when using the Internet. The task completion behaviors of humans were calculated. By the end of this discussion session, We can reach to the conclusion that, to reduce Internet anxiety, addiction and depression scale on the Internet, it is important to have many multicultural experiences, and control over own self behavior to accumulate successful experiences of behavior.

5 Conclusions and Future Work Taking the results and discussion to the logical conclusion it appears to the authors knowledge that, “Internet has lulled humans with the sense of dependency to greater extent”. Five types of hypothetical questions were answered in this study. These questions were: Do users shows increased or reduced anxiety level when using the Internet? What kinds of behaviors are shown when using the Internet? What is the role of the content? Finding types of the anxiety behaviors? , And How human process information at the internet interface? Seven major types of anxiety were studied

Study of Human Anxiety on the Internet

75

and validated: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, and general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Two types of behavior (verbal and non-verbal) were formulated from relevant literature study, empirical analysis and evaluation. The study of Brain Computer Interface concludes that signal could be send in human brain physically to control and observe behavior of humans, however the BCI techniques were not used in the study and without using BCI techniques, the study conducted discovered sample of humans showing increased level of anxiety when using the Internet. From the book by Sir Tony Hoare, The first passage in Communicating Sequential Processes reads, “Forget for a while about computers and computer programming, and think instead about objects in the world around us, which act and interact with us and with each other in accordance with some characteristic pattern of behavior”. The same idea is followed in the study of human anxiety on the Internet. • Large sample size, different demography structure and discovery of perfect user study model are needed, for larger impact and generalization. • In contrast to several findings of negative effect in the Internet addiction, anxiety and depression group, some positive effects could be determined in future by building the framework for future learning through Imagination, Investigation and Innovation. • In depth analysis and comparisons of the human brain and the network open System Interconnection (OSI) model could be performed. Despite the above limitations, undoubtedly the Internet has provided a collection of applications that is having a profound effect on mankind. Like the wheel, the plow, and steam power before it, it is a proving a truly differentiating tool in our world, changing the very ways in which we interact with each other. Progress is relatively easy to recognize if we follow technology exploration. A more challenge is to find technology we want to change ourselves and for our civilization. Understanding the human factors for the design and development of the technology, system and services to ensure successful and perfect application environment is major concern. The forms of anxiety identified suggest areas for future Internet development and research.

References 1. ISOC.org (2008), http://www.isoc.org/internet/history/brief.shtml 2. ISC.org. Millions of host on the internet (2008), https://www.isc.org/ 3. Central Intelligence Agency. The DigiWorld in the global economy. In: DigiWorld 2008 (2008), https://www.cia.gov/library/publications/ the-world-factbook/geos/xx.html; https://www.cia.gov/library/publications/ the-world-factbook/fields/2184.html 4. Golden, S.M.J.: Protecting children in the internet age (2008), http://www.senate.state.ny.us/sws/ Protecting%20Children%20in%20the%20Internet%20Age.pdf

76

S.K. Kalwar and K. Heikkinen

5. Kraut, R.E. (2008), http://www.cs.cmu.edu/~kraut/RKraut.site.files/articles/ Bessiere06-Internet-SocialResource-DepressionL.pdf 6. Chou, C.: Incidence and correlates of internet anxiety among high school teachers in taiwan. Computers in Human Behaviour 19, 731–749 (2003) 7. Skinner, B.F. (2008), http://www.bfskinner.org/aboutfoundation.html; http://www.bfskinner.org/f/Science_and_Human_Behavior.pdf 8. Presno, C.: Taking the byte out of internet anxiety: Instructional techniques that reduce computer/internet anxiety in the classroom. J. Educ. Comput. Res. 18, 147–161 (1998) 9. Jackson, L.A.: Social psychology and the digital divide. In: The 1999 Conference of the Society for Experimental Social (1999) 10. Sax, L.J., Ceja, M., Teranishi, R.T.: Technological preparedness among entering freshmen: the role of race, class and gender. Journal of Educational Computing Research 24, 363– 383 11. Jackson, L.A., Ervin, K.S., Gardner, P.D., Schmitt, N.: Gender and the internet: Women communicating and men searching. Sex Roles 44(5), 363–379 (2001) 12. Jackson, L.A., Ervin, K.S., Gardner, P.D., Schmitt, N.: The racial digital divide:Motivational, affective and cognitive correlates of internet use. Journal of Applied Social Psychology 31, 2019–2046 (2001)

The Research on Adaptive Process for Emotion Recognition by Using Time-Dependent Parameters of Autonomic Nervous Response Jonghwa Kim1, Mincheol Whang2, and Jincheol Woo1 1

Dept. of Computer Science, Sangmyung University, 7 Hongji-dong, Jongno Gu, Seoul, Korea {rmx2003,mcun}@naver.com 2 Dept. of Digital Media Technology, Sangmyung University, 7 Hongji-dong, Jongno Gu, Seoul, Korea [email protected]

Abstract. This study is to propose new method, called by TDP (time dependenet parameter) anlaysis, of physiological signal processing for emoiton recognition. TDP consised of delay, activation, half recovery and full recovery. TDP was determined from running average and normalization of physiological signals for finding tonic and phasic reponse according to emotion at entire time range from stimulating emotion to recovery. As the results of this study, TDP analysis and adaptive TDP analysis enhanced accuracy of emotion recognition in the comparison with tonic analysis. Speciallly, TDP analysis enhanced the accuracy while adaptive TDP analysis reduced the individual difference of the accuracy. Keywords: Physiological signal, GSR, ECG, PPG, Skin temperature, emotion recognition, accuracy.

1 Introduction Human emotion has been tried to be recognized by physiological measurements based on assumption that emotion was expressed by them[1]. Indentifying physiological signals to subjective emotion has been main issue of emotion recognition research. However, since the signals were vulnerable to be affected by noise, emotion has had recognition error. One of methods avoiding noise has been running average at narrow time interval of incoming signal. This signal process has been effective to treat noise in autonomic response having relatively low frequency of physiological signals [2-4]. Physiological signals have been analyzed into tonic level and phasic response which were important variables according to emotion response[5]. Data treatment should be considered to discriminate level and reaction from the signal. Physiological signal has contained information of both spontaneous and non-spontaneous response. Phasic response could provide non-spontaneous overall oscillation and specific continuous level according to stimulus[5]. Recently, normalization of stimulus state from reference state has been enabled to count both tonic level and phasic response for emotion recognition [6-8]. However, since response before and after stimulus according to time variation was not examined enough, non-specific phase response of physiological signals has less clarified to cause low accuracy of emotion recognition. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 77–84, 2009. © Springer-Verlag Berlin Heidelberg 2009

78

J. Kim, M. Whang, and J. Woo

Physiological response has been characterized by individual difference. Same emotion could be recognized by different regulation of physiological response. Therefore, the strategy or rule of emotion recognition should consider individual characteristics. Some findings showed that algorithm of emotion recognition set physiological variation automatically based on verification of subjective emotion and that this process enhances the accuracy of emotion recognition[9]. Therefore, emotion could be well recognized by considerations of noise reduction, discrimination between tonic level and phasic response of physiological signals, and individualization. Considering these issues, this study is to suggest new analysis methods of physiological signals, called by TDP (time dependent parameter) and to attempt to show effectiveness of emotion recognition when it is used.

2 Method 2.1 Research Purpose This study is to propose new analysis methods for physiological response and to prove that this method was effective for emotion recognition. This research was proceeded to compare the accuracies of emotion recognition from different methods which were tonic analysis, TDP analysis, and adaptive TDP analysis. 2.2 Definition of TDP TDP (Time dependant parameter) of physiological response was defined in this study as shown in Fig. 1. The delay was moment difference between stimulation and the activation. The activation meant time at peak form the beginning. The half recovery indicated the time at half peak and the full recovery did the time back to base state. The full recovery could be inferred from half recovery when it could not be measured. In this study, ECG (Electrocardiogram), RSP (Respiration), PPG (photoplethysmogram), GSR (Galvnic skin resistance) and SKT(Skin temperature) were processed to construct TDP curve as shown in Fig. 1.

Fig. 1. TDP (time dependent parameter) of physiological measurement

The Research on Adaptive Process for Emotion Recognition

79

2.3 Emotion Induction Emotion was tried to be induced by image pictures. The images were chosen by previous study [8]. 100 university students (33 females and 67 males) participated, and were not visually handicapped. Participants were asked to score subjective emotion after watching the images. Significant images of emotion induction were categorized into 2 dimensional emotion model[10]. 6 images were selected for evoking unpleasantness-arousal emotion and 10 for pleasantness-relaxation emotion as shown Fig. 2. The pleasantness-relaxation was called as the positive emotion, and the unpleasantness-arousal was called as the negative emotion in this study.

Fig. 2. The images evoking emotions

2.4 Experiment 4 university students (average 26.5 years old) participated in the experiments and were healthy with no problem of vision. 24 prepared images were presented to participants for inducing emotion. PPG, GSR, RSP and SKT were measured during presenting images. Experiment procedure was shown as fig. 3. Participant experienced first non-image state as a reference state for 30 seconds followed by presenting the image for 10 seconds. Then, non-image state called by neutral state was presented for 30 seconds. A procedure consisted of presenting 4 images and a participant experienced 6 times a procedure. The experiment took 190 seconds for a procedure and total experimental time was 1440 seconds for a participant.

Fig. 3. Experimental procedure

80

J. Kim, M. Whang, and J. Woo

3 Analysis 3.1 Data Acquisition Collected were 6 sets consisting of 4 physiological signals and subjective score of emotion. For purpose of analysis, data unit was set at 70 seconds including neutral state for 30 seconds, stimulation for 10 seconds and another neutral state for 30 seconds. Total 74 data (6 set * 4 pictures * 4 participants) were prepared for tonic analysis, TDP analysis, and adaptive TDP analysis. 3.2 Running Average for Noise Reduction and Normalization The running average has been effective to reduce noise [3]. In this research, the time interval for running averaging was determined by response rate of each signal. Time intervals of physiological signals were tried to be set for noise reduction. The determination was done by visual scanning for confirming signal stability. The time interval of GSR and SKT was set at 0.5 seconds while one of RSP was at 3 seconds. PPG was analyzed to convert HR by frequency analysis. Therefore, time intervals for PPG and HR were set 2 seconds. Running average was performed by sliding window method at the pre-determined time interval on all the physiological signals. Then, the stimulus state of physiological signal was normalized from the neutral state. This process was able to observe activation level (tonic level) of physiological signal. Normalization was determined by equation 1 and performed at every 0.5 seconds. Normalized state = (Stimulus state –Neutral state) / Neutral state

(1)

3.3 TDP Rule for Emotion Recognition TDP rule for emotion recognition was determined from previous study [8]. Visual stimulus from the prepared images induced their emotion as shown Fig 2. Then, physiological signals were analyzed into TDP and their threshold values of emotion recognition were set for constructing rule of emotion recognition as shown Table 1. Table 1 showed mean and standard deviation of physiological response time of emotion based on TDP definition. The range was defined by mean plus and minus standard deviation for recognizing respective neutral, positive and negative emotion. 3.4 Adaptive TDP Rule for Individualization Since physiological response on same emotion was individually different, TDP rule for emotion recognition needed to be developed for individualization. The process of adaptive TDP rule was shown as Fig. 4. First, emotion was recognized by nonadaptive TDP rule. Second, difference between measured subjective emotion and emotion by physiological signals was calculated. Then, if difference existed, TDP rule was adaptively set by individual input of subjective emotion. Otherwise, emotion recognition was performed. Through these processes, it became individual and accurate adaptively for a particular person.

The Research on Adaptive Process for Emotion Recognition

81

Table 1. The TDP rule (unit: seconds) t1(Delay) t2(Activation) t3(Half Recovery) Neg. Neu. Pos. Neg. Neu. Pos. Neg. Neu. Pos. 1.55 1.92 4.08 5.79 1.57 3.08 HR ±0.92 ±1.00 ±2.91 ±2.31 ±0.17 ±1.40 15.85 9.61 20.00 5.60 5.83 16.00 SKT ±10.00 ±0.40 ±9.21 ±5.09 ±0.74 ±6.57 3.67 1.00 RR ±1.80 ±0.05 2.65 1.56 3.69 0.79 4.51 GSR ±1.16 ±0.40 ±1.95 ±11.06 ±6.55 PPG 2.25 2.43 3.90 4.37 3.45 8.00 1.75 6.03 amp ±2.74 ±0.31 ±2.91 ±1.97 ±1.48 ±4.93 ±12.06 ±6.03

t4(Recovery) Neg. Neu. Pos. 2.85 1.44 1.92 ±1.20 ±0.24 ±1.68 -

-

-

3.33 ±2.50

1.00 ±0.05

-

-

-

-

-

-

Neg.: Negative, Neu.: Neutral, Pos.: Positive

Fig. 4. The process of adaptive TDP rule for individualization

4 Result Results showed the accuracies of emotion recognition from three different methods such as tonic analysis, TDP analysis and adaptive TDP analysis as shown table 2-5. The accuracy was determined by match rate between subjective emotion and emotion determined by physiological signals. Tonic response rule was determined from previous research of autonomic response pattern for emotion[5, 11]. If negative emotion was evoked, electrodermal and cardiovascular response were increased and thermal response was decreased and vice versa [11]. As shown in Table 2, the accuracy of emotion recognition was about 62% in negative emotion recognition and about 50% in positive emotion. In the same condition, the accuracy was enhanced up to 60% when TDP analysis was used as shown in Table 3. It was about 70% in recognition of both positive and negative. Therefore, the accuracy of emotion recognition by TDP analysis could observe responses more than one by tonic analysis. There were findings for individualization of emotion recognition as shown Table 5. Adaptive TDP rule made the accuracy enhanced little more. Interestingly, the participants having the accuracy lower than 70% increased up to 70% or more. Therefore, adaptive TDP analysis could be effective to increase accuracy of a particular person who was low. Figure 5 showed overall accuracies from three analyses. TDP analysis showed improvement of accuracy but adaptive TDP did not much. Since accuracy improved in individual showing low, overall accuracy did not contribute improvement.

82

J. Kim, M. Whang, and J. Woo Table 2. Accuracy of emotion recognition from tonic analysis

participants

Negative emotion Emotion by Subjective physiological emotion signals

Positive emotion Emotion by Subjective physiological emotion signals

Accuracy

A

8

5

10

6

61.1%

B

6

3

9

4

46.7%

C

7

4

8

4

53.3%

D

8

6

9

4

58.8%

Sum

29

18

36

18

Accuracy

62.1%

55.0%

50.0%

Table 3. Accuracy of emotion recognition by TDP rule

Participants

Negative emotion Emotion by Subjective physiological emotion signals

Positive emotion Emotion by Subjective physiological emotion signals

Accuracy

A

8

7

10

7

77.8%

B

6

3

9

6

60.0%

C

7

4

8

7

73.3%

D

8

6

9

5

64.7%

Sum

29

20

36

25

Accuracy

69.0%

69.0%

69.4%

Table 4. Accuracy of emotion recognition by adaptive TDP rule

Participants

Negative emotion Emotion by Subjective physiological emotion signals

Positive emotion Emotion by Subjective physiological emotion signals

Accuracy

A

8

6

10

7

72.2%

B

6

4

9

6

66.7%

C

7

4

8

7

73.3%

D

8

6

9

6

70.6%

Sum

29

20

36

26

Accuracy

69.0%

72.2%

70.1%

The Research on Adaptive Process for Emotion Recognition

83

Fig. 5. Accuracy comparison of three analyses such as tonic, TDP and adaptive TDP analysis

5 Conclusion TDP and adaptive TDP were newly proposed to analyze physiological response for emotion recognition. The methods were successful of enhance its accuracy. The adaptive TDP was effective to count individual difference of physiological response. Comparing accuracies of emotion recognition among tonic analysis, TDP analysis, and adaptive TDP analysis, this study concluded the followings. First of all, TDP analysis enhanced the accuracy more than tonic analysis. Accuracy by tonic analysis was an average of 55%, and one of TDP analysis was of 69%. Second, the accuracy of adaptive TDP analysis reduced the individual difference. In the case of analysis by using TDP rule, the accuracy of individual difference was between 60% and 77% while in the case of adaptive TDP rule, the accuracy of individual difference was between 66.7% and 73.3%. Results showed adaptive TDP could enhance accuracy that was relatively low but work less for a participant whose accuracy was already high. Therefore, TDP and adaptive TDP method may be useful of emotion recognition and observe detail significant response of physiological response.

References 1. Christine, L., titia, L., Fatma, N.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J. Appl. Signal Process., 1672–1687 (2004) 2. Allanson, J., Fairclough, S.H.: A research agenda for physiological computing. Interacting with Computers 16, 857–878 (2004) 3. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: First steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS, vol. 3068, pp. 36–48. Springer, Heidelberg (2004)

84

J. Kim, M. Whang, and J. Woo

4. Mandryk, R.L., Atkins, M.S.: A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of HumanComputer Studies 65, 329–347 (2007) 5. Boucsein, W.: Electrodermal Activity. Plenum Press, New York (1992) 6. Whang, M.: The emotional computer adaptive to human emotion. Phillips Research: Probing Experience 8, 209–219 (2008) 7. Whang, M., Lim, J., Boucsein, W.: Preparing computers for affective communication: a psychophysiological concept and preliminary results. The Journal of the Human Factors and Ergonomics 45, 623–634 (2003) 8. Kim, J., Whang, M., Kim, J., Woo, J.: The study on emotion recognition by timedependent parameters of autonomic nervous response. Korean Journal of the science of emotion & Sensibility 11, 637–644 (2008) 9. Fredrickson, B.L., Losada, M.F.: Positive Affect and the Complex Dynamics of Human Flourishing. American Psychologist 60, 678–686 (2005) 10. Russell, J.A.: A circumplex model of affect. Journal of personality and social psychology 39, 1161–1178 (1980) 11. Whang, M., Chang, G., Kim, S.: Research on Emotion Evaluation using Autonomic Response. Korean Journal of the science of emotion & Sensibility 7, 51–56 (2004)

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking Yu-Jin Kim, Jin Ah Bae, and Byeong Ho Jeon Dept. of Media Image Art & Technology, Kongju National University, Sinkwandong, Gongju, South Korea {yujinkim,jinabae,bhjeon}@kongju.ac.kr

Abstract. In this paper, we used eye tracking methodologies to investigate students’ visual perceptions of lectures using 3D real-time virtual studio technology. For measuring learning performance, we also gave the students multiple-choice paper quizzes at the end of the lectures. Three virtual lectures were created with different types of lecture materials (text-centered, image-centered, and lecturercentered) and 3D virtual sets (classroom, cyberspace, and lecture-theme space). Through analyzing students’ eye movements in viewing still and moving scenes of the virtual lectures, we found that layouts and movements of design elements on lecture screens significantly influenced students’ scanpaths and areas of interest (AOIs). Lecture material types affected learning performance while 3D virtual sets had no effect due to students’ inattention to the virtual background areas. We discuss effective ways to develop virtual lectures and design lecture screens for better presentation of lecture content and higher learning performance. Keywords: Virtual lectures, virtual studios, eye tracking, visual perception, learning performance, user-centered screen design.

1 Introduction Over the last two decades, advances in digital technologies brought about the emergence of new teaching pedagogies such as Computer-Assisted Instruction (CAI), Computer-Supported Collaborative Learning (CSCL), Web-Based Instruction (WBI), and others. Besides these pedagogies using computer and internet technologies, there has been growing interest in virtual studio technology, which composites images of performers shot in front of a chroma-key blue background and visual images created through real-time 3D computer graphics [1]. The virtual studio technology increases face-to-face interaction between a lecturer and students and helps the lecturer interact with various types of lecture materials in dynamic virtual environments [2]. In academia, however, little attention has been paid to the methods to create courses using virtual studios for delivering learner-centered lecture materials and enhancing learning performance. To fill this research niche, this study aims to analyze how students look at the screens of virtual lectures [3] using eye tracking technology. Eye tracking provides insight into students’ visual perceptions and cognitive processes by measuring their eye movements. We research how lecture material types and 3D virtual sets affect learning performance. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 85–94, 2009. © Springer-Verlag Berlin Heidelberg 2009

86

Y.-J. Kim, J.A. Bae, and B.H. Jeon

2 Related Work 2.1 Lectures Using Virtual Studio Technology Virtual education, by the use of cyberspace, eliminates the spatial and temporal limitations by removing the need for the lecturer and students to be present at an instructional site at a designated time [4]. Recently, the proliferation of information and communication technologies (ICTs) has spawned a boom in virtual education by creating a variety of virtual lecture content production techniques such as Flash, Web 3D, 3D real-time virtual studio, and others. Among these production technologies, several researchers have explored the effectiveness of the virtual studio application to lectures. In 2003, Morozov, Debelov, and Zhmulevskaya claimed that the application of virtual studio systems enhances the efficiency of instructional processes by providing lecturers more possibilities for diverse forms of education [5]. According to Brown and Cruickshank (2003), a virtual lecture can be more effective than classroom lecture by achieving significant savings in the costs of lecture delivery and student support, though a very large investment of time and effort is required by the lecturer [6]. Along with the previously mentioned research interests in the educational effects of the application of new technology, researchers have studied the factors affecting the delivery of lecture content in virtual education in order to increase the motivation and effects of learning. These factors include screen design [7, 8], content design [9], content suggestion style, and lecture material [10]. 2.2 Eye Tracking in Visual Information Processing Eye movements are driven both by properties of the visual world and processes in a person’s mind [11]. Therefore, many scholars have used eye tracking techniques in order to explore the relations between eye movements and visual information processing in diverse research fields. In particular, researchers in HCI have tracked eye movements to understand visual and display-based information processing, as well as to discover the factors that may impact the usability of system interfaces [12]. In fact, eye movement data provide more detailed and specific information about a user’s cognitive process in many different kinds of displays [13, 14]. They also help researchers determine the roots of some usability problems, and then come up with effective solutions. Meanwhile, the recent trend of eye-tracking studies in HCI shows that eye-tracking methods have been rapidly employed in conducting usability testing of websites. Laarni et al. (2003) tracked the eye movements of users while they were selecting and reading online news items on a small computer [15]. Cowen (2004) asked the subjects to perform two tasks on a website and measured their total fixation duration, the number of total fixations, average fixation duration, and distribution of fixations on the screen [13]. Through these measurements, he suggested effective ways to design webpages in terms of user interface. In 2007, Cutrell and Guan also measured web surfers' eye movements to observe their web search strategies [16]. Like the aforementioned studies, many researchers have analyzed the interface of and interaction with web content, mainly focusing on a sequence of still scenes of web pages.

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

87

However, there have been increasing interests in employing eye-tracking methods in moving visual images along with the proliferation of more dynamic web content, which is developed by adopting new types of web authoring tools and media, such as Flash, Web 3D, Virtual Studio, and others. 2.3 Eye Tracking Metrics The main measurements used in eye-tracking research are “fixations,” which are moments when eyes are relatively stationary, taking in or encoding information, and “saccades,” which are quick eye movements occurring between fixations [12]. From these basic measurements, a multitude of metrics are also derived [17]: (1) “gaze duration,” the cumulative duration and average spatial location of a series of consecutive fixations within an area of interest; (2) “area of interest (AOI),” the area of a display or visual environment that is of interest to the research or design team and, thus, defined by them (not by the participant); and (3) “scanpath,” a special arrangement of a sequence of fixations.

3 Hypotheses The focus of our study is the investigation of students’ visual perceptions in virtual lectures. In order to identify the patterns of students’ visual perceptions, we tracked their eye movements according to different layouts and movements of design elements on lecture screens. We also studied the effects of lecture material types and 3D virtual sets on students’ learning performance. In line with these purposes, we investigated the following research hypotheses: Hypotheses 1a. Students’ eye movements (AOIs, scanpaths) are affected by the layouts of design elements on lecture screens. Hypotheses 1b. Students’ eye movements (AOIs, scanpaths) are affected by the movements of design elements on lecture screens. Hypotheses 2a. Students’ learning performance varies according to lecture material types. Hypotheses 2b. Students’ learning performance is different according to 3D virtual set types.

4 Experiment 4.1 Virtual Lecture Prototypes In order to test the aforementioned hypotheses, we designed and conducted an experiment to analyze students’ eye movements and learning performance in the context of virtual studio-based lectures. As experimental material, three virtual lecture prototypes were created under the theme of “Media and Culture” with varying layouts and movements of lecture screen design elements (lecturer, lecture board, 3D virtual sets, text, images, and movies on the lecture board). The three prototypes (see Figure 1) were also produced with different types of lecture materials (text-centered, image-centered,

88

Y.-J. Kim, J.A. Bae, and B.H. Jeon

and lecturer-centered) and 3D virtual sets (classroom, lecture-theme space, and cyberspace): (1) Prototype 1 - text-centered lecture materials in classroom background sets; (2) Prototype 2 - lecturer-centered lecture materials in cyberspace background sets; and (3) Prototype 3 - image-centered lecture materials in lecture-theme space background sets. Figure 1 shows screen shots of these three prototypes. Three screen shots located in the first row display their 3D background sets. The other three shots in the second row illustrate how the same lecture content about the phenomena of mass communication is delivered by different types of lecture materials.

Fig. 1. Screen shots of three prototypes (left: Prototype 1, middle: Prototype 2, and right: Prototype 3)

After modeling and animating the above 3D virtual sets using 3D Studio Max 9.0., we combined these 3D background sources with the live-action footage of lecturers by a real-time chroma key technique of the "VS2000" system. In addition, we created animation buttons for controlling lecture screen design elements using the VS scripts of the "HotAction" program. 4.2 Participants Forty participants were selected for the experiment, but ten of them had problem with the calibration of the eye-tracking system. Calibration was a fine-tuning process for the experiment, so 30 participants successfully completed the experiment. The 30 subjects were divided into three groups of ten who participated in the experiment composed of three kinds of virtual lectures. The subjects were freshmen and sophomores of K University and had almost no previous knowledge of the content of the experiment lectures. They were between the ages of 18 and 23, and the gender ratio was 1:1. Twenty-four of the students had experienced distance lectures, and eight of them had been exposed to VR lectures. 4.3 Apparatus This experiment used the hardware "Eyegaze Development System" developed by LC Technologies, Inc. This device tracks the x-y coordinates of the participant’s

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

89

gazepoint on the computer screen automatically using the pupil-center-corneal reflection (PCCR) method, which directs infrared light into the eye. This system generates raw eyegaze point location data at a camera field rate of 60 Hz [18] and calculates three kinds of gazepoints: (1) a moving point that a participant looks at for only 1/60 second, then looking at other points; (2) a fixating point that a participant looks around in; and (3) a fixation completed point that a participant maintains the gaze in. This study used the software "EyeTrack v1.0" and “EyeTrackMovie v1.0” for analyzing eye movements in viewing still and moving scenes, respectively. These programs were developed by the Human Computer Interaction Laboratory (HCIL) of KAIST for the visual monitoring and analysis of eye-tracking results that were expressed in coordinates [19]. “EyeTrack” and “EyeTrackMovie” provide “Replay” and “Analysis.” "Fixation Mark" and "Color Variation" options enable the monitoring of eye-tracking data with a variety of graphic effects in chronological order. For an efficient analysis of eye movement patterns, it also provides five analysis options: (1) "shadow," which shows the area that subjects focused on and illustrates the rest in shades with gradation effects; (2) "frequent area," which marks a clear boundary between the area that the subjects focused on and the rest, highlighting the former; (3) "hotspot," which shows the amount of gaze (red meaning more gaze and green meaning less gaze); (4) "selected area," which shows the amount of gaze in a certain area and its duration in numbers; and (4) "priority order," which shows the duration of a gaze over time in circles. All participants’ eye movements could be monitored at once due to the function of the programs which can analyze multiple data simultaneously. The patterns of AOIs and scanpaths, described in Section 2.3, were discovered from these eye movements. 4.4 Experimental Design and Procedure The experiment was performed in four stages, and it took about an hour for a participant to complete the experiment: (1) questionnaires on demographics, media education level, and cyber learning experiences; (2) still scene eye-tracking with the captured scenes according to distinguishable types of screen layouts (Prototype 1: 6 scenes, Prototype 2: 9 scenes; and Prototype 3: 7 scenes); (3) moving scene eyetracking with three prototype movies; and (4) a quiz with multiple-choice questions. Eye movements were tracked twice for still and moving scenes of virtual lectures. Compared with the eye-tracking of moving scenes, the eye-tracking of still scenes allowed more detailed and accurate analysis of eye movements for the different screen layout types. On the other hand, the eye-tracking of the moving scenes enabled an additional analysis of the effects of the movements of screen layout elements on eye movements.

5 Results 5.1 AOIs According to the Layouts and Movements of Lecture Screen Design Elements Through analyzing students’ eye movements in viewing still and moving scenes of the virtual lectures, we found that layouts and the movement of lecture screen design

90

Y.-J. Kim, J.A. Bae, and B.H. Jeon

elements (the lecturer, lecture board, and 3D background set) significantly influenced students’ scanpaths and areas of interest (AOIs) [H1a and H1b]. In the case of AOIs, parts that students paid close attention to were similar in both still and moving scene tests. By pointing out the design elements that received the most attention on the sill scenes, it was found that students generally gazed into a lecturer’s face in all of the still scenes that feature a lecturer (Prototype 1: 4 scenes, Prototype 2: 5 scenes, and Prototype 3: 5 scenes) regardless of the lecturer’s size and position on the screen. Figure 2 shows the “hotspot” option analysis results of the three scenes (left: Prototype 3, middle: Prototype 2, and right: Prototype 3), where different sizes of a lecturer appear in different positions. We calculated the distribution of fixations on the face of the lecturer in the left scene of Figure 2 using the “selected area” option and found that 52.5% of the total fixations were on the lecturer’s face.

Fig. 2. “Hotspot” option analysis results (Note: The amounts of fixations gradually increased from green to red, and the reddest parts are marked in circle)

In addition, we found that the students’ points-of-regard are likely to stay not just on the lecture’s face, but also on the face of people illustrated in the lecture material images (see Figure 3). Another finding was that the student’s attention to the people whose profile or back views were shown to decrease compared with front view images.

Fig. 3. “Selected area” option analysis results (1. Front view (28%), 2. Profile view (11.9%), and 3. Back view (12.5%) of people in the image)

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

91

Fig. 4. Comparing visual attention to text areas (3. Body (40.1%) > 2. Subtitle (29.5%) > 1. Title (7.3%))

Along with the aforementioned visual attention to people’s faces in the still scenes, we also found a tendency in the moving scenes that students’ points-of regard are mostly directed at the lecturer’s face as well as the faces of people illustrated in the lecture material images. In addition, the analysis results also suggest that students’ eye movements and text sizes in lecture boards were inversely proportional. There were less eye movements and shorter gazes on larger texts. In fact, visual attention was generally highest in the body, subtitles, and title (see Figure 4). In other words, considering that titles take up more space than subtitles, but receive less visual attention, it would be effective to deliver new and important messages using subtitles. 5.2 Scanpaths According to the Layouts and Movements of Lecture Screen Design Elements In order to measure the student’s scanpaths and scanpath duration, we selected a student whose eye movement pattern can represent the visual attention of the ten students participating in the same still scene experiment and then analyzed his/her eye movement over time using the “fixation map” and “priority order” options. From the results, it turned out that the farther from the center the objects are positioned, the

Fig. 5. Scanpath tracking (left: from left to right, right: from top to bottom)

92

Y.-J. Kim, J.A. Bae, and B.H. Jeon

more likely they are to be out of the scanpath or stared at later. This analysis also verified the commonly accepted idea that people in Korean-speaking culture read words and sentences from left to right and from top to bottom (see Figure 5). On the other hand, students’ scanpaths showed different patterns in still and moving scene tests in terms of the starting point of the eye’s gaze. In the still scene test, the starting points of the students’ gazes were related to screen design element types, as well as their positions. For example, the majority of students firstly gazed into the lecturer’s face, regardless of its position, and then moved their gazes to other elements located around the center of the screens. In the moving scene test, the movement of screen design elements, rather than their type, dominantly influenced the starting points, as illustrated in Figure 6. We also found that students moved their gazes to empty parts where the next lecture content was expected to show up. Properly timed anticipation could enable the students to better perceive animated lecture content by preparing them for what will come next in the lecture.

Fig. 6. Starting points of students' gazes according to the lecture screen layout type

5.3 Students’ Learning Performance According to Lecture Materials and 3D Virtual Sets The average quiz scores for the three groups who respectively watched text-centered, image-centered, and lecturer-centered virtual lectures (12.9, 9.7, and 10.2, respectively) were significant at the alpha level of 0.05 (F=4.694). These results support H 2a, which suggests the possibility of a relationship between students’ learning performance and lecture material types. Students could easily understand and memorize lecture content in text-centered lecture material which presented the lecturer’s

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

93

explanations in clear print on the screen. While lecture material types affected learning performance, 3D virtual sets had no effect on learning performance due to students’ lack of attention to the background areas. Consequently, H 2b, which suggests a strong relationship between students’ learning performance and 3D virtual set type, was not supported. In fact, the effects of the virtual sets on the learning process are very small because students’ visual attention is usually drawn to the background only when they found some noticeable images or figures in the set.

6 Conclusions This study investigated students’ visual perceptions of virtual lectures by analyzing their AOIs and scanpaths in the still and moving scenes of the lectures. It also explored how to develop lecture materials and 3D background sets of virtual lectures for improving learning performance by testing the students’ comprehension with multiple-choice questions. Our research revealed that the following issues on virtual lecture production should be carefully considered for better presentation of lecture content: (1) the lecturer’s size and positions on lecture screens and people’s postures illustrated in lecture materials; (2) text’s sizes and positions in lecture materials; (3) movements of screen design elements (the lecturer, lecture board, animating parts on the lecture board); and (4) balanced layouts between the lecturer, lecture materials, and virtual background sets. Educators and material designers should also consider that text-centered lecture materials were the most effective for higher learning performance than imagecentered and lecturer-centered ones. Even though statistical significance was not reached between 3D background sets and learning performance, an effective background set design suited to the lecture objectives and content is necessary for enhancing students’ learning interest and motivation. Finally, it is hoped that this study will assist lecturers in understanding students’ visual information processing in virtual lectures, in designing virtual lecture content more effectively, and in improving the instructional effects of virtual lectures.

Acknowledgment We thank Kun-pyo Lee and Jung-mi Park for their comments and help in conducting our eye-tracking experiment.

References 1. Fukaya, T., Fujikake, H., Yamanouchi, Y., Mitsumine, H., Yagi, N., Inoue, S., Kikuchi, H.: An Effective Interaction Tool for Performance in the Virtual Studio-Invisible Light Projection System. NHK Science & Technical Research Laboratories, Japan (2003) 2. Dolgovesov, B.S., Morozov, B.B., Shevtsov, M.Y.: The System for Interactive Virtual Teaching Based on “Focus” Virtual Studio. In: International Conference Graphicon 2003 in Moscow, Russia (2003)

94

Y.-J. Kim, J.A. Bae, and B.H. Jeon

3. In this paper, a virtual lecture means instructional content that is created using virtual studio techniques for virtual education 4. Kuroda, K., Shanawez, H.D.: Strategies for Promoting Virtual Higher Education: General Considerations on Africa and Asia. Africa and Asian Studies 2(4), 565–575 (2003) 5. Morozov, B.S., Develov, B.B., Zhmulevskaya, M.Y.: The System for Interactive Virtual Teaching Based on “Focus” Virtual Studio. In: International Conference Graphicon, Moscow, Russia (2003), http://www.graphicon.ru/ 6. Brown, S., Cruickshank, I.: The Virtual Studio. International Journal of Art & Design Education 22(3), 281–288 (2003) 7. Kim, M.R.: Strategies on Screen Design of Learner-Centered Web-based Instructional Systems. Journal of Educational Technology 16(4), 51–65 (2008) (in Korean) 8. Shon, M., Chung, H.H.: An Analysis on the Learning Hindrance Factors in BlendedLearning Environment. Journal of Educational Information and Media 13(2), 251–276 (2007) (in Korean) 9. Ryu, I.: Factors Influencing the Effectiveness of Web-Based Distance Learning. Management Education Review 6(2), 7–27 (2003) (in Korean) 10. Kang, M.H., Gu, M.H., Moon, H.N., Jung, S.Y., Chung, J.Y., Kim, J.S.: Examining the Effects of Tutor Delivery Modes on Cognitive Presence and Learning Outcomes in Online Lectures. Journal of Educational Information and Media 13(4), 155–181 (2007) (in Korean) 11. Richardson, D.C., Spivey, M.J.: Eye-Tracking: Characteristics and Methods, 1-9. In: Wnek, G., Bowlin, G.: Encyclopedia of Biomaterials and Biomedical Engineering. Informa HealthCare (2004) 12. Poole, A., Ball, L.J.: Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future Prospects. In: Ghaoui, C. (ed.) Encyclopedia of Human Computer Interaction, Idea group (2004) 13. Cowen, L.: An eye movement analysis of web-page usability. Masters by Research in the Design and Evaluation of Advanced Interactive Systems (2001) 14. Lohse, G.L.: Consumer Eye Movement Patterns on Yellow Pages Advertising. Journal of Advertising 26(1), 61–73 (1997) 15. Laarni, J., Isotalus, P., Kojo, I., Kärkkäinen, L.: Reading News from a Pocket Computer: An Eye-Movement Study. In: Harris, C.D., Duffy, V., Smith, M.J., Stephanidis, C. (eds.) Human-Centered Computing: Cognitive, Social and Ergonomic Aspects. The Proceedings of HCI International 2003, Lawrence Erlbaum, Mahwah (2003) 16. Cutrell, E., Guan, Z.: What Are You Looking for?: an Eye-tracking Study of Information Usage in Web Search. In: Proceedings of the SIGCHI conference on Human factors in computing systems, San Jose, California, USA (2007) 17. Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human-computer Interaction and Usability Research: Ready to Deliver the Promises (Section commentary). In: Hyönä, J., Radach, R., Deubel, H. (eds.) The Mind’s Eyes: Cognitive and Applied Aspects of Eye Movements, Elsevier Science, Oxford (2003) 18. Eyegaze Development System Information, http://www.eyegaze.com 19. Park, J., Lee, K.: Eyetrack - Developing Eyegaze Analysis Visualization Software for Designers’ Use. In: KEER 2007, Sapporo, Japan, p. 10 (2007)

Toward Constructing an Electroencephalogram Measurement Method for Usability Evaluation Masaki Kimura*, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology {masaki-k,hideta-u,masao,matumoto}@is.nasit.jp

Abstract. This paper describes our pilot study toward constructing an electroencephalogram (EEG) measurement method for usability evaluation. The measurement method consists of two steps: (1) measuring EEGs of subjects for several tens of seconds after events or tasks that are targets to evaluate, and (2) analyzing how much components of the alpha and/or beta rhythm are contained in the measured EEGs. However, there only exists an empirical rule on measurement time length of EEGs for usability evaluation. In this paper, we conduct an experiment to reveal the optimal time length of EEGs for usability evaluation by analyzing changes of EEGs over time. From the results of the experiments, we have found that the time length suitable for usability evaluation was more than 0~56.32 seconds.

1 Introduction Existing usability evaluation methods include interview, think-aloud protocols [1], and questionnaires [2, 3, 4]. These methods are widely used for usability evaluation because they require no measurement apparatuses and allow usability experts to measure usability of software systems in a relatively easy way. However, they also have some shortcomings for usability evaluation. For instance, usability evaluation using the above methods often requires a huge amount of time to analyze and evaluate collected data. Also, the results of the analysis are sometimes hard to replicate because the collected data is based on qualitative and/or subjective evaluations from system users or subjects participating in usability testing. In order to complement the limitations of these methods, many studies try to quantitatively and objectively develop evaluation methods for measuring the mental or psychological state of users from biological information. In this paper, we focus on the electroencephalogram (EEG) of users after using software as a quantitative measurement method of usability and aim to construct an EEG measurement method for usability evaluation. Alpha rhythm and beta rhythm, which are the frequency component of EEG, change according to mental condition [4], and EEG measurement methods have features that do not disturb subjects using a computer. In general, the EEG measurement method is used as follows: * Graduate School of Information Science, Nara Institute of Science and Technology 8916-5,Takayama, Ikoma, Nara, Japan. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 95–104, 2009. © Springer-Verlag Berlin Heidelberg 2009

96

M. Kimura et al.

(1) experimenters (or usability experts) measure a subject’s EEG for several tens of seconds after the subject finishes target tasks for usability evaluation, and (2) experimenters analyze how much the measured EEG contains the alpha and beta rhythm components, which respectively indicate a comfortable or uncomfortable state of the subject’s mind. The results of the evaluation should change according to the time length to perform the EEG. After the tasks, subjects’ EEGs return to the usual condition as time passes. Hence, the time length of the analysis is too long, and the ratio of the EEG changed by the tasks decreases. Conversely, if the time length is too short, evaluation accuracy decreases because the ratio of the noise to the entire the EEG increases. However, the measurement time at EEG analysis has been decided by experiential standards. So we need to analyze EEGs by time series for usability evaluation. In this paper, we try to obtain the proper time length of EEG data.

2 Related Research In this paper, we quantitatively evaluate a psychological state of computer users during a system in-use, using alpha and beta rhythms composed of frequency components of brain waves. Power spectrums of alpha and beta rhythms, which are obtained by discrete Fourier transform, the ratio of alpha and beta rhythms in all brain waves, and beta/alpha, which is the ratio of alpha and beta rhythms, are often used as common indicators for observing the psychological state of human beings. Matsunaga et al. developed a brain wave measurement system for evaluating satisfaction of human beings and validated the hypothesis that people feel comfortable if the amount of information processing in the brain is small, while people feel uncomfortable if the amount of information processing in the brain is large [5]. In this paper, we also use the ratio of alpha and beta rhythms in brain waves and beta/alpha as indicators. These indicators are often used for studies using brain waves. So our experimental results are easy to compare with implications and insights from previous work.

3 Experiment 3.1 Overview Using Microsoft Excel 2003 and Excel 2007, which are the most popular spreadsheet software, participants performed eight kinds of tasks operating spreadsheets, and we measured participants’ EEGs after each task. Excel 2003 and Excel 2007 are different versions of software released in 2003 and 2007, respectively. Both versions have almost the same functions, but have a different look and feel in their graphical user interfaces (GUIs). Excel 2007 has a new GUI called “ribbon,” which is designed to improve task performance and user experience. However, the newly designed ribbon interface introduced users to a lot of changes to menus, tool bars, and working windows. So, even if users would like to use a familiar function such as “Save As,” they need to select a menu or button with different names and/or positions between the two versions. Furthermore, even if names and positions of menus are not changed, the design of working windows displayed after selecting a menu is often different from Excel 2003. In this way, not only new users but also existing users of Excel 2003 need to learn how to operate the new interface of Excel 2007.

Toward Constructing an EEG Measurement Method for Usability Eval.

97

Table 1. Subjects’ usage frequency of Excel 2003 and Excel 2007

never several times per year several times per month several times per week

Excel 2003 0 2 3 5

Excel 2007 6 1 2 1

Using Excel 2003 and Excel 2007, we investigate the relationship between the experiences of software and the attributes of brain waves without the effect of functional differences. In our experiment, we also analyzed the relationship between the results of subjective evaluation by questionnaire and the attributes of brain waves. 3.2 Participants Ten master’s students from the graduate school of information science participated in the experiment. Table 1 shows participants’ usage frequency of Excel 2003 and Excel 2007. All participants had experience in using Excel 2003 and understood basic operations and functions, but half of the participants had never used Excel 2007. 3.3 Task Participants performed eight tasks (four types of tasks for each version of Excel) operating the spreadsheets given in advance. Table 2 shows a list of tasks used in the experiment. All tasks can perform both Excel 2003 and Excel 2007. The content (a grade report) of the data file that is used in this experiment is the same in all tasks. Participants can continue the tasks until the task time exceeds five minutes. We counterbalanced the order of the tasks to minimize learning and fatigue effects. The following are the details of each task. Same Place Task In this task, the participant selects a particular menu that has the same name in the same position in different versions of Excel. The task is completed when the participant selects the objective menu. Different Place Task The participant selects a menu that has a different name in a different position in the two versions of Excel. This task is completed when the participant selects the objective menu. Same Interface Task The Same Interface Task consists of functions that have the same interfaces of modal dialog in different Excel versions. In the task, menu name and position were given to the participant before the task. Different Interface Task In this task, functions that have different dialog interfaces in Excel 2003 and Excel 2007 were used. As with the Same Interface Task, menu name and position were given to the participant before the task.

98

M. Kimura et al. Table 2. A task list used in the experiment

Task type Same place Different place

Same interface

Task name Open Clip Art Pane Filter Setting Display of version information Record of macros Format Cells Page Orientation

Different interface

Conditional Formatting Insert Bar Chart

Description Open clip art pane to select clip art from a list. Set options for data filtering. Display the version information of Excel. Change date formats from Mar-01 to 03/01. Change a page orientation to landscape and set margins. Change a page orientation to landscape and set margins. Indicate cells that have less than “C” or “Absence” as red font. Insert stacked bar chart of student's scores with chart/axis titles.

3.4 Environment The Emotional Spectrum Analysis System ESA-16 was employed to record EEGs of participants. After the task, we recorded participants’ EEGs for two minutes at 200Hz sampling frequency in the eyes-closed, resting condition. Electrode locations are based on the International 10-20 System, shown in Figure 1. We adapted referential derivation to observe the EEG, and used the right earlobe (A2) as a reference electrode. As a ground electrode, the center of the forehead (Fpz) was employed and the center of the parietal (Pz) was used as an exploring electrode to minimize the electromyogram (EMG) artifact. We also recorded electrocardiogram (ECG) from both arms. In addition to this, we used a headrest and elastic net bandage to secure electrodes placed on the head. Before the first task, each participant adjusted the height of chair and position of the mouse/keyboard. 3.5 Procedure The procedure of the experiment was as follows. 1. Preparation: Authors informed participant about experiment and EEG measurement. 2. Environment setting: Put the electrodes on the participant at the points described in Section 3.4, then set up the EEG analyzer. 3. Practice tasks: The participant performed two practice tasks to understand the procedure of EEG measurement. These tasks were excluded from analysis. 4. Perform a task: The participant performed a main task described in Table 2. 5. EEG measurement: After each task, the EEG of the participant was measured. 6. Perform eight tasks: The participant performed tasks repeatedly until finishing eight tasks and EEG measurements. 7. Questionnaire: After the tasks, the participant filled out the questionnaire described in Section 3.6.

Toward Constructing an EEG Measurement Method for Usability Eval.

99

3.6 Questionnaire After the eight tasks, participants answered the questionnaire sheet to investigate subjective satisfactions for each version of Excel and usage frequency of each function that was used in the tasks. The questionnaire was created by the authors based on the Questionnaire for User Interaction Satisfaction (QUIS). Each question about usage frequency consists of a four-point scale (from “Never” to “Few times per week”) and seven-point scale (from “Strongly disagree” to “Strongly agree”) for subjective satisfaction. Figure 2 shows a part of questionnaire sheet that used in the experiment.

Fp1 F7

T3

Fpz

Fp1

Fz

F3

C3

F4

F8

T4

C4

Cz

A2

A1 T5

P3

P4

Pz O1

T6

O2

Fig. 1. Electrode Locations in the International 10-20 System

Brain wave time[sec]

Fig. 2. Questionnaire Sheet

Method 1. Power spectral analysis at even intervals

5.12

Method 2. Power spectral analysis using different lengths of time window

5.12

5.12

5.12

10.24 15.36

Fig. 3. Two Kinds of Analysis Methods for Electroencephalogram

4 Analysis for EEG We applied power spectral analysis to EEG data we collected at a sampling frequency of 200Hz. To have a clear understanding of how frequency components of brain waves changed over time in the setting of our experiment and how analysis results varied according to lengths of the analysis window, we used two analysis methods for the EEG data as follows. Figure 3 illustrates the difference between the analysis methods.

100

M. Kimura et al.

Method 1. Power spectral analysis at even intervals This analysis aims to observe how brain waves change over time. We analyzed EEG data with a time interval of 5.12 seconds by cutting out the entire EEG data every 5.12 seconds so as not to overlap analysis data. The EEG data with 19 intervals (from 0 to 92.16 second) was analyzed. Method 2. Power spectral analysis using different lengths of time window This analysis aims to observe how analysis results differ according to lengths of analysis window. We analyzed EEG data by increasing time length of the analysis window by 5.12 seconds, without changing the start position of our analysis. The time length was increased by 5.12 seconds (min.) to 97.28 seconds (max.) (i.e., 0~5.12 sec., 0~10.24 sec , …, and 0~97.28 sec.). Next, the target data was filtered to reduce the artifacts from eye blinking, myoelectric activity and so on. We used a high-pass filter (HPF, 3Hz cutoff frequency, +6dB/oct attenuation factor), a low-pass filter (LPF, 60Hz cutoff frequency, -6dB/oct attenuation factor), and a band-elimination filter (BEF, 60Hz central frequency, 47.5Hz~72.5Hz stopband, second order). The band-elimination filter was used to remove the influence of an alternating-current power supply. After the EEG data was multiplied by the Hamming window and processed with the fast Fourier transform (FFT), we obtained the power spectrum from the EEG data. From the obtained power spectrum, we calculated the respective proportions of alpha rhythm and beta rhythm to all brain waves, and also calculated beta/alpha, which divided the proportion of alpha rhythm into the proportion of beta rhythm. In accordance with the classification of the international 10-20 system, we set the frequency components of alpha rhythm and beta rhythm to 8~13Hz and 13~30Hz respectively. We also set the range of all brain waves to 3~30Hz. Since the proportions of alpha and beta rhythms to all brain waves are widely used for observing various activities in the brain, we also decided to use them as indexes for measuring the physiological state of subjects after the tasks. However, because the proportions and intensity of alpha and beta rhythms vary from individual to individual, comparisons of brain waves with the absolute value would be inappropriate. In this paper, we normalized EEG data by an average value of each subject’s power spectrum and compared it with each data.

5 Results 5.1 Results of the Power Spectral Analysis at Even Intervals Figures 4, 5, and 6 respectively show the mean and the standard deviation of the alpha rhythm, beta rhythm and beta/alpha in the power spectral analysis at even intervals. In each graph, the left x-axis, the right x-axis, and the y-axis are the mean, standard deviation, and time respectively. Figure 4 indicates that the mean of alpha rhythms for Excel 2003 were larger than that for Excel 2007 after 56.32 seconds and that the difference of the alpha rhythms between Excel 2003 and 2007 was largest at 81.92 seconds. The standard deviation was always comparatively small and lowest at 56.32 seconds. Figure 5 shows the mean of the beta rhythms for Excel 2007 was larger than that for Excel 2003 after 46.08 seconds, and the difference of the beta rhythms between Excel 2003 and 2007

Toward Constructing an EEG Measurement Method for Usability Eval.

101

was greatest at 87.04 seconds. The standard deviation was higher on the whole than the standard deviation of alpha rhythms. The lowest standard deviation was observed at 40.96 seconds. Figure 6 presents that the mean of beta/alpha in Excel 2007 was larger than that in Excel 2003 after 46.08 seconds and that the difference of beta/alpha between Excel 2003 and 2007 was at 81.92 seconds. The standard deviation is larger than the results of alpha rhythms and beta rhythms and smallest at 40.96 seconds. 5.2 Results of the Power Spectral Analysis Using Different Length of Time Window

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

0.40

1.15

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28

1.20

Time length[sec]

Start time [sec]

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

1.20

0.40

1.15

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

Start time [sec]

Time length[sec]

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

Standard deviation

0.40

1.15

Average of normalized β/ α

Fig. 8. Beta Rhythm in Method 2

1.20

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16

Average of normalized β/ α

Fig. 5. Beta Rhythm in Method 1

Fig. 6. Beta/Alpha in Method 1

Standard deviation

0.35

5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28

0.40

1.15

Standard deviation

1.20

Average of normalized β rhythm

Fig. 7. Alpha Rhythm in Method 2

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16

Average of normalized β rhythm

Fig. 4. Alpha Rhythm in Method 1

Start time [sec]

Standard deviation

0.35

1.20

0.40

1.15

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

Time length[sec]

Fig. 9. Beta/Alpha in Method 2

Standard deviation

Excel 2007 Excel 2007 SD

5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28

1.10

Excel 2003 Excel 2003 SD

Standard deviation

1.15

Average of normalized α rhythm

0.40

1.20

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16

Average of normalized α rhythm

Figures 7, 8, and 9 respectively show the mean and the standard deviation of the alpha rhythm, beta rhythm and beta/alpha in the power spectral analysis using different lengths of time window. In each graph, the left x-axis, the right x-axis, and the y-axis are the mean, standard deviation, and time respectively.

102

M. Kimura et al.

Figure 7 shows that the mean of alpha rhythms in Excel 2003 was larger than that in Excel 2007 in the case of using time windows over 40.96 seconds. As lengths of time window became longer, the alpha rhythms in Excel 2003 tended to increase and the standard deviation decreased. Figure 8 presents that the mean of the beta rhythms in Excel 2007 was larger than that in Excel 2003 in the case of using time windows over 56.32 seconds. As lengths of time window became longer, the standard deviation tended to decrease as well as the results of the alpha rhythms. Figure 9 indicates that the mean of beta/alpha in Excel 2007 was larger than that in Excel 2003 in the case of using time windows over 40.96 seconds. As lengths of time window became longer, the standard deviation tended to decrease as well as the results of the alpha and beta rhythms. Table 3. A Result of Questionnaire Usage

Understand Productivity

frequency Excel 2003 Average 3.3 SD 0.82 Excel 2007 Average 1.8 SD 1.14 p < 0.05 yes

5.0 1.33 3.5 2.17 no

Simple Interface Easy Satisfacto use to use tion 5.4 4.8 4.9 5.3 5.0 1.26 1.32 1.45 1.34 1.15 3.5 4.0 3.1 3.4 3.3 1.72 1.63 2.13 1.90 1.95 no yes yes yes yes

5.3 Results of Questionnaire Table 3 shows the mean, standard deviation and the result of the two-sample t-test of each questionnaire item. In the table, there were significant differences in “Productivity,” “Interface,” “Easy to use” and “Satisfaction” between Excel 2003 and Excel 2007. Our subjects gave Excel 2007 lower scores than Excel 2003.

6 Discussions From the results of the power spectral analysis at even intervals, we could confirm that all three indexes tended to be relatively stable after 56.32 seconds. In fact, the alpha rhythms in Excel 2003 were larger than those in Excel 2007. The beta rhythms and beta/alpha in Excel 2007 were larger those in Excel 2003. The standard deviations of all three indexes also indicated lowest or relatively lower from 40.96 seconds to 56.32 seconds. These results would imply that EEG data from 56.32 seconds to 61.44 seconds is stable among all subjects, is less influenced by artifacts, and appropriately reflects the influence of the difference between Excel 2003 and 2007. However, for all indexes, the difference of the influence between two versions of Excel and the standard deviations constantly fluctuated. We consider that this might occur due to individual differences of subjects’ brain waves, fatigue from the long experiment duration, myogenic potential by attitude variation, and so forth. Therefore, it is necessary not to use a time window with short length, but to use one with long length, in order to conduct accurate usability evaluation.

Toward Constructing an EEG Measurement Method for Usability Eval.

103

From the results of the power spectral analysis using different lengths of time window, we could observe that the alpha rhythms in Excel 2003 were stably higher in the case of using time windows over 40.96 seconds, the beta rhythms in Excel 2007 were stably higher in the case of using time windows over 56.32 seconds, and beta/alpha in Excel 2007 was stably higher in the case of using time windows over 40.96 seconds. For all indexes, the standard deviations tended to become small as the time window became longer. This might be because the rate of EEG influenced by artifacts became small as the time window became longer. These results suggest us that we could analyze EEG data with no influence of artifacts by using a time window over 56.32 seconds. The results of our analysis showed that the alpha rhythms in Excel 2003 were larger than those in Excel 2007, and the beta rhythms and beta/alpha in Excel 2007 were larger than those in Excel 2003. It is clarified by previous studies on EEG measurement methods that the amount of alpha rhythms is decreased and the amount of beta rhythms and the value of beta/alpha are increased when a subject’s mental work load is high. The results of the questionnaire showed our subjects preferred Excel 2007 to Excel 2003. These results of questionnaire and our analysis agree with previous work. Since the results of our analysis and questionnaire also supported the results of the past studies on EEG, we can conclude EEG measurement would be useful for evaluating software usability.

7 Conclusion and Future Work In this paper, we have conducted an experiment to have a clear understanding of the appropriate timing and lengths of time window in order to analyze EEG data for accurate usability evaluation. From the results of the experiments, we could obtain the following insights. • A short length of time window (e.g. 5.12 seconds) is not suitable for usability evaluation because the frequency components of brain waves constantly fluctuate. • The accuracy of usability evaluation can be improved by using length of time windows of 56.32 seconds. In this experiment, we could not observe normal state of EEG data but EEG influenced by the tasks. In the near future, we need to conduct another experiment to observe it.

References 1. Ericsson, K.A., Simon, H.A.: Protocol analysis: Verbal reports as data. MIT Press, Cambridge (1993) 2. Osgood, C.E., Suci, G.J., Tannenbaum, P.H.: The measurement of meaning. University of Illinois Press, Urbana (1957) 3. Chin, J.P., Norman, K.L., Shneiderman, B.: Subjective user evaluation of CF PASCAL programming tools. Technical Report (CAR-TR-304) (1987)

104

M. Kimura et al.

4. Hart, S.G., StaveLand, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Handcock, P.A., Meshkati, N. (eds.) Human Mental Workload, pp. 139–183. Elsevier, Amsterdam (1988) 5. Matsunaga, H., Nakazawa, H.: A Study on Human-Oriented Manufacturing System (HOMS) – Development of Satisfaction Measurement System (SMS) and Evaluation of Element Technologies of HOMS using SMS. In: Int. Conf. Manufacturing Milestones toward 21st Century, pp. 217–222 (1997)

Automated Analysis of Eye-Tracking Data for the Evaluation of Driver Information Systems According to ISO/TS 15007-2:2001 Christian Lange, Martin Wohlfarter, and Heiner Bubb Lehrstuhl für Ergonomie, Technische Universiät München Bolzmannstrasse 15, 85747 Garching {lange,m.wohlfarter,bubb}lfe.mw.tum.de

Abstract. First of all, the most important content of the ISO/TS 15007-2:2001 standard for performing eye tracking experiments will be described. The following text includes a detailed description of how eye/gaze experiments using the Dikablis eye tracking system are conducted according to the above mentioned standard and how recorded statistical evaluations can be automated and visualized. Keywords: ISO/TS 15007-2:2001, Eye tracking, Driver Assistance Systems, Driver Information Systems.

1 Introduction Adhering to the guidelines of the European Statement of Principles ESoP, the need for good and less distracting design of driver information systems will grow enormously. The future will yield the driver information systems which will hinder or distract the driver from the driving task being performed as least as possible. The advantage of the standardized experimental conduction lies in the improvements delivered by progressive experimental comparability, error reduced and faster processing. The duration of the analysis of uplifted data is therefore enormously reduced. Subsequently, the standard conduction of gaze experiments using the Dikablis Toolkit can be proved to comply with the ISO/TS 15007-2:2001 standard.

2 Toolkits for Standardized Experimentation and Testing The Dikablis software package supports the standards required by eye tracking experimentation. In figure 1 the interference between Dikablis and the ISO/TS 150072:2001 standard regarding test planning, calculations and analysis of eye tracking data is illustrated. Dikablis will be deployed in order to evaluate the tracking of the gaze direction and eye and head movements of the test subject. The Dikablis Toolkit consists of a Recording Software, an Analysis Software for post-processing of the eye-tracking data and manual data analysis and D-LAB for completely automated data analysis. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 105–110, 2009. © Springer-Verlag Berlin Heidelberg 2009

106

C. Lange, M. Wohlfarter, and H. Bubb

Fig. 1. Workflow schema for standardized testing and experimentation of Dikablis and ISO/TS 15007-2:2001 interferences/differences

With the Recording Software the gaze behavior of the test subject can precisely be recorded. The test subject wears the Dikablis Head Unit, on which two cameras are installed. One camera is directed towards the test subject’s eye and is used to recognize the subject’s gaze behavior (pupil movements). The other camera is directed straight on in front of the subject and monitors the subject’s environment. By processing both of these pieces of video data, the gaze direction of the test subject can almost precisely be determined. Offline re-calibration and post-processing of the eye-detection can be made with the Dikablis Analysis Software using the recorded data even after the recording software has finished recording. These possibilities to adjust and post-process guarantee clean and always analyzable test results under any circumstances. The D-LAB module contains a package for conducting experiments, from test planning and defining of special Areas of Interest (AOI) visible within the gaze region, to the automatic calculation of glance durations and graphical presentation of these results. Successively, the proceedings to standardized testing and experimentation will sbe described subsequently. Herein the capital and lower case letters are always related to figure 1’s demonstrated procedural (workflow) plan and the relationships between the small angled boxes located within the figure.

Automated Analysis of Eye-Tracking Data

107

2.1 Construction of an Experimental Plan The compliancy to test partitioning presented by ISO/TS 15007-2:2001 into „experimental condition“, „task“, and „subtask“ can be defined into one test plan with DLAB (see figure 2 left). This way a test can be represented in the form of intertwined and nested intervals in which: • „experimental condition“ an entire Experiment is unfolded (Ex. driving on country roads); • “task” defines the interaction between a determined system presented within the Experiment (ex. Operation of Navigation Systems); • A specification of a “task” as “subtask” will be considered. (Ex. Operation of a navigation system via a touch screen display). D-LAB offers the possibility to additionally define „subsubtasks“ as the fourth layer, ex. in order to mark the appearance of critical events presented within the experiment (ex. sharp breaking situations) or automatic analysis of the display screen (ex. an individual input screen within the navigation system, for example inputting a destination address).

Fig. 2. Left: constructing a test sequence; Right: Automatically created block shifting diagram/interface for online test segment sequence marking

2.2 Record Gaze Behavior and Mark the Beginning and End Points of Each Trial Interval Using one of the D-Lab integrated applications one can automatically create a shifting block diagram/interface using the trial definitions, which can be seen on the right side of figure 2. Each shifting block represents a test segment. By pressing a block, a network event is created, which marks the start or ending of a task interval directly in the Recording Software. The functionality of the Dikablis recording software is described

108

C. Lange, M. Wohlfarter, and H. Bubb

in written detail in Lange et al., 2006a und Lange et al., 2006b. These events mark the beginning respectively the end of a trial segment and are synchronously saved to the calculated gaze data. The events can also be started from another data recorder such as a driving simulator. 2.3 Validating Gaze Data and Trial Intervals The Dikablis analysis software is firm in validating the gaze data after a trial and in optimizing. In preparation, the gaze data if necessary is calibrated after the trial so as to optimize offline pupil recognition (see Lange et al., 2006a and Lange et al. 2006b). Further processing in D-LAB follows after the optional jointing is successful. The first move consists of testing whether all trial intervals would be marked correctly. The marked trial intervals located under the gaze player window synchronized to the playtime line shown on the user interface layer (see figure 3). D-LAB also offers a testing function which automatically identifies inconsistencies within the trial segment. The segment can be manually adjusted or changed, tasks can be added or deleted. Figure 3 shows the D-LAB interface for management of a trial interval.

Fig. 3. Validation and post-processing of a trial segment in D-LAB

2.4 Definition of Areas of Interest The areas of interest used in the calculation of glance durations in relevant gaze regions (ex. the navigation system display, the street view, the rear view mirror, the left angle mirror, the dashboard, etc.) as required by the ISO/TS 15007-2:2001 are defined by using D-LAB’s functionality of marking specific AOIs. Thus in D-LAB a random number of AOIs can be defined in the form of a polyline and labeled with names. Figure 4 demonstrates the Head-Up Display and Dashboard as defined AOIs.

Automated Analysis of Eye-Tracking Data

109

Fig. 4. Defining selected AOIs within the gaze region such as the Head-Up Display and the Dashboard

2.5 Automatic Calculation of Glance Metrics to the Defined AOIs In order to automatically calculate glance duration and glance frequency, the pupil of the test subject as well as his/her head position in relation to the test environment (relative to the defined AOIs) must be recognized. The determination of the head position is carried out with the help of the so called marker(s) (see the quadratic black and white object in figure 4), which describe the environment reference. These markers are found using image processing in the photo of the field camera and used in the head position calculation of the test subject. For every defined and established AOI, using the D-LAB function „Calculate Gaze Durations“ it is calculated when the test subject glanced at an AOI and when their glance was away from a certain AOI. The result of this calculation is displayed analogously to the graphical representation of the trial intervals located under the gaze film player window synchronous to the playtime line. Hence, the operator can impose calculation corrections at any time simply by testing and manually correcting if necessary. The gaze specific values from the ISO/TS 15007-2 standard can be calculated using the automated determination of glance durations via the defined AOIs. Thus the operator can input the values which need be calculated for the following gaze experiment in the form of an „Analysis Series“. Therefore the specific values pertaining to specific tasks and AOIs to be calculated must be defined. The following values are available to choose from: • • • • • • • •

Total Glance Time Glance Frequency Time off road-scene-ahead Total glance time as a percentage Fixation probabilities Link value probabilities Maximum Glance Duration Mean Glance Duration

110

C. Lange, M. Wohlfarter, and H. Bubb

D-Lab hereupon calculates the inputted values and saves the results in the form of text file. These automated value calculations can be conducted for all defined AOIs, all terms of the experiment plan as well as for all gaze metrics. The text file is built concurrently so that it can be opened for further statistical analysis using the SPSS statistics program without indirection. 2.6 Graphical Representation of Calculated Metrics Next to the internal calculation of values, D-LAB additionally offers a trial result(s) graphical representation capability. For this purpose several graphical diagrams are displayed which support the interpretation of the result(s). There are two examples in figure 5 that exemplify this. To the top one can find the progressive course of glance duration on defined AOIs for a single subtask presented by