Human-Computer Interaction. New Trends: 13th International Conference, HCI International 2009, San Diego, CA, USA, July 19-24, 2009, Proceedings, Part ... Applications, incl. Internet Web, and HCI)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5610

Julie A. Jacko (Ed.)

Human-Computer Interaction New Trends 13th International Conference, HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings, Part I

13

Volume Editor Julie A. Jacko University of Minnesota Institute of Health Informatics MMC 912, 420 Delaware Street S.E., Minneapolis, MN 55455, USA E-mail: [email protected]

Library of Congress Control Number: 2009929048 CR Subject Classification (1998): H.5, I.3, I.7.5, I.5, I.2.10 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13

0302-9743 3-642-02573-0 Springer Berlin Heidelberg New York 978-3-642-02573-0 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12705719 06/3180 543210

Foreword

The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human–Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in the knowledge and effective use of computers in a variety of application areas. This volume, edited by Julie A. Jacko, contains papers in the thematic area of Human–Computer Interaction, addressing the following major topics: • • • • •

Novel Techniques for Measuring and Monitoring Evaluation Methods, Techniques and Tools User Studies User Interface Design Development Approaches, Methods and Tools

The remaining volumes of the HCI International 2009 proceedings are: • • • • •

Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 3, LNCS 5612, Human–Computer Interaction––Ambient, Ubiquitous and Intelligent Interaction (Part III), edited by Julie A. Jacko Volume 4, LNCS 5613, Human–Computer Interaction - Interacting in Various Application Domains (Part IV), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis

VI

Foreword

• • • • • • • • • • •

Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 11, LNCS 5620, Digital Human Modeling, edited by Vincent G. Duffy Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen Volume 17, LNAI 5639, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris

I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.

Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland

Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA

Foreword

Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea

Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany

Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA

Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China

Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA

Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK

VII

VIII

Foreword

Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa

Matthew J.W. Thomas, Australia Mark Young, UK

Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA

Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK

Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA

Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA

Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK

Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA

Foreword

Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA

Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China

Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria

Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan

Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA

Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA

IX

X

Foreword

Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK

Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany

Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China

Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China

Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan

Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan

In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.

Foreword

XI

I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis

HCI International 2011

The 14th International Conference on Human–Computer Interaction, HCI International 2011, will be held jointly with the affiliated conferences in the summer of 2011. It will cover a broad spectrum of themes related to human–computer interaction, including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/

General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email: [email protected]

Table of Contents

Part I: Novel Techniques for Measuring and Monitoring Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kiyohiko Abe, Shoichi Ohi, and Minoru Ohyama

3

A Usability Study of WebMaps with Eye Tracking Tool: The Effects of Iconic Representation of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ¨ Ozge Ala¸cam and Mustafa Dalcı

12

Feature Extraction and Selection for Inferring User Engagement in an HCI Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stylianos Asteriadis, Kostas Karpouzis, and Stefanos Kollias

22

Informative or Misleading? Heatmaps Deconstructed . . . . . . . . . . . . . . . . . Agnieszka (Aga) Bojko

30

Toward EEG Sensing of Imagined Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan

40

Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ying Gao, Armando Barreto, and Malek Adjouadi

49

Usability Evaluation by Monitoring Physiological and Other Data Simultaneously with a Time-Resolution of Only a Few Seconds . . . . . . . . K´ aroly Hercegfi, M´ arton P´ aszti, Sarolta T´ ov¨ olgyi, and Lajos Izs´ o

59

Study of Human Anxiety on the Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . Santosh Kumar Kalwar and Kari Heikkinen

69

The Research on Adaptive Process for Emotion Recognition by Using Time-Dependent Parameters of Autonomic Nervous Response . . . . . . . . . Jonghwa Kim, Mincheol Whang, and Jincheol Woo

77

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu-Jin Kim, Jin Ah Bae, and Byeong Ho Jeon

85

Toward Constructing an Electroencephalogram Measurement Method for Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masaki Kimura, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto

95

XVI

Table of Contents

Automated Analysis of Eye-Tracking Data for the Evaluation of Driver Information Systems According to ISO/TS 15007-2:2001 . . . . . . . . . . . . . . Christian Lange, Martin Wohlfarter, and Heiner Bubb Brain Response to Good and Bad Design . . . . . . . . . . . . . . . . . . . . . . . . . . . Haeinn Lee, Jungtae Lee, and Ssanghee Seo An Analysis of Eye Movements during Browsing Multiple Search Results Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuko Matsuda, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Development of Estimation System for Concentrate Situation Using Acceleration Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masashi Okubo and Aya Fujimura Psychophysiology as a Tool for HCI Research: Promises and Pitfalls . . . . Byungho Park Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Genaro Rebolledo-Mendez, Ian Dunwell, Erika A. Mart´ınez-Mir´ on, Mar´ıa Dolores Vargas-Cerd´ an, Sara de Freitas, Fotis Liarokapis, and Alma R. Garc´ıa-Gaona Effect of Body Movement on Music Expressivity in Jazz Performances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mamiko Sakata, Sayaka Wakamiya, Naoki Odaka, and Kozaburo Hachimura

105 111

121

131 141

149

159

A Method to Monitor Operator Overloading . . . . . . . . . . . . . . . . . . . . . . . . . Dvijesh Shastri, Ioannis Pavlidis, and Avinash Wesley

169

Decoding Attentional Orientation from EEG Spectra . . . . . . . . . . . . . . . . . Ramesh Srinivasan, Samuel Thorpe, Siyi Deng, Tom Lappas, and Michael D’Zmura

176

On the Possibility about Performance Estimation Just before Beginning a Voluntary Motion Using Movement Related Cortical Potential . . . . . . . Satoshi Suzuki, Takemi Matsui, Yusuke Sakaguchi, Kazuhiro Ando, Nobuyuki Nishiuchi, Toshimasa Yamazaki, and Shin’ichi Fukuzumi

184

Part II: Evaluation Methods, Techniques and Tools A Usability Evaluation Method Applying AHP and Treemap Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toshiyuki Asahi, Teruya Ikegami, and Shin’ichi Fukuzumi

195

Table of Contents

Evaluation of User-Interfaces for Mobile Application Development Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Florence Balagtas-Fernandez and Heinrich Hussmann User-Centered Design and Evaluation – The Big Picture . . . . . . . . . . . . . . Victoria Bellotti, Shin’ichi Fukuzumi, Toshiyuki Asahi, and Shunsuke Suzuki

XVII

204 214

Web-Based System Development for Usability Evaluation of Ubiquitous Computing Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jong Kyu Choi, Han Joon Kim, Beom Suk Jin, and Yonggu Ji

224

Evaluating Mobile Usability: The Role of Fidelity in Full-Scale Laboratory Simulations with Mobile ICT for Hospitals . . . . . . . . . . . . . . . . Yngve Dahl, Ole Andreas Alsos, and Dag Svanæs

232

A Multidimensional Approach for the Evaluation of Mobile Application User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jos´e Eust´ aquio Rangel de Queiroz and Danilo de Sousa Ferreira

242

Development of Quantitative Usability Evaluation Method . . . . . . . . . . . . Shin’ichi Fukuzumi, Teruya Ikegami, and Hidehiko Okada

252

Reference Model for Quality Assurance of Speech Applications . . . . . . . . . Cornelia Hipp and Matthias Peissner

259

Toward Cognitive Modeling for Predicting Usability . . . . . . . . . . . . . . . . . . Bonnie E. John and Shunsuke Suzuki

267

Webjig: An Automated User Data Collection System for Website Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mikio Kiura, Masao Ohira, and Ken-ichi Matsumoto ADiEU: Toward Domain-Based Evaluation of Spoken Dialog Systems . . . Jan Kleindienst, Jan Cuˇr´ın, and Martin Labsk´ y Interpretation of User Evaluation for Emotional Speech Synthesis System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ho-Joon Lee and Jong C. Park Multi-level Validation of the ISOmetrics Questionnaire Based on Qualitative and Quantitative Data Obtained from a Conventional Usability Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jan-Paul Leuteritz, Harald Widlroither, and Michael Kl¨ uh What Do Users Really Do? Experience Sampling in the 21st Century . . . Gavin S. Lew Evaluating Usability-Supporting Architecture Patterns: Reactions from Usability Professionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Edgardo Luzcando, Davide Bolchini, and Anthony Faiola

277 287

295

304 314

320

XVIII

Table of Contents

Heuristic Evaluations of Bioinformatics Tools: A Development Case . . . . Barbara Mirel and Zach Wright A Prototype to Validate ErgoCoIn: A Web Site Ergonomic Inspection Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcelo Morandini, Walter de Abreu Cybis, and Dominique L. Scapin

329

339

Mobile Phone Usability Questionnaire (MPUQ) and Automated Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Young Sam Ryu

349

Estimating Productivity: Composite Operators for Keystroke Level Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jeff Sauro

352

Paper to Electronic Questionnaires: Effects on Structured Questionnaire Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Trujillo

362

Website Designer as an Evaluator: A Formative Evaluation Method for Website Interface Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chao-Yang Yang

372

Part III: User Studies Building on the Usability Study: Two Explorations on How to Better Understand an Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anshu Agarwal and Madhu Prabaker Measuring User Performance for Different Interfaces Using a Word Processor Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tanya R. Beelders, Pieter J. Blignaut, Theo McDonald, and Engela H. Dednam

385

395

Evaluating User Effectiveness in Exploratory Search with TouchGraph Google Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kemal Efe and Sabriye Ozerturk

405

What Do Users Want to See? A Content Preparation Study for Consumer Electronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yinni Guo, Robert W. Proctor, and Gavriel Salvendy

413

“I Love My iPhone... But There Are Certain Things That ‘Niggle’ Me” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Anna Haywood and Gemma Boguslawski

421

Table of Contents

Acceptance of Future Technologies Using Personal Data: A Focus Group with Young Internet Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fabian Hermann, Doris Janssen, Daniel Schipke, and Andreas Schuller Analysis of Breakdowns in Menu-Based Interaction Based on Information Scent Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yukio Horiguchi, Hiroaki Nakanishi, Tetsuo Sawaragi, and Yuji Kuroda E-Shopping Behavior and User-Web Interaction for Developing a Useful Green Website . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fei-Hui Huang, Ying-Lien Lee, and Sheue-Ling Hwang Interaction Comparison among Media Internet Genre . . . . . . . . . . . . . . . . . Sang Hee Kweon, Eun Joung Cho, and Ae Jin Cho Comparing the Usability of the Icons and Functions between IE6.0 and IE7.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chiuhsiang Joe Lin, Min-Chih Hsieh, Hui-Chi Yu, Ping-Jung Tsai, and Wei-Jung Shiang

XIX

431

438

446

455

465

Goods-Finding and Orientation in the Elderly on 3D Virtual Store Interface: The Impact of Classification and Landmarks . . . . . . . . . . . . . . . . Cheng-Li Liu, Shiaw-Tsyr Uang, and Chen-Hao Chang

474

Effects of Gender Difference on Emergency Operation Interface Design in Semiconductor Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hunszu Liu

484

Evaluating a Personal Communication Tool: Sidebar . . . . . . . . . . . . . . . . . Malena Mesarina, Jhilmil Jain, Craig Sayers, Tyler Close, and John Recker

490

“You’ve Got IMs!” How People Manage Concurrent Instant Messages . . . Shailendra Rao, Judy Chen, Robin Jeffries, and Richard Boardman

500

Investigating Children Preferences of a User Interface Design . . . . . . . . . . Jamaliah Taslim, Wan Adilah Wan Adnan, and Noor Azyanti Abu Bakar

510

Usability Evaluation of Graphic Design for Ilmu’s Interface . . . . . . . . . . . . Tengku Siti Meriam Tengku Wook and Siti Salwa Salim

514

Are We Trapped by Majority Influences in Electronic Word-of-Mouth? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yu Tong and Yinqing Zhong

520

XX

Table of Contents

Leveraging a User Research Framework to Guide Research Investments: Windows Vista Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gayna Williams A Usability Evaluation of Public Icon Interface . . . . . . . . . . . . . . . . . . . . . . Sungyoung Yoon, Jonghoon Seo, Joonyoung Yoon, Seungchul Shin, and Tack-Don Han

530 540

Part IV: User Interface Design Little Design Up-Front: A Design Science Approach to Integrating Usability into Agile Requirements Engineering . . . . . . . . . . . . . . . . . . . . . . . Sisira Adikari, Craig McDonald, and John Campbell

549

Aesthetics in Human-Computer Interaction: Views and Reviews . . . . . . . Salah Uddin Ahmed, Abdullah Al Mahmud, and Kristin Bergaust

559

Providing an Efficient Way to Make Desktop Icons Visible . . . . . . . . . . . . Toshiya Akasaka and Yusaku Okada

569

An Integration of Task and Use-Case Meta-models . . . . . . . . . . . . . . . . . . . R´emi Bastide

579

Model-Based Specification and Validation of User Interface Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Birgit Bomsdorf and Daniel Sinnig

587

A Position Paper on ’Living Laboratories’: Rethinking Ecological Designs and Experimentation in Human-Computer Interaction . . . . . . . . . Ed H. Chi

597

Embodied Interaction or Context-Aware Computing? An Integrated Approach to Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Johan Eliasson, Teresa Cerratto Pargman, and Robert Ramberg

606

Supporting Multidisciplinary Teams and Early Design Stages Using Storyboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mieke Haesen, Jan Meskens, Kris Luyten, and Karin Coninx

616

Agent-Based Architecture for Interactive System Design: Current Approaches, Perspectives and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . Christophe Kolski, Peter Forbrig, Bertrand David, Patrick Girard, Chi Dung Tran, and Houcine Ezzedine BunBunMovie: Scenario Visualizing System Based on 3-D Character . . . Tomoya Matsuo and Takashi Yoshino Augmented Collaborative Card-Based Creative Activity with Digital Pens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motoki Miura, Taro Sugihara, and Susumu Kunifuji

624

634

644

Table of Contents

Usability-Engineering-Requirements as a Basis for the Integration with Software Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karsten Nebe and Volker Paelke Design Creation Based on KANSEI in Toshiba . . . . . . . . . . . . . . . . . . . . . . Yosoko Nishizawa and Kanya Hiroi High-Fidelity Prototyping of Interactive Systems Can Be Formal Too . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Philippe Palanque, Jean-Fran¸cois Ladry, David Navarre, and Eric Barboni

XXI

652 660

667

RUCID: Rapid Usable Consistent Interaction Design Patterns-Based Mobile Phone UI Design Library, Process and Tool . . . . . . . . . . . . . . . . . . . Avinash Raj and Vihari Komaragiri

677

The Appropriation of Information and Communication Technology: A Cross-Cultural Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jose Rojas and Matthew Chalmers

687

UISK: Supporting Model-Driven and Sketch-Driven Paperless Prototyping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vin´ıcius Costa Villas Bˆ oas Segura and Simone Diniz Junqueira Barbosa

697

Beyond the User Interface: Towards User-Centred Design of Online Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcin Sikorski

706

Designing for Change: Engineering Adaptable and Adaptive User Interaction by Focusing on User Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruno S. da Silva, Ariane M. Bueno, and Simone D.J. Barbosa

715

Productive Love: A New Proposal for Designing Affective Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ramon Solves Pujol and Hiroyuki Umemuro

725

Insight into Kansei Color Combinations in Interactive User Interface Designing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . K.G.D. Tharangie, Shuichi Matsuzaki, Ashu Marasinghe, and Koichi Yamada Learn as Babies Learn: A Conceptual Model of Designing Optimum Learnability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Douglas Xiaoyong Wang Time-Oriented Interface Design: Picking the Right Time and Method for Information Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Keita Watanabe, Kei Sugawara, Shota Matsuda, and Michiaki Yasumura

735

745

752

XXII

Table of Contents

Enabling Interactive Access to Web Tables . . . . . . . . . . . . . . . . . . . . . . . . . . Xin Yang, Wenchang Xu, and Yuanchun Shi

760

Integration of Creativity into Website Design . . . . . . . . . . . . . . . . . . . . . . . . Liang Zeng, Robert W. Proctor, and Gavriel Salvendy

769

Part V: Development Approaches, Methods and Tools YVision: A General Purpose Software Composition Framework . . . . . . . . Ant˜ ao Almada, Gon¸calo Lopes, Andr´e Almeida, Jo˜ ao Fraz˜ ao, and Nuno Cardoso Collaborative Development and New Devices for Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hans-J¨ org Bullinger and Gunnar Brink

779

789

Orchestration Modeling of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . Bertrand David and Ren´e Chalon

796

An Exploration of Perspective Changes within MBD . . . . . . . . . . . . . . . . . Anke Dittmar and Peter Forbrig

806

Rapid Development of Scoped User Interfaces . . . . . . . . . . . . . . . . . . . . . . . Denis Dub´e, Jacob Beard, and Hans Vangheluwe

816

PaMGIS: A Framework for Pattern-Based Modeling and Generation of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . J¨ urgen Engel and Christian M¨ artin

826

People-Oriented Programming: From Agent-Oriented Analysis to the Design of Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Steve Goschnick

836

Visualization of Software and Systems as Support Mechanism for Integrated Software Project Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Peter Liggesmeyer, Jens Heidrich, J¨ urgen M¨ unch, Robert Kalckl¨ osch, Henning Barthel, and Dirk Zeckzer Collage: A Declarative Programming Model for Compositional Development of Web Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bruce Lucas, Rahul Akolkar, and Charlie Wiecha Hypernetwork Model to Represent Similarity Details Applied to Musical Instrument Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuya Maeshiro, Midori Maeshiro, Katsunori Shimohara, and Shin-ichi Nakayama

846

856

866

Table of Contents

Open Collaborative Development: Trends, Tools, and Tactics . . . . . . . . . . Kathrin M. Moeslein, Angelika C. Bullinger, and Jens Soeldner

XXIII

874

Investigating the Run Time Behavior of Distributed Applications by Using Tiny Java Virtual Machines with Wireless Communications . . . . . . Tsuyoshi Miyazaki, Takayuki Suzuki, and Fujio Yamamoto

882

OntoDesk: Ontology-Based Persistent System-Wide Undo on the Desktop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . David Nemeskey, Buntarou Shizuki, and Jiro Tanaka

890

Peer-to-Peer File Sharing Communication Detection System with Traffic Mining and Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Satoshi Togawa, Kazuhide Kanenishi, and Yoneo Yano

900

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

911

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images Kiyohiko Abe1, Shoichi Ohi2, and Minoru Ohyama3 1

College of Engineering, Kanto Gakuin University, 1-50-1 Mutsuura-higashi, Kanazawa-ku, Yokohama, Kanagawa 236-8501, Japan 2 School of Engineering, Tokyo Denki University, 2-2 Kandanishiki-cho, Chiyoda-ku, Tokyo 101-8457, Japan 3 School of Information Environment, Tokyo Denki University, 2-1200 Muzaigakuendai, Inzai-shi, Chiba 270-1382, Japan [email protected], [email protected], [email protected]

Abstract. We propose a new eye blink detection method that uses NTSC video cameras. This method utilizes split-interlaced images of the eye. These split images are odd- and even-field images in the NTSC format and are generated from NTSC frames (interlaced images). The proposed method yields a time resolution that is double that in the NTSC format; that is, the detailed temporal change that occurs during the process of eye blinking can be measured. To verify the accuracy of the proposed method, experiments are performed using a high-speed digital video camera. Furthermore, results obtained using the NTSC camera were compared with those obtained using the high-speed digital video camera. We also report experimental results for comparing measurements made by the NTSC camera and the high-speed digital video camera. Keywords: Eye Blink, Interlaced Image, Natural Light, Image Analysis, HighSpeed Camera.

1 Introduction The blinking of the eye is related to factors such as human cognition, fatigue, and depressed consciousness; many studies have investigated eye blinking in relation to these factors. Most conventional methods for the measurement of the eye blink analyze eye images (images of the eye and its surrounding skin) captured by a video camera [1], [2], [3]. The NTSC video cameras that are commonly used are capable of detecting eye blinks; however, it is difficult for these cameras to measure the detailed temporal change occurring during the process of eye blinking, because eye blinks occur relatively fast (within a few hundred milliseconds). Therefore, a high-speed camera is required for an accurate measurement of the eye blink [3]. NTSC video cameras capture moving images at 60 fields/s and these field images are mixed with images that have a frame rate of 30 frames/s (fps) to field interlaced images. In this paper, we propose a new method for measuring the eye blink that uses J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 3–11, 2009. © Springer-Verlag Berlin Heidelberg 2009

4

K. Abe, S. Ohi, and M. Ohyama

NTSC video cameras. This method utilizes split-interlaced images of the eye captured by an NTSC video camera. These split images are odd- and even-field images in the NTSC format and are generated from NTSC frames (interlaced images). The proposed method yields a time resolution that is twice that in the NTSC format. Therefore, the detailed temporal change that occurs during the process of eye blinking can be measured. To verify the accuracy of the proposed method, we performed experiments using a high-speed digital video camera. Thereafter, we compared results obtained using the NTSC cameras with those obtained using the high-speed digital video camera. This paper also presents experiments that evaluate the proposed automatic method for measuring eye blinks.

2 Open-Eye Area Extraction Method by Image Analysis In general, eye blinks are estimated by measuring the open-eye area [2] or on basis of characteristics of specific moving points between the upper and lower eyelids [3]. Many of these methods utilize image analysis. It is possible to measure the wave pattern of eye blinks if the entire process of an eye blink is captured [3]. Furthermore, the type of eye blink and/or its velocity can be estimated on the basis of this wave pattern. However, it is difficult to measure the wave patterns of eye blinks by using video cameras that are commonly used for measuring eye blinks because the resulting eye images include high noise content owing to the change in light conditions. We have developed a new method for measuring the wave pattern of an eye blink. This method can be used with common indoor lighting sources such as fluorescent lights, and it can measure the wave pattern automatically. Hence, our proposed measurement method can be used under a variety of experimental conditions. In this method, the wave pattern is obtained by counting the number of pixels in the openeye area of the image as captured by a video camera. This image is enlarged for capturing the detailed eye image. We have proposed an algorithm for extracting the open-eye area in a previous study [4]. It utilizes color information of eye images. We have adapted the algorithm to our proposed method for elucidating the wave pattern of eye blink measurement. This algorithm has been developed for our eye-gaze input system, in which it compensates and traces head-movement [5]. Furthermore, the algorithm has been used under common indoor sources of light for a prolonged period. Hereafter, we describe in detail our image-processing algorithm for extracting the open-eye area. 2.1 Binarization Using Color Information on Image Many methods have been developed for the purpose of skin-color extraction; these methods are primarily focused on facial image processing, including those that utilize color information on a facial image. They mostly determine threshold skin-color values statistically or empirically [6]. We have developed an automatic algorithm for estimating thresholds of skin-color. Our algorithm can extract the open-eye area from the eye image on the basis of the skin-color.

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images

5

Using our algorithm, skin-color threshold is determined by the histogram of the color-difference signal ratio of each pixel—Cr/Cb—that is calculated from the YCbCr image transformed from the RGB image. The histogram of the Cr/Cb value has 2 peaks indicating skin area and open-eye area. The Cr/Cb value indicated by the minimum value between the 2 peaks is designated as the threshold for open-eye area extraction. 2.2

Binarization by Pattern Matching Method

The method described in Subsection 2.1 can extract the open-eye area almost completely. However, the results of this extraction sometimes leave deficits around the corner of eye, because the Cr/Cb value around the corner of eye is similar to the value on skin in certain subjects. To resolve this problem, we have developed a method for open-eye extraction without deficits by combining 2 extraction results. One of them is a binarized image using color information, as described in Section 2.1. The other extraction result is a binarized image using light intensity information, which includes in the extraction result the area around the corner of the eye. Binarization using light intensity information utilizes the threshold estimated by a pattern matching method, which determines the matching point by using the color information of the binarized image as reference data. Hence, the threshold level is estimated automatically. The original image and the extracted open-eye area image are shown in Fig. 1(a) and Fig. 1(b).

(a)

(b)

Fig. 1. Original eye image (a) and extracted open-eye area (b)

3 Measurement Method of Wave Patterns of Eye Blinks Using Split-Interlaced Images Commonly used NTSC video cameras output interlaced images. One interlaced image has 2 field images, which are designated as odd or even fields. If an NTSC camera captures a fast movement such as an eye blink, there is a great divergence between the captured odd- and even-field images. Therefore, the area around eyelids on the captured image has comb-like noise. This phenomenon occurs because of mixing of 2 field images of the fast movement of eyelids. An example of interlaced images during eye blinking is shown in Fig. 2. To describe this phenomenon most clearly, Fig. 2 has been captured at low resolution (145 × 80 pixels).

6

K. Abe, S. Ohi, and M. Ohyama

If one interlaced image is split by scanning even- and odd-numbered lines separately, 2 field images are generated. Thus, the time resolution of the motion images doubles, but the amount of information in the vertical direction decreases by half. These field images are captured at 60 fields/sec, and the NTSC interlaced moving images are captured at 30 fps; therefore, this method yields a time resolution that is double that available in the NTSC format. The duration of a conscious blink is a few hundred milliseconds; therefore, it is difficult to measure accurately the wave pattern of an eye blink by using NTSC cameras. However, the detailed wave pattern of an eye blink can be measured by using our proposed method. The split-interlaced images are shown in Fig. 3. The 2 eye images shown in Fig. 3 are enlarged in a vertical direction and were generated from the interlaced image shown in Fig. 2. Our proposed method measures the wave patterns of eye blinks from these images.

Fig. 2. Blinking eye image (interlaced)

Fig. 3. Split-interlaced image generated from Fig. 2

4 Evaluation Experiment for Proposed Method Either 4 or 5 subjects participated in experiments to evaluate our proposed method, as described in Subsections 4.1 and 4.2, respectively. The experimental setup includes an NTSC DV camera (for home use), a high-speed digital video camera, and a personal computer (PC). The PC analyzes sequenced eye images captured by the video cameras. The DV camera captures interlaced images at 30 fps, and the high-speed digital video camera captures non-interlaced images at 300 fps. In the experiments performed using these video cameras, the wave pattern of eye blinks is measured from sequenced eye images. The experimental setup is shown in Fig. 4.

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images PC

7

Display

User NTSC or High-speed digital video camera

Fig. 4. Hardware configuration of experimental system

4.1 Experiment for Eye Blink Measurement Using NTSC Camera In this experiment, sequenced eye images were captured using the DV camera at 30 fps in NTSC format. In addition, split-interlaced images are generated from these interlaced NTSC images. These split-interlaced images have a time resolution of 60 fields/s. The wave pattern of eye blinks is measured by the interlaced NTSC images and split-interlaced images. The binarization threshold for open-eye area extraction is determined automatically from the first field image of the experimental moving images. This threshold is estimated by the method described in Section 2. A typical result from this experiment is shown in Fig. 5.

Pixels of open-eye area

1.1 1 0.9 0.8

60fps

0.7

30fps

0.6 0.5 0.4 1

6

11 16 21 Sampling point (1/60 sec.)

26

Fig. 5. Wave patterns of eye blinks measured by DV (30 fps and 60 fps)

In Fig. 5, the longitudinal axis and the abscissa axis indicate pixels of open-eye area and sampling point (interval: 1/60 sec), respectively. To compare the 2 wave patterns of eye blinks, these plots are normalized using the pixels of open-eye area at the first field image. The bottoms of the plots indicate the eye-closed condition. Our proposed algorithm classifies the area of eyelid outline and cilia into the open-eye area; therefore, the pixels at the bottom of the plots are not reduced to zero. From Fig. 5, it is evident that sequenced images at 60 fields/s can be used to estimate the detailed wave pattern of an eye blink. During the eye blink, there is a great difference in the 2 plots of pixels of the open-eye area; however, this difference is not dependent on individual subjects.

8

K. Abe, S. Ohi, and M. Ohyama

Results of the wave pattern of eye blink measurements for 5 subjects are shown in Fig. 6, where the longitudinal axis and the abscissa axis show pixels of open-eye area and sampling point, respectively. These plots also are normalized in a manner similar to those in Fig. 5. From Fig. 6, it is evident that there are great differences in the results for each subject. 1.1

Pixels of open-eye area

1 0.9 0.8 0.7 0.6 0.5 0.4

Subject A Subject D

0.3

Subject B Subject E

Subject C

0.2 1

11 21 31 41 Sampling point (1/60 sec.)

51

Fig. 6. Wave patterns of eye blinks of 5 subjects measured by DV (60 fps)

4.2 Experiment for Eye Blink Measurement Using High-Speed Video Camera To verify the accuracy of the proposed method that utilizes split-interlaced images, experiments were conducted with 4 subjects; this experiment and the one described in Subsection 4.1 were conducted separately. Subjects A and E (listed in Fig. 6) were enrolled in this experiment continuously, in which sequenced images at 3 different frame rates (30, 60, and 150 fps) were generated from moving images captured by the high-speed digital video camera. These sequenced images were then analyzed to measure the wave pattern of eye blinks. The results of eye blink measurements performed using the sequenced images at 3 different frame rates and those taken at 300 fps are compared. Typical examples of measurement results are shown in Fig. 7, Fig. 8, and Fig. 9, which display results at 30, 60, and 150 fps, respectively. From Fig. 7 and Fig. 8, it is evident that the degree of accuracy of measurement at 60 fps is higher than that at 30 fps. The minimum of the wave pattern (bottom of the curve) is quite characteristic of when an eye blink occurs. Results at 60 fps show that the bottom of the plot is measured with a high degree of accuracy. Therefore, sequenced images at this frame rate are suitable for measurement of eyelid movement velocity. Moreover, our proposed method using split-interlaced images (described in Section 3) utilizes 2 field images generated from one interlaced image; that is, the

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images

9

spatial information of these field images is decreased by half. We have confirmed that this decrease in spatial information does not affect measurement accuracy via an experiment using sequenced images at 60 fps. The sequenced images at 60 fps were generated from moving images captured by a high-speed digital video camera. In this experiment, we generated half-sized eye images by extracting scanned odd-numbered lines from sequenced images at 60 fps. We estimated wave patterns of eye blinks using these half-sized images. Our results show that the measured open-eye area decreases by half, which is in agreement with the results shown in Fig.8.

Pixels of open-eye area

46000 300 fps

44000

30 fps

42000 40000 38000 36000 34000 32000 1

11

21 31 41 51 61 71 Sampling point (1/300 sec.)

81

Fig. 7. Wave pattern of eye blinks measured by high-speed video camera (30 fps)

Pixels of open-eye area

46000

300 fps

44000

60 fps

42000 40000 38000 36000 34000 32000 1

11

21 31 41 51 61 71 Sampling point (1/300 sec.)

81

Fig. 8. Wave pattern of eye blinks measured by high-speed video camera (60 fps)

10

K. Abe, S. Ohi, and M. Ohyama

Pixels of open-eye area

46000

300 fps

44000

150 fps

42000 40000 38000 36000 34000 32000 1

11

21 31 41 51 61 71 Sampling point (1/300 sec.)

81

Fig. 9. Wave pattern of eye blinks measured by high-speed video camera (150 fps)

4.3 Discussion On the basis of Fig.5, it is evident that by using split-interlaced images, the time resolution of measurement is doubled than that of the results obtained in previous studies. These split images are odd- and even-numbered field images in the NTSC format that are generated from NTSC frames. This method can also be utilized for any subject under common indoor lighting sources, such as fluorescent lights. We have shown the wave patterns of eye blinks for 5 subjects in Fig. 6. From results shown in Fig. 7, Fig. 8, and Fig. 9, it is evident that the degree of accuracy of measurement increases with increasing frame rate. A closer estimate of eye blinking velocity can be achieved if the wave pattern of an eye blink were to be measured with higher accuracy. In other words, the type of eye blink can be classified with a high degree of accuracy. In addition, our proposed method can measure the wave patterns of eye blinks efficiently even by using half-sized eye images. As shown by our experimental results presented earlier, we have verified the reliability of our proposed method described in Section 3. Thus, detailed wave patterns of eye blinks can be measured by using our proposed method.

5 Conclusions We present a new automatic method for measuring eye blinks. Our method utilizes split-interlaced images of the eye captured by an NTSC video camera. These split images are odd- and even-numbered field images in the NTSC format and are generated from NTSC moving images. By using this method, the time resolution for measurement increases to 60 fps, which is double that of conventional methods. Besides the function of automatic measurement of eye blinks, our method can be used under common indoor lighting sources, such as fluorescent lights. In evaluation experiments, we measured eye blinks of all subjects without problems.

Automatic Method for Measuring Eye Blinks Using Split-Interlaced Images

11

To verify the accuracy of our proposed method, we performed experiments using a high-speed digital video camera. On comparison of the results obtained using NTSC cameras with those obtained using a high-speed digital video camera, it is evident that the degree of accuracy of measurement increases with increased resolution time. Additionally, a decrease in area of the split-interlaced image has no adverse effect on the results of eye blink measurements. We confirmed that our proposed method is capable of measuring the wave pattern of eye blinks with high accuracy by using an NTSC video camera. In the future, we plan to develop a new method for classifying types of eye blinks using our proposed measurement method reported above. That new method will be capable of profiling eye blinks according to velocity of open-eye area changes. We also plan to apply this new method to more general ergonomic measurements.

References 1. Grauman, K., Betke, M., Gips, J., Bradski, G.R.: Communication via Eye Blinks - Detection and Duration Analysis in Real Time. In: Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1010–1017, Lihue, HI (2001) 2. Morris, T., Blenkhorn, P., Zaidi, F.: Blink Detection for Real-Time Eye Tracking. J. Network and Computer Applications 25(2), 129–143 (2002) 3. Ohzeki, K., Ryo, B.: Video Analysis for Detecting Eye Blinking Using a High-Speed Camera. In: Proc. of Fortieth Asilomar Conf. on Signals, Systems, and Computers, Pacific Grove, CA, pp. 1081–1085 (2006) 4. Abe, K., Ohyama, M., Ohi, S.: Eye-Gaze Input System with Multi-Indicators Based on Image Analysis under Natural Light. J. The Institute of Image Information and Television Engineers 58(11), 1656–1664 (2004) (in Japanese) 5. Abe, K., Ohi, S., Ohyama, M.: An Eye-Gaze Input System Using Information on Eye Movement History. In: Proc. on 12th International Conference on Human-Computer Interaction, HCI International 2007, Beijing, vol. 6, pp. 721–729 (2007) 6. Garcia, C., Tziritas, G.: Face Detection Using Quantized Skin Color Regions Merging and Wavelet Packet Analysis. IEEE Trans. on Multimedia 1(3), 264–277 (1999)

A Usability Study of WebMaps with Eye Tracking Tool: The Effects of Iconic Representation of Information Özge Alaçam and Mustafa Dalcı Human Computer Interaction Research and Application Laboratory, Computer Center, Middle East Technical University, 06531 Ankara/Turkey {ozge,mdalci}@metu.edu.tr

Abstract. In this study, we aim to conduct usability tests on different WebMap sites with eye movement analysis. Overall task performance, the effects of iconic representation of information, and the efficiency of pop-up usage were evaluated. The eye tracking technology is used for this study in order to follow the position of the users’ eye-gaze. The results show that there are remarkable differences in task performance between WebMaps. Addition, they also differ in the use of iconic representations according to results of users’ evaluation. It is also found that efficiency of pop-up windows’ usage has an effect on task performance. Keywords: Web mapping, usability, eye tracking, cognitive processes, iconic representations, and the efficiency of pop-ups.

1 Introduction Web mapping sites became widespread in many professional areas since they provide opportunities such as designing and sharing maps on the World Wide Web. Beside to their role on professional area, it also became very considerable part of our daily life since it makes the navigation easier [13]. Addition to large number of web mapping sites’ users which access these sites with their desktop and laptop PCs, improvements in technology make the internet available in nearly everywhere providing a chance to connect with mobile devices (mobile phones, smart phones, PDAs) and multiply the number of web mapping sites’ users. By the increasing number of web mapping sites’ users, researchers started to conduct usability studies of these sites and to investigate the effects of usability [2, 8, 13, 15]. The term usability is defined by ISO 9241 [9] as “the effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments” [9, 13]. Another definition from Nielson, one of the pioneers in the usability field, states usability as a quality evaluation that assesses how easy user interfaces are to use. According to his definition, usability is composed of five quality components [16]; these components are learnability, efficiency (task completion time), memorability, errors, satisfaction. Addition to these parameters obtained from usability study, usage of the eye tracking tools adds a different aspect to the usability field for the reason that it provides objective and quantitative evidence to investigate user’s cognitive processes such as visual and attentional J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 12–21, 2009. © Springer-Verlag Berlin Heidelberg 2009

A Usability Study of WebMaps with Eye Tracking Tool

13

processes [5]. Usage of eye tracking on usability field started at 1950’s [6]. However due to the difficulties in the analysis of huge data obtained from the eye tracking tools, it lost its popularity in 1970’s. With the improvements of the eye tracking technologies, eye tracking tools gain their impacts on the usability field again [10] and nowadays they are accepted as a tool to improve computer interface. In one of the studies about WebMap usability conducted by Nivala [13], severity of the usability problems were investigated. In our study, we aim to make additional analysis to find the reason of these usability problems and make them more clear by analyzing eye movements of the users. The focus of this study is to analyze the effects of the iconic representation of the information and to investigate whether the pop-ups are used efficiently by the user. The eye tracking tool is used for this study in order to follow the position of the users’ eye movements, which helps to measure the attended location on the map. It is known that eye movements provide information about cognitive processes such as perception, thinking, decision making and memory [1, 3, 4, 12, 14]. Evaluation of eye movement provided us the opportunity to focus on the iconic representations, efficiency of pop-up windows and their effects in map comprehension in different WebMaps.

2 Method and Materials 26 subjects (12 female, 14 male) either university students or graduate in a range of ages between 18 and 32 participated to this study. In order to get information about their prior knowledge about WebMap usage and to get the user’s idea about the comprehensibility of the icons and preferences about the WebMaps, a questionnaire was carried out. Each subject evaluated two different WebMaps for different places in US in random order. Six tasks shown in Table 1 were used in the experiment. Users were told that they could give up the task or the experiment whenever they wanted to. Tasks given to the users include; to find given address, to find definite places which are represented with icons (s.t airport, metro station, hospital) and to show the route to specific locations. The experiments are conducted at the Human-Computer Interaction Research and Application Laboratory at Middle East Technical University. Eye movements of users were collected by Tobii 1750 Eye Tracker and analyzed with Tobii Studio. Table 1. Task Description Task No Task Description Instruction Welcome to X City/State. You are planning to look at the city map to specify the locations that you want to visit before starting your trip 1 Point the nearest highway intersection to X International Airport You want to go from X International Airport to X University. Could you describe 2 how to arrive to that location? Find the address of the hospital nearest to X Park 3 Now, you are in X Location. Show the nearest metro/railway station to this place 4 You are searching for the library nearest to X place. Find and point it on the map. 5 Show the intersection point of the given address with X street. 6

14

Ö. Alaçam and M. Dalcı

In the Nivela et al.’s study [13], there is an evaluation of four different web mapping sites. These are Google Maps, MSN Maps & Directions, MapQuest and Multimap. However, since the MSN Maps and Directions, Multimap are based on Microsoft Virtual Earth, we replaced these sites with Live Search Maps that is also based on MS Virtual Earth. Since these are well-known and all have zooming and panning options on their 2D map applications, they are very good candidates for usability testing. Although their common properties mentioned above, they differed in terms of usage of icon representation and pop-up window properties. We conducted the usability testing of Google Maps, Live Search Map, MapQuest and Yahoo Map and investigated the effect of iconic representation of information and pop-up windows by analyzing eye movements. We use the term “The iconic representation of information” as to state the relationship between their semantics and appearance. Addition to evaluation of task completion performance (s.t. task completion score and time), eye tracking data such as fixation length, fixation count, observation length was collected.

3 Results Results are presented under three categories; task performance, analysis of the iconic representations and analysis of pop-up windows. 3.1 Task Performance Users are grouped into two categories according to their WebMap usage experience; experienced users (14 users) for high-level usage frequency and inexperienced users (12 users) for low level usage frequency. One way ANOVA test was conducted to compare mean fixation length on task completion time for experience level. Result shows that user’s experience level has a significant effect on task completion time, F(1,52)=5,30, p>.05. One of the evaluation criteria of comparing the usability of WebMaps is users’ task completion scores. Task completion score was evaluated under three categories; accomplished tasks, unaccomplished tasks and partially accomplished tasks that the users thought that they accomplished a task when they actually did not. Table 2 provides the percentage of users, who accomplished, partially accomplished and did not accomplish each task and also overall score was calculated for each WebMap site. Fig. 1 shows the overall completion score for each map. Results of one way ANOVA shows that task completion score of Google Map is significantly different than MapQuest and Yahoo Map, F(3,48)= 8.629 p<.05. It is also worth to note that significance value of difference between Live and Yahoo is .05. In addition to analysis of task completion score, mean fixation length for each task was analyzed individually (see Fig. 2 for comparison). Only fixation length on accomplished and partially accomplished tasks was counted. The results show that, for the first task, there is no significant difference in fixation length according to map type. The fixation length during task two, a significant difference was found between Live Search Map and MapQuest, Google Map and MapQuest, Yahoo Map and MapQuest F (3,43)= 12.538, p<.05. For the third task, Google Map is significantly

A Usability Study of WebMaps with Eye Tracking Tool

15

different than MapQuest, F (3,33)=3.768, p<.05. For the fourth task, Google Map is significantly different than MapQuest and Live Search Map, F(3,23)= 5.398 p<.05. The fixation length on the fifth task, Google Map is significantly different than MapQuest, Yahoo Map and Live Search Map, F(3,35)=12.058, p<.05. For the task six, only difference in Live Search Map and MapQuest is significant, F(3,41)=2.444, p<.05. Statistical analysis of mean fixation count for each task also shows that there are significant differences between the pairs given above. Table 2. Percentage of users’ task completion scores

Task2

Task3

Task4

Task5

Task6

Overall

tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished tasks accomplished tasks partially accomplished tasks unaccomplished

Live Search Map 86.7 13.3 0.0 93.3 0.0 6.7 46.7 26.7 26.7 46.7 0.0 53.3 53.3 13.3 33.3 100.0 0.0 0.0 71.1 8.9 20.0

Map Quest 100.0 0.0 0.0 50.0 35.7 14.2 64.3 14.2 21.4 14.2 7.1 78.6 57.1 0.0 42.8 57.1 0.0 42.9 57.1 9.5 33.3

120,0 Number of participants (%)

Task1

Google Map 100.0 0.0 0.0 100.0 0.0 0.0 35.7 21.4 42.8 78.5 14.2 7.1 78.5 21.4 0.0 100.0 0.0 0.0 82.1 9.5 8.3

100,0 80,0 60,0 40,0 20,0 0,0 Google Map tasks accomplished

Live Search

Map Quest

tasks partially accomplished

Yahoo Map tasks unaccomplished

Fig. 1. The percentage of users according to task completion score

Yahoo Map 100.0 0.0 0.0 63.6 0.0 36.2 36.4 27.2 36.4 9.1 36.4 54.5 45.5 18.1 36.4 45.5 18.2 36.4 50.0 16.7 33.3

16

Ö. Alaçam and M. Dalcı

600 500 400 300 200 100 0 görev1

görev2

Live Search Map

görev3 Google Map

görev4 Yahoo Map

görev5

görev6

MapQuest

Fig. 2. Fixation Length (sec.) on each task according to WebMaps

3.2 Analysis of the Iconic Representations In order to investigate the efficiency of iconic representations in these WebMaps, the observation length on icons was counted frame by frame in specific tasks. In order to analyze icon usage, first, third and fourth tasks were selected since these tasks contain specific places which can be represented by icons (such that airport icon for first task, hospital icon for task three, metro/railway icon for task 4, and pointers that appears after users’ search for all three tasks). Other icons displayed on the map during these tasks were also investigated. Fixation length on each icon and the time which they are displayed on the map were counted and then the percentage of iconic representation looking time was calculated. One-way ANOVA results shows that there is no significant difference in observation length on icons for each WebMap, F(3,48)=1,859, p>.05. In addition, no correlation between the icon usage looking time percentage and completion score was found. (Spearman’s rho=.37, p>.05) Icons are divided into two categories as, task related icons and task unrelated icons. However, since every map could not have a representation for each icon, they were investigated individually, without conducting statistical analysis. Since MapQuest had only pointer icons, these are not compared with the specific icons in other maps. Table 3 provides the percentage of looking time for each icon in each WebMap. Observation length on Airport in Yahoo Map, is 13,3% of the time that it appears on the map. However, since the other WebMaps (Google Maps, MapQuest and Live Search Map) do not have icon for airport, there can be no comparison analysis. The pointers which represent the searched location have approximately same looking time for all maps. For the metro and railway icons, Google icon has biggest looking time percentage, and then it is followed by Live Search Map and Yahoo Map respectively. Since the MapQuest and Live Search Maps do not have hospital icons, looking time of icons on only Yahoo Map and Google Map were investigated. Even tough Google and Yahoo Maps contain an icon for “Hospital”, the users are expected to zoom close

A Usability Study of WebMaps with Eye Tracking Tool

17

enough for that icon to be visible. However none of the users were inclined to zoom to that distance while performing their tasks. The eye movement analysis of users which perform the experiment on Live Map Search shows an interesting outcome; some task unrelated icons (s.t. park, hotel or sponsored link icons) have been remarkably fixated by the users. Table 3. The percentage of looking time for each icon Icon Type Google Map Task Related Icons

Unrelated Icons

Live Search

Map Quest

Yahoo Map

Airport

Na*

Na

Na

13,3

Pointers

10,6

10,4

10,9

10,5

Metro/railway

13,5

10,8

Na

7,2

University

3,5

Na

Na

Na

Hospital

0,0

Na

Na

0,0

Park

1,4

Na

Na

Na

Hotel

Na

5,5

Na

Na

Sponsored link

Na

1,7

Na

Na

*Not Applicable.

Task based investigation has been made regarding iconic representations. Correlations between the icon looking time percentage, completion score and observation length for Task 1,3 and 4 were examined individually. For Task 1, no correlation between these parameters was found. The analysis conducted for the third task, correlations have been found between the icon looking time percentage and completion score (Pearson’s r=.523; p<.05) and between icon looking time and observation length (Pearson’s r=-.368; p<.05). For Task four, a correlation has been found between icon looking time and observation length (Pearson’s r=-.289; p<.05). On the other hand, no correlation has been found between the icon looking time percentage and completion score (Pearson’s r=.165; p>.05). Addition to eye movement analysis, user evaluation questionnaire was carried out. The users were asked to predict the meaning of icons and rate their comprehensibility in a scale from 1 to 10 (the results are given in Table 4). This gives us the opportunity to evaluate the efficiency of the relationship between their semantics and appearances and show whether there is an ambiguity. The comprehensibility ratings for only correct predictions were counted. It can be claimed that there is a consistency between its appearance and semantic when both the correct predictions and comprehensibility rate of a icon is high. Although the comprehensibility evaluated by the users is high for the hospital icon in Google, only 6.3 % of the users predict the meaning of the icon correctly. The comparison of the results for metro/railway icons indicates that iconic representations in Google Maps and Yahoo Maps are comprehended more easily by the users than in Live Search Maps. None of the users predicts the meaning of the sponsored link and hotel icon in Live Search Map. When we look at the Table 3 for looking time percentage of these icons, it can be concluded that the users notice them without attaching a meaning.

18

Ö. Alaçam and M. Dalcı Table 4. Users’ ratings on iconic representations Icon

% of correct definition by users

Comprehensibility

+

6,3

9,0

Hospital (Yahoo Map)

87,5

8,5

Airport (Yahoo Map) Metro/Railway (GoogleMap) Metro/Railway (Yahoo Map) Metro/Railway (Live Search Map)

100

10

81,3 / 93,8

6,6 / 7,9

81,3

6,6

62,5

6,2

Hotel (Live Search Map)

0

Na

Sponsored Links (Live Search Map)

0

Na

Hospital (Google Map)

Moreover, users were asked to specify their icon preference and rate the usabilty of the maps which they used during the experiment. We can group the icons into three main categories; these are Pictorial, Textual and Numerical/Alphabetical icons. Textual icons like (s.t. ) are mostly prefered icons (37.5%). 31,3 % of the users prefer pictorial icons (s.t. ), to be used in maps. Another 31,3 % of the users prefer numerical/alphabetical icons (s.t. ). Additionally, users’ usabilty ratings of Web Maps are parallel with task completion score given in Table 2. Their ratings are 8.3 for Google Map 5.1 for live Search Map, 4.6 for Yahoo Map and 3,4 for MapQuest. 3.3 Analysis of Pop-Up Windows Additional analysis was conducted to investigate the usage of pop-up windows. It can be suggested that the aim of the pop-up is to direct users focus on something else where they are provided with additional information either regarding their task or not. Moreover, these pop-ups can facilitate additional search on locations that the task requires. Therefore pop-up windows are very important parts of the web maps and they frequently appear during map usage. In order to investigate the efficiency of popup usage in these WebMaps, the sections that contain pop-up windows on map area were extracted for each task. Fixation length and fixation count on pop-up windows and display time of it on map area was counted. The analysis of looking time of popup windows when they appear on the map gives us an idea about whether the users prefer to use them. The results of this analysis showed that Google Maps use the popup windows more efficiently, since % 64.8 of the fixations on the map are on the popup area. It is followed by MapQuest, Live Search Map and Yahoo Map respectively (see Fig. 3). One way ANOVA results also indicates that there is a significant difference between Google Map and Live Search Map in pop-up usage percentage. Google

A Usability Study of WebMaps with Eye Tracking Tool

19

Observation Length (sec)

200,00 180,00 160,00 140,00 120,00 100,00 80,00 60,00 40,00 20,00 0,00

Google Map

Live Search Map

MapQuest

popup

48,96

40,05

63,16

25,79

map

75,52

96,78

115,19

77,38

map

Yahoo Map

popup

Fig. 3. Observation Length on Pop-up / Map for each WebMap

Map is also significantly different than Yahoo Map. Addition, there is also difference between Yahoo Map and MapQuest, F(3,51)=5.939, p<.05. Additionally, correlations have been found between fixation count on pop-ups and overall completion score (Spearman’s rho=.396; p<.05) and between fixation length on pop-ups and overall completion score (Spearman’s rho=.423; p<.05).

4 Conclusion By analyzing eye movements which are indicators of cognitive processes, we examined the effects of iconic representations, pop-up window usage, and their roles on the usability of these mapping sites. Since comprehension of maps contains very complicated cognitive process, decreasing the cognitive load by making the icons more comprehensible and the pop-ups more usable will cause effectiveness and efficiency in the usage of web mapping sites. Task performance evaluation shows that there is a significant difference between these Webmaps. When the tasks are getting complicated (ex. to find an address of specific location near to another location), differences between these WebMaps in terms of task completion time and score has become apparent. Besides the differences in overall display organization, in a micro level framework, we examined the effect of the iconic representation of information and efficiency of the usage of popup windows. The analysis indicates that there is gab between user’s ratings on icons and their looking time percentages. Even the icons that are correctly predicted and given high comprehensibility level by the users have a low looking time. Additionally, it is also worth to be noted that icon’s looking time percentage is low for even task related icons. This means users have difficulty to detect them when they are performing their tasks although the icons are highly related with tasks. To make them more visible can help the users notice them, and increase their task completion performance. Moreover, analysis on the efficiency of pop-up windows, which are other

20

Ö. Alaçam and M. Dalcı

widely used elements of WebMaps, shows that users’ pop-up usage differs significantly according to WebMap type and this parameter is positively correlated with task completion score. In addition, as an expected outcome, experience level has a significant role on WebMap usage performance.

5 Further Studies However, icons for local traffic signs (ex. Highway numbers) were highly fixated during tasks; however these findings have been disregarded due to the user’s lack of familiarity towards to locations. Follow up studies to investigate these icons can be conducted with native users of the location. In addition, to evaluate the interaction between some particular areas of web site (s.t. search bars, menus, map area, information area which shows the results of the searched location) gives additional information about the efficiency and effectiveness of the usability of the WebMaps. Acknowledgements. We thank to the Computer Center / Middle East Technical University, and TUBİTAK (for support under grant SOBAG 104K098) for providing eye tracking system. We also thank to Yasemin Saatiçioğlu Oran and our colleagues in METU Computer Center for their valuable support.

References 1. Barkowsky, T.: Mental Processing of Geographic Knowledge. In: Montello, D.R. (ed.) COSIT 2001. LNCS, vol. 2205, pp. 371–386. Springer, Heidelberg (2001) 2. Davies, C., Wood, L., Fountain, L.: User-centred GI: hearing the voice of the customer. In: AGI 2005: People, Places and Partnerships, Annual Conference of the Association for Geographic Information, London, UK, November 8–10, 2005, Association for Geographic Information, London (2005) 3. Downs, R.G., Liben, L.S.: The Development of Expertise in Geography: A CognitiveDevelopmental Approach to Geographic Education. Annals of the Association of American Geographers 81(2), 304–327 (2005) 4. Downs, R.G., Liben, L.S., Daggs, D.G.: The Development of Expertise in Geography: A Cognitive-On Education and Geographers. The Role of Cognitive Developmental Theory in Geographic Education 78(4), 680–700 (2005) 5. Duchowski, A.T.: A Breadth-First Survey of Eye Tracking Applications. Behavior Research Methods, Instruments, & Computers (BRMIC) 34, 455–470 (2002) 6. Fitts, P.M., Jones, J.E., Milton, J.L.: Eye movements of aircraft pilots during instrument landing approaches. Aeronautical Engineering Review 9(2), 24–29 (1950) 7. Global Mapping International, http://www.gmi.org/mapping/websites.htm 8. Haklay, M., Zafiri, A.: Usability Engineering for GIS: Learning from a Screenshot. The Cartographic Journal 45(2), 87–97 (2008) 9. ISO. ISO/DIS 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability, International Organization for Standardization (1998) 10. Jacob, R.J.K., Karn, K.S.: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises (Section commentary). In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eyes: Cognitive and Applied Aspects of Eye Movements, Elsevier Science, Oxford (2003)

A Usability Study of WebMaps with Eye Tracking Tool

21

11. Montello, D.R.: Cognitive Map-Design Research in the Twentieth Century: Theoretical and Empirical Approaches. Cartography and Geographic Information Science 29(3), 283– 304 (2002) 12. Murata, A., Furukawa, N.: Relationships among display features, eye movement caharcteristics and reaction time in visual search. Human Factors 47(3) (2005) 13. Nivala, A.M., Brewster, S., Sarjakoski, L.T.: Usability Evaluation of Web Mapping Sites. The Cartographic Journal 45(2), 129–138 (2008) 14. Richardson, D.C., Dale, R., Spivey, M.J.: Eye movements in language and cognition: A brief introduction. In: Gonzalez- Marquez, M., Coulson, S., Mittelberg, I., Spivey, M.J. (eds.) Methods in cognitive linguistics, John Benjamins, Amsterdam (in press) 15. Skarlatidou, A., Haklay, M.: Public Web Mapping: Preliminary Usability Evaluation. In: Proceedings of GIS Research UK Conference, Nottingham, April 5-7 (2006) 16. Usability 101: Introduction to Usability (August 25, 2003), http://www.useit.com/alertbox/20030825.html (retrieved 16:22, September 21,2008)

Feature Extraction and Selection for Inferring User Engagement in an HCI Environment Stylianos Asteriadis, Kostas Karpouzis, and Stefanos Kollias National Technical University of Athens, School of Electrical and Computer Engineering, Image, Video and Multimedia Systems Laboratory, GR-157 80 Zographou, Greece [email protected], {kkarpou,stefanos}@cs.ntua.gr

Abstract. In this paper, we present our work towards estimating the engagement of a person to the displayed information of a computer monitor. Deciding whether a user is attentive or not, and frustrated or not, helps adapting the displayed information of a computer in special environments, such as e-learning. The aim of the current work is the development of a method that can work userindependently, without necessitating special lighting conditions and with only requirements in terms of hardware, a computer and a web-camera. Keywords: User engagement, Head Pose, Eye Gaze, Facial Feature tracking.

1 Introduction While a lot of work has been done regarding the issue of user attention estimation in multi-user environments, such as meetings [7], very few articles have been published for estimating attention (engagement) in an HCI environment [6]. In this work, we propose a method for inferring user attention based on the extraction of features deriving from facial analysis. Such systems have mainly been proposed for estimating drivers' attention. For example, in [8], a monocular system is used and color precicates are employed to detect and track the driver's facial features. Facial geometry and eye closure are calculated and driver's attention is extracted with the use of three finite state automata. In [3], the authors use solely eyes closure to determine whether a driver is attentive or not. To this aim, they use a gaussian model to describe driver's attention and, eye closure for certain amounts of time would denote inattention or fatigue. In [10], the position in space of each facial feature is detected using a stereo vision system. Based on these positions, a least square algorithm estimates the head orientation and further analysis follows for the detection of the eye gaze. The combination of the above gives a good estimate of the direction at which a user is looking, however, it requires initialization. In our work, we combine head pose with eye gaze, as well as other biometrics, in a monocular environment, not necessitating any particular lighting conditions or calibration. There is not much work in bibliography combining the two features in an unconstrained environment. Typical work is the one J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 22–29, 2009. © Springer-Verlag Berlin Heidelberg 2009

Feature Extraction and Selection for Inferring User Engagement

23

reported in [12], where facial symmetry along with Gabor filters are used for estimating head pose and eye gaze respectively. A look-up table is then built for corresponding the resulting eye gaze and head pose with the final focus of attention estimate. Here, we propose a work that can be summarized as follows: Face detection [11], followed by facial feature detection is the first step of our method, while tracking follows. Based on facial features' motion, a series of biometric measurements are extracted and their appropriateness is evaluated for inferring the level of frustration or attentiveness of a user in a Human-Computer-Interaction scenario. Our algorithm is able to recover and re-initialize in cases of occlusion or tracking failure.

2 Facial Points Detection and Tracking The method reported in [1] is used for face and eye centre and mouth corner (here, enhanced with upper and lower lip points) localization. For the detection of the eye corners (left, right, upper and lower eyelids) a technique similar to that described in [13] is used. In the current work, the point between the nostrils and two points on each eyebrow have also been used, as will be discussed later. For nostrils detection, an area around a segment of the perpendicular to the inter-ocular line was extended starting from the middle of the eyes. The darkest row of this area is considered as the vertical position of the nostrils and the middle point of this row is further used for our experiments. In a similar manner, two points on each eyebrow are extracted, as the darkest points in a neighborhood above the eye corners. The above are illustrated in Fig. 1, where the luminance values of two search areas have been projected on the vertical axis. The minimum of the projections corresponds to the features in search. Tracking is done using a three-Pyramid Lukas-Kanade algorithm. Geometrical face models, and prototypes of natural human motion are employed for recovering from erroneous tracking (see subsection 3.1).

Fig. 1. Eyebrow and nose detection search regions

3 Feature Extraction The features extracted in our method are the following: Head pose, eye gaze, eyebrow movements, head horizontal and vertical speed components, mouth horizontal and vertical opening, relative movements of the user back and forwards.

24

S. Asteriadis, K. Karpouzis, and S. Kollias

3.1 Head Pose Estimation Head rotation is calculated by examining the translation of the point in the middle of the inter-ocular line with regards to its position when the user was rotated frontally (see Fig. 2), thus providing the Head Pose Vector p = [px py], where px and py are the horizontal and vertical components of the eye middle point’s translation, respectively, normalized with the inter-ocular distance, as calculated at start-up, to cater for scale variations. The fraction of the inter-ocular distance with the vertical distance between the eyes and the mouth is monitored, and if it is restricted within certain limits with regards to its value at a frontal position, no rotation is decided. As tracking, many times, fails, thus giving false estimates of head pose (as well as the other features), a series of rules were integrated: After large rotations, some features are occluded and cannot be further recovered. In this case, when the user comes back to a frontal position, after nt1 frames, the pose length reduces in length but is above a certain threshold, as one of the eyes is not well tracked and the eye center is not at the same neighborhood as at start-up. In this case, the algorithm can re-initialize. The above can be modeled as in equations (1),(2):

p(n) < thr1 ⋅ p(n − nt1 )

(

)

var p n−nt 2:n < thr2

(1) (2)

where ║*║ denotes a vector length metric. Equations (1)-(2) are interpreted as follows: If the Head Pose Vector length at the current frame n is smaller than a fraction thr1 of its value at frame n-nt1, and its variance for the last nt2 frames (with nt2
Fig. 2. Pose changes during a video of a person in front of a monitor

Feature Extraction and Selection for Inferring User Engagement

25

smaller than thr2, the algorithm re-initializes. In our experiments, we used nt1=10, nt2=7, thr1=0.7, thr2=0.05. If the above conditions are met, but the user has not turned frontally, face detection fails and, thus, frontal rotation is not decided. In general conditions, however, the algorithm re-initializes by re-detecting the face, facial features and re-starting to track. Further constraints that are taken into account are related with the displacement of features in subsequent frames. By assuming an orthographic projection in the interval between two subsequent frames, it is expected that features are shifted in a uniform way. Finding such outliers and re-calculating the mean shift with the rest of the features helps positioning erroneous points to the position that would agree with the rest of the features’ shift. As experiments showed, the above refinement is achieved after 7-10 iterations per frame.

Fig. 3. Gaze changes during a video of a person not moving his head in front of a monitor

3.2 Eye Gaze Estimation Eye gaze is extracted by monitoring the eye centre movements with regards to a coordinate system defined by the positions of the eye corners and eyelids at each frame (see Fig. 3). The resulting displacement provides with the Eye Gaze Vector gy], where gx and gy is the horizontal and vertical component respectively.

g = [gx

3.3 Extraction of Further Features The vertical movements of the eyebrows with regards to the upper eyelids are also extracted, and the horizontal and vertical components of the speed (in pixels per frame) of the head movements are calculated. Furthermore, mouth opening is calculated, with reference to the initial distance between the mouth corners and the lips distance at start-up. Finally, the inter-ocular distance changes are monitored, and

26

S. Asteriadis, K. Karpouzis, and S. Kollias

calculated as fractions of the eye centres’ distance with regards to the first frame of each initialization of the system. In this way, when changes in inter-ocular distance are not due to head rotations, qualitative measurements of user movement back and forth are achieved.

4 Feature Selection The experiments were conducted on a database consisting of children with learning difficulties, between the age of 8 and 10. The recorded videos were 720×576 pixels and the frame-rate 25fps. A total of about 10000 and 12250 frames were used for the case of attention/non-attention and frustration/non-frustration problems respectively. The videos were annotated by experts. One of the difficulties of the dataset was that the positive instances (frustration, non-attentiveness) were very few in comparison to the negative ones, and this limited the training session prototypes. In order to evaluate the appropriateness of each of the above features, the Fisher's exact test [4] was used. To this aim, the 3-bin histogram of each of the features was calculated and the resulting distribution for positive instances was compared against the distribution of the same feature throughout all videos, regardless of the state. In our case, we chose Fisher's exact test and not any other method (e.g. chi-square method) because it is ideal for small scale data. Indeed, in many occasions (for example, horizontal speed of the head when the user is frustrated), there are only a few instances where there are low or high values at the correspondent histogram bins. Fisher's exact test is ideal in cases of such small samples. For the event of non-attentiveness, among all features, tests showed that for the features of Head Pose, Eye Gaze, Inter-Ocular Distance Changes and Head Speed, the null hypothesis (that observed and expected distributions do not differ) should be rejected with higher confidence than for the rest of the features. For the event of frustration, Head Pose, Horizontal and Vertical Head speed and Eye Gaze do not follow the null hypothesis as much as the rest of the features do, as it was expected. Figure 4 justifies the rejection of some of the features due to high p-values, while Figures 5 and 6 illustrate examples of data for each class.

Fig. 4. p-values for feature selection in attention/non-attention and frustration/non-frustration scenarios

Feature Extraction and Selection for Inferring User Engagement

27

Fig. 5. Features used for attention/non-attention classification

Fig. 6. Features used for frustration/non-frustration classification

5 Experimental Results For testing the accuracy of our system, a Sugeno-type fuzzy [9] inference system was built for each case. The idea behind using fuzzy systems has to do with the fact that behavioral states do not necessarily belong to certain classes but, rather, they are fuzzy concepts. For example, frustration or distraction can be given confidence values, and outputs of fuzzy systems are ideal in this case. Prior to training, our data were clustered using the sub-cluster algorithm described in [2]. This algorithm, instead of using a grid partition of the data, clusters them and, thus, leads to fuzzy systems deprived of the curse of dimensionality. The number of clusters created by the algorithm determines the optimum number of the fuzzy rules. After defining the fuzzy inference system architecture, its parameters (membership function centers and widths), were acquired by applying a least squares and back-propagation gradient descent method [5]. Tables 1 and 2 summarize the results of the overall accuracy of our system in estimating the behavior of a user in the cases attention/non-attention and frustration/non-frustration experiments using different sets of features with low p-value as inputs and 1 or 0 the target states (attention/non-attention or frustration/non-frustration). Testing was done by adopting a leave-one-out protocol.

28

S. Asteriadis, K. Karpouzis, and S. Kollias

From tables 1 and 2, it can be seen that, in the case of frustration, although eye gaze has low p-value (see Fig. 3), excluding it from experiments does not deteriorate the results but they are marginally higher. This is due to the fact that, in cases of frustration, in our dataset, eye gaze vector length was strongly correlated with head pose vector length. Similarly, although Head Speed (horizontal and vertical) has low p-values in the attention tests, results have shown that, excluding these parameters from our decision systems, would improve the results. More careful observation of our data and the corresponding annotation gave the following explanation: Head speed is only large at the beginning of those time segments when a person is turning his/her head away from the camera. At those time segments, head pose vector has small values but head speed is high. However, such movements can also be met during attention time-stamps, as it is very frequent for a reader to make small rapid movements without changing his head pose a lot. For the above reason, it was decided to exclude head speed from our experiments in the case of attention estimation. The database we used was acquired under normal lighting conditions, with very challenging subjects: Children with learning difficulties. Testing our system on such a dataset is challenging, not only because of its nature, but also due to the fact that annotation is subjective. However, the results obtained are extremely promising. Table 1. Neuro-Fuzzy System decision accuracy for two different sets of low p-value features for detecting User Attention Features Head Pose, Eye gaze, Distance changes, Head speed Head Pose, Eye gaze, Distance changes

Overall success rates 84.00% 88.00%

Table 2. Neuro-Fuzzy System decision accuracy for two different sets of low p-value features for detecting User Frustration Features Head Pose, Horizontal and Vertical Head speed Head Pose, Horizontal and Vertical Head speed, Eye Gaze

Overall success rates 82.00% 80.63%

6 Conclusions and Future Work We presented a method for the automatic estimation of the behavior of a person in front of an HCI environment. Our system is un-intrusive, thus, leaving space for spontaneous behavior, it does not depend on controlled conditions in terms of lighting, and this constitutes it ideal for different settings. Furthermore, since the system does not require any a-priory knowledge of the user or the camera, it does not need any kind of training or calibration beforehand. Future extensions of our work shall include the creation of a common framework for discriminating among a set of states simultaneously. To this aim, we will build a database suitable for our research and work on developing a facial feature tracker, highly specialized for such applications.

Feature Extraction and Selection for Inferring User Engagement

29

Acknowledgments This work has been funded by the FP6 IP CALLAS (Conveying Affectiveness in Leading-edge Living Adaptive Systems), Contract number IST-34800 and by the IST Project ’FEELIX’, (under contract FP6 IST-045169).

References 1. Asteriadis, S., Nikolaidis, N., Pitas, I., Pardàs, M.: Detection of facial characteristics based on edge information. In: 2nd International Conference on Computer Vision Theory and Applications (VISAPP), Barcelona, Spain, pp. 247–252 (2007) 2. Chiu, S.L.: Fuzzy Model Identification Based on Cluster Estimation. Journal of Intelligent and Fuzzy Systems 2(3), 267–278 (1994) 3. D’Orazio, T., Leo, M., Guaragnella, C., Distante, A.: A visual approach for driver inattention detection. Pattern Recognition 40(8), 2341–2355 (2007) 4. Fisher, R.A.: Statistical Methods for Research Workers. Hafner Publishing (1970) 5. Jang, J.S.R.: ANFIS: Adaptive-Network-Based Fuzzy Inference System. IEEE Transactions on Systems, Man, and Cybern. 23, 665–684 (1993) 6. Matsumoto, Y., Ogasawara, T., Zelinsky, A.: Behavior recognition based on head pose and gaze direction measurement. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Takamatsu, Japan, pp. 2127–2132 (2000) 7. Otsuka, K., Takemae, Y., Yamato, J.: A probabilistic inference of multiparty-conversation structure based on markov-switching models of gaze patterns, head directions, and utterances. In: ICMI, pp. 191–198 (2005) 8. Smith, P., Member, S., Shah, M., Lobo, N.D.V.: Determining driver visual attention with one camera. IEEE Trans. on Intelligent Transportation Systems 4, 205–218 (2003) 9. Takagi, T., Sugeno, M.: Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. Syst. Man Cybern. 15(1), 116–132 (1985) 10. Victor, T., Blomberg, O., Zelinsky, A.: Automating driver visual behavior measurement. In: 9th Vision in Vehicles Conference, Australia (2001) 11. Viola, P.A., Jones, M.J.: Rapid object detection using a boosted cascade of simple features. In: Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511–518 (2001) 12. Weidenbacher, U., Layher, G., Bayerl, P., Neumann, H.: Detection of head pose and gaze direction for human-computer interaction. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS, vol. 4021, pp. 9–19. Springer, Heidelberg (2006) 13. Zhou, Z.H., Geng, X.: Projection functions for eye detection. Pattern Recognition 37(5), 1049–1056 (2004)

Informative or Misleading? Heatmaps Deconstructed Agnieszka (Aga) Bojko User Centric, Inc. 2 Trans Am Plaza Dr, Ste 100, Oakbrook Terrace, IL 60181 USA [email protected]

Abstract. Eye tracking heatmaps have become very popular and easy to create over the last few years. They are very compelling and can be effective in summarizing and communicating data. However, heatmaps are often used incorrectly and for the wrong reasons. In addition, many do not include all the information that is necessary for proper interpretation. This paper describes several types of heatmaps as representations of different aspects of visual attention, and provides guidance on when to use and how to interpret heatmaps. It explains how heatmaps are created and how their appearance can be modified by manipulating different display settings. Guidelines for proper use of heatmaps are also proposed. Keywords: Heatmaps, attention maps, eye tracking.

1 Introduction Heatmaps are two-dimensional graphical representations of data where the values of a variable are shown as colors. Heatmaps are compelling for two reasons. First, the intuitive nature of the color scale as it relates to temperature minimizes the amount of learning necessary to understand it. From experience, we know that yellow is warmer than green, orange is warmer than yellow, and red is hot. It is not difficult to then figure out that the amount of heat is proportional to the level of the represented variable. Second, heatmaps show the data directly over the stimulus. Because the data could not be any closer to the elements to which they pertain, little mental effort is required to read a heatmap. Heatmaps can be of great value to papers, reports, and presentations because they summarize large quantities of data that would be much more difficult to grasp if presented numerically. Heatmaps help us quickly see “the big picture” including any patterns or trends that may exist in the data. In the user experience field, heatmaps can represent various types of data, such as usage (e.g., clicks, key presses), accuracy, or visual attention. This paper focuses exclusively on attention heatmaps, which have become popular over the past few years due to the increased usage of eye tracking technology. Attention heatmaps can be easily generated with the help of an eye tracking software application such as Tobii’s ClearView, Tobii Studio, SMI’s BeGaze, NYAN, or J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 30–39, 2009. © Springer-Verlag Berlin Heidelberg 2009

Informative or Misleading? Heatmaps Deconstructed

31

EyeTools. Anyone with an eye tracker and the right software can create a heatmap. No knowledge of eye movements or of how heatmaps are created is required. As a result, heatmaps are often generated unnecessarily or are misinterpreted by those who do not understand what the visualizations are really showing or, perhaps even more importantly, not showing. Heatmaps can be deceptive because they look so intuitive that we often do not realize how much we actually do not understand. This paper describes different heatmap types and their limitations, as well as settings used to manipulate the appearance of heatmaps. It also discusses when heatmaps should and should not be used. Proposed guidelines for using heatmaps correctly conclude this work.

2 Types of Attention Heatmaps Heatmaps are often shown with little, if any, description of what it is they are representing. The assumption is that they are showing “attention” or “eye movements” but knowing that is certainly not enough to be able to truly understand a heatmap. There are different aspects of eye movements that heatmaps can represent. Examples include fixation count, absolute or relative gaze duration, and percentage/proportion of participants who fixated on each area of the stimulus. Choosing the right heatmap to present depends on the study objectives and the eye movement measures that address these objectives. For example, if search efficiency was of interest to the researchers, one of the measures collected and analyzed might be the number of fixations prior to acquiring the target [1]. Therefore, assuming that the analysis would benefit from data visualization, a fixation count heatmap should be presented. A fixation count heatmap would also be appropriate if the study goal was to determine the amount of interest generated by various elements of the stimulus during a free-view task (i.e., task with no specific task instructions) [1]. However, if noticeability of a particular element was of interest, the percentage of participants who fixated on the element could be used as a measure (in addition to, for example, time to first fixation), which would warrant a participant percentage heatmap. Because each heatmap type has different limitations that impact its interpretation, it is not only important to be aware of these limitations, but also to know the types of all heatmaps included in papers, reports, and presentations. 2.1 Fixation Count Heatmap A visual fixation can be loosely defined as a relatively stationary eye position focused on a particular location of the stimulus (a more precise definition is discussed in section 3.1). Fixations are important events that provide insight into human cognition because during each fixation we extract visual information that we process [2]. A fixation count heatmap (see Fig. 1) shows the accumulated number of fixations across participants. Each fixation made by each participant adds a value to the color map at the location of the fixation [3]. This value is the same for each fixation regardless of its duration, so a 100 ms fixation is represented in the same way as a 900 ms fixation. Thus, when looking at a fixation count heatmap, we cannot assume that areas of the same color received similar total gaze time.

32

A. (Aga) Bojko

Fig. 1. Fixation count heatmap (n = 13; 0 – 12 s; free-view task)

Fixation count heatmaps can also be biased towards individuals who show high interest in elements that others do not. For example, two elements can be the same color but one attracted ten fixations from one participant only, while the other attracted attention of ten participants, one fixation from each. Therefore, we cannot assume that areas that appear similar in terms of “heat” are equivalent in terms of the number of participants who looked at them. Another limitation of fixation count heatmaps is the fact that they can be skewed towards individuals who had a longer exposure to the stimulus and thus an opportunity to produce more fixations. For example, if participant A spent twice as much time on the stimulus than participant B, participant A’s data would impact the heatmap twice as much as participant B’s data. This is something to always keep in mind when viewing heatmaps created based on unequal exposure times. 2.2 Absolute Gaze Duration Heatmap An absolute gaze duration heatmap (see Fig. 2) shows the accumulated time participants spent looking at the different areas of the stimulus. Each fixation made by each participant adds a value to the color map that is proportional to its duration [3]. For example, a 900 ms fixation will be nine times higher in color value than a 100 ms fixation. Because fixation duration is an indicator of cognitive processing [4], a heatmap that is scaled by fixation duration not only shows which areas were attended to but also represents the level of cognitive processing that the areas required.

Fig. 2. Absolute gaze duration heatmap (n = 13; 0 – 12 s; free-view task)

Informative or Misleading? Heatmaps Deconstructed

33

An absolute gaze duration heatmap can be misleading because it displays different phenomena in the exact same way. For example, this type of heatmap will make one 900 ms fixation look the same as nine 100 ms fixations. A 900 ms fixation on an element indicates that one person looked at it for a while, while nine 100 ms fixations could mean, for example, that one person made nine brief fixations to the element or nine people made one brief fixation each. In addition, similar to the fixation count heatmaps, absolute gaze duration heatmaps can be biased towards individuals who spent more time looking at the stimulus. To eliminate any bias due to unequal exposure times, the gaze duration data can be normalized to create relative gaze duration heatmaps. 2.3 Relative Gaze Duration Heatmap A relative gaze duration heatmap shows the accumulated time each participant spent fixating at the different areas of the stimulus relative to the total time the participant spent looking at the stimulus [3]. In other words, if participant A spent 6 seconds on a web page including 2 seconds on the navigation and participant B spent 60 seconds on the same web page including 20 seconds on the navigation, this type of heatmap will make their data, as it relates to the navigation, the same weight. Similar to the absolute gaze duration heatmap, this heatmap will also show a high gaze time by an individual (proportional to his/her total viewing time) the same way as several short gaze times by a number of individuals (proportional to their total viewing times). If the exposure time is equal across participants (e.g., all participants saw the page for 12 s), which is the case in the examples presented in this paper, the relative gaze duration heatmap and the absolute duration heatmap will be identical. 2.4 Participant Percentage Heatmap A participant percentage heatmap (see Fig. 3) shows the percentage of participants who fixated on the different areas of the stimulus. Each participant who looked at any given location adds a value to the color map. This value is the same for each participant regardless of the number of fixations he or she made or the fixation durations. Thus, an area that was briefly fixated once by each of the participants will be presented in the same color as an area that was fixated multiple times by each of the participants and the fixations were much longer.

Fig. 3. Participant percentage heatmap (n = 13; 0 – 12 s; free-view task)

34

A. (Aga) Bojko

3 Display Settings for Creating Heatmaps The appearance of heatmaps can be modified by manipulating various display settings. While adjusting the settings cannot change the relative distribution of attention, certain areas can be made to appear “hotter” or “colder.” This can be done, for example, by changing the fixation criteria used for the analysis, showing raw data instead of fixation data, changing the upper threshold definition of the color scale, or modifying the time segment for the presented data. Since heatmap display settings can have a great impact on the appearance of heatmaps, they must be properly selected and communicated to ensure accurate interpretation. 3.1 Changing Fixation Criteria There are several algorithms that can be used to define a fixation. A common algorithm used in commercial eye tracking software is based on duration and dispersion threshold identification. To define a fixation using this algorithm, two parameters need to be specified: minimum fixation duration (e.g., 80 ms) and maximum dispersion threshold (e.g., 0.5 degree of visual angle) [5]. A fixation defined by 80 ms and 0.5° will encompass all consecutive eye movements that occurred within 0.5° from each other for at least 80 ms. Unless noted otherwise, the heatmaps in this paper were created using these settings. Manipulating the duration and dispersion thresholds will change the number of fixations in the data. For example, increasing the minimum fixation duration from 80 ms to 200 ms will decrease the number of fixations because it will exclude all the fixations between 80 ms and 200 ms. Increasing the maximum dispersion threshold from 0.5 degree to 1 degrees of visual angle will also decrease the number of fixations because some of the fixations that are closer together will be combined. Conversely, reducing the minimum fixation duration and maximum dispersion threshold will increase the number of fixations. More fixations in the data will increase the amount of “heat” in the heatmap, as shown in Figure 4.

Fig. 4. Fixation count heatmaps based on the same data (n = 13; 0 – 12 s; free-view task). On the left: minimum fixation duration = 200 ms; on the right: minimum fixation duration = 80 ms.

The lack of explicit fixation definition is an issue that does not pertain specifically to heatmaps but to entire papers and reports. Many user experience studies that analyze fixation data never mention how these fixations were defined. However, this information is very important for two reasons. First, the results of different studies are

Informative or Misleading? Heatmaps Deconstructed

35

not comparable unless it is clear that fixations were defined in the same way. Second, if the definition is not provided, it cannot be verified whether or not the fixation criteria were appropriate for the stimuli used in the study. For example, fixation duration threshold for image viewing should be higher than for reading because image viewing tends to produce longer fixations than reading due to the fact that more information is being processed in a single fixation [6]. The fixation definition used to create heatmaps should match the definition used for data analysis. Changing the fixation duration to obtain a visualization of a particular intensity is not a good practice. 3.2 Displaying Raw Data Instead of Fixation Data One step further from decreasing the fixation duration and dispersion thresholds is presenting raw data instead of fixation data. Raw data consists of meaningful eye movements (raw fixation points) and “noise” – eye movements that have little meaning in most user experience research. The noise includes eye movements that take place during saccades (rapid eye movements between fixations) as well as drifts, tremors, and flicks that occur during fixations [5]. Adding all the noise intensifies the heatmap, increasing the area covered in red (see Fig. 5).

Fig. 5. Fixation count heatmaps based on the same data (n = 13; 0 – 12 s; free-view task). On the left: fixation data; on the right: raw data.

As a general rule, fixation data rather than raw data should be used when creating visualizations unless the stimulus has moving elements and the heatmap has to show smooth pursuit eye movements. We can assume that fixation data was used if the paper or report specifies how the researchers defined a fixation. 3.3 Changing the Definition of the Color Scale Upper Threshold Another way to manipulate the amount of heat in heatmaps is by changing the definition of the upper threshold on the scale, which is usually indicated by the red color. If the requirements for an area to be red are lowered, the amount of red in the heatmap will increase. Lowering of the upper threshold of the scale can be achieved by decreasing the minimum number of fixations in fixation count heatmaps (see Fig. 6) or by decreasing the minimum gaze length in absolute gaze duration heatmaps.

36

A. (Aga) Bojko

There is no set process for choosing the right upper threshold. The rule of thumb is to make sure that the heatmap properly captures the range of values that are of interest to the study. Threshold selection can be compared to setting the maximum value on the Y axis in a graph, where the Y axis indicates values of the independent variable. If the Y axis is too short, the data points that exceed the maximum on the axis are cut off. As a result, we only know that these data points are higher than the maximum but we do not know what they are exactly and how they differ from one another. On the other hand, if the Y axis is too high, the graph data will appear compressed and the differences between the data points will look smaller. Similarly, if a heatmap’s upper threshold is set too low, many areas will be covered in red with no differentiation between the amount of attention each attracted. Conversely, setting a heatmap’s upper threshold too high will limit the range of colors (e.g., constricting it to yellow or orange as the maximum) and no areas will be covered in red. Regardless of what criteria have been selected, they should be explicitly communicated in the figure legend or caption (e.g., “red = 10+ fixations” or “red color indicates areas that accumulated 10 s or more of gaze time”). It is also useful to put this value in context by providing the average number of fixations each participant made on the stimulus or the average time each participant spent looking at it. The heatmaps presented in this paper were created based on data with the average of 42 fixations on the page per participant and consistent exposure time of 12 seconds. If heatmaps are generated for different experimental conditions or participant groups, their upper threshold definition should be identical, so the heatmaps can be compared. If the display settings are not the same, the differences between the heatmaps may be due to factors other than the data itself.

Fig. 6. Fixation count heatmaps based on the same data (n = 13; 0 – 12s; free-view task). On the left: red = 10+ fixations; on the right: red = 3+ fixations.

3.4 Modifying the Time Segment Sometimes it may be appropriate to present data based on a shorter time segment than the total time during which a stimulus was shown to the participants. This will obviously decrease the amount of data, thus reducing the size of red areas in all heatmap types mentioned in this paper except for the relative gaze duration heatmap (see Fig. 7). Therefore, if a heatmap presents data from a time segment that is shorter than the total viewing time, this needs to be specified in the figure legend or caption.

Informative or Misleading? Heatmaps Deconstructed

37

In addition to the time segment, it should also be clear what participants were trying to do when the data presented in the heatmap was being collected. All too often heatmaps are shown without any context of the task. Eye movements are very taskdependent [7] and participants trying to log in to a website will produce very different attention distribution than if they were trying to find a product. Even if there was no specific task, it should still be noted that the data was collected in a free-view situation, which is how the heatmaps included in this paper were obtained.

Fig. 7. Fixation count heatmaps of the same free-view task (n = 13). On the left: data from 0 to 6 seconds; on the right: data from 0 to 12 seconds.

4 Heatmap Usage 4.1 Common Mistakes There are a few common mistakes when it comes to using heatmaps. The biggest of them is a belief that heatmaps are appropriate for just about any user experience study and for any research question. Sometimes creating heatmaps even becomes the objective of the study (e.g., “we just wanted to see where people look”). This popular “let’s-track-and-see-what-happens” approach has a limited value because of its lack of focus (i.e., the study design does not target specific questions) and frequent lack of proper data analysis. Researchers often draw conclusions or make recommendations based on the results of those studies, which is inappropriate. For example, we cannot say that the reason why an element did not get much attention was because of its suboptimal placement or insufficient size unless we have tested other conditions (i.e., alternative placements or sizes) and compared the data using appropriate statistics. Even if different conditions were tested, sometimes conclusions regarding differences between conditions are made just by looking at the heatmaps. However, heatmaps do not lend themselves to any systematic comparison. Without any data analysis, it is impossible to tell if there are real differences between heatmaps, even if the heatmaps appear to be different. 4.2 Proper Usage Examples Heatmaps should be used in a purposeful way and only when they add value. They can serve as illustrations of participants’ viewing behavior and distribution of attention.

38

A. (Aga) Bojko

While they can communicate data, they cannot explain it or help analyze it. Therefore, heatmaps can rarely stand on their own. To maximize their usefulness and reduce ambiguity, heatmaps should accompany a quantitative analysis. One of our studies investigated a new standardized label template for prescription drug labels [8]. The goal was to determine the impact of the template on pharmacists’ drug selection speed and accuracy as compared to the existing label designs. The eye tracking measures included the number of fixations prior to target selection as an indicator of search efficiency, average fixation duration as a measure of information processing difficulty, and pupil diameter as a measure of cognitive workload. The results were presented in the form of statistical analyses but no heatmaps were included in the report. The study was a quantitative assessment of the effectiveness of the new labels, and heatmaps showing attention distribution were simply of no value. In another study, we evaluated a new homepage design for a professional organization against the original homepage [9]. Our objective was to identify which design was better and why based on a series of tasks during which participants attempted to locate the correct entry point on the homepage. Measures of search efficiency such as the number of fixations and the number of eye visits to the target prior to target selection were analyzed. The analysis was supplemented with heatmaps to show the distribution of attention on the page and to help account for any inefficiencies that occurred. For example, several tasks were more efficient using the new design which had a more centralized navigation. The heatmaps of the original design showed scattered fixations covering multiple navigation areas, while heatmaps of the new design revealed fixations focused mostly around the targets.

5 Guidelines for Using Heatmaps Even though heatmaps are very compelling and seemingly easy to understand, they should be used with caution and according to the following guidelines, summarized based on the discussion from the previous sections of this paper: A. B. C. D.

E. F. G. H.

I. J.

Generate heatmaps only if they add value to the research. Use heatmaps for data visualization instead of data analysis. Use heatmaps to support quantitative analysis rather than on their own. Understand the different heatmap types and only use the ones that represent measures which address your study objectives (e.g., when analyzing gaze time, use a gaze duration heatmap). Specify the type of data the heatmap is representing (e.g., fixation count or absolute fixation duration). Know the limitations of each heatmap type to avoid incorrect interpretation. When creating heatmaps, use fixation data rather than raw data. Provide fixation definition and keep it consistent for analyses and visualizations within a study (e.g., min fixation duration = 100 ms and max dispersion threshold = 0.5°). Provide the definition for the upper threshold of the heatmap color scale (e.g., red = 10+ fixations). Put the upper threshold value in context (e.g., average number of fixations on the stimulus per participant).

Informative or Misleading? Heatmaps Deconstructed

39

K. Specify the time segment based on which the heatmap was created (e.g., the first 10 seconds of exposure). L. Provide task context for each heatmap (e.g., data obtained from participants during the checkout task). M. Use the same heatmap settings (e.g., upper threshold and time segment) for conditions that you are comparing. N. If a paper or report does not provide important information about its heatmaps (e.g., type, fixation definition, upper threshold definition, time segment), ask the authors for clarification before making any assumptions.

6 Conclusion Blinded by the attractiveness and apparent intuitiveness of heatmaps, we often do not realize how much information in addition to the visualization itself is necessary to fully understand a heatmap and properly interpret the data it represents. In other words, the biggest danger involved in creating and reading heatmaps is that we are often unaware of what we do not know, and thus we do not look or ask for the missing information. This paper has exposed some of these gaps in our meta-knowledge in the hope to encourage more critical thinking about the usage of heatmaps.

References 1. Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises. In: Hyona, J., Radach, R., Deubel, H. (eds.) The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, pp. 573–605. Elsevier Science, Amsterdam (2003) 2. Liversedge, S.P., Findlay, J.M.: Saccadic Eye Movements and Cognition. Trends in Cognitive Sciences 4, 6–14 (2000) 3. Tobii.: Tobii Studio 1.X User Manual (2008) 4. Duchowski, A.: Eye Tracking Methodology: Theory and Practice. Springer, Heidelberg (2003) 5. Salvucci, D.D., Goldberg, J.H.: Identifying Fixations and Saccades in Eye-Tracking Protocols. In: Proceedings of Eye Tracking Research and Applications Symposium (2000) 6. Castelhano, M.S., Rayner, K.: Eye Movements During Reading, Visual Search, Scene Perception: An Overview. In: Rayner, K., Shem, D., Bai, X., Yan, G. (eds.) Cognitive and Cultural Influences on Eye Movements, pp. 3–33. Psychology Press (2008) 7. Yarbus, A.L.: Eye Movements and Vision. Plenum Press (1967) 8. Bojko, A.: Measuring the Effects of Drug Label Design and Similarity on Pharmacists’ Performance. In: Tullis, T., Albert, W. (eds.) Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics, pp. 271–280. Morgan Kaufmann, San Francisco (2008) 9. Bojko, A.: Using Eye Tracking to Compare Web Page Designs: A Case Study. Journal of Usability Studies 1, 112–120 (2006)

Toward EEG Sensing of Imagined Speech Michael D’Zmura, Siyi Deng, Tom Lappas, Samuel Thorpe, and Ramesh Srinivasan Department of Cognitive Sciences, UC Irvine, SSPB 3219, Irvine, CA 92697-5100 {mdzmura,sdeng,tlappas,sthorpe,r.srinivasan}@uci.edu

Abstract. Might EEG measured while one imagines words or sentences provide enough information for one to identify what is being thought? Analysis of EEG data from an experiment in which two syllables are spoken in imagination in one of three rhythms shows that information is present in EEG alpha, beta and theta bands. Envelopes are used to compute filters matched to a particular experimental condition; the filters' action on data from a particular trial lets one determine the experimental condition used for that trial with appreciably greater-than-chance performance. Informative spectral features within bands lead us to current work with EEG spectrograms. Keywords: EEG, imagined speech, covert speech, classification.

1 Introduction Dewan reported in 1967 [1] that he and several others could transmit letters of the alphabet using EEG recording; their ability to control voluntarily the amplitude of their alpha waves let them send letters using Morse code. Farwell and Donchin [2] used a second method for transmitting linguistic information by EEG; the P300 response to targets let them determine which of a sequence of displayed letters a person had in mind. Finally, Suppes and colleagues [3,4] reported that they were able to classify EEG responses to heard sentences and that there was information available through EEG concerning imagined speech. The work reported here verifies this latter result and develops it further, with the aim of using EEG brain waves to communicate imagined speech. Classification experiments show that brain-wave signatures of imagined speech lets one distinguish linguistic content, with varying degree of success. These signatures include differences in alpha-, beta- and theta-band activity. Our near-term goal is to use such signatures to design filters that let one distinguish linguistic elements in real time. One application is in further experiments which provide feedback to the thinker as to how recognizable a particular thought is, with the aim of training the thinker to produce brain waves which are more discernible. The focus within is on the results of a single experiment in which subjects produce in their imagination one of two syllables in one of three different rhythms. Analysis of the data using envelopes within alpha, beta and theta frequency bands, spectral features within single bands, and spectrograms in current work show that EEG provides considerable information on imagined speech in this experimental setting. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 40–48, 2009. © Springer-Verlag Berlin Heidelberg 2009

Toward EEG Sensing of Imagined Speech

41

2 Methods Four subjects participated in an experiment with six conditions determined factorially through combination of two syllables and three rhythms (see Figure 1). A single experimental session comprised 20 trials for each of the six conditions; conditions were presented in block-randomized order. Each subject participated in six such sessions for a total of 120 trials per condition. EEG was recorded using a 128 Channel Sensor Net (Electrical Geodesics) in combination with an amplifier and acquisition software (Advanced Neuro Technology). The EEG was sampled at 1024Hz and on-line average referenced. Subjects were instructed to keep their eyes open in the dimly-lit recording room and to avoid eye and other movement during the six seconds, following the cue, during which speech was imagined without any vocalization whatsoever.

Fig. 1. Timelines for the six conditions, labeled at left, in the imagined speech experiment. Durations are indicated in eighths of a second. The syllable /ba/ was imagined in one of three different rhythms in conditions 1-3; the syllable /ku/ was imagined in conditions 4-6. The condition for each trial was cued during an initial period of duration 4.5 sec (12/8 sec = 1.5 sec; 3 x 1.5 sec = 4.5 sec). During this initial period, subjects heard through Stax electrostatic earphones either a spoken "ba" or a spoken "ku" followed by a train of clicks (arrows) indicating the rhythm to be reproduced. Subjects were instructed to speak in their imagination the cued syllable, illustrated for condition 1 by {ba}, using the cued rhythm and tempo. Desired imagined syllable onset times reproduce the cued rhythm and are indicated by asterisks.

3 Analysis and Results Our aims were to classify single trials offline according to condition and discern condition signatures needed to create online filters. EEG waveform envelopes provide encouraging results when used to classify trials according to experimental condition.

42

M. D’Zmura et al.

These envelopes were computed for each electrode in the theta (3-8Hz), alpha (813Hz) and beta (13-18Hz) frequency bands and used to construct matched filters. These filters work well to classify trials according to condition, yet further results with spectra lead us to explore a finer-grained analysis using spectrograms. 3.1 Matched-Filter Classification Using Envelopes We used waveform envelopes to compute matched filters, one for each of the six conditions (see Figure 2). The inner product of each matched filter with a particular trial's envelope provides six numbers. The matched filter which gives rise to the largest inner product is the best match, and the condition to which the matched filter corresponds is declared the best guess as to the experimental condition for that trial. Offline preprocessing steps include • segment the EEG data to provide time-varying waveforms for each condition, electrode and trial; • remove from further consideration 18 electrodes most sensitive to electromyographic artifact: those with the lowest positions about the head and spanning locations close to the eyes, low on the temple, at or below the ear, and at or below the external occipital protuberance; • remove the mean and linear trend from each segmented waveform; • low-pass filter the detrended, segmented, waveforms to remove 60Hz line noise; • use thresholds to identify and remove from further consideration filtered waveforms likely contaminated with electromyographic artifact. Alpha-, beta- and theta-band activity were computed for each electrode and trial using band-pass elliptic filters. These band-pass filtered waveforms (we[t] and bottom right plot in Fig. 2) were Hilbert-transformed to provide corresponding envelopes (ve[t] and middle right plot). The envelopes serve as input to the matched-filter classification and as data used to construct the matched filters. An electrode's average envelopes, found by averaging across trials for each of the six conditions, serve as matched filters with one further step: for each electrode, pseudoinvert the six conditions' average envelopes to provide six filters (Fec[t] and top right plot). The inversion of the average envelopes provides filters which return an inner product of one for the corresponding condition's average envelope and an inner product of zero for all other condition's average envelopes. Information may be integrated across electrodes by summing the measures pe,c across electrodes and determining the maximum of the resulting six numbers. Information may be integrated across bands by weighing individual response measures for the three bands by band reliability estimates or by a voting procedure. Classification results shown in Table 1 show that the beta band (13-18Hz) is, for all but one subject, the most informative frequency band; theta (3-8Hz) and alpha (813Hz) are comparable for all subjects but S1. The per-condition classification performance is shown for subject S2 in Fig. 3. Rows refer to a trial's actual condition, while columns refer to the condition of the matched filter for which the maximal response was obtained. Darker values indicate higher percentages of trials. These dark values describe diagonals, which indicates a match between a trial's actual condition and the condition with matched filter providing the greatest response.

Toward EEG Sensing of Imagined Speech

43

Fig. 2. Preprocessed and band-pass-filtered waveform we[t], recorded by electrode e, is Hilberttransformed to provide envelope ve[t]. Shown at bottom right is the preprocessed waveform from an electrode with mid-parietal location from subject S1 for a trial during which {ba} is spoken in imagination at times 1.5, 3.0 and 4.5 sec (condition 1). Shown at middle right is the corresponding envelope. The inner product <, > of this envelope with each matched filter Fe,c[t], one filter per condition per electrode, provides six numbers pe,c. These six numbers measure how well the particular trial's envelope matches the filter for each condition. The maximum of these six numbers is used to determine the most likely condition c˜ e . Shown at top right is a matched filter for condition c = 1.

The distributions of classification performance across the scalp are similar across the four subjects; those for subject S4 are shown in Figure 4. The most informative electrodes lie largely near the top of the head (vertex) where electromyographic artifacts have their least influence. These distributions of information differ significantly from the distributions of envelope amplitude, which follow well-known patterns for alpha activity (parietal) and theta activity (frontal).

44

M. D’Zmura et al.

Table 1. Classification performance using matched filters for envelopes in three frequency bands. The fraction of correctly-classified trials (720 trials per subject, identified in the left column) is indicated. The chance performance level in this classification among six conditions is 1/6 (0.17).

S1 S2 S3 S4

alpha 0.38 0.63 0.44 0.64

beta 0.80 0.87 0.68 0.62

theta 0.63 0.59 0.46 0.59

Fig. 3. Classification matrices for subject S2. Values along the diagonals indicate correct classification, while non-white values off the diagonal indicate errors. The black gray-value in the middle panel (beta-band) for actual condition 1 and matched filter condition 1 (top left square) represents 91% of the trials; lighter shades indicate smaller values (through white indicating 0%). Perfect performance would be indicated by all off-diagonal entries set to white.

Fig. 4. Distributions of classification performance across the scalp for alpha (α), beta (β) and theta (θ) bands for subject S4. Darker values indicate the positions of electrodes providing better classification performance.

3.2 Spectral Features

Analysis of trials’ power spectral densities shows that differences in power within single frequency bands can provide information concerning trial condition. Spectral

Toward EEG Sensing of Imagined Speech

45

differences within a single band are completely invisible to the previous analysis, which grouped together activity at all frequencies within a single band. Data for subjects S1 and S2 show that, for condition 3, there is a peak in the power spectrum in the range 5-6Hz, relative to baseline power measured at 3.5-4.5Hz. This peak is largely absent for condition 1. This difference between the average power spectra for conditions 3 and 1 is localized to electrodes with front central locations on the scalp (see Fig. 5, leftmost panels). This spectral difference alone, within the theta band, provides ~75% correct two-way classification between these two conditions. The peak and the localized spatial distribution do not exist for subjects S3 and S4 (see Fig. 5, rightmost panels).

Fig. 5. Difference between condition 3 and condition 1 theta sub-band spectral power localized primarily to electrodes with front, center locations for subjects S1 and S2; this pattern is not evident for S3 and S4

3.3 Spectrographic Analysis

The spectral data suggest several things. The first is that one would do well to represent EEG data spectrographically: record power as a fine-grained function of both frequency and time. The second and third are more cautionary. There is a tremendous amount of trial-by-trial variability in EEG recordings. Adding channels, like further frequency bands of narrow range as in spectrograms, may have the effect of masking signals more effectively. Furthermore, fine subdivision of the frequency domain may reveal individual differences (as in Fig. 5), so that averages across subjects become less meaningful. These caveats notwithstanding, our current work focuses on spectrographic means of identifying imagined speech.

4 Discussion The syllables /ba/ and /ku/ have little semantic content in and of themselves, so that differences in EEG records underlying classification performance are unlikely to reflect semantic contributions to imagined speech production. Furthermore, each experimental trial starts with a period during which the cue to the condition is presented; cue recognition and response planning occur during the cue period and are likely absent during the following period in which EEG recording is made of imagined speech. The EEG recordings analyzed here concern most likely immediate aspects of imagined speech production based on readout from working memory.

46

M. D’Zmura et al.

EEG electrodes record not only signals from brain cortex but also electromyographic activity. Common sources of the latter are muscles responsible for eye and head movements and other muscles near recording electrodes, like those in the temples and neck. Electrodes atop or close to such muscles are most sensitive to electromyographic activity. The most flagrant such offenders were excluded from analysis in the work reported here. Furthermore, recordings for which EEG signals (detrended and filtered to excluded 60Hz line noise) exceeded 30 μV in absolute value were excluded from consideration. This threshold eliminated many (but not all) recordings contaminated with artifact. The most informative electrodes are those positioned on the scalp near the top of the head (Fig. 3), and this suggests that the information is due more to cortical than electromyographic activity. Subjects were instructed to think the imagined speech without any vocal or subvocal activity: without moving any muscles involved in producing overt speech. This contrasts with what is the most successful method yet for recognizing speech produced without any acoustic signal: surface electromyographic (SEMG) recordings from face and throat muscles. Substantial progress has been made in automatic speech recognition from surface EMG recordings of vocal tract articulators made during both spoken speech [5-10] and subvocalized speech [11-17]. Further non-acoustic methods for extracting imagined speech information potentially include magnetoencephalography (MEG) and invasive methods, like electrocorticography (ECoG), which involve implanted electrodes. While there are no published positive results yet for classification of imagined speech using MEG, its high temporal resolution and moderate spatial resolution (comparable to high-density EEG) suggest its value, as does success in classifying heard speech [18-21]. MEG is most sensitive to cortical signals from surfaces perpendicular to the scalp (in sulci) while EEG is most sensitive to signals from parallel surfaces (gyri) [22], so that the two methods are complementary. While the use of electrodes implanted in the brain is limited to small clinical populations for which these electrodes are indicated in combination with neurosurgery, it can provide very useful information concerning imagined speech [23]. Functional magnetic resonance imaging (fMRI) studies have contributed significantly to our neuroscientific knowledge of language perception and production [24-27]. Yet the poor temporal resolution of fMRI and of functional near-infrared imaging (fNIR) make these unlikely candidates for speech recognition. EEG is a non-invasive, non-injurious method for probing cortical activity which has a high temporal resolution, moderate spatial resolution (~2cm), a relatively low cost and which is increasingly portable. We feel that its greatest contribution as a braincomputer interface is likely to be found in circumstances where a human subject is trained to produce activity discernible by EEG [28]. Such training can reduce the variability associated with imagined speech production and also help one emphasize aspects of imagined speech which are most easily discerned by EEG. Our next step is to close the loop by providing real-time feedback concerning imagined speech signals.

Acknowledgements We thank David Poeppel for suggesting the value of an experiment like this. This work was supported by ARO 54228-LS-MUR.

Toward EEG Sensing of Imagined Speech

47

References 1. Dewan, E.M.: Occipital alpha rhythm eye position and lens accommodation. Nature 214, 975–977 (1967) 2. Farwell, L.A., Donchin, D.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogy and Clinical Neurophysiology 70, 510–523 (1988) 3. Suppes, P., Lu, Z.-L., Han, B.: Brain wave recognition of words. Proceedings of the National Academy of Science USA 94, 14965–14969 (1997) 4. Suppes, P., Han, B., Lu, Z.-L.: Brain wave recognition of sentences. Proceedings of the National Academy of Science USA 95, 15861–15866 (1998) 5. Morse, M.S., O’Brien, E.M.: Research summary of a scheme to ascertain the availability of speech information in the myoelectric signals of neck and head muscles using surface electrodes. Computers in Biology and Medicine 16, 399–410 (1986) 6. Chan, A.D.C., Englehart, K., Hudgins, B., Lovely, D.F.: Hidden Markov model classification of myoelectric signals in speech. IEEE Engineering in Medicine and Biology 21, 143– 146 (2002a) 7. Chan, A.D.C., Englehart, K., Hudgins, B., Lovely, D.F.: Multiexpert automatic speech recognition using acoustic and myoelectric signals. IEEE Transactions on Biomedical Engineering 53, 676–685 (2002b) 8. Bu, N., Tsuji, T., Arita, J., Ohga, M.: Phoneme classification for speech synthesizer using differential EMG signals between muscles. In: Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, pp. 5962–5966 (2005) 9. Jou, S.-C., Maier-Hein, L., Schultz, T., Waibel, A.: Articulatory feature classification using surface electromyography. In: Acoustics, Speech and Signal Processing, ICASSP 2006 Proceedings, pp. I–605–I–608 (2006) 10. Jou, S.-C., Schultz, T., Walliczek, M., Kraft, F., Waibel, A.: Towards continuous speech recognition using surface electromyography. In: Interspeech 2006 – ICSLP, pp. 573–576 (2006) 11. Jorgensen, C., Lee, D.D., Agabon, S.: Sub-auditory speech recognition based on EMG signals. In: Proceedings of the International Joint Conference on Neural Networks, Portland Oregon (July 2003) 12. Betts, B.J., Jorgensen, C.: Small vocabulary recognition using surface electromyography in an acoustically harsh environment. NASA/TM-2005-213471 (2005) 13. Maier-Hein, L.: Speech recognition using surface electromyography. Diplomarbeit, Universität Karlsruhe (2005) 14. Maier-Hein, L., Metze, F., Schultz, T., Waibel, A.: Session independent non-audible speech recognition using surface electromyography. In: 2005 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 331–336 (2005) 15. Jorgensen, C., Binsted, K.: Web browser control using EMG based sub vocal speech recognition. In: Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS 2005), pp. 294–301 (2005) 16. Binsted, K., Jorgensen, C.: Sub-auditory speech recognition. In: HICSS (2006) 17. Walliczek, M., Kraft, F., Jou, S.-C., Schultz, T., Waibel, A.: Sub-word unit based nonaudible speech recognition using surface electromyography. In: Interspeech 2006 - ICSLP, pp. 1487–1490 (2006) 18. Numminen, J., Curio, G.: Differential effects of overt, covert and replayed speech on vowel-evoked responses of the human auditory cortex. Neuroscience Letters 272(1), 29–32 (1999)

48

M. D’Zmura et al.

19. Ahissar, E., Nagarajan, S., Ahissar, M., Protopapas, A., Mahncke, H., Merzenich, M.M.: Speech comprehension is correlated with temporal response patterns recorded from auditory cortex. Proceedings of the National Academy of Sciences 98, 13367–13372 (2001) 20. Houde, J.F., Nagarajan, S.S., Sekihara, K., Merzenich, M.M.: Modulation of the auditory cortex during speech: an MEG study. Journal of Cognitive Neuroscience 15, 1125–1138 (2002) 21. Luo, H., Poeppel, D.: Phase patterns of neuronal responses reliably discriminate speech in human auditory cortex. Neuron 54, 1001–1010 (2007) 22. Nunez, P.L., Srinivasan, R.: Electric Fields of the Brain: The Neurophysics of EEG, 2nd edn. Oxford University Press, New York (2006) 23. Barras, C.: Brain implant helps stroke victim speak again. New Scientist (July 2008) 24. Indefrey, P., Levelt, W.J.M.: The spatial and temporal signatures of word production components. Cognition 92, 101–144 (2004) 25. Hickok, G., Poeppel, D.: Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition 92, 67–99 (2004) 26. Hickok, G., Poeppel, D.: The cortical organization of speech processing. Nature Reviews Neuroscience 8, 393–402 (2007) 27. Poeppel, D., Idsardi, W.M., van Wassenhove, V.: Speech perception at the interface of neurobiology and linguistics. Philosophical Transactions of the Royal Society B 363, 1071–1086 (2007) 28. Fetz, E.: Volitional control of neural activity: implications for brain-computer interfaces. Journal of Physiology 579, 571–579 (2007)

Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User Ying Gao, Armando Barreto, and Malek Adjouadi Department of Electrical and Computer Engineering Florida International University Miami, FL 33174 USA {ygao002,barretoa,adjouadi}@fiu.edu

Abstract. The pupil diameter (PD) has been found to respond to cognitive and emotional processes. However, the pupillary light reflex (PLR), is known to be the dominant factor in determining pupil size. In this paper, we attempt to minimize the PLR-driven component in the measured PD signal, through an ∞ Adaptive Interference Canceller (AIC), with the H time-varying (HITV) adaptive algorithm, so that the output of the AIC, the Modified Pupil Diameter (MPD), can be used as an indication of the pupillary affective response (PAR) after some post-processing. The results of this study confirm that the AIC with the HITV adaptive algorithm is able to minimize the PD changes caused by PLR to an acceptable level, to facilitate the affective assessment of a computer user through the resulting MPD signal. Keywords: Pupil diameter (PD), Pupillary light reflex (PLR), Pupillary affec∞ tive response (PAR), Adaptive Interference Canceller (AIC), H time-varying (HITV) adaptive algorithm.

1 Introduction Affective Computing, defined as “computing that relates to, arises from, or deliberately influences emotions” by Picard in 1997 [1] requires computers to have the ability to remain aware of their users’ affective states. Therefore, the affective assessment of the computer user has been considered as one of the key challenges to overcome for improvement of the relationship between human and computers. This challenge has been addressed, among other approaches, through the processing of a variety of physiological signals from the computer users, such as the Galvanic Skin Response (GSR), the Blood Volume Pulse (BVP), etc. We are currently studying the possibility of analyzing pupil diameter (PD) variations in the computer user for the detection of affective changes. The human pupil is the variable-size opening in the iris of the eye, through which light passes to the retina. The size of pupil is known to be controlled by two opposing sets of muscles in the iris, the sphincter and dilator pupillae, which are governed by the sympathetic and parasympathetic divisions of the Autonomic Nervous System (ANS) [2]. Traditionally, light intensity has been considered as the major determinant of the pupillary response, through the Pupillary Light Reflex (PLR). However, other J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 49–58, 2009. © Springer-Verlag Berlin Heidelberg 2009

50

Y. Gao, A. Barreto, and M. Adjouadi

psychological factors controlling pupil size, such as emotional processes have recently been investigated. For example, in 2003, Partala and Surakka found, using auditory emotional stimulation, that the pupil size variation can also be seen as an indication of affective processing during human-computer interaction [3]. Therefore, in this paper, we focus our study on monitoring and processing the pupil diameter (PD) signal for the affective assessment of a computer user. To achieve the required separation of the PLR-driven component from PD changes due to Pupillary Affective Responses (PAR) we propose an Adaptive Interference Canceller (AIC), which is known to be able to remove an unwanted interference component z(k) that pollutes a measured signal s(k) using an independent measurement of the interference n(k) [4] (See Figure 1). The core portion of this AIC system is an Adaptive Transversal Filter (ATF), which performs the adaptive algorithm to implement the function of interference canceling. To ensure a successful noise removal from the AIC, it is important to select an adaptive algorithm that possesses good robustness properties. The H∞ adaptive algorithm, introduced in robust control theory, is an attempt at addressing this need, with features that are useful in safeguarding against the worst-case of model uncertainties and makes no assumption on the (statistical) nature of the signals [5].

s (k )

d (k )

d (k )

z (k )

n( k ) d (k )

n( k )

r ( k ) = n( k )

r (k )

∧

∧

y (k ) = z (k )

Σ

s ( k ) = e( k )

u (k ) = 0

Fig. 1. Adaptive Interference Canceller (AIC) Block Diagram

Thus, we propose to use an H∞ –based adaptive technique, namely the H∞ timevarying (HITV) algorithm, in the Adaptive Interference Canceller for the removal of the PLR-driven component from the pupil diameter variation. Our intent is to use the output of the AIC, the Modified Pupil Diameter (MPD), further refined by additional processing (median evaluation of a sliding window), to indicate the occurrence of affective changes (e.g., onset of stress) in the computer user.

2 Methods 2.1 AIC Overview A basic AIC block diagram is illustrated in Figure 1, above. The signal of interest, polluted with an uncorrelated noise signal, is transmitted over the top channel, and constitutes the primary input to the AIC. The bottom channel receives a signal, which

Monitoring and Processing of the PD Signal

51

is uncorrelated with the signal of interest, but correlated with the interference, constituting the reference input. The output of AIC is expected to provide a signal where the correlated interference has been removed. As described in the previous paragraph, the key equations describing the AIC are: Primary Input: Reference Input:

Output:

d (k ) = s (k ) + z (k )

(1)

r ( k ) = n( k ) + u ( k )

(2)

e(k ) = d (k ) − y (k ) = d (k ) − zˆ (k ) = sˆ(k )

(3)

where, 9 9 9 9

s(k) is the signal of interest (PD- driven by PAR); z(k) is the interference in the primary sensor (PD- driven by PLR); n(k) is the actual source of the interference (Illumination changes); u(k) is the measurement noise. (In this study, u(k) is assumed to be zero);

In the AIC system, the core element is an adaptive transversal filter (ATF), where the reference signal r(k) is processed to produce an output y (k ) = zˆ (k ) that is an approximation of z(k). The state space model of the ATF is given by [5]: w(k + 1) = w(k ) + Δw(k )

(4)

d (k ) = r (k ) w(k ) + v(k )

(5)

z (k ) = r (k ) w(k ) + υ (k )

(6)

T

T

v(k ) = s(k ) + υ (k )

(7)

In these equations, 9 9 9 9 9 9

w(k) = system state vector, is the ATF coefficient vector of size m × 1 (m is the order of ATF); d(k) = measurement sequence, is the observed pupil diameter(PD) signal. z(k) = sequence to be estimated Δw(k ) = process noise vector, represents the time variation of the ATF weights w(k). v(k) = measurement noise vector, includes s(k) (PD- driven by affective changes) and model uncertainties υ (k ) .

r (k ) = [rk , r( k −1) ,......, r( k − m +1) ]T is the interference vector of size m ×1

As shown in Figure 1, for this study we define the primary input of the AIC as the recorded pupil diameter signal, which is composed by the signal of interest s(k) (PD- driven by PAR) and the interference z(k) (PD- driven by PLR). Although, the reference input comprises both the actual source of the interference n(k) (actual illumination changes) and the measurement noise u(k), under the assumption that the measurement noise is negligible, an independent measurement of illumination in the neighborhood of the eye of the subject (IL) is used as the reference input. We expect

52

Y. Gao, A. Barreto, and M. Adjouadi

the adaptive transversal filter (ATF) to emulate the transformation of the illumination variations to pupil diameter changes, which would convert the noise n(k) into a closeenough replication of the PLR-driven components of PD (the output y(k)). Therefore, the error, e(k) (Modified Pupil Diameter), would be the estimation of the desired signal s(k), i.e., the PD variations due exclusively to affective changes (PAR). 2.2 H∞ Time-Varying Adaptive Algorithm

In this study, the adaptive algorithm we applied to the Adaptive Interference Canceller system is the H∞ time-varying (HITV) adaptive algorithm, which aims to remove the noise from the recorded signal by adaptively adjusting the impulse response of the ATF. The robustness of this HITV algorithm is derived from its minimization of the maximum energy gain from the disturbances to the estimation errors with the following solutions [6]: P% −1 (k ) = P −1 (k ) − ε g−2 r(k )rT (k )

(8)

~

g (k ) =

P(k )r(k ) ~

1 + rT (k ) P(k )r(k)

(9)

wˆ (k + 1) = wˆ (k ) + g (k )( y (k ) − rT (k ) wˆ (k ))

(10)

P(k + 1) = [ P -1 (k ) + (1- ε g-2 )r (k )r T (k )]-1 + γ 0

(11)

P (0) = η I = Π 0 , ϒ 0 = ρ I

(12)

Here, g ( k ) is the gain factor; ε g , η and ρ are positive constants. Note that ρ reflects a priori knowledge of how rapidly the state vector w(k ) varies with time, and η reflects the a priori knowledge of how reliable the initial estimate available for the state vector w(0) is.

3 Experiments 3.1 Subjects

We have collected data from twenty-two volunteer students who have completed the protocol described below. All the subjects reported to have normal color vision and had experience using computers. This paper presents results achieved through the processing methods proposed on a subset of the subject pool recorded to date. 3.2 Task

In order to observe the response of pupil diameter changes to affective stimuli, the “Stroop Color-Word Interference Test”, implemented as a flash program [7], was used to elicit mild mental stress in the participating subjects during the experiment.

Monitoring and Processing of the PD Signal

Fig. 2. Sample of Stroop Test Interface

53

Fig. 3. Stimuli schedule of the Stroop Test

In the test, a word presented to the subject designates a color that may or not match the font of the word. The subjects are instructed automatically by the program to select (out of 5 possible choices) the screen button that indicates the font color of the word presented, by clicking on it (An example, showing the appearance of the test interface is shown in Figure 2). The stimuli schedule of the test in the experiment is shown in the Figure 3. The complete protocol is composed of three consecutive sections. In each section, there were four segments. 9 9 9 9

‘IS’ – the Introductory Segment to let the subject get used to the task environment; ’C’ – the Congruent segment of the Stroop Test, in which the subject was asked to click the on-screen button naming the font color of a word that correctly spelled the font color being displayed; ‘IC’ – the Incongruent segment of the Stroop Test in which the subject was asked to click the on-screen button naming the font color of a word that spelled the name of a different color; ‘RS’ – is a Resting Segment to let the subject relax for some time.

The incongruent Stroop segments (IC) were expected to elicit mild mental stress in the subject, according to previous research found in the psychophysiological literature [8]. In contrast, the congruent Stroop segments (C) were expected to allow the subject to continue in a relaxed state. The binary numbers shown in Figure 3 represent the demultiplexed output of the stimulus generator, which is used to insert the corresponding values (1, 2, 3) in the event channel of the PD data file, and the corresponding time marks in the illumination measurement data, recorded simultaneously. 3.3 Instruments

In this study, a desk-mounted eye tracking system (TOBII T60) was used to measure the pupil diameter signals from both eyes of the subjects at 60 samples/sec. The average ((L+R)/2) was recorded as the “Measured PD” signal, which corresponds to d(k), in Figure 1. Simultaneously, the illumination intensity level present in the area around the eyes of the subjects was recorded by a system built for that purpose. This system is composed by a BS500B0F photo-diode (Sharp), placed on the forehead of the subject and connected to an amplification circuit to provide an analog output voltage that is proportional (~0.0043 v/Lux) to the illumination intensity level [9]. The “Measured IL” signal, r(k), shown in Figure 1 was finally provided by sampling the analog

54

Y. Gao, A. Barreto, and M. Adjouadi

output of the luminance meter at the frequency of 360 Hz with a Data Acquisition (DAQ) System (PCI-DAS6023 board from Measurement Computing Co.) 3.4 Procedure

In our experiments, participants were asked to remain seated in front of the TOBII screen, interacting with the “Stroop Test” for about 30 minutes, while wearing a head band with the photo-diode. During that time, all the normal lights in the room were kept on, but an additional level of illumination provided by a desk lamp placed above the eye level of the subject was switched ON and OFF, alternatively, at intervals not previously known by the subject, using a dimmer. This was done to repeatedly introduce passages of high and low illumination in the experiment, which would trigger the pupillary light reflex.

4 Results Before its application to the adaptive interference canceller, the recorded pupil diameter signal is pre-processed by a blink-removal algorithm implemented in MATLAB, which is able to: 1. Detect the PD data interruptions due to eye blinks (identified as a value of “4” in the validity code provided by the TOBII system); 2. Compensate the missing data by linear interpolation. 3. Filter out the blink responses through a low pass, 512th order FIR filter designed for a cutoff frequency 0.13Hz. Figure 4 illustrates the stages of the blink-removal process on data collected from subject 13. MEASURED PD SIGNAL 6 4 2 0 -2

0

0.5

1

1.5

2

2.5

3 4

x 10 VALIDITY CODES FROM BOTH EYES 6 4 2 0 -2

0

0.5

1

1.5

2

2.5

3 4

x 10 PD SIGNAL AFTER BLINK COMPENSATION 6 4 2 0 -2

0

0.5

1

1.5

2

2.5

3 4

x 10 PD SIGNAL AFTER LOW PASS FILTER 6 4 2 0 -2

C1 0

0.5

IC1 1

C2 1.5

IC2 2

C3 2.5

IC3 3 4

x 10

Fig. 4. PD Signal Before & After Blink-Removal

Monitoring and Processing of the PD Signal

55

The PD signal obtained after blink-removal is applied to the AIC system as the primary input signal d(k), and the reference input r(k) is the simultaneously measured illumination intensity level signal, which is down-sampled from 360Hz to 60 Hz to share the same sampling rate with the PD signal. A MATLAB program was created to apply the HITV adaptive algorithms for the ATF with 120 weights and the parameter settings of η = 0.001 , ε g = 2.0 as well as a time-varying parameter ρ , changed according to the IL value to enable the AIC system to have a quicker response when there is a sudden increase in IL. The output of the AIC, the MPD, is shown in the bottom plot of Figure 5. This signal is further processed as described below to become a useful indicator of pupillary affective response in the subject. AIC PRIMARY INPUT: PD SIGNAL 6 4 2

0

0.5

1

1.5

2

2.5

3 4

x 10 AIC REFERENCE INPUT: IL SIGNAL 2 1.5 1

0

0.5

1

1.5

2

2.5

3 4

x 10 -6

1

AIC TIME-VARING PARAMETER: RHU

x 10

0.5 0

0

0.5

1

1.5

2

2.5

3 4

x 10 AIC OUTPUT: MPD SIGNAL 2 0 -2

C1 0

0.5

IC1 1

C2 1.5

IC2 2

C3 2.5

IC3 3 4

x 10

Fig. 5. AIC Implementation with HITV Algorithm

The affective state of “Stress” is expected to cause a dilation of the pupil [2]. Therefore, the negative portions of the MPD signal, are zeroed to isolate significant MPD increases, which indicate the emergence of stress in the subject. The result of applying this non-negative restriction to the MPD signal is shown in the bottom panel of Figure 6.

56

Y. Gao, A. Barreto, and M. Adjouadi

A sliding window with a width of 1200 samples is applied throughout the nonnegative MPD signal to calculate the median value within each window. The effect of this process on both the original PD signal and the non-negative MPD signal is compared in Figure 7. In this figure, it is clear that the significant increases that have been isolated in the processed MPD signal correlate closely with the occurrence of IC segments, regardless of the presence of higher illumination passages during segments IC2 and C3. It should also be noted that the same post-processing operated on the PD signal obtained directly from the eye gaze tracking system does not set apart the IC segments as clearly. Sliding Window Median Analysis of PD

AIC Output: MPD Signal 2

6

1

5

0

4

-1

0

0.5

1

1.5

2

2.5

3

3

0

0.5

1

1.5

2

2.5

3

4

4

x 10

x 10

Non Negative MPD Signal

Sliding Window Median Analysis of MPD

2

1

1

.5

0

0 C1

-1

0

0.5

IC1 1

C2 1.5

IC2

C3

2

IC3

2.5

C1 .5

3

0

0.5

IC1 1

C2

IC2

1.5

C3 2

2.5

4

x 10

Fig. 6. MPD Signal Non Negative Processing for Stress Indication

Fig. 7. Comparison of Sliding Window Median Analysis on PD & MPD

Signal Processing Results of Subject 11 3.5 3 2.5 MPD 2 Non Negtive1.5 MPD 1 Sliding 0.5 Window Median MPD 0 C1 -0.5

0

0.5

IC1 1

3 4

x 10

GSR

IC3

C2 1.5

IC2 2

C3 2.5

IC3 3

3.5 4

x 10

Fig. 8. Signal Processing Results of Subject 11

Monitoring and Processing of the PD Signal

57

A similar result is observed for data collected from Subject 11, as shown in Figure 8, where the 3 top signals (GSR, which was recorded simultaneously, MPD and Nonnegative MPD) have been shifted up to make the display clear. In this case, also, the most significant increases of the bottom trace, after the initial adaptation that precedes segment C1, occur during the incongruent segments (IC1, IC2 and IC3). This figure shows the appearance of Skin Conductivity Responses (SCRs) and overall elevation of the GSR signal during the incongruent segments. Furthermore, the bottom plot (the result of the proposed post-processing) also shows significant increases during IC1, IC2 and IC3, with minimal output on the other segments after the initial adaptation.

5 Conclusions This study implemented an H∞ time-varying adaptive algorithm in an Adaptive Interference Canceller to discount the influence of Pupillary Light Reflex from a measured PD signal, so that the output of the AIC, the MPD, with the application of a non negative constraint and sliding window median analysis, can be used as an indicator of Pupillary Affective Responses due to, for example, subject stress. This indicates that this approach might be useful for the affective assessment of a computer user, even in the presence of illumination changes. A comparison of this result with the outcome obtained by applying the same post-processing directly to the recorded pupil diameter signal, points out the advantage of the adaptive implementation, which output relatively more distinctive increases when the incongruent Stroop segments occurred. Data from other subjects who participated in our experiment have revealed similar results. These outcomes, therefore, encourage the continued exploration of adaptive processing algorithms applied to pupil diameter signals, as a non-invasive mechanism to achieve affective assessment of computer users in ordinary environments. Acknowledgements. This work was sponsored by NSF grants CNS-0520811, HRD0833093 and CNS-0426125. Ms. Ying Gao is the recipient of a Dissertation Year Fellowship from Florida International University.

References 1. Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997) 2. Beatty, J., Lucero-Wagoner, B.: The Pupillary System. In: Cacioppo, Tassinary, Berntson (eds.) Handbook of Psychophysiology, 2nd edn., pp. 142–162. Cambridge University Press, Cambridge (2000) 3. Partala, T., Surakka, V.: Pupil size variation as an indication of affective processing. Int. J. of Human-Computer Studies 59, 185–198 (2003) 4. Widrow, B., Stearns, S.D.: Adaptive signal processing, pp. 337–339. Prentice-Hall, Englewood Cliffs (1985) 5. Hassibj, B., Kailath, T.: H∞ adaptive filtering. In: Proc. Int. Conf. Acoustics, Speech, Signal Processing, pp. 945–952 (1995) 6. Puthusserypady, S.: H∞ Adaptive filters for eye blink artifact minimization from electroencephalogram. IEEE Signal Processing Letters 12, 816–819 (2005)

58

Y. Gao, A. Barreto, and M. Adjouadi

7. Zhai, J., Barreto, A.: Significance of Pupil Diameter Measurements for the Assessment of Affective State in Computer Users. Biomedical Sciences Instrumentation 42, 495–500 (2006) 8. Renaud, P., Blondin, J.-P.: The stress of Stroop performance: physiological and emotional responses to color-word interference, task pacing, and pacing speed. Int. Journal of Psychophysiology. 27, 87–97 (1997) 9. Todoroki, A., Hana, N.: Luminance change method for cure monitoring of GFRP. Key Engineering Materials, 321–323, 1316–1321 (2006)

Usability Evaluation by Monitoring Physiological and Other Data Simultaneously with a Time-Resolution of Only a Few Seconds Károly Hercegfi, Márton Pászti, Sarolta Tóvölgyi, and Lajos Izsó Budapest University of Technology and Economics, Department of Ergonomics and Psychology, Egry J. u. 1, 1111 Budapest, Hungary {hercegfi,tim,ebediyen,izsolajos}@erg.bme.hu

Abstract. This paper outlines the INTERFACE methodology developed by researchers of our department. It is based on the simultaneous assessment of Heart Period Variability (HPV), Skin Conductance (SC), and other data. The objective and significance of this paper are (1) showing its capability of identifying quality attributes of software elements with a time-resolution of only a few seconds and (2) presenting its practical applicability in the evaluation phase of a real software development process. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics carried out a contract-based applied research project for the GeneraliProvidencia Insurance Co. Ltd. The Company was in the process of further developing the software used in its customer centers, and our Department contracted to assess the user interface. Both analytical and empirical usability evaluation methods were applied. In this paper, we highlight the new experiences of the INTERFACE testing methodology. Keywords: Usability testing and evaluation, empirical methods, case study, Heart Period Variability (HPV), Skin Conductance (SC).

1 The Frame of the Project During the last decades, the Department of Ergonomics and Psychology at the Budapest University of Technology and Economics developed mutually useful relationships with some industrial partners successfully accomplishing various Research & Development (R&D) projects 4. The Genarali-Providencia Insurance Co. Ltd. as a member of the Generali Group is owned by the Generali PPF Holding that has 9 million customers in Central and Eastern Europe. From these 9 million customers, 1.2 million are given by the Genarali-Providencia Insurance Co. Ltd. Two kinds of different core systems are used by the Genarali-Providencia Insurance Co. Ltd. to keep their data: one for the life insurance (it is called SYNPAC), and another (it is called VIAS) for the non-life insurance, e.g. liability insurance, Casco insurance, etc. The management started to establish a new system that can visualize the data of both core systems. To carry it out, the IT department of the company started to develop a new frame system called Genesys. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 59–68, 2009. © Springer-Verlag Berlin Heidelberg 2009

60

K. Hercegfi et al.

This software is used by approximately 1500 users. From these 1500 users, 300 users use it daily. The main part of the users work in personal customer centers. One smaller part of the users work as call center operators. It is a smaller group of the users, however, the time pressure in their job underlines the importance of the usability factors. The Department of Ergonomics and Psychology at the Budapest University of Technology and Economics latched on to the software development to carry out the usability assessment. The main goal of our project was to collect problems of the user interface (UI). From these misses we made a list that we gave it to the developers. Developing solutions for the found problems can be a next project. Other goal of the Company was supporting a scientifically interesting research project transferring results of foundational researches to really applicable methodology in real software development process.

2 Applied Methods The preparation of the project started in November 2007. The contract was signed in March 2008. We got in touch with the developers at the beginning of April 2008. We started our project with studying the software and collecting data and facts. While we were studying it, the system was updated, so sometimes it was difficult to be up to date. In the 2nd part of April we started to observe the users of the software, and we also made interviews with them. We performed the observation and interviews at 3 different locations: in the biggest customer service of the firm, in another type, smaller customer service, and in a call center. We obtained some logfile data to establish objective initial data for the subsequent assessment. We applied analytical methods (usability inspection methods) in May and June 2008. The backbone of the analytical evaluation was a Guideline Review, supported by Cognitive Walkthrough elements. Also GOMS model-based analysis was carried out. The main part of the assessment was a carefully planned, deep empirical series of experiments applying the INTERFACE methodology described in the following section. The series of experiment were carried out in July 2008; the analysis of the collected records was performed in August 2008.

3 Description of Our Main Methodology: The INTERFACE Figure 1 shows the conceptual arrangement of the INTERFACE (INTegrated Evaluation and Research Facilities for Assessing Computer-users' Efficiency) workstation.

Usability Evaluation by Monitoring Physiological and Other Data

61

observable behavior current screen content

data collecting and

keystrokes and mouse clicks

processing frame system

physiological signals by ISAX

Fig. 1. Conceptual arrangement of the INTERFACE user interface testing workstation

The advantage of the methodology applied in our study lies in its capability of recording continuous on-line data characterizing the user’s current mental effort derived from Heart Period Variability (HPV) and the user’s emotional state indicated by Skin Conductance (SC) parameters simultaneously and synchronized with other characteristics of Human-Computer Interaction (HCI). This way, a very detailed picture can be obtained which serves as a reliable basis for the deeper understanding and interpretation of psychological mechanisms underlying HCI. Elementary steps of HCI, like the different mental actions of users followed by a series of keystrokes and mouse-clicks, are the basic and usually critical components of using software. These steps can be modeled and analyzed by experts, but empirical studies of real users’ interactions often highlight new HCI issues or give more objective results than expert analyses. One of the key aspects of the empirical methods is measuring mental effort as it is laid down e.g. in the earlier international standard of software product evaluation (ISO/IEC 9126:1991). Hence we need methods capable of monitoring users’ current mental effort during these elementary steps. To attain the above, a complex methodology was developed earlier at the Budapest University of Technology and Economics, by Prof. Lajos Izsó and his team [3, 4, 5]. This study presents an improved methodology and a new case study. The INTERFACE simultaneously investigates the following: • Users' observable actions and behavior − keystroke and mouse events; − video record of the current screen content; − video records of users’ behavior: (1) mimics, (2) posture and gestures. • Psycho-physiological parameters − Power spectrum of Heart Period Variability (HPV), regarded as an objective measure of current mental effort – we apply this signal successfully since more than 15 years [1, 2, 3, 4]; − Skin Conductance (SC) parameters, indicating mainly the emotional reactions – recently integrated into our system.

62

K. Hercegfi et al.

A number of studies [1, 2, 3, 5, 7, 8, 9, 10] have shown that an increase in mental effort causes a decrease in the mid-frequency (MF) peak of the HPV power spectrum. The main advantage of the assessment method of the spectral components integrated into our system over the previously existing HPV-based methods is that the MF component of HPV shows changes in mental effort in the time range of several seconds (as opposed to the earlier methods with a resolution of tens of seconds at the best). This feature was achieved by an appropriate windowing data processing technique, and application of an all-pole auto-regressive model with built-in recursive Akaike's Final Prediction Error criteria and a modified Burg’s algorithm. We watch the Alternating Current (AC) component of the Skin Conductance (SC) responses focusing mainly on the emotional aspects of the HCI, in addition to our well-tried approach of mental effort. An interesting series of experiments analyzing SC responses is finished by one of our colleagues [6]. It is a good example of the promising way to use data mining techniques in empirical usability studies. Participant, operator of the Computer used Camera to Display Standard call center with standard by the record the with IP headset regularly used in the participant – facial the phone call center. The Skin according to expressions. software of the Conductance (SC) electrodes can the standard Motorized face currently call tested center tracking, be seen on the left hand, the workstation of zoomable ECG electrodes are on the torso the call center

The ISAX equipment to record the physiological signals

Camera to record the body posture

Computer of the experimenter – during the session, online curves of the physiological signals, video images of the cameras and the editor window for the comments can be seen

Fig. 2. The experimental arrangement applied during the sessions of the INTERFACE usability testing installed on a standard workstation of the call center

Usability Evaluation by Monitoring Physiological and Other Data

63

4 Applying INTERFACE in the Current Series of Experiments The empirical sequence of experiments applying the previously introduced INTERFACE assessment was performed with the call center operators in the middle of July 2008. Due to the impersonality caused by the phone calls, on one hand the simulation was more authentic than it could be in the personal customer centers. On the other hand, because of the specialty of the call center, we could count on concentrated, quick problem-solving. What more, the usability problems are the most critical in the call centers due to the time pressure. The tasks occurred in the call center are usually not solved only by means of the Genesys, but these other software are not being examined in this project. However,

The screen just seen by the user Two cameras: Facial expression and body posture

Keyboard and mouse actions

Experimenter’s comments

Upper (blue) curve: AC component of the Skin Conductance (SC). The higher deviation means more emotional event.

Signals derived from the ECG, related to mental effort. Red RR curve in the middle: periods between the subsequent heart beats in ms. Last (green) profile curve: the MidFrequency (MF) power of the variability of the RR curve. Its low values mean significant mental effort; peaks mean relief, relaxation.

Fig. 3. The INTERFACE Viewer screen with a record of the empirical test of the Genesys software. As it can clearly be seen, currently the user makes significant mental effort – it is shown by the facial expression and gesture, and the low value of the last green profile curve of the Mid-Frequency (MF) power of the Heart Rate Variability (HRV) at the cross-hair.

64

K. Hercegfi et al.

we had to give controlled, simulated tasks to the users in order that they can solve them in the Genesys system. These simulated tasks helped us to be able to compare the 12 sessions. We used the version of the Genesys from the test server that is substantially equal to the version actually used in real. 12 real operators were involved as participants, each of whom we recorded a onehour-long session with. The quantity of data gained from these sessions is really significant according to the depth of the enquiry. Due to the real life situation the users participating the series of experiments, were more or less disturbed by their colleagues’ calls or talks. Since these are the employees’ real conditions, they have got used to them, and so they could work on typical, real workstations of their workplace. These advantages gave reason not to hold this usability testing in laboratories. Nevertheless, we chose a workstation that was located in the corner of the operators’ room, in order not to disturb the others. Behind this workstation, the team leader’s glass wall can be found, and so our staff could sit and make simulated phone calls from behind this pane. As mentioned before, during the recorded sessions, the users were given tasks from the real life, with quasi-real data, names, problems, questions, etc.; the main difference was that the customers were not real customers, but from our staff. Three ECG electrodes were put on the users’ torso and one on the left hand (in case of a right-handed person) for measuring Skin Conductance. After that the users put on their headsets and adjusted their seats. Figure 2 shows the experimental arrangement applied during the sessions of the INTERFACE usability testing installed on a standard workstation of the call center. Figure 3 shows the INTERFACE Viewer screen with a record of the empirical test of the Genesys software. At the beginning of the session, we asked the users to relax for two minutes. We told them the aim and the details of the assessment. We always emphasized that we are not willing to assess they themselves, but the Genesys software by means of their help. Relaxation was followed by two-minute mental effort: mental arithmetic. The result of the counting was not important; we only wanted to generate mental effort. These periods were planned for “calibrating” the physiological curves. After that, the hard work started. The first customer (from our staff) rang the phone and asked some questions. To answer these questions the operator had to use the software under testing. Later 4 more similar calls came. Each call contained 2 to 4 questions. One part of the questions were just to “warm up”, others were really difficult. The subtasks were based on the interviews, observations and expert analyses performed earlier. The last part of the INTERFACE session was an interview.

5 Validation As it was mentioned, the periods of relaxation and mental arithmetic were planned for “calibrating” the physiological curves. The curves shown in the upper part of the Figure 4 were recorded during session #11, the ones shown at the bottom were recorded during session #10.

Usability Evaluation by Monitoring Physiological and Other Data

relaxation

65

mental arithm.

relief

relaxation

mental arithm.

relief

Fig. 4. The typical pattern of the relaxation and mental arithmetic in cases of two participants

In both cases, the three curves are the blue curves of the AC of Skin Conductance (SC), the red RR curves (Heart Periods), and the green profile curves of the MidFrequency (MF) power of Heart Rate Variability (HPV). The blue curve of AC of SC is relatively smooth during both the relaxation and the mental arithmetic. During these sections, there are not any emotional peaks, and these two participants can be characterized as “stabile” type according to the typology of physiology. However, the beginnings and the ends of the sections are followed by peaks. During relaxation, the MF component of the HPV increases, then the red RR curve has zigzags, and the green profile curve is relatively high. (In case of perfect relaxation, the profile curve should be consistently high. However, this is not expected in this experimental situation. The curve can be considered as high, especially in comparison with the next section.) During mental arithmetic, the red RR curve gets smoother, and the green profile curve is significantly low. After the “calibration” tasks, the participants really relieve. During this short period of relief, the participants get more relaxed than during the conscious, intended relaxation: the green curves have their highest peaks here.

66

K. Hercegfi et al.

Mean of MF power of HPV [ms2]

These “calibration” tasks prove a validation of our method. The values of the MF power of the HPV were significantly higher during relaxation than during mental arithmetic. A non-parametric statistical method, the Wilcoxon Signed Ranks Test proves the difference (sig. 0.037 – Figure 5).

100,0

80,0

60,0

40,0

20,0

0,0 relaxation

mental arithm.

Fig. 5. Validation of measuring Mid-Frequency (MF) power of Heart Rate Variability (HPV) as an indicator of mental effort: the MF power of HPV was significantly higher during relaxation than during mental arithmetic (sig.0.037)

It is a significant difference, in spite of the non-perfect relaxation. However, the mental arithmetic task works better: the significance of the difference between the values of MF power of HPV during mental arithmetic and in general, during the whole software usage section is better: the Wilcoxon test results sig. 0.002. The values of the deviation of the AC component of Skin Conductance (SC) do not differ during the relaxation and the mental arithmetic significantly. As it was described earlier, this is the expected result. However, the deviations of the AC of SC during the relaxation and the mental arithmetic are significantly lower than in general, during the whole software usage section: the Wilcoxon tests result sig. 0.009 and sig. 0.017. After these results, we can say that the low value of the curve of the MF power of HPV really means mental effort, and the high deviation of AC of SC probably means higher emotions. Than, in the section of software usage, we look for moments with relatively high (and unwanted) mental efforts and high (and unwanted and not positive) emotions. This method gives us the key to find the problems of the UI.

6 A Sample UI Problem Identified by the INTERFACE Methodology Commercial sensitivities prevent publication of the most of the details of the particular software problems found. However, Figure 6 gives an illustration.

Usability Evaluation by Monitoring Physiological and Other Data

67

Fig. 6. The 11th participant during the first task of the second call. The mental effort can clearly be seen: it is shown by the facial expression and gesture, and the low value of the last green profile curve of the Mid-Frequency (MF) power of the Heart Rate Variability (HRV) at the cross-hair. In this case, the problem was caused by a bad design solution to ensure choosing a time period for a list view. This problem can also be found by analytical methods – but the INTERFACE highlighted this problem and gave objective evidence for it.

7 Conclusion Based on the results presented here as well as in related papers, it can be stated that the INTERFACE methodology in its present form is capable of identifying the relative weak points of the HCI. By this methodology and the related workstation, it was possible to study events occurring during the HCI in such detail and objectivity that would not have been possible using other methods presently known to us. The sophisticated Heart Period Variance (HPV) profile function integrated into the INTERFACE system is a powerful tool for monitoring events in such a narrow time frame that it can practically be considered as a time-continuous recording of relevant elementary events. Measuring the Skin Conductance (SC) is a new opportunity to modulate the results. Acknowledgements. The authors would like to thank Dr. Eszter Láng for the earlier developments, the Generali-Providencia Insurrance Co. Ltd. for the support of the deep research, and the participants of the series of experiments for their valuable contribution.

68

K. Hercegfi et al.

References 1. Chen, D., Vertegaal, R.: Using Mental Load for Managing Interruptions in Physiologically Attentive User Interfaces. In: Proc. CHI 2004, pp. 1513–1516. ACM Press, New York (2004) 2. Hercegfi, K., Kiss, O.E., Bali, K., Izsó, L.: INTERFACE: Assessment of HumanComputer Interaction by Monitoring Physiological and Other Data with a Time-Resolution of Only a Few Seconds. In: Proc. ECIS 2006, ECIS Standing Comm., pp. 2288–2299 (2006) 3. Izsó, L.: Developing Evaluation Methodologies for Human-computer Interaction. Delft University Press, Delft, The Netherlands (2001) 4. Izsó, L., Hercegfi, K.: HCI Group of the Department of Ergonomics and Psychology at the Budapest University of Technology and Economics. In: Ext. Abstracts CHI 2004, pp. 1077–1078. ACM Press, New York (2004) 5. Izsó, L., Láng, E.: Heart Period Variability as Mental Effort Monitor in Human Computer Interaction. Behaviour Information Technology 19(4), 297–306 (2000) 6. Laufer, L., Németh, B.: Predicting User Action from Skin Conductance. In: Proc. IUI 2008, pp. 357–360. ACM Press, New York (2008) 7. Lin, T., Imamiya, A.: Evaluating Usability Based on Multimodal Information: An Empirical Study. In: Proc. ICMI 2006, pp. 364–371. ACM Press, New York (2006) 8. Mulder, G., Mulder-Hajonides van der Meulen, W.R.E.H.: Mental Load and the Measurement of Heart Rate Variability. Ergonomics 16, 69–83 (1973) 9. Orsilia, R., Virtanen, M., Luukkaala, T., Tarvainen, M., Karjalainen, P., Viik, J., Savinainen, M., Nygard, C.-H.: Perceived Mental Stress and Reactions in Heart Rate Variability – A Pilot Study Among Employees of an Electronics Company. International Journal of Occupational Safety and Ergonomics (JOSE) 14(3), 275–283 (2008) 10. Rowe, D.W., Sibert, J., Irwin, D.: Heart Rate Variability: Indicator of User State as an Aid to Human-Computer Interaction. In: Proc. CHI 1998, pp. 18–23. ACM Press, New York (1998)

Study of Human Anxiety on the Internet Santosh Kumar Kalwar and Kari Heikkinen Lappeenranata University of Technology, Department of Information Technology, Lappeenranata, Finland [email protected], [email protected]

Abstract. In this paper a conceptualization of human anxiety on the Internet is introduced; it is built on the understanding of human behavior with regard to technology. The objective of this paper is to conceptualize the human anxiety. An integral part of understanding is an inter-disciplinary (psychology science, cognitive science, behavioral science and communication technology) literature review, of which and overall summary is presented. The understanding is conceptualized by designing, implementing and evaluating through a developed user study model. In this paper the preliminary result of utilizing the developed user study found seven particular anxiety areas which need further studies. Keywords: Human, study, anxiety, internet.

1 Introduction Twenty-seven years ago, Internet was like a small dog at the bottom of the application pile, fighting for recognition, today, vast numbers of people are using Web Services through the Internet. One of the fascinating things about the Internet is that it comes with outstanding tools for usage in regards to recreation, education, society and much more. Current estimated statistics have shown that more than 63 million humans use the Internet in homes and the importance of the Internet in daily activities is growing. There is little which cannot be accomplished from the comfort zone of home; paying bills, research work, buying, selling, updating, uploading, downloading, shopping and, most importantly, communicating with your family and friends half way across the world. Since the establishment of the WWW, the number of Internet users has grown from an estimated 16 million in 1995 to more than 500 million in 2002 – explosive growth to say the least [1]. The rather recent but enormous and rapidly progressing emergence of the Internet has not gone unnoticed by scholars working in many different fields of research [1]. The Internet has become both an object of study and a tool of research. A basic definition of the term host in the case of a computer network is a computer connected to the Internet. The number of hosts connected to the Internet has grown rapidly. In 1983, when the Transmission Control Protocol/Internet Protocol (TCP/IP) standard was first adopted there were only about 250 hosts connected to the Internet. By late 1993, figure had grown to two million. During 2005, the number of hosts on the Internet had increased to 317.6 million. In recent years, the number has J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 69–76, 2009. © Springer-Verlag Berlin Heidelberg 2009

70

S.K. Kalwar and K. Heikkinen

skyrocketed to 541 million [2]. The fact and digital data from various sources confirms that number of humans accessing the Internet is growing at a rapid pace. As the Internet evolves in terms of number of human online, it feels as if it is evolving as social community. Worldwide, the Internet population is growing at a rapid pace. The number of people getting access to information, learning, and going online is booming like never before. It should be remembered that as late 1988, only a few countries were connected to the Internet. According to the 2004 CIA World Fact book, over 50 countries have at least one million humans using the Internet [3]. Humans once found it difficult and expensive to communicate during the times of voice telephones but with the rapid technological development communication has improved drastically. The distance has been shortened from family, friends and from seeking information which can be seen as the replacement of very important daily interactions. Boundaries of time, distance and identity are broken by the transfer of simple applications like e-mail to the complex world of virtual communities. Together with the positive growth, its negative effects are growing too. According to the U.S. Department of Justice, the Internet is an anonymous and effective way for many predators to find and groom children for illegal activities [4]. The fear of using the internet is further amplified by social disintegration, psychological and cognitive implications. 1.1 Objective and Scope The main research question is, “how can we address the challenges such as Internet addiction, psychology and human computer interaction it is currently facing now?” To understand the objectives, some of the questions were researched in details. The paper aims to find answers to the following hypotheses: • • • • •

Do users shows increased or reduced anxiety level when using the Internet? What kinds of behaviors are shown when using the Internet? What is the role of the content? Finding types of the anxiety behaviors? How human process information at the internet interface?

The scope of the work includes design, implement and evaluate methods used in understanding the behavior of humans using the Internet technology. It has wide range of scope from the field of psychological perspective to cognitive science, behavioral science and communication technology. The scope is complex. However, To study human and their tasks and how to relate information to design style, human behavior theories, standards, procedures or guidelines in order to build an appropriate model of interaction with the help of some existing methods is investigated. 1.2 Internet Anxiety A lot has been written in past about the negative use of the Internet anxiety, Internet addiction and full dependence on the internet is welling up [5] [6] [7]. The service disruption because of network faults, software bugs, administrator mistakes and version upgrade could seem less tolerable. Millions of human around the world use Internet to search, inform, find, communicate, work and play. Internet should not be

Study of Human Anxiety on the Internet

71

only viewed as negative such as addiction and pathological nor should it be vilified. One must be aware of negative consequences of overuse of the Internet by understanding the behavior of themselves and from others. Four types of Internet anxiety was identified by Presno. C,, using qualitative study method [8]. These are: Internet terminology anxiety: anxiety produced by an introduction to a host of new vocabulary words and acronyms. Net search anxiety: anxiety produced by searching for information in a maze-like cyberspace. Internet time delay anxiety: anxiety produced by busy signals, time delays, and more and more people clogging the Internet. General fear of Internet failure: a generalized anxiety produced by fear that one will be unable to negotiate the Internet, or complete required work on the Internet. Additional three areas of the Internet anxiety from the qualitative study were found in this research. Experience anxiety: an anxiety produced by lack of concentration or focus. Usage anxiety: a generalized anxiety produced by excessive usage of the Internet. Environment and attraction anxiety: anxiety produced by content on the Internet. For example: interactive game, pornography and larger number of colorful applications. Table 1. Types of Anxiety recorded for Subject I and Subject II Types of anxiety Terminology Search Time delay General fear Experience Usage Environment & attraction

p1

p2

p3

x x x x x

x x x x x

p4 x x x x x x

p5

p1 x

x x

p2 x

p3 x x

x x

x

x x x

x x x

x x

x x x x

p4

p5

x x x

x x x x x

x

2 Inter-disciplinary Literature Review Sociability is important as well as usability of applications in the Internet. While usability is concerned with making sure that the application, software and system is consistent, predictable, and easy and satisfying to use, sociability and the social aspect of building and maintaining an online community focuses on processes and styles of social aspect in interaction that support human behavior on the internet to some extent. Research has shown certain social groups to be under-represented on the Internet [9] [10] not simply because of a lack of access, but more because of cognitive, motivational and affective factors [11]. Psychology therefore has an important role in advancing the understanding of why humans choose to use or not to choose the usage of the Internet [12]. There will always be an argument to model psychology with technology or technology with psychology however; combining psychology with technology will give rise to new technology called psycho technology. To understand the Internet technology in broader ways, interaction between human and the technology through the Human Computer Interaction becomes essential. Brain Computer

72

S.K. Kalwar and K. Heikkinen

Interface techniques are only studied in this research work, no practical implementation was carried out due to the limitation of resources. In BCI, skill developed by a human involves proper control of electro physiological signals which are easily adapted and modulated by the brain for better feedback. In understanding human anxiety, BCI techniques could be used for testing and analyzing different activities on the Internet. The ethnographic study of the Internet can be divided into two categories. First, user-based and second, content based. User-based analysis is about the investigation, examination and the study of humans using the Internet. Whereas, content based analysis is mainly focused on text. Humans are capable of providing reasons to support their points of view if asked a question such as “What color is my shirt?” and are capable of knowing without explicit Deduction or reasoning answers to questions like, “If I were you, I would hate myself”, whereas computers cannot function without specific programming instructions.

3 Methods, Design and Implementation The user study model used both qualitative and quantitative research methods. The qualitative research was conducted using interviews and observational analysis. The quantitative research was conducted in three iterations by using questionnaires and surveys. The Questions were analyzed with the help of different types of questionnaires being constructed. Intentionally, Participants were given very general types of question to answer. Two types of Questionnaires were used: Using the Internet and Using the Pen and Paper method. Task calculation was carried out by dividing the task into subtask. Two types of subject were categorized based on skill level (novice, intermediate and expert human): Subject I and Subject II. Usability Test recording form was used for each participant. This form was used to record both the verbal and non-verbal behaviors of humans using the Internet. The task was chosen which was based on the

Fig. 1. The user study model implemented to test behavior of humans using the Internet

Study of Human Anxiety on the Internet

73

Internet. The task contained three modules which were divided into task 1, task 2 and task 3 based on the level of difficulties. It was found from the task analysis that humans using the Internet took less than a second to complete all of these tasks. There is a division of the participant’s overt behavior into two major general categories: verbal and nonverbal. Verbal behavior includes anything a participant says and Nonverbal behaviors include various activities that the participants actually do. In non verbal behavior mostly facial expression such as smiling, looks of surprise, furrowing brow or showing body gestures such as leaning close to the screen, rubbing the head is shown by the participants these are facial expression, eye-tracking, pupil diameter, skin conductance among others.

4 Results and Discussion The goal of the study was to determine “how can we address the challenges such as Internet addiction, psychology and human computer interaction it is currently facing now?” In order to evaluate the main research question, the main research question was broken down into hypothesis. There were five different hypothesis formulated in the beginning of the research. Now, Let us try to discuss these hypotheses to see our method, design, evaluation and analysis of the research was supported (fully supported, partially supported and not supported) or not. H1: Do users shows increased or reduced anxiety level when using the Internet? Hypothesis 1 was fully supported. Human shows the sign of increased level of anxiety when using the Internet. It appears that, with any given task on the Internet number of anxiety increase in human. More number of participants said “yes” to five or more items from the QS 2, which indicates problematic Internet Usage. Using HADS and PHQ-9 it was observed that, one participant seemed to have a Case of higher depression scale. Therefore, in this particular case users showed increased anxiety level when using the Internet. H2: What kinds of behaviors are shown when using the Internet? Hypothesis 2 was fully supported. Literature review revealed that there are two types of behavior shown by human using the Internet: Verbal and Non verbal. During Observation of behavior for the participants, Most of the times humans were laughing, smiling, drumming their fingers on the table and looking aimlessly around. These behaviors pattern was verbal and non-verbal. These types of the behavior patterns were observed among narrowly selected group of the participants. H3: What is the role of the content? Hypothesis 3 was partially supported. Role of content could determine the predicted or unpredicted human behavior on the Internet. Such as addiction, anxiety and stress of using the Internet, since humans were successfully able to complete the task with ease, it could be predicted that-any sort of given task for humans is very easy to do on the Internet. Therefore, the role of content has principal impact on how human behaves on the Internet.

74

S.K. Kalwar and K. Heikkinen

H4: Finding types of the anxiety behaviors? Hypothesis 4 was fully supported. It was found that in this particular case there are seven main area of anxiety in the humans: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Using Observation methodology and Comparing two types of subjects: Subject I and Subject II, it concludes that- all the participants showed these above cited anxiety. H5: How human process information at the internet interface? Hypothesis 5 was partially supported. When human interacts with the internet interface, it appears that, everything that human senses such as sight, hearing, touch, smell and taste are processed as the information in the mind. This information could result in behavior such as verbal and non-verbal. Even if behavior initially disappears, it may partially return as undamaged parts of the brain reorganize their linkages. The human behavior in totality of processing information includes internal cognitive processes which can result in observable behavior. Processing of the information at the internet interface has the realistic approach such as thinking of mental processes as several railroad lines that all feed to same terminal. Two schools of thought have emerged which confirms with the hypothesis that, there are two types of the behavior while using the Internet: Verbal and Non-Verbal behavior. In more general terms human using the Internet can use the content available on the Internet in two different ways: Positive or Negative. The gestures or types of human behavior shown could lead to anxiety. Seven major types of anxiety were studied and validated: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, and general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Two types of behavior (verbal and non-verbal) were formulated from relevant literature study, empirical analysis and evaluation. The study of Brain Computer Interface concludes that signal could be send in human brain physically to control and observe behavior of humans, however the BCI techniques were not used in the study and without using BCI techniques, the study conducted discovered sample of humans showing increased level of anxiety when using the Internet. The task completion behaviors of humans were calculated. By the end of this discussion session, We can reach to the conclusion that, to reduce Internet anxiety, addiction and depression scale on the Internet, it is important to have many multicultural experiences, and control over own self behavior to accumulate successful experiences of behavior.

5 Conclusions and Future Work Taking the results and discussion to the logical conclusion it appears to the authors knowledge that, “Internet has lulled humans with the sense of dependency to greater extent”. Five types of hypothetical questions were answered in this study. These questions were: Do users shows increased or reduced anxiety level when using the Internet? What kinds of behaviors are shown when using the Internet? What is the role of the content? Finding types of the anxiety behaviors? , And How human process information at the internet interface? Seven major types of anxiety were studied

Study of Human Anxiety on the Internet

75

and validated: Internet terminology anxiety, Internet search anxiety, Internet time delay anxiety, and general fear of Internet failure anxiety, experience anxiety, usage anxiety, and environment and attraction anxiety. Two types of behavior (verbal and non-verbal) were formulated from relevant literature study, empirical analysis and evaluation. The study of Brain Computer Interface concludes that signal could be send in human brain physically to control and observe behavior of humans, however the BCI techniques were not used in the study and without using BCI techniques, the study conducted discovered sample of humans showing increased level of anxiety when using the Internet. From the book by Sir Tony Hoare, The first passage in Communicating Sequential Processes reads, “Forget for a while about computers and computer programming, and think instead about objects in the world around us, which act and interact with us and with each other in accordance with some characteristic pattern of behavior”. The same idea is followed in the study of human anxiety on the Internet. • Large sample size, different demography structure and discovery of perfect user study model are needed, for larger impact and generalization. • In contrast to several findings of negative effect in the Internet addiction, anxiety and depression group, some positive effects could be determined in future by building the framework for future learning through Imagination, Investigation and Innovation. • In depth analysis and comparisons of the human brain and the network open System Interconnection (OSI) model could be performed. Despite the above limitations, undoubtedly the Internet has provided a collection of applications that is having a profound effect on mankind. Like the wheel, the plow, and steam power before it, it is a proving a truly differentiating tool in our world, changing the very ways in which we interact with each other. Progress is relatively easy to recognize if we follow technology exploration. A more challenge is to find technology we want to change ourselves and for our civilization. Understanding the human factors for the design and development of the technology, system and services to ensure successful and perfect application environment is major concern. The forms of anxiety identified suggest areas for future Internet development and research.

References 1. ISOC.org (2008), http://www.isoc.org/internet/history/brief.shtml 2. ISC.org. Millions of host on the internet (2008), https://www.isc.org/ 3. Central Intelligence Agency. The DigiWorld in the global economy. In: DigiWorld 2008 (2008), https://www.cia.gov/library/publications/ the-world-factbook/geos/xx.html; https://www.cia.gov/library/publications/ the-world-factbook/fields/2184.html 4. Golden, S.M.J.: Protecting children in the internet age (2008), http://www.senate.state.ny.us/sws/ Protecting%20Children%20in%20the%20Internet%20Age.pdf

76

S.K. Kalwar and K. Heikkinen

5. Kraut, R.E. (2008), http://www.cs.cmu.edu/~kraut/RKraut.site.files/articles/ Bessiere06-Internet-SocialResource-DepressionL.pdf 6. Chou, C.: Incidence and correlates of internet anxiety among high school teachers in taiwan. Computers in Human Behaviour 19, 731–749 (2003) 7. Skinner, B.F. (2008), http://www.bfskinner.org/aboutfoundation.html; http://www.bfskinner.org/f/Science_and_Human_Behavior.pdf 8. Presno, C.: Taking the byte out of internet anxiety: Instructional techniques that reduce computer/internet anxiety in the classroom. J. Educ. Comput. Res. 18, 147–161 (1998) 9. Jackson, L.A.: Social psychology and the digital divide. In: The 1999 Conference of the Society for Experimental Social (1999) 10. Sax, L.J., Ceja, M., Teranishi, R.T.: Technological preparedness among entering freshmen: the role of race, class and gender. Journal of Educational Computing Research 24, 363– 383 11. Jackson, L.A., Ervin, K.S., Gardner, P.D., Schmitt, N.: Gender and the internet: Women communicating and men searching. Sex Roles 44(5), 363–379 (2001) 12. Jackson, L.A., Ervin, K.S., Gardner, P.D., Schmitt, N.: The racial digital divide:Motivational, affective and cognitive correlates of internet use. Journal of Applied Social Psychology 31, 2019–2046 (2001)

The Research on Adaptive Process for Emotion Recognition by Using Time-Dependent Parameters of Autonomic Nervous Response Jonghwa Kim1, Mincheol Whang2, and Jincheol Woo1 1

Dept. of Computer Science, Sangmyung University, 7 Hongji-dong, Jongno Gu, Seoul, Korea {rmx2003,mcun}@naver.com 2 Dept. of Digital Media Technology, Sangmyung University, 7 Hongji-dong, Jongno Gu, Seoul, Korea [email protected]

Abstract. This study is to propose new method, called by TDP (time dependenet parameter) anlaysis, of physiological signal processing for emoiton recognition. TDP consised of delay, activation, half recovery and full recovery. TDP was determined from running average and normalization of physiological signals for finding tonic and phasic reponse according to emotion at entire time range from stimulating emotion to recovery. As the results of this study, TDP analysis and adaptive TDP analysis enhanced accuracy of emotion recognition in the comparison with tonic analysis. Speciallly, TDP analysis enhanced the accuracy while adaptive TDP analysis reduced the individual difference of the accuracy. Keywords: Physiological signal, GSR, ECG, PPG, Skin temperature, emotion recognition, accuracy.

1 Introduction Human emotion has been tried to be recognized by physiological measurements based on assumption that emotion was expressed by them[1]. Indentifying physiological signals to subjective emotion has been main issue of emotion recognition research. However, since the signals were vulnerable to be affected by noise, emotion has had recognition error. One of methods avoiding noise has been running average at narrow time interval of incoming signal. This signal process has been effective to treat noise in autonomic response having relatively low frequency of physiological signals [2-4]. Physiological signals have been analyzed into tonic level and phasic response which were important variables according to emotion response[5]. Data treatment should be considered to discriminate level and reaction from the signal. Physiological signal has contained information of both spontaneous and non-spontaneous response. Phasic response could provide non-spontaneous overall oscillation and specific continuous level according to stimulus[5]. Recently, normalization of stimulus state from reference state has been enabled to count both tonic level and phasic response for emotion recognition [6-8]. However, since response before and after stimulus according to time variation was not examined enough, non-specific phase response of physiological signals has less clarified to cause low accuracy of emotion recognition. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 77–84, 2009. © Springer-Verlag Berlin Heidelberg 2009

78

J. Kim, M. Whang, and J. Woo

Physiological response has been characterized by individual difference. Same emotion could be recognized by different regulation of physiological response. Therefore, the strategy or rule of emotion recognition should consider individual characteristics. Some findings showed that algorithm of emotion recognition set physiological variation automatically based on verification of subjective emotion and that this process enhances the accuracy of emotion recognition[9]. Therefore, emotion could be well recognized by considerations of noise reduction, discrimination between tonic level and phasic response of physiological signals, and individualization. Considering these issues, this study is to suggest new analysis methods of physiological signals, called by TDP (time dependent parameter) and to attempt to show effectiveness of emotion recognition when it is used.

2 Method 2.1 Research Purpose This study is to propose new analysis methods for physiological response and to prove that this method was effective for emotion recognition. This research was proceeded to compare the accuracies of emotion recognition from different methods which were tonic analysis, TDP analysis, and adaptive TDP analysis. 2.2 Definition of TDP TDP (Time dependant parameter) of physiological response was defined in this study as shown in Fig. 1. The delay was moment difference between stimulation and the activation. The activation meant time at peak form the beginning. The half recovery indicated the time at half peak and the full recovery did the time back to base state. The full recovery could be inferred from half recovery when it could not be measured. In this study, ECG (Electrocardiogram), RSP (Respiration), PPG (photoplethysmogram), GSR (Galvnic skin resistance) and SKT(Skin temperature) were processed to construct TDP curve as shown in Fig. 1.

Fig. 1. TDP (time dependent parameter) of physiological measurement

The Research on Adaptive Process for Emotion Recognition

79

2.3 Emotion Induction Emotion was tried to be induced by image pictures. The images were chosen by previous study [8]. 100 university students (33 females and 67 males) participated, and were not visually handicapped. Participants were asked to score subjective emotion after watching the images. Significant images of emotion induction were categorized into 2 dimensional emotion model[10]. 6 images were selected for evoking unpleasantness-arousal emotion and 10 for pleasantness-relaxation emotion as shown Fig. 2. The pleasantness-relaxation was called as the positive emotion, and the unpleasantness-arousal was called as the negative emotion in this study.

Fig. 2. The images evoking emotions

2.4 Experiment 4 university students (average 26.5 years old) participated in the experiments and were healthy with no problem of vision. 24 prepared images were presented to participants for inducing emotion. PPG, GSR, RSP and SKT were measured during presenting images. Experiment procedure was shown as fig. 3. Participant experienced first non-image state as a reference state for 30 seconds followed by presenting the image for 10 seconds. Then, non-image state called by neutral state was presented for 30 seconds. A procedure consisted of presenting 4 images and a participant experienced 6 times a procedure. The experiment took 190 seconds for a procedure and total experimental time was 1440 seconds for a participant.

Fig. 3. Experimental procedure

80

J. Kim, M. Whang, and J. Woo

3 Analysis 3.1 Data Acquisition Collected were 6 sets consisting of 4 physiological signals and subjective score of emotion. For purpose of analysis, data unit was set at 70 seconds including neutral state for 30 seconds, stimulation for 10 seconds and another neutral state for 30 seconds. Total 74 data (6 set * 4 pictures * 4 participants) were prepared for tonic analysis, TDP analysis, and adaptive TDP analysis. 3.2 Running Average for Noise Reduction and Normalization The running average has been effective to reduce noise [3]. In this research, the time interval for running averaging was determined by response rate of each signal. Time intervals of physiological signals were tried to be set for noise reduction. The determination was done by visual scanning for confirming signal stability. The time interval of GSR and SKT was set at 0.5 seconds while one of RSP was at 3 seconds. PPG was analyzed to convert HR by frequency analysis. Therefore, time intervals for PPG and HR were set 2 seconds. Running average was performed by sliding window method at the pre-determined time interval on all the physiological signals. Then, the stimulus state of physiological signal was normalized from the neutral state. This process was able to observe activation level (tonic level) of physiological signal. Normalization was determined by equation 1 and performed at every 0.5 seconds. Normalized state = (Stimulus state –Neutral state) / Neutral state

(1)

3.3 TDP Rule for Emotion Recognition TDP rule for emotion recognition was determined from previous study [8]. Visual stimulus from the prepared images induced their emotion as shown Fig 2. Then, physiological signals were analyzed into TDP and their threshold values of emotion recognition were set for constructing rule of emotion recognition as shown Table 1. Table 1 showed mean and standard deviation of physiological response time of emotion based on TDP definition. The range was defined by mean plus and minus standard deviation for recognizing respective neutral, positive and negative emotion. 3.4 Adaptive TDP Rule for Individualization Since physiological response on same emotion was individually different, TDP rule for emotion recognition needed to be developed for individualization. The process of adaptive TDP rule was shown as Fig. 4. First, emotion was recognized by nonadaptive TDP rule. Second, difference between measured subjective emotion and emotion by physiological signals was calculated. Then, if difference existed, TDP rule was adaptively set by individual input of subjective emotion. Otherwise, emotion recognition was performed. Through these processes, it became individual and accurate adaptively for a particular person.

The Research on Adaptive Process for Emotion Recognition

81

Table 1. The TDP rule (unit: seconds) t1(Delay) t2(Activation) t3(Half Recovery) Neg. Neu. Pos. Neg. Neu. Pos. Neg. Neu. Pos. 1.55 1.92 4.08 5.79 1.57 3.08 HR ±0.92 ±1.00 ±2.91 ±2.31 ±0.17 ±1.40 15.85 9.61 20.00 5.60 5.83 16.00 SKT ±10.00 ±0.40 ±9.21 ±5.09 ±0.74 ±6.57 3.67 1.00 RR ±1.80 ±0.05 2.65 1.56 3.69 0.79 4.51 GSR ±1.16 ±0.40 ±1.95 ±11.06 ±6.55 PPG 2.25 2.43 3.90 4.37 3.45 8.00 1.75 6.03 amp ±2.74 ±0.31 ±2.91 ±1.97 ±1.48 ±4.93 ±12.06 ±6.03

t4(Recovery) Neg. Neu. Pos. 2.85 1.44 1.92 ±1.20 ±0.24 ±1.68 -

-

-

3.33 ±2.50

1.00 ±0.05

-

-

-

-

-

-

Neg.: Negative, Neu.: Neutral, Pos.: Positive

Fig. 4. The process of adaptive TDP rule for individualization

4 Result Results showed the accuracies of emotion recognition from three different methods such as tonic analysis, TDP analysis and adaptive TDP analysis as shown table 2-5. The accuracy was determined by match rate between subjective emotion and emotion determined by physiological signals. Tonic response rule was determined from previous research of autonomic response pattern for emotion[5, 11]. If negative emotion was evoked, electrodermal and cardiovascular response were increased and thermal response was decreased and vice versa [11]. As shown in Table 2, the accuracy of emotion recognition was about 62% in negative emotion recognition and about 50% in positive emotion. In the same condition, the accuracy was enhanced up to 60% when TDP analysis was used as shown in Table 3. It was about 70% in recognition of both positive and negative. Therefore, the accuracy of emotion recognition by TDP analysis could observe responses more than one by tonic analysis. There were findings for individualization of emotion recognition as shown Table 5. Adaptive TDP rule made the accuracy enhanced little more. Interestingly, the participants having the accuracy lower than 70% increased up to 70% or more. Therefore, adaptive TDP analysis could be effective to increase accuracy of a particular person who was low. Figure 5 showed overall accuracies from three analyses. TDP analysis showed improvement of accuracy but adaptive TDP did not much. Since accuracy improved in individual showing low, overall accuracy did not contribute improvement.

82

J. Kim, M. Whang, and J. Woo Table 2. Accuracy of emotion recognition from tonic analysis

participants

Negative emotion Emotion by Subjective physiological emotion signals

Positive emotion Emotion by Subjective physiological emotion signals

Accuracy

A

8

5

10

6

61.1%

B

6

3

9

4

46.7%

C

7

4

8

4

53.3%

D

8

6

9

4

58.8%

Sum

29

18

36

18

Accuracy

62.1%

55.0%

50.0%

Table 3. Accuracy of emotion recognition by TDP rule

Participants

Negative emotion Emotion by Subjective physiological emotion signals

Positive emotion Emotion by Subjective physiological emotion signals

Accuracy

A

8

7

10

7

77.8%

B

6

3

9

6

60.0%

C

7

4

8

7

73.3%

D

8

6

9

5

64.7%

Sum

29

20

36

25

Accuracy

69.0%

69.0%

69.4%

Table 4. Accuracy of emotion recognition by adaptive TDP rule

Participants

Negative emotion Emotion by Subjective physiological emotion signals

Positive emotion Emotion by Subjective physiological emotion signals

Accuracy

A

8

6

10

7

72.2%

B

6

4

9

6

66.7%

C

7

4

8

7

73.3%

D

8

6

9

6

70.6%

Sum

29

20

36

26

Accuracy

69.0%

72.2%

70.1%

The Research on Adaptive Process for Emotion Recognition

83

Fig. 5. Accuracy comparison of three analyses such as tonic, TDP and adaptive TDP analysis

5 Conclusion TDP and adaptive TDP were newly proposed to analyze physiological response for emotion recognition. The methods were successful of enhance its accuracy. The adaptive TDP was effective to count individual difference of physiological response. Comparing accuracies of emotion recognition among tonic analysis, TDP analysis, and adaptive TDP analysis, this study concluded the followings. First of all, TDP analysis enhanced the accuracy more than tonic analysis. Accuracy by tonic analysis was an average of 55%, and one of TDP analysis was of 69%. Second, the accuracy of adaptive TDP analysis reduced the individual difference. In the case of analysis by using TDP rule, the accuracy of individual difference was between 60% and 77% while in the case of adaptive TDP rule, the accuracy of individual difference was between 66.7% and 73.3%. Results showed adaptive TDP could enhance accuracy that was relatively low but work less for a participant whose accuracy was already high. Therefore, TDP and adaptive TDP method may be useful of emotion recognition and observe detail significant response of physiological response.

References 1. Christine, L., titia, L., Fatma, N.: Using noninvasive wearable computers to recognize human emotions from physiological signals. EURASIP J. Appl. Signal Process., 1672–1687 (2004) 2. Allanson, J., Fairclough, S.H.: A research agenda for physiological computing. Interacting with Computers 16, 857–878 (2004) 3. Haag, A., Goronzy, S., Schaich, P., Williams, J.: Emotion recognition using bio-sensors: First steps towards an automatic system. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS, vol. 3068, pp. 36–48. Springer, Heidelberg (2004)

84

J. Kim, M. Whang, and J. Woo

4. Mandryk, R.L., Atkins, M.S.: A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of HumanComputer Studies 65, 329–347 (2007) 5. Boucsein, W.: Electrodermal Activity. Plenum Press, New York (1992) 6. Whang, M.: The emotional computer adaptive to human emotion. Phillips Research: Probing Experience 8, 209–219 (2008) 7. Whang, M., Lim, J., Boucsein, W.: Preparing computers for affective communication: a psychophysiological concept and preliminary results. The Journal of the Human Factors and Ergonomics 45, 623–634 (2003) 8. Kim, J., Whang, M., Kim, J., Woo, J.: The study on emotion recognition by timedependent parameters of autonomic nervous response. Korean Journal of the science of emotion & Sensibility 11, 637–644 (2008) 9. Fredrickson, B.L., Losada, M.F.: Positive Affect and the Complex Dynamics of Human Flourishing. American Psychologist 60, 678–686 (2005) 10. Russell, J.A.: A circumplex model of affect. Journal of personality and social psychology 39, 1161–1178 (1980) 11. Whang, M., Chang, G., Kim, S.: Research on Emotion Evaluation using Autonomic Response. Korean Journal of the science of emotion & Sensibility 7, 51–56 (2004)

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking Yu-Jin Kim, Jin Ah Bae, and Byeong Ho Jeon Dept. of Media Image Art & Technology, Kongju National University, Sinkwandong, Gongju, South Korea {yujinkim,jinabae,bhjeon}@kongju.ac.kr

Abstract. In this paper, we used eye tracking methodologies to investigate students’ visual perceptions of lectures using 3D real-time virtual studio technology. For measuring learning performance, we also gave the students multiple-choice paper quizzes at the end of the lectures. Three virtual lectures were created with different types of lecture materials (text-centered, image-centered, and lecturercentered) and 3D virtual sets (classroom, cyberspace, and lecture-theme space). Through analyzing students’ eye movements in viewing still and moving scenes of the virtual lectures, we found that layouts and movements of design elements on lecture screens significantly influenced students’ scanpaths and areas of interest (AOIs). Lecture material types affected learning performance while 3D virtual sets had no effect due to students’ inattention to the virtual background areas. We discuss effective ways to develop virtual lectures and design lecture screens for better presentation of lecture content and higher learning performance. Keywords: Virtual lectures, virtual studios, eye tracking, visual perception, learning performance, user-centered screen design.

1 Introduction Over the last two decades, advances in digital technologies brought about the emergence of new teaching pedagogies such as Computer-Assisted Instruction (CAI), Computer-Supported Collaborative Learning (CSCL), Web-Based Instruction (WBI), and others. Besides these pedagogies using computer and internet technologies, there has been growing interest in virtual studio technology, which composites images of performers shot in front of a chroma-key blue background and visual images created through real-time 3D computer graphics [1]. The virtual studio technology increases face-to-face interaction between a lecturer and students and helps the lecturer interact with various types of lecture materials in dynamic virtual environments [2]. In academia, however, little attention has been paid to the methods to create courses using virtual studios for delivering learner-centered lecture materials and enhancing learning performance. To fill this research niche, this study aims to analyze how students look at the screens of virtual lectures [3] using eye tracking technology. Eye tracking provides insight into students’ visual perceptions and cognitive processes by measuring their eye movements. We research how lecture material types and 3D virtual sets affect learning performance. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 85–94, 2009. © Springer-Verlag Berlin Heidelberg 2009

86

Y.-J. Kim, J.A. Bae, and B.H. Jeon

2 Related Work 2.1 Lectures Using Virtual Studio Technology Virtual education, by the use of cyberspace, eliminates the spatial and temporal limitations by removing the need for the lecturer and students to be present at an instructional site at a designated time [4]. Recently, the proliferation of information and communication technologies (ICTs) has spawned a boom in virtual education by creating a variety of virtual lecture content production techniques such as Flash, Web 3D, 3D real-time virtual studio, and others. Among these production technologies, several researchers have explored the effectiveness of the virtual studio application to lectures. In 2003, Morozov, Debelov, and Zhmulevskaya claimed that the application of virtual studio systems enhances the efficiency of instructional processes by providing lecturers more possibilities for diverse forms of education [5]. According to Brown and Cruickshank (2003), a virtual lecture can be more effective than classroom lecture by achieving significant savings in the costs of lecture delivery and student support, though a very large investment of time and effort is required by the lecturer [6]. Along with the previously mentioned research interests in the educational effects of the application of new technology, researchers have studied the factors affecting the delivery of lecture content in virtual education in order to increase the motivation and effects of learning. These factors include screen design [7, 8], content design [9], content suggestion style, and lecture material [10]. 2.2 Eye Tracking in Visual Information Processing Eye movements are driven both by properties of the visual world and processes in a person’s mind [11]. Therefore, many scholars have used eye tracking techniques in order to explore the relations between eye movements and visual information processing in diverse research fields. In particular, researchers in HCI have tracked eye movements to understand visual and display-based information processing, as well as to discover the factors that may impact the usability of system interfaces [12]. In fact, eye movement data provide more detailed and specific information about a user’s cognitive process in many different kinds of displays [13, 14]. They also help researchers determine the roots of some usability problems, and then come up with effective solutions. Meanwhile, the recent trend of eye-tracking studies in HCI shows that eye-tracking methods have been rapidly employed in conducting usability testing of websites. Laarni et al. (2003) tracked the eye movements of users while they were selecting and reading online news items on a small computer [15]. Cowen (2004) asked the subjects to perform two tasks on a website and measured their total fixation duration, the number of total fixations, average fixation duration, and distribution of fixations on the screen [13]. Through these measurements, he suggested effective ways to design webpages in terms of user interface. In 2007, Cutrell and Guan also measured web surfers' eye movements to observe their web search strategies [16]. Like the aforementioned studies, many researchers have analyzed the interface of and interaction with web content, mainly focusing on a sequence of still scenes of web pages.

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

87

However, there have been increasing interests in employing eye-tracking methods in moving visual images along with the proliferation of more dynamic web content, which is developed by adopting new types of web authoring tools and media, such as Flash, Web 3D, Virtual Studio, and others. 2.3 Eye Tracking Metrics The main measurements used in eye-tracking research are “fixations,” which are moments when eyes are relatively stationary, taking in or encoding information, and “saccades,” which are quick eye movements occurring between fixations [12]. From these basic measurements, a multitude of metrics are also derived [17]: (1) “gaze duration,” the cumulative duration and average spatial location of a series of consecutive fixations within an area of interest; (2) “area of interest (AOI),” the area of a display or visual environment that is of interest to the research or design team and, thus, defined by them (not by the participant); and (3) “scanpath,” a special arrangement of a sequence of fixations.

3 Hypotheses The focus of our study is the investigation of students’ visual perceptions in virtual lectures. In order to identify the patterns of students’ visual perceptions, we tracked their eye movements according to different layouts and movements of design elements on lecture screens. We also studied the effects of lecture material types and 3D virtual sets on students’ learning performance. In line with these purposes, we investigated the following research hypotheses: Hypotheses 1a. Students’ eye movements (AOIs, scanpaths) are affected by the layouts of design elements on lecture screens. Hypotheses 1b. Students’ eye movements (AOIs, scanpaths) are affected by the movements of design elements on lecture screens. Hypotheses 2a. Students’ learning performance varies according to lecture material types. Hypotheses 2b. Students’ learning performance is different according to 3D virtual set types.

4 Experiment 4.1 Virtual Lecture Prototypes In order to test the aforementioned hypotheses, we designed and conducted an experiment to analyze students’ eye movements and learning performance in the context of virtual studio-based lectures. As experimental material, three virtual lecture prototypes were created under the theme of “Media and Culture” with varying layouts and movements of lecture screen design elements (lecturer, lecture board, 3D virtual sets, text, images, and movies on the lecture board). The three prototypes (see Figure 1) were also produced with different types of lecture materials (text-centered, image-centered,

88

Y.-J. Kim, J.A. Bae, and B.H. Jeon

and lecturer-centered) and 3D virtual sets (classroom, lecture-theme space, and cyberspace): (1) Prototype 1 - text-centered lecture materials in classroom background sets; (2) Prototype 2 - lecturer-centered lecture materials in cyberspace background sets; and (3) Prototype 3 - image-centered lecture materials in lecture-theme space background sets. Figure 1 shows screen shots of these three prototypes. Three screen shots located in the first row display their 3D background sets. The other three shots in the second row illustrate how the same lecture content about the phenomena of mass communication is delivered by different types of lecture materials.

Fig. 1. Screen shots of three prototypes (left: Prototype 1, middle: Prototype 2, and right: Prototype 3)

After modeling and animating the above 3D virtual sets using 3D Studio Max 9.0., we combined these 3D background sources with the live-action footage of lecturers by a real-time chroma key technique of the "VS2000" system. In addition, we created animation buttons for controlling lecture screen design elements using the VS scripts of the "HotAction" program. 4.2 Participants Forty participants were selected for the experiment, but ten of them had problem with the calibration of the eye-tracking system. Calibration was a fine-tuning process for the experiment, so 30 participants successfully completed the experiment. The 30 subjects were divided into three groups of ten who participated in the experiment composed of three kinds of virtual lectures. The subjects were freshmen and sophomores of K University and had almost no previous knowledge of the content of the experiment lectures. They were between the ages of 18 and 23, and the gender ratio was 1:1. Twenty-four of the students had experienced distance lectures, and eight of them had been exposed to VR lectures. 4.3 Apparatus This experiment used the hardware "Eyegaze Development System" developed by LC Technologies, Inc. This device tracks the x-y coordinates of the participant’s

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

89

gazepoint on the computer screen automatically using the pupil-center-corneal reflection (PCCR) method, which directs infrared light into the eye. This system generates raw eyegaze point location data at a camera field rate of 60 Hz [18] and calculates three kinds of gazepoints: (1) a moving point that a participant looks at for only 1/60 second, then looking at other points; (2) a fixating point that a participant looks around in; and (3) a fixation completed point that a participant maintains the gaze in. This study used the software "EyeTrack v1.0" and “EyeTrackMovie v1.0” for analyzing eye movements in viewing still and moving scenes, respectively. These programs were developed by the Human Computer Interaction Laboratory (HCIL) of KAIST for the visual monitoring and analysis of eye-tracking results that were expressed in coordinates [19]. “EyeTrack” and “EyeTrackMovie” provide “Replay” and “Analysis.” "Fixation Mark" and "Color Variation" options enable the monitoring of eye-tracking data with a variety of graphic effects in chronological order. For an efficient analysis of eye movement patterns, it also provides five analysis options: (1) "shadow," which shows the area that subjects focused on and illustrates the rest in shades with gradation effects; (2) "frequent area," which marks a clear boundary between the area that the subjects focused on and the rest, highlighting the former; (3) "hotspot," which shows the amount of gaze (red meaning more gaze and green meaning less gaze); (4) "selected area," which shows the amount of gaze in a certain area and its duration in numbers; and (4) "priority order," which shows the duration of a gaze over time in circles. All participants’ eye movements could be monitored at once due to the function of the programs which can analyze multiple data simultaneously. The patterns of AOIs and scanpaths, described in Section 2.3, were discovered from these eye movements. 4.4 Experimental Design and Procedure The experiment was performed in four stages, and it took about an hour for a participant to complete the experiment: (1) questionnaires on demographics, media education level, and cyber learning experiences; (2) still scene eye-tracking with the captured scenes according to distinguishable types of screen layouts (Prototype 1: 6 scenes, Prototype 2: 9 scenes; and Prototype 3: 7 scenes); (3) moving scene eyetracking with three prototype movies; and (4) a quiz with multiple-choice questions. Eye movements were tracked twice for still and moving scenes of virtual lectures. Compared with the eye-tracking of moving scenes, the eye-tracking of still scenes allowed more detailed and accurate analysis of eye movements for the different screen layout types. On the other hand, the eye-tracking of the moving scenes enabled an additional analysis of the effects of the movements of screen layout elements on eye movements.

5 Results 5.1 AOIs According to the Layouts and Movements of Lecture Screen Design Elements Through analyzing students’ eye movements in viewing still and moving scenes of the virtual lectures, we found that layouts and the movement of lecture screen design

90

Y.-J. Kim, J.A. Bae, and B.H. Jeon

elements (the lecturer, lecture board, and 3D background set) significantly influenced students’ scanpaths and areas of interest (AOIs) [H1a and H1b]. In the case of AOIs, parts that students paid close attention to were similar in both still and moving scene tests. By pointing out the design elements that received the most attention on the sill scenes, it was found that students generally gazed into a lecturer’s face in all of the still scenes that feature a lecturer (Prototype 1: 4 scenes, Prototype 2: 5 scenes, and Prototype 3: 5 scenes) regardless of the lecturer’s size and position on the screen. Figure 2 shows the “hotspot” option analysis results of the three scenes (left: Prototype 3, middle: Prototype 2, and right: Prototype 3), where different sizes of a lecturer appear in different positions. We calculated the distribution of fixations on the face of the lecturer in the left scene of Figure 2 using the “selected area” option and found that 52.5% of the total fixations were on the lecturer’s face.

Fig. 2. “Hotspot” option analysis results (Note: The amounts of fixations gradually increased from green to red, and the reddest parts are marked in circle)

In addition, we found that the students’ points-of-regard are likely to stay not just on the lecture’s face, but also on the face of people illustrated in the lecture material images (see Figure 3). Another finding was that the student’s attention to the people whose profile or back views were shown to decrease compared with front view images.

Fig. 3. “Selected area” option analysis results (1. Front view (28%), 2. Profile view (11.9%), and 3. Back view (12.5%) of people in the image)

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

91

Fig. 4. Comparing visual attention to text areas (3. Body (40.1%) > 2. Subtitle (29.5%) > 1. Title (7.3%))

Along with the aforementioned visual attention to people’s faces in the still scenes, we also found a tendency in the moving scenes that students’ points-of regard are mostly directed at the lecturer’s face as well as the faces of people illustrated in the lecture material images. In addition, the analysis results also suggest that students’ eye movements and text sizes in lecture boards were inversely proportional. There were less eye movements and shorter gazes on larger texts. In fact, visual attention was generally highest in the body, subtitles, and title (see Figure 4). In other words, considering that titles take up more space than subtitles, but receive less visual attention, it would be effective to deliver new and important messages using subtitles. 5.2 Scanpaths According to the Layouts and Movements of Lecture Screen Design Elements In order to measure the student’s scanpaths and scanpath duration, we selected a student whose eye movement pattern can represent the visual attention of the ten students participating in the same still scene experiment and then analyzed his/her eye movement over time using the “fixation map” and “priority order” options. From the results, it turned out that the farther from the center the objects are positioned, the

Fig. 5. Scanpath tracking (left: from left to right, right: from top to bottom)

92

Y.-J. Kim, J.A. Bae, and B.H. Jeon

more likely they are to be out of the scanpath or stared at later. This analysis also verified the commonly accepted idea that people in Korean-speaking culture read words and sentences from left to right and from top to bottom (see Figure 5). On the other hand, students’ scanpaths showed different patterns in still and moving scene tests in terms of the starting point of the eye’s gaze. In the still scene test, the starting points of the students’ gazes were related to screen design element types, as well as their positions. For example, the majority of students firstly gazed into the lecturer’s face, regardless of its position, and then moved their gazes to other elements located around the center of the screens. In the moving scene test, the movement of screen design elements, rather than their type, dominantly influenced the starting points, as illustrated in Figure 6. We also found that students moved their gazes to empty parts where the next lecture content was expected to show up. Properly timed anticipation could enable the students to better perceive animated lecture content by preparing them for what will come next in the lecture.

Fig. 6. Starting points of students' gazes according to the lecture screen layout type

5.3 Students’ Learning Performance According to Lecture Materials and 3D Virtual Sets The average quiz scores for the three groups who respectively watched text-centered, image-centered, and lecturer-centered virtual lectures (12.9, 9.7, and 10.2, respectively) were significant at the alpha level of 0.05 (F=4.694). These results support H 2a, which suggests the possibility of a relationship between students’ learning performance and lecture material types. Students could easily understand and memorize lecture content in text-centered lecture material which presented the lecturer’s

Students’ Visual Perceptions of Virtual Lectures as Measured by Eye Tracking

93

explanations in clear print on the screen. While lecture material types affected learning performance, 3D virtual sets had no effect on learning performance due to students’ lack of attention to the background areas. Consequently, H 2b, which suggests a strong relationship between students’ learning performance and 3D virtual set type, was not supported. In fact, the effects of the virtual sets on the learning process are very small because students’ visual attention is usually drawn to the background only when they found some noticeable images or figures in the set.

6 Conclusions This study investigated students’ visual perceptions of virtual lectures by analyzing their AOIs and scanpaths in the still and moving scenes of the lectures. It also explored how to develop lecture materials and 3D background sets of virtual lectures for improving learning performance by testing the students’ comprehension with multiple-choice questions. Our research revealed that the following issues on virtual lecture production should be carefully considered for better presentation of lecture content: (1) the lecturer’s size and positions on lecture screens and people’s postures illustrated in lecture materials; (2) text’s sizes and positions in lecture materials; (3) movements of screen design elements (the lecturer, lecture board, animating parts on the lecture board); and (4) balanced layouts between the lecturer, lecture materials, and virtual background sets. Educators and material designers should also consider that text-centered lecture materials were the most effective for higher learning performance than imagecentered and lecturer-centered ones. Even though statistical significance was not reached between 3D background sets and learning performance, an effective background set design suited to the lecture objectives and content is necessary for enhancing students’ learning interest and motivation. Finally, it is hoped that this study will assist lecturers in understanding students’ visual information processing in virtual lectures, in designing virtual lecture content more effectively, and in improving the instructional effects of virtual lectures.

Acknowledgment We thank Kun-pyo Lee and Jung-mi Park for their comments and help in conducting our eye-tracking experiment.

References 1. Fukaya, T., Fujikake, H., Yamanouchi, Y., Mitsumine, H., Yagi, N., Inoue, S., Kikuchi, H.: An Effective Interaction Tool for Performance in the Virtual Studio-Invisible Light Projection System. NHK Science & Technical Research Laboratories, Japan (2003) 2. Dolgovesov, B.S., Morozov, B.B., Shevtsov, M.Y.: The System for Interactive Virtual Teaching Based on “Focus” Virtual Studio. In: International Conference Graphicon 2003 in Moscow, Russia (2003)

94

Y.-J. Kim, J.A. Bae, and B.H. Jeon

3. In this paper, a virtual lecture means instructional content that is created using virtual studio techniques for virtual education 4. Kuroda, K., Shanawez, H.D.: Strategies for Promoting Virtual Higher Education: General Considerations on Africa and Asia. Africa and Asian Studies 2(4), 565–575 (2003) 5. Morozov, B.S., Develov, B.B., Zhmulevskaya, M.Y.: The System for Interactive Virtual Teaching Based on “Focus” Virtual Studio. In: International Conference Graphicon, Moscow, Russia (2003), http://www.graphicon.ru/ 6. Brown, S., Cruickshank, I.: The Virtual Studio. International Journal of Art & Design Education 22(3), 281–288 (2003) 7. Kim, M.R.: Strategies on Screen Design of Learner-Centered Web-based Instructional Systems. Journal of Educational Technology 16(4), 51–65 (2008) (in Korean) 8. Shon, M., Chung, H.H.: An Analysis on the Learning Hindrance Factors in BlendedLearning Environment. Journal of Educational Information and Media 13(2), 251–276 (2007) (in Korean) 9. Ryu, I.: Factors Influencing the Effectiveness of Web-Based Distance Learning. Management Education Review 6(2), 7–27 (2003) (in Korean) 10. Kang, M.H., Gu, M.H., Moon, H.N., Jung, S.Y., Chung, J.Y., Kim, J.S.: Examining the Effects of Tutor Delivery Modes on Cognitive Presence and Learning Outcomes in Online Lectures. Journal of Educational Information and Media 13(4), 155–181 (2007) (in Korean) 11. Richardson, D.C., Spivey, M.J.: Eye-Tracking: Characteristics and Methods, 1-9. In: Wnek, G., Bowlin, G.: Encyclopedia of Biomaterials and Biomedical Engineering. Informa HealthCare (2004) 12. Poole, A., Ball, L.J.: Eye Tracking in Human-Computer Interaction and Usability Research: Current Status and Future Prospects. In: Ghaoui, C. (ed.) Encyclopedia of Human Computer Interaction, Idea group (2004) 13. Cowen, L.: An eye movement analysis of web-page usability. Masters by Research in the Design and Evaluation of Advanced Interactive Systems (2001) 14. Lohse, G.L.: Consumer Eye Movement Patterns on Yellow Pages Advertising. Journal of Advertising 26(1), 61–73 (1997) 15. Laarni, J., Isotalus, P., Kojo, I., Kärkkäinen, L.: Reading News from a Pocket Computer: An Eye-Movement Study. In: Harris, C.D., Duffy, V., Smith, M.J., Stephanidis, C. (eds.) Human-Centered Computing: Cognitive, Social and Ergonomic Aspects. The Proceedings of HCI International 2003, Lawrence Erlbaum, Mahwah (2003) 16. Cutrell, E., Guan, Z.: What Are You Looking for?: an Eye-tracking Study of Information Usage in Web Search. In: Proceedings of the SIGCHI conference on Human factors in computing systems, San Jose, California, USA (2007) 17. Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human-computer Interaction and Usability Research: Ready to Deliver the Promises (Section commentary). In: Hyönä, J., Radach, R., Deubel, H. (eds.) The Mind’s Eyes: Cognitive and Applied Aspects of Eye Movements, Elsevier Science, Oxford (2003) 18. Eyegaze Development System Information, http://www.eyegaze.com 19. Park, J., Lee, K.: Eyetrack - Developing Eyegaze Analysis Visualization Software for Designers’ Use. In: KEER 2007, Sapporo, Japan, p. 10 (2007)

Toward Constructing an Electroencephalogram Measurement Method for Usability Evaluation Masaki Kimura*, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology {masaki-k,hideta-u,masao,matumoto}@is.nasit.jp

Abstract. This paper describes our pilot study toward constructing an electroencephalogram (EEG) measurement method for usability evaluation. The measurement method consists of two steps: (1) measuring EEGs of subjects for several tens of seconds after events or tasks that are targets to evaluate, and (2) analyzing how much components of the alpha and/or beta rhythm are contained in the measured EEGs. However, there only exists an empirical rule on measurement time length of EEGs for usability evaluation. In this paper, we conduct an experiment to reveal the optimal time length of EEGs for usability evaluation by analyzing changes of EEGs over time. From the results of the experiments, we have found that the time length suitable for usability evaluation was more than 0~56.32 seconds.

1 Introduction Existing usability evaluation methods include interview, think-aloud protocols [1], and questionnaires [2, 3, 4]. These methods are widely used for usability evaluation because they require no measurement apparatuses and allow usability experts to measure usability of software systems in a relatively easy way. However, they also have some shortcomings for usability evaluation. For instance, usability evaluation using the above methods often requires a huge amount of time to analyze and evaluate collected data. Also, the results of the analysis are sometimes hard to replicate because the collected data is based on qualitative and/or subjective evaluations from system users or subjects participating in usability testing. In order to complement the limitations of these methods, many studies try to quantitatively and objectively develop evaluation methods for measuring the mental or psychological state of users from biological information. In this paper, we focus on the electroencephalogram (EEG) of users after using software as a quantitative measurement method of usability and aim to construct an EEG measurement method for usability evaluation. Alpha rhythm and beta rhythm, which are the frequency component of EEG, change according to mental condition [4], and EEG measurement methods have features that do not disturb subjects using a computer. In general, the EEG measurement method is used as follows: * Graduate School of Information Science, Nara Institute of Science and Technology 8916-5,Takayama, Ikoma, Nara, Japan. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 95–104, 2009. © Springer-Verlag Berlin Heidelberg 2009

96

M. Kimura et al.

(1) experimenters (or usability experts) measure a subject’s EEG for several tens of seconds after the subject finishes target tasks for usability evaluation, and (2) experimenters analyze how much the measured EEG contains the alpha and beta rhythm components, which respectively indicate a comfortable or uncomfortable state of the subject’s mind. The results of the evaluation should change according to the time length to perform the EEG. After the tasks, subjects’ EEGs return to the usual condition as time passes. Hence, the time length of the analysis is too long, and the ratio of the EEG changed by the tasks decreases. Conversely, if the time length is too short, evaluation accuracy decreases because the ratio of the noise to the entire the EEG increases. However, the measurement time at EEG analysis has been decided by experiential standards. So we need to analyze EEGs by time series for usability evaluation. In this paper, we try to obtain the proper time length of EEG data.

2 Related Research In this paper, we quantitatively evaluate a psychological state of computer users during a system in-use, using alpha and beta rhythms composed of frequency components of brain waves. Power spectrums of alpha and beta rhythms, which are obtained by discrete Fourier transform, the ratio of alpha and beta rhythms in all brain waves, and beta/alpha, which is the ratio of alpha and beta rhythms, are often used as common indicators for observing the psychological state of human beings. Matsunaga et al. developed a brain wave measurement system for evaluating satisfaction of human beings and validated the hypothesis that people feel comfortable if the amount of information processing in the brain is small, while people feel uncomfortable if the amount of information processing in the brain is large [5]. In this paper, we also use the ratio of alpha and beta rhythms in brain waves and beta/alpha as indicators. These indicators are often used for studies using brain waves. So our experimental results are easy to compare with implications and insights from previous work.

3 Experiment 3.1 Overview Using Microsoft Excel 2003 and Excel 2007, which are the most popular spreadsheet software, participants performed eight kinds of tasks operating spreadsheets, and we measured participants’ EEGs after each task. Excel 2003 and Excel 2007 are different versions of software released in 2003 and 2007, respectively. Both versions have almost the same functions, but have a different look and feel in their graphical user interfaces (GUIs). Excel 2007 has a new GUI called “ribbon,” which is designed to improve task performance and user experience. However, the newly designed ribbon interface introduced users to a lot of changes to menus, tool bars, and working windows. So, even if users would like to use a familiar function such as “Save As,” they need to select a menu or button with different names and/or positions between the two versions. Furthermore, even if names and positions of menus are not changed, the design of working windows displayed after selecting a menu is often different from Excel 2003. In this way, not only new users but also existing users of Excel 2003 need to learn how to operate the new interface of Excel 2007.

Toward Constructing an EEG Measurement Method for Usability Eval.

97

Table 1. Subjects’ usage frequency of Excel 2003 and Excel 2007

never several times per year several times per month several times per week

Excel 2003 0 2 3 5

Excel 2007 6 1 2 1

Using Excel 2003 and Excel 2007, we investigate the relationship between the experiences of software and the attributes of brain waves without the effect of functional differences. In our experiment, we also analyzed the relationship between the results of subjective evaluation by questionnaire and the attributes of brain waves. 3.2 Participants Ten master’s students from the graduate school of information science participated in the experiment. Table 1 shows participants’ usage frequency of Excel 2003 and Excel 2007. All participants had experience in using Excel 2003 and understood basic operations and functions, but half of the participants had never used Excel 2007. 3.3 Task Participants performed eight tasks (four types of tasks for each version of Excel) operating the spreadsheets given in advance. Table 2 shows a list of tasks used in the experiment. All tasks can perform both Excel 2003 and Excel 2007. The content (a grade report) of the data file that is used in this experiment is the same in all tasks. Participants can continue the tasks until the task time exceeds five minutes. We counterbalanced the order of the tasks to minimize learning and fatigue effects. The following are the details of each task. Same Place Task In this task, the participant selects a particular menu that has the same name in the same position in different versions of Excel. The task is completed when the participant selects the objective menu. Different Place Task The participant selects a menu that has a different name in a different position in the two versions of Excel. This task is completed when the participant selects the objective menu. Same Interface Task The Same Interface Task consists of functions that have the same interfaces of modal dialog in different Excel versions. In the task, menu name and position were given to the participant before the task. Different Interface Task In this task, functions that have different dialog interfaces in Excel 2003 and Excel 2007 were used. As with the Same Interface Task, menu name and position were given to the participant before the task.

98

M. Kimura et al. Table 2. A task list used in the experiment

Task type Same place Different place

Same interface

Task name Open Clip Art Pane Filter Setting Display of version information Record of macros Format Cells Page Orientation

Different interface

Conditional Formatting Insert Bar Chart

Description Open clip art pane to select clip art from a list. Set options for data filtering. Display the version information of Excel. Change date formats from Mar-01 to 03/01. Change a page orientation to landscape and set margins. Change a page orientation to landscape and set margins. Indicate cells that have less than “C” or “Absence” as red font. Insert stacked bar chart of student's scores with chart/axis titles.

3.4 Environment The Emotional Spectrum Analysis System ESA-16 was employed to record EEGs of participants. After the task, we recorded participants’ EEGs for two minutes at 200Hz sampling frequency in the eyes-closed, resting condition. Electrode locations are based on the International 10-20 System, shown in Figure 1. We adapted referential derivation to observe the EEG, and used the right earlobe (A2) as a reference electrode. As a ground electrode, the center of the forehead (Fpz) was employed and the center of the parietal (Pz) was used as an exploring electrode to minimize the electromyogram (EMG) artifact. We also recorded electrocardiogram (ECG) from both arms. In addition to this, we used a headrest and elastic net bandage to secure electrodes placed on the head. Before the first task, each participant adjusted the height of chair and position of the mouse/keyboard. 3.5 Procedure The procedure of the experiment was as follows. 1. Preparation: Authors informed participant about experiment and EEG measurement. 2. Environment setting: Put the electrodes on the participant at the points described in Section 3.4, then set up the EEG analyzer. 3. Practice tasks: The participant performed two practice tasks to understand the procedure of EEG measurement. These tasks were excluded from analysis. 4. Perform a task: The participant performed a main task described in Table 2. 5. EEG measurement: After each task, the EEG of the participant was measured. 6. Perform eight tasks: The participant performed tasks repeatedly until finishing eight tasks and EEG measurements. 7. Questionnaire: After the tasks, the participant filled out the questionnaire described in Section 3.6.

Toward Constructing an EEG Measurement Method for Usability Eval.

99

3.6 Questionnaire After the eight tasks, participants answered the questionnaire sheet to investigate subjective satisfactions for each version of Excel and usage frequency of each function that was used in the tasks. The questionnaire was created by the authors based on the Questionnaire for User Interaction Satisfaction (QUIS). Each question about usage frequency consists of a four-point scale (from “Never” to “Few times per week”) and seven-point scale (from “Strongly disagree” to “Strongly agree”) for subjective satisfaction. Figure 2 shows a part of questionnaire sheet that used in the experiment.

Fp1 F7

T3

Fpz

Fp1

Fz

F3

C3

F4

F8

T4

C4

Cz

A2

A1 T5

P3

P4

Pz O1

T6

O2

Fig. 1. Electrode Locations in the International 10-20 System

Brain wave time[sec]

Fig. 2. Questionnaire Sheet

Method 1. Power spectral analysis at even intervals

5.12

Method 2. Power spectral analysis using different lengths of time window

5.12

5.12

5.12

10.24 15.36

Fig. 3. Two Kinds of Analysis Methods for Electroencephalogram

4 Analysis for EEG We applied power spectral analysis to EEG data we collected at a sampling frequency of 200Hz. To have a clear understanding of how frequency components of brain waves changed over time in the setting of our experiment and how analysis results varied according to lengths of the analysis window, we used two analysis methods for the EEG data as follows. Figure 3 illustrates the difference between the analysis methods.

100

M. Kimura et al.

Method 1. Power spectral analysis at even intervals This analysis aims to observe how brain waves change over time. We analyzed EEG data with a time interval of 5.12 seconds by cutting out the entire EEG data every 5.12 seconds so as not to overlap analysis data. The EEG data with 19 intervals (from 0 to 92.16 second) was analyzed. Method 2. Power spectral analysis using different lengths of time window This analysis aims to observe how analysis results differ according to lengths of analysis window. We analyzed EEG data by increasing time length of the analysis window by 5.12 seconds, without changing the start position of our analysis. The time length was increased by 5.12 seconds (min.) to 97.28 seconds (max.) (i.e., 0~5.12 sec., 0~10.24 sec , …, and 0~97.28 sec.). Next, the target data was filtered to reduce the artifacts from eye blinking, myoelectric activity and so on. We used a high-pass filter (HPF, 3Hz cutoff frequency, +6dB/oct attenuation factor), a low-pass filter (LPF, 60Hz cutoff frequency, -6dB/oct attenuation factor), and a band-elimination filter (BEF, 60Hz central frequency, 47.5Hz~72.5Hz stopband, second order). The band-elimination filter was used to remove the influence of an alternating-current power supply. After the EEG data was multiplied by the Hamming window and processed with the fast Fourier transform (FFT), we obtained the power spectrum from the EEG data. From the obtained power spectrum, we calculated the respective proportions of alpha rhythm and beta rhythm to all brain waves, and also calculated beta/alpha, which divided the proportion of alpha rhythm into the proportion of beta rhythm. In accordance with the classification of the international 10-20 system, we set the frequency components of alpha rhythm and beta rhythm to 8~13Hz and 13~30Hz respectively. We also set the range of all brain waves to 3~30Hz. Since the proportions of alpha and beta rhythms to all brain waves are widely used for observing various activities in the brain, we also decided to use them as indexes for measuring the physiological state of subjects after the tasks. However, because the proportions and intensity of alpha and beta rhythms vary from individual to individual, comparisons of brain waves with the absolute value would be inappropriate. In this paper, we normalized EEG data by an average value of each subject’s power spectrum and compared it with each data.

5 Results 5.1 Results of the Power Spectral Analysis at Even Intervals Figures 4, 5, and 6 respectively show the mean and the standard deviation of the alpha rhythm, beta rhythm and beta/alpha in the power spectral analysis at even intervals. In each graph, the left x-axis, the right x-axis, and the y-axis are the mean, standard deviation, and time respectively. Figure 4 indicates that the mean of alpha rhythms for Excel 2003 were larger than that for Excel 2007 after 56.32 seconds and that the difference of the alpha rhythms between Excel 2003 and 2007 was largest at 81.92 seconds. The standard deviation was always comparatively small and lowest at 56.32 seconds. Figure 5 shows the mean of the beta rhythms for Excel 2007 was larger than that for Excel 2003 after 46.08 seconds, and the difference of the beta rhythms between Excel 2003 and 2007

Toward Constructing an EEG Measurement Method for Usability Eval.

101

was greatest at 87.04 seconds. The standard deviation was higher on the whole than the standard deviation of alpha rhythms. The lowest standard deviation was observed at 40.96 seconds. Figure 6 presents that the mean of beta/alpha in Excel 2007 was larger than that in Excel 2003 after 46.08 seconds and that the difference of beta/alpha between Excel 2003 and 2007 was at 81.92 seconds. The standard deviation is larger than the results of alpha rhythms and beta rhythms and smallest at 40.96 seconds. 5.2 Results of the Power Spectral Analysis Using Different Length of Time Window

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

0.40

1.15

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28

1.20

Time length[sec]

Start time [sec]

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

1.20

0.40

1.15

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

Start time [sec]

Time length[sec]

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

Standard deviation

0.40

1.15

Average of normalized β/ α

Fig. 8. Beta Rhythm in Method 2

1.20

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16

Average of normalized β/ α

Fig. 5. Beta Rhythm in Method 1

Fig. 6. Beta/Alpha in Method 1

Standard deviation

0.35

5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28

0.40

1.15

Standard deviation

1.20

Average of normalized β rhythm

Fig. 7. Alpha Rhythm in Method 2

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16

Average of normalized β rhythm

Fig. 4. Alpha Rhythm in Method 1

Start time [sec]

Standard deviation

0.35

1.20

0.40

1.15

0.35

1.10

0.30

1.05

0.25

1.00

0.20

0.95

0.15

0.90

0.10

0.85

0.05

0.80

0.00

Time length[sec]

Fig. 9. Beta/Alpha in Method 2

Standard deviation

Excel 2007 Excel 2007 SD

5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16 97.28

1.10

Excel 2003 Excel 2003 SD

Standard deviation

1.15

Average of normalized α rhythm

0.40

1.20

0.00 5.12 10.24 15.36 20.48 25.60 30.72 35.84 40.96 46.08 51.20 56.32 61.44 66.56 71.68 76.80 81.92 87.04 92.16

Average of normalized α rhythm

Figures 7, 8, and 9 respectively show the mean and the standard deviation of the alpha rhythm, beta rhythm and beta/alpha in the power spectral analysis using different lengths of time window. In each graph, the left x-axis, the right x-axis, and the y-axis are the mean, standard deviation, and time respectively.

102

M. Kimura et al.

Figure 7 shows that the mean of alpha rhythms in Excel 2003 was larger than that in Excel 2007 in the case of using time windows over 40.96 seconds. As lengths of time window became longer, the alpha rhythms in Excel 2003 tended to increase and the standard deviation decreased. Figure 8 presents that the mean of the beta rhythms in Excel 2007 was larger than that in Excel 2003 in the case of using time windows over 56.32 seconds. As lengths of time window became longer, the standard deviation tended to decrease as well as the results of the alpha rhythms. Figure 9 indicates that the mean of beta/alpha in Excel 2007 was larger than that in Excel 2003 in the case of using time windows over 40.96 seconds. As lengths of time window became longer, the standard deviation tended to decrease as well as the results of the alpha and beta rhythms. Table 3. A Result of Questionnaire Usage

Understand Productivity

frequency Excel 2003 Average 3.3 SD 0.82 Excel 2007 Average 1.8 SD 1.14 p < 0.05 yes

5.0 1.33 3.5 2.17 no

Simple Interface Easy Satisfacto use to use tion 5.4 4.8 4.9 5.3 5.0 1.26 1.32 1.45 1.34 1.15 3.5 4.0 3.1 3.4 3.3 1.72 1.63 2.13 1.90 1.95 no yes yes yes yes

5.3 Results of Questionnaire Table 3 shows the mean, standard deviation and the result of the two-sample t-test of each questionnaire item. In the table, there were significant differences in “Productivity,” “Interface,” “Easy to use” and “Satisfaction” between Excel 2003 and Excel 2007. Our subjects gave Excel 2007 lower scores than Excel 2003.

6 Discussions From the results of the power spectral analysis at even intervals, we could confirm that all three indexes tended to be relatively stable after 56.32 seconds. In fact, the alpha rhythms in Excel 2003 were larger than those in Excel 2007. The beta rhythms and beta/alpha in Excel 2007 were larger those in Excel 2003. The standard deviations of all three indexes also indicated lowest or relatively lower from 40.96 seconds to 56.32 seconds. These results would imply that EEG data from 56.32 seconds to 61.44 seconds is stable among all subjects, is less influenced by artifacts, and appropriately reflects the influence of the difference between Excel 2003 and 2007. However, for all indexes, the difference of the influence between two versions of Excel and the standard deviations constantly fluctuated. We consider that this might occur due to individual differences of subjects’ brain waves, fatigue from the long experiment duration, myogenic potential by attitude variation, and so forth. Therefore, it is necessary not to use a time window with short length, but to use one with long length, in order to conduct accurate usability evaluation.

Toward Constructing an EEG Measurement Method for Usability Eval.

103

From the results of the power spectral analysis using different lengths of time window, we could observe that the alpha rhythms in Excel 2003 were stably higher in the case of using time windows over 40.96 seconds, the beta rhythms in Excel 2007 were stably higher in the case of using time windows over 56.32 seconds, and beta/alpha in Excel 2007 was stably higher in the case of using time windows over 40.96 seconds. For all indexes, the standard deviations tended to become small as the time window became longer. This might be because the rate of EEG influenced by artifacts became small as the time window became longer. These results suggest us that we could analyze EEG data with no influence of artifacts by using a time window over 56.32 seconds. The results of our analysis showed that the alpha rhythms in Excel 2003 were larger than those in Excel 2007, and the beta rhythms and beta/alpha in Excel 2007 were larger than those in Excel 2003. It is clarified by previous studies on EEG measurement methods that the amount of alpha rhythms is decreased and the amount of beta rhythms and the value of beta/alpha are increased when a subject’s mental work load is high. The results of the questionnaire showed our subjects preferred Excel 2007 to Excel 2003. These results of questionnaire and our analysis agree with previous work. Since the results of our analysis and questionnaire also supported the results of the past studies on EEG, we can conclude EEG measurement would be useful for evaluating software usability.

7 Conclusion and Future Work In this paper, we have conducted an experiment to have a clear understanding of the appropriate timing and lengths of time window in order to analyze EEG data for accurate usability evaluation. From the results of the experiments, we could obtain the following insights. • A short length of time window (e.g. 5.12 seconds) is not suitable for usability evaluation because the frequency components of brain waves constantly fluctuate. • The accuracy of usability evaluation can be improved by using length of time windows of 56.32 seconds. In this experiment, we could not observe normal state of EEG data but EEG influenced by the tasks. In the near future, we need to conduct another experiment to observe it.

References 1. Ericsson, K.A., Simon, H.A.: Protocol analysis: Verbal reports as data. MIT Press, Cambridge (1993) 2. Osgood, C.E., Suci, G.J., Tannenbaum, P.H.: The measurement of meaning. University of Illinois Press, Urbana (1957) 3. Chin, J.P., Norman, K.L., Shneiderman, B.: Subjective user evaluation of CF PASCAL programming tools. Technical Report (CAR-TR-304) (1987)

104

M. Kimura et al.

4. Hart, S.G., StaveLand, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Handcock, P.A., Meshkati, N. (eds.) Human Mental Workload, pp. 139–183. Elsevier, Amsterdam (1988) 5. Matsunaga, H., Nakazawa, H.: A Study on Human-Oriented Manufacturing System (HOMS) – Development of Satisfaction Measurement System (SMS) and Evaluation of Element Technologies of HOMS using SMS. In: Int. Conf. Manufacturing Milestones toward 21st Century, pp. 217–222 (1997)

Automated Analysis of Eye-Tracking Data for the Evaluation of Driver Information Systems According to ISO/TS 15007-2:2001 Christian Lange, Martin Wohlfarter, and Heiner Bubb Lehrstuhl für Ergonomie, Technische Universiät München Bolzmannstrasse 15, 85747 Garching {lange,m.wohlfarter,bubb}lfe.mw.tum.de

Abstract. First of all, the most important content of the ISO/TS 15007-2:2001 standard for performing eye tracking experiments will be described. The following text includes a detailed description of how eye/gaze experiments using the Dikablis eye tracking system are conducted according to the above mentioned standard and how recorded statistical evaluations can be automated and visualized. Keywords: ISO/TS 15007-2:2001, Eye tracking, Driver Assistance Systems, Driver Information Systems.

1 Introduction Adhering to the guidelines of the European Statement of Principles ESoP, the need for good and less distracting design of driver information systems will grow enormously. The future will yield the driver information systems which will hinder or distract the driver from the driving task being performed as least as possible. The advantage of the standardized experimental conduction lies in the improvements delivered by progressive experimental comparability, error reduced and faster processing. The duration of the analysis of uplifted data is therefore enormously reduced. Subsequently, the standard conduction of gaze experiments using the Dikablis Toolkit can be proved to comply with the ISO/TS 15007-2:2001 standard.

2 Toolkits for Standardized Experimentation and Testing The Dikablis software package supports the standards required by eye tracking experimentation. In figure 1 the interference between Dikablis and the ISO/TS 150072:2001 standard regarding test planning, calculations and analysis of eye tracking data is illustrated. Dikablis will be deployed in order to evaluate the tracking of the gaze direction and eye and head movements of the test subject. The Dikablis Toolkit consists of a Recording Software, an Analysis Software for post-processing of the eye-tracking data and manual data analysis and D-LAB for completely automated data analysis. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 105–110, 2009. © Springer-Verlag Berlin Heidelberg 2009

106

C. Lange, M. Wohlfarter, and H. Bubb

Fig. 1. Workflow schema for standardized testing and experimentation of Dikablis and ISO/TS 15007-2:2001 interferences/differences

With the Recording Software the gaze behavior of the test subject can precisely be recorded. The test subject wears the Dikablis Head Unit, on which two cameras are installed. One camera is directed towards the test subject’s eye and is used to recognize the subject’s gaze behavior (pupil movements). The other camera is directed straight on in front of the subject and monitors the subject’s environment. By processing both of these pieces of video data, the gaze direction of the test subject can almost precisely be determined. Offline re-calibration and post-processing of the eye-detection can be made with the Dikablis Analysis Software using the recorded data even after the recording software has finished recording. These possibilities to adjust and post-process guarantee clean and always analyzable test results under any circumstances. The D-LAB module contains a package for conducting experiments, from test planning and defining of special Areas of Interest (AOI) visible within the gaze region, to the automatic calculation of glance durations and graphical presentation of these results. Successively, the proceedings to standardized testing and experimentation will sbe described subsequently. Herein the capital and lower case letters are always related to figure 1’s demonstrated procedural (workflow) plan and the relationships between the small angled boxes located within the figure.

Automated Analysis of Eye-Tracking Data

107

2.1 Construction of an Experimental Plan The compliancy to test partitioning presented by ISO/TS 15007-2:2001 into „experimental condition“, „task“, and „subtask“ can be defined into one test plan with DLAB (see figure 2 left). This way a test can be represented in the form of intertwined and nested intervals in which: • „experimental condition“ an entire Experiment is unfolded (Ex. driving on country roads); • “task” defines the interaction between a determined system presented within the Experiment (ex. Operation of Navigation Systems); • A specification of a “task” as “subtask” will be considered. (Ex. Operation of a navigation system via a touch screen display). D-LAB offers the possibility to additionally define „subsubtasks“ as the fourth layer, ex. in order to mark the appearance of critical events presented within the experiment (ex. sharp breaking situations) or automatic analysis of the display screen (ex. an individual input screen within the navigation system, for example inputting a destination address).

Fig. 2. Left: constructing a test sequence; Right: Automatically created block shifting diagram/interface for online test segment sequence marking

2.2 Record Gaze Behavior and Mark the Beginning and End Points of Each Trial Interval Using one of the D-Lab integrated applications one can automatically create a shifting block diagram/interface using the trial definitions, which can be seen on the right side of figure 2. Each shifting block represents a test segment. By pressing a block, a network event is created, which marks the start or ending of a task interval directly in the Recording Software. The functionality of the Dikablis recording software is described

108

C. Lange, M. Wohlfarter, and H. Bubb

in written detail in Lange et al., 2006a und Lange et al., 2006b. These events mark the beginning respectively the end of a trial segment and are synchronously saved to the calculated gaze data. The events can also be started from another data recorder such as a driving simulator. 2.3 Validating Gaze Data and Trial Intervals The Dikablis analysis software is firm in validating the gaze data after a trial and in optimizing. In preparation, the gaze data if necessary is calibrated after the trial so as to optimize offline pupil recognition (see Lange et al., 2006a and Lange et al. 2006b). Further processing in D-LAB follows after the optional jointing is successful. The first move consists of testing whether all trial intervals would be marked correctly. The marked trial intervals located under the gaze player window synchronized to the playtime line shown on the user interface layer (see figure 3). D-LAB also offers a testing function which automatically identifies inconsistencies within the trial segment. The segment can be manually adjusted or changed, tasks can be added or deleted. Figure 3 shows the D-LAB interface for management of a trial interval.

Fig. 3. Validation and post-processing of a trial segment in D-LAB

2.4 Definition of Areas of Interest The areas of interest used in the calculation of glance durations in relevant gaze regions (ex. the navigation system display, the street view, the rear view mirror, the left angle mirror, the dashboard, etc.) as required by the ISO/TS 15007-2:2001 are defined by using D-LAB’s functionality of marking specific AOIs. Thus in D-LAB a random number of AOIs can be defined in the form of a polyline and labeled with names. Figure 4 demonstrates the Head-Up Display and Dashboard as defined AOIs.

Automated Analysis of Eye-Tracking Data

109

Fig. 4. Defining selected AOIs within the gaze region such as the Head-Up Display and the Dashboard

2.5 Automatic Calculation of Glance Metrics to the Defined AOIs In order to automatically calculate glance duration and glance frequency, the pupil of the test subject as well as his/her head position in relation to the test environment (relative to the defined AOIs) must be recognized. The determination of the head position is carried out with the help of the so called marker(s) (see the quadratic black and white object in figure 4), which describe the environment reference. These markers are found using image processing in the photo of the field camera and used in the head position calculation of the test subject. For every defined and established AOI, using the D-LAB function „Calculate Gaze Durations“ it is calculated when the test subject glanced at an AOI and when their glance was away from a certain AOI. The result of this calculation is displayed analogously to the graphical representation of the trial intervals located under the gaze film player window synchronous to the playtime line. Hence, the operator can impose calculation corrections at any time simply by testing and manually correcting if necessary. The gaze specific values from the ISO/TS 15007-2 standard can be calculated using the automated determination of glance durations via the defined AOIs. Thus the operator can input the values which need be calculated for the following gaze experiment in the form of an „Analysis Series“. Therefore the specific values pertaining to specific tasks and AOIs to be calculated must be defined. The following values are available to choose from: • • • • • • • •

Total Glance Time Glance Frequency Time off road-scene-ahead Total glance time as a percentage Fixation probabilities Link value probabilities Maximum Glance Duration Mean Glance Duration

110

C. Lange, M. Wohlfarter, and H. Bubb

D-Lab hereupon calculates the inputted values and saves the results in the form of text file. These automated value calculations can be conducted for all defined AOIs, all terms of the experiment plan as well as for all gaze metrics. The text file is built concurrently so that it can be opened for further statistical analysis using the SPSS statistics program without indirection. 2.6 Graphical Representation of Calculated Metrics Next to the internal calculation of values, D-LAB additionally offers a trial result(s) graphical representation capability. For this purpose several graphical diagrams are displayed which support the interpretation of the result(s). There are two examples in figure 5 that exemplify this. To the top one can find the progressive course of glance duration on defined AOIs for a single subtask presented by four independent test subjects. To the bottom the course of the users mean glance fixation duration in a critical situation is shown in order to allow conclusions on the basis of mental strain.

Fig. 5. Top: Glance duration on the defined AOIs on all trial experiments. Bottom: User glance fixation duration in critical situations.

References 1. ISO/TS 15007-2:2001: Road vehicles - Measurement of driver visual behavior with respect to transport information and control systems - Part 2: Equipment and procedures 2. Lange, C., Yoo, J.-W., Wohlfarter, M., Bubb, H.: Dikablis - Operation mode and evaluation of the human-machine interaction. In: Spring Conference of Ergonomics Society of Korea, Seoul, May 12 (2006) 3. Lange, C., Wohlfarter, M., Bubb, H.: Dikablis - engineering and application area. In: Proceedings IEA 2006 16th World Congress on Ergonomics, Maastricht the Netherlands (2006)

Brain Response to Good and Bad Design Haeinn Lee1, Jungtae Lee2, and Ssanghee Seo2 1

107A Kiehle Visual Arts Center, 720 Fourth Avenue South St. Cloud, MN 56301-4498, USA 2 6409-1 Dept of Computer Science & Engineering, Pusan National University Jangjeon-dong, Geumjeong-gu, Busan, 609-735, Republic of Korea

Abstract. This paper is about the decision of whether good or bad design is the result of the human brain process. Our research team has used the technique of functional MRI and Electroencephalogram (EEG) to address the question of how the brain answers while subjects viewed different designs. Classifying the good or bad designs, subjects chose a mouse button to decide their perception of good or bad design and we analyzed their patterns of EEG rhythms and fMRI. The results of fMRI showed that the perceptions of different feelings of designs are associated with the frontal lobe and the occipital lobe. After analyzing the EEG by the Event-related brain potentials (ERP) method, we also found that the amplitude of ERP components in perception of bad design is greater and latency is shorter than that of good design. Therefore, the human brain responds sooner and stronger in perception of bad feeling. Keywords: Human Behaviors, EEG, fMRI, ERP, Interaction and Interface Design, Usability Test, Brainwork, Visual Brain.

1 Introduction With advances in computer video display technologies such as EEG, fMRI, PET, and SPECT, brain research has become a contemporary issue. Most people thought that the brain is just black box and is not an interesting area to explore, but because of developed new technologies such as EEG, PET, and fMRI, research of the brain revealed how it is structured and worked. [1] The weight of the human brain is about 1400g and contains between 1 billion and 100 billion neuron, and is divided into the brain stem, limbic system, and cortex. The brain stem is the major route by which the forebrain sends information to and receives information from the spinal cord and peripheral nerves. The limbic system is a system of nerves in the brain involving several different areas concerned with basic emotions such as fear and anger, and basic needs such as the need to eat and to have sex. [1] The Cortex forms 90 percent of the brain, and plays a key role in memory, attention, perception, awareness, thought, language, and consciousness. Also this part of the brain is divided into four sections: the occipital lobe, the temporal lobe, the parietal lobe, and the frontal lobe. Functions, such as vision, hearing, and speech, are distributed in these selected regions. Brain research has become a contemporary issue and J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 111–120, 2009. © Springer-Verlag Berlin Heidelberg 2009

112

H. Lee, J. Lee, and S. Seo

has explored diverse aspects of many different major areas such as medical science, engineering, psychology, biotechnology, linguistics, economics, music, etc. This research has provided new insight into underlying disease mechanisms and is beginning to suggest new treatment. Also using the Neuro-Lingustic Programming (NLP) method, researchers understand human behavior and cure the human mind to lead a better life. [2] The brain controls body activities, feels the five senses, and shapes thoughts, hopes, dreams, and imaginations. In short, the brain is what makes people human. [1] Recently, there are lots of brain researches relating to linguistics and music activities. For example, a person hears two different sentences; “He takes coffee with cream” and “He takes coffee with dog” and examined their brain movements. There are certain distinctions between the two sentences. The human brain has a strong reaction when people hear the latter sentence that is not normal. Likewise, there are certain differences of brain movements when people hear natural and unnatural melodies from music. [3,4,5] It can be linked to the art and design areas, so that research can be about how the human brain acts when people do artistic activities, and how people feel when they look at artistic works or designs. If people look at the smile of the Mona Lisa, they usually get a good feeling. On the other hand, if people look at a photo that shows a scene with a terrible car accident, they get a bad feeling. According to this fact, if our researchers can clarify and measure how the brain works when people get a good or bad feeling, we can come up with an objective evaluation of which part of design makes people get a good feeling or bad feeling. Also we can determine the main aspects of a design with good value. We have used the technique of functional MRI and EEG to address the question of how the brain answers while subjects viewed different designs. Twenty-five subjects participated in this research and viewed fifty design images randomly. Classifying the good or bad designs, subjects chose a mouse button to decide their perception of good or bad design. We analyzed their patterns of EEG rhythms and fMRI to find the characteristics of brain activity based on good and bad designs.

2 Brain Structure and Movement 2.1 Brain Structure As I mentioned before, the brain is divided into four sections: the occipital lobe, the temporal lobe, the parietal lobe, and the frontal lobe, and functions such as vision, hearing, and speech, are distributed in these selected regions. The occipital lobe is located in the back of the brain and plays a role in processing visual information. (Fig.1) The parietal lobe plays a role in sensory processes, particularly determining spatial sense and navigation, attention, and language. The frontal lobe has a role in controlling movement and in the planning and coordinating of behavior. The temporal lobe is involved in auditory processing and is home to the primary auditory cortex. [1].

Brain Response to Good and Bad Design

113

Fig. 1. Brain Classifications [1]

2.2 Brain Movement and Technology Functional magnetic resonance imaging (fMRI). People use the MRI, which provides a high quality, three-dimension image of organs and structures inside the body to examine a body structure. But to examine the brain activity people need to use fMRI and this is most popular neuroimaging technique today. This technique compares brain activity under resting and active conditions, and it combines the high spatial resolution, which is a correlate for neuronal activity. This technique allows for more detailed maps of brain areas underlying human mental activities in health and disease. Our team has used the technique of fMRI to find out which part of brain is more activated when subjects viewed good or bad design. [1]. Electroencephalogram (EEG). Many of the recent advances in understanding the brain are due to the development of techniques that allow scientists to directly monitor neurons throughout the body. Electroencephalogram is the recording of electrical activity along the scalp produced by the firing of neurons within the brain. In this method, electrodes placed in specific parts of the brain, which vary depending on which sensory system is being tested-make recordings that are then processed by a computer. [1]. EEG has several strong sides as a tool of exploring brain activity; for example, its time resolution is very high (on the level of a single millisecond). Other methods looking at brain activity, such as PET and fMRI have time resolution between seconds and minutes and it measures the brain’s electrical activity directly. EEG can be used simultaneously with fMRI so that high-temporal-resolution data can be recorded at the same time as high-spatial-resolution data. [6]. Our team has used the technique of EEG (Electroencephalogram) to address the question of how the brain answers while subjects viewed images. Most important thing to use EEG is which part of brain we need to place electrodes to measure the brain waves. Electrode location and names are specified by the 10-20 system for most research applications. This system is an internationally recognized method to describe and apply the location of electrodes in the context of an EEG test. [6] (Fig.2).

114

H. Lee, J. Lee, and S. Seo

Fig. 2. Electrodes location to check the EEG (10-20 System) [8]

Event-related potential (ERP). During the experiment, we used event-related brain potentials (ERP), which is a method in checking an electroencephalogram in certain moment when subjects received the event like viewed designs. ERP is any measured brain response that is directly the result of a thought or perception. It can be reliably measured using the EEG, a procedure that measures electrical activity of the brain through the skull and scalp. [7]. There are two important components in ERP waveform, which are P300 and N400. The N400 ERP component is described as a negative voltage deflection occurring approximately 400ms after stimulus onset, where as P300 component describes as a positive voltage deflection 300ms after stimulus onset. The presence, magnitude, topography and time of this signal are often used as metrics of cognitive function in decision making processes. While the neural substrates of this ERP still remain hazy, the reproducibility of this signal makes it a common choice for related researches. [6].

3 Brain Response to Good and Bad Design Our team has used the technique of functional MRI and EEG to address the question of how the brain answers while subjects viewed different designs. Twenty-five subjects participated in this research and viewed fifty design images randomly. First, as I am a professional graphic designer, I subjectively chose twenty-five designs that evoked good feelings and twenty-five that evoked bad feelings. (Fig.3) Because the author’s judgment of good or bad designs was purely subjective, the individual subjects were asked to categorize each design as good or bad. As they viewed each image, they either clicked the right mouse button to classify the image as good, or the left mouse button to classify the image as bad. As they did this, we analyzed their patterns of EEG rhythms and fMRI.

Brain Response to Good and Bad Design

Fig. 3. Design image examples for experiment

Fig. 4. The fMRI response of good design

115

116

H. Lee, J. Lee, and S. Seo

3.1 The Results of the fMRI Based on Good and Bad Design Our team used fMRI machine (ISOL fORTE) in the Brain Science Research Center at Korea Advanced Institute of Science and Technology (KAIST) and shooting variable is as follows: TR=3000ms, TE=35ms, Number of Slice=25, FOV=24cm, Image natrix=64x64, Slice Thickness=5mm. The scenario of the experiment is as follows: 1. Shows “+” on a screen for 3seconds. 2. Shows good and bad design randomly for 3seconds. 3. Shows “+” on a screen for 3seconds again. We kept repeating this process to show all fifty designs. At the same time, subjects chose a mouse button to decide their perception of good or bad designs. Fig 4 presents the part of brain that showed activity when one subject saw a good design. The top left brain image shows the midsagittal section, the right image is showing the coronal section, and the bottom image shows a horizontal section. The left part of the midsagittal is the location of the occipital lobe and the right part is the location of the frontal lobe. The top part of horizontal section is the right part of midsagittal section and bottom part is the left part of midsagittal section.

Fig. 5. The fMRI response of bad design

Brain Response to Good and Bad Design

117

The bottom table of fig4 shows the coordinates of where the brain was activated. Using these coordinates we can indicate the broadmann area in the activated brain areas. After observing the broadmann area, the most activated areas were points 19,7,11,and 6. These areas are the occipital lobe, the parietal lobe, the prefrontal lobe, and the frontal lobe. The decisions to judge images either good or bad are made in the occipital lobe and the frontal lobe. The broadmann 7 area, which controls visualmotor coordination, is located in the parietal lobe, but it is close to the occipital lobe. Fig 5 presents the part of brain that showed activity when one subject saw a bad design. When subjects judged the design as bad, their brain was activated in broadmann area 18, 19,7,11,6, and 20 and these areas are location of the occipital lobe, parietal lobe, prefrontal lobe, frontal lobe, and temporal lobe. The broadmann 20 area, which controls the high-level visual processing and recognition, is located in the temporal lobe. Similarly, The occipital lobe, frontal lobe, and parietal lobe were activated while subjects looked a bad design. We noticed that there were differences in specific activated areas, for example, the brain action when subjects had a bad feeling was much stronger than when subjects had a good feeling. Also, there was little difference in brain action between the left and right brain when the subject perceived the design as good, but left-brain action was strong when subjects perceived the design as bad. 3.2 The Results of the EEG Based on Good and Bad Design Based on the fMRI result, our researchers decided to attach electrodes in the area of the occipital and the frontal lobe, which is Fp1, Fp2, O1, and O2 from 10-20 system, and examine the EEG in the areas. (See Fig.2.) Twenty-five subjects participated this research and the scenario of the experiment is as follows: 1.Shows “+” on a computer screen for 1minute to be a steady state of brain. 2. Shows good and bad design randomly for 3seconds. 3. Subjects choose a mouse button to decide their perception of good or bad designs. 4. Shows “+” on a computer screen for 1minute to be a steady state of brain again. We kept repeating this process to finish all fifty designs. During the experiment, we used event-related brain potentials (ERP), to check EEG in certain moment when subjects received the different feelings. Fig6 image shows the amplitude of the occipital lobe when the subjects perceived the design as good or bad. The latency of P300 component from channel O1 and O2 when subjects had a bad feeling appeared faster than when subjects had a good feeling. Also the amplitude of the perception of bad feeling is higher than the amplitude of the perception of good feeling. Fig 7 shows the result of difference of latency based on subjects’ perception of good or bad design. From this figure, the x-axis represents the result of the latency (G) of a bad design subtracts from the latency (B) of a bad design, and the y-axis is the subjects’ number. Most points placed in plus area which means the latency (G) of a good design is longer than the latency (B) of a bad design. Therefore, the human brain responds sooner and stronger action when subjects perceived the design as bad. We also examined the latency of the frontal lobe when the subjects perceived the design as good or bad, and the result is similar with the occipital lobe. As I mentioned before, the human brain responds sooner to in perception of bad feeling. However, these results are the mean value of subjects, there are people responded sooner in perception of good feeling.

118

H. Lee, J. Lee, and S. Seo

Fig. 6. Comparison of the occipital lobe characteristic

Fig. 7. Comparison of the occipital lobe latency

4 Conclusion With advances in computer video display technologies such as EEG, fMRI, PET, and SPECT, brain research has become a contemporary issue. Based on the issue, many different major areas such as Medical Science, Engineering, Psychology, Linguistics,

Brain Response to Good and Bad Design

119

Biotechnology, Economics and Music explored diverse aspects of brain research. Hence, there should be lots of possibilities to study brain research relative to Art and Design. When one makes the decision that something is good or bad his/her decision is a result of the human brain process. It is obvious that the brain works differently depending on a good feeling or a bad feeling. In other words, the brain will answer different ways when people look at good design or bad design. Although there would be a difference in determining design values, it depends on the individual. In the case of a masterpiece, most people admit its artistic value. According to this fact, if our researchers clarify how the brain works when people look at good and bad design, we come up with an objective evaluation of how people judge the design value. Based on the result, we also determine what can be the main aspects of a design with good value. We have used the technique of functional MRI and EEG to address the question of how the brain answers while subjects viewed different designs. The results of fMRI show that the perceptions of different feeling for designs are associated with the frontal lobe and the occipital Lobe. The occipital lobe is placed in back of the brain part and has a visual cortex so controls with the organ of vision. The frontal lobe is placed in front of the brain part and takes care of complicated functions such as thinking, planning, and deciding. Based on the fMRI result, our researchers decided to attach electrodes in the area of the occipital and the frontal lobe and examine the EEG in the areas. During the experiment, we used the ERP, which is a method in checking the EEG in certain moment when subjects received the event like viewed designs. After analyzed the EEG by the ERP method, we also found that the amplitude of ERP components in perception of bad design was higher and latency was shorter than that of good design. From all examinations, we found there is a significant distinction between an individual because of a characteristic of brain. Some people have strong action in leftbrain and some are not. But in general, the human brain responds sooner and stronger to action in perception of bad feeling. Thus, we assume it can be an objective fact to determine good value of design when brain responds slower and exhibit a long latency. Considering this issue, we also can determine what aspect that makes good value of design and apply as a solution to create a better design. In the future, the research includes not only pure design but also can be applied into the human interface design. One of the most important issues in the human interface design is a usability test to grasp a person’s individual feeling and measure the emotional satisfaction. The idea of this paper will show as a great method of usability test to classifies the good or bad interface design as well as link with ease of use. Acknowledgments. I want to give a special thank you to my research team and family: my father, mother, and brother for their constant and unconditional love, encouragement, and support. I am deeply indebted to them and dedicate this study to them.

References 1. Society for Neuroscience: Brain Facts, A primer on the Brain and Nervous System (2005), http://www.sfn.org 2. Mark, F., Barry, W., Michael, A.: Neuroscience: Exploring the Brain, 3rd edn. Lippincott Williams & Wikins (2006)

120

H. Lee, J. Lee, and S. Seo

3. Lee, J.Y.: Neurophysiology and Brain-imaging study of Music-music& language, music & emotion. Nangman Music Magazine 18(3) (2006) 4. West, W.C., Rourke, T., Holcomb, P.J.: Event-Related Brain Potentials and Language Comprehension: A Cognitive Neuroscience Approach to the Study of Intellectual Functioning, Tufts University (1998) 5. Lu, H., Wang, M., Yu, H.: EEG Model and Location in Brain when Enjoying Music. In: Proceedings of the 2005 IEEE Engineering in Medicine and Biology, Shanghai, China, pp. 2695–2698 (2005) 6. Wikipedia, encyclopedia: http://en.wikipedia.org/wiki/EEG 7. Coles, Michael, G.H., Rugg, M.D.: Event-related brain potentials: an introduction, Electrophysiology of Mind, Oxford scholarship Online Monographs, pp. 1–27 (1996) 8. Kim, D.S.: Electroencephalogram, Korea Medical (2001)

An Analysis of Eye Movements during Browsing Multiple Search Results Pages Yuko Matsuda, Hidetake Uwano, Masao Ohira, and Ken-ichi Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology 8916-5,Takayama, Ikoma, Nara, Japan {yuko-m,hideta-u,masao,matumoto}@is.nasit.jp

Abstract. In general, most search engines display a certain number of search results on a search results page at one time, separating the entire search results into multiple search results pages. Therefore, lower ranked results (e.g., 11thranked result) may be displayed on the top area of the next (second) page and might be more likely to be browsed by users, rather than results displayed on the bottom of the previous (first) results page. To better understand users' activities in web search, it is necessary to analyze the effect of display positions of search results while browsing multiple search results pages. In this paper, we present the results of our analysis of users' eye movements. We have conducted an experiment to measure eye movements during web search and analyzed how long users spend to view each search result. From the analysis results, we have found that search results displayed on the top of the latter page were viewed for a longer time than those displayed on the bottom of the former page. Keywords: Eye tracking, Web search, User activity, Search results page.

1 Introduction Web search engines are designed to search for useful information on the World Wide Web. Each search engine uses different algorithms to rank web pages, but their interfaces are similar to each other; that is, users type some words into a query box and receive a rank-ordered list of web pages that is relevant to the words. Better understanding user activities would provide us insights into improvements of interactions during web search and the usefulness of the interfaces for search engines. Up to now, many studies have reported analysis results of users’ web search activities based on eye tracking, which is well-known as an effective means to understand what users are looking for during web search. One of the studies [2] showed that users tend to spend most time browsing a few, top results on a search results page while they spend less time browsing the bottom of the page. The study concluded users strongly rely on the correctness of a ranked order presented by a search engine. As with the above study, most previous studies also have focused on user interactions on the first page of the entire search results. However, many users browse multiple results pages in using a search engine [3]. Neglecting this fact might lead to an incomplete understanding of user activities in web search. When a number of search J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 121–130, 2009. © Springer-Verlag Berlin Heidelberg 2009

122

Y. Matsuda et al.

results exist, most search engines separate the entire results into multiple search results pages and display one results page at a time (e.g., 10 search results on a page. The numbers of results displayed on one page depends on user preference.) In this case, lower ranked results (e.g., 11th result) may be displayed on the top area of the next page and might be more likely to be browsed by users, rather than results displayed on the bottom of the previous (first) results page. Therefore, it is important to analyze and understand user activities in searching multiple search results pages in order to provide users with a means or new interface that enables them to more naturally browse search results in a ranked order recommended by a search engine. In this paper, we have conducted an experiment to observe effects of search result positions on user activities in web search. We measured eye movements during web search tasks and analyzed them to understand how long users spend to browse each result.

2 Related Work There have been many studies on user activities in web search. A popular approach to analyzing user activities in web search is to use data of browsing histories or access logs [6][7][8]. Although using history data helps us understand a user’s access paths or interests in a specific web page, it cannot be used for analyzing a user’s attention to search results during web search and influences of displayed positions of search results on web search activities. Another approach is to use eye tracking instruments to capture user activities based on eye movements. Cutrell, et al. [1] used eye tracking to analyze the influence of the length of a site summary (a snippet text) presented in web search results on a user’s search activities. The results of their experiments indicated that a long site summary has an effect of decreasing search time and increasing a user’s search correctness in informational search tasks where users are trying to find a specific web site or homepage. In contrast to informational search tasks, in navigational search tasks where users are trying to find web pages that include some kind of information, it has an effect on the increase of search time and the decrease of user’s search correctness. Guan et al. [2] also measured eye movements and then analyzed the influence of positions of target results that users are looking for. They found that users took longer to search the target results and were less successful in finding the targets when the search targets were placed in low positions on a search results page. Although the study provides useful insights into the design of a new interface for web search engines, it only focused on the first search results page. So it still fails to capture user activities in browsing multiple search results pages. Lorigo et al. [9] analyzed differences of task types in web search. They reported that users performing informational search tasks took longer to complete those tasks than navigational ones and spent more time to stay at pages linked by search results. However, the study also did not focus on time consumption for each search result and user interactions with multiple search results pages. In this paper, using multiple search results pages and two types of tasks (i.e., informational and navigational tasks), we measure and analyze users’ eye movements for each set of search results to provide a new understanding of user activities in web search.

An Analysis of Eye Movements during Browsing Multiple Search Results Pages

123

3 Experiment 3.1 Overview To observe how users look at each set of search results during browsing multiple search results pages, we analyzed total time of eye movements on each search result. In the experiment, participants were asked to search for appropriate web pages (target results) from the search results pages of Google to find particular information with predetermined words. The information and words were specified by experimenters. The experimenters measured eye movements of the participants during the tasks. To analyze user’s eye movements helps us understand how users browse search results. In the experiment, WebTracer [10] was used as an eye tracking system. WebTracer allows us to collect and analyze data of a user’s eye movements and operations (e.g., mouse and keyboard operations) during web search. After the tasks, participants answered a questionnaire about their usual search activities and were interviewed about observed interactions in the tasks. Participants of the experiment were 21 undergraduate students studying information science. All participants used web search in daily life and used Google as their main search engine. 3.2 Apparatus In the experiment, the following equipment was used. • Display: 21-inch LCD monitor (Viewable screen size: H30 x W40cm, Resolution 1,024 x 768 pixels) • Distance from subject’s face to display: approx. 50cm • Device for measurement of sight-line: NAC, EMR-NC (View angle: 0.28 degrees, resolution on the screen: approx. 2.4mm) • Recording and playing of sight-line data: WebTracer 3.3 Task The tasks performed by participants were (1) to search for appropriate web pages linked by search results from search results pages, (2) to find particular information specified by the experimenters and (3) to bookmark the target pages.

Fig. 1. Example of Rearrangement Search Results

124

Y. Matsuda et al.

The time limit of each task was ten minutes whether participants could complete the task or not. In the experiment, participants needed to use predetermined words and they were prohibited from changing search words during the tasks. Since the purpose of this experiment was to observe user’s activities in using multiple web search results pages, participants were only permitted to move to web pages linked by Google’s search results. The order of the tasks was counterbalanced to consider the learning effect. The tasks themselves were based on the test collection provided by NTCIR (NTCIR4 WEB) [11][12]. In this experiment, the two types of tasks were selected as follows. • Informational Task: required participants to find specific information (e.g., web pages including information on university entrance exams). The task was completed by finding three web pages linked by target results and bookmarking them. • Navigational Task: required participants to find specific web pages (e.g., official web page of the university). The task was completed by finding a web page linked by a target result and bookmarking it. Each participant performed ten tasks (five tasks for each task type). 3.4 Design of Web Search Results Pages To prevent the bias effect of the numbers of target results and their positions, we modified the results pages that were saved in a local computer when we searched with Google. The participants performed the search tasks with the modified search results pages. The previous study showed users search about 2.35 pages [3]. Therefore, we prepared three search results pages and allocated target results randomly on the search results pages. Advertisements and information irrelevant for web search were removed. We used Google’s default setting in which 10 search results are displayed at a time. In addition, Google’s original (unmodified) search results pages were used for fourth or later search results pages. Note that each search result and display position of search results followed Google’s page rank. To prevent participants from finishing their search only on the first page, we allocated target results on the second or third page. Figure 1 is an example of a way of inserting target search results.

Fig. 2. Inserted Positions of Target Search Results

An Analysis of Eye Movements during Browsing Multiple Search Results Pages

125

For the design of search results pages, we prepared four rearrangement patterns of search results (Figure 2). In the Informational Task, we displayed target results in top (I-1), middle (I-2), bottom (I-3) and even (I-4) in second and third search results page. In Navigational Tasks, we displayed target results in the top (N-1, N-3) and bottom (N-2, N-4) on the second and third search results pages. Since participants may notice the experimenters’ intention (i.e., target results were displayed only after the second page), we insert a dummy task (original search results of Google) into each task. 3.5 Experimental Procedure 1. Explanation of the experiment and preparation: Experimenters explained the experiment and the eye tracking system to participants. 2. Configuration of the eye tracking system: We configured devices for measurement of sight line and checked sight line (calibration). 3. Task for practice: To understand the flow of the experiment, participants practiced a task. The task was an Informational Task. Original search results pages of Google were used. 4. Performing tasks: Experimenters explained each search task and participants started to search. This was repeated until all the tasks were finished. 5. Questionnaire: At the end of the experiment, participants were asked to answer a questionnaire about their daily use of web search engines. 6. Interview: Participants were also asked to answer an interview about observed characteristic activities during the tasks.

4 Results 4.1 Eye Movement Figure 3 shows the example of eye movements gathered in the experiment. The vertical axis shows the position of search results and the horizontal axis shows the time of appeared sight line. In the Figure, horizontal line describes eye movements on search results and the circle shows user click to the search result. The figure showed this user searched the results from top to bottom. The total time of eye movements on the clicked result was longer than other results. 0

Eye Movements Mouse Click

)k nar 10 (lt us eR hrc ae 20 S 30 0:03

0:37

1:13

1:57 2:14 2:41 3:41 Search Time(min:sec)

4:01

4:16

Fig. 3. Eye Movement and Clicked Search Result during the Task

5:05

126

Y. Matsuda et al. Table 1. Classification of Search Completion Pages

Group

Last Search Result

Last Search Results Page

Number of Task

G0

~5

Less than 1

Informational 6

Navigational 5

G1

6~15

1

18

21

G2

16~25

2

33

38

G3

26~

Over 3

27

20

4.2 Analytic Procedure To calculate the total time of eye movements, we separated the tasks by search completion pages. The search completion page is calculated from the position of the lowest search result looked at by each subject. Table 1 shows the classification of search completion pages by group. In this paper, we would like to analyze users’ activities searching multiple search results pages. Hence, we analyze the tasks which finish at the second page (G2) and after the third page (G3). We use the length of time to analyze the eye movements. Even if a users’ gaze appears at a certain search result, it does not necessarily mean that the reviewer has interest that line. Hence, we have to distinguish a focus (i.e., interest) from users' eye movements. In this paper, we defined focus as the eye remaining on a certain search result more than 100ms. To increase correctness of the analysis, we also remove eye movements that stay a long time on a particular position. When the user reads a search result intensively, the time of eye movements to the results greatly increases. However, this increase is not the effect of display position but the effect of the content of the result itself, that is, title of the web page, snippet (description of the web page), and URL. To distinguish a user’s intensive reading, the average reading time of clicked search results was adopted. Basically, users read the results before the clicking, to decide to move to the web page. Hence, it is reasonable to remove the eye movements to search results that stay more than the average time of clicked search results. In this experiment, the average time in the Informational Task was 3.18 seconds, and 2.44 seconds for the Navigational Task. 4.3 Analysis Result Figures 4 and 5 describe the mean time of eye movements on each search results classified as G2 and G3, respectively. The vertical axis shows the mean time of eye movements and the horizontal axis shows the rank of results. In both groups, users tend to view search results longer in informational tasks than navigational tasks. To evaluate effects of search result positions on user activities in web search, we calculated the total time of eye movements on top search results and bottom search results. Table 2 shows the average time of eye movements for the three results that were ranked high and displayed at bottom of the page (HB), and for the three results that were ranked low and displayed at top of the page (LT) in G2. Table 3 shows the average time of eye movements for the three HB results and that for the three LT results in G3. The table describes that the mean time of eye movements to LT is

An Analysis of Eye Movements during Browsing Multiple Search Results Pages

yeE fo e im T

）c 3 es 2.5 （ 2 st ne 1.5 1 em vo 0.5 M 0

Informational Task

1

3

5

7

9

11

13 15 17 19 Search Result （rank）

21

127

Navigational Task

23

25

27

29

Fig. 4. Mean Time of Eye Movements on each Search Result in Informational Task (square) and Navigational Task (triangle) of G2

yeE fo em iT

c）e 3 s（ 2.5 st 2 ne 1.5 m ev 0.51 oM 0

Informational Task

1

3

5

7

9

11

13 15 17 19 Search Result （rank）

21

Navigational Task

23

25

27

29

Fig. 5. Mean Time of Eye Movements on each Search Result in Informational Task (square) and Navigational Task (triangle) of G3 Table 2. Mean Time of Eye Movements for High-Rank Results Displayed in Bottom Area 3 and Lower-Rank Results Displayed in Top Area 3 in G2

Group

Search Results

LT_1 HB_1 LT_2 HB_2

1~3 8~10 11~13 18~20

Mean Time of Eye Movements (sec) Informational Navigational 1.67 1.24 1.46 1.04 1.51 0.96 0.66 0.79

Table 3. Mean Time of Eye Movements for High-Rank Results Displayed in Bottom Area 3 and Lower-Rank Results Displayed in Top Area 3 in G3

Group

Search Results

LT_1 HB_1 LT_2 HB_2 LT_3 HB_3

1~3 8~10 11~13 18~20 21~23 28~30

Mean Time of Eye Movements (sec) Informational 1.38 1.56 1.43 1.25 1.48 0.75

Navigational 0.95 0.88 0.84 0.83 0.95 0.61

almost the same as HB, or longer than HB in some cases (e.g. between HB_2 and LT_3 at G3). This result indicated users did not view the search results in proportion with page rank.

128

Y. Matsuda et al.

In particular, we focused on the top-three search results (LT) of each page (see Figures 4 and 5). In Figure 4 (eye movements of G2), the mean time of eye movements to the first result of each page (ranks 1 and 11) was shorter than the second and third results (ranks 2, 3, 12, and 13) at both task types. Also at G3 (Figure 5), eye movements to the first result of the page were shorter than the second and third search results.

5 Discussion 5.1 Effect of Task Differences The result of the experiment describes that users tend to view search results longer in informational tasks than navigational tasks. In the Navigational task, users read the snippet of the search result, then decide to click the search result or not. On the other hand, in the Navigational Task, users read the title and URL of the result instead of the snippet to decide to click the result. Reading the title and URL of the result requires less time than reading the snippet, therefore the length of the eye movements in Informational Tasks is longer than Navigational Tasks. The result suggests that when users browse multiple web search results pages, they adopt different reading patterns for each task. 5.2 Effect of Position within a Result Page/Screen The result of the experiment describes that time of eye movements on LT is longer than HB. The result shows that users are impressed not only by the rank but also the position of the search results within the results page. That is, the time length of eye movements on the search results is influenced by the position within a results page. The detailed analysis showed the time of eye movements on second and third results in each page are longer than the first result. This result suggests users’ eye movements are attracted by the position on the screen. In the experiment, users viewed the middle area of the screen more than the top area. The second and third results are displayed on the middle of the screen when the user went to the search results page. On the other hand, the first result is displayed on the top of the screen. The first result is turned out by the scroll, hence, the user interest moved to the second/third results. This assumption was verified from interviews of the subjects. 5.3 Design Implications Using the results of the experiment, we propose a design encouraging users to browse the search results based on the rank. To increase the time of the eye movements of the users, the results displayed on the bottom of the page should be emphasized to get more attention from the users. In the Navigational Task, users concentrated their eye movements on the title of the web page or URL since the result page was a goal of the task. Hence, a thumbnail and/or attribute (e.g. the official page or blog) of the Web page are useful information for the users searching a specific web site.

An Analysis of Eye Movements during Browsing Multiple Search Results Pages

129

6 Conclusion In this paper, we experimentally analyzed the effect of the result position on the results pages. In the experiment, we measured users’ eye movements during web search tasks to analyze how long users spend on each result of the results pages. As a result, we found the results displayed on the bottom of the page were viewed for a shorter time than the results displayed on the top of the next page. Also, we found the tendency that the second/third results of each page were viewed longer than the first result of the results page. As a future work, we will analyze the effect of the display position on the screen. Acknowledgments. We would like to thank all the participants. In this paper, we used a part of the data of the NTCIR-4 WEB task that is sponsored by the National Institute of Informatics, as organizers of the NTCIR-4 WEB task project.

References 1. Cutrell, E., Guan, Z.: What Are You Looking For?: An Eye-tracking Study of Information Usage in Web Search. In: CHI 2007: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 407–416 (2007) 2. Guan, Z., Cutrell, E.: An Eye Tracking Study of the Effect of Target Rank on Web Search. In: CHI 2007: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 417–420 (2007) 3. Jansen, B.J., Spink, A., Saracevic, T.: Real life, real users, and real needs: a study and analysis of user queries on the web. Inf. Process. Manage. 36(2), 207–227 (2000) 4. Broder, A.: A taxonomy of web search. SIGIR Forum 36(2), 3–10 (2002) 5. Rose, D.E., Levinson, D.: Understanding User Goals in Web Search. In: WWW 2004: Proceedings of the 13th international conference on World Wide Web, pp. 13–19 (2004) 6. Murata, T., Saito, K.: Extraction and Visualization of Web Users’ Interests Using SiteKeyword Graphs. Journal of Japan Society for Fuzzy Theory and Intelligent Informatics 18(5), 701–715 (2006) (in Japanese) 7. Clarke, C.L.A., Pan, B., Agichtein, E., Dumais, S., White, R.W.: The Influence of Caption Features on Clickthrough Patterns in Web Search. In: SIGIR 2007: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 135–142 (2007) 8. Otsuka, S., Toyoda, M., Kitsuregawa, M.: A Study for Analysis of Web Access Logs with Web Communities. Transactions of Information Processing Society of Japan 44(18), 32– 44 (2003) (in Japanese) 9. Lorigo, L., Pan, B., Hembrooke, H., Joachims, T., Granka, L., Gay, G.: The influence of task and gender on search and evaluation behavior using Google. Inf. Process. Manage. 42(4), 1123–1131 (2006) 10. Sakai, M., Nakamichi, N., Shima, K., Nakamura, M., Matsumoto, K.: WebTracer: A New Web Usability Evaluation Environment Using Gazing Point Information. Transactions of Information Processing Society of Japan 44(11), 2575–2586 (2003) (in Japanese)

130

Y. Matsuda et al.

11. Eguchi, K., Oyama, K., Aizawa, A., Ishikawa, H.: Overview of the Informational Retrieval Task at NTCIR-4 WEB. In: Proceedings of the Fourth NTCIR Workshop on Research in Information Access Technologies Information Retrieval, Question Answering and Summarization (2004) 12. Oyama, K., Eguchi, K., Ishikawa, H., Aizawa, A.: Overview of the NTCIR-4 WEB Navigational Retrieval Task 1. In: Proceedings of the Fourth NTCIR Workshop on Research in Information Access Technologies Information Retrieval, Question Answering and Summarization (2004)

Development of Estimation System for Concentrate Situation Using Acceleration Sensor Masashi Okubo and Aya Fujimura Doshisha University, 1-3 Miyakodani, Tatara, Kyotanabe, Kyoto, 610-0321, Japan [email protected], [email protected]

Abstract. Recently, to discipline to increase powers of concentration is popular. One of the reason, it is difficult to concentrate something in these days because of a flood of information. However we discipline our concentration by using the how-to books and the portable games, we cannot evaluate the training effect on the practical life. In this paper, we propose an evaluation system for user’s powers of concentration in which the method for the estimate user’s sitting situation is utilized. This system is constructed by two kinds of method, one is the method which estimates the sitting situation and the other is the evaluation method for user’s powers of concentration situation. These methods use user’s motion that is obtained from the acceleration sensor that is fixed on the chair. And we prepare the three kinds of Graphical User Interface (GUI) which presents the concentration situation to the user. Keywords: Powers of concentration, GUI, Sensory evaluation and Selfmanagement.

1 Introduction The modern society can be described as information society because of a flood of miscellaneous information. It is getting difficult to handle with these information obtained by Internet and e-mail as well as newspapers and televisions with full understanding. We have a difficulty to deliberate and concentrate something in our life environments. On the other hand, since the launch of game software aiming at brain activation, commonly known as ‘Notore’, which means brain training, various games to train brains, how-to books for improving our concentration are becoming popular among wide range of age groups． One of main factors of this phenomenon is attributed to middle-aged and senior people’s demands for prevention of brain aging and wide-range age groups’ desires for obtaining abilities to deal with a large number of information in a short time. Since these trainings are mostly performed by recitations and simple calculations, users can improve the ability as though they were playing some kind of games. Moreover the results of the trainings can be described as ‘age of brain’ so that they can easily check their abilities. This is one of the reasons that ‘brain training’ software are widely accepted among various age groups. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 131–140, 2009. © Springer-Verlag Berlin Heidelberg 2009

132

M. Okubo and A. Fujimura

However, even our thinking ability and concentration powers can be improved in such games and trainings, we cannot evaluate the effects in the practical life. Currently, the method using biosignals including breathing, heartbeat, and brain waves…etc. is proposed in order to measure person’s powers of concentration [1]. For example, it is found that a brain wave called Frontal Midline theta is generated when solving a problem and concentrating on abstract thinking. Also it has been researched that transition of learner’s skin potential level meets his interest in lectures. However, this sort of method put a heavy burden on a measured person as well as it is costly and time-consuming. Therefore in this study, assuming the situation as if users are working with a computer, studying and doing some paper works in sitting situation, we propose an evaluation system for user’s powers of concentration by processing information obtained by motion sensor fixed on a chair and presenting the concentration situation to the user. Along with development of the actual system, we also prepare the three kinds of Graphical User Interface (GUI) and perform the sensory evaluation experiments for the proposed system and the interface evaluation using these GUI.

2 Hardware Configuration of Estimation System in Sitting Situation Hardware configuration of proposed system is showed as Fig. 1. The motion sensor sends acceleration data to Personal Computer (PC) through Bluetooth communication after measuring a movement of a chair. User’s sitting or standing up situation and his power of concentration are estimated on PC according to the received data, and the estimation result is presented to the user. Monitor

User Bluetooth communication Acceleration sensor

PC

Chair

Fig. 1. System configuration

In the proposed system, the user’s sitting situation on a revolving chair that is generally used in households and offices as shown in Fig. 2 is estimated. Since the revolving chair reflects the various users’ movements, we can estimate the user’s movements by measuring the acceleration of the movements and swings using the motion sensor.

Development of Estimation System for Concentrate Situation

133

We use a remote controller of Nintendo Wii games as the motion sensor. This remote controller uses Bluetooth for connecting to the main unit so that we can easily connect to PC. Since Wii game is available at a relatively low cost, it has already been popular among many households. Fig.2 shows that we fix the remote controller on the upper side of the backrest of the chair because the inclination of the backrest and the revolution of seating face appear prominently there and yet it doesn’t disturb the user. The three dimensional motion sensor is installed near A button located around the center of the remote controller surface. Fig.2 also shows the coordinate axes with putting it lengthwise: X axis is horizontal direction, Y axis is vertical direction and Z axis is depth direction [2].

Y X Z

Fig. 2. Coordinate of 3-axis accelerometers

3 Software Configuration of Estimation System in Sitting Posture 3.1 Estimation of Time of Sitting and Leaving This proposed system estimates if the user is sitting down or standing up and user’s powers of concentration in sitting situation. At first we propose the method for estimating if the user is sitting down or standing up. There are the following two kinds of methods for estimating user’s situation. The first one is a method for estimation by momentary change of acceleration at the time of sitting down and at the time of standing up. The second one is a method for estimation by the change of acceleration along with time. In the proposed system, we use these two methods to ensure the reliability. Estimation by Momentary Change of Acceleration Speed. We sample the acceleration around 100Hz from Wii remote controller via Bluetooth. We instruct several subjects to sit down, work and stand up the chairs with the remote controller fixed on and perform the preparatory experiment to measure the acceleration of these movements. As a result, we find that the accelerations significantly change at the time of sitting down and standing up in the all axes directions. However, we also find that relatively greater acceleration is measured in X-axis and Z-axis at the time of sitting down. Therefore we estimate whether sitting down or standing up by Y-axis acceleration which shows less change. Moreover we focus on momentary change of

134

M. Okubo and A. Fujimura

acceleration at the time of sitting and standing up, respectively. Fig. 3 shows an example of transition of Y-axis acceleration at the time of sitting down and standing up. It shows that the acceleration shifts to a negative direction after shifting greatly to a positive direction. This occurs due to the rise of the seat front as a reaction when the subject is being seated and the seat front sinks down once. On contrary, the acceleration shifts to a positive direction when standing up after shifting greatly to a negative direction. This occurs due that the seat front sinks down as a reaction when the subject is standing up and the seat front rises, which means that the Y-axis acceleration shifts from a positive direction to a negative direction when sitting down and shifts from a negative direction to a positive direction when standing up. Moreover, these shifts of the acceleration such as positive to negative by sitting down and negative to positive by standing up occur within 50ms. Using these shift patterns of the acceleration, we estimate the subjects’ sitting down and standing up situations. Acceleration G(×9.8m/s2)

Time(ms) (a) When sitting down

Time(ms) (b) When standing up

Fig. 3. Transition of the Y-axis acceleration by (a) sitting down and (b)standing up

Now we perform the preparatory experiment for setting the threshold value against the acceleration in order to distinguish the movements among sitting down, standing up and working in sitting situation. Among the nine subjects, maximum value of 0.27G is measured by two subjects and minimum value of -0.23G is measured by another subject. Therefore we set the threshold value of the acceleration to estimate sitting down and standing up as 0.3G in the positive direction and -0.3G in the negative direction. We summarize the estimation method for sitting down and standing up using the momentary acceleration change. At first if more than 0.3G acceleration is measured, and if less than -0.3G is measured within 50ms during standing up, we estimate the subject is sitting down. Also if less than -0.3G acceleration is measured during sitting down and if more than 0.3G acceleration within 50ms is measured during sitting down, we estimate the subject is standing up. Estimation Using the Acceleration Changes with Time Course. The acceleration measured at the time of sitting down and standing up varies with each individual. Sometimes it is difficult to estimate if the acceleration occurred by sitting down and standing up or movements generated during sitting down. According to the maximum and minimum value of Y-axis acceleration when sitting down and standing up, the

Development of Estimation System for Concentrate Situation

135

acceleration of one subject shows less change when sitting down; at 0.11G and -0.19G. If the acceleration is more than 0.1G and less than 0.3G, and if it is less than -0.1G and more than -0.3G when sitting down and standing up, the estimation will be inaccurate simply by using the momentary acceleration. Therefore, we also use the estimation method using the power spectrum sequentially obtained by sampling the acceleration in a certain period of time. At first we seek the power spectrum by executing Fourier transform of 256 pcs sample data among about 2.5s acceleration in the three axes. The frequency component 0.01Hz being regarded as noise is eliminated. We can find that the power spectrum is barely measured in the all frequency domains when absence but is measured in many frequency domains during sitting. Then we add up the value of power spectrum except the spectrum at 0.01Hz frequency. We will judge between absence and sitting by sum of the power spectrum obtained as above. Fig. 4 shows the transition of sum of 3-axis acceleration power spectrum when absence, sitting, and working by using the acceleration data obtained by preparatory experiment Sitting Not available down

Working

Sum total of diff of acceleration

Rotation of the seat

Inclining the rear

Standing up

of

X-axis Y-axis Z-axis

Time(s)

Fig. 4. Transition of sum of 3-axis acceleration power spectrum

Comparing between absence and working in Fig.4, we find that the sum of Y axis accelerations power spectrum does not change very much comparing to that of other axes. On the other hand, the sum of X and Z axis acceleration power spectrum changes greatly depending on the user’s motion. For example, the sum of X axis acceleration power spectrum increases by the user’s motion like rotating the seat front and the sum of Z axis acceleration power spectrum increases by the user’s motion like inclining the backrest of the seat. We summarize the estimation method for sitting down and standing up using the power spectrum. In case more than -0.3 G and less than -0.1G acceleration is measured when absence and the sum of power spectrum exceeds 50 afterwards, we estimate for “sitting down”, otherwise we estimate for “absence”. On contrary, in case more than -0.3G and within -0.1G acceleration, or more than 0.1G and under 0.3G acceleration are measured and the sum of power spectrum is between 5s and 10s, we estimate for “standing up”, otherwise we estimate for “working”.

136

M. Okubo and A. Fujimura

3.2 Relationship of Sitting Situation and Power Spectrum Experiment Objective. Generally, it is considered that a person’s motion strongly tends to lessen when concentrating. That is, when concentrating, it is highly possible that the sum of X-axis and Z-axis acceleration power spectrum obtained by the motion sensor fixed on the chair (hereinafter called the sum of the acceleration power spectrum) become smaller. We perform the preparatory experiments in order to examine this assumption. In the experiments the subjects are instructed to type on the computer for quantitative evaluation for concentration. Experiment ・ Evaluation Method. In this experiment we examine the relationship between the sum of the acceleration power spectrum and the typing speed per unit of time while three subjects engage in typing on the computer for 30 minutes. The subjects are instructed to sit down the chair for 30 minutes and engage in either typing or taking a rest. Except the first 1 minute and last 1minute in each of 30-minutes data, we divide the remaining 28-minutes data into 1 minute to find the cross-correlation functions. Evaluation result. The transition of the sum of power spectrum and typing speed, and the examples of the cross-correlation function is as shown in Fig.5. According to Fig. 5 (a), we find that the sum of power spectrum increases when the typing speed becomes 0, when taking a rest. Also Fig.5 (b) shows the fact that approximately -0.7 negative correlation is obtained due to the result as mentioned above. Fig.5(c) shows the decrease of the sum of power spectrum when increasing the typing speed. Fig.5 (d) shows that approximately -0.7 negative correlation is obtained likewise. The average and standard deviation of the minimum of cross-correlation function of the three subjects are as shown in Fig.6. The average of over -0.4 negative correlation function can be seen in Fig.6. Thus we understand that the strong negative crosscorrelation exist in the sum of the power spectrum of user’s motion and the typing speed per unit of time. Also, the validity of estimation that the subjects are concentrating when the sum of the power spectrum become smaller is clarified. Power spectrum Time(s) (b)

60

Time(s) (a)

30

60

Time(s) (c)

Power spectrum

0

Typing speed Power spectrum of acc.

Time(s) (d)

Fig. 5. Cross-correlation functions (b),(d) between power spectrum of user’s motion and typing speed(a),(c)

Development of Estimation System for Concentrate Situation

Subject A

Subject B

137

Subject C

Fig. 6. Average and standard deviation of the minimum of cross-correlation function

3.3 Interface Set-Up Recently, cars installed instantaneous fuel consumption meters are popular in the market. Drivers are likely to loosen gas pedals because of this meter by knowing his situation at the moment and by reviewing his own behavior. That is also the aim of this proposed system. The situations of users who are sitting are described with avatar so that the users can monitor their own situation objectively. The four types of avatars are prepared: “NOT AVAILABLE”,“AVAILABE”, “WORKING”, “PLAYING” and two patterns such as “male”(upper) and “female”(lower) are created. If the system is used as a self-management tool, it becomes important not only to present the user’s situation whether concentrating or not concentrating, but also to present the powers of the concentration. Therefore, as well as displaying the avatar, we create a line graph that present the powers of concentration in the long term and three patterns of GUI using a level meter to describe momentary powers of concentration. Fig.7 (a) shows the GUI displaying the avatar only. In the Fig. 7 (b), GUI that presents transition of the powers of the concentration is added to the avatar. As we assume that the less the sum of the power spectrum decreases, the more the degree of the concentration increases, we set the upper limit of the sum of the power spectrum at 500. Then we reduce the sum of the power spectrum from 500 so that the line goes up when the Graph of power of concentration while long term

Line graph of power of concentrations while short term.

Level meter (a) GUI (avatar only)

(b) GUI (with line graph)

Fig. 7. Graphical User Interface

(c) GUI (with level meter)

138

M. Okubo and A. Fujimura

degree of the concentration becomes high. The graph in the lower right of GUI presents the powers of the concentration in the previous 30s. Also the graph in the upper side of GUI presents the powers of the concentration for a last few minutes for users to check the record. In Fig. 7 (c), the level meter which displays the momentary change of the powers of concentration nonlinearly is added. Even if the sum of the power spectrum is small when concentrating, up down of the scale of the meter is visually confirmed by decreasing the sum per one scale.

4 System Sensory Evaluation Experiment 4.1 Experiment Objective We examine the usefulness and usability of the proposed system and understandably of the powers of concentration presented in three types of GUI. To be more specific, we confirm the operation capability of the estimation system for powers of concentration and perform the sensory evaluation experiments for aforementioned three types of GUI. 4.2 Experiment Conditions Methods In the experiment, ten subjects ( five males and five females in their twenties) are instructed to sit down the chairs with Wii remote controller fixed on, and to engage in typing on computers which the proposed system are running (Fig.8). They are asked to type letters printed on papers in word processing program for five minutes. The GUI is displayed in the upper left of the screen so that it will not disturb their works and the subjects can check it during the works. GUI

Fig. 8. Experimental scene

The experiments are conducted under the four kinds of conditions as follows: (1) without the proposed system, (2) with GUI using only avatar, (3) with GUI using avatar and line graph, and (4) with GUI having avatar and level meter. Taking into account of the order effect, the order of the experiment conditions changes every subject. After the experiment, we give the subjects evaluation questionnaires by five grades. 4.3 Result of Experiment The average and standard deviation obtained by scoring the answers of questionnaire on understandability of the powers of concentration and the interface is shown in

Development of Estimation System for Concentrate Situation

139

Fig.9. These results show that comparing to when having no system and having GUI using avatar only; the subjects finds easier to understand their degree of the concentration when having GUI with the line graph and level meter. Also we conduct a sign test for all combinations in all questionnaires. About the understandability of the powers of the concentration, significant differences at 5% significant level are seen among the combination between having no system and having the line graph, and between having no system and having the level meter. This result also shows that GUI using the line graph and level meter is easier to understand the powers of concentration than having no system. 4.4 Discussion The GUI with the line graphs and the one with level meters were preferred alike as mentioned previously. The subjects who prefer the line graphs evaluated the ability to check the archival record and the others who prefer the level meter evaluated the ability to understand the powers of concentration. However, some said that they were preoccupied by the system and couldn’t concentrate. So the system which users can choose their favorite GUI and save the archival record to look back the day is desirable. Simplicity of understanding for power of concentration

P<0.05

Good

Neutral

No good No system Only avatar Line graph Level meter

Fig. 9. Result of questionnaire

5 Conclusion This study aims that users conduct self-management by presenting their situations while they are using the systems. We develop the systems which estimate user’s powers of the concentration and sitting down and standing up situation, and also present the powers of concentrations using interface to the users. We conduct further experiments on the sensory evaluation to evaluate the operation effectiveness of the systems and interface. As a result, interface using the line graphs which presents the transition of the long-term powers of concentration and interface using the level meter which presents transition of the momentary powers of concentration are preferred equally by the users.

140

M. Okubo and A. Fujimura

The proposed system is developed in order to review users’ own behaviors by understanding their own situations. Although we consider that this system enables to facilitate the concentration, long hours of concentration turns out to be stress and have a bad influence on mental and health. Therefore, we think it is possible to build up the system to facilitate taking appropriate rests during long hours of concentration. We will validate its effectiveness in the long-term experiments along with improvement of the existing system and interface.

References 1. Tamura, H.: Human Interface. Ohmsha, 44–68 (1998) (in Japanese) 2. http://www.wiili.org/index.php/Wiimote

Psychophysiology as a Tool for HCI Research: Promises and Pitfalls Byungho Park Graduate School of Information and Media Management Korea Advanced Institute of Technology (KAIST) 207-43 Cheongryangri 2-dong, Dongdaemun-gu, Seoul, 130-722, Korea [email protected]

Abstract. Psychophysiology, an area of psychology that measures individual's physiological responses to refer to one's psychological state, can provide a set of useful measures HCI researchers can take advantage of. However, there are limitations to the method itself and room for misinterpretation. This paper introduces psychophysiology, and also shows how research methods psychophysiology offer can be used for HCI research, advantages and disadvantages of using research tools from psychophysiology.

1 Introduction A student essay written and posted online by MIT in 1998 [14] reads "...from studying how human physiology and psychology, we can design better interfaces for people to interact with computers. Work in this domain is only beginning (indeed the number of papers written on this topic has increased in the past few years), and there is much that we don't yet know about the way the human mind works that would allow more perfect user interfaces to be built. (p.4)" Yes, ten years has passed. Though it may have not been as magnificent as this (presumably) young student predicted, there were small yet solid advances made here and there in the area of psychology research. There were also technological advances made for physiology research. Today, as Kim et al. [7] put, HCI researchers are calling for study of human interaction, cognition and physiology to apply such knowledge in system development and implementations. However, it seems that measuring and analyzing physiology data is still not commonly found in HCI research today. Part of the reason may be entrance barrier (understanding of the relationship between human psychology AND physiology, learning to collect physiological data, high cost of physiology equipment, etc.). Another reason may be lack of effort to introduce and share knowledge on using physiology for HCI research. This paper is an attempt to introduce the advantages (and also warn about the possible mistakes made) of applying psychophysiology for HCI research. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 141–148, 2009. © Springer-Verlag Berlin Heidelberg 2009

142

B. Park

2 Psychophysiology and HCI 2.1 Psychophysiology as an Academic Discipline Psychophysiology as an academic discipline that studies the interrelationships between the physiological and psychological aspects of human behavior [15]. A typical study in this area observes human’s physiological responses to understand human’s psychological status and/or changes. Due to its interdisciplinary nature, research from various disciplines including, but not limited to, psychology, cognitive science, medicine, anatomy and neuroscience are incorporated. The type of physiological responses used for psychophysiology studies include blood flow pattern in the brain (functional magnetic resonance imaging; fMRI), heart rate variability, sweat production (skin conductance; SC), respiration, Electroencephalography (EEG; commonly known as ‘brain wave’), muscle contraction (Electromyography; EMG), eye movement, and much more. 2.2 Psychophysiology Providing Tools for HCI Research A wide variety of techniques ranging from qualitative to quantitative are used in HCI studies. The qualitative ones such as in-depth interviews and focus group interviews can provide great depth of insight. However, they are open to misinterpretation and requires a great deal of experience to avoid this. The quantitative ones can be further divided to self-reports and observational ones. Former includes questionnaire and survey, and latter includes response time, counting number of errors (or attempts), and so on. The advantage of using quantitative methods is that the data produced can be used for inferential statistical analysis, providing insight beyond descriptive statistics. However, many quantitative techniques, especially the self-report types, are also open to serious errors. The data may be less-than-accurate because the subject cannot remember certain things clearly, or even motivated to lie (e.g., “have you voted during the last election?”). The research tools psychophysiology offers are quantitative and observational. What makes these tools interesting is that they provide a way to tap into human mind. A large body of literature in psychophysiology provides a fairly solid ground for interpreting the data. In other words, it can provide an alternative way to find how much attention is being paid to the task or how difficult the interface is to use – both which subjects might provide inaccurate answers to when asked using a self-report questionnaire, depending on circumstances.

3 Useful Psychophysiological Research Techniques As mentioned above, a large variety of physiological responses are subject for study in psychophysiology. This section will introduce a few of them that HCI researchers are more familiar with, or may find more useful than others.

Psychophysiology as a Tool for HCI Research: Promises and Pitfalls

143

3.1 Sweat Production (SC) and Stress It is known that people sweat under stress. This is also true when computer users are stressed out from the difficulty of the task given. In order to find how easy or difficult an interface is to use, HCI researchers may measure the amount of sweat production, or skin conductance (SC) of the subject [11, 16]. Deep down, skin conductance is directly related to the activation of sympathetic system of the central nervous system. This is good because it means skin conductance is independent from the activation of the parasympathetic system, which can cause misinterpretation of the activity of many human organs or systems which are under influence of both central nervous systems (for example, human heart may beat faster because the sympathetic system is activated, or because the parasympathetic system is deactivated, or both). However, there are downsides for using skin conductance. One of them is the sensor (electrode) placement. Sweat glands in humans are concentrated in palms and soles. Many computer-related tasks and their interfaces require users’ hands. Collecting skin conductance data from the palm may restrict subjects from using both hands freely. Furthermore, in certain tasks that require both hands (such as typing text with a keyboard), using palm is out of question. Alternatively, sole of the foot may be used, but this is inconvenient since subjects are required to remove their socks and have their foot kept lifted through the whole session to prevent electrodes touching the floor. Using skin conductance as an index of task (or interface) difficulty is it’s slow response speed. Rather than constantly producing sweat, human sweat glands ‘spout out’ sweat. It takes about six seconds for sweat glands to respond to an arousing event, though exact timing varies by individual. This makes it hard for researchers to pinpoint the exact timing of a particular event that causes stress in the human subject. For this reason, comparing the total amount sweat or average skin conductance level over time within a subject during different tasks (which takes longer than 10 seconds to complete) is recommended rather than attempting to identify the exact moment that causes stress [15]. 3.2 Heart Rate (HR) Variability and Attention Human heart is affected by both sympathetic and parasympathetic systems of the central nervous system. This means that its activity is affected by both arousal and paying attention to external stimuli. Research shows that the speed of heart beat slows down when one is paying attention to a stimulus presented [17]. This can be explained from an evolutionary psychology perspective: when an unknown change is noticed in the environment, the body automatically responds to it by calming down (or slowing down) the activities of the internal organs until proper assessment of the situation is made. If the change turns out to be life-threatening, then the famous ‘fight or flight’ reaction, which includes intensive activity in sweat glands and quick acceleration of heart rate follows. If the change does not pose any harm to human, then the heart rate gets faster, but no faster than the normal speed.

144

B. Park

It is important to acknowledge that deceleration of heart rate is an index of external attention. Measurement of external attention is useful for HCI research since it serves as an index of whether the subject is paying attention to the menu of the computer screen, or whether that gentle chime has actually got the subject’s attention. The other type of attention is called internal attention, which happens when one puts mental effort to solve math questions [9]. When this happens, heart rate acceleration is observed. Hence, it is important to see the context of the settings heart rate data has been collected and determine whether the heart rate getting fast is a heart rate acceleration which is a byproduct of internal attention, or simply a heart rate deceleration from not paying attention. Also to put into consideration is the subject’s arousal level – excitation leads to arousal in the sympathetic system, which triggers heart to beat faster. When analyzing heart rate data, context matters a lot. There are two ways to measure heart rate. One is to measure the electrical pulse produced by the human heart every time it contracts and pumps blood out to the whole body. This measurement is called electrocardiogram, or ECG. The other way is to measure the blood flowing in and out to the tip of the finger and/or toe, which is called photoplethysmography (PPG). Photoplethysmography is typically measured by emitting infrared light into the skin. The level of light absorption changes by the amount of blood flowing underneath the skin, and this is used to measure heart rate. ECG monitors the electrical activity of heart, while PPG monitors the mechanical activity of heart. In a perfect world, both has to be able to serve as equally perfect indices of heart activity. But neither is perfect. ECG requires three electrodes, attached on both arms, both legs, or above the chest. Chest placement is rarely used outside of a hospital setting, though it provides the clearest ECG. Attaching electrodes on arms or legs is more practical for HCI research, but the distance from the heart tends to make it more vulnerable to noise (caused by subject’s body movement and internal organ activities) and weak pulse. From my personal experience, arms tend to provide cleaner ECG than legs. PPG needs to have one sensor attached to a finger or a toe. Generally, fingers are better than toes for collecting PPG data because they are closer to the heart. Nevertheless, if the subject’s heart is relatively weak and the hand or foot is in a position that makes it hard for blood to flow in, there may not be enough blood flowing in and out for the infrared light sensor to notice and record the PPG. For these reasons, both are somewhat vulnerable to errors, especially by missing individual heart beats. However, many of these issues can be prevented by preparing the best possible settings for the subject. As long as the study condition is fine, ECG is as reliable as PPG [13]. Sensor placement again can be a problem: ECG requires three locations on the arm or the leg. PPG may require only one but it has to be either a finger or a toe. And it is best to avoid the sensors touching hard surface. As a result, type of data to be collected for heart rate and location for sensor placement depends on the type of task for the subjects. 3.3 Eye Tracking and Attention As the maxim “eye is the window to mind” suggests, it has been long thought that human gaze reflects the top priority of cognitive processes [6]. This is one of the

Psychophysiology as a Tool for HCI Research: Promises and Pitfalls

145

reasons HCI researchers find eye tracking to have great potential to be an important research tool. In the past, electrooculography (EOG) that measures resting potential of the retina was used to track the eye movements. However, EOG can only show to which general direction the eye has moved to; not exactly where the eye gaze if fixed to [6, 15]. With advances in real-time optical data processing, monitoring of the retina by using infra-red light became more sophisticated. Data produced by eye tracking is useful not only since it provides quantitative data that can be subjected for statistical analysis, but also because it can be visualized in ways that are intuitive for audience when presented. One is heat map, which colorcodes each area of the computer screen where usually red is colored to the area that the eye fixation happened longest and dark blue (or no color) represents little or no eye gaze in the area. The other is an animation of which part of the screen the gaze has been moved to (which, some companies call “gaze replay”) in accordance to time. When used properly, both visualization techniques are powerful enough to convince audience with little or no knowledge in statistics [3, 4, 6]. In the past, eye tracking equipments required a head-mounted camera that monitors the retina movement and alto a head-mount that fixes the subject’s head since the equipment was not capable of making adjustments according to head movements. Today, there are equipments in the market that does not require head-mounted cameras. Some of them are capable to make adjustments to minor (and gentle) head movements. However, eye tracking technology available today still has its own limitations. The largest challenge is that it is hard to use on subjects with seriously bad vision. This is not because of the vision itself, but because of the lenses (both glasses or contact lenses). In theory, the equipment should be able to be adjusted for such lenses, but in practice, virtually no machine in the market provides such calibration. However, there are indeed quite a few equipments that automatically adjust to subjects who wear glasses for relatively light myopia. Incapability to track vision to sudden and/or fast head movement is another limitation for contemporary eye tracking technology. 3.4 Facial Muscle Activity (EMG) and Emotion Muscle fibers of human (as well as other animals) contract when they are triggered by electrical current. Electromyography (EMG) is a measure of the electrical activity of muscles, which directly index muscle activities. Facial EMG refers to indexing of facial muscle activities. Psychophysiology researchers have been using facial EMG to monitor emotional changes in human [5, 12]. There are three muscle groups monitored for this purpose: Currugator supercilii, Zygomatic major, and Orbicularis oculi [15]. The Corrugator supercilii muscle group is the “frowning” muscle. It is located between the nose and the forehead and when activated, it draws the eyebrow slightly downward and creates vertical wrinkles between the eyebrows. Corrugator activity is typically used as an index of negative emotion. The Zygomatic major is a muscle group that is located at both ends of the lips. When activated, it pulls the lips from both sides and mostly used when one is smiling. For this reason, it is used as an index of positive emotion. However, it is also

146

B. Park

activated when one is experiencing other types of non-positive emotion (e.g., One screaming “Eeeek!” out of disgust), so the data has to be analyzed with caution. The Orbicularis oculi is another muscle group that is getting attention as an alternative for the Zygomatic major muscle group as an index for positive emotion. The Orbicularis oculi muscle group is located right below the lower eye lids, and it is primarily responsible for eye blinks. It is also responsible for gathering of skin around the eyes when one is smiling, which is called the Duchenne smile. A Duchenne smile involves contraction of both the zygomatic major muscle group and the orbicularis oculi muscle group, and it is seen as an indicator of one experiencing true happiness (a Non-Duchenne smile only involves the zygomatic major muscle group and interpreted as an indicator of social smiling which does not involve happiness; for review, see Ekman, Davidson, and Friesen [2]). One of the challenges using facial EMG is that the muscle groups are small and the electrical activity is very weak. Especially, orbicularis oculi muscle group is so small that researchers are forced to place two mini electrodes (diameter of 4-millimeters) very close (about 5-millimeters), which makes it vulnerable to errors during data collection.

4 Conclusion: No One Measure Is Perfect This paper has reviewed skin conductance, heart rate, eye tracking, and facial EMG as tools for HCI research. There are, of course, other tools such as Electroencephalography (EEG; also known as ‘brain wave’), functional Magnetic Resonance Imaging (fMRI; also known as ‘brain imaging’), and many more. Each has its own unique advantages and disadvantages as a tool for research. Though psychophysiology promises to provide useful research tools for HCI, there are limitations that generally apply to them [8]. The largest one is external validity. As mentioned above, skin conductance, heart rate, facial EMG all requires some kind of sensor to be attached on parts of the subject’s body. And this adds to the awkward feeling to already unnatural setting (e.g., being put into a usability lab with a one-way window and, possibly, video camera rolling) the subject may experience during a session. Eye tracking is no less artificial than other measurements. The least intrusive technology still requires participants keep their head movement minimal and if needed, move gently. The subject has to be reminded of this, and this inevitably causes unnatural tension in the subject which, depending on the type of study, may have fairly large negative impact on the data collected. Another limitation is that the data collected is open to interpretation. Though many people tend to believe psychophysiology provides an absolutely objective data and interpretation. For example, Hornbaek (2005) argues that physiological measures are objective, using "physiological measures of fun in playing computer games (p.92)" as an example. Unfortunately, this view can be wrong – especially in the context of HCI research. Studies in psychology tend to use simple stimuli, which helps avoid unexpected factors to interfere. But in many cases, HCI research has to use real-life equipments and/or interfaces as stimuli that are much more complex than those stimuli used in a

Psychophysiology as a Tool for HCI Research: Promises and Pitfalls

147

typical psychology experiment. So the data collected always have to be interpreted with the context in mind. Could the heart rate have picked up speed because the subject was not paying attention, or because the subject was puzzled by the menu system and had to think about it? Was the subject sweating more because the interface was hard to use or because there was an image of a huge spider shown on the screen? When designing the session, it would be the best to eliminate all factors that may result unwanted interference during data collection, but even after data has been collected, it would be the best to go back and think if there is any room for alternative explanation for certain physiological responses. And researchers should keep in mind that though research tools provided by psychophysiology may offer new insights, none of them is perfect by itself and it is best to be used in combination with other research methods. Through his online column, Jacob Nielsen, a well-known HCI guru and consultant, advices design agencies who are seeking ways to convince clients to pay for usability testing, "using sound methodology is the true sign of professionalism" and they have to "point out usability's astounding return on investment."[10] Including psychophysiology to the set of tools available may also be a good idea for design agencies for improving usability of the final product. Psychophysiology should be able to make good friends with both HCI researchers and HCI practitioners. It’s just the matter of choosing the right tool and applying it the right way to get the right answer to the right question.

References [1] Allanson, J., Wilson, G.M.: Physiological Computing. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI 2002), pp. 912–913 (2002) [2] Ekman, P., Davidson, R.J., Friesen, W.V.: The Duchenne Smile: Emotional Expression and Brain Physiology II. Journal of Personality and Social Psychology 58(2), 342–353 (1990) [3] Hewig, J., Trippe, R.H., Hecht, H., Straube, T., Miltner, W.H.R.: Gender Differences for Specific Body Regions When Looking at Men and Women. Journal of Nonverbal Behavior 32, 67–78 (2008) [4] Jacob, R.J.K., Karn, K.S.: Eye Tracking in Human–Computer Interaction and Usability Research: Ready to Deliver the Promises. In: Hyona, Radach, Deubel (eds.) The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, Oxford, England (2003) [5] Jones, C.M., Dlay, S.S.: The Face as an Interface: The New Paradigm for HCI. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), vol. 1, pp. 774–779 (1999) [6] Just, M.A., Carpenter, P.A.: Eye fixations and cognitive processes. Cognitive Psychology 8, 441–480 (1976) [7] Kim, S., Godbole, A., Huang, R., Panchadhar, R., Smari, W.: Toward an Integrated Human-centered Knowledge-based Collaborative Decision Making System. In: Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, LasVegas, NV, November 2004, pp. 394–401 (2004) [8] Lin, T., Hu, W., Omata, M., Imamiya, A.: Do physiological data relate to traditional usability indexes? In: Proceedings of the 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future (OZCHI), vol. 122, pp. 1–10 (2005)

148

B. Park

[9] Mansell, W., Clark, D.M., Ehlers, A.: Internal versus external attention in social anxiety: An investigation using a novel paradigm. Behaviour research and therapy 41(5), 555–572 (2003) [10] Nielsen, J.: Convincing Clients to Pay for Usability (May 19, 2003), http://www.useit.com/alertbox/20030519.html [11] Pecchinenda, A., Smith, C.A.: The affective significance of skin conductance activity during a difficult problem-solving task. Cognition and Emotion 10(5), 481–504 (1996) [12] Prendingera, H., Mori, J., Ishizuka, M.: Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game. International Journal of Human-Computer Studies 62, 231–245 (2005) [13] Selvaraj, N., Jaryal, A., Santhosh, J., Deepak, K.K., Anand, S.: Assessment of heart rate variability derived from finger-tip photoplethysmography as compared to electrocardiography. Journal of Medical Engineering & Technology 32(6), 479–484 (2008) [14] Spiegel, D.: Human Computer Interaction (October 22, 1998), http://xenia.media.mit.edu/~spiegel/papers/HCI.pdf [15] Stern, R., Ray, W.J., Quigley, K.S.: Psychophysiological Recording, 2nd edn. Oxford University Press, New York (2001) [16] Svebak, S.: The effect of task difficulty and threat of aversive electric shock upon tonic physiological changes. Biological Psychology 14(1-2), 113–128 (1982) [17] Ward, R.D., Marsden, P.H.: Physiological responses to different WEB page designs. International Journal of Human-Computer Studies 59, 199–212 (2003)

Assessing NeuroSky’s Usability to Detect Attention Levels in an Assessment Exercise Genaro Rebolledo-Mendez1,3, Ian Dunwell1, Erika A. Martínez-Mirón2, María Dolores Vargas-Cerdán3, Sara de Freitas1, Fotis Liarokapis4, and Alma R. García-Gaona3 1

Serious Games Institute, Coventry University, UK 2 CCADET, UNAM, Mexico 3 Facultad de Estadistica e Informatica, Universidad Veracruzana, Mexico 4 Interactive Worlds Applied Research Group, Coventry University, UK {GRebolledoMendez,IDunwell,Sfreitas, F.Liarokapis}@cad.coventry.ac.uk, [email protected], {dvargas,agarcia}@uv.mx

Abstract. This paper presents the results of a usability evaluation of the NeuroSky’s MindSet (MS). Until recently most Brain Computer Interfaces (BCI) have been designed for clinical and research purposes partly due to their size and complexity. However, a new generation of consumer-oriented BCI has appeared for the video game industry. The MS, a headset with a single electrode, is based on electro-encephalogram readings (EEG) capturing faint electrical signals generated by neural activity. The electrical signal across the electrode is measured to determine levels of attention (based on Alpha waveforms) and then translated into binary data. This paper presents the results of an evaluation to assess the usability of the MS by defining a model of attention to fuse attention signals with user-generated data in a Second Life assessment exercise. The results of this evaluation suggest that the MS provides accurate readings regarding attention, since there is a positive correlation between measured and selfreported attention levels. The results also suggest there are some usability and technical problems with its operation. Future research is presented consisting of the definition a standardized reading methodology and an algorithm to level out the natural fluctuation of users’ attention levels if they are to be used as inputs.

1 Introduction This paper presents a usability evaluation of NeuroSky’s MindSet (MS) device. An aspect of interest was to investigate whether MS readings can be combined with usergenerated data. The amalgamation of physiological and user-generated data would allow the programming of more sophisticated user models. An experimental setting was set up to analyze MS usability in an assessment exercise in Second Life. The assessment [10] is based on a multiple-choice questionnaire in the area of programming for Computer Science undergraduate students. The questionnaire is presented by an Artificial Intelligence-controlled avatar (AI-avatar) who is aware of the levels of attention of the J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 149–158, 2009. © Springer-Verlag Berlin Heidelberg 2009

150

G. Rebolledo-Mendez et al.

person interacting with it. The MS1 also provides a measurement of the user’s meditative state (derived from alpha wave activity). In this paper, however, only the levels of attention are used, given their role and importance in educational settings. The objective of this study is threefold: firstly, the MS general usability is examined. Secondly, an analysis of how well it is possible to fuse information generated as part of normal interactions with brain activity. Thirdly, an analysis of the MS adaptability to different ableusers is provided. The significance of this work lies in that it presents evidence of the usability of a commercially available BCI and its suitability to be incorporated into serious games. The paper is organized in five sections. Section two presents a literature review about Brain Computer Interfaces and their use for learning. Section three describes the Assessment exercise used as test bed and presents the materials, participants and methodology followed during the evaluation. Section four presents the results of the evaluation and, finally, section five provides the conclusions and future research.

2 Brain Computer Interfaces (BCI) Brain Computer Interface (BCI) technology represents a rapidly emerging field of research with applications ranging from prosthetics and control systems [6] through to medical diagnostics. This study only considers BCI technologies that use sensors that measure and interpret brain activity (commonly termed neural bio-recorders [14]) as a source of input. The longest established method of neural bio-recording, developed in 1927 by Berger [3], is the application of electrodes that measure the changes in field potential over time arising from synaptic currents. This forms the basis for EEG. In the last two decades, advances in medical imaging technology have presented a variety of alternative means for bio-recording, such as functional magnetic resonance imaging (fMRI), magnetoencephalography (MEG), and positron emission tomography (PET). A fundamental difference between bio-recording technologies used for diagnostic imaging, and those used for BCI applications, is a typical requirement for real or quasi-real time performance in order to translate user input into interactive responses. In 2003, a taxonomy by Mason and Birch [8] identified MEG, PET and fMRI as unsuitable for BCI applications, due to the equipment required to perform and analyze the scan in real-time, but more recent attempts to use fMRI as a BCI input device have demonstrated significant future potential in this area [12]. Bio-recording BCIs have become a topic of research interest both as a means for obtaining user input, and studying responses to stimuli. Several studies have already demonstrated the ability of an EEG-based BCI to control a simple pointing device similar to a mouse [9, 12] and advancing these systems to allow users more accurate and responsive control systems is a significant area for research. Of particular interest to this study is the use of BCI technologies in learning-related applications. The recent use of fMRI to decode mental [4] and cognitive [11] states illustrates a definite capability to measure affect through bio-recording, but the intrusiveness of the scanning equipment makes it difficult to utilize the information gained to provide feedback to a user performing typical real-world learning activities. In this study, the effectiveness of one of the first commercially available lightweight EEG devices, NeuroSky’s MS, is considered. Via the application of a single 1

The MB is a developer-only headset. NeuroSky’s newest headset has been designed to address comfort and fitting problems and is available to both developers and consumers.

Assessing NeuroSky’s Usability to Detect Attention Levels

151

electrode and signal-processing unit in a headband arrangement, the MS provides two 100-state outputs operating at 1Hz. These outputs are described by the developers as providing separate measures of ‘attention’ and ‘meditation’, and it is thus assumed these readings are inferred from processing beta and alpha wave activity respectively. Although the MS provides a much coarser picture of brain activity than multielectrode EEG or the other aforementioned technologies, the principle advantage of the MS is its unobtrusive nature, which minimises the aforementioned difficulties in conducting accurate user studies due to the stress or distraction induced by the scanning process. Research into EEG biofeedback as a tool to aid individuals with learning difficulties [5] represents an area for ongoing study, and the future widespread availability of devices similar to the MS to home users presents an interesting opportunity to utilize these technologies in broader applications.

3 An Assessment Exercise in Second Life An assessment exercise was developed to examine the MS. The exercise works in combination with a model of attention [10] built around dynamic variables generated by the learner’s brain (MS inputs) and the learner’s actions in a computer-based learning situation. The combination of physiological (attention) and data variables is not new [7, 1]. Our approach, however, fuses MS readings (providing a more accurate reading of the learner’s attention based on neural activity) with user-generated data. In our model, attention readings are combined with information such as the number of questions answered correctly (or incorrectly), or the time taken to answer each question, to model attention within the assessment exercise. The MS reads attention levels in an arbitrary scale ranging from 0 to 100. There is an initial delay of between 7 and 10 seconds before the first value reaches the computer and newer values of attention are calculated at a rate of 1Hz (one value per second, see Figure 1). A value of -3 indicates no signal is being read and values equal to or greater than 0 indicate increasing levels of attention with a maximum value of 100. Given the dynamic nature of the attention patterns and the potentially large data sets obtained, the model of attention underpinning the assessment exercise is associated to a particular learning episode lasting more than one second. The model of attention not only determines (detects) attention patterns but also provides (reacts) feedback to the learner [10]. The assessment exercise consists of presenting a Second Life2, AI-driven avatar able to pose questions, use a pre-defined set of reactions and have limited conversations with learners in Second Life. The AI-driven avatar was programmed using C# (C-sharp) in combination with the lib second life library3. Lib Second Life is a project aimed at understanding and extending Second Life’s client to allow the programming of features using the C# programming languages. This tool enables the manipulation of avatars’ behaviors so that they respond to other avatars’. To do so, the AI-driven avatar collects user-generated data during the interaction including MS inputs. The current implementation of the AI-driven avatar asks questions in a multiple-choice 2 3

http://secondlife.com/ http://www.libsecondlife.org/

152

G. Rebolledo-Mendez et al.

format, while dynamically collecting information (answers to questions, time taken to respond, and whether users fail to answer). The data generated by the MS is transmitted to the computer via a USB interface and organized via a C# class which communicates with the AI-driven avatar. In this way, the model of attention is updated dynamically and considers input from the MS as well as the learner’s performance behavior while underpinning the AI-driven avatar’s behavior. For the purposes of assessing MS’s usability, the assessment exercise consisted of ten questions in the area of Informatics, specifically for the area of Algorithms. This area was targeted since it has been noted first year students in the Informatics department often struggle with the conception and definition of algorithms, a fundamental part of programming. The assessment exercise asked Fig. 1. Attention readings as read by the NeuroSky nine theoretical questions and presented three possible answers. For example, the avatar would ask ‘How do you call a finite and ordered number of steps to solve a computational problem?’ while offering ‘a) Program, b) Algorithm, c) Programming language’ as possible answers. The assessment exercise also includes the resolution of one practical problem, answered by the learner by hand while still wearing the MS. 3.1 Materials To evaluate the MS’s reliability, two adaptations of the Attention Deficit and Hyperactivity disorder (ADHD) test and a usability questionnaire were defined. The attention tests consisted of seven items based on the DSV4-IV criteria [2]. The items chosen for the attention test were: 1. Difficulty to stay in one position, 2. Difficulty in sustaining attention, 3. Difficulty to keep quiet often interrupting others, 4. Difficulty to follow through on instructions, 5. Difficulty to organize tasks and activities, 6. Difficulty or avoidance of tasks that require sustained mental effort and 7. Difficulty to listen to what is being said by others. Each item was adapted to assess attention both in the class and at interaction time. To answer individual questions, participants were asked to choose the degree which they believed reflected their behavior in a Likert type scale with 5 options. For example, question 1 of the attention questionnaire asked the participant: ‘How often is it difficult for me to remain seated in one position whilst working with algorithms in class/during the interaction?’ with the answers 1) all the time, 2) most of the time, 3) some times, 4) occasionally and 5) never. Note that for both questionnaires the same seven questions were asked but were rephrased considering the class for the pre-test 4

Diagnostic and Statistical Manual of Mental Disorders.

Assessing NeuroSky’s Usability to Detect Attention Levels

153

or the interaction for the post-test. The usability questionnaire consisted of adapting three principles of usability into three questions (a) comfort of the device; (b) easiness to wear; and (c) degree of frustration. To answer the usability questionnaire participants were asked to select the degree to which they felt the MS faired during the interaction via a Likert type scale with 5 options. For example, question 1 of the usability questionnaire asked the student: ‘Was using Neurosky’ 1) Very uncomfortable, 2) Uncomfortable, 3) Neutral, 4) Comfortable, 5) Very comfortable. Note that to report the usability of the MS, other factors were also considered such as battery life, light indicators and data read/write times and intervals. 3.2 Participants and Methodology An evaluation (N=40) to assess the usability of the MS was conducted among firstyear undergraduate students in the Informatics Department at the University of Veracruz, Mexico. The population consisted of 28 males and 12 females, 38 undertaking the first year of their studies and 2 undertaking the third year. 26 students (65%) of the population were 18 years old, 12 students (30%) were 19 years old and 2 students (5%) were 20 years old. The participants interacted with the AI-avatar for an average of 9.48 minutes answering ten questions posed by the avatar within the assessment exercise (see previous section). During the experiment, the following procedure was followed: 1) students were asked to read the consent form, specifying the objectives of the study and prompted to either agree or disagree, 2) students were asked to solve an online pre-test consisting of the adaptation of the attention deficit and hyperactivity disorder (ADHD) questionnaire to assess their attention levels in class, 3) students were instructed on how to use the learning environment, and finally 4) the students were asked to answer an online post-test consisting of the usability questionnaire and the adaptation of the ADHD questionnaire to assess their attention levels during the interaction in the assessment exercise. Individual logs registering the students’ answers and attention levels as read by the MS were kept for analyses. All students agreed to participate in the experiment but in some cases (N=6) the data was discarded since the MS did not produce readings for these participants. See the results section for a description of these problems. Cases with missing data were not considered in the analysis.

4 Results The results of this evaluation are organized to consider the MS’s usability, how well the model fuses user-generated data and attention readings and the MS’s adaptability. 4.1 Usability and Appropriateness of MS for Assessment Exercises The main aspect of interest was MS’s usability considering the responses to three questions (see materials section). This questionnaire considered three aspects to assess the usability of new computer-based devices: Comfort, Ease of Use, and the Degree of Frustration. The answers to the questionnaire are organized around each aspect considered. There was one question associated to every usability aspect.

154

G. Rebolledo-Mendez et al.

Comfort The results showed that for 5% (N=2) the MS was uncomfortable, for 10% (N=4) somewhat uncomfortable, for 35% (N=14) neither comfortable nor uncomfortable, for 25% (N=10) somewhat comfortable and for 25% (N=10) comfortable. Ease of Use The results showed 15% (N=6) students found the MS difficult to wear, 12.5% (N=5) found it somewhat difficult to wear, 37.5% (N=15) thought it was neither easy nor difficult to wear, 12.5% (N=5) found it somewhat easy to wear and 22.5% (N=9) thought it was easy to wear. Degree of Frustration The answers showed 2.5% (N=1) found the experience frustrating, 2.5% (N=1) thought it was somewhat frustrating, 22.5% (N=9) found the experiment neither frustrating nor satisfactory, 25% (N=10) thought it was somewhat satisfactory and 47.5% (N=19) had a satisfactory experience using the MS. There were three aspects that only became apparent once the evaluation was over. The first aspect of interest was in relation to the pace and the way readings were collected. The attention model [1] considered readings in the space of time used by learners to formulate an answer for each question. The pace in which data was collected by the model was 10Hz which produced repeated measurements in some logs. This method of collecting data is inefficient as plotting attention fluctuations considering fixed, regular intervals might be difficult. People interested in programming the MS device should consider that, due to a hardware processing delay, the MS outputs operate at 1Hz, and need to program their algorithms accordingly. The second aspect of interest is in relation to difficulties wearing the device. When connection is lost, there is a delay of 7-10 seconds before a new reading is provided. Designers should consider this as a constant input might not be possible. The third aspect refers to MS’s suitability as an input device for interface control. Developers need to consider that attention levels (and associated patterns) vary considerably between users (see Figure 2), as expected. If developers employ higher levels of attention as triggers for interface or system changes, they should consider some users normally have higher levels of attention without being prompted to put more attention. This normal variability creates the need to research and develop an algorithm to level-out initial differences in attention levels and patterns. On a related topic, MS’s readings vary in a scale from 0 to 100 (see Figure 1): however, it is not yet clear what relationship exists between wave activity and processed output, whether the scale is linear, or whether the granularity of the 100-point scale is appropriate for all users. Finally, there were some usability problems that caused data loss, in particular: 1. In 3 cases the MS did not fit the participant’s head properly leading to adjustments by the participants leading to not constant and unreliable readings. Another problem was people with longer hair having problems wearing the device to allow sensors touch the skin behind the ears at all times. During the experiment, extra time was required to make sure people with longer hair placed the device adequately. 2. In other 3 cases the MS ran out of battery. The battery was checked before each participant interacted with the assessment exercise using NeuroSky software via its associated software. However, despite the precautions taken and after having checked the green light on one of the device’s side, battery life was very short. The

Assessing NeuroSky’s Usability to Detect Attention Levels

155

device does not alert the user when battery levels are low, so it was not clear when batteries needed to be replaced. This was a problem at the beginning of the experiment but later on batteries were replaced on daily basis. 4.2 Adaptability to Different Users One of the characteristics of the MS reader is that it can be worn by different users producing different outputs. This would allow for adaptation of the model [10] in the frame of the assessment exercise. It was expected MS outputs would vary for different users reflecting varying levels of attention. Furthermore, this adaptation would be fast and seamless without the need to train the device for a new user. To throw some light onto the issue of adaptability, it was speculated attention readings would be different for individuals. It was also hypothesized there would be a positive correlation between the readings and the self-assessment attention test (see materials section). To assess variability among participants, a test of normality was done to see the distribution of the participants’ average attention levels. Table 1 shows descriptive statistics of the readings for the population (N=34). The results of a test of normal distribution showed that the data is normally distributed (Shapiro-Wilk = .983, p = .852) suggesting there is not a tendency to replicate particular readings. Figure 2 illustrates the Q-Q plot for this sample suggesting a good distribution of average attention levels during the assessment exercise. Table 1. Descriptive statistics

N 34 34

Student's attention levels Student's self reported attention

Min 14.99 3.0

Max 88.00 5.0

Mean 53.40 4.27

Std. Dev. 16.69 .44548

Another test designed to see whether MS readings adapted to individual participants, was a correlation between the readings and the self-reported attention using the post-test questionnaire. A positive correlation was expected between these two variables. Table 1 shows the descriptive statistics for the two variables. To calculate self-reported attention levels, the mean of the answers to the 7 items of the attention posttest was calculated per participant; lesser values indicate lesser attention levels. The results of a PearObserved Value son’s correlation between the two variables indicated a significant, Fig. 2. Q-Q plot of students’ average attention positive correlation (Pearson’s = levels during the assessment exercise -.391, p = .022). .3

.2

.1

0.0

Dev from Normal

-.1

-.2

-.3

-.4

20

30

40

50

60

70

80

90

156

G. Rebolledo-Mendez et al.

4.3 Fusing User-Generated Information with MS Readings One way to analyze whether the data was fused correctly was to check the logs for missing or incorrect data. The results of this analysis showed that there were six participants (15%, original sample N=40) for which the MS did not produce accurate readings. An analysis of the logs for the remaining participants (N=34) showed the device produced readings throughout the length of the experiment (average time = 9.48 minutes) without having an erroneous datum (attention = -3). The causes for the lack of readings in 6 cases were due to usability problems (see following section). Another way of throwing some light on how well the MS readings and usergenerated data were fused consisted of analyzing the logs to see whether there was a variation on the model’s reactions for the sample. Since the reactions given by the AI avatar could be of six types [1], the frequency was calculated for each reaction type for the entire population with correct NeuroSky readings (N=34), see Table 2. Table 2. Frequencies associated to the model’s reaction types for the population (N=34)

Reaction Type Frequency

6 128

5 172

4 0

3 77

2 13

1 0

It was expected the frequencies for reaction types 4 and 1 would be 0 given the averages of the four binary inputs. Reaction Types 5 was the most common type followed by Reaction Types 6, 3 and 2. Given the 8 possible results of averaging out the four binary inputs [10], it was expected Reaction Type 3 would be the most frequent. However, this was not the case suggesting the model did vary and the reactions type provided were in accordance to the variations in attention, time, and whether answers were correct. Finally, the responses to two questions in the post-test questionnaire gave an indication of students’ subjective perceptions about how well the reaction types were adequate to their attention needs. The first question asked: ‘how frequently the reactions helped you realize there was something wrong with the way you were answering the questions?’ The answers showed 25% (N=10) of students felt the avatar helped them all the time, 20% (N=8) said most of the time, 35% (N=14) mentioned some times, 12.5% (N=5) said rarely and 7.5% (N=3) stated never. The second question asked ‘how appropriate they thought the combined use of MS and avatars was appropriate for computer-based educational purposes?’ Students’ answered with 65% (N=26) saying it was appropriate, 12.5% (N=5) saying it was appropriate most of the time, 15% (N=6) saying it was neither appropriate nor inappropriate, 2.5% (N=1) saying it was somewhat inappropriate and 5% (N=2) saying it was inappropriate.

5 Conclusions and Future Work The reliability of MS readings to assess attention levels and to amalgamate with usergenerated data was evaluated in an assessment exercise in Second Life, N=34. The results showed there is variability in the readings and they correlate with self-reported attention levels suggesting the MS adapts to different users providing accurate readings of attention. The results of analyzing the device’s usability suggest some users

Assessing NeuroSky’s Usability to Detect Attention Levels

157

have problems with wearing the device due to head sizes or hair interference and that the device’s signals to indicate flat batteries are poor. By analyzing individual logs it was possible to determine that, when the device fits properly, the MS provides valid and constant data as expected. Log analyses also helped establish the frequency different reactions types were provided in the exercise in the light of attention variability. The frequencies suggested the model did not lean to the most expected reaction (Type 3) but that it tended to be distributed amongst Reaction Types 5 and 6, providing an indication that user-generated data was fusing adequately with attention readings. When asked about their experience, 35% of the population said the avatar helped them realize there was something wrong with how s/he was answering the questions and 65% indicated using a MS in combination with avatars was appropriate in computer-based educational settings. When asked about comfort, 35% thought the device was neither comfortable not uncomfortable, 37.5% thought it was neither easy nor difficult to wear and 47.5% said they had a satisfactory experience with the device. There were other results that were apparent only after the evaluation. In particular, it was found that: 1) sampling rates need to be considered to organize data in fixed, regular intervals to determine attention. 2) Developers need to be aware there is a delay when readings are lost due to usability issues. 3) Variability imposes new challenges for developers who wish to use levels of attention as input to control or alter interfaces. Work for the future includes the combination of MS readings with other technologies such as using learner’s gaze, body posture and facial expressions to read visual attention. Future work will be carried out to find out the degree of attention variability to program an algorithm capable of leveling-out different patterns of attention. In addition, future work will explore how attention data can be used to develop learner models that help understanding attention and engagement for informing gamebased learning design and user modeling.

Acknowledgements The research team thank the NeuroSky Corporation for providing the device for testing purposes. We also thank the students, lecturers and support staff at the Faculty of Informatics, Universidad Veracruzana, Mexico.

References 1. Amershi, S., Conati, C., McLaren, H.: Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games. In: Workshop in Motivational and Affective Issues in ITS, 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan (2006) 2. Association, A.P.: Diagnostic and Statistical Manual of Mental Disorders. American Psychiatric Press (1994) 3. Berger, H.: On the electroencephalogram of man. In: Gloor, P. (ed.) The fourteen original reports on the human electroencephalogram, Amsterdam (1969) 4. Haynes, J.D., Rees, G.: Decoding mental states from brain activity in humans. Nature Neuroscience 7(7) (2006)

158

G. Rebolledo-Mendez et al.

5. Linden, M., Habib, T., Radojevic, V.: A controlled study of the effects of EEG biofeedback on cognition and behavior of children with attention deficit disorder and learning disabilities. Applied Psychophysiology and Biofeedback 21(1) (1996) 6. Loudin, J.D., et al.: Optoelectronic retinal prosthesis: system design and performance. Journal of Neural Engineering 4, 72–84 (2007) 7. Manske, M., Conati, C.: Modelling Learning in an Educational Game. In: 12th Conference on Artificial Intelligence in Education, IOS Press, Amsterdam (2005) 8. Mason, S.G., Birch, G.E.: A general framework for brain-computer interface design. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11, 70–85 (2003) 9. Poli, R., Cinel, C., Citi, L., Sepulveda, F.: Evolutionary brain computer interfaces. In: Giacobini, M. (ed.) EvoWorkshops 2007. LNCS, vol. 4448, pp. 301–310. Springer, Heidelberg (2007) 10. Rebolledo-Mendez, G., De Freitas, S.: Attention modeling using inputs from a Brain Computer Interface and user-generated data in Second Life. In: The Tenth International Conference on Multimodal Interfaces (ICMI 2008), Crete, Greece (2008) 11. Sona, D., Veeramachaneni, S., Olivetti, E., Avesani, P.: Inferring cognition from fMRI brain images. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D.P. (eds.) ICANN 2007. LNCS, vol. 4669, pp. 869–878. Springer, Heidelberg (2007) 12. Sitaram, R., et al.: fMRI Brain-Computer Interfaces. IEEE Signal Processing Magazine 25(1), 95–106 (2008) 13. Trejo, L.J., Rosipal, R., Matthews, B.: Brain-computer interfaces for 1-D and 2-D cursor control: designs using volitional control of the EEG spectrum or steady-state visual evoked potentials. IEEE Transactions on Neural Systems and Rehabilitation Engineering 14(2), 225–229 (2006) 14. Vaughan, T., et al.: Brain-computer interface technology: a review of the second international meeting. IEEE Transactions on Neural Systems and Rehabilitation Engineering 11(2), 94–109 (2003)

Effect of Body Movement on Music Expressivity in Jazz Performances Mamiko Sakata1, Sayaka Wakamiya2, Naoki Odaka2, and Kozaburo Hachimura3 1

Faculty of Culture and Information Science, Doshisha University, 1-3 Tatara Miyakodani, Kyotanabe City, 610-0394 Japan [email protected] 2 Graduate School of Human Development and Environment, Kobe University [email protected], [email protected] 3 College of Information Science & Engineering, Ritsumeikan University [email protected]

Abstract. In this study, we tried to examine empirically how body motion contributes to music expressivity, both in terms of intensity and manners, during impromptu jazz performances. Psychological rating experiments showed that music expressivity in jazz performances are assessed in two aspects, namely power and aesthetic quality. In the assessment of musical performances, the music itself basically contributed to how observers evaluated its expressivity. However, it was also shown that body motion had a greater influence on assessing the quality of music in terms of “hard or soft” and “light or heavy.”As a result of the three-dimensional motion analysis using motion capture, we learned that the characteristics of the player’s body motions changed with the playing style and the playing dynamics. The player, therefore, is making music not only by producing the “sound,” but by also showing “body motions” for creating that sound. Keywords: Jazz Performances, Music Expressivity, Body Movement, Motion Capture.

1 Introduction When performing classical music, the musicians share a general, predetermined understanding of the expression of their music and timing and play under the conductor’s direction. In impromptu jazz performances, however, the musicians begin to play their music only after agreeing on minimal requirements, such as the chord progression and structure of the musical composition they are about to perform. This means that each player must understand, on a real-time basis, the music expressivity of the other players and respond to or go along with them. Many studies on music expressivity have been conducted concerning its technical aspects, such as performance methods and their relationship with the impressions aroused by music, but hardly any empirical research has been conducted up to this point concerning body movements as representative of music expressivity; in other J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 159–168, 2009. © Springer-Verlag Berlin Heidelberg 2009

160

M. Sakata et al.

words, the visual expression of body movements. Recently, some studies (those by Davidson [1], Okada [2], and Maruyama [3] and others, or example) have made some reference to the role played by the body in musical performances, but such works are sporadic, and the discussion has really only just begun. This study aims to illuminate “the visual roles of body movements during impromptu performances of jazz music” and empirically show the modes and intensity of body movements that contribute to music expressivity. For this purpose, we employ “Kansei” information processing techniques, motion capturing, feature extraction from motion data and some statistical analyses. “Kansei” is a Japanese word whose meaning is close to “feeling” or “sensibility” in English. Kansei information processing is a method of extracting features related to Kansei conveyed by the media we receive. Conversely, it is also a method of adding or generating some Kansei factors to media produced by computers [4]. We employ motion capturing techniques for obtaining images of human body motions. This technique is now used commonly in movie and CG animation production. Several systems are commercially available nowadays. This study uses motion capturing to analyze jazz performances and quantitatively analyze the roles played by body movements.

2 Study Subjects For study subjects, we prepared materials with the help of professional jazz musicians in order to study the role of body movements in music expressivity. We asked a 10year veteran male alto-saxophone player (24 years old) to play the jazz standard “Summertime.” He played the front theme1, an ad-lib solo in the middle and the back theme for two choruses (with each chorus containing 16 bars). He was asked to play them in three different modes: “ordinary,” “expressionless” and “over expressive.” In this study, these three different modes of expression are defined as “expression dynamics.” In order to retain the characteristic feature of freestyle jazz performance, which is to perceive each other’s music expressivity in real-time, respond to them or follow them, we asked other players to join in the performance of our subject. The back band consisted of a drummer, a wood base player and a guitarist. The drummer was asked to keep the BPM=120 tempo, and all the other players were asked to follow the alto-saxophonist’s performance.

3 Motion Capture System We used an optical motion capture system (Motion Analysis Corporation, EvaRT with Eagle cameras) to measure body movements during a jazz performance. Fig. 1 shows a scene from the motion capturing session in our studio. Reflective markers were attached to the joints on the player’s body, and several high-precision, highspeed video cameras were used to track the motion. In our case, 33+2(on the instrument) markers were put on the player's body (see Fig. 2), and the movement was 1

The pre-composed part of a jazz number is called “theme”. In common jazz performances, musicians play the theme first, then the solo, and then the theme again.

Effect of Body Movement on Music Expressivity in Jazz Performances

161

measured with 10 cameras. The acquired data can be observed as a time-series using the three-dimensional coordinate values (x, y, z) of each marker in each frame (frame rate is 120 fps).

Fig. 1. Motion Capture

Fig. 2. Positions of markers

4 Psychological Rating Experiments In order to determine what kind of impressions are perceived from jazz performances, the object of our study, and how the different modalities, namely the “sound” and the “body,” contribute to musical expressivity, we conducted psychological rating experiments. Thirty-eight observers (20 men and 18 women) participated in this experiment. The mean and standard deviations of age among the 38 observers were 20.5 and 1.43, respectively. All observers had some training in jazz. 4.1 Stimuli for Experiments The motion measured by the motion capture system described above was filmed by a digital camera (SONY) and then edited to produce experimental stimuli. The stimuli were obtained by editing the performances of the front theme and ad-lib solo played with three different expression dynamics, and then further edited using the three modalities of “sound only,” “visual images only” and “sound and visuals.” Table 1 shows the order in which the stimuli were presented (the modalities, dynamics and styles) and the length of each stimulus. 4.2 Procedure We briefed the observers on the experiment, asked them to answer questions concerning their personal attributes and then presented the stimuli, one type at a time, in the order of “visual images only” -> “sound only”-> “sound and visuals.” We provided an interval after showing one type of stimulus twice. In the first showing of the stimuli, the subjects were asked to closely and carefully observe the stimuli. In the second showing, they were asked to fill in the Answer Sheet using the Assessment Words on

162

M. Sakata et al.

a scale from 1 to 7 for each word in the adjective pair, which are shown in Table 22. The videotaped recording was temporarily stopped during the intervals between showing the different types of stimuli. The new stimulus was presented after making sure all subjects finished filling in their Answer Sheet. Table 1. Order of stimuli Order of display 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Modality

Expression dynamics

Style

Duration

visual images only visual images only visual images only visual images only visual images only visual images only sound only sound only sound only sound only sound only sound only sound and visuals sound and visuals sound and visuals sound and visuals sound and visuals sound and visuals

ordinary expressionless over expressive over expressive expressionless ordinary expressionless over expressive ordinary over expressive expressionless ordinary over expressive expressionless over expressive expressionless over expressive ordinary

solo theme theme solo solo theme solo theme theme solo theme solo theme solo solo theme theme solo

64 sec 66 sec 64 sec 62 sec 63 sec 65 sec 63 sec 64 sec 65 sec 62 sec 66 sec 64 sec 65 sec 63 sec 62 sec 66 sec 64 sec 64 sec

Table 2. Assessment Words 1 2 3 4 5 6 7 loose --+--+--+--+--+--+--+-soft --+--+--+--+--+--+--+-powerful --+--+--+--+--+--+--+-clear-cut --+--+--+--+--+--+--+-impressive --+--+--+--+--+--+--+-have presence-+--+--+--+--+--+--+-neat --+--+--+--+--+--+--+-plain --+--+--+--+--+--+--+-happy --+--+--+--+--+--+--+-light --+--+--+--+--+--+--+-unique --+--+--+--+--+--+--+-fantasy-like--+--+--+--+--+--+--+-rich --+--+--+--+--+--+--+-beautiful --+--+--+--+--+--+--+-fast --+--+--+--+--+--+--+-warm --+--+--+--+--+--+--+-subdued --+--+--+--+--+--+--+-inarticulate--+--+--+--+--+--+--+-favorable --+--+--+--+--+--+--+-good --+--+--+--+--+--+--+--

2

tight hard weak unclear unimpressive no presence messy passionate sad heavy ordinary realistic poor ugly slow cold bright articulate unfavorable bad

In defining the Assessment Words used in our experiment, we referred to Iwamiya [5] and added our own 10 pairs of adjectives to create a group of Assessment Words and started our preliminary experiments using these words. After running a factor analysis, we deleted the terms that were not affecting any of the factors, deleted one of the highly-correlated pairs and finally selected 20 pairs of adjectives.

Effect of Body Movement on Music Expressivity in Jazz Performances

163

4.3 Results of Kansei Assessment Experiment The results of the assessments of each stimulus were converted into scores from 1 to 7 using the SD method. We also obtained the average for the adjectives. Extraction of KANSEI information from the stimuli. After conducting a principal component analysis based on the Kansei Assessment Scores obtained, we extracted two principal components with a characteristic value greater than 1. (The cumulative contribution rate was 0.879 up to the second principal component.) Table 3 shows the values of factor loading of each word pair to the two principal components. The shaded areas in the Table indicate the significant image word pair ratings to each principal component with a magnitude larger than 0.8. Table 3. Results of PCA for the rating experiment Assessment Words

PC1

PC2

Table 4. Result of multiple regression analysis Assessment Words

Standardized Coefficients Sound

Adjusted R2

Visuals

loose-tight

.882

.122

soft-hard

.650

.025

powerful-weak

.959

-.141

clear-unclear

.948

.252

powerful-weak

0.926

0.055

0.876

impressive-unimpressive

.964

.108

clear-unclear

0.942

0.081

0.932 0.928

loose-tight

0.851

0.204

0.931

soft-hard

0.593

0.601

0.897

.974

.158

impressive-unimpressive

0.999

-0.031

-.649

.737

0.931

0.077

plain-passionate

.932

.102

have presence-no presence

happy-sad

.749

-.524

neat-messy

0.773

0.222

0.914

light-heavy

.673

-.385

plain-passionate

0.841

0.193

0.959

happy-sad

0.617

0.433

0.847

light-heavy

0.587

0.656

0.959

unique-ordinary

-

-

-

fantasy-like-realistic

-

-

-

rich-poor

1.028

-0.047

0.991

beautiful-ugly

0.943

0.051

0.945

fast-slow

0.670

0.348

0.861

-

-

-

0.657

0.405

0.910 0.969

have presence-no presence neat-messy

unique-ordinary

.974

-.076

fantasy-like-realistic

.859

.036

rich-poor

.789

.589

beautiful-ugly

.308

.942

fast-slow

.743

-.653

warm-cold

.618

.640

subdued-bright

-.701

.706

inarticulate-articulate

-.464

.853

favorable-unfavorable

.490

.846

good-bad

.468

.865

Eigenvalue

11.708

5.877

Variance (%)

58.539 87.925

warm-cold subdued-bright

0.926

inarticulate-articulate

0.687

0.341

favorable-unfavorable

0.915

0.102

0.898

good-bad

0.884

0.145

0.988

From Table 3, one can interpret the PC1 to be the variable concerned with the “power” of a musical performance, and PC2 to be the variable concerned with “aesthetic quality.” We plotted the principal components of PC1 and PC2 on the x- and y-axes and also plotted 18 types of stimuli on a graph (see Fig. 3). On this graph, the presence or power of a performance increases as you move toward the right, while the aesthetic quality increases as you move upward. From the PCA results, it became clear that musical expressivity in jazz performance is perceived from two aspects, namely the “power” and “aesthetic quality.”

164

M. Sakata et al.

Fig. 3. Plot of PCA score for each motion

The Body and the Music in Music Expressivity. To examine which of the two factors, namely “body motions” and “sound,” contributes more to music expressivity, we made a multiple regression analysis for each of the adjectives listed in Table 2, using Kansei Assessment Scores obtained by showing “visual images only” (in other words, body motions only) and “sound only” (in other words, music only) as independent variables. The Kansei Assessment Scores obtained by showing the “sound and visuals” were treated as dependent variables. As a result, we obtained a multiple regression equation (p<0.05) with a high contribution rate for many of the adjectives, as shown in Table 4. Based on the information in Table 4, we see that music contributes more than the body motion (visual images) in the Kansei assessment, as expressed by the words “loose-tight,” “powerful-weak,” “clear-unclear,” “impressive-unimpressive,” “have presence-no presence,” “neat-messy,” “plain-passionate,” “happy-sad,” “rich-poor,” “beautiful-ugly,” “fast-slow,” “subdued-bright,” “inarticulate-articulate,” “favorableunfavorable” and “good-bad.” On the other hand, the body motion (visual images) contributes more than the music in the Kansei assessment, as expressed by words like “soft-hard” and “light-heavy.”

5 Feature Values for Body Motion In the current study, the angles of each body part (back, sides, and knee), velocity (finger tips, elbow, sacral, head, and toe) and the distance moved on the floor (heel movement per unit time) were adopted as “feature values for body motion.” 5.1 Extracting Physical Parameters Angle. This parameter shows how the various body parts change in a time-series manner during musical performances. In our study, we measured the angles of the back, the body’s side and the knee. In Fig. 2, the angle created by marker numbers 5, 18 and 31 shows the angle of the back. The angle created by marker numbers 7, 4 and

Effect of Body Movement on Music Expressivity in Jazz Performances

165

18 is the angle of the side of the body. The angle created by marker numbers 19, 21 and 31 is the angle of the knee. For example, in the case of the back, we set the origin at marker no.18 (x2, y2, z2) and measured the angleθ created between marker no. 5 (x1, y1, z1) and marker no. 31 (x3, y3, z3). Then, using Equation (1) below, we calculated cosθ and returned the obtained cosine radian to a radian using the Arc Cosine. Then we used the Degrees function to convert the angles in degrees into numerical values of the angle. (1) Velocity. This parameter shows the time-series change in the movement of the body parts during a musical performance. In the current study, we measured the speeds of the fingertips of the right hand (no.11), elbow (no.7), sacral (no.18), head (no.2) and the toe (no.27). For each marker, we obtained the Euclidian distance in the frames from the data expressed by the x, y and z coordinates. We then multiplied the Euclidian distance with the frame rate to obtain the time-series data of the velocity. For example, when the x, y and z coordinates of the marker in Frame i are expressed as xi, yi and zi, we can obtain the distance d from Equation (2). (2) Then, by multiplying the d with the motion data frame rate, 120[Frame/sec], we can obtain the velocity |v|. |v| = d * 120

(3)

For the elbow, we used the relative coordinates by using the shoulders as the origin. The sacral were set as the origin to obtain the relative coordinates for the head and toe. This means that the velocity of the elbow, head and toe is expressed by a relative velocity based on the shoulder and the sacral. We obtained the velocity of the fingertips of the right hand and the sacral by using absolute coordinates using an origin determined in the capture area. Floor Travel Distance. This parameter shows how much the player moved on the floor during the performance. We obtained the distance traveled by the left heel (no.32) frame by frame. 5.2 Feature Values for Body Motion By conducting a principal component analysis after obtaining each parameter (raw data) described in Section 5.1 above, we extracted three components with characteristic values greater than 1. (The cumulative contribution rate up to the third principal component was 0.782.) From the values of factor loading shown by the Table 5, we can interpret that PC1 is the component showing the velocity of the upper part of the body, PC2 is the component showing flow travel distance, and PC3 is the component showing the bending of the body.

166

M. Sakata et al. Table 5. Results of PCA for the motion capture data Physical parameters Angles of the back

PC1

PC2

PC3

-.088

.181

.240

.285

.601

Angles of the knee

-.396

.628

-.107

Speed of the hand

.925

.251

-.053

Speed of the elbow

.922

.278

-.093

Speed of the head

.922

.258

-.155

Speed of the sacral

.913

.065

.000

Speed of the toe

.883

-.312

.091

Floor travel distance

.422

-.744

.190

4.572

1.371

1.094

Angles of the body’s side

Eigenvalue Variance (%)

.801

50.798 66.029 78.186

Fig. 4. Plot of PCA score for each motion by motion capture data

In the left graph of Fig. 4, the centers of gravity for PC1 and PC2 are plotted on the x and y axes, and six types of stimuli are plotted. In this figure, the more you move to the right, the greater velocity at which the top part of the body moves. The lower you go on the figure, the more distance covered in floor travel. Looking at each stimulus, the player showed greater floor travel when playing solo than when playing the theme. In terms of expression dynamics, the velocity of the top part of the body increased in this order: “ordinary” -> “expressionless” -> “over expressive.” Likewise, we plotted the centers of gravity for the principal components of PC1 and PC3 on the x and y axes and plotted each stimulus in a graph, which is shown in the right graph of Fig. 4. In this graph, the more you go to the right, the greater velocity of the top part of the body, and the lower you go, the more bending there is of the body. This graph shows that playing solo, rather than playing the theme, resulted in greater body bending. And when playing either solo or the theme, the player’s body bending was greatest during the “ordinary” mode of playing.

Effect of Body Movement on Music Expressivity in Jazz Performances

167

5.3 Relationship between Kansei Assessment and Feature Values for Body Motion We calculated the average and standard deviations for each stimulus for the nine parameters obtained in Section 5.1. In order to examine the relationship between the Kansei Assessment and the Feature Values for Body Motion, we calculated the coefficient of correlation between the principal component score of each performance obtained in Chapter 4 and the characteristic value of body motion of each performance(see table 6). The shaded area shows the combinations that showed a significant correlation of 5%. In the “Power Component,” we found a significant correlation between all Power Components and body motion parameters, except for the parameters “average angle of body’s side,” “average foot travel distance” and “standard deviation of foot travel distance.” For the Aesthetics Components, on the other hand, we could not find any correlation with any of the body motion parameters. Table 6. Correlation matrix Physical parameters Angles of the back Angles of the body’s side Angles of the knee Speed of the hand Speed of the elbow Speed of the head Speed of the sacral Speed of the toe Floor travel distance

Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD

PC1 Power -0.889* 0.880* 0.377 0.879* -0.949* 0.880* 0.894* 0.927* 0.908* 0.954* 0.931* 0.968* 0.908* 0.944* 0.881* 0.859* 0.710 0.678

PC2 Aesthetics 0.130 0.163 0.040 0.254 0.080 0.211 0.244 0.242 0.241 0.226 0.209 0.194 0.247 0.212 0.256 0.270 0.269 0.198

*p<0.05

This means that in musical performance, body motion contributes a large measure to the Power, but not to the Aesthetics.

6 Discussion and Conclusion In this study, we tried to examine empirically how body motion contributes to music expressivity, both in terms of intensity and manners, during impromptu jazz performances. Psychological rating experiments showed that music expressivity in jazz performances are assessed in two aspects, namely power and aesthetic quality. In the Kansei assessment of musical performances, the music itself basically contributed to how observers evaluated its expressivity. However, it was also shown that body motion

168

M. Sakata et al.

had a greater influence on assessing the quality of music in terms of “hard or soft” and “light or heavy.” As a result of the three-dimensional motion analysis using motion capture, we learned that the characteristics of the player’s body motions changed with the playing mode and the playing dynamics. The player, therefore, is making music not only by producing the “sound,” but by also showing “body motions” for creating that sound. It was found that body motions had a great role in creating “power,” but were not much related to the “aesthetics quality.” Naturally, the Kansei emanated from the sound itself is central to music expressivity. However, we have shown empirically that the body motions people make when making music also contribute greatly in music expressivity. This study offers a basic examination of the role of body motions in musical performances; however, many challenges and problems still remain to be explored. Acknowledgments. This work was supported in part by the Grant-in-Aid for Scientific Research, for Young Scientists (B) No. 19700493 of the Ministry of Education, Culture, Sports, Science and Technology, Japan. The authors would like to express their sincere gratitude to Mr. Junya Kondo for his cooperation with our research. Thanks are also due to Mr. Takahiro Yorino, Ms. Hitomi Toyoka and Mr. Naoyuki Okamoto for their kind help in motion capturing experiment.

References 1. Davidson, J.W.: Visual Perception of performance manner in the movements of Solo Musicians. Psychology of Music 21, 103–113 (1993) 2. Okada, A.: The body playing the piano. Shunjusha Publishing Company (2003) (in Japanese) 3. Maruyama, S.: The Embodied Sense of Music: Case Studies on the Rhetorical Function of Bodily Gestures by Highly Practiced Musicians. Cognitive studies: bulletin of the Japanese Cognitive Science Society 14(4), 471–493 (2007) (in Japanese) 4. Sakata, M., Hachimura, K.: KANSEI Information Processing of Human Body Movement. In: Smith, M.J., Salvendy, G. (eds.) HCII 2007. LNCS, vol. 4557, pp. 930–939. Springer, Heidelberg (2007) 5. Iwamiya, S.: Multimodal Communication of Music and Image. Kyushu University Press (2000) (in Japanese) 6. Iwamiya, S.: Design of Sounds. Kyushu University Press (2007) (in Japanese) 7. Nagashima, Y.: Drawing-in Effect on Perception of Beats in Multimedia. The Journal of the Society for Art and Science 3(1), 108–148 (2004)

A Method to Monitor Operator Overloading Dvijesh Shastri, Ioannis Pavlidis, and Avinash Wesley Computational Physiology Lab, Department of Computer Science, University of Houston, Houston, TX, 77204 {dshastri,ipavlidis,awesley}@uh.edu

Abstract. This paper describes research that aims to quantify stress levels of operators who perform multiple tasks. The proposed method is based on the thermal signature of the face. It measures physiological function from a standoff distance and therefore, it can unobtrusively monitor a machine operator. The method was tested on 11 participants. The results show that multi-tasking elevates metabolism in the supraorbital area, which is an indirect indication of increased mental load. This local metabolic change alters heat dissipation and thus, it can be measured through thermal imaging. The methodology could serve as a benchmarking tool in scenarios where an operator’s divided attention may cause harmful outcomes. A classic example is the case of a vehicle driver who talks on the cell phone. This stress measurement method when combined with user performance metrics can delineate optimal operational envelopes. Keywords: Human-Machine Interaction, divided attention, stress, thermal imaging.

1 Introduction In many occasions, machine operators simultaneously perform more than one task. When a combination of overlapping events demands critical decisions and rapid actions, they raise the operator’s alertness. If this persists, it develops to stress that overloads the operator. Stress due to operator’s divided attention may lead to degradation of his/her performance. This work aimed to analyze stress-induced change in facial physiology during divided attention situations. The supraorbital thermal signature emerged as the physiological variable of interest, which can be measured through thermal imaging. In fact, thermal imaging based stress monitoring is an increasingly popular approach [1] [2] [3]. In contrast to probe based stress monitoring methods, it is totally unobtrusive [7] [8]. Although concurrent performance of multiple tasks is part of human life, insufficient research has been done to understand its effect on human emotional states and performance. The purpose of this study is to develop an effective tool to gauge stress load of operators engaging in multi-tasking. The experimental design focused on cellphone communication during driving simulation [9][10][11] and was tested on 11 participants. However, other task pairs could have been chosen, such as reading in the presence of background distractions. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 169–175, 2009. © Springer-Verlag Berlin Heidelberg 2009

170

D. Shastri, I. Pavlidis, and A. Wesley

2 Methodology During dual tasking there is considerable temperature increase in the supraorbital region of a participant. This locally elevated temperature is the result of increased metabolic activity due to activation of the forehead muscle group. The phenomenon is consistent with findings in prior experiments involving Stroop color conflict testing [1]. In that previous work, the stress signal was extracted from the evolution of the mean thermal footprint of the entire supraorbital region (see Fig. 1. a). This approach, however, introduced noise in the extracted signal, partly due to the wide probing area and partly due to sub-optimal tracking performance [4]. In the present work, the tracking region was differentiated from the measurement region. An even bigger area that included sharp contrasts (e.g., skin versus hair) was selected for tracking. This improved tracking performance. However, only a small subset within the tracking region was selected for the thermal measurement. This subset was confined in the area were metabolic changes are more dramatic, to reduce the effect of probing noise (Fig. 1. b).

(a)

(b)

Fig. 1. (a) Tracking and measurements regions coincide in legacy method. (b) Measurement region (in pink) is a subset of tracking region in the current method.

For every participant in the experiment, tracking and measurement regions of interest were selected as described above. The mean temperature of the measurement region of interest was computed for every frame in the thermal clip. Thus, a 1D supraorbital signal was produced from the 2D thermal data. Any residual noise in the supraorbital signal was suppressed by a noise cleaning algorithm based on Fast Fourier Transform (FFT) [5]. The supraorbital signal was split into segments corresponding to the phases of the experiment (resting, initial single task, dual task, latter single task, and cool-off). Each segment was approximated with a linear fit (Fig. 2), which described the local metabolic rate at the time.

3 Experimental Design A high quality Thermal Imaging (TI) system was used for data collection. The centerpiece of the TI system was a ThermoVision SC6000 Mid-Wave Infrared (MWIR) camera (FLIR Systems) (sensitivity = 0.025oC) [12]. The experimental protocol included thermal imaging of the participant’s face while he was resting, engaging in a

A Method to Monitor Operator Overloading

171

driving simulation game (single task), engaging in a driving simulation game and talking on the cell phone (dual task), and relaxing. The dataset featured participants of both genders, different races, and with varying physical characteristics. The participants were placed 6 feet away from the thermal camera (Fig. 3). We used a XBOX-360 game console and the Test Drive: Unlimited game to simulate driving. The participants were asked to follow all traffic signs, drive normally, and not race during the experiment. They were given an opportunity to testdrive before the experiment began to acquaint themselves with the driving simulation. In the first formal phase of the experiment, the participants were asked to rest for 5 min while being imaged. This helped to isolate effects of other stress factors that participants may have carried inside them. This was the baseline phase.

Fig. 2. Supraorbital raw thermal signal (marked in blue color), noise cleaned thermal signal (marked in pink color) and linear fitting (marked in yellow color). Slope values for the linear segments are shown in blue colored text.

Next, the participants were asked to play the driving simulation game. This phase of the experiment also lasted about 5 min. After around 1 min of driving simulation (initial single task phase), the participant received a cell phone call that played a set of prerecorded questions in the following order: Instruction: Please do not hang up until you are told so. Q1: Are the lights ON in the room, yes or no? Q2: Are you a male or female? Q3: Who won the American civil war, the north or the south? Q4: What is 11 + 21? Q5: How many letters ‘e’ are in the word experiment? Q6: I am the son of a mom whose mother in law's son hit. How am I related to the other son? Q7: My grandma's son hit his son. How are the sons related? Q8: A man is injured in 1958 and died in 1956. How is that possible? Q9: What is 27 + 14? Instruction: You may now hang up the phone and pay attention to the game.

172

D. Shastri, I. Pavlidis, and A. Wesley

The question set was a combination of basic, logical, simple math, and ambiguous questions. The order of the questions was designed to build-up pressure on the participants. Additional pressure was achieved by repeating one more time every question that was incorrectly answered. The participants were supposed to drive while talking on the cell phone (dual task phase). At the end of the phone conversation, participants put the phone down and continued driving till the end of the experiment (the latter single task phase). Finally, the participants relaxed for 5 minutes. The purpose of this so-called cool-off segment was to monitor physiological changes after the simulated driving experiment.

Fig. 3. Experimental setup: Participant, imaging equipment, xBOX-360

4 Experimental Results The slopes of the linearly fitted segments, computed according to the method described in Section 2, were used as stress indicators. Fig.4 shows the mean slope values of the various segments for the entire data set (statistically constructed mean participant). The graph clearly indicates that the temperature increase during dual tasking is the highest among all segments (sole exception was participant S-6). Since the temperature increase is correlated to metabolic rate, the results indicate elevated metabolism in the supraorbital region during dual tasking. This is presumably due to strong muscle activation, associated with frowning, a facial expression autonomically associated with mental engagement. Stress during the latter single task phase was stronger than the initial single task phase (Fig. 4). Apparently, this was due to residual effect from the dual task that preceded the latter single task phase. Most of the participants admitted during debriefing that they were thinking about their dual task performance while performing the latter single task.

A Method to Monitor Operator Overloading

173

Fig. 4. Mean slope value of the experimental segments. This stress indicator is the highest during dual tasking.

Interestingly, baseline stress was a bit higher than the initial single task stress. This indicates that either the baseline was poorly designed (just sitting idle can be stressful) or participants carried some residual stress from the informal test-driving phase that preceded it. Participant S-6 is an interesting case. It appears that his stress started decreasing in the middle of dual tasking (Fig. 5). On careful examination of the data, the investigators found that this is the only participant who started perspiring in the middle of the experiment, apparently due to overwhelming stress. Perspiration reduces the thermal signature and the current method wrongly interprets this as lower metabolic rate and thus, lower stress. It is exactly the opposite. A method that identifies the emergence of perspiration and switches measurement metrics is needed to overcome this issue. In all cases, the rate of thermal change of the cool-off segment had an opposite global trend than that of the dual task segment. In most cases, the rate of thermal change of the cool-off segment had also an opposite global trend than that of the initial and latter single task segments. This illustrates that the participants indeed felt relaxed after 5 minutes of intense mental activity but at various degrees. Those participants who were thinking about their performance during the cool-off period exhibited slow recovery. This is an interesting finding, as it illustrates that not only an action but also thoughts about the action (past action in this case) could affect stress levels. Performance of the drivers degraded during the dual task segment, as measured by the point system of the simulator. This was inversely proportional to the mean stress level measured through the supraorbital channel.

174

D. Shastri, I. Pavlidis, and A. Wesley

Fig. 5. During the dual task period, the supraorbital signal (marked in blue color) of S6 showed ascending global trend in the first half and then descending global trend in the second half (marked in green color). The culprit is the onset of perspiration.

5 Conclusion This research brings to the fore a stress quantification method ideally suited to situations where the attention of the machine operator is divided. Unobtrusive quantification of stress and its correlation to operator performance and emotions are of singular importance in man-machine interaction. A feedback system can be developed that alerts the operator about his/her stress status based on the facial thermal signature. Results from a pilot experiment on the effect of cell phone communication during driving are more than encouraging. They open the way for a plethora of other multitasking experiments drawn from daily life. At the technical level, the issue of perspiration, which corresponds to the onset of extreme stress, cannot be handled with the current method. A method that identifies perspiratory patterns and handles thermal computation in a different manner from that point onward is needed in the future.

Acknowledgments This material is based upon work supported by the National Science Foundation under Grant No. #ISS-0812526, entitled “Do Nintendo Surgeons Defy Stress.” Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

References 1. Puri, C., Olson, L., Pavlidis, I., Levine, J., Starren, J.: StressCam: Non-contact measurement of users’ emotional states through thermal imaging. In: CHI 2005 Extended Abstracts on Human Factors in Computing Systems, pp. 1725–1728 (2005)

A Method to Monitor Operator Overloading

175

2. Palvidis, I., Eberhardt, N.L., Levine, J.: Human behavior: Seeing through the face of deception. Nature 415, 35 (2002) 3. Palvidis, I., Levine, J.: Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine 21(6), 56–64 (2002) 4. Dowdall, J., Pavlidis, I., Tsiamyrtzis, P.: Coalitional tracking. Computer Vision and Image Understanding 106, 205–219 (2007) 5. Tsiamyrtzis, P., Dowdall, J., Shastri, D., Pavlidis, I., Frank, M.G., Ekman, P.: Imaging facial physiology for the detection of deceit. International Journal of Computer Vision 71(2), 197–214 (2006) 6. Standage, D.I., Trappenberg, T.P., Klein, R.M.: A continuous attractor neural network model of divided visual attention. In: Processing of IEEE international Joint Conference on Neural Networks, vol. 5, pp. 2897–2902 (2005) 7. Yamakoshi, T., Yamakoshi, K., Tanaka, S., Nogawa, M., Shibata, M., Sawada, Y., Rolfe, P., Hirose, Y.: A Preliminary Study on Driver’s Stress Index Using a New Method Based on Differential Skin Temperature Measurement. In: 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 722–725 (2007) 8. Yamaguchi, M., Wakasugi, J., Sakakima, J.: Evaluation of Driver Stress using Biomarker in Motor-vehicle Driving Simulator. In: 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1834–1837 (2006) 9. Yili, L.: A queuing network model of human performance of concurrent spatial and verbal tasks. IEEE Transactions on Systems, Man and Cybernetics 27(2), 195–207 (1997) 10. Kenmochi, A., Takaki, Y., Fukuzumi, S.: Psychological tension estimation during the use of a driving simulator: A finger and ear pulse volume study. In: Proceedings of the 18th Annual International Conference of the IEEE Bridging Disciplines for Biomedicine, vol. 5, pp. 1804–1805 (1996) 11. Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems 6, 156–166 (2005) 12. FLIR Systems, 70 Castilian Dr., Goleta, California 93117, http://www.flir.com

Decoding Attentional Orientation from EEG Spectra Ramesh Srinivasan, Samuel Thorpe, Siyi Deng, Tom Lappas, and Michael D’Zmura Department of Cognitive Sciences, UC Irvine, SSPA 3151, Irvine, CA 92697-5100 {r.srinivasan,sthorpe,sdeng,tlappas,mdzmura}@uci.edu

Abstract. We have carried out preliminary experiments to determine if EEG spectra can be used to decode the attentional orientation of an observer in threedimensional space. Our task cued the subject to direct attention to speech in one location and ignore simultaneous speech originating from another location. We found that during the period where the subject directs attention to one location in anticipation of the speech signal, EEG spectral features can be used to predict the orientation of attention. We propose to refine this method by training subjects using feedback to improve classification performance. Keywords: EEG, attention, orienting, classification.

1 Introduction This report summarizes our preliminary studies towards developing a BCI to decode intended direction, i.e., the attentional orientation of an individual, from EEG recordings alone. This effort is part of a larger project (ARO 54228-LS-MUR, Silent Communication Among Dispersed Forces) to develop a BCI to decode intended speech and intended direction from EEG signals. The most obvious indicators of attentional orientation are head and eye orientation. However, even when head and eyes are oriented in one direction, an observer can attend to other directions. In experimental psychology, this is referred to as covert attention[1-3]. Covert attention involves all the senses including visual, somatosensory, and auditory information processing [4-5]. Auditory information processing is particularly relevant to covert attention, because the auditory system can identify sources which lie outside the visual scene, including to the side of and behind the observer. Mechanisms which monitor and detect sources throughout the environment selectively are essential to guide head and eye movements. The close relationship between attention and orienting eye and head movements is supported by experimental studies and theoretical models which suggest a motor programming role for attention [6-7]. Most previous EEG and fMRI studies focused on the effects of orienting attention on the response to a sensory stimulus [8-9]. Across all of these studies, the main effect is that larger responses are recorded to attended stimuli. More recently, a more nuanced view has emerged which suggests that distinct brain networks respond to attended and unattended stimuli [10-11]. Our goal is to identify the neural signature of orienting attention in one direction even when there is no stimulus at that location. Thus, we are interested in the J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 176–183, 2009. © Springer-Verlag Berlin Heidelberg 2009

Decoding Attentional Orientation from EEG Spectra

177

top-down instruction to orient attention in one direction (or to one location) rather than the orienting elicited by a salient stimulus (bottom-up). Recently, a number of EEG and fMRI studies have been directed at this question and have demonstrated preparatory neural activity when attention is directed by instruction (cued) to one location on a computer screen [12-15]. The fMRI studies examined retinotopically mapped areas of the visual system and demonstrated preparatory increases in neuronal activity as indexed by the BOLD signal. EEG studies showed increased alpha rhythm in the parietal cortex ipsilateral to the attended visual field. Although these studies have elucidated some of the mechanisms of top-down attentional orienting, they are limited in their usefulness towards developing a BCI that can decode attentional orientation. In general, orienting attention takes place in a larger sensory field, not just a limited sector of visual space within 10-12 degrees of fixation on a computer monitor. In audition, perception of sources takes place in all directions, even behind the subject. Thus, in our experiment observers selectively attend to auditory rather than visual stimuli in order to investigate attentional orientation in a wider span of sensory space. We have carried out experiments directing attention to one of two directions and identify the attended direction by classification of the EEG spectra. Our results suggest that attentional orientation can potentially be decoded from the EEG but that further work is needed to train the observers and improve classification methods.

2 Methods Procedure. Six subjects participated in a speech perception experiment (see Figure 1). The subject was seated in a dimly lit room between two speakers (each at 1 m distance) and instructed to fixate on a point. There were two experimental conditions – attend left or attend right. The subject is given the instruction through both speakers. After a variable ISI (500, 700, 900, 1100, or 1300 ms), two different speech stimuli were presented one through each speaker. The speech stimuli were synthesized (http://cepstral.com) in two distinct male voices, one played through each speaker. The assignment of voices to speakers was independently randomized on each trial. The stimuli were a simplified version of the Coordinate Response Measure corpus [16]. These sentences were structured as “(Arrow, Baron, Eagle or Tiger) go to (Blue, Green, Red, or White) now”. The subjects’ task was to identify the two words played through the attended speaker with a response on a keypad. In the example shown in the figure the correct response is “Baron” and “Blue”. The experiment was designed in this manner to demand that the subject direct attention to the correct speaker before the speech was played; otherwise the subject would miss the first code word. The variable ISI and randomized voice ensured that the observer quickly deployed and maintained attention to the appropriate speaker. In addition an adaptive staircase procedure was used to control subject performance. When the subject responded correctly to the first word, the volume was reduced by 5 % on the attended speaker and increased by 5% on the unattended speaker on the next trial. When the subject responded incorrectly to the first word, the volume was increased by 10% on the attended speaker and decreased by 10% on the unattended speaker. This procedure resulted in subjects performing the task correctly about 70% of the time.

178

R. Srinivasan et al.

Fig. 1. The experimental setup. A. The physical layout of the experiment. Note that each speaker was 45 degrees away from fixation and could not be seen by observer without moving the eyes. B. The time course of each trial in the experiment. In the example shown, the instruction is to attend to the right and after a variable ISI, two distinct sentences are played through each speaker. In the example shown, the subject responds “Baron” and “Blue”, and received feedback indicating a correct response.

At this threshold, the amplitude of the attended speaker was typically 30 dB below the unattended speaker. A single experimental session comprised 200 trials, presented in two 100 trial blocks with a break. Each subject participated in three such sessions, each lasting around an hour. EEG recording. EEG was recorded using a 128 Channel Geodesic Sensor Net (Electrical Geodesics, Inc., Eugene, OR, USA) in combination with an amplifier and acquisition software (Advanced Neuro Technology, Inc, Enschede, NL). The EEG was sampled at 1024 Hz and on-line average referenced. Artifact editing was performed through a combination of automatic editing using an amplitude threshold and a manual editing to check the results. Trials with excessive bad channels (> 15 %) were first discarded and then channels with bad trials were discarded. This typically yielded 80100 usable EEG channels (out of 128) and 500-550 usable trials. Data Analysis. We focused on three intervals within the ISI – 0-500, 200-700, and 400-900 ms. For the later intervals, we discarded trials where the target stimulus had already started, resulting in different numbers of trials for the three intervals – 600, 480, and 360. The data were Fourier transformed using an FFT (Matlab) for each 500 ms interval ( f = 2 Hz) and the power spectrum calculated as the squared magnitude of the Fourier coefficients. We limited further analyses to the interval from 4-22 Hz.

Decoding Attentional Orientation from EEG Spectra

179

This was motivated by the goals of this study to identify robust spectral features that can predict attentional orientation. Below 4 Hz EEG is often contaminated with movement and eye blink artifacts. Above 20 Hz the EEG is often contaminated with EMG artifacts. EEG power at each frequency was log transformed and normalized against the total power from 4-22 Hz. Classification. Classification was performed using a naïve Bayes classifier (Matlab) which assumes independent variables, i.e., a diagonal covariance matrix. The data were divided into three conditions based on instruction and performance: Correct Left, Correct Right, and Incorrect. There were roughly equal numbers of Correct Left

Fig. 2. Examples of classification performance versus the number of variables for the three classification intervals. Overall S0 had the best classification performance and S5 had the worst performance. The other subjects showed intermediate levels of classification performance.

180

R. Srinivasan et al.

and Correct Right trials, and 10-20% fewer Incorrect trials. To facilitate comparisons in classification performance between the analysis intervals, we used a fixed number (determined by the 400-900ms interval) of randomly selected “training” trials (typically 150) to calculate a linear classifier, which was applied to classify another 150 “test” trials. The classification proceeded in two steps. First we evaluated the performance of each individual variable (80-100 channels x 10 frequencies) in classification of the “test” data, using 30 random samples of “training” and “test” trials. Then we evaluated the performance of the best 10, 20, 50, 75, and 100 variables using 50 random samples of “training” and “test trials”.

Fig. 3. Power spectra from the channel that provided the best three-way classification for each subject. For 3 subjects (S0, S3, and S7) the channel shown is located over parietal cortex. For the other subjects (S2, S4 and S5) the channel shown is located over the frontal lobes. Note that spectral differences are present at most frequencies.

Decoding Attentional Orientation from EEG Spectra

181

Fig. 4. Topographic distribution of channel sensitivity to attentional orientation. Each channel classification ability was scored and averaged across frequencies. The results were plotted as a topographic map, where white shows the highest predictive power and black indicates no predictive power.

3 Discussion In a simple cued spatial attention experiment we were able to demonstrate preliminary results indicating that the EEG contains information that can be used to decode the orientation of attention in space. Our approach here has been very direct and overly

182

R. Srinivasan et al.

simplistic. We have made use of spectral features of the EEG and a naïve Bayes classification scheme. Both approaches can be significantly improved. Despite the limitation of these methods, we were able to achieve as high as 75% classification performance in 2-way classification and 60% classification performance in 3-way classification. This indicates that EEG signals clearly contain information about attentional orientation that can be used to decode attentional orientation for BCI applications. The data also suggest that the signatures of attentional orientation can be more robustly decoded at a longer interval following the cue. In our comparisons, 400-900 ms after the cue provided the best classification in each subject. This is consistent with theoretical and experimental studies of the episodic nature of attention, which suggested that following the cue the attention window will take at least 300 ms to open [17]. How long this window remains open and whether more robust classification can be obtained remains to be tested. Estimating the duration for which the neural signatures of attentional orientation are sustained will require a larger amount of data. Our results suggest that electrodes over frontal and parietal cortex have the greatest sensitivity to attentional orientation. This finding is consistent with a large body of experimental research with EEG and fMRI that indicate that large-scale networks spanning parietal and frontal cortex mediate selective attention [18-20]. More surprising was the lack of frequency specificity in our classification results. Previous reports had suggested occipital/parietal alpha rhythms would be sensitive to attentional orientation [15, 21]. However, those studies used visual displays where attention was directed to one of two regions within 4-6 degrees of fixation. Thus, those results were specifically related to attentional orienting within the narrowly defined retinotopically mapped visual space. Our results relate to orienting to two regions of auditory space separated by 90 degrees and out of the field of view while fixating. The results of this paper are preliminary and indicate the potential for obtaining an attentional orientation BCI. An important factor we have not yet considered is training subjects to optimize the BCI. Our current work extends this study through a consideration of the full 360 degrees of auditory space surrounding the subject.

Acknowledgements This work was supported by ARO 54228-LS-MUR.

References 1. Desimone, R., Duncan, J.: Neural mechanisms of selective visual attention. Annu. Rev. Neurosci. 18, 193–222 (1995) 2. Egeth, H.E., Yantis, S.: Visual attention: control, representation, and time course. Annual Reviews in Psychology 48, 269–297 3. Kastner, S., Ungerleider, L.G.: Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience 23, 315–341 (2000) 4. Spence, C., Pavani, F., Driver, J.: Crossmodal links between vision and touch in covert endogeneous spatial attention. Journal of Experimental Psychology: Human Perception and Peformance 26, 1298–1319 (2000)

Decoding Attentional Orientation from EEG Spectra

183

5. Spence, C., Driver, J.: Crossmodal attention. Current Opinion in Neurobiology 8, 245–253 (1998) 6. Moore, T., Armstrong, K.M., Fallah: Visuomotor origins of covert spatial attention. Neuron 40, 671–683 7. Sheliga, B.M., Riggio, L., Rizzolatti, G.: Orienting of attention and eye movements. Experimental Brain Research 98, 507–522 (1994) 8. Hillyard, S.A., Anllo-Vento, L.: Event-related brain potentials in the study of visual selective attention. PNAS 95, 781–787 (1998) 9. Kastner, S., Pinsk, M., De Weerd, P., Desimone, R., Ungerleider, L.: Increased activity in human visual cortex during directed attention in the absence of visual stimulation. Neuron 22, 751–761 (1999) 10. Corbetta, M., Shulman, G.L.: Control of goal- and stimulus-driven attention in the brain. Nature Reviews Neuroscience 31, 201–215 (2002) 11. Ding, J., Sperling, G., Srinivasan, R.: SSVEP power modulation by attention depends on the network tagged by the flicker frequency. Cerebral Cortex 16, 1016–1029 (2006) 12. Sylvester, C., Jack, A., Corbetta, M., Shulman, G.: Anticipatory Suppression of Nonattended Locations in Visual Cortex Marks Target Location and Predicts Perception. Journal of Neuroscience 28(26), 6549–6556 (2008) 13. Shulman, G., Ollinger, J., Akbudak, E., Conturo, T., Snyder, A., Petersen, S., Corbetta, M.: Areas Involved in Encoding and Applying Directional Expectations to Moving Objects. The Journal of Neuroscience 19(21), 9480–9496 (1999) 14. Fu, K., Foxe, J., Murray, M., Higgins, B., Javitt, D., Schroeder, C.: Attention-dependent suppression of distracter visual input can be cross-modally cued as indexed by anticipatory parieto–occipital alpha-band oscillations. Cognitive Brain Research 12, 145–152 (2001) 15. Worden, M., Foxe, J., Wang, N., Simpson, G.: Anticipatory Biasing of Visuospatial Attention Indexed by Retinotopically Specific alpha band Electroencephalography Increases over Occipital Cortex. The Journal of Neuroscience 20 (2000) 16. Moore, T.J.: Voice communication jamming research. In: AGARD Conference Proceedings 331: Aural Communication in Aviation, vol. 2, pp. 1–6 (1981) 17. Weichselgartner, E., Sperling, G.: Dynamics of automatic and controlled visual attention. Science 238, 778–780 (1987) 18. Gitelman, D., Nobre, A., Parrish, T., LaBar, K., Kim, Y., Meyer, M., Mesulam, M.: A large-scale distributed network for covert spatial attention, Further anatomical delineation based on stringent behavioural and cognitive controls. Brain 122(6), 1093–1106 (1999) 19. Posner, M., Petersen, S.: The attention system of the human brain. Annual Review Neuroscience 13, 25–42 (1990) 20. Coull, J.T.: Neural correlates of attention and arousal: insights from electrophysiology, functional neuroimaging and psychopharmacology. Progress in Neurobiology 55(4), 343– 361 (1998) 21. Foxe, J., Simpson, G., Ahlfors, S.: Parieto-occipital ~10 Hz activity reflects anticipatory state of visual attention mechanisms. NeuroReport 9, 3929–3933 (1998)

On the Possibility about Performance Estimation Just before Beginning a Voluntary Motion Using Movement Related Cortical Potential Satoshi Suzuki1, Takemi Matsui1, Yusuke Sakaguchi1, Kazuhiro Ando1, Nobuyuki Nishiuchi1, Toshimasa Yamazaki2, and Shin’ichi Fukuzumi3 1

Tokyo Metropolitan University, Asahigaoka 6-6, Hino, Tokyo 191-0065, Japan Kyusyu Institute of Technology, Kawazu 680-4, Iizuka, Fukuoka 820-8502, Japan 3 NEC Common Platform Software Research Laboratories, Shibaura 2-11-5, Minatoku, Tokyo 108-8557, Japan [email protected] 2

Abstract. The present study aimed to investigate this tripartite relationship, regarding MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance. Experiments were conducted 'reaching' task; the subject touches the target appears 300 pixels away from the start point in a vertical direction on the touch sensitive screen with the forefinger. During experiments, EEG, EMG as trigger, image by high-speed camera and the efficiency of task were acquired. As a result, significant differences between the high and poor performance groups were clear on the NS in MRCP acquired from Fz (p < 0.05), Cz (p < 0.05) and Pz (p < 0.05). Furthermore, the difference was confirmed on the duration of ballistic movement. Based on our findings, we attempted to extract MRCP rapidly and automatically without using signal averaging and discuss whether it is possible to estimate accuracy just before the motion is executed. Keywords: Accuracy, ballistic movement, movement-related cortical potential (MRCP), reaching, voluntary motion.

1 Introduction Movement-related cortical potential (MRCP) is a feature of brain waves that appears toward the midline of the head approximately one to two seconds before the onset of voluntary motion [1, 2, 3]. Although the divisions differ slightly between researchers, MRCP is usually separated into the Bereitschafts Potential (BP), the Intermediate Slope (IS) and the Negative Slope (NS). The BP and IS appear at the front and top of the head and have a gradual slope from 2000-500 msec before movement, while the NS starts approximately 500 msec before motion and, in the case of hand movement, has an amplitude of 10 – 15 μV [3]. It can also be observed if the subject imagines the motion in their head without undertaking any real activity [4]. Recently, some studies have demonstrated a relationship between MRCP and eye movements [5], and J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 184–191, 2009. © Springer-Verlag Berlin Heidelberg 2009

On the Possibility about Performance Estimation

185

to use as a method of medical diagnostics [6], although not all details have been fully clarified. Previous work has also shown that the thinking process and information processing within the brain is broken into several steps. Many useful models of this information processing in the head are proposed [7, 8]. These models commonly have 4 steps: perception, cognition, motion planning and motion. Related to this model, MRCP is generally assumed to reflect a preparation or planning stage for beginning the motion [9]. On the other hand, “reaching”, such voluntary goal-directed movement, is known to be one of the most important components of motion by human arms. Mathematical models considering jerk, torque and dynamics of the musculoskeletal system have been used to understand how the motion of reaching is planned and controlled in the brain [10, 11, 12]. Movement efficiency has also been studied and Fitts’ Law proved that reaction time during reaching constantly changes according to the difficulty of the task [13, 14]. During the motion process reaching has two characteristics, the ballistic and corrective movements [15, 16], which influence both the accuracy and duration time of the motion. Observation of these characteristics enables us to comprehend the efficiency of the task performance. There is believed to be a relationship between the ballistic movement and MRCP as the former is a feed-forward movement and MRCP reflects a planning stage for this motion. Building on this, a relationship has also been suggested between MRCP, the ballistic movement and the accuracy of motion (Figure 1). Considering the tripartite relationship among MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance has meaning for ergonomics field. Motion Start Perception Cognition Controlled Processing

Information Processing

Action

MRCP EEG

Motion

BP

IS

NS N-P feedforward

feedback

Ballistic Movement

Collective Movement

Task Performance

Motion Time Accuracy

Fig. 1. The concept of this research. The aim of this study is an investigation the tripartite relationship, regarding MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance.

186

S. Suzuki et al.

The present study aimed to investigate this tripartite relationship, regarding MRCP as a physiological index, ballistic movement as an index of operation and accuracy of the task performance within the field of ergonomics. Based on our findings, we attempted to extract MRCP rapidly and automatically without using signal averaging. We explain and consider the results of this extracting system and discuss whether it is possible to estimate accuracy just before the motion is executed.

2 Methods 2.1 Electroencephalogram MRCP is generally observed at scalp positions Fz, Pz and Cz, on the midline of the frontal and parietal regions of the head. To achieve high spatial resolution around these regions, electroencephalogram (EEG) data were acquired using a 128ch sensor net (Geodegic Sensor Net, Electrical Geodesics Inc., OR, USA) and analyze system (Net Station 4.3, Electrical Geodesics Inc., OR, USA). Electrode impedance was set between 10-50 kΩ. EEG was recorded using a 0.1–50 Hz bandpass (3 dB attenuation). Signals were sampled at 1 kHz and were digitized. 2.2 EMG and Trigger The trajectory of voluntary motion was observed using a high-speed camera (Fastcam512PCI, Fotron Co., Tokyo, Japan) with 125 fps (Figure 2). The trigger signal was generated using the surface EMG on the common digital extensor muscle. Data from the EEG and the high-speed camera were clipped at each trial using triggers. EMG data were acquired using a bio-amp system (Biotop, NEC-Sanei Co., Tokyo, Japan) with a sampling rate of 10 kHz to accurately synchronize and analyze in real time using a Field Programmable Gate Array (FPGA) module (PCI-7831, National Instruments Co., TX, USA). 2.3 Subjects and Task Procedure Experiments were conducted on eight healthy male subjects ranging in age from 22 to 24 years old (average 22.88 +/- 0.83 years). All subjects were right-handed as confirmed by the Edinburgh handedness test [17]. Experiments were conducted in an electromagnetic-shielded room using the following protocols: the subject touches the center of a 17-inch touch-sensitive screen (LCD-AD172F2-T, I/O Data Co., Tokyo, Japan) located 30 cm in front of them with their forefinger. A small cross-shaped target then appears 300 pixels away from the point previously touched in a vertical direction (screen pixel pitch 0.264 × 0.264 mm). The subject moves their forefinger to the center of the displaying target and touches it. The trial is repeated a total of 50 times in two sets. Specific instructions regarding motion speed and accuracy during trial performance were not given to all subjects. The subject’s head was placed on a jaw rest and the right forearm on an armrest to reduce artifacts other than motion of the forefinger.

On the Possibility about Performance Estimation

187

2.4 Analysis MRCP and high-speed camera data were clipped in each trial by a trigger signal generated using the surface EMG on the common digital extensor muscle. EEG data were clipped from 1500 msec before the start of the movement to 500 msec afterward, according to previous studies. The gap from the center of the target to the position actually touched was used as an evaluation index for the accuracy of the voluntary movement. Based on the gap average of task performance, we divided data of performance in each trial into two

EMG System EMG Amp.

PC1 (Full-wave Rectification) Trigger Signal 1

EEG System Shielded room

EEG Electrodes

128ch EEG Amp.

EMG Electrode

PC2 (EEG Analysis)

Motion Analysis System High Speed Camera

PC3 (Motion Analysis)

Lenz

Touch Panel

Trigger Signal 2

Task System PC4 (Task Control)

Fig. 2. Block diagram and data flow. Experiments were conducted on eight healthy male subjects using EEG, high-speed camera, EMG and task systems.

EEG DATA

Ti …..

T1T2T3T4

T50 Trigger

Task performance Average

A

High Accuracy Low Accuracy

EEG DATA

B

A BB

AA B

A B

….. …..

T5 T7T8 T10 ….. Signal T2 T9 ….. Averaging T1 T3T4 T6

MRCPgroupA MRCPgroupB

Comparison

Fig. 3. Analytical process. MRCP and high-speed camera data were clipped in each trial by a trigger signal generated using the surface EMG. Based on the gap average of task performance, the clipped EEG and high-speed camera data from each trial were separated into two groups corresponding to the ‘A’ and ‘B’ groups. The typical waveforms in each group were calculated by averaging and compared with each other.

188

S. Suzuki et al.

groups: a high performance group ‘A’ and a low performance group ‘B’. The clipped EEG and high-speed camera data from each trial were separated into two groups corresponding to the ‘A’ and ‘B’ groups. The typical waveforms in each group were calculated by averaging and compared with each other (Figure 3).

3 Results 3.1 Task Performance Figure 4 shows the distribution of gaps in each trial according to a subject (S1). The distribution resembled a gamma distribution with an average gap of 5 pixels. The trend of the distribution form was seen in all subjects. EEG signals corresponding to trials when the gap was less than 5 pixels were placed in group A and those over 5 pixels were placed in group B, in this subject S1’s case. Typical waveforms corresponding to the two groups were calculated by averaging. High-speed camera data were also divided into two groups. 3.2 MRCP Figures 5 a) and Figure 5 b) show sample of typical MRCP waveforms corresponding to the two groups acquired from Fz and Cz based on the same subject S1 shown in Figure 4. Differences between BPs (observed from 2000 msec before movement) and ISs (observed from 900 to 500 msec before movement) of groups A and B were not obvious. However, differences in NSs (observed from 500 msec before movement) were clearly observed between the two groups, with group A showing a steeper slope than group B. This trend was confirmed from around Fz, Cz, and Pz of all subjects (Figure 6) and the difference in average values between the two groups was found to be significant (Fz: p < 0.05 (p = 0.019), Cz: p < 0.05 (p = 0.050), Pz: p < 0.05 (p = 0.017)). This suggests that there is a relationship between the performance of the arm and the NS slope. Average

20

A group

B group

frequency

15

10

5

0 0

5

10

15

20

Accuracy (pixel)

Fig. 4. Sample data of result of performance (Subject 1). The distribution resembled a gamma distribution with an average gap of 5 pixels, in this subject S1’s case.

On the Possibility about Performance Estimation

A group

B group

A group

Trigger

B group

189

Trigger

-

-1500

-1000

-500

0

500

msec

-1500

5μV

-1000

-500

0

500 msec

5μV

+

+

a) Fz

b) Cz

Fig. 5. Sample data of MRCP (Subject 1). Typical MRCP waveforms corresponding to the two groups acquired from Fz (in Fig. 5 a)) and Cz (in Fig. 5 b))based on the same subject S1 shown in Figure 4. *

15

*

*

Slope (μV/sec)

A group B group

10

(*:p<0.05)

5

0 Fz

Cz

Pz

Fig. 6. Comparison of NS slopes in 2 groups (All subjects). Differences in NS (observed from 500 msec before movement) were clearly observed between the two groups, with group A showing a steeper slope than group B.

3.3 Process of the Reaching Motion Figure 7 shows the movement distance of the forefinger peak. A small difference in ballistic movement between groups A and B was confirmed, but we could not confirm a difference in corrective movement. This is because the experimental task is simple and primitive and the quantity allocated for corrective movement was few so it is not necessary to allocate for corrective movement. Movement distance (mm) (mm)

500

400 400

300

A group B group

200 200 100

00

1 Ballistic Movement

2

Corrective Movement

Fig. 7. Comparison of movement processes in 2 groups (All subjects). A small difference in ballistic movement between groups A and B was confirmed.

190

S. Suzuki et al.

4 Discussion and Conclusion In the present study, we attempted to confirm the tripartite relationship between MRCP, ballistic movement and accuracy of task performance. We could not confirm any difference between high-performance group ‘A’ and low- performance group ‘B’ on BP and IS in MRCP acquired from Cz and Fz. However, significant differences between the groups were clear on the NS in MRCP acquired from Cz and Fz. Furthermore, the difference was confirmed on the duration of ballistic motion. As NS is generally believed to represent a preparation stage for voluntary motion, it appears that the observed difference between groups influences the process of motion and the task performance. These results show the possibility of estimating execution of the performance just before beginning the voluntary motion using MRCP. On the other hand, the motion “reaching” is nearly optimized in terms of smoothness over the entire movement in the field considering with mathematical models. Various optimal criteria are proposed for trajectory planning of this motion in terms of multi-joint arm movements, like human arms. The minimum-jerk criterion [10] plans smooth trajectories in the extrinsic task space, while the minimum-joint-anglejerk criteria, the minimum-torque-change criteria [11], and the minimum-motorcommand-change criteria plan smooth trajectories [12] in the intrinsic body space. These models and criteria are discussed under the assumption that humans plan the trajectory of arm movement before beginning the motion. However, Hogan & Wolpert [18] pointed out that it is difficult to explain the biological relevance of factors such as jerk or torque change in previous models of arm trajectory. They showed that the movement can be achieved by reducing the variance of errors at the end of the movement. The final goal of the “reaching” movement is to minimize the gap at the end of the motion, as suggested by their minimum-variance theory. Although it is not clear how the cerebrum and cerebellum contribute to achieving this movement, this model is an effective way of considering this movement. If we place our own results in context with this model, the concept becomes validated as it is known that humans plan and learn the trajectory of reaching with optimal efficiency just before beginning the motion. Finally in the current study, we attempted to develop a prototype system to derive the NS slope in MRCP from EEG data automatically and in real time. When we observe the P300 and N200, the importance of the shape of the waveform necessitates signal averaging. If the event-related potentials (ERP), such as N200 and P300, was observed, the technique signal averaging is generally used not only to remove the artifact but also to extract more information, i.e. latency, amplitude, the shape of the waveform and so on. However, in the case of MRCP, we only need to observe slopes during the 500 ms just before the motion is executed, namely, just before the trigger. We are therefore attempting to develop a prototype system that obtains a sequentialmemory of the value during 2000 ms of EEG and sequentially calculates the value of the NSs slope in 500 ms. This prototype system is currently being developed using LabVIEW (National Instruments Co., TX, USA) and we already confirmed that the performance can be estimate with 75 % maximum of the time using an index based on NS values. Thus, in the future, our concept and method appears to be promising for use controlling devises safety, such as at car driving.

On the Possibility about Performance Estimation

191

References 1. Kornhuber, H.H., Deecke, L.: Hirnpotentialanderungen bei Willkurbewegungen und passiven Bewegungen des Menschen. Bereitschafts potential und reaffernte potentiale. Pfligers Arch, Gesamte Physiol Menschen Tiere 284, 1–17 (1965) 2. Barrett, G., Shibasaki, H., Neshige, R.A.: Computer-assisted method for averaging movement-related cortical potentials with respect to EMG onset. Electroencephalogr. Clin. Neurophysiol. 60(3), 276–281 (1985) 3. Shibasaki, H., Hallett, M.: What is the Bereitschaftspotential? Clin. Neurophysiol 117(11), 2341–2356 (2006) 4. MacKay, D.M., MacKay, V.: Behind the eye. Basil Blackwell, Malden (1991) 5. Yamamoto, J., Ikeda, A., Satow, T., et al.: Human eye fields in the frontal lobe as studied by epicortical recording of movement-related cortical potentials. Brain 127, 873–887 (2004) 6. Barrett, G., et al.: Cortical potential shifts preceding voluntary movements are nomal in parkinsonism. Electroencephalogr. Clin. Neurophysiol. 63, 340–348 (1986) 7. Card, S.K., Moran, T.P., Newell, A.: The psychology of human-computer interaction. Lawrence Erlbaum Associates, New Jersey (1983) 8. Gopher, D., Sanders, A.F.: S-Oh-R: Oh Stages! Oh Resources! In: Prinz, W., Sanders, A.F. (eds.) Cognition and motor processes, Springer, Heidelberg (1984) 9. Revelle, W.: Indivisual differences in personality and motivation: ’non-cognitive’ determinants of cognitive performance. In: Baddeley, A., Weiskrantz, L. (eds.) Awareness Control, Clarendon Press, Oxford (1991) 10. Flash, T., Hogan, N.: The Coordination of Arm Movements: An Experimentally Confirmed Mathematical Model. Journal of Neuroscience 5, 1688–1703 (1985) 11. Uno, Y., Kawato, M., Suzuki, R.: Formation and control of optimal trajectory in human multijoint arm movement - Minimum torque-change model. Biological Cybernetics 61(2), 89–101 (1989) 12. Kawato, M.: Optimization and learning in neural networks for formation and control of coordinated movement. In: Meyer, D., Kornblum, S. (eds.) Attention and performance XIV, pp. 821–849. MIT Press, Cambridge (1993) 13. Fitts, P.M., Peterson, J.R.: Information capacity of discrete motor responses. Journal of Experimental Psychology 67, 103–112 (1964) 14. Accot, J., Zhai, S.: Beyond Fitts’ Law: Models for trajectory-based HCI tasks. In: CHI 1997, pp. 295–302 (1997) 15. Flowers, K.: Ballistic and corrective movement on an aiming task. Neurology 25, 413–421 (1975) 16. Pratt, J., Abrams, R.A.: Practice and component submovements: The roles of programming and feedback in rapid aimed limb movements. Journal of Motor Behavior 28(2), 149–156 (1996) 17. Oldfield, R.C.: The assessent and analysis of handedness: The Edinburgh Inventory. Neuropsychologia 9, 97–113 (1971) 18. Harris, C.M., Wolpert, D.M.: Signal-dependent noise determinations motor planning. Nature 394, 780–784 (1998)

A Usability Evaluation Method Applying AHP and Treemap Techniques Toshiyuki Asahi, Teruya Ikegami, and Shin’ichi Fukuzumi Common Platform Software Research Laboratories, NEC Corporation, 8916-47, Takayama-Cho, Ikoma, Nara 630-0101, Japan [email protected]

Abstract. This report proposes a visualization technique for checklist-based usability quantification methods. By applying the Treemap method, the hierarchical structure of checklists, weights of check items and evaluation results for target systems can be viewed at a glance. Effective support for usability analysis and its presentation tasks of usability evaluation results are expected. A prototype tool was implemented on a PC and experimental studies assuming actual usability evaluation tasks were conducted. The results indicate that the proposed method improves performance time of some typical tasks. Usability engineers gave higher subjective scores on the usefulness of the proposed method than that of printed table presentation .

Keywords: Usability quantification, checklist, Treemap, visualization, analytic hierarchy process, design tools.

1 Introduction Usability quantification has been considered one of the key technologies for systematizing and spreading user-centered design and has been studied for some time [1]. Several methods such as 1) performance analysis/prediction, 2) subjective questionnaire, 3) checklist-based heuristic and 4) biological metric measurement/analysis have been developed as quantitative usability evaluation methods. Among those, checklist-based heuristics can provide early stage evaluation and synthetic rating even for software systems with many functions and can also pinpoint the user interface elements or interaction techniques to be improved when the checklist is designed carefully and goes into detail [2]. Therefore, this method is expected to fit for benchmarking and usability improvement of large-scale software applications. However, in addition to its relatively higher execution cost, it has a disadvantage in that it is not easy for evaluators or developers to grasp the evaluation result because of the huge number of check items and their complex (hierarchical) structure. One of the solutions to this problem is to visualize the evaluation results properly. Yamada et al. [3] devised an explanation of the score distribution of target systems along with several check item categories by using segmented bar graphs. Although this method seemed to be successful in showing and comparing target systems’ total scores, J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 195–203, 2009. © Springer-Verlag Berlin Heidelberg 2009

196

T. Asahi, T. Ikegami, and S. Fukuzumi

it was still hard to read out which and how check items contribute to (or lower) the synthetic scores since the detailed check items below the second hierarchy (88 items) are not presented on the graph. In this paper, we propose to apply the well-known technique, Treemap, to visualize the evaluation result of checklist-based heuristics. One of the quantification techniques using checklist will be explained first; then the Treemap visualization techniques for that method will be demonstrated with the examples on the PC tool prototype. Section 4 describes the procedure and the results of the experimental study validating the effectiveness of the proposed method. Finally, we discuss the result and its implications for future study.

2 Checklist-Based Usability Quantification Treemap visualization was applied on a checklist-based heuristic method developed by Ikegami et al. [4]. The checklist consists of 126 check items categorized into five sections and 18 sub-sections, which were extracted and arranged from various user interface (UI) guidelines, ISO standards [5][6] and consultation know-how. Each check item is described from the viewpoints of UI components or system functions such as “Are titles attached to each window?” or “Are substitutive operations provided for double-click operations?” However, the usability score is expected to be given from user viewpoints (ex. learnability, memorability) when it is utilized in usability testing or benchmarking. Therefore, a weighting value was given to each checklist item for selected user viewpoints. Several techniques have been known for weight calculation such as expert opinion, task usage frequency or task importance, KANO model, entropy model, geometric mean of pair-wise comparison result in AHP (Analytic Hierarchy Process) and number of problems collected [7]. Ikegami et al. adopted the AHP method considering reliability and execution cost. A few usability experts applied a pair comparison method to give weighting value sets for every viewpoint. Thus, four weighting value sets, which correspond to “learnability,” “memorability,” “efficiency” and “low error rate” (selected by referring to [8]), were given to the check items, their sections and sub-sections (Fig. 1).

3 Usability Visualization Using Treemap In order to visualize the evaluation result of the checklist effectively, we think the major requirements are: 1. Giving the overall view of the checklist including its hierarchical structure, and 2. Displaying detailed information such as each check item’s weight and checking result. In other words, both structural and quantitative information, and both overall and detailed information should be displayed in one view in a form easy to understand. We assumed the Treemap method [9] developed at the University of Maryland would fill these requirements and tried to apply it to the checklist heuristics. The following parts of this section describe detailed techniques of Treemap implementation.

A Usability Evaluation Method Applying AHP and Treemap Techniques

197

Fig. 1. Outline of checklist weighting method [4]

3.1 Visualize the Checklist Treemap techniques were applied to the checklist introduced in Section 2. Figure 2 shows examples of checklist visualization, (a) without weights (assuming all sibling items or sections have equal weighting value), and (b) with the weight set given for the “efficiency” viewpoint. The “slice and dice” and “offset” techniques introduced in reference [9] were adapted to utilize the display area effectively and to present check item structure and titles clearly (full titles of check items appear by simple mouse-over operation). It is obvious that the structure and weight distribution can be viewed in a given display area even for a checklist consisting of more than 100 items. By providing four sets of weighting values and the corresponding map, each of which represents one of the four viewpoints, the entire checklists can be visualized. Although it is quite easy to merge them into one map by adding one more level of hierarchy, we chose to show them separately because the merged Treemap looked too busy and there seemed to be less need for viewing four viewpoints simultaneously.

198

T. Asahi, T. Ikegami, and S. Fukuzumi

(a) No weight

(b) With a set of weights for "Efficiency" Fig. 2. Checklist visualization

3.2 Visualize the Checking Result Two large-scale Web application systems (tentatively named “A” and “B”) were evaluated along with the checklist, and the result was displayed as shown in Fig. 3. The preceding study [10] and Ikegame proposed to do checklist evaluation with yes (meaning the target user interface satisfies the corresponding check item) or no judgment for every check item in order to minimize the effect of individual difference. In the example of Fig. 3, colored areas (yellow for A, red for B) mean “yes” and black areas represent “no” judgment. The total scores, which are the summations of colored areas, are displayed as the bar chart below the map. Intuitively, it is quite easy to read out which check items have bigger influence on the total score, or why system B gets a higher score than A from the viewpoint of

Black: means "no" for A or B Yellow: means A satisfied the check item Red: means B satisfied the item

Fig. 3. Visualization of checking result with a bar chart

A Usability Evaluation Method Applying AHP and Treemap Techniques

199

efficiency, for instance. On the other hand, it is hard to explore the checking result for the low-weighted items. (However, this problem has been partly solved by implementing the zooming function, which enables the zooming of any node (rectangle) out to the base rectangle area with its subsidiary nodes.)

4 Experimental Study In the previous sections, Treemap visualization for a checklist-based quantification method was proposed. The simulated map seems to be useful for usability analysis tasks and for reporting their results. In this section, we try to validate its usefulness experimentally assuming actual data analysis tasks in usability consultation. 4.1 Experimental Design We assumed two kinds of situations in which Treemap visualization would be helpful: 1) usability engineers analyze the checking result, and roughly estimate the usability level of the target systems or detect where they should be changed for effective usability improvement, and 2) usability engineers or consultants explain the analysis result to developers of targeted systems or clients of the consultation. The tasks in the experiment were designed assuming situation (1). Outline. The Treemap on the PC screen (such as Fig. 2) and the table forms printed on paper sheets, both of which represent the checklist items, their weighting values, and checking results for two systems (A, B), were displayed to subjects. Each subject executed tasks assuming situation (1). Task completion time was measured by an experimenter along with the correctness of the answers. Subjective questionnaires were applied after completing all tasks. Table 1. Experimental tasks (task set TA)

a b c d e f

g

Task description Meaning of tasks From the “learnability” viewpoint, what Understand which category is supis the most significant category? posed significant in each viewpoint. Select all viewpoints in which system A’s Grasp target systems’ usability featotal score is higher than that of B. ture roughly. From the “low error rate” viewpoint, Understand which check item is what is the most important check item? most influential on the final score. From the “low error rate” viewpoint, Estimate which check item is not so what is the least important check item? critical. From the “memorability” viewpoint, Compare overall scores among tarwhich system’s score is higher? get systems. From the “memorability” viewpoint, Identify where to improve for raiswhich check item should be improved to ing usability score most effectively. raise system B’s score most effectively? Count up the number of check items in Compare system usability for cersub-category “data output,” where only B tain (restricted) usability aspect. is OK.

200

T. Asahi, T. Ikegami, and S. Fukuzumi

Participants. Nine usability engineers with usability testing or consulting experience ranging from 20-40 years old participated in the experiment. They were asked to do tasks supposing they were in a consulting situation. Tasks. Seven simple tasks supposed to be typical in actual usability analysis work were selected as shown in Table 1. In addition to the seven tasks (task set “TA”), similar tasks (doing the same thing using another data viewpoint) a’ - g’ were also prepared as task set “TB.” Each subject tried TA first, then tried TB. Half of them used table data for TA and Treemap data for TB, and the other half executed TA with the Treemap data and TB with the tables. This order is considered to minimize the training effect possibly appearing in the performance time. Just after completing TA and TB, all subjects were asked to check the seven subjective questionnaires. All subjects could complete all tasks and questionnaire responses in 40-60 min without any serious problem. 4.2 Results Figure 4 shows mean and standard deviation of task completion time and number of correctly executed tasks. In five of seven tasks (except for d and e), completion times on the map seem shorter than those on the table data when simply comparing with mean value. ANOVA shows significant difference in task c (t=2.67, P<0.02) and in task f (t=4.94, P<0.01). Task completion time （sec ） 80

Table Map

60

40

20

0 a

b

c

d

e

f

g Task

Correct

9

9

9

9

7

8

6

8

9

9

8

8

7

6

Not correct

0

0

0

0

2

1

3

1

0

0

1

1

2

3

Fig. 4. Task completion times and numbers of tasks correctly executed or not

A Usability Evaluation Method Applying AHP and Treemap Techniques

201

Figure 5 shows the result of subjective rating for seven questionnaires. As for the overall impression, participants (usability engineers) gave higher scores to table form for “simplicity,” to map form for “comfortableness,” and the same score for “easiness to understand.” As for the readability of check item weights, higher scores were given to Treemap. (As for the check item structure, the same scores were given.) Also, participants tended to feel the Treemap was more useful for both usability analysis and presentation tasks. 4.3 Discussion of Experimental Result From this experiment only, we could not reach a clear conclusion because the number of subjects was not sufficient for strict statistical analysis. One of the reasons was that participants were screened according to their experience as usability engineers. However, some implications or tendencies about the usefulness of the proposed method can be extracted. The experimental results indicate the Treemap presentation should: …be useful in data analysis tasks for usability consultation. We are paying attention to the result that significant task performance improvement was seen in tasks c and f. These tasks are to “select the most important check item (c)” and “pinpoint the check item for improving overall score most effectively.” In many cases, usability engineers need to examine heavily weighted check items prior to others for improving target system usability effectively and promptly. Tasks c and f were designed with the intention of checking the adaptability to this requirement. Effectiveness and usefulness of the Treemap method are highly expected since significant improvement was observed in both tasks. …not be suitable for examining low-weight check items. Although it was not statistically significant, performance time on tables was higher than that on Treemap (approximately 10% shorter mean time) in task d (select the lowest weight check item). When there are a lot of check items, sometimes weights are set smaller than 1/1000. In Treemap representation, where the weights are displayed as areas of rectangles, it is often hard to read out these areas exactly. In real situations, there are not many cases that require elaborate examination of low weight check items, and Treemap may not support such tasks sufficiently. (If engineers are accustomed to using the zooming function, the task completion time will be greatly improved, though.) …not have clear advantages in “rough examination” tasks. As for tasks a, b, e and g, significant difference in task completion time was not observed, contrary to the authors’ expectations. These tasks include roughly comparing target system characteristics, such as selecting viewpoints in which system A’s score is higher. In these tasks, participants did not have to search for items by comparing weighting values, and they could complete the task easily just by reading numerical values in the tables. …be useful by usability engineers. Participants gave higher subjective scores on Treemap presentation for two questionnaires about usefulness. Taking into consideration that every participant was a beginner in using Treemap, this indicates their expectations of Treemap are considerably high. More experience with Treemap and additional software functions that support usability analysis tasks will raise the subjective rating further.

202

T. Asahi, T. Ikegami, and S. Fukuzumi

Very easy to understand

Easy to understand

Average

Simple

Average

Confusing

Very confusing

Very comfortable

Comfortable

Average

Painful

Very painful

Very easy to understand

Easy to understand

Average

Hard to understand

Very hard to understand

Hard to understand

Very hard to understand

Overall impression Very simple

Structure of check items

Weights of check items

For usability analysis

For presentation to developers or clients

Very easy to understand

Easy to understand

Average

Hard to understand

Very hard to understand

Very useful

Useful

Average

Useless

Quite useless

Very useful

Useful

Average

Useless

Quite useless

Printed table

Treemap

Fig. 5. Result of subjective questionnaires

5 Concluding Remarks and Future Work In this report, a method applying the Treemap visualization to checklist-based heuristics was proposed. In some tasks that were intended to simulate usability analysis, improvement of task completion time was observed. Subjective ratings were higher than table form presentation on the usefulness for both data analysis and presentation tasks. Users’ (usability engineers’) experience and additional software functions will raise this score further. We think the following three issues should be overcome for making the checklist-based quantification method practical and widespread. 1. Provide theoretical and reliable bases to the quantification method. 2. Enable objective evaluation by eliminating/minimizing the score difference caused by individual skills or impressions. 3. Provide practical and useful tools for evaluation and analysis tasks.

A Usability Evaluation Method Applying AHP and Treemap Techniques

203

As for issue (2), Ikegami has claimed it will be achieved to some extent by designing check items and their terms elaborately and tuning them iteratively through experimental studies [4]. The proposed method was intended to contribute to resolving issue (3) and some effect was shown in the experimental studies. Of course, we will need many more functions and tools to support real usability analysis tasks. We are considering a tool for weight assignment by using the Treemap as a data input tool [11]. In order to create breakthroughs on issue (1), we need to present scientific bases of quantification, but it is hard to accomplish with short-term research. Although preceding studies [2][3][4][10] have tried to add reliability by adopting well-used guidelines or regulations, they have not become widespread as established methods. We think we need to ensure the validity of the method by developing or refining user cognitive/behavioral models for the checklist-based quantification as with the GOMS and KLM models for performance prediction with CogTool. Both scientific research and practical field activities should be merged harmoniously to develop reliable and systematic methodologies.

References 1. Hirasawa, N.: Usability Engineering for Software Development. IPSJ Magazine 44(2), 136–144 (2003) 2. Smith, S.L., Mosier, J.N.: A Design Evaluation Checklist for User-System Interface Software, Technical Report ESD-TR-84-358, MITRE, MA (1984) 3. Yamada, S., Tsuchiya, K.: A Study of Usability Evaluation of PC – Discussion of Model on PCs. The Japanese Journal of Ergonomics 32, 350–351 (1996) 4. Ikegame, T., Okada, H.: Toward Quantification of Usability. NEC Technical Journal 3(2), 53–56 (2008) 5. ISO 9241-10: Ergonomic requirements for office work with VDTs –Dialog principles (1996) 6. ISO 9241-12: Ergonomic requirements for office work with VDTs –Presentation of information (1998) 7. Ham, D.-H., Heo, J., Fossick, P., Wong, W., Park, S.-H., Song, C., Bradley, M.: Model-based approaches to quantifying the usability of mobile phones. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4551, pp. 288–297. Springer, Heidelberg (2007) 8. Nielsen, J.: Usability Engineering. Academic Press, London (1993) 9. Johnson, B., Shneiderman, B.: Tree-Maps: A Space-Filling Approach to the Visualization of Hierarchical Information Structures. In: Proc. of the 2nd Conference on Visualization 1991, pp. 284–291 (1991) 10. Kato, S., Horie, K., Ogawa, K., Kimura, S.: A Human Interface Design Checklist and Its Effectiveness. Transaction of Information Processing Society of Japan 36(1), 61–69 (1995) 11. Asahi, T., Turo, D., Shneiderman, B.: Using Treemaps to Visualize the Analytic Hierarchy Process. Information Systems Research 6(4), 357–375 (1995)

Evaluation of User-Interfaces for Mobile Application Development Environments Florence Balagtas-Fernandez and Heinrich Hussmann Media Informatics Group, Department of Computer Science, University of Munich, Germany {florence.balagtas,heinrich.hussmann}@ifi.lmu.de

Abstract. This paper discusses about the different user-interfaces of mobile development and modeling environments in order to extract important details in which the user-interfaces for such environments are designed. The goal of studying such environments is to come up with a simple interface which would help people with little or no experience in programming, develop their own mobile applications through modeling. The aim of this research is to find ways in order to present the user interface in a clear manner such that the balance between ease-of-use and ease of learning is achieved.

1 Introduction Nowadays, the development of software applications is no longer bounded within the confines of people with programming skills. People are no longer limited to just being end-users of an application, but are encouraged to be the creators of their own applications as well. An example of this is the growth of the world wide web and how the creation of web pages are no longer restricted to people who have skills in writing HTML code and scripts. The introduction of WYSIWYG HTML Editors such as Microsoft Front Page and Google Page Editor has made this possible. By hiding the HTML code in the background and allowing components to be dragged and dropped on to a page makes it easy for novices to create their own web pages. The same thing is happening now to the mobile industry. Mobile phone users are no longer limited to use pre-installed applications on their devices or buying readymade mobile applications for their personal purposes. People now have the power to create their own applications given the right motivation, creativity, skills and tools. Mobile phone companies and organizations have now opened up their application programming interfaces (APIs) that would allow anyone to develop their own applications for their mobile devices. Examples of these are the Java Platform Micro Edition (Java ME) API1 from Sun Microsystems, the Android API2 from the Open Handset Alliance and the iPhone API from Apple3. However, even though many users may have ideas for novel applications for mobile phones, software development is simply too difficult for most people. It takes a large amount of skill and familiarity with how 1

http://java.sun.com/javame/index.jsp http://code.google.com/android/ 3 http://developer.apple.com/iphone/ 2

J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 204–213, 2009. © Springer-Verlag Berlin Heidelberg 2009

Evaluation of User-Interfaces for Mobile Application Development Environments

205

the framework is used before a person can create a decent amount of code for a simple application. Even setting up the programming environment is a complex task, let alone, trying to figure out how to use the APIs, compiling, running and deploying the application on the actual device. Other things that make developing applications for mobile devices more difficult as compared to desktop applications are factors such as device limitations (e.g. screen size, computing power, power consumption) [4], different operating systems for mobile devices, different data representation and additional device capabilities (e.g. Bluetooth, Wifi, GPS, Camera-enabled) which are not standard to all devices and therefore should be considered when developing a uniform application that can be run on different mobile devices. In this research, we are investigating ways to make application development accessible to people with low or no programming skills. We propose applying modeldriven development (MDD) which is an approach to creating complex software systems by first creating a high-level, platform-independent model of the system, and then generating specific code based on the model to the target platform [5]. In ordinary software development, models are just thought of as tools for getting system requirements and for documentation purposes, however, in MDD, the models are actually part of the implementation of the system. The basic idea of our work is to come up with a modeling environment that is specific for modeling mobile applications which targets non-experts as the main users. Non-expert users here are defined to be people who have little or no experience in programming for mobile platforms. We want to present to the user one application that they could use to model their mobile applications without having to worry about low-level coding. In order to do this, we are developing a tool called Mobile Applications Modeler (Mobia). The focus of discussion for this paper is on the design of the user-interface for the Mobia modeling environment. The aim here is to find out which user interface design concepts are most suitable in order for non-expert users to develop their own mobile application with ease. The goal is to present the interface such that the balance between ease-ofuse and ease-of-learning [10] is achieved. We have focused on non-expert users in this research and do not include expert users in general since these two types of users often differ in their experiences and needs [6]. Unlike existing modeling tools such as Magic Draw4 and Eclipse with the Eclipse Modeling Framework (EMF)5 plugins that are more general purpose modeling tools, we want to present to the user a more domain-specific modeling tool that specializes only on modeling mobile applications. The focus of this part of our research is on how to present the user interface of the Mobia modeler such that it is easy to use for non-expert users. The rest of the paper is organized as follows: section 2 will discuss a few related researches to our work particularly in the area of model-driven development. In section 3 we will discuss the different user interfaces of existing development and modeling tools that are the basis of some of our designs. And then, the remainder of the paper will be a discussion of our approach such as, the design of our prototypes and evaluation results.

4 5

http://www.magicdraw.com/ http://www.eclipse.org/modeling

206

F. Balagtas-Fernandez and H. Hussmann

2 Related Work Integrated development environments (IDEs) are tools which are made to ease the application development process. Most IDEs provide an environment that features a text editor, compiler, debugger and simulator to name a few, which are all integrated into one application. They have evolved throughout the years, adding more features (e.g. GUI designer, version control, etc.) that would help the developer in accomplishing their tasks in the most efficient way. For mobile applications development in particular, examples of IDEs that allows plugins for mobile application development are the Netbeans, Eclipse and XCode development environments. A problem with IDEs for mobile application development though is that, different mobile phones have different application programming interfaces and platforms. Thus, creating a common application that would run on different platforms of mobile phones tends to get tedious and redundant since developers have to develop different code for each of them. One solution to this problem is by applying the model-driven approach in which models are used to describe the application and through transformation tools, these models are transformed to code that would run on specific platforms [5]. An example research that applies MDD is the Multimedia Modeling Language (MML) which is a platform-independent language used for the model-driven development of multimedia applications [7]. MML models are transformed into Flash models which can then later on be loaded into the Flash authoring tool for further completion of the application [8]. The approach [7,8] is usually for teams wherein graphical designers and software designers need to work together in a certain project. Each group of users has their own expertise in terms of skills and tools they are using. However, when non-expert users are involved in the development process, this approach can be quite complicated. Extensive knowledge on how to make UML models is necessary in order to create the applications using this approach, and since the tools are not yet integrated, mastery in using these tools is a must [8]. Dunkel and Bruns [3] also presents a model-driven way of producing business applications for mobile devices with BAMOS (Base Architecture for Mobile Applications in Spontaneous networks) as the target platform. Their models are expressed in UML activity diagrams to specify control flow and the description of mobile services through a DSL they have defined using UML profiles. As with MML, the approach uses different tools which are not yet integrated [3]. Another research that applies the MDD process and targets non-experts as the primary developers is the Simple Mobile Services (SMS) project. This project aims to create service authoring tools and mobile services that are simple to use, find and set up[9]. They focus on non-expert users as the people assembling these mobile services on their own. SMS applies the MDA [5] approach in building their services [2]. Our approach is similar to SMS in a way that we target non-expert users as main users of our tool for developing mobile applications. However, while SMS focuses on mobile web-based services, our research focuses on mobile-based applications. In the next section, we will discuss the different user-interface components present in various development and modeling environments. We want to find out which existing approaches in the UI and some new ones are most suitable for non-experts.

Evaluation of User-Interfaces for Mobile Application Development Environments

207

3 A Closer Look into the User Interfaces In this section, we would like to compare user-interfaces of existing IDEs that supports mobile application development (Netbeans and Eclipse), and a modeling tool (MetaEdit+) that supports domain specific modeling of mobile applications. We want to explore what features these tools have, and which of these features are essential parts of an environment in which non-experts can benefit in it.

Fig. 1. General Parts of a Development/Modeling Environment

In studying these tools, we have identified four basic areas that are usually present in such environments. For the purpose of discussion in this paper, we will attach a general name to each of the areas, which may be identical or not to how they are labeled in such environments. Fig. 1. shows the typical default location of the main areas and their names. Table 1 contains the main areas and some of the possible contents that may appear in those areas. Table 1. The different areas and their possible contents Area Navigation/Browsing Area Main/Central Area Palette/Properties Area Toolbar Area Output Area

Possible Contents Different components in a certain development project (e.g. files and folders, classes and packages) The component in which the user is currently working on (e.g. source code, design for a user interface, data source) Components that can be dragged and dropped to the main/central area (e.g. UI components, Datasets) Button controls (e.g run, debug), editing controls (e.g. copy, paste) Program output, Compiler errors, Debugging messages, etc.

Shown in Fig. 2. is an overview of the Netbeans 6.5 environment. The components described in our general UI model for a development environment are present in the Netbeans environment. One additional feature of Netbeans is the ability to switch to different views in the main area depending on what the user is focusing on. The source view allows the user to make changes to the source code; the screen view allows drag and drop design of the mobile application’s user interface; the flow view allows adding logic to the program by dragging flow arrows between the different

208

F. Balagtas-Fernandez and H. Hussmann

screens; and the analyzer view shows unused resources and MIDP compliancy. Switching through the different views changes the contents of the palette area, depending on what components are needed in that certain view. Netbeans also has the ability to bind a screen component’s data to information taken from a database.

Fig. 2. Overview of the Netbeans Environment

The next IDE interface that we are going to discuss is the Eclipse IDE. There are several projects that aim to develop plugins for Eclipse to allow mobile applications development (e.g. EclipseME6, Eclipse Plugin for Android7). For the purpose of this paper, we will focus on analyzing the interface for the Eclipse IDE used for developing Android applications since the basic components of the IDE are similar anyway. As shown in Fig. 3, the positioning of the components in the environment are similar to that of Netbeans. However, the features offered by the Eclipse environment are just a subset of Netbeans. As of the moment this paper is written, it does not feature a drag-and-drop GUI environment for developing Android applications, but through editing an XML file for the placement of the GUI components on the screen, or by directly adding lines to the source code for the GUI. Although, as expected in the future, developers might add more features for easy GUI development as the platform matures. DroidDraw8 is one example application UI editor for the Android platform which generates an XML file that can be copied to the main code. The MetaEdit+ Modeler is a DSM tool that allows the modeling of different domain specific applications (e.g. Mobile, Automotive, Telecom, Embedded, etc.). One supported domain is for modeling smartphone applications. Fig. 4 shows an overview of the basic user-interface of MetaEdit+ for modeling mobile applications. Unlike the first two IDEs we have described, the MetaEdit modeler features a simpler interface with several of its components positioned at different areas. The palette area contains fewer number of components as compared to the first two tools discussed. It features specialized constructs which is specific for such mobile platforms. The navigation area contains a list of components in the model. Below the navigation area is the properties area, which contains information about the current component in focus. 6

http://eclipseme.org http://code.google.com/android/intro/installing.html 8 http://www.droiddraw.org/ 7

Evaluation of User-Interfaces for Mobile Application Development Environments

209

Fig. 3. Overview of the Eclipse Environment

From these three examples, we want to extract the most desirable feature of each that can be applied to the design of Mobia. The Netbeans environment for instance, features the ability to change to different views, which can allow the user to concentrate on one task at a time. However, it contains so much features that it can take awhile before the user can actually take advantage of such features. The MetaEdit tool on the other hand contains only a limited number of components. It contains specialized constructs that could easily be identified by the user. All the tasks such as designing the screen and adding flow to the program is modeled in one view. The disadvantage about this though, is that since it is very specialized, the user is restricted to the type of application they can create. The Eclipse environment also offers a very simple interface and not shows too much features. It is clearly a tool for expert developers, who basically know what source code to type in for the applications they are developing. In the next section, we will discuss more about our approach in finding the ideal user-interface for the Mobia Modeler such that non-expert users will be able to use it. We apply some design patterns that is shown in the previous tools that we have described, and try to evaluate it in order to find out which features are most desirable for such an environment.

Fig. 4. Overview of the MetaEdit+ Modeler

210

F. Balagtas-Fernandez and H. Hussmann

4 The Mobia Modeler User Interface The Mobia Modeler is a modeling tool specifically designed to allow modeling for mobile applications. The target users for Mobia are non-expert users who are people that have little or no experience at all in programming for mobile platforms. For this particular study, we feature a module of Mobia that is focused on modeling applications in the domain of mobile health monitoring. As of the moment, we want to focus on one type of domain, since different domains may offer different modeling constructs. For this module, the users will model a certain type of application that can be used for health monitoring and feature modeling constructs that represent data from different medical gadgets or medgets (e.g. ECG meter, Thermometer, etc.). In order to find the ideal interface for Mobia, we have created two prototypes using Flash, which offers two different UI designs. Just to clarify, these prototypes are simply focused on evaluating the different user interface designs and interactions and do not yet have code transformation features. 4.1 Mobia with One View Shown in Fig. 5 is a screenshot of the first version of our Mobia prototype which we called Mobia One View. The first prototype offers one view for the user which means that, the user can design an instance of the mobile screens, add data and application control flow all in one view. The user can concentrate on designing a single screen by zooming into that area, and try to see an overview of the whole system by zooming out. The palette on the right side of the screen contains screen components that can be dragged on to the mobile screen. For our prototypes, we only feature a subset of the possible screen components that a mobile application can have. The right palette also contains data input which we call medget (short for medical gadget) input. The medget constructs in the medget input palette contains abstract representations of information that comes from health monitoring devices capable of sending their data to a mobile device. More information about the different representations of medget data will not be discussed in this paper, but in a separate paper [1].

Fig. 5. Mobia with One View. (In the foreground) The main area is zoomed-in to see the screen designs better.

Evaluation of User-Interfaces for Mobile Application Development Environments

211

4.2 Mobia with Multiple Views The second design approach that we did for Mobia is what we called multiple views which is shown in Fig. 6. This is similar to the Netbeans IDE in which the main area features different kinds of views depending on the specific task that the user is doing. The reason behind the choice of this type of design is that we want the user to focus on one task at a time.

Fig. 6. Mobia with Multiple Views (Design, Data and Navigation View)

The default view is the Design View in which the user can design individual screens by dragging and dropping screen components from the palette to the screen. The left panel contains all the mobile screens for that application. Clicking on an individual screen in the left panel shows it in the main view to be further edited and designed. Screens can be added and deleted by pressing the add and delete buttons respectively. This design is borrowed from presentation programs such as Microsoft Powerpoint and OpenOffice Impress wherein each slide can be viewed from a panel and allows switching from one slide to another by clicking on the mini versions of the slides in the panel. The Data View is similar to the design view except for the fact that the palette contents on the right panel changes to medget data. In this view, the users can concentrate on how they want data taken from health monitoring devices be displayed on the screen. These medget components act as placeholders into which the real information from the devices will appear in the real application. The last view is the Flow View, which shows all the screens in the model and how the screen transitions from one to the next. The user can add basic control logic to the application by dragging on arrows and linking the screens together. In this view, a small component palette contains buttons in which the users can drag to the screens. The logic behind this design approach is that, in the application, only by pressing a control component such as a button can trigger going from one screen to the next.

5 User Study Evaluation and Results Given the prototypes that we have described in the previous section, we want to find out which of the prototypes provides a simpler UI for the user and gets the task done

212

F. Balagtas-Fernandez and H. Hussmann

quickly. For a more subjective evaluation, we also want to find out which design is more fun and easier to use. In order to do this, we have conducted a user study in which each user is given a task to accomplish using both prototypes. In order to measure efficiency, we get the time in which the user accomplishes a certain task. In order to eliminate the bias towards the second prototype, we alternate which prototype each participant uses first in doing the tasks. The participants were instructed not to ask any questions from the evaluators. The goal here is to allow the participants to explore the tool themselves and learn how to use it by themselves without any outside intervention. At the beginning of the user-study, the participant was asked to explore the prototypes and give comments. After they are done studying the tool in whichever method they choose, they are given two tasks which are to design the contents of the screens and then later on to add control flow to the screens. They were asked to create 3 screens with some screen components in them. After designing the screens, they were asked to add control flow in which allows switching to a different screen whenever a button is pressed. Table 2. The average times accomplishing the tasks using the two prototypes

Version Mobia One View Mobia Multiple Views

Average Time (Minutes) Screen Design Adding Control Task Flow Task 4.036 minutes 1.126 minutes 5.833 minutes 2.223 minutes

There were 10 participants to our user study: 60% have backgrounds in the field of Computer Science and the rest from the fields of Educational Psychology, Archeology, Architecture and Social Welfare. Only 10% of the participants have a background in programming for mobile platforms which is also in the very basic level. Table 2 shows the average times of when the users accomplished the tasks while, Table 3 shows the results for the subjective evaluation in terms of which prototype is easier and more fun to use. Based on the results shown in the tables, Mobia One View allows the users to do the tasks faster as compared to the multiple view. A factor that might contribute to this is the fact that in multiple views, the user has to switch from one to the next in order to add a different component or do another design task. Based on the subjective feedback of the participants, the Mobile one view poses an environment that is both easy and fun to use. Table 3. Subjective evaluation for the Mobia Prototypes Version Mobia One View Mobia Multiple Views None

Criteria (Percentage of Users) Easier to Use More fun to Use 60% 50% 40% 40% 0% 10%

Evaluation of User-Interfaces for Mobile Application Development Environments

213

6 Summary and Future Work In this paper, we have presented different design ideas for a mobile application modeling environment that targets non-experts as the main users. The design and results presented here are just the initial phase of our iterative approach to finding the ideal interface for a tool that would help accomplish tasks with ease. Our future work aside from continuing to polish the user interface design of Mobia, is to come up with an underlying framework to support code transformation from the models. An approach to have a user-adaptive tool that changes according to each user’s existing skills and preferences to enhance user experience and learning is also envisioned.

Acknowledgments We would like to thank the German Academic Exchange Service (DAAD) for funding this research. We would also like to thank Ugur Örgün for helping with the prototypes and to the people who participated in our user study.

References 1. Balagtas-Fernandez, F., Hussmann, H.: Modeling Information From Wearable Sensors. In: MDDAUI 2009- Model Driven Development of Advanced User Interfaces 2009. CEUR Proceedings (2009) 2. Bartolomeo, G., Casalicchio, E., Salsano, S., Melazzi, N.B.: Design and Development Tools for Next Generation Mobile Services. In: International Conference on Software Engineering Advances, ICSEA 2007, p. 16 (2007) 3. Dunkel, J., Bruns, R.: Model-Driven Architecture for Mobile Applications. Business Information Systems, pp. 464–477 (2007) 4. Gaedke, M., Beigl, M., Gellersen, H.-W., Segor, C.: Web Content Delivery to Heterogeneous Mobile Platforms. In: ER 1998: Proceedings of the Workshops on Data Warehousing and Data Mining, pp. 205–217. Springer, London (1998) 5. Kleppe, A., Warmer, J., Bast, W.: MDA Explained: The Model Driven Architecture: Practice and Promise. Pearson Education, Inc., Boston (2003) 6. Petre, M.: Why looking isn’t always seeing: readership skills and graphical programming. Commun. ACM 38, 33–44 (1995) 7. Pleuss, A.: MML: A Language for Modeling Interactive Multimedia Applications. In: ISM 2005: Proceedings of the Seventh IEEE International Symposium on Multimedia, pp. 465– 473. IEEE Computer Society, Washington, DC, USA (2005) 8. Pleuß, A., Vitzthum, A., Hussmann, H.: Integrating heterogeneous tools into model-centric development of interactive applications. In: Engels, G., Opdyke, B., Schmidt, D.C., Weil, F. (eds.) MODELS 2007. LNCS, vol. 4735, pp. 241–255. Springer, Heidelberg (2007) 9. The SMS Project, http://www.ist-sms.org 10. Weiss, S.: Handheld Usability. John Wiley and Sons, Chichester (2002)

User-Centered Design and Evaluation – The Big Picture Victoria Bellotti1, Shin’ichi Fukuzumi2, Toshiyuki Asahi2, and Shunsuke Suzuki2 1

Palo Alto Research Center, 3333 Coyote Hill Road, Palo Alto, CA 94304, USA [email protected] 2 NEC Corporation, 8916-47, Takayama-cho, Ikoma, Nara 630-0101, Japan [email protected], [email protected], [email protected]

Abstract. This paper provides a high-level overview of the field of usability evaluation as context for a panel “Systematization, Modeling and Quantitative Evaluation of Human Interface” in which several authors report on a collaborative effort to apply CogTool, an automated usability evaluation method, to mobile phone interfaces and to assess whether usability predictions made by CogTool correlate with user subjective impressions of usability. If the endeavor, which is still underway at the time of writing, is successful, then CogTool may be applied economically within the product development lifecycle to reduce the risk of usability problems. Keywords: Usability evaluation, methods, metrics, systematization.

1 Introduction User-centered design and usability evaluation go hand-in-hand. It is generally accepted amongst members of the HCI community that you cannot guarantee a successful user experience without testing it in some way [15], the earlier in the process, the better [34]. Indeed, it has even been argued, perhaps unfairly, that user-centered design is often little more than trial and error, guided by checks with users in evaluations [14]. Iterative design, prototyping and usability evaluation certainly ranked very high amongst all methods cited in two surveys of HCI professionals [20, 48]. Early, rapid and low-fidelity prototyping approaches (e.g., paper or look-and-feel mock-ups) allow ideas to be evaluated for usability with real users and then refined and tested again before design decisions are finalized and high-fidelity software prototyping affords even more thorough usability evaluation during the design process. Evaluation performed during the design process is known as formative evaluation and can be contrasted with summative evaluation that can be used to assess the final merits of a system at the end of the design process [44]. Formative evaluation is useful for improving a design and summative evaluation for supporting claims to its efficacy and developing requirements for a future release of the design product. However, regardless of when and why evaluation is required, there is some considerable debate as to what usability evaluation methods are most effective (e.g. [18, 21 and 49]). This paper briefly discusses the big picture of some of the issues that can influence the choice of a usability evaluation method within the user-centered design process. It J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 214–223, 2009. © Springer-Verlag Berlin Heidelberg 2009

User-Centered Design and Evaluation – The Big Picture

215

is presented as context for a series of position papers within a panel that reflects on one particular effort to systematize usability evaluation in a large corporation where a number of fairly common constraints such as lack of availability of user-centered design experts and tight product deadlines and budgets apply.

2 Challenges for Usability Evaluation in Design Despite the obvious importance of evaluation in user-centered design (obvious at least to our own HCI community), it is not always the case that applications built to be placed in the hands of hapless end-users enjoy the benefits of any kind of objective evaluation; i.e., a method relying on more powerful evidence than the intuitions of inexperienced engineers [see, for example, 6]. And HCI researchers have expressed concern that the evaluations that do take place are inadequate (for example, they may involve the wrong user representatives [19]; or only quality control testers [41]). In this section I review three obstacles that may stand as explanations for this unfortunate phenomenon. 2.1 A Plethora of Methods to Choose From One key obstacle to understanding what usability evaluation methods one should adopt is the abundant diversity of methods and tools starting with “quick-and-dirty” methods such as expert reviews or guidelines walkthroughs [18] and simple tools such as surveys [e.g. 42] all the way through to extended in situ evaluation methods incorporating multiple data sources such as logging and interview-based data collection methods [e.g. 33] or sophisticated tools such as eye-tracking systems [e.g. 11]. Each method has its strengths and drawbacks and is appropriate in different circumstances, for example discount usability methods [36] are appropriate when resources are constrained, even if they are not as sensitive at picking up problems as a full-scale evaluation. If there were a one-size-fits-all solution available for standardized usability evaluation, it would surely be easier to train designers and developers to apply it in all projects that are likely to impact end users. But instead diversity opens the doorway to confusion and suboptimal choice. 2.2 A Diversity of Influential Design Circumstances At the time of writing in 2009, based on my own experiences interacting with representatives of a variety of commercial and research application development organizations, it is still not uncommon for a design effort to take place without a serious usability evaluation. We are all familiar with the baffling results of such endeavors, which we encounter regularly in our interactions with hardware, software and webbased user interfaces. Many circumstances can exert influence over whether usability evaluation takes place at all and over what type of evaluation with what metrics is most appropriate. Consider the following variables (which are both contextual and inherent to the design) as examples: • •

Application domain Standards and performance criteria that pertain to the application domain

216

• • • • • •

V. Bellotti et al.

Target users and their particular characteristics Novelty of the design and its interaction elements User-centered design expertise within the design team Budget Time available Organizational culture around the design team and perceptions of the importance of usability

Let me briefly illustrate the kinds of impact these factors can have. In my own past work exploring novel solutions in the application domain of personal information management (PIM) [8, 9 and 10], it quickly became apparent that experimental evaluations made no sense, since the proof of the PIM pudding so-to-speak can only be in the extended use of a solution with one’s own real personal information, leading to a need for in situ evaluation of real use over weeks rather than the hours that Nielsen [38] suggests can usefully be applied to web-site evaluation. As another example, Grudin [19] reported extensively on various organizational related constraints that can lead to suboptimal design results and Bak et al. [6] more recently also highlight organizational obstacles as significant, both in the literature and in their own survey, together with developer mindset (a culture of greater focus on functionality and efficient code and lack of user-centered design expertise). Perhaps the key factor impacting usability evaluation is the overall culture of the host organization for the design effort, (or even the culture within which that organization exists) which can in turn influence other factors that I listed above. Specifically, if few members of the organization are aware of user-centered design as a discipline and the value of a good user experience and fewer still have the relevant skills to apply the appropriate methods, then budget and time will not be allocated to a serious effort to evaluate the user experience and people with the required expertise will of course not be available to engage in that effort. In many countries today usability experts (or engineers who also have user-centered design skills) are indeed still an extremely rare species and, even if a corporation wishes to hire them, they may find that they simply cannot find them. In such circumstances, how might a large corporation make the best of limited usability expertise? This is an issue to which I will return in a subsequent section. 2.3 Metrics Usability is defined in the ISO 9241-11:1998 standard as the “extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” Taking this standard as widely agreed upon, effectiveness, efficiency and satisfaction thus have to be measured somehow in order to know how usable a prototype or product is. Unfortunately metrics also present something of a challenge to evaluators since, as Sauro & Kindlund [43] point out without overlooking the obvious irony, “Usability Metrics Need to be Easier to Use.” Bak et al. [6] surveyed 2795 papers in the HCI literature, amongst which they found 28 with a focus on usability in organizations. Out of these, 11

User-Centered Design and Evaluation – The Big Picture

217

mentioned poor understanding of usability as an obstacle (behind resource demands, 17/28; test participant issues such as identification and access to users, 14/28; and organizational obstacles including anti-usability culture, 14/28). In particular, according to Bak et al., usability is often confused with functionality, which, at least to this author’s mind, may explain why so many applications have so many unused, often hard to discover, and near useless features. In fact there are five basic (although not cleanly independent) usability metrics that have been applied repeatedly and seem to be accepted as fairly standard within the HCI community [e.g. 32, 34 and 43]. These are: • • • • •

Time taken to learn to execute tasks Time taken to execute tasks Task completion rate (proportion of tasks in an evaluation that can be completed to some standard of correctness) Number of errors (deviations from viable task completion paths or production of a result or state that must be undone) User satisfaction (a composite of a variable host of subjective assessments)

Other less common metrics such as objectively measurable stress [e.g. 45] and analytically derived cognitive complexity [27] that can be correlated with at least one of these four have also been discussed in the literature. However, the five basic metrics can be all measured directly without special equipment (although perhaps not always as accurately as with special equipment) and cover the most significant possible consequences of bad design. The question then is, should all of these dimensions be measured or do some matter more than others in different design circumstances? In fact, design circumstances can heavily weight the importance of one metric over another and may even require trade-offs to be made between metrics. For example, a UI optimized for novice users with lots of easy to find and learn menus and buttons will tend to be slower and less efficient for an expert who will usually look for keyboard shortcuts that are faster to execute. So the evaluator must understand the importance of the metric to the design circumstances at hand and the extent to which any given tool or method is likely to provide reliable values for the metrics that matter. Different usability evaluation methods vary in the extent to which they are able to provide these metrics. For example, a cognitive walkthrough [40] will not allow the evaluator to measure task times very accurately, although it may be better at measuring the extent to which a system is likely to be error-prone. A laboratory experiment may allow an evaluator to measure time, task completion and errors quite accurately, but render no reliable measurement of user satisfaction. A user survey may measure satisfaction quite well, but provide only subjective (and thus unreliable) appraisals of time, task completion and error proneness. Of course, it can make sense to combine multiple methods in one evaluation such as an experiment and a survey, usually at a reduced cost for each method, since study participants need only be recruited, scheduled and paid once for a single session to perform more than one exercise. Whatever the case may be, it requires some expertise to know what aspects of the design situation to pay attention to in deciding what evaluation metrics are best.

218

V. Bellotti et al.

3 Systematizing Usability Evaluation Given the above challenges for usability evaluation in the design process, it is hardly surprising that some professionals have sought to develop systematic methods in an attempt to help those with less expertise assess usability without the added time and expense of bringing in real users or an expert who may be hard to find. Three approaches to systematization are: • • •

Guidelines Procedures Automation

Usability guidelines have been common for quite some time. For example, Jakob Nielsen describes participating in a US Airforce exercise to compile “existing usability knowledge into a single, well-organized set of guidelines for its user interface designers” between 1984 and 1986 [37]. Many corporate, governmental and quite a few international user interface design guidelines have been compiled and updated since then [e.g. 5, 24, and 28]. They seek to describe what the designer must aim to accomplish or constraints she or he must work within. However conforming to guidelines can be a tricky business for the unskilled, especially when they are not well articulated as in the ISO guideline for “Suitability for learning” which is articulated thus, “A dialogue is suitable for learning when it supports and guides the user in learning to use the system.” Procedures is a term I want to use here to refer to well-defined methods for usability evaluation and count as a subset of the methods described in section 2.1 of this paper. The Cognitive Walkthrough [40] is one such procedure that has been evaluated [31] to show that it can be followed by a knowledgeable person and, depending on the extent of that person’s skill, produce consistent predictions without requiring a complex modeling effort or a real user evaluation. Another similar approach is the Heuristic Evaluation method [39], which, in evaluation, has been shown to be better performed by usability experts and best of all by application domain specialist usability experts [e.g. 34]. These general-purpose methods have also been accepted for a long time and have stood the test of time, still being in use even over 15 years after their invention [22]. Earthy at al. [17] provides a review of the ISO 13407 humancentred design processes which represents an attempt to set standards for interactive systems design in general. Other evaluation procedures have been developed more recently for specific platforms (e.g. the mobile phone [30]) and specific application domains (e.g. e-learning [50]). Automation may be the Holy Grail of usability evaluation since the possible cost savings in design endeavors are immense. Card, Moran & Newell developed the foundational example of an evaluative human information processing model and the GOMS (Goals, Operators, Methods and Selection rules) approach to computational modeling of human interaction with computers in the early 80’s [13]. Since then, many attempts have been made to achieve full automation [25]. Quite a number of early efforts to develop approaches only focused on specifying the rules that would need to be learned by a user to operate a system and were never automated and took far too long to apply successfully [7]. More successful has been the work based on sustained development of working software models of a human information processor

User-Centered Design and Evaluation – The Big Picture

219

such as ACT [1] and its descendents (ACT* [2], ACT-R [3 and 4] and ACT-R/PM, [12], which is still under development at Carnegie Mellon University (CMU). Another, albeit less well-known, example of such a system is the SOAR cognitive architecture [29] developed in the UK. Building upon the ACT-R computational cognitive architecture Bonnie John at CMU and her colleagues and students have developed a GOMS-based system called CogTool [26] that has perhaps come closest to achieving minimal effort from the system developer. CogTool uses performance measurements taken from real user interactions and is able to generalize them to specifications of user interfaces that contain the same basic features (e.g. buttons, menus and other GUI elements). The user interface specification is provided to CogTool in the form of a storyboard (based on sketches or screenshots) that preserves the dimensions of the target GUI upon which the evaluator demonstrates to CogTool the actions required to execute tasks. Using its models of user thinking times and actions (plus expected system response times), CogTool is able to output a metric of task completion time predictions for a skilled user and also completion times and deviating actions together with the time they will take for novices. CogTool uses an augmentation of ACT called SNIF-ACT [17] which assumes novice users read text labels and click on items that are semantically close to their goal, sometimes this will lead to mistakes since interfaces often contain ambiguous or misleading elements [46 and 47].

4 Seeking Systematization in the Enterprise The HCI International 2009 panel, “Systematization, Modeling and Quantitative Evaluation of Human Interface” with which this paper is associated includes a number of positions from collaborators who have participated in an effort to systematize usability evaluation in a large corporation, NEC, based in Japan, that frequently develops software and hardware products for both consumers and for business use. The discussion in this paper has sought to provide some context for the approach adopted in the work reported, by addressing some of the key considerations that relate to its rationale. The chosen approach reflects a desire to simplify the choice of usability evaluation method in NEC where usability expertise is not as pervasive as would be ideal and where tight deadlines and budgets always apply. A small team of usability specialists in the research division of NEC began a collaboration effort with researchers at the Palo Alto Research Center and at Carnegie Mellon University to validate the use of CogTool (introduced above) in assessing the usability of products under development. CogTool was chosen as an ideal method because of its being easy to apply to a graphical UI specification (possibly early in the design process) without much expertise. This approach sidesteps the problems experienced by the corporation of lack of expertise and limited time and budget for usability. By providing one general-purpose systematic approach, it also seeks to get around possible bewilderment by non-experts working at the corporation at the number of methods available and the reliance of many of the more suitable, under the circumstances, economical and fast discount usability methods (such as Heuristic Evaluation) on experts to apply them well. Because of their importance to the corporation as a product category and the discrete, and thus easy to model, nature of the tasks users perform on them, mobile

220

V. Bellotti et al.

phones were chosen as having an ideal user interface upon which to first test the CogTool approach. However, CogTool at the time of writing mainly generates predictions of the occurrence of user behaviors such as looking, thinking and gesturing and the time they each take during task execution (which may include trial-and-error exploration for novice users). So it was necessary to determine whether these predictions correlate with the kind of usability that really matters to a product vendor developing applications where user performance demands (i.e. time to execute tasks and error rates) are not stringent; the subjective experiences of both experts and novices. Experts, of course, will form this opinion over extended periods of use, but novices can only form this opinion based on hearsay from existing users (probably experts) or their own initial exploration of the product, perhaps in a store (also known as “shelf usability”) or when given the opportunity to try the product for the first time by a friend or colleague. A literature search was undertaken to find the best possible method for assessing user subjective impressions of usability. Out of many possible candidates, the Mobile Phone Usability Questionnaire developed by Ryu [42] was chosen because it built upon and systematically refined questions from a number of previously well-accepted subjective usability assessment questionnaires. At the time of writing, an intensive effort is underway to obtain naïve and expert user performance data on a set of tasks across three types of mobile phone interface (between subject measures) and to correlate that data with user subjective impressions of usability obtained using the MPUQ both before and after using the mobile phones. The anticipated outcomes of the research will be: • • •

A comparison of three mobile phone models in terms of time taken by expert and naïve users to complete a set of tasks on each of the three phones. CogTool predictions for both experts and naïve users on each of the phone interfaces and an assessment of the accuracy of those predictions. A comparison between the real user data and the CogTool predictions and user subjective impressions of usability.

If the team is able to demonstrate that predictions of CogTool do indeed correlate with user subjective impressions, then we have evidence that CogTool may be used by commercial mobile phone developers to improve the usability of their mobile phones by using its predictions as a means to identify usability problems and areas for improvement in new phone interface design efforts. Whilst this might not be the ideal method for formative usability evaluation in the product development lifecycle, it will be a practical solution that can be used by non-experts under non-ideal circumstances and should reduce instances of at least some types of usability problem going undetected before product release.

References 1. Anderson, J.R.: Language, Memory and Thought. Erlbaum Associates, Hillsdale, NJ (1976) 2. Anderson, J.R.: The Architecture of Cognition. Harvard University Press, Cambridge, MA (1983)

User-Centered Design and Evaluation – The Big Picture

221

3. Anderson, J.R.: Rules of the Mind. Lawrence Erlbaum Associates, Hillsdale (1993) 4. Anderson, J.R.: ACT: A Simple Theory of Complex Cognition. American Psychologist 51, 355–365 (1996) 5. Apple: Apple Human Interface Guidelines for Mac OS X (2008), http://developer.apple.com/documentation/UserExperience/ Conceptual/AppleHIGuidelines 6. Bak, J.O., Nguyen, K., Risgaard, P., Stage, J.: Obstacles to Usability Evaluation in Practice: A Survey of Software Development Organizations. In: Proceedings of the 5th Nordic Conference on Human-Computer interaction: Building Bridges, NordiCHI 2008, vol. 358, pp. 23–32. ACM, New York (2008) 7. Bellotti, V.: Implications of Current Design Practice for the Use of HCI Techniques. In: Jones, D.M., Winder, R. (eds.) People and Computers IV, pp. 13–34. Cambridge University Press, Cambridge (1988) 8. Bellotti, V., Dalal, B., Good, N., Bobrow, D.G., Ducheneaut, N.: What a To-do: Studies of Task Management Towards the Design of a Personal Task List Manager. In: ACM Conference on Human Factors in Computing Systems, CHI 2004, pp. 735–742. ACM, New York (2004) 9. Bellotti, V., Ducheneaut, N., Howard, M.A., Smith, I.E.: Taking Email to Task: The Design and Evaluation of a Task Management Centered Email Tool. In: CSCW 2002 Workshop: Redesigning Email for the 21st Century, New Orleans, LA, ACM, New York (2003) 10. Bellotti, V., Smith, I.: Informing the Design of an Information Management System with Iterative Fieldwork. In: Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, ACM, New York (2000) 11. Benel, D.C.R., Ottens, D., Horst, R.: Use of an Eye Tracking System in the Usability Laboratory. In: Proceedings of the Human Factors Society 35th Annual Meeting, Santa Monica, Human Factors and Ergonomics Society, pp. 461–465 (1991) 12. Byrne, M.D.: ACT-R/PM and Menu Selection: Applying a Cognitive Architecture to HCI. International Journal of Human-Computer Studies 55, 41–84 (1999) 13. Card, S.K., Newell, A., Moran, T.P.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates Inc., Hillsdale (1983) 14. Constantine, L.L.: Beyond User-Centered Design and User Experience: Designing for User Performance. Cutter IT Journal 17(2), 16–25 (2004) 15. Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction, 2nd edn. PrenticeHall, Englewood Cliffs (1993) 16. Earthy, J., Sherwood Jones, B., Bevan, N.: The Improvement of Human-centred Processes – Facing the Challenge and Reaping the Benefit of ISO 13407. International Journal of Human Computer Studies 55(4), 553–585 (2001) 17. Fu, W.-T., Pirolli, P.: SNIF-ACT: A Cognitive Model of User Navigation on the World Wide Web. Human-Computer Interaction 22, 355–412 (2007) 18. Gray, W.D., Salzman, M.C.: Damaged Merchandise? A Review of Experiments that Compare Usability Evaluation Methods. Human Computer Interaction 13(3), 203–261 (1998) 19. Grudin, J.: Systematic Sources of Suboptimal Interface Design in Large Product Development Organizations. Human-Computer Interaction 6(2), 147–196 (1991) 20. Gunther, R., Janis, J., Butler, S.: The UCD Decision Matrix: How, When, and Where to Sell User-Centered Design into the Development Cycle (2001), http://www.ovostudios.com/upa2001/ (retrieved January 24, 2009) 21. Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria for Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction 15(1), 145–181 (2003)

222

V. Bellotti et al.

22. Hollingsed, T., Novick, D.G.: Usability Inspection Methods After 15 Years of Research and Practice. In: Proceedings of the 25th Annual ACM international Conference on Design of Communication, SIGDOC 2007, pp. 249–255. ACM, New York (2007) 23. ISO 9241-11:1998. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) – Part 11: Guidance on Usability. International Organization for Standardization (1998) 24. ISO 9241-110:2006 Ergonomics of Human-System Interaction – Part 110: Dialogue Principles (2006) 25. Ivory, M.Y., Hearst, M.A.: The State of the Art in Automating Usability Evaluation of User Interfaces. ACM Computing Surveys 33(4), 470–516 (2001) 26. John, B.E., Prevas, K., Salvucci, D.D., Koedinger, K.: Predictive Human Performance Modeling Made Easy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2004, pp. 455–462. ACM, New York (2004) 27. Kieras, D., Polson, P.: An Approach to the Formal Analysis of User Complexity. Int. Journ. of Man-Machine Studies 22, 365–394 (1985) 28. Koyani, S.J., Bailey, R.W., Nall, J.R.: Research-Based Web Design & Usability Guidelines. U.S. Department of Health and Human Services (HHS) and the U.S. General Services Administration (GSA), Washington DC (2004), http://www.usability.gov/pdfs/guidelines.html#1 (retrieved February 21, 2009) 29. Laird, J., Rosenbloom, P., Newell, A.: Universal Subgoaling and Chunking: the Automatic Generation and Learning of Goal Hierarchies. Kluwer Academic Publishers, Dordrecht (1986) 30. Lee, Y.S., Hong, S.W., Smith-Jackson, T.L., Nussbaum, M.A., Tomioka, K.: Systematic Evaluation Methodology for Cell Phone User Interfaces. Interacting with Computers 18(2), 304–325 (2006) 31. Lewis, C., Polson, P.G., Wharton, C., Rieman, J.: Testing a Walkthrough Methodology for Theory-Based Design of Walk-up-and-use Interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 1990, pp. 235–242. ACM, New York (1990) 32. Macleod, M., Bowden, R., Bevan, N., Curson, I.: The MUSiC performance measurement method. Behaviour & Information Technology 16(4-5), 279–293 (1997) 33. Muller, M.J., Geyer, W., Brownholtz, B., Wilcox, E., Millen, D.R.: One-Hundred Days in an Activity-Centric Collaboration Environment Based on Shared Objects. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2004, pp. 375– 382. ACM, New York (2004) 34. Nielsen, J.: The Usability Engineering Life Cycle. Computer 25(3), 12–22 (1992) 35. Nielsen, J.: Usability engineering. Academic Press, Boston (1993) 36. Nielsen, J.: Using Discount Usability Engineering to Penetrate the Intimidation Barrier. In: Bias, R.G., Mayhew, D.J. (eds.) Cost-Justifying Usability, Academic Press, London (1994) 37. Nielsen, J.: Durability of Usability Guidelines. Jakob Nielsen’s Alertbox (January 17, 2005), http://www.useit.com/alertbox/20050117.html (retrieved January 27, 2009) 38. Nielsen, J.: Cost of User Testing a Website. Alertbox (May 3, 1998), http://www.useit.com/alertbox/980503.html (retrieved January 24, 2009) 39. Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1990, pp. 249–256. ACM Press, New York (1990)

User-Centered Design and Evaluation – The Big Picture

223

40. Polson, P.G., Lewis, C., Rieman, J., Wharton, C.: Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces. International Journal of Man-Machine Studies 36, 741–773 (1992) 41. Poltrock, S.E., Grudin, J.: Organizational Obstacles to Interface Design and Development: Two Participant-Observer Studies. ACM Trans. Comput.-Hum. Interact 1(1), 52–80 (1994) 42. Ryu, Y.S.: Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods, Doctoral dissertation, State University, Blacksburg, VA, USA (2005) 43. Sauro, J., Kindlund, E.: A Method to Standardize Usability Metrics into a Single Score. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2005, pp. 401–409. ACM, New York (2005) 44. Scholtz, J.: Usability Evaluation. National Institute of Standards and Technology (2006), http://www.itl.nist.gov/iad/IApapers/2004/ Usability%20Evaluation_rev1.pdf (retrieved January 24, 2009) 45. Stickel, C., Scerbakov, A., Kaufmann, T., Ebner, M.: Usability Metrics of Time and Stress - Biological Enhanced Performance Test of a University Wide Learning Management System. In: Holzinger, A. (ed.) Proceedings of the 4th Symposium of the Workgroup HumanComputer interaction and Usability Engineering of the Austrian Computer Society on HCI and Usability For Education and Work. Lecture Notes In Computer Science, vol. 5298, pp. 173–184. Springer, Heidelberg (2008) 46. Teo, L., John, B.E.: Towards Predicting User Interaction with CogTool-Explorer. In: Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting, pp. 950– 954 (2008) 47. Teo, L., John, B.E., Pirolli, P.: Towards a Tool for Predicting User Exploration. In: CHI 2007 Extended Abstracts on Human Factors in Computing Systems, CHI 2007, pp. 2687– 2692. ACM, New York (2007) 48. Vredenburg, K., Mao, J., Smith, P.W., Carey, T.: A Survey of User-Centered Design Practice. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 471–478. ACM, New York (2002) 49. Wixon, D.: Evaluating Usability Methods: Why The Current Literature Fails the Practitioner. Interactions 10(4), 28–34 (2003) 50. Zaharias, P.A.: Usability Evaluation Method for E-Learning: Focus on Motivation to Learn. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems, pp. 1571–1576. ACM, New York (2006)

Web-Based System Development for Usability Evaluation of Ubiquitous Computing Device Jong Kyu Choi, Han Joon Kim, Beom Suk Jin, and Yonggu Ji Dept. of Information and Industrial Engineering, Yonsei University, Seoul, Korea {jk.choi,khjoon,kbf2514jin,yongguji}@yonsei.ac.kr

Abstract. Recently, with the development of electronic technology, information technology (IT) devices that satisfy user requirements, such as PMP (Portable Multimedia Player), PDA (Personal Data Assistant), UMPC (Ultra Mobile Personal Computer) and mobile phones have been developed. These devices are making wireless communication and network communication more accessible, and by the ubiquitous paradigm, provide accessibility of information everywhere. The appearance of these devices and the development of the technology are integrating and converging in the IT devices. Therefore, there are significant changes in the purpose and environment of IT device applications. This is due to the modification of the environment in which the device is used (not only in a passive state but also in a motional state), which has a greater influence on usability. Therefore, a new methodology is required to evaluate the usability of the devices. In previous studies, by gathering and integrating the usability factors and ubiquitous characteristics, the Ubiquitous Evaluation Factor was obtained. For each factor of ubiquitous devices, deconstruction was accomplished for each usability evaluation. Through this process, components of ubiquitous devices could be extracted. Evaluation scores of ubiquitous device components and the score of the evaluation of each usability factor could be obtained from the usability evaluation. This evaluation framework was developed as a Webbased system to let the users perform the usability evaluation without having trouble with the location. This system was developed in Windows Server 2003 Enterprise Edition platform. Web Server IIS (Internet Information Server) 6.0 was used, and MS-SQL 2000 was used for the database server. For development of language, ASP (Active Server Page) was used, which is run in IIS. This study is meaningful in that through a Web-based system, various people could easily access the device, and in that evaluation of a portion of the device as well as the entire device is possible. Keywords: Ubiquitous computing device, usability, web-based system, system development.

1 Introduction The development and improvement of electronic and related technology, as well as diverse IT devices are being introduced. The appearance of Mobile devices such as PDA (Personal Digital Assistant), PMP (Portable Multimedia Player), UMPC (Ultra Mobile Personal Computer) and mobile phones is making wireless communication J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 224–231, 2009. © Springer-Verlag Berlin Heidelberg 2009

Web-Based System Development

225

more accessible and providing the possibility of having access to information everywhere[1]. Therefore, the development of these IT devices and the improvement upon technology together requires integration and convergence [2]. The environments where the IT devices are being used and the purposes for their utilization are changing. For instance, IT devices are not only used in the passive state, but also in a state of motion, which influences the usability of the device. The distinguishing characteristic of ubiquitous computing is that it is a communication system [3] that allows the users to obtain the required information in any place. Therefore, previous usability evaluation tools need to be improved, taking into consideration new user environments and ubiquitous computing [1]. From previous studies, we selected and integrated the usability factors and ubiquitous characteristic factors, developing new Ubiquitous Evaluation Factors. Each part of the ubiquitous device usability evaluation was deconstructed so the separate components of the ubiquitous device could be obtained. Through the usability evaluation, the evaluation score and each usability factor evaluation score of a ubiquitous device component can be obtained. In this study, an evaluation framework was developed as a Web-based system, allowing users to perform a usability evaluation anywhere.

2 Background 2.1 Ubiquitous Computing Device Ubiquitous computing technology development and mobile information device convergence development provide information to the user everywhere at every moment with any device. The basis of ubiquitous computing is to provide service at the request of a user and to grasp the intention of a user and situation. This results in one service system actively supporting another; that is to say the ubiquitous computing service. A ubiquitous computing device is a device for the ubiquitous service that allows a user to interact with the service anywhere at any time. Also, it grasps the intention of the user and situation to support the user. Ubiquitous devices function in a state where people do not realize that we acquire information about embedded, pervasive, portability and mobility functions; that is, to realize the ubiquitous environment [5]. Table 1. Characteristics of ubiquitous device

Researcher M.Weiser M.Weiser Burnet & Rainsford Kwon et al. uKoreaForum, 2006

Characteristics of ubiquitous device Pervasiveness Ubiquity Diversification Poratbility Interconnectivity

2.2 Previous Ubiquitous Computing Research By understanding the ubiquitous computing user’s intention and utilizing the user’s envi ronmental characteristics, it was possible to reflect the interaction with the user [4]. Ther efore, it is possible to say that the Context-Aware Computing is similar to the condition

226

J.K. Choi et al.

of the user; especially when it was focused on context-of-use of the mobile devices. Thu s, the Context-Aware Computing [6] model and the ubiquitous computing model are si milar in many ways, especially when focused on the context-of-use of mobile devices. Consequently, it is possible to say that the Context-Aware Computing model and the ubiquitous computing model have many similar points [8,9]. J. Scholtz and S. Consolvo [10] have presented a framework (UEA, Ubiquitous computing Evaluation Area) to evaluate the ubiquitous computing application. The evaluation domains for the ubiquitous computing were: attention, conceptual model, appeal with each conceptual measure and metric. In relation to Context-Aware Computing, Nigel and Miles [11] had presented the idea that to confidentially calculate the usability, it is necessary to evaluate the representative environment, user and task. Thus, it is essential to have a deep understanding of the context of use of the product. 2.3 Limitation J. Scholtz [10] defined things that are important to take into account in ubiquitous computing as “area,” and then categorize them to experiment with a systematic analysis to present a conceptual measurement variable. However, this had a major focus on the ubiquitous service; it did not focus on the device usability evaluation, so there was insufficient consideration of the user’s task. Taken from Nigel’s study[12,13], in most of the context-aware computing studies, only information on the diverse types of context is presented, lacking a concrete connection with usability principles.

3 Framework The Ubiquitous device’s usability evaluation framework was established from previous studies [14]. New suggestions on usability evaluation were proposed after a modification on the context deconstruction. Figure 1 shows how the main user and main task were selected by having the user information of the device. In this way, a specification of the device context information was achieved, which later will be used in an evaluation checklist. Consideration of each device characteristic makes further ubiquitous device usability evaluations possible.

Fig. 1. Evaluation Framework

Web-Based System Development

227

Fig. 2. Generating Evaluation Factors

3.1 Evaluation Factors Figure 2 shows the extractions made in the usability factors for the evaluation framework of the ubiquitous computing environment property. In previous studies, the basic properties for usability evaluation were: efficiency, effectiveness and satisfaction – that is to say ‘General Evaluation Factors.’ The General Evaluation Factors are considered suitable for general device evaluation, but are not specified for ubiquitous computing devices. Proposing factors for a ubiquitous device demands creation of pervasive computing quality and ubiquitous computing quality tool. Table 2. Ubiquitous Device Evaluation Factors

Factor Adaptability Controllability Interconnectivity Mobility Predictability Simplicity Transparency

Description Adaptable or easily adjusted to the changes in context Able to control device in any circumstances Interconnected network among devices, allowing sharing of information The station of the device can be mobile as the user carries it with him From past experience, the result of the system execution can be predicted User interface and instruction are simple Provides the current status of system as well as when it is running an execution

Table 2 shows some comparative computing of related studies that were used for ubiquitous service or ubiquitous software studies: Adaptability, Controllability, Interconnectivity, Mobility, Predictability, Simplicity and Transparency. This is an assortment and integration of ubiquitous device related factors. Usability evaluation factors of devices are organized as: (visual) Clarity, Accessibility, Affect, Compatibility, Consistency, Effectiveness, Efficiency, Error prevention, Feedback, Forgiveness, Helpless, Learnability, Memorability, Multi-threading, Responsiveness, Safety and User tailorability. 3.2 Evaluation Area Figure 3 shows an elementary device deconstruction for a device evaluation. It implements the usability evaluation on each device so as to obtain the degree of usability

228

J.K. Choi et al.

(high or low) that each factor has. The developed factors can be applied to evaluate each device components: LUI (Logical User Interface), GUI (Graphical User Interface), and PUI (Physical User Interface), respectively. The device component can be individually evaluated by making a separation.

Fig. 3. Evaluation area

LUI is divided into: Application software, Menu structure and Contents, System Awareness and System Acceptance. GUI is divided into Indicator, Icon and Menu. In H/W Area, Device H/W is separated in Body and Screen while PUI is separated into Control key and Touch screen. In the case of Touch screen, it is necessary to separately subdivide by input methods, and when evaluating it is risky to use different factors and checklists as standard controllers. Consequently, when evaluating devices with a touch screen, PUI of the touch screen is performed. If there is no touch screen, that evaluation factor is not taken into account. 3.3 Context of Use Figure1 shows the context of information that is solidified as: user type, device type, task type and use type. Use type is information about the environment and condition (situation) in which the user is using the device. Each context information framework has significance on the evaluation target, information access and entertainment systems. User type is divided into novice and expert, while device type is divided into PMP, Music Player, PDA, UMPC, Smartphone and Game Device. Through the expert evaluation, depending on different contexts, each evaluation factor and checklist was evaluated, giving the results of a usability evaluation with relative importance. Each evaluation factor has its own weight, which changes the importance of each checklist, depending on the device and context information.

4 System Development 4.1 System Structure In this study, the system was developed as a Web-based system to let the users perform an evaluation without having trouble with location. This system is composed for a client and a Web server or database server. The client indicates work to the Web server through browsers connected to the Internet after accessing the Web. The Web server then sends a Web page to the client

Web-Based System Development

229

and provides data that the client requires of the database server. The database server is able to query the user regarding the data that the user wants from the Web server, and carries out the work, finally returning the results to the Web server. In Figure 4, the system was developed using Windows Server 2003 Enterprise Edition, IIS (Internet Information Server 6.0) Web Server and MS-SQL 2000 Database Server. ASP (Active Server Page) was used for developing language.

Fig. 4. System structure

4.2 Evaluation Procedure The first step shows a Web page where information of each context type has to be inputted. The information about the context type that was saved in the database is recalled, then shown. In this step, the desired device to be evaluated is selected, and the user to be evaluated is identified as a novice or expert of the device in User Type. In Device Type, a selection is made between PMP Type, MP3 Type and PDA Type. Task type is divided into: video player, music player, information reading and game recognition. Use type depends on the wearable shape and portable form. After that, there is a step to input all the data in the form of a questionnaire. It gives a description of each context type so the user can understand easily, and it also gives an option of ‘not considering’ for the context types that the user does not wish to evaluate. In the second step, after having selected the information for the context, the information about the user is inputted in the server session and a Web page appears showing the different evaluation areas that can be selected to continue. This action recalls the information that is saved in the database and displays it. To allow more than one selection of the area the user wisher to evaluate, the options are selected by checking a box. Moreover, to support the user’s need of knowing more about the area to be evaluated, a description of each area is provided. In the third step, the information about the area that is going to be evaluated is saved in the server session and is shown in the corresponding checklist of each area in the database. In the upper part of the page where the checklist was selected, information about the area that is being evaluated is displayed. Each area is displayed on a different page to reduce and avoid confusion and disorder. In the last step, the user’s selection of the checklist, the information of the context type that was saved in the sever session in previous steps, and the information of the

230

J.K. Choi et al.

evaluation area are saved. From the data that was saved about a determined device, it is possible to obtain the average evaluation score. After being saved in the database, the visible result is shown in a page with an eight-column graph. The results from the evaluation of ubiquitous characteristics are shown in a graph that indicates the score of each ubiquitous factor. The results from the evaluation of general characteristics are also shown in a graph. Moreover, by providing a graph for each factor with a score over 100, we can see insufficient areas more clearly. Also, the result of the device evaluation area (LUI, GUI, PUI, Device H/W) of each of them is represented in a graph with a score over 100 so as to show the areas that have to be improved.

Fig. 5. Evaluation system

5 Conclusion and Further Study This study developed a Web-based system of a framework to evaluate the usability of ubiquitous computing devices. There are three important aspects. First, through the development of a Web-based system, the user can evaluate everywhere where the Internet is accessible. Also, it is more comfortable, as it allows the user to see the results in a moment. Secondly, the user is able to select the area that he desires to evaluate. A complete or part evaluation of the selected areas is possible. Thirdly, as this system uses a database, the evaluated data can be saved. Through this saved data, it is possible to see an average of all the other data of previous and other evaluations. However, this system has only been implemented for a small number of devices, and not in every type of device. Therefore, in further studies it is necessary to increase the validity of the system by performing an evaluation of a more diverse range of devices. Then, after obtaining the validity of the system, it will be possible to make updates.

Web-Based System Development

231

References 1. Zhang, D., Adipat, B.: Challenges, Methodologies, and Issues in the Usability testing of Mobile Applications. International Journal of Human-Computer Interaction 18(3), 293– 308 (2005) 2. Rondeau, D.B.: Branding is Experience. Communication of the ACM 48(7), 61–66 (2005) 3. Park, I.C., Kim, S.C., Choi, M.S.: A Study on the Development Methods of gestureoriented Natural Interface for the Control of Ubiquitous Device. Koeran Society of Basic Design & Art 6(3), 265–274 (2005) 4. Scholtz, J., Richter, H.: Report from ubicomp 2001 workshop: Evaluation methodologies for ubiquitous computing, SIGCHI Bulletin (January/February 2002) 5. Kim, J.Y.: Ubiquitous Computing: Business model and prospects, Samsung Economic Research Institute (2003) 6. Schilit, B.N., Adams, N., Want, R.: Context-Aware Computing Applications. In: IEEE Workshop on Mobile Computing Systems and Applications (1994) 7. Betiol, A.H., de Abreu Cybis, W.: Usability Testing of Mobile Devices: A Comparison of Three Approaches. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 470–481. Springer, Heidelberg (2005) 8. Kwon, O.B., Kim, J.H.: A Multi-Layered Approach to Assessing Level of Ubiquitous Computing Services. Information System Review 8(1), 43–61 (2006) 9. Nam, J.H., Choi, M.S.: A Study on the Multi Interface Factor Base the Service Evolution on Ubiquitous Computing Environment. Korea Digital Design Council 11, 357–366 (2006) 10. Scholtz, J., Consolvo, S.: Toward a framework for evaluating ubiquitous computing applications. IEEE 3(2), 82–88 (2004) 11. Bevan, N., Macleod, M.: Usability measurement in context. Behavioral and Information Technology 13, 132–145 (1994) 12. Lee, I.S., et al.: Use Contexts for the Mobile Internet: A Longitudinal Study Monitoring Actual Use of Mobile Internet Services. International Journal of Human-Computer Interaction 18(3), 269–292 (2005) 13. Barnard, L., Yi, J.S., Jacko, J.A., Sears, A.: Capturing the effects of context on human performance in mobile computing systems. Personal and Ubiquitous Computing 11, 81–96 (2007) 14. Kim, H.J., Choi, J.K., Ji, Y.: Usability Evaluation Framework for Ubiquitous Computing Device. In: International Conference on Convergence and Hybrid Information Technology (2008)

Evaluating Mobile Usability: The Role of Fidelity in Full-Scale Laboratory Simulations with Mobile ICT for Hospitals Yngve Dahl1, Ole Andreas Alsos2, and Dag Svanæs2 1

Telenor Research & Innovation, Otto Nielsensvei 12, 7004 Trondheim, Norway [email protected] 2 Department of Computer and Information Science, Norwegian University of Science and Technology, Sem Sælandsvei 7-9, 7491 Trondheim, Norway {Ole.Andreas.Alsos,Dag.Svanes}@idi.ntnu.no

Abstract. We have applied full-scale simulations to evaluate the usability of mobile ICT for hospitals in a realistic but controllable research setting. Designing cost-effective and targeted simulations for such a purpose raises the issue of simulation fidelity. Evaluators need to identify which aspects of the research setting that should appear realistic to simulation participants, and which aspect that can be removed or represented more abstractly. Drawing on research on training simulations, this paper discusses three interrelated fidelity components—equipment/prototype fidelity, environmental fidelity, and psychological fidelity. These components need to be adjusted according to which design aspects evaluators want to gather feedback on. We present examples of how we have configured the components in various simulation-based usability assessments of mobile ICT for hospitals. The paper concludes by providing a set of guiding principles concerning the role of fidelity in simulation-based usability evaluations. Keywords: Clinical information systems, fidelity, evaluation, human factors, mobility, simulation, training simulation, usability, user-centered design.

1 Introduction In this paper we investigate the concept of fidelity in the context of full-scale laboratory simulations and usability evaluations of mobile ICT for hospitals. Fidelity, as it is commonly understood in HCI literature, is the extent to which a computer application or system reproduces visual appearance, interaction style and functionalities during evaluation [1]. In conventional desktop-based usability testing the physical and social aspects of the use situation are of little concern for evaluators. Interaction with desktop-based systems is highly uniform and static with regard to the physical and social aspects of the use situation—one user sitting in front of a PC, with his or her attention directed mainly on the computer screen. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 232–241, 2009. © Springer-Verlag Berlin Heidelberg 2009

Evaluating Mobile Usability

233

As human-computer interaction moves “beyond the desktop” and into highly dynamic work settings, such as hospitals, the old standards for usability testing arguably no longer hold water. Work situations in hospitals often involve mobility and bodily work [2, 3]. This makes the usability of mobile ICT more subject to external factors that are not related to the GUI and the software being evaluated as such. These are factors that fall beyond what evaluations conducted in conventional usability laboratories can reveal. We have attempted to meet these challenges by means of full-scale laboratory simulations—Simulated, natural-like hospital environments, in which nurses and physicians act out clinical scenarios using mock-ups or prototype systems. Such an approach arguably raises the need for evaluators to think about the fidelity, or level of detail, of the research setting. A critical issue for applying simulations as a usability evaluation methodology in a cost-effective way is deciding on the right level of fidelity. This paper aims first to show that fidelity in usability evaluations of mobile ICT is a concept that extends beyond the software prototype being evaluated. Particularly, when addressing hospital settings, where the technology is likely to be used as part of work activities requiring manual labor with hands and feet in addition to high situational awareness, the physical environment and the work tasks become vital components of the total system being simulated. The second objective of this paper is to demonstrate how the test environment, the prototype, and the test scenario can be adjusted to achieve targeted evaluations and induce behavior among participant that is desirable in terms of informing specific design aspects relevant for the usability of mobile systems in hospitals.

2 Methodological Motivation The work conditions that ICT supporting clinical care is used in are very different than those for office settings and desktop-based computer interaction. The call for mobile ICT in hospitals essentially stems from the distributed nature of clinical work, and rapidly changing work situations. In order to conduct valid usability assessments of mobile interactive technology for such environments, then, the design solutions must be evaluated in a relevant use context [4]. Consequently, aspects that are characteristic for clinical work, such as mobility, clinician-patient interaction, and frequent context shifts, must be reflected in the setting in which the evaluation takes place. Conducting usability evaluations in actual clinical situations is challenging. Firstly, the hospital is a high-risk environment, in which it can be critical to avoid affecting ongoing work. Secondly, patient information confidentiality is likely to prevent video and audio recording. At the same time, conventional laboratories intended for controlled desktop-based usability evaluations are unsuited for reconstructing the rapidly changing conditions of hospital work. The combined need for a realistic, yet controllable, research setting, has motivated us to build a customized full-scale model of a hospital ward section, with advanced video and audio recording facilities.

234

Y. Dahl, O.A. Alsos, and D. Svanæs

3 Simulation Fidelity Principles HCI literature in general provide little practical guidance on how to compose fullscale simulations with the aim of evaluating usability of mobile ICT. This has motivated us to study literature on training simulations in search of guiding principles. Given the relatively long history of training simulators, we will first highlight some of the central principles described in related research literature. Next, we will provide a brief overview of studies within the field of mobile HCI, where simulations have been employed to evaluate prototypes and early concepts. 3.1 Simulations Applied for Training Purposes Within high-risk industries such as aviation, naval shipping, health care, and nuclear power production there has been a long tradition for using simulations for training purposes in risk-free environments. Obviously, the objective of training simulations and simulations for usability purposes differ. Simulations applied in the context of usability assessment aim at gathering data about the effectiveness, efficiency, and user satisfaction of a product used by specific users in a realistic situation. The focus is on product performance. Training simulations, on the other hand, are typically used for educational purposes and for maintaining or enhancing human work-related skills. In this sense, they are human-centric rather than product-centric. Despite these differences in focus, many of the concepts developed from research on training simulations are highly relevant when designing simulation-based usability evaluations. Among the most central concepts described in research literature on training simulations are equipment fidelity, environment fidelity, and psychological fidelity [5]. Equipment fidelity refers to the extent to which the appearance and feel of real tools, devices, or systems that simulation participants operate on are duplicated. For example, aircraft cockpit procedures have been trained both with hi-fi representations of aircraft instruments and with lo-fi mock-ups [6]. Environment fidelity concerns the extent to which physical characteristics of the real-world environment (beyond the training devices) are realistically represented in the simulation. High-fidelity aircraft simulators are full-size replica of cockpits that duplicate of the operational aircraft environment and motions to great detail [7]. In flight training environments of less fidelity (e.g., desktop evaluation environments), visual and motional cues are typically reduced or lacking. Lastly, psychological fidelity, relates to the realism of the simulation as perceived by its participants. In other words, it is extent to which participants are able to engage in the simulated situation, as they would have done in the natural setting. This is intimately dependent on the psychological demands the simulated tasks place on the participants. Human perception, attention, decision-making, memory, and action are factors that may influence psychological fidelity [8, p. 420]. Developing scenarios that replicate task demands of the real-world system is a common technique for enhancing psychological fidelity [5]. In training simulations, psychological fidelity is often considered the most important, because it is the most relevant attribute for learning. Prophet and Boyd [6] found that the transfer of training was equal for students practicing ground cockpit procedures in real airplanes, and student practicing the same routines on lo-fi representations of the relevant devices.

Evaluating Mobile Usability

235

Each of the three components described above can be set along a continuum ranging from low to high fidelity (Fig. 1). As pointed out by Beaubien and Baker [5], the level of fidelity for each component typically depends on the purpose of the training simulation. For example, low-fidelity role-plays have been used to train teamwork related attitudes and skills, while simulations of higher fidelity are required if the goal is to learn the specific consequences of actions. Taken together equipment, environment, and psychological fidelity form the overall simulation fidelity (Fig. 1).

Fig. 1. Three interrelated simulation fidelity components. Each component can be set along a continuum ranging from low to high fidelity.

3.2 Mapping the Simulation Fidelity Concepts to HCI Terminology In the context of HCI and usability assessments we consider equipment fidelity to be the equivalent of computer system or application fidelity (we will refer to this as prototype fidelity). As previously noted, this component encompasses physical appearance, interaction style, and functionalities. Environmental fidelity is the realism of the physical use setting (i.e., the point of interaction), while psychological fidelity corresponds to the user-perceived realism of the tasks they are given as part of a test. 3.3 Simulations Applied in Mobile HCI The use of simulations in mobile HCI is in many ways a result of the recognition that conventional usability laboratories and testing do not duplicate factors affecting mobile usability [9, 10]. It can also been considered an efficient approach to overcome many of the challenges related to studying mobile systems in the field [11]. Examples of usability studies in which contextual features have been simulated in laboratory settings are decribed in work by Bohnenberger et al. [12], Pirhonen et al. [13], and Kjeldskov and Skov [14]. Some simulation-based usability studies of relevance to mobile ICT in hospital settings can be found in Refs. [4, 15, 16].

4 Applying the Fidelity Principles in Practice In the current section we will provide examples from conducted simulations, showing how we have attempted to carefully adjust simulation fidelity to promote reflection

236

Y. Dahl, O.A. Alsos, and D. Svanæs

among participants regarding specific design aspects. Our examples also highlight the close interrelationship between the various simulation fidelity components presented earlier. 4.1 Case Study: Point-of-Care Scenarios There are many hospital scenarios in which mobile ICT may prove helpful [2]. We have concentrated on a limited set of situations that have been found appropriate for usability testing in our full-scale hospital ward model. In particular, we have focused on situations where nurses and physicians are located at the patient bedside, i.e., at the point of care. Examples of hospital routines were such situations occur include ward rounds, during administration of medicine, and in response to patient calls. These situations form suitable test candidates for mobile ICT because they occur frequently in hospital wards, require mobility, and involve personnel who need quick and effortless access to patient related information. 4.2 Environment Fidelity As previously pointed out, the circumstances under which mobile ICT supporting hospital work are used are radically different than that for office work and conventional desktop computer interaction. Human-computer interaction in clinical settings is more physical in nature both in the sense that hospital workers are using them while on the move, and in situations requiring physical interaction with patients (e.g., assistance and examination). We have observed that to capture physical and bodily usability aspects of mobile ICT used at point of care, prototype solutions must be evaluated in research environments that closely mimic the physical environment of real patient rooms. For example, realistically propositioned rooms furnished with patient beds makes it possible for participants to move naturally in the model both around and between patient beds. For simulations addressing the usability of point-of-care systems this is essential, because it can help give participants a realistic impression of how well different solutions are physically adapted to the care situations. Examples of physical design factors, which according to simulation participants can increase the usability of mobile ICT at point of care, include the possibility easily to share screen content with patients (Fig. 2, left) and the opportunity to have digital media ready at hand to be to be used and put aside depending on what the immediate care situation calls for (Fig. 2, right). Both examples illustrate the intimate relationship between the physical environment and the physical placement and form factors of interactive devices.

Fig. 2. Examples showing the close relationship between the physical environment and the physical placement (left) and form factors (right) of digital media

Evaluating Mobile Usability

237

There are often subtle details that need to be in place to capture physical and bodily usability factors. For example, to form a realistic impression of how well an interaction design solution accommodates the dialogue between clinicians and in-bed patients, patients need to be represented by human actors in real hospital beds (Fig. 2, left). Likewise, simulation participants need to wear their daily work uniforms, to bring forth or temporarily put aside digital media (Fig. 2, right). 4.3 Prototype Fidelity The Spectrum of Prototype Fidelity. Prototypes can generally be divided into low, medium, or high fidelity. In low-fidelity prototyping, props (e.g., foam and cardboard models, paper and post-it notes, etc.) are often used as physical representations of interactive devices with rough sketches of envisioned graphical interfaces. Mediumfidelity prototypes are functional (computer-based) models of systems. They generally have simplified GUIs, but little or no functionality behind the GUI elements. High-fidelity prototypes are sophisticated and functional versions of envisioned designs. They may show sample information content. Prototype Fidelity for Point-of-Care Scenarios. In our studies of mobile ICT applied in hospital settings we have used prototypes of different fidelities depending on design phase. As part of specifying user requirements for mobile ICT for point-ofcare usage, we have conducted simulations using lo-fi props in realistic models of patient rooms [17]. The main rational for this is to put focus on the context in which the technology will be used, rather than details concerning GUIs and software functionality. By applying mock-ups one can avoid restricting reflection among participants by committing to particular hardware and interfaces. In simulations with functional prototypes, we have attempted to be more particular about design aspects we wanted to address. For these simulations we have applied mixed fidelity prototypes [18], i.e., models that combine different fidelities with regard to GUIs, functionalities, interaction styles, and information content. For example, in simulations addressing the usability of location-based access to medical information at the point of care (the full study is described in Ref. [16]) we found it useful

Fig. 3. Mixed fidelity prototype providing location-based access to a patient’s medical record. High-fidelity sensors and radio tags detect the physical position of a test participant and near patients. The graphical representation of the medical record, which is automatically retrieved and presented on the bedside terminal, is of low fidelity.

238

Y. Dahl, O.A. Alsos, and D. Svanæs

to implement the interaction techniques using high-fidelity hardware and sensors to give participants a realistic impression of actual use. At the same time, we deliberately kept the details of the GUI at a low level and avoided to link the prototype to realistic medical sample data. The main rational for this delimitation is that we primarily wanted to help participant focus on and give feedback regarding the usability of interaction styles, rather than GUI related aspects of the design. Fig. 3 shows the mixed fidelity prototype that we applied in the example described above. 4.4 Psychological Fidelity This section gives a brief summary of different techniques we have employed to increase the psychological fidelity of our simulations. Domain Expertise. As pointed out above, developing scenarios that mimic the task demands of the real-world system can increase psychological fidelity. To facilitate this, and make sure that the simulations reflected a sufficient degree of realism, the scenarios were designed with assistance from domain experts (physicians and nurses). Baseline Scenarios. To help participants relate the prototypes and evaluated concepts to their everyday workday, we have learned that running an introductory baseline scenario reflecting current paper-based practices are useful (Fig. 4). These “as-is” scenarios can effectively act as a reference or benchmark for the participants when they later act out the same scenario using functional prototypes. This help participants “make sense” of scenarios involving information media they are not familiar with.

Fig. 4. Baseline scenarios reflecting current paper-based practices (left) can act as references for test subjects when they later try out digital solutions for the same purposes (right)

Targeted Simulations. Because the conducted simulations mainly have focused on evaluating early concepts and partial prototypes, we have tried to tailor the scenarios to help promote feedback on the particular design aspects. In some cases, this has resulted in delimitations that, depending on the purpose of the simulation, possibly might reduce psychological fidelity. Using only mock-up representations of electronic medical records in some of the simulations, as explained above, is an example of such a limitation of scope. This, however, we consider necessary compromises to achieve targeted evaluations. Feedback from the participants indicated that they found the simulations realistic in spite of such simplifications. Scripted Simulations. We have also experimented with ways to increase the participants’ perceived realism of the simulations by trying to integrate their professional experience. One technique we have applied is to use patient actors instructed to reveal

Evaluating Mobile Usability

239

certain pieces of information during the scenario, but leave it for the participants (i.e., clinicians) to decide how to act on that information [19]. For other types of simulations we have found it sufficiently to give the patient actors a more passive role, e.g., acting as physical markers in the simulations.

5 Setting the Scene Right A fundamental dilemma related to simulations, whether applied for usability assessment or training purposes, is specifying a priori a sufficient fidelity level. Typically, approximating mobile use contexts involves trade-offs between control over the research setting, realism, and available resources [4]. In the current section we will present and discuss some guiding principles for “setting the scene right” for simulation-based usability evaluations of mobile ICT for hospitals. 5.1 Psychological Fidelity First In Sect. 3.1 we pointed out that psychological fidelity is often considered the most significant fidelity component in training simulations. This is because it represents the transfer of skills learnt in the simulated setting back into the real world. Based on our experiments, we argue that the same component is also the most critical when designing full-scale simulations for usability assessment purposes, but for different reasons. We see psychological fidelity first and foremost as a key premise for provoking reflection among simulation participants. This is especially valuable in early phases of the design process before critical design choices are taken, and when end user feedback is most likely to help inform the actual design. In order to motivate reflection among simulation participants it is essential that they see the on-the-job relevance of the simulation. As discussed in Sect. 4, a realistic test environment and prototypes with certain functional features can help evoke the central psychological mechanism triggered in everyday clinical work. 5.2 How Much Fidelity Is Enough? In our full-scale simulations we have attempted to follow a “just enough” principle with regard to the amount of realism reflected. That is, duplicating aspects that we, along with domain experts, consider most likely to affect the perceived usability of the prototypes that are to be tested. This, as we have pointed out earlier, depends closely on the object of the simulation. Because the usability of mobile ICT for hospitals are highly contextual, there is no “one size fits all” approach when it comes to fidelity of simulation-based usability evaluations.

6 Summary and Conclusions In this paper we have investigated the role of fidelity in full-scale laboratory simulations used for usability assessment of mobile ICT for hospitals. Drawing on training simulation research, we identified three components of relevance—Environment fidelity, equipment or prototype fidelity, and psychological fidelity. We have shown

240

Y. Dahl, O.A. Alsos, and D. Svanæs

by examples from practical simulations how the different components can be modified to match the focus of different evaluations. The key principles the current paper has suggested regarding fidelity of simulationbased usability assessments of mobile ICT for hospitals are as follows: • Simulations need to be specific about the design aspects that are being evaluated. Cost-effective and targeted simulations should replicate features of the mobile ICT solution, physical environment, and work tasks, that are considered relevant for the design aspect one wants to gather feedback on. • There is no direct correlation between the overall fidelity of experimental simulations and effectiveness in terms of informing design. Realistic prototypes and work environments may enhance the perceived realism of the simulation, but this does not guarantee that feedback from participant are valuable in terms of informing design. • Simulations for usability assessment purposes should prioritize psychological fidelity. This component is most relevant for provoking reflection in design among simulation participants. To provoke such reflections it essential that participants are able to relate the design concept to their everyday work. • The requirements to simulation fidelity will typically increase as the mobile ICT solution is developed. We expect that future simulations in our full-scale ward model will enable us to further explore the role fidelity plays in simulation-based usability evaluations of mobile ICT for clinical work.

Acknowledgements The current work has been supported in part by the Norwegian Research Council by grant 176761 (POCMAP) of the VerdIKT program, DIPS ASA, The Industrial Research Fund for NTNU, St. Olav University Hospital, Akershus University Hospital, NTNU, and Telenor R&I.

References 1. Rudd, J., Stern, K., Isensee, S.: Low vs. high-fidelity prototyping debate. Interactions 3, 76–85 (1996) 2. Sørby, I.D., Melby, L., Nytrø, Ø.: Characterising cooperation in the ward: framework for producing requirements to mobile electronic healthcare records. International Journal of Healthcare Technology and Management 7, 506–521 (2006) 3. Bardram, J.E., Bossen, C.: Mobility Work: The Spatial Dimension of Collaboration at a Hospital. Computer Supported Cooperative Work 14, 131–160 (2005) 4. Kjeldskov, J., Skov, M.B., Als, B.S., Høegh, R.T.: Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In: Mobile HCI 2004, pp. 61–73 (2004) 5. Beaubien, J.M., Baker, D.P.: The use of simulation for training teamwork skills in health care: how low can you go? Qual Saf Health Care 13 (2004)

Evaluating Mobile Usability

241

6. Prophet, W.W., Boyd, H.A.: Device-Task Fidelity and Transfer of Training: Aircraft Cockpit Procedures Training. Tech. Report. Human Resources Research Organization, Alexandria, VA (1970) 7. Rehmann, A., Mitman, R., Reynolds, M.: A handbook of flight simulation fidelity requirements for human factors research. Tech. Report No. DOT/FAA/CT-TN95/46. Wright-Patterson AFB, OH: Crew Systems Ergonomics Information Analysis Center (1995) 8. Patrick, J.: Training. In: Tsang, P.S., Vidulich, M.A. (eds.) Principles and Practice of Aviation Psychology, CRC Press, Boca Raton (2002) 9. Johnson, P.: Usability and mobility; Interactions on the move. In: Proceedings of the First Workshop on Human-Computer Interaction with Mobile Devices (1998) 10. Graham, R., Carter, C.: Comparison of speech input and manual control of in-car devices while on-the-move. In: Mobile HCI 1999 (1999) 11. Pascoe, J., Ryan, N., Morse, D.: Using while moving: HCI issues in fieldwork environments. Transactions on Computer-Human Interaction 7, 417–437 (2000) 12. Bohnenberger, T., Jameson, A., Krüger, A., Butz, A.: Location-aware shopping assistance: Evaluation of a decision-theoretic approach. In: Paternó, F. (ed.) Mobile HCI 2002. LNCS, vol. 2411, pp. 155–169. Springer, Heidelberg (2002) 13. Pirhonen, A., Brewster, S., Holguin, C.: Gestural and Audio Metaphors as a Means of Control for Mobile Devices. In: Proceedings of the SIGCHI conference on Human factors in computing systems (2002) 14. Kjeldskov, J., Skov, M.B.: Creating Realistic Laboratory Settings: Comparative Studies of Three Think-Aloud Usability Evaluations of a Mobile System. In: Interact 2003, pp. 663– 670. IOS Press, Amsterdam (2003) 15. Alsos, O.A., Svanæs, D.: Interaction techniques for using handhelds and PCs together in a clinical setting. In: NordCHI, Oslo, Norway, pp. 125–134. ACM, New York (2006) 16. Dahl, Y., Svanæs, D.: A comparison of location and token-based interaction techniques for point-of-care access to medical information. Personal and Ubiquitous Computing 12, 459– 478 (2008) 17. Svanæs, D., Seland, G.: Putting the users center stage: role playing and low-fi prototyping enable end users to design mobile systems. In: Proceedings of the SIGCHI conference on Human factors in computing systems, Vienna, Austria, ACM, New York (2004) 18. Petrie, J.N., Schneider, K.A.: Mixed-fidelity prototyping of user interfaces. In: Doherty, G., Blandford, A. (eds.) DSVIS 2006. LNCS, vol. 4323, pp. 199–212. Springer, Heidelberg (2007) 19. Alsos, O.A., Dabelow, B.: Stylus, Finger, or Buttons? A Comparative Evaluation Study of Interaction Techniques for PDAs in Point-of-Care Situations (submitted manuscript)

A Multidimensional Approach for the Evaluation of Mobile Application User Interfaces José Eustáquio Rangel de Queiroz and Danilo de Sousa Ferreira Federal University of Campina Grande, Electrical Engineering and Computer Science Center – Computer Science Department, Av. Aprígio Veloso, s/n – Bodocongó, Campina Grande, CEP 58109-970, Paraíba, Brazil {rangel,danilo}@dsc.ufcg.edu.br, {rangeldequeiroz,danilo.sousa}@gmail.com

Abstract. This paper focuses on a hybrid approach for the evaluation of mobile application UI, based upon a set of well known techniques for usability evaluation. Two perspectives of the problem are focused: (i) the user’s perspective, which is expressed by user’s perception of the application; and (ii) the specialist’s perspective, which is expressed by his/her considerations from the point of view of the user-application interaction, and from the point of view of the HCI community as well. Further comparisons between a lab and field evaluation approaches are given for a case study involving an Internet tablet. Conclusions are given concerning on how to apply the experience acquired by evaluating conventional UI in the mobile technology domain. Keywords: Usability evaluation, mobile devices, multidimensional approach.

1 Introduction In recent years, a variety of mobile computing devices has emerged, including portables, palmtops, and PDAs. In consequence, the evolution of mobile computing devices has imposed a clear need for evaluation methods that are specifically suited to mobile devices. Nonetheless, the more evolved is the computing market toward mobile computing systems, the more difficult becomes for HCI evaluators to choose among the approaches for mobile application UI usability evaluation recently proposed in the literature. It is a fact that a user is likely to be mobile is the single greatest difference in context between users of mobile and desktop devices, and of course that mobility leads to dynamic changes in users’ context. But it has to be also taken into account that it is difficult, if not impossible, to compare between different studies, based on a plethora of claims made without solid statistical results. Given the inherent features of these devices (e.g., mobility, restrictive I/O and storage capabilities, and dynamic use contexts), they has imposed a clear need for specifically suited evaluation methods. One of the major questions in the literature is related to the possibility of adapting concepts, methodologies, and approaches commonly used in traditional lab and field testing of desktop applications to mobile ones. Another question has been related to whether to adopt a field or a lab approach. However, little discussion is given of which technique or combination of them is best suited for a specific application and its context of use. Beyond these J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 242–251, 2009. © Springer-Verlag Berlin Heidelberg 2009

A Multidimensional Approach

243

levels of choice and for a successful choice, it seems equally essential to know about the effectiveness of the chosen approach. Practitioners need to know which methods are more effective and in what ways and for what purposes. Otherwise the evaluation process may result in a big effort with a small payoff.

2 Usability Evaluation for Mobile Devices Usability data typically consists of any kind of information which can be used as measures or identification keys for factors affecting the usability of a system. Such kinds of data are collected by usability evaluation methods and techniques that can assign values to usability dimensions for evaluating different kinds of UI [1] and/or indicate usability problems or other design deficiencies in UI [2]. Usability data are usually gathered via either analytic or empirical methods [2][3]. Analytic or expert-based methods are often conducted by HCI experts and do not involve human participants performing the tasks, i.e. they rely frequently on the specialists' judgment. On the other hand, in spite of being also conducted by HCI experts, empirical or used-based methods involve data collection of human usage. Usability diagnosis basically begins with raw observational data often categorized into models/ frameworks emphasizing either (i) the nature/fidelity of the artifact being evaluated; (ii) the context of use (involving user and social relations, tasks and psychological factors, and environmental aspects); (iii) the approach adopted for capturing the data (including the expended resources, the involved degree of formality, rigor, and amount of designers/evaluators and users); or (iv) the goal of the collection effort [4][5][6]. Some of those models and frameworks are aligned to ISO usability dimensions [7], which are commonly taken to include the aspects efficiency, effectiveness, and subjective satisfaction in a HCI process. It is undeniable that usability evaluation effort for desktop systems has grown especially in the last decade. In spite of debates still taking place within the HCI area, they are often based on a tacit understanding of basic concepts. Extensive guidelines have been written for describing how usability evaluation in controlled environments should be conducted (e.g., [3][8]). Further, experimental results highlighting pros and cons of different techniques are available to be applied (e.g., [9]). Especially in the past decade, technological advances and methodological approaches in HCI have been challenged by the growing focus on applications for mobile computing devices. Several authors (e.g., [10][11]) argue that mobile computing demands not only real users but also a real or simulated context with device interaction tasks as well as real tasks or realistic task simulations. The question about carrying on mobile device evaluation in a lab or field context has also been discussed (e.g. [9][12]), the effectiveness of the approach depending on the relevance of the results presented, and on the quality of the data analysis process as well. However, despite presenting data analysis results, the reports usually omit some important details of the data gathering and the analysis process. For sure, they could guide choices, and give a comprehensive view of the approach. While a strong effort of HCI research has been devoted on alternatives for data collection issues, data analysis/validation are presented in rare cases (e.g., [3][12]). In consequence, the evaluator is unable to replicate appropriately and successfully the reported findings in other contexts. As for empirical data analysis, many methods and techniques have been employed for field testing data, video data, expert data, or

244

J.E.R. de Queiroz and D. de Sousa Ferreira

head-mounted video and cued recall [3][13] [14]. In essence, the usual method triangulation seems to be a field testing without or with video analysis, and transcriptions of usability test sessions. The absence of an in depth usage data analysis seems be due to it is often not applicable to industrial purposes due to several constraints. Nonetheless for research purposes it is strongly recommended to provide sufficient detail to allow for replication.

3 The Multidimensional Evaluation Approach The present approach was originally proposed for evaluating desktop application UI [15], and further adapted to evaluate the usability of mobile application UI [16]. It is based upon a hybrid strategy which encompasses the best features of: (i) standards inspection; (ii) user performance measurement; and (iii) user inquiry. It is based on the premises that (i) each evaluation technique provides a different level of information, which will help the evaluator to identify usability problems from a specific point of view; and (ii) triangulation is used to compare the data collected from the various techniques with the aim to produce complementary and more robust results. 3.1 Product Standard Conformity Assessment, User Performance Measurement and User Subjective Satisfaction Measurement According to [7] conformity assessment means checking whether products, services, materials, processes, systems, and personnel measure up to the requirements of standards. For conformity assessment, the desktop version, of the multidimensional approach adopts the standard ISO 9241 (Ergonomic Requirements for Office Work with Visual Display Terminals). In its mobile application UI evaluation version, and more specifically for the Internet tablet case study presented in this paper, it was found that only some parts of ISO 9241 could be applied: 14 [17], 16 [18], and 17 [18]. Some other standards were applied to this kind of device such as ISO/IEC 14754 [20], ISO/IEC 24755 [21]. In general, user performance measurement aims to enable real time monitoring of user activities, providing data on the effectiveness and efficiency of his/her interaction with a product. It also enables comparisons with similar products, or with previous versions of the same product along its development lifecycle, highlighting areas where the product usability can be improved. When combined with the other methods, it can provide a more comprehensive view of the usability of a system. The major change introduced in the original evaluation approach concerns the introduction of field tests as a complement to the original lab tests. The measurement of user subjective satisfaction has been widely adopted as a measure of IS success, and the subject of a number of researches since the 1980s (e.g., [22][23]). User satisfaction diagnosis provides an insight into the level of user satisfaction with the product, highlighting the relevance of the problems found and their impact on the product acceptance. In this approach, user subjective satisfaction data are gathered from three methods: (i) automated questionnaires administered before and after test sessions; (ii) informal think-aloud trials performed during test sessions; and (iii) unstructured interviews conducted at the end of test sessions.

A Multidimensional Approach

245

In essence, ISO defines usability as an extent to which a product can be used by specified users, in a specified context of use, to achieve specified goals with effectiveness, efficiency and satisfaction [7]. It also defines that at least one indicator in each of these aspects should be measured to determine the level of usability achieved. As briefly exposed in this section, the multidimensional approach presented here meets the requirements set by ISO 9241-11 because it is used: (i) the task execution time as an efficiency indicator; (ii) the number of incorrect actions, the number of incorrect choices, the number of repeated errors, and the number of accesses to the online/ printed help as effectiveness indicators; and (iii) the think-aloud comments, the unstructured interview responses, and the questionnaire scores as subjective satisfaction indicators.

4 Comparative Study of Lab versus Field Use of an Internet Tablet The main objective of this study was to investigate the need for adapting the original evaluation approach to the context of mobile UI applications, based on the analysis of the influence of the context - lab versus field, mobility versus stationary interaction. 4.1 Experiment Design The experiment was designed to investigate the influence of the context (field and lab and related aspects, e.g., mobility, settings) and the user experience on the evaluation results. Consequently, independent and dependent variables were chosen, as well as objective and subjective usability indicators were defined. The independent variables chosen were: (i) Task context, which comprised external factors (e.g., noise level and light intensity) and internal factors (e.g., stress or other health conditions) that could affect the user behavior and performance during the usability test; (ii) User mobility, which referred to conditions under which the task was being performed (e.g., moving between places or stand still wandering is working while being mobile); and (iii) User experience level, which referred to the user knowledge regarding mobile devices and desktop computers, in general. On the other hand, the dependent variables chosen were: (i) Task execution time (time taken by a device user to perform a task); (ii) Number of incorrect choices (number of times the user has made incorrect choices while selecting menu options in the interface); (iii) Number of incorrect actions (number of times the user has performed incorrect actions while selecting menu options in the interface, excluding menu incorrect choices); (iv) Number of repeated errors (number of times the same error was made by the user while performing a task, excluding the number of incorrect choices); (v) Number of accesses to the online/printed help (number of times the user accessed the online and/or printed help while performing a task); (vi) Perceived usefulness (user opinion about the usefulness of the mobile application for the prescribed task); and (vii) Perceived ease of use (user subjective satisfaction when using the mobile device). The chosen usability objective indicators were: (i) Task execution time; (ii) Number of incorrect actions; (iii) Number of incorrect choices; (iv) Number of repeated errors; and (v) Number of accesses to the online help and/or printed manuals. Additionally, the chosen usability subjective indicators were: (i) Product easy of use; (ii)

246

J.E.R. de Queiroz and D. de Sousa Ferreira

Task completion easiness; (iii) Input mechanism easy of use; (iv) Text input modes easy of use; (v) Ease-of-understanding terms and labels; (vi) Ease-of-understanding messages; and (vii) Ease-of-use of help instructions. 4.2 Test Environment, Materials and Participants In both realistic test environments, all the elements (e.g., tasks, informal think aloud, unstructured interviews) were identical, only the test environment was different. The lab test was conducted in a typical usability lab, while the field test was conducted in an environment in which users could walk, stand still, sit or do whatever they would normally do while performing their tasks. To minimize moderator bias, the tests were conducted with three experienced usability practitioners with 3 to 12 years of experience in usability testing. The instructions given to participants were predefined. All moderators participated in data gathering and data analysis. Statistical analysis was performed by one moderator, and revised by another one. The chosen mobile device for this experiment was the Nokia 770 Internet Tablet and some of its native applications. Tests were performed in a controlled environment (a usability lab) and in the field as well. In the first one, the interaction was recorded by using three cameras installed in the room, one focused on the user facial expressions, a second one wider focused on the table where the user performed test tasks with the device fixed to the table or free in his/her hand. In the field experiment, a micro-camera connected to a transmitter was coupled to the device to remotely record and transmit user-device interaction data to the lab through a wireless connection (see Fig. 1).

Fig. 1. Apparatus to support the video micro-camera

Additionally, a remote screen capture software (VNC) was used to take test session screenshots, and a web tool named WebQuest [24] was used as well for supporting the user subjective satisfaction measurement. WebQuest supports the specialist during data collection, automatic score computation, performs statistical analysis, and generates graphical results. Currently WebQuest supports two questionnaires: (i) a pre test questionnaire, the USer (User Sketcher), conceived to raise the profile of the system users; and (ii) a post test questionnaire, the USE (User Satisfaction Enquirer), conceived to raise the user degree of satisfaction with the system. The participants were divided into two groups of 20 for the field and lab tests. According to their experience levels, both groups were then subdivided into three subgroups. A ratio of 8 beginners to 8 intermediates to 4 experts was adopted. For the lab

A Multidimensional Approach

247

tests only, the 20 participants were subdivided again into two subgroups of 10 to perform the task script with the device fixed on the table and free in their hands. A ratio of 4 beginners to 4 intermediates to 2 experts for each subgroup was adopted. 4.3 Experimental Procedure Observation and retrospective audio/video analysis for quantitative and qualitative data were employed. Participants were required to provide written consent to be filmed prior to, during and immediately after the test sessions, and to permit the use of their images/sound for research purposes without limitation or additional compensation. On the other hand, the evaluation team was committed to do not disclose the user performance or other personal information. According to the approach basis, the first step consisted in defining the evaluation scope for the product as well as in designing a test task scenario, in which the target problems addressed were related to the: (i) shape/dimensions of the device; (ii) mechanisms for information input/output; (iii) processing power; (iv) navigation between functions; and (v) information legibility. Since the test objectives focused on (i) investigating the target problems; and (ii) detecting other problems which affect usability, a basic but representative set of test tasks was selected and implemented. The test tasks consisted of (i) initializing the device; (ii) searching for books in an online store; (iii) visualizing a PDF file; (iv) entering textual information; (v) using the e-mail; e (vi) using the audio player. After planning, 2 pilot tests (lab and field) were conducted to verify the adequacy of the experimental procedure, materials, and environment. Aiming to prevent user tiredness, the session time was limited to 60 minutes, and the test scenario was re-dimensioned to 6 tasks. Thus, each test session consisted of (i) introducing the user to the test environment by explaining the test purpose and procedure; (ii) applying the pre-test questionnaire (USer); (iii) performing the six-task script; (iv) applying the post-test questionnaire (USE); and (v) performing a non-structured interview. For the participants who declared not having had any previous contact with the Internet tablet, an introductory explanation about the device I/O modes and its main resources was given, considering that, at the time of the experiment, it was not yet widely spread in Brazil.

5 Results Conformity assessment results can be summarized by computing an Adherence Rating (AR), which is the percentage of the Applicable recommendations (Ar) that were Successfully adhered to (Sar) [17]. The results of the conformity assessment are summarized in Table 1. As it can be observed, all the ARs, excluding that one related to ISO 14745, are higher than 75%, which means successful results. As for ISO 14745, the result indicates the need to improve the input text via write recognition. Those results corroborate with the idea that the efficacy of standards inspection can be considerably improved if it is based upon standards conceived specifically for mobile devices, which could evidence more usability problems.

248

J.E.R. de Queiroz and D. de Sousa Ferreira Table 1. Nokia 770 conformity assessment with standards

STANDARD

#Sar

#Ar

AR (%)

ISO 9241 Part 14 ISO 9241 Part 16 ISO 9241 Part 17 ISO 14754 ISO 24755

45,0 26,0 47,0 4,0 6,0

53,0 33,0 52,0 11,0 7,0

84,9 78,8 90,4 36,4 85,7

As for the user subjective satisfaction measurement, both questions and answers of the post test questionnaire (USE) were previously configured. The questionnaire was applied soon after the usability test and answered using the mobile device itself, with the purpose to collect information on the user degree of satisfaction with the device by means of 38 questions about menu items, navigation cues, understandability of the messages, ease of use functions, I/O mechanisms, online help and printed manuals, users’ impression, and product acceptance level.. With the support of the pre test questionnaire (USer), the user sample profile was drawn. It was composed of 16 male and 24 female users, of which 16 were undergraduate students, 17 post-graduate students, 5 had graduate level and 2 had post-graduate level. The age varied between 18 and 29 years. They were mainly right handed and used some sort of reading aid (glasses or lenses). All of them had at least 1 year of previous experience with computer systems, were currently using computers on a daily basis, and had previous experience with mobile devices. The ranges for USE normalized user satisfaction are 0.67 to 1.00 (Extremely Satisfied), 0.33 to 0.66 (Very satisfied), 0.01 to 0.32 (Fairly satisfied), 0.00 (Neither satisfied nor unsatisfied), 0.01 to -0.32 (Fairly dissatisfied), -0.33 to -0.66 (Very dissatisfied), and -0.67 to -1.00 (Extremely dissatisfied). The normalized user satisfaction achieved was 0.330 (Very satisfied) for the lab experiment, and 0.237 (Fairly satisfied) for the field experiment . During the test sessions were identified 23 usability problems. 21 problems (91.3%) were detected in the lab experiment, while 14 ones (60.8%) were found in the field experiment . On the other hand, 12 problems (60.0%) were found with the device fixed on the table, while 15 (75.0%) were identified with the device free, in the user’s hands. Since the multidimensional approach is based upon the triangulation of results, Table 2 summarizes the usability problem categories which were identified during the evaluation process. For each category, the number of problems identified by each technique is given. As can be seen, some of the usability problem categories were more associated to the performance measurement (e.g. hardware aspects, help mechanisms) whereas others (e.g. menu navigation, presentation of menu options) were identified by the conformity assessment. The combination of the results from the post-test questionnaire to the comments made during the test sessions and the nonstructured interviews at the end of each experiment session showed that the user opinion was in agreement (e.g., location and sequence of menu options) or disagreement (e.g., menu navigation) with the results obtained from the other two evaluation techniques. This discrepancy can originate from the users’ perception of product quality, and from the perception of their own skills to perform the task.

A Multidimensional Approach

249

Table 2. Overlay of results obtained from different techniques described above PROBLEM CATEGORY Location and sequence of menu options Menu navigation Presentation of menu options Information feedback Object manipulation Symbols and icons Text entry via stylus (writing recognition) Text entry via virtual keyboard Processing power Hardware issues Fluent tasks execution Online and offline help Form manipulation

SI 9 (06) 9 (01) 9 (04) 9 (01) 9 (06) 9 (02) 9 (07)

9 (05)

Legend: SI – Standards Inspection PM – Performance Measurement SM – Satisfaction Measurement

PM 9 (01) 9 (01) 9 (02)

9 (01) 9 (04) 9 (05) 9 (06) 9 (03) 9 (01)

SM 9 (07) 8 (01) 9 (04) 8 (03) 9 (02) 9 (07) 8 (01) 9 (04) 8 (05) 9 (06) 9 (03)

9 Consistent findings 8 Contradictory findings

The statistic analysis consisted of: (1) building a report with univariate statistics; (2) generating the covariance matrices for the predefined objective and subjective indicators; (3) applying the one-way F ANOVA test to the data obtained from the previous step in order to investigate possible differences; and (4) applying the TukeyKramer process to the one-way F ANOVA results aiming to investigate if the found differences were statistically significant to support inferences from the selected sample.According to the results (see Table 3), the series of two-factor ANOVA involving Time, Errors (Incorrect actions, Incorrect choices, and Repeated errors), and Help accesses showed that the user experience level had a more significant effect on the number incorrect choices in the field experiment than in lab one. Pre and post-test questionnaire analysis and informal interviews results reinforced that domain knowledge and computer literacy have significant influence on user performance concerning the incidence of errors, both in lab and in the field. Table 3. Lab x Field and Fixed x Free experiment results

Lab

p-Value (α=0.05) Field Fixed

Free

Experience x Task Time

0.019

0.056

0.025

0.026

Experience x Incorrect Actions

0.003

0.003

0.001

0.043

Experience x Incorrect Choices

0.049

0.0006

0.164

0.270

Experience x Repeated Errors

0.017

0.127

0.194

0.133

Experience x Help Accesses

0.164

0.563

0.148

-

Variable Pair

250

J.E.R. de Queiroz and D. de Sousa Ferreira

6 Final Considerations Studies in the literature fit basically in two categories: (i) user mobility while using the device, inside of a lab or outdoors; and (ii) user distraction in pervasive computing. This study considered both aspects as part of the task context. In field test subjects were free to choose between moving or remaining still as they performed the task with the mobile device. The movement registered was limited to situations while the user waited for some device processing (e.g. web page downloads). During the field tests, while the user was moving, there was a clear interference of the environment on the user attention. Outdoors, in the ambient light, the device’s legibility was reduced/ aggravated by glare and reflections on the screen. Although the user’s opinion was that the camera apparatus did not interfere with the task execution the vast majority decided to lay the device down during task execution. Confirming previous findings, the experiments demonstrated that applications that require a lot of interaction and user attention are inappropriate for performing while walking due to attention distraction. This reinforces that, in spite of the mobility of the device targeted in this study, the evaluation settings did not need to differ substantially from the one employed in the evaluation of stationary devices, since the users tend not to wander while performing tasks that demand their attention. Until recently, studies have been published which deal with new paradigms and evaluation techniques for mobile devices. Few of the proposed new techniques are really innovative if compared to the ones traditionally employed. The data gathered and analyzed support the initial assumption that minor adaptations in the traditional evaluation techniques and respective settings are adequate to accommodate the evaluation of the category of mobile devices targeted by this study. The above comments corroborate with the views of the authors and [15] that of the laboratory and field evaluations do not diverge but are complimentary. As shown in this study they both add to the evaluation process, producing data that is significant to the process reinforcing the relevance of a multidimensional approach for the mobile device usability evaluation.

References 1. Rosson, M.B., Carroll, J.M.: Usability Engineering: Scenario-Based Development of Human-Computer Interaction. Academic Press, San Diego, CA (2002) 2. Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria for evaluating usability evaluation methods. IJHCI 15(1), 145–181 (2003) 3. Nielsen, J.: Usability engineering. Academic Press, Boston (1993) 4. Wixon, D., Wilson, C.: The usability engineering framework for product design and evaluation. In: Helander, M., Landauer, T.K., Prabhu, P. (eds.) Handbook of humancomputer interaction, 2nd edn., pp. 653–688. John Wiley and Sons, Chichester (1997) 5. Jones, M., Marsden, G.: Mobile Interaction Design. John Wiley and Sons, Inc., Chichester, West Sussex (2006) 6. Danielson, D.R.: Usability data quality. In: Ghaoui, C. (ed.) Encyclopedia of humancomputer interaction, pp. 661–667. Idea Group Reference (2006)

A Multidimensional Approach

251

7. ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. International Organization for Standardization, Geneva, Switzerland (1998) 8. Dumas, J.S., Loring, B.A.: Moderating Usability Tests: Principles and Practices for Interacting, illustrated edn. Morgan Kaufmann, San Francisco (2008) 9. Kjeldskov, J., Stage, J.: New techniques for usability evaluation of mobile systems. IJHCI 60(5-6), 599–620 (2004) 10. Ballard, B.: Designing the Mobile User Experience. John Wiley and Sons, Chichester (2007) 11. Goren-Bar, D., Graziola, I., Pianesi, F., Zancanaro, M., Rocchi, C.: Innovative Approaches for Evaluating Adaptive Mobile Museum Guides. In: Stock, O., Zancanaro, M. (eds.) PEACH - Intelligent Interfaces for Museum Visits, pp. 245–265. Springer, Heidelberg (2007) 12. Po, S., Howard, S., Vetere, F., Skov, M.B.: Heuristic evaluation and mobile usability: Bridging the realism gap. In: Proceedings of Mobile HCI, pp. 49–60 (2003) 13. Sanderson, P., Fisher, C.: Usability testing of mobile applications: A comparison between lab and field testing. Human-Computer Interaction 9, 251–317 (1994) 14. Omodei, M.A., Wearing, J., McLennan, J.P.: Head-mounted video and cued recall: A minimally reactive methodology for understanding, detecting and preventing error in the control of complex systems. In: Proceedings of 21th European Annual Conference of Human Decision Making and Control (2002) 15. de Queiroz, J.E.R.: Abordagem Híbrida para avaliação da usabilidade de interfaces com o usuário. Tese de Doutorado, UFPB, Brazil, p. 410 (2001) (in Portuguese) 16. Turnell, M.F.Q.V., de Queiroz, J.E.R., Ferreira, D.S.: Multilayered Approach to Evaluate Mobile User Interfaces. In: Lumsden, J. (ed.) Handbook of Research on User Interface Design and Evaluation for Mobile Technology, vol. 1, pp. 847–862. IGI Global (2008) 17. ISO 9241-14: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 14: Menu dialogues. ISO, Geneva, Switzerland (1997) 18. IS09241-16: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 16: Direct manipulation dialogues. ISO, Geneva, Switzerland (1999) 19. IS09241-17: Ergonomic requirements for office work with visual display terminals (VDTs) - Part 17: Form filling dialogues. ISO, Geneva, Switzerland (1998) 20. ISO/IEC 14754: Information technology - Pen-based interfaces - Common gestures for text editing with pen-based systems. ISO, Geneva, Switzerland (1999) 21. ISO/IEC 24755: Information technology - Screen icons and symbols for personal mobile communication devices. ISO, Geneva, Switzerland (2007) 22. Bailey, J.E., Pearson, S.W.: Development of a Tool for Measuring and Analyzing Computer User Satisfaction. Management Science 29(5), 530–545 (1983) 23. Aladwani, A.M., Palvia, P.C.: Developing and validating an instrument for measuring user-perceived Web quality. Information & Management 39, 467–476 (2002) 24. De Oliveira, R.C.L., de Queiroz, J.E.R., Vieira Turnell, M.F.Q.: WebQuest: A Configurable Web Tool to Prospect the User Profile and User Subjective Satisfaction. In: Salvendry, G. (ed.) Proceedings of the 2005 Human-Computer Interaction Conference, vol. 2, Lawrence Erlbaum Associates, Nevada (2005) (U.S. CD-ROM Multi Platform)

Development of Quantitative Usability Evaluation Method Shin’ichi Fukuzumi1, Teruya Ikegami1, and Hidehiko Okada2 1

NEC Corporation, Common Platform Software Research Laboratories, 7-1, Shiba 5-chome, Minato-ku, Tokyo 108-8001, Japan {s-fukuzumi@aj,t-ikegami@ct}.jp.nec.com 2 Kyoto Sangyo University, Kamigamo Motoyama, Kita-ku, Kyoto 603-8555, Japan [email protected]

Abstract. A variety of evaluation methods are practiced in order to make more appealing and improve the usability of computer systems. The authors have developed a quantitative usability evaluation method that uses a checklist that outlines an evaluation procedure and clarifies judging standards. This paper describes this quantitative usability evaluation method that is not influenced by an evaluator’s subjective impression. Moreover, such clear and precise definitions makes checklist-based evaluations more repeatable (thus more reliable) and less affected by differences among evaluators. The effectiveness of our checklist has been evaluated by the experiments with novice and experienced evaluators. This article reports the method and results of the experiments. Keywords: Usability, evaluation, checklist.

1 Introduction A usability evaluation method using a checklist [1], which is typical in usability evaluations [2], can be applied to the later stages of a development process [3]. However, obtaining justifiable evaluation results is difficult because they depend on the evaluators’ skill, experience, and subjectivity. To solve this problem, we have developed a usability checklist to minimize deviation of evaluation results for realizing usability quantification [4-5]. In this paper, we introduce this checklist, validate it, and apply it to system operation products.

2 Development of a Checklist 2.1 Problem of Usability Quantification and Approach for Its Solution Exclusion of an evaluator’s skill. Generally, a score of goodness of fit for an evaluation target is obtained by using a checklist evaluation method. However, when evaluating over five stages, the results will blur due to the subjectivity of evaluators [6]. Also, sometimes an evaluator cannot understand the meaning of the contents of the item and misinterprets it. For example, the names of UI parts, such as “a list box” and “a pull-down menu”, vary in meaning even among expert evaluators. Because of this, wrong items are chosen as evaluation targets. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 252–258, 2009. © Springer-Verlag Berlin Heidelberg 2009

Development of Quantitative Usability Evaluation Method

253

The authors made sure to judge each item as “has a problem”, “No problems” or “irrelevant” for a clarifying procedure, an evaluation target, and a gauge. Moreover, to stop evaluators’ different intelligibilities and interpretations causing results to blur, samples in a checklist and a collection of terminology definitions were also prepared. Visualization of the effect to the user. An evaluation axis often consists of an element directly connected with design and development, such as a layout, and a button because it is assumed that a specialist in UI design and a developer generally use a checklist, and the effect for the user is hard to determine. To measure the degree to which the effect of each item of the checklist satisfies the user, it is important to weight the qualities of each item. The authors weighted the items using the analytic hierarchy process (AHP) method [7], and evaluation results were decided on the basis of four qualities: “efficiency”, “ease to learn”, “errors”, and “ease to memorize”. 2.2 Maintenance of a Checklist Selection of the item. The authors referred to the various standards and guidelines, made a rough draft of the checklist, selected items, improved them through verifying them, and then evaluated them. Moreover the AHP method of weighting the items was contemplated, and the chapter construction of the item was decided so that no part of the layer became too deep (table 1). Table 1. Chapter of the checklist

1 2 3

Section name Consistency of indication/ operation Legibility of information Presentation of the present state

4 5

Conformability to the user/ environment Conformability to the work

Item

Procedure

Content

Fig. 1. Items of the checklist

The number of items 17 8 22 18 19

Target

Weight

254

S. Fukuzumi, T. Ikegami, and H. Okada

Procedure of an evaluation. This checklist consists of five sections and 84 items and is equipped with “evaluation procedure”, “evaluation target”, and “the weight (4 axes)” for every item (figure 1). The flow of the evaluation was procedure-ized, and a gauge and a result in each step were described clearly to make sure that evaluation results could be judged correctly for each designated evaluation target. When a button or pull down menu was right clicked and the appropriate operation was performed, the judgment result was “No problems”. When the appropriate operation was not performed, the result was “Has a problem”. Additionally, in case that necessity which fits in is low which it may occur problems not to satisfy the precondition and a presence of customization, it is judged “Irrelevant”. Figure 2 shows items of the checklist and a case by an illustration.

Evaluation item Evaluation target

Evaluation Procedure

Example

Fig. 2. Item of the checklist (details and example)

Development of Quantitative Usability Evaluation Method

255

Weight of item. To decide the weight of each item, the AHP method was applied. The feature of the AHP method is to apply paired comparison to an evaluation target according to some gauges. This method can calculate the high score of the validity compared with deciding about the weight of each element overall. Of the five special usability qualities Nielsen advocates to writers [1], the authors chose four: “ease to learn”, “errors”, “ease to memorize” and “efficiency” by a gauge and decided the weight of all of these qualities in each item. The value of a paired comparison of the item was decided by conference of 3 specialists of user interface.

3 Effectiveness Evaluation of Checklist 3.1 Experiment Method Evaluation targets. Three or five GUI windows of an e-mail software were selected as evaluation targets. Participants (Evaluators). In total, 50 people participated in this experiment. Of these, 30 were novice college students without much experience or understand of the software’s usability. On the other hand, the remaining 20 participants were expert evaluators who had experience and understanding of the software’s usability. They are the researchers who are emplyed by a company. Procedure. In accordance with the gauge described in section 2, participants evaluated some of the GUI windows prepared as evaluation targets, as explained above. By comparing novices and experts’ reported results, novice participants’ results can be verified to see whether it is possible to obtain results the same as those of experts. 3.2 Experimental Result Each evaluation result was judged as “Has a problem”, “No problems”, and “Irrelevant”. By comparing results, the possibility of both experts and novices obtaining the same results was tested. As an index the degree is identical between experts and novices’ results, concordance rate is defined as follows [4]. The concordance rate (%) = 100 * (number of novices whose results agreed with those of an expert (%))/(Number of novices) The average concordance rate obtained in this experiment was 73.75%. The average concordance rates of “Has a problem”, “No problems”, and “Irrelevant” are show in Table 2. Table 2. Average concordance rate

Evaluation results Has a problem No problems Irrelevant

Concordance rate 50.6 % 78.7% 80.6%

256

S. Fukuzumi, T. Ikegami, and H. Okada

3.3 Discussion As shown in Table 2, concordance rates of “No problems” and “Irrelevant” are relatively high while that of “Has a problem” is relatively low. When problems existed, it can be said that in a lot of cases novice evaluators overlooked the problem. However, the concordance rate of 50.6% means that the rate of “more than one person of inside of n persons is correspondence” is 1- (1-0.506) n, and big sufficiently with 94.0% at n=4, 87.9% at n=3. From this, a right result is expected to be obtained when more than three people estimate a result.

4 Practical Use of the Checklist This section describes the operation procedure of this checklist. 4.1 Application of the Evaluation Items An evaluator judges whether an evaluation target is described as “Has a problem”, “No problems”, or “Irrelevant” in accordance with an evaluation procedure. The goodness of fit of the evaluation result is calculated by the weight of the judgment result and the item. Next, the methods of judging and calculating the goodness of fit of the evaluation result are described. Judgment of a result. In the method for evaluating represented information and the consistency of operation, these items are applied to the whole screen. When they found a part of a screen with a problem by the item, evaluators judged it as “Has a problem”. If there was a problem only in a part of the screen among screen groups, even when being unified during other pictures, evaluators judged it as “Has a problem” (figure 3). Each screen and part that was an evaluation target was evaluated in terms of the items regardless of the consistency.

Without problem Consistency of Display / operation Layout of button

There is a problem There is a problem

Layout order of table / list

Fig. 3. Evaluation of consistency (the arrangement location of the button)

Calculation of the goodness of fit of the evaluation results. Even in the same item, there exist several evaluation targets that had different a judgment results, so goodness of fit of the evaluation results needs to be calculated by integrating the results. An

Development of Quantitative Usability Evaluation Method

257

evaluator passes a basic overall judgment by prioritizing “Has a problem”, over “No problems”, and then “Irrelevant”. For example, if one “Has a problem” is in the results, the overall judgment is also “Has a problem”. The item judged as “No problems” is weighted to check the goodness of fit of the evaluation results in accordance with the respective evaluation axes. 4.2 Calculation of an Evaluation Result The amount the results concur by the same weight of each quality is the overall score on the evaluation axes. Evaluation result examples for three similar products are indicated in figure 4. Since product A clearly has the highest efficiency and product B is obviously the easiest to learn, it is possible to grasp the special quality of each product. Thus it becomes possible to confirm from four angles the evaluation result about usability by using this checklist. Efficiency Product A Product B Product C

Little error

Easy to learn

Easy to memorize

Fig. 4. Example of Evaluation Result

5 Summary The authors have developed a usability quantification method in which a checklist is used that excludes blurring of results by detailing an evaluation target, an evaluation procedure, and acceptance standard. Even when an evaluator knows little about system usability, he or she can obtain objectives results on it by using this evaluation method.

References 1. Nielsen, J.: Usability Engineering. Academic Press, London (1993) 2. Ravden, S., Johnson, G.: Evaluating Usability of Human-Computer Interfaces: A Practical Method. Prentice-Hall, Englewood Cliffs (1989) 3. ISO 13407: Human-centred design processes for interactive systems (1999)

258

S. Fukuzumi, T. Ikegami, and H. Okada

4. Ikegami, T., Okada, H., Yoshizaka, S., Fukuzumi, S.: Proposal of usability quantification method (1) –checklist for exclusion blurring among evaluators. In: Annual conference of Information Processing Society Japan (2008) (in Japanese) 5. Okada, H., Ikegami, T., Yoshizaka, S., Fukuzumi, S.: Proposal of usability quantification method (2) –experiment for validation of a checklist. In: Annual conference of Information Processing Society Japan (2008) (in Japanese) 6. Kato, S., Horie, K., Ogawa, K., Kimura, S.: A Human Interface Design Checklist and Its Effectiveness. Transaction of Information Processing Society of Japan 36(1), 61–69 (1995) (in Japanese) 7. Ham, D.-H., Heo, J., Fossick, P., Wong, W., Park, S.-H., Song, C., Bradley, M.: Model-based approaches to quantifying the usability of mobile phones. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4551, pp. 288–297. Springer, Heidelberg (2007)

Reference Model for Quality Assurance of Speech Applications Cornelia Hipp and Matthias Peissner Fraunhofer Institute for Industrial Engineering (IAO), Nobelstr. 12, 70569 Stuttgart, Germany {cornelia.hipp,matthias.peissner}@iao.fraunhofer.de

Abstract. The acceptance of speech applications is still very low in Germany. The German market of speech industry identified this problem and makes an effort to improve the quality of speech applications, which should lead to higher user acceptance. To ensure higher quality standards, a reference model has been developed with special regard to the needs of interactive voice response systems (IVR). This model includes instructions to improve the process quality of the development process as well as methods, measurements and quality criteria to evaluate the product quality. Furthermore, the presented reference model differentiates between eight application types of IVR and describes which methods, measurements and quality criteria are especially important for each application type. Keywords: Quality, speech, interactive voice response, automatic speech recognition, measurement, method, voice, speech interaction, reference model, application type.

1 Introduction Speech dialogue systems are struggling with user acceptance. While interaction with mouse and keyboard has reached absolute standard usage, the potential of speech interaction is not adequately exploited yet. A survey made in Germany 2006, called “Acceptance of Speech Application” (German orig. “Akzeptanz von Sprachapplikationen”) [1], shows that the publicity of speech applications is continually increasing in Germany (70% of interviewees said 2006, they know speech applications, compared to only 54% in 2003), however the user satisfaction has not risen. Counterpart to these findings is the continual improvement in the technology for interactive voice response systems and long term predictions that advantages of speech control will be more appreciated [2]. Regarding the difference between user acceptance and potential (e.g. usage while being on the move or benefit for visual impaired people who have problems with graphical user interfaces) we assume that quality control and management is not mature yet within the speech application industry. We believe that an improvement of quality assurance would systematically optimize interactive voice response systems and would improve product image, user experience and call frequency of speech applications. After research, we recognized that there are several solutions for quality J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 259–266, 2009. © Springer-Verlag Berlin Heidelberg 2009

260

C. Hipp and M. Peissner

assurance within the areas of software management [3][4], classical engineering and usability engineering [5][6]. Additionally there are first solutions for evaluating speech applications [7][8] and guidelines to improve the usability of speech applications [9]. But there is a lack of knowledge regarding the systematic improvement in quality of speech applications within an adequate reference model. There are ambitions of the German Initiative Voice Business with their yearly congress Voice Days and their Voice Award which is awarded to best German-based speech application [10]. Although the used method of testing is a solid basis for quality measurement, it is not usable within development projects as a model for continuously controlling and optimising the quality of speech applications.

2 Intention For the reasons mentioned above, we aimed to find a suitable approach for measuring and improving the quality of speech applications. We realized quickly that a holistic approach to improve the dialogue and the user experience of interactive voice response systems is needed. Different factors within the development of speech applications are affecting the quality of the IVR and they are dependent on each other. Within this paper we describe a new holistic reference model for quality assurance of speech applications. The model has been developed at Fraunhofer Institute for Industrial Engineering (IAO) Germany in cooperation with the Initiative Voice Business (IVB) and with exchange of 36 experts within the German speech application market. Additionally, the efforts made were supposed to encourage the voice industry on the German-speaking market and to find solutions which are industry-orientated and easy to transfer to concrete projects. The work is supposed to sharpen the public awareness of the subject of quality in speech applications and to show possibilities of improving the quality.

3 Description of Reference Model First, we adopted a common approach within the software development, the differentiation between product quality and process quality [4]. Product quality refers to the product itself, in this case the IVR. Process quality in contrast refers to rules, strategies and requirements of the development process. Securing a high-quality process doesn’t assure a high quality of the end product but chances are very good, that it is highly qualitative. Therefore in this holistic model, actions are described to monitor the process development as well as to continuously evaluate the product quality. For the product quality, we identified ten quality criteria explicitly for speech applications [11]. These ten criteria describe a good speech application in a holistic way so that criteria are defined within the areas voice user interface and usability as well as strategy and business logic, dialogue platform and integration and speech technology and linguistics. But the quality criteria are not assigned precisely to one specific area and can affect several.

Reference Model for Quality Assurance of Speech Applications

261

Fig. 1. Overview of the Presented Reference Model with Components and their Dependencies

The ten quality criteria can be evaluated with the help of measurements. In total, there are 34 measurements defined to the special needs of IVRs, like e.g. caller frequency or no match rates. Accordant to the quality criteria, the measurements are defined for the four different areas and do not have to be assigned to only one area. The measurements are used to supply concrete values, which can be used for comparison, either between different speech applications or different versions/ development stages of one application. With the help of methods, the described measurements can be performed to achieve concrete values. In total, there have been 23 methods identified with special regard to speech applications, like load test, wizard of oz test and expert evaluation. The methods are important elements within the reference model for the process quality. They are assigned to different steps of the process, which are differentiated between project preparation and analysis, concept and design, implementation, integration and bringing into service and operation. Methods and measurements are supposed to be used in an iterative process to control and achieve good performance.

262

C. Hipp and M. Peissner

Furthermore, quality criteria, measurements and methods have different weight for different application types. Therefore eight different application types have been identified [12] with partly very different priorities for criteria, measurements and methods. In the first step, when using the reference model, it has to be identified which of the defined application types is going to be implemented. Subsequently, important quality criteria can be looked up and then, which measurements are meaningful for the particular criteria. After that, it can be looked up, which methods are usable in order to get results for the measurements. 3.1 Quality Criteria Within the introduced reference model ten quality criteria are defined to show what applications have to achieve to be considered as a good IVR. They are described with regard to the holistic approach of the reference model and cover the four mentioned predefined themes voice user interface and usability, strategy and business logic, dialogue platform and integration and speech technology and linguistics. With the aid of measurements, data can be collected to check whether the quality criteria are achieved. Following the ten quality criteria are listed: Appropriate Functionality Coverage and Content Offering: A speech application is good, if an added value is created for the costumer by means of an attractive and complete offer of functionality. Faultless Operability and Capability: A speech application is good, if a secure and faultless functioning with high performance is assured – on peak loads as well. Administrability and Efficient Operations: A speech application is good, if technical efforts after launching the IVR can be kept at a minimum. Expandability and Scalability: A speech application is good, if the system architecture easily allows future enhancements and changes. Profitability: A speech application is good, if the service is economical profitable. Reliable Recognition of User Utterances: A speech application is good, if speech recognition reliably recognizes an appropriate amount of prospective user utterances. Effective Management of Errors: A speech application is good, if recognition errors and errors of usage do not cause major damage. Effective and Flexible Dialogue Flow: A speech application is good, if the navigation structure supports users to reach their aim fast and secure. Comprehensible and Goal-Orientated System Output: A speech application is good, if the acoustic system output supports the user within orientation and in formulating goal-orientated utterances. Impression and emotional addressing: A speech application is good, if a positive and appropriate attitude of the user towards the speech application, its use and its operator can be reached.

Reference Model for Quality Assurance of Speech Applications

263

3.2 Measurements Within the presented reference model, 32 measurements have been worked out. With their aid, data can be collected in order to identify whether a quality criteria is fulfilled or not. The measurements are performed by means of the defined methods. Measurements are defined based on the following characteristics: name, synonyms, brief description, reference to quality criteria, reference to application components, actual use in practice, usable methods for collecting data for this measurement, appraisal of profitability. As an example, the measurement routing rate is displayed in the following: Name: routing rate Synonyms: correct routing rate Brief Description: percentage of calls which can be successfully transferred (according to the wishes of the costumer) in proportion to the total amount of callers Reference to Quality Criteria: faultless operability and capability, profitability, reliable recognition of user utterances, effective management of errors, impression and emotional addressing Reference to Application Components: model, view, control and access – an optimal co-operation between all components is necessary (referring to the architectural pattern model-view-controller, enhanced with the component access) Actual Use in Practice: frequently – comment: measurement is only relevant for applications where routing has high importance Usable Methods for Collecting Data for this Measurement: logfile analysis and reporting Appraisal of Profitability: very high 3.3 Methods 23 different methods are defined within this reference model for quality assurance of speech applications. They are attached to specific process steps, but do not necessarily have to be attached only to one process step. By means of the methods, data can be collected for different measurements. Subsequently, with the aid of the measurements, quality criteria can be evaluated. Within the reference model, differentiation has been done between methods which should be necessarily done and methods which should be applied to achieve an excellent process. This classification differs between the eight application types, listed later on. Methods are defined based on the following characteristics: name, synonyms, brief description, which measurements can be covered or optimized, reference to quality criteria, reference to application components, relevance to thematic areas, reference to process steps, maturity of method, actual use in practice, potential for use in practice, requirements for this method, appraisal of profitability As an example, the method in-service test is displayed subsequently: Name: in-service test Synonyms: watchdog test, keep-alive test, availability test

264

C. Hipp and M. Peissner

Brief Description: The in-service test is a permanent test regarding the availability and functionality of a speech portal in operation. The system generates external controlling calls cyclically over a long timeframe. Time intervals, type of calls and test scripts are freely selectable. Which Measurements can be Covered or Optimised: service availability, service accessibility, answering time for costumer, correctness of system output Reference to Quality Criteria: faultless operability and capability, profitability Reference to Application Components: access – comment: affected are the telephone functionalities (referring to the architectural pattern model-view-controller, enhanced with the component access) Relevance to Thematic Areas: dialogue-platforms and integration Reference to Process Steps: in operation Maturity of Method: high Actual Use in Practice: occasional/ seldom Potential for Use in Practice: high Requirements for this Method: test equipment, test services Appraisal of Profitability: good – minor costs compared to high benefit 3.4 Product Quality The final aim of the reference model is to reach a high quality of the final product of the IVR. This can be achieved with the help of high process quality and checked with the ten defined quality criteria. 3.5 Process Quality The described reference model differentiates between process quality and product quality. While product quality focuses on evaluation and optimisation of the final result (the IVR) the process quality focuses on the process while developing the product. The notion that improving the process quality will lead subsequently to an improvement of the product quality is the reason why process quality has an important part in the reference model. The process steps which are differentiated within the development of a speech application are defined as project preparation and analysis, concept and design, implementation, integration and bringing into service and operation. These process steps are not strictly separated and should be seen as an iterative process. Additional evaluation should be done through the whole process and not included at a specific time-frame of the development. Within the reference model, methods are defined which should be carried out at specific process steps to ensure a high quality of the process. Furthermore, the model discriminates between an ensurance of a minimum of process quality and the achievement of an excellent process. Therefore, it defines methods which should be necessarily done for the first and lists additional methods for the latter. For instance, it is necessarily recommended to do functional tests during the implementation phase of self service portals. To achieve an excellent process, it is recommended in addition to already do a friendly user test during this phase.

Reference Model for Quality Assurance of Speech Applications

265

3.6 Application Types Measurements, methods and quality criteria are differently useful and meaningful for different application types. E.g. the quality criterion impression and emotional addressing is very important for marketing applications, but not that much for authentication services. Therefore the reference model does not allocate data to specific measurements for all of the speech applications, but differentiates between unequal application types. With the help of this discrimination, it is possible to compare results between applications within the same application type and identify potentials and weaknesses of them. Within the reference model, the following eight application types are defined, with different benchmarks for quality criteria, measurements, methods and process steps: Call Routing: Incoming calls are sorted thematically, and are transferred to the correct person in charge within the call center. Information Service: Costumer can receive information inexpensive, swiftly and upto-date via speech-based information service. Reminding and Alerting Service: Alerting services trigger automatic calls in the event of an emergency (e.g. catastrophes like earthquakes, hurricanes). Reminding services are calling in case of pre-defined important appointments (e.g. delivery dates or taking medication). Authentication Service: Speech application verifies whether caller is in fact the person he is declaring to be based on the unique characteristics of the human voice. Automated Telephone Switchboard: In case of absence of the callee, the automated telephone switchboard can e.g. transfer the call to a colleague or start an answering machine. Track & Trace-System: With the aid of track & trace-systems, companies can permanently provide information of the actual state of their services. Marketing Application: Companies can use IVR for marketing purposes, like lottery, advertisement or usage of voices of prominent people. Self Service Portal: Costumers can execute different transactions and employ information services using self service portals by themselves.

4 Concluding Remarks In Germany, the need for improving the quality of speech applications has been realized, due to the reason that German-speaking voice industry has an issue with the user acceptance in IVR. The potential of speech interaction is not adequately exploited yet and several German speech companies are working together to find consolidated solutions. Quality criteria, methods and measurement have already been defined with special regards of eight application types. But there are still open questions, e.g. how to compare applications within the same type, but with different degree of complexity. Furthermore, the ongoing work should conclude in an acknowledged standard to sensitize customers to quality differences in of speech applications.

266

C. Hipp and M. Peissner

Acknowledgements. Ongoing work to find solutions for quality of speech applications is done with great support of 36 experts in Germany. We would like to thank the following companies and persons: Cirquent (Dr. Bettina Attallah), D+S solutions (Kerstin Sehnert), E.ON Hanse (Frank Oldorf), Genesys Telecommunications Laboratories (Giancarlo Boi), HFN Medien (Dr. Frank Wanning), IBM Germany Research and Development (Ludovica De Sio, Dr. Carsten Günther, Dr. Marion Mast), mind Business Consultants (Sebastian Paulke, Bernhard Steimel), NEXT ID (Ralf Poplawski), Nortel (Dr. Oliver Huber), SemanticEdge (Jörn Kreutel, Dr. Lupo Pape), Sikom Software (Jürgen Hoffmeister, Dietmar Kneidl), Sparda-Bank Hamburg eG (Jürgen Mehring), SpeechConcept (Dr. Uwe Lay), Strateco (Mark Gutmann), Sympalog Voice Solutions (Dr. Jürgen Haas), tech2biz (Dr. Christian Dugast), Deutsche Telekom Laboratories (Caroline Clemens, Dr. Florian Metze, Prof. Dr. Sebastian Möller, Wiebke Johannsen), Telenet Communication Systems (Dr. Florian Hilger, Markus Kesting), T-Mobile Deutschland (Dr. Guntbert Markefka), T-Systems Enterprise Services (Frank Oberle), Unisys Deutschland (Andreas Schaub), VMA (Dr. Guntbert Markefka, Andreas Schaub, Dr. Frank Wanning), voiceandvision (Tom Houwing), Voice & Visual Design (Paul Hubert Vossen), 4Com (Dennis Jehne).

References 1. Peissner, M., Sell, D., Steimel, B.: Acceptance of Speech Applications (German orig. Akzeptanz von Sprachapplikationen). Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart (2006) 2. Wu, E.: Bill Gates predicts software revolution. MIS ASIA (August 14, 2008), http://mis-asia.com/news/articles/ bill-gates-predicts-software-revolution 3. Sommerville, I.: Software Engineering. Pearson Education Germany GmbH, Munich (2001) 4. Ludewig, J., Licher, H.: Software Engineering. dpunkt.verlag, Heidelberg (2007) 5. Nielsen, J.: Usability Engineering. Academic Press, San Diego (1993) 6. Mayhew, D.: The Usability Engineering Lifecycle: A Practitioner’s Handbook for User Interface Design. Academic Press, San Diego (1999) 7. Dybkjaer, L., Hemsen, H., Minker, W.: Evaluation of Text and Speech Systems. Springer, Dordrecht (2007) 8. Möller, S.: Quality of Telephone-Based Spoken Dialogue Systems. Springer Science + Business Media, New York (2005) 9. Hempel, T. (ed.): Usability of Speech Dialog Systems. Springer, Berlin (2008) 10. Initiative Voice Business, http://www.voice-award.de 11. Peissner, M., Hipp, C., Steimel, B.: Quality Criteria, Measurements and Methods for Speech Applications (German orig. Qualitätskriterien, Maße und Verfahren für Sprachapplikationen). Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart (2007) 12. Hipp, C., Paulke, S., Peissner, M., Steimel, B.: Quality Guideline – Cookbook for Good Speech Applications (German orig. Qualitätsleitfaden – Kochbuch für gute Sprachapplikationen). Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart (2008)

Toward Cognitive Modeling for Predicting Usability Bonnie E. John1 and Shunsuke Suzuki2 1

Human-Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Ave. Pittsburgh, PA, 15213, USA [email protected] 2 NEC Corporation, 8916-47, Takayama-cho, Ikoma, Nara 630-0101, Japan [email protected]

Abstract. Historically, predictive human performance modeling has been successful at predicting the task execution time of skilled users on a desktop computer. More recent work has predicted novice behavior in web searches. This paper reports on a collaborative effort between industry and academia to expand the scope of predictive modeling to the mobile phone domain, both skilled and novice behavior, and how human performance relates to the perception of usability. Since, at this writing, only preliminary results to validate models of mobile phone use are in, we describe the process we will use to progress towards our modeling goals. Keywords: Cognitive modeling, GOMS, KLM, CogTool, Information Foraging.

1 Introduction Predictive human performance modeling has been an HCI “holy grail” for decades. If the field had a computational model of a human that could perform like a human (including perception, cognition and motor action), make errors like a human, learn like a human, and experience emotions like a human, then we could test our design ideas as they emerge in the design process, quickly and inexpensively. Just as an automotive crash dummy saves consumers from poorly designed vehicles, a cognitive crash dummy could expose design ideas that would harm, or at least annoy, users before they were brought to market. This would save companies substantial investment in building and marketing products that people would have a hard time learning or using or would not enjoy. The first model to make progress toward these goals was Card, Moran and Newell’s Model Human Processor in 1983 [1]; researchers have been making slow but steady progress since then. The most reliable models today predict task execution time of skilled users. Models such as Keystroke-Level Model (KLM; [2]) and GOMS (Goals, Operators, Methods, and Selection rules; [3]) have been validated in over 100 research papers by scores of authors, primarily on tasks performed on a desktop computer. Information Foraging Theory [4], a more recent entry into the realm of computational predictive modeling, predicts exploratory behavior of people searching for information. It has been operationalized in an ACT-R [5] model called SNF-ACT [6] and validated against human data on web search tasks. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 267–276, 2009. © Springer-Verlag Berlin Heidelberg 2009

268

B.E. John and S. Suzuki

Despite research progress creating and validating theory, UI developers have not adopted predictive human performance modeling as a frequently-used tool for design. Recent work has embodied these theories into tools that allow practicing developers to achieve the benefits of modeling without investing considerable time in learning to model and constructing each new model (e.g., [7, 8, 9]). However, it is difficult to make a trustworthy tool for practical design problems. This paper explains the process of doing so in the context of collaborative research between NEC, PARC and Carnegie Mellon University. Our project is aimed at producing a tool for predicting task execution time of skilled users, novice exploration to accomplish a goal, and subjective perception of the usability of mobile phones.

2 The Process of Making a Trustworthy, Practical Tool for Design The process of making a trustworthy, practical tool for design is shown in Figure 1. Each time a new domain is entered, or a new metric is added, the theory, tool and models must be validated with data from appropriate users to produce a trustworthy tool for prediction. If the models’ predictions do not match the human data sufficiently, either the theory or the tool, or both, must be revised until valid predictions are produced. The next question is whether the tool is learnable and usable by UI designers in their work process. User-centered design techniques should be used to design, evaluate, and redesign, until the tool is practical for design.

Fig. 1. General process of human performance modeling research that leads to a practical tool for design

Our project started with CogTool, a tool that allows UI designers to create valid Keystroke-Level Models in one tenth the time of doing them by hand as originally demonstrated by Card, Moran and Newell [2]. It has been shown to be easily learnable by users with no background in psychology or cognitive modeling ([8] and

Toward Cognitive Modeling for Predicting Usability

269

tutorials at professional conferences like HFES, BRIMS, and HCII). Recent research with CogTool has extended it beyond KLM and predictions of skilled task execution time to information foraging theory and predictions of novice exploration behavior [10, 11]. From this starting point, we set out to expand CogTool’s ability to predict human behavior to a new domain, mobile phones, and to a new metric, subjective impressions of usability as measured by the Mobile Phone Usability Questionnaire (MPUQ) developed by Ryu [12, 13]. Thus, this project will touch all the points in Figure 1. We start by using CogTool as it exists to make predictions of skilled execution time and novice exploration behavior and test those predictions against human data on mobile phones, fully expecting that adjustments to the underlying theory and tool will need to be made. After making changes to the theory and tool to produce valid predictions of these metrics, we intend to correlate various aspects of the predictions with people’s perceptions of usability. After verifying that we can make trustworthy predictions, we will determine whether CogTool can be used by mobile phone designers and adjust CogTool’s UI until it becomes a practical tool. At this writing, we are at the first part of the process, making predictions with CogTool as it exists and comparing those predictions to human data. The remainder of this paper will describe the current state of the research.

3 The New Domain – Mobile Phones Mobile phones were chosen as the domain in which to pursue this approach. This product category is important to the corporation and the discrete nature of the tasks users perform on mobile phones makes them relatively easy for collecting human data and to model. In addition, CogTool had previously been shows to make good predictions of skilled use of a similar hand-held device (PDAs, [14, 15]). Although the project will evaluate several different mobile phones, this paper will use the N905i, shown in Figure 2, as an example of our research process. The tasks we are examining are varied, as follows. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Call a number from a phone book Store a number into a phone book Put an event into a Schedule Change a Security Setting View a previously sent mail message Set a previously stored picture to be the wallpaper. Delete a previously stored picture Add a function into a shortcut Check memory info Shoot a movie, check it, and save it

Data was collected from skilled users who had owned their phones for at east two months and from novice users who had never used this model of phone. The phone screen was captured on video, which was later transcribed to identify which buttons

270

B.E. John and S. Suzuki

Fig. 2. N905i mobile phone shown at the screen that was the start for each task

were pressed, when each button was pressed, and how long it took the phone to respond to each button press (system response time).

4 CogTool and Initial Models CogTool is a prototyping and cognitive modeling tool created to allow UI designers to rapidly evaluate their design ideas. A design idea is represented as a storyboard (inspired by the DENIM Project [16]), with each state of the interface represented as a node and each action on the interface (e.g., button presses) represented as a transition between the nodes. Figure 3 shows the start state of the storyboard, where buttons are placed on top of an image of the phone. Figure 4 shows a storyboard for six instances of the first task, calling a person who is already listed in the phone’s contact list. The first action at the start state is to press the down button called out in Figure 3. Because different contacts are located at different points of the phone book, the task takes different paths from the start screen to completion of the task. We will use Calling Person4 as the example in the remainder of this paper. After creating the storyboard, the next step is to demonstrate the correct actions to do the task. CogTool automatically builds a valid Keystroke Level Model from this demonstration. It creates ACT-R code that implements the Keystroke-Level Model and runs that code, producing a quantitative estimate of skilled execution time and a visualization of what ACT-R was doing at each moment to produce that estimate (Figure 5). Since mobile phones are a new domain for CogTool, we do not expect that the predictions it makes “out of the box” will be very accurate when compared to human data. We expect to have several iterations of comparing the predictions to human data and fixing the underlying theory and CogTool’s implementation of that theory, before we can make trustworthy predictions to help design. The next section presents preliminary analysis of one such iteration.

Toward Cognitive Modeling for Predicting Usability

271

Each Button in the picture of the phone has a button “widget” drawn on top. Actions on widgets follow transitions as defined in the storyboard (Fig 4).

If you tap the down button, CogTool transitions to the next frame in the storyboard (Fig. 4)

Fig. 3. Start screen of the CogTool prototype

Start screen

Calling Person6

Calling Person5

Calling Person1

Calling Person2

Calling Person3

Calling Person4

Fig. 4. Storyboard of the screens a person would pass through to accomplish Task 1 (making a call from the phone book) for six different instances of the task, i.e., calling six difference people in the phone book. We will use the instance of Calling Person4 as an example throughout this paper.

272

B.E. John and S. Suzuki

4.1 Comparing Initial Models to Human Performance Data The first step in comparing human performance data to the predictions of models is to make sure the same metrics are used in both the data and the models. For example, CogTool models predict not only when a button will be pressed, but also the thinking time and visual perception that precedes pressing the button. Only button presses were recorded in the empirical study, so we cannot directly compare the “total” time predicted by the model against the “total” time observed in the experiment. Adjusting for this difference, and comparing the time from first key press to the appearance of “Calling Person4” on the screen, the CogTool model predicted 11.049 seconds. The mean of five skilled participants was 9.770 seconds, an over-prediction of the average by 13% and an average absolute percent error of 15% between the predicted time and each observed time. This level of prediction is within the 20% error typically claimed by KLM and is an excellent prediction for an initial foray into a new domain and device. The next step is to go to a deeper level of comparison and look at the predictions for each individual action. We expect the quantitative comparisons to get worse, as explained by Card, Moran and Newell [1], but we are looking for patterns in behavior at this point, not absolute quantitative match. The first types of patterns we hope to see are those predicted by the model. Consider Figure 5, a timeline of the model’s predictions for the Calling Person4 task provided by a CogTool as a visualization of its behavior. The rows in the timeline represent different types of actions in the model. The changes in the phone’s screens are on the top gray line, with the estimates of system wait time (between button press and when the screen can be read) in the second row (light gray). The three purple rows show activity associated with vision: Vision-Encoding, Eye Move–Execute and Eye Move–Preparation. The central gray row represents the cognition that controls behavior, both the long “Mental operators” empirically established by Card, Moran and Newell, and the short ACT-R cognitive acts that control vision and hand motions. The bottom red row shows the button presses, in this case, with the right hand (the thumb). The model predicts a pattern: 1 press (at time=0), pause, 6 presses, pause, 8 presses, pause, 1 press.

Fig. 5. Timeline of a CogTool model prediction

Consider Figure 6, where the data from five participants are placed below the model’s timeline, aligned so that their first key presses all start at 0.0 sec. The top four participants display a pattern in keeping with the model’s prediction (1 press, pause, 6 presses, pause, 8 presses, pause, 1 press), except for P9 who does not pause for long

Toward Cognitive Modeling for Predicting Usability

273

before the last key. However, the bottom participant, P1 does not show this pattern at all. When we went back to the video of this participant, we found that although P1 used the same number of keys to complete the task, he did not use the same keys as the other participants or the model. Further investigation is needed to understand whether this was due to an error or whether it represents an alternative correct method for this task. Either way, in the majority of cases of this small sample, CogTool automatically predicted a pattern of behavior that was observed in human performance, even without modifying CogTool for the mobile phone domain.

Fig. 6. Timeline of a CogTool model prediction with keypress data from five participants aligned below it

Looking more closely at the data of the people who used the same keystrokes as the model (the top four), another pattern can be seen, one not predicted by the unmodified CogTool. Each participant shows two grouping of keys pressed close together in time, one of six keys in the beginning of the task and one of eight keys at the end of the task. Of these eight groupings, six show a distinct pause before the last keystroke in the group (P6, 1st group; P9, both groups; P12, 2nd group; P15, both groups). The groupings of six are repeated pressing of the S3 key to move across a set of icons at the top of the screen, some of which drop down a list of items that can be selected. When the desired icon is reached and its list drops down, the user then hits the Down key eight time to move down to the desired contact and hits the Call button to complete the task. The pauses come before the last S3 key press and the last Down

274

B.E. John and S. Suzuki

key press. In both cases, the user is watching a highlight move across (or down) the screen and can anticipate when the next key press will bring the highlight to the desired item. The pause before the last key press might represent a strategy to avoid over-shooting. This monitoring activity was not included in the original systems tested by Card, Moran and Newell and therefore is not represented in the original Keystroke-Level Model. Thus, we have identified a case where we may need to develop new theory about monitoring and anticipatory keystrokes (i.e., iterate on the theory) and build it into the tool (i.e., iterate on the tool) before we can produce trustworthy predictions in this domain. Another case where the predictions do not match the data is in the inter-keystroke times. All four users who did the task in the same way as the model pressed the same key far faster than CogTool did, as seen by the denser grouping of keystrokes in the participants’ timelines than in the model timeline. In this case, we will have to iterate on the underlying theory of motor movement to allow it to produce faster keystrokes. The timeline shows us that CogTool inserts visual perception of a key between each keystroke, which likely to be wrong for repeated keystrokes, especially given the monitoring activity described above where the user’s eyes are presumably on the screen not the buttons. With a model of just one instance of one task and data from five participants, the timeline visualization has suggested that the model is making reasonable predictions of the grouping of actions but is missing some important patterns of human behavior. More tasks and more data will have to be analyzed to be sure it is necessary to change the underlying theory and build it into CogTool to get trustworthy predictions. However, this small example illustrates the process of model validation this project has undertaken.

5 Future Work In addition to following the process in Figure 1 for skilled task execution time predictions on mobile phones, this project will also examine the prediction of novice exploration behavior, with CogTool-Explorer ([10, 11], a version of CogTool that predicts novice behavior). As with skilled behavior, we do not expect CogTool-Explorer to be able to predict a new domain (mobile phones instead of web searches) in a new language (Japanese instead of English) without iteration on the theory and tool. We have already identified improvements to the tool required for mobile phones, for example, mobile phones have “soft keys” where the label of the key is displayed on the screen instead of being printed on the key and CogTool-Explorer was not originally designed to represent that relationship. Perhaps more interestingly, when we have succeeded in producing trustworthy predictions of behavior, we intend to correlate this behavior with subjective impressions of usability as measured by the Mobile Phone Usability Questionnaire (MPUQ) developed by Ryu [12, 13]. Unlike empirical methods that can only correlate observed behavior, like time on task or number of errors, with questionnaire results, we can extract much more varied metrics from the models against which to correlate

Toward Cognitive Modeling for Predicting Usability

275

subjective impressions. For example, total time on task may not correlate with subjective impressions, but time spent in cognition may. Or more complex measures may be needed, like time spent in cognition that is not in parallel with motor movements for skilled users. Or number of keys looked at by CogTool-Explorer before making a choice, for the subjective impressions of novice users. Or amount of system response time not in parallel with cognition (i.e., making the user wait). Because CogTool produces a process model of perception, cognition and motor actions necessary to do a task, many combinations of actions can be explored to see if any can explain a significant part of the variance in subjective impressions. If a significant correlation can be found, then the predictive human performance models will be extended to a subjective metric, moving the field closer to the holy grail.

References 1. Card, S.K., Moran, T.P., Newell, A.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale (1983) 2. Card, S.K., Moran, T.P., Newell, A.: The Keystroke-Level Model for User Performance Time with Interactive Systems. Commun. ACM 23(7), 396–410 (1980) 3. Card, S.K., Moran, T.P., Newell, A.: Computer Text-Editing: An Information-Processing Analysis of a Routine Cognitive Skill. Cognitive Psychology 12, 32–74 (1980) 4. Pirolli, P., Card, S.: Information Foraging in Information Access Environments. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 1995), pp. 51–58. ACM Press/Addison-Wesley Publishing Co., New York (1995) 5. Anderson, J.R., Bothell, D., Byrne, M.D., Douglass, S., Lebiere, C., Qin, Y.: An Integrated Theory of the Mind. Psychological Review 111(4), 1036–1060 (2004) 6. Fu, W.-T., Pirolli, P.: SNIF-ACT: A Cognitive Model of User Navigation on the World Wide Web. Human-Computer Interaction 22, 355–412 (2007) 7. Blackmon, M.H., Kitajima, M., Polson, P.G.: Tool for Accurately Predicting Website Navigation Problems, Non-Problems, Problem Severity, and Effectiveness of Repairs. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI 2005, pp. 31–40. ACM, New York (2005) 8. John, B.E., Prevas, K., Salvucci, D.D., Koedinger, K.: Predictive Human Performance Modeling Made Easy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2004, pp. 455–462. ACM, New York (2004) 9. Wu, C., Liu, Y.: Usability Makeover of a Cognitive Modeling Tool. Ergonomics in Design 15(2), 8–14 (2007) 10. Teo, L., John, B.E.: Towards Predicting User Interaction with CogTool-Explorer. In: Proceedings of the Human Factors and Ergonomics Society 52nd Annual Meeting, HFES, pp. 950–954, Santa Monica (2008) 11. Teo, L., John, B.E., Pirolli, P.: Towards a Tool for Predicting User Exploration. In: CHI 2007 Extended Abstracts on Human Factors in Computing Systems, CHI 2007, pp. 2687– 2692. ACM, New York (2007) 12. Ryu, Y.S.: Development of Usability Questionnaires for Electronic Mobile Products and Decision Making Methods, Doctoral dissertation, State University, Blacksburg, VA, USA (2005) 13. Ryu, Y.S., Smith-Jackson, T.L.: Reliability and Validity of Mobile Phone Usability Questionnaire (MPUQ). Journal of Usability Studies 2(1), 39–53 (2006)

276

B.E. John and S. Suzuki

14. Luo, L., John, B.E.: Predicting Task Execution Time on Handheld Devices Using the Keystroke-Level Model. In: Proceedings of the International Conference on Human Factors in Computing System (CHI 2005), pp. 1605–1608. ACM Press/Addison-Wesley Publishing Co., New York (2005) 15. Luo, L., Siewiorek, D.P.: KLEM: A Method for Predicting User Interaction Time and System Energy Consumption during Application Design. In: Proceedings of the 11th International Symposium on Wearable Computers (ISWC 2007), pp. 69–76. IEEE Press, New York (2007) 16. Lin, J., Newman, M.W., Hong, J., Landay, J.A.: Denim: An Informal Tool for Early Stage Web Site Design. In: CHI 2001 Extended Abstracts on Human Factors in Computing Systems (CHI 2001), pp. 205–206. ACM, New York (2001)

Webjig: An Automated User Data Collection System for Website Usability Evaluation Mikio Kiura, Masao Ohira, and Ken-ichi Matsumoto Graduate School of Information Science, Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara, Japan {mikio-k,masao,matumoto}@is.naist.jp

Abstract. In order to improve website usability, it is important for developers to understand how users access websites. In this paper, we present Webjig, which is a support system for website usability evaluation in order to resolve the problems associated with the existing systems. Webjig can collect users’ interaction data from static and dynamic websites. Moreover, by using Webjig, developers can precisely identify users’ activities on websites. By performing an experiment to evaluate the usefulness of Webjig, we have confirmed that developers could effectively improve website usability. Keywords: Web usability, usability evaluation, analysis of user interactions, dynamic websites.

1 Introduction It has been found that there are various benefits associated with improving website usability, and so, in recent times, there have been increasing interests on website usability. For instance, in the case of an EC (electronic commerce) site, website usability has an impact on conversion rates. In the case of a website used in workplaces, website usability can also affect the work efficiency. In addition, the cost of user support can be reduced by improving website usability. It also influences a company’s image. Developers must understand how users access a website in order to improve website usability [1]. Usability testing is widely used for understanding users’ interactions on a website [2]. In usability testing, users perform some specific tasks within a set time under the supervision of usability engineers (or experts). The engineers observe how the users follow certain steps to accomplish the tasks. Through usability testing, developers can presume users’ intellectual process and observe their interaction; these developers can identify problems, clarify design issues, or come up with new ideas. Although usability testing is a prevalent approach for improving website usability, it cannot be applied to every website. Usability testing requires stakeholders (e.g., users, developers, and experts) to spend large amounts of time and money. Developers cannot conduct usability testing easily [3]. So far, several systems [4, 5, 6, and 7] for website usability evaluation have been proposed so that developers can understand how users access a website over a network with low cost. J.A. Jacko (Ed.): Human-Computer Interaction, Part I, HCII 2009, LNCS 5610, pp. 277–286, 2009. © Springer-Verlag Berlin Heidelberg 2009

278

M. Kiura, M. Ohira, and K. Matsumoto

However, these systems are designed to collect data only from static websites. Developers cannot figure out users’ interactions on a dynamic website (e.g., automatically created webpages by CGI or server-side script, and webpages developed by JavaScript in a Web browser). By using JavaScript, developers can implement an interface that can switch the displayed contents by tabs, drop down menus, or dragand-drop methods without URL transitions. On a website using such interfaces, the existing systems cannot obtain the previously displayed contents accessed by users because these contents would change. In this paper, we propose Webjig, which is a support system for website usability evaluation for both dynamic and static websites; this system records users’ interactions related to the contents that are displayed in users’ Web browser. Developers can exactly understand users’ interactions on a website by using Webjig. Thus, they can efficiently improve website usability.

2 Related Work The traditional approach to resolving problems of website usability it to use the Web server accesses logs [4]. Developers can know various kinds of information including users’ IP address, accessed time, request data, and Web server’s response from the Web server access log. The advantage of using the Web access sever log is that the access log is automatically saved in a Web server and can be used by developers with low cost. If developers can easily use the Web access log to improve website usability, however, they cannot know users’ interactions such as mouse motions, mouseclick positions, and mouse-click timings on a website [5]. Several systems have been proposed to automatically collect the data of users’ interaction on a website (e.g., MouseTrack [6], UsaProxy [7]). The systems solved the problem above, by identifying users’ mouse motions, mouse-click positions, and mouse-click timings by embedding JavaScript codes into a webpage. The systems helped developers understand users’ interactions on a website at a considerably low cost. Previous studies have suggested that there is a correlation between the point of gaze and the position of mouse cursor. Chen et al. have reported that there is a strong correlation between the point of gaze and the position of mouse cursor; further, the developer can predict a point in the website where the user interested in and they may chart a pattern of the user by users’ interaction [8]. In addition, Muller et al. reported that 35% of users traced a sentence with a mouse cursor when they read the sentence in a website. These results show that developers can detect the problems of website usability by studying users’ interaction on it.

3 Webjig In this paper, we introduce Webjig, which is a new system used to solve the problems of the existing systems. Webjig can handle data from static and dynamic websites. By analyzing DOM (Document Object Model) of HTML, Webjig can collect the data of contents clicked by users, including timings, positions, and motions. This mechanism

Webjig: An Automated User Data Collection System for Website Usability Evaluation

279

allows usability engineers and developers to solve the problems associated with the existing system, i.e., the existing system could not precisely identify users’ interactions on a dynamic website. We present the system architecture of Webjig in Fig.1. Webjig is a client/server system. The client is implemented by using JavaScript, which executes in a Web browser. The server is implemented by using PHP. The system consists of Webjig::Fetch, Webjig::Analysis, and Webjig::DB. Webjig::Fetch is a subsystem that automatically collects the data of users’ interactions on a website. Webjig::Analysis is a subsystem that shows the information of users’ interactions to developers. Webjig::DB is a subsystem that holds the data of users’ interactions and provides API to access the data.

Fig. 1. System architecture of Webjig

3.1 Webjig::Fetch Webjig::Fetch is a subsystem that automatically collects the data of users’ interactions on the website. Table 1 shows the data collected and stored by Webjig. During the time in which a user stays on a webpage, the data may be changed, except for the name and version of the Web browser. The system monitors a change in the data at intervals of dozens of milliseconds and sends the data to Webjig::DB at intervals of few seconds and at the time when the user exits the webpage. Table 1. Collected data usign Webjig Data type Name and version of Web browser Inner size of Web browser Position of scroll bar Position of mouse cursor Timing and type of mouse click Timing and type of key pressed Contents displayed in a Web browser

Timing of data collection Loaded Changed Changed Changed Pressed Pressed Changed

Timing of data transmission Loaded Intervals and exit Intervals and exit Intervals and exit Intervals and exit Intervals and exit Intervals and exit

280

M. Kiura, M. Ohira, and K. Matsumoto

For collecting users’ interactions data, developers have to install Webjig::Fetch in a webpage. what developers have to do is only to insert a line <script src=”URL of Webjig::Fetch”> in the HTML source code of the webpage that targets the usability evaluation using Webjig. Fig.2 is an example of Webjig installed in an HTML source code. Webjig works even if the developer may insert the script tag at the any place in the HTML source code. However, a mainstream Web browser interprets the HTML source code from the top and displays the contents. Therefore, we recommend inserting the script tag at the bottom of the HTML source code so that Webjig does not disturb the original contents.