Human-Computer Interaction. Interacting in Various Application Domains: 13th International Conference, HCI International 2009, San Diego, CA, USA, ... Applications, incl. Internet Web, and HCI)
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany
5613
Julie A. Jacko (Ed.)
Human-Computer Interaction Interacting in Various Application Domains 13th International Conference, HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings, Part IV
13
Volume Editor Julie A. Jacko University of Minnesota Institute of Health Informatics MMC 912, 420 Delaware Street S.E., Minneapolis, MN 55455, USA E-mail: [email protected]
Library of Congress Control Number: 2009929048 CR Subject Classification (1998): H.5, I.3, I.7.5, I.5, I.2.10 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI ISSN ISBN-10 ISBN-13
0302-9743 3-642-02582-X Springer Berlin Heidelberg New York 978-3-642-02582-2 Springer Berlin Heidelberg New York
The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human–Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of the design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. This volume, edited by Julie A. Jacko, contains papers in the thematic area of Human-Computer Interaction, addressing the following major topics: • • • •
eLearning and Education Games and Entertainment Work, Collaboration and Business Advanced Applications
The remaining volumes of the HCI International 2009 proceedings are: • • • • • •
Volume 1, LNCS 5610, Human–Computer Interaction––New Trends (Part I), edited by Julie A. Jacko Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 3, LNCS 5612, Human–Computer Interaction––Ambient, Ubiquitous and Intelligent Interaction (Part III), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis
VI
Foreword
• • • • • • • • • •
Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 11, LNCS 5620, Digital Human Modeling, edited by Vincent G. Duffy Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen Volume 17, LNAI 5639, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris
I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.
Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland
Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA
Foreword
Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea
Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany
Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA
Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China
Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA
Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK
VII
VIII
Foreword
Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa
Matthew J.W. Thomas, Australia Mark Young, UK
Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA
Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK
Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA
Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA
Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK
Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA
Foreword
Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA
Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China
Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria
Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan
Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA
Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA
IX
X
Foreword
Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK
Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany
Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China
Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China
Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan
Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan
In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.
Foreword
XI
I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis
HCI International 2011
The 14th International Conference on Human–Computer Interaction, HCI International 2011, will be held jointly with the affiliated conferences in the summer of 2011. It will cover a broad spectrum of themes related to human–computer interaction, including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/
General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email: [email protected]
Table of Contents
Part I: eLearning and Education Arab Children’s Reading Preference for Different Online Fonts . . . . . . . . . Asmaa Alsumait, Asma Al-Osaimi, and Hadlaa AlFedaghi Adaptation Decisions and Profiles Exchange among Open Learning Management Systems Based on Agent Negotiations and Machine Learning Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Silvia Baldiris, Ram´ on Fabregat, Carolina Mej´ıa, and Sergio G´ omez Accessing e-Learning Systems via Screen Reader: An Example . . . . . . . . . Maria Claudia Buzzi, Marina Buzzi, and Barbara Leporini
Optimal Affective Conditions for Subconscious Learning in a 3D Intelligent Tutoring System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pierre Chalfoun and Claude Frasson
39
Computer-Based Learning to Improve Breast Cancer Detection Skills . . . Yan Chen, Alastair Gale, Hazel Scott, Andrew Evans, and Jonathan James Virtual Classroom and Communicability: Empathy and Interaction for All . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francisco V. Cipolla Ficarra
49
58
Communicability for Virtual Learning: Evaluation . . . . . . . . . . . . . . . . . . . . Francisco V. Cipolla-Ficarra, Miguel Cipolla-Ficarra, and Pablo M. Vera
68
Attention and Motivation in Hypermedia Systems . . . . . . . . . . . . . . . . . . . . Francisco V. Cipolla Ficarra and Miguel Cipolla-Ficarra
The Application of the Flexilevel Approach for the Assessment of Computer Science Undergraduates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mariana Lilley and Andrew Pyper
140
Development of Ubiquitous On-Demand Study Support Environment for Nursing Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yukie Majima, Yumiko Nakamura, Yasuko Maekawa, and Yoichiro So
149
The Effects of Prior Knowledge on the Use of Adaptive Hypermedia Learning Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Freddy Mampadi, Sherry Chen, and Gheorghita Ghinea
156
Supporting Learners in Adaptive Learning Environments through the Enhancement of the Student Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luca Mazzola and Riccardo Mazza
166
The Concept of IMPRESSION: An Interactive Instruction System and Its Practice for Real-Time Distance Lessons between U.S. and Japan . . . Takashi Mitsuishi, Fumiko Konno, Yuki Higuchi, and Kentaro Go
176
Improving Children’s Writing Ability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joana Pereira, Lu´ıs Carri¸co, and Carlos Duarte From Paper to Module – An Integrated Environment for Generating SCORM Compliant Moodle Courses Out of Text and Multimedia Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hans-Martin Pohl, Benedikt Deicke, and Jan-Torsten Milde Development of a Simulator of Abacus: Ancient Analog Calculator on a Mobile Phone as a Teaching Material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kenta Saito, Yuki Makita, Vu Quang, and Hitoshi Sasaki
186
196
204
Table of Contents
A Proposal for a Framework for an e-Alumni Program Using SNS . . . . . . Hiroshi Sano Supporting End-User Development of Personalized Mobile Learning Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marco de S´ a and Lu´ıs Carri¸co
The Experience of Presence in 3D Web Environment: An Analysis of Korean Second Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SungBok Park, Ha Sung Hwang, and Myungil Choi
387
Influence of Real-World Ten-Pin Bowling Experience on Performance during First-Time Nintendo Wii Bowling Practice . . . . . . . . . . . . . . . . . . . . Kirsten A. Peters
396
Emotionally Adapted Games – An Example of a First Person Shooter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Timo Saari, Marko Turpeinen, Kai Kuikkaniemi, Ilkka Kosunen, and Niklas Ravaja DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tatsushi Takeuchi, Koichiro Watanabe, Tomoo Inoue, and Ken-ichi Okada
406
416
Part III: Work, Collaboration and Business New Health Information Systems (HIS) Quality-in-Use Model Based on the GQM Approach and HCI Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reem Al-Nanih, Hana Al-Nuaim, and Olga Ormandjieva
429
Table of Contents
An Information Visualization Approach to Hospital Shifts Scheduling . . . Carmelo Ardito, Paolo Buono, Maria F. Costabile, Rosa Lanzilotti, and Adalberto L. Simeone Designed to Fit: Challenges of Interaction Design for Clothes Fitting Room Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bo Begole, Takashi Matsumoto, Wei Zhang, Nicholas Yee, Juan Liu, and Maurice Chu Usability for Poll Workers: A Voting System Usability Test Protocol . . . . Dana Chisnell, Karen Bachmann, Sharon Laskowski, and Svetlana Lowry
XIX
439
448
458
CAD and Communicability: A System That Improves the Human-Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Francisco V. Cipolla Ficarra and Roc´ıo A. Rodr´ıguez
468
A Novel Visualization Tool for Evaluating Medication Side-Effects in Multi-drug Regimens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jon Duke, Anthony Faiola, and Hadi Kharrazi
478
Design of a Web Intervention to Change Youth Smoking Habits . . . . . . . . Kim Nee Goh, Yoke Yie Chen, Emy Elyanee Mustapha, Subarna Sivapalan, and Sharina Nordin
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work: The Case of IT Service Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eser Kandogan, Eben M. Haber, John H. Bailey, and Paul P. Maglio
504
Human Computer Interaction in Virtual Standardized Patient Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrick G. Kenny, Thomas D. Parsons, and Albert A. Rizzo
514
Towards Standardized Pen-Based Annotation of Breast Cancer Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Suzanne Kieffer, Annabelle Gouze, Ronald Moncarey, Christian Van Brussel, Jean-Fran¸cois De Wispelaere, Fran¸coise Kayser, and Benoˆıt Macq ImproV: A System for Improvisational Construction of Video Processing Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Atsutomo Kobayashi, Buntarou Shizuki, and Jiro Tanaka E-Assessment: A Suitable Alternative for Measuring Competences? . . . . . Martin Kr¨ oll
Arab Children’s Reading Preference for Different Online Fonts Asmaa Alsumait1, Asma Al-Osaimi2, and Hadlaa AlFedaghi2 1 Computer Engineering Dep., Kuwait University, Kuwait Regional Center For Development of Educational Software, Kuwait [email protected], {alosaimi,hadlaa}@redsoft.org 2
Abstract. E-learning education plays an important role in the educational process in the Arab region. There is more demand to provide Arab students with electronic resources for knowledge now than before. The readability of such electronic resources needs to be taken into consideration. Following design guidelines in the e-learning programs’ design process improves both the reading performance and satisfaction. However, English script design guidelines cannot be directly applied to Arabic script mainly because of difference in the letters occupation and writing direction. Thus, this paper aimed to build a set of design guidelines for Arabic e-learning programs designed for seven-to-nine years old children. An electronic story is designed to achieve this goal. It is used to gather children’s reading preferences, for example, font type/size combination, screen line length, and tutoring sound characters. Results indicated that Arab students preferred the use of Simplified Arabic with 14-point font size to ease and speed the reading process. Further, 2/3 screen line length helped children in reading faster. Finally, most of children preferred to listen to a female adult tutoring sound. Keywords: Child-Computer Interfaces, E-Learning, Font Type/Size, HumanComputer Interaction, Information Interfaces and Presentation, Line Length, Tutoring Sound.
studied adults’ preferences regarding font type and size from measuring adults’ reading efficiency, adults’ reading time, legibility, attractiveness, and adults’ font preference [1]. They found that Times and Arial font type with size 12-point were read faster than other fonts and sizes that were tested. Further, Arial and Courier font type were considered the most legible fonts. Again, Arial font type with 12-point size was the most preferable to read online. Another study carried out for adults compared the fonts used in previous editions of Windows with those new created for Windows Vista [3]. They found that the new fonts designed for Windows Vista such as Cambria and Constantia were more legible than traditional Times New Roman. While studies conducted on children indicated that the children prefer to read English text that is Arial font type with 14-point size and Comic font type with 12-point size [2]. Moreover, in English, adults prefer to read medium line length (approximately 65 to 75 CPL) and children prefer narrow line length (approximately 45 CPL) [4, 6]. On the other hand, few studies were conducted to find adult users’ preferences for Arabic script. For example, Hemayssi et. al. concluded that in order to increase legibility, it is preferred to use bold fonts, colors and clear icons [5]. Most existing research in this area is oriented to build guidelines for designing English e-learning programs. Therefore, in this paper, we examined the preferences of seven to nine years old Arab children's for five font types at sizes 12- and 14-point and their screen line length using an electronic story.
2 Usability Evaluation Process The usability evaluation of the Arabic electronic story for font type, size, and line length was conducted on July 2008. Participants were asked to spend half an hour with the e-story under the supervision of the ReDSOFT team member experimenter. At the beginning of the usability evaluation, participants were asked to complete a user profile questionnaire. The background questionnaire revealed that thirty five participants (16 males, 19 females) aged between seven and nine years old. Moreover, majority (86%) of the participants use computer at home, and almost all of them (97%) had previous experience with electronic reading. Each page of the e-story was written with a different combination of font type and size. In total, the e-story was presented in 10 pages representing the five font types (Arial Unicode MS, Courier New, Microsoft Sans Serif, Simplified Arabic, and Traditional Arabic) and two font sizes (12 and 14). Figure 1 shows sample of a page of the e-story.
Fig. 1. A sample of a page of the e-story
Arab Children’s Reading Preference for Different Online Fonts
5
After reading each page of the e-story, participants were asked to answer four questions to measure their satisfaction regarding font type and size used. 1. Is this font type and size easy to read? 2. Do you think you can read faster with this font type and size? 3. Is this font type and size attractive? 4. Would you like this font type and size to be used in your schoolbook? Text was displayed in the center of the screen. Fonts were black presented on a white background. A stopwatch was used to record the time participants took to read the paragraphs and the experimenter also noted the number of incorrectly pronounced words. To confirm findings, participants were asked to read a simple sentence with all font type and size combinations in a single page and select their preferable font type and size. Also participants were asked to choose the most comfortable screen line length while reading. The same font type and size (Simplified Arabic with size 14-pionts) script was presented to the participant in different line length conditions; 1/3 screen line length, 2/3 screen line length, and full screen line length. The participant was asked to rate his/her reading comfortance and satisfaction with the three different screen line lengths. We also explored a new direction for e-learning computer tutoring sound characters, which we believe will maximize students’ learning gains and enjoyment. The traditional scenario allows students to interact primarily with a single coach or tutor character sound on-screen. In this evaluation process, we allowed the child to select his own tutor sound to be a teacher, a boy, or a girl. The usability evaluation process measured seven main factors in the reading process: the ease of reading factor, reading faster factor, font attractiveness factor, the desire to use font combination in schoolbooks factor, the font preferences factor, the screen line length factor. And finally the tutor character sound factor. Table 1 shows different font types and sizes used in the experiment. Table 1. Font types and sizes studied Font type Arial Unicode MS Courier New Microsoft Sans Serif Simplified Arabic Traditional Arabic
3 Results 3.1 Ease of Reading Ease of reading measures whether a specific font type and size gave children an impression that the reading process was simple and trouble free. Analysis showed that
6
A. Alsumait, A. Al-Osaimi, and H. AlFedaghi
61% of the participants considered the Simplified Arabic font as an easiest to read font. The lowest ease of reading score was given to the Courier New and Traditional Arabic. Further analysis is done to find font sizes that help children in reading faster. The results showed that there is small difference between fonts that are sized 12-point and fonts that are sized 14-point. 57 % of the children thought that 14-point sized font are easy to read in oppose to 45% who thought that 12-point sized font is easy to read. The results revealed that using Simplified Arabic with 14-point font size was considered the easiest to read font for Arab children. Guideline 1 Use Simplified Arabic with 14-point font size to ease the reading for Arab children.
3.2 Reading Faster The second factor to be measured was reading speed. It is a variable that measures whether a specific font type and size gave children an impression that the reading process was done quicker. Analysis showed that font type has an affect on children's reading speed. Simplified Arabic and Microsoft Sans Serif had the highest participants’ score preference; 54% for reading faster. On the other hand, Courier New had the lowest preference score regarding the reading faster factor, only 34 % of the participants thought that they can read faster with it. When it comes to font size, results showed that children (47%) had preferred to read fonts that are sized 14-point. Fonts with size 12-point got a lower preference score. In addition, results showed that 60% of the participants believed that the Simplified Arabic with 14-point font size helped them read faster. In contrast, only 26% of the participants thought that Traditional Arabic with font size 12-point, will help them read faster. Guideline 2 Use Simplified Arabic with 14-point font size to speed the reading for Arab children.
3.3 Font Attractiveness Font Attractiveness measures whether children found a specific font type and size nice looking and eye catching. 66% of the participants considered the Arial Unicode MS as the most attractive font type among the types we tested. When it comes to font size, results showed that it has no difference in attractiveness scores. Font sizes 12-point and 14-point had similar attractiveness scores (around 60%). According to experimental results, Arial Unicode MS with 14-point font size got the best score when it comes to font type/size attractiveness.
Arab Children’s Reading Preference for Different Online Fonts
7
Guideline 3 Use Arial Unicode MS with 14-point font size when considering font attractiveness for children.
3.4 Desire to Use Font Type and Size Combination in Schoolbooks Desire to use font combination in schoolbooks factor measures whether children want to use a specific font type and size in their schoolbooks. The results showed that Simplified Arabic with font size 12-point and Arial Unicode MS with font size 14-point are considered the most desired font type and size combinations preferred to be used in schoolbooks. Guideline 4 Use Simplified Arabic with font size 12-point or Arial Unicode MS with font size 14-point when considering children's preferences for schoolbooks.
3.5 Confirm Font Preference 3.5.1 12-Point Font Size Figure 2 shows font type preferences percentage with size 12-point. Arial Unicode MS was selected as the most preferred (45%) font type with size 12-point.
50
Percentage (%)
40
30
20
10
0 Arial Unicode Courier New MS
Microsoft Sans Serif
Simplified Arabic
Traditional Arabic
Fig. 2. The font type preference percentage with size 12-point
3.5.2 14-Point Font Size Figure 3 shows font type preferences percentage with size 14-point. Arial Unicode MS was considered to be the most preferred (57%) font type with size 14-point.
8
A. Alsumait, A. Al-Osaimi, and H. AlFedaghi
60
50
Percentage (%)
40
30
20
10
0 Arial Unicode Courier New MS
Microsoft Sans Serif
Simplified Arabic
Traditional Arabic
Fig. 3. The font type preference percentage with size 14-point
Figure 4 shows which font size is preferred by participants. Result showed that participants preferred to read using 14-point font size. These results are in agreement with the results obtained through the usability evaluation process of reading the estory. 80 70
Percentage (%)
60 50 40 30 20 10 0 12-point
14-point
Fig. 4. Best font size selected by participants
3.6 Tutoring Sound At the end of the experiment, we asked the child to choose a character sound to tutor him/her through the final part of the survey. Figure 5 (a, b, c) shows the percentage of the selected tutoring sound character by the participants. The result showed that most children (48 %) regardless of their gender had selected the female teacher as a sound tutoring (Figure 5.a). This result is related to the fact that at this specific age (seven to nine years old) children are used to be taught by female teachers at the schools.
Arab Children’s Reading Preference for Different Online Fonts
60
50
Percentage
40
30
20
10
0 The teacher character "Hanan"
The boy character "Salem"
The girl character "Abeer"
Fig. 5a. The percentage of the selected tutoring sound character by all participants
60
50
Percentage (%)
40
30
20
10
0 The teacher "Hanan"
The boy character "Salem"
The girl character "Abeer"
Fig. 5b. The percentage of the selected tutoring sound character male participants
60
50
Percentage (%)
40
30
20
10
0 The teacher "Hanan"
The boy character "Salem"
The girl character "Abeer"
Fig. 5c. The percentage of the selected tutoring sound character by female participants
9
10
A. Alsumait, A. Al-Osaimi, and H. AlFedaghi
Interestingly, results also reveled that the male child participants have selected only the boy sound charter "Salem" (50%) along with the teacher sound character "Hanan" (50%) (Figure 5.b). Similarly, the female child participants have selected only the girl sound character "Abeer" (53%) along with the teacher character "Hanan" (47 %) (Figure 5.c). This indicates that children prefer only the teacher sound character along with the child tutor sound of the same gender. Guideline 5 To motivate both genders and gain their concentration, use neutral character (e.g. a teacher). To motive gender-specific participants' group and gain their concentration, use either a neutral character or an equivalent gender.
3.7 Line Length This research also investigated the effect of screen line length on reading performance and satisfaction. The same paragraph was presented in three different screen line lengths. Despite the fact that there were no significant differences in satisfaction scores (see Figure 6), a line length that supports faster reading could impact the overall experience of e-learning programs. Reading rates were found to be the fastest (4.59 sec/word) with the 2/3 screen line length, (5.44 sec/word) with the full screen line length and to be the slowest (6.56 sec/word) at the 1/3 screen line length. Participants reported either liking or disliking the extreme screen line lengths (1/3 and full line length). Those that liked the 1/3 screen line length indicated that the short line length helped faster reading and was easier because it required less eye movement and the paragraph seemed neat and clear. Those that liked the full screen line length stated that they liked having more information on one line and believe that it seemed to them that they are reading less information.
40
Percentage (%)
30
20
10
0 1/3 Screen Line Length 2/3 Screen Line Length Full Screen Line Length
Fig. 6. The percentage of the preferred screen line length
Arab Children’s Reading Preference for Different Online Fonts
11
Although some participants reported that they felt like they were reading faster at 1/3 screen line length, this condition actually resulted in the slowest reading speed. From this study, it may be beneficial to use 2/3 line lengths when possible. Guideline 6 Use 2/3 screen line length to speed the reading.
4 Conclusion Readable and satisfying e-learning interfaces are achieved through following e-learning design guidelines. These design guidelines need to be configured to suite some language characteristics. Therefore, this paper aimed to build a set of design guidelines for Arabic e-learning programs designed for seven-to-nine years old children. The results of evaluating the e-story indicated that children preferred to read text using Simplified Arabic or Arial Unicode MS with size 14-point. These font type and size combinations were considered, among the different font type and size combinations used in this study, to be the easiest and fastest font type and size combinations to read. Conversely, children, for font attractiveness, chose other preference. They preferred to read using Arial Unicode MS with size 14-point. Generally, font types with 14-point size were preferable over the other font types with 12-point size. In addition, participants were able to read faster with 2/3 screen line length. When it comes to the tutoring sound character, children preferred to listen to a female adult tutoring sound, which is closer to their actual life style. Tutoring sound characters can play a critical success factor in learner acceptance of e-learning programs. Larger combinations of font types and sized need to be assessed by Arab children. Further assessments are also needed to investigate the influence of other variables on the reading experience. Examples of those variables are: text color, effect of emphasizing pieces of text; bold, italic or underlined text, paragraph spacing, eye movements, and scrolling movements.
References 1. Bernard, M.L., Lida, B., Riley, S., Hackler, T., Janzen, K.: A comparison of popular online fonts: Which size and type is best?. Usability News. 4.4 (2002) 2. Bernard, M.L., Melissa, M., Talissa, F., Jan, M.: Which Fonts do Children Prefer to Read Online?. Usability News. 3.1 (2001) 3. Chaparro, B., Dawn Shaikh, A., Chaparro, A.: Examining the Legibility of Two New ClearType Fonts. Usability News. 8.1 (2006) 4. Dyson, M.C.: How physical text layout affects reading from screen. Behaviour & Information Technology 23(6), 377–393 (2004) 5. Hemayssi, H., Sanchez, E., Moll, R., Field, C.: Designing an Arabic user experience: methods and techniques to bridge cultures. In: Proceedings of the Conference on Designing for User Experiences DUX 2005 (2005) 6. McPherson, M.A., Nunes, J.M., Zafeiriou, G.: New Tutoring Skills for Online Learning: Are e-Tutors adequately prepared for e-learning delivery? In: Proceedings of EDEN 2003 The Quality Dialogue; Integrating Quality Cultures in Flexible, Distance and e-learning, Rhodes, Greece, June 15-18, 2003, pp. 347–350 (2003)
Adaptation Decisions and Profiles Exchange among Open Learning Management Systems Based on Agent Negotiations and Machine Learning Techniques Silvia Baldiris, Ramón Fabregat, Carolina Mejía, and Sergio Gómez Institute of Informatics and Aplications (IIiA), Universitat de Girona, Spain [email protected], [email protected], {carolina,sergiog}@eia.udg.edu
Abstract. We have developed some projects [1,2] for addressing the heterogeneity problem in open learning management systems (LMS). In [3], an independent adaptation platform to support competences development through personalization is presented. Three user characteristics (competences profile, learning style, and accessing context) are modeled by means of analyzing user interaction data in a LMS. This process is supported by the assigment of independent adaptation tasks to different JADE intelligent agents. In this paper we introduce some negotiation strategies among those intelligent agents in order to: 1) select the best types of adaptation through collaborative tasks, and 2) generate standards and exchangeable user profiles based on the inferred user characteristics, describing the mechanisms to mobilize these profiles between different LMSs. These profiles support the generation of specifics learning designs for each particular user. Keywords: competence development, adaptation, intelligent agents, adaptive hypermedia, machine learning.
Adaptation Decisions and Profiles Exchange among Open LMS
13
main issues to be addressed in the context in an open learning environment because of the impact that it could have in the learning design process. In this paper, we have focused on how to establish collaborative work among the agents in the ADA+ adaptation platform and we present a general schema for the IMS specification exchange among e-learning platforms. For the first problem we propose the definition of an optimization problem to relate the tasks developed by each agent in the platform and the cost associated to the development of these tasks. The second problem will be addressed by combining semantics and structural techniques for similarity analysis among XML documents. The paper is structured as follows. First, in the section 1 an approach for the state of the art is presented, and its relationships with the problems related in the paper. Next, the two problems mentioned above are described. In the section 4, a detailed description about the proposal is presented and finally some conclusions and future works are shown.
2 Related Works In this section we introduce some related work and its relationship with our research works. The first research area related with our proposal is the use of intelligent agents technologies in education. Many educational systems are known to reduce navigation effort, time to achieve the learning goal, and learner retention, and to increase quality of learning. In particular, Shaboo [5], Mas-PlanG [1], MAS-SHAAD [2] and aLFanet [6] are projects developed by Universidad Industrial de Santander, University of Girona and UNED, respectively, which show how user modelling processes and adaptation take into account different user characteristics independently to improve learning. Some of these projects use intelligent agents to develop adaptation tasks but frequently this set of agents do not collaborate among them to achieve a common objective. Our research efforts are oriented to include some characteristics from adaptive hypermedia systems in the context of LMS, taken into account the autonomy and mobility of the agents. An adaptation platform [3] was developed with this purpose but it is necessary to improve our work including collaborative task among the agents in the platform. This is the first objective in this paper. Conceptual modeling to facility the interoperability among LMS is the second research line of our interest. IMS specifications are one of the main elements in the user modeling and adaptation process. They are exchangeable XML documents with an associated schema. The analysis of the structural and semantic similarities between XML and with the problem of detecting change among these kinds of documents is another large research area. Tendencies for the analysis of the structural similarity in a XML document can be summarized into three groups. The first one considers a XML document as an ordered tree [7]. The second one is based on the tag similarity [8], and the third one consist on the interpretation of the XML document as a time series which applies the Discrete Fourier Transform [9], analyzes the frequency of this signal and infers the similarity measure.
14
S. Baldiris et al.
Some algorithms such as MH-Diff [10] and LaDiff [11] have been developed for detecting change between structured documents algorithms. For the specific XML documents, XMLTreeDiff [12], XyDiff [13], X-Diff [14] has been developed. Our intention is not propose a new algorithm but to use the best one with the purpose of generating a full user model exchange. The user modelling process defines and maintains up-to-date user models, where user models are a representation of information about an individual user that is essential for an adaptive system to provide the adaptation effect [15]. The result of this process is a user model that consists in different user characteristics and can be expressed in different formats according to the system that stores and exchanges it. User model servers or shells are systems in charge of maintaining the user models and they generally have the follows characteristics: 1) it use a particular language to represent the user characteristics, 2) require specific collaborative filtering techniques in order to create stereotypes of user groups, 3) count with a user history recorder, and also 4) apply techniques such as user models retrieval, update and validation. Some of the developments in this area are GUC [16], UM [17], BGP-MS [18], DOPPELGÄNGER [19], TAGUS [20], GUMS [21], PROTUM [22], UMT [23]. A main issue regarding the exchange of information between user model servers is the analysis of the information that it contains. This problem is related for us with the XML similarity analysis.
3 Problem Description Our research works are based on the following two hypotheses: The first one is to support the learning process with the aim to improve the learning. In this case, the knowledge about the users (user modeling) such as previous competences, individual preferences and human values has to be considered [1] and the learning process has to be adaptive according to this knowledge. The second hypothesis is that in an Table 1. Characterization of the framework elements to support competence development
Specifications Exchange Service Agents Technologies • Technological Standard Based Learning Adaptation Process Agent Technologies • Platform Independence • Technological Standard Based Decision Generation Process Machine Learning Techniques Collaborative Filtering • Platform Independence • Technological Standard Based User Modelling Process Machine Learning Learning style, competences, etc. Independent Agents Platform LEARNING PROCESS MODELLING (Competences, Course Structure, Learning Resources and Activities, Evaluation Another Necessary Models)
Adaptation Decisions and Profiles Exchange among Open LMS
15
open learning environment the possibility to interchange user profiles based on competences can facilitate the management of the learning process. Table 1 presents a general vision of different process to verify these hypotheses. In [4] we describe a user modeling and e-learning adaptation process based in the use of different IMS Specification. The user modeling process defines the method for inferring some user characteristics. In particular, the specific competence level, the collaborative competences level, the user learning style and the context. User modeling process supports a specific learning adaptation process [3]. The result of this process can be an IMS Learning Design (LD) level B, which is a course that takes into account some user features. This adapted learning design permits to deliver learning objects (LO) and activities according to the values of user characteristics presented above. The ADA+ adaptation platform was developed with this proposes and consists in different intelligent agents, which have particular adaptation tasks (see table 2). Table 2. Specific Agent Task in the adaptation platform
Specific Agent Task Planner Agent Specific Competence Agent Collaborative Competence Agent Learning Style Agent
Context Agent
Task description − Generates an adapted IMS-LD (level B) for users. − Generates the planning problem using competence definition and the LO metadata. − Defines the collaborative level of the user using a grouping behaviour. − Defines the delivering order of the learning resource type using automatic classification techniques. − Monitors the change in the user learning style according to the user interaction. − Selects equivalent LO according to the user access device and accessibility user preferences.
Actually, agents in ADA+ adaptation platform carry out some tasks independently. The reason behind this is that we are interested in creating a flexible collaborative environment among intelligent agents. This environment is meant to consider additional users’ characteristics when needed. However, we have analysed that the problem of context limitations in the virtual environment can produce undesired adaptation results. For these reason, one of the purposes of this paper is to introduce an approach based on the definition of an adaptation decision process in order to identify the types of adaptation that can be offered in a specific moment according to the learning context limitations. On another hand, one of the main purpose for applying specifications and standards in education technologies is to facilitate the information exchange among LMS.
16
S. Baldiris et al.
IMS consortium has worked in the generation of different kind of specifications with the objective of creating a standard e-learning language and increase the systems interoperability. These languages permit to specify products and sub-process of the learning process such as competence, learning designs, and assessments among others. We present the Specification Exchange Service (SES) as a mechanism for exchange information among e-learning platforms, in particular, IMS specifications. It proposes a general framework supported in an adjusted version of X-Diff algorithm [24]. SES could support the decision process by increasing the quantity of resource than can be used by the agents in ADA+.
4 Proposal Description In this section, we present our proposal to establish a collaborative work among the agents in the adaptation platform, and present a general schema for the IMS specification exchange among e-learning platforms. 4.1 Adaptation Decision through Detection of Context Constrain and Negotiation When the adaptation idea is introduced in the context of a virtual learning process, the starting point is still the modelling of the teaching-learning process as in the traditional (face to face) process. However, the initial work for the teacher increases because she needs to perform new activities using a LMS and adjust the environment according to the requirements. If the teacher does not provide the necessary information to the systems, multiples context limitations can be detected in the environment. These limitations impact negatively the automatic generation of the adaptive learning design. Table 3 summarizes the different limitations that can be found in the learning environment. Table 3. Agent tasks vs.context limitations
Specific Agent Task Specific Competence Agent Collaborative Agent Learning Style Agent
Context Agent
Context Limitations − Existence and quality of the Competence definition. − Existence and quality of the LO metadata according to the needed model. − Amount and quality of users interaction in collaborative tools. − Amount of learning objects according to the different LO metadata resource types. − Amount and quality of users interaction with the different LOM resources types. − Existence of the equivalent LO for the topics.
Adaptation Decisions and Profiles Exchange among Open LMS
Description 1 if competence definition exist % 1 if collaborative interaction exist % 1 if learning objects exist % 1 if equivalent learning objects exist % 1 if objects metadata exist % %
These constraints can be expressed in terms of the specific variables. Table 4 shows some of them. In ADAPTAPlan project each particular agent can define the value of specific variables and they use them as the input of the decision process. The natures of the variables permit us model the decision problem as an optimization problem, where the objective is to reduce the cost of accessing distributed resource repositories in order to obtain the necessary information to generate the adaptation task. In this way, the decision process can be supported in a cost matrix where different source and its cost are related in terms of different characteristics as network traffic needed, purchase cost, availability, and importance, among other. Taken into account the cost associated to the necessary resources for each particular adaptation task and also the cost associated to combinations of adaptation tasks, the decision is defined. As a result of this collaboration process based on the cost of the different resources, an adapted IMS-LD is generated. 4.2 Specification Exchange Service (SES) Consider two different e-learning platforms where the corresponding students developed different learning activities. Each particular platform is associated to a specific user modeling system and user model shell system. As was described above, the user modeling system obtains user model for the users through the analysis of user interaction and the user model shell system maintain the model (see figure 1). The question is: how can the information in these systems be shared? In last years, XML have become the most common language to represent information to be shared about the learning process and the people. Indeed, IMS specifications use XML as a representation language and schemas for the structure definition and validation.
18
S. Baldiris et al.
Fig. 1. Exchange Problem
For this reason we have tested some algorithms for the analysis of the similarities between XML documents, algorithms were shown in the section 2. We have selected diffX algorithm [24] in order to detect the structural changes in the XML documents and for supporting the generation of a new user model which is a mix of previous two documents. diffX algorithm was selected for including move operation that another tested algorithm does not implement. The process of the new user model generation is shown in the figure 2.
Fig. 2. Complete User Model Generation
The first part of the process, matching, considers the XML documents as an ordered tree, identifying isolated tree fragment mapping (i.e. an iterative top-down mapping technique). The result of this mapping is the definition of the set of sub trees that are the same in both trees. This identification permits to determinate the set of necessary operations to transform one document into another. The set of operations are applied to the document of the LMS that has generated the request. Then the document is sent to the requester LMS. Semantic analysis of the XML document in the matching process it is supported in the schemas associated to different IMS specifications.
5 Conclusion and Future Works In this paper we deal with two problems associated to the use of adaptive e-learning platforms and we propose an approach for each of them. The first one is about how the incomplete learning process modeling can produce undesired adaptation results in the LMS. We propose a strategy based on the collaboration among the intelligent agents in order to generate an adequate adaptation decision. The decision strategy is supported by a cost matrix where different source and its cost are related in terms of
Adaptation Decisions and Profiles Exchange among Open LMS
19
different characteristics as network traffic needed, purchase cost, availability, and importance. The second problem is related to the need of information exchange among elearning platforms, user profiles exchange, competence definition or any other type of information, in order to address this problem one approach supported in diffX algorithm was introduced. Proposals presented in the paper are under construction in the context of A2UN@ project (TIN2008-06862-C04-02/TSI). In particular, different types of repositories for supporting the agent collaboration are under construction. Testing process for comparing different algorithms have been developed and the integration of different syntactic and semantic algorithms is been implemented.
References 1. Peña, C.I.: PhD Thesis: Intelligent agents to improve adaptivity in a web-based learning environment. Universidad de Girona (2004) 2. Mérida, D., Cannataro, M., Fabregat, R., Arteaga, C.: MAS-SHAAD a Multiagent System Proposal for an Adaptive Hypermedia System. In: Proceedings of IJCEELL journal Special issue: Adaptivity in Web and Mobile Learning Services (2004) 3. Baldiris, S., Santos, O., Huerva, D., Fabregat, R., Boticario, J.G.: Multidimensional Adaptations for Open Learning Management Systems. Accepted at TUMASA Workshop of 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Sydney (Australia), December 9-12 (2008) 4. Baldiris, S., Santos, O.C., Barrera, C., Boticario, J.G., Velez, J., Fabregat, R.: Integration of educational specifications and standards to support adaptive learning scenarios in ADAPTAPlan. International Journal of Computer Science and Applications (IJCSA). Special Issue on New Trends on AI techniques for Educational Technologies 5, 1 (2008) 5. Moreno, G.D., Baldiris, S.M.: Degree project memories: Adaptive Hypermedia System for Teaching Object Oriented Programming. Universidad Industrial de Santander (2003) 6. Santos, O.C., Boticario, J.G.: Meaningful pedagogy via covering the entire life cycle of adaptive eLearning in terms of a pervasive use of educational standards: the aLFanet experience. In: Nejdl, W., Tochtermann, K. (eds.) EC-TEL 2006. LNCS, vol. 4227, pp. 691–696. Springer, Heidelberg (2006) 7. Shasha, D., Zhang, K.: Fast algorithms for the unit cost editing distance between trees. Journal of Algorithms 11, 581–621 (1990) 8. Buttler, D.: A Short Survey of Document Structure Similarity Algorithms. In: International Conference on Internet Computing 2004, pp. 3–9 (2004) 9. Flesca, S., Manco, G., Masciari, E., Pontieri, L., Pugliese, A.: Fast Detection of XML Structural Similarity. IEEE Transaction on Knowledge and Data Engineering 17(2), 160–175 (2005) 10. Chawathe, S., Garcia-Molina, H.: Meaningful change detection in structured data. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, Tucson, Arizona, USA, May 13-15 (1997) 11. Chawathe, S., Rajaraman, A., Garcia-Molina, H., Widom, J.: Change detection in hierarchically structured information. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, Montreal, Quebec, Canada, June 4-6 (1996) 12. Cubera, F., Epstein, D.: Fast Difference and Update of XML Documents, March 1999. Xtech, San Jose (1999)
20
S. Baldiris et al.
13. Cobena, G., Abiteboul, S., Marian, A.: Detecting changes in XML documents. In: Proceedings of the 18th International Conference on Data Engineering, San Jose, California, USA, February 26 - March 1 (2002) 14. Wang, Y., DeWitt, D., Cai, J.: X-Diff: An effective change detection algorithm for XML documents. In: Proceedings of the 19th International Conference on Data Engineering, Bangalore, India, March 5-8 (2003) 15. Brusilovsky, P., Millán, E.: User Models for Adaptive Hypermedia and Adaptive Educational Systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 3–53. Springer, Heidelberg (2007) 16. Van Der Sluijs, K., Houben, G.: A generic component for exchanging user models between web-based systems. International Journal of Continuing Engineering Education and Life-Long Learning 16(1/2), 64–76 (2006) 17. Kay, J.: The um Toolkit for reusable, long term user models. User Modeling and UserAdapted Interaction 4(3), 149–196 (1995) 18. Kobsa, A.: Modeling the user’s conceptual knowledge in BGP-MS, a user modeling shell system. Comput. Intelligence 6, 193–208 (1990) 19. Orwant, J.: Heterogenous learning in the Doppelgänger user modeling system. User Model. User-Adapted Interact. J. Personal. Res. 4(2), 107–130 (1995) 20. Paiva, A., Self, J.: TAGUS: A user and learner modeling workbench. User Model. UserAdapted Interact. J. Personal. Res. 4(3), 197–226 (1995) 21. Finin, T.W., Drager, D.: GUMS1: A general user modeling system. In: Sixth Canadian Conference on Artificial Intelligence, Montreal, Canada, pp. 24–29 (1986) 22. Vergara, H.: PROTUM: a prolog based tool for user modeling. WIS-Report 10, WG Knowledge-Based Information Systems, Department of Information Science, University of Konstanz, Germany (1994) 23. Brajnik, G., Tasso, C.: A shell for developing non-monotonic user modeling systems. Int. J. Human-Computer Studies 40, 31–62 (1994) 24. Al-Ekram, R., Adma, A., Baysal, O.: diffX: An Algorithm to Detect Changes in MultiVersion XML Documents. School of Computer Science, University of Waterloo (2005)
Accessing e-Learning Systems via Screen Reader: An Example Maria Claudia Buzzi1, Marina Buzzi1, and Barbara Leporini2 1
Abstract. The evolution of the Information and Communication Technology (ICT) and the rapid growth of the Internet have impelled the pervasive diffusion of e-Learning systems. This is a great opportunity for visually-disabled people provided that both the interactive environment, created by the Learning Management Systems, and the Learning Objects, created by teachers, are properly designed and delivered. In this paper we investigate interaction of the blind user with an open source Virtual Learning Environment (Moodle) and discuss how the use of the W3C Accessible Rich Internet Applications (ARIA) suite may improve the experience of navigation via screen reader. Keywords: e-Learning, accessibility, usability, blind, ARIA.
When designing for blind users, it is necessary to consider the three main interacting subsystems of the Human Processor Model: the perceptual, motor and cognitive systems [5]. Sightless persons perceive page content aurally and navigate via keyboard. This makes the “reading process” time-consuming and sometimes difficult and frustrating, if the contents are not designed with special attention to their needs. The cognition part of the interaction is important, since many learning techniques are only relevant to people with good vision and may not apply to someone with a visual impairment. Thus, alternative ways to deliver the same content should be provided. Furthermore, a blind person may develop a different mental model of both the interaction and the learning processes, so it is crucial to provide an easy overview of the system and contents. Non-visual perception can lead to: 1. Content serialization. The screen reader reads the contents sequentially, as they appear in the HTML code. This process is time-consuming and annoying when part of the interface (such as menu and navigation bar) is repeated in every page. As a consequence, blind users often have to stop the screen reading at the beginning, and they prefer to navigate by Tab Keys, from link to link, or explore the content row by row, via arrow keys. 2. Content and structure mixing. The screen reader announces the most important interface elements such as links, images, and window objects as they appear in the code. For the blind user, these elements are important for figuring out the page structure, but require additional cognitive effort. 3. Table. If the table’s content is organized by columns the screen reader (which reads by rows) announces the content of the page out-of-order, and consequently the information might be confusing or misleading for the user. 4. Lack of context. When navigating by screen reader the user can access only small portions of text and may lose the overall context of the page; thus it may be necessary to reiterate the reading process. 5. Lack of interface overview. Blind persons do not perceive the overall structure of the interface, so they can navigate for a long time without finding the most relevant contents. 6. Difficulty understanding UI elements. Links, content, and button labels should be context-independent and self-explanatory. 7. Difficulty of working with form control elements. The new JAWS version (v. 10) simplifies the interaction with forms since it can automatically activate the editing modality (for text input) when the virtual focus arrives at the text box (for instance when the user presses the tab key). However, with previous screen reader versions the user may have great difficulties since switching between exploration and editing modalities is required (i.e. form mode on/off). 8. A blind person is unable to access multimedia content such as video streaming, video conferencing, and captioning. If an alternative description is not present the user may lose content. In this paper we analyze accessibility and usability for the blind of two demo courses offered by Moodle (http://www.moodle.org/), a very popular open source eLMS. Then we illustrate how ARIA, the suite developed by the Web Accessibility Initiative (WAI) group of W3C, may facilitate interaction for the blind. Lastly, the paper presents our conclusions.
Accessing e-Learning Systems via Screen Reader: An Example
23
2 Related Works E-learning systems pose new challenges with respect to classic user-centered product design, where the target is a set of homogeneous users. Learner-centered Design must answer to the needs of multiple learner categories due to differences in learning strategies, know-how, experiences, motivation to learn and, not least, user age and ability. If appropriately designed and implemented, e-Learning systems are more effective and useful than classroom learning [4]. Various studies focus on the usability of e-Learning systems and some also include a general discussion on accessibility, but to our knowledge only a few focus on totally blind persons. Fifteen years ago Nielsen proposed an informal method for evaluating systems and design usability based on verifying the conformance to a set of principles of usable software design (heuristics) performed by experts [8]. This approach, which detects a high percentage of problems, is cost-effective and easy to implement compared to usability testing. However, to be effectively applied to the e-Learning domain these general principles needed to be further refined. A few years later, Squires and Preece embedded usability heuristics in the socio-constructive theory and specified criteria ad hoc for e-Learning [12]. However, even today various researchers criticize the lack of accurate studies in this field. In [1] the authors take the first steps in defining a methodology for the rigorous evaluation of e-Learning applications, but accessibility for special needs students is not analyzed. Furthermore, Zaharias critically examined the usability of e-Learning applications and proposed a new usability measure: the student’s intrinsic motivation to learn [15]. Developing a usability evaluation method based on a questionnaire, he carried out two large empirical studies showing the reliability of this approach. For Sloan et al. the goal of universal accessibility on the Web is inappropriate and instead it is necessary to explore multiple routes to provide equivalent experiences [11]. As Kelly et al. argued, rather than demanding that an individual learning resource be universally accessible, it is the learning outcome that needs to be accessible [7]. Based on user profiles, metadata and dynamic connection to resources, the user’s experience can be customized to match his/her abilities. Then an appropriate design is crucial for improving the accessibility and usability of eLearning Systems. De Marsico et al. [5] defined methodological guidelines involving users with disabilities as well as pedagogical experts in the development process, believing that input of different know-how may enrich the quality of e-Learning applications, and provide a more satisfying learning experience. They also include two examples of building and providing learning objects accessible respectively to visually- and hearing- impaired students. Rodriguez et al. describe a project for improving e-Learning experience for the visually impaired, based on ethnomethodology and taking into account psychosocial issues, the user context and experience [9]. Next they created different learning object formats suitable for the blind, including DAISY (Digital Accessible Information SYstem). However, although authors describe the methodology used to improve learning materials no general guidelines are offered to the reader. Within the framework of a project aimed at providing an accessible e-Learning platform for disabled and adult learners, Santos et al. [10] illustrate a methodology for developing standard based accessible courses which use two-step evaluations. However for the totally blind, more specific UI features are necessary than those provided
24
M.C. Buzzi, M. Buzzi, and B. Leporini
in this study, such as providing a page overview, full control of interface elements and easy and rapid navigation via keyboard. E-Learning is a great opportunity for visually-disabled people, provided that both the interactive environment (created by the LMS) and the learning objects are properly designed and delivered. In this paper we mainly focus on accessibility and usability of the virtual environment for the blind, taking as an example the environment offered by Moodle. Concerning Moodle, Debevc et al. [4] compared usability of two LMSs, i.e. Moodle vs a proprietary system, when delivering an identical package of educational materials. This usability study was carried out with a SUMI (Software Usability Measurement Inventory) evaluation that is not specific for e-Learning environments. However indications derive from this study, that point out Moodle’s weaknesses (vs the proprietary system) in efficiency as well as learnability.
3 The Moodle Virtual Learning Environment To evaluate accessibility and usability of a Virtual Learning Environment (VLE) via screen reader, we chose Moodle, an Open Source LMS which offers a rich VLE. Thanks to its large spectrum of available tools and options it has become very popular worldwide. Specifically it offers (http://www.moodle.org): Assignment modules, Blogs, Chat, Choices with multiple response options, Course resources (Moodle pages, uploaded files or web links), Databases, Forums, Glossaries, Interaction, Lessons, SCORM packages, Surveys, Quiz module, Wikis and Workshop modules. The richness of the VLE is Moodle’s strength but the environment‘s complexity can also create difficulties in interaction via screen readers. 3.1 Evaluation Methodology In the first step of our research, we analyzed two demo courses available on the website (http://demo.moodle.org) by using the screen reader JAWS for Windows (http://www.freedomscientific.com) v. 9.0 and 10. We used both the MS IE version 7.0 and the Mozilla Firefox version 3.0.5 browsers. The test was carried out by all the authors independently; afterwards, outcomes were compared and integrated. One author has been totally blind since childhood and uses the JAWS screen reader every day; thus she knows this tool’s functions very well and is able to use advanced commands. By analyzing the test results we noticed that in spite of her great expertise using JAWS, she was unable to perceive the exact structure of the layout. This paper’s sighted authors carried out the test using only JAWS basic commands, but viewing the Moodle UI allowed them to rapidly understand the origin of obstacles encountered when interacting via screen reader. Therefore, integrating both these outcomes led to a more accurate analysis. The different experiences of the authors when using JAWS allowed us to cover a variety of interaction modalities: i.e. both basic commands, which simulate the use of beginner users, and advanced screen reader functions. 3.2 Exploring the Moodle Demo To illustrate how a blind user interacts with the Moodle environment, we explore several pages related to two selected courses, according to specific features to be
Accessing e-Learning Systems via Screen Reader: An Example
25
tested: main page functions, topic outline of a selected course, interacting environment, and so on. When opening the demo page, before doing the log-in, JAWS announces four headings. Through the special command “Insert+F6” JAWS shows the headings list. Although different levels are assigned to these headings, the levels order used is unsuitable. In fact, level 1 is associated with the course title; thus, by pressing the key “1” the virtual focus moves to the beginning of the page. In this case, it would be more appropriate to assign the level 1 to the “Log in” section, which logically is the first part of the interface the user must navigate to enter the system. After the login, the list of available courses is shown. Once the chosen course is clicked, its page is loaded and opened. The screen reader informs the user about headings as well as the number of links. Basically, this page seems to be accessible, because several accessibility features are detected (for instance, a hidden link pointing to the “main content”) by the screen reader. Also, headings are immediately identified and announced by JAWS. But, are all these features suitable for comfortable navigation by blind users? Let us consider the “Higher Education Film Studies Module” (in the following referred to as Course A), available as a demo course.
Fig. 1. Moodle demo - Course A
After selecting this course the following elements are detected by JAWS: • Headings: 32 headings are announced. By exploring the heading list (Insert+F6 command) we note that most of them refer to the course, but the others are related to the e-learning tools (e.g. chat, blog, and so on). Those headings are too numerous to be navigated comfortably. Although headings are used, the page is too long to be read by keyboard via screen reader. Furthermore, it is not very clear which headings refer to the course modules and which ones to the interacting platform functions (see below).
26
M.C. Buzzi, M. Buzzi, and B. Leporini
• Layout table: to show the Topic outline a table has been used for the layout. Specifically two tables are used, the second nested in the first cell of the first table, but JAWS announces only one table. Although headings are used to split the long content, a table should not be used for the rendering. Moreover, the “summary” attribute of the table is “table layout”, totally useless. A more appropriate summary value could be “table of contents”; in this way, by just pressing the letter “t” JAWS will announce immediately “Table of contents”. If a layout table is used, at least a meaningful summary should be applied. • Number of links. The page contains too many links (in our case 115 links). • Images. Many of the images in the pages of the course are not perceived by the screen reader since no alternative text is provided or descriptions are negligible. After the log-in as teacher, we tried to insert an image and found that the alternative text is now mandatory. However there is no control on the number of characters inserted, so it is possible to skip this edit field with only a space press. In order to understand whether the issues encountered in the selected course could be generalized, we attempted to navigate another demo course. Specifically, we picked up the course “Moodle Features Demo” (in the following referred to as Course B). Also the main page of the second course includes a great number of links, two tables used for visual layout, and 25 headings. However, the level assigned to each heading as well as their arrangement differ from Course A. In this case, first the headings for interacting with the Moodle environment are available, and then the Topic outline. Consequently, it seems the UI is not consistent among the courses even if they are very similar (probably due to changes in the Moodle versions used to create these courses). However, headings arranged in this way might make the page structure somewhat unclear. In fact, it is difficult to know how many modules compose the course and their titles and contents, since the module number has not been included in the tag and thus does not appear in the headings lists (Fig. 3); to obtain this information the user must read the page in a sequential way, as shown in the Fig. 2 – left section. All headings are mixed and not well grouped by clearly indicating which are modules (See Fig. 3), and which are associated with the virtual environment (on the right side in Fig. 1). In other words, the main regions: “menu”, “topic outline” and “interacting actions” should be better indicated. For example, by opening the page related to “eXe SCORM package”, the screen reader recognizes some unclear elements: • various graphical icons; the image tags have the ALT attribute, but the alternative descriptions used are not clear for the type of the icon. For instance, several links are preceded by an icon labelled “Completed” (See Fig. 2 – right section). It is not clear what this means. • Frame: in the page a frame called “scoframe1 frame” is identified by JAWS, which is not particularly clear. Even if frame content is explored in a sequential way, is unclear what the student. According to the last version of accessibility guidelines [WCAG 2.0] frames should be avoided (deprecated). The following figure reports the page content as it is interpreted by the screen reader JAWS.
Accessing e-Learning Systems via Screen Reader: An Example
COURSE B Features Demo: eXe SCORM package Heading level 1 Moodle Features Demo You are logged in as Link Sam Student (Link Logout) Heading level 2 You are here list of 4 items Link Moodle Demo/ ... ? eXe SCORM package list end … eXe Scorm Package Link Expand/Collide Graphic Completed Link eXe Scorm Package list of 5 items nesting level 1 Graphic Completed Adding an image Graphic Completed ... list end nesting level 1 list end Review Mode scoframe1 frame Adding an image Graphic img6_2 The eXe logo In eXe it is easy to include images with your web content. ... scoframe1 frame end
Fig. 2. Page segments sequentially read by JAWS (up/down arrows): on the left Course A main page, on the right a SCORM module function of Course B
Fig. 3. List of headings of the demo course A (“Insert+F6” JAWS command)
28
M.C. Buzzi, M. Buzzi, and B. Leporini
In short, although the pages seem to be accessible, our initial interaction using the screen reader JAWS shows some usability issues when interacting via keyboard and in a sequential manner. The few pages considered in our preliminary evaluation highlighted some difficulties orienting oneself among different information (i.e. headings and links), as well as in handling conceptual information. For instance, in some cases pictures or graphical representations are used to provide information on modules or on intersecting concepts. In a learning system it is also important to make those concepts accessible and usable by alternative modalities.
4 Applying ARIA When designing a web interface it is essential to keep in mind both usability and accessibility principles. Accessibility is a basic pre-requisite for allowing access to page content, while usability guarantees an easy, simple, efficient, rapid and satisfactory navigation and interaction. Navigation is vital for special-needs persons, and in particular for the blind, because it is crucial for them to be aware of their current location on the webpage and how to return to the beginning, or how to reach a certain point in the materials [4]. The use of ARIA would enhance Moodle’s usability in many ways, as explained and exhaustively illustrated by the ARIA best practises document [13]: • Reduce the amount of unnecessary text announced. Specifically the use of Table as layout may be silently ignored by the screen reader if the table is tagged with the presentation role :
…
• Definition of regions to allow the user get a page overview. Moodle utilizes heading levels to structure the page and to allow the user to perceive the whole structure. Using only standard (X)HTML code, the usability of keyboard navigation is reduced so blind users are forced to use tabbing for accessing active elements (form elements and links). To simplify interaction and allow easy jumping to main interface regions, developer usually relay on creating a link to the main content or use heading levels to structure the page (since the screen reader gives a table of headings). However the use of headings to mark sections is not consistent across web sites [13]. ARIA allows marking sections with standard specifying XHTML landmarks or defining customized regions:
…
• Simplifying keyboard navigation. The use of landmarks/regions also allows to simplifying navigation via keyboard since the user may jump from one region to the next by pressing a key (in JAWS v.10 the “;”). Furthermore, the developer using the attribute “flowto” defines the order in which regions should be visited:
These are basic considerations to improve the interaction via screen reader with the VLE offered by the LMS, that should be integrated on the basis of all observations reported below. Further improvements are possible in the specific learning object
Accessing e-Learning Systems via Screen Reader: An Example
29
prepared by the teacher; thus it is very important to provide automatic tools to support the easy creation of accessible and usable learning objects.
5 Conclusion In this paper we analyze the usability of the Moodle Learning Environment for the blind. Specifically we analyzed two demo courses provided as examples by the system, highlighting features that could be improved. We mainly focus on the navigability of the virtual environment provided by Moodle, which by offering several educational tools integrated into one system, may create complex interfaces. It is important to notice that LMSs may greatly favours the student learning process since the same educational material may be transmitted anywhere, anytime, at any learning rhythm, in a format suited to each individual's ability. On the other hand, since LMSs automatically add a virtual environment to the educational material, if the virtual environment layout is not appropriately designed with a thorough knowledge of accessibility and usability issues, it may induce problems that could be spread to the learning objects themselves. This highlights the importance of considering usability issues from the beginning of the development of every LMS. Making a VLE suitable for the abilities and skills of all users offers many challenges. When defining the graphical UI it is fundamental to consider the needs of sighted users but the needs of the blind should also be kept in mind when writing the UI code. Specifically, the same information should be provided through both visual and auditory channels, the design should be optimized for reading via screen reader, the UIs should be easy to use via keyboard and no additional cognitive effort should be required of the blind user. In conclusion, we believe that our findings could have general applications and that applying ARIA would enhance usability via screen reader in any Virtual Learning Environment.
References 1. Ardito, C., Costabile, M., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V.: An Approach to Usability Evaluation of e-Learning Applications. Universal Access In the Information Society 4(3), 270–283 (2005) 2. Card, S.K., Moran, A., Newell, T.P.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates Inc., New Jersey (1983) 3. Debevc, M., Bele, J.L.: Usability testing of e-learning content as used in two learning management systems. European Journal of Open, Distance and E-Learning (2008), http://www.eurodl.org/materials/contrib/2008/Debevc_Bele.htm 4. Debevc, M., Verlic, M., Kosec, P., Stjepanovic, Z.: How Can HCI Factors Improve Accessibility of m-Learning for Persons with Special Needs? In: Stephanidis, C. (ed.) HCI 2007. LNCS, vol. 4556, pp. 539–548. Springer, Heidelberg (2007) 5. De Marsico, M., Kimani, S., Mirabella, V., Norman, K.L., Catarci, T.: A proposal toward the development of accessible e-Learning content by human involvement. UAIS Journal 5(2), 150–169 (2006)
30
M.C. Buzzi, M. Buzzi, and B. Leporini
6. Eijl, P., Pilot, A., Voogd, P.: Effects of Collaborative and Individual Learning in a Blended Learning Environment. Education and Information Technologies 10(1-2), 51–65 (2005) 7. Kelly, B., Phipps, L., Howell, C.: Implementing a holistic approach to e-Learning accessibility (Retrieved), http://www.ukoln.ac.uk/web-focus/papers/alt-c2005/accessibility-elearning-paper.doc 8. Nielsen, J.: Usability inspection methods. In: Heuristic evaluation, pp. 25–62. John Wiley & Sons, Inc., New York (1994) 9. Rodriguez, E.P.G., Domingo, M.G., Ribera, J.P., Hill, M.A., Jardi, L.S.: Usability for All: Towards Improving the E-Learning Experience for Visually Impaired Users. LNCS, pp. 1313–1317. Springer, Heidelberg (2006) 10. Santos, O.C.B., del Viso, J.G., de la Cámara, A.F., Sánchez, S.P., Gutiérrez, C.R., Restrepo, E.: HPCN-Europe 1994. LNCS, pp. 796–805. Springer, Heidelberg (2007) 11. Sloan, D., Heath, A., Hamilton, F., Kelly, B., Petrie, H., Phipp, L.: Contextual web accessibility - maximizing the benefit of accessibility guidelines. In: Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (2006) 12. Squires, D., Preece, J.: Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers 11(5), 467–483 (1999) 13. W3C. WAI-ARIA Best Practices. W3C Working Draft 4 February (2008), http://www.w3.org/TR/wai-aria-practices/ 14. Wilson, R., Landoni, M., Gibb, F.: A user-centered approach to e-book design. The Electronic Library 20(4) (2002) 15. Zaharias, P.: A usability evaluation method for e-learning: focus on motivation to learn. In: Proceedings of CHI 2006 extended abstracts on Human factors in computing systems (2006)
Using Tablet PCs and Pen-Based Technologies to Support Engineering Education Ignacio Casas1, Sergio F. Ochoa2, and Jaime Puente3 1
Department of Computer Science, Pontificia Universidad Catolica de Chile, Chile 2 Department of Computer Science, Universidad de Chile, Chile 3 Microsoft Research, Redmond, WA, USA [email protected], [email protected], [email protected]
Abstract. Several experiences and results of the Tablet PC adoption have been reported, mainly in American universities. Although the benefits seem to be highly interesting, it is not clear if they are replicable in developing countries. In order to try to understand the impact of Tablet PCs on engineering education in Chile, the authors conducted several experiments at the two traditional Chilean universities. This paper reports the experiences and the obtained results, comparing them with those obtained in American universities. Keywords: Tablet PCs, Pen-based Technologies, Engineering Education, Mobile Computing.
by the authors in order to determine the impact that pen-based computing have in engineering education in the two main Chilean universities. Section 5 discusses the obtained results. Finally, section 6 shows the conclusions and the future work.
2 Tablet PCs in High Education During the last years several researchers have been doing experiments in order to determine the impact of using Tablet PCs in education. Some of them believe that a “Tablet PC has the potential to dramatically alter the educational process. This new technology significantly changes the way students and teachers interact. It adds completely new dimensions to classroom interaction by providing digital ink and drawing tools for writing, sketching, and drawing; and for real-time collaboration” [1]. At the moment we have seen reports of particular experiences where Tablet PCs have became interesting tools in certain scenarios. Now the challenge is to identify in which scenario the Tablet PCs can effectively add value. For example, Anderson et al. show how Tablet PCs can be used to improve the knowledge delivery and facilitate the interaction between students and the instructor during a distributed lecture [2]. Berque et al. report the results of using Tablet PC and DyKnow tools to support collaborative problem solving [3]. Davis et al. and Anderson et al. report experiences of using Tablet PC to take notes during a class [2, 4]. In addition, the authors have shown, in previous works, how these mobile devices can be used to improve the evaluation process in computer science courses [9] and also to support the activities of communities of practice in science education [5]. There are also several articles describing educational experiences on physics, mathematics and agronomy, in which Tablet PCs have been used as the supporting platform for the teaching-learning process. Considering the experiences reported in recent literature, and focusing just on the main features of the Tablet PCs, we can say that the main activities to be supported with these devices are the following ones: 1. Taking notes. Students can take notes on the slides the professor shares with them, or just on a digital blank sheet. These digital notes can be organized, shared, searched, emailed and linked to other resources [6]. In that sense, this functionality seems to be highly useful, and better than the paper notebook or even a laptop. 2. Enrich the lecture presentation. Instructors can use Tablet PCs to enrich the information they are presenting during a lecture. It could be used, for example, to do marks or make annotations on the slides, or use the Tablet PC as a replacement for (or supplement to) the black/whiteboard [2]. Since digital annotations made on these devices can be shared, the cognitive load of the attendees is reduced. 3. Support group work. During work meetings the group members (i.e. students and/or instructors) usually carry out brainstorming, discuss proposals/alternatives to carry out the job, and validate ideas or the work done. Usually a black/whiteboard is used to support these activities; however Tablet PCs could also be used for this purpose [3]. A benefit of using these devices instead of whiteboards is that notes written on the space shared by the attendees are easier to record and share. In addition, it is not longer needed to erase the whiteboard because its physical space is completely used.
Using Tablet PCs and Pen-Based Technologies to Support Engineering Education
33
In this list we have excluded the activities that can be performed with a laptop. In other words, we included just the activities requiring the special features of a Tablet PC. Please note there are activities that are variants of the listed ones, which were not included in the list. Summarizing, it seems the use of these computing devices could replace the use of the paper notebook and also the black/whiteboard.
3 Impact on the Students and Instructors Experiences reported in the literature show the students are usually comfortable using Tablet PCs to take notes and perform problem-solving activities [13, 14, 15]. Something similar happens with the instructors’ opinion [5, 15]. In addition, most of the researchers that explore the use of these devices in educational scenarios state that Tablet PCs facilitate the active learning [12, 16][Roschelle et al., 2007]. Considering these previous works, some of the main contributions reported about the use of Tablet PCs, are the following ones: • Benefits for students. Students recognize the possibility to take notes is an important contribution of Tablet PCs usage. The free-style handwriting possibility makes the students more comfortable to express their ideas in sketches or annotations [14]. The possibility to exchange digital resources among the students (or with the instructors) was recognized as an important feature [16]. Finally, students find valuable the possibility to store and manage their courses information in digital format [16]. • Benefits for instructors. Many instructors have found that Tablet PCs not only can replace the black/whiteboard, but also extend the resources pool to be used during a lecture [6]. It makes the lectures more dynamic and interactive [Roschelle et al., 2007]. In addition, the annotations the instructor performs during a lecture can be easily stored and shared with the students. Several researchers report an increment of the students’ interest during the classes and also an improvement in the way used by the instructors to deliver knowledge [13, 16]. Based on the authors’ experience, many of the envisioned benefits are inherent to the use of laptops during lectures. Other benefits, for example the change of students’ and instructors’ attitude during the lectures, could be the result of the redesign of the teaching-learning process performed to take advantage of the Tablet PCs features. Of course, there are several benefits that really are a consequence of these devices usage, for example the possibility to make handwriting annotations on the lecture slides. Next section describes a preliminary study carried out in two traditional Chilean universities, which tries to determine the impact of Tablet PCs in computer science and engineering courses. The study also discusses the obtained results and compares them with the findings reported by other researchers in USA.
4 Empirical Study In order to try to understand the impact that the use of Tablet PCs could have in engineering education in Chile, the authors conducted an empirical study in two traditional
34
I. Casas, S.F. Ochoa, and J. Puente
Chilean universities: Universidad de Chile and Pontificia Universidad Catolica de Chile. This study was performed in computer science courses at the engineering school and it involved several types of Tablet PCs and also software products. 4.1 Experience in the Universidad de Chile This experience involved students of a computer science course (CC51A: Software Engineering) during two semesters: fall 2008 (29 students) and spring 2008 (20 students). Ten Tablet PCs were delivered among the students in order to support two main activities: course examinations using MOCET [9] (Figure 1), and group problem-solving using MS OneNote [10]. In addition, the instructor used Classroom Presenter [2] during half of the course lectures, and students using Tablet PCs took notes during lectures using this software. When the course was finishing, the instructor and the students filled a survey that evaluated the experience of using these devices. Some of the issues identified by the participants were the following:
Fig. 1. Use of MOCET at University of Chile
• The hardware matters. Students using particular models of Tablet PCs (DELL Latitude XT and HP Pavilion TX1000) consistently reported problems to write/erase annotations, regardless of the software they were using. In addition, these students also reported precision problems when these devices recognize the stylus location on the screen. • The working conditions matters. Students and the instructor identified the importance of avoiding situations where only some of the students had Tablet PCs. Some educational activities designed to take advantage of the Tablet PC features, for example the group design, were difficult to perform for students that did not have available these devices or a whiteboard. Therefore, if you are going to perform an activity that takes advantage of these devices’ features, be sure that all the students count with Tablet PCs. • Training is required. Students and the instructor reported an important improvement in the usefulness of the devices and the effectiveness of their work, once they were trained in the use of Tablet PCs. • The instructor mobility becomes reduced. Because the Tablet PC must be connected to the projector through a physical cable during a lecture, the mobility of the
Using Tablet PCs and Pen-Based Technologies to Support Engineering Education
•
•
•
• •
•
35
instructor becomes reduced. After some sessions the instructor started to use MaxiVista [7] as intermediary to recover the mobility. Although the solution was good, it required to have two computers in the classroom to deliver the lecture. Tablet PCs do not replace the blackboard. Although the Tablet PC’s functionalities are comparable to those of a whiteboard (and even better in some aspects), the information shown to the attendees through these devices is replaced when the instructor change the slides. In the case of the whiteboard, it acts as an extended screen of the instructor presentation; therefore it keeps visible the information written on it, even when the instructor changes the slides. In other words, students and the instructor think the whiteboard and the Tablet PC are complementary. Useful to make free-style annotations. Students and the instructor highlighted the capability of these devices to make free-style handwriting annotations. Several software products were used to carry out this activity; all of them were useful and comfortable for the users. They are similar to laptops in several aspects. Students and the instructor identified as an important contribution the capability of mobile computing devices to record annotations and manage information in digital format. They also highlighted the capabilities of these devices to share information and communicate among them in a simple way, even when the users are on the move. However, these functionalities are not only available in Tablet PCs, but also in laptops. Something similar occurs with the possibility to perform simulations, or execute or compile programs during the lectures. In other words, several of the advantages reported as result of the Tablet PCs usage are also present when using laptops. The instructor and the students feel comfortable using Tablet PCs. Users felt comfortable using these devices during the experiences. Although some training was required, the learning effort was worthy. Computing devices could be distracters during lectures. Students and the instructor agreed that mobile computing devices can be a distracting element during the lectures. If they do not have a clear role to play during the lecture, their use compete with the instructor’s speech. The redesign of the lecture style makes them more active. The participants in these experiences recognized the lectures became more active and participative. However, it seems to be a consequence of the lectures style redesign more than a result of the use of Tablet PCs.
4.2 Experience in Pontificia Universidad Catolica de Chile This section describes the experiences of introducing Tablet PCs in two courses at the Pontificia Universidad Catolica de Chile during 2007: “Computing Software Tools for Engineering” which is a workshop course (iic2100) and “Introduction to Programming” (iic1102). The main objective of the experiment was to motivate engineering students to learn and enjoy the art of problem solving and project team work supported by computers. In order to reach such a goal, we started to use Tablet PCs in the classroom and we modified the dynamic of the teaching-learning process in these courses. Tablet PCs were used in the classroom for modeling, problem solving and programming.
36
I. Casas, S.F. Ochoa, and J. Puente
Fig. 2. Use of OneNote at Pontificia Universidad Catolica de Chile
The course iic2100 was the subject of this experimentation during two terms in 2007. Forty students participated in each semester. Twenty one Tablet PCs (HP Compact TC4200) were delivered among the students. These devices were shared by the students and they were used to support problem solving during lectures (Figure 2). The impact of the Tablet PCs usage was measured based on opinion surveys from students, teaching assistants and professors. A similar experimentation process was conducted in the course iic1102: Introduction to Programming, during the Fall and Spring terms in 2007. One hundred students participated each semester. Twenty one Tablet PCs (HP Compact TC4200) were delivered among the students in order to support group problem solving during lectures. In spite of all the positive aspects of the use of technologies in the classroom, a few problems were observed; for instance, annoying delays at the beginning of some activities due to instability of the wireless network and software server (a technical assistant was present in the classroom to solve this kind of incidents). Problems also occurred with software tools and their different versions. It was also observed that at the beginning of the class some students had difficulties to turn on the Tablet PC or manage the stylus-pen, due to simple lack of knowledge (this problem was easily solved). Importantly, some students became distracted with the technology and engaged in Internet searching, chatting, playing or e-mailing, but after several sessions this distraction diminished. The students highlighted, as a positive factor, the possibility to solve problems during classes. They considered the use of Tablet PCs helped to make the lectures more active and interesting. However, they also identified limitations in the problemsolving process because not all of the students counted with one of these devices. Most of the students agreed that the use of technology motivated them. It is also important to note that in both courses there were no students considering Tablet PCs as an inhibitor factor; however they recognized that some training is required to take advantage of these devices features. They considered that Tablet PCs facilitated the discussions and the group work during lectures.
5 Discussion At the moment, the use of Tablet PCs in computer science courses seems to have positive effects. However, there is not a comprehensive and scientific study that
Using Tablet PCs and Pen-Based Technologies to Support Engineering Education
37
shows which are the real benefits of including this kind of technology in our courses. Although using mobile computing devices during lectures brings several benefits, it does not seem to be consequence of using just Tablet PCs. Our experiments have also identified that computing devices can become a distracter factor during lectures, if there is not a clear role assigned to them. These experiments also have shown that the use of Tablet PCs motivates to students and professors, however it is not clear if the use of laptops generates the same effect. Provided the courses involved in the experiments tried to take advantage of the Tablet PCs features, and following the recommendations of other researchers in the area, we redesigned their teaching-learning process. Now these courses are more active and there is more interactive work during the lectures, which produced a positive impact among the participants. However this result seems to be a consequence of the redesign process more than the inclusion of Tablet PCs. The experiences help us to identify that not all of the Tablet PCs provide an adequate support for these activities, and the mix of technology and paper-based work is not a good combination; particularly, if students have to use paper notebooks because there are not enough computers for everybody. Although the students recognized the importance of Tablet PCs for taking notes and facilitating group work, the use of these devices requires a training process. Instructors were happy when using Tablet PCs. They think these devices help to improve the quality of the lectures; however they are not able to replace the black/whiteboard. In one of the experiences, a lack of the instructor mobility was identified because of the use of Tablet PC to deliver the lecture. Such problem was then solved using additional software technology.
6 Conclusions and Future Work The use of Tablet PCs in engineering education seems to be growing more and more every day. Several researchers have reported experiences of the Tablet PCs adoption, mainly in USA. They have identified a list of benefits derived from the use of these devices. In order to try to understand the impact of this type of computers t on engineering education in Chile, two experiences were conducted at two traditional Chilean universities. The obtained results show some similarity with those obtained in American Universities. However there are others that are a bit different. The lack of resources for every student opens new challenges for the adoption of these technologies in developing countries. The experiments have also shown that several benefits associated to the Tablet PCs adoption are more a consequence of others activities around this situation. At the moment it is not clear which benefits are the direct consequences of the Tablet PCs adoption, however the balance seems to be positive. More experimentation and scientific work is required to understand its impact of this emerging technology.
Acknowledgements This work was partially supported by Fondecyt (Chile), grant Nº 11060467 and also by Hewlett Packard through the Technology for Teaching Higher Education Grant Initiative.
38
I. Casas, S.F. Ochoa, and J. Puente
References 1. Alvarado, C., Anderson, R., Prey, J., Simon, B., Tront, J., Wolfman, S., Tablet, P.C.: Computing Curriculum. In: Summary of the Tablet PC and Computing Curriculum Workshop, Seattle, USA, August 4 (2004) 2. Anderson, R., Davis, P., Linnell, N., Prince, C., Razmov, V., Videon, F.: Classroom Presenter: Enhancing Interactive Education with Digital Ink. IEEE Computer 9(40), 56–61 (2007) 3. Berque, D., Bonebright, T., Dart, J., Koch, Z., O’Banion, S.: Using DyKnow Software to Support Group Work: A Mixed-Method Evaluation. In: Prey, Reed, Berque (eds.) The Impact of Tablet PCs and Pen-based Technology on Education, pp. 11–20. Purdue University Press (2007) 4. Davis, K.M., Kelly, M., Malani, R., Griswold, W.G., Simon, B.: Preliminary Evaluation of NoteBlogger: Public Note Taking in the Classroom. In: Prey, Reed, Berque (eds.) The Impact of Tablet PCs and Pen-based Technology on Education, pp. 33–42. Purdue University Press (2007) 5. Herrera, O., Ochoa, S., Neyem, A., Betti, M., Fuller, D., Aldunate, R.: Mobile Portfolio to Support Communities of Practice in Science Education. In: Proceedings of HCI International 2007, Beijing, China (July 2007) 6. Huettel, L.G., Forbes, J., Franzoni, L., Malkin, R., Nadeau, J., Nightingale, K., Ybarra, G.A.: Transcending the traditional: Using tablet PCs to enhance engineering and computer science instruction. In: Frontiers in education conference (FIE 2007) (2007) 7. Bartels Media GmbH. MaxiVista (Last visit, December 2008), http://www.maxivista.com/ 8. Nguyen, H., Bilen, S., Devon, R., Wise, J.: Adopting Tablet PCs in Design Education: Student Use of Tablet PCs and Lessons Learned. In: Richards, G. (ed.) Proceedings of World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2007, pp. 1172–1177. AACE, Chesapeake (2007) 9. Ochoa, S., Neyem, A., Bravo, G., Ormeño, E.: : MOCET: a MObile Collaborative Examination Tool. In: Proc. of HCII 2007, Beijing, China (July 2007) 10. Microsoft Corp. OneNote (2007), http://office.microsoft.com/es-es/ onenote/FX100487703082.aspx (Last visit December 15, 2008) 11. Prey, C., Reed, R.H., Berque, D.A. (eds.): The impact of Tablet PCs and Pen-based Technology on Education. Purdue University Press (2007) 12. Prey, C., Weaver, A.: Tablet PC Technology: The Next Generation. IEEE Computer 9(40), 32–33 (2007) 13. Price, E., Simon, B.: A Survey to Assess the Impact of Tablet PC-based Active Learning: Preliminary Report and Lessons Learned. In: Prey, Reed, Berque (eds.) The Impact of Tablet PCs and Pen-based Technology on Education, pp. 97–105. Purdue University Press (2007) 14. Sommerich, C., Collura, K.: Learning with Mobile Technology in high School: A HumanFactors Perspective. In: Prey, Reed, Berque (eds.) The Impact of Tablet PCs and Penbased Technology on Education, pp. 127–136. Purdue University Press (2007) 15. Toto, R., Lim, K., Wise, J.: Supporting Innovation: The Diffusion and Adoption of Tablet PCs in College of Engineering. In: Prey, Reed, Berque (eds.) The Impact of Tablet PCs and Pen-based Technology on Education, pp. 147–155. Purdue University Press (2007) 16. Tront, J.G.: Facilitating Pedagogical Practices through a Large-Scale Tablet PC Development. IEEE Computer 9(40), 62–68 (2007)
Optimal Affective Conditions for Subconscious Learning in a 3D Intelligent Tutoring System Pierre Chalfoun and Claude Frasson Département d’informatique et de recherche opérationnel, Université de Montréal, Montréal, Canada {chalfoun,frasson}@iro.umontreal.ca
Abstract. In this paper we take a closer and in-depth look at initial results obtained from a previous novel experiment conducted with a 3D subliminal teaching Intelligent Tutoring System. Subliminal priming is a technique used to project information to a learner outside of his perceptual field. Initial results showed great promise by illustrating the positive impact of the subliminal module on the overall emotional state of the learners as well as their learning performances. Indeed, since emotion monitoring is critical in any learning context, we monitored the physiological reactions of the user while they learned and while they answered questions. We present a detailed and precise look at the optimal affective conditions that set the best learners apart. We will also explain a most surprising finding: the positive long term impact of subliminal priming on the entire learning process. Keywords: optimal affective conditions, HCI, subconscious learning, 3D ITS.
perception or complete awareness to the task at hand [5-8]. In fact, the existence of perceptual learning without perception has been neurologically proven and accepted [9]. Furthermore, recent work has put forth the performance increase in performance when using a subliminally teaching Intelligent Tutoring System [10]. Yet, subconscious learning systems are still widely absent in the HCI community. We intend to investigate in this paper the optimal emotional state of learners when using a subliminal teaching ITS by stating two research questions. First, in learning to solve a problem in a 3D virtual system, is there a significant emotional state in which the best learners are that sets them apart from the rest? Second, in answering questions following a learning session, what significant relationship can we establish between learners’ emotional state and subliminal projections? The organization of this paper is as follow: In the next section, we will present and discuss the previous work related to various aspects of our research. The following section describes the experiment setup and depicts the various aspects related to subliminal stimuli in a virtual 3D tutoring system. The obtained results will follow the experiment section leading to the last section where we conclude and present future work.
2 Related Work To the best of our knowledge, only a handful of papers in various fields have claimed the use of subliminal priming as a support for memory in the HCI community. The first and most referred to is Wallace’s text editor program [11]. In this experiment, Wallace and colleagues put forward two important findings: (1) the projected stimuli must take into account the specifications of the computer such as screen resolution and refresh rate (2) that the frequency at which subjects requested help was much lower when the requested information was projected subliminally. The Memory Glasses by [5] used wearable glasses that projects subliminal cues as a strategy for just-in time memory support. The objective was to investigate the effect of various subliminal cues (correct and misleading) on retention in a word-face learning paradigm and compare recall performance. Another use of priming for memory support can be found in the thesis of [12] where the author assesses the effects of brief subliminal priming on memory retention during an interference task. Finally, our most recent work showed the positive impact of subliminal stimuli on the learner’s performance [10]. Besides seeming to impact memory, subliminal priming can also have an emotional consequence on learners. Indeed, subliminal priming can have an emotional impact on the self-attribution of authorship of events [13]. Subjects were asked to compete against a computer in removing non words such as “gewxs” from a computer screen in the fastest time possible. However, after a determined amount of time, the computer would remove the word. Subliminal primes of self-associated words like “I” and “me” before an action increased the personal feeling that it was the participant that eliminated the non word and not the computer, thus increasing the feeling of selfauthorship of events. Furthermore, visual subliminal stimulus has been neurologically proven to have an impact in many physiological signals, namely the galvanic skin response (correlated to arousal) [14].
Optimal Affective Conditions for Subconscious Learning in a 3D ITS
41
Since we also use physiological sensors to monitor the emotional reactions of the learner, it would be relevant to sum some of the work related to using physiological sensors to record and analyze emotions that can occur in a learning environment. Physiological signals are generally correlated with emotions by associating specific signals, such as skin conductance and heart rate, to valance and/or arousal [15]. Indeed, the Empathic Companion is a good example where multiple physiological sensors, namely galvanic skin response (also referred to as skin conductance), heart rate and respiration were taken in real-time to analyze and adapt the tutor to the emotional reactions of the learner in a virtual 3D ITS [16]. Further research has analyzed a more detailed and relevant emotional significance of physiological signals, either in complex learning or gaming [17-19].
3 Experiment The current experiment uses precise and timed subliminal projections in a 3D intelligent tutoring system while monitoring the physiological reactions of the learner. In the same time we record the actions on the screen as well as the facial movements of the learners. Those visual recording are crucial to remove noise and identify events of special interest. Moreover, we constructed the subliminal cues in a way which would accelerate the learning process by triggering and enhancing an already possessed knowledge. 3.1 Design of the Experiment Indeed, the focus of the experiment is to visually teach, in a virtual 3D environment, the construction of an odd magic square of any order with the use of neither a calculator nor one mental arithmetic operation. A magic square of order n is a square containing n2 distinct integers disposed in a way such as all the n numbers contained in all rows, columns or diagonals sum to the same constant. The first part of Fig. 1. below depicts such a square. Magic Square
Trick #1
Trick #2
Trick #3
Fig. 1. Experiment design : Magic square and the three tricks taught
To construct the following square, one must successively apply three simple tricks. These tricks are illustrated in Fig. 1 and labelled trick 1 to 3 respectively. We decided to show the learners multiple examples of each trick without explaining how the trick works. As an example, the first trick to construct any magic square is to place the following number one square above and two squares to the right of the previous one (exactly like a knight’s move in chess). If we look at the second picture of Fig. 1, we notice that number 15 is placed one square above and two squares to the right of
42
P. Chalfoun and C. Frasson
number 14. The same logic applies to numbers 1 and 2, 4 and 5 and so forth. Instead of giving away the answer to the first trick, we ask the subjects to deduce the rule by themselves. This is where the subliminal stimulus comes into play. We will have two groups, one group will take part of the experiment without subliminal stimuli (control group) and the tutor will subliminally send the answer to the other group. We will then compare performances, trick completion time, question completion time as well as physiological signal variations. The teaching material is separated into parts, or PowerPoint-like slides, and displayed at a slow rate to give every learner an equal chance at fully reading each “slide”. The subliminal stimuli and threshold were carefully chosen following the neural bases of subliminal priming [9]. Each stimulus was preceded by a 271 ms pre-mask of random geometrical figures, a 29 ms prime and a 271 post-mask of random geometrical figures. The subliminal stimuli that will be presented to one of the two groups will be displayed at significant places before and after specific slides. The experiment intends to “boost” learning by priming the answer before showing the corresponding slide. Fig. 2. shows a diagram of the way subliminal priming will take place between slide 1 and slide 2 when learning to deduce the inner working of the first trick.
271 ms
mask
29 ms prime 271 ms Slide 1
mask
Priming between slides
Slide 2
Fig. 2. Subliminal priming of the solution between 2 slides
The learners were instructed to answer a series of two to three questions following each learned trick to test their knowledge. The learners were instructed to finish the experiment as quickly and efficiently as possible. No time limit was imposed. A base line for the physiological signals preceded all monitored activities. A questionnaire preceded the experiment aiming at collecting demographical data as well as the gaming experience of the subjects. Another series of questions were asked at the end of the experiment to evaluate the learner’s appreciation and more importantly their overall appreciation of the system. 3.2 The 3D Virtual Environment Learning takes place in a game-like environment called MOCAS [20] as show in Fig. 3. The experiment has three rooms like the one illustrated on Fig. 3. Each room teaches one trick. MOCAS takes place in full-screen mode for a better immersion and less window distracting events. Furthermore, the system clock is hidden so users don’t get distracted by continuously monitoring the time they have spent on each lesson. The interactions between the avatar’s learner and the pedagogical agents are done via mouse clicks. This interaction is important because learners have a time
Optimal Affective Conditions for Subconscious Learning in a 3D ITS
43
window of 30 seconds to answer each question. If they feel that time was not enough, they can simply re-click on the agent and the question restarts. This re-click factor was important in distinguishing good from bad learners. The learners are instructed to continue once they are convinced they have discovered the inner working of each trick. They are then asked to answer a series of questions (two to three) by another set of visually different pedagogical avatars. Each question is related to the last trick learned. The agent asks the user to correctly place a number in a magic square. The learner responds by choosing the path that correctly answers the question. Physiological signals of the learners were also monitored in real-time and saved for further analysis. The used signals were heart rate, galvanic skin response, respiration rate and skin temperature. The signals are managed by the ProComp Infinity encoder [21].
Path leading to the Question Agents
Avatar of a pedagogical agent
Clicking on the Pedagogical agent starts the lesson.
Teaching material
Avatar of the learner
Fig. 3. 3D virtual learning environment
3.3 Learners Tested A total of 31 healthy volunteers, 16 men and 15 women, took part of the experiment. One participant had to be removed because of a major recording issue in the videosignal synchronisation module. The sample’s mean age was 26 (SD = 4.51). Only two volunteers had extensive video gaming experience. All the others gaming experience ranged equally anywhere from weak to moderate high. A repartition of the learners can be found in table 1. Table 1. Participants’ distribution
Group A : no subliminal stimuli Group B : primed with subliminal stimuli Total
Men 8 7 15
Women 7 8 15 30
44
P. Chalfoun and C. Frasson
4 Results The first aspect we wanted to examine was the existence, if any, of a relationship between affective variations and subliminal projections while learning the tricks. Fig. 4 shows the average quantitative affective variations of learners with regards to valence and arousal when learning all three tricks with and without the subliminal module. The signal used and correlated with valence is the heart’s inter-beat interval (IBI) and galvanic skin response was used and correlated with arousal (GSR) [15]. These signal values are normalized by mean-shifting, that is subtracting each signal’s value from the signal’s baseline mean then dividing the result by the signal’s standard deviation. This widely accepted technique enables us to compare learners’ results for it solves the problem of extra-individual physiological differences. Fig. 4 shows the average affective values for a period of 4 second following every subliminal stimulus. The full brown bars represent the average value of the signal for all subliminal stimuli at the precise moment the stimulus was projected (t=0s, s is for seconds). The horizontal dashed bars represent the same averaged value except for it’s computed for the 4 seconds following that projected stimulus (T=t + 4s). Since group A was not primed with subliminal stimuli, we placed markers for each learner at the precise moment where subliminal cues would have been projected if these learners would have been taking the course with the subliminal module. For example, the first four numbers from the left (-0.7, -0.7, 0.3 and 0.6) represent the following situation: on average, all learners in the experiment have had a normalized valence change of zero (-0.7 at moment t=0 and -0.7 after 4 seconds) when learning without the subliminal module compared to a normalized valence increase of +0.3 (0.3 at moment t=0 and 0.6 after 4 seconds) when learning with the subliminal module. Average Quantitative Affective Variations Of Learners While Learning The Tricks Group A
Normalized Mean-Shifted Values With Regards To Baseline
8.0
6.8 6.7
7.0
Group B
Group A
6.0 5.0
4.4 4.4
5.5
Group B
4.0
Group Group B A
3.6 3.2 2.9
3.0
Group B
2.0 1.0
Group A 0.3
Group B
1.7
Group A
0.6
Group B
2.5 2.5
1.0
0.8
Group A
1.1 0.5
0.4
0.0 Valence
-1.0
-0.7 -0.7
Arousal
Valence
All Learners
Arousal Best Learners
-0.9 -0.9
Valence -0.6 -0.6
Arousal Worst Learners
All 3 Tricks
-2.0 Valence And Arousal For All Tricks Broken Down By Best, All and Worst Learners
Without Subliminal Module - Average Value At t=0s
Without Subliminal Module - Average Value At T=t+4s
Fig. 4. Quantitative affective variations when subjected to subliminal stimuli while learning
Optimal Affective Conditions for Subconscious Learning in a 3D ITS
45
The results shown in Fig. 4 are not only statistically significant (p<0.001, α=0.05) but very important for they enable us to distinguish between the best and worst learners in terms of valence and arousal but also in terms of how much variation is considered optimal for success. In this case, having an average positive valence variation increase of about 0.8 and arousal increase between 2.5 and 2.9 is what our system should be looking for. In fact, we can clearly see at the far right part of Fig. 4 that the worst learners, those who made the most mistakes, were the ones who had a negative valence variation. Checking the results with Lang’s two dimensional space [15] informs us that a negative valence leads to a negative emotional state and thus not optimal for learning. Since subliminal projections increase valence variations, our system could then detect this negative emotional state and start projecting more stimuli until an optimal state is reached. The second aspect we wanted to investigate was the affective state of learners when answering questions. It is important to mention that no subliminal priming took place when answering questions. Group A and group B can then be compared without bias. Fig. 5 displays the quantitative affective variations when answering questions grouped by trick. The horizontal dashed bars represent the values for group B, that is the group projected to subliminal stimuli during learning. The results are surprising and the difference showed here is more than statistically significant for alpha is equal to 0.01 (p<0.001, α=0.01). Not only does the subliminal module help increase valence and arousal variations to optimal levels as discussed previously, but the effect seems to last throughout the experiment. Indeed, we can see that the best learners from group B are twice less stressed when answering all the questions and much less aroused than the best learners from group A. Affective Variations Of Learners (Valence And Arousal) When Answering Questions
Normalized Mean-Shifted Values With Regards To Baseline
14.0
11.9
12.0
10.0
9.5 8.2
8.0 6.9 6.3
6.3
6.0
5.5
5.3
5.4
5.2
3.9
4.0
3.2
2.0 -0.1
-0.3
-0.5
-0.6 -0.2
-0.1
-0.5
0.0 Valence
Arousal
-0.9 All Learners
-2.0
Valence
Arousal
-1.1 Best Learners
1st Trick Questions
Valence
Arousal
-0.9 All Learners
Valence
Arousal
-1.0 Best Learners
Valence
-0.9 All Learners
2nd Trick Questions
Without Subliminal Module
Arousal
Valence
Arousal
Best Learners
3rd Trick Questions
With Subliminal Module
Fig. 5. Affective variations of learners while answering questions
The results of the two previous figures give us an insight into the complex innerworkings of unconscious perception. It seems that positive arousal variations (between 2.5 and 2.9) and positive valence variations (between 0.6 and 0.8) seem to yield
46
P. Chalfoun and C. Frasson
the best results when answering questions. Furthermore, subliminal projections seem to produce a cumulative positive effect because of the observed results for the third trick. Indeed, the last trick is the most difficult because it requires the use of the first two, henceforth the very high arousal variations for group A. We demonstrated in [10] that the subliminal module helped reduce dramatically the number of mistakes made. Fig. 5. has just explained why in terms of valence and arousal. Impact Of Affective Variations On Perfomance When Answering Questions 80.0 73.0
70.0 61.9
60.0 49.1
50.0 41.0
40.0 34.0
41.5
41.0
39.0
35.0
33.8
30.0
25.0
24.9
20.0
17.0 12.0
10.0
13.0 10.0
9.0
7.0
0.0 1st Trick Questions
2nd Trick Questions
3rd Trick Questions
All Learners
Total Reclics - Without Subliminal Module Total Errors - Without Subliminal Module Average Completion Time (s) - Without Subliminal Module
Total Reclics - With Subliminal Module Total Errors - With Subliminal Module Average Completion Time (s) - With Subliminal Module
Fig. 6. Impact of the affective variations on results, broken down by trick
The affective variations examined throughout this section lead us to carefully examine the effect on performance when answering questions of the third trick. Indeed, we can surmise that a learner can correctly construct a magic square if he successfully answers all questions but more specifically the third trick questions. Fig. 6. displays the results of our analysis by comparing group A’s results with group B’s with regards to total re-clicks (number of times learners asked to restart the question), total errors and more importantly completion times. The results present another argument for the combination effect of subliminal stimuli. Indeed, we can see that the total re-clicks for group B is almost the same as group A’s for the first trick but dramatically decreases as the subliminal projections increase throughout the experiment. The same phenomena can be observed with errors and completion times. The last aspect, completion times, is very important because it represents the foundation of what we are aiming at: faster and more efficient learning. We are very pleased with the results because they clearly show the significant contribution of the subliminal module (p<0.001, α=0.05) to the dimension of performance as well as emotions.
5 Conclusion We presented in this paper the optimal affective conditions for subconscious learning in a 3D virtual ITS. Subconscious learning is done with the use of carefully
Optimal Affective Conditions for Subconscious Learning in a 3D ITS
47
engineered subliminal projections aiming to accelerate and enhance learning. We also illustrated the importance that subliminal stimuli can have on the long term process of learning and pattern recognition. In contrast to the previous work regarding the use of subliminal priming and emotions, our work differs in three ways. First, the subliminal priming was used in an HCI to teach a more complex lesson than simply face recognition or item association. Second, in contrast with the vast majority of researchers, we discussed and placed forward the use of quantitative, and not just qualitative, affective variations in building an effective HCI to detect the best learners from the rest. Third, we placed forward the long-term impact that subliminal cues can have on the entire learning process and not just simply on simple tasks such as face memorizations or item pairing. As future work, we intend to use these results to construct an intelligent real-time HCI system that detects the emotional variations of the learner and adjusts the subliminal projections accordingly. Acknowledgments. We acknowledge the support for this work from the Fond Québecois pour la Recherche sur la Société et la Culture (FQRSC).
References 1. Villon, O., Lisetti, C.L.: A User-Modeling Approach to Build User’s Psycho-Physiological Maps of Emotions using Bio-Sensors. In: IEEE International Symposium on Robot and Human Interactive Communication, Session Emotional Cues in Human-Robot Interaction. Human-Robot Interaction, United Kingdom (2006) 2. Damasio, A.: Descarte’s Error - Emotion, Reason and the Human Brain. Putman Press, New York (1994) 3. Blanchard, E., Chalfoun, P., Frasson, C.: Towards advanced Learner Modeling: discussions on quasi real-time adaptation with physiological data. In: 7th IEEE conference on Advanced Learning Technologies: ICALT 2007, Niigata, Japan (2007) 4. McQuiggan, S.W., Lester, J.C.: Learning empathy: a data-driven framework for modeling empathetic companion agents. In: International Conference on Autonomous Agents, Hakodate, Japan (2006) 5. DeVaul, R.W., Pentland, A., Corey, V.R.: The Memory Glasses: Subliminal vs. Overt Memory Support with Imperfect Information. In: IEEE International Symposium on Wearable Computers. IEEE Computer Society, New York (2003) 6. Dijksterhuis, A., Nordgren, L.F.: A Theory of Unconscious Thought. Perspectives On Psychological Science 1 (2006) 7. Watanabe, T., Nanez, J.E., Yuka, S.: Perceptual learning without perception. Nature 413 (2001) 8. Nunez, J.P., Vincente, F.D.: Unconscious learning. Conditioning to subliminal visual stimuli. The Spanish Journal of Psychology 7 (2004) 9. Del Cul, A., Baillet, S., Dehaene, S.: Brain Dynamics Underlying the Nonlinear Threshold for Access to Consciousness. PLoS Biology 5 (2007) 10. Chalfoun, P., Frasson, C.: Subliminal Priming Enhances Learning and Performance in a Distant 3D Virtual Intelligent Tutoring System. In: AACE World Conference on Elearning in Corporate, Government, Healthcare, & Higher Education: E-LEARN 2008, Las Vegas, Nevada (2008) 11. Wallace, F.L., Flaherty, J.M., A. K.G.: The Effect of Subliminal HELP Presentations on Learning a Text Editor. Information Processing and Management 27 (1991)
48
P. Chalfoun and C. Frasson
12. Schutte, P.C.: Assessing the Effects of Momentary Priming on Memory Retention During an Interference Task. Computer Science, Vol. Master Of Science. Virginia Commonwealth University, Virginia (2005) 13. Dijksterhuis, A., Preston, J., Wegner, D.M., Aarts, H.: Effects of subliminal priming of self and God on self-attribution of authorship for events. Journal of Experimental Social Psychology 44 (2008) 14. Tranel, D., Damasio, A.: Knowledge without awareness: an autonomic index of facial recognition by prosopagnosics. Science 228 (1985) 15. Lang, P.J.: The emotion probe. American Psychologist 520 (1995) 16. Prendinger, H., Ishizuka, M.: The Empathic Companion: A Character-Based Interface That Addresses Users’ Affective States. Applied Artificial Intelligence 19 (2005) 17. Conati, C.: Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence 16 (2002) 18. DiMello, S.K., Taylor, R., Graesser, A.C.: Monitoring Affective Trajectories during Complex Learning. In: Trafton, D.S.M.J.G. (ed.): Proceedings of the 29th Annual Cognitive Science Society, Austin, TX (2007) 19. Picard, R., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions Pattern Analysis and Machine Intelligence 23 (2001) 20. Blanchard, E., Frasson., C.: Easy Creation of Game-like Learning Environments.: Workshop on teaching with robots and agents. In: conjunction with ITS 2006, Jhongli, Taiwan (2006) 21. Thought_Technology (2008), http://www.thoughttechnology.com
Computer-Based Learning to Improve Breast Cancer Detection Skills Yan Chen1, Alastair Gale1, Hazel Scott1, Andrew Evans2, and Jonathan James2 1
Applied Vision Research Centre, Department of Computer Science, Loughborough University, United Kingdom 2 Nottingham Breast Institure, Nottingham City Hospital, Nottingham, United Kingdom {Y.Chen3,A.G.Gale,H.Scott}@lboro.ac.uk
Abstract. In breast cancer screening it is important both to improve and maintain cancer detection skills at their highest levels. The introduction of digital imaging enables computer-based learning to be undertaken outside breast screening centres using a range of different devices. The potential for providing computer-based interpretation training using low-cost devices is detailed. The results demonstrated that naive observers can be trained to recognise certain key breast cancer appearances using a low cost display monitor along with a range of HCI techniques. Keywords: mammogram interpretation, training, eye movement, visualization, Human-Computer Interaction (HCI).
nationally. Additional training for individuals which targets such known discrepancies can then be proposed which should help in maintaining skill levels of cancer detection. Currently most training in the interpretation of screening mammograms needs to be undertaken where there is a mammo-alternator (a large viewing device, where several hundred cases can be loaded, which comprises a back-illuminated surface on which numerous mammographic cases can be presented simultaneously for examination) or other suitable light box (where only one single mammographic case can be presented at a time) on which to view the mammograms. This then limits the locations and availability where training can take place. However, digital breast screening, where images of the breast are digitally acquired and the resultant images examined using very high resolution and expensive dedicated workstations in breast screening centres is gradually being introduced nationally. Typically these workstations have a dual monitor set up where each has a resolution of 5 Mega pixels (2048 x 2560) and is capable of displaying 10-bit greyscale images with a high contrast ratio. Each monitor can display various combinations of mammographic views and is used primarily to display a single mammogram which can then be zoomed into and manipulated. The widespread adoption of such digital technology renders additional training opportunities using other less expensive and lower resolution displays and in various locations possible for certain purposes if suitable Human-Computer Interaction (HCI) techniques can be derived which are acceptable to radiologists. Primarily when examining digital mammographic images for diagnostic purposes then an appropriate workstation is fundamentally required. However it may be possible to target training of key mammographic appearances using other lower resolution displays which would make web delivered training using a range of viewing devices both potentially feasible as well as acceptable. This research investigates the utility of employing low-cost devices to provide individualised training with the support of HCI techniques and then determines what types of image interaction and manipulation techniques are required by potential end users. It was hypothesised that using a lower resolution display would hamper the identification of small signs of breast cancer (e.g. microcalcifications) but that larger signs (e.g. masses) would still be visible. Different HCI approaches were used as an initial approach to training based on facets of typical routine clinical training or reporting. In particular, the possibility of using aspects of visual search (as an HCI technique) to assist in mammographic interpretation training is explored.
2 Methods Firstly, a questionnaire was sent to all (601) UK screening personnel to determine their satisfaction with current training opportunities and their views on the potential for individualised training as well as on the use of other image display devices for examining the high resolution mammographic images. This included details of current screening (e.g. profession; experience of digital mammography) and the individual’s current usage of mammographic interpretation training (e.g. forms of training available; the amount of training opportunities; any difficulties of current training; advantages and disadvantages of current training).
Computer-Based Learning to Improve Breast Cancer Detection Skills
51
Secondly, as a ‘worst case’ scenario investigation twenty experienced breast screening radiologists were shown a series of mammograms on a small PDA (screen size 3.5”) and questioned as to whether they thought such displays could ever be used for training purposes in breast screening and if possible then what types of HCI image manipulation techniques would they envisage requiring (Figure 1).
Fig. 1. A radiologist views mammograms (MLO views) on a 3.5” screen PDA
An experimental investigation was then undertaken which examined different aspects of computer based training for breast screening interpretation. Initially, both the eye movements and the audio description of an expert breast screening radiologist were recorded whilst a series of recent screening cases were examined. From these eye movement data key fixations were identified (based on visual dwell measures) which fell on both abnormal areas as well as other normal image areas which attracted visual attention. Twenty naïve participants were then first given a short standardised introductory computer-based presentation about breast cancer; mammograms and the appearances of two key mammographic features; masses and calcifications. These two features were deliberately chosen as whilst both can be difficult to detect on a mammogram, masses are generally relatively large and calcifications are fairly small irregular abnormalities which can appear singly or in clumps and of various size. This was then followed by an image examination exercise. During this, each participant was required to examine 21 recent mammographic screening cases (7 normal cases, 7 cases containing a mass and 7 cases containing calcifications) on two occasions. Their performance in identifying abnormal appearance was measured as they first examined these cases, then they either undertook a short training exercise or were assigned to a control condition and then re-examined the cases. On both occasions the participants were asked to identify whether each case contained
52
Y. Chen et al.
either of the two features. If they thought an abnormality was present then they also had to indicate its location. Participants’ eye-movements were recorded throughout. The different forms of training used were: whole mammogram presented but with regions of interest highlighted; only regions of interest (where only a magnified portion of the mammographic image around a potential abnormality site was presented); playback of the expert’s visual search behaviour, and playback of the expert’s audio instruction on each case. Participants Twenty naive observers participated (seventeen research students in various subjects, and three university employees) were involved in the study. None of them had any experience in mammography reading. Participants were split into four experimental groups and a control group (without undertaking any training) with the experimental groups undertaking different forms of computer based training. Materials Visual stimuli: 21 sets of recent digital mammographic images were used. Each image set comprised both the Medio Lateral Oblique (MLO) views and Cranio Caudal (CC) views of both breasts. Fourteen of the pairs featured a specific abnormality which had been grouped into two types (namely: Mass and Calcification) with the abnormality visible on either one or both views. Seven of the sets featured no abnormality.
Fig. 2. Schematic of the different training and control approaches
Computer-Based Learning to Improve Breast Cancer Detection Skills
53
Hardware: The experiment was run on a PC with a LCD monitor (display size: 21.5”; resolution: 1680 x 1050 ) for displaying images; eye movements were recorded using a remote oculometer – a Tobii X50 eye-tracker. This records eye movements with no attachment to the participant and allows them some degree of free head movement (within 30 x 16 x 20 cm ; W x H x D) at 60 cm from the device. It has a reported accuracy of 0.5-0.7 degrees of visual angle. Design Training sets: each training set included the 21 cases comprising both MLO and CC views of each case but were presented in four different formats (Figure 2). These formats were: 1. T1 – whole image: the MLO views of both breasts were presented then, where appropriate, the feature area was highlighted by a circle with descriptive text, followed similarly by the CC views. 2. T2 – image with portion: the MLO views were presented (as in format T1) but then the area of interest around the abnormality was highlighted followed by this area being magnified and shown alone (size: 8”x 8”). This was followed similarly by the CC views. 3. T3 - eye movements: the MLO views were shown and then overlaid with annotated fixation locations of where the initial expert radiologist had looked and in the order in which visual search had been performed. Also, the area of abnormality was highlighted. This was followed similarly by the CC views. 4. T4 - comments: This was similar to format T1 above but with the addition of the expert’s audio descriptions concerning the case. 5. control: a control activity was undertaken which took the same length of time as the other training sessions. Procedure For each participant the eye tracker was first calibrated. Each was then given a short introduction concerning breast cancer, mammogram images, and were familiarised
A
B
Fig. 3. A shows a participant taking the test viewing the MLO mammograms of a case.; B shows a participant undertaking the audio training whilst viewing the CC images of a case.
54
Y. Chen et al.
with the appearance of the two different key breast cancer features in such images (masses and calcification). They then visually examined two practice cases. Subsequently they completed a computer-based image examination task whilst their eye movements were discretely recorded (Figure 3). During this task the participants first fixated on a small centrally presented fixation cross and then this was replaced with the MLO views of the case which in turn was followed by the CC views. Participants were asked visually to examine each case view, identify whether the view was normal or contained an abnormality. If the latter, they also had to specify the feature type (i.e. mass or calcification) and its location.
3 Results For the questionnaire study some 273 questionnaires were returned (a 45% response rate) which covered the main professions in breast-screening film reading, e.g. 152 consultant radiologists, 78 advanced practitioners (in the UK these are specially trained technologists), and others. These all indicated that screeners considered that whilst existing training was adequate they would like to have individualised and targeted training which is only really possible using digital images. The responses concerning the most useful mammographic interpretation training type is shown in Figure 4.
No.PPT consier it 'Most Useful'
100 90
Review of individual film reading data Arbitration/consensus PERFORMS MDT's Interval cancer review Discussion/interaction Radiology meetings/sessions CPD course Conference
1 2 3 4 5 6 7 8 9
80 70 60 50 40 30 20 10 0 1
2
3
4
5
6
7
8
9
Training Type Fig. 4. Participant’s response on ‘The most useful training type’
‘Interval cancer review’, ‘Arbitration/consensus’ and ‘PERFORMS’ were considered to be the fist three most useful. According to the characteristic of these three training types, it shows that being able to read a large amount of images followed by other forms of information, such as, expert’s opinions on the case, other people’s feedback on the case, etc. were considered to be the key useful training approaches.
Computer-Based Learning to Improve Breast Cancer Detection Skills
55
25% % of All correct answer
T est Session 1 T est Session 2
20% 15% 10% 5% 0% Visual
Audio
Control
Training type Fig. 5. Percentage of both correct answers (feature & location) of the training types (both test sessions)
Test session one Test session two
Fig. 6. Mean performance values of each participant group (both test sessions)
56
Y. Chen et al.
With regard to viewing images on the PDA, initially all participants were highly sceptical as the display had approximately 1/10th of the resolution of a digital mammographic workstation but it was determined that such a display could be useful for training (but not for diagnostic) purposes if suitable simple HCI techniques were available which would enable rapid image pan and zoom with contrast level manipulation. In the experimental study of naïve observers some 880 responses were collected and of these only 124 correctly identified both features and locations on both views of a case. Initially the training types were considered as either visual (whole image, portion, eye movements), audio (comments) or control - details are shown in Figure 5. In audio, performance decreased after training whereas for both the visual and control conditions there was a slight, non significant, increase. The performance of these participants in the first and second test was then attributed points concerning whether they correctly identified location and features on both views (Figure 6) where a score of 4 indicates identifying both the correct location and feature on both views. Most participants scored 1-1.5 indicating that they only correctly scored either location or feature in either view. In the three visual conditions there was a slight increase in performance in test 2, in the audio condition performance decreased and in the control condition there was little variation. In all conditions participants were significantly better in identifying features correctly than in identifying correct location (p<.05). The performance in identifying features between test 1 and test 2 significantly improved (p<.05). In terms of mammographic features then masses were better identified after any visual training and only slightly increased after audio training. Calcifications were detected worse after all three types of visual training as well as the audio training. In the control condition calcification identification improved. Normal cases were reported worse on every second trial. In the control group the correct identification of normal cases dropped on the second test but mass and calcification detection increased. In the audio condition calcification and normal identification fell on the second test but mass identification improved slightly. In the three visual conditions only mass identification improved.
4 Discussion From the returned questionnaires, these all indicated that screeners considered that whilst the existing training was adequate they would like to have individualised targeted training [5]. With regard to viewing images on the PDA then for this to be at all useful, easy to use HCI techniques which would allow rapid image manipulation were clearly required [6]. Using a PDA for some training purposes would be possible if this was done in a appropriately lit environment with a low ambient light level. The experimental study reported here was an initial investigation utilising a single low cost monitor to deliver mammographic interpretation training as compared to using one or two high resolution workstation monitors. Different training regimes were developed which presented observers with images enhanced using different HCI approaches which were considered suitable in this domain. The approaches used were firstly simply showing the full views of both breast simultaneously which mimicked the display on a workstation but at a much lower resolution. Secondly the full view
Computer-Based Learning to Improve Breast Cancer Detection Skills
57
was show followed by a magnified view of the area of interest to mimic basic image manipulation on a workstation. The third approach taken was to utilise areas of interest which attracted the visual attention of an expert together with the scanning path which were overlaid on the image. The fourth approach tried to replicate an expert describing how they examine a case for abnormalities which is a commonly adopted approach in real screening as shown in the questionnaire survey. The study set out to use short training approaches to examine their effects on naïve observers. This demonstrated that such observers can be trained to recognise certain key breast cancer appearances using a low cost display monitor along with a range of HCI techniques. Two mammographic appearances were studied; small calcifications because these can be difficult to detect in routine breast screening and larger masses. Calcifications were not detected well presumably due to the shortage of image manipulation techniques used here. Naïve observers were used in this study to see how they responded to the different training types.
5 Conclusion Overall, it is argued that these findings taken together indicate that low cost devices can be used for training purposes in digital breast screening with appropriate HCI techniques. These then extend the opportunity for training beyond the clinical workplace. Ongoing research is further investigating the delivery of computer-based training using a variety of HCI methods to breast screeners.
References 1. World Health Organization “Cancer”, Fact sheet 297 (2006) (Accessed: 01 March, 2009) 2. National Statistics Online, Breast Cancer: Incidence rises while deaths continue to fall, Office for National Statistics (2007) (Accessed: 01 March, 2009) 3. Gale, A.G.: PERFORMS-a self assessment scheme for radiologists in breast screening. Seminars in Breast Disease 6(3), 148–152 (2003) 4. Gale, A.G., Scott, H.J.: Measuring Radiology Performance. In: Michell, M. (ed.) Breast Screening. Contemporary Issues in Cancer Imaging – Breast Cancer. Cambridge University Press, Cambridge (in press) 5. Chen, Y., Gale, A.G., Scott, H.J.: Mammographic interpretation training profile in the UK: current difficulties and future outlook. In: Manning, D., Sahiner, B. (eds.) Image Perception, Observer Performance, and Technology Assessment. Proceedings of SPIE (in press) 6. Chen, Y., Gale, A.G., Scott, H.J.: Mammographic interpretation training: how useful is handheld technology? In: Manning, D., Sahiner, B. (eds.) Image Perception, Observer Performance, and Technology Assessment. Proceedings of SPIE, vol. 6917, pp. 691712– 691712-10 (2008)
Virtual Classroom and Communicability: Empathy and Interaction for All Francisco V. Cipolla Ficarra1,2 HCI Lab. – F&F Multimedia Communic@tions Corp. ALAIPO: Asociación Latina de Interacción Persona-Ordenador 2 AINCI: Asociación Internacional de la Comunicación Interactiva Via Pascoli, S. 15 – CP 7, 24121 Bg, Italy [email protected] 1
Abstract. We present the main empathy components in the design of interactive systems aimed at classroom and E-learning education. These components have a bidirectional relationship with communicability and usability. Each of them depicts an intersection of communication, semiotics, interface design, software engineering, usability engineering and human-computer interaction. Additionally, we present a table which can be used as a communicative quality guide. Its content is the result of 20 years of design and heuristic assessment of on-line and off-line interactive systems, mainly for Americans and European users. Keywords: Education, Virtual Classroom, Hypermedia, Communicability, Design, Information, Empathy, Interaction, Accesibility.
Virtual Classroom and Communicability: Empathy and Interaction for All
59
phones in Italy, for instance, and yet they keep on demanding more functions), multimedia systems are based on partial design models, with primitives not unanimously accepted in the computer context, of human-computer interaction, usability, etc. [5]. Consequently, the lack of a unique model causes in many cases time loss of human resources. The only way to solve these problems is by resorting to basic notions of the historic concepts within international circulation. In the current work there is a brief description of the state of the art at the beginning, then follows the eradication of ambiguities in the essential notions of communicability, information and communication. Later on the empathy and the interaction is analyzed from a diachronic perspective with examples and finally an heuristic table is presented for the first assessment of the empathy applied to the educative and interactive systems, on-line and off-line.
2 Eliminating Ambiguity in Hypermedia Systems It is important to carry out in some cases the anchoring operations of the relationship of signification and significance of the terms, as is the way in semiotics framework and linguistics [6]. For instance, in the later stage of the evolution of interactive multimedia systems, we have the notion of hypermedia, which is an hybrid word between hypertext and multimedia. Here are assembled the advantages of both technologies inside the multimedia communication process. As has been observed in the origins of hypertext, where the textual aspects of the first systems prevails (including static graphics, in a broad sense), where it is established the associative character in the structure of information (these aspects generate a less frequent denomination in hypermedia systems, such as is the case of the electronic book). Whereas in multimedia and through the intersection of media there is a dynamic content of information: video, computer and audio animations. This dynamic component entails the time or synchronization factor among diverse means, which in our case we call panchronism. (From Greek ‘pan’ meaning all and ‘chronos’ meaning time [5]. A panchronic approach to, say, language is an approach including every aspect or all dimensions of time. Measurable quality criteria is very important in the communicability of on-line and off-line contents of hypermedia systems [7]. Consequently, hypermedia in a classical sense of the notion allows: selected access to those parts determined beforehand by the user, a greater degree of detail in the structure of information, that is to say, it is possible to resort to the richness of the content, in regard to several means used in the transmission of messages and reinforcing the communication process [8]. For instance, it is present in the use of CD/DVD-Roms and the DVDs for education, commerce, tourism, the ecological and cultural patrimony of many cities, regions or countries of our planet, etc. As a side note, in our projects in this area, until now we have undertaken our research only with users without physical disabilities. We hope and wish to include a wider range of users to study hypermedia systems in our future research. Along the evolutionary path of hypermedia, it can be said that it has become an interactive extension of multimedia. Consequently, this is the reason why the notion of multimedia is used indistinctly with that of hypermedia in many research works,
60
F.V. Cipolla Ficarra
although the relationship between the signification and the significant may be not strictly symmetrical according to Ferdinand de Saussure [6]. In hypermedia the synchronism of the active and static and dynamic means is essential. Besides, the amount of interaction required by the system and control of the fruition of the user over the system are two quality criteria for making assessments in current hypermedia systems [7]. 2.1 Communication, Information and Communicability In the context of the factual and formal sciences there are plenty of theoretic and practical works which talk about communication [9]. In this research it is easy to observe several definitions, where there are common elements among them. However, the momentum of verbose communication in the daily and colloquial environment has generated a sort of deficiency in its scientific sense, that is to say, a terminological status is missing. This deficiency becomes even more obvious when we talk about communicability in multimedia systems. Therefore, we will try to rank different points of view on both notions. In the case of communications, inside the context of the social sciences and more specifically in the area of social communication, some experts traditionally made a study of the differences between communication and information (Abraham Moles, David Berlo, Frank Dance, Luka Brajnovic, Rau Birdwhistell, Wilbur Schramm, etc. [10]). Others, in contrast, exclude the notion of information. Currently in the design of interactive systems the word communication is linked to contents, that is to say, the communication of contents. A short analysis of the word communication from its genesis leads us to the noun “communication” and the verb “communico”. Both have their origin in the word “communis” formed by cum (with) munis (duties-lins) [11], entailing implicitly the meaning to unite, to bind, to link to, that’s to say, there is a relationship with another person or system, in our case. There is an implicit dynamic process. For instance, through the relationship with the stored information in multimedia on-line and off-line systems, human beings can expand their knowledge. Regarding this, Schramm [6] considers communication as a real established relationship which consists in the discovery of the “I”, the “other” and “others”, and a donation of content which entails a duality of terms between emitter and receptor, who coexist in a contained environment and which is the foundation to its corresponding process. Within the notion of an emitter, we understand it to be the designer of multimedia systems, whereas the receptors are the potential users of that interactive system. Now, Schramm shows the need of cultural understanding between both to obtain optimal results in the communication process. On his side Luka Brajnovic tells us about communication and information differentiating each one in the following way: “communication consists in getting in touch with two or more people, things or bodies, in their different combination possibilities and its meaning can be manifold, and have diverse procedures and effects” [10]. It is a sort of direct channel or creative encounter that ties presences and distances, sometimes, without the informational goal, although it can be a vehicle of information. Therefore, information can be included in communication. When the information is incorrect there is no feedback, in the interactive communication process, for instance, because human communication is a psychosocial process.
Virtual Classroom and Communicability: Empathy and Interaction for All
61
Moreover, if we consider that communication is a psychosocial phenomenon, we must admit that this phenomenon is a given in the human being with all his rational and creative possibilities to organize the message and interpret its reception. That is to say, that from the point of view of social communication, it is not a simple direct channel of relationships, but rather a socialization process. Therefore, communication is a social fact, updated by the human being, through a process among which the integrants who make it up, where the interchange of experiences entails the enrichment of the participants, through the internalization of the messages that have been expressed in a given space and time. Among human beings there is a will to encounter in communications and a desire to have or create something in common. Spontaneous binding achieves the integration of man with nature, cultural patrimony, the community he inhabits, etc. We not only communicate data and facts but additionally we can communicate our ideas, experiences, feelings, and real or imaginary events, the objective and the subjective, the presence of two or more people in their different combinatory possibilities. Once we have quality in communication we are faced with the notion of communicability. That is to say, that communicability automatically includes quality. In our case, we sometimes use both notions simultaneously and redundantly to strengthen the idea of quality in the process of interactive communication. However, communication and communicability are two synonymous terms in the current multimedia theoretical framework. In the design of interactive multimedia systems it is easy to detect the presence of communicability, although as happens with the notion of beauty, it requires time to describe each one of the elements that make it up. In short, it is starting from this interrelation “cum” technological and “cum” the individual when the relationship in other research environments take place, such as the cognitive models used in the development of interfaces. At the moment of the design of an interactive system, the designer must consider the cultural factors, the types of users, the geographical location of the system, etc. [12], [13], [14], [15]. These variables mean that since the software sector at the start of the nineties the need of incorporating sociologists, anthropologists and sociologists has been affirmed to improve the quality of interactive systems. However, in the new millennium it is necessary to talk of experts in communicability [7].
3 Empathy and Interaction: Interactive Design We can define the empathy in the interactive design as the interactive systems designer's mental ability to put himself in the shoes of the potential user. It is the result of the triad confirmed by the cultural knowledge, mental ability to occupy the place of the other in the communicative process and the competence in advancing the user's behaviour in front of certain situations [16], [17]. In multimedia design traditionally we talk about cognitive models, that is to say, the solution would be to frame it in the psychological context. Obviously, it is a valid alternative for the first hypertext and multimedia systems in the late eighties and the decade of the nineties. With the advent of the use of information networks, whether it is Internet or extranet from international entities, since the late nineties it has been a matter of communicability. A communicability which stems from the design process in the
62
F.V. Cipolla Ficarra
interactive systems and is translated to its usability. If we analyze some multimedia products aimed at the education of the nineties, we can find how in the design of their structure one resorts continuously to two quality attributes such as are prediction and self-evidence [16]. A priori, prediction and self-evidence can seem similar, but it is not so. In selfevidence, the navigation of pages with dynamic elements (i.e. audio, video, animation, etc.) and the structure of the system can be anticipated by the user from the first moment, even if the user has scarce experience in the use of hypermedia. On the contrary, in prediction the user must have previous ability in order to navigate efficiently and overpass complex situations, after having previously navigated the hypermedia system. In prediction there is more previous experience in the navigation or knowledge of structural parts. Nevertheless, in some hypermedia, where the designers resort to reusability of information, prediction and/or self-evidence can be prejudiced if the behaviour of the dynamic media is different according to the place that it has in the structure. For example, in some sites the user can interrupt the animation and go to another page of the structure, while in others he cannot do so until all the animation is finished [18]. These two attributes are related to the concept of isotopies inside the context of communication. That is to say, those elements that must be maintained continuously in each one of the design categories to favour the interaction of the users with the content of the multimedia system [18] [19]. For instance, the location itself of the navigation keys in the different screens. The same modes of activating and disactivating the dynamic means, the synchronization between the audio and the images in movement, regardless of whether they refer to a video or an animation, etc. The presence of the isotopies in the interactive design indicates a high degree of empathy towards the potential users. In the first multimedia off-line systems it is observed a certain local or national empathy at the expense of international empathy. The reason of this failure in the internationalization of the multimedia contents lies in the fact that these systems counted with a design team for human factors, where would be located the empathy with psychologists and anthropologists. The origin of this deformation in communicative empathy and in the future of communicative evolution lies in software engineering. Some authors in the nineties maintained that in order to increase the quality of the software it was necessary to include some representatives of the social sciences, referring only to psychologists and anthropologists [20]. Obviously, some commercial products of multimedia software have followed these rules but they have forgotten the social communicators, whose training is in many cases an intersection of the fact and social sciences. Before making these statements to increase the quality of the software it is necessary to know beforehand the university study plans which are usually different in each of the continents in our planet. It is through the social communication professionals that a bigger international empathy is reached, which makes easier the circulation of on-line multimedia content. Throughout the history of interactive systems interfaces there have been some works that have made apparent the need to overcome cultural barriers among the different international users and the designer.
Virtual Classroom and Communicability: Empathy and Interaction for All
63
These works refer to the design categories presentation and content of the hypermedia systems. The acceptance of these premises of the interfaces design are related to the operative systems of the computers, that is to say, the interface and the PC Macintosh icons are not the same as compared with a PC that works with the last Windows version. In this regard, the contribution made regarding compatibility matters, such as the meaning of icons, that has been undertaken by manufacturers of operating systems in order to increase the empathy and interactivity with the multimedia systems, should be praised. Although some icons and functionalities of these have changed in the last years, this does not mean that they are objects of criticism because from the point of view of their usability and communicability it is necessary to consider them from a diachronic view. To make an heuristic and synchronic study of these; without considering the temporal variable only mistaken results are obtained. Making a diachronic study of the different interfaces used for the teaching of languages and those aimed at the virtual campuses, we can see how the interactivity increases during the teenage years in off-line systems, for instance. The interactivity with the system is related to the designer's empathy. It is for this reason that in the categories of presentation and content of the components of the interfaces for children we include the use of primary colors, sounds and melodies that draw the attention, characters from tv series, etc. [21] [2]. The more that the user's age increases, the designer will increasingly try to immerse him/her in real life situations to go deeper into the knowledge acquired in the learning process of a language. The relationship between user and computer shows some variables during interaction [22]. These variables are related among the individual user, the interactive system, and the context. In the first case, we have to consider elements like age, physical condition for autonomy of interaction, education or previous experience with multimedia systems and the aims of fruition, above all. In the second case, we have to consider access to information and the kind of support for the information, basically. The last variable is the time to fruition through the use of an interactive system.
4 Lessons Learning In the design process of a multimedia system the aim is to communicate content. Now, communication implies behaviour. Consequently, it is an activity that the designer develops with the purpose of obtaining a behaviour from other people through the information he has about the potential users. However, communication has a goal. This goal is flexible, since the very communicative process is modified, or rather, that goal is being determined progressively during communication. Therefore, communication does not only modify behaviour, but also the goal of the action to be communicated. That is why some of the quality attributes of a multimedia system refer to self-evidence, prediction, etc. The notion of wealth as a quality attribute is a solution that the designer has when he lacks enough information in front of the potential users of the system [23]. Nonetheless, that wealth entails a metamorphosis in the categories of navigation and structure of the system, that is to say to adapt the content and the ways of presenting the information in the interface of the computer
64
F.V. Cipolla Ficarra
screen, iPod, E-Book, etc. in front of the eventual users, globalization and the future of standardization [24], [25], [26]. It is here where the ability of the designer is seen and his empathy, that is to say, the bigger is the experience the more you can see how the goals are being determined with regard to the options that are presented in a progressive way. In other words, the goal of the interactive communication must be adapted in a constant, progressive and detailed way, as the users' behaviour indicates a bigger interaction with the system’s contents. Therefore, currently it is mistaken to set all the goals to be implemented at the beginning of a communicative process in a rigid way, for instance, eliminating the attribute of wealth or richness, whether it is via bidirectional navigation among the nodes of a referential link, activation or disactivation of dynamic means, etc. A parallel of this situation exists in the communication process in a classroom, when the teacher lays down the goals of a lesson at the beginning of the class. Although the goals are shown in a synthetic way, there must also exist the malleability of this order and/or contents with regard to the feedback inside the interaction process with the students [27], [28]. Therefore, communication always entails an action of a previous flow chart, where it is implicitly laid down that is intended to be obtained in the human computer interaction, for instance, repetition of the most frequent sentences in a language, overcoming the self-assessment tests, etc. that is to say, the fruition of the greatest possible amount of information stored in the database, explicitly, such as using the multimedia information to increase the cultural level of a student. In the next graphic some of the main goals are summarised, accompanied by interfaces where there is an excellent empathy towards the potential users of encyclopedia and English course. In figures 1 and 3 we find the richness atribute and a menu of the options. The users can active or desactive the sounds, transition of the frames and dimension of the video, for example.
Fig. 1. Main empathy components in the design of interactive systems (content and presentation categories)
Virtual Classroom and Communicability: Empathy and Interaction for All
Fig. 2. Enciclopedia de la ciencia [29]
Fig. 4. Richness option
65
Fig. 3. Kiyeko –5 languages for children [30]
Fig. 5. Learning English Kids –www.britishcouncil.org/kidsenglish
5 Conclusions The designer's empathy in face of potential users is the key issue for interactive systems of the following types –multimedia on-line, off-line, immersion multimedia, Web 3.0, virtual reality, etc. The current training of the designer and heuristic assessor
66
F.V. Cipolla Ficarra
of interactive systems demands a real intersection of factual and social sciences. Besides, being a communicability expert is also necessary. That is to say, to reach an utmost quality in the design of interactive systems, in the least possible time and with reduced costs. That is why the communicability guide presented is a constructive addtion to partially reaching that goal. We say partially because the current guide is aimed at users without physical disabilities, however it will extend and adapt itself to users with disabilities in the future. It is in the educational process that empathy gains a paramount role, since it is an essential element in boosting the motivation of potential users. Aditionally, a good design of multimedia teaching materials with an excellent communicability level allows E-learning education to be offered not only inside the borders of a country, but also other adjacent countries or those who have a language in common. Consequently, empathy is an economic factor, whether it is because of the gains or losses which can pertain to the educative or industrial entity due to the generation of multimedia content. A way of detecting its presence or absence is through the users' satisfaction at the moment of interaction with the interactive systems. The intersection can be measured through the time spent on the websites or with the classical CD-ROMs or DVDs in the case of languages, for instance. The more interaction, the more communicability, and additionally it means there is a greater designer's empathy at the moment of designing the multimedia system. Acknowledgments. A special thanks to Emma Nicol (University of Strathclyde) and Maria Ficarra (Alaipo & Ainci – Italy and Spain) for their helps. Also, I would like to thank all producers and distriibutors of multimedia/hypermedia off-line: Electronic Arts (Madrid –Spain), Microsoft (Madrid –Spain), ZetaMultimedia (Barcelona – Spain), Ubisoft (Paris –France) and DeAgostini (Novara –Italy).
References 1. Bruner, J.: The Culture of Education. Harvard University Press, Massachusetts (1996) 2. Reeves, B., Nass, C.: The Media Equation –How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Cambrigde (1998) 3. Botto, F.: Multimedia, CD-ROM & Compact Disc. Sigma Press, Wilmslow (1992) 4. Nickerson, R., Landauer, T.: Human-Computer Interaction: Background and Issues. In: Handbook of Human-Compputer Interaction. Elsevier, Amsterdam (1997) 5. Cipolla-Ficarra, F., Cipolla-Ficarra, M.: Interactive Systems, Design and Heuristic Evaluation: The Importance of the Diachronic Vision. In: New Directions in Intelligent Interactive Multimedia, vol. 142, pp. 625–634. Springer, Heidelberg (2008) 6. Nöth, W.: Handbook of Semiotics. Indiana University Press, Indianapolis (1995) 7. Cipolla-Ficarra, F.: Communicability design and evaluation in cultural and ecological multimedia systems. In: Proc. 1st ACM Workshop CommunicabilityMS 2008, pp. 1–8. ACM Press, New York (2008) 8. Lang, M., Fitzgerald, B.: Hypermedia Systems Development Practices –A Survey. IEEE Software 22, 68–75 (2005) 9. Bunge, M.: The Science –Your method and your philosophy. Siglo XXI, Buenos Aires (1981) 10. Dance, F.: Teoría de la comunicación humana. Troquel, Buenos Aires (1973)
Virtual Classroom and Communicability: Empathy and Interaction for All
67
11. Birdwhistell, R.: Diccionario de las ciencias sociales. Aguilar, Madrid (1974) 12. Shih, E., et al.: IT Diffusion in Developing Countries. Communications of the ACM 51(2), 43–48 (2008) 13. Fernández, T.: Global Interface Design –A Guide to Designing International User Interfaces. Academia Press, Boston (1995) 14. Kellog, W., Thomas, J.: Cross-Cultural Perspective on Human-Computer Interaction. SIGCHI 25, 40–45 (1993) 15. Dubberly, H., Pangaro, P., Haque, U.: What is Interaction? Are There Different Types? Interactions, pp. 69–75 (2009) 16. Cipolla-Ficarra, F.: HEDCDEH: A Heuristic Evaluation Disk for Communication and Design in Hypermedia. In: CD-ROM Proc. HCI International 2005, Las Vegas (2005) 17. Preece, J.: Empathic Communities. Interactions 5, 32–43 (1998) 18. Cipolla-Ficarra, F.: An Evaluation of Meaning and Content Quality in Hypermedia. In: CD-ROM Proc. HCI International, Las Vegas (2005) 19. Cipolla-Ficarra, F.: Dyadic for Quality in Hypermedia Systems. In: CD-ROM Proc. Applied Human Factors and Ergonomics, Las Vegas (2008) 20. Basili, V., Musa, J.: The Future Engineering of Software –A Management Perspectiqe. IEEE Computer 24, 90–96 (1991) 21. Cipolla-Ficarra, F.: Multimedia and Languages for Children: Semiosis for Universal Access. In: CD-ROM Proc. HCI International, Las Vegas (2005) 22. Marcus, A.: When is a user not a user? Who are we? What do we do? Interactions 5, 28–34 (2003) 23. Cipolla-Ficarra, F.: Evaluation Heuristic of the Richness. In: Proc. International Conference on Information Systems Analysis and Synthesis, ISAS, Orlando, pp. 23–30 (1999) 24. Sears, A., Lund, A.: Creating Effective User Interfaces. IEEE Software 14, 21–24 (1997) 25. Marcus, A.: Icons, Symbols, and Signs: Visible Languages to Facilitate Communication. Interactions 10, 37–43 (2003) 26. Robson, R.: Globalization and Future of Standardization. IEEE Computer 39, 82–84 (2006) 27. Mason, R., Rennie, F.: Elearning –The Key Concepts. Routledge, New York (2006) 28. Edmundson, A.: Globalized E-learning –Cross Cultural Dimension. VDM Verlag, Saarbrücken (2008) 29. Enciclopedia de la ciencia CD-ROM. Dorling Kindersley-ZetaMultimedia, Barcelona (1998) 30. Kiyeko CD-ROM. Ubisoft, Paris (1995)
Communicability for Virtual Learning: Evaluation Francisco V. Cipolla-Ficarra1,2, Miguel Cipolla-Ficarra2, and Pablo M. Vera2 HCI Lab. – F&F Multimedia Communic@tions Corp. ALAIPO: Asociación Latina de Interacción Persona-Ordenador 2 AINCI: Asociación Internacional de la Comunicación Interactiva Via Pascoli, S. 15 – CP 7, 24121 Bg, Italy [email protected], [email protected] 1
Abstract. An assessment is made of the work of design from the perspective of communicability and usability in multimedia aimed at E-learning, mainly through the off-line interactive systems (commercials) and on-line (open software). The method used is accompanied by a series of heuristic results along time to stress the validity or not of some of the design components. Besides, a novel strategy of organizing the textual content is presented for teenagers and the young: the truncated inverted pyramid. Finally, those quality attributes are mentioned that are related to the dynamic and static means, at the moment of heuristically assessing the communicability of a hypermedia system which has as its main goal college education. Keywords: Virtual Learning, Evaluation, Content, User-Centered Design, Open Software.
professors and students. When the instruments are presented in a rigid way towards the teachers, it will be difficult to implement e-learning in the time planned. As is the case with other platforms aimed at e-learning, in almost all the centres where Moodle has been used,there has been a need to create seminars, demonstrations, interchange of experiences, etc. for the use of the platform, training courses for the staff who will use the Moodle, assistance in the design of the courses (even in the transformation of the traditional material that normally use the professors for their schoolroom teaching), and activation of a help desk. It is very important that the help desk be located inside the university and that it is not an outsourced activity [4]. There is an endless series of negative experiences in this sense inside the software context and computers because the user needs access to a person who can explain or solve the problem. In Northern Italy, for instance, the assistance is only available by phone in the commercial world, and involves having to wait until an external technician arrives at the premises where the problem has originated. The costs are high because they include the commuting to the place where the problem is to be found. Outsourcing is avoided as an ideal solution in the software [4]. The current research work is structured in the following way: a brief state of the art of the software used in Elearning college education currently. Later on, the advantages of organizing textual information under the form of truncated inverted pyramid are considered. Then the diachronic results obtained from the heuristic assessment of the off-line multimedia systems off-line are compared, from the point of view of usability and those components of communicability are considered which have a great validity in the current on-line systems whose main purpose is teaching.
2 E-Learning and Open Software University centres keep spending an important sum of their financial resources for the software licenses and the maintenance of the hardware of large servers. In contrast Moodle is a free web application that educators can use to create effective on-line learning sites. Today, the great universities are adopting Moodle for the realization of distance courses or support to presential lessons. Perhaps the main advantage lies in the fact that many applications with a cost equal to zero already exist. These have been evaluated and corrected in other university teaching centres. However, a series of problems to be solved persists before implementing those systems can happen. These problems are related to issues known as human factors of software engineering; guaranteeing an efficient use of the technological instruments from the point of view of usability; helping teachers in the design process, elaboration, assessment, implementation and management of a distance course; to use a design of interactive system that does not change according to the contents and types of studies made inside the academic curriculum; that is to say, that the interface used in a technical engineering in computers has to be similar for superior engineering and a master in design and credibility of the information in the web (for instance), inside the university teaching centre itself. Now we understand by design not only the distribution of the contents of dynamic and static means on the computer screen, that is to say, the presentation, not only navigation, the structure, panchronism, etc. In
70
F.V. Cipolla-Ficarra, M. Cipolla-Ficarra, and P.M. Vera
order to satisfactorily overcome each one of these tasks it is necessary to count on an expert in communicability of interactive systems [5]. In e-learning, it is always necessary, from a financial point of view to use multimedia technologies within the reach of the greatest number of local users and in particular to the potential international users in the emerging areas. Therefore, it is incomprehensible and mistaken that a university under the teaching model "dual mode" plumps for iPod in E-learning, when the dean and over 75% of the professors do not have a digital culture. This entails resorting to outsourcing for the generation of teaching material and even the use of servers for academic management or ERP such as AS/400, creating teaching platforms in Lotus Domino, with very high costs in maintenance and management of software, hardware, professor training, content creation, etc. [6]. There are currently excellent gratis models [7] based integrally on Linux [8] in some Spanish regions as is the case in Extremadura. In this region, Elearning usually has a modest impact from the financial point of view (yearly computer budget) compared to some realities of "dual mode" universities in Lombardy (Bergamo city, for example). These costs may also have an impact on the ondine laboratories dedicated to teaching which exist in the main universities of a country or a region. These laboratories have as their main goal the support of lessons outside of teaching hours to go deeper into subjects, small lessons tending to the stretching of the students, etc. In some cases these activities are under the CAI initials, –Computer Aided Instruction–. CAI may include activities referring not only to the students, but also to the professors. In the case of the professors it has as its essential goal the updating of the latest technological news and knowledge of the potential of the technologies for a qualitative, straight and fluent communication with the students [2], [9], [10].
3 Truncated Inverted Pyramid and Textual Contents On-Line Historically, the textual contents and images are very important in educational multimedia systems [11], [12], [13]. The communicability proposed an efficient way of organizing the textual content in the hypermedia systems: the truncated-invertedpyramid. In the inverted pyramid the most important things are at the beginning, while the least important ones are at the end [14]. But this organization of the text was not detected in the multimedia systems analysed, even if it is more important than the usability of the systems. The use of the inverted pyramid is useful to increase the speed of interaction between user and computer, above all in the case of encyclopaedias, where usually the text prevails over the images. In the textual static elements analysed broadly prevails the organization of the text as a normal pyramid, especially for dictionaries and encyclopaedias. The normal pyramid admits a division in five related areas: 1. 2.
Presentation or exposition of the theme (also called lid). In this area there is the content of the rest of the textual element (textual sememe). Explanation of the principal idea. It is used to enlarge the presentation of the theme.
Communicability for Virtual Learning: Evaluation
3. 4. 5.
71
Sub-themes. It presents other aspects related to the principal idea. Contextual information or background. It creates a context for the principal data of the theme presented. More information about the presentation of the theme or lid. In this area there is a redundancy of information that consequently increases the space of storage of the information in the database. At the end, there are the least important data or a conclusion.
Areas two and three of the normal pyramid have been unified in the inverted pyramid. Therefore there are four related areas: 1. 2. 3. 4.
The lid is the principal area where there is a summary or a conclusion of the theme in no more than thirty words. Explanation of the principal idea and development of the sub-themes. Contextualization of the information. Amplification of the lid.
The use of the inverted pyramid is a writing technique that simplifies the content, as the user has access to the textual element core from the beginning. This is the reason why it is better to organize information in the form of inverted pyramid in the hypermedia systems, particularly when the user has no experience in using computers, as for example the systems developed by UOC –Open University of Catalonian [14]. A graphical representation of the areas of a normal, inverted and inverted-truncated pyramid is shown in figure 1:
Fig. 1. Normal, inverted and inverted-truncad pyramids
In each one of the areas of the pyramid it is necessary to determine the extension of the paragraphs. From the experience carried on in the UOC [14], it is suggested that in the normal pyramid the number of the words in the first area should be between 15 and 20. For the inverted pyramid, the words should be about 30. While in the other areas the paragraphs should not be composed by more than 20 words. But usually this method is not followed in the multimedia systems as you can see in the following image representing the textual element of the definition of the “Hypermedia” – English language in the Wikipedia:
72
F.V. Cipolla-Ficarra, M. Cipolla-Ficarra, and P.M. Vera
Fig. 2. Definition of the “Hypermedia” in the form of a normal pyramid with 70 words in the lid (first paragraph)
An analysis of the organization of the texts is important in order not to compromise the communicability of the multimedia system, as the understanding of the text can be difficult for the inexpert user, especially when the sentence of the text is composed of more than 20 words and there are only a few verbs (one or two). A good proportion is seven words for every verb [14]. The textual elements can be divided according to the type of pyramid presented for a detailed analysis of its content. Nevertheless, the analysis of the content should be different from the presentation or visualization of the content of the textual elements, as now the trend is to title or divide the paragraphs in order to fractionate the content of a large textual element, as for example the information concerning a country. To follow, there is an example where the textual information of the element Australia in the Zanichelli encyclopaedia [15] is divided in paragraphs with direct access. The text can be accessed directly from the titles of the paragraphs located in the inferior part of figure 3:
Fig. 3. Textual element divided and presented in paragraphs
Currently the so-called truncated pyramid is imposing itself in the texts of digital newspapers. The truncated pyramid structure is used when you have to combine the interest that comes from the facts with the need of giving a chronological order to the
Communicability for Virtual Learning: Evaluation
73
account. It consists of the following elements: headline, starting paragraph, main body of the news (main data in decreasing order of interest and secondary data in a growing order of interest), and new essential data. In the truncated inverted pyramid the principles of the lid are maintained with the extension of the words that make up the paragraphs. However, it is necessary to cut down to 50% the explanation of the main ideas and the contextualization of information, which can be replaced by referential links. These four areas of the inverted pyramid are ideal for adult and senior users of interactive systems, who have had previous experience in the off-line multimedia systems and are readers of the traditional books in paper support, including on-line. The young users feel boredom in the face of some of the components of the design of a hypertext structure, and consequently the interest towards the fruition of the content drops. Two examples may be: guided links with texts in the inverted pyramid format and of lineal structure, or rather textual referential links in the truncated pyramid format. Even in the case of using the truncated pyramid modality, many of them only read the lid or first paragraph. Therefore, it is necessary to keep in mind the use of the inverted or truncated pyramid, avoiding the use of the normal pyramid. In the first pyramids in it is necessary to sum up as far as possible the important information at the beginning, placing titles and subtitles that draw the attention of the potential users. Obviously, a senior or adult user whose purpose is research or seeing information online will be grateful in the end for new essential data or referential links.
4 Systematic Assessment Systematic assessment is a set of abstract tasks defined by communicability specialists [5]. These tasks were carried out by a group of users who are experts in the computer environment in particular, in operating systems, or hypermedia systems. A systematic assessment in our case may consist in evaluating four categories of design such as presentation, content, navigation and panchronism. Obviously, the problems that were put to the users have required a previous analysis of the inconsistencies detected in the design [16], [17]. The analysis was the responsibility of a specialist in heuristic assessment in multimedia/hypermedia system. These inconsistencies or problems can refer to the guided tours [18]. A problem that can occur within them is when you can't have access to the nodes that make them up as it happens with the system. At first a guided tours is selected and the navigation through the nodes is made in a directional way. However, there is the possibility of going back to the visited node by choosing the historic listing option, if that component is present in the design category. The latter can only be known at the moment of navigating through the application. For instance, in a guided tours that refers to an artist's collection in a given period of his life, sometimes the paintings are presented with the node number that is being visited and the total of nodes that make up the collection. (node 7 out of a total of 45, for example). In some zone of the interface we can read 7/45. This helps the user in the orientation of the navigation. However, it is necessary that the information presented in the interface agrees with the structure of the nodes and links, that is to say, that you are really in node 7 of a collection of 45 nodes. Inside a set of nodes belonging to a group, joined by a common denominator from the
74
F.V. Cipolla-Ficarra, M. Cipolla-Ficarra, and P.M. Vera
point of view of the content, for instance a historic era, a painting style, a study subject, etc. there are many nodes that have been used, that is to say, inserted in other groups but which do not adapt correctly to the new context. Finally, there can be dynamic means inside the guided links in which the advance or regression of each one of the nodes, can leave active these means: video, animation, music, etc. and prevent navigation, or there is no synchronism among them, thereby affecting the category of the panchronism. An expert in communicability will carry out the inspection of the multimedia system for the latter definition of a set of activities in a heuristic assessment with users. These activities are called abstract tasks. The communicability expert will consider the list of abstract tasks and the list of potential problems the user will find. With this information the assessor defines a series of actions or tasks to be developed by the user. These tasks that are enumerated and described next, may refer to an educational multimedia/hypermedia encyclopedia: 1. To find the geographical data of a country and the continent where it is to be found; 2. To locate guided links and make them navigate forwards and backwards; 3. To determine the borderline countries of that country; 4. Go to the screen of a region or province of that country to then be able to activate or de-activate the dynamic means, for example. Before the realization of the described tasks, it was explained to the users which were the objectives to be reached and the kind of multimedia system that had to be analyzed. In the laboratory a computer system is available for the automatic recording of the time spent and of the answers of the users at the moment of carrying out the tasks. The tables listed the averages of the values for each tasks. Besides, the users' mistakes were indicated with an asterisk (*). If the user ignored the way to carry out the task, the situation was indicated with the letters (i). These letters mean that the user did not know how to do it. The data in the tables have made it possible to identify which were the critical points during the interaction person-computer. The user test has made it possible to check in this way the problems detected in the systematic inspection by the expert. Knowing the time spent and the total number of actions carried out by the users in each task, it can be seen how the problems found by the assessor in the systematic inspection that had previously made. 4.1 Communicability Diacronism and Heuristic Evaluation With the purpose of comparing communicability diacronism and taking account of the evolution of design of the multimedia systems, the MEHEM methodology and the MECEM metrics [19] have been applied. Now the main results related to the application of the table of heuristic assessment are being presented, as described in [20], and the experiments carried out with users to verify the presented methodology. The results of the table are related to the universe of study which has enabled the elaboration of the suggested method. This randomly chosen universe of study is made up by 40 commercial CD-ROMs of international distribution (science and technology, ecology, travels, arts, history, social sciences, hobbies, sports, and games). In table #1
Communicability for Virtual Learning: Evaluation
75
Table 1. Multimedia Systems Off-line Analyzed All Movie Guide; Anatomia 3D; Argentina Vision; Astronomical explorations; CD atlas; Autodesk Texture Universe; Castelos de Portugal; Cdull; Cuba; Curso de Windows; Diccionario de la Lengua Española; Einstein; El Museo Thyssen Bornemisza; El teatro mágico; Encarta; Encicopledia del cine español; Enciclopedia del Universo; Encicopledia Zanichelli; Garden Encyclopedia; Green Bear; Historia Universal del Arte; Instrumentos Musicales; Kiyeko; La aventura de los dinosaurios; La máquina del tiempo; L’Egitto dei Faraoni; Maps Facts; Multimedia Beethoven; NN’n Toy Markers; Peter y los números; Rayman; Red Rhino; The Complete Herman; Travel Talk; Velázquez; Wild Board Games; World’s Greatest Classic Books; Peter Gabriel’s Secret World, Van Gogh and Zoo.
are described the off-line multimedia systems which have been used to analyze the static and dynamic graphic components that have been analyzed. The verification process of the presented methodology consisted firstly in the assessment in a usability lab and secondly in the elaboration and assessment of the applications developed for the UOC [14]. Other heuristic assessors and users have intervened in the verification. Some of these assessors have worked inside a usability laboratory. The users who have taken part in the assessment belong to two categories: expert and inexpert in the use of multimedia systems. Next, the obtained results and in brackets the percentage of the presence of the design component that has been assessed is depeicted: Table 2. Heuristic Evaluation of Communicability Design • • • • • • • • • •
The sound is present in most of the analyzed multimedia systems (75%) The sound prevails in relation to the animations and the video (25%) The sound is the most used device to draw the attention of the user in the navigation (20%) The audio is the device most used to foster the navigation in the analyzed systems (20%) In the base texture of the screens there prevails a combination of texts, colors and images (25%) The lighting effects that prevail are of the ambient kind (50%) The 3D simulations prevail in the 3D with shadow edges (45%) The 3D objects are simply presented with a final rendering (25%) The 2D animations are the ones that prevail in the interface (55%) The explanation of the functioning of the system lies in an help option (75%)
These results which encompass several design categories also enable the designers to have a state of the art summing-up of the commercial multimedia systems off-line in the decade of the 90s. We can see the quality results by categories of design in the annex # 1. The results obtained in the assessment with users inside the laboratory have made it possible to elaborate a list of actions and conclusions which are next presented: • Usability must be divided into several levels. One can start from very general problems which are common to all the interactive systems and go to the specific problems of specific systems. That is to say, at a first level there are the interactive systems in general terms; at a second level the categories of systems such as are the hypermedia or the multimedia and at a third level the kinds of systems, such as for instance an information stand at a museum. In each level there must be a preparatory stage, a listing of abstract tasks and a set of assessment criteria.
76
F.V. Cipolla-Ficarra, M. Cipolla-Ficarra, and P.M. Vera
• Upon evaluating a multimedia/hypermedia system it is necessary in the first place for an inspection by an expert in heuristic assessment of multimedia systems. The results of the inspection are: some application outlines, a list of the potential problems and a set of abstract tasks. The tasks must be specific activities for users in the heuristic assessment. • Empirical evaluation is more effective in relation to the quality of the results and cost when there is a list of abstract tasks. • The inspection of the expert must be efficiently combined with the empirical assessment.
5 Conclusions The heuristic assessment carried out by the communicability and usability expert in the current systems and those in off-line multimedia supports have confirmed the obtained results with the tests made in the usability laboratories in the past decade. Counting on a usability and communicability expert is very positive since it allows the tasks the users must carry out at the moment of the test to be established beforehand. Having precise tasks available allows money to be saved since the designer knows beforehand the possible inconveniences that the users will find at the moment of interaction. In the current work it has been detected how in the multimedia systems made with open source software, they maintain many of the design components of the off-line multimedia systems of the nineties. Therefore, in the current work it has been seen that it is important to consider those educative systems and adapt them to the on-line contents. In the case of the text, which is one of the design categories that prevail in the current on-line interactive systems, it is necessary to resort to organization in the shape of an inverted truncated pyramid. The presented quality attributes and the multimedia design components that have lasted throughout time will be immediately applied in the realization of multimedia products for a virtual campus. Acknowledgments. Thanks to Emma Nicol (University of Strathclyde) and Maria Ficarra (Alaipo & Ainci – Italy and Spain) for their helps.
References 1. Beverly, A.: Instructional and Cognitive Impacts of Web-based Education. Idea Group Publishing, Hershey (2000) 2. Paymal, N.: Pedagogia 3000. Editorial Brujas, Cordoba (2008) 3. Burleson, W., Picard, R.: Gender-Specific Approaches to Developing Emotionally Intelligent Learning Companions. IEEE Intelligent Systems 22, 62–69 (2007) 4. Iacovou, C., Nakatsu, R.: A Risk Profile of Offshore-Outsourced Development Projects. Communications of the ACM 51, 89–94 (2008) 5. Cipolla-Ficarra, F.: Communicability design and evaluation in cultural and ecological multimedia systems. In: Proc. 1st ACM Workshop CommunicabilityMS 2008, pp. 1–8. ACM Press, New York (2008)
Communicability for Virtual Learning: Evaluation
77
6. Willcocks, L., Sykes, R.: The Role the CIO and IT Function in ERP. Communications of the ACM 43, 32–38 (2000) 7. Campbell-Kelly, M.: Will the Future of Software be Open Source? Communications of the ACM 51, 21–23 (2008) 8. Torvalds, L.: The Linux Edge. Communications of the ACM 42, 38–39 (1999) 9. Hammnd, N., et al.: Educational Multimedia for Conceptual Understanding. In: Proc. ACM Multimedia, pp. 447–456 (1995) 10. Larochelle, M., Benarz, N., Garrison, J.: Costructivism and Education. Cambridge University Press, Cambridge (1998) 11. Pea, R.: Learning through Multimedia. IEEE Computer Graphics & Applications 1, 58–68 (1991) 12. Garrand, T.: Writing for Multimedia. Focal Press, Boston (1997) 13. Stephenson, J.: Teaching and Learning Online: Pedagogies for New Technologies. Kogan Page, London (2001) 14. Cipolla-Ficarra, F.: Evaluation and communication techniques in multimedia product design for on the net university education. In: Multimedia on the NetVienna, pp. 151–165. Springer, Heidelberg (1996) 15. Zanichelli Enyclopaedia CD-ROM. Zanichelli Publishing, Bologna (2004) 16. Nielsen, J., Mack, R.: Usability Inspection Methods. Willey, New York (1994) 17. Lewis, J.: Sample Sizes for Usability Tests: Mostly Math, Not Magic. Interactions 13, 29– 33 (2006) 18. Trigg, R.: Guided Tours and Tabletops : Tools for Communicatiing in Hypertext Environment. ACM Transactions on Office Systems 6, 398–414 (1988) 19. Cipolla-Ficarra, F.: Communication Evaluation in Multimedia –Metrics and Methodology. LEA 3, 567–571 (2001) 20. Cipolla-Ficarra, F.: Table of Heuristic Evaluation for Communication of the Multimedia Systems. In: Proceedings of the HCI International, pp. 940–944. LEA, Crete (2003)
Annex #1: Evaluation of Multimedia Systems
Attention and Motivation in Hypermedia Systems Francisco V. Cipolla Ficarra1,2 and Miguel Cipolla-Ficarra2 HCI Lab. – F&F Multimedia Communic@tions Corp. ALAIPO: Asociación Latina de Interacción Persona-Ordenador 2 AINCI: Asociación Internacional de la Comunicación Interactiva Via Pascoli, S. 15 – CP 7, 24121 Bg, Italy [email protected], [email protected] 1
Abstract. We present the results of a heuristic analysis of a set of multimedia off-line systems aimed at boosting the mental skills of the users through reflexes, maths, etc. To this purpose two metrics have been created to assess the motivation of the users and the degree of help implicit in the multimedia system. The metrics are based on a group of primitives aiming at increasing the communicability of commercial multimedia systems and targeted to the public at large, regardless of age and previous experience in the use of computers. Keywords: Attention, Motivation, Hypermedia, Design, Navigation, HumanComputer Interaction, Help.
support. From 1995 onwards plenty of small and medium businesses sprang up with the purpose of digitalizing the contents in paper support. Here takes place the first transference of the concepts of the graphical arts to the design of computer interfaces. Obviously, the dimension of the screen was bigger than the classical A4 folio. Many principles of topography, colour, disposition of the images, with their texts, etc., responded to the graphical arts of the paper [4]. To design a screen at that time was to organize the background of the screen, that is to say, all the elements in the screen that contribute to the appearance and the behaviour of the interface. The background carried all the burden of the design because it covered a great part of the screen. The background, from the point of view of motivation and the design fulfills two functions: First, it has an influence in the appearance and the visual balance, and the position of all the elements. Second, it fills the void space so that other elements are not flying objects. In both cases, all the elements must be differentiated, but at the same type an isotopy must be created among them [5]. Currently, we find university websites where the background is textual, full of colours that do not respect any chromatic order for the interfaces design [6] and the text shifts vertically, joined to dissolution transitions by dots among the frames of the different nodes, at the moment of navigation. That is to say, a style from the early 90s for off-line multimedia systems but which can't be applied in the era of Web 2.0 and 3.0, especially in the university environment. An interface with such backgrounds immediately undermines motivation of the contents' fruition.
2 Motivation: Dynamics and Statics Means The motivation for this study is the whole set of dynamic means and structural resources that boost the navigational quality of a multimedia/hypermedia system. With dynamic means and structural resources, it is intended to increase or maintain interest towards the system, especially when the user is in a student role [7], [8]. The motivation seeks to focus the user’s attention on the need to continue fruition of the system. Attention on the screen must be sustained, that is, the user keeps an attitude of permanent expectation with regard to the system. It is mainly virtual resources that trigger very good results in attention and acceptance towards the system, especially when there is no previous computer experience by the user. Without going deeper into psychological aspects, there are two factors related to the content and structure which allow attention to be drawn to an interface. If the content of the information is specialized, it means that the target of the multimedia system is an expert, and therefore the information must be relevant and be well-organized. Here the attention on the interface is of a cognitive character [8]. There is an “affective” link of the user towards the system because of the linearity of the message. This lineal factor forces the user to utilise the system until the end of a sequence. As a rule, the establishment of a sequence series through the use of structural methods is due to the fact that the user studies the content. These cognitive or affective factors can be used to draw attention to a multimedia/hypermedia system in a joint or individual way [9]. Choosing one way or the other will depend on the kind of user and the purpose of the system, such as educational, consultation, entertainment, etc. But both factors are present in the motivation or boosting of navigationn multimedia systems one resorts continually to dynamic means to motivate fruition:
80
1.
2.
3.
4.
F.V. Cipolla Ficarra and M. Cipolla-Ficarra
Music; Sound effects that are activated upon reaching a goal, such as are: to pass an educational subject with success, to go through a higher level inside a tale, to win a game, etc. Voice; The narrator or speaker incorporates sentences to promote advancment through the system. For instance, in the fragmented and gradual assessment of content in a computer-assisted teaching system. Text and images linked to transitional effects, such as blurring, sweeping, fadeins, fade-outs. The textual sentences have expressions of the kind: “excellent”, “go on,” and “continue”. 2D and 3D animations of characters and objects (in the universe of study they are 2D animations, made with Flash [10]).
Fig. 1, 2. Interfaces with 2D animations in Allenamente DVD-ROM
Examples of structural resorts which maintain and boost interest in the system: The outlines that at the end of a guided link make it possible to switch to other entities; An Index, and nodes that link to other components of the same entity. The existence of resources to promote or maintain motivation have a positive influence on the richness of a multimedia system [11]. The relationship between the attributes of wealth and motivation are based mainly on resources that belong to the visual component, such as the images and interactive help. Interaction with the system can generate a series of mistakes due to either hardware or software. Hypermedia systems usually have means to solve these hardware errors as well as those of software. There are several ways of presenting help in the interface. This help can be classified in regard to the access and the presentation in the form of tutorials, outlines and exploratory texts: 1. 2.
3.
Help in the shape of a tutorial consists of a collection of screens with an index on the first of them (can be activated from all the screens in the system –fig. 3 [12]). Help of the outline kind is that in which on a single screen are explained the functions of the main components of the interface, as it can be observed in figure 4 [13]. The exploratory help is that in which as the cursor moves along the keyboard, the explanation is activated, for example, in figure 5 [14]. In the automatic correction
Attention and Motivation in Hypermedia Systems
81
of Gmail spelling, for instance, sometimes we see how the first word remains hidden, thereby preventing a quick visualization of the possible correct words. (figure 6). A good example of this is the interface of the Cinemanía system as it can be seen in the figure 4.
Fig. 3. Help in the form of a tutorial
Fig. 4. Help of the outline
Fig. 5. Upon placing the cursor on the clock icon the meaning appears
Fig. 6. Gmail spellchecker –the first word remains hidden
82
F.V. Cipolla Ficarra and M. Cipolla-Ficarra
There are several ways to activate help for the explanations of the components of a screen, such as the shifting of the cursor on the interface; through an option as has been seen in figure 5 or in the presentation of the system. In this last case it is shown in the shape of an automatic animation before it appears on the main screen. However, 63% of the analyzed multimedia systems lack available aids on all their screens. Access to these aids should be constant in all the screens of the system regardless of the place within the structure in which the user finds themselves. Although the purpose of the aid is to facilitate interaction, there are systems in which the help is not a solution to the user, since they increase disorientation and demotivation within the interaction. The use of some kind of help or another at the moment of design will depend on the user of the system [15]. Among the different kinds of access to help, the most positive to the expert user are the outline and exploratory modalities. The outline modality allows users watching on a single screen all the explanations of the components in the shape of an outline which speeds up reading, but obviously avoiding the failure in figure 5. For the inexperienced user, it is positive to have a simple movement of the mouse from which the meaning of the subject indicated on the screen is revealed through exploration. However, for an expert user the exploratory help can turn out to be negative by watching information he/she already knows. This is due to redundancy of information, and to the fact that exploration requires a longer time at the moment of interaction with the system. For these kind of users it is advisable to use other means of help.
3 Attention and Users The treatment of the attention stimuli must pursue a simplification, hence the organization of the framing of the visual dynamic means is important. Creating excessively informative situations must be avoided in order not to wear down the user. This can only fully handle a small fraction of the whole information that reaches his/her senses. Through the capacity for selective attention the user pays attention discreetly to some stimuli while ignoring others. In any development in which a great amount of elements are combined, as it happens in any message of the visual dynamic means, this process takes place. If a big part of the development of the speech is left to the user's selective attention ability, you run the risk of not being able to establish an effective communication, because it is liable to trigger many limited interpretations and in some cases overlapping interpretations. Here we do not refer to the participation margin, to the stimulation of the user's imagination that any hypermedia must present [9], [16] but to the lack of structuring or conductive order of the sense. In this sense the camera movements (video or animation) are important. For instance, in the horizontal movements there is a tendency to split the screen in two halves, one to the left where we place the I and now, and another to the right, where we place the “you”, the future and any pictorial object located in this part tends to look bigger. Therefore, the movements from left to right symbolize a movement from I to you, the movement in direction to the world, the progressive movement. Its perception demands lesser effort than those aimed at the right. The movements from right to left are related to traditionalism and interiorization, it is a conservative movement, in direction to the origin, to the I. The direction of the movement also affects the
Attention and Motivation in Hypermedia Systems
83
perception speed by the user [17], in general the descending motions and from left to right are faster than ascending ones, and those from right to left. Here is the reason of the importance of the elements that make up the visual composition of the dynamic means. The sound is another element to be combined during the composition has a similar treatment, in front of the attention and the stimuli that make it work, for example in games [18], or the audio channel, by establishing relations to the visual side, constitutes a semantic and/or expressive boost in the whole articulation of the multimedia content. The obtained results make it clear that in the set of analyzed systems, motivation is only of 20% value, and the quality of the help is equal to 35% of the analyzed cases, as revealed through heuristic techniques. Consequently, the analyzed systems are not communicable to those users who do not have previous wide experience in the use of computers. 3.1 Attention, Users and Audio-Visual Factors With the passing of time and the reduction of the size of screens it was necessary to include other components from the dynamic means in order to appeal to the attention and the motivation of the users. One of those was the video. A good image that speaks and moves has a greater communicative power than that of a photograph or a graphic, for instance. However, it loses that ability to draw attention and motivate the user if the content does not convey a message or a captivating experience. Evidently, the fact that the longest video segments require a greater variety and more attention to the coordination of those segments (something missing here!). Twenty seconds with an anchorman who makes you think of a speaking statue may seem like eternity to a teenager, whereas a 75 second long action sequence will seem short as compared with the first. Now, from the point of view of design of the interface it is necessary that the video merges to some extent with the rest of the objects that make it up. In multimedia off-line systems the use of video was cut down to a minimum for several reasons: the room occupied in megabytes in the PC, the low quality of the full screen reproduction with which they were cut down to 10-12 cm. long small squares or rectangles (the user was accustomed to seeing these kind of images in the screens of computers which were over 20 inches big), the copyrights, etc. The computer animations through the video games consoles would breed another generation of users of multimedia systems dedicated to a pastime who didn't use the keyboard or the PC mouse [17]. In those systems, the animations tried to draw the attention of the users through the audio, that is to say, the music, the special effects, etc. , which were especially created for each one of the interactive games. In order to successfully integrate the sound in the interface it was necessary to pay attention to the synchronism among the music levels, the human voice in the narration (where it exists) and the sound effects. All of them were well-balanced in order to achieve the desired effect on the user, especially in the design category known as panchronism. In the combination of the audio with the animations there was an excellent coherence of the colours, lightning and scenography, the characters the script of the plot or adventure, etc. Many of these design strategies in the dynamic and static means have been boosted through lighting, for instance.
84
F.V. Cipolla Ficarra and M. Cipolla-Ficarra
4 Visual Attention Evidently it is very important that the illumination component and special effects (FX) enter into learning process independently of the contents and animations 2D and/or 3D. We have three main effects: • Lighting effects: The effects stemming from lighting such as are shadows, reflections, diffusion, etc., must also remain constant in the whole hypermedia system. The lighting effects derive from the kind of light that is used in scenes with tridimensional objects. This kind of light can be environmental or directional. In environmental lighting the objects that receive the light generate shadows, whereas in directional lightening a beam of light is emitted over a given object of the scene. When there is more than a beam of light, lighting is multidirectional. In the following figures extracted from the off-line multimedia systems Braincity [18] and Universal History of Art [19], there are examples of lighting. The correct use of environmental light can be seen in all the scenes that make up the application and in the Universal History of Art. Through directional lighting the main options of the system are shown and activated.
Fig. 7. Directional lighting (center of the image)
Fig. 8. Environmental lighting
• The effects of the reflection or shining simulate the arrival of the light to a spot in an object. This light can be depicted in the shape of a star. The shine can be either static or dynamic. It is dynamic when an animation is associated with it. The animation may consist in a switch of 360º over its axis or in an increase or diminution of the shining. This effect is used mainly when in the background of the screen the black colour prevails, the scale of grey and blue shades, and all the metalized colours such as gold, silver, copper, etc. The shine means prestige or superiority of an element in relation to the others that make up the interface. They can also be useful to draw or attract the attention to an element in the screen. In the Movie Guide we can find a dynamic shine (the intention is to stress a “search wizard” which can be activated from any place in the structure of the system). • Relief effect: There was an attempt in the 90s to emulate the shadows produced by environmental lighting in the screen elements. The purpose in these first
Attention and Motivation in Hypermedia Systems
85
hypermedia systems was that these bidimensional elements “acquire” tridimensionality. Emulation is obtained by correctly darkening the edges of those elements that are intended to be stressed in the interface. Today it is a common element in the web 2.0 and web 3.0 Between these two elements the navigation keys can be mentioned, typography, picture frames, etc. It is a positive effect in all the navigation keys, since they are easier to recognize in an interface, as it can be seen by comparing figures 9 [21] and 10 [22]. The interface of a multimedia system has a greater degree of realism with the emulation of the shadows, which favors their acceptance.
Fig. 9. A greater degree of realism
Fig. 10. A normal degree of realism
Without any doubt, lightning, colours, shapes, the organization of the contents, etc. have a very important role in the motivation and in the attraction of the users in the on-line and off-line multimedia systems. A good or bad lightning of the components that make up the interface can boost or not the interaction of the potential users, regardless of their content and the goal of the interaction: entertainment, educational, informative, etc.
5 Heuristic Evaluation: Results In our case we have assessed as a representative example a set of 15 multimedia systems in DVD-ROM support from the Allenamente collection [10], which theoretically favors language, logic, memory, etc. Besides, it contains an endless series of games and exercises for self-assessment the more the user advances in the contents. The search for the presence of the motivation and the attention in the ensemble of the analyzed off-line multimedia systems has led us to create a series of metrics. For their elaboration we have resorted to the primitives stemming from the evolution of the interactive systems: hypertext, multimedia and hypermedia. The listing of primitives and the design categories to which they refer are: in annex #1. The graphic showing the obtained results. The graphic demonstrates the scarce quality from the motivation point of view and the attention of the analyzed systems.
86
F.V. Cipolla Ficarra and M. Cipolla-Ficarra
6 Conclusions The motivation and the holding of the users' attention in the content of interactive systems: multimedia, virtual reality, etc. is one of the main goals at the moment of the design. Now, the 3D graphic informatics has more available resources in this sense than 2D informatics. The dynamic means play a very important role at the moment of drawing and keeping the users' attention. The metrics presented in this work have made it apparent how in the off-line multimedia systems with 2D animations there is a scarcity of animated graphic resources and a lack of creativity in the planning of the contents. An interesting content without resorting to communicability may turn out to be negative for the potential users to keep on navigating through the different systems that make up a wide multimedia systems collection. The motivation and the attention to the design of an interactive system must not only involve the content, but also each one of the following categories: presentation, navigation, structure and synchronization of the dynamic means, that is, panchronism. All these categories must include the quality attributes for multimedia/hypermedia systems off-line and on-line. Acknowledgments. Thanks to Emma Nicol (University of Strathclyde), Maria Ficarra (Alaipo & Ainci – Italy and Spain), Quim Romo-Ferrer and Carlos for their helps.
References 1. Soh, J., Tan, B.: Mobile Gaming. Communications of the ACM 51, 35–39 (2008) 2. Cipolla-Ficarra, F., Cipolla-Ficarra, M.: Interactive Systems, Design and Heuristic Evaluation: The Importance of the Diachronic Vision. In: New Directions in Intelligent Interactive Multimedia, pp. 625–634. Springer, Heidelberg (2008) 3. Silva-Salmerón, J., Rahman, M., El Saddik, A.: Web 3.0: A Vision for Bridging the Gap between Real and Virtual. In: Proc. 1st workshop communicability design and evaluation in cultural and ecological multimedia systems, pp. 9–14. ACM Press, New York (2008) 4. Kahn, P., Lenk, K.: Principles of Typography for User Interface Design. Interactions 6, 15–29 (1998) 5. Cipolla-Ficarra, F.: Evaluation of Multimedia Components. In: Proc. IEEE Multimedia Conference on Multimedia Computing Systems, pp. 557–564 (1997) 6. Cipolla-Ficarra, F.: HECHE: Heuristic Evaluation of Colours in HomepagE. In: DVDROM Proc. Applied Human Factors and Ergonomics, Las Vegas (2008) 7. Cipolla-Ficarra, F.: Evaluation and communication techniques in multimedia product design for on the net university education. In: Multimedia on the Net, pp. 151–165. Springer, Heidelberg (1996) 8. Edwards, A., Holland, S.: Multimedia Interface Design in Education. Springer, Berlin (1992) 9. Card, S., et al.: The Pychology of Human-Computer Interaction. Hillsdale, New Jersey (1993) 10. Allenamente DVD-ROM. GreenTeam, Bologna (2008) 11. Cipolla-Ficarra, F.: Evaluation Heuristic of the Richness. In: Proc. International Conference on Information Systems Analysis and Synthesis, ISAS, Orlando, pp. 23–30 (1999)
Attention and Motivation in Hypermedia Systems 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22.
87
Enciclopedia de la Ciencia CD-ROM. ZetaMultimedia, Barcelona (1998) Cinemania CD-ROM. Microsoft, Madrid (1996) Musée d’Orsay CD-ROM. Montparnasse Multimedia, Paris (1996) Dworman, G.: Arbitration of a Help System. Interaction 14, 39–42 (2007) O’Neill, S.: Interactive Media –The Semiotics of Embodied Interaction. Springer, London (2008) Terzopoulus, D.: Artificial Life for Computer Graphics. Communications of ACM 42, 32–42 (1999) Raghuvanshi, N., et al.: Real-Time Sound Synthesis and Propagation for Games. Communications of the ACM 50, 66–73 (2007) Braincity CD-ROM. Digital Illusion, Barcelona (1995) Historia Universal del Arte CD-ROM. Espasa-Calpe, Madrid (1996) Explorama CD-ROM. Anaya, Madrid (1995) All Movie Guide CD-ROM. Corel, Ottawa (1995)
Annex #1: Primitives and Graphic Results Table 1. (P)rimitives for heuristic analysis of the motivation and attention (design catogories: content, dynamics, structure, presentation and panchronic) Analepsis (P): Content and Dynamic; Continuum (P): Content and Dynamic; Element (P): Content; Element Type (P): Dynamic and Structure; Entity (P): Content and Presentation; Frame (P): Presentation, Content, Dynamic and Panchronic; Frame Principal (P): Presentation, Content, Dynamic and Panchronic; Guided Tour (P): Presentation, Structure and Dynamic; Hierarchical Links (P): Structure and Dynamic; Index (P): Structure and Dynamic; Hypertrails (P): Structure and Content; Keyword Links (P): Content; Link (P): Structure; Node (P): Content; Polytopes (P): Presentation and Content; Polysemy (P): Content; Referential Links (P): Structure and Content; Sememe (P): Content.
A Web-Based, Interactive Annotation Editor for the eCampus Development Environment for SCORM Compliant E-Learning Modules Benedikt Deicke, Jan-Torsten Milde, and Hans-Martin Pohl University of Applied Sciences Fulda Competence Center Human Computer Interaction [email protected], [email protected], [email protected]
Abstract. The eCampus development environment was created in an interdisciplinary project at the University of Applied Sciences Fulda. Today it is a fully webbased application for the easy creation of E-Learning modules complying the SCORM standard. The webbased, interactive annotation editor for the eCampus development environment is used to both automatically and manually annotate existing OpenOffice documents in order to transform them into E-Learning modules. The editor is build using Open Source software and frameworks such as Ruby on Rails. Keywords: E-Learning, Web, SCORM, eCampus, OpenOffice, Ruby, Ruby on Rails, JavaScript, user friendly, annotation, transformation.
1 Introduction The creation of standard compliant E-Learning modules is a complex task. In order to simplify the process, the University of Applied Sciences Fulda created the eCampus development environment [5]. It is based on OpenDocument files which are transformed into SCORM [1] compliant E-Learning modules using XSL Transformations. The transformation relies on annotations added to the OpenDocument file using OpenOffice.org [3]. Nevertheless, the handson experience showed that the annotation process using OpenDocument styles within OpenOffice.org is still too complicated for most of the users. With the “eCampus” development environment now being a web-based and integrated system for the generation of SCORM compliant E-Learning modules, the creation of an interactive, web-based annotation editor is the next logical step. To simplify the annotation process even more an automatic analysis and annotation process is implemented, too.
A Web-Based, Interactive Annotation Editor for the eCampus Development Environment
89
the eCampus project at the University of Applied Sciences Fulda. In this interdisciplinary project, content is being created for the faculties Nutritional Sciences, Food and Consumer Sciences, Applied Computer Science, Nursing & Health Care as well as Food Technology. Within the first years of the project, many E-Learning modules were produced. All modules address prominent introductory courses of the particular program of study. This was only possible by using the eCampus framework. The central target of the project is the implementation of a tool that decreases the complexity of the process of learning module production. The central objectives of the learning module production are an increase in the interactivity of the course, as well as an increase in the quality and quantity of self learning.
3 The Tools The annotation editor is based on the existing eCampus web application which is built using Ruby [8] and the Ruby on Rails framework [7]. It is running as an E-Service on a centralized server. In order to use it, the author has to create a user account. This is required to protect uploaded documents and generated E-Learning modules from unauthorized access. The user interface is built using XHTML and CSS. It is enhanced by JavaScript utilizing the Prototype and Script.aculo.us frameworks [4].
4 The Process Using the webbased annotation editor the process of creating E-Learning modules from OpenDocument files is condensed into five simple steps: Uploading of the unannotated OpenDocument file, automatic analysis and annotation of the file, manual annotation and correction of the resulting document, transformation into a SCORM compliant E-Learning module, downloading of the finished module. 4.1 Reading and Modifying OpenDocument Files In order to read and manipulate the OpenDocument a library has been developed. It encapsulates the required steps within a simple API. In order to read the contents of the document it is extracted using RubyZIP. Afterwards the content. xml file, which contains the actual content, and the styles.xml file are parsed using Ruby’s built-in XML library [6]. Utilizing Ruby’s dynamic nature, the library creates a class for every node type contained in the document’s content and instantiates it as needed. This provides the possibility to dynamically overwrite the default behavior for specific node types. 4.2 Automatic Analysis and Annotation The process of the automatic analysis and annotation is separated into different annotation engines. The engines analyze every stylable element in the OpenDocument’s content structure using different approaches. If the analysis step is
90
B. Deicke, J.-T. Milde, and H.-M. Pohl
not successful, the next annotation engine tries to analyze the current element. This is continued until no annotation engines are left or an approach is successful and annotates the element. This process is repeated for every annotated element of the input text. Figure 1 visualizes this concept. Currently five annotation engines are implemented: Keep, Tags, Styles, Bayes and Default. The Keep engine stops the analysis chain if the element is already annotated with eCampus styles by checking each elements formatting style. The Tags engine analyzes the type of the current element and its context and tries to map this onto eCampus styles. For example text:h nodes with an outline level of 1 are annotated with EC_Modul, nodes with outline level 2 as EC_Session, etc. The Styles engine simply translates default OpenDocument styles into eCampus styles where possible, such as Text body to EC_Text. The Bayes engine tries to analyze the current elements style properties in order to detect elements like headlines or important terms which were just changed visually, but not by using using built-in OpenDocument styles. In order to achieve this, the Bayes engine is trained with correctly annotated documents. Based on the annotation and the visual style, the engine learns possible visual variations. While visiting each node in the document the Bayes engine builds “sentences” from the nodes style attributes. This results in sentences like “12pt bold italic” which are handed to a bayes classifier that returns a corresponding eCampus style or unknown if no style could be identified.
Fig. 1. The automatic annotation process
As well as the final transformation into a SCORM compliant E-Learning module, the automatic analysis and annotation takes several seconds (or even minutes, depending on the documents size) to finish. Therefore it needs to run asynchronously in the background. This is done marking the documents as waiting for analysis in the database. See Figure 2 for a complete overview of possible states. A separate processes, triggered by CRON, checks for waiting documents and runs the analysis on them. On the client side the user can see the status of the transformation. It is automatically updated using a polling AJAX request.
A Web-Based, Interactive Annotation Editor for the eCampus Development Environment
91
Fig. 2. Possible states of a document
4.3 The Interactive, Web-Based Annotation Editor The front end of the annotation editor for manual annotation is implemented on the client side using JavaScript. It uses Ajax techniques [2] to save changes to the document on the server. The GUI is designed to be simple and clean and focusses on the important functions needed to annotate the document. Figure 3 shows a screenshot of the final GUI. It consists of a toolbar on top, which includes buttons for navigating through the document and buttons for saving and exiting the editor. The tooltip on the right is used to actually annotate the currently highlighted element. The user is able to select the desired style by selecting it from the select-box. The JavaScript implementation is designed to be as modular as possible. It utilizes the observer pattern to separate the various GUI components from the underlying logic. On startup of the annotation editor, every component registers itself with the AnnotationEditor core class. Based on the users interaction with the GUI the elements fire events on the core class, which executes the requested functionality and fires events on all registered components as a result. This enables the possibility of additional annotation methods, other than the tooltip described above. Imaginable annotation methods are: dragging and dropping the styles on the elements or a fill format mode like known from OpenOffice or Microsoft Word. To display the OpenOffice document it is read on the server and transformed into a simple HTML representation. Each HTML elements relation to a node in the original OpenOffice document is described by adding an attribute containing the nodes XPath to the element. The nodes annotation (such as EC_Definition, EC_Section, EC_Image, etc.) as well as its family (Paragraph, Text or Graphic) are represented using CSS class attributes. Additional every HTML element that can be modified later the CSS class annotatable is added. During the manual annotation the underlying CSS class attributes identifying the annotation are added, modified or removed. On modification of an element it gets
92
B. Deicke, J.-T. Milde, and H.-M. Pohl
marked as changed. When the user decides to save the changes, all marked elements are collected and serialized into JSON before they are transfered to the server using an Ajax request. On the server the JSON is deserialized and the changes are applied to the OpenOffice document using the library described above.
Fig. 3. Screenshot of the Graphical User Interface
5 Results and Visions The combination of a simple user interface for the annotation and the automatic analysis and annotation process the creation of standard compliant E-Learning modules out of existing content simplifies the creation of standard compliant ELearning modules. Being integrated into the eCampus web application the annotation editor inherits useful features like easy updating and extending. The annotation editor completes eCampus to a full web based development environment for SCORM compliant E-Learning modules. Future development could include the possibility to rearrange elements within the document or even change the content. Additionally the current user interface needs to be fully evaluated during end-user tests. As described above other annotation methods could be implemented as well. Furthermore the automatic annotation engines offer a broad scope for new additions.
References 1. ADL. Scorm, http://www.adlnet.gov/scorm/ 2. Garret, J.J.: Ajax: A new approach to web applications (August 5, 2008), http://www.adaptivepath.com/ideas/essays/archives/000385.php 3. OpenOffice.org. The free and open productivity suite, http://www.openoffice.org/ 4. Porteneuve, C.: Prototype and script.aculo.us. The Pragmatic Programmers, 1st edn. (2007)
A Web-Based, Interactive Annotation Editor for the eCampus Development Environment
93
5. Pohl, H.-M., Tulinska, P., Milde, J.-T.: Efficient creation of multi media eLearning modules. In: Smith, M.J., Salvendy, G. (eds.) HCII 2007. LNCS, vol. 4558, pp. 457–465. Springer, Heidelberg (2007) 6. REXML. Ruby standard library documentation (August 18, 2008), http://www.rubydoc.org/stdlib/libdoc/rexml/rdoc/index.html 7. Thomas, D., Heinemeier-Hannson, D.: Agile Web Development With Ruby On Rails, 1st edn. The Facets Of Ruby Series. The Pragmatic Programmers (2005) 8. Thomas, D.: Programming Ruby - The Pragmatic Programmers’ Guide, 2nd edn. The Pragmatic Programmers (2005)
An Innovative Way of Understanding Learning Processes: Eye Tracking Berrin Dogusoy1 and Kursat Cagiltay2 Computer Education and Instructional Technology Department, 1 Mersin University, 2 Middle East Technical University {bdogusoy,kursat}@metu.edu.tr
Abstract. This paper aims to present findings on the use of eye-tracking technology as a new approach from an educational perspective. The studies in this paper on relationship between learning and eye-movements have focused on concept-map formation, learning from multimedia materials, designing materials with different cognitive strategies, multimodal comprehension of language and graphics with and without annotation, computer games and cognitive style effects of computer based interfaces and hypertext environment. The results of the Middle East Technical University (METU) Human Computer Interaction (HCI) research group’s eye-tracking based research studies presented and discussed how this approach helps educators to better understand learning processes of humans. Understanding and using this innovative approach is important for both educators and researchers in terms of comprehending learning processes deeply. Keywords: Eye-tracking, web based learning processes, concept maps, computer games, learning from multimedia.
An Innovative Way of Understanding Learning Processes: Eye Tracking
95
potential to give a broaden perspective for the researchers providing both application opportunities and the different analysis techniques. This paper aims to provide information on this new and innovative approach from an educational perspective. Our studies on learning and eye-movements have focused on concept-map formation, learning from multimedia materials and computer games. The results of eye-tracking based research studies [5] were presented and discussed how this approach may help educators to better understand learning processes of humans. Understanding this innovative approach is important for both educators and researchers in terms of comprehending the learning process deeply by using its new techniques. 1.1 Related Eye-Tracking Studies There are many studies which were conducted by using eye-tracking approach. Eye tracking approach gives researchers opportunity to collect information on user behavior in definite tasks. This device also provides researchers various data related with the user through the process of task; task completion duration, transition number, fixation count, average fixation duration, time to first fixation, hotspot, fixation order, gaze time and gaze replay data [6]. Under METU HCI group, eye-tracking studies have been done by using a usability laboratory environment. Baran, Dogusoy and Cagiltay (2007) conducted an eyetracking study to understand how adults solve Tangram (Chinese puzzle) based geometry problems [7]. The sampling of this study consists of twenty participants. To analyze the problem solving process, the eye movements of the participants were recorded while they were working on two problems with different levels. The results showed that the participants use different strategies to solve problems given in two different difficulty levels. The researchers indicated that there might be relationship between the task completion duration and the difficulty levels of the problems. Moreover, eye-tracking device enabled the researchers to use the Area of Interests (AOI) option to determine different areas on the screen and these areas were used while analyzing the users’ behaviors. In this study, the AOIs were used by dividing the screen into two parts (geometric object screen-the objects given in this screen and problem screen-the solving process is eventuated) in order to determine the different behavior patterns of the participants in problems with two different levels. The researchers argued that there is a relation between the problem complexity level and the focused place on the screen. The participants tended to focus on the problem screen rather than the geometric objects screen. This was explained by the researchers as participants used both inductive and deductive strategies for solving the problems with different levels. This study have potential in terms of giving beneficial clues to educators both using different strategies and support the learners in the development of strategies for solving the problems in different difficulty levels. User analysis is an important issue for instructional designers and in another study the main issue which was explored is the presumptive effects of cognitive style on the interaction ways while using computer based interfaces [8]. Since cognitive style is frequently researched issue by the researchers especially the potential influence of cognitive style on the interaction with interfaces, they grouped the participants by using Cognitive style test, Group Embedded Figures Test (GEFT) and they were
96
B. Dogusoy and K. Cagiltay
informed about the process. The process includes studying the predetermined web site and after that the participants were required to take a test about the web site content. The results showed that although there wasn’t a statistically significant difference between the participants in terms of the eye-tracking data, there were differences between the two groups in terms of the fixation duration and fixation places. It was suggested by researchers to study with wider groups and using think-aloud protocol during the process, studying the website in further studies. Another eye-tracking study was related with the multimodal instructional materials and the behavior of the users was conducted [9]. The main purpose of this study was to investigate the potential effects of the instructional environments which were designed by using multimodal information on the participants’ behaviors. Fifteen participants’ behaviors were recorded and their eye movements were analyzed by using eye-fixation counts and durations. The researchers indicated that the participants used both slide and video presentations in the study. Also, the participants much more focused on the video screen than the slide presentation screen which was obtained from eye-fixation duration data. They suggested replicating the study by using different combinations of the slide and video materials, that it might be possible to offer effective designs for this kind of instructional materials. Eye-tracking is used as a beneficial approach in terms of observing the general patterns of the participants while completing a specific task. In another study, this approach is used for observing the general pattern of the prospective teachers during the concept map construction [10]. With the aid of eye-tracking device it was aimed to determine whether there was a general pattern among the participants while developing a concept map. Especially the focusing time on concepts, links and cross links were explored. Also, the quality of the created concept maps was considered while analyzing the eye-tracking data on process. Sixteen prospective teachers attended this study from the science education related departments. The results showed that the participants tended to develop hierarchic concept maps, and synchronize strategy while writing the links and concepts. The researchers indicated that with the aid of this study, it may be possible to give some hints to teachers about the effective use of concept maps and their construction process. Usability is a crucial issue in education that design process is directly affected by the usability related problems. In another research study, the main aim was to gather information on the issues related with the computer games especially usability aspect with the aid of eye-tracking device [11]. They used a computer game which was not familiar for the participants and tracked the behaviors of them during the learning process of a new game. Sixteen participants attended to the study. The researchers pointed that they used not only quantitative but also qualitative methods. Eye tracking approach provided information related with the fixation number, total duration and gaze time of the participants. In addition to this, the comments of the participants were taken for determining the main usability issues related with game. Researchers explained that during the learning process the participants tended to use a trial and error approach. Although there were no gender and level differences among participants in terms of education effect on participants’ learning process, the participants marked many usability issues related with the game. Also, they emphasized that the game should be re-designed with the aid of the information gathered from the users.
An Innovative Way of Understanding Learning Processes: Eye Tracking
97
The researchers in another study aimed to investigate the cognitive style effects on navigation patterns during an interaction occurred within a hypertext environment [12]. The sampling of this study consisted of twenty participants and they were grouped with the aid of the Group Embedded Figures Test (GEFT) and also the knowledge and ability of the participants on computer was another factor. A predetermined task was given to the two groups (Field-dependent and field-independent) and they were asked to navigate within a hypertext environment. Data was gathered by using eye-tracking device; the fixation duration and gaze points of the participants which were recorded during the task completion process also the navigation complexity, visited and revisited page numbers were traced. The researchers observed the participants and interviewed with them. Also for strengthening the process, think aloud protocol. This protocol was used to comprehend the process. It was explained that these data would help to verify the eye-tracking data. Moreover, the participants took a recall test to understand if there was a difference among groups. According to the results of this study, there was no significant effect of cognitive style on the participants’ navigational patterns. The researchers suggested that this study should be done with a wider sampling and this may give more reliable results in terms of the cognitive style differences. Ozcelik, Arslan and Cagiltay (unpublished manuscript), proposed a study about the effectiveness of signaling the pertinent keywords or sentence on the performance of the participants [13]. In this study, for determining the influence of signaling on reading they used eye-tracking device. The sampling consisted of twenty-eight undergraduate students, one group was given signaled keywords and the other group was given signaled sentence. The results of this study showed that there is no difference between the groups in terms of their performance. The retention and transfer scores were not influenced from the different designs. However, the eye-tracking data contributed the results of the study that signaling affected the students in a positive way, increased attention, less visual search and gave attention to the necessary information from the material. In the same vein, a similar study on signaling and its impacts on learning was conducted [14]. In this study, the researchers used two groups, one of them is signaled and the other one is nonsignaled. They preferred to use color signaling in their study, the transfer test was an indicator of their study in terms of the performance of the participants. The researchers pointed out the inconsistencies in the literature about the retention and transfer. Also, they emphasized the importance of eye-tracking in the study with giving the four main reasons that the signaling contributed to the meaningful learning. First of all, they stated that it was supported by the number of fixations on the necessary information. Secondly, they emphasized that much more time spent on the persistent information given that the gaze duration and average fixation supported this argument. Thirdly, they explained that the time of the visual search was decreased. Lastly, in terms of the effective positioning the relevant information, the data was beneficial since increased fixations gave valuable information about the process. Yet another study was conducted on the effects of color coding on multimedia learning and visual search of the participants [15]. The purpose of this study is to differentiate whether color-coded split texts and split format text make a change of the participants’ recall, transfer, matching performance and their eye tracking data. The researchers explained that color-coded materials enhance the recall and transfer
98
B. Dogusoy and K. Cagiltay
performance. On the other hand, they found that there was no significant difference between participants in terms of the matching performance. The eye tracking data contributed the results with giving information on the participants’ eye fixations on color-coded materials which was longer than the other material. The researchers emphasized the existent literature on the fixation duration and its relation with cognitive processing. Also, they stated that the positive effects were related with the cognitive processing. They suggested that future research should be done on this issue to find our whether there are other reasons or not. Dogusoy and Cagiltay (2007) studied on another issue by using eye-tracking approach [16]. They pointed out the importance of finding information and search engines’ role in this process. In this study, the researchers aimed to examine the possible search behavior of the participants with the aid of eye-tracking approach. An unfamiliar search engine was chosen and nine participants used this search engine for completing four predetermined tasks. According to the results of this study, it was observed that the participants generally had problems during the searching process with the search engine. They tended to behave in a centralized manner while looking for the necessary information that the eye-tracking data supported this result. In addition to this, it was reported that the participants seem to use an “icon-based approach” that the eye tracking data showed that bigger and colored icons were much more focused on the screen than the other icons. They suggested that there needs to be more research on visual based search engines and their formations. Since the sampling is limited, the future studies might be conducted with wider and diverse sampling. They explained that these kinds of studies may give hints about the effective design processes especially the icon and keyword based structures and gathering data from users might help for constructing more effective visual search engines. Another research study was conducted on an interesting issue, multimodal comprehension of language and graphics graphs with and without annotation [17]. They explained their aim as to determine the function of “annotated textual elements on graphs in multimodal documents” [17]. At this point, eye tracking was used for gathering data on the participants’ eye movements and this may help them to have information on the visual attention of the participants. In this study, the sampling consists of thirty-two participants and they were taken verbal and spatial working memory span tests. The results of this study showed that the eye movement characteristics changed in accordance with the annotations on graphs. Also they stated that if the fixation duration data showed the participants’ processing difficulties, this was supported by the results of the study that the participants’ fixation duration is greater on the Graph AOI in graph-only format than the annotated format. They explained it with the possibility of the cognitive attempts of the participants in the process. Also, it was evidence that the participants’ gaze time on the Graph AOI was greater in the graph-only format. They emphasized that the findings of this study is on the same parallel with Split Attention effect [18]. They proposed the future works importance and introduced some of the possibilities. They explained that a detailed analysis of the Graph AOI is essential and with this, it might be possible to comprehend the annotated textual elements function in the process. Moreover, experimental studies might be replicated to have more observable results and lastly post-test might be included to the procedure for determining the individual differences and the connections between the features of the eye-movements of participants.
An Innovative Way of Understanding Learning Processes: Eye Tracking
99
2 Conclusion and Discussion The results of the eye-tracking based research studies under the METU HCI group were presented [5] in this paper. Since eye-tracking enables the researchers to study on different dimensions of human and cognitive schema, this may provide beneficial clues for both educators and researchers in order to understand the learning process in depth. The studies which were presented in this paper aimed to show less explored aspects of the educational applications and the attempts on using different techniques and strategies. Learning is a very complex process and it is related with the human beings’ complex thinking structure. The main issue which was pointed out in this paper was the design of better learning environments. Many papers focused on this issue and they tried to represent different design possibilities and their effectiveness. It can be seen that eye-tracking approach has a potential to provide information on learning processes and especially the general view on user’s behaviors. Although many studies resulted in with “no significant difference”, eye tracking approach has potential to provide information on the process which helps to represent a bigger picture for analyzing the process more easily. Also, new developments in technologies and more studies in this field, especially the analysis process, enable us to use this device more effectively and more accurately. At this point, eye-tacking approach may also give beneficial information on the process of understanding the nature of the cognitive processing. The studies generally focused on the data related with the fixation duration and place which gives information about the visual attention. Moreover, the gaze replay data is beneficial to the researchers to analyze the process deeply which enable them to observe the participants in the task completion process. Major limitation in eye-tracking studies is related with the small sampling. It is really hard for researchers to generalize the results with limited sampling. Also, in the literature the limitation of the eye-tracking studies were explained as they might give some answers about the users’ behavior and their ocular behavior but understanding the reasons under this behavior is not an easily understandable by researchers [19]. Video recordings which provide data on the participants’ facial expressions and verbal protocols of participants may help researchers in eye-tracking research studies. Although there are limitations related with this approach, the potential of these kinds of studies should not be ignored so that they may assist instructional designers and researchers in order to better understanding the learning process of human beings with predicating the deficiencies of the designs and how they can be strengthen. Acknowledgments. This study was supported by TUBITAK under grant SOBAG 104K098 and METU Human Computer Interaction research group (http://hci.metu. edu.tr).
References 1. Just, M.A., Carpenter, P.A.: Using eye fixations to study reading comprehension. In: Kieras, D.E., Just, M.A. (eds.) New methods in reading comprehension research, pp. 151– 182. Erlbaum, Hillsdale (1984)
100
B. Dogusoy and K. Cagiltay
2. Rayner, K.: Eye movements and cognitive processes in reading, visual search, and scene perception. In: Findlay, J.M., Walker, R., Kentridge, R.W. (eds.) Eye movement research: Mechanisms, processes, and applications, pp. 3–21. Elsevier Science, New York (1995) 3. Rayner, K.: Eye movements in reading and information processing: Twenty years of research. Psychological Bulletin 124, 372–422 (1998) 4. Just, M.A., Carpenter, P.A.: Psychological Review: A Theory of Reading: From Eye Fixations to Comprehension. American Psychological Association 87, 4 (1980) 5. Middle East Tehnical University (METU) Human Computer Interaction (HCI), METU HCI Research Group (2007), http://hci.metu.edu.tr 6. Clearview User Manual. Tobii, Sweden (2006) 7. Baran, B., Dogusoy, B., Cagiltay, K.: How do adults solve digital tangram problems? Analyzing cognitive strategies through eye tracking approach. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4552, pp. 555–563. Springer, Heidelberg (2007) 8. Yecan, E., Çagiltay, K.: Cognitive Styles and Students’ Interaction with and Instructional Web-Site:Tracing Users through Eye-Gaze. In: The 6th IEEE International Conference on Advanced Learning Technologies, Kerkrade, Netherlands, 5-7 July (2006) 9. Yecan, E., Sumuer, E., Baran, B., Cagiltay, K.: Tracing Users’ Behaviors in a Multimodal Instructional Material: An Eye-Tracking Study. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4552, pp. 755–762. Springer, Heidelberg (2007) 10. Sancar, H., Dogusoy, B., Cagiltay, K., Cakiroglu, J.: Exploring Cognitive Process of Concept Map Construction: An Eye-Tracking Study. In: Association for Educational Communications and Technology (AECT), Orlando, USA (2008) 11. Sancar, H., Karakus, T., Cagiltay, K.: Learning a New Game: Usability, Gender and Education. In: 3rd Technology-Enhanced Learning Enlargement Workshop: Young Researchers for the European Future of the Technology-Enhanced Learning, Sofia, Bulgaria (2007) 12. Karakus, T., Sancar, H., Cagiltay, K.: An Eye Tracking Study: The Effects of the Individual Differences on Navigation Patterns and Recall Performance on Hypertexts Environments. In: ED-MEDIA, World Conference on Educational Multimedia, Hypermedia & Telecommunications, Vienna, Austria (2008) 13. Ozcelik, E., Arslan, I., Cagiltay, K.: Signaling the Keywords or Sentences: An Eyetracking Study (unpublished manuscript) 14. Arslan, İ., Ozcelik, E., Cagiltay, K.: Effect of signaling on multimedia learning: An eye tracking approach (unpublished manuscript) 15. Ozcelik, E., Karakus, T., Kursun, E., Cagiltay, K.: Color Coding Effect on Multimedia Learning and Visual Search: An eyetracking approach. In: The American Educational Research Association (AERA), Chicago, USA (2007) 16. Dogusoy, B., Cagiltay, K.: Evaluation of Visually Categorized Search Engine. In: 3rd Technology-Enhanced Learning Enlargement Workshop: Young Researchers for the European Future of the Technology-Enhanced Learning, Sofia, Bulgaria (2007) 17. Acarturk, C., Habel, C., Cagiltay, K., Alacam, O.: Multimodal Comprehension of Language and Graphicsgraphs with and without Annotation. ECEM. In: 14th European Conference on Eye Movements, Potsdam, Germany, August 19-23 (2007) 18. Chandler, P., Sweller, J.: The split-attention effect as a factor in the design of instruction. British Journal of Educational Psychology 62, 233–246 (1992) 19. Pan, B., Hembrooke, H., Gay, G., Granka, L., Feusner, M., Newman, J.: The Determinants of Web Page Viewing Behavior: An Eye Tracking Study. In: Proc. ETRA (2004)
A Set of Rules and Strategies for UNSAM Virtual Campus Jorge Fernández Niello1,2,5, Francisco V. Cipolla Ficarra3,4,5, Mario Greco1,5, Rodolfo Fernández-Ziegler1,5, Silvia Bernatené1,5, and Maria Villarreal1,5 1
Universidad Nacional de San Martín Comisión Nacional de Energía Atómica 3 ALAIPO: Asociación Latina de Interacción Persona-Ordenador 4 AINCI: Asociación Internacional de la Comunicación Interactiva 5 Ayacucho 2197 (1650) San Martín Provincia de Buenos Aires - Argentina [email protected], [email protected], {mario.greco,rziegler,sbernatene}@unisam.com.ar 2
Abstract. We present a first set of strategies for the establishment of a virtual campus. Additionally, a set of pedagogic and communicative rules is established with the purpose of achieving a better diffusion of the university contents among the students. These rules are constantly updated according to the teaching supply and demand, and the requirements of the teachers and the students. Additionally the different components of interactive education are present as well as their main functions in the virtual campus. Keywords: E-learning, Campus Virtual, Pedagogy, Design, Hypermedia, Moodle, Software, Hardware.
campus. Consequently, before setting in motion these projects it is necessary to analyze each one of the cost variables to avoid unforeseen circumstances in the future. Obviously, some of the main goals of distance teaching are: to increase the quality of education, to widen the teaching offered and to raise the social level of the community to which the university belongs, among others. In the current work a state of the art of the software and hardware within E-learning education and the first strategies and rules to foster the use of the virtual campus is presented. Simultaneously, the interface of the current system is made public as well as a series of attributes of quality to boost communicability.
2 Software and Hardware The university costs in these projects entail the training of the teaching body in the use of these technologies, the promotion inside and outside the university through corporative publicity, to generate multimedia material, to dedicate space and personalized staff in the management and maintenance of the software-hardware, etc. With regard to the software the virtual campus manages, currently the costs are minimal as compared to the investments made by other American, Asian and European universities in the mid-nineties. The price reduction lies in what is known as open software [7], [8]. Thanks to this there are systems such as Moodle (Modular Object-Oriented Dynamic Learning Environment) aimed at E-learning. Concerning this there are colleges that have implemented this software with excellent results among students and teachers. This saving in the budget can be aimed at the realization of multimedia teaching materials of excellent quality and which guarantee the constant updating throughout time. When it comes to hardware, in our case, one might have two mirror servers. that is to say, that in the case that one fails, the second is automatically activated, using Linux as an operating system, thus avoiding additional costs through the yearly renewal of licenses in software material as it happens with other operative systems. Both servers should have firewalls available. Two PCs with the same configuration may be a transitory solution in the case that one of them breaks down. All these computer devices and network must have available a continuous energy supply in the face of potential electrical blackouts and in rooms that comply with the quality rules for the correct functioning of the computer system, that is to say, air conditioners. We see how the costs in computer material have also gone down as compared with the realities of the past decade. In spite of this, there are still European universities, for instance, that continue using servers for the internal management of the university with the campus or virtual classroom, with very high maintenance and great updating costs with regard to server hardware components [9]. In our case, we have available a magnificent system for SIU-Guaraní academic management, with the possibility of making excellent detailed statistic graphics in real time. However, no aspect related to classrooms or virtual campuses is developed yet. Consequently, we have started to project the main strategies to reach that target in a short term. The SIU-Guaraní is a system of college information that administers academic management since the students enroll in college in a standardized way and profiting
A Set of Rules and Strategies for UNSAM Virtual Campus
103
Fig. 1. SIU-Guaraní system –database interface for academic gestion
data to the maximum. It has a single federal database, that is, it allows information to be added and for it to be reviewed in a decentralized way. Some of its main functionalities are planning, fees management, classroom management, exam management, administration and expedition of certificates, management of pupils polls, diffusion of memos and communiqués through e-mails or SMS (students, teachers, and academic authorities), management of several interfaces: follow-up of the students who have finished their studies, accountancy management, etc. and even has an interface with Moodle, but which is in the development stage and will be aimed at E-learning. Access is through the university intranet and internet. It fosters cooperative and cooperation work, starting from the interaction among the different sectors and actors (authorities/student staff/teachers/students) [10]. Additionally, here also are to be found accessibility bases; since it is an on-line system it makes possible its availability around the clock, 365 days of the year. From the point of view of software engineering it has the main attributes of quality, that is to say, privacy of information, reliability, flexibility, adaptability, etc. [11], [12], [13], [14]. Some of the operations the students can do are: examinations inscription, reviewing the study plan and academic record, seeing the chronology of the partial assessments, class agenda, requesting certificates, updating of personal data, reception of files and messages, etc. On their side, professors can look up the students who have registered to their examinations, to register the exam grades in the records, to consult the schedule of the exams, etc. In short, the SIU-Guaraní system is a very interesting tool of control over academic management and which makes easier to some extent some of the activities that the tutors/teachers have to carry out and the future students in the virtual classrooms.
104
J. Fernández Niello et al.
3 Human Factors In common with other public or private entities with a high number of people who interact among themselves, the human factor is essential to set in motion the projects, in order to avoid resistance to it. Within the context of the new or latest technologies in the education sector there are those whom Umberto Eco would call apocalyptic or integrated [15], that is to say, you are either in favour of or against their use. In the metropolis of the economically developed societies these problems virtually do not exist. It is a population that constantly includes new technologies in their daily life activities [16], [17]. In other places it is necessary to boost the motivation among its participants, always trying to establish a horizontal communication, among peers, for which it is necessary to count on accurate rules to be followed by the virtual community that is generated with e-learning, and so that the democratization phenomena of distance teaching may reach each one of the most distant places in a national and international territory [17], [18] since we live in Mc Luhan's ‘global village’ [20]. It is necessary to remember that we have overcome the era of computer use and we are going through the quality era of communicability [21]. It is also necessary to remember that many students do not finish their mid-level studies, therefore, a campus or virtual classroom can motivate them for such a purpose, resorting to the public institutions. There are countries in Southern Europe where these studies can be finished in two years resulting in some cases even to obtaining a masters degree. Oddly enough, these private teaching centres guarantee the finalization of studies in a 100% of cases. Obviously, this is a reality that belongs more to the commercial sector than to education. Now through social programs and having available sufficient economic and financial means, the causes of school-quitting might be traced, and thus put at the disposal of the potential students a whole range of pedagogical and technological resources to end the studies and later on, if they want to, they might follow their university studies. The idea here is to introduce to the students a quick way to study and even if they are working, they have the alternative of finishing the studies for which they have registered. In other words, a well-structured and systematically functional virtual campus can speed up the timing in the process of teaching and learning. Simultaneously, a virtual classroom may overcome the problems originated by the workload complaints of the university teaching and non-teaching staff and which usually end up in some places in long strikes. The students who have the virtual material can follow their exercises, see the exam schedule, see the grades, etc. It is a way of not damaging the students. A way to overcome the difficulties that the students usually have at the moment of having access for the first time to the virtual environments is through the implementation of a call center. The call center has to be active 24 hours per day, along all the weeks, including holidays. Perhaps a good strategy to overcome the human factors barriers in the first moments for the use of the instruments of the virtual classroom is to explain their main advantages, both for the teachers and the students, through a corporative or institutional information campaign, activation of an exclusive website with multimedial demos which explain the functioning of each one of the main
A Set of Rules and Strategies for UNSAM Virtual Campus
105
components of the virtual campus, training of tutoring students for the virtual community, etc. In the annex #1 a list with the main components. All these lines of action have to be coordinated and follow the common goal of the democratization of the use of the virtual campus, inside and outside the college facilities [3], [22], [23], [24], [25].
4 Campus Virtual UNSAM: First Step Next a series of active interfaces that have allowed navigation, and the first experiences with the main professors involved in the project: One of the solutions to set in motion projects of this stature consists in external advice through other university realities (in this case, PUC of Valparaíso). However, we must remember that it is feasible for us to insert a foreign model 100% in the university context since the sociocultural variables such as the mentality of the peoples changes from one state to another. However, the experiences of the bordering countries will always be more positive and adaptable to home reality than those coming from overseas. Without any doubt,
Fig. 2. UNSAM Virtual Campus: A simple iconography and minimal style desing of the interfaces. Two email options: intern. between students and professors/tutors (adove) and extern between the users and system administrator of virtual campus (down).
106
J. Fernández Niello et al.
Fig. 3. A ‘classic’ chat system and menu of the colors for texts. It is a public area for all users with autorization –virtual campus system aministrator.
Fig. 4. Marks and person responsable for evaluation, additional information, observations and details. In the right zone a list with all active virtual classrooms. Neat data and content disposition.
A Set of Rules and Strategies for UNSAM Virtual Campus
107
Fig. 5. Notes, summaries, practical homework, etc. option. Eassiness for attach files, add contents description and/or remarks, and selection of the categorie.
the current virtual campus contains activities that are currently carried out in the virtual environments aimed at E-learning. These are instruments which are not complicated to use, but which necessitate time for its learning by both tutors and students. The presented interfaces have made it clear that they have a very simple design, easy to use and not ambiguous [2], [26], [27]. The communicability for an expert user is optimal, however it is necessary to think of those users who have little experience in the use of computers. The activation and disactivation of the static and dynamic means, as well as their functioning, are similar to the main components of the Windows operating system [28].
5 Conclusions Establishing a virtual campus entails following some general and secondary targets akin to the former, where achieving the common good prevails among each one of its
108
J. Fernández Niello et al.
members whether it is as course organizers, multimedia content generators, Elearning tutors, active members of the virtual community, etc. The rules to be followed must be determined in a protocol of principles for the whole university community. These rules must respect usability and communicability of the interactive contents aimed at E-learning. A priori, the set of rules and strategies may seem to the final user of the system as a kind of normal pyramid, that is, at the top are the academic authorities, the professors, tutors, etc. However, it is not so, because it is precisely in the opposite sense. The switch of the pyramid is achieved thanks to the virtual community that is born from the E-learning process. That is to say, with the passing of time and through the use of the instruments provided by the new technologies of the virtual campus, we are in front of an inverted pyramid where we find the students at the summit. Besides, the use and circulation of free software make it possible to speed up this process. Axiomatically and undisputedly, it is necessary at the beginning to count with the greatest possible amount of available information, even from similar experiences in other universities and the cooperation of some outsourcing services. However, the final goal must be to transfer even these outer services inside the university context. In our case, we have available an efficient federal academic management system, but for the moment it lacks a virtual campus. Consequently, the set of rules and strategies presented are not static but rather dynamic, such as E-learning communication and education process. Finally, the costs concerning software and hardware have considerably lowered nowadays as compared to the past decade to set in motion a project of virtual campus. Acknowledgments. The authors wish to acknowledge all colleagues in our Universidad Nacional de San Martín and PUC of Valparaíso for the support and their helps.
References 1. Saussure, F.: Curso de Lingüística General. Losada, Buenos Aires (1982) 2. Cipolla-Ficarra, F.: Evaluation and communication techniques in multimedia product design for on the net university education. In: Multimedia on the Net, Vienna, pp. 151– 165. Springer, Heidelberg (1996) 3. Paymal, N.: Pedagogia 3000. Editorial Brujas, Cordoba (2008) 4. Eletti, V.: Che cos’è L’E-learning. Carocci, Roma (2002) 5. La Noce, F.: E-learning. Franco Angeli, Milano (2001) 6. Harasim, L., et al.: Learning Networks. MIT Press, Cambridge (1998) 7. O’Reilly, T.: Lessons from Open-Source Software Development. Communications of the ACM 42, 32–37 (1999) 8. Schlesinger, D.: Working with Open Source: A Practical Guide. Interactions 14, 35–37 (2007) 9. Lamborghini, B. et al.: European Information Technology Observatory. FrancoAngeli, Milano (2005) 10. Mamtyla, K.: Blending E-Learning: the Power is in the Mix. McGraw Hill, New York (2001) 11. Ternoven, I.: Support for Quality-Based Design and Inspection. IEEE Software 13, 44–54 (1996)
A Set of Rules and Strategies for UNSAM Virtual Campus
109
12. Frakes, W., Isoda, S.: Success Factors of Systematic Reuse. IEEE Software 11, 14–19 (1994) 13. Ozkaya, I., et al.: Making Practical Use of Quality Attribute Information. IEEE Software 25, 25–33 (2008) 14. Safayeni, F., et al.: Requirements Engineering in New Product Development. Communications of the ACM 51, 77–82 (2008) 15. Eco, U.: Apocalittici e integrati. Bompiani, Milano (2001) 16. Beverly, A.: Instructional and Cognitive Impacts of Web-based Education. Idea Group Publishing, Hershey (2000) 17. Toyama, K., Dias, B.: Information and Communication Technologies for Development. IEEE Computer 41, 22–25 (2008) 18. Ramakrishnan, R., Tomkins, A.: Toward a PeopleWeb. IEEE Computer 40, 63–72 (2007) 19. Balasubramaniam, S., et al.: Situated Software: Concepts, Motivation, Technology, and the Future. IEEE Software 25, 50–55 (2008) 20. McLuhan, M., Power, B.: The Global Village: Transformations in World Life and Media in the 21st Century. Oxford University Press, Oxford (1992) 21. Cipolla-Ficarra, F.: Communicability design and evaluation in cultural and ecological multimedia systems. In: Proc. 1st workshop Communicability MS 2008, pp. 1–8. ACM Press, New York (2008) 22. Trentin, G.: Insegnare e apprendere in rete. Zanichelli, Bologna (1998) 23. Gatti, R., Gherardi, V.: Le scienze dell’educazione. Carocci, Roma (1999) 24. Norman, D., Spohrer, J.: Learner-centered Education. Communications of the ACM 39, 24–27 (1996) 25. Dziuban, C., Moskal, P.: Evaluating Distributed Learning in Metropolitan Universities. Metropolitan Universities 1, 41–49 (2001) 26. Nielsen, J.: The Usability Engineering Life Cycle. IEEE Computer 2, 12–22 (1992) 27. Sharp, H., Rogers, Y., Preece, J.: Interaction Design: Beyond Human-Computer Interaction. John Wiley, West Sussex (2007) 28. Shneiderman, B.: Designing the User Interface. Addison Wesley, Massachusetts (1998)
110
J. Fernández Niello et al.
Annex #1: First Set of the Components for a Virtual Campus 1) Hypermedia On-line and Off-line –Design Categories − − − − −
Content Navigation Panchronism Presentation or layout; Structure
2) Quality Attributes − Acteme is the study of communication as behavioural system. − Adaptability of the content, that is, variety of styles of navigation, plenty of different media (text, graphics, maps, animations, videos, music and sounds), a wide control over active media. − Behaviour-animated help analyses the attributes universality, simplicity, originality and humour in animated pedagogical agents. − Competence considers how the systems adapts itself to skills of the main groups of users for navigation. − Isomorphism sets down a range of constant formal features among the different components and design categories, i.e., content, layout, structure, navigation, etc., that is, it researches those components that remain equal in the different categories and especially the topology of the elements’ distribution on the frame. Isomorphism seeks ‘regularity in irregularity’. − Motivation is the set of the dynamic means and structure resources that strengthen navigation in multimedia systems. − Semiosis for universal access makes reference to the main components that converge in this attribute for systems multimedia: linguistic, iconic, attention, narration and education/formation. − The control of fruition is the degree of autonomy in navigation that the structure of the multimedia system gives the user. − The naturalness of metaphor assesses the user’s ability to understand the set of images that make up the structure of the interface. − The phatic function asserts the direct communication in the human-computer interaction process without generating mistakes. − The transparency of meaning analyses the usage mainly of terms (also, images and sounds related to the words) of the interface that do not cause ambiguities between content and expression. − The triple dynamic coherence analyses the relationship between text, image and audio in the dynamic media, nevertheless, this attribute does not consider the synchronization between audio and dynamic image (animation and video). − Unlimited iconic consists of those icons and their components that, depending on the national and/or international cultural setting, have different meaning, for example, the gestures, the colours, the words, etc. − Acteme is the study of communication as behavioural system.
3) Experts − − − − −
Communicability Human-Computer Interaction Human-Computer Interface Pedagogy Software (analysis and programming) and Hardware (systems)
4) On-line Learning –Main Elements − − − − − −
Call center: info and problem resolution support Interactive communication: Email, chat, netmeeting, etc. Portals (website –Internet and intranet) Teaching materials: book style and support (digital and/or analogic), production, publishing, etc. Tutors: virtual or real Virtual community : integrants and administrators
HCI Professional Involvement in k-12 Education: On Target or Missing the Mark? Martin Jelin, Adrian Sudol, Jeffrey Damon, Douglas McCadden, and David Brown Computer Science Department, Maine School of Science and Mathematics, 95 High Street, Limestone ME, 04750, United States {sjelin,asudol,jpdamon,mccaddend,dr.david.brown}@gmail.org
Abstract. The state of learning across geographic, socioeconomic, age, and gender boundaries can be enhanced greatly by Human-Computer Information (HCI) infusion into blended learning [1][2][3] or Course Management System (CMS) software [4][3][5]. The major thrust of this paper is to examine problematic issues examined in popular software such as Moodle™ in which the HCI community could be beneficial. By regarding the ultimate students’ goal, i.e. grades, and the desirable benefit of course material understanding, one can develop an understanding of what CMS software needs and CMS software users expect on the high school level. Keywords: blended learning, e-learning, Course Management System (CMS), Virtual Leaning Environment (VLE), Human-Computer Interaction (HCI), k-12 Education, magnet schools.
1.1 Blended Learning and Course Management Systems (CMS) “Blended learning” is a method where students are being taught by two or more sources of information, generally traditional classrooms and online classrooms [10]. Often times, blended learning is implemented by using a Course Management System (CMS). CMS, also known as Virtual Learning Environments (VLE), are online tools for teachers to interact with their students outside of the physical classroom [8]. Interaction may occur in a variety of educational modalities.
2 Review of Literature Blended learning and CMS software can improve learning for all students [1][2][3][4][5]. The use of blended learning has been increasing quickly over the past few years, both in school and businesses [7][11][12]. Many college campuses have realized this and moved to CMS environments to streamline and augment organization, reporting, teaching and learning [1][9][13][14]. These programs have been generally well received, with both students and teachers finding the programs beneficial [6][9][13][14]. These institutions are funded well enough to absorb the costs of retail offerings such as First Class™, Blackboard™, and WebCT™ (now owned by Blackboard™), as well as technology training programs. High school funding does not allow this luxury. Moodle™ is an open-source CMS commonly used in high schools that cannot afford more expensive programs [8]. The software vehicle establishes the platform and potential groundswell of increased activity with HCI and the high school communities [8]. Moodle™, educational communities are international in scope with as many diverse uses as cultural approaches to education [2][3][15]. As a vehicle, Moodle™ has tools such as blogs, wikis, online tests, instruction, email, and asynchronous collaboration [8]. It's flexibility has caused it to outshine many other open source CMS programs, and even Blackboard™ in some cases [16][17]. Moodle™ is well known for focusing on student interaction, often considered the main advantage of blended learning. The educational philosophy that advocates this is known as social constructivism [8][18]. According to Berger and Luckmann [22], this theory posits knowledge is actually created by students and teachers through interaction with themselves and their environment. They contend that social constructivism is important to learning [22]. However, despite this focus on interaction, there is sparse research on the actual content of online student interaction [19]. A study of a blended learning class at Blackpool and The Fylde College found that 77% of students in the class preferred a normal learning environment, and more reflective students tended to avoid the online portion of the class as much as possible [9]. Martin Dougiamas, the creator of Moodle™, did a similar study in which he found Moodle™ did not even allow social learners to properly interact with each other [18]. Hetlighen and Anderson surmise that information overload is another large problem facing CMS users. This occurs when the students are given too much information absorbing little [20][21].
HCI Professional Involvement in k-12 Education: On Target or Missing the Mark?
113
A 2000 study by The US National Center for Education Statistics found that of k12 teachers who used the Internet, only 39% used it for instructional material, and just 16% used it for gathering information to plan class lessons [6].
3 Methodology The sample, drawn from a magnet school population in the Northeastern United States, contains grades 10-12. Socio-economic statuses range from multimillionaire families to wards of the state. Gender distribution is approximately 66% male and 34% female in a population of 114 students. Diversity of the population results in reduced bias. It should be noted that all students at the school have Internet 75% of the day. Two computer labs are available for a required, introductory technology class (32 students). Increased exposure to technology and high academic rigor could bias the survey responses toward CMS software and blended learning. The school uses the open-source CMS Moodle™ [8]. Approximately two thirds of the population used the CMS for at least one of their classes at the time of the data collection. The students were given an anonymous survey on their perception of Moodle’s™ influence. This volunteer survey resulted in 63% response. NonMoodle™ users’ responses were excised. The survey consisted of two parts. In the first part, students gave information about their general experience with Moodle™. In the second, students gave information about their experience specifically within their class which used Moodle™ the most. 3.1 Semantic Differential Students used the following semantic differential to rate effects: Table 1. Semantic Differential Choice Large Negative Effect Medium Negative Effect Small Negative Effect Small Positive Effect Medium Positive Effect Large Positive Effect Not Applicable or Not Used
Acronym LN MN SN SP MP LP N/A
4 Results Students were first asked to give information about their experience with Moodle™ in general. The students were asked to furnish data concerning their experiences with Moodle™, specifically within the class that used the CMS the most.
114
M. Jelin et al. Table 2. Moodle™ General Results Question What is your gender? What grade are you in?
Results Male – 26 Sophomore – 11 Senior – 16
How many classes are you taking this semester? About how often do you log onto Moodle™?
Four – 15 Five – 17 Seven – 4 Eight – 1 Never – 0 1 to 3 Days Per Month – 7 1 to 3 Days Per Week – 23 4 to 6 Days Per Week – 8 Daily – 10
About how often do you have difficulties when using Moodle™? Overall, what effect has Moodle™ had on your grades?
Never –13 Rare – 20 Mildly Often – 6 Very Often – 9 LN – 0 MN – 4 SN – 8 SP – 26 MP – 9 LP – 1
Overall, what effect has Moodle™ had on your understanding of the subjects of your classes?
LN – 1 SP – 25
Female – 22 Junior – 21
MN – 2 MP – 11
Four – 15 Seven – 4
SN – 7 LP – 0
Table 3. Moodle™ Intensive Class Results Question
Approximately how many days a week does your teacher required you to log onto Moodle™ for this class? Approximately how many days a week are you advised to log onto Moodle™ for this class? Approximately how many days a week do you log onto Moodle™ for this class? Rate the effect Moodle™ has had on your grade in the class.
Results Zero – 19 Three – 5 Six – 1
One – 8 Four – 3 Seven – 2
Two – 5 Five – 5
Zero – 6 Three – 4 Six – 2
One – 7 Four – 1 Seven – 14
Two – 6 Five – 8
Zero– 6 Three – 4 Six– 2
One – 11 Four – 7 Seven – 7
Two– 8 Five – 3
MN – 3 MP – 9
SN – 8 LP – 3
MN – 2 MP – 7
SN – 5 LP – 2
LN – 1 SP – 5 N/A – 23
MN – 1 MP – 7
SN – 6 LP – 5
LN – 1 SP – 10 N/A – 23
MN – 0 MP – 5
SN – 7 LP – 2
LN – 0 SP – 19 N/A – 6 Rate the effect Moodle™ has LN – 1 had on your understanding of SP – 23 N/A – 6 the class subject.
Rate the effect Moodle's™ Assignment Page has had on your grade in the class. Rate the effect Moodle's™ Assignment Page has had on your understanding of the class subject.
HCI Professional Involvement in k-12 Education: On Target or Missing the Mark?
Rate the effect Moodle's™ Weekly/Topic Outline has had on your grade in the class. Rate the effect Moodle's™ Weekly/Topic Outline has had on your understanding of the class subject. Rate the effect Moodle's™ Discussion Board (forums) has had on your grade in the class. Rate the effect Moodle's™ Discussion Board (forums) has had on your understanding of the class subject.
LN – 0 SP – 12 N/A – 19
MN – 1 MP – 9
SN – 4 LP – 3
LN – 0 SP – 15 N/A – 19
MN – 3 MP – 7
SN – 4 LP – 0
LN – 2 SP – 13 N/A – 23
MN – 0 MP – 4
SN – 5 LP – 1
LN – 2 SP – 11 N/A – 23
MN – 1 MP – 5
SN – 3 LP – 3
115
Results of “N/A” on the last few questions indicate that the student responding did not use that particular feature of Moodle™ in his or her class which used the CMS most. 4.1 Graphs Only data relevant to the conclusion were graphed.
25 20 15 10 5 0 Never
Rare
Mildly Often
Very Often
Fig. 1. Frequency of Difficulty
30 25 20 15 10 5 0
E ffect o n G rade E ffect o n Understanding
LN
MN
SN
SP
MP
Fig. 2. Overall Effect of Moodle™
LP
116
M. Jelin et al.
20 O v e ra ll Ef f e c t
15
A s s ignm e nt P a ge ' s E f f e c t
10
We e k ly O ut line ' s E f f e c t
5 0 LN
MN
SN
SP
MP
LP
D is c us s io n B o a rd ' s E f f e c t
Fig. 3. Moodle’s™ Effects on Grade by Components 25 20
O v e ra ll E f f e c t
15
A s s ignme nt P a ge 's E f f e c t
10
We e k ly O ut line 's E f f e c t
5 0 LN
MN
SN
SP
MP
LP
D is c us s io n B o a rd's E f f e c t
Fig. 4. Moodle’s™ Effects on Understanding by Components
4.2 Significant Differences Using the Kruskal-Wallace test, a nonparametric that compensated for unequal cell sizes, the following significant differences were obtained: Table 4. Significant Differences Independent Gender Frequency of Difficulty Frequency of Difficulty Frequency of Difficulty Frequency of Difficulty Frequency of Difficulty Class Subject Class Subject
Dependent Frequency of Difficulty Effect of CMS on Class Grade Effect of CMS on Subject Understanding Effect of CMS' Assignment Page on Subject Understanding Effect of CMS' Weekly Outline on Class Grade Effect of CMS' Weekly Outline on Subject Understanding Effect of CMS on Class Grade Effect of CMS on Subject Understanding
HCI Professional Involvement in k-12 Education: On Target or Missing the Mark?
117
5 Conclusion Seventy-three percent of the sample perceived problems with the CMS. The significant differences found indicate the importance of enhancing ease of use, ease of navigation, clarity of command and menu structures, as well as other factors of HCI. Poole’s study found no students having difficulties with their CMS [9]. This contrasts starkly with the 73% of students at the secondary school. The large difference demonstrates that studies on higher education’s use of blended learning cannot be generalized to the high school level. Of approximately 40 articles reviewed for this paper, only four scholarly sources could be found which focused on k-12 education. Current research on blended learning does not address the needs of secondary schools or the HCI components that may need reconstruction or clarification. A new body of research is needed for high schools. The significant difference between the genders when crossed with perceived difficulties gives a direction to improve CMS in high schools. Both psychological and sociological HCI concerns should be addressed. Historically, young females have addressed computers as antagonists, with some trepidation [23]. The current generation of students is facing a greater risk of information overload [21]. If the CMS adds to the learning curve, rather than simplifying the experience, the learning process becomes obfuscated. Therefore, a CMS needs to be as simple relative to HCI standards as possible. In follow-up conversations with the student body, students did not notice or take advantage of help pages. In fact, 80.4% of the sample was unable to locate and access the help modules. The help pages were perceived as confusing and detracting from the subject matter. Although Moodle™ is used throughout the world due to its opensource nature; it is believed that HCI professional community can contribute to its efficacy by improving HCI concerns. Despite the problems facing blended learning, the literature signifies great potential. Previous studies conducted on blended learning on the college level have shown that college students appreciate the process of blended learning [8][9][13][18]. High school students in this study believed that blended learning helped increase their knowledge acquisition (See Figure 2). As such, blended learning as an idea is not flawed, but needs more data based underpinnings. Students found the broader aspects of the information delivered by the CMS to be more relevant to both grades and understanding than those components that fostered depth. This may also lead to the supposition that a better navigational experience would have enhanced the students’ use of the blended learning tools (see Figure 3 and Figure 4).
Acknowledgements The authors would like to thank Gary Parker for his initial contributions, the student body for their participation in surveys and questionnaires, the faculty who responded to surveys and interviews, and an administration that facilitated the financial aspects of delivering this paper.
118
M. Jelin et al.
References 1. Garrison, D.R., Vaughan, N.D.: Blended Learning in Higher Education: Framework, Principles, and Guidelines. Jossey-Bass, Hoboken (2007) 2. Hargadon, S.: Implementation Study #3: Moodle. K-12 Open Technologies (2008), http://www.k12opentech.org/implementation-study-3-moodle 3. U.S. Department of Education, http://www.ed.gov 4. Voogt, J.M., Knezek, G.A.: International Handbook of Information Technology in Primary and Secondary Education. Springer, Heidelberg (2008) 5. Awidi, I.: Critical Factors in Selecting a Course Management System for Higher Education in Ghana. Educause Quaterly (2008), http://connect.educause.edu/Library/EDUCAUSE+Quarterly/ CriticalFactorsinSelectin/46028 6. National Center of Education Statistics, http://nces.ed.gov 7. Chapman, G.: Federal Support for Technology in K-12 Education. Brookings Papers on Education Policy, 307–343 (2000) 8. Moodle.org, http://www.moodle.org 9. Poole, J.: E-learning and learning styles: students’ reactions to web-based Language and Style at Blackpool and The Fylde College. Language and Literature 15(3), 307–320 (2006) 10. Voos, R.: Blended Learning: What it is and Where it Might Take Us? Sloan-C View 2(1) 11. Mullins, J.: E-Learning in the Inspector General Community: The Federal Government Is Using E-Learning to Raise Inspector General Skill Levels, Reduce Training Costs, and Increase Training Hours and Training Opportunities-Strengthening the Community’s Core Competencies Overall. The Public Manager 35(1), 46–50 (2006) 12. Carlivati, A.P.: E-learning evolve. ABA Banking Journal 94(6), 49–56 (2002) 13. Crisp, P.: E-learning and Language and Style in Hong Kong. Language and Literature 15(3), 277–290 (2006) 14. Plummer, P., Busse, B.: E-learning and Language and Style in Mainz and Münster. Language and Literature 15(3), 257–276 (2006) 15. Computers in the Classroom: Uses, Abuses, and Political Realities. EMCParadigm Publishing, http://www.emcp.com/intro_pc/reading1.html 16. Graf, S., List, B.: An Evaluation of Open Source E-Learning Platforms Stressing Adaptation Issues. In: Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies, pp. 163–165 (2005) 17. Munoz, K., Duzer, J.: Blackboard vs. Moodle: A Comparison of Satisfaction with Online Teaching and Learning Tools. Humboldt State University (2005), http://www.humboldt.edu/~jdv1/moodle/all.htm 18. Dougiamas, M., Taylor, P.C.: Interpretive analysis of an internet-based course constructed using a new courseware tool called Moodle. In: HERDSA 2002 Conference (2002) 19. Ziegler, M., Paulus, T., Woodside, M.: Creating a Climate of Engagement in a Blended Learning Environment. Journal of Interactive Learning Research 17(3), 295–308 (2006) 20. Hetlighen, F.: Change and Information Overload: negative effects. Principia Cybernetica Web (1999), http://pespmc1.vub.ac.be/chinneg.html 21. Anderson, N.: 2008: Year of Information Overload? Ars Techina (2007), http://arstechnica.com/news.ars/post/20071226-interruptionsinfo-overload-cost-us-economy-650-billion.html 22. Barger, P.L., Luckmann, T.: The Social Construction of Reality: A Treatise in the Sociology of Knowledge. Anchor Books (1966) 23. Beckwith, L., Burnett, M., Grigoreanu, V., Wiedenbeck, S.: Gender HCI: What About the Software? Computer (2006)
A Language Learning System Utilizing RFID Technology for Total Physical Response Activities Harumi Kashiwagi, Yan Xue, Yi Sun, Min Kang, and Kazuhiro Ohtsuki Kobe University, 1-2-1 Tsurukabuto, Nada, Kobe, 6578501, Japan {kasiwagi,kang,ohtsuki}@kobe-u.ac.jp, [email protected], [email protected]
Abstract. In this paper, we present a method of integrating a CALL system with RFID tags in a classroom in order to provide the basic support for listening activities based on the concept of Total Physical Response (TPR) Approach. We designed and developed a prototype system with the function of providing corrective feedback. The prototype system has the following three features: (1) Real objects are used as the options for responding to audio questions. (2) A considerable amount of attribute information is used to increase the variation in the questions. (3) The system has a function of providing error messages and additional questions, depending on the degree of a learner’s mistakes. Results from the experiment suggest that integrating real objects into the learning system by using RFID tags has a potential impact on the language learners. Keywords: RFID, Language learning, TPR, Corrective feedback, Interaction.
used as the options for responding to audio questions. The respective object is linked to its attribute information. (2) A considerable amount of attribute information is also used to increase the variation in the questions. Thus, multiple target listening points are included in one question. (3) The system has a function that provides not only appropriate error messages depending on the learner’s incorrect responses but also provides additional questions as supplementary exercises, depending on the degree of the learner’s mistakes. The remainder of the paper is organized as follows. Chapter2 describes the basic concept of the system. In chapter 3, the outline and features of the prototype system have been discussed. Chapter 4 presents the implementation of the system, and chapter 5 explains the experiment and its results. Finally, the conclusions as well as future suggestions are presented in chapter 6.
2 System Concept In this chapter, the basic concept of the system has been described. This includes the concept of the TPR approach, the manner in which the questions and answer options should be arranged so as to cope with the variation in and difficulty level of the questions, and the manner in which feedback messages should be prepared in response to the learner’s mistakes. 2.1 The TPR Approach The TPR approach is a method in which teachers first provide a command in the target language, and then perform the corresponding action with the students. In this manner, the students learn not only by observing the actions but also by performing them[7]. In lessons that employ the TPR approach, teachers should avoid introducing new commands in quick succession. It is generally recommended that they present three commands at a time. As the students increase their knowledge of the target language, a longer series of connected commands can be provided. In this method, the actions clarify the meaning of the commands. Moreover, memory is also activated by connecting the actions with the language. From the teachers’ perspective, they will be able to immediately determine whether or not the students understand the command by observing their actions. The prototype system designed and developed in this study is based on this method. 2.2 Question and Feedback Structure Varying the Question Sentence Patterns and Controlling the Difficulty Level of the Questions. When creating listening exercises, it is necessary to consider the variation in the question sentence patterns and the difficulty level or complexity of the questions. The latter is related to certain aspect of language, i.e., vocabulary and phrases, grammatical structures, and length of sentences. Thus, a listening sentence that includes many new words and high-level grammatical structures will be difficult to comprehend. In addition to vocabulary and grammatical complexity, the difficulty level of a question will also be affected by the length of the sentence[10]. Therefore, vocabulary, grammatical structures, and length of sentences are the determinants of
A Language Learning System Utilizing RFID Technology for TPR Activities
121
the difficulty level of questions. However, the following can be employed to reduce the difficulty level. 1. Give specific examples: When using terms such as “carbonated drinks” in a question, provide specific examples such as “Coke” or “Sprite.” 2. Give external characteristics: When the word “glue” is mentioned in a question, provide supplementary information about its external characteristics such as "It has the shape of stick and has a blue cap." 3. Simplify the text: Consider the following sentence, "Please give me cold lemon juice in a plastic bottle." This text can be simplified by breaking it up into two sentences, such as "I want a cold drink. A lemon juice is good, and a plastic bottle would be better." Thus, providing examples, giving supplementary information, and breaking up the questions into manageable parts is helpful for the learners. Controlling the Difficulty Level of the Questions by Varying the Response Options. The second factor to be considered is the response options. Typically, teachers instruct mixed-level learners, and hence, by observing the level of each learner, they need to keep changing the difficulty level of the questions. However, we can also control the difficulty level of the questions by varying the response options to the questions. For example, when a question such as "Please choose cold lemon juice which costs ¥130" is presented, the learner will attempt to listen selectively for specific information such as cold, lemon juice, and ¥130. If the learner is required to choose a response from the following four options, he/she will have to listen to the question and identify all the key words: (1) cold, lemon juice, and ¥130; (2) cold, lemon juice, and ¥150; (3) cold, apple juice, and ¥130; (4)hot, lemon juice, and ¥130. However, if the learner is given the following options, he/she will be correct even if only one key word is correctly identified: (1) cold, lemon juice, and ¥130; (2) hot, orange juice, and ¥200; (3) cold, banana juice, and ¥150; (4) cold, apple juice, and ¥150. In this manner, the difficulty level of questions can be controlled, depending on the learner’s understanding, by merely changing the response options. This method would be helpful particularly when teaching lower-level learners. Thus, when developing our system, we introduced the above-mentioned concept of changing the objects that are used in the response options. Corrective Feedback. In general, teachers react to the mistakes made by learners in many different ways. One such manner is by providing feedback; it helps the learners in altering their outputs in a constructive manner. The typical types of corrective reactions have been described in Chaudron's research[4,5] in the following manner: _Emphasis: Using repetition, or questions, to identify the aspect that is incorrect _Explanation: Providing information on the different causes of the error _Original question: Repeating the original question _Altered question: Altering the original question syntactically, but not semantically _Verification: Assuring that the correction has been understood
122
H. Kashiwagi et al.
In addition, the following can be regarded as the other types of corrective reactions: _Selecting & reviewing: Selecting & reviewing important points in order to guide the learners toward the right direction _Breaking down: Breaking down the learning items into smaller parts/steps to ease understanding Based on the above-mentioned works, we considered the function of providing feedback in the prototype system.
3 Prototype System 3.1 Flow of Instructions Fig. 1 illustrates the prototype system developed in this study. When a learner scans a question card, the system reads out the corresponding message, e.g., “Please choose cold lemon juice in a plastic bottle” (Fig. 2).
Fig. 1. Prototype system
Fig. 2. Method to use the system
The flow of instructions has been presented in Fig. 3. When the learner responds to the question by choosing an appropriate object and allowing the scanner to scan it, the system will check his/her answer. For example, if the learner chooses a cold, orange juice can, the system will detect an error in the following two attributes: kind and container type. Thereafter, it will detect the degree of the error, i.e., the data pertaining to two attributes among the three targeted attributes are wrong. These errors are represented as follows: cold = cold, lemon ≠ orange, and plastic bottle ≠ can. Then, the system will read out an error message, e.g., “No, it is not lemon juice. It is not in a plastic bottle,” followed by an additional question regarding the wrong response, e.g., “Please choose lemon juice.” If the learner provides the correct response in this case, the system would state, “OK, good,” and then read out the second additional question for another wrong response, i.e., “Please choose a plastic bottle.” If the learner correctly responds to all the additional questions, the system would again read out the initial question, i.e., “Please choose cold lemon juice in a plastic bottle.” At this instance, when the learner responds correctly, the system will move on to the next question.
A Language Learning g System Utilizing RFID Technology for TPR Activities
123
Fig. 3. Flow of instructions
3.2 System Features The Use of Real Objects for Providing Responses. In this system, learners aree required to choose an appro opriate object after listening to the question. The systtem checks whether or not the object selected is correct. In order to realize this, real objects with RFID tags are used u as the response options. The respective objects are linked to a considerable amount a of attribute information that includes factual ddata about the object, such as kind, k price, container type, and size, as shown in Fig. 44. A set of correct attribute inforrmation is also prepared in advance for each question. T The system checks the responses made by the learners by comparing the difference between the attribute data of the t correct response and that of the selected response. Questions with Multiple Target T Listening Points. The attribute information of eeach object is also used to createe questions with multiple target listening points. Here, the target listening points referr to the points in a question that a learner is requiredd to listen to. For instance, in the t question “Please choose cold lemon juice in a plaastic bottle,” the target listening points are cold, lemon juice, and plastic bottle. As m mentioned above, a set of correect attribute information is prepared for each question. F Further, a question sentence paattern is also prepared in advance. For example, in Figg. 5, the attribute data—(1) cold d, (2) lemon juice, and (3) plastic bottle—are set as the correct attribute information n. When we frame a question using the above data, “Pleease choose [hot/cold] [kind of juice] j in a [container type],” the following question willl be read out: “Please choose co old lemon juice in a plastic bottle.” Error Messages and Addittional Questions. When considering feedback messages and questions, many of the exissting CALL materials, including the commercially availaable ones, only have a “Good-W Wrong” feedback function, leaving much to be desired. In this respect, more detailed feedb back and step-by-step support can be expected, dependingg on the learner's level of understtanding. Taking the above-mentioned into consideration, the prototype system verifies th he responses given by the learner and reads out correspoonding error messages as well as a additional questions. These error messages and additioonal questions are produced baseed on the degree of the learner’s mistakes.
124
H. Kashiwagi et al.
Fig. 4. Attribute information n of the objects
Fig. 5. Process of creating a question
Fig. 6 illustrates the man nner in which error messages and additional questions are selected and produced. 1. To count the number of incorrect i attributes (n) in a question 2. To calculate the ratio (n//m) between the number of incorrect attributes (n) and the total number of attributees in one question (m), by which the degree of the learnner's mistakes are graded baseed on the error level 3. To determine the scenariio pattern (feedback pattern) of the error messages and the additional questions, whiich are adapted to the learner's error level 4. To produce error messag ges and additional questions in the decided scenario pattern For example, when the learner listens to the question “Please choose cold lem mon juice in a plastic bottle” and chooses a cold, orange juice in a can, the system deteects that the attributes of kind and a container type are incorrect. It then calculates the degree of the error, i.e., two of the three targeted attributes are wrong, representedd as cold = cold, lemon ≠ orang ge, and plastic bottle ≠ can. In this case, the error leveel is determined to be Level B, and the scenario pattern decided is S3. The system tthen reads out the error messagee, “No, it is not lemon juice. It is not in a plastic botttle.” Thereafter, it reads out an additional question for the wrong answer, “Please chooose lemon juice.” If the learnerr provides the correct answer, it reads out the second quuestion, “Please choose a plasstic bottle.” If the learner answers all the additional quuestions correctly, the system reads r out the initial question again.
4 Implementation The prototype system is ou utlined in Fig. 7. The system consists of a PC, an RFID tag reader unit (RFID-RS232C C READ/WRITE BOARD), and question cards and rreal objects with RFID tags (Ph hilips, Hitag2). The tag unit can read data from and w write data into the RFID tags witthin a distance of about 5 cm. The system is implemennted using Java and JavaScript. The PC functions as a local web server in the system. As shown in Fig. 7, two thread ds, i.e., the thread of detecting the scanning activity of the RFID tags and that of obseerving the clipboard, are developed by Java; the questtion sentences, response checkin ng, error messages, and additional questions are develooped by JavaScript. JavaScript iss used because the patterns of the questions and feedbback messages can easily be addeed.
A Language Learning System Utilizing RFID Technology for TPR Activities
Fig. 6. Process of producing feedbacks
125
Fig. 7. Outline of the prototype system
The process of how the system works is described in Fig. 7. Here, one question sentence pattern and one feedback message pattern are implemented in the prototype system. The information about the question number and the kind of card (a question card or an answer option card) is transferred by using the method of “get.” The information representing whether the system produces an initial question or an additional question, correct attribute information for each question, and attribute information of the object that the learner chose as the response object, are transferred by using a “Cookie.” First, when a question card with an RFID tag is scanned, the related information is read by using the “get” method in Thread 1 of the Java application. The information about the kind of card and the question number are obtained and embedded into a URL. Then, Internet Explorer is activated with the URL (http://localhost/sample/index.html?k=40&p4=0001) in which the query string is added. In Fig. 7, k represents the kind of card, and p4 represents the question number. The file index.html provides a question or checks a response based on the information of the kind of card that is scanned. If a question is provided, the file quest.html is activated; on the other hand, when an answer is provided, the file judge.html is activated. Further, the file quest.html provides an initial question or an additional question based on the information maintained in the “Cookie.” If the initial question is to be produced, quest0.html is activated, and if the additional question is to be produced, quest1.html is activated. Similarly, with regard to the file judge.html, the “Cookie” determines whether the response is for the initial question or for the additional question. If it is for the initial question, judge0.html is activated, and if it is for the additional question, judge1.html is activated. In Thread 2, the system observes the clipboard, and produces questions or feedback messages in synthetic voices.
5 Experiment 5.1 Participants and Procedures Thirteen students were selected to evaluate the system; their evaluations were assimilated with the help of a questionnaire. There were seven undergraduates, five graduates, and one postgraduate. Two exchange students from the U.S. whose native language was English were among the seven undergraduates. They worked as
126
H. Kashiwagi et al.
teaching assistants in the free conversation room at the University. After providing them with the necessary instructions on how to use the system, each student attempted to use the system with two different feedbacks: (1) only with “OK, good”, and “Wrong, Once again,” without detailed error messages or additional questions, and (2) with detailed error messages and additional questions. Subsequently, they were required to complete a questionnaire on the usability of the system, the use of real objects as response options, and the feedback messages and additional questions depending on the degree of the learners’ mistakes. 5.2 Results and Discussion System Usability and the Use of Real Objects as Learning Materials. According to the results of the questionnaire, as shown in Table 1, all the participants were of the opinion that the system with RFID tags was easy to use. Further, twelve of them agreed to the question “Is using real objects as learning materials useful for learning vocabulary.” The results suggested that the system does have a potential impact on language learners, albeit further empirical studies are required. Detailed Feedback, Depending on the Degree of the Learner’s Mistakes. Participants were required to choose the statements that best described what they observed from among the options presented in Table 2, on the basis of the two abovementioned feedback functions. According to the results, (1) 70%–80% of the participants stated that when they used the system with detailed error messages and additional questions, they could follow the exercises by themselves; further, they found that the error messages were effective in identifying their mistakes. (2) Half of them believed that this function could enhance the interaction between the system and the learner. In fact, some of them also commented that the “Good-Wrong” feedback function was monotonous and that the detailed error messages and additional Table 1. Results of the questionnaire on system usability and use of real objects Questionnaire Yes Is the system with RFID tags easy to use? 13 (100%) Is using real objects as learning materials useful for 12 (92%) learning vocabulary?
No 0 (0%) 1 (8%)
Table 2. Results of the questionnaire on the two feedback functions Statements with regard to the two feedback functions We can identify our mistakes with the help of error messages. We can follow the exercises with the help of feedback messages. This function can enhance the interaction between the system and the learner.
“Good-Wrong” feedback 3 (23%)
Detailed feedback
4 (31%)
9 (69%)
2 (15%)
6 (46%)
10 (77%)
A Language Learning System Utilizing RFID Technology for TPR Activities
127
questions, depending on the degree of the learner’s mistakes, were good because each learner’s mistake is different from the others. Based on these results, it appears that the participants showed positive responses toward detailed error messages and additional questions, depending on the degree of the learner’s mistakes. At the same time, they also stated that variation in the feedback messages should be increased. Additional Questions. Finally, the participants were required to choose the statements that best described their opinion from among the options presented in Table 3, with regard to additional questions. In Table 3, the results are classified into two groups—native speakers (NS) and nonnative speakers (NNS). According to the results, the two exchange students (NS) agreed that one additional question was sufficient for multiple errors, while the seven NNS believed that one additional question was required for every error. Table 3. Results of the questionnaire on additional questions Statements regarding additional questions One additional question is sufficient for multiple errors. It is better to provide an additional question for the error. We can check each mistake with the help of additional questions. Additional questions are short and easy to understand.
NS (two participants) 2
NNS (eleven participants) 0
0
7
0
6
1
5
This result is consistent with the result that about 50% of the NNS felt that they could check each error with the help of additional questions. Further, five NNS agreed that the additional questions were short and easy to understand. Based on these results, it can be stated that feedback messages and short question sentences are helpful for learners who are unfamiliar with listening to a foreign language. It is expected that feedback comprising one additional question for an error is sufficient for helping foreign language learners.
6 Conclusion In this study, we proposed a CALL system with RFID tags in order to provide the basic support for listening activities in a classroom. Results from the experiment suggest that integrating real objects into the learning system by using RFID tags has a potential impact on language learning. Further studies on the variation in the feedback messages and additional questions are required to improve the system. It is assumed that the present prototype system can only be used by one user at a time. The individual user is not authenticated, and the learning history data are not saved. Using an RFID tag card as an identification card and saving history data on the RFID tags would help cope with these issues.
128
H. Kashiwagi et al.
References 1. Akiyama, I., Suenaga, S., Matsumura, Y., Minegishi, Y., Mano, S., & Iguchi, N.: The Structure of the IC Tag and its Impact. Soft Research Center (2004) (in Japanese) 2. Bandoh, H., Sato, H., Otsuki, Y., Baba, Y., Sawada, S., Ono, K.: Design and Trial Production of a Tool for Active Playtime in Preschools by Using RFID. Information Processing Society of Japan, IPSJ SIG Notes 2006(74) 2006-CE-85-(6), 41–48 (2006) (in Japanese) 3. Bhuptani, M., Moradpour, S.: RFID Field Guide. Prentice-Hall, Englewood Cliffs (2005) 4. Chaudron, C.: A descriptive model of discourse in the corrective treatment of learners’ errors’. Language Learning 27, 29–46 (1977) 5. Chaudron, C.: The role of error correction in second language teaching. In: Das, B.K. (ed.) Patterns of classroom interaction in Southeast Asia. RLC Anthology Series, vol. 17, pp. 17–50. Regional Language Centre, Singapore (1987) 6. Ito, E., Kojima, T., Takeuchi, H., Aoki, T., Miyazaki, S., Todoroki, S., Kitazawa, S., Yonekubo, S., Kazama, T.: Voice Output Communication Aid Used by the Picture Card with RFID. In: Proc. of the 21st Japanese Conference on the Advancement of Assistive and Rehabilitation Technology, pp. 327–328 (2006) (in Japanese) 7. Larsen-Freeman, D.: Techniques and Principles in Language Teaching. Oxford University Press, Oxford (2000) 8. Ogata, H., Akamatsu, R., Yano, Y.: Computer supported ubiquitous learning environment for vocabulary learning using RFID tags, TEL (Technology Enhanced Learning) 2004, France (2004) 9. Ogata, H., Yano, Y.: CLUE: Computer Supported Ubiquitous Learning Environment for Language Learning. Transactions of Information Processing Society of Japan 45(10), 2354–2363 (2004) (in Japanese) 10. Wilson, J.J.: How to teach listening. Pearson Longman (2008) 11. Yatani, K., Onuma, M., Sugimoto, M., Kusunoki, F.: Musex: A System for Supporting Children’s Collaborative Learning in a Museum with PDAs. The Transactions of the Institute of Electronics, Information and Communication Engineers J86-D-I(10), 773–782 (2003) (in Japanese)
Promoting Metacognition in Immersive Cultural Learning Environments H. Chad Lane University of Southern California Institute for Creative Technologies Marina del Rey, CA [email protected]
Abstract. Metacognition, defined as active control over cognitive processes during learning, is a critical component in the development of intercultural competence. Progression through stages of intercultural development requires self-assessment, self-monitoring, predictive, planning, and reflection skills. Modern virtual learning environments now provide a level of immersion that enable meaningful practice of cultural skills, both in terms of visual and experiential fidelity. This paper discusses their potential role in intercultural training, and the use of intelligent tutoring and experience manipulation techniques to support metacognitive and intercultural development. Techniques for adapting the behaviors of virtual humans to promote cultural learning are discussed along with the role of explicit feedback. The paper concludes with several suggestions for future research, including the use of existing intercultural development metrics for evaluating learning in immersive environments and on the balance between implicit and explicit feedback to establish optimal conditions for acquiring intercultural competence. Keywords: intercultural competence, metacognition, intelligent tutoring systems, immersive learning environments, experience manipulation.
way, training should be built around identifiable stages of development [3,8,9]. For example, if one is rushed to the point of behavior adjustment with limited or no understanding of the underlying cultural reasons, it could greatly hinder their overall development. True intercultural competence requires (at least) a heightened sense of self-awareness, an ability to self-assess, enhanced perceptive abilities, and a proclivity to reflect on experience. In other words, intercultural development requires concomitant metacognitive growth. This paper is about this process and how immersive learning environments and artificial intelligence can be used to promote intercultural learning. Metacognition involves active control over cognitive processes during learning. For example, when a learner is solving an algebra equation, cognition refers to the activities necessary to solve the problem, such as identifying rules to apply, applying them, finding a solution, and so on. Metacognition refers to a higher order of thinking that operates on these cognitive activities, such as planning, analyzing, assessing, monitoring, and reflecting on problem solving decisions and performance. Metacognition also enables more effective learning. A learner who is able to accurately gauge his or her own understanding is better equipped monitor his or her own progress and make productive decisions. This typically involves self-questioning and is part of the larger notion of metacognitive regulation [5]. Metacognitive skills can be taught. Numerous classroom studies have shown that explicitly teaching metacognitive strategies in the context of a specific domain (e.g., physics) can improve learning outcomes [4, p.19]. Strategies taught in these studies integrate metacognitive activities with cognitive and seek to make the steps of analyzing, planning, assessing, and reflection habitual in the learner. Studies have also shown that learning is more effective when learners explain worked out solutions to themselves. This phenomenon, which better learners do spontaneously, is known as the self-explanation effect. It is also possible to scaffold and promote the use of selfexplanations [6]. More recently, computer tutors focusing on teaching metacognitive skills have shown positive effects on learning behaviors (e.g., [1]).
2 Developing Intercultural Competence The Development Model of Intercultural Sensitivity (DMIS), developed by Bennett [3], is a model of intercultural development that has undergone rigorous validation. It is intended to explain how people construe cultural difference and how this ability
Fig. 1. The Developmental Model of Intercultural Sensitivity (DMIS) [3]
Promoting Metacognition in Immersive Cultural Learning Environments
131
becomes more flexible with time. By construe, Bennett is referring to Kelly’s [11] constructivist view that experience is a function of how one assigns meaning to events that occur in their lives. It is not simply a matter of being present during some event or set of events, but rather how those events are interpreted, encoded into memory, and later remembered. An underlying assumption of the DMIS is that as one’s ability to construe cultural differences evolves, intercultural competence also increases. According to Bennett, “it is the construction of reality as increasingly capable of accommodating cultural difference that constitutes development” [3, p. 24]. What constitutes a cultural difference for someone? It depends on that person’s cultural worldview, which is defined as the set of distinctions the person draws from to construe events in the world. A monocultural person – one who has primarily experienced a single culture in his or her life – will be unable to construe perceived differences from outside that cultural worldview. On the other hand, a person with a broader understanding is generally able to understand, even assume, other cultural worldviews. Hammer and Bennett summarize: “The crux of the development of intercultural sensitivity is attaining the ability to construe (and thus to experience) cultural difference in more complex ways” [8, p.423]. The DMIS is posits two broad worldview orientations: ethnocentrism and ethnorelativism. Each consists of three stages described below. The model is summarized in figure 1. Ethnocentrism is defined as an assumption that “the worldview of one’s own culture is central to all reality” [3, p.30]. Further, an ethnocentric person will implicitly assume that all others share this same worldview. The first ethnocentric stage is denial of difference in which the learner ignores or neglects differences. The next stage is defense against difference which includes recognition of cultural difference, but with negative evaluation. This stage is characterized by an “us vs. them” mindset and overt, negative stereotyping. The last ethnocentric stage is minimization of difference and includes the first signs of considering another cultural worldview. In this stage, the learner emphasizes similarities between cultures and recognizes only superficial cultural differences. Comments such as “we are all the same” are common at this stage. Guidance is especially important because some learners believe minimization is the ultimate stage of growth. When reality sets in that cultural differences are truly significant, there is a risk of withdrawal [3, p. 44]. The remaining three stages represent a shift to the ethnorelative orientation and are characterized by a basic understanding that one’s culture is but one out of many valid worldviews. The first ethnorelative stage is acceptance of difference in which the learner recognizes and appreciates cultural differences. Cultural difference evokes positive feelings, such as curiosity, in the learner for the first time. In the next stage, adaptation to difference, the learner makes an asserted effort to take the perspective of others. Because of this new perspective-taking or “frame shifting” ability, the learner can more easily interact with people from other cultures. The final stage is integration of difference: the learner has internalized multiple cultural worldviews and can easily assume different perspectives. Integration is an advanced stage often requiring years of experience to achieve. Metacognitive skills are critical for advancement through the DMIS stages. Given that the model is based on how one construes cultural differences, it follows that a learner must become aware of the construal process [11], how it works and how to assess their own ability to flexibly construe observed differences. The following metacognitive skills (at least) are related to the DMIS:
132
H.C. Lane
• Enhanced perceptive abilities are needed to consciously recognize cultural differences without reacting to them immediately. • Self-assessment shortcomings can hinder progress through the DMIS stages if the learner is not able to identify communicative failures or misunderstandings. This feeds into self-monitoring and tracking through the stages. • Cultural self-awareness, defined as an understanding of one’s own culture, is an important aspect of movement from ethnocentrism to ethnorelativism [2]. • Self-regulated learning is necessary for advancement: “Generally, people in the later phase of adaptation know how to orchestrate their own learning" [3, p. 59]. • Planning and goal-setting can support progression through DMIS stages, such as seeking to reach a specific stage or to understand specific cultural differences. Bennett addresses metacognition in his discussion of the final stage, integration of difference. He explains: “By being conscious of this dynamic process, people can function in relationship to cultures while staying outside the constraints of any particular one.” [3, p.60]. Of course, reaching this stage typically requires years of firsthand experience and is out of scope for any training program; but, it does make a strong case for nurturing an intercultural learner’s metacognitive skills. This idea is consistent with many intercultural training programs [9]. The rest of this paper explores how immersive learning environments may provide additional support.
3 Intelligent Techniques for Guided Cultural Learning 3.1 Immersive Cultural Learning Environments and Virtual Humans Technologies such as virtual reality and photoreal 3D graphics are particularly important when considering cross-cultural training. High fidelity simulations make it possible to create realistic portrayals of the products of different cultures, such as architecture, art, dress, sounds, language, attitudes, and even smells. This can promote the learner’s sense of immersion and provide a foundation for identifying cultural differences. Two such environments are shown in figure 2. The first is the Tactical Language and Culture Training System (TLCTS) developed by Alelo, Inc. [10]. TLCTS provides an experiential learning environment for acquiring language and cultural skills. In the mission environment (from Tactical Dari, shown on the left side of figure 2), a learner is free to explore an Afghani village, hear the sounds, speak to locals in Arabic using free speech, and make gestures. The clothing, buildings, and surroundings are realistic and thus can give a learner a sense of what it might be like to walk around an actual village. In this way, the system is already in a position to aid in the learner’s identification of cultural differences. The screenshot on the right in figure 2 is from the Adaptive Thinking and Leadership (ATL) simulation game [14]. ATL is a team-training system that uses human role players for both sides in intercultural scenarios. In-game assessment is performed by peers and instructors who observe play and after-action review (AAR) facilities are available to convey the outcomes to trainees. Learners are often assigned to role play as people from different cultures, with appropriate backstories and goals. Role-playing is a well-developed technique in the cross- cultural training literature and consistent with the DMIS with respect to the goal of understanding different cultural worldviews.
Promoting Metacognition in Immersive Cultural Learning Environments
133
Fig. 2. (Left) The Tactical Language and Culture Training System (Tactical Dari) by Alelo, Inc. [10] and (right) the Adaptive Thinking and Leadership simulation [14]. Reproduced with permission.
In multi-player environments, like the ATL system, inhabitants are human roleplayers. This can be costly and sometimes challenging to control from an educator’s point of view. Research on virtual humans provides an alternative or supplement to cultural team-training in immersive learning environments. Virtual humans combine artificial intelligence (AI) research in cultural and emotional modeling, speech processing, dialogue management, natural language understanding, and gesturing, among others, to enable natural feeling communication and interaction with computercontrolled characters that listen and respond to the user. Virtual humans are driven by detailed models of tasks, emotion, body language, and communication [16]. The underlying representations readily support explanation, which can be useful for learning [7]. In the case of intercultural education, it is important to endow virtual humans with the ability to explain their actions and reactions in terms of their cultural worldview. It is also important that their behavior be controllable in order to establish conditions that best promote learning. 3.2 Experience Manipulation and Implicit Feedback Generally speaking, computer simulations simulate real world phenomena as accurately as possible. There are circumstances when it is appropriate to consider goals other than fidelity when deciding how a simulation should behave and what events should occur. For instance, to enhance entertainment value, a popular basketball video game in the 80’s included special modes that allowed players to jump well over ten feet high and be “on fire”. In this case, the goal of entertaining the player trumps the goal of simulating basketball completely realistically. In the case of learning, the same idea applies: if a certain event or situation will promote learning, then the simulation should seek to produce that experience. The caveat is that in the case of learning, the adaptations should probably fall within bounds of feasibility. We refer to this general technique as experience manipulation and now discuss several ways it might be used to promote metacognitive growth and cultural learning.
134
H.C. Lane
Fig. 3. Expressions of anger, skepticism, and appreciation by virtual humans [7,16]
In a face-to-face intercultural situation, there are several ways that intelligent interventions could promote learning: • • • •
recognizing that an error was committed (or a good action was taken) finding a causal link between the action taken and the observed reaction understanding the reason(s) and culpable underlying cultural differences learning to avoid the same mistake in the future (or sustain good actions)
If a learner commits a cultural error, for example, and simply concludes to avoid the same behavior in the future, it will contribute little in their progression through the DMIS stages since this does not get to underlying causes. Also, the stage a person is in impacts how cultural differences are interpreted. Someone in the denial phase may not even be willing to accept the fact that a cultural error even occurred, for example. A learner in the other two ethnocentric stages (defense and minimization) may be aware an error occurred, but unwilling to take blame or perhaps place the onus on the virtual human to be the one who should adapt. Based on this understanding, the reaction of a virtual human to a cultural error should be appropriate for that learner. Feedback from the simulation itself, such as the oral and nonverbal reactions of virtual humans, is called implicit feedback. To support recognition of cultural errors, there are several strategies that can be used adjust implicit feedback to promote learning. One of the simplest is to accentuate verbal responses of characters to draw more attention to anger or negative feelings, in the case of an error, thus supporting the learner in recognizing a cultural difference that exists. Similarly, implicit positive feedback can be achieved by accentuating positive and laudatory responses to correct user actions. In some cases, it may even be appropriate for the virtual human to deliver an impassioned mini-lecture regarding the cultural issues in question. The choice of words by the virtual human can be designed to refer directly to actions taken by learner to support the pedagogical goal of linking cause and effect in the learner’s mind. In addition, the virtual human might also drop hints regarding the underlying cultural differences. Other forms of implicit feedback, such as body language and gestures, can have a dramatic effect on the communicative power of utterances. Figure 3 shows several virtual humans in different emotional states and displaying a variety of gestures. The timing and emphasis of these gestures can be adjusted to meet pedagogical goals in a way similar to the utterance content. Aside from body
Promoting Metacognition in Immersive Cultural Learning Environments
135
language, other features that might be adjusted in virtual humans are facial expressions, speech rate, intonation, and tone, emotional state, and personality traits. 3.3 Experience Management and Interactive Narrative The techniques described in the previous section all focused on emphasizing specific details of interacting with virtual humans to support the learner in recognizing cultural difference and improving their ability to self-assess in an interpersonal context. Of course, explanations for why certain behaviors are culturally offensive can be very complex. They may involve fundamental differences between worldviews, varying ethical standards, social structure, historical and geographical factors, and so on. A deep understanding of culture includes these advanced notions and may enable a learner to go beyond rote learning by providing the knowledge needed to reconstruct appropriate surface behaviors later. Cultural simulations should provide diverse cultural experiences that go beyond one-onone interactions and manage how events are presented and experienced by the learner. One such technology focusing on experience management is automated story directing (ASD) [15]. The goal is similar to that of a traditional tutoring system: allow users to feel as much freedom as possible, but keep them on certain paths that consist of certain experiences. The “path” in an interactive storytelling system is a storyline consisting of plots, arcs, events, and other narrative elements. Users may “break” a storyline by taking actions in the virtual world, and so ASD systems use a variety of techniques, like reactive planning, to repair storylines and re-plan when new events are deemed desirable. Often, the aim is to maximize engagement. For cultural training, the additional aim is to create situations and conditions that specifically address intercultural development of a learner and their specific areas of need. Learning issues involving metacognitive skills come into play when we consider the learner’s role in the narrative. She or he must be aware that the actions being taken are being observed by the AI agents in the simulation and that choices being made have observable outcomes. Just as behavioral details of interactions can be manipulated to highlight differences, so can story elements. For example, if a learner makes a gender-related error early in a game, the ASD may decide to propagate this knowledge through the social network so other story elements and characters can exploit the weakness. Here the goal is not only to teach gender specific cultural differences, but also to encourage consideration of unintended cultural consequences of earlier actions. This requires the metacognitive ability to continuously self-assess over an extended period, reaching back further than just the most recent action, and the ability to predict (another metacognitive skill). After dealing with negative consequences of actions, it is hoped that a learner will become more likely to anticipate possible unintended outcomes of actions before taking them. 3.4 Guidance and Feedback Inherent risks exist in unguided environments, such as inefficient learning, the formation of misconceptions, and development of incomplete or fragmented knowledge [12]. Experience manipulation and implicit feedback can certainly mitigate these risks to a certain degree, but to adequately address the needs of novice and intermediate learners, explicit feedback from a human tutor, pedagogical agent, disembodied
136
H.C. Lane
coach, or other form of intelligent tutor has the potential to greatly enhance learning. Explicit guidance can come in different forms in an immersive learning environment. A pedagogical agent who plays a role in the underlying simulation is a popular approach. The TLCTS [10] and the mission rehearsal exercise described in [16] both provide pedagogical agents in the form of a knowledgeable companion who can give hints on how to succeed. No matter what the modality, explicit feedback generally provides more direct and understandable guidance than implicit – this is especially important for novices [12]. Immersive learning environments can be overwhelming at times. With respect to cultural learning, explicit hinting and feedback can help learners in several ways: 1. 2. 3. 4. 5.
confirm a learner’s interpretation of observed virtual humans behaviors explain the cultural differences in play during specific interactions explain the “under the hood” reasoning of a virtual human hint about ideal actions to take or warn against certain risks suggest the learner identify possible outcomes and desirable end states
Explicit tutorial feedback removes a level of interpretation for the learner. Rather than guessing or inferring the cognitive and emotional states of virtual humans, a clear statement by a tutor can act as a strong scaffold for learning in immersive cultural environments. There are certainly cognitive aspects to the tactics listed above, but they also address the metacognitive demands of intercultural development. Tactics 1-3 encourage self-assessment by describing the impact of a learner’s actions on a virtual human. Because this is feedback being delivered in a real-time environment, the issues of distraction and cognitive overload need to be considered. Thus, it is ideal to keep “in-game” feedback short and precise, saving the longer explanations for postpractice reflection. Hinting (tactic 4) can be direct (and at the cognitive level), but also can be used to encourage the learner to think about pros and cons of taking different actions – this is especially important in ill-defined domains where assessment is inherently challenging [13]. The content of tactics 1-4 are precisely the what should be a part of the learner’s deliberations before acting in the environment. Such cognitive activities by the learner would constitute attempts at self-explanation. Tactic 5, identifying potential end states, is a purely metacognitive tactic that is geared towards supporting goal formation and identifying “what right looks like.” Encouraging the learner to “think before acting”, to engage in planning and simulate hypothetical actions, and “reflect after acting” are at the core of metacognitive growth and a fundamental requirement for advancement through the DMIS stages. Tutorial sub-dialogues are rarely possible in real-time environments, and so there is time only for very brief periods of reflection. However, once a practice session or exercise is complete, there is time to carefully target metacognitive skill development. Immersive learning environments should therefore provide supporting tools such as video playback. Reflective tutoring is an appropriate supplement to guide the use of these tools and to fill in the gaps from feedback that was delivered during practice. The reflective tutoring system built for the virtual humans [7] walks the student through three questions: (1) What happened? (2) Why did these events occur the way they did? and (3) How can good performance be sustained and poor performance be improved? A promising approach here is to leverage explanation facilities of virtual humans to uncover their thinking via explanation and discover what other actions may
Promoting Metacognition in Immersive Cultural Learning Environments
137
have produced better outcomes. An advanced tactic is to restart the simulation to give the learner a second chance (a “mulligan”). Reflection at this point may enhance selfassessment skills and intercultural growth.
4 Conclusions and Suggestions for Future Research This paper has argued that achieving intercultural competence requires strong metacognitive abilities. Although cross-cultural training programs frequently adopt metacognitive approaches to teaching intercultural competence, the connection is rarely made explicit. The Peace Corps describes this growth as a continuum from unconscious incompetence (not knowing anything and being blissfully unaware of differences) to unconscious competence (full awareness of differences and appropriate behavior is second nature). By describing these stages, the learner put in a position to self-monitor their advancement. This then requires application of other metacognitive skills, such as self-assessment, self-explanation and self-regulation, to progress through the stages. The Developmental Model of Intercultural Sensitivity (DMIS) is an empirically derived and validated model of intercultural development based on notion of how cultural differences are “construed” by a learner [3,11]. To develop intercultural competence, it is necessary to construe cultural difference in progressively more complex ways, such as from different cultural worldview perspectives. Growth here also requires mature metacognitive abilities and it may be possible to promote these skills in modern immersive learning environments through a combination of experience manipulation and explicit guidance. The DMIS suggests cultural difference as the pivot point for intercultural growth, and so careful direction of virtual humans and delivery of feedback that targets self-assessment, predictive, and reflection skills has the potential to speed the growth to intercultural competence. Most of the computer simulations built for cultural education have not undergone rigorous experimental evaluations for learning or for intercultural development. Hammer and Bennett’s Intercultural Development Inventory (IDI) [8] has been used to validate the DMIS and may provide a suitable metric for determining the valueadded, if any, that comes from augmenting cultural training programs with immersive learning environments – especially if the IDI can be administered repeatedly and rates of change can be tracked. Other more general questions about feedback are suggested for further study. For example, • • • •
Does implicit pedagogical feedback break immersion? If so, what is the cost (if any) to breaking immersion with respect to learning? What is the interplay between implicit and explicit feedback? What situations merit the use of explicit feedback?
The use of implicit feedback and experience manipulation is perhaps one of the most important open questions to address. The instructor interface to the ATL serious game [14] provides the ability to throw “curve balls” to teams during their missions, such as helicopter fly-overs. These are intended to support the development of adaptive thinking skills under stress. This is related to research in the ITS community on finding the appropriate level of challenge for a learner. These connections need further exploration, as does the reasoning behind expert instructors’ decisions to throw curve balls:
138
H.C. Lane
What are the triggers? How do instructors decide which curve ball to throw? The answers may not always involve metacognitive skills, but as this paper has attempted to argue, manipulations of this sort in an intercultural context may be ideal to highlight cultural differences to give learners practice in dealing with them and in developing intercultural competence.
Acknowledgments The project described here has been sponsored by the U.S. Army Research, Development, and Engineering Command (RDECOM). Statements and opinions expressed do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred.
References 1. Aleven, V., McLaren, B., Roll, I., Koedinger, K.: Toward meta-cognitive tutoring: A model of help-seeking with a cognitive tutor. Int. J. of Art. Int. in Ed. 16, 101–130 (2006) 2. Bennett, J.M.: Intercultural communication training in cultural self-awareness for study abroad. PhD thesis, University of Minnesota (October 1985) 3. Bennett, M.J.: Towards ethnorelativism: A developmental model of intercultural sensitivity. In: Paige, R.M. (ed.) Education for the Intercultural Experience, pp. 21–71. Intercultural Press, Yarmouth (1993) 4. Bransford, J.D., Brown, A.L., Cocking, R.R. (eds.): How People Learn: Brain, Mind, Experience, and School. National Academy Press, Washington (2000) 5. Brown, A.L.: Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In: Weinert, F.E., Kluwe, R.H. (eds.) Metacognition, motivation, and understanding, pp. 65–116. Lawrence Erlbaum Associates, Hillsdale, New Jersey (1987) 6. Chi, M.T.H., de Leeuw, N., Chiu, M.H., LaVancher, C.: Eliciting self-explanations improves understanding. Cognitive Science 18(3), 439–477 (1994) 7. Core, M.G., Traum, D., Lane, H.C., Swartout, W., Gratch, J., van Lent, M., Marsella, S.: Teaching negotiation skills through practice and reflection with virtual humans. Simulation 82(11), 685–701 (2006) 8. Hammer, M.R., Bennett, M.J., Wiseman, R.: Measuring intercultural sensitivity: The intercultural development inventory.I. J. of Intercultural Relations 27, 421–443 (2003) 9. Hoopes, D.S.: Intercultural communication concepts and the psychology of intercultural experience. In: Pusch, M.D. (ed.) Multicultural education: A cross cultural training approach, pp. 10–38. Intercultural Network, Inc., La Grange Park (1979) 10. Johnson, W.L., Vilhjalmsson, H., Marsella, S.: Serious games for language learning: How much game, how much AI? In: Looi, C.-K., McCalla, G., Bredeweg, B., Breuker, J. (eds.) Proc. of 12th Int. Conf. on Artificial Intelligence, pp. 306–313. IOS Press, Amsterdam (2005) 11. Kelly, G.A.: Theory of personality: The psychology of personal constructs. Norton, New York (1963) 12. Kirschner, P.A., Sweller, J., Clark, R.E.: Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist 41(2), 75–86 (2006)
Promoting Metacognition in Immersive Cultural Learning Environments
139
13. Ogan, A., Aleven, V., Jones, C.: Culture in the classroom: Challenges for assessment in illdefined domains. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053. Springer, Heidelberg (2006) 14. Raybourn, E.M.: Applying simulation experience design methods to creating serious game-based adaptive training systems. Interacting with Computers 19, 206–214 (2007) 15. Riedl, M.O., Stern, A., Dini, D., Alderman, J.: Dynamic experience management in virtual worlds for entertainment, education, and training. Int. Tran. on Systems Science and Applications, Sp. Iss. on Agent Based Systems for Human Learning 4(2), 23–42 (2008) 16. Swartout, W., Gratch, J., Hill, R., Hovy, E., Marsella, S., Rickel, J.: Toward virtual humans. AI Magazine 27(2), 96–108 (2006)
The Application of the Flexilevel Approach for the Assessment of Computer Science Undergraduates Mariana Lilley and Andrew Pyper University of Hertfordshire, School of Computer Science, College Lane, Hertfordshire AL10 9AB, United Kingdom {m.lilley,a.r.pyper}@herts.ac.uk
Abstract. This paper reports on the use of the flexilevel approach for the formative assessment of Computer Science undergraduates. A computerized version of the flexilevel was designed and developed, and its scores were compared with those of a traditional computer-based test. The results showed that the flexilevel and traditional scores were highly and significantly correlated (p<0.01). Findings from this study suggest that the flexilevel approach is a viable adaptive testing strategy, and may be a good candidate for smaller applications where IRT-based CATs may be too demanding in terms of resources. Keywords: e-assessment, flexilevel, adaptive testing strategies.
The Application of the Flexilevel Approach for the Assessment of Computer Science
141
Other studies on the use of adaptive testing in a higher education environment include research previously conducted by one of the authors [7]. In this work, it was shown that the application of an IRT-based approach to adaptive testing supported proficiency level estimates comparable to those obtained using traditional (i.e. static) tests. The IRT-based approach to adaptive testing was also shown to be effective at tailoring the level of difficulty of a test to the proficiency level of individual students. By tailoring the level of difficulty of the question or task to individual proficiency levels, students were challenged by test items at an appropriate level, rather than demotivated by items that were above or below their proficiency level. Despite these positive findings, procedural factors impeded a greater uptake of the adaptive approach as proposed in the research [7]. The implementation of IRT-based adaptive testing applications is a complex task. Most importantly, the IRT model requires a large and calibrated database of questions. The calibration process usually requires a large item pool and specialist calibration programs, such as Xcalibre [1, 5]. This paper reports on a pilot study that investigated the implementation of computerized adaptive testing based on the flexilevel approach [8, 11] as an alternative to IRT-based algorithms. An overview of the flexilevel approach is presented in the next section of this paper.
2 The Flexilevel Approach Like IRT, flexilevel algorithms attempt to match the level of difficulty of questions to the proficiency level of individual students. The Flexilevel approach, however, is based on a fixed branching strategy that is less complex to implement than IRT-based algorithms. Database calibration is also simpler; indeed Lord [8, p. 117] suggests that “any rough approximation” of the difficulty, of the questions will be adequate. The difficulty of a question is determined by the following formula (adapted from Ward [12]):
⎛ n × 100 ⎞ ⎟⎟ D = 1 − ⎜⎜ p n r ⎝ ⎠
(1)
In Equation 1, np is the number of students who answered the question correctly, and nr is the total number of students who answered the question. In addition to simpler calibration, smaller question databases are required. The flexilevel approach requires a database of 2n-1 question (where n is the number of questions to be administered during the test). A flexilevel test typically starts with a question of medium difficulty. If the student answers the question correctly, a more difficult question is presented. Conversely, if the question is answered incorrectly, an easier question follows. The flexilevel approach was initially devised as a self-scoring paper-and-pencil test [8]. As an example, consider a paper-based test with 19 questions; the difficulty of the questions ranging from 0.05 (easiest) to 0.95 (hardest).
142
M. Lilley and A. Pyper [Q10, difficulty = 0.50] Red 1 [Q9, difficulty = 0.45] Blue 1 [Q11, difficulty = 0.55] Red 2 [Q8, difficulty = 0.40] Blue 2 [Q12, difficulty = 0.60] Red 3 [Q7, difficulty = 0.35] Blue 3 [Q13, difficulty = 0.65] Red 4 [Q6, difficulty = 0.30] Blue 4 [Q14, difficulty = 0.70] Red 5 [Q5, difficulty = 0.25] Blue 5 [Q15, difficulty = 0.75] Red 6 [Q4, difficulty = 0.20] Blue 6 [Q16, difficulty = 0.80] Red 7 [Q3, difficulty = 0.15] Blue 7 [Q17, difficulty = 0.85] Red 8 [Q2, difficulty = 0.10] Blue 8 [Q18, difficulty = 0.90] Red 9 [Q1, difficulty = 0.05] Blue 9 [Q19, difficulty = 0.95]
Fig. 1. Adapted from Lord [8]. Layout of a paper-based flexilevel test, in which Q1 is the easiest question and Q19 is the hardest question
In the example shown in Fig. 1, the student will start the test by answering Q10 (i.e. question of medium difficulty). The answer sheet will inform the student whether each response was correct or incorrect; for example, a red spot appears where the student has selected an incorrect answer and a blue spot appears where the student’s response was correct. Each time a correct answer is given, the question to be answered next is the lowest numbered “blue” question not previously answered. In the event of an incorrect response, the next question to be answered is the lowest numbered “red” question not previously answered. A student who answers all questions correctly will answer the following sequence of questions: Q10, Q11, Q12, Q13, Q14, Q15, Q16, Q17, Q18, and Q19. A student who answers all questions incorrectly will answer the following sequence of questions: Q10, Q9, Q8, Q7, Q6, Q5, Q4, Q3, Q2, and Q1. A student who answers the first question incorrectly, and all the following questions correctly will answer the following sequence of questions: Q10, Q9, Q11, Q12, Q13, Q14, Q15, Q16, Q17, and Q18. Following the directions for a paper-based flexilevel test can be an onerous and disengaging task. This issue can be avoided by developing a software application to select, administer and score the questions, and this is the focus of the study reported here.
3 Methodology The experiment was designed to model a real formative assessment situation as far as possible, given the constraints of the experimental design. So, whilst participants were free to leave without consequence and they were not the subject of the test, they were encouraged to engage with the test as if it were a real formative assessment. 3.1 Participants Twenty-four final year Computer Science undergraduates participated in the experiment. They had previously studied the topic that was the subject of the test, but were briefed that the experiment was a test of the software application and not of them.
The Application of the Flexilevel Approach for the Assessment of Computer Science
143
3.2 Software The software prototype was developed using VB.NET with an Access database to store the questions and responses. There were 29 questions; these were separated into two pools, one to populate the flexilevel test (19 questions), and the other to populate the statically selected test (10 questions). The questions had previously been used in standard testing and calibrated based on the calibration formula described previously. The order of presentation of the two tests was randomised. The two different sets of questions had different light background colours on the question screens. This was done in order to enable the experimenters and participants to refer to the different tests without naming them. The colours used were selected because they maintained a clear contrast between the text of the questions and the background of the screen. Fig. 1 shows the layout of the question screen.
Fig. 2. Screenshot of the flexilevel software application
Finally, the software allows for the timing of a test, and displays time elapsed and number of questions answered throughout the test. 3.3 Questionnaire The questionnaire document consisted of a briefing section setting out information about the flexilevel approach, test and questionnaire and 15 statements that participants could rate using a 5 point Likert scale from 1: Strongly Disagree to 5: Strongly Agree. The statements are shown in Tables 3 (ease of use) and 4 (perceived usefulness). The statements were designed to elicit participants’ views on the ease of use of the software application and the perceived usefulness of the software application.
144
M. Lilley and A. Pyper
A further 3 questions that asked for free text responses were included, and are shown in Table 1 below. Table 1. Open questions included in the questionnaire
16
Statement What problems, if any, can you see to the uptake of this adaptive testing approach?
17
Is there a question that you would like to have been asked? If so, what is it and how you would answer it?
18
Can you see any benefits of this adaptive testing approach?
3.4 Experimental Procedure The participants were briefed that they were about to engage in an assessment using a new form of software assessment. They were told that the test was of the software and not of them. However, in order to encourage them to answer the questions to the best of their ability, participants were informed that the participant with the highest overall score would be given £20.00. They were also told that they would be asked to fill in a questionnaire. All participants were offered £10.00 as a thank you for participating, but it was made clear that they could leave at any time and did not have to complete the test or the questionnaire in order to receive the money. The participants took the test in a computer lab in test conditions. They started the test in their own time and had 25 minutes to complete the whole test; this was monitored by the software application. This enabled the collection of the performance of participants in the two tests and the reporting of their results at the end of the test. Once the participants had completed the tests they were asked to fill in the questionnaires that had been placed by their computers. This enabled the collection of data associated with the views of participants. 3.5 Hypothesis The hypothesis of the experiment was that there would be a significant correlation between students’ results on the statically generated test and the flexilevel generated test.
4 Results A summary of students’ performance is presented in Table 2. Table 2 shows that the range of scores achieved by participants was relatively large for both test conditions and that there is little apparent difference between the conditions.
The Application of the Flexilevel Approach for the Assessment of Computer Science
145
Table 2. Summary of student scores for each section of the test (N=24) Test Flexilevel Static
Minimum
Maximum
1.0 2.0
9.0 8.5
Mean 4.62 4.89
Std. Deviation 2.10 1.91
A Pearson product-moment correlation coefficient was computed to assess the relationship between the flexilevel and static scores shown in Table 2, r=0.764, n=24, p<0.01, Sig. (2-tailed)=.000. A paired-samples t-test showed no significant difference between the flexilevel and the static scores, t(23) = -.954, p=.350. The scatterplot shown in Fig. 2 illustrates the results.
Fig. 2. Scatterplot diagram showing the scores achieved by participants in the two test conditions
Student responses to the questionnaires were also analysed. Table 3 below shows that the majority of participants agreed with statements concerning the ease of use of the software application. As can be seen from Table 4, participants were also positive when responding to statements about the perceived usefulness of the software application. This was particularly the case in terms of using the flexilevel for formative assessment (statement 4), areas that the participants might need to work on (statement 6), or not (statement 11) and that if flexilevel tests were made widely available that they would use them (statement 13).
146
M. Lilley and A. Pyper Table 3. Ease of use. Mode and median for the responses. Statement Median Mode 1. Learning to use the Flexilevel software application would be 4 4 easy for me. 2. I would find it easy to remember how to perform tasks (e.g. 4 4 how to answer a question) using the Flexilevel software application. 3. I would find the Flexilevel software application easy to use. 4 4 Table 4. Perceived usefulness. Mode and median for the responses. Statement Median 4. I would find the Flexilevel approach useful for practice 4 tests. 5. I would find the Flexilevel approach useful in summative 3.5 tests (i.e. the test score counts towards my final grade). 6. The adaptivity supported by the Flexilevel approach 4 would help me to identify the areas in which I need to work harder more quickly. 7 The adaptivity supported by the Flexilevel approach would 4 enhance my overall assessment experience. 8 For practice tests, I would prefer using the Flexilevel 4 approach to other forms of objective testing. 9. For summative tests, I would prefer using the Flexilevel 3 approach to other forms of objective testing. 10. The system used to score a Flexilevel test makes sense to 4 me. 11. I would find the score provided by the Flexilevel 4 approach useful at identifying how much I have learned. 12. I would find it useful if the level of difficulty of a test is 4 tailored to my level of understanding. 13. Assuming the Flexilevel software was available to me for 5 practice tests, I predict that I would use it on a regular basis. 14. In practice tests, test questions that are too easy are less 4 engaging than those questions that are tailored to my level of understanding. 15. In practice tests, test questions that are too difficult are 3 less engaging than those questions that are tailored to my level of understanding.
Mode 5 3 5
4 4 3 4 4 4 4 4
3
Participants were less positive in their responses to the use of the flexilevel approach in summative assessments (statement 5) and tended not to agree that more difficult questions were less engaging (statement 15). The open questions were included to enhance the richness of the data collected, but were too few to be subjected to content analysis. As such, they will be used to inform the discussion of the results that follows, rather than being reported in this section.
The Application of the Flexilevel Approach for the Assessment of Computer Science
147
5 Discussion The results of this pilot study are very encouraging, there is a high and significant correlation between participants’ performance on the two different types of test and participants were positive about their use of the software. Additionally, participants were positive about the use of the flexilevel approach in formative assessment. Moreover, there was a reasonably wide distribution of scores. So, it seems that performance of participants in the two tests is consistent across a range of attainment. Nonetheless, this is a pilot study, and as such there are aspects of this study that future experiments will need to replicate. The participants are all final year undergraduate students. It may be expected that such a sample would have no problems with using the software application. The measures of ease of use are important in controlling for potential extraneous variables, and also to identify any usability issues in these early stages of development. In this sense a sample of Computer Science undergraduates was a good choice for the pilot study. Expert analysis of the interface indicated no serious usability issues that might be expected to trip up even users with a high degree of technical literacy. The results of the study bear this out. Clearly the downside to using Computer Science students as participants is that this sample of participants may not be representative of a wider student population. Whilst the participants in this study reported no problems in using the software application, it seems likely that extrapolating this to a wider population would be unwise. Clearly an important part of future experiments would be to involve participants from a wider population of students. One reservation raised about the software application was that “You can't go back and correct a previous answer”. This is a necessary feature of the system, however, because if participants were able to go back and review the questions, they would soon be able to enhance their marks simply by going back and attempting the questions until they found the correct response. In terms of the perceived usefulness of the application, there seems to be a difference in the way participants perceived the flexilevel approach as a tool for formative and summative assessment. Participants were positive about the formative use of the flexilevel approach. One participant commented that “providing multilevel questions after one another is good”. As noted previously, participants believed it could support them in their academic progress – “...would really help to outline where problems in understanding lie and help students to address those areas.” Also “...good approach to learning giving better students harder questions.” This attitude did not extend to the use of the flexilevel approach in summative contexts: “all students may not be tested equally” and “I think that there would be a smaller gap in marks between good and bad students than in normal tests which would not be good for assessment”. Participants did suggest that the flexilevel approach would enhance their overall assessment experience (statements 7 and 12), but it seems that this enhancement would mostly be realised in the formative assessment they engage in. An informal observation has been that students were uncomfortable using a system in which the scoring system was not clear to them; for example, maximum likelihood estimates, used in IRT-based adaptive testing may not be obvious to all. In this study, participants indicated that they did indeed understand how the scoring system worked
148
M. Lilley and A. Pyper
(statement 10). This could be taken to be support for the use of a simpler adaptive system than those that had been previously employed. However, this bears further scrutiny in future work given that participants felt that high-performing students may be disadvantaged by this scoring in summative assessments.
6 Conclusion This pilot study was concerned with the efficacy of the flexilevel approach to adaptive testing when compared to standard computer based testing. The results show a highly significant correlation between scores achieved by participants on the two tests. This provides a basis upon which to replicate and extend this research to include larger groups of students and those in studying in different academic disciplines. This will be the subject of future work.
References 1. Assessment Systems Corporation, XCALIBRE Marginal Maximum-Likelihood Estimation, http://assess.com/xcart/product.php?productid=270&cat=0& page=1 2. Challis, D.: Committing to quality learning through adaptive online assessment. Assessment & Evaluation in Higher Education 30(5), 519–527 (2005) 3. Conejo, R., Guzmán, E., Millán, E., Trella, M., Pérez-de-la-Cruz, J.L., Ríos, A.: SIETTE: A Web–Based Tool for Adaptive Testing. International Journal of Artificial Intelligence in Education 14, 1–33 (2004) 4. Conejo, R., Millán, E., Pérez-de-la-Cruz, J.-L., Trella, M.: An empirical approach to online learning in SIETTE. In: Gauthier, G., VanLehn, K., Frasson, C. (eds.) ITS 2000. LNCS, vol. 1839, pp. 604–614. Springer, Heidelberg (2000) 5. Gierl, M.J., Ackerman, T.: Software Review: XCALIBRE — Marginal MaximumLikelihood Estimation Program, Windows Version 1.10. Applied Psychological Measurement 20(3), 303–307 (1996) 6. Joy, M., Muzykantskii, B., Evans, M.: An Infrastructure for Web-Based ComputerAssisted Learning. ACM Journal of Educational Resources 2(4), 1–19 (2002) 7. Lilley, M.: The Development and Application of Computer-Adaptive Testing in a Higher Education Environment, Unpublished PhD thesis, School of Computer Science, University of Hertfordshire, Hertfordshire (2007) 8. Lord, F.M.: Applications of Item Response Theory to Practical Testing. Lawrence Erlbaum Associates Inc., Mahwah (1980) 9. Ríos, A., Conejo, R., Trella, M., Millán, E., Pérez-de-la-Cruz, J.L.: Aprendizaje automático de las curvas características de las preguntas en un sistema de generación automática de tests. In: Actas de la Conferencia Española para la Inteligencia ArtificialCAEPIA 1999 (1999) 10. Ríos, A., Millán, E., Trella, M., Pérez-de-la-Cruz, J.L., Conejo, R.: Internet Based Evaluation System. In Open Learning Environments: New Computational Technologies to Support Learning, Exploration and Collaboration. In: Proceedings of the 9th World Conference of Artificial Intelligence and Education AIED 1999, pp. 387–394. IOS Press, Amsterdam (1999) 11. Wainer, H.: Computerized Adaptive Testing: A Primer. Lawrence Erlbaum Associates Inc., Mahwah (2000) 12. Ward, C.: Preparing and Using Objective Questions. Nelson Thornes Ltd., Cheltenham (1980)
Development of Ubiquitous On-Demand Study Support Environment for Nursing Students Yukie Majima1, Yumiko Nakamura2, Yasuko Maekawa2, and Yoichiro So3 1
Department of Liberal Arts and Science, Osaka Prefecture University 1-1 Gakuencho, Naka, Sakai, Osaka 599-8531, Japan [email protected] 2 School of Nursing, Osaka Prefecture University 3-7-30 Habikino, Habikino, Osaka 583-8555, Japan [email protected], [email protected] 3 Production Systems Research Lab. KOBELCO 1-5-5 Takatsukadai, Nishi, Kobe, Hyogo 651-2271, Japan [email protected]
Abstract. We were selected to work with “Development of e-Learning Program to help enhance human resource ability based on needs,” which is a government supported program of fiscal year 2005 to address modern education needs (Modern GP). The program has therefore been underway since 2005. The objectives of the program are to use e-Learning to further improve education and teaching practices qualitatively for nursing education, to supply high-level nursing practice capabilities, and provide a new environment in which students can study independently in an efficient manner. The main targets to achieve the program goals are: i) to produce e-Learning training materials that include examples of nursing to use for the study of nursing practice and support students’ acquisition of problem-solving abilities during nursing operations; and ii) to construct an environment to support ubiquitous on-demand studies in which students can study for themselves easily, at any time, and anywhere on campus or on the actual practice site using this e-Learning methodology. This paper presents a report of an actual lecture class conducted in using such training materials. Keywords: Education of nursing, Nursing practice ability, E-learning, Ubiquitous on-demand study.
and instruction of nursing more qualitatively, and to provide students with a new environment in which they can study efficiently. Among all educational activities, the most effective way for students to acquire actual nursing capabilities is to have the learner join on-site practice training, which is arequired part of the nursing education curriculum. Through on-site practice, students can learn comprehensively by communicating directly to patients or those who require nursing care. However, such practicing facilities are spread out in various locations. For that reason, the situation available on site is not necessarily as good a study environment as that available on campus. The campus might provide facilities such as libraries, Internet capability, and IT equipment. By making e-Learning available at every on-site practice location, we believe that we can not only improve the study environment situation, but also help to raise the quality level of the study itself. Under these circumstances, the School of Nursing at the Osaka Prefecture University was selected to work with “Development of e-Learning Program to help enhance human resource ability based on needs,” which is a government-supported program of fiscal year 2005 to address modern educational needs (Modern GP). The actual title is “e-Learning to help acquire nursing practice ability –Construction of a ubiquitous on-demand study supporting environment–.” The program began activities last year as the “CanGo” project. “CanGo” stands for “Communication”, “Art”, “Nursing”, “Good practice”, “Osaka Prefecture University”. This paper reports the actual classes taught in 2006-2007 using materials that were developed specifically for this project based on project activities undertaken in 2005, and of the results obtained from on-site training support we gave to students.
2 CanGo Project Summary The CanGo project objective is to fill the gaps that are readily noticeable from one study method to another, to improve the efficiency and usefulness of every study activity on-campus or off-campus, and to improve students’ attitudes toward study more aggressively. This objective must be achieved so that students can acquire the ability to solve study-related problems that might often occur in lectures, or during practice sessions made on-campus or on-site, all of which are conducted in the regular curriculum of nursing education. The backbones of our activities are: i) to produce e-Learning materials for the study of nursing practice and examples for use by students to acquire problem-solving ability; and ii) to construct a ubiquitous on-demand study-supporting environment in which students can study independently through e-Learning at any time, anywhere, and without any difficulty whatsoever. Through these four-year studying opportunities on and off campus, we are hoping to help students acquire capabilities to solve all kinds of nursing problems efficiently and aggressively so that, eventually, we can produce in our society good human resources with full proficiency in advanced nursing practice capabilities. Specifically, the following three unique environments are available to support students’ study.
Development of Ubiquitous On-Demand Study Support Environment
151
Utilizing USB memory that stores the recorded study history, anyone can easily download examples of nursing from the nursing training materials server machine. Each student is allowed to compile a digital nursing dictionary by downloading the necessary sub-training materials before the on-site practice starts. We believe that any student can study independently while compiling such a “personal digital nursing dictionary.” During that study process, a learner would be able not only to review and summarize the contents of past study, but also to proceed with meta-recognition that would point to further study or related activities, and what knowledge and nursing technology must be acquired before the on-site practice starts. Finally, to support selfstudy, a bi-directional study support environment would be prepared in which a
Fig. 1. Overall structure of CanGo project Table 1. CanGo project targets and its accomplishments Targets 1) Develop 100 examples of nursing by re-structuring the existing nursing examples documents, and construct a training-material database 2) Construct a ubiquitous on-demand study support environment that is openly available to students Accomplishments 1) Development of 107 examples in four specialized nursing areas, and a total of 1901 references for the digital nursing dictionary 2) Supplying equipment for study support environment 272 portable multimedia terminals 15 laptop PCs to support lecture classes 1 wireless attachment for a projector
152
Y. Majima et al.
student can receive guidance from the instructor through e-mail that might be available using a cell phone or other media equipment. Figure 1 depicts the overall structure of the CanGo project. Table 1 shows its targets.
3 e-Learning Materials for Nursing Examples Simulation We produced nursing training materials with examples for each specialized nursing area based on prototype training frames for e-Learning of nursing, developed by Majima et. al. and Seta et. Al.[1,2,4,5] to improve nursing students’ problem-solving abilities.
Fig. 2. Training frames on nursing examples
Fig. 3. Digital nursing dictionary
Development of Ubiquitous On-Demand Study Support Environment
153
Training frames (Figure 2) consist of two study modes. In study mode 1, a series of nursing processes can be studied through “Understanding patients’ examples”, “Problem studies on nursing”, and “Evaluation of analyzed results”. Study mode 2 is available for each example of a patient, in which the student can self-study, when necessary, through the contents of basic and required knowledge of specialized nursing practices, nursing techniques, and national-examination-related materials. Training materials were produced using multimedia techniques incorporating sound effects, animation, and illustrations. Contents that are organizing the training materials can be reorganized and reused for other digital sub-training materials, allowing the creation or addition of any new training materials without difficulty. Moreover, after downloading, they can be used to create a new digital nursing dictionary (Figure 3). In the other mobile method, Maag[3] introduced an emerging technology using by podcasting in nursing education.
4 Practice and Example in Nursing Education From 2006 to 2007, we hosted a series of lectures using training materials developed in from 2005 to 2007, and used portable multimedia terminals for on-site practice. Following are the ideas and opinions given to us after that on how to improve practices, as well as other useful advice. 4.1 Lectures Using Training Materials with Nursing Examples Instructors who joined in the creation of training materials actually used them in their classes. We asked them to report about how they used the training materials in their classes in addition to the responses they received from their students. To have them conduct such training in their classes, the study-support environment development team of the project organization supplied the training materials, computers and assemblies, and other materials to instructors who requested them. The execution evaluation team helped create questionnaires for students and the reporting formats to be used by the instructors. The actual lectures were intended mainly for third-year nursing department students (about 120–140) studying for work in five different nursing fields. Following are some examples of how training materials were used: i) Starting with the introduction of the patient starring in the training materials of examples on nursing (giving self-introduction narration by the patient), the patient subsequently presented scenes in which nursing problems are described; then the students were separated into work groups and each was expected to make a nursing care plan, and to present it before others, which were finally evaluated by the instructor with appropriate comments provided. ii) Introduction of some nursing examples and scenes of nursing care problems were posed. iii) Providing the students with some nursing care problems beforehand to use them in later work to be done by every group. Therefore, different approaches seem to have been taken by each instructor. Some opinions from the students are as follows: “It was easy to grasp the images of the examples,” “It was easy to read the patient’s emotions shown in the examples,” “It was realistic, so it was easy to understand,” “It was realistic and practical,” “It
154
Y. Majima et al.
was good from a visual perspective with the image pictures provided,” “It was easy to know the necessary information,” “It was good to hear the information given by voice,” and “I was able to understand the priority of the nursing care problems.” Additionally some students pointed out not only about how to collect information, but also about the methodology of how nursing-care assistants should recognize situations to move forward to create opportunities to educate patients. 4.2 On-Site Practice Support by Portable Multimedia Terminals We prepared portable multimedia terminals (PSP, Play Station Portable; Sony Corp.) loaded with training material contents prepared for the associated nursing practice areas, and lent them each to students who agreed to monitor them. Subsequently, we requested that they use them during the entire practice period. We requested them to give us opinions regarding the operability of the training materials and other information when they had to return the terminal. Following are some of the comments. Portable Multimedia Terminal 1. It was good to be able to find and view the necessary document without taking much time. 2. Because it is basically a game machine, its operability is good. 3. It was easy to use the training materials in this way. 4. The images in the training materials are useful, as if we were playing a game. 5. It is easy to carry because it is small and light. 6. The screen was easier to see than I thought. 7. The battery power duration time is too short. 8. It would be nice if there were a search function available everywhere in the training materials. Training Materials’ Contents 1. They were helpful in the practice class. It would have been nice if they had been available before. 2. What we wanted to see out of the training materials was restricted for our practice class. 3. It was very convenient to be able to see technically related pictures close at hand. 4. It was helpful to understand things using images. 5. We can see it anytime anywhere. Therefore, it is helpful when we study at home after the nursing technique practice is over.
5 Discussions After having had the training materials and the contents we produced used in the actual classes, we found that the students’ evaluations were good partially because it is something new to them. Nevertheless, their comments and opinions accurately reflected their experience, and helped us to determine the issues clearly and work continuously to correct and improve them in future versions of the system. Some
Development of Ubiquitous On-Demand Study Support Environment
155
instructors were not very happy with manipulation of information equipment, which underscores the importance of support them during use in actual classes. Through the process of having the training materials used in actual classes, we can continue to search for the most appropriate training materials. In so doing, we can review previous teaching methods and adapt and reconstruct them, if necessary. Additionally, using such training materials in actual classes, we can review the class proceedings and teaching methods the classes with the materials combined. Above all, the fact that the instructors felt that teaching a class was fun should create a positive impact on future educational activities resulting in leading to develop of our faculty.
6 Conclusions This report described our introduction of e-Learning in the School of Nursing at the Osaka Prefecture University (as the CanGo project), and reported comments and opinions given when we had the nursing practice example materials and sub-material contents we produced from 2005 to 2007 used for the actual classes from the first half of 2006 to 2007. In addition, we reported those comments and opinions given from the experiences we had when supporting the actual on-site practices. Future studies will further introduce e-Learning techniques based on the plan, along with regular and constructive review. We would be happy if the knowledge and experience we received were referenced and used anywhere else.
Acknowledgments This project was conducted through the aid of the Modern Education Needs Supporting Program of the year 2005 offered from the Ministry of Education, Culture, Sports, Science and Technology.
References 1. Kazuhisa, S., Yukie, M., Yoichiro, S.: Nursing task ontology based learning system redesign for enabling adaptive on-demand learning environment. In: Proceedings of the 3rd WSEAS/IASME International Conference on Engineering Education, Greece, pp. 114–119 (2006) 2. Kazuhisa, S., Yukie, M., Yoichiro, S.: Towards Building Adaptive On-Demand Learning Environment Based on Nursing Ontology. WSEAS Transaction on Advantages in Engineering Education, 563–570 (2006) 3. Margaret, M.: Podcasting: An emerging technology in nursing education. Studies in Health Technology and Informatics 122, 835–836 (2006) 4. Yukie, M., Yoichiro, S.: Development of E-learning for Problem solving Approach of Nursing Students. Studies in Health Technology and Informatics 122, 919 (2006) 5. Yukie, M., Yoichiro, S., Kazuhisa, S.: Framework for Problem-Solving Based Learning in Nursing Domain – An Experimental Study. Learning by Effective Utilization of Technologies: Facilitating Intercultural Understanding, pp. 625–628 (2006)
The Effects of Prior Knowledge on the Use of Adaptive Hypermedia Learning Systems Freddy Mampadi, Sherry Chen, and Gheorghita Ghinea School of Information Systems, Computing and Mathematics, Brunel University, Uxbridge, Middlesex, UB8 3PH, UK {Fred.Mampadi,Sherry.Chen,George.Ghinea}@brunel.ac.uk
Abstract. Prior knowledge and cognitive styles are considered important determinants in adaptive hypermedia learning systems (AHLSs) as they influence how learners select information to put into memory. However, there is a need to investigate how they influence learner performance and perceptions prior to comparing them and establishing if they can be used together to maximise learning in AHLSs. To this end, this study investigated the effects of prior knowledge on the use of AHLSs to set the foundation for the comparison. 60 students participated in this study. The results showed that, in general, adapting to prior knowledge improves learner performance and perceptions, especially for users with low prior knowledge. However, the results also indicated that the relative improvement in learner performance is significantly higher than that of perceptions when using the AHLS. The implications of the design of AHLSs are discussed by the paper. Keywords: Learning performance, Perceptions, Novice, Experts, Evaluation, Cognitive styles, Learning styles.
knowledge improves learner performance and/or perceptions, as per the previous empirical studies, prior to making comparison with systems that adapt to cognitive styles. The objective of this study is to, therefore, develop the adaptive hypermedia learning system that adapts to individuals’ prior knowledge and then compare it with an ordinary hypermedia system as an evaluation process to determine whether the adaptive version has a different effect from the ordinary version.
2 Related Research There have been a number of adaptive hypermedia systems that have been developed that takes into consideration, the user’s level of prior knowledge. These systems are normally grouped under Adaptive Educational Hypermedia Systems (AEHS), which are also known as Adaptive Hypermedia Learning Systems (AHLS). Examples of such systems are ELM-ART II [22], and InterBook [2]. The evaluation of the developed AHLSs mostly analysed one of the adaptive techniques without taking the totality of the system. That is, adaptation in an adaptive hypermedia system is driven by several adaptive techniques that work in conjunction to improve the learning of an individual. However, previous research (e.g. [2]) investigated adaptive navigation support alone while Boyle and Encarnacion [1] investigated adaptive presentation. Our research, however, combines the previously research adaptive techniques, also basing on the proposed frameworks (e.g. [7]) to develop an AHLS that could be tested against an ordinary hypermedia system that does not offer adaptivity. Also, according to the review of the literature, previous research focused on the improvement of learning performance as a determinant for the importance of prior knowledge in driving adaptation in AHLSs. A little research targeted the perceptions of use for such systems. Higher test scores were, in effect, transcribed as a representation for positive perceptions as several studies have indicated that students performed better in learning environments of which they had more positive perceptions [6,13]. To this end, there is a need to establish whether learning performance or perceptions (or both) are influenced by adapting to prior knowledge in AHLSs. We conducted a controlled experiment investigating the effects of prior knowledge on the use of AHLSs. Our experiment was based on AHLS developed with previous Table 1. Summary of preferences for low prior knowledge and high prior knowledge users (From [7])
• • • •
Low Prior Knowledge Perform better in hierarchical structure Need advance organizers and advisement Prefer guided navigation Prefer concept maps
• •
High Prior Knowledge Perform better in network structure Prefer free navigation
158
F. Mampadi, S. Chen, and G. Ghinea
research findings that combined the conceptual framework for designing AHLSs [7] and preferences of novices and experts in hypermedia environments (Table 1 and 2). The experiment was aimed at answering two research questions: (1) whether adapting a hypermedia learning system to an individual’s level of prior knowledge has an effect on learning performance; (2) whether adapting a hypermedia learning system to an individual’s level of prior knowledge has an effect on perceptions.
3 Experiment 3.1 Method Sixty participants took part in this experiment. In order to determine whether or not the AHLS was better than the ordinary hypermedia system, a between-subjects design was used. In other words, this meant that each student used either of the systems once but not both. The experiment was controlled. The same content was used for both systems without incurring the practice and fatigue effects in the experiment. Furthermore, each participant went through the same procedures in order to minimise bias. 3.2 Procedure Participants started with a pre-test which consisted of 19 questions before perusing the learning content (either Ordinary or AHLS). The AHLS used adaptive techniques in Table 2 to cater for novices and experts. When the participants felt they had completed the material, they followed a link to post-test before finishing the experiment by answering an exit questionnaire. The questionnaire gathered perceptions and attitudes of use. Table 2. Differences between novices’ and experts’ interfaces Adaptive Hypermedia Link hiding Adaptive layout Additional support Annotated Links
Rich links Alphabetic Index No advisements No annotations
4 Discussion of Results The sample mean scores for novices and experts using the ordinary hypermedia system were determined and treated as the expected means for the respective populations. The means for novices and experts using the adaptive hypermedia system were also calculated and then compared to the expected means of the population. In other words, the calculated means for the ordinary hypermedia system were compared to the calculated means for adaptive hypermedia system. For learning performance, an independent one-sample t-test was used. The onesample t-test compares the mean score of a sample to a known value. It measures
The Effects of Prior Knowledge on the Use of AHLS
159
whether a sample value significantly differs from a hypothesized value [18]. In our case the hypothesized value is the mean calculated scores resulting from the use of the ordinary hypermedia system. The sample mean is calculated from results drawn using the adaptive hypermedia system. The resultant two means can, therefore, be compared. The means of gain scores and post-test scores were used to determine the differences in learning performances. To make a comparison between the perceptions of those students that used the ordinary hypermedia system with the ones that used the adaptive hypermedia system that adapts to their levels of prior knowledge, we employed the independent onesample t-test as used in the analysis of learning performance. The means of certain questions from the exit questionnaires tailored to novices and experts using the ordinary interface were calculated, and treated as the hypothesised mean for the population, which was then compared to the means for the adaptive hypermedia version using the one-sample t-test. The statistical analysis, to make all the comparisons on the data, was conducted using Statistical Package for the Social Sciences (SPSS) for Windows version (release 15.0). A significance level of p < .05 was adopted for the study. 4.1 Learning Performance Analysis of learning performance was measured using the post-test scores and the gain scores (post-test score minus the pre-test score) for novice students using the ordinary hypermedia and the adaptive hypermedia versions. The ordinary version was used to hypothesise the scores in order for the mean to be compared with that of the adaptive version. The results from the one-sample t-test are summarised in Table 3. With respects to post-test score means, the results show that there was a significant difference in learning performance between users of the ordinary hypermedia version and the adaptive version (t (21) = 3.465, p = .002). In other words, the average posttest score of novices using the ordinary hypermedia version (M = 62.65, SD = 11.875) is significantly different from the novice using the adaptive hypermedia version (M = 72.23, SD = 12.965). The result shows that there was significant improvement in posttest scores when novice students used the adaptive hypermedia that was adapted to their levels of prior knowledge. The results of the gain scores indicate that there was a significant difference in learning performance between novices of the ordinary hypermedia and adaptive hypermedia versions (t (21) = 2.590, p = .017). In other words, the average gain score Table 3. Summary of the learning performance results (one-sample t-test) Novice Variable Po st-test Score Ga in Score
Mean (SD) M ean (SD)
Expert
Ordinary Hypermedia
Adaptive Hypermedia
Ordinary Hypermedia
Adaptive Hypermedia
62.65 (11.875)
72.23 (12.965)
73.40 (7.542)
74.61 (7.678)
35.42 (14.396)
43.09 (13.894)
10.32 (11.235)
15.33 (8.485)
160
F. Mampadi, S. Chen, and G. Ghinea
of novices using the ordinary hypermedia version (M = 35.42, SD = 14.396) was significantly different from that of novice using adaptive hypermedia version (M = 43.09, SD = 13.894). The result shows that there was significant improvement of gain score when novice students used the adaptive hypermedia system that was adapted to their levels of prior knowledge. With respects to expert users, the one-sample t-test indicate that there was no significant difference in mean post-test scores between users of the ordinary hypermedia and adaptive hypermedia versions (t(17)=.625, p = .540). This means that the average performance of experts, with respect to mean post-test scores, using the adaptive hypermedia version (M = 74.61, SD = 7.678) was not significantly different from that of experts using the ordinary version (M = 73.40, SD = 7.542). The results, however, showed that with respect to gain scores, there was slightly significant difference between users of the ordinary hypermedia and adaptive hypermedia versions (t(17)=2.507, p = .023). This means that the average performance of experts, with respect to mean gain scores, using the adaptive hypermedia version (M = 15.33, SD = 8.485) was significantly higher than that of experts using the ordinary hypermedia version (M = 10.32, SD = 11.235). In this respect, this result shows that the adaptive hypermedia system that adapt to individuals’ prior knowledge can improve students’ learning performance. These results are consistent with the majority of reported studies (e.g. [9,17]). 4.2 Perceptions and Attitudes To analyse the perceptions and attitudes of novice and expert users towards adaptive hypermedia that adapts to their levels of prior knowledge, we analyses users’ responses from the questionnaires. More specifically, a comparison is conducted between the results of the questionnaire from the ordinary hypermedia system and those from the adaptive hypermedia system. A one-sample t-test was used to determine if there was any positive improvement in perceptions and attitudes when students used the adaptive hypermedia version. The summary of results is shown in Table 4. Table 4. Summary of results for perceptions (one-sample t-test) Novices Statements
Experts
Ordinary Hypermedia
Adaptive Hypermedia
Ordinary Hypermedia
Adaptive Hypermedia
I felt the structure of this tutorial is not clear
Mean (SD)
2.57 (1.025)
1.77 (0.922)
2.16 (0.879)
1.72 (0.904)
I found the sequence of topics was logical It is hard to find a route for a specific topic in this tutorial Sometimes I found it hard to keep track which bits I had learnt.
Mean (SD)
3.01 (0.950)
4.41 (0.596)
3.42 (0.982)
4.44 (0.602)
Mean (SD)
3.68 (0.875)
1.77 (0.923)
3.07 (0.987)
2.50 (0.930)
Mean (SD)
3.64 (0.822)
2.77 (0.922)
2.46 (0.932)
2.06 (0.994)
The Effects of Prior Knowledge on the Use of AHLS
161
Students’ views on content structure. The adaptive hypermedia version was designed such that it offered overall a topic map for students with low prior knowledge and a topic index for students with high prior knowledge of XML. In the map version (used by novices), the content was organised into three levels, with topics, subtopics and pages, presenting a global picture of the entire hierarchical structure. In the index version (used by experts), the content was organised alphabetically with all of the nodes covering the topics, subtopics and pages in a network structure. The ordinary hypermedia system did not employ adaptive hypermedia techniques. In comparing the ordinary version and the adaptive version basing on the structural differences, the results indicated that there were significant differences in attitudes and perceptions between the groups. Two questions were extracted from the questionnaire to determine the views of students using the adaptive hypermedia system. The response was in comparison to those students that used the ordinary hypermedia version. With respect to novices, the results of the independent on-sample t-test (Table 4) indicate that there was a significant difference in perceptions and attitudes towards structure of the tutorials between users of ordinary hypermedia system and adaptive hypermedia version, t(21)=-4.055, p = .001. This means that students that used the adaptive hypermedia version (M = 1.77, SD = 0.922) found the structure of the tutorial to be clearer as compared to those that used the ordinary hypermedia version (M = 2.57, SD = 1.025). The results of the one-sample t-test also indicate that there was a significant difference in perceptions and attitudes towards sequence of topics between users of the ordinary hypermedia XML tutorial and adaptive hypermedia version, t(21) = 11.116, p = .000. This means that the students that used the adaptive hypermedia version (M = 4.41, SD = 0.596) found the sequence of topics to be more logical than those that used the ordinary hypermedia version (M = 3.01, SD = 0.950). The same questions were posed to experts to determine their views on structure of their respective tutorials. There were significant differences in attitudes and perceptions between students that used the adaptive hypermedia version and those that used the ordinary hypermedia version. The one sample t-test results, t(17) = 2.470, p = .024 shows that in terms of clarity of structure experts users using the adaptive version (M = 1.72, SD = 0.904) found it slightly clearer than those using the ordinary hypermedia version (M = 2.16, SD = 0.879). A significant difference was also found when experts were asked about how they felt about the logical sequence of topics. As showed in Table 4, the results (t(17) = 8.500, p = .000), show that the experts students that used the adaptive hypermedia version (M = 4.44, SD = 0.602) found the sequence of topics to be more logical as compared to those that used the ordinary hypermedia version (M = 3.42, SD = 0.982). In summary, with respect to structure, the results of this investigation support prior research [15,19] that indicate that adaptive hypermedia systems that adapt the structure of the content to learners’ prior knowledge improve learning performance and are perceived as useful and enjoyable. Students’ views in terms of navigational structure. In terms of getting lost using the tutorial, novice users using the adaptive hypermedia version enjoyed using their version while their counterparts using the ordinary hypermedia version felt frustrated probably because they did not have a suggested route (implemented through the
162
F. Mampadi, S. Chen, and G. Ghinea
pre-requisite structure of the tutorial) to follow through the learning system (Table 4), hence prone to getting lost. The results of the independent one-sample t-test indicate that there was a significant difference in perceptions and attitudes towards getting lost using the tutorials between users of the ordinary hypermedia and adaptive hypermedia versions (t(21)=-11.010, p = .000). This means that students that used the adaptive hypermedia version (M = 1.77, SD = 0.923) found it easy to find a route for a specific topic in the tutorial as compared to those that used the ordinary hypermedia version (M = 3.68, SD = 0.875). The problem of navigation and getting lost was not as significant for experts as was for novices. However, experts using the adaptive hypermedia version felt that the navigational structure was better as compared to those that used the ordinary hypermedia system which did not have an index. As showed in Table 4, (t(17) = 2.202 , p = .042), experts found it slightly easier to find a specific topic when using the adaptive hypermedia version (M = 2.50, SD = 0.930) than the ordinary hypermedia version (M = 3.07, SD = 0.987). The results of the experiment support the findings of previous studies that matching individuals with their preferred navigational features improves the likeability of a hypermedia system. The studies have evaluated the effectiveness of different navigation tools for high and low prior knowledge users. They found that these tools influence users’ achievement and attitudes (Chen et al., 2006; Farrell and Moore, 2001). The adaptive hypermedia version of the tutorial used in this experiment used the same principles to adapt navigational structure to novice and experts. It used a map for novice students and an index for experts. Students’ view in terms of additional support. To provide additional support for novices the adaptive hypermedia system uses different colours for hyperlinks and the list of pages read and those that still need reading. Novice students using the ordinary hypermedia version (M =3.64, SD = 0.822) found it harder to track which bits of information they have learnt as compared to novice students that used the adaptive hypermedia version (M = 2.77, SD = 0.922) of the tutorial as shown in Table 4 (t(21) = -4.411, p = .000). For the experts, there was no significant difference between those who used the ordinary hypermedia version (M = 2.46, SD = 0.932) as compared to the ones that used the adaptive hypermedia version (M = 2.06, SD = 0.994). The results show that for t(17) = -1.966, p = .066. This may be due to the fact that both experts using the two systems did not find it hard to keep track of what bits the have learnt. A possible reason for this finding may be that they have the ability to restructure the material because they are already familiar with the basic concepts [14]. The results of the experiment are in support of previous research which shows that additional support when provided can help learners with a low level of prior knowledge in hypermedia learning, particularly in free navigation conditions [7,14]. Advisement, which provides learners with visual aids and recommended navigation paths, is helpful in preventing disorientation in non-linear hypermedia learning. As novice learners cannot rely on their prior knowledge to help them structure the text,
The Effects of Prior Knowledge on the Use of AHLS
163
graphical overviews and structural cues are powerful and beneficial in providing navigation guidance so as to ease disorientation problems[7].
5 Conclusions Two research questions were examined in this study: (1) whether adapting a hypermedia learning system to an individual’s level of prior knowledge has an effect on learning performance; (2) whether adapting a hypermedia learning system to an individual’s level of prior knowledge has an effect on perceptions. The answer to the first research question is that an adaptive hypermedia learning system that adapts to individuals’ levels of prior knowledge improves their learning performance and therefore affects it. The answer to the second research question is that an adaptive hypermedia learning system that adapts to individuals’ levels of prior knowledge affects perception of use. This study demonstrated that the adaptive hypermedia learning system is more beneficial than ordinary hypermedia learning system, especially for users with low prior knowledge. The study also supports the results of previous research which also emphasises on the usefulness of adaptive hypermedia learning systems tailored to learners’ prior knowledge. The study revealed, however, that not everybody benefits from the adaptive hypermedia learning systems, especially users with a high level of prior knowledge. A possible reason for this finding may be that they have the ability to restructure the material because they are already familiar with the basic concepts. Therefore, the major challenge lies with improving the learning performance and perceptions of high prior-knowledge users with adaptive hypermedia systems that adapt to their levels of knowledge. With respect to perceptions, the study has demonstrated that although students had more positive perceptions using the adaptive version over the ordinary version, there was a slight difference between experts using the adaptive version and those using the ordinary version. The study also revealed that, in comparing the results of learning performance and perceptions on the adaptive version, learning performance scores seemed to be slightly more significant than those of perceptions. That is, students seem to do well in terms of test scores but not showing as much positive attitudes towards the interfaces provided to them. Hence, this study recommends that interfaces design incorporate mechanisms that are dedicated to improving learner performance like pre-requisite structure driven by a learner model. However, there should be an investigation on how to improve perceptions of use for such systems, by analyzing the importance of cognitive styles. This type of design could maximize performance of learners using adaptive hypermedia systems. There is also a need to incorporate both prior knowledge and cognitive styles and determination of the right blend to maximise learning, in adaptive hypermedia systems design could then be established, with its associated pros and cons. This was a small-scale study, hence we recommend that further studies be undertaken with a larger sample to provide additional evidence. We believe that such evidence will not only help to improve the design of adaptive hypermedia learning systems, but also will also be useful for the development of other personalized applications, such as e-commerce systems and virtual training environments.
164
F. Mampadi, S. Chen, and G. Ghinea
References 1. Boyle, C., Encarnacion, A.O.: MetaDoc: An Adaptive Hypertext Reading System. User Modeling and User-Adapted Interaction 4(1), 1–19 (1994) 2. Brusilovsky, P., Pesin, L.: Adaptive navigation support in educational hypermedia: An evaluation of the ISIS-Tutor. Journal of Computing and Information Technology (1998) 3. Brusilovsky, P.: WebEx: Learning from examples in a programming course. In: Fowler, W., Hasebrook, J. (eds.) Proceedings of WebNet 2001, World Conference of the WWW and Internet, Orlando, FL, AACE, October 23-27, 2001, pp. 124–129 (2001) 4. Brusilovsky, P., Eklund, J., Schwarz, E.: Web-based education for all: A tool for developing adaptive courseware. Computer Networks and ISDN Systems 30(1-7), 291–300 (1998) 5. Caramel, E., Crawford, S., Chen, H.: Browsing in Hypertext: A Cognitive Study. IEEE Transactions on Systems, Man, and Cybernetics 22(5), 865–883 (1992) 6. Chen, S.Y.: A cognitive model for non-linear learning in hypermedia programmes. British Journal of Educational Technology 33, 449–460; Ghinea, G., Chen, S.Y.: The impact of cognitive styles on perceptual distributed multimedia quality. British Journal of Educational Technology 34, 393–406 (2003) 7. Chen, S.Y., Fan, J., Macredie, R.D.: Navigation in Hypermedia Learning Systems: Experts vs Novices. Computers in Human Behavior 22(2), 251–266 8. Cleary, C., Bareiss, R.: Practical Methods for Automatically Generating Typed Links. In: The Proceedings of the Seventh ACM Conference on Hypertext, Washington, DC (1996) 9. Dochy, F.: Assessment of prior knowledge as a determinant for future learning. Lemma BV/Kingsley Publishers, Ultrecht (1992) 10. Farrell, I.H., Moore, D.M.: The effect of navigation tools on learners’ achievement and attitude in a hypermedia environment. Journal of Educational Technology Systems 29, 169–181 (2001) 11. Ferguson, W., Bareiss, R., Bimbaum, L., Osgood, R.: ASK systems: An approach to the realization of story-based teachers. The Journal of the Learning Sciences 2, 95–134 (1992) 12. Ford, N., Miller, D.: Gender differences in Internet perception and use. In: Electronic Library and Visual Information Research. Papers from the third ELVIRA conference, April 30, 1996, pp. 87–202. ASLIB, London (1996) 13. Fullerton, K.: The interactive effects of field dependence- independence and Internet document manipulation style on student achievement from computer-based Instruction. Ed.D Dissertation. University of Pittsburgh (2000) 14. Last, D.A., O’Donnell, A.M., Kelly, A.E.: Using hypermedia: Effects of prior knowledge and goal strength. In: The annual meeting of the Society for Information Technology & Teacher Education, Washington, D.C (1998) 15. McDonald, S., Stevenson, R.J.: The effects of text structure and prior knowledge of the learner on navigation in hypertext. Human Factors 40, 18–27 (1998) 16. Mitchell, T.J.F., Chen, S.Y., Macredie, R.D.: Hypermedia learning and prior knowledge: domain expertise vs. system expertise. Journal of Computer Assisted Learning 21, 53–64 (2005) 17. Moerkerke, G.: Assessment for flexible learning. Lemma, Utrecht (1996) 18. O’Mahony, M.: Sensory adaptation. J. Sensory Studies I, 237–258 (1986) 19. Shin, E., Schallert, D., Savenye, W.: Effects of learner control, advisement, and prior knowledge on young students’ learning in a hypertext environment. Educational Technology, Research and Development 42(1), 33–46 (1994)
The Effects of Prior Knowledge on the Use of AHLS
165
20. Spyridakis, J.H., Isakson, C.S.: Hypertext: a new tool and its effect on audience comprehension. In: IEEE-IPCC 1991, pp. 37–44 (1991) 21. Weber, G., Specht, M.: User modeling and adaptive navigation support in WWWbased tutoring systems. In: Jameson, A., Paris, C., Tasso, C. (eds.) User Modeling: Proceedings of the Sixth International Conference, UM 1997, pp. 289–300. Springer, Vienna (1997) 22. Weber, G., Brusilovsky, P.: ELM-ART an adaptive versatile system for web-based instruction. International Journal of Artificial Intelligence and Education, this volume (2001)
Supporting Learners in Adaptive Learning Environments through the Enhancement of the Student Model Luca Mazzola and Riccardo Mazza University of Lugano, Faculty of Communication Sciences Institute for Communication Technology Via Buffi 13, CH-6904 Lugano, Switzerland Ph.: +41 58 666 4674; Fax: +41 58 666 4647 {luca.mazzola,riccardo.mazza}@lu.unisi.ch
Abstract. This positional paper presents our research aimed at finding some possible research directions towards the enhancement of the use of open student models in the field of Technology Enhanced Learning and Adaptive Systems. Starting from the historical evolution of the learner model, we will describe some possible uses of learner models and propose some possible directions of enhancement. We will present 6 possible directions of research, and 11 dimensions on analysis. The 6 directions have been evaluated against the dimensions, and tentative ranking has been proposed. The result of this analysis will guide the work on open learner models which will be undertaken in the context of the European Union funded project GRAPPLE [1] aimed at building an infrastructure for adaptive learning systems that will adopt the strategy of opening learner models to the course learners and instructors Keywords: Technology Enhanced Learning, independent Open Learner Model, Human Computer Interaction, adaptation.
Supporting Learners in Adaptive Learning Environments
167
This paper aims to explore the historical evolution of the student model, to identify the different approaches to model the level of knowledge of the learners and, based on this analysis, we will propose some possible directions of research on student models to support learners and encourage reflection in adaptive learning environments. The motivation behind this research is that adaptive learning systems have been found to be useful in engaging the learner more in the educational experience [6]. The necessity to offer a higher level of commitment with the learning environment, especially when the most engaging human aspects inside these platforms are not so evident, became one of the key success factors for the wide diffusion of computerbased learning. A large part of the research in the field of adaptive learning environments focuses mainly on providing the learner with a customized learning environment that adapts the content of the course to the user's knowledge and preferences [7]. These systems don’t usually allow for the fact that the learner, besides being a user of the learning platform, is part of a “community of learners” (i.e. the class). A number of researches on social aspects of learning [8] stress the importance of the social interactions that occur during the learning processes. Our proposal aims to contribute to the research on adaptive systems by exploring some possible ideas to extend the adoption of user models. In this view, the open student model is not only an internal component of an adaptive system, but is also a useful source of information that can be stressed to enhance the user’s commitment to the online learning experience. The next section begins with an historical analysis of the state of the art in the field of student models. We have identified 6 directions of research that could be explored in the research of learner models with the aim of encouraging the learner to inspect his/her model, and promote reflection as learning. Then an analysis, driven by a rating system based on 11 characteristics, proposes the most promising idea which will be investigated in the context of the EU-funded project GRAPPLE [1].
2 Modeling Learners in Technology Enhanced Learning Environments Our attention is drawn to the field of Technology Enhanced Learning environments. We believe that it is important to improve the user commitment and the effectiveness along the whole learning process undertaken by using these learning environments, while also considering the importance of education for humanity and the progress of civilization and science. We consider the application of adaptive features in learning systems particularly well-suited to achieving this goal. From the literature, we have identified four different approaches to the use of student models in educational systems, which are listed in the following sections. 2.1 Internal Models This approach, also known as “close models”, was developed to build an internal representation of the learners’ progress with the course, mainly to implement adaptive features. Research has been conducted on internal users and students models for a long time, with the aim of profiling the user in different domains and applications.
168
L. Mazzola and R. Mazza
Internal modeling procedures collect data from the users' actions and feedback, store it in an internal database, and provide the input data that can be used by the adaptive functionalities for user data reasoning. The modeling process is quite complex and involves different steps: deciding and structuring the information to be collected, collecting raw data, scrubbing, extracting new information with reasoning rules, and, if necessary, updating existing data. Internal models are built for system internal use only, as a black-box component: neither students nor instructors have the possibility of exploring the contents of the student model. 2.2 Open Models The idea of either partially or totally opening learner models to the learner for inspection was developed mainly to promote student self-reflection and awareness of the adaptive features of the learning environment. The term “scrutable user models” was introduced by Judy Kay in 1999 [9]. The notion of scrutability is related to the possibility of the learners to scrutinize the model to see, not only what information the system holds about them, but also the process used by the system to collect the data about the learners and the inferences based on that data. It must be noted that scrutability concerns inherently convey a complementary but different view on personalization, which stresses upon the learner's awareness of the personalization process he/she is committed to. Recent insistence on scrutability or “inspectable open learner models” [10] advocate for explicit communication to learners of the pedagogical aspects framing the personalized learning experience designed for them by an adaptive learning technology. Student models can also be opened to peers, but this raises new problems, such as privacy, control over personal data, and trustworthiness of the system. One possible approach to deal with some of these problems presented is to distinguish between friendship networks and peers' groups. Users can decide which part of their profile to release in a named (or anonymous) form, and who is authorized to access this data representation (peer models). It could be observed that opening the models to peers can foster collaboration (with friends) and competition (with peers) [11]. 2.3 Group Models The recent achievement of the social network centrality into the social constructivist theory provides reasons to investigate models that take social and group aspects into account. Group modeling is a recent field of research, in which the learners are modeled as a group instead of a set of single individuals. These model the characteristics of an identified group of learners, and aim also to present the position and the relative distance of profiles, in order to allow learners to compare and understand their own situations. Opening group models to the users may offer some advantages. It can help learners to reflect on their progress in the group context and understand the problems of other group members [12]. Group models have been used to support the collaboration between learners of the same group, and to foster competition in a group of learners [13]. Right now, only simple methods have been used to mine the group models. The most common is using the average individual values representing a particular aspect considered in the model.
Supporting Learners in Adaptive Learning Environments
169
2.4 Interactive Models Interactive models introduce some levels of interactivity that can be expressed in two ways: the first, called inspection methods, is the capability to manipulate the graphical representation of the model, change the appearance, apply filters to the visualization and thus achieve a more in-depth understanding of the model. The other option, called interactive methods, is the possibility to influence the model, both by changing the data used by the system to represent the user itself, or by challenging and convincing the system that the current profile doesn't represent the learner status well. In this case, one problematic aspect could be the verification of the real understanding that the user has of the model, and which data support the claim for changing the internal profile according to the user request. Some systems tackle this problem by using a challenging process: they ask learners to solve a problem related to the particular aspect that should be changed and decide if this modification will be done based on the performance reached by the learner in this process. In the next section we will take the most interesting aspects from each of the previously described modeling approaches into account, and will investigate how they could be extended with input from other fields to provide the user with a better experience.
3 Extending the Use of Open Learner Models Traditionally, research on Open Learner Models (OLM) has been carried out by scholars active in adaptive systems and educational technologies. This is evident as the main OLM applications are in adaptive systems. However, we believe that contributions from other fields of research could be beneficial to extend and improve the adoption of OLM. For instance, one of the key issues in OLM is how to graphically represent the model; techniques from Information Visualization could provide indications on how to select and encode data in a graphical format suited to the representation task of the user. Another aspect is how to aggregate information and find correlations between data; techniques from artificial intelligence could help in this aspect. Other useful works carried out in different educational research could be adopted in student models. For example, Glahn et al. [14] propose the adoption of “smart indicators” as a way to aggregate user model information in a compact and intuitive way through a visual indicator that draws attention to ongoing relevant events only when really necessary. Erickson & Kellogg [15] at IBM T.J. Watson Research Center propose the “social translucence” idea as an approach to designing systems that support social processes: socially translucent systems are digital systems that “support coherent behaviour by making participants and their activities visible to one another and they have three main characteristics—visibility, awareness, and accountability—which enable people to draw upon their experience and expertise to structure their interactions with one another” [15, p.59]. More specifically, works from other disciplines have led us to identify some possible directions of research for OLM: P1
Positioning the learner with respect to the class or to a group of learners. This aims to use the OLM as a tool for supporting socialization, by making the student aware of the social context in which the learning experience is
170
L. Mazzola and R. Mazza
taking place, as suggested by the social translucence theory of Erickson & Kellogg [15]. P2
The introduction of innovative graphical interfaces. Contributions for Information Visualization [16] may help in investigating how to design innovative graphical interfaces, in order to reduce their complexity and the cognitive load required to interpret the underlying data, and adopt interaction techniques that enable the exploration of different aspects of user data.
P3
The representation of the temporal evolution of the model. The student model is not static, it changes as the student performs any action in the learning environment. A research work could explore whether opening this dynamic evolution of the learner model could be beneficial to the students (or to the instructors).
P4
The use of adaptive representations of OLM. An initial attempt to introduce some of the ideas of adapting the graphical representation of OLM comes from the work on “smart indicators” by Glahn et al. [17]. The idea is that the introduction of adaptive features in the graphical representation could help students to perceive the OLM in a more meaningful way. This allows the production of more engaging and effective external representations of OLM, and could provide a better experience to the users of the learning applications.
P5
The exploration of techniques that allow the definition of a global student model, by integrating different independent student models from different courses. Traditionally, student models keep track of the learner's knowledge and the skills acquired during the learning process on a course basis. Although a learning platform may run different courses, and students can enroll in several courses, each course has an independent student model. This is primarily due to practical, as opposed to theoretical, reasons: it is very hard to define a global ontology that models the concepts for several courses.
P6
The introduction of a metric, a function that defines a “distance” between students, based on the data stored in the learner model. This also allows the identification of groups of students having similar profiles.
Proposals P1, P2, P3, and P4 deal with an externalization of one or more aspects of the learner model, where the information is related to a specific course. P5 and P6 deal, instead, with more complex subjects: the creation of a global model of the user (P5) and the definition of a metric to measure the distance on learner models (P6) are very complex tasks for which research is still in the early stages. P5 and P6 are possible through data mining techniques, that work mainly with student tracking data (logs) to extract relations between data and can work even if a global model of the domains of the courses is missing [18]. In order to investigate how the research directions specified above could be effective in the learning process, we have identified a number of dimensions of analysis. These dimensions take into account the level of difficulty in implementing
Supporting Learners in Adaptive Learning Environments
171
the idea, the difficulty in managing the new functionalities proposed, the expected impact at the system level, and the estimated benefits offered to the student in terms of the metacognitive skills that can be acquired. The dimensions that we have considered are: D1 To what extend does this research enhance socialization among students? D2 How much effort is required for the knowledge extraction and for reasoning over the data? D3 The computational complexity needed to maintain the model. D4 The difficulties in identifying one or more metrics. D5 The granularity of representation of the problem space (continuous, stepped or discrete) [16]. D6 The amount of data required to have a reliable model. D7 The difficulty in identifying the most useful data to collect and the level of aggregation. We also propose other dimensions of analysis, which are more related to the experience, interaction, and mental model of the learner: D8 The novelties and the benefits introduced by the new graphical interface. D9 The impact on the cognitive load of the learner caused by the new information presented and the new way of presentation. D10 The complexity of the rules that drive the creation of the model and the speed of their convergence into a stable state. D11 The level of interactivity and the interaction type (continuous, stepped, passive or composed) [16].
4 Analysis of Proposed Dimensions To investigate which dimension seems useful for exploration in our EU project Grapple, we have created an evaluation space and divided it into 4 areas. Each area represents an aspect that we would like to stress: the social aspects that can be influenced, the innovation in the graphical user interfaces that a research approach may bring, a representation of the temporal evolution, and the clustering of data that comes from learner models. The first step was an identification on the 4 areas of the 6 proposed directions of research (see Fig. 1). In order to identify which proposals among P1 ... P6 are the most promising and worth investigating, we decided to set up a ranking system based on the 11 dimensions (D1 ... D11) defined previously. These dimensions were used to rate every proposal according to a 5 level scale (from a to e). Since some dimensions describe positive aspects (such as enhancing socialization, D1) and others denote problematic aspects (such as the computational complexity needed to maintain the model, D3), we divided them into two groups (D1, D5, D8, D11 and D2, D3, D4, D6, D7, D9, D10) and rated them using a direct scale (e is the highest value in the scale)
172
L. Mazzola and R. Mazza
Fig. 1. The projection of the 6 proposals on a 4-area evaluation-space
for the first group and a negative one (a is the highest value in the scale) in the second. Then, each proposal P1, ..., P6 has been matched with each dimension D1, ..., D11 and the match has been empirically assigned with a value in the scale (a ... e) by a team of experts in educational technology at the laboratory of eLearning at the University of Lugano. The result of this exercise is illustrated in Table 1. This table is also used to derive a sort of ranking among the different proposals (columns Pos). Table 1. Matching proposals against dimensions D1
D5
D8
D11
Pos
D2
D3
D4
D6
D7
D9
D10
Pos
Tot
P1
a
c
d
e
3
b
c
a
b
c
c
d
2
2
P2
e
a
e
e
6
a
b
a
a
d
c
e
1
3
P3
d
d
c
e
5
d
d
d
d
e
e
e
6
6
P4
e
a
a
b
1
d
b
a
c
c
b
d
2
1
P5
c
c
d
c
3
d
e
d
e
c
e
d
4
3
P6
b
a
d
c
2
e
d
e
e
c
e
c
5
3
The data shown on Table 1 has been visually encoded into a star plot graphical representation [19]. From the visualization depicted in Fig. 2, according to the findings on dimensions defined in Table 1, the proposal P4 (The use of adaptive representations of OLM) and P1 (Positioning the learner with respect to the class or to a group of learners) appears to be the most promising. After this analysis, we decided to concentrate our work on the direction of introducing graphical interfaces adapted to the learners' characteristics stored in the learner model. Moreover, we want to integrate some support for social aspects in our work, such as the positioning of learners in the class or group (P1). The final two proposals (P5 and P6) are very interesting from the point of view of possible
Supporting Learners in Adaptive Learning Environments
173
Fig. 2. Star plot of analysis dimensions for every proposal
outcomes in the user experience, pose a great number of open issues and will be classified as possible research extensions of this work.
5 Conclusions We have analyzed some possible research directions towards the enhancement of the use of open learner models in the field of Technology Enhanced Learning and Adaptive Systems. The aim of this study was to identify, among a list of possible research directions, the most promising according to a list of 11 dimensions. The proposals of research have been evaluated against the dimensions, and a tentative ranking has been proposed. We located the most promising and plan to explore them in the context of the EU-funded project GRAPPLE. A further aim of this paper was to stimulate the cooperation of researchers from different disciplines. The research directions proposed in this paper had the specific purpose of collecting feedback and asking other researchers to share their experiences and to foster collaboration.
174
L. Mazzola and R. Mazza
We believe that this work could help, as an initial seed, to lead to a self-analysis by researchers into the improvements that could bring Personal Learning Environments to a new level of commitment and awareness in the users' experience.
Acknowledgments This work was supported by the 7th Framework Program European project GRAPPLE ('Generic Responsive Adaptive Personalized Learning Environment'). The first author wishes to thank all of his colleagues and the coordinators of the Ph.D. School Red-Ink at the University of Lugano, as well as the researchers involved in the NewMinE lab and the eLab at USI for the interesting and useful discussions and ideas provided.
References 1. GRAPPLE project, http://grapple-project.org 2. Esposito, F., Licchelli, O., Semeraro, G.: Discovering Student Models in e-learning Systems. Journal of Universal Computer Science 10(1), 47–57 (2004), http://dx.doi.org/10.3217/jucs-010-01-0047 3. Brusilovsky, P., Peylo, C. (eds.): Adaptive and intelligent Web-based educational systems. International Journal of Artificial Intelligence in Education 13(2-4) 4. Graf, S., Kinshuk: Learner Modelling Through Analyzing Cognitive Skills and Learning Styles. In: Adelsberger, H., Kinshuk, P., Pawlowski, J., Sampson, D. (eds.) Handbook on Information Technologies for Education and Training, 2nd edn., pp. 179–194. Springer Publishing Company, Incorporated, Heidelberg (2008) 5. Bull, S.: A Simple Student Model to Make Students Think. In: Jameson, A., Paris, C., Tasso, C. (eds.) User Modelling Proceedings from 6th International Conference, UM, pp. 315–326 (1997) 6. Conlan, O., O’Keeffe, I., Brady, A., Wade, V.: Principles for Designing Activity-based Personalized eLearning. In: IEEE International Conference on Advanced Learning Technologies (ICALT 2007), pp. 642–644 (2007) 7. Dolog, P., Henze, N., Nejdl, W., Sintek, M.: Personalization in distributed e-learning environments. In: Proceedings of the 13th international World Wide Web Conference on Alternate Track Papers &Amp; Posters, WWW Alt. 2004, pp. 170–179. ACM, New York (2004) 8. Dillenbourg, P., Fischer, F.: Basics of Computer-Supported Collaborative Learning. Zeitschrift für Berufs- und Wirtschaftspädagogik. 21, pp. 111–130 (2007) 9. Kay, J.: A scrutable user modelling shell for user-adapted interaction, PhD Thesis, Basser Department of Computer Science, University of Sydney, Australia (1999) 10. Bull, S., Nghiem, T.: Helping Learners to Understand Themselves with a Learner Model Open to Students, Peers and Instructors. In: International Conference on Intelligent Tutoring Systems 2002 - Workshop on Individual and Group Modelling Methods that Help Learners Understand Themselves (2002) 11. Bull, S., Mabbott, A., Abu-Issa, A.: UMPTEEN: Named and Anonymous Learner Model Access for Instructors and Peers. International Journal of Artificial Intelligence in Education 17(3), 227–253 (2007)
Supporting Learners in Adaptive Learning Environments
175
12. Vassileva, J.I., Greer, J.E., McCalla, G.I.: Opennes and Disclosure in Multi-agent Learner Models. In: Proceedings of the Workshop on Open, Interactive, and Other Overt Approaches to Learner Modelling at AIED 1999, Lemans, France (1999) 13. Vassileva, J.: Open Group Learner Modeling, Interaction Analysis and Social Visualization. In: Dimitrova, V., Tzagarakis, M., Vassileva, J. (eds.) Proceedings of Workshop on Adaptation and Personalisation in Social Systems: Groups, Teams, Communities. Held in conjunction with UM 2007, Crete (2007) 14. Glahn, C., Specht, M., Koper, R.: Reflecting on Web-readings with Tag Clouds. In: Conference Paper, Computer-based Knowledge & Skill Assessment and Feedback in Learning Settings (CAF) Special Track at the 11th International Conference on Interactive Computer aided Learning (ICL 2008), September, 24-26, 2008, Villach, Austria (2008) 15. Erickson, T., Kellogg, W.A.: Social translucence: using minimalist visualizations of social activity to support collective interaction. In: Höök, K., et al. (eds.) Designing information Spaces: the Social Navigation Approach, pp. 17–41. Springer, Berlin (2003) 16. Spence, R.: Information Visualization: design for interaction, 2nd edn. Pearson Education/Prentice Hall, Harlow (2007) 17. Glahn, C., Specht, M., Koper, R.: Smart indicators to support the learning interaction cycle. International Journal of Continuing Engineering Education and Lifelong Learning 18(1), 98–117 (2008) 18. Minaei-Bidgoli, B.: Data Mining for a Web-Based Educational System. Doctoral Thesis. Michigan State University (2005) 19. Mazza, R.: Introduction to Information Visualization. Springer, London (2009)
The Concept of IMPRESSION: An Interactive Instruction System and Its Practice for Real-Time Distance Lessons between U.S. and Japan Takashi Mitsuishi1, Fumiko Konno1, Yuki Higuchi2, and Kentaro Go3 1
Graduate School of Educational Informatics, Tohoku University Kawauchi 27-1, Aoba-ku, Sendai, Japan {takashi,fumiko}@ei.tohoku.ac.jp 2 PRO & BSC Corporation Honcho 2-1-8, Aoba-ku, Sendai, Japan [email protected] 3 Interdisciplinary Graduate School of Medicine and Engineering,University of Yamanashi Takeda 4-4-37, Kofu, Japan [email protected]
Abstract. In order to perform flexible and effective lesson, we proposed “Double Loop Instructional Design Process Model” and developed an interactive instruction system; IMPRESSION based on the proposed model. In this paper, we show the concept of IMPRESSION and also describe the realtime distance lessons we conducted continuously in the first semester of 2007 with IMPRESSION and Skype by connecting between Mountain View, CA, U.S. and Tohoku University, Sendai, Japan via the Internet. As a result of these lessons, we confirmed that although we had a little time lag and some QoS problems of video stream caused by narrow bandwidth and very long distance, we could perform flexible and effective real-time distance lessons by using IMPRESSION and a videoconference system like ordinary lessons such as chalk and talk lessons in a classroom. However, we also found some points to be improved on for continuous use of IMPRESSION. Keywords: real-time distance lesson, interactive electronic whiteboard system, multimedia materials, instructional design process model.
The Concept of IMPRESSION: An Interactive Instruction System
177
modify lesson plan immediately based on formative evaluations in a lesson, and developed an interactive instruction system; IMPRESSION[6]. In this paper, we show the concept of IMPRESSION and also describe the result of the real-time distance lessons we performed continuously for half a year by connecting between U.S. and Japan via the Internet with IMPRESSION and a videoconference system.
2 Double Loop Model and IMPRESSION IMPRESSION is an instruction system for flexible and effective lesson with multimedia materials. It is a sort of sharable and interactive electronic whiteboard system based on Double Loop Instructional Design Process Model (Double Loop model) we proposed [7]. 2.1 Double Loop Model Double Loop model is an instructional design process model for designing interactive lessons by using information technologies and it forms double loop shown in Fig 1. The outer loop defines the activities of designing a lesson, and is consists of (1) Plan, (2) Apply, and (3) Evaluate phases. The inner loop of Apply phase defines the activities in a lesson, and is consists of (1) Implement, (2) Check, and (3) Modify phases.
Plan Apply Implement
Modify
Evaluate
Check
Fig. 1. Double Loop Instructional Design Process Model
As mentioned above, Double Loop model, differ from existing models, defines modification process of instructional design in a lesson in addition to instructional design process of a lesson. Thus, it clarifies interaction between a teacher and students, and allows us to cope with unexpected learners’ reactions flexibly in a lesson. It improves effectiveness of a lesson. And we are also able to review processes of performed lessons after them and confirm the modifications from plans. It facilitates us to redesign next class with the results of actually performed lessons.
178
T. Mitsuishi et al.
2.2 Facilities and Features of IMPRESSION IMPRESSION was designed and implemented as an instruction system based on our Double Loop model and allows us to perform activities in Apply phase. It consists of two types of terminals; a terminal for instructor and terminals for student, which are equipped with pen-based input device such as Tablet PC, and a relay server as shown in Fig. 2. We are able to apply this system to both in-class lesson and distance lesson. And it also allows us to utilize various multimedia data provided by web servers over the Internet as educational materials. A teacher registers a set of multimedia data to use as educational materials in advance. During a lesson, he/she select ones at will from them, present them on a
Terminal for Instructor
Terminal for Student Lesson Records < ?x m l v er s i on= " 1. 0" en cod n i g= " euc - j p" ? > < e l c ur t e> < dr aw it m e= " 18 37" >
operations
g= " euc - j p" ?> < e l c ur t e> < st ar t da t a="2 005 0 / 5/ 04/ 1 6: 13 : 16" /> < /d r aw > < op er at e ti m e= " 11 62"> < end d at e=" 200 5/ <pr05/ es0en 4/ 17: t i d="I 1 6:mag 27" /e1 > - 1" /> < /l ect ur e> 0 " e nco di ng =" e uc- p j " ?>
operations
< end d e=" at e=" 200" 5/ 05/ 0 4/ 17: 1 6: 27" / > < /l ect ur e> < m ov i e d i ="M o vi e1- 1"> < see k t i m e= " 36. 21 79 83 2"/ > < /m o vi e> <en d dat e= " 20 05 0 / 5/ 04/ 1 7: 16: 2 7" / >
operations
Relay Server Students
Teacher Multimedia Materials Internet Web Servers
Fig. 2. System Construction of IMPRESSION
Fig. 3. A Screenshot of IMPRESSION terminal for instructor
The Concept of IMPRESSION: An Interactive Instruction System
179
whiteboard, handle them (e.g. move, resize), and draw annotations on them by handwriting with an IMPRESSION terminal for instructor. Fig 3 shows a screenshot of the terminal. With these operations, it facilitates us to conduct interactive lessons according to reaction of students with various multimedia materials not only in a class but also a distance class. Thus, it makes possible to conduct flexible lessons based on formative evaluation which was defined as Apply phase of Double Loop model.
3 Practice of Real-Time Distance Lessons between U.S. and Japan This time, we conducted a series of lessons as real-time distance lessons continuously every week in the first semester of 2007 with IMPRESSION and Skype by connecting between U.S. and Japan via the Internet. 3.1 Content and Class Format of Practiced Lessons The series of lessons we conducted this time is a course named “Fundamental of IT Education B.” This course has been opened at Graduate School of Educational Informatics, Tohoku University from 2002 and provided to graduate students who have not always studied computer science or engineering. The first author lectures on fundamental computer engineering and information technologies for education (e.g. logical operation, database, computer network, and so on) in this course. This time, the first author stayed at California for a business in 2007, so we decided to practice distance lessons. We conducted fourteen lessons through 13:00 to 14:30 on every Tuesday of Japan Standard Time (through 21:00 to 22:30 on Monday of Pacific Daylight Time) from April to July. Participated students were five, and the second author, who is a doctoral candidate, also participated as a teaching assistant in order to help students and operate computer systems for IMPRESSION and a videoconference system. So far, the teacher had been preparing a lecture note and providing it before each lesson, assigning homework after every lesson, marking it and giving feedback to it, and returning it with comment at a next lesson. Therefore, this time, we provide lecture notes, receive reports for assignment, and return marked reports through a LMS of ISTU[8]. But, TA marked reports and gave feedback to them at this moment. 3.2 Implemented Environment and System Configuration Fig. 4 shows an implementation environment and a system configuration for the practiced distance lessons. This time, we connected between a room in an apartment house at Mountain View, CA, U.S., where the first author lived in, and a classroom at Tohoku University, Sendai, Japan, where students participated to lessons, via the Internet, and used Skype[9] for videoconference in addition to IMPRESSION as shared whiteboard. At the teacher’s room, we prepared a Table PC (Toshiba PORTEGE M400) for both IMPRESSION and Skype and connect it to the Internet via DSL. At Tohoku University, we prepared two Tablet PCs, one (Toshiba PORTEGE M400) with video projector is for IMPRESSION, and another (Toshiba Dynabook SS 3500) with 50in.
180
T. Mitsuishi et al.
Student Student side side (Tohoku (Tohoku University, University, Japan) Japan) Screen for Skype
Screen for IMPRESSION
Teacher Teacher side side (U.S.) (U.S.)
Terminal for IMPRESSION
TA
U.S. → Japan: about 70Kbps
USB Camera teacher
Terminal for Skype Internet Internet
TAINS/G
students
USB Camera
Japan → U.S.: about 110Kbps
DSL Terminal for IMPRESSION and Skype
Fig. 4. Implemented Environment and System Configuration of Practiced Distance Lessons
wide screen plasma monitor is for videoconference by Skype. In order to capture the video, we prepared USB camera on both site. We also prepared speakers in the classroom. The teacher used Bluetooth wireless headset in order to prevent howling. 3.3 Results of the Practices Fig. 5. (a) shows the front of the classroom where students participated and Fig. 5. (b) shows the terminals which TA operated in Japan. Fig. 5. (c) shows the teacher conducting a lesson by operating the terminal. Fig. 6 shows a screenshot of IMPRESSION in a practiced lesson.
Video from Teacher (Skype)
Shared Whiteboard Screen (IMPRESSION)
Terminal for Skype
Terminal for IMPRESSION
Terminal for IMPRESSION and Skype
(a) Front of a Classroom in Japan (b) Terminals in Japan (c) A Terminal in U.S. Fig. 5. Scenes of Practiced Distance Lessons
As results, although there were some problems that were necessary to reboot Skype for a few times because video streams froze and some audio communication failure occurred, IMPRESSION worked with no problem and we were able to conduct whole planned lessons almost well. However it was not communication failure, there exited about 1 second time lag in audio and video streams which may be caused very long distance communication and encoding and decoding of streams. By restricted bandwidth, quality of video images was also not so clear. Therefore, we could not distinguish expression of each student very much even if he/she was closer to the camera when we wanted to see whole students.
The Concept of IMPRESSION: An Interactive Instruction System
181
Fig. 6. A Screenshot of the Shared Whiteboard Screen in a Conducted lesson
4 Evaluation Here we describe subjective evaluation by the teacher and also evaluation by participated students. 4.1 Subjective Evaluation by the Teacher This time, although we had some problem on videoconference by Skype, we confirmed that we were able to conduct distance lessons with a videoconference system such as Skype and IMPRESSION as an interactive instruction system in combination. And it seems that they almost stand comparison with ordinary in-class chalk & talk lessons from result of assignments and a term-end exam. As we mentioned in the results, there exists time some lag in video streams and resolution of video images are restricted. We confirmed that these lacks of quality were affecting communication between the teacher and students. For example, even though sounds are clear enough, time lag upset timing to speak and prevent smooth communications. And the teacher sometimes felt difficulty to decide to put forward or explain again with different words because it was difficult to grasp whether students certainly understood or not from vague video images. On the other hand, IMPRESSION worked well with its features although there are some restriction such as screen size and resolution in comparison with ordinary blackboards. For example, we are able to present any materials at will by registered them in advance, it reduced the load to write complex figures and tables in a lesson and the teacher could concentrate on explanations. IMPRESSION is able to go back to the screen in the past. With this feature, the teacher showed again the drawn screen after putting forwarding explanations. Such a progress is impossible in ordinary lesson with blackboard. And the more, by giving the right of operation to students, the teacher and students were able to interact with each other through the numbered cards on the screen of IMPRESSION when we discussed sorting algorithm. Therefore it facilitated the teacher to conduct not only flexible lessons which were the original objective of IMPRESSION but also smooth and effective lessons.
182
T. Mitsuishi et al.
However, we also found some problems to be solved as will be discussed later in this paper. 4.2 Evaluation by Students In order to investigate the effectiveness of distance lessons with IMPRESSION and Skype, we conducted the survey with questionnaires to students [10]. Fig. 7 shows part of the result of questionnaires; from Q1 through Q7 on the systems and Q8 and Q9 on help and feedback by TA.
22
Q1: Could you see the screen of videoconference enough? ……….……….……….…………..
11
22
11
22
11
Q2: Could hear the voice of videoconference enough? ……….……….……….…………………
11
Q3: Did you think you could communicate with the teacher by videoconference? ………........
11
33
11
Q4: Could you see the screen of whiteboard system enough? ……….……….……….………..
11
33
11
Q5: Did you think materials and annotations help your understanding? ………………………..
22
22
11
Q6: Did you think you could interact with the teacher via whiteboard system? ……….…….....
22
22
11
Q7: Did you think these distance lessons were enough in comparison with ordinary lessons?
22
22
11
44
Q8: Did you think consultation with TA helped your study? ……….……….…….…………........
33
Q9: Did you think feedback to assignments from TA help your study? ……….……….….…….
0 Yes
Relatively Yes
Neutral
11
1
11 2
Relatively No
3
11 4
5
No
Fig. 7. Results of Questionnaire to Students Table 1. Reasons why they felt not enough with these distance lessons
- Face to face lessons seem better than distance lessons. - It seems a little difficult to communicate with the teacher. - Sometimes voice had disconnected. - Much better communication line seems necessary. Although it doesn’t have statistical meaning because the amount of participants were a very few at this moment, we can grasp a rough tendency. As a result, students felt that the systems were not enough but not too bad. About a question; “when you felt the systems are not enough, which are the problems of them or how to solve them?” they answered to this question as shown in Table 1. From these opinions, most of the reasons why they did not feel enough or satisfied with the distance lessons are lack of stability of Skype causing freeze of video streams and audio communication failure, lack of quality of video images caused restricted bandwidth and time lags caused very long distance.
The Concept of IMPRESSION: An Interactive Instruction System
183
As for drawing and presentation with IMPRESSION, although we had some negative answers to the question Q6: “Can you interact with the teacher via shared whiteboard screen?” we had relatively positive answers to the question Q5: “Is the explanation by drawing annotations and presenting materials on shared whiteboard screen helpful for your understanding?” It shows the effectiveness of features of IMPESSION for distance lessons. The reason of negative answers to Q6 might be assumed that we could prepare only one Table PC for IMPRESSION in a classroom, students should move to the terminal in order to operate it, and we could not interact at will via IMPRESSION. And these are not about the systems such as IMPRESSION and Skype, the second author helped students, checked reports for assigned homework and gave feedback as TA. Q8 and Q9 were questions asking how affect these help and feedback by TA and the answers were almost very well. Thus, we could assume these help and feedback by TA made up for restrictions and problems in communication between the teacher and students. 4.3 Confirmed Problems of IMPRESSION to Be Solved Although we find some restrictions of bandwidth and lack of stability of the videoconference system we used, these matters will be solved in the near future and we confirmed the effectiveness of IMPRESSION to sum up. However we also found some problems of IMPRESSION to be solved. The load to prepare materials and manage them is especially one of big problem of them. This time was the first time to use IMPRESSION in this course and the teacher should prepare materials. So, the load to prepare was of course heavy. However, in addition to the load to prepare materials, we found following two problems; 1) it was burdensome to register materials to IMPRESSION or to delete registered materials from IMPRESSION, 2) it was difficult to find an aimed material when so many materials were registered. Current IMPRESSION is just prototype to demonstrate interactive presentation of multimedia materials and has features for this purpose. Therefore it has only simple features and user interfaces to handle and manage materials. For example, it allows us to register materials one by one, but we are not able to register a set of materials at once. And it does not have a feature to sort out registered materials nor retrieve an aimed material by query. So, when we select a material to present in a lesson, we need to search the aimed material from all registered materials. This time we conducted lessons continuously every week and materials for each lesson were different from others. So we should register a new set of materials every week. And, we assumed it was difficult to find an aimed material to present when so many materials were registered, we deleted materials which were registered for previous lessons. These were very burdensome tasks. Furthermore, even when we regard some materials were not necessary and deleted them, sometimes they became necessary in order to review previous lessons. It could be said as lack of reusability. In order to solve these problems and facilitate to select an aimed material from many materials and to reuse registered materials in previous lessons, it is necessary not only to improve user interfaces but also to develop some features to sort and manage materials based on lesson plans.
184
T. Mitsuishi et al.
5 Conclusion This time, we conducted real-time distance lessons continuously for half a year by connected between U.S. and Japan with Skype and IMPRESSION via the Internet. As a result, we confirmed that although there were several restrictions on videoconference via the Internet such as time lags and resolution of video images, we were able to conduct real-time distance lessons with a videoconference system and our IMPRESSION, which allows interactive presentation of multimedia materials and drawing annotations on a shared whiteboard screen among a teacher and students, and the effectiveness of such lessons in comparison with ordinary lessons in a classroom. However, we also found the problems of IMPRESSION to be solved in order to conduct lessons continuously such as usability of user interfaces and features to handle and manage many materials. We are developing such features[11]. On the other hand, we will be able to record conducted lessons when we conduct them with a system like IMPRESSION, and it could be expected to create contents for review by students or to prepare data for reflection by the teacher[12][13][14][15][16]. We will solve the confirmed problems, improve our system, and propose novel educational environments as our future work. Acknowledgments. A part of this work was supported by KAKENHI (16700550 and 20700631).
References 1. Yoshino, T., Inoue, Y., Yuizono, T., Munemori, J., Ito, S., Nagasawa, Y.: Development and Application of a Supporting System for Distance Learning Classroom Using Personal Computers via Internet. IPSJ Journal 39(10), 2788–2801 (1998) 2. Murakami, M., Yagi, K., Kakusho, K., Minoh, M.: Analysis of Evaluation Factor of Distance Learning System Focusing on Effect of Experience and Custom in Lecture (in Japanese). IEICE Transactions on Information and Systems J84-D-I(9), 1421–1430 (2001) 3. He, A., Kara, A., Cheng, Z., Go, K., Koyama, A., Huang, T., Imamiya, A.: RIDEE-SPS: A Presentation System for Realtime Interactive Distance Education Environment (in Japanese). IPSJ Journal 44(3), 700–708 (2003) 4. Hayashi, T., Watanabe, K., Otani, M., Tanaka, H., Okazaki, Y., Hayashida, Y., Kondo, H.: Design and Implementation of a Remote Lecture Based on Instruction with Blackboard Using High-Quality Media Devices and High Speed Information Network (in Japanese). Journal of Japanese Society for Information and Systems in Education 22(1), 3–14 (2005) 5. Matsuura, T., Maeda, K., Kohno, E., Kishida, T.: A Telepresentation System to Assist Presenting Rich Multimedia to Audiences in Remote Locations (in Japanese), Technical Report of JSiSE Chugoku branch 5(4), pp.16-19 (2005) 6. Higuchi, Y., Mitsuishi, T., Go, K.: An Interactive Multimedia Instructional System: IMPRESSION for Double Loop Instructional Design. IEICE Transaction on Information and Systems E89(6), 1870–1884 (2006) 7. Higuchi, Y., Konno, F., Mitsuishi, T., Go, K.: A Double Loop Instructional Design Process Model for Interactive Instructions. Japan journal of educational technology 3(4), 457–468 (2008)
The Concept of IMPRESSION: An Interactive Instruction System
185
8. Mitsuishi, T., Kumai, M.: ISTU: Internet School of Tohoku University (in Japanese). Journal of IEICE 86(11), 816–820 (2003) 9. Skype web site, http://www.skype.com/ 10. Mitsuishi, T., Konno, F.: A Report on Practice of Distance Class between Japan and U.S. with ISTU and Real-time Distance Education System (in Japanese). Annual report of ISTU 4, 9–21 (2006) 11. Suzuki, T., Konno, F., Ohkawa, Y., Mitsuishi, T.: Design and Implementation of a Lesson Plan Based Data Management Mechanism for Educational Materials in IMPRESSION. In: Proc. of 33th JSiSE Annual Conference, pp. 296–297 (2008) 12. Konno, F., Higuchi, Y., Mitsuishi, T.: A proposal of a methodology of supporting teacher reflection by presenting differences between planned and implemented actions. In: Proc. of AACE SITE 2007, pp. 1059–1064 (2007) 13. Konno, F., Higuchi, Y., Mitsuishi, T.: Teacher’s Awareness in New Reflection Methodology Using Highlighted Process Displays. In: Proc. of AACE SIT 2008, pp. 2665–2673 (2008) 14. Kanno, Y., Konno, F., Ohkawa, Y., Hashimoto, K., Mitsuishi, T.: A Proposal of Teacher Reflection Supporting System Based on Observation of Reviewing Activities with Handouts and Video Records (in Japanese). Educational Informatics Research (7), 1–8 (2008) 15. Konno, F., Higuchi, Y., Mitsuishi, T.: A Study on a Teacher Reflection Method by Presenting Differences between Lesson Plan and Actual Instructions (in Japanese). Japan journal of educational technology, 32(4) (printing) 16. Konno, F., Kanno, Y., Mitsuishi, T.: Effects of Teacher Reflection with Highlighted Level Process Displays: Presenting Differences between Lesson Plan and Implemented Actions. In: Proc. of AACE SITE 2009, pp. 1236–1243 (2009)
Improving Children’s Writing Ability Joana Pereira, Luís Carriço, and Carlos Duarte LaSIGE, Faculty of Sciences of the University of Lisbon, Portugal [email protected], {lmc,cad}@di.fc.ul.pt
Abstract. This paper presents IWA, a platform to aid children when learning how to write. The proposed system offers both tutor and child a certain degree of autonomy. IWA provides tutor and child with different interfaces. The features available to the tutor allow the definition and configuration of repetition exercises comprising letters, numerals and freeform gestures. The child interface supports the child in the task of solving those exercises. The system has been evaluated in two sessions with children. From the evaluation results and the feedback provided by a school teacher we conclude this to be a very promising system towards optimizing the repetition process required for perfecting hand-writing. Keywords: Hand-writing, Children, Education, Multimedia, Evaluation.
to support interaction mechanisms similar to the ones used in traditional learning techniques by having the child write directly on the device, in the same way she would do with paper. This will allow lessening the child’s adaptation efforts while offering a rich and motivating interaction experience. The paper reports on the development of such platform, named IWA (Improving Writing Ability), which applies gesture evaluation techniques in teaching to write scenarios. The platform strives to perfect some of the writing and calligraphy characteristics trough contextualized challenges and task execution, while making use of appropriate feedback mechanisms capable of stimulating children of target age ranges to a healthy and constructive learning experience. The next section presents a summary of related work. Afterwards, we present the IWA system supporting concepts, the tutor and pupil interfaces, the data storage and logging mechanisms and the gesture grading algorithm. This is followed by a description of the preliminary evaluation conducted and a presentation of the results obtained. Finally we discuss those results and finish with some concluding remarks and future work directions.
2 Related Work One very often applied technique in learning to write support methods based on recent technologies is gesture recognition and evaluation, from gestures performed with the assistance of haptic systems [1, 2, 3, 4]. Telemaque [1] is a system which employs a force feedback ready pen to help children learning how to write. According to its authors this system achieved positive results regarding the adoption of a proactive strategy to control handwriting movements by the children who participated in the evaluation [1]. However, the force feedback system limits the children’s movements to a specified area around the letter that has to be drawn, accompanying the movement. This system, aimed at reducing the errors made, also limits the child’s independence and, potentially, the child’s confidence to operate without the system’s assistance. Other systems targeted at improving the performance of people perfecting their writing skills have also been explored in the context of learning a second language. ITOUCH [2] is one such a system. It was conceived to help learning Japanese characters, with an animation completed by voice instructions teaching how the character should be drawn. This system also showed benefits, with a significant improvement in the classifications obtained by the persons who used the system over the ones who didn’t. However, the classification process had to be made by an independent teacher. IWA supports automatic classification and additionally is capable of providing feedback during the shape’s execution. Other studies [5] show how technology can help children to develop their reading skill and improve their knowledge. Systems using technology together with children appealing content, like games or animations, show great potential when correctly conceived. Given that children are one of the most complex target audiences, some guidelines to assist in the development of systems targeting them have been defined [6, 7]. One of those studies concerns the capability of a system or application to engage children [6]. The author presents some fundamental factors to achieve this
188
J. Pereira, L. Carriço, and C. Duarte
capacity, like supporting rich interactions and always provide adequate feedback, amongst others, while pointing out a set of precautions that should be considered when designing such systems. IWA follows these guidelines, thus becoming an engaging system, providing the sufficient independence to the child and the tutor, ensuring appropriate feedback is given to the child, and, additionally, being able to present important information regarding the child’s evolution to the tutor.
3 The IWA System The IWA system aims to provide the required autonomy for both tutors and pupils during the calligraphy learning, repetition and perfecting process. The system was developed targeting both user groups: teachers and pre-school and first school years children. For the teachers, the system provides a set of tools allowing them to define and configure the exercises to present to the pupils. For the pupils, the system offers a rich graphical interface for presenting the teacher defined exercises, developed taking into account their literacy level, and taking care to provide immediate feedback according to the child’s performance. Each exercise is presented to the child as a challenge. A challenge is based in two concepts: a template and a task. A template is a set of sequentially ordered points defining the shape of a gesture. The task defines a set of presentation and interaction settings. Current settings allow defining how the template is to be presented (with a solid or dashed line), what scale is to be used (thus varying the size of the presented shape) and how many repetitions are required for the challenge to be considered successful. A challenge is thus defined as an association of a task to a template. It is the responsibility of the tutor to define and configure each challenge before it being presented to the pupil. A set of challenges is named learning unit. In this fashion, the teacher can prepare a set of learning units to use in her classes. After being presented with a challenge, the pupil executes the assigned tasks and waits for the system attributed score. Each challenge overcome by the child provides an immediate reward and the enabling of new challenges. In the following sections we will present in greater detail both profiles available in the IWA system, the logging mechanisms offered and the algorithm employed to grade the child’s performance and on which is based the presented feedback. 3.1 The Tutor Interface The IWA user profile dedicated to the teachers’ population has as main goal supporting the definition of the challenges to present to the children. Prior to presenting the challenges to the children, these must be defined and appropriately configured by the teacher. To this end, the IWA system offers two main groups of features: one allowing the definition of templates and the other supporting the association of tasks to templates in order to create challenges. When defining templates the tutor is responsible for identifying the class of gesture to ask the children to do. Currently, IWA supports for classes of gestures: upper case letters, lower case letters, numerals and freeform. When defining a letter or a numeral the tutor can select the desired symbol by pressing the corresponding button on the
Improving Children’s Writing Ability
189
IWA tutor interface. Figure 1 presents this process for the letter “o”. The template is stored on the local file system, using the symbol as an identifier. When defining a free form gesture the tutor is requested a symbol name to act as its identifier. After defining the identifier the tutor draws the gesture in the drawing space on the right side of the interface (see figure 1). The tutor can define as many templates as whished for each letter, numeral or freeform gesture.
Fig. 1. Creating a template for the letter “o” in the IWA tutor interface
Once the templates have been defined the tutor can proceed with the challenges creation. All the previously defined templates are available to the tutor during the challenge creation process. After selecting the template, the tutor must configure the task the child will have to perform. As described above the task consists on a line following assignment, and the tutor can configure how the template is presented (line type and shape size) and how many repetitions will be requested. Additional details also have to be selected, like the voice instructions (which can be recorded at this point if not already available) or the difficulty settings for the scoring algorithm. Figure 2 illustrates this process.
Fig. 2. Configuring the challenge in the IWA tutor interface
190
J. Pereira, L. Carriço, and C. Duarte
The tutor interface also supports browsing of the challenges completed by pupils. The teacher can load the results of the challenge that has been submitted by the pupil and, besides seeing the score achieved and the time it took for the child to draw the shape, she can play an animation of how the child draw the shape, which will assist in identifying possible problems, like the child drawing the shape in the wrong direction, for instance. 3.2 The Pupil Interface Figure 3 shows the pupil graphical interface where the challenges will be solved by the children. The interface is organized in three functional areas: the top area consists of a set of buttons corresponding to the challenges; in one of the sides a colored progress bar is used to present the child’s gesture score; and a workspace with a horizontal guide line where the challenges are presented and the gestures executed.
Fig. 3. The IWA pupil interface
The challenges are presented and executed in a precise sequence. The challenge execution process starts with the child pressing the first button in the top area. In response to the button press, the challenge is presented in the workspace area through an animation exemplifying how the gesture is to be made. This animation can be replayed anytime the child wishes to, by pressing the corresponding button. After the gesture animation stops the shape’s line is drawn according to the teacher definition and a sound cue is played signaling the child can start the gesture drawing. Once the drawing task is concluded the child requests the grade by pressing the colored progress bar. In response the system slides the progress bar to the obtained score and follows up with one of two behaviors: plays a sound associated with a less achieved performance and then displays the same exercise for the child to repeat and perfect; plays a successful sound and unlocks the next challenge by enabling the related button and sharpening its image. This is expected to captivate the child, motivating him to reach good scores and unlocking new challenges. The described process is repeated until all the challenges are overcome. However, the child is able to repeat a challenge whenever he desires it.
Improving Children’s Writing Ability
191
The drawing process is supported by some assistance mechanisms. After the shape has been presented, or whenever the child lifts the pen from the drawing surface during the challenge execution, a graphical animation presents a hint of where the child should continue drawing the shape, and in what direction. Additionally, when the difference between the shape being drawn by the child and the template rises above a teacher defined threshold the line drawn by the child gets thicker and changes color as can be seen in figure 4.
Fig. 4. The drawing line thickens and changes color when the pupil strays away from the template
The images and sounds used in the pupil interface are tailored to child’s age and gender in order to try to motivate them as most as possible. 3.3 Data Storage and Logging Mechanism Every data related to challenge creation and their execution are stored in XML files. These files comprise a set of metadata with the challenge configuration and the set of sequentially ordered points defining the template used in the challenge, or the shape drawn by the child. The metadata pertaining to a challenge is the information characterizing both the template and the tasks defining the challenge. The metadata pertaining to a challenge execution consist of not only the challenge’s full characterization, but also of every point drawn with temporal information and the score achieved. For each attempt at solving a challenge a complete independent file is generated. The files storing the challenge solving attempts can be browsed and visualized later in the tutor’s interface to assist in the identification of difficulties the child may be going through. 3.4 The Grading Algorithm The grading of the child’s executed gestures is based on two main criteria. The first criterion is the average absolute distance between the drawn points and the template points. The identification of what is the closest point in the template for each point in the drawn shape is not a trivial problem when lines cross in the shape. This happens in
192
J. Pereira, L. Carriço, and C. Duarte
many of the alphabet letters. To prevent the algorithm to identify as the closest point, a point that is not in the same stroke, the comparison between the current point and the template points does not include all the template points. Instead, it includes just the next thirty points in sequential order from the last matched point. This prevents the matching with incorrect point, while supporting situations where there is a need to lift the pen to continue drawing in another part of the workspace. The second criterion is the percentage of the template that has been covered in the child’s shape. This prevents the situations where the child only draws part of the shape, even if they were drawn close to the template, which would result in a good grade if only the first criterion was used. Based on these two criteria, and the internal parameters configured by the teacher, the shape’s score is computed. The algorithm also works in a proactive manner, keeping an internal count of the consecutive failed attempts in the same challenge. This allows the mechanism that adjusts the line thickness and color according to the distance between drawn point and template point, to operate after the child has failed the challenge the number of times specified by the teacher. This provides an immediate feedback to the child on his performance, as presented on figure 4.
4 Preliminary Evaluation To assess the proposed system two preliminary evaluation sessions were conducted. Our goal was to assess the usability of the pupil interaction platform while a child is executing the challenges, and also our assumptions regarding the usefulness of a system like the one described towards an increase in motor skills related to handwriting tasks. To this end we set up two evaluation sessions, with a three and a half years old girl participating in one session and a four and a half years old boy participating in the other session. For each session twelve distinct challenges were prepared, totaling twenty four challenges distributed over two learning units. All challenges were presented in dashed lines, keeping the same scale that was used in the template definition. Each evaluation session took approximately twenty minutes. The application was executed in a LG P100 tablet PC. From the first to the second session, some usability improvements were made. This will not impact our conclusions since we will not try to establish any comparison between both sessions’ results. The sessions were conducted at the children’s homes, in order to stress them as little as possible. The children’s parents were present during the evaluation. They were briefed prior to the sessions, and agree on having the sessions recorded on tape. The tape was later used to assess the stress level felt by each child during the evaluation. During the first evaluation a school teacher, who is the child’s grandparent, was also present. Instructions on what was expected of the children were explained to them at the beginning of the trial, and every time they requested it, or was felt needed by one the evaluation team members. During the evaluations, both children provided very positive reactions regarding the system’s feedback mechanism, and, on their own initiative, wanted to repeat the
Improving Children’s Writing Ability
193
challenges after completing them successfully once. This allowed us to record results on two rounds per learning unit. After the session’s completion, results and information gathered were processed and analyzed, resulting in three metrics regarding the executed challenges: best score, average number of attempts until successfully completing the challenge, and average time taken per attempt. The score is assessed in a 0 to 100 scale, with 100 being the perfect score. Table 1 presents these results for both children. Table 1. Evaluation sessions‘ results
Best Score
Girl Boy
Round 1 72.13 69.16
Average # of tries Round 2 73.48 72.83
Round 1 1.75 1.5
Round 2 1.5 1.2
Average time per try (s) Round 1 Round 2 10.32 10.66 8.79 9.93
A quick analysis of the presented results shows that from the first to the second round in both sessions there was an increase in the best score, a reduction in the number of attempts per challenge and a slight increase in the time taken per attempt. Due to the small number of evaluation sessions conducted we feel it is not adequate to perform a detailed quantitative analysis of the results. A qualitative discussion of the evaluation sessions conducted is presented in the next section.
5 Discussion During the first evaluation session we identified some usability problems in the pupil interface. The school teacher present in the session also offered some constructive criticism regarding some pedagogical aspects employed in the IWA system. The scoring progress bar was placed on the right side of the workspace. For righthanded children, as was the case of both children involved in the evaluation session, this placement meant their arm covered the score bar. We noticed that after completing the shape the children did not remove the arm immediately, and failed to watch the progress bar animation. For the second session, the bar was placed on the left side of the workspace, in a way that was always visible to the child. During the first session it was noticed the child failed to follow closely the animation that exemplifies how the shape should be executed. We introduced a different cursor, with a pencil shape and a bigger size, which proved better at drawing the child’s attention. When the challenge used a template requiring multiple strokes (e.g. “E”), we noticed that, many times, the child did not know where to proceed after finishing a stroke and lifting the pen. We introduced a mechanism to hint at where the child should continue the shape’s execution, and in what direction, every time the pen is lifted. The way the score is presented was also changed. In the initial version the score indicator began its animation in the middle of the progress bar. This meant that when the achieved score was low, the indicator dropped. For the second session, this
194
J. Pereira, L. Carriço, and C. Duarte
behavior was altered. The score indicator starts at the bottom of the progress bar, which means that whatever the score is, the indicator always rises, improving the child’s motivation. Although only two evaluation sessions have been performed, it can be noticed a slight increase in the scores obtained and in the time taken to perform the shapes from the first to the second round. This can be a sign that the children were trying to perfect their execution to improve the classification achieved. Regarding the boy who participated in the second session it was perceivable that the degree of difficulty demanded (a score of 50 points or better to unlock the next challenge) proved to be a demotivating factor, due to the child’s proficiency level for the requested tasks. This effect can be avoided if the difficulty can be adjusted in execution time, based on the scores that have been achieved in the previously executed challenges.
6 Conclusion and Future Work This paper presented IWA, a system to assist both teachers and pupils in the handwriting learning process. The system employs pen-based interaction techniques in order to mitigate the effort required of the children to adapt to a technological solution. The paper also presented the results of a preliminary evaluation of the proposed system. The results were positive and extremely encouraging, regarding both the improvements perceived in the evaluation sessions and the enthusiasm shown by the children. Also the feedback provided by the school teacher was very positive and her suggestions were used to improve the system for the second evaluation session. However, from the small number of evaluations performed, we can envision several evolutions to the current system in order to improve both pupil and tutor interface and the grading algorithm. Perhaps the biggest evolution we could make to the system is to endow it with adaptive capabilities to adjust to difficulty level used in the grading algorithm in accordance to the acuity revealed by the child in the previous challenges.
References 1. Palluel-Germain, R., Bara, F., Hillairet de Boisferon, A., Hennion, B., Gouagout, P., Gentaz, E.: A visuo-haptic device - Telemaque - increases kindergarten children’s handwriting acquisition. In: Proceedings of the Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC 2007), pp. 72–77. IEEE Computer Society, Los Alamitos (2007) 2. Eid, M., Mansour, M., El Saddik, A., Iglesias, R.: A Haptic Multimedia Handwriting Learning System. In: Proceedings of the international workshop on Educational multimedia and multimedia education (EMME 2007), pp. 103–108. ACM Press, New York (2007) 3. Bara, F., Gentaz, E., Colé, P.: The visuo-haptic and haptic exploration of letters increases the kindergarten-children’s reading acquisition. Cognitive Development 19, 433–449 (2004)
Improving Children’s Writing Ability
195
4. Henmi, K., Yoshikawa, T.: Virtual lesson and its application to virtual calligraphy system. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 1275–1280. IEEE Press, Los Alamitos (1998) 5. Liang, P., Johnson, J.: Using technology to enhance early literacy through play. Computers in Schools 15(1), 55–64 (1999) 6. Said, N.: Towards a ‘model of engagement’ Designing Multimedia Application for Children. Digital Learning 3, 1 (2007) 7. Wyeth, P., Purchase, P.H.: Using Developmental Theories to Inform the Design of Technology of Children. In: Proceeding of the 2003 conference on Interaction Design and Children, pp. 93–100. ACM Press, New York (2003)
From Paper to Module – An Integrated Environment for Generating SCORM Compliant Moodle Courses Out of Text and Multimedia Elements Hans-Martin Pohl, Benedikt Deicke, and Jan-Torsten Milde Fulda University of Applied Science HCI Research Center Heinrich-von-Bibra-Platz 3, 36037 Fulda, Germany [email protected], [email protected], [email protected]
Abstract. ECampus is a project spanning all departments at the University of Fulda. It has been started to create a uniform learning environment at the university. The objective is to research and develop a user-friendly easy-to-use editor to generate SCORM 2004 conform E-Learning modules. This editor is based on Open Source software and new technologies such as XSL transformations and the Google web toolkit. The whole system is provided as a web application and embedded within the E-Learning environment of the university. Some E-Learning modules can be developed with the system immediately. These modules are now being used during the lessons with great success. Keywords: SCORM 2004, XSLT, transformation, creation of content, E-Learning, modules, lesson, user friendly, style sheet, LOM, moodle, LMS.
2 State of the Art While there are currently a lot of different Learning Management Systems, the use is limited to but a few in Germany. Hereby Moodle (see http://www.moodle.org) and ‘Ilias’ (see http://www.ilias.de/ios/) play the greatest role. In contrast to the LMS there are only a limited number of editors for creating an ELearning module. HTML editors are widely-use but don’t provide adequate functionality. Furthermore E-Learning modules generated by an editor can mostly only be used within a proprietary LMS. Additionally, a lot of effort is required to learn how to use these editors. Such a system fails entirely if there has to be consistent development and presentation of tests and learning success. The advantage of these systems is the functionality for communication between users or between tutor and student. In our case this can be neglected because this functionality is captured by the LMS ‘System2Teach’ (see also http://www.system2teach.de) which is a proprietary development of the University of Applied Sciences Fulda. This paper focuses on easy and comfortable development of E-Learning modules based on textual content by using the eCampus framework. Concerning the technology for state of the art web applications there are many choices. Many different systems support the web based approach. Besides PHP (see http://www.php.net) or JSP (see http://java.sun.com/products/jsp/index.jsp) it is also possible to use Ruby (see http://www.ruby-lang.org) in combination with the Ruby on Rails framework (see http://www.rubyonrails.com).
3 The eCampus Project This research focuses on the development of a user-friendly system for writing E-Learning units, allowing authors to concentrate on the content and the didactic concept of the unit, instead of worrying about the underlying technology. It is part of the eCampus project at the University of Applied Sciences Fulda. In this interdisciplinary project, content is being created for the faculties Nutritional Sciences, Food and Consumer Sciences, Applied Computer Science, Nursing & Health Care as well as Food Technology. Within the first years of the project, many E-Learning modules were produced. All modules address prominent introductory courses of the particular program of study. This was only possible by using the eCampus framework which is described below. The central target of the project is the implementation of a tool that decreases the complexity of the process of learning module production. The central objectives of the learning module production are an increase in the interactivity of the course, as well as an increase in the quality and quantity of self learning. Currently LMS with SCORM 2004 support are not commonly available. Until Moodle is SCORM 2004 capable the format of transformed E-Learning modules will be SCORM 1.2.
198
H.-M. Pohl, B. Deicke, and J.-T. Milde
Fig. 1. System Architecture
3.1 Overview System Architecture The learning module production takes place in three phases (see Figure 1): • the structural annotation of the textual content • the automatic generation of the E-Learning unit • the automatic transmission into the LMS and allocation as SCORM compliant E-Learning module In phase one, most of the (textual) content is created in a standard word processor (e.g. Open Office, [10]). Here the text is structurally annotated using a predefined formatting template. The text is stored and automatically converted into a simple XML file, which provides the basis for subsequent transformation steps. This approach works very well for static content and is used in a number of systems (e.g. eLAIX (see http://www.boldt-media.de/)). In the second phase, the file prepared in phase one is transformed into a SCORM 2004 compliant E-Learning module automatically. For this, the document is unpacked and analyzed. To get by on cascaded XSL transformations (EXtensible Stylesheet Language) each element is identified and transformed with specific instructions (see Figure 2). Finally all contents are packed into an E-Learning module and multiplexed with the necessary auxiliary files. The available metadata is separated and attached to the module in a LOM (Learning Objects Metadata) compliant form. The complete process was formerly controlled by the Apache ant build tool ([4]) but is now ported to Rake (see http://rake.rubyforge.org). It uses saxon8 (see http://saxon.sourceforge.net/) as a transformation engine.
From Paper to Module – An Integrated Environment
199
Fig. 2. Transformation Tree of eCampus Framework
In the last phase the author is able to either download the education module or to transfer it to a Moodle server, where a new course is created automatically. Unfortunately Moodle currently lacks a stable web service API. Due to this the automatic transfer and creation of courses is not yet supported until Moodle fully implements this. In addition the transformation server itself also provides a RESTful Web service API. These interfaces make it possible to integrate the transformation server into other applications. 3.2 User Front End The user front end is very simple and easy to use. All main functions can be controlled by the main screen (see Figure 3). The menu bar on top contains the administration functions and an online help. The main part is the dark green beam with the input box for the source file. If the path is unknown the file can be selected using the browser's file chooser. Afterwards the transformation can be started with a single click. In advanced mode there is the possibility to set the name of the target file or assign a style for the presentation as single HTML files within a browser. Additionally it is possible to add media files like images, flash animations, audio or video. The lower part of the screen lists all transformations and displays their current state. It is possible to get a detail view and to download the results of completed transformations. The detail view (see Figure 4) displays the state of the transformation and contains the included source files, the resulting learning units and some functions to reannotate or retransform the source file. 3.3 Transformation Process The complete process is controlled by the Rake (see http://rake.rubyforge.org) build tool and uses saxon8 (see http://saxon.sourceforge.net/) as a transformation engine supported by a graphical user interface (GUI).
200
H.-M. Pohl, B. Deicke, and J.-T. Milde
Fig. 3. User Front End Main Screen
Fig. 4. Detail View of a Transformation
After starting transformation, the written Open Office document is unpacked. After that the file ‘content.xml’, which contains the content, is accessible and can now be transformed (see Figure 4). The first step of the transformation is to transform this ‘content.xml’ into a new file called ‘puretext.xml’. This new file is in an intermediate format. Thereby the
From Paper to Module – An Integrated Environment
201
annotated elements are assigned an explicit format template. Furthermore all elements without annotation are deleted. The result of this transformation is a universal intermediate format without direct correlation to the producer application. The transformation is controlled by the XSLT stylesheets ‘content2pure_1.xsl’ and ‘content2pure.xsl’. The advantage of this intermediate format is that the resulting transformations are totally independent of the content collecting process or application. As a result a content collecting process using other applications is also possible in the future, e.g. Microsoft Word. Only the transformation into the intermediate format has to be adapted.
Fig. 5. Abstract of the unpacked *.odt
The file in intermediate format can be transformed into the output format (here HTML) by applying the XSLT stylesheets ‘generateHTML.xsl’ and ‘html.xsl’ For this purpose each element from ‘puretext.xml’ is analyzed and the corresponding transformation instruction is searched. If the correct instruction is found, it is applied to the specific element. The transformation result is stored into the HTML file. To print out the content easily a further version is generated. In contrast to the first form all learning objects, sessions and sections, are stored within one page. This page can easily be printed by using the print functionality of the browser. An overall glossary is generated by applying the stylesheet ‘glossary.xsl’. Hereby all elements which are annotated as glossary entries are transformed into a separate glossary file. To use the learning module without an LMS there must be extra navigation. In this case all headlines from sections and sessions are transformed into the file ‘menu.xml’ supported by ‘menu.xsl’ The metadata which is stored within the ‘content.xml’ is read and transformed into a further intermediate format ‘intermediateMD.xml’. From this the ‘generate LOMMetadata.xsl’ generates the ‘modulID.xml’ and ‘parenttitle.xml’ which
202
H.-M. Pohl, B. Deicke, and J.-T. Milde
conforms to the LOM standard. Subsequently the complete content and all metadata is transformed into XML files. 3.4 Integration in LMS It is possible to manually download and import the resulting ZIP archive into any SCORM compliant LMS. Additionally, as the University of Applied Sciences Fulda is currently using Moodle, it is planned to implement an automatic transfer and import of generated learning modules into this LMS. Unfortunately Moodle's SOAP based web service API is currently under development and scheduled for its 2.0 release. As soon as the required functionality is implemented, the eCampus web application will be extended accordingly. By clicking a single button the user will be able to automatically create a new course in the Moodle LMS based on the previously generated learning unit.
4 Discussion and Prospects As the transformation process is run on a single central server, it becomes relatively simple to update and extend the system functionality. An expensive deployment onto multiple computers is not needed anymore and it is ensured that all authors are always able to use the up-to-date transformation process. The transformation server is based on the dynamic scripting language Ruby and the programming language Java. Both are using virtual machines which are available for various operating systems. Although the transformation server is currently running on Linux it is possible to run it on other platforms, too. Future works will be concentrating on the development of a web based editor making it possible to replace OpenOffice. This editor should be able to import documents in different formats, automatically analyse the document structure and provide functions to annotate these documents online.
References 1. Crockford, D.: RFC 4627. The application/json Media Type for JavaScript Object Notation (JSON), http://www.json.org/ (accessed, 13.01.2009) 2. David Eisenberg, J.: OASIS OpenDocument Essentials. LuLu.com (2004) 3. Hodgins, W., Duval, E., et al.: Draft Standard for Learning Object Metadata. Report, Learning Technology Standards Committee. IEEE, Los Alamitos (2002) 4. Holzner, S.: Ant - The Definitive Guide. O’Reilly Media, Sebastopol (2005) 5. Google Inc. GWT: the Google Web Toolkit, http://code.google.com/webtoolkit/ (accessed 13.01.2009) 6. Kay, M.: XSLT 2.0 programmer’s reference. Wiley, Indianapolis (2004) 7. Koper, R.: Modeling units of study from a pedagogical perspective (2001), http://eml.ou.nl/introduction/articles.html 8. Advanced Distributed Learning. SCORM 2004 (2004), http://www.adlnet.gov/scorm (accessed, 13.01 2009)
From Paper to Module – An Integrated Environment
203
9. Schulmeister, R.: Grundlagen hypermedialer Lernsysteme: Theorie - Didaktik - Design. Oldenbourg Verlag, München (2002) 10. OpenOffice.org team. OpenOffice, http://www.openoffice.org/ (accessed 13.01.2009) 11. W3C. XML, http://www.w3.org/MarkUp/ (accessed, 13.01.2006) 12. Wilde, E., Lowe, D.: XPath, XLink,XPointer, and XML. Addison-Wesley Professional, Reading (2002)
Development of a Simulator of Abacus: Ancient Analog Calculator on a Mobile Phone as a Teaching Material Kenta Saito, Yuki Makita, Vu Quang, and Hitoshi Sasaki Faculty of Engineering, Takushoku University, 815-1 Tatemachi, Hachioji, Tokyo 193-0985, Japan {myrte,makita}@eitl.cs.takushoku-u.ac.jp, [email protected] Hitachi Software Engineering Co., Ltd. 4-12-7, Higashishinagawa, Shinagawa, Tokyo 140-0002, Japan
Abstract. Portable electric devices such as mobile phones and video game consoles are becoming high performance and they are becoming ubiquitous. And a lot of people are in situations that allow for use of these applications. On one hand, it is useful for students to study out of class if they can use teaching materials at school with a portable device. And Japanese people had widely been using an abacus until the electronic digital calculator appeared recently. Even today, the abacus is used to teach math in Japanese primary schools teaching. We are developing a simulator of abacus as one of the computerizations of the teaching materials in this research. Keywords: abacus, android, education, teaching material, mobile phone.
developed functions which display the number to understand expression of peculiar number for abacus and set simple addition problem to practice the calculation for using this application.
2 Overview of Japan Style Abacus The arrangement of an abacus is five beads per row as in Figure 1. A single row shows one digit. The top five beads are shown of 5. Four beads under the top show one respectively. And as shown in Figure 2, number from 0 to 9 is shown by sequence of the beads. Students using an abacus learn skills that not only the calculation using an abacus is quick but also a mental calculation that the student images the abacus their mind. If a student can use the mental calculation, a student can come out calculations quickly without the abacus [4].
Fig. 1. Sharpe of a Japanese abacus
An abacus is used to understand a basic calculation at classes of primary school. However, they don’t have enough time to study it. Therefore almost students can’t master the mental calculation using abacuses. A lot of students who can make the mental calculation have been to the abacus classroom besides school classes.
Fig. 2. How to show number from 0 to 9
3 The Android Platform 3.1 Overview The Android is a platform that it was developed Google and Open Handset Alliance (OHA) [5]. OHA is a group of forty seven technology and mobile companies. They
206
K. Saito et al.
released Android as an open source. We can use Android and a development environment for free. T-Mobile releases G1 [6] as a mobile phone equipped Android. A million units had sold from October 2008 to December 2008 in USA. In addition, Google have sold Android Dev phone that was removed to SIM lock and Hardware lock from G1 since December 2008. There are some features in Android. One of features is using the touch sensor and the acceleration sensor. So, a user can sensuously operate Android with there sensors. And the Android implements OpenGL ES as a 3D graphic function and a database corresponding to SQLite. Moreover, Android can synchronize Google account. Therefore, a user can take the schedule the user made and message of Gmail. 3.2 Execution Environment of Android Applications The Android runs on a Linux operating system. Applications for the Android operate by using Dalvik Virtual Machine (DVM) of Java Virtual Machine (VM) implemented on Linux. DVM is what Google developed originally for the Android, it is suitable architecture for operation on a mobile phone. DVM is different from normally Java VM. Google develops originally for Android. And this is suitable shape to operate on mobile phone.
4 Development of Application for the Android 4.1 Background and Purpose Various teaching materials like the abacus are used in schools. However, it is costly to the purchase of a lot of teaching tools and student can't freely take these teaching materials out of classroom. We developed such a teaching material as application. Students can use a teaching material anywhere by using the teaching material application. Then in this research, we paid attention to the Android platform for mobile phones. The mobile phone is small. So, students can use the application any time and everywhere. Students easily can get the mobile phone for mass-production. The used mobile phone is cheaper than other portable electronic instruments. Therefore, there is a possibility of spreading because people in the poor segment of the population and the developing country can buy it. In addition, Android is free that license fee as for use regardless of the efficient platform. Therefore, there is a good possibility that famous companies sell Android mobile phone and people using other mobile phone use Android. 4.2 Operation Method and the Main Function We develop the simulator of abacus using a touch sensor function. The start screen of a simulator of abacus is shown in Figure 3. It uses like an actual abacus. Students can move a bead to touch it as shown Figure 4. When a leaner touches it, the application emits the sound in which the abacus moves. Also, if the student moves their finger like the arrow shown in Figure 5, all beads are initialized as shown Figure 3. Of course, this operation is the same as an actual abacus. Thus, we expect that the effect of the calculation power improvement as well as an actual abacus to reproduce the operation feeling similar to an actual abacus.
Development of a Simulator of Abacus
207
Fig. 3. Start screen of a simulator of abacus
Fig. 4. An example of beads operations
Fig. 5. Initialization operation
And we developed the function to display number shown with the abacus by numbers to understand expression of peculiar number to abacus as shown Figure 6. Therefore, the inexperienced student to handling the abacus becomes easy to understand expression of numerical value that uses abacus. In addition, we developed the computational problem mode. This mode set questions of the addition. Students can see a problem and current answer at the lower side of screen as shown Figure 7. Students operate numerical values using the abacus. It is possible to become accustomed to the abacus by solving the problem of addition in this function with an abacus, and the calculation ability improvement can expect it.
Fig. 6. Display of numerical value
208
K. Saito et al.
Fig. 7. An example of the computational problem mode
5 Conclusion We developed of the simulator of abacus using Android. This simulator achieved similar to an actual abacus by using Android, and students came to be able to use the abacus anytime anywhere. We plan to add the function that students can practice the abacus and the calculation test to use the abacus application. As a practice function of the abacus, we will develop the calculation function that the student listen to the sound reading number and add them and the correction function when students mistake the operation of the abacus. And we will develop an application of Android not only abacus but also other teaching materials. Moreover, we opened the source file of this simulator of abacus to the public because we would like for this application to be used by many students. The application can be improved in the future by taking into account student’s opinions.
References 1. Android, G., http://code.google.com/android/ 2. Hikaru, A.: Mental Arithmetics by Soroban Method and Teachers of Soroban. Annual report of researches in environmental education 11, 97–110 (2003) (in Japanese) 3. Japan Ministry of Public Management: The survey of telecommunications trends in 2007 (2009) (in Japanese) 4. Oskar, Y.M.K., Jesse, C.Y.W., Blake, C.J.Y., Chi-Jen, L., Tak-Wai, C.: Maintaining Student Engagement in Extensive Practice by Implanting Gaming Factor. In: The 16th International Conference on Computers in Education, pp. 721–728 (2008) 5. Open Handset Alliance, http://www.openhandsetalliance.com/ 6. T-Mobile G1 with Google, http://www.t-mobileg1.com/
A Proposal for a Framework for an e-Alumni Program Using SNS Hiroshi Sano Faculty of Foreign Studies, Tokyo University of Foreign Studies 3-11-1 Asahi-cho, Fuchu-shi, Tokyo, 183-8534 Japan [email protected]
Abstract. In Japan, there has been some argument that many academic programs today are often inadequate from the viewpoint of practical education. However, it is difficult for universities to secure enough human resources to satisfy their students’ needs with sufficient service. TUFS, or Tokyo University of Foreign Studies, also faces the same problem as other universities do in Japan. One proposed idea to help solve this problem is to exploit the collective intelligence of alumni who have considerable expertise and experience in the real world. This paper introduces TUFS’s new development of a SNS application on academic education for the purpose of improving its services. A framework of the knowledge management of participants’ collective intelligence is also suggested here. This attempt proposes a general framework of SNS application on practical education of universities. Keywords: SNS, Implicit knowledge, Education.
resource to bridge the gap between the current situation at TUFS and students’ needs. In order to put this idea into action, a project was initiated to make a supportive alumni network using SNS (Social Networking Service applications for the students’ practical needs. The plan, titled “Student overseas support through an e-alumni program”, has been adopted as a MEXT (the Ministry of Education, Culture, Sports, Science and Technology) financially assisted program of 2008 under the category (or heading?), “Support program for contemporary educational needs” [1]. Though the initial purpose of the e-alumni program was to support students studying abroad, this paper is not limited to presenting TUFS’s experience of creating an alumni knowledge network with SNS. The experience can be fully extended to propose a general framework of SNS applications on practical education of universities. 1.1 Background Since TUFS’s degree programs feature diverse language and culture studies, many of its students study abroad for at least a short term. Since there is a wide variety of destinations for the students’ overseas studies, assistance for the students is acutely needed in addition to offering detailed information on daily life in the countries that they choose. However, it is hard to meet the needs with the limited budget and human resources TUFS has. In other words, the university’s organization alone cannot cover all of the needs of its students. In this dilemma, TUFS has started the “Student overseas support through an e-alumni program”. The project aims at organizing TUFS alumni’s collective knowledge through Information and Communication Technology, or more specifically SNS. The system will facilitate active communication among TUFS, its alumni and students, and enable the exchange of necessary, practical information. 1.2 Overview of the Project The e-alumni program can be described as an experiment to help solve some of the issues that TUFS and most other Japanese universities have. TUFS expects SNS to be
Fig. 1. E-alumni project outline
A Proposal for a Framework for an e-Alumni Program Using SNS
211
applicable to the project because of the following reasons. SNS sites have been extremely popular as a means of sharing information and contents on the Internet. The idea of SNS is often associated with one of the most popular websites in Japan called Mixi. Although SNS sites are considered a kind of circle or community, they are thought to evolve into a virtual world for people who can share their expertise with people in similar fields. There are two points that are discussed in the following sections describing our attempts to make this a successful project. Section 2: A framework of SNS settings for an e-alumni program Section 3: A methodology for e-alumni knowledge management
2 A Framework of SNS Settings for an e-Alumni Program In the course of planning the e-alumni program, we found that an effective alumni network has to meet two needs: 1. Form identifiable specialist groups in a virtual world 2. Provide a usable ubiquitous network environment for users to share technical knowledge in their groups These terms can be met technically, but need to give more consideration to facilitate the use of the network. The points are: 1. Organize user groups, paying careful attention to their expertise, and operate the groups successfully 2. Profile users need to contribute to intended educational purposes so that users can accept and send information effectively 2.1 Organizing Users To organize users, an administrator of TUFS will be in charge of forming an SNS community. When alumni wish to make a new community, the administrator must permit and create the requested community. This is to prevent the SNS from forming Table 1. User permissions
permission community forming
invitation community
to
University
alumni users
Students
(under the community guidelines for forming specialists groups)
(partially allowed for the purpose of alumni communications)
(allowed to make a profile for himself )
(under the purpose of committing educational plans for the students)
(partially allowed for the purpose of alumni communications)
(partially allowed for the purpose of user communications)
212
H. Sano
unexpected communities which do not suit the university’s purpose of offering its students practical applications. Once the community is prepared, the applicant will take over its management. By making a closed society shown in Table 1, users’ privacy is protected and therefore it facilitates the entry of alumni with expertise and keeps the use of the SNS secure.
Fig. 2. Administrated community
2.2 User Administration User profile information is divided into two categories, “basic information” and “registered information”. When users log in, their home will be a community selected through the use of the basic information. The registered information allows users to search any expert groups that they wish to access. The profiles are used mainly for three types of purposes as follows: 1. To confirm the identification of the community members by each member 2. To confirm the identification of the alumni by the university 3. To confirm the identification of students by the university The following tables are suggested settings for the profile information and the range of disclosure (Tables 2 and 3). Table 2.
classification
administrator read write
person himself read write
other members read write
1 2
optional
3
optional
A Proposal for a Framework for an e-Alumni Program Using SNS
213
Table 3.
classification 1 2 3
range of disclosure person himself and administrator person himself and MY members All
Fig. 3. Three forms of access to resources
The detailed user profiles allow users to accept and send messages efficiently among special interest groups. Once the network has started its service, the contents that the users create themselves will become part of TUFS’s valuable educational resources. The resources will be accessed in three forms: (1) sharing information in the community, (2) exchanging information between members, and (3) browsing diaries of members mutually. 2.3 Operation of SNS Alumni Network Since the network includes a large amount of personal data, a stringent operation manual will be developed. To secure adequate operation, a server will be placed in the TUFS Information Collaboration Center. A full-time overseas study coordinator will also be posted at the center to be in charge of office matters. TUFS plans to make a test installation of the network in mid-2009.
3 A Methodology for e-Alumni Knowledge Management Along with the SNS network development, it is important to build a methodology for e-alumni knowledge management. Information provided by the alumni has to be effectively stored and has to be ready to be searched through and referenced in order to maximize the use of the expertise. To develop a usable method, two elements have to be considered: 1) to identify the features of the communities, 2) to establish the best way to categorize the knowledge so that it is suitable for the specialized communities.
214
H. Sano
3.1 Features of e-Alumni Students at a specialized college like TUFS receive professional education even when they are undergraduate students. Graduates usually engage in highly-specialized work where they contribute to society in various fields. To take TUFS for instance, the Faculty of Foreign Languages has seven (or six?) courses and 26 majors where language study is the main focus. Its purpose is to train language specialists who will work in the international arena. For example, many graduates are involved in various global intellectual professions such as import-export, global sales and marketing, or supervising an overseas subsidiary with their language expertise. 3.2 Organizing Knowledge Once the characteristics of the communities are defined, the next question is how to compile appropriate databases for the prospective knowledge exchanged on the SNS.
Fig. 4. Knowledge management process
A Proposal for a Framework for an e-Alumni Program Using SNS
215
The goal is to give a proper categorization of the specialized BOK, or Body of Knowledge. In the initial stage of usage of the SNS, the users exchange their knowledge based on the provided communities and concepts as described in Section 2. Database design should be started after some period of monitoring and storing the knowledge of the users. Since any gathered body of knowledge will exhibit some tendencies, appropriate categorization should be attempted after monitoring the stored data thoroughly by university professors. After the categorization is carried out, accordant indexes are released to the users through the top of the blog menu. Then the users will be able to follow the indexes when they post new information. The users can give the university feedback on the usability of the indexes, and the university will reform them accordingly. Through the continuous repetition of this process, the indexes will become effective enough in a few years. Actually, these indexes themselves will be a source of valuable knowledge that the university should use.
4 Summary of the Process in Designing the Framework for the e-Alumni Project As described, this project suggests the general framework of a SNS to be used for a university alumni network. The process is summarized here again, so that the concept will be easily understood and reproducible by other universities. 1. A university forms specialized communities that connect alumni and current students following and pursuing its educational policies and goals. 2. The users exchange information through SNS features such as chat, blogging, or Q&A. 3. The university monitors and stores the knowledge for a certain period of time, and its professors categorize the data with appropriate specialized indexes. By using constant feedback and maintenance of the categorization, the indexes will gradually become more sophisticated as time advances. These indexes and the indexing process themselves will serve as unique and valuable knowledge for the university. The indexing itself forms one part of the knowledge management system that the university should value. TUFS will execute this project this year and will experience the creative process involved in this knowledge management system. Then we can expect a specific report of the results that will reveal just how effective our proposed database will be.
5 Conclusion This paper proposed a framework of an alumni network through SNS technologies as a means of practical education. TUFS will inspect the framework thoroughly to prove its efficiency through its actual effort in supporting students with the e-alumni program. With more elaborate observation and development of the framework, it is hoped that practical applications in university education will be improved greatly.
216
H. Sano
References 1. Application for 2008 Ministry of Education, Culture, Sports, Science and Technology (MEXT) grant, Support Program for Contemporary Educational Needs ‘Student overseas support through an e-alumni Program’ application form, Tokyo University of Foreign Studies (2008) (in Japanese) 2. The Central Council for Education, daigaku bunkakai seido kyoiku department, gakushi katei kyouiku no kouchiku ni mukete (2008) (in Japanese), http://www.meti.go.jp/press/20080627007/20080627007-4.pdf 3. The Ministry of Economy, Trade and Industry, syakaijin kisoryoku ni kansuru kennkyuukai interim report (2008) (in Japanese), http://www.meti.go.jp/press/20060208001/20060208001.html
Supporting End-User Development of Personalized Mobile Learning Tools Marco de Sá and Luís Carriço LaSIGE & University of Lisbon Campo-Grande, 1749-016, Lisboa, Portugal {marcosa,lmc}@di.fc.ul.pt
Abstract. Mobile devices present great features for the support of pervasive learning and content personalization. This paper presents a framework which takes advantage of these features and supports end-users while creating their customized tools for their students. Additionally, the framework comprises means for teachers to include persuasive and motivational mechanisms and hints, promoting student engagement while pursuing their learning activities. We describe the framework’s architecture, its features, including the supporting guidelines and development process, and detail some of the already developed material and the results that emerged during initial trials and case studies, also stressing their contributions to the field of m-learning. Keywords: Mobile devices, personalization, multimedia content.
to students on a dynamic manner, reacting and behaving according to their needs. Globally, despite the great benefits that some of these experiences demonstrated [10,11], there is still the need to provide personalized content and, most importantly, personalized attention and access to the educational material. This paper describes a development framework which supports teachers and educators while creating and defining personalized educational artifacts and tools (e.g., books, tutorials, questionnaires, tests, videos) which can be accessed by their students through personal mobile devices. It aims at tackling the abovementioned shortcomings of existing software by providing means for developers (e.g., teachers) and users (e.g., students) to interact with the same tools but achieving different goals. In general, the framework encompasses a set of modules that support the creation and utilization of different learning tools which range from simple homework forms, to elaborate tests, annotations and multimedia books. The framework also offers the possibility of exchanging information and supports collaboration between teachers and students, including analysis tools and synchronization mechanisms. All the tools take into account users’, devices’ and content requirements, allowing teachers to adjust the final user interface to various dimensions, including content and domain, accessibility features and multimodal capabilities. We address the framework’s various functionalities and features, stressing its architecture and extensibility choices also focusing the inclusion of multimodalities to support universal access to the learning material and to the inclusion of behavior in order to extend the teacher’s presence outside classes. The goal is to arrange persuasive user interfaces that compel and request students to engage in certain endeavors throughout their daily lives or whenever accomplishing their educational tasks. We explain the requirements that emerged throughout the design stage and present the solutions that were created in order to overcome existing challenges. Afterwards, we detail the framework’s functionalities that pertain directly to accessibility, adaptation and persuasion, addressing the issues that are focused and the support that is provided while end-users develop their own material and tools. Finally, as a validation we present two case studies where teachers defined their own content and created the adequate containers, including presentation and interaction dimensions, to support a better learning process for their students.
2 Framework’s Architecture The end-user development framework is divided into several tools, available for two main platforms. The first and main tool is the interface/artifact builder. It allows endusers/power-users (e.g., teachers or tutors) to design and create the educational artifacts, selecting content, arranging the user interface layout and personalizing the overall artifact to several dimensions (e.g., student’s needs, location, content). This tool is available for desktop computers. As a result from the utilization of this tool, and as depicted in figure 1, XML files containing the artifact’s specifications are created and can be later interacted with on the runtime environment. The runtime environment is the tool that allows students to engage on their learning activities while using mobile devices. Globally, it reconstructs the specification of the artifact, contained within the XML file and materializes it into an interactive
Supporting End-User Development of Personalized Mobile Learning Tools
219
Fig. 1. Overall framework architecture
application which contains the selected content (e.g., images, textual questions, videos, e-books) and allows students to browse the content, respond to questions, take annotations and transfer data to other students as required. Currently, there are runtime environments for Windows Mobile Devices and PalmOS devices. A Windows version was also developed in order to allow students to use their learning material on TabletPCs or laptops. During the utilization of the artifacts on the runtime environment, logs and results are generated, containing information about usage patterns, the student’s selections and answers [3]. These can be exported to a desktop computer and analyzed on the framework’s last tool, the result analyzer. The result analyzer includes means to review logs through a video like mode or to browse results screen by screen or element by element. On a lower level, the framework is divided into several components which support standardization of formats, reutilization of software elements and, most importantly, the framework’s scalability. As shown in figure 2, User Interface guidelines and artifact templates feed the developer tool allowing end-users to utilize them and compose the educational artifacts accordingly. In addition, different element libraries, created for specific contexts/domains (e.g., drawing elements for arts classes; pictorial books for small children; multiple-choice questions for homework) can be used. These are used during development and can be easily arranged and dragged into the educational artifact and are later used on the actual artifacts. Each artifact is composed by a set of screens and each screen is composed by a set of elements (e.g., question and image; video and audio clip). The UI design guidelines and artifact templates can be added to the developing tool as XML files. New element libraries need to be programmatically created but can also be imported into the developing environment.
220
M. de Sá and L. Carriço
Fig. 2. Component Architecture
3 Development Support Given the highly dynamic content, structures and domains that the developed framework targets; its main contribution over previous work is the ability that it provides, for end-users, to create their own educational artifacts. Besides allowing teachers/educators to customize content and the way it is presented and used by students, it also copes with the need to adjust the artifacts whenever required without any intervention by software programmers or designers. However, taking into account that most teachers/tutors do not have any design background or programming skills, the design and development of these artifacts has to be supported, wizard-based and must not require previous programming/UI design knowledge. Accordingly, the developing environment is composed by a wizard-based interface which allows teachers to create screens sequentially, dragging and dropping the selected elements that compose the screen. Figure 3 shows the overall layout for the artifact developing tool. On the top, teachers can select items and elements from the available pool and libraries. Beneath the selection pane, there are the two configuration options. The first allows teachers to configure the artifact’s and elements’ presentation while the second displays the various interaction possibilities. On the right side of the tool, an automatic preview of the resulting screen is presented along with the artifact’s main data. In addition to the various configuration options that pertain to interaction, presentation and content, the developing tool is also able to enforce usability guidelines, which ensure that the resulting artifact will be usable by the end-users. To achieve so, it loads the XML file with the design guidelines as previously detailed. Usability guidelines that pertain directly to the used device and its characteristics (e.g., screen resolution; touch screen or physical keyboard) can also be used (Figure 4).
Supporting End-User Development of Personalized Mobile Learning Tools
221
Fig. 3. Artifact development wizard. Configuration options and available elements on the left and screen preview on the right.
The guidelines automatically arrange screen elements according to specific values that can be configured and updated according to the targeted domain. Currently integrated guidelines allow the automatic docking and adjustment of elements to screen sizes and layouts. Moreover, the amount of elements per screen, as well as the amount of content per element (e.g., number of sentences within a label, size of an image, number of options within a list) can be limited according to various aspects (e.g., screen resolution, orientation). As a complement for these guidelines, when available, the system is also able to provide suggestions and alternative approaches to the current design. For instance, combo-boxes can be automatically replaced by radio buttons whenever options are longer than 25 characters or vice-versa when the list contains more than 5 items.
Fig. 4. Artifact development wizard. Device related usability guidelines.
222
M. de Sá and L. Carriço
Overall, the entire design and artifact construction process is straightforward and supported by the design guidelines and offered features which help end-users while creating and personalizing the educational material.
4 Accessibility and Persuasion To provide access to different students with different dis/abilities, the framework includes elements that allow teachers to create accessible artifacts. Besides the various configurations and common interaction options that can be used to interact (e.g., text input, multiple-choices) or to visualize the content (e.g., images, video), multimodal elements are also available. These include text-to-speech features which recreate the content through audio output. For instance, a question that was provided through a text-label element can be automatically recreated as an audio file that can be listened to instead of read. Other type of content can also be transformed into audio. For instance, images can be tagged with a description which will be transformed into audio or can contain an audio-only description directly recorded by the developer/teacher. The same principle applies to all other elements that can be used to create the artifact. Gesture-based navigation was also included for situations where eyes-free interaction is necessary. Furthermore, to facilitate eyes and hands-free interaction, the main basic navigation options (e.g., next, previous, play, pause, stop) can also be issued by voice, allowing the artifact to be utilized by, for instance, visually impaired users. Additionally, as a complement to the accessibility features, and in order to create proactive digital learning documents/artifacts that offer hints and aids to users, manuals that omit or show new information according to the user's performance while completing it, and overall motivate students whenever needed, the design framework also includes the customization of the artifacts' behavior. Behavior can be configured through the definition of rules. Rules are composed by time, interaction or content triggers and a consequent action. They can be attached to a specific element or to the entire artifact. On the former, depending on the interaction or usage within a specific element, a certain action can take place. On the latter, the interaction within various elements of the artifact (e.g., sequence of navigation or time to browse through various elements) defines the entire artifact's behavior. Time-based triggers can be configured to prompt warnings or change elements according to the time the user is taking to review/complete them. They can also be used to define alarms that alert users to use or complete an artifact at a given time. Interaction triggers analyze the user's interaction with the device by counting clicks or detecting where on the screen the user has interacted. Content based triggers activate actions depending on the content of the elements. For example, when the user chooses an option from a list or a value within a defined threshold, a certain action can be triggered (e.g., selecting "YES" displays a warning). In concert with the rules that are defined for each item or artifact, four different types of actions can be selected. The first one prompts a message that is composed by the user while the second jumps from the current screen or element to another within the artifact (e.g., first screen, end of the artifact, drawing, previous element). The two remaining actions hide or show elements (e.g., if a user checked a radio button a textbox is shown).
Supporting End-User Development of Personalized Mobile Learning Tools
223
5 Case Study and Examples The design of the developing framework and the experimentation of the resulting artifacts followed a user centered design approach. Accordingly, throughout the entire process, end-users were involved and participated by experimenting and providing their opinions on both the features and usability of the resulting tools. However, to fully validate its potential and features, the framework was also used to create educational artifacts for young students, for a variety of subjects. The main goal was to provide students with evaluation tests that could be answered on personal mobile devices (e.g., PocketPCs), regardless of their whereabouts. The framework was provided to two teachers in order to create the evaluation tests. Tests included several questions with different input and output elements (figure 5) and were distributed to a group of students. Two different sessions took place. One took place inside the students’ class room and the second outside the class-room, simulating a mobile scenario. All the results were gathered after the class and later reviewed by the teachers. Initial tests, composed by 20 screens/pages each, with two elements per page (e.g., one for output and one for input) took roughly one hour to create. On the second iteration (the tests that were completed outside the class), teachers composed the artifacts with video and audio elements and took between 20 and 40 minutes (figure 5, left). Students were given one hour to complete each test. Log revision showed that most students completed the tests in less than 30 minutes and revised their questions an average number of two to three times before locking their results. On a qualitative analysis, students responded to questionnaires and results showed that they appreciated the fact of conveying within one device their annotations, exams and manuals provided by the teachers. Hints, the varied pictorial content and the visual appeal were also very well received. The ability to respond to their questions through different modalities and access different content through different means was also one of the most highlighted benefits.
Fig. 5. Artifacts on the runtime environment - created with the development environment
224
M. de Sá and L. Carriço
For teachers the major contributions were the ability to configure artifacts according to various dimensions, adjusting them to the students’ needs, subject and content. Additionally, the ground breaking analysis that the log player features provided for class and homework and the inclusion of the persuasion/motivation mechanisms were deemed very important and useful.
6 Conclusions and Future Work The advances within mobile technology allow their utilization on a daily basis for a wide set of activities. As a result, they are now strongly integrated within the lives of a large set of the population and are especially appreciated and used with ease by the young adults, teenagers and children. Moreover, as they features and processing capabilities are further extended, they are becoming multimedia and multimodal devices which offer better support to highly interactive and demanding tasks. Educational activities and the m-learning concept are gradually making use of mobile devices and capitalizing these potentialities to new levels, enhancing both learning and teaching processes. However, the absence of personalized tools which simultaneously cope with the large variety of domains and subjects that are currently taught, added with the limited persuasion and motivational features, still hinder the widespread of m-learning tools. In this paper we presented a tool which supports end-user development, endowing teachers and tutors with the ability to create their own m-learning material. Additionally, the framework includes features which allow users to overcome previous limitations by personalizing content, taking into account accessibility concerns and introducing motivational and persuasive mechanisms to maintain student engagement even while away from classes and their teachers. To facilitate this process, the tool integrates design guidelines and offers a guided development process which, through the case studies has proved to be easy to utilize. Initial results have demonstrated the framework’s potentialities regarding customization and personalization and have inspired teachers to create multimedia and highly interactive educational artifacts that promote cooperation between students. For students, initial trials have also shown promise, especially the ability to receive aid and help from their teachers through a deferred modality and while away from classes. Current and future work is directed toward the definition of new element libraries and the inclusion of new interaction mechanisms (e.g., gesture recognition) and the integration of the mobile device artifacts with large displays, commonly used during classes. The enhancement of the developing and of the analysis tools is also envisioned. In particular, a mobile version of each tool is being designed, allowing the on-the-spot analysis of student’s results and the adjustment of the artifacts on-the-fly, which will be useful for mobile scenarios such as field trips to museums or other locations.
Acknowledgements This work was supported by LaSIGE and FCT, through the Multiannual Funding Programme and individual scholarship SFRH/BD/28165/2006.
Supporting End-User Development of Personalized Mobile Learning Tools
225
References 1. Virvou, M., Alepis, E.: Mobile educational features in authoring tools for personalised tutoring. Computers & Education 44, 53–68 (2005) 2. Sá, M., Carriço, L.: Handheld devices for cooperative educational activities. In: Proceedings of SAC 2006, Dijon, France (2006) 3. Sá, M., Carriço, L.: Detecting Learning Difficulties on Ubiquitous Scenarios. HumanComputer Interaction. In: Proceedings of HCI Applications and Services, 12th International Conference, HCI International 2007, Part IV, Beijing, China, July 22-27, 2007, pp. 235–244 (2007) 4. Theng, et al.: Mobile G-Portal Supporting Collaborative Sharing and Learning in Geography Fieldwork: An Empirical Study. In: Proceedings of JCDL 2007, Vancouver, Canada, pp. 462–471. ACM, New York (2007) 5. Costabile, et al.: Explore! Possibilities and Challenges of Mobile Learning. In: Proceedings of CHI 2008, Florence, Italy, pp. 145–154. ACM, New York (2008) 6. Sánchez, et al.: Mobile Science Learning for the Blind. In: Extended Abstracts, CHI 2008, Florence, Italy, pp. 3201–3206. ACM, New York (2008) 7. Lindquist, D., et al.: Exploring the Potential of Mobile Phones for Active Learning in the Classroom. In: Proceedings of SIGCSE 2007, Kentucky, USA, pp. 384–388. ACM, New York (2007) 8. Pownell, D., Bailey, G.D.: The next small thing – handheld computing for educational leaders. In: Learning and Leading with Technology, vol. 27(8), International Society for Technology in Education (2000) 9. Kortenkamp, U., Materlik, D.: Geometry teaching in wireless classroom environments using Java and J2ME. Science of Computer Programming 53 (2004) 10. Inkpen, K.M.: Designing handheld technologies for kids. Personal technologies 3, 81–89 (1999) 11. Alford, K.L., Ruocco, A.: Integrating Personal Digital Assistants (PDAs) in a Computer Science Curriculum. Frontiers in Education, Reno (October 2001)
Didactic Models as Design Representations Chris Stary University of Linz, Department for Business Information Systems, Austria [email protected]
Abstract. The contribution focuses on the role of didactic knowledge when designing interactive e-learning environments. Several representational approaches for the preparation of domain content and learning support have been developed. However, for the context-sensitive design of interactive artifacts not only the representation of particular aspects of learning is essential, but rather the propagation of didactic knowledge to functional services and interaction facilities. Such an endeavor requires the explicit representation of relationships between structure and behavior elements. Model-driven design supports the distinctive representation of multiple perspectives while allowing the mutually tuned refinement of design elements. In this paper a model-based approach for self-organized e-learning is presented. It supports the design of learnercentered knowledge acquisition by specifying user roles and learning tasks. We discuss the required enrichments of traditional model-based design approaches, due to the consistent tuning of high-level design elements, and the coherent propagation of task and user information to interaction services. Keywords: model-based design, e-learning, learning management, coherence, consistency, integrated specification.
Although several techniques for guiding the preparation and representation of content and transfer of knowledge have been developed, the benefits of model-based design have barely been explored. However, for context-sensitive design both, the representation and propagation of didactic knowledge to functional services, and the intertwining of model elements and relations, such as between the user and the domain model, are essential. They are discussed revisiting existing model-based approaches and reflecting recent e-learning platform developments.
2 Constructivist e-Learning: The Scholion Project Constructivist approaches to e-learning provide environments to explore information and guide learners actively to build individual mental processes that occur during the construction of mental representations. Active (re-)construction is seen as particularly beneficial for learners as they can pursue their individual interests, while they are motivated to communicate their understanding to others. As we know from studies in constructionism, the situated and public nature of any construction activity is identified as important [3]. Both, the individualized content, and the social aspect of learning processes have to be tackled by learning management. Such an understanding of e-learning is grounded in mathetics, a special science of (scientific) teaching centered on the operational processes of the learning, and dealing in particular with the principal aspects of conceptualization and cognitive organization. Didactics, defined as the strict scientific core of pedagogy, are thought of as inconceivable from any viewpoint other than a mathetic one [13]. Computer technologies for learning have opened up new avenues for designing content. In the sense of mathetics they can trigger active learner participation along knowledge provision and acquisition processes. To meet the requirements for computer-mediated context-sensitive and collaborative learning, it must be possible for learners to explore different categories of information in virtual environments and to communicate, so that meaningful learning of a domain can proceed in tandem with establishing communities of practice. Still, the ultimate goal is to create personally meaningful mental representations [6]. The significance of being able to treat learning as a socially valid exploratory activity, rather than a linear, pre-planned activity in detail, is recognized by coaches and developers step-by-step. One appreciation can be gained through looking at deeper issues than domain-specific structures of knowledge, web-design of user interfaces, or domain-specific methods. It addresses context from different perspectives: (i) the mathetic knowledge that drives the provision and acquisition of knowledge – developers have to look for a proper preparation process, (ii) communication channels utilized for learning and guidance – developers have to look for links of communication entries to content items, and (iii) information beyond the core of domain content, such as cultural issues like ethno-computing in computer science education – developers have to look for additional information to facilitate comprehensive understanding a topic. Our research so far has not only been targeting at mathetically effective content development, but also towards context-rich guidance and situation-sensitive learning ([12], scholion.ce.jku.at, www.blikk.it).
228
C. Stary
The frame of reference developed in the Scholion project initially requires the didactically relevant decomposition of learning material into so-called blocks, such as ‘definition’. They serve as focal point in the learning process and might be encoded into different media (text streams, images, videos etc.). Hence, multiple (re)presentations of content (polymorph content) may exist. Hyperlinks between blocks and media are likely, once linear learning material (information) is decomposed and de-linearized. In addition, different levels of detail (LoD) for each block can be specified, e.g., providing a slide for a definition on the top level (LoD 1) based on the full text of the definition on LoD 2, e.g., representing a textbook. Annotations constitute individual views on content items by commenting, linking, or highlighting items, block elements, or enriching content blocks. Some of those annotations can be links to communication entries of the Scholion communication components (chat, forum, infoboard etc.). In this way communication elements are directly linked to content blocks and vice versa. Communication can be established among peers for learning, as well as between learners and coaches. The latter, as quality managers, are responsible for improving content and structures based on learner input and feedback. In Scholion the content is arranged according to the aforementioned didactically relevant information types, conforming to current e-learning standards (cf. www.imsproject.org). They represent leaves in the hierarchy courseÆ moduleÆ learning unit. Currently, 15 generic types of this sort are available as part of an XML schema. They comprise definition, motivation, background, directive, example, self test, and other domain-independent content structures. Domain-specific block types can be added to support dedicated applications, such as proof when handling content from mathematics. Each block type can be visualized in Scholion through a certain layout, e.g., green background for proof. Block types allow learners and coaches to browse content along specific categories using a filter function. The workspace then contains only filtered block types. In this way learners might follow their interests and habits, such as starting to learn with studying background information. In order to establish self-organized learning processes Intelligibility Catchers (ICs) are offered to learners [11]. ICs are assignments made available through the bulletin board in Scholion. They have been designed to foster in-depth understanding of a topic. Their orientation section addresses the stage of capacity building when the IC should be used and what learners can expect when accomplishing the IC tasks. This section should motivate learners. It can be compared to an advanced organizer. The objectives set the scope in terms of the topics that are addressed and the understanding that should result from exploring and processing the topic information. It reflects the didactic value of the concerned learning unit(s), and serves as a means of orientation for learners. The task section contains a list of tasks to be achieved. It comprises a documented work and an intellectual work part. The documented work lays ground for the tasks to be represented in task-based model representations (see next section). It encourages active information search and processing as well as focused communication. The conference section sets the rules for establishing a corresponding community of practice, in particular when meetings or interactions should occur. The reference section provides links to material that helps to accomplish the tasks. The bulletins can be dynamically created, and are available in the Scholion information
Didactic Models as Design Representations
229
board. They are intended to facilitate the completion of the assignment. Finally, the departmental cuts reveal the estimated individual effort to meet the objectives. It might be expressed in terms of credit points or the estimated IC completion time. The structure combines organizational with subject-specific information. The information is arranged from a mathetically relevant perspective. For instance, the orientation section in the beginning informs coaches and learners when to use this IC, addressing competencies, the content involved, the rationale for choosing this content and for co-constructing mental representations. Initially, the learners are encouraged to identify those blocks of learning units where information is already available, i.e. part of the prepared content. In case of an IC for learning UML this can be design class diagrams. Then the learners are asked to enrich particular content items, e.g., UML class diagrams with task-specific information stemming from a case study of their choice. Typically after some practical work, such as modeling in UML, all results are shared with peers. In Scholion it is enabled by dedicated views on the original content. Views are like transparent slides put on top of the prepared content. They contain all content enrichments, such as comments, individual examples, and various types of links. As such, the content items are directly linked to discussion items according to the learning task, e.g., to develop an understanding about the proper use of UML class diagrams. All results can be validated by the coach through feedback, in order to ensure correct learner representations. The annotation facility of Scholion is considered as the key to individualization. It is based on a flexible hypermedia scheme for the representation of content elements. It enables learners to (i) mark a specific position in a content element for learning, (ii) post questions, answers or comments, and (iii) additionally link the contribution to a discussion theme from the system’s discussion board when working with content. The latter link may guide the user to the adjacent discussions of content. In case of realtime online connections, e.g., chats, the questions and answers can pop up immediately on the displays of all connected users (available in a buddy list). The referenced content elements can be displayed at the same time. In the discussion board topics of discussions can be created either manually by users or triggered by coaches posting a learning input. For the sake of traceability, each discussion contains a vector of contributions, being part of a certain discussion group, and ordered by relevant themes.
3 Model-Based Design Representations In model-based development the identification of models has been dominated by a top-down approach to the design and implementation of user interfaces[8]: The task and domain (object) model is located at the top layer of abstraction, as this model is expected to capture all activities that need to be performed in order to reach user goals. They might be detailed through properties (attributes), and arranged along a hierarchy representing causal or temporal mutual relationships. In addition, this model is expected to capture (business) objects that have to be manipulated in order to perform tasks. The abstract user interface model is located below the top layer, as it contains the logical structure required for task accomplishment at the user interface.
230
C. Stary
Abstract elements of interaction styles are captured in their mutual context, e.g., an area for data manipulation in a browser-like way. The relationships to user tasks or goals are encoded, e.g., to indicate that at a certain point in time options for navigating in a task hierarchy have to be offered. The final codality of information for presentation (text, graphics a.t.l.) and user control (modality) are not specified at that layer of abstraction. The concrete user interface model captures the refinements of all abstract interaction elements to more concrete ones. The initial abstract specification of style elements is replaced with one containing concrete attributes and relationships between interaction elements, including the modality of interaction. At that point, platform- and media-specific information is attached, in order to enable physical access to an application. The instantiation of the concrete model leads to an actual user interface, utilizing (standard) programming technologies, such as Java. The envisioned interplay of the models in the course of task-based design can be exemplified as follows: Consider booking a course at the university. It requires several sub tasks, such as finding proper options, selecting class data a.t.l.. This information is kept on the task (model) level, in addition to information for selection and manipulation, such as course. The subsequent abstract user-interface model captures all (categories of) interaction elements that are supposed to support each subtask. For instance, to identify course options an entry field for search has to be provided. Finally, the concrete user-interface model captures all specific interaction elements. For instance¸ in a web interface, search entry fields are supported by text bars after selecting a search engine (and site). The actual user interface can be constructed based on the specification of the concrete user interface model. Such a top-down approach focusing on layered design perspectives facilitates the use of multiple platforms and various arrangements of interaction elements. It encodes rather than makes transparent Gestalt design knowledge - knowledge that is mainly created in the course of mapping the results of user and work analysis (located in the top level task/domain model) to abstract interaction elements. The initially acquired (and represented) context of an interactive application does not directly guide concrete user-interface designs. A contextualized process throughout development requires a shift in model-based design (support), namely (i) a more detailed and flexible representation of users, work tasks, and domain elements, at least to the same extent as user-interface representations, and (ii) explicit conceptual relationships among design elements (inter- and intra-model-specific), considering universal access from a usability and softwareengineering perspective in a more integrative but self-contained way. Such an approach has to take a more networked rather than hierarchic perspective on model-based development. It requires a set of models that still enable the specification of an application model (in the sense sketched above), but details the user- and use-relevant elements, e.g., in terms of a task, user, domain, and an interaction model. The traditional engineering perspective has to be enriched with design variants capturing interactivity before considering functional implementation. The following set of models lays ground for model-based tools that recognize the need for supporting diverse user communities and situations of use [10]. The task model contains details of the organization at hand, the tasks, and the users’ perception of work. Task modeling includes modeling of the objectives
Didactic Models as Design Representations
231
users want or have to meet using an interactive application, probably in response to particular situations or events, as well as the different activities users have to perform to accomplish their tasks, as, e.g., given by global business processes. The user model contains a role model by reflecting specific views on tasks and data (according to the functional roles of users in organizations). It also captures individual user characteristics, capabilities or specific needs that developers need to take into account. The domain (data) model addresses all data, material, and resources required for task accomplishment in the application domain. The interaction model is composed of device and style specifications that designers need to construct a user interface or to generate a user-interface prototype. Hence, the structure of modalities and its related behavior are captured in that model, both at an abstract, and concrete level. For the developers’ convenience platform-specific specifications can become part of the interaction model. In the case where behavior specifications are missing, the preprogrammed, platform-inherent behavior might restrict the design space. The adaptation model contains all rules that trigger a particular structure and/or behavior of an interactive application. As such it enables interaction adapted to usermodel elements, such as left-hand assignments to interaction media (e.g., mouse buttons) based on consistent representations. It also enables the multiple presentation of task-information for navigation, e.g., tree views and acoustic trees. Since its scope includes all the previously listed models and their relationship, adaptation may concern task, data, interaction elements, and their inter-relationship.
Fig. 1. Model-based specification for e-learning tasks in ProcessLens
232
C. Stary
The application model integrates all five models from a static and dynamic perspective. In this way, it defines the entire structure and behavior of an interactive application. For interaction the users’ privileges, preferences, special needs, and tasks to be performed are taken into account when mapped to a concrete device or widgets of the interaction model. The ProcessLens approach [2] also expands the model-based design space, providing a design ontology for task-driven design. It can be used for task-based e-learning design as shown. Figure 2 demonstrates the enrichment to traditional design. It leads to a context-sensitive specification based on the mentioned models. For learning management the task model has to contain the tasks of the documented work-part of the IC for learners (see section 2) and the preparation of the IC by the coach. ‘Capture UML’ is decomposed into ‘IC-Specification’, ‘View Management’ and ‘Discuss’. ProcessLens also shows the role-specific responsibilities through a role-based user model. The coach ‘handles’ the preparation and the view management, whereas the learners handles views and forum entries in the course of discussion (see subtasks of ‘Discuss’ in Figure 2).
Fig. 2. Part of e-learning task hierarchy and roles in ProcessLens
A typical example of a design relationship is ‘handles’ between roles and activities. It binds responsibilities and allows for role-specific interaction domains. ‘Before’ demonstrates a causal relationship that has to be transformed to a set of temporal constraints in the behavior specification. A behavior specification is exemplified in Figure 3. It shows a coherent propagation to problem domain elements for being invited to a discussion. In the course of ‘Discuss’ peers have to be invited to join a certain community of practice. It is done by e-mail which requires to handle e-mails from the task and the problem-domain perspective. As content and communication are of equal importance in e-learning, communication facilities have to be represented in the problem domain model besides the content (decomposed to block types).
Didactic Models as Design Representations
233
Fig. 3. Model-based behavior coupling in ProcessLens
The interaction model is also decomposed into a structure and behavior part. The latter contains the life cycle of each widget and interaction style as required for device types. For instance, Scholion instances can also be used on web mobiles (see www.mobilearn.at). In ProcessLens the abstract and concrete layer of description is captured within a single model. The coupling of behavior sequences from other models is done analogously to the synchronization shown in Figure 3. Dedicated synchronization relationships have been developed to ensure behavioral consistency of design specifications. In order to implement a user interface for learners, relations have to be set between the task model and interaction model, as well as between the problem domain and the interaction model. The first link is required to navigate according to the tasks of the IC-documented work part. The latter is required to annotate content, and to communicate on the basis of linking content elements to communication entries. For interaction modeling besides generic UML structures (as used in ProcessLens), e.g., for modalities [7], XML derivates, such as USiXML [4] are widely used. They facilitate transformations as well as the dynamic creation of models, relying on various models. For instance, XIML [9] as user-interface description language supports the specification of various (linked) interaction models simultaneously. Its definition allows to capture task, user, presentation and dialog models as well as platform details. However, there is no executable relationship between the various interaction model parts, as provided by ProcessLens.
234
C. Stary
4 Conclusions As the pedagogical value of conveyed content and the related interaction features for learning support has become a distinctive factor of e-learning environments, design representations have moved to the center of interest in development. They have to capture the structure and dynamic context of subject items. Model-based design techniques not only allow different perspectives to be addressed, they also allow for capture of mutual relations between design models, both from the structure and the behavior point of view. In this paper we have proposed a model-based frame of reference and a corresponding tool supporting the development of e-learning applications. It focuses on the relationships between model elements to ensure consistent tuning and coherent propagation of design knowledge for self-managed learning processes. Of major importance are learning tasks to be accomplished by learners, as they require dynamic handling of domain content and communication elements at the user interface. Although the proposed adaptation model provides design constructs to represent this system dynamics, there is still research to be performed to achieve algorithmic support for automated model transformation.
References 1. Calvary, G., Coutaz, J., Dâassi, O., Balme, L., Demeure, A.: Towards a new generation of widgets for supporting software plasticity: The ”Comet”. In: Bastide, R., Palanque, P., Roth, J. (eds.) DSV-IS 2004 and EHCI 2004. LNCS, vol. 3425, pp. 306–324. Springer, Heidelberg (2005) 2. Dittmar, A., Forbrig, P., Heftberger, S., Stary, C.: Tool Support for Task Modelling – A ‘Constructive’ Exploration. In: Bastide, R., Palanque, P., Roth, J. (eds.) DSV-IS 2004 and EHCI 2004. LNCS, vol. 3425, pp. 59–74. Springer, Heidelberg (2005) 3. Farmer, R.A., Hughes, B.: A Situated Learning Perspective on Learning Object Design. In: Proc. ICALT 2005. IEEE, Los Alamitos (2005) 4. Limbourg, Q., Vanderdonckt, J.: Addressing the Mapping Problem in User Interface Design with UsiXML. In: Proceedings TAMODIA 2004, pp. 155–163. ACM, New York (2004) 5. Meder, N.: Didaktische Ontologien. In: Ohly, G.R.H.P., Siegel, A., Ergon, W. (eds.) Globalisierung und Wissensorganisation. Neue Aspekte für Wissen, Wissenschaft und Informationssysteme, vol. 6, pp. 401–406 (2000) 6. Norman, D.A., Spohrer, J.C.: Learner-Centered Education. Communications of the ACM 39(4), 24–27 (1996) 7. Obrenovic, Z., Starcevic, D.: Modeling Multimodal Human-Computer Interaction. IEEE Computer 37(9), 65–72 (2004) 8. Paternò, F.: Model-Based Design and Evaluation of Interactive Applications. Springer, Berlin (1999) 9. Puerta, A., Eisenstein, J.: XIML: A Common Representation for Interaction Data. In: Proceedings IUI 2002 International Conference on Intelligent User Interfaces. ACM Press, New York (2002)
Didactic Models as Design Representations
235
10. Stary, C.: TADEUS: Seamless Development of Task-Based and User-Oriented Interfaces. IEEE Transactions on Systems, Man, and Cybernetics, Part A 30(5), 509–525 (2000) 11. Stary, C.: Intelligibility Catchers for Self-Managed Knowledge Transfer. In: ICALT 2007, 7th International Conference on Advanced Learning Technologies, pp. 517–522. IEEE, Los Alamitos (2007) 12. Stary, C.: Sustainable e-Learning Communities. In: Akoumianakis, D. (ed.) Virtual Community Practices and Social Interactive Media: Technology Lifecycle and Workflow Analysis, IGI Global, Hershey, pp. 398–411 (2009) 13. Verpoorten, D., Pumay, M., Ledercq, C.: The Eight Learning Events Model: A Pedagogic Conceptual Tool Supporting Diversification of Learning Methods. Interactive Learning Environments 15(2), 151–160 (2007)
Interactive Learning Panels Ricardo Tesoriero, Habib Fardoun, José Gallud, María Lozano, and Victor Penichet LoUISE Research Group – Computer Science Research Institute (I3A), Campus Universitario S/N, Castilla-La Mancha University, 02071 Albacete, Spain {ricardo,habib,jgallud,mlozano,vpenichet}@dsi.uclm.es
Abstract. New sensing technologies, as RFID readers, are being incorporated into mobile devices to provide users with new interaction experiences. And, the combinations of these new technologies open up new challenging application scenarios. One of the areas that could exploit this potential is the design of interactive solutions for context-aware applications applied to learning environments. This article describes an m-learning environment that is enriched with new interaction features that are, or may, be provided by actual or future mobile device technologies. The proposed gesture based interface allows users to relate ideas and concepts through the improvement of traditional methods. This environment is based on the reuse of existent physical resources, such as the learning panels used in school classes. For instance maps, historical posters, timelines, and so on. These panels are improved with the low cost and widely used RFID technology that enables students to interact with them through mobile devices, encouraging the interest the students applying the constructivist education theory. Keywords: HCI, RFID technology, mobile devices, context awareness, collaborative environment, m-learning, social software.
New developments in mobile phone technologies are not only offering rich multimedia experiences to the users, but the possibility to implement new applications that exploits the physical environment of the users; for instance, GPS, accelerometers and RFID technologies among many other. The application of mobile technologies to the learning process is basically named mobile learning. So, from a pedagogical perspective, mobile learning provides a whole new dimension to support either traditional or novel educational processes. Some characteristics of this type of applications described in [1] include: the urgency of learning need, the initiative of knowledge acquisition, the mobility of learning setting, the interactivity of the learning process, the “Situatedness” of instructional activities and the integration of instructional content. These characteristics make mobile learning quite different compared to traditional classroom learning environment, where all the educational activities are carried out at a designated time and place. Although the use of laptops extends the range of education to places where wired connections are available; mobile technologies go beyond wires and take education possibilities to places where wireless connection is feasible. Another issue to take into account is the adaptation of mobile technology to contextual life-long learning, which is defined as the knowledge and skills people need to prosper throughout their lifetime. These activities are not confined to scheduled times and places as traditional education requires that are so difficult to achieve when people finishes formal learning. Thus, mobile technologies become a powerful tool to support contextual life-long learning, by being highly portable, individual, unobtrusive, adaptable to the context of learning, the learner’s evolving skills and knowledge [2]. Our proposal can be seen as an interaction improvement to m-learning systems inheriting m-learning advantages, as long-life learning is. So, mobile learning frees users from being anchored to a specific space, providing the possibility to explore the environment and interact with the world outside the desktop. This is an interesting point to highlight because it restricts student explorations to a fixed place limiting one of the most meaningful ways of building knowledge from a constructivist point of view that encourages the idea of discover to learn. Therefore, the main idea of this proposal is a novel interaction way where users are able to interact with the environment to build knowledge from relationships acquired from the ambient. There are many ways of building these relationships, in this paper we present a new device called the “Interactive Learning Panel” where students relate information that is presented by the mobile application to a physical region of a panel. It is based on the idea of relating concepts with lines, or multiple choice questionnaires. Concretely, we have implemented a PDA web based application where users have to relate a flag to a region on a map. For instance, if a flag is given on the PDA screen, students have to relate it to the country depicted on the map and vice-versa. These panels are equipped with RFID tags that represent concepts on the panel and the PDA is equipped with a RFID reader that is able to read these tags and detect how user relate these concepts through their readings. So, to relate the concepts exposed in the PDA screen to those on the panel, users have to put the reader (mobile device) over the graphical representation in the panel.
238
R. Tesoriero et al.
The structure of the article is organized as follows. In Section 2 we present the informal learning concept that is the learning theory foundation of this approach, and then in Section 3 we perform a review to the most related informal learning applications in the field. Section 4 presents the most relevant functional and technological aspects of “Interactive Learning Panels”. Then, in Section 5 a case of study where we expose a fully functional application applying these technologies. Finally on Section 6 we present conclusions and future work.
2 Formal and Informal Learning Combs in [3] defines informal learning as “the spontaneous, unstructured learning that goes on daily in the home and neighborhood, behind the school and on the play field, in the workplace, library and museum, and through the various mass media, informal learning is by far the most prevalent from of adult learning”. As principal features we can mention that it has no place in education to be carried out in normal life or professional practice. There is no curriculum and is not organized professionally instead, originates accidentally, sporadically, and sometimes, depending on the requirements of practical situations. Besides, it is not systematically planned pedagogically according to unconsciously, related to problems in a holistic manner, and related to the situations management. And it is directly experienced in his “natural” role as a tool to live and survive. The concept of informal learning, as used by Dewey in an initial stage and then by Knowles, has experienced a renaissance, especially in the context of development policies. At first, informal learning was bounded on formal school and non-formal learning in courses [3]. From organizational informal learning processes are not formally organized and are not funded institutions [4]. A broader approach is that of Livingstone which is geared towards learning and self-directed and self emphasizes self of the process by the student [5]. The form of learning from people in their informal work is 80%. The workers learn a lot more than look to others, trial and error, asking colleagues for help that formal training. However, defining informal learning is not so simple and is subject of an ongoing debate [6, 7, 8 and 9]. A complete vision is presented in [9] and suggests that informal learning should be defined as a learning process that occurs independently and casually without being tied to a directive or instructive curriculum by presenting a typology. Thus in the intentional informal learning, learning goals and process are explicitly defined by the teacher or the institution. And the learner is the one who defined the goals and the process of the intentional informal learning. On the other hand, in the unintended informal learning targets are not defined beforehand, and there is no prescribed learning process, but can be developed at runtime when a learning occasion arises. There are also hybrid types of learning, such as museums and schools. Although the case study presented in this article is suited to formal learning, it is not limited to it. As reader may have noticed, panels can be placed almost anywhere, for instance bus stops, train stations, museums, libraries, town halls, schools, etc., leading to a every time – everywhere learning environment.
Interactive Learning Panels
239
We have to highlight the fact that in this article we present a new way of interaction to learn, not the learning method itself. Besides, this interaction capability can be applied in both formal and informal education seemingly.
3 Learning Using Digital Technologies in Museums, Science Centers and Galleries This section describes the most relevant aspects of m-learning as part of learning in general. In this case, we relied on the report by Roy Hawkey, from King’s Collage of London. This report describes the experiences in the field of education at the scene, inside museums, science centers and galleries. The goal of this section goal is to show that we can relate ubiquitous computing concepts, through context awareness (mainly location awareness), to the learning activity. Moreover, our intent is to provide specific points of contact on this domain and the collaborative learning networks. 3.1 On-Site Learning The work presented in [10] indicates that there are two types of exhibits, static and dynamic. The later one is divided into three categories: automatic, operational and interactive. According to this classification, this approach suits into the dynamicinteractive category. On the other hand, [11] makes a distinction between exhibit (broadcasts facts) and informal learning. Main difference is the support level of personal interaction among users involved in the system. Participation is an important point in all kinds of social activities. Examples of different types of participation can be observed in: The “In Touch” that allows visitors to create their own page and access it after the visit, or at “Bristol, get connected” where visitors can compare their ideas in a variety of topics. Another example is the “Victoria and Albert Museum” that allows students to create their own digital images. And finally, the “Wish you were here” where visitors can use a digital camera and an editing program to create postcards. The collaboration is an important aspect related to education. The article [12] reveals that there are manly three ways of collaboration in these scenarios: real collaboration between learners (now present in an exhibition), virtual students (online) and in 3D virtual reality environment. Among the most prominent examples are: the “STEM project” [13] promotes the publication of ideas on the part of museum visitors physical or virtual, in an educational use of the National Museum of Science and Industry on the Web [14], and the “Keystone online project” of Franklin Institute, in which the activity-based research and professional development opportunities are combined into a Web site devoted to facilitating the teaching of science-based research. Our approach is based on real collaboration through the Web. Students are able collaborate with their teachers to check practices and even see exam corrections by email automatically. Even more, from this point of view, students are collaborating with their environment. In a producer-consumer scenario, panels play the role of information producers; and students play the role of information consumers.
240
R. Tesoriero et al.
3.2 Personalization and Mobility Flexibility is crucial for students to pursue their own paths and can do so in their times. In [2] they have developed the theory of long-life learning through mobile devices such as PDAs and others, considering the hardware, software, communications and the interface design. Among the features of these systems we can mention: y The high portability and availability to anyone who wants to learn. y They should be individually adapted to the learners’ abilities, knowledge and learning styles; and designed to withstand the personal learning, instead of the general office work. y They should be not obstructive, so that we can capture the apprentice situation and retrieve knowledge that the technology clogging the situation. y They should be available on either side, to allow communication between teachers, experts and peers. y They should cater the evolution of knowledge and skills of the apprentice. y They should be able to manage learning over time, in this way the apprentice accumulates resources and knowledge that will be immediately accessible despite changes in technology. For instance, reference work. y They should be intuitive for people who are inexperienced with the technology. The Sparacino article [15] describes a study in the MIT Museum, more specifically in the exhibit “Robots and Beyond” where the system tries to “understand the use and produce an output based on the interpretation of the user context intention”. To do so, it is based on the behavior (places time, visited objects, etc.). On the other hand, the project “The Electronic Guidebook” at the Exploratorium in San Francisco makes effective use of the PDA to make “Bookmarks” materials and then visit them [16] One negative aspect appears in two dimensions, mechanical and cognitive skills. The need to hold the device reduces the activities related with their hands while reading the conversation inhibited demand. The article viewpoint [17] describes the “investigation nomadic” in which the trainees can manipulate the information and conduct research while moving the physical exhibits. In this learning type, is threatening to replace the Excel interaction with gestures “mediated talks with others and cognitively challenging”. Discarding this, which requires a careful design of instruction, trainees may benefit from their mobility within a context of physical objects and without feeling exhibits socially or physically isolated. There are many alternatives of applying this approach. The case study set in this paper is about evaluation. However, the same idea can be applied to retrieve information that may be hidden at first sight. It provides users with the ability to explore information through a learning panel just by going over the panel with the mobile device. For instance, students can easily explore the panel surface discovering or augmenting information that may not fully available for physical space.
4 ILP Interactive Learning Panels This section will explain the implementation details of the system. This description starts with a conceptual description of how the system works, it continues with the
Interactive Learning Panels
241
software system architecture, then it explains the interaction model it was employed, and finally it presents the interface design that has been followed to model the interaction. 4.1 Conceptual Description The System is based on three key elements: the panel, the mobile device and the service provider. The panel acts as a medium to interact between the user and the system. Basically it defines “hot spots” where: commands can be selected from the panel just by the gesture of putting the device near a “hot spot”. For example, in a questionnaire, this gesture may be used to select an answer for a question, to go to the next question, go back to the previous one, to end the questionnaire and so on. On the other hand, the mobile device is a tool that enables users to interact with the panel and perform tasks according to the user behavior. It provides the physical link between a physical representation (in the panel) and information received from service provider. Thus, the service provider: It is responsible for interpreting information of the user (acquired through the mobile device) and the information store into the database system. Although the system can be easily extended, it supports the recovery and playback of multimedia content (audio, text, video, etc.). It also supports a learning specific application, the questionnaire. This tool asks users about information that is represented into the panel, and they have to answer pointing the right graphical representation on the panel. For instance, if users are asked about which region on the map is related to a defined flag, they have to point the region on the map to answer that question. The application will be exposed in detail on Section 5. 4.2 System Architecture Users receive multimedia content on their mobile device through a Web based architecture. As explained on Fig. 1, in a first step, the user perform a simple gesture of bringing their mobile device to the panel, exciting, through the mobile device RFID reader magnetic field, any tag deployed on the reverse of the panel. This action results on the retrieval of the tag unique identifier. Immediately afterwards, the mobile device sends this identifier to the Web server. The server processes this request interpreting this identifier as a command, and generates a response to the action. The
Fig. 1. System Architecture
242
R. Tesoriero et al.
answer may result on a resource transfer (video, photography, audio, text, etc.) or information, stored in the database system (for example, the result of a question). Thus, users finally perceive on their mobile devices the result of the interaction between them and the panel. 4.3 Interaction Modeling The system was implemented using the MVC (model-view-controller) architecture depicted on Fig. 2. Through this architecture we mapped the rugged natural gestures of user commands to certain events and actions that are processed by our domain model.
Fig. 2. The distributed Model View Controller implemented by the system
We are facing a distributed system. So, on the client side, the RFIDReader entity signals to the EcoPanelModel entity that a tag was read. The EcoPanelModel transmits the identifier to the server through a proxy represented by the ServerProxy class. This identifier is processed by the server and is mapped to a Command (for instance: next question) or ToolModel (a questionnaire) as necessary. Thus, as a result of the previous process, EcoPanelModel, which is part of the customer, receives the result as a set of serialized instances.
5 Case of Study For the case of study we will present one of the developed projects we made in our research laboratory. It treats the idea of teaching the Geography in the school, the library or any other learning environment. What we want to present is a connection between the student and the material provided by this subject. This material will be presented in such a simple and intuitive way that students can understand and interact with it without the help of any tutor. The goal of the panel is the identification of spatial entities as countries, cities, seas, oceans, etc. in a given map (the panel). On Fig. 3-a the system asks students for personal information and operation modes. Students are free to choose between two operation modes: the exam or a practice. The difference between these two modes lies on whether results are sent to the professor or not.
Interactive Learning Panels
(a)
(b)
243
(c)
Fig. 3. Main application screens
Once an operation mode is selected, a question is displayed. The question may be the identification of a region in the map or a flag corresponding to a defined region on the PDA screen (Fig. 3-b). For instance: “Select a state flag and then locate it into the map”. At the right bottom of the page the user can observe the number of the question he/she is answering and the total number of questions. As mentioned before, to select something, he/she has just to bring the PDA device near to the flag or region image depicted in the panel. It is a simple and clear process to develop. Commands and information retrieval are preformed analogously. Once questionnaire was fulfilled by a student, the results are displayed on the PDA screen (Fig. 3-c). The panel contains the map, flags, and other interactive buttons that the user will need to realize the practice or evaluation process. The next item to describe is the interaction panel based on RFID tags. While Fig. 4-a shows the map panel from the student point of view, Fig. 4-b shows the implementation of the panel based on a RFID tag matrix. On the right of Fig. 4-a we can see the map of the political division of the country and on the left we can see the list of flags that represent all the states of this county. At the bottom of the panel, there should appear a control panel. Usually, they are: the go back or forward panel buttons, to go back and forward through the questions, the confirm and cancel panel buttons to accept or cancel any action, and finally the exit panel button, to exit with or without saving the activity sending results by email.
(a)
(b) Fig. 4. The map panel
244
R. Tesoriero et al.
6 Conclusions This article describes an m-learning solution that exploits mobile technologies that are actually available, or will be available in the near future allowing students to interact directly with the materials they have to learn. Besides, ILPs also provide information to teachers about students’ progress. This system is implemented using several technologies, including mobile devices which support communication, RFID technology enabling gestural interaction, and the subject material which is introduced to enhance the student participation. And among main advantages of the system is worth to highlight the following issues: y y y y y y
Implementation of new interfaces for any mobile device. New resources can be introduced in a scalable and open way through the Web. Accessibility features, as audio-text for blind people, are easy to add. Multi-lingual support for resources on the PDA side. Great flexibility due to the panels can be updated quickly and efficiently. Economic deploy, because passive lo priced RFID tags and RFID popularity.
As future work, we can mention the extension of this interfaces to existing social networks to expand the service, for instance to Google Maps. Besides this form of interaction is not limited to retrieving information, instead it allows the execution of parameterized commands that can lead to the creation of active control panels that can evolve dynamically. Acknowledgements: We would like to thank CENIT project (CENIT 2008-1019MIO!) and CICYT project (TIN2008-06596-C02-01) for funding this work.
References 1. Chen, Y.S., Kao, T.C., Sheu, J.P., Chiang, C.Y.: A Mobile Scaffolding-Aid-Based BirdWatching Learning System. In: Milrad, M., Hoppe, H.U. (eds.) IEEE International Workshop on Wireless and Mobile Technologies in Education, pp. 15–22. IEEE Computer Society, Los Alamitos (2002) 2. Sharples, M.: The design of personal mobile technologies for lifelong learning. Computers & Education 34, 177–193 (2000) 3. Coombs, P., Ahmed, H.: Attacking rural poverty. How nonformal education can help, Baltimore (1974) 4. Watkins, K., Marsick, V.: Informal and Incidental Learning in the Workplace (1990) 5. Livingstone, D.W.: Mapping the Iceberg. NALL Working Paper number 54 (2002) 6. Mocker, D.W., Spear, G.E.: Lifelong Learning: Formal, Nonformal, Informal and SelfDirected. Information Series No. 241. ERIC Clearing House on Adult, Columbus (1982) 7. Hawkey, R.: Learning with Digital Technologies in Museums, Science Centres and Galleries. A Report for NESTA Futurelab (Report 9). Future Lab. King’s College, London (2004), http://www.nestafuturelab.org/research/reviews/09_01.htm (Last access 26.08.05)
Interactive Learning Panels
245
8. Sefton-Green, J.: Literature Review in Informal Learning with Technology Outside School (2004), http://www.nestafuturelab.org/research/reviews/07_01.htm (Last access 26.08.05) 9. Vavoula, G.: KLeOS: A Knowledge and Learning Organization System in Support of Lifelong Learning. PhD Thesis, University of Birmingham, UK (2004) 10. Miles, R.S., Alt, M.B., Gosling, D.C., Lewis, B.N., Tout, A.F.: The Design of Educational Exhibits. George, Allen & Unwin, London (1982) 11. Bradburne, J.: A new strategic approach to the museum and its relationship to society. Museum Management and Curatorship 19(1), 75–84 (2001) 12. Galani, A., Chalmers, M.: Can you see me? Exploring co-visiting between physical and virtual visitors. In: Museums and the Web 2002, Archives & Museums Informatics, Pittsburgh (2002) 13. The STEM project, http://www.sciencemuseum.org.uk/education/item 14. Bazley, M.: The internet: who needs it? Journal for Education in Museums 19, 40–43 (1998) 15. Sparacino, F.: The museum wearable: real-time sensor-driven understanding of visitors interests for personalized visually-augmented museum experiences. In: Museums and the Web 2002, Archives & Museums Informatics, Pittsburgh (2002) 16. Semper, R., Spasojevic, M.: Devices and a wireless web-based network to extend the museum experience. In: Museums and the Web 2002 (2002) 17. Hsi, S.: A study of user experiences mediated by nomadic web content in a museum. Journal of Computer Assisted Learning 19(3), 308–319 (2003)
WebELS: A Content-Centered E-Learning Platform for Postgraduate Education in Engineering Haruki Ueno, Zheng He, and Jingxia Yue National Institute of Informatics, Japan 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, Japan {Ueno,Hezheng,Yue}@nii.ac.jp
Abstract. This paper proposes a general purpose e-Learning platform WebELS to support higher education in engineering and science especially for PhD education. WebELS consists of three major modules, i.e., Learning for selflearning, Meeting for Internet-based on-line meeting, and Lecture for Internetbased distance lecture, as an “all-in-one” system. Using an easy-to-use authentication interface non-IT users can edit their own e-Learning contents on their personal computer as a series of slides from PPT, pdf, image and video data and upload to the WebELS server for e-Learning. Audio and cursor can be recorded onto each slide and be play-backed in a synchronized manner for helping understanding. WebELS is a Java-based server system and functioning in a low-speed Internet environment. The WebELS software is available as an open source system and is used in universities and industry in Japan and Asian countries. Keywords: e-Learning, Internet, higher education, postgraduate education, engineering education, software platform, open source, Internet meeting, distance lecture.
such as advanced networking technologies, multimedia information processing technologies, and sophisticated software technologies, at lower cost and higher quality at global scale [2]. The role of universities is requested to change in the circumstance, in such a stream. The traditional model of Engineering Education in university is based on a combination of undergraduate education and graduate education by classroomoriented lecture. Most of Engineering Education in Japan is still in this way. Some universities are trying to introduce Internet-based e-Learning for correspondence courses for working students. This is because most professors are conservative on introducing e-Learning. However, social demand for graduate education in international scale is increasing, that results in a use of Internet-based e-Learning especially in a PhD program. In this paper we propose a general purpose e-Learning platform WebELS to support flexibility and globalizing of higher education in engineering and science especially for PhD education by means of advanced information technology. According to our analysis of PhD education features of e-Learning system should be different from those of undergraduate education. For example, typical lecture style in PhD program is discussion-based meeting. Presentation at an international conference is also a typical style of education. Test for evaluating the quality of knowledge that is one of key functions in undergraduate education is not applied. Standardized texts are not used in lecture. In addition, globalization of higher education in science and technology become strongly important. It should be noted that e-Learning platform should be available not only in advanced countries where high quality Internet is widely in use but also should be available in developing countries such as in Asia and Africa where low speed Internet is normally used. WebELS is designed to provide high quality eLearning environment for higher education in science and technology to meet these demands. Since WebELS is designed as a general purpose e-Learning platform, it is useful not only in education but also for business meeting in institutions and industry. Combined use of WebELS with Moodle [4] is also suitable idea of use such that functions of both systems would be featuring together. WebELS consists of three major modules, i.e., WebELS Learning for self-learning, WebELS Meeting for Internet-based on-line meeting, and WebELS Lecture for Internet-based distance lecture, as an “all-in-one” system. Using an easy-to-use authentication interface non-IT users can edit their own e-Learning contents very easily as a series of slides converted from PPT, pdf, image and video data on their personal computer and upload to the WebELS server for dissemination. Audio and cursor can be recorded onto each slide and be play-backed in a synchronized manner for helping understanding. WebELS is a Java-based server system that consists of Editor, Viewer and Manager, and functioning on a small PC server in a low-speed Internet environment. Every functioning module is downloaded from the server onto a user’s computer automatically as a Java Applet when he/she chooses a specific function in using the system via Internet browser such as IE. The WebELS software is available as an open source system and is used in universities and industry in Japan and Asian countries.
248
H. Ueno, Z. He, and J. Yue
2 Background Concepts of WebELS WebELS is designed to provide an advanced e-Learning platform for globalizing graduate education in science and technology focusing on authoring and dissemination of multimedia contents, aiming to assist instructors to archive their learning materials on the web for on-demand learning, on-line meeting and on-line lecture. We have analyzed the characteristics of PhD education from the point of views of eLearning [1,6,7]. Some key characteristics are as in the following. 1. PhD students are research partners as well as students, and activities as individual scientists are involved in education. Joining research meetings, presenting at international conferences are typical examples. PPT-based presentation followed by discussions is a typical style. 2. PPT-based lecture is a typical style of classroom lecture, and PPT-based playback with voice and synchronized cursors on a learner’s computer seems to be reasonable for on-demand self learning. High quality PPT slides with voice and cursor are requested to replay in a narrowband Internet. 3. Powerful authoring features for non-IT users are strongly requested so that professors can edit their own educational materials on their personal computers and upload them onto the WebELS server. 4. The e-Learning system must be used on multiple operating systems which include Windows, Mac OS and Linux in a global situation over the Internet. WebELS was designed to meet these requirements mainly for supporting global PhD education as a contents management e-Learning system (CMS). Java programming language was chosen for achieving a multiple OS system to be used in Windows, Mac OS and Linux. WebELS is an all-in-one e-Learning system consisted of three modules, i.e., Learning for self-learning, Meeting for Internet-based on-line meeting, and Lecture for Internet-based on-line distance lecture. PPT contents are converted into a series of image slides in an authoring session using the easy-to-use editor. Then, voice and synchronized cursor are attached to each slide by means of a voice recording function of the editor. Pdf contents are converted to images, i.e., slides as well. Video clip can be inserted into the list of slides by means of specially designed function of the editor. So, it is easy to combine PPT data, pdf data and video data on a user’s computer using the editor. The editor is automatically downloaded from the WebELS server when the user clicks editing function of the contents list as discussed in the following chapters. Image-based contents are easy to maintain and safety for copyright protection. Export and Import functions are available so that the author of specific contents could be moved from a server to a server. This function is useful to maintain WebELS contents in a distributed WebELS server systems environment. Same multimedia contents can be used for three styles of usage in a similar manner. That is, a content edited and uploaded to the WebELS Learning database is able to use for WebELS Meeting as well by copying it within the server by means of the Export-Import function. By means of an embedded voice meeting function it is easy to achieve an Internet meeting and Internet lecture in a narrowband Internet environment with high quality image slides. The quality is accomplished by means of a pre-downloading feature of the system. On-line zooming and cursor pointing functions support
WebELS: A Content-Centered E-Learning Platform
249
easy-to-understand slide-based on-line meeting and lecture. It should be stressed that the design concepts as well as key features have been carefully considered by the analysis of users. We are going to keep this policy in the future. An offline-viewing feature, named OFV, was added according to student’s request. By means of OFV a user can download a specific content from the server onto his/her computer and bring it without Internet connection for viewing later in offline. An OFV content is a package of viewer and content. In addition to bring it on user’s computer for personal learning it is possible to send by an e-Mail attachment and insert to another contents list maintained by another platform such as Moodle for distribution. Another policy in developing the system is that the WebELS software should be distributed as an open source. Because of this universities in developing countries will be able to install the WebELS system on their PC server through the Internet with free of cost. We believe that this policy would be one type of international contributions in globalization of higher education in science and technology.
3 Outline of the System Fig.1 shows an overview of the WebELS system. As shown in the figure, WebELS is a server-based system functioning on a Linux server. Every user can use the system over the Internet using a browser, such as IE. The editor is downloaded onto user’s computer by accessing to the server for authoring contents, and viewer is downloaded to user’s computer when he/she access to the server for self-learning, Internet meeting and distance lecture. The editor and viewer are implemented in Java and thus downloaded as Java Applets. Windows users are requested to pre-install JRE (Java Runtime Environment) through the Internet. Java programming language was chosen because it is powerful in developing a variety of functions and a multi-OS system was realized, i.e., the system provides every function for Windows, Mac and Linux users equally. Multi-language feature is another important character of the system to support a global use of e-Learning in science and engineering. The user interface is automatically switched to English or Japanese language by detecting Browser language. It is possible to append other languages in this way. It should be noted that the editor as well as the viewer, i.e., player, is working on a client computer in off-line processing. The processing in the Learning session is monitored by the server for managing the user’s operations that include Login, Logout, Downloading, Uploading, and so on. For on-line meeting and lecturing the server’s role is more. It includes On-line slide changing, On-line cursor positioning, On-line zooming, Voice meeting, and so on. Since PPT-based slides are predownloaded onto every members of a meeting as well as a distance lecture, high quality slide sharing by multiple locations are available in a low-band Internet environment. Downloading time is varying according to the speed of data transmission over the Internet from several seconds to several minutes. Figure 2 shows an example of viewing windows for on-demand self learning of WebELS Learning. Synchronization of sound and cursor makes easy to understand PPT-based lectures on user’s computer in a narrow-band Internet environment [3]. A
250
H. Ueno, Z. He, and J. Yue
Fig. 1. Overview of the WebELS system
Fig. 2. An example of playback for self-learning (Shimamoto)
content with viewer is downloaded from the server onto the user’s computer in advance, then playback is executed on it offline. The zooming function is available so that a learner can zoom-in at any position of the slide. A voice and cursor synchronization helps him/her in understanding a lecture. A user-friendly editing interface to create the content is installed within the editor as discussed later. Fig. 3 shows an example of two slides in on-line multi-point Internet meeting by using WebELS Meeting module. Several groups are able to organize Internet meetings independently by using the embedded functions such as ad-hoc presenter
WebELS: A Content-Centered E-Learning Platform
251
management, embedded voice line, on-line cursor control, on-line zooming. A virtual meeting room is automatically created when several users access to a specific content with specific password. One of the members can obtain the presenter’s right to control presentation just by clicking a presenter bottom on the viewing window. Presenter can be switched to another member by clicking this bottom any time during the session. The server has the monitoring function to manage this kind of multi-point meeting. Since WebELS is a download-based system, every user is requested to download the content onto his/her computer before starting the meeting. During the meeting session only control signals for changing slides and pointing cursor in addition to voice signal are transmitted over the Internet, and then high quality slides can be shared even in a narrow band Internet environment.
Fig. 3. An image of Internet meeting (Bourgeois/UNESCO)
One of the most important functions of WebELS is that easy-to-use authoring interface is available for non-IT users. Users can create their own contents on their own computer from existing PPT, pdf, image and video data by means of a specially designed editing interface. Fig. 4 shows the editing interface of the WebELS system. In creating a new multimedia content a user is requested to click the “Create a New Course” bottom on the contents list window assigned to him/her first. Then the authoring system is downloaded from the server onto the client computer as a Java Applet followed by initialization process to start functioning, and the editing interface appears as in the figure. Editing procedure is quite simple as follows. For a newly creating content, enter the title of a course into the first column, choose a category and subcategory, set a password to protect from unauthorized viewing if needed. Creation of a content is done as in the following procedure. Choose one of four types of data from PPT, pdf, image, and video by clicking a button and press the “Add” button as shown in the
252
H. Ueno, Z. He, and J. Yue
Fig. 4. Editing interface
figure. Then, choose a data-file on the computer according to the guidance. The identified data are converted to a standardized WebELS slide format and recorded onto the working file for further editing procedure. The slide list appears in the window as in the figure. It is possible to combine four types of data into one content using this interface by means of Add, Remove, Up and Down functions.
Fig. 5. Voice and cursor recording interface
Fig. 5 shows the user interface for recording voice and cursor onto a specific slide that is identified in the editing interface of Figure 4. Choose one in the slide list window, click the “Preview Record” button, then the slide appears as in Figure 5. Recording voice and cursor can be done by quite simple manner. Just click the “Record
WebELS: A Content-Centered E-Learning Platform
253
Audio & Cursor” button and recording process begins. Voice recording is achieved just by talking toward a microphone. If a cursor is used during talking, then synchronized record of cursor is done. By clicking the “x” button (erasing-window button) of the window the recorded slide is moved to the working file and a “recorded mark” appears in the front of each slide in the window.
4 Implementation Some key technologies in the implementation of the WebELS system are discussed in this chapter. More discussions are in [3,5]. 4.1 System Configuration In WebELS system, implementation of editor, player and manager is based on a typical Browser/Server (B/S) structure. Under this structure, all user interfaces are implemented on web pages. Main tasks are executed by server side and only small work is carried out on browser side, it will much lighten the loads on client-side computers. Moreover, this kind of system can be accessed and used without downloading or installing other tools beforehand. Fig. 6 describes the architecture of WebELS system.
Fig. 6. Architecture of WebELS system
Web server is responsible for users’ requirements. A customized HTTP server handles web-based user interface that includes HTML and JSP pages. Java Servlets response the interaction with the Java tools on client-side and perform data transformation with database server. Database server is to provide data storage for achieved contents, supports contents editing, manage user information, and verify course dependency. And all user account information is maintained by SQL database server.
254
H. Ueno, Z. He, and J. Yue
4.2 Voice and Cursor Synchronization In the WebELS system, a cursor file is embedded into each slide of the contents to simulate lectures’ action. Users can edit the slide and point a cursor onto corresponding slide to tip the topic they are talking on, or show the points they want to strengthen. Each slide has its own timeline in order to: !Syntax Error, .) record the cursor positions (x, y) chronologically; ii) store the information (s, f) for the embedded audio stream file simultaneously, in which s is the current state (play or stop) and f is frame numbers.
Fig. 7. Synchronization of cursor positions with audio stream
As in Figure 7, the cursor positions are synchronized with audio stream in each slide file by the time shaft. For examples: i) when t=t0, (s, f)=(play, 0) and (x, y)=(0, 0), the audio stream is started, and there is no cursor action; ii) when t=t1, (s, f)=(stop, f1) and (x, y)=(x1, y1), the audio stream is paused and the cursor is pointed to the location (x1, y1) on this slide; iii) when t=ti, (s, f)=(play, fi) and (x, y)=(xi, yi), the audio stream has processed fi frames and the cursor is pointed to the location (xi, yi) on this slide. Also a continuous audio stream can be dispersed at different cursor positions according to the frame numbers recorded by the timeline. Therefore the duration of audio stream is normally shorter than cursor action although both of the actions are simultaneously recorded. Compared with real-time recording system, the file size of the audio stream which was edited based on timeline was decreased.
5 Concluding Remarks WebELS is designed to provide an all-in-one e-Learning platform to be used in a narrow band Internet to support flexible education and internationalization of graduate education based on the analysis of requirements of Ph.D education in an international situation. WebELS is also useful for such as undergraduate education, employee education, life-long education and online-meeting for international projects. WebELS is available over the Internet at a Web-site such as [9]. Since WebELS is designed as a content-centered e-Learning platform an integrated use with LMS such as Moodle [4] should be useful for under-graduate education.
WebELS: A Content-Centered E-Learning Platform
255
Acknowledgement The authors would like to express sincere thanks to all persons who support the WebELS project of NII Japan, especially to Dr. Vuthichay Unpornaranveth for his contributions in designing and implementing the WebELS system, and to Prof. Nobuo Shimamoto for his contributions in concept formation and design of user interface from a point of views of non-IT users. The project is funded by such as Science Research Foundation of Japan, Research Organization of Information and Systems and The Graduate University of Advanced Study. We express sincere thanks to the eLearning project of UNESCO Jakarta office for further collaborations using WebELS.
References 1. Ampornarambeth, V., Zhang, T., Hadiana, A., Shimamoto, N., Ueno, H.: A Web-Based eLearning Platform for Postgraduate Education. In: Proc. the 5th IASTED International Conference on Web-Based Education, pp. 388–393 (2006) 2. Blinco, K., Mason, J., McLean, N., Wilson, S.: Trends and Issues in E-Learning Infrastructure Development. DEST, Australia and JISC-CETIS, UK (2004) 3. He, Z., Yue, J., Ueno, H.: WebELS: A Multimedia E-Learning Platform for Nonbroadband Users. In: Proc. ICCET 2009 (2009) 4. Moodle: http://moodle.org/ 5. Rahman, M.M., He, Z., Sato, H., Ampornaramveth, V., Shimamoto, N., Ueno, H.: WebELS e-learning system: Online and offline viewing of audio and cursor synchronized slides. In: Proc. ICCIT 2007, pp. 1–6 (2007) 6. Rahman, M.M., He, Z., Sato, H., Ampornarambeth, V., Shimamoto, N., Ueno, H.: WebELS E-Learning System: Online and Offline Viewing or Audio and Cursor Synchronized Slides. Proc. ICCIT 2007, 106–110 (2007) 7. Ueno, H.: Role of e-Learning in Engineering Education - Background and Outline of WebELS. In: Proc. International Workshop on Engineering Education (2008) 8. Ueno, H.: Internet-Based Distance Learning for Lifelong Engineering Education - A personal view and Issues. J. of Information and Systems in Education. 1(1), 45–52 (2002) 9. WebELS: Web E-Learning System, http://webels.ex.nii.ac.jp/
A Pen-Based Teaching System for Children and Its Usability Evaluation Danli Wang1, Tingting Ying1, Jinquan Xiong2, Hongan Wang1, and Guozhong Dai1 1
Institute of Software, Chinese Academy of Sciences, Beijing 100090, China 2 Jiangxi Institution of Education, Nanchang 330029, China [email protected]
Abstract. The computer has become more and more important in children’ life and learning. Various issues exist in the application of multimedia edutainment software and courseware. Therefore, we analyze the current situation and requirement of preschool education software, and present the development of pen-based teaching system for children. Lecturing courseware, annotation, adding contents have been provided, which make targeted classroom teaching convenient, and children interactive courseware function has also been provided to arouse their learning initiative and enthusiasm. After analyzing the feedback, we design and develop the improved version of the system, and evaluate the two versions through experiments. Finally, we propose some suggestions for its modification. Keywords: pen-based interaction, children, teaching system, usability.
A Pen-Based Teaching System for Children and Its Usability Evaluation
257
teachers’ need. Some software for Intelligence and skill development lacks of systematic, and not suitable for realizing whole classroom teaching. In addition, most of software is using the keyboard and mouse operation basically. Lecture activities need frequent outline, so the keyboard and mouse operation is less free and easy to operate than pen, and against children’s participation in classroom activities. In recent years, pen-based interactive technology has been widely used. Pen-based interaction has great advantage in teaching: the operation is free and simple in keeping with cognitive and operation habit. It will provide an effective means of interaction in teaching activities; In addition, Pen-based interaction is an effective tool to solve the input problem of Chinese characters computers. It is convenient to writing for teachers and helpful to learn Chinese writing for children [3]. According to the current situation of preschool education software and requirement of preschool education, combining with advanced teaching theory, we analyze the characteristics of children, and propose an pen-based teaching system and realize a teaching environment where teachers can fully play a leading role, while children can play more initiative and active, and participate in the teaching in order to promote the teaching effect. Through using the software in kindergarten, software functions and performance have been improved continuously according to the users’ feedback, and moreover based on the earlier version (version 1), a new version(version 2) has been designed and developed. These two versions have been evaluated in usability. We will analyze the evaluation process and results in detail and give the system improvement comments and suggestions.
2 Related Works 2.1 Pen-Based Interaction Technology Natural interaction and human interface is the development trend of human-computer interaction. Pen-based user interface develops with the emergence of pen-based interaction devices like the handwritten pen, the touch screen and so on. The earliest pen-based interaction devices appeared before mouse and graphical user interface, which appeared in the system of Sketchpad in 1963[4]. With the progress of computer software and hardware technology, pen-based computing environment with features of pen-based interaction is rapidly developing. One of the obvious characteristic in pen-based interaction is easy to control, effective to use, natural to outline [5]. Both text and graphics can be naturally outlined and displayed on the screen; therefore, applying pen-based interaction to education has become a new hotspot. It not only makes the full use of natural pen and paper way of working, but also make up for the limit of estimating the position of mouse click on and contents input from keyboard, simply teachers’ operation, reduce controlling difficulties [3]. 2.2 Educational Software for Children Computer technology is affecting the education and learning, especially network and multimedia technology, which is very suitable for helping learning. Computer technology has enough potential to improve students' score. Multimedia electronic
258
D. Wang et al.
courseware combined with text, video, and images has completely changed the traditional teaching and learning mode. The multimedia teaching process can be regarded as the process in which teachers make use of multimedia technology to present study materials and let students to obtain information/knowledge. In the learning process by using multimedia, multimedia is not only an information transmission tool, but also means of helping students understand knowledge [6]. Multimedia technology which combines various media such as text, images, audio, video, animation and so on, can attract children's attention, arouse their interest, create the scene atmosphere, arouse their emotional development, enhance the teaching effect through the organic combination of teaching information, promote children's multisensory development through improvement of interpersonal interaction [7]. Currently, many researchers perform in-depth studies in children’s education software, and have developed some education software. Britain’s Sussex University studied the learning environment that supports children’s learning of biological concepts [8]. Carrie Heeter et al. studied software suitable for girls to learn animals [9]. Several large software companies also marketed multimedia course development tools, e.g. Microsoft’s PowerPoint allows users to conveniently produce texts, graphs, adding images, audio, motion pictures, video, etc. It also allows design of presentation effects based on needs. Macromedia’s Authorware is an editing platform based on icons and lines, multimedia production software for developing Internet and on-line learning applications [10]. Software such as Action, Flash provides tools for producing course materials. However, the cost of learning this kind of software is high, the operation is complex, and special training is necessary. Therefore it will take ordinary teachers long time to make courseware, the efficiency is very low. Software are not satisfied with the requirement for preschool education, because of not considering the characteristics of preschool teachers and the situation of children education, therefore, they cannot be used for most preschool teachers. In addition, there are some series of courseware, such as WaWaYaYa etc. [11], which are made according to some teaching materials. Coursewares provide abundant content of courses and multimedia display effect. However, this kind of software does not allow teachers to modify the content of courses and reuse courseware materials, which constraints teachers’ activities. Therefore, developing the teaching system for children is proposed in the paper. It is suitable for Chinese children, convenient to teaching activities for teachers. Simultaneously, allow children to participate in teaching activities, improve their learning enthusiasm and initiative.
3 Design and Realization Pen-Based Interaction System 3.1 Platform Architecture Based on above analysis, we developed a pen-based children teaching system, the framework of which is shown in figure 1. The top layer is pen-based user interface; the middle layer is data processing layer; the lowest layer is system database, primarily comprised of courseware base, resource base, constrain base, gesture base and text base. Users can communicate with system through pen-based user interface. Input
A Pen-Based Teaching System for Children and Its Usability Evaluation
259
Fig. 1. Architecture of System
gestures are transmitted into system. The system first analyzes and comprehends the input. Through the support of recognition algorithms, the system obtains the input intents of the user, and performs the corresponding operations. Then the system’s responding results are output to the user interface. 3.2 System Function The function of the system is shown in figure 2. It is divided into four modules: basic operation module, media interaction operation module, handwriting operation module and auxiliary operation module. Among them, the media interaction operation includes pictures interaction, text interaction, audio, Gif animation control, and track-based animation control, all of which have their own interaction properties (such as: click to show, click to hide etc.), and allow users to interact freely.
Fig. 2. System Function
3.3 System Realization The teaching system was developed in VC++ 6 based on our lab existing pen-based software development platform. This platform provides basic pen-based interface and gesture lib. Users go through pen to interact with the system. The system first analyzes and comprehends the input, carry on the according processing and operation, then the system’s responding results are output to the user interface. The system interface of Version 1 is showed in figure 3-4. This system has been provided for more than kindergartens to try out. Analyzing the users’ feedback, we propose an improved version (Version 2), which is showed in figure 5-6.
260
D. Wang et al.
Fig. 3-4. Interface of Teaching System (Version 1)
Toolbar in left
Toolbar in bottom
Fig. 5-6. New Interface of Teaching System (Version 2)
4 Usability Evaluation of the Teaching System 4.1 The Evaluation Method Among all of the evaluation methods, user testing and questionnaire are selected to evaluate the teaching system. The tester first finishes a group of tasks according to the requirements, then he or she answers the questions. In order to let the tester learn the teaching software quickly, the training scheme and testing tasks were devised using interface scenarios, and this method has been proven to efficiently improve learning rate of the user and finishing rate of the tasks. 4.2 Evaluation Process Participants and Software The users of our system are mainly teachers who teach students by the teaching system. Therefore, we choose 6 testers three male and three female. All of them are 20 to 30 years old. In addition, some of the testers were ever teachers. The evaluation software is the teaching system developed earlier (to be called version 1 indicated with V1 later), another is the new teaching system in this article (called version 2, indicated with V2 later). The differences between two versions are that we add a dragged function toolbar in the version 2. The purpose of this evaluation is to understand what kind of interface testers like more and test various availability indicators of the teaching system. In this evaluation we adopts contrast experiments for two versions, we will also carry on the contrastive analysis for two versions’ test data.
A Pen-Based Teaching System for Children and Its Usability Evaluation
261
Evaluation training The staff member first introduced the purpose and arrangement of this experiment, then explained the usage of the software with a scenario-based method, and offered a demo. Then, the testers had 10 minutes exercise. At last, the instructor introduced the testing requirements and tasks. Testing and questionnaire After the exercise, the testers began to test according to the task requirements. The staff members recorded the errors and other problems during testing. All the errors and problems did not be prompted. After testing, the testers filled out the questionnaires. 4.3 Result Analysis We mainly analyze the test result from two aspects. One is objective aspect, analyze data recorded during the test procedure, such as wrong number, time of finishing tasks and so on; The other is subjective aspect, testers’ questionnaire survey results. Objective data z Integrity Table 1 shows the integrity which the testers finished each task in, and the data is analyzed using statistical method, and the average integrity and deviation of finishing every task were obtained. Table 1. Integrity of finishing tasks (the highest is 1) Task V1 Lesson preparation V1 class begin V2 Lesson preparation V2 class begin
： ： ： ：
Mean 1 0.98 1 0.98
s.d. 0 0.03 0 0.03
From table 1, we can see the achievement of testers in two versions is the same. According to task completion rate, the whole task can be basely finished. In version II, we have joined the dragged toolbar, however from the objective data, it is not difficult for testers to understand and operate. Integrity of the task doesn’t reduce. Therefore, the data shows that testers feel tasks difficult to understand or to implement during the test, and the tasks and operations are within their cognitive workload and operation scope. z The errors During the whole evaluation process, when tester operates, problems may occur. The reason may be the habit of the testers, or the operation error, or it may be that the system produced the errors. The statistical results of the errors are: during the process of finishing tasks for two versions, the numbers of system errors are both 0. It shows that our systems are running stable relatively. The error frequency of operation is also low overall. The number of errors in version 1 is more than that in version 2, such as that the pen attributes Toolbar in version 1 has hidden page turning buttons, thus results in wrong operations. The data is shown in the Table 2.
262
D. Wang et al. Table 2. The statistics of the errors Version V1
Group System error Operation error System error Operation error
V2
Error number 0 2 0 1
The occurrence rate of both in version 1 and version 2, errors are very low. Testers used the teaching system very fluently. It implies that the software has high reliability and naturalness. However, since errors occurred, there were some aspects should be improved in the system. We will focus on these errors, and get the clue of the further perfection. The results of questionnaire For the design questionnaire, the teaching system includes three parts: the first one is the overall evaluation of the teaching system, the second is evaluation of version 1, and the last one is the evaluation of version 2. In the 8 questions of part 1, the 1st, 3rd, 5th and 7th questions are proposed to version 1, the others are proposed to version 2. The average score of version 1 is 2.71, and 3.79 for version 2. Therefore, the synthetic evaluations for version 2 are higher than version 1. Testers prefer to use version 2 which has the dragged function toolbar. The questionnaire is consisted of 18 questions, adopting 5 degree evaluation table. It provides the overall evaluation of our teaching software version 1, including easy-learning, easy use, reliability, naturalness, interaction and satisfaction. The results are shown in Table 3. The usability evaluation score of version 1 shows in figure 7. Table 3. Usability evaluation for version 1
A Pen-Based Teaching System for Children and Its Usability Evaluation
263
Similarly, we also evaluate version 2 on easy-learning, easy-use, reliability, naturalness, interaction and satisfaction. The results are shown in Table 4. The usability evaluation score of version 1 shows in figure 8. Table 4. Usability evaluation for version 2
From above data, the average scores of each availability indicator in both versions are more than 4, which are high scores. The standard deviations are also in the normal scope. It shows that the evaluation result of the teaching system is satisfied. Especially in terms of Naturalness, the system has been given higher scores. It is mainly because we fully taken into account the users’ habits during the process of design and realization. Synthesized comparison between two versions is shown in Table 5 and in Finger 9. The scores of two versions are not very different, synthetical score in version 2 is higher. It shows that as a whole, synthesized evaluation for version 2 is higher than version 1. And in easy-learning, easy-use, reliability and satisfaction, the evaluation score in two versions is closed. It shows that joining the dragged toolbar doesn’t make users feel more difficulties. However, the evaluation of version 2 is higher than that of version 1 in two aspects of naturalness and interaction. It shows the dragged function toolbar makes users more convenient when using the teaching system, while in version 1, users cannot drag the toolbar at will. If the position of teachers is distant from the function buttons like page up or page down buttons, then the teacher may have to go to distant locations to find those function buttons, causing a certain amount of inconvenience. The introduction of the dragged function toolbar strengthens interface interaction of the entire system. According to testing situation and questionnaire survey, users authorize this interactive approach of the dragged toolbar.
264
D. Wang et al. Table 5. Synthesized comparison between two versions
Fig. 9. Synthesized comparison between two versions
4.4 Overall Evaluation For both version 1 and version 2, the integrity of finishing task is very high. Simultaneously the error rate is also quite low. The testers basically can complete the testing tasks well. Looking from questionnaire survey, testers have given good evaluations in both version 1 and version 2. The improvement that is the dragged function toolbar in version 2 has been approved. In addition to evaluation questionnaires, we have gotten some testers’ ideas from their feedback for the system, such as their favorite functions, improvement points. Testers feel that adding animation in the course of teaching has enhanced the effectiveness of teaching. Whiteboard function is also very appropriate for the system design; teachers can open "white board" for writing at any time. However, through the evaluation process, we have found the shortage of the software design. For instance, the dragged function toolbar can sometimes hide other buttons, or when the dragged toolbar moves to upper boundary, it cannot automatically close the border in accordance with the placing of the border, as well as some users want to operate more naturally and so on. We will improve it and do further research on how to make teachers interact with the teaching system more smooth and natural.
5 Conclusion This paper, focused on the issues in multimedia teaching and the needs of children’s education software, presented the development of a pen-based children courseware system. The platform provided for the teachers a convenient environment for lecturing
A Pen-Based Teaching System for Children and Its Usability Evaluation
265
courseware to facilitate multimedia teaching for the teachers, as well as for the children an interactive and interactive environment, where children can be allowed to participate in multimedia courseware explanation to deepen their understanding and memorization of the knowledge, so as to improve their learning results. This paper combined pen-based interface technology and the design of natural user interface to provide for the teacher and the children an easy-to-learn, easy-to-use teaching system. Through the use of system and feedback, we have modified the design of the system and obtain the new software version. Through comparing the new version with the old one, we present some comments and suggestions for improvement. Acknowledgments. The authors would like to acknowledge the support of Zhang Jie, Peng Fang, Wu Guangyu, Xu Li Shuang, among others. This research is supported in part by the National Natural Science Foundation of China (under Grant No. 60373056), and the CAS Pilot Project of the National Knowledge Innovation Program (Grant No. KGCX2-YW-606).
References 1. Mai, N., Neo, T.K.: Multimedia in education: developing a student-centred learning environment in the classroom, http://www.vschool.net.cn/english/gccce2002/lunwen/ gcccelong/41.doc 2. He, K.: The challenges in the development of educational informationalism. China Educational Technology 6, 5–11 (2006) 3. Wang, D., Chiu, S.C., Wu, G., Dai, G., Wang, H.: Pen-Based Children’s Instructional Platform Based on Cognition Theory of Multimedia Learning. In: PDPTA 2008, pp. 742–748 (2008) 4. Sutherland, I.E.: SketchPad: A Man-Machine Graphical Communication System. In: AFIPS Spring Joint Computer Conference, pp. 329–346 (1963) 5. Li, Y.: Research on Pen-based User Interfaces—Theory, Technique and Implementation. PhD dissertation (2002) 6. Mayer, R.E.: Multimedia Learning. Cambridge University Press, Cambridge (2001) 7. Li, X., Atkins, M.S.: Early Childhood Computer Experience and Cognitive and Motor Development. Pediatrics 113(6), 1715–1722 (2004) 8. Michael, S., Yvonne, R., Frances, A., Matt, D.: Designing for or designing with? Informant design for interactive learning environments. In: CHI 1997 (1997) 9. Carrie, H., Brian, M.W., Darcy, D.G.: Theories meet realities: designing a learning game for girls. In: DUX 2005: Proceedings of the 2005 conference on Designing for User Experience (2005) 10. Calvert, Children Journeys Through the Information Age (1999) 11. WaWaYaYa, http://www.wawayaya.net/guide/
Development of a Visualised Sound Simulation Environment: An e-Approach to a Constructivist Way of Learning Jingjing Zhang1, Beau Lotto2, Ilias Bergstom2, Lefkothea Andreou2, Youzou Miyadera3, and Setsuo Yokoyama4 1
Department of Education, University of Oxford, UK [email protected] 2 Lottolab Studio, University College London, UK 3 Division of Natural Science, Tokyo Gakugei University, Japan 4 Information Processing Centre, Tokyo Gakugei University, Japan
Abstract. In this paper, the design and implementation of a visualised sound simulation environment is presented as an initial step to further laboratory experimentation. Preliminary laboratory experiments showed a positive learning curve in human auditory perception. Learning occurred when new information was processed with relevant existing knowledge in this simulation environment. While the work towards the truth of the empirical hypothesis is still under discussion, this project has been expanded beyond the scope that was originally envisaged and the developed environment showed its potential to be adopted on mobile devices for many educational purposes. This initiative not only brings scientists and educators together, but it is also hoped that it represents a possible e-approach to a constructivist way of learning. Keywords: visualisation, simulation, constructivist, learning, mobile, visual, auditory.
Development of a Visualised Sound Simulation Environment
267
information from arrays of stimulation that specify the events, layout and objects of the world”. In this paper, we put focus on parts of the learning process that look to be somewhat independent from conscious or critical forms of learning (such as learning to solve a mathematical problem) but that are involved with rather low-level acquisition through the central perception system. Perception has long been considered as a modular function, with the different sensory modalities working separately from each other (Schlottmann, 2000). Recent studies in neuroscience and psychology, however, challenge this view by suggesting that crossmodal interactions are in fact more common than originally thought (Martino & Marks, 2000). A number of experiments involved with cross modality have been carried out (e.g. Butler & Humanski, 1992; Carlile, 1990). Recently, research on sensory perception has become both popular and integrated with fast development of computer simulation technology. However, the fundamental problem for any sensory-guided system (natural or artificial) is that the information it receives from its environment is ambiguous. Particularly, most computer simulated sound systems fail under natural conditions, or when forced to contend with an environment that is not explicitly represented in the system. How natural systems resolve this challenge is currently not known, though one increasingly popular hypothesis is that the problem is resolved empirically. That is, the system encodes the statistics of its experience with past sources of stimuli1. Despite of the increasing support for this view, however, there is currently no one that has tested it. One reason is that to truly test this view we would have to provide someone with a wholly new sensory modality, and then look to see how that person’s brain deals with the underlying ambiguity of the information it receives. This is of course impossible. What is possible, however, is to present one of the senses with a new kind of statistical experience in a simulated environment. Therefore, a visualised sound simulation environment was designed and implemented, as an initial step to further laboratory experimentation. The environment presented the visual stimuli with a new auditory experience in order to prove the ambiguity problem in vision is resolved empirically. Preliminary laboratory experiments showed a positive learning curve in human auditory perception. Learning occurred when new information was process with relevant existing knowledge in this sound simulation environment. While the work towards the truth of the empirical hypothesis is still under discussion, this project is extended beyond its initial establishment and the developed environment shows its potential to be adopted in different real-world educational settings, e.g. a music lesson and a drawing course. This initiative not only brings scientists and educators together, but it is also hoped that it represents a possible e-approach to a constructivist way of learning.
2 Mapping Model A Mapping Model (M) was proposed for translating any pixel in a real 2D image to a unique virtual sound pixel of an imaginary sound panel in two continuous steps. That 1
Constructivists such as Helmholtz H. and Gregory R. argue that external world cannot be directly perceived because of the poverty of the information in the retinal images. Since information is not directly given, we have to interpret the sensory data in order to construct perception.
268
J. Zhang et al.
is, a real image pixel is mapped to a ‘visual pixel’, which is subsequently converted as a ’sound pixel’ by using this Mapping Model (M). M
=
{R, I, S,
,
}
The elements of Mapping Model (M), are justified as follows: R: a matrix, which represents the real image input (which can be anything from hand drawn images to photographs or streaming video). I: a
matrix, which represents an imaginary visual panel.
S: a a virtual sound panel.
matrix, which symbolises
: a mapping function from R to I. A scaled and projected representation I of the real image R is produced by this mapping function : Map any point , to , by:
: a mapping function from I to S. It creates a sound panel isomorphic to the visual panel: Map any point , to , by:
1 1 1
Fig. 1. Mapping Model - w is the width of the matrix R, while h is the height of the matrix R. xNum is the width of the matrix I; yNum is the height of the matrix I; (panMAX -panMIN) is the width of the matrix S; y(freqMAX -freqMIN) is the height of the matrix S.
As we can see, The Mapping Model shows the process of converting a real image pixel (Pr) in an image R to a visual pixel (Pi) on a visual panel I and then subsequentis a ly mapped to a sound pixel (Ps) on a sound panel S. The mapping function scale formula to trim the input of a real image into a required standard image, which
Development of a Visualised Sound Simulation Environment
269
can be then mapped to a virtual sound panel. The mapping function then converts , to a sound pixel with an each pixel of the product of the mapping function unique property, such as loudness, frequency, and pan.
3 System Design and Implementation The invention of the screen, either on a computer or a mobile device, has completely changed the traditional work of designing. However, while it adds more interactivity and flexibility into most of our products in the daily life, it also unavoidably brings in complexity and ambiguity. It is said that miscellaneous designs has given most users a difficult time to get used to the interface. Building upon the proposed model, the designed system updated from the first working prototype with pop-up panels to a unique three-tier structure for maximising such seamless translation. The final design was based on and has been further refined by the feedback obtained during the usability testing. The look of such design was thus simple and appealing to users.
Fig. 2. Three Hierarchy Design - - showing the three-tier architecture
The top architectural level of the system was a two part top-and-bottom skeleton (Fig. 2), which included (in the top part) a menu bar and a painting canvas (in the bottom part). The second architectural level within the ‘painting’ canvas also consisted of a top-and-bottom structure. On the top, there was a ‘Space Panel’, which included an ‘Image Space Panel’ and a ‘Sound Space Panel, and at the bottom a ‘Control Panel’ (i.e. an ’Image Control Panel’ and a ’Sound Control Panel’). The third level was outlined as a left-and-right structure contained within the painting canvas, separated as ‘image’ and ‘sound’ (from left to right) in both the ‘Space Panel’ and ‘Control Panel’. That is, on the painting canvas, there were four panels, from top left - an ‘Image Space Panel’, top right - a ‘Sound Space Panel’, bottom left – an
270
J. Zhang et al.
‘Image Control Panel’ for controlling the ‘Image Space Panel’, bottom right - a ‘Sound Control Panel’ to control the ’Sound Space Panel’. The four panels were independently resizable and can be switched on and off for maximising the screen of any device, such as a computer or a mobile phone. The ‘Image Control Panel’ allowed users to change the resolution of the above image panel and the sound panel. There was also an image colour chooser panel consisted of two parts: the Preview panel and the Colour palette. The Colour Palette consisted of four tabbed panels: Swatches, HSB, RGB, and Gray Scale. The Gray Scale was a slider bar which adjusted the colour intensity of image pixels. The ‘Sound Control Panel’ provided different selections of 128 MIDI sounds and 1 sine sound predefined for sound pixels above with the option to choose Loop or not. The system was implemented in Java, and so can be run as a standalone program on a local computer or an applet on the Web (the interface is shown in 2). The success of the developed system was tested in three methods: Code Reading testing, Black Box testing, and Integration testing. In addition, the system has been used to carry out several experiments to test the empirical hypothesis on nearly 20 subjects. Both the testing of the system and the results of the experiments indicated that the algorithms behind the Mapping Model successfully mapped simple visual images into sound ‘images’. It is suggested that it can be further developed and installed on mobile devices in the future. It is also believed that this sound simulation would work better on a touch screen and could receive better learning outcomes whilst on the move.
Fig. 3. System Interface (two screen captures) - showing the main windows of the user interface, which consists of four split panels with changeable size. They are: Visual Space Panel (on the top left), Sound Space Panel (the top right), the Visual Control Panel (on the bottom left), the Sound Control Panel (on the bottom right). The five menu bars are on the top of the interface: file, experiment, training, subject and help.
4 Laboratory Experiments Early work shows that vision dominates in multi-sensory processing. This means vision is always considered as active, but hearing is usually thought of as passive. However, research has shown that while vision may dominate spatial processing, hearing dominates temporal processing (Guttman, Gilroy, & Blake, 2005). Therefore, the success of testing the ’empirical’ basis of learning possibility by translating visual
Development of a Visualised Sound Simulation Environment
271
information into auditory information lies in the right design of appropriate experiments, which can take best advantage of the temporal processing of hearing. A series of experimental tasks were developed, and the data from such experiments can be analysed in a great many ways, resulting in the discovery of many unknown aspects of learning. In this paper attention focuses mainly on two sets of experimental tasks accomplished by eleven subjects. There were 6 males and 5 females of them, ranging in age from 20 to 30 years old. The hearing abilities of the 11 subjects were at average level. 4.1 Experiment 1 - Sound Localisation The sound localisation experiment was designed to measure the ability of the brain to process spatial information. This experiment differed from other sound localisation experiments in several ways. Firstly, earlier experiments always used special facilities to organise the experiment such as geodesic spheres housing an array of 277 loudspeakers (e.g. Hartmann & Acoust, 1983). This increased the cost of experiments and decreased the practicality. In contrast, this experiment used a computer simulated system, which can automatically accomplish the experiment without manual input and supervision. In addition, the system was developed in Java and can be easily extended to a mobile device in the future. Secondly, real sound sources (e.g. speakers) were usually used in other experiments earlier whereas our experiments used virtual 2D sound space, in which each ’sound pixel’ represented a unique virtual sound source thereby again saving on cost, practicality and time. Finally, unlike 3D cave-like virtual environments where sound localisation system involved with head-related transfer functions (HRTFs), each image pixel in this developed system was translated directly to a sound pixel. To add more detail, in the horizontal direction, the interaural time difference (i.e. Inter-aural level difference and Inter-aural time difference) was used to determine horizontal position. In the vertical direction, frequency was used to perceive vertical position in this experiment. These two parameters are used to distinguish each ‘sound pixel’ in experiment with respect to the Mapping Model (M). All the mouse-click responses were recorded and stored in a text file. In the series of this experiment, the resolutions of the sound panel increased from 3 × 3, 7 × 7, to 11 × 11. In each one, four different combinations of trials were used to
Fig. 4. Experiment 1 Sound Localisation - the Square with gray colour is the square where the subject clicked to predict the location of the sound
272
J. Zhang et al.
Fig. 5. Results of Experiment 1 - showing the total number of correct clicks from step 1 to step 5. The step 1 and step 5 were the test stages, while the step 2, 3, and 4 served as the training. The number of correct clicks of the step increased after 3 different kinds of training. The step 4 (adjacent sound with a reference sound) achieved the best results. The number of correct clicks decreased as the sound space dimension increased.
explore a better pattern for identifying sound. Subjects were first introduced to a random sound, and then were given a reference sound from the pixel in the centre of the sound panel followed by a random sound. After this, subjects were given a series of sounds which were adjacent to each other. Then, subjects were given a reference sound followed by an adjacent sound to it. Last, subjects were provided by a random sound again to find out whether learning occurred from the training. As we can see in Fig. 5, the sound localisation ability is clearly improved with training. Subjects were also better able to locate sound if given an initial reference sound. The chart on the right shows that the ability of the subjects to locate sound decreased with increased resolution. By comparing the ability of identifying sound horizontally and vertically, if the space dimension is smaller than 7 × 7, subjects can find the horizontal position of a sound better than its vertical position in the space; if more than 7, it seemed to be easier to identify a sound in the vertical axis. As expected, the subject’s ability to specifically locate sound correctly in the 11 × 11 space was very low. Despite this, as shown in Fig. 6, while the absolute position was incorrect, subjects were very good at finding the relative location of the sound source, given the previous test sound. Thus, in deciding on the location of the pixels in this sound space, subjects used their previous responses to previous stimuli in order to estimate the positions of future test sounds. This indicates that the human auditory system predicts new information based on past information. Thus the past information can be used to aid in the ambiguity that they experience when trying to predict the new test sound, and enhance the current attempt. The positive parts of the experiments have shown that humans are able to learn from the wholly new translated sensory information (as shown by the increasing learning curves from the sound localisation experiment data). We suspect a long-term training process would help to better develop subjects’ visual and audio sense with this new kind of audio-visual experience.
Development of a Visualised Sound Simulation Environment
273
Fig. 6. Total Number of Correct Clicks (General Pattern of Movement) – The red numbers show the sequential locations of the test sounds sound generated by the experiment. The black numbers represent an example subjects clicking responses.)
4.2 Experiment 2 - Object Recognition While Experiment 1 was based on one sound, the object recognition experiment was further designed to increase the sound sources. Nine 20 × 20 simple black and white images, consisted of the same number of pixels (and therefore of the same overall intensity), were used. The images were randomly loaded from a local folder on a computer. There was the same number of white pixels in each image. Pixels in white were the one able to produce sound simultaneously, while pixels in black were silent ones. Subjects were first introduced by a compound sound from a random image, and asked to pick up a right image to pair with the sound. They were then presented again with a compound sound with feedback. That is, if subjects chose a wrong image, the right image would be highlighted to correct them. Then, an identical image panel would be switch on automatically on the left to allow subjects to mark a small area (in which sound pixels were able to be generated) and navigate to different directions. Sound would be generated if subjects moved onto corresponding white pixels hidden underneath. Last, subjects were presented with a random compound sound from images again to test their learning experiences. It is clear that the ability of object recognition without training was poor (16%). However, their ability to recognise the object increased sharply when they were given the opportunity to explore it (72%), which increased further after four trial explorations (where the subjects were correct 100% of the time). The statistical data also showed that motion aided in recognition of the masked object as opposed to polyphonic compound sounds generated by the static object. That is, motion was easier to recognise than the static object. By moving around, the brain could construct an internal map of the image from experience. Although it is clear that in moving the rectangular window around in a sometimes arbitrary fashion, only a temporary memory was utilised to intuitively construct a mental model of the 2D environment. We believe that without using any exploration of the masked Sound Space to obtain temporary experience and perceptions, the brain instead needs the experience stored in the brain by long-term training.
274
J. Zhang et al.
Fig. 7. Experiment 2 – the interface on the left shows the object recognition experiment in Step 1, 2 and 4. The interface on the right shows the object recognition with exploration in step 3. There is the masked image panel on the left containing one of the nine images on the right hand side underneath. The subject can drag a rectangle by mouse and move it by pressing the up, down, left, right arrow keys to explore this image. After exploration, the subject clicked the one he/she thought was correct on the right hand side. The red image button then showed the correct answer.
5 From Laboratory to Real World As stated earlier, this study, was at first intended to be a three-year project in a neuroscience laboratory to prove the empirical vision theory, but later expanded beyond the scope that was originally envisaged, as the adoption of this developed sound simulation in real educational settings became increasingly feasible. Especially, when mobile technologies hold out the promise of ubiquitous leverage to this Java-based application, the study moved on to the discussion of the social values of such sound simulation on mobile devices. Statistical data showed that global mobile phone use was set to pass 3.25 billion by 2007 (Reuters, 2007) - around half the world's population. With regard to this, it is widely accepted that mobile and ubiquitous computing can genuinely support learning, but it is crucially important to ask how such simulated environment can be combined with mobile technologies to be immersed into various kinds of learning experiences in different contexts. It is a very complex undertaking to design interfaces for mobile devices. Visual interfaces have unpleasant limitations in small-screen devices, as they have a confined space on which to display information (Rinott, 2004; Smith & Walker, 2005; Walker & Lindsay, 2006). While mobile devices were getting smaller and lighter, this developed environment with its three-tier structures was able to switch on and off four different panels in the real time. That is, it potentially enlarged the small screen four times. Such an interface with layered view also promised seamless switching between different layers to a great extent of graphic rending. More importantly, the designed interface was sonically enhanced, and required less visual attention. The use of audio feedback thus to certain extent augmented the display of a mobile device, and helped users to be able to navigate in unfamiliar interface more easily. Not only can the interface of a mobile device be improved by this developed sound visualisation simulation environment, but also can the created applications widely be used for many educational purposes. Visual and auditory sensory are closely involved with each other. On the one hand, hand drawing exercises with sound in this
Development of a Visualised Sound Simulation Environment
275
developed environment provided a good opportunity to bring sound or music into the drawing practice. On the other hand, the database of 128 midi music sounds in two dimensions (frequency and pan) enhanced the traditional music teaching. The concise design integrated with drawing function was likely to be of more appeal to learners. More importantly, the environment held a record of every single experiment in the background, and was capable of providing a learning curve or analysis. Furthermore, the possibility to tailor learner-centred experiments by learner themselves gave this environment a great deal of flexibility and openness. It allowed a healthy cycle of further development in the future.
6 Conclusion The developed sound visualisation environment was able, to a certain degree, to translate the visual stimuli into sound stimuli although we did encounter some shortcomings in the experimental implementation. Furthermore, this system is planned to be implemented on mobile devices. It is believed that this sound simulation would work better on a touch screen and could result in better learning outcomes whilst on the move. This initiative not only brings scientists and educators together, but it is also hoped that it represents a possible e-approach to a constructivist way of learning.
References 1. Butler, R.A., Humanski, R.A.: Localisation of Sound in the Vertical Plane with and without High-frequency Spectral Cues. Percept. Psychophys. 51, 182–186 (1992) 2. Carlile, S., King, A.J.: Monaural and Binaural Spectrum Level Cues in the Ferret: Acoustics and the Neural Representation of Auditory Space. Journal of Neurophysiology 71 (1994) 3. Dewey, J.: How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process (New edn.). D.C. Heath, Lexington (1933) 4. Gibson, E.J.: An Odyssey in Learning and Perception. MIT Press, Cambridge (1991) 5. Guttman, S., Gilroy, L., Blake, R.: Hearing What the Eyes See: Auditory Encoding of Visual Temporal Sequences. Psychological Science 16(3), 228–235 (2005) 6. Hartmann, W.M., Acoust, J.: Localisation of Sound in Rooms I. Soc. Am. 74, 1380–1391 (1983) 7. Martino, G., Marks, L.E.: Cross-modal interaction between vision and touch: the role of synesthetic correspondence. In: Perception 2000, vol. 29, pp. 745–754 (2000) 8. Mayer, R.E.: Memory and Information Processes. In: Weiner, I.B. (ed.) Handbook of Psychology, vol. 07. Wiley, Hoboken (2003) 9. Purves, D., Lotto, R.B.: Why We Wee What We Do: An Empirical Theory of Vision (November 2002) 10. Reuters: Global Mobile Phone Use to Pass 3 Billion (2007) 11. Rinott, M.: Sonified Interactions with Mobile Devices. In: Proceedings of the International Workshop on Interactive Sonification, Bielefeld, Germany (2004) 12. Schlottmann, A.: Is perception of causality modular? Cognitive Sciences 4 (2000) 13. Smith, D.R., Walker, B.N.: Effects of Auditory Context Cues and Training on Performance of a Point Estimation Sonification Task. Applied Cognitive Psychology 19(8) (2005) 14. Walker, B.N., Lindsay, J.: Navigation Performance with a Virtual Auditory Display: Effects of Beacon Sound, Capture Radius, and Practice. Human Factors 48(2), 265–278 (2006)
Causal Links of Presence Donghun Chung1 and Chae-Hwan Kim2 1
School of Communication Kwangwoon University 447-1 Wolgye-Dong, Nowon-Gu, Seoul Korea 139-701 [email protected] 2 Department of Mass Communication and Journalism Tongmyoung University [email protected]
Abstract. The purpose of this paper1 is to examine antecedent variables and an outcome variable of presence. Presence has been used to explain the extent to which technology users are immersed and involved in a technology-created experience. In video gaming, gamers frequently don’t distinguish between reality and the game world, and they identify characters with themselves. This comes from a high level of presence. So what makes technology users have greater presence? The present study proposes a causal model which includes attitude and empathy as antecedent variables that lead to a degree of presence and then para-social interaction as an outcome of presence level. The results showed that path analysis of the model was successfully supported. Keywords: Presence, Attitude, Empathy, Para-Social Interaction, Wii.
1 Introduction Presence is a psychological state or subjective perception in which, even though part or all of an individual's current experience is generated by and/or filtered through human-made technology, part or all of the individual's perception fails to accurately acknowledge the role of the technology in the experience [1]. Actually, there are many different conceptual and operational definitions of presence, and scholars both within the same field and in different fields use their own terminologies. Yet these terminologies are for the most part interchangeable. Many scholars commonly agree that presence is, at least on a fundamental level, the perception of non-mediation [2]. Many scholars have examined causes or effects of presence [2][3][4][5][6]. However, only a few studies have revealed causal relationships of presence with some antecedent variables. This is important to know because when we find out reliable variables that increase or decrease the level of presence, we can manipulate them. Consequently, we can find out what outcomes presence influences and how it does so, for presence is not an ultimate outcome itself. Therefore, presence has an important role as a mediated or moderated variable. 1
The present Research has been conducted by the Research Grant of Kwangwoon University in 2008.
In the development of state-of-the-art video game technology, people are more likely to experience presence. There are a few reasons why gaming is important in presence research. First, gamers are active users. According to uses and gratification theory, most game players have motivations which lead to spending more time playing games and other activities [7][8]. Since they are active players, it is easy to be immersed in the gaming situation. Second, gaming has developed very quickly and dramatically. Many aspects of gaming have increased in depth and complexity, including the game story, graphics, characters, background, and even controllers, and these are all significant factors in influencing the degree of presence. Third, interactivity has been shown to be an important function in gaming. Interaction definitely plays an important role in presence [2]. Fourth, the systems on which games are played have also been developed. For instance, the modern TV set provides game players with clear images, high resolution (even HD quality), big screens, and vivid colors in console games. Computer monitors offer the same advantages in computer games. In addition, modern surround sound stereo systems enable a fully immersive auditory experience with crystal-clear quality. These are all significant factors affecting presence. Finally, based on the previous results, game research has been successfully performed. Many studies have raised presence issues using games; for instance, aggression [9], arousal [5], social relationship [10], violence [11], etc. In the present study, we will investigate causal relationships among attitude, empathy, presence, and para-social interaction in gaming. These variables directly or indirectly linked with one another in the previous literature. More rationales will be described in the next section.
2 Proposed Model Witmer and Singer [6] assert that involvement and immersion are necessary for experiencing presence. Involvement depends on focusing one’s attention and energy on a coherent set of stimuli while immersion depends on perceiving oneself as a part of the stimulus flow. More specifically, involvement is defined as a psychological state experienced as a consequence of focusing one’s energy and attention on a coherent set of stimuli or meaningfully related activities and events. Although immersion is a psychological state like involvement, it is characterized by perceiving oneself to be enveloped by, included in, and able to interact with an environment that provides a continuous stream of stimuli and experiences. They propose that a valid measure of presence should address factors that influence involvement as well as those that affect immersion. Many studies have investigated causes and effects of presence. One of the more interesting studies was Cummins’ dissertation [12] which examined the effects of direct address on empathy, interactivity, presence, and entertainment value in reality television shows. He found significant causal links among empathy, presence and para-social interaction as a partial path model. In his model, he disclosed that empathy as a predictor leads to presence and subsequently para-social interaction in watching television entertainment programs. This model can be applied in the gaming study because gaming also has entertainment value as an ultimate goal. Unlike Cummins’ model, this research has
Causal Links of Presence
281
attitude as an antecedent variable. A few reasons exist as to why attitude plays an important role. First, as a driving force, attitude works as a predisposition. Although it is hard to have one universal definition of attitude, the central ideas are readiness for response and motivation [13]. This is a preparation for behavior. Second, as an evaluation, attitude leads to involvement or attention. Attitude can be defined as favorableness or unfavorableness which ultimately decides a person’s further psychological disposition or behavior. Lastly, according to previous attitude and empathy literature [14][15], there was a significantly positive relationship between the two variables. From the extensive literature, the present research has the following proposed model . This model explains that 1. attitude towards a game character influences empathy with it, 2. empathy with a game character influences presence, and 3. presence influences para-social interaction with a game character. Ultimately, because these relationships are interrelated, the fit of the causal model will be tested. Attitude
Empathy
Presence
Para-Social Interaction
Fig. 1. Proposed path model
3 Method 3.1 Participants All of the participants were women and non-game players. Before they joined this project, they answered Witmer and Singer’s (1998) immersive tendency measure and were found to be homogeneous. Fourteen participants joined this project. Ages ranged from 19 to 25 and the average age was 21.36 (SD=1.55). 3.2 Procedure All participants filled out a consent form and then entered into a game lab. A researcher explained how to play the tennis game in the Nintendo Wii. The researcher showed them how to serve, do a forehand, and do a backhand. They were asked to practice the tennis game for 5 minutes as a training session. All of the lights were turned off to get a more immersive experience. The researcher helped guide them during that time. After confirming that no problems existed, the researcher restarted the game and let them play alone. After 15 minutes of playing, the game was stopped and the participants were given a main questionnaire that asked about their attitude toward character, presence, empathy, and para-social interaction. 3.3 Instruments The questionnaire was mainly composed of four parts: attitude toward character, empathy, presence, and para-social interaction. Attitude toward character had four items: “useless/useful,” “unimportant/important,” “foolish/wise,” and
282
D. Chung and C.-H. Kim
“unpleasant/ pleasant.” This measure simply asked participants’ general feeling of favorableness or unfavorableness for their characters and five-point semantic differential scale was employed. One item (unimportant/important) was deleted. (M=3.45, SD=0.67) and it was found reliable (α=.70). Second, eight questions were asked to measure empathy defined as feeling the same way that an observed character is feeling. Two items were deleted and it was reliable (M=2.75, SD=0.72, α=.85). Third, presence was composed of two parts: involvement and immersion. Fourteen questions were provided and all of the items were retained and reliable (M=3.03, SD=0.83, α=.93). Lastly, para-social interaction was measured. Para-social interaction, which is the imaginary one-way relationship that viewers develop with people on television, was coined by Horton and Wohl [16]. Para-social interaction between gamers and game characters was measured by eight items and all of the items were retained and reliable (M=2.54, SD=0.93, α=.92). 3.4 Gaming System The Nintendo Wii was chosen for this research. The Wii is what emerged from an attempt to change the way video games are perceived. The Wii has several significant differences from the other next-generation gaming platforms. For one thing, it is very small, at 8.5 cm x 6 cm x 2 cm and 3.84 lbs. It also has wireless connectivity.
Fig. 2. Overview of the experimental condition
Causal Links of Presence
283
The most innovative feature of the Nintendo Wii is the controller, called the Wii Remote. It contains sophisticated motion-sensing technology that enables a variety of gaming functions. Gamers can swing the controller like a tennis racket to play a tennis game, grab the controller with both hands and steer it like a steering wheel, or point and shoot in first-person shooting games. With an additional controller connected to the Remote, gamers can box an opponent by engaging in a punching and blocking motion using both hands. The Remote also contains other features that may contribute to a more immersive experience. It has a rumble feature to supply kinesthetic feedback. It also has a small speaker built into the controller, adding an auditory component to a player’s movements. A LG 42 inch LCD TV was used with the Wii. It is 46.3 x 30.2 x 11.8 (in) and 90.4 (lbs) with the stand. The resolution is 1366 x 768 (Dot) and the television system is NTSC-M, ATSC, 64 & 256 QAM. For better sound, a Panasonic SA-HT940 home theater system was used. It has 5 +1 channels.
4 Result Path analysis was used to test the fit of the causal model. The AMOS program was used to obtain estimates of global fit. Various fit indices such as chi-square, the Comparative Fit Index (CFI), the Goodness of Fit Index (GFI) and the Root Mean Squared Error of Approximation (RMSEA) were examined. Hu and Bentler [17] note that CFI values range from 0 to 1.0, with values greater than .90 indicating close fit. A GFI of greater than .9 is conventionally considered to indicate an acceptable fit. With RMSEA, values less than .05 indicate close fit. In addition, the obtained chi-square value is compared with the predicted value and the resulting p value is provided. Overall, the fit of this model to the data with the AMOS analysis was good (χ2(3) = 1.82, p > .05, CFI = 1.000, GFI = .942, RMSEA = .000) . Table 1 presents the zero-order correlations of the study variables.
Attitude
Empathy
.67**
Presence
.70**
Para-Social Interaction
.63*
Note. ** indicates p < .01, * indicates p < .05.
Fig. 3. Obtained path model Table 1. Zero-order Correlations of the Study Variables Attitude Attitude 1 Empathy 0.633* Presence 0.517 Para-social interaction 0.490 Note. * indicates p < .05, two-tailed.
Empathy
Presence
Para-social interaction
1 0.603* 0.474
1 0.559*
1
284
D. Chung and C.-H. Kim
5 Discussion This study aimed at disclosing causal links of presence composed of attitude, empathy, and para-social interaction. More specifically, the causal model proposed that attitude influences empathy, empathy influences presence, and presence influences para-social interaction. Since they were not independent of each other, path analysis was performed to test the causal model. To test the proposed model, participants played Nintendo Wii tennis games. Since the Wii system has unique interactive functions, we expected gamers to experience a greater sense of presence. Results indicated that all hypotheses were supported and the data were judged to be consistent with the model. Each causal link has significant implications. First, the relationship between attitude toward a game character and empathy with it is significantly positive. Attitude is simply a favorable or unfavorable manner toward the game character in this research. This link confirmed that the more positive the attitude toward a game character, the greater empathy with it. Containing affective, cognitive, and behavioral components, attitude influences a person’s psychological traits, attributions, characteristics, and behaviors. Does having a positive or negative attitude toward the game character influence intent to play the game? What if gamers have negative attitudes toward the game character? Are they going to play it? They may want to change their characters or even the game. For emotional reconciliation, the role of attitude toward the character may be as important as stories, images, and so on. Of course, attitude also plays an important role in presence research because the degree of involvement and immersion in the game environment partially comes from people’s characteristics. This will be further described in the empathy and presence relationship. A question about attitude toward the character will be developed as above. Therefore, we need to know gamers’ or technology users’ attitude toward an object when we have individual characteristics variables. Second, empathy and presence have a significant positive relationship. It means that the greater the empathy with a game character, the greater the sense of presence. Many presence studies have been interested in particular technological attributes as causes of presence; for instance, screen size, resolution, interactivity, and so on. However, as Cummins [12] pointed out, this relationship revealed that individual characteristics can influence presence as well. More specifically, an individual’s ability to engage in empathic processes was a significant predictor of presence. Empathy in this research was defined as feeling the same way that an observed other is feeling (Hoffman, 1975). Of course, an observed other means the game character here. Since gamers had a positive attitude toward the game character, they were drawn into the empathic development, and subsequently empathic engagement increased immersion and involvement in the game. In order to have homogeneity, the present research filtered out samples using gender and game experience as well as Witmer and Singer’s immersive tendency scale [6]. Even after this filtering process, individual characteristics predicted the level of presence experienced. Future studies will find out more promising attribute variables for presence. Lastly, there is a significant positive relationship between presence and para-social interaction, meaning that the greater the experience of presence, the greater the parasocial interaction with a game character. Simply speaking, para-social interaction
Causal Links of Presence
285
means gamers’ perceived or psychological interaction with game characters. Lombard and Ditton [2] explained that para-social interaction had mostly been discovered with TV characters and less empirical evidence of it existed with computer characters. As a matter of fact, little research has empirically examined the relationship between presence and para-social relationship. Therefore, this study demonstrates evidence that presence leads to para-social interaction with the game character. In addition to the outcome of presence, para-social interaction should be investigated concerning what this will bring about as a predictor and an influence of presence. This study has some limitations by nature. First, the causal links are too simple. There were neither moderated variables nor multiple exogenous and endogenous variables. Although it is an advantage that the simple relationships clearly explain the outcomes, it cannot explain various causes and effects of presence. Second, the sample size is too small to perform path analysis. However, generally speaking, the reason that we need relatively big samples (about 200) is to detect a difference between the model and the data because standard errors are large in the small sample size. In this research, although small samples were provided, the path model was successfully supported. Finally, it is necessary for future research to find out various cause-and-effect variables of presence that are not limited to the technology (or its functions) but psychological attributes, characteristics, and traits.
References 1. International Society for Presence Research.: The Concept of Presence: Explication Statement (2000), http://ispr.info/ (Retrieved February 10, 2007) 2. Lombard, M., Ditton, T.: At the Heart of It All: The Concept of Presence. Journal of Computer-Mediated Communication 2 (1997) 3. Chung, D.: Something for Nothing: Understanding Purchasing Behaviors in Social Virtual Environments. CyberPsychology & Behavior 6, 538–554 (2005) 4. Skalski, P., Tamborini, R., Glazer, E., Smith, S.: Effects of Humor on Presence and Recall of Persuasion Messages. In: 56th Annual International Communication Association Conference, Dresden, Germany (2006) 5. Tamborini, R., Westerman, D., Skalski, P., Weber, R., Kotowski, M., Chung, D.: Repeated Exposure to Virtual Video Game Violence: Presence and Hostile Thoughts. In: The 91st annual meeting of the National Communication Association, Boston, MA (2005) 6. Witmer, B.G., Singer, M.J.: Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments 3, 225–240 (1998) 7. Jansz, J.: Gaming at a LAN Event: The Social Context of Playing Video Games. New Media & Society 3, 333–355 (2005) 8. Sherry, J.L., Lucas, K., Greenberg, B.S., Lachlan, K.: Video Game Uses and Gratifications as Predictors of Use and Game Preference. In: Vorderer, P., Bryant, J. (eds.) Playing Video Games: Motives, Responses, and Consequences, pp. 213–224. Lawrence Erlbaum Associates, New Jersey (2006) 9. Nowak, K.L., Krcmar, M., Farrar, K.: Examining the Relationship Between Violent Video Games. Presence, and Aggression. In: The Presence Conference, Cleveland, Ohio (2006)
286
D. Chung and C.-H. Kim
10. Lee, K.M., Park, N., Song, H.: Can a Robot Be Perceived as a Developing Creature? Effects of a Robot’s Long-Term Cognitive Developments on Its Social Presence and People’s Social Responses Toward It. Human Communication Research 4, 538–563 (2005) 11. Tamborini, R., Eastin, M., Lachlan, K., Skalski, P., Fediuk, T., Brady, R.P.: Hostile Thoughts, Presence and Violent Virtual Video Games. In: The 51st annual convention of the International Communication Association, Washington, D.C. (2001) 12. Cummins, R.G.: The Entertainment Appeal of Reality Television: The Effect of Direct Address on Empathy, Interactivity, Presence, and Entertainment Value. Unpublished Dissertation, University of Alabama, Tuscaloosa, AL (2005) 13. Allport, G.W.: Attitudes. In: Murchison, C. (ed.) A Handbook of Social Psychology, pp. 798–844. Clark University, Worcester (1935) 14. Ascione, F.R.: Enhancing Children’s Attitudes about the Humane Treatment of Animals: Generalization to Human-Directed Empathy. Anthrozoos 3, 176–191 (1992) 15. Bagshaw, M., Adams, M.: Nursing Home Nurses’ Attitudes, Empathy, and Ideologic Orientation. International Journal of Aging & Human Development 3, 235–246 (1986) 16. Horton, D., Wohl, R.R.: Mass Communication and Para-Social Interaction: Observations on Intimacy at a Distance. Psychiatry 3, 215–229 (1956) 17. Hu, L.-t., Bentler, P.M.: Evaluating Model Fit. In: Hoyle, R.H. (ed.) Structural Equation Modeling: Concepts, Issues, and Applications. Sage, Thousand Oaks (1995)
Games Design Principles for Improving Social Web Applications Ines Di Loreto DICO – Dept. of Informatics and Communication Università degli Studi di Milano, Via Comelico, 39, Italy [email protected]
Abstract. Most young people (at least in countries were social communication technologies are established from a long run) carry mobile devices, surf the Internet and download music. They are always connected and live in a world where the distinction between virtual and real fade. The design of new interfaces becomes, in this context, a complex activity that involves a series of methodological problems. To one side designers have to create interfaces using basic HCI principles, on the other side they have to merge them with others ICT principles able to support social aspects, bearing in mind that they are addressing the above described generation of younger. Is our opinion that designers can find useful suggestion in game design strategies. Looking at games interface design choices, in fact, can help software engineers to improve the usability of other types of - more conventional - applications. In order to demonstrate this we will compare two social web applications: the new Facebook website and a French online game, Hordes. Keywords: Game Design, HCI, Social Interaction.
Y”1, that include people born between 1981 and 2000. However this way of partitioning generations doesn’t works anymore. Digital natives [2] cannot be defined only through vital statistics, but through a set of characteristics and common experiences. Most young people in many societies (at least in countries were social communication technologies are established from a long run) carry mobile devices, surf the Internet and download music. They are always connected, with a preference for multitasking, and using technology in new ways. For instance they use technologies for staying always in touch with friends trough Instant Messaging services like Messenger and Skype, for finding old friends lost over time (Facebook), for sharing any kind of material (e.g., using Flickr and YouTube), and so on. This “generation” explore the world in an entirely new way as they interact with these technologies: they live in a world where the distinction between virtual and real fade. As Palfrey and Gasser claim, digital natives “are connected to one another by a common culture. Major aspects of their lives- social interaction, friendships, civic activities – are mediated by digital technologies. And they’ve never known any other way of life.” [3]. We can say that digital natives are characterized by a set of common practices, including the amount of time they spend using digital technologies, rather than by their age. However, being familiar with these technologies doesn’t necessarily implies a deep understand of the used technologies nor implies the desire to understand them. In fact, for digital natives Internet and cell phones are “goods” as for other generations were Radio and TV. What this scenario imply for interface design? As for the previous generations, the User Interface digital natives use influences significantly their impression of any application. However, is our opinion that, because of the pervasive role digital technologies (consoles, internet games, mobile devices, social networks) play in the early life of digital natives, the user interface they are used to see and interact with in these early moments influence their expectations about the interface of any other application they will eventually interact with. Studies at the Pew Internet & American Life Project [4] show that virtually all college students play video, computer or Internet games and 73% of teens do so (as a result, for example, they become accustomed with a style of learning that takes place informally[13]). This game adoption rate (and its consequences) is, in our opinion, of crucial relevance in order to understand new users expectations. For example, observing that players become frustrated or confused if interfaces don’t give them the right feedback and the right amount of control, we claim that new generation of users (that are often, as we have already said, technologically literate, but that does not necessarily are media literate) feel frustrated if they don’t have the right amount of control over their computer-use experience. The design of users interfaces becomes then -more than before because of the above described scenario- a complex activity. In fact, on the one side designers have to create interfaces using basic HCI principles -mostly applied to single humanmachine interaction, on the other side, they have to merge them with others ICT 1
Generation Y is a cohort which consists of those people born after the Generation X cohort. Its name is controversial and is synonymous with several alternative names including The Net Generation, Millennials, Echo Boomers, and iGeneration.
Games Design Principles for Improving Social Web Applications
289
principles, able to support social aspects. It is our opinion that, designers of social interactive systems, as social web applications are, can find some useful suggestion in game design strategies. In fact, we believe that looking at games interface design choices can help software engineers to find useful guidelines for addressing interaction problems (as we will show in par.2) and for improving the usability of other types of (more conventional) applications. 1.1 Game Design and HCI Firstly it is worth recognizing that most of the applications used by digital natives in the above described scenario well fit in the framework of computer-mediated human interaction (i.e. the design of human-computer interfaces that will be used for social interaction). We can think, for example, to social networking software applications in web environments (Facebook, MySpace, and the like – Facebook is also accessible through cell phone), but also to consoles (Windows Xbox live community, Nintendo Wii and Mii channels, and PlayStation Home). Finally, online collaborative game play (e.g. Everquest and World of Warcraft) has increased dramatically over the last five to ten years with the growth of usage of the internet. The phenomenon has led to over 4 million users of Everquest worldwide and 6 million users of World of Warcraft.[5] Because of the focus on user performance and user satisfaction, a lot of interesting ideas about new ways of “communication” between users and interfaces arose in games in order to fulfill users expectations. Game designers face the challenge of creating games that can be easily learned, effectively played, and emotionally enjoyed by gamers (otherwise the game will not sell). As Ye and Ye claim in [6] no guidelines or well founded principles have been established to help game designers do their work. With limited theoretical foundations and little research or data on gamers, they have to mainly rely on their intuition and experience in order to find solution to their problems. Most of these solutions, are practical, non conceptualized, solutions, which, however, fit perfectly into HCI principles. Moreover, they are more appealing for new generations because of their look and feel. Without any ambition of completeness, we believe worth considering the following basic criteria used in game design: 1. Ease the learning curve. Players - as users often do in other settings - do not read the manual, so game designers have developed a quite sophisticated approach to ease the learning curve of a game (and appeal to a more casual gamer). On the one hand, they present the basic instructions as part of the game, on the other hand, they hide instructions in the interface itself, making them available on demand. 2. Support modifications of the UI. Game designers enable the gamers to modify and extend their UI and consider this possibility a fundamental feature of the system. For instance, most games allow the player to disable interface features in order to increase the size of the action window. 3. Avoiding information overload. Presenting too much information and features to the players all at once causes many players to give up on a game. So, quite often, designers hide non-essential information in places that the player can access when
290
I. Di Loreto
needed. Since the interface is part of the feedback mechanism, they make sure that the players cannot only access information they need –only when they need it - but also that the information they receive is useful.[7] In this paper we use the above mentioned criteria to analyze help facilities in web environments, comparing them with help facilities in games environments. In particular, we focus on metalinguistic signs[8]- signs that communicate a message about the use of the application itself - and we analyze in which way these signs in existing websites can be improved using games interfaces ideas. In particular, we consider the new Facebook (www.facebook.com) interface and the interface of a French online game, Hordes (www.hordes.fr). This choice is due to the fact that both of them - in different ways - can be seen as social web applications, i.e., applications characterized by dynamic public content that changes on the basis of the input of many people. Facebook private home pages are exactly this. In fact, each homepage is the result of friends posts, comments, and others kinds of inputs (video, links, and so on). In the same way, each game session in Hordes is the sum of the dynamic “real” game part, plus the overall discussions about strategies (we will explain this more in depth in par.4). 1.2 Contextual Help in an Online Social Networking Site: Facebook Facebook is a popular social networking website that is owned by Facebook, Inc. People can add friends, send them messages, and update their personal profile to notify friends about themselves. The website currently has more than 150 million active users worldwide [9]. Facebook has a number of features for people to interact with. They include the Wall, a space on everyone profile page that allows friends to post messages for the user to see; Photos, where participants can upload albums and photos; Status, which allows people to inform their friends of their whereabouts and actions, and so on. On September 2006, a new feature, News Feed, was introduced, which appears on everyone homepage and highlights information including profile changes, upcoming events, and birthdays related to the user's friends. Initially, the News Feed caused dissatisfaction among Facebook users: most of them were concerned it made it too easy for other people to track down individual activities (such as changes in relationship status, events, and conversations with other users)[10]. In response to this dissatisfaction, developers included appropriate customizable privacy features. Since then, users have been able to control, in all the Facebook applications, what types of information they want to share with friends.[11] Bearing in mind the above mentioned games design principles, we can claim that this experience demonstrate that Facebook designers understood that people need multiple levels of detail and personalization in their applications (the second principle) and let people customize their interface. On July 2008, Facebook introduced "Facebook Beta", a redesign of the user interface (one for all: profiles were separated into tabbed sections, and an effort was made to create a "cleaner" look). This upgrade can also be seen as another attempt to give users a (little) more customizable interface. However if we look at the way Facebook implemented in this occasion the help facilities (essential for the transition from the old to the new interface), we find that these feature are totally unaware of the
Games Design Principles for Improving Social Web Applications
291
concept of different level of details. In fact, the developers used a main (textual, static) help (Fig.1) where the users can find what they need by search (however, the link to the help is well hidden in the bottom of the page). Moreover, for a short amount of time, a set of contextual (textual and static) little windows of helps have been available for the new added features (Fig.2). Contextual help can only be closed, but not being opened again if closed by accident. In this way developers not only were not able to lower the learning curve – the first criteria we listed - but they also weren’t coherent in their design choices. While they succeeded in avoiding the information overload, they failed providing the right amount of information. New users’ first impression of Facebook was of a complicated service with a complex interface. 1.3 Contextual Help in an Online Game: Hordes The principles we listed in par.2 were draw basing on “classical” games, with a dedicated client. But what happens to these principles when the client is the web browser? We will try to understand it through the analysis of a website game, Hordes. Hordes is a game create by Motion Twin, a provider of online games for the French-speaking public. Created in 2001, Motion Twin has 7 million accounts, divided into more than 40 casual minigames, a word game, a RPG, a multiplayer online portal, an arcade platform game, and so on[14]. The game is a strategic game played entirely in a browser. However, unlike the majority of web sites games, Hordes is a social game, where everything depends on the relationship each gamer establishes with other players.
Fig. 1. Classical help in Facebook
292
I. Di Loreto
Fig. 2. Interactive help in Facebook
In a few words, Hordes is a game of survival in a world invaded by zombies. The main goal is to take the city you are living in as long as possible, constructing various buildings and defenses to deal with zombies attacks. By registering, the player joins a town of 40 players (the great majority of the players don’t know each other). To maintain the town as long as possible, players must learn to cohabit and to aim towards a common goal. In order to organize themselves, citizens can use a set of tools. The most important one is a forum where every player can freely post. It’s very interesting to note that the town’s forum reflect different social behavior: some of the players (without a top-down decision or an election) candidate himself as major, deciding the objectives of the town; others refuse the authority; others again report negative actions of the neighbors, and so on. In this case we are talking about a particular social application -an online gamewere a set of features are given to the users in order to accomplish their task (play and chat with others). The set of features developers give to the players are not so innovative (but for an interactive map of the city and of the environment, where all the players are showed), and the interface doesn’t offers personalization features. However they found an interesting solution to the problem of the help. They used a main (textual, static) help (Fig.3), with a list of the more frequently asked questions. Moreover, they created also a dynamic step-by-step contextual help (Fig. 4) that can be easily reached every time the user wants and needs (if closed it can be easily opened again).
Games Design Principles for Improving Social Web Applications
293
Fig. 3. Classical help in Hordes
Fig. 4. Interactive help in Hordes
In this case the designers well understood they have to avoid information overload, and they also eased the learning curve. However, they didn’t provide a sufficient level of personalization: e.g., not all the players use the supplied tools, while several of
294
I. Di Loreto
them prefer external applications created by other players. This result in various webbrowser tab opened at the same time, and the user have to switch between them while playing. The integration of external application into the game would be an interesting upgrade in the direction of personalization.
2 Conclusions While it is true that the main purpose of an interface - either for a social networking software, a game, or any other kind of application - is to allow users to communicate with the software[8], we cannot forget that for the kind of applications we are analyzing the main purpose is the communication between peoples, so the interface have to dissolve[15]. With a view on what we have just said, both of the sites developers have to learn something from the game design principles we described in par.2. It is also evident from the analysis, that the use of a web browser as client doesn’t allows a straightforward application of game design practical solutions, but require further elaboration. Finally, from this analysis and comparison, we want now draw some considerations on “good practice” in web applications design. First we suggest that designers have to take into account the de facto standards used in interfaces adopted by members of social groups that can become future lead users. Second, designers cannot forget the HCI assumption that the functioning of a technology have to be self-evident. In order to do this, designers have to provide multiple levels of detail, at increasing depth of complexity, allowing people to choose the features they need. More in general, we claim for the re-use of some solution and features developed in games design (as seen in par.2), in order to improve social web applications usability.
References 1. Lotito, G.: Emigranti digitali. Origini e futuro della società dell’informazione dal 3000 a.C. al 2025 d.C., Mondadori Bruno, Milano, IT (2008) 2. Prensky, M.: Digital Natives, Digital Immigrants (October 2001), http://www. twitchspeed.com/site/Prensky%20,%20Digital%20Natives, %20Digital%20Immigrants%20-%20Part1.htm (Retrieved January 13, 2009) 3. Palfrey, J., Gasser, U.: Born Digital: Understanding the First General of Digital Natives. Basic Books, New York (2008) 4. Lenhart, A., et al.: Teens, Video Games, and Civics, Pew Internet & American Life Project (2008), http://pewInternet.org/pdfs/PIP_Teens_Games_and_Civics_ Report_FINAL.pdf (Retrieved January 13, 2009) 5. Freitas, S.d.: Learning in immersive worlds: a review of game-based learning. London: Joint Information Systems Committee (2006) 6. Ye, Z., Ye, D.: HCI and Game Design: From a Practitioner’s Point of View (2004), http://www.yebrothers.com/documents/HCIGAMEDESIGN.pdf 7. Saunders, K., Novak, J.: Game development essentials. Thomson Delmar Learning, Clifton Park (2007)
Games Design Principles for Improving Social Web Applications
295
8. de Souza, C.S.: The Semiotic Engineering of Human–Computer Interaction. The MIT Press, Cambridge (2004) 9. Facebook Statistics, http://www.facebook.com/press/info.php?statistics (Retrieved January 13, 2009) 10. Lacy, S.: Facebook Learns from Its Fumble. Business Week, http://www.businessweek.com/technology/content/sep2006/tc200 60908_536553.htm?campaign_id=rss_tech (Retrieved January 13, 2009) 11. Gonsalves, A.: Facebook Founder Apologize. In: Privacy Flap; Users Given More Control. Information Week, http://www.informationweek.com/news/internet/ebusiness/show Article.jhtml?articleID=192700574 (Retrieved January 13, 2009) 12. Kundanis, R.M.: Children, teens, families, and mass media: The millennial generation. LEA, Mahwah (2003) 13. Kirriemuir, J., McFarlane, A.: Literature review in games and learning. Report 8. Bristol. Nesta Futurelab (2004) 14. Motion Twin description, http://www.motion-twin.com/english (Retrieved January 13, 2009) 15. Weiser, M.: Creating the invisible interface (invited talk). In: ACM Conf on User Interface Software and Technology (UIST), 1 (1994)
A Multiple-Level 3D-LEGO Game in Augmented Reality for Improving Spatial Ability Trien V. Do and Jong-Weon Lee Mixed Reality & Interaction Laboratory, Sejong University 98 Gunja-dong, Gwangjin-gu, Seoul, 143-747, Republic of Korea [email protected], [email protected]
Abstract. Inspired by the real LEGO game, an Augmented Reality 3D LEGO game is introduced. With multiple levels, the game provides a tool to improve spatial ability for a wide range of ages. Through the game, users can practice many spatial skills such as analyzing a 3D model’s structure, figuring out what to do to make a primitive geometry become a component of a 3D model, assembling components to create a complex model. The users mainly use their hands controlling physical makers to play the game. A user study was also carried out to evaluate the game and to compare it with the real LEGO game. The game is believed to be a useful and interesting tool to enhance not only human’s spatial ability but also human’s creation in 3D reconstruction. Keywords: LEGO, Augmented Reality, Human Computer Interaction, Spatial ability, Serious Game.
A Multiple-Level 3D-LEGO Game in Augmented Reality for Improving Spatial Ability
297
LEGO game, it does not require any special devices. Just with a personal computer, a cheap webcam, and some printed markers, any users can start playing the game at once. The 3DAR-LEGO introduces the users a flexible, convenient, and effective human-computer interaction to play the LEGO game in computer by combining both the traditional input method (mouse/keyboard) and the tangible input method (markers). The users use a mouse to select operations on a toolbar then control markers to interact with the game.
2 Related Work Lego is a line of construction toys manufactured by the Lego Group [7]. It consists of colorful interlocking plastic bricks and an accompanying array of gears, small figures and other parts. Lego bricks can be assembled and connected in various ways in order to construct such objects as vehicles, buildings and even robots. Normally, players will look at 2D images of 3D models and reconstruct physical models. Certainly, the 2D images cannot convey all the information of 3D models. Recently, the Lego Group has collaborated with Metaio to launch the “DIGITAL BOX” [8]. Metaio utilizes the innovative technology, Augmented Reality, to develop software programs which together with a camera and a display screen let LEGO packages reveal their contents fully-assembled within live 3D animated scenes. Therefore, the potential buyers can hold both the box and the finished product in their hands and are able to see exactly how much fun there is inside. Thanks to this idea, players now have a better view of what they have to assemble. However, the idea of the DIGITAL BOX just stops at intuitively displaying 3D models to assist in playing the game. Nothing is changed in the way of interacting with the Lego. The image of a DIGITAL BOX terminal from the LEGO Group in Fig. 1 shows exactly how a Lego model will look like when it has been fully assembled, even before the box is opened.
Fig. 1. A Lego digital box
298
T.V. Do and J.-W. Lee
Before the introduction of the Digital Box, the use of Lego had been already brought to the Augmented Reality in several systems. However, bricks of Lego are often used as the tangible parts for these systems. AR Lego [9] implements a playful machine assembly and maintenance scenario. The system provides step-by-step assembly instructions of which block should be mounted next and how to verify whether the user is at the correct stage in the construction. LEGO House Platform [10] is a real miniature house using LEGO bricks. Instead of building the house completely with furniture and decorations by LEGO bricks, authors augment the physical LEGO house with virtual furniture using AR technology.
3 Game Overview Inspired by the famous Lego game, the idea of the 3DAR-LEGO is that: first the 3DAR-LEGO allows users to load a sample 3D model from a file, then it decomposes this model into “pieces”; the users have to use these pieces to reconstruct the 3D model while observing the sample model. Depending on the selected level of the game, a piece can be a precise component of the sample model or just a primitive geometry: − Level 1: each piece is a precise component of the sample model. All the users have to do is to pick it and put it at an appropriate position to recreate the model. No transformation operation is required. − Level 2: each piece is a primitive geometry such as a sphere, a cube, a cone etc., which can be customized to become a precise component of the sample model. The users have to customize pieces then place them together to reconstruct the model. The customizing operations include scaling, rotating, resizing, coloring or texture mapping. − Level 3: no piece is provided at this level. By observing the sample model, the users have to analyze to figure out its structure. Then they select, customize and place primitive geometries together to reconstruct the model.
4 Hardware Setup Keeping the wish of bringing the 3DAR-LEGO to a large volume of audiences in mind, a desktop marker-based AR game with minimum requirement of computer skills and hardware devices is developed. After printing all the markers, turning on the camera, the users can at once enjoy the game. Fig. 2 shows a typical hardware setup for the game. In order to play the 3DAR-LEGO, a user needs: − A personal computer with a normal monitor: no kind of 3D glasses is required. − A webcam with a tripod (preferable) − A Cardboard containing multiple markers: this is the working space of the game, so the users can freely interact with the game without too much worry about the marker occlusion problem. − An Object Marker: a component of a model is displayed on this marker and added to a model on the cardboard later. This marker is attached to a block to provide a convenient tangible interaction device.
A Multiple-Level 3D-LEGO Game in Augmented Reality for Improving Spatial Ability
299
− A Sample Marker: this is a block with five markers so that the users can observe a sample 3D model from any direction. − A Selecting Marker: this marker is used to select any components of a model. All transformation operations are applied to selected components. − Two Editing Markers: change attributes of a piece.
Fig. 2. A typical hardware setup for the 3DAR-LEGO
5 How to Play the Game The 3DAR-LEGO has three levels as mentioned above. Depend on each level, skills are required differently. In this section, how to play the game at the Level 1 will be described in details first. The level 2 and 3 will be explained after that. At the Level 1, playing the 3DAR-LEGO is similar to playing the real Lego game. It follows the steps below: 1) Loading a sample 3D model from a file: the game provides some sample models (from simple to complex ones). Later, users themselves can create their own sample models. After a model is selected, it will be displayed on the Sample Marker (See Fig. 3). By observing this visual 3D model, the users will know what they have to assemble. On the other hand, it will be decomposed into pieces. These pieces are stored in a palette on the right side of the windows. Each piece is located in a box of the palette. This palette is dynamically resized depending on the number of the pieces (See Fig. 3). 2) Observing pieces: when the users move the mouse over a box, the piece in that box will be displayed on the Object Marker. Thanks to this, the users can have a comprehensive observation of that piece.
300
T.V. Do and J.-W. Lee
3) Picking a piece: if the current piece on the Object Marker is suitable, the users can click the mouse to firmly bind it to Object Marker (See Fig. 3). 4) Assembling a piece to reconstruct the 3D model: holding the Object Marker, the users freely move their hands to an appropriate position to place it down by clicking on the Add button on the toolbar. The aligning feature makes this operation simple and accurate (See Fig. 3). If by mistake, a piece is placed at a wrong position, it can be attached back to the Object Marker by clicking on the Cut button on the toolbar. 5) Step by step, the users will complete their model. Transformation operations may be required depending on the complexity of the sample model.
Fig. 3. The 3DAR-LEGO interface
Level 2 and Level 3 require the users to have good ability to analyze a 3D model’s structure. The users have to figure out what to do to customize a primitive geometry to a model’s component, and then assemble components to recreate a complex model. To create and customize primitive geometry, the users need to use functions provided by the 3DARModeler [11] which is already integrated into the game. The 3DARModeler can be considered a simple version of 3D Studio Max but in Augmented Reality environment. It provides necessary functions for a modeling system such as creating 3D objects, customizing object’s attributes, aligning objects, grouping, applying texture, adding animation etc. The models created by the 3DARModeler are saved to files to serve as sample 3D models. Therefore, sample models can be designed by users themselves to meet their own need in a specific area which they want to improve spatial ability. For example, Fig. 4 shows the 3DAR-LEGO at Level 3 where no piece is provided. In this game, the sample model is a tower. While observing it, the users have to realize that the bottom component is essentially a cube. Then a cube should be created and textured. After that it needs to be scaled along the three axes by Editing
A Multiple-Level 3D-LEGO Game in Augmented Reality for Improving Spatial Ability
301
Fig. 4. The 3DAR-LEGO interface
Markers (Details of how these Editing Markers perform other operations are explained in [11]). Step by step, other components are added to complete the 3D model.
6 Evaluation and Conclusion Through the game, players can practice many spatial skills such as analyzing a model’s structure, figuring out what to do to customize a primitive geometry to a model’s component, arranging components to create a complex model. To get feedbacks from users, a user study has been carried out through an HCI class, and the Kid & Edu exhibition in Korea. The main goal is to evaluate the system and to compare it with the real Lego game. 63 participants were divided into three groups − Group 1: from 7 to 11 years old (29 participants) − Group 2: from 12 to 15 years old (13 participants) − Group 3: 16 years old above (21 participants)
Fig. 5. A simple model of a table
302
T.V. Do and J.-W. Lee Table 1. Evaluation results of the 3DAR-LEGO
Aspects Interesting Practical Similar to the real Lego Intuitive Easy to play at Level 1 Easy to play at Level 2 Easy to play at Level 3 Able to improve spatial abilities Able to enhance the ability of analyzing a model’s structure
All the participants were asked to assemble a simple model of a table (see Fig. 5) at all the three levels then fill in a questionnaire. We used the 7 scale to evaluate each aspect of the 3DAR-LEGO. The Table 1 above shows the results of the user study. All the participants were able to play the game at Level 1 easily. However only 2 participants of group 1, 4 participants of group 2, and 7 participants of group 3 rated above 6 for the “Easy to play” at Level 2 and 3. In compare with the real Lego, all the participants agreed with the following advantages of the 3DAR-LEGO: − − − − −
Easy to play Do not worry about the loss of pieces Do not need space to store pieces Easy to carry to any place Can see the complete 3D models before playing the game
With the supportive results of the user study, we believe that our system will be a useful and interesting tool to improve not only human’s spatial ability but also human’s creation in 3D reconstruction.
Acknowledgments This work was sponsored and funded by Seoul R&BD Program (20070387).
References 1. Spatial Intelligent Home Page of University of Limerick, Limerick, Ireland, http://www.ul.ie/~mearsa/9519211/ 2. Eckstrom, R.B., French, J.W., Harman, H.H., Derman, D.: Kit of Factor-Referenced Cognitive Tests. Educational Testing Service, Princeton, NJ (1976) 3. Souvignier, E.: Training räumlicher Fähigkeiten (Training spatial abilities). In: Klauer, K.J. (Hrsg.) Handbuch Kognitives Training, pp. 293–319. Hogrefe, Göttingen (2001)
A Multiple-Level 3D-LEGO Game in Augmented Reality for Improving Spatial Ability
303
4. Durlach, N., Allen, G., Darken, R., Garnett, R.L., Loomis, J., Templeman, J., Von Wiegand, T.E.: Virtual environments and the enhancement of spatial behavior: Towards a comprehensive research agenda. In: PRESENCE - Teleoperators and Virtual Environments, vol. 9, pp. 593–615 (2000) 5. Rizzo, A.A., Buckwalter, J.G., Neumann, U., Kesselman, C., Thiebaux, M., Larson, P., Van Rooyen, A.: The Virtual Reality Mental Rotation Spatial Skills Project. Cyber Psychology and Behavior 1, 113–120 (1998) 6. ARToolkit Home Page, http://www.hitl.washington.edu/artoolkit/ 7. Lego Home Page, http://www.lego.com 8. Metaio Home Page, http://www.metaio.com/media-press/press-release/ lego-digital-box/ 9. István, B., Dieter, S.: Augmented Reality Agents in the Development Pipeline of Computer Entertainment. In: Proceedings of the 4th International Conference on Entertainment Computing, pp. 345–356 (2005) 10. Irawati, S., Ahn, S.C., Kim, J.W., Ko, H.D.: VARU Framework: Enabling Rapid Prototyping of VR, AR and Ubiquitous Applications. In: Proceedings of IEEE Virtual Reality Conference, March 2008, pp. 201–208 (2008) 11. Do, T.V., Lee, J.W.: 3DARModeler: a 3D Modeling System in Augmented Reality Environment. International Journal of Computer Systems Science and Engineering 4(2), 145–154 (2008)
An Online Survey System on Computer Game Enjoyment and Personality Xiaowen Fang, Susy Chan, and Chitra Nair School of Computing, College of Computing and Digital Media DePaul University, USA [email protected], [email protected], [email protected]
Abstract. This paper discusses the development of an online survey instrument to measure the game enjoyment and player characteristics like age, gender and personality traits. A research framework of game play is proposed based on a review of prior research on computer game enjoyment, game characteristics, personality theories, effects of computer game play, and technology acceptance model. The proposed framework suggests that an appropriate fit between characteristics of the player and gaming technology will result in greater enjoyment while social influence may moderate effects of the fit. The survey will allow the researcher to establish the fit profiles between player characteristics and game play. Keywords: computer games, game play, personality, enjoyment.
An Online Survey System on Computer Game Enjoyment and Personality
305
It will be important from the perspective of practitioners in computer game industry to understand the player characteristics and personality differences in their target audiences. A comprehensive framework for examining the interaction between player characteristics and game features is needed for a better understanding of the process of game play and its impacts on players. Based on literature review, we propose a framework of computer game play. As discussed in section 4, this framework emphasizes a fit between the player’s characteristics, such as personality, and game technology could lead to enjoyment. This paper reviews relevant research for the framework and discusses the development of an online survey system to investigate computer game enjoyment and personality traits. In the following sections, we will discuss: (1) theories and research findings of computer game enjoyment and media enjoyment, (2) game players’ characteristics; (3) computer game play; (4) the proposed framework, (5) research method, and (6) next steps.
2 Background Literature on Game Play and Enjoyment This section provides a review of studies in game play and enjoyment, video game preference, and several theories about media enjoyment. 2.1 Game Play and Enjoyment Player enjoyment is central to playing computer games ([10] and [11]). Few studies to date measure the player’s emotional responses, or game enjoyment, during game play ([12], [13], and [14]). In spite of a rising interest in academic game research, the actual experience of playing digital games is underrepresented in the gaming literature ([12]). Rajava and colleagues ([14]) have measured psychophysiological responses (facial EMG, skin conductance and cardiac RBI to game events) in the video game Monkey Bowling 2. Their study tests the premise that emotions would be expected to play an important role in gaming behavior and enjoyment. They note that different video game events elicit a reliable psychosociological response and arousal, and there may be large individual differences in the player’s emotional response to games and game events. Emri and Mayra ([15]) have studied immersion in the game world and proposed a model consisting of three different components of immersion: sensory, challengebased and imaginative immersion (SCI-Model). Sensory immersion refers to multisensory properties of a game; challenge-based immersion involves cognitive and motor skils of the games; imaginative immersions refers to immersions within the fantasy world created through the game and depends on the richness of the narrative structure of the game. Klimm ([16]) proposes that game enjoyment is based on three experiential factors: experience of affectence or immediate feedback to the player as a causal agent, feelings of suspense, relief and self esteem, and the feeling of being drawn to a fictional world.
306
X. Fang, S. Chan, and C. Nair
Karolein and colleagues ([12]) suggest that the social context (game competition between friends) and negative experiences such as in-game frustration or tension should be examined to understand the overall game experience. They used the focus group metohodolgy and summarized their finding into a comprehensive categorization of digital game play experiences. Based on their categorization, fun, amusement, pleasure and relaxation are the in-game experiences of enjoyment and the feelings of being energized, satisfaction, and relaxation are the post-game experiences. Other researchers have also investigated the emotional responses during game play. Chumbley and Griffiths ([17]) have found that skill and in-game reinforcement characteristics significantly influence a number of affective measures, most notably excitement and frustration. Some researchers have used physiological responses to objectively evaluate a player’s experience with entertainment technology ([18] and [19]). 2.2 Video Game Preference Research on video game uses and gratifications has focused on the main appeal of video games. Based on a survey of 244 10- to 24-year olds, Selnow ([20]) isolates five gratification factors that attract players to arcade video game play. These factors show that a video game: (1) is preferable to human companions, (2) teaches about people, (3) provides companionship, (4) provides activity/action, and (5) provides solitude/escape. Another study of video games ([21]) reveals a similar set of gratifications for arcade game use: excitement, satisfaction, and tension reduction. A survey conducted by Phillips, Rolls, Rouse, and Griffiths ([22]) suggests several uses of video game play, including “to pass time,” “to avoid doing other things,” “to cheer oneself up,” and “just for enjoyment.” Furthermore, Griffith’s ([6] and [23]) research on video game addiction includes additional uses and gratifications: arousal, social rewards, skill testing, displacement, and stress reduction. In several comprehensive studies, Sherry and his colleagues ([24], [25], and [26]) have enumerated a set of video game uses and gratifications based on focus group research and surveys of over 1,000 participants ranging in age from 10 to 24 years old. These factors include competition—to prove to other people who have the best skills and who can react or think the fastest; challenge—to solve the puzzles to achieve goals such as reaching the next level or beating the game; social interaction—to use video games to interact with friends and learn about the personalities of others; diversion—the use of games to avoid stress or responsibilities and to fill time, relax, escape from stress, and/or because there is nothing else to do; fantasy—to do things they normally would not be able to do, such as driving race cars, playing professional football, or flying; and arousal—the stimulation of emotions resulting from fast actions and high quality graphics. Grodal ([27]) explains that much of the fascination with video games can be attributed to the ability of players to control the game in terms of outcomes (i.e., deciding how the “plot” will unfold), the speed at which the game progresses, and mastery of the game or mastery over other players. Grodal further argues that video games are a tool for emotional control, whereby desired arousal levels can be maintained through playing. As such, video games are enjoyed most when the level
An Online Survey System on Computer Game Enjoyment and Personality
307
and speed of the game match the player’s optimal mental and motor capacity. Vorderer, Hartmann, and Klimmt ([28]) have provided support for the idea that game play is more enjoyable when there are a variety of ways to solve a challenge offered in a video game. 2.3 A Tripartite Model of Media Enjoyment Based on a review of various terms for capturing the concept of media enjoyment, Nabi and Krcmar ([30]) introduce a tripartite model of media enjoyment—affective, cognitive, and behavioral dimensions of enjoyment (Figure 1). In this conceptual model, the underlying affective dimension focuses largely on empathy; positive and negative moods and specific affective states (e.g., horror, sadness, and suspense) could also be considered to feed this component. The cognitive component focuses primarily on judgments of game characters’ actions, though other judgments, like general enjoyment as attitudes toward story assessments (e.g., perceived realism, story coherence, message quality) or more personal evaluations (e.g., relevance, similarity) could also be included in this category. Finally, the behavioral component is logically connected to selective exposure in terms of the player’s viewing intent as well as behaviors during viewing, including the act of viewing itself. A player’s affective, cognitive, and behavioral reactions are influenced by a number of other factors including prior knowledge, direct experience, personality traits, and current mood. However, these factors are expected to operate by influencing the three components of enjoyment, which, in turn, serve to inform perceptions of enjoyment.
Fig. 1. A tripartite model of media enjoyment ([30])
3 Research on Game Player’s Characteristics Several individual characteristics may influence enjoyment of computer game play. A literature review reveals gender, age, and personality as key characteristics.
308
X. Fang, S. Chan, and C. Nair
3.1 Gender One of the most consistent results in studies of video game usage is the striking difference between boys and girls in the amount of play (e.g., [22] and [23]). Some scholars argue that this difference may be due to access. The annual Annenberg Public Policy Center survey on family media use points out that video games are played in 76% of homes with at least one boy, as compared to 58% of homes with at least one girl ([35]). Other studies emphasize that the gender gap in game use may have less to do with access than the content of the games ([36] and [37]). Bonanno and Kommers ([38]) discover that a high percentage of females prefer puzzle, adventure, fighting, and managerial games, and males prefer first-person shooters, role playing, sport, and strategy games. They suggest that these tendencies can be viewed as a process of accommodation to different underlying gender-related neurocognitive processes. 3.2 Age An online survey conducted by Griffiths, Davies, and Chappell ([39]) compares adolescent and adult online game players. Significant patterns emerge among adolescent gamers. They were more likely to be male, less likely to gender swap their characters, and more likely to sacrifice their education or work to play video games. In relation to favorite aspects of game play, significantly more adolescents than adults claimed violence as their most favorite aspect of playing. Results also show that, in general, the younger the player, the more time they spent each week playing online games. 3.3 Personality Personality can be defined as a stable set of tendencies and characteristics that determine the commonalities and differences in people’s psychological behavior (thoughts, feelings and actions) that have continuity in time ([40]). Personality is one of the most elusive areas of psychology, difficult to understand, and difficult to test. Nevertheless, psychologists have developed several theories to explain personality ([41]) based on two principles: core of personality and periphery of personality ([40]). Core of personality addresses the inherent attributes of human beings which do not change over the course of living. They are used to explain the similarities among people. Periphery of personality, on the other hand, focuses on learned attributes. It helps to identify the differences among people. There are many personality theories: psychoanalytic theories, biopsychological theories, behaviorist theories, cognitive theory, humanistic theory and many more ([42]). Personality type and trait theories have served as the basis of several personality tests widely used commercially and in academic research. Type theory focuses on personality type or the psychological classification of different types of people. It originated from the ideas of Swiss psychiatrist Carl Jung. The basic assumption is that the personal unconscious is a potent part of human
An Online Survey System on Computer Game Enjoyment and Personality
309
psyche. Reliable communication between the conscious and unconscious parts of the psyche is necessary for wholeness ([43]). Myers Brigg Type Indicator or MBTI ([44]), one of the most widely used instrument for non-psychiatric populations in the area of counseling and personality testing ([45] and [46]) is based on Jungian based type theory. The inventory has four bipolar discontinuous scales: Introversion-Extraversion, Sensation-Intuition, Thinking-Feeling and Judging Perceiving. Respondents are classified into one of 16 point score obtained for each scale. Carlson ([47]) points out that although MBTI has been used unsystematically in a wide range of areas, it has a favorable validity assessment. However, critics of this tool ([48], [49], [50], and [51]) have reported its psychometric shortcomings and observed that the types are quite strongly stereotyped by professionals. Another personality type theory is Type A and Type B personality theory ([52]). Jenkins Activity Survey ([53]) is a self administered questionnaire to assess an individual's A/B type. Type A tends to be intense and hard-driving while type B people tend to be relaxed and less competitive, and lower in risk. Though widely used in clinical setting, Type A/B theory has been reported as having questionable perspective validity ([54], [55], and [56]). Trait theory addresses personality traits, or a person’s habitual patterns of behavior, thought, and emotion ([57]). Trait theorists generally assume that traits are relatively stable over time; they differ among individuals and influence behavior. Many studies and questionnaires are based on the trait theory, for example, a 300 item Adjective Checklist -ACL ([58]), Catell's 16 PF ([59]), 44 Item Trait Descriptor Adjectives (TDA) ([60]) . Over the years, the five-factor model (e.g. [61], [62], [63], [64], and [65]) has gained acceptance among researchers because it establishes a common taxonomy ([66]). It contains the following five dimensions of personality: Extraversion outgoing and stimulation-oriented vs. quiet and stimulation-avoiding ; Neuroticism emotionally reactive, prone to negative emotions vs. calm, imperturbable, optimistic; Agreeableness - affable, friendly, conciliatory vs. aggressive, dominant, disagreeable ; Conscientiousness - dutiful, planful, and orderly vs. laidback, spontaneous, and unreliable ; Openness to experience - open to new ideas and change vs. traditional and oriented toward routine. The Big Five traits have been researched and validated by many different psychologists (e.g. [67], [65], [68], and [69]) and are at the core of many personality questionnaires. Many studies have also established that four constructs of MBTI converge with the Big Five traits (e.g., [70]). Among the many measurement questionnaires, NEO-PI-R developed by Costa and McCrae ([71]) is well validated and widely used. International Personality Item Pool (IPIP) ([72]) also gained significant attention over the past decade. It is available over the internet and includes over 2000 items. The rationale for the collaboratory effort was to allow faster and continuous refinement of the set of personality scales by other scientists over a public domain. It is non proprietary and allows free use of items and constructs by researchers.
310
X. Fang, S. Chan, and C. Nair
Perceived Ease of Use
Player Characteristics -Demographics - Personality Traits
Fit Profiles
Enjoymen t
Intention to Play
Gaming Technology
Social Influence
Fig. 2. A framework of computer game play
4 A Framework of Computer Game Play Based on media enjoyment theories, personality theories, and the technology acceptance model ([8]), we propose a conceptual model of computer game play as depicted in Figure 2. This framework suggests that enjoyment derived from game play is the result of a fit between characteristics of the player and elements of gaming technology. The better the player-technology fit, the more enjoyment. Social influence will moderate the effects of player-technology fit. Based on this framework, enjoyment and perceived ease of use are two determinants of user intention to play computer games. This framework conceptualizes the impact of player characteristics and social influence on game play enjoyment that have been observed in many earlier studies ([3], [4], [5], [7], [9], [22], and [23]). Furthermore, the framework incorporates the gaming technology and its interactions with the player in affecting enjoyment of game play and intention to play. The following sections (4.1 to 4.6) discuss the main constructs of this framework. 4.1 Fit Fit has been examined in some detail in the strategic management literature. Three definitions of fit in structural contingency theory have been identified: fit as congruence, fit as interaction, and fit as internal consistency ([73]). Venkatraman ([74]) has extended these ideas to six unique perspectives on fit in the strategy literature: fit as moderation, as mediation, as matching, as gestalts, as profile deviation, and as covariation. The most promising perspective for player-technology fit in a gaming context is the idea of fit as an ideal profile. From this perspective, fit is viewed as feasible sets of equally effective alternative designs of games. Each design should be internally consistent and matched to a player’s characteristics. This conceptualization of fit translates well into a gaming environment. The player-technology fit can be defined as ideal profiles composed of an internally consistent set of player characteristics and gaming elements that affect game play enjoyment. A higher degree of adherence to an ideal profile increases the game
An Online Survey System on Computer Game Enjoyment and Personality
311
play enjoyment. A test of player-technology fit would require three steps: (1) identifying distinct player characteristics, (2) specifying ideal gaming technology for each set of player characteristics, and (3) testing player enjoyment resulting from player-technology alignment. 4.2 Enjoyment Strong empirical evidence indicates that the motivational basis of human activity relies on two rather independent systems: a so-called approach system and an avoidance system ([75]). Activation of the approach system results in pleasure, whereas activation of the avoidance system leads to pain ([76]). Research in psychology and neuroscience most often uses the term pleasure to describe agreeable reactions to experiences in general. Most communication researchers have used the term enjoyment to describe and explain such positive reactions toward the media and its contents. Our framework uses enjoyment to describe and explain positive reactions derived from game play. The tripartite model of media enjoyment proposed by Nabi and Krcmar ([30]) suggests three types of reactions to enjoyment: affective, cognitive, and behavioral reactions. Their model effectively conceptualizes player reactions to enjoyment as discussed in other media enjoyment theories such as model of complex entertainment experience ([29]), transportation theory ([31]), disposition theory of drama ([32]), and flow theory ([33]). These three types of reactions to enjoyment can be applied to develop an instrument to measure the degree of enjoyment derived from game play. 4.3 Player Characteristics Previous research has shown that game play is linked to gender (e.g., [22] and [23]), age ([39]), and personality ([3], [4], [5], and [7]). Media enjoyment theories also imply close relationships between a player’s personality and enjoyment ([29], [30], [31], [32], and [33]). In our proposed framework, a player can be described in terms of gender, age, and personality traits. 4.4 Social Influence Various theories suggest that social influence is crucial in shaping user behavior. For example, in the theory of reasoned action (TRA) ([77]), a person’s behavioral intentions are influenced by subjective norms as well as attitude. Hsu and Lu ([78]) indicate that social influences, including perceived critical mass and social norms, significantly and directly, but separately, affect player attitude and intention of playing online games. Choi and Kim ([9]) also note the importance of social interactions on continuing to play online games. Our proposed framework of game play posits that social influence impacts enjoyment as a moderator. It moderates the effects of player-technology fit. 4.5 Extending TAM to Game Play The technology acceptance model (TAM) ([8]) is one of the most widely used models for IT adoption. According to TAM, an individual’s IT adoption is influenced by
312
X. Fang, S. Chan, and C. Nair
perceived usefulness and perceived ease of use. Perceived usefulness (PU) is defined as the degree to which a person believes that using a particular system would enhance his or her job performance. Perceived ease of use (PEOU) refers to the degree to which a person believes that using a particular system would be free of effort. In a gaming context, perceived usefulness is no longer applicable and therefore not an appropriate measure of extrinsic motivation ([79]). Enjoyment is deemed as a more appropriate measure of extrinsic motivation to play games because enjoyment measures how the gaming technology helps achieve the task-related objective -enjoyment -- and it becomes an outcome expectancy.
5 Method 5.1 An Online Survey System In order to test the framework depicted in Figure 3, an online survey system will be developed to continuously collect data on player characteristics and the enjoyment of playing different computer games. This survey system is hosted on a permanent web server. It will allow researchers to establish the fit profiles between game technology and player characteristics. We have set up a web server and are in the process of completing a website for online surveys. The online survey system will also allow researchers to conduct longitudinal studies.. The initial survey we plan to conduct contains three types of questions about each player: demographics and gaming experience, personality traits, and enjoyment of playing a particular computer game. The instruments measuring computer game enjoyment and personality traits are discussed next. 5.2 Game Enjoyment Instrument Based on the tripartite media enjoyment model, we have developed an 11-item instrument ([80]) to measure computer game enjoyment following the method recommended by Moore and Benbasat ([81]) for measurement development. We developed an initial set of 60 items to measure affective, cognitive, and behavioral dimensions of game enjoyment. Afterwards, we conducted expert consultation, exploratory and confirmatory card sorting, and an extensive survey. These steps resulted in an 11-item instrument with strong validity and reliability measures. 5.3 Personality Instrument Personality traits will be measured using the 50-item IPIP inventory in the five domain constructs -- E: Extraversion vs. Introversion; A: Agreeableness vs. Antagonism; C: Conscientiousness vs. Lack of direction; N: Neuroticism vs. emotional stability O: Openness vs. closedness. IPIP items have been developed to measure the constructs in several inventories, for example, NEO-PI-R ([71]), 16 Personality Factor Questionnaire -16PF ([82]), Hogan Personality inventory-HPI ([83]) etc. Tables comparing the psychometric characteristics of the original scales with IPIP item are available for all the major inventories on the website: http://ipip.ori.org/newValidity.htm
An Online Survey System on Computer Game Enjoyment and Personality
313
6 Next Steps This paper presents a conceptual framework of computer game enjoyment. Based on the framework, we propose to design and conduct online surveys to understand the role of player characteristics and personality differences in game enjoyment. An appropriate fit between characteristics of the player and gaming technology could result in greater enjoyment. A study of such nature has not been done before. We have developed an instrument to measure game enjoyment. The next step is to establish the relationships between player characteristics, personality traits in particular, and game enjoyment. The findings are expected to have profound implications to future development of computer games. Based on our research finding, we plan to conduct additional studies to validate our proposed framework of game play.
References 1. Griffiths, M.D.: The therapeutic use of videogames in childhood and adolescence. Clinical Child Psychology and Psychiatry 8, 547–554 (2003) 2. Robillard, G., Bouchard, S., Fournier, T., Renaud, P.: Anxiety and presence during VR immersion: A comparative study of the reactions of phobic and non-phobic participants in therapeutic virtual environments derived from computer games. CyberPsychology & Behavior 6, 467–476 (2003) 3. Anderson, C.A.: An update on the effects of playing violent video games. Journal of Adolescence 27, 113–122 (2004) 4. Anderson, C.A., Bushman, B.J.: Effects of violent video games on aggressive behavior, aggressive cognition, aggressive affect, physiological arousal, and prosocial behavior: a meta-analytic review of the scientific literature. Psychological Science 12(5), 353–359 (2001) 5. Anderson, C.A., Dill, K.E.: Video games and aggressive thoughts, feelings, and behavior in the laboratory and in life. Journal of Personality and Social Psychology 78(4), 772–790 (2000) 6. Griffiths, M.D.: The observational analysis of adolescent gambling in U.K. amusement arcades. Journal of Community and Applied Social Psychology 1, 309–320 (1991) 7. Uhlmann, E., Swanson, J.: Exposure to violent video games increases automatic aggressiveness. Journal of Adolescence 27, 41–52 (2004) 8. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly 13(3), 319–339 (1989) 9. Choi, D., Kim, J.: Why people continue to play online games: in search of critical design factors to increase customer loyalty to online contents. Cyberpsychology & Behavior 7(1), 11–24 (2004) 10. Peter, V., Tilo, H., Christoph, K.: Explaining the enjoyment of playing video games: the role of competition. In: Proceedings of the second international conference on Entertainment computing (2003) 11. Vorderer, P., Bryant, J.: Playing Video Games: Motives, Responses, And Consequences. Lawrence Erlbaum Associates, Mahwah (2006) 12. Karolien, P., Yvonne de, K., Wijnand, I.: It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. In: Proceedings of the 2007 conference on Future Play (2007)
314
X. Fang, S. Chan, and C. Nair
13. Niklas, R., Mikko, S., Jussi, H., Timo, S., Jari, L., Aki, J., et al.: Emotional response patterns and sense of presence during video games: potential criterion variables for game design. In: Proceedings of the third Nordic conference on Human-computer interaction (2004), http://portal.acm.org/citation.cfm?id=1028068# 14. Rajava, N., Saari, T., Laarni, J., Kallinen, K., Salminen, M.: The Psychophysiology of Video Gaming: Phasic Emotional Responses to Game Events. In: The Changing Views: Worlds in Play (2005) 15. Ermi, L., Mayra, F.: Fundamental components of the gameplay experience: Analysing immersion. In: de Castell, S., Jenson, J. (eds.) Changing Views: Worlds in Play. Selected papers of the 2005 Digital Games research Association’s Second International Conference (2005) 16. Klimm, C.: Dimensions and determinants of the enjoyment of playing digital games: a three-level model. In: Copier, M., Raessens, J. (eds.) Level up: Digital games research conference, Utrecht University, Faculty of Arts (2003) 17. Chumbley, J., Griffiths, M.: Affect and the Computer Game Player: The Effect of Gender, Personality, and Game Reinforcement Structure on Affective Responses to Computer Game-Play. Cyberpsychology & Behavior 9(3), 308–316 (2006) 18. Hazlett, R.L.: Measuring Emotional Valence during Interactive Experiences: Boys at Video Game Play. In: Proceedings of CHI 2006, pp. 1023–1026. ACM, New York (2006) 19. Mandryk, R.L., Inkpen, K.M., Calvert, T.W.: Using psychophysiological techniques to measure user experience with entertainment technologies. Behaviour & Information Technology 25, 141–158 (2006) 20. Selnow, G.W.: Playing videogames: The electronic friend. Journal of Communication 34(2), 148–156 (1984) 21. Wigand, R.T., Borstelmann, S.E., Boster, F.J.: Electronic leisure: Video game usage and the communication climate of video arcades. Communication Yearbook 9, 275–293 (1985) 22. Phillips, C.A., Rolls, S., Rouse, A., Griffiths, M.D.: Home video game playing in schoolchildren: A study of incidence and patterns of play. Journal of Adolescence 18, 687– 691 (1995) 23. Griffiths, M.D.: Are computer games bad for children? The Psychologist: Bulletin of the British Psychological Society 6, 401–407 (1991) 24. Sherry, J.L., Desouza, R., Holmstrom, A.: The appeal of violent video games in children. In: The Broadcast Education Association annual conference, Las Vegas, NV (2003) 25. Sherry, J.L., Desouza, R., Greenberg, B., Lachlan, K.: Relationship between developmental stages and video game uses and gratifications, game preference, and amount of time spent in play. In: The International Communication Association annual conference, San Diego, CA (2003) 26. Sherry, J., Holmstrom, A., Binns, R., Greenberg, B., Lachlan, K.: Gender differences in video game use and preferences. In: The National Communication Association annual conference, Miami, FL (2003) 27. Grodal, T.: Video games and the pleasure of control. In: Zillmann, D., Vorderer, P. (eds.) Media entertainment: The psychology of its appeal, pp. 197–213. Erlbaum, Mahwah (2000) 28. Vorderer, P., Hartmann, T., Klimmt, C.: Explaining the enjoyment of playing video games: The role of competition. In: Proceedings of the Second International Conference on Entertainment Computing, Carnegie Mellon University, Pittsburgh (2003) 29. Vorderer, P., Klimmt, C., Ritterfeld, U.: Enjoyment: at the heart of media entertainment. Communication Theory 14(4), 388–408 (2004) 30. Nabi, R.L., Krcmar, M.: Conceptualizing media enjoyment as attitude: implications for mass media effects research. Communication Theory 14(4), 288–310 (2004)
Playability Testing of Web-Based Sport Games with Older Children and Teenagers Xavier Ferre, Angelica de Antonio, Ricardo Imbert, and Nelson Medinilla Universidad Politecnica de Madrid Campus de Montegancedo 28660 - Boadilla del Monte (Madrid), Spain {xavier,angelica,rimbert,nelson}@fi.upm.es
Abstract. Playability occupies a central role in videogame design. Heuristics may help for establishing the game concept, but some testing is essential for ensuring a wide acceptance in the target user population. The experience of designing and testing a set of web-based sport videogames is described, focusing on the heuristics employed and the testing approach. The results show that an emphasis on a simple set of game controls and the introduction of humorous elements has obtained a positive response from older children and teenagers. Keywords: videogame design, playability heuristics, testing with older children and teenagers, sport web-based games, Olympic Games.
For the purpose of designing videogames, as in any interactive software development effort, evaluation plays a very important role. Evaluation with representative users drives the whole design process, ensuring that the game concept is well understood and that playability reaches an appropriate level; and it may also provide some evidence about the degree to which the videogame is engaging and easy to learn for the target population. In this paper, the process of developing a set of web-based online sport games about the History of the Olympics is described, emphasizing the evaluation approach taken and describing the results obtained so far. The remainder of the paper is organized as follows. Section two describes the project structure. In section three, the development approach is presented. Section four includes the heuristics employed, while section five analyzes the usability testing carried out. Next, the main results of the evaluation through usability testing are presented in section six. Finally, section seven summarizes the findings and presents the conclusions.
2 Online Games about the History of Olympics Modern Olympic Games started with the 1896 games in Athens, and they have taken place every four years since then, with some gaps due to world wars. Madrid is one of the four candidate cities currently bidding to host the 2016 edition of the Olympic Games. The Playability Evaluation of Games about the History of the Olympics project aims to deliver the Olympic spirit and knowledge about the history of the modern Olympic Games, sponsored by the Madrid town council, through a series of webbased online games that portray the reality of the Olympics as they evolved starting in Athens 1896. The main target users of these games are older children (10-12) and teenagers (13-16), who will be in an appropriate age to become Olympic volunteers in 2016, when the Olympic Games might take place in Madrid. In order to act as a support to the Madrid 2016 candidature by attracting the greater number of visitors to the website, the project intends to provide an entertaining experience to the online game player, considering playability as the primary focus throughout the development effort. 2.1 Games Structure The website structure is organized around each Olympic Games edition. In a first phase, the project covers the first eight modern Olympic Games, from Athens 1896 to Amsterdam 1928. A particular sport has been chosen for every Olympic Games edition, recreating in the videogame the ambiance and rules at that time, but there are additional elements around the sport game contributing to the educational aim of the project, as shown in Fig. 1. When accessing a particular Olympic Games edition, the player is presented a world map where it is to be located the place where the Olympics took place. In case the player chooses a point that is not close enough to the target city, the player is redirected to a card game where points can be gained in order to get additional clues
Playability Testing of Web-Based Sport Games with Older Children and Teenagers
317
start geographical game
athlete selection
sport game
card game
podium
multimedia newspaper Fig. 1. Game structure for each Olympics edition
in the world map. The card game is based on swapping adjacent cards to form horizontal or vertical combinations of three or more cards with the same design. After successfully locating the city in the world map, the player chooses the gender and race of the character he or she will play with, and the competition begins in the
318
X. Ferre et al.
sport game. After winning a medal or a trophy1, a multimedia newspaper is presented to the player, with a voice-over reading its contents, so that interesting historical facts are presented to the player. Afterwards, the player is given access to the following Olympic Games edition in chronological order.
3 Development Approach Before starting the development of the sport games, a study was performed about existing similar games freely available on the World Wide Web. The great majority were written in Adobe Flash, as it is a multimedia platform that eases the tasks related to animation graphics and interaction in a web environment, so Flash was chosen as programming platform. Regarding the game size (in pixels) in the web pages where they are embedded, the average size in the analyzed web-based sport games is around 550x400. A slightly bigger size has been decided for this project, 640x480, in order to allow games with a greater interactive area, while still fitting the great majority of desktop PC screen resolutions. With respect to Flash file size, the chosen game size is still between reasonable limits in terms of user bandwidth consumption. Concerning ease of learning, while the majority of the analyzed games did not provide a training mode, a significant minority did. For the development of each game, guidance was provided by a team of sport history experts. This guidance included a detailed description of the scenery where the sport event depicted in the game took place, of judges, sportsmen and sportswomen equipment, including their attire; the content of each Olympic Game newspaper (pictures, video and text); and the rules for the chosen sport at that time. The choice of the sport to depict in each Olympic Game and the design of the game concept was performed jointly by the technical team and the team of sports history experts. Particularities or special events that characterized the chosen sport in such edition of the Olympic Games were introduced in the game concept, in order to improve the game in terms of historical resemblance (see section 6.1). Playability heuristics were considered in the early design of each game (see section 4). According to the historical guidelines and the game concept definition, a first prototype was designed, running in parallel with the design of the graphical elements and decorations. The aim of this first prototype was just allowing the game to be played, with graphical aspects reduced to a sketch level, and without additional elements like instructions, podium screen, educational newspaper and so on. With the first playable prototype some evaluation was performed with children and teenagers at collaborating schools and high schools, in order to check the understandability of the game mechanics, and the operability of the game controls. After feedback had been gathered from this first playability test, the game mechanics and controls were reconsidered, along with the development of the whole graphical part and the inclusion of the remaining details in the game. A second round of playability testing was performed afterwards with a new set of test participants in the target age range, in order to check if the objectives had been met. Sections 5 and 6 detail the usability tests carried out for playability evaluation and the results obtained. 1
Depending on the particular Olympic Games edition, event winners were awarded either trophies or medals.
Playability Testing of Web-Based Sport Games with Older Children and Teenagers
319
4 Heuristics Several sources have been considered for choosing the playability heuristics to be taken into account in early design activities: Generic ones for any kind of game [4][6], and heuristics for mobile games [7]. Chosen heuristics are the ones that better fit the kind of simple games to be developed in the project (see Table 1). Table 1. Chosen playability heuristics and their sources # 1
2 3 4 5 6 7 8 9
10
11 12 13 14 15 16 17 18 19 20 21 22 23
Heuristic description The player should be always able to identify the score/status and goal in the game. The player sees progress in the game and can compare results with other players. The interface embodies metaphors with physical or other systems that the user already understands. The interface uses randomness in a way that adds variety without making controls unreliable. The interface uses humour appropriately. Player's fatigue is minimized by varying activities and pacing during game play. The game is enjoyable to replay The game is fun for the player first, the designer second and the computer third (put non-expert player’s experience first). The first player action is painfully obvious and should result in immediate positive feedback. Pace the game to apply pressure but not frustrate the player. Vary the difficulty level so that the player has greater challenge as he or she develops mastery (easy to learn, hard to master). Challenges are positive game experiences, rather than a negative experience (results in their wanting to play more, rather than quitting). Mechanics/control actions have consistently mapped and learnable responses. Shorten the learning curve by following the trends set by the gaming industry to meet user’s expectations. Provide immediate feedback for user actions. Upon initially turning the game on the player has enough information to get started to play. Sounds from the game provide meaningful feedback or stir a particular emotion. Players do not need to use a manual to play the game. The interface should be as non-intrusive to the player as possible. Art should be recognizable to player, speak to its function and support the game. Indicators are visible The player understands the terminology The player does not have to memorize things unnecessarily The game contains help The player is in control.
5 Usability Testing for Playability Evaluation Every game has been evaluated through usability testing [8]. Representative users play with the game while test facilitators observe their performance, taking notes about their comments, and about the problems they experience when trying to play the games. After the test each user is asked to fill a satisfaction questionnaire to gather his or her impressions about the game. A total of seven test sessions have been carried out, with the participation of 148 older children and teenagers in the 10-16 year old range, belonging to five schools and high schools. This section presents the methodology followed for the usability testing. 5.1 Previous Planning Test participants fall in the defined target age range, and the task they are asked to perform is to play the game in order to be the Olympic winner. Tests are always performed in the test participants' educational institution, usually in the computer classroom. Test planning involves preparing in advance the following materials: Content of the questionnaires for test participants to fill in; selection and preparation of any promotional gift offered to thank them for their participation; consent forms to be distributed between parents; and a detailed description of test procedures, including a script for the verbal instructions to be offered to participants. Games are previously tested in the computers where the test will take place, in order to avoid possible problems either with the Internet connection or related to software compatibility. Whenever possible a pilot test is performed in order to review test setup and planning. 5.2 Test Sessions Before granting them access to the game, test participants are given a brief introduction to the project objectives and their role as game testers is highlighted. They are asked to follow the instructions on the screen and to ask about any aspect of the game that feels strange or incomprehensible to them. Test sessions usually cover the testing of two different games. A fixed time of around 15 minutes is established for playing with each game, but test facilitators may suggest a particular participant to advance to the following game (after filling the corresponding questionnaire), in case he or she gets tired of the previous game or is winning easily. Test facilitators observe test participants, taking notes about any comment or remarkable behavior from them; and answering any question test participants may ask. After playing with each game, test participants are requested to fill in a satisfaction questionnaire, and when the test session ends they are thanked for their participation and the gifts are distributed. 5.3 Usability Report Data gathered in the test session includes test participants' performance with the games (automatically logged during the test session), their answers to the satisfaction
Playability Testing of Web-Based Sport Games with Older Children and Teenagers
321
questionnaire and notes taken by test facilitators. These pieces of data are analyzed and presented in the form of a usability report that details the main usability problems identified, and the positive and negative comments to different parts of the game. When suggestions for improvement have been mentioned by test participants they are also included in the document. The report is discussed by the development team in order to evaluate the priority and severity of the problems identified, and an action plan is established in order to make the appropriate changes to game design. When test results are satisfactory according to project objectives, the game is released in the project website [5].
6 Results The tests carried out with the different versions of the games have allowed for an important improvement in terms of playability, both between the final version of a specific game compared to the previous versions tested, and between the first games developed and the rest of them. Children and teenagers who have tested the games have liked them in general terms, with the answers to the satisfaction questionnaires showing that they agreed to the sentence "I would like to play often this game" to a high (47% of participants) or very high extent (32% of participants). Additionally, most of test participants showed reactions when playing with the final version of the games that suggested they were actually enjoying the games in the test sessions. The main two issues raised in the usability testing are related to the particular elements introduced in the games to increase the "fun" aspect, and to the appropriateness of including a guided training tutorial. 6.1 Inclusion of Funny Elements In certain games, funny or peculiar elements that attract the attention of the player have been introduced in the game concept, in order to make the overall player's experience more enjoyable. Whenever possible, these elements highlight particularities of the sport event as it happened in the corresponding Olympic Games edition, specially those ones that may be considered striking nowadays. This is part of the educational aim of the project, for the users to know the reality of the first editions of the modern Olympic Games. For example, in the marathon game - for London 1908 - there are some bottles and sponges with water, which have to be picked by the player in order to maintain the stamina and be able to win the race. But a glass of champagne appears randomly as well, with the effect of losing all the stamina (see Fig. 2). In this way it is shown to players that it was not uncommon for sportsmen to receive any kind of drink from watchers willing to help. In the tests performed with the marathon, children commented about how much they liked having objects appearing on the road, and that they would have liked having even more objects appearing in the game.
322
X. Ferre et al.
Fig. 2. Objects in the marathon game: Champagne glass, sponge and water bottle
Whenever these elements are funny, the reaction from older children and teenagers is even better. In the high jump game - Athens 1896 - there is a preparation phase before the actual jump where a correct posture must be achieved, and getting the sportsman to lean forward too much makes him fall to the ground, as shown in Fig. 3.a. In the sack racing game - Saint Louis 1904 - there is a chicken appearing randomly between the sportsmen or sportswomen (see Fig. 3.b). The observed reaction of young test participants to these elements has been very positive, since they cheered out loudly and they hurried to tell their friends about these particular elements, showing that they took their attention. With the inclusion of these funny ingredients, the objective of offering players an enjoyable experience has been reinforced, since these elements have been very well received by test participants, who seemed to appreciate startling elements in the game. 6.2 Training Tutorial vs. Simple Set of Controls For a web-based online game, ease of learning plays an important role, since players need to be enjoying their playing experience from the very beginning. When the player does not understand the game concept or gets stuck because of ignorance about game controls, there is a high risk that he or she will abandon the game to go to any of the thousands of other games freely available on the Internet. In the first game, the high jump, a relatively complex game concept was designed, since there is a preparation phase before the actual jump where a correct posture must be achieved, and there is a jump phase where accurate pressing of control keys influences the jump result. For it to be challenging, the jump phase happens very quickly, but this makes it hard for the novice player to grasp the game concept. Therefore, a guided training tutorial was introduced at the beginning of the game. In this tutorial, the player may try to perform a jump slower than in competition, and feedback is offered about the posture and key pressing. Comments from test participants about this game have shown that, while they enjoyed the actual competition, they did not specially enjoy having to go through a training tutorial.
Playability Testing of Web-Based Sport Games with Older Children and Teenagers
323
Fig. 3. a) Fallen sportsman in high jump game; b) chicken in sack racing game
In the following games tackled, the design of the game concept was focused on reducing to a great extent the complexity of controls, so that there was no need for a training tutorial. A screen with brief instructions is presented instead, which is shown the first time the game is played. Test participants were asked in the satisfaction questionnaire if the game was easy to understand and play, and results showed that the first game including a training tutorial was perceived as less easy to understand and play (by 11%) than the average of the other games, which were designed with a simpler and common set of controls. Additionally, test participants had a better initial performance in the games with a simpler set of controls that didn't need a training tutorial.
7 Conclusions In the frame of the Playability Evaluation of Games about the History of the Olympics project eight games have been developed and evaluated through usability testing with older children and teenagers. Even if further testing should be done in order to discover why some particular elements work better than others, results already show some hints that could help developers of web-based simple games, similar to the sports games developed in the project. When the game concept is kept simple, it seems to favor game acceptance between users in the 10-16 age range, especially in terms of game controls. In that direction, when the game is simple enough so that basic controls may be explained in an introductory screen and no training tutorial is needed, players tend to perceive it as easier to control and their initial performance improves. Startling elements included in a game have been highlighted by test participants as funny, showing their appreciation by cheerful comments with peers. The project will go on evaluating the educational potential of the developed sport games. It is planned to carry out an experimental study in physical education classes, comparing the knowledge about the history of the Olympic Games between a group of children taught in the traditional way (lectures) and a group of children that use the games as a way to actively learn about this subject.
324
X. Ferre et al.
Only the first eight editions of the modern Olympic Games have been covered so far, with encouraging partial results, but the development of additional games to cover the rest of Olympics editions will be conditioned by the results obtained in the experimental study.
Acknowledgments The authors wish to thank the members of the sport history experts team (M. Teresa Gonzalez Aja, Vicente Gomez Encinas, Elena Merino Merino and Alejandro de la Viuda Serrano) for having the original idea of the project and making it happen, and to the Madrid town council for their financial sponsorship to the project.
References 1. OnlineGames.Net - Play Free Games Online, http://www.onlinegames.net/ 2. Games at Free Online Games.com, http://www.freeonlinegames.com/ 3. Jones, S., Fox, S.: Pew Internet Project Data Memo. Generations Online 2009. Pew Internet & American Life Project (2009), http://www.pewinternet.org/pdfs/PIP_Generations_2009.pdf 4. Malone, T.W.: Heuristics for designing enjoyable user interfaces: Lessons from computer games. In: Proc. of the 1982 Conference on Human Factors in Computing Systems, pp. 63– 68. ACM, New York (1982) 5. Playability Evaluation of Games about the History of the Olympics site (2009), http://raptor.ls.fi.upm.es/olimpiadas/ 6. Desurvire, H., Caplan, M., Toth, J.A.: Using Heuristics to Evaluate the Playability of Games. In: CHI 2004 Extended Abstracts on Human Factors in Computing Systems, pp. 1509–1512. ACM, New York (2004) 7. Forum Nokia: Mobile Game Playability. Version 1.0. Nokia (2006), http://sw.nokia.com/id/5ed5c7a3-73f3-48ab-8e1e-631286fd26bf/ Mobile_Game_Playability_Heuristics_v1_0_en.pdf 8. Dumas, J.S., Redish, J.C.: A Practical Guide to Usability Testing. Revised Edition. Intellect, Exeter, England (1999)
Exploring the Elements and Design Criteria of Massively-Multiplayer Online Role-Playing Game (MMORPG) Interfaces Chun-Cheng Hsu1 and Elvis Chih-Hsien Chen2 1
Department of Communication and Technology, National Chiao Tung University, 1001 Ta-Hsueh Rd., Hsinchu 300, Taiwan 2 Institute of Communication Studies, National Chiao Tung University 1001 Ta-Hsueh Rd., Hsinchu 300, Taiwan 300 [email protected], [email protected]
Abstract. A great many people play online games and sales of online games are considerable, but research has further shown that a major reason behind the failure of games in the market is poor user interface design or usability, highlighting even more the importance of these issues in games design. This research uses surveys and focus groups to explore the factors influencing the usability and interface design of online games. First the definition and different types of game are discussed, and then the composition and features of online game interfaces analyzed. Second, a review is made of Human-Computer Interaction (HCI) research literature relating to design criteria for game interfaces. Finally, in discussion with experts, this study isolates the design criteria that should be emphasized when designing each key element of an online game interface. Keywords: Massively-multiplayer online role-playing games (MMORPG), interface design, elements of game interfaces.
games fail in the market, a major reason for which is that principles of HCI research have not been implemented, forcing players to rely solely on their intuition and experience to operate games. Besides this, currently there exists in the games industry a lack of criteria and guidance that games designers can follow. Many interface design criteria are excessively abstract and general, lacking any detailed explanation of how they may be met [1].
2 Purpose of Research With this in mind, the goal of this research is to explore the elements that comprise online games interfaces and criteria for interface design. First, this study looks back over the context of the development of online games in order to understand the basic definition of online games and the types and composition of interfaces. Second, taking MMORPGs as a case study, this paper analyzes and comes to some conclusions about the elements that make up the interfaces of online games. Finally design criteria important within the field of HCI are discussed, and in the process corresponding detailed design criteria for each interface element are identified. It is hoped that these findings will provide a reference for online game development and interface design.
3 Games and Interface Design 3.1 Definition and Types of Computer Game Computer games generally refer to electronic games played on personal computers (Zhou Rong, 1998), and are one of the important forms of game to emerge in the digital age. On the basis of past definitions and revisions, Juul [7] concluded that games possess six characteristics: they are rule-based, provide different possible outcomes, attach positive or negative values to these outcomes, involve player effort, players are attached to the outcomes and negotiate any consequences of success or failure. The specialist computer game magazine Computer Gaming World has divided electronic games into eight genres: action/arcade, adventure, role-playing, simulation, sports, strategy, classic/puzzle and war games. Novak [9] alternatively categorized games into action/arcade, adventure, arcadeadventure, casino, puzzle, role-playing, simulation and strategy games. Rollings & Adams [10] divided games into action/arcade, strategy, role-playing, sports, vehicle simulation, construction and management simulation, adventure, growth and puzzle games. MMORPGs currently dominate the online games market, and so accordingly this study will focus mainly on exploring this type of game, discuss the course of the development of online role-playing games and briefly describe their characteristic qualities. 3.2 Online Role-Playing Games Online role-playing games originate with pencil and paper tabletop role-playing games (TRPG). In a TRPG, using the game’s rulebook and scenario, players play
Exploring the Elements and Design Criteria of MMORPG Interfaces
327
characters in a story and interact with other players playing other characters in accordance with the rules of the game. Dungeons and Dragons is the most wellknown TRPG, and has deeply influenced the way that later multiplayer role-playing games are played, the arrangement of their plots and framework of their stories (Shuffle Alliance, 2002). Embryonic forms of today’s online role-playing games, MUDs (Multi-user Dungeons), first appeared in the late 1970s, and were conducted entirely in text, lacking any graphics [4]. Today’s online role-playing games can be said to be a kind of graphical MUD. A MUD allows many players to simultaneously log into a computer program and participate in adventures, roam around and interact. In this virtual space, players can play their characters and interact with other players, as well as create their own objects and environments to their own liking (Su Fenyuan, 1996). In an ordinary single player role-playing game, the player tries to reach successive break points and finally complete the game. However in a MUD, there is no such end point or success and failure. In fact, players can add new objects and scenes to its database at any time, expanding the entire MUD world. As in current online games, in a MUD players can simultaneously interact, communicate and participate in social interaction with many other players [4]. Although online role-playing games have inherited the characteristics of MUDs, they differ in that they are conducted via completely graphical user interfaces (GUI). A GUI not only increases the entertainment factor of the games, but also creates a more complete virtual reality. (Zheng Rongji, 2006). According to past related research, online role-playing games share four characteristics (Lin Peiyuan, 2007; Rolling & Adams, 2003): role-playing attributes, combat and adventure, a field of play and group interaction. To summarize, MMORPGs are games conducted in a virtual environment in which players attempt to increase their standing within the game, carry out tasks and missions, or simply enjoy interacting with other players within the rules of the game. However, a game can only progress by means of players’ interaction with the game mechanism via a control interface. Below some discussion is now given and conclusions drawn regarding the constituent elements of games interfaces according to past related research. 3.3 Games Interfaces The definition and types of games interfaces. Novak [9] believes a game’s interface allows players to be able to control characters in a game, move through the whole environment of the game, and make decisions during the course of the game. If there were no games interface, then a game would be merely a story, an animation or a static environment – a game that would be impossible to actually play. Consequently, an interface provides the connection between the player and the world of the game, allowing the player to then interact with the content of the game. As far as the definition of an interface is concerned, Clanton (1998) distinguished two components of interfaces: a motor component, including for example how a joystick operates a game; and a perception component, including the design of menus, tools and buttons and how they are displayed on the screen. Ding Zhaochen (2006) defined a game interface as the spatial environment on a screen of a game and its
328
C.-C. Hsu and E.C.-H. Chen
interactive mechanism. The above shows that there are broad and narrow definitions of games interfaces, Clanton’s (1998) definition being the former, includes both hardware input devices and graphic display; the latter refers solely to the display of illustrative and operational screens. Rollings & Adams [10] point out that the main function of a game interface is to allow the user to interact with the game, and can be split into two parts. The first is to receive the player’s commands, and the second is to inform the player of the state of play during the course of the game, and to endeavor to display all options open to the player with images. These different functions and capabilities require interfaces to be made up of different constituent elements. Saltzman [11] divided a game’s interface into exterior interface and user interface components; the former refers to menu screens mainly used for game-related setup options such as adjusting sound, changing game type, equipment control and saving the game; the latter refers to the screens that the player will encounter during the course of the game, such as the appearance of avatars, maps of the environment and skill buttons. Looking at interfaces from the perspective of presentational form, Boyd [2] divided games interfaces into two types, configurational interfaces and pictorial interfaces. A configurational interface chiefly refers to the control panes in a game, the majority of which are located on the edges of the screens of a game which provide the player with appropriate hints and messages; the emphasis in a pictorial interface differs from a configurational interface’s emphasis on function, in that it resembles a real picture, namely the spatial environment of the game during play. Novak’s [9] definition of a game interface is somewhat broader, including both a non-video interface (namely operating interface) and a video interface. The former includes hardware components such as control buttons, keyboards and mice, which allow the player to carry out choices, such as use a particular weapon, walk around within the game space, or to interact with computer characters or other players. The video interface refers to those functional features displayed on the screen that allow the player to know the status of other avatars within the world of the game, the location of the enemy, distance and direction and other information important to game play. Players base decisions about their own behavior in the game upon this information. The video interface may be further divided into an active video interface and passive video interface, according to whether or not it is under the player’s control. The active video interface refers to that with which a player may interact. Normally this is carried out by clicking on options displayed on the interface, normally presented on the interface in the form of menus or actional systems. The passive video interface consists of those interface elements with which a player is unable to directly control or change, such as players’ avatar status, score and time remaining. This information may be grouped in the same place on the screen, or may be scattered in different places [9]. The above is summarized in Table 1. From this table it can be seen that Novak’s [9] video interface category encompasses the categorizations given by both Saltzman [11] and Boyd [2]. If we then differentiate according to whether or not a player has control, then the active video interface encompasses the external interface, user interface and configurational interface; and the passive video interface includes the user interfaces and pictorial interface. With these categorizations in mind, this research will now discuss the common elements constituting the video interfaces of ordinary games and online games.
Exploring the Elements and Design Criteria of MMORPG Interfaces
329
Table 1. Categorizations of game interface types (source: this paper) Scholar Saltzman [11]
Boyd [2]
Type External Interface User interface Configurational Interface Pictorial Interface
Novak [9]
Active video interface Passive video interface
Definition refers to menu screens, mainly used for game-related setup options refers to the screens that the player will see during the course of the game chiefly refers to the control panes in a game the screens displaying the spatial environment of the game during play Players can click to carry out interaction, normally presented on the interface in the form of menus or actional systems Elements of an interface the player is unable to directly control or change, such as score and time remaining
Constituent elements of a game interface. As described above, the interface within the screens of a game can be regarded as its video interface, and divided into active and passive types depending on whether the player has control. Novak [9] remarks that both the active and passive video interfaces in games share many common constituent parts, including information that the player can utilize and actions that a player must take in order to complete tasks in the game. This normally includes interface elements such as points/level, number of lives/stamina, maps, the character and the game startup menu. If one then divides these according to whether or not the player can control them, then score/level and number of lives/stamina and maps all belong to the passive interface; the character and game startup menu are both part of the active interface. This is clarified in table 2. Table 2. Game interface element types (source: this paper)
Video interface
Game interface type External Configurational interface Interface Active User Configurational interface Interface Configurational Interface User Passive interface Pictorial interface
Constituent elements of games interface Game startup menu Character (Avatar’s actions and functions) Points/level, number of lives/stamina, maps Maps (the spaces and scenes in the game), character (appearance of avatar)
The start menu and the avatar’s actions and functions are components that the player can click to control, and so both form part of the active video interface. They may be further categorized as exterior interface and user interface according to their function during the course of the game. In the passive video interface are included the player’s score, stamina and maps. The player has no way to directly interfere with or manipulate these, and they belong to the configurational interface; the pictorial interface mainly consists of a representation of the game’s virtual space and the player’s avatar. The elements that constitute interfaces differ with game type; those that make up MMORPG interfaces in particular are now dealt with below.
330
C.-C. Hsu and E.C.-H. Chen
Elements of MMORPG interfaces. This paper will look at three games that occupy a relatively large share of the market in Taiwan – ‘World of Warcraft’, ‘Maple Story’ and ‘Perfect World’, summarize the elements common to their interfaces, and then one by one categorize them according to previous scholars’ schemes (Table 4).
Fig. 1. MMORPG game user interface screen (grabbed from World of Warcraft)
A game’s active video interface chiefly incorporates two main types of interface, the user interface and the exterior interface. The former is primarily for game-related setup before the player logs in, while the latter includes the chat/message channel, avatar information, shortcut keys, backpack/toolbag, social activities, game menu/options and seek help/report abuse functions; these all closely related to the player during the course of the game. Manipulating or clicking these functions enables the player to interact with the game mechanism or other players. For details see fig.1. In a game’s passive video interface, although the player does not possess control, this kind of information informs the player of current circumstances and status, and so is very important to game play. This mainly appears at the top of the game screen, and includes the functions avatar status and map information. The former shows the present status of the player’s avatar, providing the player with information on which to base subsequent actions. The latter tells the player his physical location within the game’s spatial environment, aiding orientation. The game’s pictorial interface occupies the center of the screen, and includes a graphical representation of the spatial environment and the full view of the player’s avatar.
4 Discussion of Criteria for Game Interface Design Traditional HCI research has accumulated a great many interface design criteria and evaluation methods, and has already successfully been utilized in the design of interfaces of many systems and web pages. Scholars have repeatedly also cited
Exploring the Elements and Design Criteria of MMORPG Interfaces
331
Table 3. Correspondence of MMORPG interface elements to design criteria Design criteria 1. Affordance 2. Consistency 3. Ability to cancel 4. Grouping
5. Visibility
6. Easy to learn 7. Ability to undo 8. Feedback 9. Effectiveness 10. Easy to remember 11. Customizability
examples of design principles in games interfaces, but at present the actual use of design principles is still overly general [1]. The elements present in game interfaces differ with game type and function, and so detailed design criteria for each interface element are analyzed at this stage. 4.1 Method A review of relevant literature and a focus group are the main methods used at this stage to analyze the important criteria for the design of online game interfaces. This research summarizes 22 criteria from traditional HCI and related literature, as follows: affordance, consistency, ability to cancel, grouping, visibility, easy to learn, ability to undo, feedback, effectiveness, easy to remember, customizability, mapping, efficiency, error correction, notification upon completed actions, shortcut keys, minimizes mental load, satisfies player’s desire to control, standardization, easy to use, aesthetic appeal and distinctiveness. For sources and references for the definitions and explanations of each design criteria, please see Table 3. This study uses a focus group to collect qualitative data. A focus group is a kind of group interview where people share ideas [8]. This study uses a small group consisting of five experienced game or interface designers. First the research topic is explained to the participants, and then there follows a discussion of the relationships between 11 elements of MMORPG interfaces and the
332
C.-C. Hsu and E.C.-H. Chen
22 design criteria. The 11 interface elements all appear on a computer screen and the group must openly discuss decide the importance of the criteria in interface design. 4.2 Result The focus group decided slightly different design criteria are important for different interface types (See Table 4). The group considered that a game’s exterior interface should be easy to remember and use, allow the player to undo actions and be laid out so that related functions are grouped closely together. As for the design of a game’s chat and messaging channel, the group decided that ease of use is important in order to facilitate rapid communication with other players. They also believed that in terms the provision of information, an interface should provide rapid, timely feedback, allowing the player to know the current state of play at any time during the game. The group of experts indicated that the important criteria in avatar information design are aesthetic appeal, the ability to customize according to player preferences, visibility and mapping. Thus in the design of avatar-related images and colors, the primary emphasis should be on aesthetic appeal, and at the same time the connection between the images and their meaning should be clear, in order to enable the player to know his own status. In designing shortcut keys, it was decided that the important design criteria should include grouping of closely related functions, affordance, mapping and easiness to learn. Thus buttons with closely-related functions should be grouped together to allow a new player to easily learn how to use the interface, and at the same time the design of images should be such that the player can quickly understand their meaning, function and method of manipulation. When designing images for the toolbag and social interaction functions, first and foremost the principles of grouping related functions, distinctness of images and standardization should be borne in mind, so that the player is able to readily pick out the desired function from among many images. The design of game menus and help functions, apart from requiring the same emphasis on distinctness and standardization, also needs to emphasize affordance and customizability, to enable the player to immediately see how to operate each function, and at the same time be able to adjust the functions of the interface to suit his needs. In a game’s passive visual interface, the focus group considered it important that the design of the avatar status indicator be visually distinct and clear, enabling the player to be able to see his current status at a glance, and quickly notice any updated information. In map design, importance should be placed on good mapping, minimization of load on the player’s short term memory, ease of use and effectiveness. This is to reduce the level of difficulty the player encounters using the map, to efficiently assist the player’s movements in the game. In terms of the design of the pictorial interface, the group decided that aesthetic appeal, ease of use, and satisfying the player’s desire for control are all important design criteria, the goal being to make the player feel more entertained and a heightened sense of immersion in the game.
Exploring the Elements and Design Criteria of MMORPG Interfaces
333
Table 4. Categorization chart of elements of MMORPG interfaces and design criteria
User/Game Interface
Video Interface
Active
External/Syst em Interface
Game Interface Type
MMORPG Interface Element (function) Choose game server, create new character, choose character, delete character
Easy to use, Easy to remember, Ability to cancel, Grouping, Customizability, Aesthetic appeal
Chat/message channel
Channel selection, channel setup
Easy to use, Feedback, Effectiveness, Notification upon completed action, Aesthetic appeal
Representation of the game’s spatial environment and avatars
Grouping, Affordance, Mapping, Easy to learn, Aesthetic appeal Grouping, Distinctness, Standardization Grouping, Distinctness, Standardization Affordance, Distinctness, Customizability, Standardization Affordance, Distinctness, Customizability, Standardization Affordance, Distinctness, Customizability, Standardization Mapping, Minimizes mental load, Easy to use, Effectiveness, Easy to learn, Aesthetic appeal, Customizability Aesthetic appeal, Easy to use, Satisfies player’s desire for control, Easy to learn, Customizability
5 Conclusion The online games industry continues to flourish, and has become one of the focal points of development in digital entertainment media. How the design of an online game’s interface can enable a player to more easily get into a game, reduce difficulty in learning a game, and then more easily become immersed in the world of the game are extremely important issues. However, much research shows that the current criteria used in the design of interfaces for online games are excessively abstract and general, lacking the detail and methodicalness of real design. This study uses surveys and focus groups to explore the characteristics of the constituent elements of online game interfaces. Second, it analyzes related HCI research into the criteria in game interface design. Finally the important criteria that should be emphasized when
334
C.-C. Hsu and E.C.-H. Chen
designing the constituent elements of online game interfaces are discussed with experts. It is hoped that the findings of this research provide both practical design suggestions and also a foundation for future research into the design of online game interfaces.
References 1. Adream, B.E., Zender, M.: User interface design principles for interaction design. Design Issues 24(3), 85–107 (2008) 2. Boyd, D.S.: Representing space: The pictorial imperative. In: 2004 Workshop on Space and Spatiality, Napier University, Edinburgh (2004) 3. Cornett, S.: The usability of massively multiplayer online roleplaying games: Designing for new users. In: CHI 2004, Vienna, Austria (2004) 4. Curtis, P.M.: Social phenomena in text-based virtual realities. In: Kiesler, S. (ed.) Culture of Internet, pp. 121–142. Lawrence Erlbaum Associates, Mahwah (1997) 5. Federoff, M.A.: Heuristics and usability guidelines for the creation and evaluation of fun in video games. Unpublished thesis, Indiana University, Bloomington (2003), http://melissafederoff.com/thesis.html 6. Jorgensen: Marrying HCI/Usability and computer games: a preliminary look. In: Proceedings of the third Nordic conference on Human-computer interaction (2003) 7. Juul, J.: The game, the player, the world: Looking for a heart of gameness. In: Copier, M., Raessens, J. (eds.) Level up. Digital games research conference, pp. 166–186. Universiteit Urencht and DIGRA, Utrecht (2003) 8. Krueger, R.A., Casey, M.A.: Focus group: A practical guide for applied research, 3rd edn. Sage, Thousand Oaks (2000) 9. Novak, J.: Game development essentials. Thomson Delmar Learning, NY (2005) 10. Rollings, A., Adams, E.: On Game Design. New Riders Publishing (2003) 11. Saltzman, M.: Game Design: Secret of The Sages, 2nd edn. Brady Games, Indianapolis (2000) 12. Ye, J., Ye, D.: HCI and game design: From a practitioner’s point of view (2004), http://www.ye-brothers.com/documents/HCIGAMEDESIGN.pdf
Healthcare Game Design: Behavioral Modeling of Serious Gaming Design for Children with Chronic Diseases Hadi Kharrazi, Anthony Faiola, and Joseph Defazio Indiana University Purdue University Indianapolis, School of Informatics 535 W. Michigan St., Indianapolis, Indiana 46202 {kharrazi,afaiola,jdefazio}@iupui.edu
Abstract. This article introduces the design principles of serious games for chronic patients based on behavioral models. First, key features of the targeted chronic condition (Diabetes) are explained. Then, the role of psychological behavioral models in the management of chronic conditions is covered. After a short review of the existing health focused games, two recent health games that are developed based on behavioral models are overviewed in more detail. Furthermore, design principles and usability issues regarding the creation of these health games are discussed. Finally, the authors conclude that designing healthcare games based on behavioral models can increase the usability of the game in order to improve the effectiveness of the game’s desired healthcare outcomes. Keywords: Serious Gaming, Diabetes, Hypoglycemia, Behavioral Modeling, Patient Empowerment, Compliance to Treatment, Adolescent Diabetic Drivers.
Healthcare Game Design: Behavioral Modeling of Serious Gaming Design INDIVIDUAL PERCEPTIONS
MODIFYING FACTORS
Socio-demographics: age, gender, duration, etc
Perceived Susceptibility
Perceived Threat
Self-efficacy
337
LIKELIHOOD OF ACTION
Perceived benefit Perceived barriers
Intention to Take Action
Cues to Action
Fig. 1. The Health Belief Model
The Planned Behavioral Model (PBM) According to the theory of planned behavior, human action is guided by three components: “beliefs about the likely outcomes of the behavior and the evaluations of these outcomes (behavioral beliefs), beliefs about the normative expectations of others and motivation to comply with these expectations (normative beliefs), and beliefs about the presence of factors that may facilitate or impede performance of the behavior and the perceived power of these factors (control beliefs)” [10]. Indeed behavioral beliefs make a behavior favorable or unfavorable (attitude), normative beliefs produce a social pressure (subjective norm) and control beliefs show the intensity of control factors (perceived behavioral control). Fig. 2 represents a schematic view of the theory. Based on this model, behavioral changes can be achieved by targeting any of the factors: attitudes, subjective norms, or perceptions of behavioral control. The result of such an intervention should produce changes in behavioral intentions and, given adequate control over the behavior, the new intentions will be carried out under appropriate circumstances [11]. The common approach in behavioral models is determining the major elements that can affect participant’s (patient’s) behavior substantially enough to improve their behavior toward their chronic condition.
Fig. 2. The Planned Behavioral Model [10]
338
H. Kharrazi, A. Faiola, and J. Defazio
2 Background 2.1 Patient Empowerment and Serious Gaming Diabetic educators have defined patient empowerment as “the discovery and development of one's inherent capacity to be responsible for one’s own life. [And the necessity to] influence their own behavior and that of others to improve the quality of their lives” [12]. The major hurdle in traditional methods of patient empowerment is motivating patients to change their behavior and maintain that change. This problem becomes even more pronounced in adolescents. One of the most significant factors in better health outcomes for children with long term or chronic disorders is empowering the patients to consistently comply with the treatment regimen, even when they are not experiencing direct effects. Ubiquitous digital games, now considered a mass medium [13], can be exploited to achieve these health objectives. The factors that make digital games so engaging can be applied successfully in health contexts where motivation and engagement are necessary for the management of chronic conditions. Today, the world of computer gaming has expanded rapidly to include applications other than entertainment, such as education. This new genre of games is named “serious games” [14][15][16]. As such, games differ in their use of visual, textual, and auditory channels for feedback, scaffolding challenges, visible goal indicators, overviews and schematics, and ease of learning [17]. This is in stark contrast to the typical “casual games” that are entertainment-centric. The process of learning how to play, how to improve skills, and how to succeed is much more natural in most games, and for applications of diabetes self-management and nutrition education, they continue to show promise. 2.2 Examples of Serious Games for Type 1 Diabetes “Packy and Marlon” is a side-scrolling adventure game (played on a Super Nintendo console) that helps children and teens with type 1 diabetes self-management. “Glucoboy” is a glucometer that plugs into several video game devices, e.g., Nintendo Advance® and Nintendo Game Boy®. The game is intended to encourage blood sugar monitoring and testing in kids with diabetes, with an advanced blood glucose meter that is extremely accurate and highly precise. “Escape from Diab” is a serious videogame adventure in healthy eating and exercise, with the goal to help prevent kids from becoming obese and developing diabetes and other related illnesses. “Nanoswarm” is another game that is about diabetes education that provides the player with a rich, immersive, interactive game. The game’s developers hope that through role-playing and a blend of sci-fi action, they can bring the player to change their behavior toward a healthier life style. Developed by the USDA, the “MyPyramid Blast Off” is an interactive online game where kids can reach Planet Power by fueling their rocket with food and physical activity. Fuel tanks for each food group help children keep track of how their choices fit into MyPyramid. “Insulot” [18] is a cellular phone-based health education learning tool developed for T1D children in Japan. The game is intended to encourage, motivate, and boost the confidence of T1D patients, with outcomes to change behavior. “FatWorld” is a game about the politics of nutrition; a game the challenges the player to decide to be fit or fat, to live or to die.
Healthcare Game Design: Behavioral Modeling of Serious Gaming Design
339
3 Current Research 3.1 Serious Games and T1D Compliance to Treatment In a study with T1D patients, a game framework based on the Theory of Planned Behavior was used to increase the compliance/adherence rate to treatment. The main research question of this study is “Can games improve adherence to treatment in children with chronic diseases?” Parents reported the compliance rates of their children and the children (patients) were awarded in the game based on their compliance to treatment. Health points collected by compliance can be used in the game to buy new items or play additional mini-games. The study showed a significant increase in compliance rates [6]. Fig. 3 includes some screenshots of the game.
Fig. 3. Screenshots of a serious game designed for Type 1 Diabetic patients [6]. Left to right: Top - the login screen; main interface; using scores to buy new items. Bottom - using scores to change options; compliance report for children, and Playing mini-games based on points.
Conceptual Framework Theoretical models of empowerment, compliance and behavioral changes were broken down into its elements and possible matching game elements were identified for each of them. Many of the conceptual elements in empowerment, compliance and behavioral models overlapped each other and therefore were removed. For example, control factors are included in multiple models such as PBM, compliance models and patient empowerment models. Then, the elements that have close conceptual meaning were merged. For example, normative belief in PBM is close to family and peer support in contextual-behavioral model for patient empowerment. The remaining list was purged based on multiple focus group results. For example, none of the focus groups showed an interest in knowing the side effects of a given medication in the game. The final list of elements was refined based on the practicality of the game elements that are matched with the behavioral models. After final considerations and defining the practical solutions some feasible elements were identified and chosen as the key functional game elements that map to
340
H. Kharrazi, A. Faiola, and J. Defazio
different factors in empowerment, compliance and behavioral change models. The final framework included: Knowledge base and educational content, Motivational factors such as Pointing systems and Virtual mentoring systems, Peer Pressure and Measurement tools such as Questionnaires and Compliance reports. Usability Study Usability of this game framework is measured by questionnaires. Some of the questions are Likert-scale while some others are comment based. Usability questions are categorized as: Efficiency which reflects the goal of the game in affecting compliance; Satisfaction which indicates how much fun the game has been; Learnability that shows how easy learning the game was; and Memorability which refers to the ease of memorizing game features; and finally Error that measures the rate of error in the game’s functionality (Fig. 4):
Fig. 4. Mean usability score based on different categories
3.2 Game Pilot Study for Adolescent Diabetic Drivers The pilot study, titled, Testing for Hypoglycemia While Driving, is an educational game aimed at adolescent diabetic drivers with T1D. This study was a required first step (Phase 1) in developing a larger (Phase 2) and more intensive intervention within the context of a computer-assisted blood glucose awareness education program. As such, the resources required to develop and implement a large-scale intervention to change behaviors was beyond the scope of what is outlined here, i.e., in Phase 1. The purpose of the study, however, is to change adolescents’ and parents’ attitudes about hypoglycemia and driving, as well as to change their knowledge of perceived susceptibility, perceived seriousness and threat, perceived benefits to taking action, and barriers to taking action. Finally, the ultimate intent is to decrease the risk of hypoglycemia while driving among adolescents with diabetes.
Healthcare Game Design: Behavioral Modeling of Serious Gaming Design
341
Phase 1: Gaming Pilot Design and Testing Stage In Phase 1, the authors modified and adapted the Cox intervention to a brief educational intervention to be used on a laptop computer [19][20]. In this phase, gaming scenarios were designed and developed, along with the HBM attitudinal and behavioral measures. The theoretical underpinning of game design was HBM, because it addresses the understanding that: (1) a person will take a health-related action if they feel that a negative health condition, such as low blood sugar, can be avoided, (2) a person will have a positive expectation that adhering to the recommended action will avoid a negative health condition, and (3) a person believes that they can take a recommended health action with a successful outcome. Four learning scenarios were developed during the analysis and design stages of the project (Fig. 5). For example, Scenario 1 allowed the participant to make choices prior to and during the drive to school. Scenario 2 allowed the participant to make choices prior to and during the drive home. The theoretical grounding for the multimedia design and testing of Phase 1 was the Multimedia Development Model (MDM) [21], which was derived from the ADDIE (Analysis, Design, Development, Implementation, and Evaluation) instructional model. As reflected in ADDIE, MDM provided an iterative process that allowed the designers to re-visit each development stage during production. An important aspect of the model is its non-linear framework that allowed a repeatedly review and revise of the product. During the analysis phase, plans for content creation were reviewed. During the design phase, flowcharts and storyboards were developed using prewritten scenarios. Formative reviews of each asset and it’s functionality within each scenario helped to ensure the quality and standard of the design during production. HBM guided information presentation (susceptibility/severity of condition, benefits of behavioral change) and ultimate performance (behavioral cues) as the foundation for this design. During the analysis and design phases, graphical, textual, audio, and narrative assets were created. Once created, assets were incorporated into
Fig. 5. Scenario Two - Phase 1 of Educational Gaming for Adolescent Drivers with Diabetes
342
H. Kharrazi, A. Faiola, and J. Defazio
the design and reviewed for appearance, appropriateness, functionality, and fit. The final phase of the design cycle implemented usability testing on several subjects to ensure functionality and flawless execution of the design. Usability Testing Adolescent participants were recruited and the education game modules were tested for content clarity and usability through a usability and user experience evaluation of the product. After the usability findings were collected and analyzed the appropriate changes were made. The following figure (Fig. 6) depicts the average score for the participants’ responses to the usability questions (Likert-scale questions):
5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 1 Q
2 Q
3 Q
4 Q
5 Q
6 Q
7 Q
8 Q
9 Q
0 1 Q
1 1 Q
2 1 Q
3 1 Q
4 1 Q
5 1 Q
6 1 Q
7 1 Q
8 1 Q
9 1 Q
0 2 Q
Axis Title
Fig. 6. Average score of responses to usability questions
Phase 2: Gaming Pilot Testing Stage In Phase 2, researchers will administer the gaming modules to 150 adolescents ages 15 -18 and one of their parents. Information on behavior, attitude, and knowledge acquisition will be collected and analyzed to better predict the likelihood that adolescents and their parents will take action concerning the existing health condition.
4 Conclusion Interactive media, including serious games, are becoming an inescapable part of our everyday life. With the emerging digital gaming culture, serious gaming will become an increasingly vital part of healthcare education for upcoming generations. Advanced interactive user interfaces now provide new opportunities for serious games that can be used not only to educate younger patients about their disease, but also to empower them for a positive change in their behavior both now and into full adulthood. Chronic care constitutes a large slice of national healthcare expenses. Serious gaming can be an economical solution to educating, motivating, tracking, and empowering chronic patients for long term management of diabetes and prevention of complications. Serious games can lower not only the cost of chronic diseases but also
Healthcare Game Design: Behavioral Modeling of Serious Gaming Design
343
the additional costs associated with side effects. Moreover, serious games aimed at health education do not involve large expenses as minimum healthcare staff is required to interact with the patient. In an ideal game, the game’s behavioral model can be integrated into both the patient’s personal health record and the hospital/doctor’s health record system. The health record integration can help the system to reward the patient in the game based on actual improved health results, and it can also help the physician/healthcare staff to track the patient’s compliance to treatment; including other age groups such as adults and elderly patients. Artificial intelligence agents can be used to alert both the patient and the physician if the health status deteriorates. Moreover, different behavioral change models should be experimented with to find the most suitable model for an interactive patient empowerment approach. Managing hypoglycemia in adolescents is critical due to their lack of insight into their disease and the lower compliance rate to treatment. Behavioral models have been used to change the behavior of the patients toward their disease and to improve their adherence to treatment. In this article, Planned Behavioral Model and Health Belief Model were the underlying behavioral models. Two studies were conducted: one to increase compliance to treatment in T1D and another to reduce severe driving consequences of hypoglycemia ion T1D. Both studies showed significant results in usability measures of effectiveness and satisfaction. In summary, advancements in gaming technology, social and health informatics, and interaction design principles and practices, have all provided new knowledge and approaches to facilitating patient education and patient empowerment to achieve higher compliance [22][6].
Acknowledgement Portions of this chapter are used with permission from Dr. Donald Orr, Adolescent Medicine, Indiana University, School of Medicine, Indianapolis, Indiana.
References 1. Hannon, T., Rao, G., Arslanian, S.: Childhood Obesity and Type 2 Diabetes Mellitus. Pediatrics 116(2), 473–480 (2005) 2. Baranowski, T., Archimage, I.: Escape from Diab (Accessed, 2008) 3. Lee, C.: New Video Games Not Just for Fun. Washington Post (October 2006) 4. Hayes, B., Aspray, W.: The Informatics of Diabetes: A Research Agenda for the Socially and Institutionally Sensitive Use of Information Technology to Improve Healthcare. MIT Press, Cambridge (2009) (in press) 5. Lustria, M.: Can interactivity make a difference? Effects of interactivity on the comprehension of and attitudes toward online health content. Journal of the American Society for Information Science and Technology 58(6), 766–776 (2007) 6. Kharrazi, H., Watters, C., Oore, S.: Improving behavioral stages in children by adaptive applications. Journal on Information Technology in Healthcare 6(1) (2008)
344
H. Kharrazi, A. Faiola, and J. Defazio
7. Winn, B.: The Heart of Serious Game Design. In: Annual meeting of the International Communication Association, San Francisco, CA (2008), http://www.allacademic.com/meta/p170925_index.html 8. Hochbaum, G.: Health Belief Model: Why People Seek Diagnostic X-rays. Public Health Reports 71, 377–380 (1956) 9. Hochbaum, G.M., Sorenson, J.R., Lorig, K.: Theory in Health Education Practice. Health Education Quarterly 19(3), 293–313 (1992) 10. Ajzen, I.: The Theory of Planned Behavior. Organizational Behavior and Human Decision Processes 2(50), 179–211 (1991) 11. Ajzen, I.: From Intentions to Actions: A Theory of Planned Behavior, 11–39 (1985) 12. Funnell, M.M., Anderson, R.M., Arnold, M.S.: Empowerment: a winning model for diabetes care. Practical Diabetol 10, 15–18 (1991) 13. Wolf, M.: The Medium of the Video Game. University of Austin Press, Austin, USA (2001) 14. Blackman, S.: Serious games and less! ACM SIGGRAPH Computer Graphics 39(1), 12– 16 (2005) 15. Thompson, D.J., Baranowski, T., Buday, R., Baranowski, J., Juliano, M., Frazior, M., Wilsdon, J., Jago, R.: In Pursuit of Change: Youth Response to Intensive Goal Setting Embedded in a Serious Video Game. Journal of Diabetes Science and Technology 1(6), 907–917 (2007) 16. Ye, Z.: Genres as a Tool for Understanding and Analyzing User Experience in Games. In: CHI 2004 Extended Abstracts on Human Factors in Computing Systems, Vienna, Austria, pp. 773–774 (2004) 17. Dyck, J., Pinelle, D., Brown, B., Gutwin, C.: Learning from Games: HCI Design Innovations in Entertainment Software. In: Graphic Interface, Halifax, pp. 159–169 (2003) 18. Aoki, N., Ohta, S., Okada, T., Oishi, M., Fukui, T.: INSULOT: A Cellular Phone-Based Edutainment Learning Tool for Children with Type 1 Diabetes. Diabetes Care 28(3), 760 (2005) 19. Cox, D.J., Gonder-Frederick, L.A., Kovatchev, B.P., Julian, D.M., Clarke, W.L.: Progressive Hypoglycemia’s Impact on Driving Simulation Performance. Diabetes Care 23, 163–170 (2000) 20. Cox, D.J., Penberthy, J.K., Zrebiec, J., Weinger, K., Aikens, J.E., Frier, B.: Diabetes and Driving Mishaps: Frequency and Correlations from a Multinational Survey. Diabetes Care 26, 2329–2334 (2003) 21. Defazio, J.: An Innovative Approach for Developing Multimedia Learning Modules. In: ISECON 2001, Cincinnati, vol. 18, p. 34 (2001) 22. Faiola, A.: Designing Humane Technologies: A Potential Framework for HumanComputer Interaction Design. The International Journal of the Humanities 2(3), 1877–1886 (2006)
Analyzing Human Behaviors in an Interactive Art Installation Takashi Kiriyama and Masahiko Sato Graduate School of Film and New Media, Tokyo University of the Arts 2-5-1 Shinko, Naka-ku, Yokohama 231-0001, Japan {kiriyama,sato}@gsfnm.jp
Abstract. Arithmetik Garden is an interactive art installation designed to perform arithmetic operations by using the body. Analysis of data collected during its exhibitions shows that viewers behave differently than optimal solutions generated by computer. There is also an indication that viewer’s emotional changes can be detected by monitoring interactions. Keywords: Interaction, behavior, emotion.
1 Introduction Arithmetik Garden is an interactive art installation designed to provide the viewer with an experience of performing arithmetic operations by using the body. The viewer feels as if he or she became a number that is transformed when it goes through gates. There are eight gates in the installation, including the entrance gate, the exit gate marked with 73, and arithmetic gates of 5, 8, 3, 7, 4, and 2. The viewer picks up a card at the entrance and hangs it around the neck. An initial number -8, -1, 2, 4, 5, 7, 8, 36, 87, or 91 is printed on the surface of each card.
Starting from the initial number, the viewer tries to make the number equal to 73 by going through the arithmetic gates. For instance, if the viewer starts with 2 and goes through the 7 gate, the current number will become 2 7 14. One can walk along a path such as 2
14
22
11
77
73
to arrive at the goal number 73, where A
op
B
means that the current number A becomes B by going through the gate . The 2 gate works only with even numbers. If the viewer becomes uncertain about the current number, he or she can go to the equation monitor to see the current number and the path walked so far.
Fig. 2. Cards with initial numbers
Fig. 3. Equation monitor
The viewer can successfully leave the exit gate if the current number becomes 73. Outside the exit gate, an equation of the path is printed on a piece of paper such as 2 7 8 2 7 4 73 .
Analyzing Human Behaviors in an Interactive Art Installation
347
The state transition diagram mounted on the wall shows all paths leading to 73 (Figure 5). The viewer can reflect on the experience by tracing the path in the diagram, while discovering other possible paths. Arithmetik Garden was created in 2007 and first exhibited at Mori Art Museum, followed by the second exhibition at NTT Inter-Communication Center [ICC] in 2008, both in Tokyo, Japan. Over the two exhibitions, a total of 72,000 visitors experienced Arithmetik Garden. Some visitors reflected on their experience such that they were getting close to the goal but at some point they got lost. Other visitors mentioned to the enlightening moment in which they found a path to the goal. Such changes are of interest for interaction design because they make the experience engaging. We analyzed event logs and video recording to know how visitors walked in Arithmetik Garden and when their behaviors changed. In the rest of this paper, we discuss the methods and results of analysis.
2 Installation Arithmetik Garden employs RFID technology. When the viewer goes through a gate, an RFID sensor inside the gate reads the tag embedded in the card. Each event of passing through a gate is sent to the server and stored in the database. The server maintains the current number and the path of each viewer. The underlying mechanism of this installation is hidden from the user. People walk through gates without being bothered by menus and buttons. In fact, they do not touch anything from the entrance to the exit, because all sensing is done wirelessly.
Fig. 4. System layout
348
T. Kiriyama and M. Sato
Fig. 5. State diagram
Analyzing Human Behaviors in an Interactive Art Installation
349
3 Human Behaviors 3.1 Intuitive Decision During the exhibitions of Arithmetik Garden, we collected paths walked by viewers. Each record consists of an initial number and a sequence of arithmetic operations with timestamps. The records are represented in an XML format after exported from the SQL database. Once represented in XML, we can use XQuery and XSLT for data analysis. They are convenient to look for patterns in a large amount of data [1]. By comparing paths taken by viewers against the shortest paths calculated by computer, we discovered gaps between human decisions and logical optimality. For instance, if the current number is 10, there are two shortest paths to the goal; 10
15
11
77
73 and 10
6
11
77
73.
But among 6,259 cases in which the current number became 10, the largest number of people (2,694) chose the ×7 gate as the next (Fig.6). In fact, going to the ×7 gate takes the viewer away from the goal from distance 4 to distance 6. Here, distance d of a number N to the goal 73 is defined as the minimum number of steps needed to go to 73. Number 10 is of distance 4 to the goal because the two shortest paths to 73 shown above contain 4 steps.
Fig. 6. Selections from 10
350
T. Kiriyama and M. Sato
We suppose that the 7 gate appears to be most promising for the viewer whose current number is 10, as 10 7 70 is numerically close to 73. It seems that people have chosen this gate by intuition. Table 1 shows average time spent to choose the next gate after the current number became 10. It took 15.5 seconds in average to decide to go through 7, which was one of the two transitions that took the shortest time. This fact suggests that people intuitively chose the 7 gate because the resulting number was close to 73. Table 1. Number of people and average time for transitions from 10
gate
transition
number
time (sec)
7
10
70
2,694
15.50
3
10
30
1,175
17.18
5
10
15
1,079
15.14
8
10
18
482
24.45
4
10
6
448
21.48
2
10
5
381
21.28
6,259
17.22
total
3.2 Sticking to a Pattern Sometimes people are occupied in an immediate goal and do not look for alternatives. For instance, 65 is only one step away to the goal. One can make the number 73 by going through the 8 gate. Of a total 8,097 cases in which the current number was 65, 6,646 people went to 73. However, 1,123 people went through the 5 gate instead, such that; 65
70.
It means that in 14% of the total 8,097 cases, people missed the goal. Among the 1,123 people, 552 (49% of 1,123) came from the 5 gate as in; 60
65
70.
Moreover, 234 people (21% of 1,123) went through the +5 gate three times in a row such as;
Analyzing Human Behaviors in an Interactive Art Installation
60
65
70
351
75.
Fig. 7 illustrates these transitions. The pattern of repeating to go through the 5 gate can be explained that people stick to a simple pattern of adding 5 to the current number. Once people started to follow this pattern, they tend to forget about looking for alternatives.
Fig. 7. Sticking to a pattern of adding 5
3.3 Emotion Before the exhibition of the Arithmetik Garden, we did a public testing at university. We videotaped 340 people to analyze how they behave in the installation. By watching video, we found that people started to run near the end of the path since they became certain about how to make the goal. Some visitors were clearly seen an emotional change on the face [2]. It is also found in data that the intervals from a gate to the next became shorter near the end of path. We located such changes of intervals in data and reviewed the video of that moment. The video recording indicated that people were indeed delighted at knowing how to reach the goal. We believe that by monitoring data we can detect emotional changes It may be useful for making interactive systems respond to emotional changes of people [3]. Although initial numbers are 4 to 6 steps away from the goal, people spend 8 to 12 steps in average to reach the goal. We assume that people need to understand the situation and the task by exploring the space. Once the idea of embodiment of transformation becomes clear, they can start concentrating on calculation. If it is true, we may be able to detect another type of emotional change from exploration to concentration by watching the tendency of selecting gates.
352
T. Kiriyama and M. Sato
4 Conclusions We found that event data collected during exhibitions were extremely useful to study human behaviors in Arithmetik Garden. The data indicate that paths taken by humans were different from computer-generated optimal solutions. People tend to prefer to a seemingly promising path such as 10
70
over the shortest paths to the goal. The fact that people in average spent only a short duration of time on the transition to 70 indicates that this selection was made intuitively. Another finding is that people stick an immediate goal. While increasing the current number to 73 by going through the +5 gate multiple times, people tend to miss a solution such as 60
65
73.
It was also found that people started to run when they became certain about the way to the goal. Such a change can be detected by monitoring intervals between gates. Monitoring the user behavior will be useful to design interactive systems that respond to user’s emotional changes. Future research includes applying data analysis techniques developed in this study to other interactive art installations for better understanding of human behaviors.
Acknowledgements The exhibition of Arithmetik Garden was supported by Hayao Nakayama Foundation for Science & Technology and Culture, Dai Nippon Printing Co.,Ltd., Takaya Corporation, and Total Interior Sugahara Co.,Ltd. Data gathering in Arithmetik Garden was made possible by the corporation of Mori Art Museum and NTT InterCommunication Center [ICC]. We thank Masasuke Yasumoto, Masaya Ishikawa, Tomomi Kito, and Yoshikazu Fujita for their contribution in developing Arithmetik Garden.
References 1. Kiriyama, T.: Representing User Interaction Data of the Arithmetik Garden. In: Asian Topic Maps Summit 2007 (2007) 2. Kiriyama, T., Sato, M.: Observing Human Behaviors in an Interactive Art Installation. In: Desmet, P.M.A., Tzvetanova, S.A., Hekkert, P., Justice, L. (eds.) Proceedings from the 6th Conference on Design & Emotion (2008) 3. Norman, D.A.: Emotional Design. Basic Books (2004)
The Effects of Quest Types and Gaming Motivations on Players’ Knowledge Acquisitions in an Online Role-Playing Game Environment Jiunde Lee and Chih-Yi Chao Institute of Communication Studies National Chiao Tung University
Abstract. The study explores how the design of quest types and different players’ gaming motivations might affect knowledge acquisitions in an online role-playing game environment. An experiment was conducted to collect data. The results showed that “immersion motivation” had the most significant influence on knowledge acquisitions. The bounty-collection quest significantly affected the procedure knowledge of subjects with high immersion motivation, whereas the fed express quest affected declarative knowledge of subjects with high immersion motivation. Keywords: Online Role-Playing Games, Quest Types, Gaming Motivations, Declarative Knowledge, Procedural Knowledge.
1 Introduction As internet access has become more widespread, the computer game market has seen tremendous growth within the last few years. The explosion of computer games as a mainstream media form combined with captivating interactive technologies is viewed as a potential way of providing solutions for problems faced by divisions of instructions, corporations, governments, and public interest groups [22]. Such assumptions are largely based on the concept of computer games as a means of solving problems. In order to complete game quests, players with various gaming motivations become actively involved in the process of problem solving to make decisions, apply tactics, and construct necessary knowledge. Players’ gaming motivations and quest types appear to be the critical factors which engage players in this process. This study aims to explore how gaming motivations and quest types might affect players’ knowledge acquisitions in terms of declarative and procedural knowledge.
of challenges and quests (problem solving). According to Sawyer [22], the core concept of gameplay is actually a problem-solving process. Such a problem-based environment aims to promote dialogical interchange, argumentative decision making and reflective thinking [18]. Scholars [8, 15] have proposed that computer games could benefit the development of peoples’ perceptive and cognitive abilities such as inductive discovery, parallel processing, and spatial representation. Three issues are particularly worth considering within a game-based context: gaming motivations, quest types (problem spaces), and knowledge acquisitions. 2.1 Gaming Motivations One of the most cited papers about players’ gaming styles is Bartle’s 1996 study of player classification [2]. By using two coordinate axes - Acting / Interacting and Players / World, Bartle categorized game players into four gaming styles: Killer, Achiever, Socialiser, and Explorer. Yee [26], however, doubted the appropriateness of applying such a classification to define players, since it is quite common that one player might demonstrate more than one gaming style during gameplay. In fact, Bartle [2] himself also did not completely rule out the possibility that these four gaming styles might overlap with each other to a certain degree. Consistent with Malone’s [14] intrinsic motivations of playing games, Yee [27] suggested that players’ gaming motivations might be a better way to distinguish their different behavioral tendencies in games. There are three types of gaming motivations: Achievement, Social, and Immersion. Achievement motivation includes advancement, mechanics, and competition attributes. Social motivation includes socializing, relationship, and teamwork attributes. Immersion motivation includes discovery, role-playing, customization, and escapism attributes. In gameplay, players possess three types of gaming motivations simultaneously with one probably predominating the others. The present study adopts Yee’s viewpoint to observe how players’ gaming motivations might affect their knowledge acquisitions in different quest types. 2.2 Quest Types Newell and Simon [17] noted that a “problem space” is an organized unity made up of a set of operators and problem representation activities. As mentioned above, a computer game can be viewed as a problem space. By trial and error and hypothesis testing, players constantly search for solutions and construct knowledge of goals, rules, and concepts (operators) through this inductive discovery process [9]. Among the studies of investigating game quests [6, 12, 13, 23, 24, 25], Aarseth ‘s [1] and Dickey’s [4] studies particularly focus on quest design and have been frequently cited by scholars. Aarseth divided computer quests into three main formats: place-oriented, time-oriented, and object-oriented. In the place-oriented format, players have to find a path to the destinations; the time-oriented format usually fixes the time limit for the completion of missions; and the object-oriented format will set concrete targets for players to acquire, for instance, special gear from a monster NPC. These three quest formats work collaboratively to create a highly entertaining quest. Dickey [4] developed a more detailed category. Computer quests are defined as bounty quest, fed express quest, collection quest, escort quest, goodwill quest, and
The Effects of Quest Types and Gaming Motivations on Players’ Knowledge Acquisitions 355
messenger quest. Bounty and collection quests are quite similar in terms of goals and missions. Players in both quests have to contact NPCs for information then collect certain numbers of specific things by exploration, or defeat opponents. Escort quests ask players to protect a designate NPC to safely move around different game zones. The exciting part of this quest type is the conflicts which arise during the journey. The main purpose of goodwill quests is to enhance social interactions among players with different experience levels. Messenger quests are simply designed to let players interact with the game-controlled objects, such as NPCs, and learn the necessary information in order to complete quests. The present study adopts Dickey’s quest category to filter appropriate quest types for the experiment need. 2.3 Knowledge Acquisition Ever since Papert’s Logo [19] successfully encouraged meaningful knowledge creation through hands-on activities, computer games have been proposed by scholars [5, 20] as a possible constructivist tool to facilitate knowledge formation. Dickey [4] noted that quest types might affect different types of knowledge acquisitions in an online role-playing game environment. For instance, collection quests can help to strengthen players’ declarative knowledge of a game; bounty quests and escort quests set players in a more complicated situation in which they have to explore new territories, applying knowledge of game rules and concepts, and framing tactics to fight against enemies. Both types of quests highly demand procedural knowledge.
Fig. 1. Study conceptual model
356
J. Lee and C.-Y. Chao
Declarative knowledge and procedural knowledge are termed respectively by Jonassen, Bessner, and Yacci’s designations [10]: knowing that and knowing how. The knowledge types knowing that and knowing how are sometimes also correspondingly termed schema and schemata [21]. Essential to a schema [3] for an entity or an event is a collection of information or discrete concepts which help us to identify that entity or event. Schemata are packets of schema that are composed of the interrelationships among schemas. In other words, declarative knowledge is what a player receives the information for a set of facts relevant to the execution of a particular skill during a quest. These facts represent general procedures to generate behaviors. They are interpreted and stored in the player’s declarative memory in the form of statements. Procedural knowledge is acquired after the factual knowledge has been translated into a series of procedures that can be applied automatically without other interpretive activities; tuning steps follow to help players apply this knowledge more appropriately. In sum, this study mainly tries to explore how players’ gaming motivations and quest types might affect their knowledge acquisitions in terms of declarative knowledge and procedural knowledge within an online role-playing game environment.
3 Methodology 3.1 Research Questions Three research questions were generated accordingly to test out study hypotheses. Q1. How might the player’s gaming motivations affect his/her knowledge acquisition in terms of declarative knowledge and procedural knowledge? Q2. How might the game quest types affect the player’s knowledge acquisition in terms of declarative knowledge and procedural knowledge? Q3. How might the interaction effects between the player’s gaming motivations and game quest types affect player’s knowledge acquisition in terms of declarative knowledge and procedural knowledge? 3.2 Study Design The experiment environment, Fulade Online (http://cb.fulade.com.tw), was chosen to be based on two reasons. First, one of the primary goals of the present study was to find out how different quest types and the gaming motivations might affect players’ knowledge acquisitions. Thus, the content of this game environment should be as unfamiliar as possible to most players to increase the possibility of showing signs of change in subjects’ knowledge acquisitions. The author first excluded the top ten most popular online RPG games from the poll results (March 2007~March 2008) of Gamer (www.gamer.com.tw), the biggest online game social networking service (SNS) site in Taiwan, as well as Game world of Yahoo Taiwan. Three coders were invited to review three selected online RPG games. The intercoders reliability was calculated (Cronbach’s α = .90) which fits Fleiss’ [7] agreement level of “excellent”. Fulade Online was considered as the most appropriate experiment environment for its eight beginner-level quests closely fit with the operational definitions of Bounty-collection quest and Fed express quest by this study.
The Effects of Quest Types and Gaming Motivations on Players’ Knowledge Acquisitions 357
A total of 64 eligible subjects participated in this experiment. Upon arrival, subjects were given a Gaming Motivation Questionnaire adopted from Yee [27] to identify their gaming motivations in terms of achievement, social, and immersion. Then, the researcher randomly assigned subjects into the Bounty-collection quest group and the Fed express quest group. Each group included approximate numbers of participants. After completing the required 4 quests of each group, subjects moved on to answer the Declarative Knowledge Test developed based on the framework of Ju and Wagner [11] and the Procedural Knowledge Test developed based on the framework of McClure, Sonak and Suen [16]. The average time required to complete the experiment was two hours.
4 Results The study results show that subjects’ “immersion motivation” has the most significant influence on knowledge acquisition. With higher “immersion motivation”, players appear to acquire higher declarative knowledge and higher procedural knowledge. Meanwhile, interactions between “immersion motivation” and “social motivation” decisively affect players’ declarative knowledge. Interestingly, players with low “social motivation” and high “immersion motivation” hold better declarative knowledge than players with high “social motivation” and high “immersion motivation”. In addition, the study results are consistent with Ju and Wagner [11] suggestion in which the computer games are able to facilitate the knowledge construction through the problem-solving tasks such as game quests. In order to accomplish the assigned quests, players have to identify problematic issues, find available resources, explore new game spaces, and apply or adjust techniques as well as strategies. As a result, players can easily form the procedural knowledge through the process of problem solving. In turn, their procedural knowledge becomes the base from which they can continually strengthen and reinforce their declarative knowledge to the end of the game.
References 1. Aarseth, E.: From Hunt the Wumpus to Everquest: Introduction to Quest Theory. In: Kishino, F., Kitamura, Y., Kato, H., Nagata, N. (eds.) ICEC 2005. LNCS, vol. 3711, pp. 317–328. Springer, Heidelberg (2005) 2. Bartle, R.: Hearts, Clubs, Diamonds, Spades: Plaers Who Suit MUDs (1996), http://www.brandeis.edu/pubs/jove/HTML/v1/bartle.html (Retrieved November 20, 2007) 3. Bartlett, F.: Remembering: A Study in Experimental and Social Psychology. Cambridge University Press, Cambridge (1932) 4. Dickey, D.: Game Design and Learning: A Conjectural Analysis of How Massively Multiple Online Role-Playing Games(MMORPGs) Foster Intrinsic Motivation. Education Technology Research Development 55, 253–273 (2006) 5. Di Blas, N., Paolini, P., Poggi, C.: 3D Worlds for Edutainment: Educational, Relational and Organizational Principles. In: Pervasive Computing and Communications Workshops, Third IEEE International Conference (2005) 6. Eladhari, M., Lindley, C.: Story Construction and Expressive Agents in Virtual Game Wolrds. In: The Other Players conference, Center for Computer Games Research, IT University of Copenhagen, Denmark (2004) 7. Fleiss, J.L.: Statistical Methods for Rates and Proportions. John Wiley, New York (1981)
358
J. Lee and C.-Y. Chao
8. Greenfield, P.M.: Video Games and Cognitive Skills. In: Baughman, S.S., Clagett, P.D. (eds.) Video Games and Human Development: A Research Agenda for the 80’s, Cambridge (Mass.), Harvard Graduate School of Education, pp. 19–24 (1983) 9. Greenfield, P.M., Camaioni, L., Ercolani, P., Weiss, L., Lauber, B.A., Perucchini, P.: Cognitive Socialization by Computer Games in Two Cultures: Inductive Discovery or Mastery of An Iconic Code. Journal of Applied Developmental Psychology 15, 59–85 (1994) 10. Jonassen, D.H., Bessner, K., Yacci, M.: Structural Knowledge: Techniques for Representing, Conveying, and Acquiring Structural Knowledge. Erlbaum, Hillsdale (1993) 11. Ju, E., Wagner, C.: Personal Computer Adventure games: Their Structure, Principles, and Applicability for Training. ACM SIGMIS Database 28(2), 78–92 (1997) 12. Juul, J.: Half-Real: Video Games Between Real Rules and Fictional Worlds. The MIT Press, London (2005) 13. Kucklich, J.: Perspectives of Computer Game Philology. Game Studies 3(1) (2003), http://www.gamestudies.org/0301/kucklich/ (Retrieved December 11, 2007) 14. Malone, T.W.: What Makes Things Fun to Learn: Heuristics for Designing Instructional Computer Games. In: Proceedings of the 3rd ACM SIGSMALL. Symposium and the First SIGPC Symposium on Small Systems, pp. 162–169 (1980) 15. Loftus, G.R., Loftus, E.F.: Mind at Play: The Psychology of Video Games. Basic Books, New York (1983) 16. McClure, J.R., Sonak, B., Suen, H.K.: Concept Map Assessment of Classroom Learning: Reliability, Validity, and Logistical Practicality. Journal of Research in Science Teaching 36(4), 475–492 (1999) 17. Newell, A., Simon, H.: Human Problem Solving. Prentice-Hall, Englewood Cliffs (1972) 18. Nurmi, S., Lainema, T.: Problem-based Learning in The Business Context – Can Simulation Games Improve Problem Solving? In: Proceedings of the International Conference on Computers in Education (ICCE) 2004, Melbourne, pp. 227–235 (2004) 19. Papert, S.: Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, New York (1980) 20. Rieber, L.P.: Seriously Considering Play: Designing Interactive Learning Environments Based On the Blending of Microworlds, Simulations, and Games. Educational Technology Research & Development 44(2), 43–58 (1996) 21. Rumelhart, D.E., Ortony, A.: The Representation of Knowledge in Memory. In: Anderson, R.C., Spiro, R.J., Montague, W.E. (eds.) Schooling and the Acquisition of Knowledge, pp. 99–136. Erlbaum, Hillsdale (1977) 22. Sawyer, B.: Emergent Use of Interactive Games for Solving Problems Is Serious Effort. In: 2004 Game Developers Conference - The Serious Games Summit, San Jose, CA, March 22-23, 2004, pp. 22–23 (2004) 23. Tosca, S.: The Quest Problem in Computer Games. In: Proceedings of the First International Conference on Technologies for Interactive Digital Storytelling and Entertainment, Darmstadt, Germany (2003) 24. Tronstad, R.: Semiotic and Nonsemiotic MUD Performance. In: Proceedings of 1st Conference on Computational Semiotics for Games and New Media, Amsterdam, The Netherlands (2001) 25. Wibroe, M., Nygaard, K., Bøgh Andersen, P.: Games and Stories. In: Qvortrup, L. (ed.) Virtual Interaction. Springer, London (2001) 26. Yee, N.: Facets: 5 Motivation Factors for Why People Play MMORPG’s (2002), http://www.nickyee.com/facets/home.html (Retrieved December, 12, 2007) 27. Yee, N.: Motivations for Play in Online Games. CyberPsychology and Behavior 9(6), 772–775 (2006)
Self-movement Feeling Generation in Sports Watching with Screen Movement via Pan-Tilt Steerable Projector Hiroshi Noguchi1,2, Kei Yoshinaka2, Taketoshi Mori2, and Tomomasa Sato2 1
2
CREST, Japan Science and Technology Agency Graduate School of Information Science and Technology, Univesity of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656 Japan {noguchi,yoshinak,tmori,tomo}@ics.t.u-tokyo.ac.jp
Abstract. This paper describes system displays video frame of sports broadcasting with screen movement by an active projector. Since the screen moves as the camera operator controls their camera in the stadium by automatic detection of background movement from video frames, the watchers are force to move their heads in home environment. The head movement helps the watchers to feel self-movement as if they ware located in the stadium. The experiments demonstrate that the head movement when the watchers track the moving screen with their eyes and heads generates sense of self-movement and active feeling about players. Keywords: Active Projector, Sense of Self-Movement, Screen Movement, Sports Watching.
watching on television contains no interaction. The user only watches on television and does not contact to television. Therefore, control factors for perception are unavailable. Since the system does not contain no computer graphics image, realism factors is unrelated. The place on watching in home environments is usually fixed. Distraction factors are difficult to provide. Rather than these factors, generation of self-movement feeling is easy to realize. Examples of self-movements in sports watching are head movement and body twist. The spectators for such sports that need large play areas as football and basketball move their gazes and heads as the players and balls move in the large stadium. On the other hand, in watching on television, a camera operator moves their camera to keep the players and the balls center in the video frame. The spectators on television do not need to move their head and body. They cannot perceive self-movement. If the system can provide sense of self-movement to the watchers, they can feel presence on watching in the stadium. Our research aim is construction of display system that stimulates perception of self-movement in watching sports in television. Target video image is TV program about sports broadcasting. In video image, there are various scenes such as scene switching and zooming. Since our goal is generation of sense about self-movement, we treat with only tracking scene about the balls and the players including overall of field.
2 Generation of Self-movement Sense with Screen Movement There are two methods to generate movement of watcher’s head and body. One method is that the system moves the watcher's body directly in accordance with scenes on TV program. The other is that the system moves gaze area of the watcher. The former requires large devices for user movement. It is unsuitable for home use. We utilize the latter method. If the user watches TV program on large display, the user seems to move his/her head. However, as above mentioned, in TV program, the players and balls are usually on center in the frame. The user does not need to watch sports with head movement. For head movement, the system is required to move the gaze area. In watching television, gaze area is equivalent of screen area. The screen area of the display system is needed. The movement of gaze area also generates distraction factors for presence. The narrow gaze area and movement create attention of the watchers. The attention helps the users to feel involvement and immersion in watching. There are 3 types of approaches for screen movement. • Head Mount Display (HMD) One approach is that the system covers the user’s field of view with special display device such as HMD and the system provides screen with head position of the user. This method’s merit is that the system can generate screen movement of all directions and the degree of immersion is high. Cost and constriction of the special devices for the users are demerits. The measurement about head position is needed. It also forces the users to wear special sensors for head measurement. In watching sports, the spectators
Self-movement Feeling Generation in Sports Watching with Screen Movement
361
do not watch all directions but the front of them. The field of all directions is not needed. This approach is not suitable for home use. • Large Display Another approach is that the system equips large display such as 4K display and high-resolution projector. The system displays only clipped area of whole screen and moves the clipped area with video images. This method has the merits; easy implementation of screen movement, fast moving and resizing of screen. The method also contains the problems. The large display requires large area for equipment. The movable area on screen movement is limited to display size. • Active Projector The other method is the system physically moves screen of the display device. While the TV monitors are too heavy to move on the wall, the projectors are easy enough to move their bodies. The pan and tilt movement of the projector body realizes screen movement on large area. The movable projector has demerits; projected image is distorted, screen area tends to be small and speed of screen movement is limited. These methods contain merits and demerits. We utilize active projector for realization of screen movement. This is because 1) the active projector covers larger area than large display and 2) the screen includes no frame [2]. Once the user notices the frame of the display, the user finds out limits of screen movement. This prevents immersion and decreases degree of sense on presence. Display system without aware of screen frame is important. The several research projects [2][3][4][5] utilize the active projectors. The main usage of the projector is display of static image everywhere. In the addition to the disply of static image, the display system with screen movmenet is required to synchornize displaying the image and moving the screen. This is interesting topic about active projector.
3 Display System Implementation We have been utilized the pan-tilt streerable projector for informational support of the occupants in the room [6]. Performance of the projector is shown in Table 1. The projector consists of traditional projector, tilt motor and pan motor. The projector is controllable with DMX communication. Pan and tilt angles are measurable with equipped rotary encoders. The projector’s focus and zoom are also controllable. In usage of screen movement, change of distance between the projector and projection plane is small. In the paper, we do not control focus and zoom parameter. Configuration of the projection system is illustrated in Fig. 1. The display system projects video images by the following proceduce. Table 1. Specification of pan-tilt steerable projector Range (degree) Pan Tilt
-180 to 180 -120 to 120
Maximum Speed (degree/sec.) 80 80
Measurable Angle Resolution (degree) 0.09 0.05
362
H. Noguchi et al. video source background detector
display target pos.
pan and tilt control
calculated result
control data image distortor 画像補正部 pan and tilt control
image frame フレーム画像 テクスチャ化 texturization
target pos. image
pan and tilt
3D 次元 renderer レンダラ
target pos.
Offline phase
Online phase
Fig. 1. Display System Diagram
3.1 Detection of Image Background Movement Ideally speaking, position of moved screen is desired to be equal to the field of the spectator’s view in the stadium. However, the information related to position of the watching view has disappeared in video image of TV program. Fortunately, if switching of camera does not occur, the camera operators control their cameras to track the players, the ball or something to focus. It is assumed that this camera operation is similar to movement of eyeshot in watching sports on spectator’s seat. Therefore, screen control with the parameter of estimated camera movement from the video images is suitable to generate sense of presence in watching sports on the display system. Video images that track the players or balls in the playfield consist of large area of moving background and small area of tracking target objects. Background movement is generated with camera movement. Thus, camera operation is estimated from detection of background image movement. For detection of background movement, we utilize optical flow of the video images. The display system detects feature points in each image converted from video frame and calculates optical flows from Bouguet’s method [7],
Fig. 2. Detected Background Motion: White line means detected optical flows of feature points. Large allow in the bottom represents 5-times vector of detected background motion.
Self-movement Feeling Generation in Sports Watching with Screen Movement
363
which is the method that generates optical flow hierarchically from harsh image to high resolution image. Vectors of calculated optical flows are quantized in units of length and angle. The mode of the quantized vectors is regarded as camera movement (Fig. 2). 3.2 Planning of Projector Control Screen movement parameters are easily calculated from detected background movement. Displacement value of screen (dxb, dyb) in single video frame is calculated by given parameters: screen size (Wd, Hd), screen resolution (Wr, Hr) and movement of background (dxb, dyb). (1) The position at the video frame is calculated from integration of screen movement vectors. The needed size of screen movement in total video frames is also found. The system can calculate appropriate screen size from calculated total area of movement and projectable plane size. The maximum speed of projector pan-tilt is also found with distances of screen movements in frames. The system decides whether the video image can display by specification of the active projector. The system calculates pan and tilt value of the active projector in each frame from needed screen position above calculated. The active projector is regarded as a robot system that includes 2 degrees of rotation freedom and 1 degree of liner freedom, which is a kind of virtual liner joint between optical center and projection plane (Fig. 3). In the robot system, holomorphic tilt and pan values are found by inverse kinematics method.
Fig. 3. Freedom of the pan-tilt projector
3.3 Image Correction Projected image on plane is distorted as long as projection ray’s direction does not equal to normal direction of the plane. Distorted image decreases sense of presence. Mismatch between target screen position and calculated position arises from difference between the optical center of the projector and the center of rotation movement. The projectors contain the opposite characteristics of the cameras (Fig. 4). The camera calibration techniques resolve these problems. In fact, intrinsic parameters are calculated with the projected grid images and measured position. Because of non-linearity with offset between the optical center and the center of rotation, extrinsic parameters are difficult to calculate. We give the parameters from blueprint of the projector. The lens of the projector contains no deformation. Distortion calibration is not needed.
364
H. Noguchi et al.
Measurement in Pattern Projection at Known Postures
rendering plane camera
screen plane
Intrin sic Paramete r Calculation
rendering image
corrected image projection
Fig. 4. Projector Calibration and Image Correction
The system generates distorted images with 3D rendering technique. The system renders the 3D scene where optical center of the projector is virtual camera and the polygon whose texture is target image locates at the target position (Fig. 4). 3.4 Synchronization of Video Images, Projection Image Generation and Pan-Tilt Control In online projection phase, in order to watch without feeling of strangeness, accurate synchronization of video images, distortion image generation and pan-tilt control of the projector. There are several delays in the system; delay of projector control, delay of video image transmission and duration of computation for distortion. The transmitting delay from distorted video frames to the projector is the longest. The system makes delays of pan-tilt control in the time duration of transmission for synchronization. The values of controlled pan and tilt do not always match with target values because of motor performance, control noise and control delay. Before online generation of screen movement, the system controls the projector with calculated route of the screen and measures pan and tilt angles from the projector. The parameters for projection image distortion are decided from measured pan-tilt values. If the system detects mismatch between target pan-tilt value and measured value, the system modify the parameter at the image distortion phase. This mechanism covers as high speed as the projector cannot chase the target screen and absorbs vibration movement of pan-tilt.
4 Experiments about the Display System We evaluate that the display system make the watchers feel self-movement. Total 14 subjects watch the 3 kinds of videos. Experimental room design is shown in Fig. 5. The prepared videos are approx. 15 second-length videos about soccer, valley ball and video game. The screen size is equivalent of 50-inch display television. The video features are shown in Table 2. The video scenes of the soccer video are shown in Fig. 6. The video game is not a sport, but in the video game, background except the leading character moves. We assumed that the screen movement makes the watcher feel something except presence with video that includes background movement. In order to investigate relationship between head movement and sense of self-movement, the subjects located in 1, 2 and 2.3 [m] (directly below the projector) front of the screen. The experiment protocol is the following. Firstly, the subject watches the video with no movement of the screen. Secondly, he/she watches the same video with movement of the screen. After watching, the subject writes the relative score 1(bad) to 5 (good)
Self-movement Feeling Generation in Sports Watching with Screen Movement 2.325m
365
x
2.51m
Active Projector
1m
1m
Active Projector 1.9m(Height)
z
Projection Plane
: Watching Points
Fig. 5. Experimental Setup Table 2. Features of the video images in experiments Image Source Soccer Valley Ball Video Game
Movement Range (W x H [m]) 1.3 x 0.1 1.0 x 0.0 2.3 x 1.1
Maximum Speed in Width (m/s) 0.2 0.6 0.4
Maximum Speed in Height (m/s) 0.1 0.0 0.5
Fig. 6. Experiment Scenes in Soccer Video
against the video watching in no screen movement. The questionnaire items are 1) visibility of scenes, 2) self-movement feeling and 3) active feeling about target players and characters. Result about visibility of scenes is shown in Fig. 7. The subject scores are low rather than fixed screen. In the screen movement, it is difficult to watch the scenes. Scores of the video about valley ball that includes the highest speed screen movement are low. This means that fast movement decreases the degree of visibility.
366
H. Noguchi et al. relationship between videos and scores Valley Ball 12
12
10
10
6 4 2
Score
1
2
3
Score
4
4 3.5
8 6 4
3 2.5 2 1.5
2 0
5
4.5
Score
8
0
5
Video Game Number Of People
Number Of People
Number Of People
Soccer
1
1
2
3
Score
4
0.5
5
0
2.3m
Soccer
Valley Ball
Video Game
Fig. 7. Result about Visibility of Scenes
Result about self-movement feeling is illustrated in Fig. 8. As expected, the sense of self-movement is higher in screen movement rather than fixed screen. The scores are related to distances of subject’s locations. The subject in the location near the screen should move their head widely. The subject feels high degree of self-movement. This shows degree of head movement with screen movements are tightly related to sense of self-movement. 5
Video Game
Valley Ball
4 3.5
Score
Number Of People
Number Of People
Number Of People
Soccer
relationship between distances and scores
4.5
3 2.5 2 1.5 1 0.5
Score
Score
Score 2.3m
0
1m
2m
2.3m
Fig. 8. Result about Self-Movement Feeling
Result about active feeling about target players and characters is shown in Fig. 9.Active feeling is also higher in screen movement rather than fixed screen. On the contrary of the self-movement feeling, active feeling is not related to distances. This feeling arises from only screen movement without the subject’s head movement. This effect appears in video game. The implemented system has capability of increasing the degree of self-movement feeling and active feeling for something except sports. relationship between videos and scores Valley Ball
Score
Video Game
Score
Number Of People
Number Of People
Number Of People
Soccer
Score
Score 2.3m
Soccer
Valley Ball
Fig. 9. Result of Active feeling about Target Players and Characters
Video Game
Self-movement Feeling Generation in Sports Watching with Screen Movement
367
In all experiments, in the location directly below the projector, although the subjects watch the video at the similar view of the camera operator, the scores are not dramatically different from scores in the other locations. This means accuracy about gaze movement is not necessary for sense of self-movement and active feeling.
5 Conclusion We constructed the display system including the active projector moves the screen with video frames for the watchers to feel self-movement. The system displays the sports scenes of TV programs with screen movement by detection of background movement in video frames. The experiments demonstrate the system enhance the watchers’ perception of self-movement and active feeling about the players rather than televisions, whose displays are fixed. The system currently cannot treat with the video including zoom and switching of scenes. The system needs offline measurement about pan-tilt value of the projector. Resolution about these problems is a future task. The current system only generates sense of self-movement. It is so far from perception of presence. We will challenge to generate other factors about presence in watching sports.
References 1. Witmer, S., Singer, M.: Measuring presence in virtual environment: A prensece questionnaire. Presence: Teleoperation and Virtual Environment 7, 225–240 (1998) 2. Pinhanez, C., Podlaseck, M.: To frame or not to frame: The role and design of frameless display in ubiquitous applications. In: Beigl, M., Intille, S.S., Rekimoto, J., Tokuda, H. (eds.) UbiComp 2005. LNCS, vol. 3660, pp. 340–357. Springer, Heidelberg (2005) 3. Pinhanez, C.: The everywhere displays projector: A device to create ubiquitous graphical interfaces. In: Abowd, G.D., Brumitt, B., Shafer, S. (eds.) UbiComp 2001. LNCS, vol. 2201, pp. 315–332. Springer, Heidelberg (2001) 4. Yang, R., Welch, G.: Automatic and continuous projector display surface estimation using everyday imagery. In: WSCG, pp. 328–335 (2001) 5. Raskar, R., Baar, J., Beardsley, P., Willwacher, S., Rao, S., Forlines, C.: ilamps: Geometrically aware and selfconfiguring projectors. In: SIGGRAPH, pp. 809–818 (2003) 6. Noguchi, H., Mori, T., Sato, T.: Attentive Information Support with Massive Embedded Sensors in Room. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4551, pp. 883–892. Springer, Heidelberg (2007) 7. Bouguet, J.: Pyramidal implementation of the lucas kanade feature tracker: description of the algorithm. Technical report, OpenCV Document (2000)
Design of Interactive Emotional Sound Edutainment System Myunjin Park and Kyujung Kim School of Media, Soongsil Univeristy 1-1 Sangdo-1-dong, Dongjak-gu, Seoul, Korea {cooljin,kyu}@ssu.ac.kr
Abstract. This paper introduces an emotional sound edutainment system for children to learn basic musical composition called as musical education sound interactive game (MESIG) employing a new type of user interface. Developed interactive game interface provides children to enjoy the game, so that they learn how to compose musical notes with touching the tangible objectives instead of using ordinary input devices. This way on experiencing and playing the computer games has been evolved to use the body and hands’ movement so as to interact with the game in virtual environment, which brings out interest for the children and their learning capability becomes more effectively improved. This system introduced in this paper requires a single camera and carries out skin color model tracking function to detect hand gesture as input device for playing the game. This computer vision technique based on image processing makes possible to operate an expressive interactive musical education system. To exploit the effectiveness, evaluation and analysis works are accomplished upon the realization of sound edutainment game. Keywords: Interactive games, edutainment, skin color model, computer vision.
Design of Interactive Emotional Sound Edutainment System
369
The developments of interactive and participatory environments that combine the physical and virtual have brought as a natural continuation to the computer game industries. This development area not only restricts to entertainment domain but indirectly creates a learning environment for children. A critical review of examples of immersive virtual reality worlds created for children, with particular attention given to the role and nature of interactivity, is attempted. Interactivity is examined in relation to learning, play, narrative, and to characteristics inherent in virtual reality, such as immersion, presence, and the creation of illusion [3]. Usual virtual reality games try to make the user interface transparent or invisible, where user does not realize the existing of interface to immerse into the games. The conventional method, the player has to use special devices such as gloves or head-mounted displays [4][5]. This motivates us to study a simple manner of children to interact with interesting game contents in interactive digital environment without any bulky device. The interactive digital environment is referring as a system which adapts to the user’s action and allows varied degrees of freedom to control over the time or space [6]. The study of the user interaction in virtual environment based on activity theory framework [7] is analyzed and decision to design activity contents to generate larger motivating in game. In this paper, we introduced a tangible input to interact with the virtual environment. Human motion interacts with the virtual environment to gain the children attention and interest to play the games. This interactive games interface able to develop and gain children attention, interest, and learning how to compose simple musical notes throughout their gaming time experience. The proposed MESIG System is described in section 2 including the explanation of the details of how to track user’s body and hand movement and the process for generation and rendering snowflakes and musical notes object by using DirectX. In the last subsection of section 2, the way how to perform the collision detection is described. Section 3 shows the experimental results. The conclusion is summaries in section 4.
2 Proposed MESIG System The proposed MESIG system provides emotional playing activity game to user, such that appropriate sounds relevant to action of user’s body and hand and snow flake movements. The functions of the proposed system can be categorized into three actions. As the first, the image of snow falling is provided to the passengers drawing their attractions if there is no one in front of camera. As the second action, when the specific user stands at the camera, all the snowflakes turns their shape into musical notes having their own colors such that the user feels like to touch. As the last action, by tracking user movement, when the user’s hand touches the musical notes, the collided snow flake is disappeared and the corresponding sound classified by its color is emitted. The procedure for processing of this system can be summarized in Figure 1 as shown below. The overall system determines whether or not there is a person in front of screen. Towards this, the image captured by camera should be changed a binary image using a background subtraction method and recognized user’s body and hand using a
370
M. Park and K. Kim
Camera
Projector
All Object
Input Image Background Subtraction
Each Snow Object Coordinate Compute
Object Image Using Component Labeling and Threshold
Render Show Object
User Object Image
Each Object Projector Coordinate to Screen Coordinate
Camera Coordinate to Screen Coordinate
Each Snow Object Move
Each Note Object Move
Object Test Snow Object
Yes
Note Object
No Change Texture From Snow to Note
Collision Detection No Yes Output Sound
Delete Object
Fig. 1. Overall structure of the proposed system MESIG
component labeling method. As soon as the existence is confirmed, all of the snowflakes subsequently changed into musical notes. Then, utilizing the position information about regions associated with formerly extracted hand and face, the collision detection is performed if the user’s hand touches the projected musical notes. Whenever the collision is detected, corresponding musical note becomes disappeared with emitting the sound, at sequel, the new object is appeared. Surely, repeating this process provides the interactive environment to the user controlled by tracking and moving user’s hand, so that the snowflake is rendered at the instant of sensing the touch of snowflake with the help of DirectX engine. 2.1 How to Track User’s Movement Since the user described as a kind of interface, when the user touches snow flake, its shape turns out to be musical notes, and relevant sound is generated. Here the importance is how to track the user movement. Towards this, this system uses background subtraction technique in order to segment out objects of interest from captured image in real-time. Objects are generated using background subtraction and shadow detection based on color difference [8]. From the objects images, we can change from snow flake objects to musical nodes and the musical is disappeared. We used threshold value of RGB
Design of Interactive Emotional Sound Edutainment System
371
color to subtract the background and foreground images. Figure 2 shows the silhouettes pixels extraction based on RGB color. The object of interest generation uses two methods. First, we use the distance in order to separate an object pixel and a background pixel, and the relevant equation for the distance is the following;
d (i, j ) = cr (i, j ) − cb (i, j )
(1)
where c represents the norm of a vector c. Second, we use difference to separate shadow can be expressed by (2) where • is the dot product operator.
distance Background Image C (R , G , B ) b
b
b
Run-time Image C (R , G , B ) r
r
r
r
b
difference
Fig. 2. Overview of objects pixels extraction of (i,j)th pixel color
We define a background image whose (i,j)th pixel color is denoted by the vector Cb(i,j) or Cb(Rb, Gb, Bb), and the run-time image whose (i,j)th pixel color is denoted by the vector Cr(i,j) or Cr(Rr, Gr, Br). The distance is computed by using intensity difference between the run-time and background pixels. If the intensity difference is very large, then the pixel should be a silhouette pixel. If the intensity difference is very small, then the pixel must be a background pixel. Shadow pixel is determined by performing pixel determination to exclude silhouette and background pixel as described above. The value difference is the angle between the vectors Cr and Cb in the RGB color domain and hence is a measure of the color difference between the run-time and background pixels in Figure 3. After background subtraction, the extracted object is shown in Figure 3. The noise could be appeared due to musical notes and snowflakes when real-time captured image is extracted as shown in Figure 3(b)-(c) based on background subtraction. In order to discriminate the noise and the person, each component is classified into the pure noise and the person by comparing those with the threshold value. Here, it is easy to determine whether the component is
372
M. Park and K. Kim
(a) Background Image
(d) Object Image A
(b) Capture Image A
(c) Capture Image B
(e) Object Image B
(f) Result of Labeling
Fig. 3. MESIG of user tracking
the noise or the person since the component corresponding to the person has more large number of pixels. In the proposed system, the threshold is fixed to 1200, so that the component is recognized into the person if the number of pixels is more than 1200, otherwise the noise. Figure 3(f) is the result after the discrimination and the suppression of noise from the subtraction image shown in Figure 3(e), and the shaded component is recognized as the person. 2.2 How to Perform Generation and Rendering Snow Flake Object Using DirectX This section utilizes DirectX as tool for generating realistic snowflakes or musical notes. It is well-known that DirectX provides comfort API to control the functions of high performed hardware. Thus, more powerful multimedia applications can be easily developed. In order to project realistic snowflakes or musical notes to the user, the eight steps are processed as shown in Figure 4 by using DirectX. First, the user sets up the total number of snowflakes. If the number of snowflakes is not enough as determined, residual snowflakes are generated by random selection of the number of snowflakes in random. After performing this step, as the second step, all the snowflakes has its own projection coordinate, and its magnitude and moving path are determined one by one. Each snow flake object follows its own moving path. As the third step, the presence of the user is testified. At fourth step, all the snowflakes turn their shapes into musical notes. Then, the collision detection is processed whether there is any touch between hand or face and musical notes in fifth step. If there is any collision, the corresponding musical note emits a sound and a color in sixth step. Here, each musical note can have 7 colors or sounds. The collided
Design of Interactive Emotional Sound Edutainment System
1
373
Initial Object Set 8
2
Moving Objects
3
User Object Test
4
Change Texture From Snow to Note
5
Collision Test
Moving Objects
Elimination Snow Object
7
Output Sound
6
Collision
No Collision
Fig. 4. Processing steps for the proposed MESIG system
musical note becomes disappeared in seventh step, whereas the object without collision keeps moving its position in eighth step. After the sound note is eliminated, the number of snowflakes will be decreased by the number of the disappeared. These 1 to 8 steps are repeated until all the snowflakes are disappeared. Figure 4 shows these processing steps for generation, movement, change and elimination of snowflakes. 2.3 How to Perform Collision Detection It is important to detect the event that snowflakes collide with user movement. Since the user performs interaction by using body or hand movement, this interface can provides very interesting situation. In order to perform the collision detection, the functions for tracking the user’s hand or face explained in Sec. 3 and the localization of projected snow flake explained in Sec. 4 should be performed. However each snow flake has its own coordinate, it hard to perform the collision detection [8] by only using the location of projected snowflakes. In other words, the coordinate oriented from the camera tracking the user’s hand or face is totally different from that from projector for snowflakes, it is necessary to use the common coordinate. This common coordinate can be updated by transforming the coordinate of projector into that of screen. After all, the collision detection is performed by this coordinate transformation. Since the coordinate of screen can be calculated by the projected region, it can easily test whether the collision happens or not. To make these coordinate changes, the keystone correction technique can be generally utilized for providing the corrected image wherever the image is projected from the projector with considering relative positions between the camera and the projector. Here, in order to detect the collision, the following equations (3) and (4) are used to transform the coordinates of the camera and the projector into the common screen coordinate. First of all, the relationship between the coordinate of projector and that of camera is designated as in LHS of equation (3) with whose corresponding transformation matrix T . And secondary, the relationship between the coordinate of
374
M. Park and K. Kim
screen and that of camera is defined as in RHS of equation (4) by using C . Using these relations, the relationship between the coordinate of screen and that of projector can be expressed in equation (4), whose relevant transform matrix defined as P
⎛ x proj ⎞ ⎛ xworld ⎞ ⎜ ⎟ ⎜ ⎟ C × T × ⎜ y proj ⎟ ≡ ⎜ yworld ⎟ , P = C −1 × T ⎜ 1 ⎟ ⎜ 1 ⎟ ⎝ ⎠ ⎝ ⎠
(4)
−1
Utilizing equations (3) and (4), the coordinate change from camera to screen can be accomplished, and Figure 4 visually depicts these coordinate transformations step-bystep. From the results of these transformations, the collision detection is performed by matching the coordinate of snow flake with that of either user’s hand or face.
3 Experimental Results and Analysis In this section, we present result of the proposed game MESIG. The proposed game required a camera (LG – color CCD camera), a projector, screen and a personal computer (Intel® Xeon™ CPU 3.20GHz, 2.56GB RAM, Windows XP OS) to execute the MESIG. The display screen will be change when the real-time camera of the system detect human or hand skin color. Then the musical notes display will replace the snowflakes display. Figure 5 (a) shows the overall system setup of MESIG which is projected to a wide screen; (b) shows the initial display of snowflakes when no player detected in real-time camera. The display screen environment will change from snowflakes to musical notes when a player face or hand skin color is detected. The musical notes have 7 basic elements of sound notes which are represented in rainbow colors. When player hand touches on a particular note, the sound notes will be generated. Figure 6 shows the result of MESIG with different players captured from the normal digital camera. The system shows the interactive game environment by detect the player skin color. When the player’s hand touch on the musical notes, a sound will be generated based on the notes representation. This MESIG will motivate the children interest and creativity to compose musical notes. The sound notes only response to one time hand touch of the musical notes.1 This analysis is accomplished for 30 children as the examination group whose age ranges from 4 to 7. The pre-exam is conducted before the children play, then after the post-exam is executed to analyze the degree of effectiveness. In order to determine how the educational interest and achievement is improved, the basic questionnaires are assigned to the children, which are including the preference and the skill about 1
www.cooljin.net/conf08/iadis_elearing
Design of Interactive Emotional Sound Edutainment System
(a) Overall system
375
(b) Initial display of snow flakes
Fig. 5. Proposed MESIG system environment
(a) Initial display of snow flakes
(b) Player near to the screen
(c) Player1 touch the notes
(d) Player2 touch the notes
Fig. 6. Experimental result shows the player plays the MESIG using hand to touch the musical notes to generate a sound notes
computer games. For fair evaluation, the corresponding questionnaires are selected from the pool made by the people who have the major on child education. Experimental results are subsequently analyzed by conducting SPSS program. Among those, firstly, the sound educational achievement is exploited by evaluating T-value a priori. At second, after the end of executing game, to concrete the inference a priori made, the posteriori T-value evaluation process is conducted at sequel. Significance level of all the experiments is fixed, whose corresponding value is less than 0.05. Table 1 shows the results of evaluation before and after play of the sound edutainment game.
376
M. Park and K. Kim Table 1. Level of interest for sound edutainment game
Table 2. Achievement of sound perception education Subject
Evaluation
Sound
preexamination postexamination
perception
Num. of Average Student (N)
Standard Deviation
30
11.47
3.42
30
13.84
2.65
T-value
P-value
5.567
0.01
As shown in Table 1, before the game play, the level of interest is not so high, but after the game play the level of interest is positively improved. In Table 1, among examination group, it can conclude that the number of children having high level of interest is increased from 27% up to 43%. Besides the increment of interest, the capability of sound perception is also enhanced. The level of achievement corresponding to the children in examination group is improved by 2.37 points. According to t=5.567 and p=0.01. From the statistical point of view, the results in Table 2 can be said to be meaningful because p < 0.05. And the results show that the sound perception education via the interactive game is quite effective to the children.
4 Conclusion This paper introduced an interactive game for children to learn basic musical composition. The proposed system is an interactive game named as musical education sound interactive game (MESIG), which can be treated as a new type of user interface. Developed system is designated an interactive game interface for children to enjoy and play the games while learning how to compose musical notes using tangible input instead of ordinary input devices. Thus, the proposed MESIG system provides emotional playing activity game to user, such that appropriate sounds corresponding to action of user’s hand and snow flake movements. The functions of the proposed system can be categorized into three actions. To verify the effectiveness of the proposed game for sound perception edutainment, the fair statistical analysis is conducted. All the results show the improvement of sound perception capability as well as the increment of interest. This indicates that the interactive game controlled by body-motion can strongly motivate the reason for learning, so that this new way of
Design of Interactive Emotional Sound Edutainment System
377
education via edutainment in a form of interactive game increases the level of learning interest and improves the capability of sound perception more effectively. Acknowledgments. This work was supported by Soongsil Research Fund.
References 1. Madej, K.: Towards digital narrative for children: from education to entertainment, a historical perspective. Comp. in Entertainment (CIE) 1(1) (2003) 2. Stromberg, H., et al.: A group game played in interactive virtual space: design and evaluation. Symposium on Designing Interactive Systems (2002) 3. Roussou, M.: Learning by doing and learning through play: an exploration of interactivity in virtual environments for children. Computer in Entertainment (CIE) 2(1) (2004) 4. Bobick, A., et al.: Perceptual user interfaces: The KidsRoom. Communication, ACM 43(3), 60–61 (2000) 5. Pausch, R., et al.: First Steps Toward Storytelling in Virtual Reality. In: Computer Graphics Proceedings, ACM SIGGRAPH. Annual Conference Series, pp. 193–203 (1996) 6. Talin: Real interactivity in interactive entertainment. In: Dodsworth Jr., C. (ed.) Digital Illusion: Entertaining the Future with High Technology. Addison-Wesley, Reading (1998) 7. Nardi, B.A.: Context and Consciousness. In: Activity Theory and Human-Computer Interaction. MIT Press, Cambridge (1996) 8. Kong, G., et al.: A real time system for robust 3d voxel reconstruction of human motion. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition., vol. 2, pp. 714–720 (2000) 9. Sukthankar, R., et al.: Smarter Presentations. Exploiting Homogra-phy in Camera-Projector Systems. In: Proceedings of International Conference on Computer Vision, vol. 1, pp. 247–253 (2001)
Understanding Online Game Addiction: Connection between Presence and Flow SungBok Park1 and Ha Sung Hwang2,*
*
1 Hanyang University, visiting professor, Graduate School of Journalism and Mass Communication 17 Haengdang-dong, Seongdong-gu, Seoul, South Korea [email protected] 2 Dongguk University, assistant professor, dept. of communication studies 26, 3 Phil-dong, Chung-gu, Seoul, South Korea [email protected]
Abstract. Addictive behavior in online gaming has been an important research topic since it has been one of the most popular activities in entertaining for younger people in Korea. However, despite the growing popularity of online games, empirical studies about the effects of immersion to the online game behavior are relatively rare. By applying two psychological concepts –presence and flow-the present study investigates how different types of immersion affect on online game addiction. Results show that both presence and flow play significant roles in online game addiction, however, flow mediates the relationship between presence and online game addiction. Based on these findings, implications and suggestions for future studies are discussed. Keywords: Presence, Flow, Online Game, Addiction, Virtual Reality.
Understanding Online Game Addiction: Connection between Presence and Flow
379
studies. Hence, the present studies argue that immersion into online games is after all a result of experiences of presence and this feeling of presence may be correlated to addictive behavior in online games. On the other hand, Choi and Kim [2] found that people continue to play online games if they have optimal experience commonly know as the concept of flow. In addition, previous research found that flow state positively related with the addiction to online game. Some scholars have insisted that presence can be defined as a special type of flow experience that occurs during teleoperations. For example, Fontaine [3] states that flow experience produces peaks of involvement that seem to be similar to the vividness of presence. Although previous research suggested that the two concepts-presence and flow—share similarities such as the immersive component and intensive feelings of involvement, it is not well understood relationships between these two concepts and how experiences of presence and flow are related to online games addictions. Therefore, this study aims to examine whether presence and flow are related in the context of online game. Moreover, the study explores the impact of the experience of presence and flow on online game entertainment and addiction.
2 Theoretical Background: Social Presence 2.1 Presence and Online Game The concept of presence has become central to theorizing about the advanced humancomputer interaction such as virtual reality systems, as well as traditional media such as television, film and books. Researchers have begun to realize that the feeling of presence is at the heart of all mediated communication environments, because presence is at the heart of human’s desire to use media to move beyond the limits of body and sensory channels[4]. After an extensive review of presence-related concepts and their explications, Lombard and his colleagues define presence as “the perceptional illusion of non-mediation” [5,6]. This illusion of nonmediation refers to a phenomenon in which “…a person fails to perceive or acknowledge that the existence of a medium in his or her communication environment and responds as he or she would if the medium were not there.” Therefore, presence describes a state of consciousness that gives the impression of being physically present in a mediated world. Commonly, presence is defined as the sensation of ‘being there’ in a mediated environment Since presence is primarily a subjective sensation, it has been argued that ‘subjective report is the essential basic measurement’. Indeed, the majority of studies measure presence through post-test questionnaires and rating scales, which have the advantage that they do not disrupt the media experience and are easy to administer. For instance, Slater et al [7] asked participants, after exposing them to a Virtual reality systems, to answer three questions on Likert scales (1-7) that served as an indication of presence: (1) their sense of ‘being there’ in the computer-generated world, (2) the extent to which there were times when the experience of the computergenerated world became the dominant reality for the participants, thereby forgetting about the ‘real world’ outside, and (3) whether they remembered the computergenerated world as ‘something they had seen’ or as ‘somewhere they had visited’.
380
S. Park and H.S. Hwang
According to Slater et al.[7], the ‘experiencing-as-a-place’ is central to understanding presence in Virtual environments: people are there, they respond to what is there and they remember it as a place. As such, presence has received substantial attention from the virtual reality programs, it is becoming increasingly relevant to online game environment. Presence has been recently been identified as a potentially important variable in online game research because it may affect use and a variety of outcomes of exposure, ranging from enjoyment to aggression. Although few studies have examined the relationship between the experience of presence and online game use, Tambrini et al. [8] found that playing a game created a strong sense of presence than observing a game, presumably due to the additional interactivity. Even though many technological features of online games are expected to contribute to the sense of “being there”, Bracken and Skalski [9] suggest that image quality impact both the level and types of presence dimensions experienced by video game players. In addition, previous studies provide compelling evidence for the mediating effect presence in the context of e-commerce [10], human-robot interaction [11] and entertainment games [12]. Especially in the context of entertainment games, the enjoyment and evaluation of video games are mediated by feelings of presence during game playing [12,13]. Consequently, the above research findings indicate that the experience of online games relatively complies with the psychological state of “being there” in a mediated virtual environment. Following this logic, are game users who are more immersed into online games more likely to become addicted? Few studies, however, concern with this topic, the investigation is worthwhile of intense exploration. 2.2 Flow and Online Game Flow theory can be referred to as the ‘psychology of optimal experience,’ which in recent years has been applied to the Internet behavior by some research. Studies suggest that cyberspace behavior has been reported as highly correlated with flow experience. As in his early definition, Csikszentmihalyi [14] sees "the holistic experience that people feel when they act with total involvement" (p.36) Further, he states (p.3) that flow experience are those optimal and enjoyable experience in which we feel “in control of our actions, masters of our own fate…we feel a sense of exhilaration, a deep sense of enjoyment” [15]. As a result, flow experience is more an emotional state during the process of the user’s activity. Csikszentmihalyi [14] originally identified four flow components: control, attention, curiosity, and intrinsic interest. To supplement Caikszentmihalyi's concept of flow experience, Trevino and Webster characterized four dimensions of flow [16]. According to them, within the human-computer interaction experience, flow incorporates the extent to which (1) the user perceives a sense of control over the computer interaction, (2) the user perceives that his or her attention is focused on the interaction, (3) the user’s curiosity is aroused during the interaction, and (4) the user finds the interaction intrinsically interesting. In Hoffman and Novak [17] flow is defined in terms of the experience of flow (intrinsic enjoyment, loss of self-consciousness), behavioral properties of the flow activity (seamless sequence of responses facilitated by interactivity with the
Understanding Online Game Addiction: Connection between Presence and Flow
381
computer and self-reinforcement), and its antecedents (skill/challenge balance, focused attention, and telepresence). Drawing upon this suggestion, Ghani and Deshpande [18] further argue that two important components-enjoyment and concentration- leads to diminished sense of time during the particular activity. It seems that while in the flow state, a user experiences a sense of happiness, accompanied by a feeling of an exploratory distortion in time perception, which often occurs in the absence of time pressure when conducting a specific activity that brings in positive feedback. As a result, the time allocation for the activity increases. However, similar to the distortion in time perception, during activities users may experience another perception--distortion of the sense of space. Such perception is related to the concept of presence as Heeter [19] uses the term "(tele) presence" to describe the sense of presence when an individual is physically far away from the scene. Indeed, some scholars insist that flow and presence shares similarities because these are terms to describe immersive component and intense feeling of involvement [3]. It is believed that flow experience produces peak of involvement that seem to be similar to the vividness of presence. Similar to presence, some studies argue that flow is significantly related to online game behavior since in online games, continuous scoring, promotion, immediate feedback, and achievement of self satisfaction have become the channels for upgrading individual self-esteem of the Internet users. Therefore, the experience described by flow state such as clear objective and immediate feedback, challenge encounter and adequate skill, combination, sense of control, curiosity, loss of self consciousness, purposeful experience, and inner interests are the states which can be experienced and accomplished by online games. Indeed, in study of Taiwan college students’ Internet behavior, Hwang [20] found that the college students actually experience flow state when using the Internet. Also Choi and Kim [2] found that people continue to play online games if they have optimal experience and flow state had an impact on user’s loyalty. In addition, motivations for online gaming influence users’ flow state and online game addiction. Kim and Park [21] identified seven needs for online game in Korean generation: escaping, leisure, community, character/compensation, satisfaction, entertainment, and pass time. Among these factors, they also found that the need for entertainment has an effect on immersion, while the need for community has a significant impact on addictive behavior in online game. In sum, evidence from flow studies has shown that enjoyment is of key value to generate optimal flow. Though the state of flow is temporal and highly subjective, it is suspected that people who enjoy flow experience during an activity may develop a tendency to repeat the activity and repetition of a particular activity may eventually develop into a tendency toward addiction. Therefore, flow experiences may play a key role in activating addiction through repetition of online game activities.
3 Research Questions Based on the claims and findings examined above, this study explores whether presence and flow are related in the context of online game especially focusing on the
382
S. Park and H.S. Hwang
relationships with entertainment and moreover, explores the impact of the experience of presence and flow on online game entertainment and addiction. Therefore, the following research questions are proposed. RQ1. How the experience of flow and a sense of presence are related online game entertainment? RQ2. How the experience of flow and presence predict online game addiction?
4 Method 4.1 Participants The Participants for this study were Korean college students. The total number of the sample was 133, with 78 males and 55 females. The average participant was 22 years old and the distribution of academic level was freshman (43.3%, N=57), sophomore (24.1%, N=31), junior (21.1%, N=27), and senior (8.3%, N=11). 4.2 Measures The survey questionnaires were distributed to participants and measure includes presence, flow, entertainment, and addictive behavior. Presence. Presence was measured with presence scale by selection from previous studies [5,6,8,22]. The scale represents three dimensions: arrival (the feeling of being present in a mediated environment, a sense of ‘being there’ in the virtual environment), departure (the feeling of no longer being present in the physical environment, my body was in this room, but my mind was inside the world created by online game), and perceived realism (extent to which the virtual environment became the dominant reality, extent to which the virtual world seems to be real). Six items were used and responses were scored on 5-point scales ranging from strongly disagree to strongly agree. The presence index was reliable (Cronbach’s alpha=.89). Flow. Participants’ flow state while playing online game was measured by the scale developed by Choi et al.[23]. The measurement includes 5 items such as attention of focus, sense of potential control, loss of self-consciousness, sense of ecstasy, and time distortion. Participants received that definition of flow with a short description at the beginning of questionnaire. Responses were scored on 5-point scales ranging from strongly disagree to strongly agree. The flow index was reliable (Cronbach’s alpha=.90). Online Game Addiction. Online game addiction was measured with the scale developed by Kim and Park [21]. This measurement includes 5 items to test addictive online game behavior for Korean people [21]. The items include ‘how often do you fear that life without the online game would be boring, empty, and joyless?, ‘how often do you try to cut down the amount of time you spend online game and fail?, ‘how often do you find yourself anticipating when you will go on-line again?, ‘how often do your grades or school work suffer because of the amount of time you spend online game?’, ‘how often do you lose sleep due to late-night gaming?’ Responses were scored on 5-point scales ranging from strongly disagree to strongly agree. The index was reliable (Cronbach’s alpha=.88).
Understanding Online Game Addiction: Connection between Presence and Flow
383
Entertainment. Online game entertainment was measured with two items. These are ‘how much you enjoy the game?’, ‘To what extent the online game entertains you? Responses were scored on 5-point scales ranging from not at all to very much. 4.3 Analysis Person correlations were computed for the relationships among presence, flow, and entertainment. Then regression was conducted to see the predictors of online game addiction. To investigate further how these two concepts--presence and flow--predict online game addiction, a path analysis was conducted using Amos.
5 Results First, to examine the relationships among presence, flow, and entertainment Pearson correlations were conducted. The findings suggest that presence(r=.18, p<.05) and flow (r=.13, p<.05). were significantly correlated with online game entertainment. However the findings indicated that presence has a more positive relationship with online game entertainment than flow does. Korean college student in the sample reported that the more they experience of being there in the online game environment, they feel more that the online game entertained and made them enjoyable. Next, to investigate the predictive power of presence and flow in online game addiction, multiple regressions were conducted. Due to multicollinearity, stepwise regression was used to identify that best predictor among the independent variables. The result indicates that both presence (β=.23, p<.01) and flow (β=.19, p<.01) had an impact on online game addiction (see table 1). This means that the more online game users feel sense of presence and experience of optimal state, the more they are addicted to online gaming. To investigate further how the concept of presence and flow predict online game addiction, a path analysis was conducted using Amos. The result shows that flow mediates the relationship between presence end online game addiction (see figure 1). The model fit that data. (df=1, p=.11, GFI=.98, RMSEA=.07). This result indicates that feeling of presence facilitates the occurrence of flow and flow in turn seems to enhance online game addiction. When analyzing the data in another way, the result shows that flow does mediate the relationship between presence and online game addiction. Table 1. Stepwise Regression: Predictors of Online game addiction Model
Fig. 1. Path model off relationships among presence and flow and online game addiction
6 Discussion The present study aims to examine whether presence and flow are related in the context of online game and moreover, explores the impact of the experience of presence and flow on online game entertainment and addiction. Within our investigation, presence and flow are correlated in relation to online game addiction, and flow mediates the connection between presence and online game addiction. Thus, the experience of being there in which immersed in a virtual reality facilitates the occurrence of mental state operation in which a person is highly involved in playing gaming. What following is a discussion of the implications of the present study with respect to online game activity in virtual environments. First, this study provides empirical evidence supporting the anticipated powerful effect of experience of flow in online games. More importantly, it was found that online game players experience high level of flow state and their addiction to online games can be explained by the flow experience. This finding seems to be consistent with previous research [9] and thus, the study confirms that flow might play an important role of addiction to online games. Second, the present study shows that there are significant mediating effects of presence between flow and online game addiction. Having experience of sense of presence applies significantly to the experience of flow and addictive behavior to online game activity. Theoretically, this result is in line with previous literature about the mediating effects model of presence in human-computer interaction especially, with respect to entertainment games. These findings, however, call for further investigation to elaborate the mediating role of presence in the context of online games. As a final remark, we would like to provide some suggestions for future studies. A limitation of the present study is that this was conducted with Korean college students as the convenience sample, thus future studies need to examine the effects of presence and flow on online game addiction with more diverse topics in other countries in order to validate and corroborate the present findings. Another interesting extension of the present study would be to examine individual difference in terms of feeling of presence and flow in relation to online game addiction. Furthermore, since this study merely employed the survey research, it might not reveal the authentic Internet situation. In the future, researchers could conduct field studies or field experiments to examine the finding obtained in this research.
Understanding Online Game Addiction: Connection between Presence and Flow
385
References 1. Korea Game Industry Agency: 2007 Korean games white book. Korea Game Industry Agent: South Korea (2007) 2. Choi, D., Kim, J.: Why people continue to play online games: In search of critical design factors to increase customers loyalty to online contents. Cyberpsychology & Behavior 7, 11–24 (2004) 3. Fontaine, G.: The experience of a sense of presence in intercultural and international encounters. Presence: Teleoperators and Virtual Environment 1(4) (1993) 4. Biocca, F., Kim, T., Levy, M.R.: The vision of virtual reality. LEA’s communication series, p. 401. Lawrence Erlbaum Associates Inc., Hillsdale (1995) 5. Lombard, M., Ditton, T.B.: At the Heart of it all: The conpcet of presence. Journal of Computer-Mediated Communication 3(2) (1997), http://www.ascusc.org.jcmc/vo13/issue2/ 6. Lombard, M., Reich, R.D., Grabe, M.E., Bracken, C., Ditton, T.B.: Presence and Television: The role of screen size. Human Communication Research 26(1) (2000) 7. Slater, M., Usoh, M., Steed, A.: Depth of presence in virtual environments. Presence: Teleoperators and Virtual Environment 3, 130–144 (1994) 8. Tamborini, R., Eastin, M., Skalski, P., Lachlan, K., Fediuk, T., Brady, R.: Violent virtual video games. Journal of Broadcasting and Electronic Media 48(3), 335–357 (2004) 9. Bracken, C., Skalski, P.: Presence and video games: The impact of image quality and skill level. In: The 8th Annual International Workshop on Presence, Clevland, Ohio, USA, August 24-26 (2006) 10. Friedman, T.: Making sense of software: Computer games and interactive texuality. In: Jones, S.G. (ed.) Cybersociety: Computer-mediated communication and community, pp. 73–89. Sage, London (1995) 11. Lee, K.M., Park, H.: Can a robot be perceived as a developing creature? Effects of robot’s long term cognitive developments on its social presence and people’s social responses toward it. Human Communication Research 31(4), 538–563 (2005) 12. Lee, K.M., Jin, S., Park, S.: Kang.: Effects of narrative on feelings of presence in computer/video games. In: The Annual Conference of the International Communication Association, New York, USA (May 2005) 13. Lee, K.M., Peng, W., Jin, S., Yan: Can robots manifest personality? An empirical test of personality recognition, social response, and social presence in human-robot interaction. Journal of Communication 56, 722–754 (2006) 14. Csilkszentmihalyi, M.: Beyond Boredom and Anxiety. Jossey-Bass., Sanfrancisco (1975) 15. Csilkszentmihalyi, M.: The psychology of optimal experience. Harper& Row, New York (1990) 16. Trevino, L.K., Webster, J.: Flow in computer-mediated communication: Electronic mail and voice mail evaluation and impacts. Communication Research 19, 539–573 (1992) 17. Hoffman, D.L., Novak, T.P.: Marketing in Hypermedia Computer-Mediated Environments: Conceptual Foundations. Journal of Marketing 60, 50–68 (1996) 18. Ghani, J.A., Deshpande, S.P.: Task characteristics and the experience of optimal flow in human-computer interaction. Journal of Psychology, 381–391 (1994) 19. Heeter, C.: Being There: The Subjective experience of presence. Presence: Teleoperators and Virtual Environments 1(2), 262–271 (1992) 20. Hwang, C.: Internet usage of Taiwan’s college students: The flow theory perspective. Master’s thesis. National Chicago Tung University, Taiwan (2000)
386
S. Park and H.S. Hwang
21. Kim, Y., Park, S.: A Study on the online game use influences in game flow and addiction: Focusing on the uses and gratifications approach. Korean Journal of Journalism and Communication Studies 50(5), 355–377 (2007) 22. Kim, T., Biocca, F.: Telepresence via television: Two dimensions of telepresence may have different connections to memory and persuasion. Journal of Computer Mediated Communication 3(2) (1997) 23. Choi, D., Kim, H., Kim, J.: A cognitive and emotional strategy for computer game design. Journal of MIS Research 10, 165–187 (2000)
The Experience of Presence in 3D Web Environment: An Analysis of Korean Second Life SungBok Park1, Ha Sung Hwang2,∗, and Myungil Choi3 1
Hanyang University, visiting professor, Graduate School of Journalism & Mass Communication 17 Haengdang-dong, Seongdong-gu, Seoul, South Korea [email protected] 2 Dongguk University, assistant professor, Dept. of Communication Studies 26, 3 Phil-dong, Chung-gu, Seoul, South Korea [email protected] 3 Namseoul University, full-time lecturer, Dept. of Advertising & Public Relation 21 Mae ju-ri, Seconghwan-eup, Seobuk-gu, Chonan, South Korea [email protected]
Abstract. Second Life is a 3D virtual web environment that has aspects of visualization and sense of presence, as well as text and audio interaction. The whole research aims to explore the sense of presence experienced by online users in Korean Second Life assessing with what ways online users who engage in Korean Second Life perceive a sense of presence. This research leads us to understanding possible factors creating the sense of presence in Korean Second Life as well as in 3D web environments of the near future. Instead of providing specific response from research subjects who have experienced Second Life, this paper presents the theoretical backgrounds to speculate on Second Life constructs and features for enhancing users’ feeling of presence. Keywords: Presence, 3D Web environment, Second Life, Virtual place, Avatar, Emoticons.
1 Introduction The ways of communication through digital media, especially the Internet, have been changing drastically with 3D virtual technology on the Internet. Second Life is the best known of this 3D online virtual world completely created and owned by its residents. Second Life has several characteristics differentiating it from other existing online environments. Especially, the combination of all conceivable types of real-time interaction and digital embodiment found in Korean cultural variation may create various virtual human interactions unlike anything else online. Thus, this study aims to explore the sense of presence experienced by online users in Korean Second Life assessing with what ways online users who engage in Korean Second Life perceive a ∗
sense of presence. This study leads us to understanding possible factors creating the sense of presence in Korean Second Life as well as in 3D web environments of the near future.
2 Overview of Second Life Bing accessible via the Internet, Second Life is an online virtual world initiated by Linden Lab that launched on June 23, 2003. It creates a highly visual, spatial, and auditory chat environment. It is also an excellent example of the current trend toward graphical, interactive domains on the Internet. Thus it is an online 3D virtual world or multi-user virtual environment (MUVE). It provides an immersive environment within witch users can create digital characters, and interact with people from around the world. Residents can explore, socialize, meet other people, participate in individual and group activities, and create and trade virtual property and services with one another, or travel throughout the world, which residents refer to as the grid which is a 3D virtual technology platform. As of April, 2008, Second Life has a total number of over 13 million registered members, geographically dispersed around the country. The most recent estimates put the number of Korean users in Second Life at about 200,000.
3 Presence in SL Environment Advances of web environments in immersive, interactive technology, combined with its increasing availability and quality, have resulted in a practical concern with the manner in which people interact with technologically mediated 3D virtual web environments and online games such as Second Life. Terms of presence or virtual presence are used interchangeably to describe the extent to which people perceive that they are actually present in an artificially created environment. Many attempts have been made to define such an experience - the feeling of being in an environment that is virtually created - and identify its determinants. In the following sub-sections, we will describe the concept of presence and illustrate influential factors affecting users’ presence in a technologically mediated environment, Second Life. 3.1 Overview of Presence First introduced as “Telepresence” by Minsky [1], the term presence is investigated by researchers in a diverse set of disciplines, such as communications, psychology, computer science, engineering, and others. Both terms refer to user’s subjective experience of a medium as a sense of “being there.” This notion of presence is concerned with the subjective feeling of existence within a mediated environment [2,3,4,5]. According to Lombard and Ditton [6], presence is considered as a multidimensional concept. They identified six different conceptualizations of presence by reviewing a diverse set of literatures. These conceptualizations include, for instance, (1) presence as social richness (e.g., the extent which a medium is perceived as
The Experience of Presence in 3D Web Environment
389
sociable, warm, sensitive, personal), (2) presence as realism (e.g., the extent to which a medium can produce seemingly accurate representations of objects, events, and people), (3) presence as transportation (the feeling of “you are there”, “It is here”, and/or “we are together” , and (4) immersion (the extent of perceptual and psychological immersion in a mediated environment). Lombard and Ditton [6] combined all of these conceptualizations into a single conceptual definition of presence as the perceptual illusion of nonmediation. An illusion of nonmediation occurs when an individual is willing to overlook or fails to perceive the existence of a medium in his/her communication environment and responds as he/her would if the medium were not there [7]. This illusion leads to the subjective experience of being in environment, even when individual physically inhabit another. There has been relatively research conducted to investigate factors that contribute to a sense of presence [e.g., 5,6,7,8]. Studies have suggested characteristics of a medium (e.g., dimensionality, interactivity, image quality, and number of sensory outputs) and characteristics of the medium user (personality type, prior experience with the medium, willingness to suspend disbelief) are important factors that contribute a sense of presence. For instance, Steuer [5] argues that the vividness and interactivity found in a medium help to evoke presence, and that identifying these attributes in different media will allow a user to predict the type of presence he/she will experience. Witmer and Singer [8] identify involvement and immersion as necessary psychological states for the experience of presence, and indicate that they are critical determinants of both the learning and performance outcomes from media use. As such, presence is determined by formal features of a medium and characteristics of the medium user [6]. 3.2 Sense of Virtual Place from Virtual Constructs in Second Life There does not exist a specific territoriality in cyberspace, rather there exists only perceived spatiality or territoriality that can be technologically mapped in computer networks enabling a person to perceive that everything exists and is located as in offline space. Rather than examining the Internet as a place for locality, scholars on the specific technical and communications features of the Internet often like to stress its ‘placeless’ nature [9]. Due to the conceptual characteristics of cyberspace placelessness, web environments seem to have no territoriality. Community studies show that a sense of place plays a crucial role in forming and sustaining community attachments and community life [10]. Some scholars also identify an evident sense of place in online communities [11,12,13]. Thus, attachment to specific virtual place in Second Life may, then, be considered to be an important factor in forming and sustaining online users’ feeling of presence. Second Life environment, as the best example to illustrate the implications of territoriality and spatiality of cyberspace, give tacit evidence to this assumption. Only a few scholars suggest that the geographical elements of physical environments play a key role in virtual environments [13]. Such may be because cyberspace is a computer-generated conceptual space with no necessary geographical analogue. However, it seems that Second Life designers understand the purposes and effects of virtual landscapes. They make systematic efforts to increase the number of choices available to Second Life members or participants.
390
S. Park, H.S. Hwang, and M. Choi
Fig. 1. Dokdo Island and Korean traditional ancient ships
Currently, text-based systems are giving way to virtual worlds equipped with more advanced 3D graphic interfaces. In Second Life, the metaphor of place and space has a highly visible presence, and produces hyper-real simulations that imitate physical place and space. Such a shift in virtual environments from text-based systems, such as MUDs or MOOs, to advanced 3D virtual systems, such as Second Life and various virtual worlds suggests that virtually constructed geographical landscapes may be essential for virtual environments. In addition, providing fascinating new social spaces that exist only in cyberspace, virtual environment systems can make valuable contributions to the understanding of the geographical structure of virtual space [14]. For instance, Second Life has a number of theme categories in which there are different advanced 3D graphical virtual landscapes and constructs for participants. Second Life is one of a number of commercially developed virtual reality (VR) systems that are publicly available on the web environments. Using avatars, Second Life provides a sense of geographic space, with freedom to move in different directions, and feelings of physicality [14]. In Second Life, users are able to own land and build homesteads, thereby constructing their own places for online social interaction. The world of Second Life provides the most persuasive example of how a virtually constructed place (as place is one of the essential aspects of a physical environment) plays an important role in web environments. In addition, virtual games and virtual environment systems remind us that although landscapes are virtually constructed, they are patterned on geographical landscapes in the physical world. In more detail, all virtual constructs including weapons, artifacts, landscapes, and even most human actions in Second Life are patterned after those in the physical world. This indicates that the virtual environment itself is conceptually constructed on the basis of the physical environment which may provide enhanced feeling of presence.
The Experience of Presence in 3D Web Environment
391
Fig. 2. The scenery around the South of Gate in Seoul
Second Life comprises recognizable elements like buildings with floors and walls, towns with homes and streets, or landscapes with mountains, boulders, meadows and blue sky backgrounds. Many scenes recreate offline life public meeting places like bars, art galleries, parks and plazas. Second Life can create fantastical creatures of all kinds. Thus, we emphasize the importance of such spatial constructions in the continuity of a feeling of presence in Second Life, such as mediated proximity and belonging which may give us a sense of presence when we are in Second Life. 3.3 Digital Embodiment of Virtual Human: Avatar In virtual environments, users are generally represented by virtual, or digital, embodiments, commonly referred to as avatars. As a virtual representation of a human being, avatars are graphical icons used to represent online participants, usually in chat rooms with visual interfaces. The use of avatars in cyberspace provides a new mode for rich personal interaction and communication, for the avatars provide the ability to communicate nonverbally. Many sophisticated 3-D avatars in Second Life are animated and change shape according to what the user is ‘doing’, such as walking or sitting. Avatars’ appearance includes the way a person dresses, cuts his or her hair or decorates his or her body. In Second Life one can choose his or her online representation, although this choice may depend on the software used and the world visited. The uses of avatars can facilitates feelings of “being there” or “being together” and re-introduce spatial elements and nonverbal cues usually deficient in traditional CMC environments [15], so that people tend to regard them as realistic representations in cyberspace. In their early get-acquainted period, online community members are likely to engage in making their online identity solid and positive in cyberspace through various, such as the
392
S. Park, H.S. Hwang, and M. Choi
Fig. 3. Example avatar of a Korean user
manipulation and use of textual graphics, the application of other technologies that capture image from audio and visual to the Internet interface, and most importantly, through digital embodiments like avatars. An avatar is as a virtual representation of human being in cyberspace representing as an online participant in a virtual world. An avatar functions as a virtual body. A visitor uses the avatar to get around the cyber world. When people meet virtually, their presence is partially indicated through their avatars’ presence. Visitors interact through the body movements and gestures of the avatars. In a graphical virtual environment, the avatar looks like a human, though not necessarily the human it represents. An observer responds to the other avatar similar to the way they would behave towards another person’s physical presence. Visitors move their avatars together when they are speaking with one another and separate them when the conversation ends. They change the avatar’s facial expression to indicate emotion. They manipulate the avatar’s limbs in order to make gestures. Vilhjalmsson [16], in an analysis of the conversation transcripts, shows significant improvement in the overall conversational process and significantly fewer messages spent on channel maintenance in the avatar groups. The avatars also significantly improved the users’ perception of each others’ efforts. Green [17] articulates that being digital of body in cyberspace is one means of embodiment formalized by practices of representation systems. Taylor [18] also admires avatars as digital embodiments of ourselves that can provide the aspects of presence, social integration, and communication. As digital bodies, avatars disclose something about a person in cyberspace. They are public signals of who online participants are. They also shape and realize how users internally experience their selves in cyberspace. Consequently, the digital embodiment and avatar of virtual humans is to give the user as sense of presence and as the representation of one self and other self in cyberspace. As elements in Second Life, avatars—including their appearance, behaviors and controls—condition the expectations and behaviors of Second Life participants or visitors. Based on extensive databases, geographical information systems,
The Experience of Presence in 3D Web Environment
393
multi-media and virtual reality modeling, in 3D graphical virtual worlds users can “step into and move around” within these cities, contribute to discussions on different kinds of forums, and run daily errands [19, p.154]. Such virtual avatar worlds are the expression of the desire of humans to create in the digital environment a world as rich as the offline reality in which we live. Virtual human, computer-generated entity that looks and acts like people and engage in conversation in virtual environments have become prominent in the study of communication. The idea of virtual human envisions future computer systems that are social actors rather than tools. Techniques from artificial intelligence, computer animation, and human–computer interaction are increasingly converging in the field of embodied conversational virtual agents. Such agents are envisioned to have similar properties to humans in face-to-face communication, including the ability to generate simultaneous verbal and non-verbal behaviors. 3.4 Emotional Indicators: Emoticons As people continue to interact and maintain relationships in cyberspace, they gradually move toward deeper areas of their mutual interpersonal relationships through verbal indicators, such as the emoting function and emoticons, in order to compensate for the limits of social context and the lack of nonverbal cues which may enhance a feeling of presence in virtual environments. In addition, CMC media are shifting away from primarily text-based interaction to more audio and visual-based interaction. In Web environment and in the new multimedia possibilities of the Internet, text, sound, video and image produce a wider range of digitized symbolic cues and these compensate for emotions and presence assumed to be lacking in online interaction. There are ways to make up for the lack of physical presence in the CMC environment. Using emotional indicators, such as emoting, emoticon, and avatar, can create a sense of presence in which we can identify with the other person. For instance, emoticons (emotional icons) are used to compensate for the inability to convey voice inflections, facial expressions, and bodily gestures in written communication. Emulating a facial expression and expressing a feeling that supplements the message, emoticons are used most in meet and greet situations where one wants to appear approachable. Some researchers illustrate how emoticons are useful for making up for the lack of nonverbal context cues in virtual environments [20,21,22,23]. Emoticons may indicate the degree of intimacy in a personal relationship. Utz [24] argues that the more emoticons a person use, the more friendships that person builds. Baym [20] also states that community members in cyberspace create expression through emoticons and abbreviations. The combination of multiple features of CMC technology and using emotional indicators in online interaction can be a powerful element for enhancing sense of presence in Second Life.
4 Overview of Research Design This research used a qualitative analysis of the user experience in Second Life in 4 undergraduate classes. As one of course assignments, total number of students who participant in this research for a week was 214. The process to evaluate the student
394
S. Park, H.S. Hwang, and M. Choi
experience in Second Life is inductive method using grounded theory and cultural theory as a foundation. Every student was asked a series of open-ended questions designed to gather their views about and experience with Second Life activities. The central question posed to the students for this paper was “in what ways and to what extent do you feel a sense of presence to Second Life?” The generated data were categorized and refined into common themes.
5 Conclusion Second Life is social spaces formed and maintained by CMC technologies, human interaction, and the individual’s interpretations. Today’s advanced technology provides a user with a unique experience that one has left real world and is now “present” in the virtual environment. This notion of presence in the virtual world has been central to both VR researchers and designers since it is introduced [1]. There have been many attempts to examine what presence is, how it occurs, and what cases and effect are. Yet, the benefits of presence, either to enhance interactive VR design or human performance, have not been clearly discovered. Given that, this paper suggests several interposed factors that may facilitate the perceived presence of users interacting with technologically mediated environments such as Second Life. Because our web environment is still undergoing constant development, future web environment is continuing in the direction of more complicate and enhanced GUI providing an enhanced sense of presence.
References 1. Minsky, M.: Telepresence. Omni 3, 45–51 (1980) 2. Heeter, C.: Being there: The subjective experience of presence. Presence: Teleoperators and Virtual Environments 1(2), 262–271 (1992) 3. Kim, Biocca: Telepresence via Television: Two dimensions of Telepresence May Have Different Connections to Memory and Persuasion. Journal of Computer-Mediated Communication 3(2) (1997), http://www.ascusc.org.jcmc/vo3/issue2/ 4. Sheridan, T.B.: Musings on telepresence and virtual presence. Presence: Teleoperators and Virtual Environments 1(1), 120–126 (1992) 5. Steuer, J.: Defining virtual reality: Dimensions determining telepresence. Journal of Communication 42(4), 73–93 (1992) 6. Lombard, M., Ditton, T.B.: At the Heart of it all: The conpcet of presence. Journal of Computer-mediated communication 3(2) (1997), http://www.ascusc.org.jcmc/vo13/issue2/ 7. Lombard, M., Reich, R.D., Grabe, M.E., Bracken, C., Ditton, T.B.: Presence and Television: The role of screen size. Human Communication Research 26(1) (2000) 8. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: A presence questionnaire. Presence: Teleoperators and Virtual Environments 7, 225–240 (1998) 9. Rheingold, H.: The virtual community: Homesteading on the electronic frontier, revised eds. The MIT Press, Cambridge (2000) 10. McMillan, D.W., Chavis, D.M.: Sense of community: A definition and theory. Journal of Community Psychology 14(1), 6–23 (1986)
The Experience of Presence in 3D Web Environment
395
11. Baym, K.N.: The emergence of on-line community. In: Jones, S.G. (ed.) Cybersociety 2.0: Revisiting Computer-Mediated Communication and Community, pp. 35–68. SAGE, Thousand Oaks (1998) 12. Baym, K.N.: Tune in, Log on: Soaps, Fandom, and Online community. SAGE Publications, Thousand Oaks (2000) 13. Gotved, S.: Spatial dimensions in online communities. Space & Culture 5(4), 405–414 (2002) 14. Dodge, M.: Explorations in Alpha World: The geography of 3-D virtual worlds on the Internet. In: The RGS-IBG Annual Conference for Virtual Reality in GeographyWorkshop and Special Session, Leicester, England, January 4-7 (1998) 15. Mantovani, G.: The Psychological construction of the Internet: From information foraging to social gathering to cultural mediation. CyberPsychology & Behavior 4(1), 47–56 (2001) 16. Vilhjalmsson, H.H.: Avatar augmented online conversation. Doctoral dissertation. Program in Media Arts and Sciences at the Massachusetts Institutes of Technology (2003) 17. Green, N.: Beyond Being Digital: Representation and virtual corporeality. In: Holmes, D. (ed.) Virtual politics: Identity & Community in Cyberspace, pp. 59–78. SAGE, London (1997) 18. Taylor, T.L.: Living Digitally: Embodiment in Virtual Worlds. In: Schroeder, R. (ed.) The social life of avatars: Presence and interaction in shared virtual environments, Springer, London (2002), http://www.itu.dk/~tltaylor/papers/Taylor-Living Digitally.pdf (Retrieved on January 20, 2009) 19. Ridell, S.: The web as a space for local agency. Communications 27, 147–169 (2002) 20. December, J.: Units of analysis for Internet communication. Journal of Communication 46, 14–38 (1996) 21. Talamo, A., Ligorio, M.: Identity in the cyberspace: The social construction of identity through on-line virtual interaction. In: 1st Dialogical Self conference, Nijmegen, June 2326 (2000) 22. Wolf, A.: Emotional expression online: Gender differnces in emoticon use. CyberPsychology & Behavior 3, 827–833 (2000) 23. Utz, S.: Social information processing in MUDs: The development of friendships in virtual worlds. Journal of Online Behavior 1 (2000), http://www.behaviro.net/JOB/v1n1/utz.html
Influence of Real-World Ten-Pin Bowling Experience on Performance during First-Time Nintendo Wii Bowling Practice Kirsten A. Peters User Centric, Inc. 2 Trans Am Plaza Dr, Ste 100 Oakbrook Terrace, IL 60181 USA [email protected]
Abstract. In order to understand if proficiency in a real-world activity influences performance or movement characteristics in a simulation of that activity, six expert real-world ten-pin bowlers, ten novice real-world ten-pin bowlers and eight expert Wii bowlers completed 3 games of Nintendo Wii Sports’ bowling game. Two values were recorded for each throw: the (score) and the total range of motion (ROM) for the participant’s throwing arm (using two-dimensional motion capture). Averages across the first five trials were compared to averages across the last five trials. From the first five to last five trials, there were significant increases in both the mean score and mean ROM values, when collapsing data across experience level. While there was a significant main effect of experience level on the overall ROM values, differences between each experience group’s ROM values were not detected. A larger sample size is necessary to confirm if real-world ten-pin bowling experience influences score and ROM during first-time Wii bowling. Keywords: Motor program, motor learning, motion capture, gesture-based interaction, video games, transfer of skill, real-world vs. simulation.
Influence of Real-World Ten-Pin Bowling Experience
397
activity would impact performance or the arm movements used when simulating that same activity using the Nintendo Wii. Nintendo's Wii Sports bowling game was chosen as the primary stimulus because it is closely modeled after the rules of the real-world game of ten-pin bowling and the outcome is wholly dependent on user input. Novice and expert ten-pin bowlers participated in the current study, as well as a control group of expert Wii bowlers. Performance was measured using the score of the first throw. The total range of motion (ROM) of the dominant bowling arm during the first throw was measured using a two-dimensional motion capture system. The primary goal of this exploratory study was to clarify the relationship between experience with a real-world activity, performance simulating the real-world activity, and arm movements used to simulate the real-world activity. Specifically, the study examined if either group’s score or ROM values changed during a first time Wii bowling practice period.
2 Background 2.1 Ten-Pin Bowling In the modern American sport of ten-pin bowling, players score points by rolling a weighted ball down a lane and knocking over pins. While real-world bowling technique varies from athlete to athlete, most players use a similar arm swing movement and a four- or five-step approach technique. In one throw, a bowler receives kinesthetic, visual, and auditory feedback. Based on the outcome of a delivery, the bowler can choose to use the same movement or modify the movement in the next frame [2]. In ten-pin bowling, repetition is key. The more precisely a bowler can duplicate the same series of motions each time they throw the ball, the higher his or her score will be. It can take years to master proper bowling technique with consistent precision and accuracy. While the amount of time and experience needed to become an advanced bowler varies, technique is likely to be fairly consistent and well established once a bowler reaches this level. 2.2 Nintendo’s Wii Sports Bowling Game The Wii Sports video game, which is a part of every Wii console, includes a bowling game. The game (see Fig. 1) uses the same rules and scoring conventions as realworld ten-pin bowling and the virtual environment is modeled after the environment and playing area seen at a real bowling alley. In Wii Sports bowling, players hold the remote and move their arm like they would in the real-world game of ten-pin bowling. When a player is ready to roll the ball, he or she holds down the trigger button located near their index finger and then raises the remote to their chest. While holding down the trigger button, the player swings as if he or she was rolling a real bowling ball [3]. When the player is ready to release the ball, he or she releases the trigger button at the bottom of their swing.
398
K.A. Peters
Fig. 1. The Nintendo Wii Remote video game controller (left) and a screen view of Nintendo’s Wii Sports bowling game (right)
2.3 Motor Schema Theory and Transfer of Skill Over time, a real-world bowler learns the series of body motions necessary to produce high scores. Schmidt’s motor schema theory attempts to explain how motor skills, such as those used to develop proper bowling technique, are generally learned [4]. Learning motor skills involves acquiring schemata which define the “relationships of the information involved in the production and evaluation of motor responses" [5]. These rules are learned through experience and used by the motor programs to produce new actions [6]. Schema theory describes an overall mental construct, a “generalized motor program”, which uses the schemata to determine specific actions. For example, over time, a bowler generates a ‘bowling motor program’. Motor learning is the process of improving motor skills where one utilizes a “set of processes that underlie the changes in a capability for movement” [5]. During the process of motor learning, a person’s motor movements will change as they develop the internal capability for the movement. While motor learning can be neither directly observed nor defined as a change in behavior, the processes taken together will lead to some particular product, state or change. One can infer the presence of motor learning based on the observed changes in motor behavior over time. When a bowler’s approach starts producing consistent results, they have learned what movements are necessary for their expert bowling motor program. If the tasks of throwing a ball in real-world bowling and Wii bowling are similar enough, skill may be easily transferred. The amount of transfer of skill between two similar tasks depends on the degree of similarity of the two tasks. As the tasks become more dissimilar, the amount of transfer, as measured by performance, can be expected to decrease [7]. It has yet to be determined whether a purely quantitative relationship exists between the degree of task similarity and subsequent change in the amount of transfer. According to the motor schema theory, if a task is modified in some way and this similar-yet-modified task is practiced routinely for a non-trivial length of time, it is possible that a separate motor program will eventually be created. When tasks are practiced in simulations prior to being actually performed, a positive transfer of skill has been consistently observed in previous studies [8, 9]. Unfortunately, little research has been done on human performance when skills are transferred from a real task environment to a virtual environment. However, there is some evidence that this transfer of skill can be positive in certain applications [8, 10].
Influence of Real-World Ten-Pin Bowling Experience
399
3 Experimental Goal and Hypotheses When a user is first exposed to a system, they choose a technique based on prior knowledge and motor schemas associated with the task requirements. Therefore, when players are told that a video game is a bowling simulation, they may assume the two activities to be similar and start the video game with a similar movement technique to one they have used for real bowling. While proper real-world bowling technique has been documented, proper (i.e., most effective) Wii bowling technique has yet to be determined. The nature of the Wii bowling simulation is such that it does not give users adequate kinesthetic feedback from the heavy ball. Players may start out using their general bowling motor program, but with practice, may modify their schema and arm movement behavior for Wii bowling. With experience, Wii bowlers may develop a modified arm swing movement that does not require the transfer of momentum to the ball from the full arm swing motion, but still achieves the goal of knocking down pins. Though it may be largely based on the original generalized motor program, another modified program may develop over time. Given the aforementioned topics, questions begin to arise. When expert real-world bowlers play Nintendo’s Wii Sports bowling game for the first time, does their proficiency at the real task help or hurt their performance at the simulated task compared to novice bowlers? Will either groups’ movement characteristics or performance abilities change during their first bowling session and, if so, how? How do each group’s movement characteristics and performance abilities compare to expert players of Nintendo’s Wii Sports bowling video game? The present study was developed in an attempt to examine these questions by observing the movement characteristics and performance abilities of novice and expert real-world bowlers over one experimental period of Wii bowling. Because they both had limited experience with the Wii bowling game, the expert real-world bowlers and novice real-world bowlers were expected to start the study with similar ROM and scores. However, because they would not be limited by the real-world bowling motor program, the novice real-world bowlers were expected to have a greater decrease in ROM and greater increase in score compared to the expert realworld bowlers. Therefore, the novice bowlers would end the experimental period with a smaller ROM and higher score than the expert real-world bowlers. As the control group, the expert Wii bowlers were expected to show no change in ROM or score over the experimental period. Their ROM was expected to be the smallest and their scores were expected to be the highest of the three groups.
4 Methodology 4.1 Participants A sampling method of convenience was used to recruit three groups of participants: Expert real-world bowlers, novice real-world bowlers, and expert Wii bowlers. Table 1 shows a summary of the participant characteristics.
400
K.A. Peters Table 1. Summary of group characteristics
Group
Real-world bowling experience Mean score Frequency of per game playing
Wii bowling experience Mean score Frequency of per game playing
> 160
≥ 1x/week
< 100
< 2x/month
< 100
< 2x/month
< 100
< 2x/month
NA
NA
>160
≥ 3x/month
The group of expert Wii bowlers was used as a control group. The data collected from this group was treated as a baseline comparison for the data collected from the other two groups. The real-world ten-pin bowling experience from this control group was not taken into consideration, as their Wii bowling technique (i.e., a unique schema) has already been established regardless of their ten-pin bowling experience. 4.2 Procedure The experimental tasks were performed on the Nintendo Wii. Participants used the Wii Remote to interface with the video game system. Participants were given a 6 ½ foot by 3 foot playing area within which to move. This allowed them to play the game with free and natural movement while allowing us to record their movements. Two reflective markers for the motion capture system were attached to Velcro strips and placed around participants’ right humerus. The approximate length of the participant’s humerus was measured and the two strips were placed at the first and third quartiles of the upper arm (see Fig. 2). Participants were then given a short orientation to become familiar with the Nintendo Wii interface and control of the Wii Remote within the Wii bowling game. Regardless of group assignment, all participants performed the same experimental task. They were asked to play the Wii bowling game with the goal of achieving the highest score. Participants completed three games of ten frames each, for a total of 30 trials. A trial was defined as a full frame which may have included a second throw, as per standard ten-pin bowling rules. Participants were required to stand during the trials and allowed to walk and lean forward as long as they stayed within the designated area needed for the motion capture system to function properly. 4.3 Data Collection Two values were recorded for each throw: the number of pins knocked over (score) and the total range of motion (ROM) for the participant’s arm. The arm movement was recorded with a motion capture video system and later translated into a ROM value represented as an angle. Motion capture is the process of digitally recording body movements and translating them into a two- or three-dimensional model [11]. A 2D motion-capture video system was used in this study. The system consisted of a digital video camera,
Influence of Real-World Ten-Pin Bowling Experience
401
passive reflective markers placed on a participant’s upper right arm, a light source pointed at the participant (to increase the amount of light reflected by the markers), and a computer running the digital video capture and Peak Motus software package [12]. The camera faced participants laterally and recorded the projection of each reflective marker at 60 frames per second. Accounting for the position and orientation of the camera, the software used the digitized video to transform these projections into a 2D position of each marker. The motion capture system was configured to allow for the capture of the entire arm swing motion of each throw. 4.4 Data Analysis In a game of real-world bowling, the goal of the first throw in a frame is always the same: knock over as many pins as possible. The conditions during the second throw of a frame are highly variable and entirely dependent on the result of the first throw. Therefore, our primary analysis focused only on the data collected during that first throw. However, participants were asked to complete both throws of every frame. A statistical analysis of the score data was conducted for all three participant groups. Since the change in performance over the experimental period was the primary variable of interest, the scores of the first throws from the first five and last five trials were included in the analysis. In order to calculate the ROM joint angles, the digital video data needed to be digitized by the motion capture software [12]. In this process, for each first throw, the software located the position of the centroid of each reflective marker from each frame of the video such that the participants’ arm motion was converted to a series of x-y coordinates. Maximum amount of shoulder flexion and hyperextension were deduced and combined to create the total ROM for each swing. Similar to the performance score analysis, only ROM data from the first throws of the first five and last five trials were analyzed for each participant group.
5 Results 5.1 Wii Bowling Score A 2 x 3 (trial period by experience level) split plot Analysis of Variance (ANOVA) was conducted on the score data where trial period (first five trials and last five trials) was the within-subjects variable and experience level (expert real-world bowlers, novice real-world bowlers and expert Wii bowlers) was the between-subjects variable. The purpose of this analysis was to determine if experience level influenced how much a participant’s score increased from the first five trials to the last five trials. A paired t-test was also conducted on the mean scores of the first and last five trials for each group. Figure 2 shows the average score for each group during the first five and last five trials. The main effect of trial period was found to be significant (F1, 21 = 4.89, p < .05). Overall, the scores from the last five trials (M = 8.23) were significantly higher than the scores from the first five trials (M = 7.83). There was no main effect found for experience level. This means that overall no one group scored better than another
402
K.A. Peters
Average Scores 9.0 8.65 8.5
Score
8.28
8.20
8.0 7.92 7.72 7.5
Expert bow lers
7.43
Novice bow lers Expert Wii-bow lers
7.0 First 5 Trials
Trial period
Last 5 Trials
Fig. 2. Average scores of each group for the first and last five trials of the session
group. Furthermore, we found no interaction between the trial period and experience level. From the first five to last five trials, no one group’s scores increased significantly more than the other groups’ scores. 5.2 Range of Motion A 2 x 3 (trial period by experience level) split plot ANOVA was conducted on the ROM data where trial period (first five trials and last five trials) was the withinsubjects variable and experience level (expert real-world bowlers, novice real-world bowlers and expert Wii bowlers) was the between-subjects variable. Similar to the score data, the purpose of this analysis was to determine if experience level influenced how much a participant’s ROM changed between the first five trials and the last five trials. Figure 3 shows the mean ROM for each group during the first five and last five trials. The ROM results show a significant main effect of trial period (F1, 21 = 17.43, p < .001). Overall, participants’ ROM increased significantly from the first five trials (M = 157.76°) to the last five trials (M = 180.19°). Post-hoc t-tests showed a significant increase in ROM for the expert real-world bowlers, (t5 = 3.094, p < .05) and novice bowlers (t9 = 4.19, p < .005) but not for the expert Wii bowlers.
Average ROM Values
225
209.95
ROM (degrees)
205 190.84 185
176.40 165
162.61 154.83 Expert bow lers
145 140.26
Novice bow lers Expert Wii-bow lers
125 First 5 Trials
Trial period
Last 5 Trials
Fig. 3. Average ROM values of each group for the first and last five trials of the session
Influence of Real-World Ten-Pin Bowling Experience
403
There was also a significant main effect of experience level (F2, 21 = 3.89, p < .05). Post hoc comparisons showed no significant differences between the individual experience levels, although there was a non-significant trend for expert real-world bowlers to have a larger ROM than novices (p = .055) and expert Wii bowlers (p = .074). The results also revealed no interaction between the trial period and experience level. This means that the three groups did not differ in the amount of ROM increase. 5.3 Correlation The Pearson product-moment correlation coefficient was calculated between the score and ROM variables to determine if there was a relationship between the increase in ROM values and the increase in score. This analysis did not reveal a significant correlation between the mean ROM and mean scores. There were also no significant correlations between the mean ROM and mean scores for each individual group.
6 Discussion The purpose of this study was to determine if proficiency in ten-pin bowling influenced performance abilities or movement characteristics when playing simulated Wii bowling. In general, the data did not support the original hypotheses that novice real-world bowlers would have a greater decrease in ROM and greater increase in score than the expert real-world bowlers. In several cases, test statistics approached significance, suggesting that the lack of support for some hypotheses may be at least partially due to the small number of observations analyzed in this study. 6.1 Wii Bowling Score While the overall Wii bowling scores increased significantly from the first five to the last five trials, there was no significant main effect for experience level nor was there a significant interaction between experience level and trial period. These findings indicate that novice real-world bowlers did not have a greater increase in score than expert real-world bowlers. An observed range effect may have prevented the detection of a relationship between experience level, trial period and Wii-bowling scores. While ten-pin bowling scores can range from 0-10 pins per trial, participants’ average scores only ranged from 7.43 to 8.65. 6.2 Range of Motion The significant increase in ROM during the session is contrary to the original hypothesis. The training period for this study only included one frame of Wii bowling. It is possible that participant’s bodies were not fully warmed up when they started their first trial and the overall increase in ROM was related to the loosening of their joints and increased activity of their muscles. The fact that the expert real-world bowlers had the largest overall mean ROM suggests these participants may have applied their established real-world bowling motor program to play Wii bowling, which requires a large arm swing movement to
404
K.A. Peters
propel the weighted ball down the lane. The significant increase in ROM for these participants can then be explained by the warm-up theory described above. The novice bowlers should not have such developed real-world bowling motor programs. However, the expert real-world bowlers maintained a bowling motor program for a reason: they were experts at real-world bowling. The non-significant trend for novice bowlers to have a smaller ROM than the expert real-world bowlers may be a result of novice bowlers not having a real world bowling motor program to define how large of a movement to use during Wii bowling. The significant main effect of experience level on ROM points toward a confirmation of Schmidt’s motor schema theory and the existence of generalized motor programs [4]. With a larger sample size, it may have been possible to determine if experience in real-world bowling influenced the increase in ROM over the experimental period. 6.3 Correlation While the ROM data was collected for this study using a 2D motion capture system, real-world and Wii bowling can involve 3D arm and body movements. For example, many expert real-world bowlers twist their wrist right before releasing the ball to include a spin. No correlation between ROM and score data was discovered. A likely explanation for the lack of correlation between ROM and score data is that data collection was solely focused on the 2D arm swing movement at the shoulder. The influence of any other body part and the overall body motion in all three dimensions was ignored. Additionally, as discussed before, the increase in ROM may not be related directly to the score performance because participants’ bodies may have been warming up throughout the thirty trials. As the joints were loosened and the muscle activity increased, the ROM at the shoulder may have increased because their arm muscles were able to stretch further. This may be another reason why no correlation was found between the overall mean ROM and overall score.
7 Conclusion Based on the overall increase in score, participants playing Wii bowling for the first time were able to improve their performance in just three games. These results indicate that the Wii bowling game was designed to allow for quick improvement, which is often not the case in real-world bowling. This function may be part of a strategy to motivate new users to continue playing Wii bowling and other games on the Nintendo Wii. Motion capture systems are often used by video game designers when creating realistic character movements. However, motion capture and video games do not often come together for academic research purposes. The motion capture system used in this study focused on capturing the amount of flexion and hyperextension of participants’ arm movements at the shoulder. While this 2D movement is a major component of the bowling arm swing motion, we believe that future studies attempting to understand the bowling arm movement should utilize a 3D system and
Influence of Real-World Ten-Pin Bowling Experience
405
help examine the role of the more subtle arm movements and their impact on performance. Further research is also needed to explore the non-significant trends observed during this study. A longer trial period should be used as the thirty trials may not have been enough to observe the true change in ROM and scores associated with learning to play Wii bowling, or the differences between the groups. Additionally, a larger sample size is necessary to determine if real-world ten-pin bowling experience has an influence on performance and movement during an initial Wii bowling practice.
References 1. Nintendo, What is Wii?, http://wii.nintendo.com/whatiswii_index.jsp 2. Bellisimo, L.: The bowler’s manual, 4th edn. Prentice-Hall, Inc., Englewood Cliffs, NJ (1982) 3. Nintendo of America. Wii Sports [Video game software and instruction booklet]. Nintendo of America, Redmond, W.A (2006) 4. Schmidt, R.A.: A schema theory of discrete motor skill learning. Psychological Review 82, 255–260 (1975) 5. Lee, T., Sherwood, D.: Schema Theory: critical review and implications for the role of cognition in a new theory of motor learning. Research Quarterly for Exercise and Sport 74(4), 376–382 (2003) 6. Newell, K.M.: Motor skill acquisition. Annual Review of Psychology 42, 213–237 (1991) 7. Schmidt, R.A., Young, D.E.: Transfer of movement control in motor skill learning. In: Cormier, S.M., Hagman, J.D. (eds.) Transfer of learning: Contemporary Research and applications, pp. 47–79. Academic Press, Inc., London (1987) 8. Kenyon, R.V., Afenya, M.B.: Training in virtual and real environments. Annals of Biomedical Engineering 23(4), 445–455 (1995) 9. Torkington, J., Smith, S.G., Rees, B.I., Darzi, A.: Skill transfer from virtual reality to a real laparoscopic task. Surgical Endoscopy 15(10), 1076–1079 (2001) 10. Bürki-Cohen, J., Boothe, E.M., Soja, N.N., DiSario, R., Go, T., Longridge, T.: Simulator fidelity – the effect of platform motion. In: Proceedings of the International Conference on Flight Simulation – The Next Decade, Royal Aeronautical Society, London, UK (2000) 11. Furniss, M.: Motion Capture. MIT Communications Forum (2004), http://web.mit.edu/comm-forum/papers/furniss.html 12. Peak Motus [Computer software]. Vicon Motion Systems, Oxford, UK
Emotionally Adapted Games – An Example of a First Person Shooter Timo Saari1, Marko Turpeinen2, Kai Kuikkaniemi2, Ilkka Kosunen2, and Niklas Ravaja3 1 Temple University, 1801 N. Broad Street, Philadelphia, PA, USA, and Center for Knowledge and Innovation Research (CKIR), Helsinki School of Economics, Finland and Helsinki Institute for Information Technology (HIIT), Finland [email protected] 2 Helsinki Institute for Information Technology (HIIT), Finland {Marko.turpeinen,kai.kuikkaniemi,ilkka.kosunen}@hiit.fi 3 Center for Knowledge and Innovation Research (CKIR), Helsinki School of Economics, Finland [email protected]
Abstract. This paper discusses a specific customization technology – Psychological Customization - which enables the customization of information presented on a computer-based system in real-time and its application to manipulating emotions when playing computer games. The possibilities of customizing different elements of games to manipulate emotions are presented and a definition of emotionally adaptive games is given. A psychophysiologically adaptive game is discussed as an example of emotionally adapted games. Keywords: Customization, adaptive systems, psychological effects, emotion, games, emotionally adapted games, psychophysiological measurement, Psychological Customization.
1 Introduction Emotions or emotion-related variables (e.g., competitiveness) play a critical role in gaming behavior [1, 2]. People seek, and are eager to pay for, games that elicit positive emotional experiences and enjoyment; however, an enjoyable game may not elicit only positive emotions but possibly also negative ones (e.g., anger, fear). Thus, one of the major goals for video game designers is to elicit optimal emotional responses or response patterns. It is possible to build systems which either automatically or semi-automatically adapt games to create optimal emotional responses and response patterns. Such customization systems entail the manipulation of the game per player to elicit specific emotional responses or to create desired response patterns. We propose a system called Psychological Customization to be used in the adaptation of games for manipulating emotional states of players.
Emotionally Adapted Games – An Example of a First Person Shooter
407
Psychological Customization entails the customization of transient (i.e. short-term) user experiences (i.e. psychological effects) when interacting with media- and communication technologies. Experience-based customization entails the automatic or semi-automatic adaptation of information per user, task and context in an intelligent way with information technology. A subset of Psychological Customization is to vary the form of information (modality for instance) per user profile, task and context, which may systematically manipulate (approach, avoid, modify intensity, frequency and duration, create combinations, create links to behavior) different psychological effects. Psychological effects can be considered transient states, such as emotion, mood, types of cognition, learning, flow, presence, involvement and enjoyment. [e.g. 3, 13] Psychological Customization works on the principle of target experiences which can be set by using the system either by providers of a service or by users. Target experiences are different types of transient psychological states that have varying durations, frequencies, intensities, combinations, and motivational and action tendencies as well as a linked stimulus class which facilitates a particular state. The system is set up to either approach or avoid a certain target experience within the other parameters of customization such as altering the intensity, duration or frequency of a certain effect or creating simultaneous combinations or links to probable behavior of different target experiences. [3] Psychological Customization can infer customer needs via user models and various feedback loops observing customer behavior and responses. Psychological Customization also provides different adaptations of presenting information to different customers based on the customer interacting with the configuration settings of the product or service. The basic functioning of the system is based on a classic control theory model, the biocybernetic loop. It defines two kinds of control loops in complex and adaptive systems that can be established: negative (avoid an undesirable standard) and positive (approach a desirable standard) loops of feedback [e.g. 4, 5]. Target experiences are then controlled by this type of reasoning in the system based on real-time feedback from user responses and/or based on ready-made design-rule databases. Emotion. Although various definitions of emotions have been proposed, the most general definition is that emotions are biologically based action dispositions that have an important role in the determination of behavior [e.g., 6 ]. It is generally agreed that emotions comprise three components: subjective experience (e.g., feeling joyous), expressive behavior (e.g., smiling), and physiological activation [e.g., sympathetic arousal, 7]. There are two main competing views of emotions. Proponents of the basic distinct emotions argue that emotions, such as anger, fear, sadness, happiness, disgust, and surprise, are present from birth, have distinct adaptive value, and differ in important aspects, such as appraisal, antecedent events, behavioral response, physiology, etc. [8]. In contrast, according to a dimensional theory of emotion, emotions are fundamentally similar in most respects, differing only in terms of one or more dimensions. Proponents of the dimensional view have suggested that all emotions can be located in a two-dimensional space, as coordinates of valence and arousal [or bodily activation; e.g., 6, 9]. The valence dimension reflects the degree to which an affective experience is negative (unpleasant) or positive (pleasant). The arousal
408
T. Saari et al.
dimension indicates the level of activation associated with the emotional experience, and ranges from very excited or energized at one extreme to very calm or sleepy at the other. Other theorists have, however, suggested that the two main, orthogonal dimensions of emotional experience are negative activation (NA) and positive activation (PA) that represent a 45° rotation of the valence and arousal axes [10]. The NA axis extends from highly arousing negative emotion (e.g., fear) on one end to low-arousal positive emotion (e.g., pleasant relaxation) on the other, while the PA axis extends from highly arousing positive emotion (e.g., joy) to low-arousal negative emotion (e.g., depressed affect). The self-report NA and PA dimensions have been suggested to represent the subjective components of the BIS and BAS, respectively [e.g., 10, 11]. We adopt the latter definition of emotion. On the NA axis we call the high arousal negative emotion anxiety and stress while the low arousal emotion can be termed as pleasant relaxation. On the PA axis we see the high arousal emotion as joy and the low arousal emotion as depression. In our studies we have successfully used both psychophysiological measurements and self-report to index emotional processes, also when playing computer games. In computer games, there is a dynamic flow of events and action, games potentially eliciting a multitude of different emotions varying across time. A serious limitation of prior game studies is that they have used tonic, rather than phasic, psychophysiological measures. Tonic measures (e.g., the mean physiological value during the game minus pre-game baseline) do not enable the examination of the varying emotions elicited by different instantaneous game events. Given that psychophysiological measurements can be performed continuously with a high temporal resolution, it is possible to quantify phasic responses to instantaneous game events (e.g., by comparing the local pre-event baseline to physiological activity immediately following event onset). [12] It is then evident that psychophysiological measurement of emotional states when playing a computer game is both feasible and fruitful in providing an account of some aspects of the moment-to-moment experience of the user [13]. Naturally, psychophysiology could be extended to function as a feedback loop into the gaming engine making real-time adaptation of the game relative to the emotional state or mood of the user a possibility.
2 Emotionally Adapted Games Emotionally adapted gaming can be seen as based on gaming templates which are parts of the meta-narrative of the game. Hence, a basic approach to an element to be adapted inside a game is a psychologically validated template which creates a particular psychological effect. A broad view of templates may be that the whole game consists of a database of psychologically validated templates that are dynamically presented by the gaming engine in sequences during gameplay. A limited view entails that a smaller collection of templates is used. The element of psychological evaluation means that the selected psychological influence (such an emotional response) of the template on a particular type of user is sufficiently predictable. These psychologically evaluated templates may consist of i) manipulating
Emotionally Adapted Games – An Example of a First Person Shooter
409
the substance of a game, such as story line (initiating events, new characters etc.) and manipulating the situations specifically related to the character of the player (such as putting the character into sudden and dangerous situations inside the game) and ii) manipulating the form or way of presentation of the game (such as visual elements, shapes, colors, types of objects, sound effects, background music, level of interactivity and feedback etc.). The difficulty level of the game may also be continuously automatically be adjusted, thereby keeping the skills and challenges in balance, which results in a maintenance of an optimal emotional experience and possibly also a flow-state. [14] Why and when then to manipulate emotion in gaming on the basis of avoiding or approaching a specific emotional state? First, there are the transient basic emotional effects of games that are dependent of the phase of the game or some specific events. These are emotions such as happiness, satisfaction, sadness, dissatisfaction, anger, aggression, fear and anxiousness. These emotions are the basis of narrative experiences, i.e. being afraid of the enemy in a shooting game, feeling aggression and wishing to destroy the enemy and feeling satisfaction, even happiness, when the enemy has been destroyed. Emotional regulation systems in these instances most naturally may focus on manipulating the event structures, such as characters, their roles, events that take place and other features of the narrative gaming experience. [14] Second, there are possibilities for emotional management, especially in the case of managing arousal, alertness and excitation. Also, one may wish to manage negative emotions, such as sadness, dissatisfaction, disappointment, anger, aggression, fear and anxiousness. The case for managing these emotions is twofold. On the one hand, one may see that these emotions could be eliminated altogether in the gaming experience. This can happen via either eliminating, if possible, the emergence of such an emotion in the game. For example, one can make a deliberately happy game with level-playing monkeys in a far away island throwing barrels at obstacles and gathering points. This would include minimum negative emotions. Or, in a game where negative emotion is a basic part of the game, one may wish to limit the intensity, duration or frequency of the emotions via manipulating gaming events and gaming elements so that sadness or fear are at their minimum levels, or that gaming events do not lead to sadness at all. [14] Similarly, managing level of arousal or the intensity, duration and frequency of select negative emotions may be quite feasible in the case of children as a form of parental control. On the other hand, one may wish to maximize arousal, alertness and excitation, perhaps even anger, fear and aggression for hardcore gamers. Third, there are possibilities related to the avoidance of certain types of emotions that are typically indicative of a poor gaming experience. Inactivity, idleness, passivity, tiredness, boredom, dullness, helplessness as well as a totally neutral experience may be indicating that there is some fundamental problem in the usergame interaction. This could be due to poor gaming skills of the user vs. the difficult challenges of the game or some other factors, such as the user is stuck in an adventure game for too long and can not proceed without finding a magic key to enter the next level or so. When a gaming engine detects these emotions in the user, it may adapt its behavior to offer the user more choices of selecting the difficulty level of the game or offer the user some clues as to how to go forward in the game. The game can also adapt its level of difficulty to the player’s skill level. [e.g. 14]
410
T. Saari et al.
Fourth, it is also possible to create different combinations of emotional states (satisfied and angry) or emotional states and other psychological states (pleasant relaxation and efficient information processing) or emotional states and behavior (using specific motivational and action tendencies). [3] All of these possibilities may be relevant. However, the elimination or minimization of certain emotions may be specifically feasible in the case of indicated overly poor gaming experience in which the game may adapt its behavior to assist the user. It should be noted that events in games may change quickly and produce complex situations and hence complex emotions that may change rapidly. Consequently, one should better integrate these approaches into the genre or type of the game, such as driving simulator, first person shooter, sports game such as golf, or an adventure game, or a level-playing game for children. [14]
3 Example: Psychophysiologically Adaptive First-Person Shooter Game We will now present a basic system schematic of an emotionally adapted game in Figure 1. The process of a typical gaming engine is depicted on the left-hand side of the diagram. The engine continuously monitors user input, which is typically collected using a keyboard, a joystick, or other game controllers. This input data is then processed and transferred to the layer that handles the game’s internal logical state, and the user input may influence the game state. After the logical state of the game is defined the system alters the actions of the synthetic agents in the game world. For example, these include the actions of computer-controlled non-player characters. The complexity of this AI layer varies greatly depending on the game. Based on the game state and the determined actions of the synthetic agents, the physics engine determines the kinetic movements of different objects within the world. Finally, the game world is synthesized for the player by rendering the graphical elements and producing and controlling the audio elements within the game. [see 14] The proposed emotional regulation can be implemented as a middleware system that runs parallel to the actual game engine. The input processing layer of the game engine can receive a data flow of captured and pre-processed sensor data. The realtime signal processing may consist of different forms of amplifying, filtering and feature selection on the psychophysiological signals. This data flow may directly influence the state of the game world, or it can be used by the emotional regulation sub-module of the emotion feedback engine. This module consists of the rules of emotional balancing for different player profile types and gamer-related explicitly set preferences controlled by the “emotion knob”. In addition, it contains a collection of design rules for narrative constructions and game object presentation within the game world. The emotional regulation module also receives input from the game engine’s logical layer to make selections related to desired emotional balance and narrative structures within the game. [14] The outputs of emotional regulation engine may then be applied to various different levels of the actions of the game engine: i) the logical state of the world may be re-directed, ii) the actions of the synthetic agents may be controlled, iii) the
Emotionally Adapted Games – An Example of a First Person Shooter
411
kinetics of the game may be altered and iv) the rendering of the game world may be changed. First two options are more relevant to high-level and story-related structures of the game, whereas the last two are more directly related to the selection of presentation of objects within the virtual environment. [e.g. 14]
Main game engine Input processing layer
processing player’s input data flows in real-time
Game logic layer
Emotion feedback middleware engine Data acquisition
collecting and pre-processing biofeedback sensor data
Emotional regulation
determining the state of the game world
balancing emotions in the game for different player profile types
Artificial Intelligence layer
story-based design rules to control the game content
Narrative rules
determining how synthetic agents act in the world
Presentation rules
Kinetics layer enforcing the world’s physics laws on game objects
database of presentation rules for different game objects
Emotion knob
Synthesizing layer
rendering displayed images and producing game audio
player-set content preferences
Fig. 1. Emotional adaptation system design for games. Adapted from [14].
With our system design for games it is possible for the game designer as well for the user to set desired emotional targets to be approached or avoided. The system uses both positive and negative feedback loops to determine the ideal adaptations case-bycase for game play for various emotional effects to be realized and managed. Indeed, to implement and evaluate some of the ideas presented, we have explored novel technical solutions and tested different kinds of psychophysiological adaptations that can be implemented. EMOShooter is a prototype platform for psychophysiologically adaptive 3D first-person shooter (FPS) gaming. It is built on top of open-source graphics engine (OGRE 3D) and physics engine (ODE). In this experimental platform we have the possibility to modify practically any game world element, player avatar, avatar outlook, or control parameter. EMOShooter is a simple psychophysiologically adaptive game and hence a part of our emotionally adapted games definition. The system uses psychophysiological signals to influence the ease of use of the controls of the game hence affecting game play difficulty and game play experience. The system does not have target experiences systematically implemented at this moment nor does it have an emotion knob to tune the system. However, the EMOShooter game is a valuable example of one type of emotionally adapted games in demonstrating one feasible link between real-time emotional state measurement with psychophysiology and the game play. The goal of the EMOShooter game is to kill cube-like enemies either with sniper or machine gun. We have been testing various adaptation patterns with EMOShooter by primarily EDA and respiration as psychophysiological signals in our adaptive
412
T. Saari et al.
feedback system regards how these signals can be meaningfully connected to the actual game play via adapting game controls. Adaptation of game controls includes changes in rate of fire, recoil, movement speed and shaking. If a player is aroused this will be reflected in EDA and respiration signals which in turn will make rate of fire and movement slower and will make the aim shaky. Hence, for a highly aroused player the game becomes more difficult. For a mildly aroused or calm player the controls become more efficient and easy to use hence facilitating performance at game play. Game events are mostly arousing. The amount of cubes to shoot, their approach and firing on the user, the amount of health left after being hit and the sound effects all are geared to drive up arousal in the game. The player’s task is to be calm as indexed by psychophysiological signals to be able to operate the controls more efficiently. In our tests of the game we have collected also EMG data to infer the valence dimension of emotion during game play. In addition to the psychophysiological signals we have collected data from the players using behavioral game logging, video capture, interviews and questionnaires. During our tests we noticed that proper calibration and base lining of the psychophysiological signals is very important for the adaptations to work. We also noticed that having robust stimuli in the game is crucial for the adaptations to work because in many cases the stimulus functioned as a trigger in adaptation. The psychophysiological signals used are calibrated by using dynamic range (basically a variation of dynamic signal normalization algorithm), which has a memory buffer of a few seconds (depending on signal). Dynamic range is easy to use and effective calibration mechanism, and relative change seems to be more practical than absolute values in this kind of gaming. According to our early analysis, there are three key issues in designing psychophysiologically adaptive games i) understanding the meaningful emotionally adaptive gaming patterns, ii) implementation of adaptation algorithms and signal processing, and iii) purposeful use of sensors in the game context [15]. The design patterns used in emotionally adaptive gaming must be meaningful and enjoyable for the player, and the utilization of signals must also obey the overall goal of the game. In order to achieve the goal player should find the right rhythm or balance of playing the game and control of psychophysiological responses and signals. Signals should be analyzed as close to real-time as possible in psychophysiologically adaptive gaming in order to keep the feedback loop in pace with the game adaptations and game events. We have used time-series analysis with short sample windows. In practice, ECG, EEG and EMG always require extensive data processing, but EDA and respiration can be almost used as such to create the adaptation signal. This implies that not all psychophysiological signals are equally open to be used as real-time inputs into an adaptive game at least in this stage of signal processing hardware and software development. Usability of psychophysiological recording devices remains quite poor. Respiration, HR [heart rate] and EDA are probably the easiest to implement. Also in case of emotional adaptation the design of the game may include the physical design of the sensors, e.g. “Detective hat” for EEG sensors or “Sniper-gloves” for EDA sensors. Hence, the sensors could be designed as part of the game story rather than presented as cumbersome and invasive laboratory-originated equipment.
Emotionally Adapted Games – An Example of a First Person Shooter
413
In future versions of EMOShooter we may also employ the system design of emotionally adapted games including setting of explicit experiential targets and their parameters for gaming sessions and the emotion control knob.
4 Discussion Gaming, as we have presented it in this article, is perhaps one of the most promising application areas of Psychological Customization. We see that both casual and hardcore gamers could benefit from the use of our system and entirely new types of games can be created. Psychological Customization would enable game designers to use our tools both when developing the game and testing it with users in a rapid manner as well as part of the final product. From the user’s point of view, using various types of control knobs of emotion or other experiences enables them to customize and have more control over their gaming experiences. Good games are composed of delicate synthesis of the components creating a pleasant game balance and challenge for players. Introducing emotional adaptation increases the complexity of game design tasks involved. However, regards the economics of game development our system would not induce a dramatic cost. The system automatically establishes gaming patterns and structures which would fill a target experience and its parameters. Our system could be a modular toolset that can be adapted to various types of gaming platforms and gaming engines. The emotional tuning knob could be integrated into existing game controls including sliders for level of graphic violence in the game, for instance. Of course, development work is needed to create an easy-to-use game adaptation interface for users to set their preferences for game play. There are several challenges for Psychological Customization systems in games. We see three main areas which are critical: i) measurement of experience, ii) quality of reasoning in an adaptive system and iii) acceptability by users. The measurement of experience in the first place is a key challenge. However, we have presented an approach to concentrate on those aspects of experience, such as emotion, which are perhaps better defined than experiential states in general. Despite this focus, there is still disagreement in theorizing, operationalizing and measuring emotions. The quality of reasoning in an adaptive system is often a bottle-neck in performance. If a closed system is produced with fixed design-rules it would inevitably encounter situations, stimuli and users which would challenge the systems fixed rules and produce errors in adaptations. While there are several deep and sophisticated technical and mathematical approaches to this problem including different ways of machine learning and reasoning (which are beyond the scope of this article), we propose a higher-level solution possibility. Our answer is to rely on the co-evolutionary potential of our system design with changing user models and emergent design-rules. Of course, this approach needs working algorithms and techniques to form the necessary metadata and other data structures to be processed by the system. User acceptance of invasive psychophysiological measurement as input to the game is critical. Hardcore gamers may be more suspect to accept new peripheral
414
T. Saari et al.
devices linking them to game than gaming novices or casual gamers. However, the culture of connecting one’s body to a game is already evolving. Think of Wii as an example with a controller tied to one’s wrist, constantly touching the skin. It would not be unimaginable to think of psychophysiological sensors embedded in similar controls as people are more used to “semi-invasive” gaming controls beyond the use of a mouse and keyboard. The solution here could be to design sensors as embedded into essential existing or new types of gaming peripherals. A driving wheel with EDA and ECG sensors or driving gloves with similar sensors with added blood pressure, muscle tension and finger movement sensors could be used as easily acceptable controls of a driving simulator, for example. In our tests of the psychophysiologically adaptive game as a first prototype of emotionally adapted games we have been able to produce meaningful gaming patterns and game adaptations. We argue that our approach to emotionally adapted games is novel and creates new opportunities for designing games. We feel that our approach may result in a new type of enabling technological platform focused on the customization of gaming experiences. This new enabling technology platform can facilitate the development new types of games but can also be used with existing types of games and gaming platforms.
References 1. Grodal, T.: Video games and the pleasures of control. In: Zillmann, D., Vorderer, P. (eds.) Media entertainment: The psychology of its appeal, pp. 197–212. Lawrence Erlbaum Associates, Mahwah (2000) 2. Vorderer, P., Hartmann, T., Klimmt, C.: Explaining the enjoyment of playing video games: The role of competition. In: Marinelli, D. (ed.) Proceedings of the 2nd International Conference on Entertainment Computing (ICEC 2003), Pittsburgh, pp. 1–8. ACM Press, New York (2003) 3. Saari, T., Turpeinen, M., Ravaja, N.: Technological and Psychological Fundamentals of Psychological Customization Systems – An Example of Emotionally Adapted Games (A manuscript submitted for publication) 4. Pope, A.T., Bogart, E.H., Bartolome, D.S.: Biocybernetic system evaluates indices of operator engagement in automated task. Biological Psychology 40, 187–195 (1995) 5. Wiener, N.: Cybernetics: Control and communication in the animal and the machine, 2nd edn. MIT Press, Cambridge (1948) 6. Lang, P.J.: The emotion probe. Studies of motivation and attention. American Psychologist 50, 372–385 (1995) 7. Scherer, K.R.: Neuroscience projections to current debates in emotion psychology. Cognition and Emotion 7, 1–41 (1993) 8. Ekman, P.: An argument for basic emotions. Cognition and Emotion 6, 169–200 (1992) 9. Larsen, R.J., Diener, E.: Promises and problems with the circumplex model of emotion. In: Clark, M. (ed.) Review of personality and social psychology, vol. 13, pp. 25–59. Sage, Newbury Park (1992) 10. Watson, D., Wiese, D., Vaidya, J., Tellegen, A.: The two general activation systems of affect: Structural findings, evolutionary considerations, and psychobiological evidence. Journal of Personality and Social Psychology 76, 820–838 (1999)
Emotionally Adapted Games – An Example of a First Person Shooter
415
11. Gray, J.A.: The neuropsychology of temperament. In: Strelau, J., Angleitner, A. (eds.) Explorations in temperament: International perspectives on theory and measurement, pp. 105–128. Plenum Press, New York (1991) 12. Ravaja, N., Laarni, J., Saari, T., Kallinen, K., Salminen, M.: Phasic Psychophysiological Responses to Video Game Events: New Criterion Variables for Game Design. In: Proceedings of HCI 2005, Las Vegas, NV, USA (2005) 13. Saari, T., Turpeinen, M.: Towards Psychological Customization of Information for Individuals and Social Groups. In: Karat, J., Blom, J., Karat, M.-C. (eds.) Personalization of User Experiences for eCommerce. Kluwer, Dordrecht (2004) 14. Saari, T., Ravaja, N., Turpeinen, M., Kallinen, K.: Emotional Regulation System for Emotionally Adapted Games. In: Proceedings of FuturePlay 2005 conference, Michigan State University, USA, October 13-15 (2005) 15. Kuikkaniemi, K., Laitinen, T., Kosunen, I.: Designing emotionally adaptive gaming. In: Proceedings of EHTI 2008: The First Finnish Symposium on Emotions and HumanTechnology Interaction, University of Tampere, Tampere, Finland, pp. 13–18 (2008)
DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities Tatsushi Takeuchi1, Koichiro Watanabe1, Tomoo Inoue2, and Ken-ichi Okada1 2
1 Graduate School of Science and Technology, Keio University Graduate School of Library, Information and Media Studies, University of Tsukuba {takeuchi,watanabe,okada}@mos.ics.keio.ac.jp, [email protected]
Abstract. This paper describes a system called DiamondTheater that supports creative activities in theater using a tabletop tangible interface. This system is used to aid in the planning of certain aspects of theater production, such as actor positioning, and sound and lighting cues. Without having some considerable experience in production, it is difficult to create a mental picture of an actual production. Thus, a miniature theatrical stage is reproduced on the tabletop surface to facilitate the user’s creation of such a picture and the sharing of ideas. Users collaboratively construct a stage by placing small dolls to represent actors and many kinds of miniature stage sets. In addition, the system allows users to reproduce other aspects of theater production such as sound and lighting. We performed a user study of this system and demonstrated that DiamondTheater appears to effectively assist the user’s activity in theater production design. Keywords: collaborative work, theater, tabletop tangible interfaces.
DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities
417
on the process of designing and arranging theater productions for actual public performance. By constructing a miniature theatrical stage on a tabletop surface, DiamondTheater helps users imagine and design theater productions, assisting with aspects of theatrical productions such as lighting and sound, as well as the interplay between actors and other theatrical elements. Each user can work while sharing the entire image of the production with the other users.
2 Creative Activity in Theater Theater as an artistic production is a branch of the performing arts and is usually created by utilizing a combination of diverse elements, such as stage lighting, sound effects, special effects, and stage sets. Staff members work according to their roles. A general example of a work flow in theater leading up to a public performance is shown in Figure 1. The cue sheet in Figure 1 is a form or template that lists information about cues. A theatrical cue is the trigger for an action to be carried out at a specific time. An action could be, for example, a lighting or sound change. Usually information about execution, timing, sequence, and details of the production are also written on the cue sheet. The board operators and running crews operate the theater productions they are involved in during a performance while confirming the information written on the cue sheet dealing with their department.
Fig. 1. A simple example of the work process leading up to a public performance
3 Related Work Many researchers have studied systems that support a wide variety of creative activities, such as story creation, animation and drawing[3,4]. Although many systems supporting two-dimensional creative activities have been studied frequently, systems for three-dimensional creative activities, such as a dance and theater, have not been developed well. Little is known about systems to support activities for these branches
418
T. Takeuchi et al.
of the performing arts. Several researchers have developed systems for theater activities, almost all of which use computer graphics and virtual spaces to represent the theater stage[5,6]. These systems using virtual spaces help users imagine the actual stage used in theater. However, virtual images are somewhat lacking in their ability to represent reality. Moreover, these systems help users to understand basic theatrical concepts, but are not systems that can be used to support real-life work. Of the few systems that support real-life staff works activities in theater, Avatar[7] is an application for supporting the distributed collaborative production of theater shows. The authors use virtual reality to represent certain theater scenes. When designers cannot actually be co-located to collaborate with each other on the design, Avatar displays the virtual scene using computer vision at the remote locations to share the scene in real-time. In Avatar, small objects with AR markers are used as input devices to the virtual world that is being shared by remote directors who are participating in the design. The poses and positions of the objects are recognized by using a webcam to observe the markers. This idea would be useful to facilitate the collaboration of remote users in production design. However, it is probably better to use real mockups in the case of co-located collaboration because the designers can share the workspace. There are many commercially produced systems for designing theater productions. For example, Matrix3[8] is a system for audio show control, and wysiwyg[9] is used to plan, design, and program lighting productions. These systems assist user's theater production design; however, users who are inexperienced in theater would be not able to operate these systems effectively since almost all such systems are tools for professionals. Moreover, the interfaces of these systems are usually PC-based and use a GUI, and these are normally systems for personal use for that reason. Thus, such systems are not assumed to be used by multiple users. We are interested in a system that can support the real-life work of staff, especially co-located collaboration in theater. We focus on the pre-production process of theater regarding theater production design. When compared with people that have a wide range of experience, it is difficult for staff without much experience to picture the actual stage, which includes a wide variety of theater elements such as sound and lighting, when thinking about theater productions. Furthermore, a cue sheet is created after the discussion of the theater production, but it is very complicated to write out a cue sheet by hand or using a PC based on the theater productions that were decided regardless of whether the person has experience or not.
4 System Design Our objective is to use our system to solve some of the above-mentioned problems. The major objectives of our system are (1) to help users create a mental picture of the actual stage when designing and considering a plan for theater productions, (2) to provide users with opportunities to share the ideas of each user, and (3) to reduce the amount of effort required to create a cue sheet. This system helps users visualize the actual stage and theater productions. Users can collaboratively discuss the theater productions for actual live performance using this system. The interface is based on an interactive tabletop surface, on top of which users can reproduce a small stage. Our system uses MERL's DiamondTouch table[10]
DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities
419
to support face-to-face collaborative work, and allows users to collaboratively design theater productions while sharing ideas. Users can also easily reproduce theater productions that they conceive, such as lighting and sound, on the reproduced stage, and easily confirm the relationship between the actors and other elements. Because the theatrical stage is represented three-dimensionally on the DiamondTouch surface, it is easier to capture the entire image of the actual stage, including depth and height, than when using theaters with virtual spaces.[5,6,7] The theater production data that users design and finalize is saved as electronic data in an XML (Extensible Markup Language) file. A cue sheet is automatically created based on the theater production data, and the users can obtain cue sheets as needed. As a result, the system saves users the trouble of making cue sheets. At the present stage we intend for our system to be used by people who are inexperienced in theater, such as children and students. It is difficult for inexperienced people to design theater productions while comprehensively creating a mental picture of them. We believe that our approach is an effective way for such people to enhance their creativity and to support creative activities in theater.
5 Implementation 5.1 Overview and Architecture A system image of DiamondTheater is shown in Figure 2. This system consists of two types of displays: DiamondTouch for reproducing theater productions and the stage, and a semi-transmissive display for showing information about the script and theater productions that the user has selected. We represent a theatrical stage on top of the DiamondTouch table. This is placed on a large rack, as shown in Figure 2. Placing a doll as a surrogate for an actor and placing small stage sets, the system reproduces a stage on the surface as shown in Figure 2. Specific graphics are drawn on the tabletop using an overhead video projector. The stage and theater productions are graphically projected onto the table from above as shown in Figure 3. The DiamondTouch senses the position and movement of the dolls placed on the tabletop surface.
Fig. 2. System image in which users are thinking about the production collaboratively
420
T. Takeuchi et al.
Fig. 3. An operating panel and a theatrical stage projected onto the tabletop surface
Fig. 4. Production and columnar writing script (Japanese style) data projected onto a semitransmissive display
The DiamondTouch can identify the manipulator by recognizing an electric signal that is transmitted through users when they touch the DiamondTouch surface. Then we add a round metal plate to the bottom of each doll and wrap a copper wire around the doll to enable it to be recognized by the system. Using the metal bases enables the electric signal to be carried from the user's body through the metal and wire to the table. When the users place a doll on the tabletop surface, the metal and wire carries the electric signal from the user to the DiamondTouch, and the DiamondTouch recognizes the signal and identifies the manipulators. Moreover, in order to assess the differences among the dolls, we add different-sized metal plates at the bottom of each doll. Therefore, by calculating the size of the metal plate placed on the tabletop surface, the system can recognize which objects the user is manipulating. As a result, the system can distinguish between the positions to which the user may move an object
DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities
421
without the use of any tools other than the DiamondTouch table. If metal plates of different sizes are added to the dolls used, the system can theoretically distinguish between any numbers of dolls. The semi-transmissive display is used to display various additional information. This display is placed upright beside the rack as shown in Figure 2. A sample image of this display is shown in Figure 4. The semi-transmissive display shows data about the theater productions and script information. The data about the theater production that users have decided on at the time of using the tabletop appear at the bottom of the screen. Script information, such as the lines, stage direction, speakers, and page numbers, appear at the top of the screen. These data are projected onto the semitransmissive display from the rear projector. Users can use a script (tangible object) that is actually used in practice. Instant memos and ideas are often written into scripts during practice or during discussions between staff. These actions are important to support the staff's creative activities, and it is for that reason we use a real script. We use RFID technology to load data about the script into the system. We add RFID tags to the pages of the script to be able to identify particular pages when they are viewed. When the page of a real script is turned over, the RFID reader attached to the rack loads the data and reflects the information of the selected page and projects it onto the semi-transmissive display. 5.2 Operation Procedure Users mainly take three steps to determine theater productions with DiamondTheater: (1) Choose line or stage direction to coordinate the theater production, (2) design the theater production while actually reproducing it using the system, and (3) finalize and save the theater production. Users first select a line at which they want to think about the theater production. Using RFID technology, DiamondTheater can identify which page of the script the users are looking at. Every time a page is turned over by the users, the tag data is recognized by the RFID reader, and information about the script on the page is read. The system provides users with a button to select a particular line on the tabletop surface. Users can select a line simply by touching this button while watching the data shown on the semi-transmissive display. Users design the theater productions while reproducing them and participating in collaborative discussions. Each theater production can be determined in association with a single or multiple lines/stage directions. The results of the theater productions that have been designed are saved as electronic data in an XML file. If a user requires a cue sheet, it can be automatically and easily output using DiamondTheater functionality without the need for complicated work. 5.3 Reproduction of Theater Productions The system can reproduce three kinds of stage effects: one of sound and two of lighting. The two types of lighting that can be reproduced are global lighting (border light) and spotlight. The look of turning a spotlight onto a doll instead of an actor is shown
422
T. Takeuchi et al.
in Figure 5. Each lighting effect can be turned on or off using a simple interaction with the system. Therefore, by simply touching a finger to the tabletop surface, users can control the lighting. In addition, sound is also available via a simple interaction. When a user selects by touch a piece of music that he or she wants to play, the music will play from two separate speakers placed near the tabletop. The standing position of an actor on stage is represented using a small doll. An example of the dolls used is shown in Figure 5. The system can recognize the actor's position if a user places a doll on the tabletop surface. Based on the position that the user places the doll, an image which expresses the actor's position is graphically projected on the tabletop (A or B shown in Figure 3). The users consider and determine the actor's position. Then, when the doll is quickly touched twice to the tabletop surface at the designated position, the position data is determined and saved in an XML file.
Fig. 5. A spotlight turned on a doll used to represent an actor
5.4 Use of Production Data We have defined a new language to save it, called CueML (Cue sheet Markup Language), which is based on XML. CueML is the language used to save data about the theater productions and to produce the cue sheet. The data for each production is saved and related to a single or multiple line(s)/stage direction(s). A cue sheet for reference during the actual performance is automatically produced by the system using the CueML data. A variety of XML elements are defined using XML tags in CueML. We can save more detailed information of a theater production in addition to the data that can be saved using the current implementation of DiamondTheater. For example, we can save data regarding volume and a sound effect, intensity and color for lighting, and so on. The current version of DiamondTheater cannot, however, deal with such detailed information. However, we will be able to extend the functionality of DiamondTheater to include these functions in the future. The system allows users to refer to theater productions that other groups saved previously by reading in data from a previous or current CueML file. This allows users to design theater productions by referencing other ideas. This process will support users in getting new ideas when users come to a deadlock.
DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities
423
6 User Study 6.1 Overview We conducted a user study to clarify the effects of DiamondTheater on users. The objective of this study was to observe how the system would affect the users' theater production planning, and how users would interact with the system. Fourteen university participants aged 21 to 24 years were asked to participate in this experiment. The participants worked in pairs during the experiment. All of the participants were inexperienced in regards to theater activity, and were novice users of the interface that we developed. Participants were asked to freely and collaboratively design theater productions about a script that we prepared using the interface. We used another interface using a PC in order to compare DiamondTheater to a system without DiamondTheater. The participants received an explanation of how to use each system before starting the experiment. A script that we produced for two-page spread was used for this experiment. The participants were asked to decide on three types of theater productions without a time restriction using each system and the script. The three types were sound, lighting, and actor positioning. The experiment had the following two system setups: DiamondTheater, and a PC and dolls. y DiamondTheater: Participants were provided with two dolls associated with actors. The positions of these dolls when placed on tabletop surface were automatically identified by DiamondTheater. The user received an explanation of how to control and turn on and off the sound and stage lighting. After designing the theater productions, the participants were asked to save their data. y PC and dolls: This case was similar to the work usually undertaken by theater production staff. The sound was played using a PC. However, stage lighting was not available because it cannot be easily reproduced in the normal environment. Participants were provided with two dolls to help them create a mental picture of their idea, the same as was done with the DiamondTheater-based system. Participants were asked to write the results of their designs on the script by hand. y We wanted to understand how DiamondTheater would allow participants inexperienced in theater to create a mental picture of their actual theater productions. We observed the user interaction in each setup, and analyzed whether DiamondTheater supported user activities related to theater production design even if the users had little or no experience in theater. After designing theater productions, we asked participants whether they noticed any particular aspects of the system. 6.2 Results Through this experiment, we were able to confirm some practical advantages of DiamondTheater in the process of planning theater productions. Participants seemed to actively interact with DiamondTheater while communicating with each other. With both system setups, participants designed a theater production freely with their own ideas while showing each other their ideas. The considerable difference between the two setups was the number of interactions with each system. Participants clearly interacted with DiamondTheater more actively
424
T. Takeuchi et al.
compared to the setup using a PC and dolls. DiamondTheater was likely a more familiar and intuitive interface than the PC and dolls setup. The participants were able to use the system easily by touching the tabletop screen or using physical objects (dolls and a script). Therefore, the participants were able to quickly become accustomed to the operations when using DiamondTheater. With DiamondTheater, almost all of the participants proactively tried to reproduce their ideas for the theater productions as they came up with them. As a result, the participants could design theater productions and share specific images while having an actual mental picture of their ideas. On the other hand, participants often thought about the three kinds of theater productions separately when using the setup using a PC and dolls. Although the participants were able to confirm the sound and actor positions, reproducing the stage lighting was normally difficult. Participants were not able to create a comprehensive mental picture of their production idea due to this constraint. This is a problem that could possibly occur in the process of actually designing a theater production. In addition, when using the PC and dolls setup, the participants sometimes selected incorrect productions for certain scenes. We define an incorrect production as incongruous or not appropriate for a certain scene. For example: y All lighting is turned off when there are lines of dialog. y Only one spotlight is turned on when two actors are talking. y The distance between the standing positions of the actors is too distant when they are talking with each other. We counted the number of incorrect productions in each single line or stage direction based on the theater production data collected from participants. Therefore, if an incorrect production occurred for three consecutive lines, the number was three. As a result, we found only one incorrect production when the DiamondTheater setup was used. However, nine incorrect productions were found when the PC and dolls setup was used. Incorrect productions increased dramatically when DiamondTheater was not used. All of the incorrect productions were found to concern stage lighting. Incorrect productions occurred at stage direction (for example, "He said it with a grim expression"), where participants decided on theater productions without a light turned on the actor in such situations. If all of the lighting were turned off, the audience would be not able to know what the actor's expression was. The number of incorrect productions was dramatically reduced when using DiamondTheater than when not. This result suggests that DiamondTheater adequately supported the participant's activities. Judging from the experiment, the participants worked while actually confirming their ideas with DiamondTheater. This is why the participants would seem to be able to accurately understand the situation of the theater productions that they were thinking about. However, with the setup using the PC and dolls, the participants were not able to adequately create a mental picture of their production ideas, especially in regards to stage lighting. This is because the participants could not try out their lighting plans with such a system, which was not the case with DiamondTheater.
DiamondTheater: A System for Reproducing Theater and Supporting Creative Activities
425
7 Conclusion In this paper, we have presented a system called DiamondTheater. This system in its current form is for use by people who are inexperienced in theater. DiamondTheater provides users with the opportunity to reproduce productions on a miniature theatrical stage. With the ability to reproduce various productions as well as an actual theatrical stage, users can design productions efficiently with a clear image of the production and collaborate when conceiving ideas. Participants in our user study were more successful and efficient when using DiamondTheater at design tasks in spite of their lack of experience. Thus, DiamondTheater allows users to use free-thinking when working in order to develop good ideas for diverse theater productions, and supports real-life creative work activities using simple operations.
Acknowledgments We would like to acknowledge Mitsubishi Electric Research Laboratories for donating a DiamondTouch table.
References 1. Patten, J., Ishii, H.: Mechanical constraints as computational constraints in tabletop tangible interfaces. In: ACM CHI 2007, pp. 809–818 (April/May 2007) 2. Ishii, H., Ullmer, B.: Tangible bits: Towards seamless interfaces between people, bits, and atoms. In: ACM CHI 1997, March 1997, pp. 234–241 (1997) 3. Raffle, H., Vaucelle, C., Wang, R., Ishii, H.: Jabberstamp: embedding sound and voice in traditional drawings. In: The 6th international conference on IDC 2007, pp. 137–144 (2007) 4. Drori, T., Rinott, M.: Pixel materiali - a system for creating and understanding pixel animations. In: IDC 2007, pp. 157–160, June 6-8 (2007) 5. Lewis, M.: Bowen virtual theatre. In: ACM SIGGRAPH 2005 conference on Web graphics 6. Geigel, J., Schweppe, M.: Virtual theatre: a collaborative curriculum for artists and technologists. In: SIGGRAPH 2005 (2005) 7. Dompierre, C., Laurendeau, D.: Avatar: a virtual reality based tool for collaborative production of theater shows. In: CRV 2006, pp. 35–41 (June 2006) 8. Matrix3, http://www.meyersound.com/lcs/matrix3/ 9. Wysiwyg, http://www.cast-soft.com/cast/software/products.jsp? SUBCATID=2 10. Dietz, P., Leigh, D.: Diamondtouch: A multi-user touch technology. In: User Interface Software and Technology (UIST 2001), pp. 219–226 (2001)
New Health Information Systems (HIS) Quality-in-Use Model Based on the GQM Approach and HCI Principles Reem Al-Nanih1, Hana Al-Nuaim1, and Olga Ormandjieva2 1 Department of Computer Science, King Abdul Aziz University, Saudi Arabia Dept of Computer Science & Software Engineering, Concordia University, Canada [email protected], [email protected], [email protected] 2
Abstract. Human Computer Interaction (HCI) is concerned with the design, evaluation, and implementation of interactive computing systems for human use. HCI is important in Health Information Systems (HIS), because misunderstandings arising because of poorly designed interfaces may lead to medical errors. This paper proposes a Quality-in-Use Model for HIS user interfaces, which identifies HIS-specific quality-in-use goals based on HCI principles, such as Mental Model, Metaphor, Visibility, Affordance, and Feedback. The Goal Question Metric (GQM) method was applied to build the new quality-inuse model applicable to most HIS systems. The resulting quality model is tailored for use in the medical field and reflects the values and viewpoints of the various user groups affected (e.g. doctors and nurses). Its qualitative and quantitative feedback can play a constructive and instructive role in medical institutions such as hospitals, and improve user productivity, satisfaction, and performance. Keywords: Health Information System (HIS), Human Computer Interaction (HCI), Goal Question Metrics (GQM), Quality-in-Use.
and satisfaction. The proposed quality-in-use model was evaluated on the Phoenix Health Information System (PHIS) currently in use at the King Abdulaziz University Hospital (KAUH). The application of our HIS Quality-in-Use model to the PHIS and the analysis of the quality feedback helped us produce a system (PHIS2) designed to minimize human error, reduce user frustration, and help doctors − with little or no training − work in a more pleasant, efficient, and effective way. The rest of the paper is organized as follows: Related work on HIS evaluation and HIS quality modeling is reviewed in section 2. Section 3 introduces the HCI principles required to understand the quality-in-use model proposed in this paper. The model itself is described in section 4. The results of the empirical validation on a case study are summarized in section 5. The conclusions and directions for future work are outlined in section 6.
2 Motivation and Related Work Computers are an integral part of modern medicine and diagnosis. This is especially so in hospitals, as they constitute the core of all advanced medical investigation methods such as ultrasound, computerized tomography, and magnetic resonance imaging [2]. HIS users need interfaces tailored to their specific requirements to facilitate their work and to help them avoid misunderstandings which may lead to medical errors. This need is still underappreciated by software designers [2]. The above motivated the research reported in this paper. HCI and human factors have a significant role to play in increasing the quality of HIS, and consequently in reducing the number of errors, especially those associated with the use of medical devices [5]. Finding design problems in an existing system related to human factors can be a challenge, however, because flaws in user interfaces can be subtle [6]. To the best of the authors’ knowledge, very little has been published on the quality modeling of HIS user interfaces. There has been a study of a hospital's order-entry system which identified twenty-two ways in which the system caused patients to be administered the wrong medicine [7]. Most of the issues involved were related to usability problems; for example, at times, users had to review up to twenty screens to see all of a patient's medications. In [3], the role of evaluation in the design of HIS is emphasized, and a framework for considering evaluation methods ranging from controlled experimental approaches to naturalistic observational approaches applicable throughout a system’s entire development life cycle is proposed. The problem of defining a quality model for evaluating HIS design is addressed in [4]. The model, based on the ISO/IEC 9126 external and internal product design quality, has been interpreted to meet the requirements of some classes of typical HCI system applications, and exploits experience gained both in the field of medical informatics and in the assessment of software products. The values that result from weighing the quality characteristics according to evidence of their criticality constitute a quality profile which can be used both for evaluation and certification of HIS design and implementation from developers’ point of view. The research reported in this paper differs considerably from the related work in the view on quality of HIS user interfaces. Our objective is to build the HIS Qualityin-Use model from the user's view in terms of HCI principles, which encompass the richness in HIS functionality and represent a broader view than software usability.
New Health Information Systems (HIS) Quality-in-Use Model
4 HIS Quality-in-Use Model The objective of this section is to present the Quality-in-Use model tailored specifically to HIS user needs. The Goal Question Metric (GQM) approach [1] was applied to define the user needs for the HIS Quality-in-Use goals, and how these goals are further decomposed into more technical sub goals and the corresponding indicators based on HCI principles.
432
R. Al-Nanih, H. Al-Nuaim, and O. Ormandjieva
GQM. The GQM approach is based on the fact that, for an organization to measure in a purposeful way, it must first specify goals for itself and its projects, then it must trace those goals to the data that are intended to define them operationally, and, finally, it must provide a framework for interpreting the data with respect to those goals [1]. The result of applying the GQM approach is a description of a measurement model targeting a particular set of issues and a set of rules for the interpretation of the measurement data. A GQM model is a hierarchical structure (Figure 1) which starts with a goal. That goal is refined into several sub goals using appropriate questions which usually break the issue down into its quantitative or qualitative indicators. Each indicator is then refined into a measurement procedure and rules for data interpretation. Quality-in-Use. Quality-in-use is the user's view of the quality of an environment containing software and is a broader view than software usability. It is measured based on the results of using the software in the environment (here, the medical field), rather than on properties of the software itself [14]. The ISO/IEC 9126-4 International Standard describes a two-part model for software quality: a) internal quality and external quality, and b) quality-in-use. Quality-in-use is defined as the capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety, and satisfaction in specified contexts of use [14]. In ISO/IEC 9126-4, effectiveness is defined as the capability of the software product to enable users to achieve specified goals with accuracy and completeness; productivity is defined as the capability of the software product to enable users to expend appropriate amounts of resources, such as time, to complete the task or the user’s effort, in relation to the effectiveness achieved; safety is described as the capability of the software product to achieve acceptable levels of risk of harm to people, businesses, software, property, or the environment; and, finally, satisfaction is expressed in terms of the user’s response to interaction with the product, which includes attitudes towards the use of the product).
Fig. 1. Hierarchical Quality-in-Use model for HIS (User Performance subgoal)
HIS Quality-in-Use. The quality-in-use model proposed in this research is rooted in the ISO/IEC 9126-4 definition of quality-in-use [14] and is aimed at helping medical personnel execute work-related tasks, in terms of: i) reducing the medical errors by making the use of HIS more pleasant and easier to manipulate, and ii) increasing their performance, productivity and satisfaction by means of HCI principles. The GQM approach was applied to further decompose user performance, user productivity, and user
New Health Information Systems (HIS) Quality-in-Use Model
433
satisfaction sub goals into their corresponding indicators providing feedback on the achievement of the user sub goals, and measurement procedures for obtaining the data required for the feedback. The decomposition is motivated by the HCI principles in the specific context of use, namely, HIS in the medical field. Figures 1, 2, and 3 illustrate the above decomposition for user performance, productivity, and user satisfaction. Validation of the HIS Quality-in-Use model proposed in this paper consisted of demonstrating the improvement of quality-in-use characteristics user performance, productivity, and satisfaction on existing software, The Phoenix Health Information
Fig. 2. Hierarchical Quality-in-Use model for HIS (Productivity subgoal)
Fig. 3. Hierarchical Quality-in-Use model for HIS (User Satisfaction subgoal)
434
R. Al-Nanih, H. Al-Nuaim, and O. Ormandjieva
System (PHIS, see section 5.1), which, after applying the HCI principles, produced a new version of the system (PHIS2, see section 5.2).
5 Case Study An example of medical software is the Phoenix Health Information System (PHIS). The creation of the PHIS began in 1998 after a team from New Zealand and other countries met in Dubai to evaluate current health information systems. King Abdulaziz University Hospital (KAUH) started to use the PHIS in 2004 to replace the Oasis system. The quality-in-use quality criteria were formally evaluated on the PHIS by novice and expert users from the hospital to identify problems they were having with the system. The researcher concluded that all the problems found based on the tasks doctors commonly perform on a daily basis were rooted in a violation of the HCI principles Mental Model, Metaphor, Visibility, Affordance, and Feedback. 5.1 PHIS Identifying and selecting users. The PHIS is divided into three parts. Each part serves a different group of people in the hospital: administrators, nurses, and doctors. This study focuses on the PHIS subsystem used by doctors and the tasks they commonly carry out on a daily basis. The doctors at the KAUH are listed in three categories: interns, residents, and consultants. Interns use the system the most frequently. The characteristics of the interns are summarized in Table 1. Table 1. Characteristic of the Interns Type of Intern Work Place Time Using the PHIS Training Time Attended the training session
Experienced
Novice In various wards KAUH From 8:00 AM to 5:00 PM From 3 to 7 months From 3 to 5 days Two hour training session available for all new doctors 80% 90%
Collecting data and identifying problems. The two data collection methods used to gather information from experienced and novice users were interviews and observations. Responses and feedback from both groups were recorded: Experienced users: Ten experienced PHIS users were interviewed and observed. These interviews provided the researchers with a clear picture of the problems users encountered regarding learnability and ease of use. The interviews also identified the daily tasks most commonly performed by the doctors on the system, such as: i) review the result of a laboratory or radiology test and check a patient’s prescribed medication, ii) request a new test, and iii) book an appointment for an operation. Novice users: Ten novice PHIS users were interviewed and observed while they carried out a list of tasks on the PHIS to evaluate their ability to use the system without help (Table 2).
New Health Information Systems (HIS) Quality-in-Use Model
435
Table 2. Results of the Task Questionnaire
User 1 User 2 User 3 User 4 User 5 User 6 User 7 User 8 User 9 User 10 Total
Task1 Check a laboratory result × 9 × × × 9 9 9 × × 4/10
Task2 Print a laboratory result × × × × × 9 × × × × 1/10
Task3 Request an order for radiology × 9 9 9 × × × 9 × × 4/10
Task4 Book an appointment for an operation × × × × × × × × × × 0/10
All the interviewees were found to be competent computer users. Five tasks were presented to the users in the questionnaire (Table 2): a check (9) means that the user performed the task successfully, while a cross (×) means that the user was unable to finish the task or to start the task. Forty percent of the users were able to complete tasks 1, 3, and 5, while the remaining users were no. Only 10% could complete task 2 and none of the users was able to complete task 4. Even though the users were proficient in terms of their computer skills and attended the training session, fewer than 50% were able to complete the tasks given to them. Table 3 lists the HIS tasks most commonly performed by the doctors, and the applicable HCI principles: Table 3. Tasks Given to the Novice Group Tasks Task 1.1 Check the last HB Task 1.2 Print the result Task 1.3 Compare CBC Tests? Task 1.4 Print HB from Compare Item Task 2 Order CBC Task 3 Booking Task 4 Check the radiology result Task 5 Print medication report
HCI Principles Mental Model Metaphor Metaphor Mental Model and Visibility Visibility and Feedback Affordance Mental Model, Visibility, and Feedback Mental Model and Feedback
Summary of HCI Problems in the PHIS. The results of interviewing the ten experienced users and the ten novice users indicated that the PHIS is not user-friendly and not easy to learn, and there are several interface design faults which need attention. The proposed quality indicators helped reveal the existing PHIS HCI problems, as outlined below (the problems are grouped by indicator for clarity): Mental Model • Lengthy steps required to reach a target window • Selecting a ward does not filter the teams • Team abbreviations are inconsistent from one ward to another
436
R. Al-Nanih, H. Al-Nuaim, and O. Ormandjieva
Metaphor • Use of the same icon for different functions, e.g.
and
Feedback • Function name of the window title displayed, instead of the window title • In the Profile tab: − there is a list of buttons using vague abbreviations (e.g. MAR/IVAR) − patient's demographic information is not displayed Visibility • Invisible order window Affordance • Diagnosis list not arranged alphabetically or by category • Doctors tried to click on a time text box without noticing the background color The obstacles and problems encountered by the doctors were solved based on the principles of HCI from a design point of view to minimize human error and user confusion, as well as to increase the efficiency and effectiveness. The result was PHIS2, a redesigned and improved version of the PHIS developed through a process that included pilot testing, redesigning, and redeveloping its user interface when it did not reflect HCI principles. 5.2 PHIS2 The following illustrates the improvement of the quality-in-use of PHIS2 by comparing the PHIS and PHIS2 quality indicators per sub goal: User Productivity improvement. After conducting the formal quality assessment of the PHIS and identifying the HCI principles that had been violated, all the recommendations from the findings were applied to improve user productivity with PHIS2. The productivity obstacles and problems encountered by the doctors were solved based on HCI principles from a design point of view to minimize human error caused by the poor quality user interface design of the PHIS. User Performance improvement. The ability of the user to complete a task successfully is used to evaluate the user’s performance. The performance measures are defined in Table 4. Table 4. Performance Measurement Performance measures Time needed by the novice user to perform task successfully How many times he/she chose the wrong icon How many times he/she made the wrong menu choice
Metric End Time- Start Time # of wrong icons selected # of wrong menu choices
Based on the technique suggested by Dumas and Redish [15], the performance of three experienced users was considered as the baseline for judging the time taken by participants. If their performance was considered excellent, more time was considered acceptable and much more time was considered unacceptable. Table 5 shows an example of the selected performance measures for the ‘Check the last HB’ task. Each
New Health Information Systems (HIS) Quality-in-Use Model
437
Table 5. Setting Criteria for Measuring Performance Measure Task 1.1: Check the last HB Time for task M= # of wrong menu choices I= # of wrong icon choices
Excellent < 1.5 M 1 1
Acceptable 1.5-2 M 2 -3 2 -3
Unacceptable >2 M More than 3 More than 3
task for each participant was measured and the data compared to that in Table 5, which classifies user performance as excellent, acceptable, and unacceptable. The results show that the performance of the novice group was excellent for more than 80% of the users for Tasks 1.3, 2, 4, and 5, and for 70%, 80%, and 60% of the users for Tasks 1.1, 1.2, and 3 respectively. For Task 1.4, performance was excellent for 50% of the users and acceptable for 50% of the users. In only one case was performance unacceptable, that is, for 20% of the users for Task 1.1. The data on the number of wrong menu choices show that performance was excellent for Task 4 and Task 5 for 100% of the users and for Task 1.1 for 90% of the users. The data on the number of wrong icons chosen show that performance was excellent for all tasks except Task 3 for 100% of the users, with performance for Task 3 being excellent for 80% of the users. It was interesting to compare the performance of the PHIS with the performance of PHIS2 by ten novice users performing four identical tasks. These tasks represent the Mental Model in Task 1.1, Metaphor in Task 1.2, Feedback and Visibility in Task 2, and Affordance in Task 3. The performance measure in the PHIS was accomplishment of the task, irrespective of the time taken and the number of attempts. Using PHIS2, the novices were able to accomplish their tasks without previous training, while using the PHIS, despite the in-house training, fewer than 40% were able accomplish their tasks. From the user performance evaluation, it is clear that the novice users were able to perform all the tasks with PHIS2 in excellent time, which reflects the accuracy of the proposed design. It also shows that their performance with respect to the number of wrong menus choices and icons was also excellent, which reflects the efficiency and effectiveness of the new UI design. User Satisfaction improvement. The users were asked to complete a User Satisfaction Questionnaire to determine users' impressions and opinions of the system, evaluating the user response to the interaction with PHIS2. The results of the questionnaire revealed that all the users gave ratings of ‘very easy’ and ‘easy’ for the tasks regarding learning the program, using the program, performing the particular task, finding the features, understanding the instruction, and recovering from error. In addition, 100% of the experienced users preferred using PHIS2 to using the PHIS, and 100% of the participants asserted that PHIS2 helps them improve the quality of their work.
6 Conclusion and Future Work The purpose of this research was the application of HCI principles, based on a sound goal-driven approach, to derive a new quality-in-use model from the medical field users’ view of the quality of HIS interfaces which will help to minimize human error and user frustration, and to make the tasks of doctors − with little or no training − more pleasant, as well as efficient and effective. Such a model can serve not only to
438
R. Al-Nanih, H. Al-Nuaim, and O. Ormandjieva
evaluate the quality-in-use of an existing HIS, but also as a guide for the user interface design of new medical software. The PHIS evaluation based on our HIS Quality-in-Use model resulted in major improvements to the user interface design in its improved version, PHIS2. Results of the formal evaluation of PHIS2 reveal that the system is a success in terms of helping users perform tasks in excellent or acceptable time, efficient and effective in terms of limiting the number of wrong menu choices made and wrong icons selected, and more pleasant to use, even with no training whatsoever. More case studies will be evaluated in our future work to determine the applicability of our quality-in-use model to different types of HIS.
References 1. Basili, V., Caldiera, G., Rombach, H.D.: The Goal Question Metric Approach (1) Institute for Advanced Computer Studies, Department of Computer Science, University Of Maryland, College Park, Maryland (2) FB Informatik, Universität Kaiserslautern, http://wwwagse.informatik.uni-kl.de/pubs/repository/ basili94b/encyclo.gqm.pdf 2. The Canadian International Development Agency (CIDA) and managed by the Canadian Society for International Health, The South Caucasus Health Information Project (2004), http://www.csih.org/what/schip/March 3. Kushniruk, A.: Evaluation in the design of health information systems: application of approaches emerging from usability engineering. Computers in Biology and Medicine 32, 141–149 (2002) 4. Fabbrini, F., Fusani, M.: Evaluation of Quality Characteristics in Health Care Information Systems, ERCIM News No.28 (January 1997), http://www.ercim.org/publication/Ercim_News/enw28/ fabbrini.html 5. Burnham. Dr. W.: The Human Factor, http://www.humanfactorsmd.com/hfandmedicine.html (2005) 6. Sawyer, D.: An Introduction to Human Factors in Medical Devices, Office of Health and Industry Programs, U.S Department of Health and Human Services, Center for Devices and Radiological Health (1996), http://www.fda.gov/cdrh/humfac/doitpdf.pdf 7. Nielsen, J.: Medical Usability: How to Kill Patients through Bad Design (2005), http://www.useit.com/alertbox/20050411.html 8. Marcus, A.: Metaphor Design in User Interfaces: How to Manage Expectation, Surprise, Comprehension, and Delight Effectively. California (1997), http://www.sigchi. org/sigchi/chi97/proceedings/tutorial/am.htm 9. Dix, A., Finlay, J., Abowd, G., Beale, R.: Human-Computer Interaction, 3rd edn. British Library Cataloguing, England (2004) 10. Booth, P.: An Introduction To Human-Computer Interaction, 4th edn. Lawrence Erlbaum Associates, Mahwah (1989) 11. Norman, D.: The Design of Everyday Things. Doubleday/Currency, New York (1988) 12. McCracken, D., Wolfe, R.: User-Centered Website Development: A Human- Computer Interaction Approach. Pearson Education, USA (2004) 13. Norman, D.: Introduction to HCI, B.Sc. Applied Computing (2004), http://hamilton.bell.ac.uk/btech/hci/hciintro.pdf 14. ISO/IEC 9126-4:2001, Software engineering — Product quality — Part 4: Quality –in-Use 15. Dumas, J.L., Redish, J.C.: A Practical Guide to Usability Testing, 3rd edn. Intellect, Wiltshire (1999)
An Information Visualization Approach to Hospital Shifts Scheduling Carmelo Ardito, Paolo Buono, Maria F. Costabile, Rosa Lanzilotti, and Adalberto L. Simeone Dipartimento di Informatica, University of Bari Via E. Orabona 4, Bari 70125, Italy {ardito,buono,costabile,lanzilotti,simeone}@di.uniba.it
Abstract. Scheduling staff shift work in a hospital ward is a well-known problem in the operation research field but, as such, it is very often studied from the algorithmic point of view and seldom from the human-computer interaction perspective. In most cases, the automatic solutions that operations research may provide do not satisfy the involved people. After discussing the inconveniences of an automatic approach with physicians, we have designed a staff scheduling system that combines an expert system with an information visualization (IV) system; in this way the schedule generated by the expert system is presented through the IV system to the schedule manager, who can modify the results if last minute changes are necessary, by directly manipulating the visualized data and obtaining immediate feedback about the changes made. Keywords: Information Visualization, Shift Scheduling.
can alternate appropriately. For these reasons, it is useful to employ a computer system able to store the history of work shifts for the last few years. However, fully automatic solutions cannot effectively cope with last minute changes: if someone gets sick, the overall schedule needs to be redefined. A mixed approach, combining algorithms that automatically compute a schedule with Information Visualization (IV) techniques capable of involving users to drive decision making process appears valuable. Visualizations can thus become the mean through which users and computers can cooperate to obtain efficient results with a semi-automatic process that leaves final decisions to users. This paper presents the development of a system composed by two modules: 1) an expert system, 2) an IV system. The first module automatically generates schedules, which are then visualized on the computer screen, so that users can interact and modify them. The user interface also allows users to specify some employees' constraints and preferences, which the expert system tries to satisfy. According to User-Centered Design methodologies [2], the system has been developed by taking into account the needs of final users, which have been involved in our design. We have collaborated with physicians of the Paediatric Hospital “Giovanni XXIII” in Bari, Italy. After an initial requirement gathering phase, some paper prototypes were devised and evaluated with the medical staff at the hospital. This led to several refinements of the prototypes that contributed to enhancing the usability of the system. The final system is the result of all the feedback received during the prior stages of the design phase. The paper has the following organization. Next section reports related work in this field. Section 3 describes how the system requirements have been gathered. Section 4 describes the developed scheduling system. Finally, Section 5 we report our conclusions.
2 Related Work Different works exist in literature that have explored the issue of hospital personnel scheduling. As shown in Burke et al. [3], most analyses focus on comparing different algorithmic solutions with the results obtained from other approaches and less on the day-to-day reality of hospitals. Another literature survey is provided in [4]. Other works study different aspects of the schedule generation process such as satisfying work rules, taking into account personal preferences, and performance in general. These systems are often developed for testing special purpose algorithms and not with the intent of being actually deployed in hospitals. In this work, the focus is on those studies who detailed experiences of complete systems being used in real world situations. Among these, Scheduler solves the generation problem algorithmically [5]: the application permits both total or partial schedule generation and manages unlimited work contracts. The schedules obtained with Scheduler can be visualized either as Gannt-diagrams, text or histograms, whereas our system offers different views, depending on the time granularity. Scheduler has been in use at several sites that could make use of its features: at Kolmårdens Djurpark, a Swedish zoo, users reported positively on its adoption, after an initial testing period. Very appreciated was the fulfilment of all work and law obligations, a requirement that is open to be much more error prone if worked on by humans, that
An Information Visualization Approach to Hospital Shifts Scheduling
441
may overlook or forget certain rules. In H&M Rowells’ call center, a mail order subsidiary of Hennes & Mauritz (a Swedish based clothes manufacturer and retailer), before the introduction of Scheduler, staff was used to get the breaks they wanted (with people they knew), while the optimised breaks generated by the application have maximised staff availability for high demand periods. Ghosh et al in their study [6] propose a computer aided system that supports managers in their work by computing optimal staff allocation based on a number of parameters. They report that the adoption of the system brings several benefits such as ensuring optimal nurse availability and simulate different staffing scenarios. The multitude of controls present in the interface disorients people not accustomed to the particular domain and would probably require some training before being able to use it with proficiency. From a human-computer interaction perspective, there is certainly room for improvement. The application presented by Beliën et al [7] enables managers to allocate the resources needed for the master surgery unit, through a graphical user interface. Both the schedule and the available resources are shown in the same screen. The schedule area is further subdivided by days per week and according to the various operating rooms. Schedules have to be built manually, by assigning each surgeon to the desired time slot. The application helps the schedule manager by instantly visualizing the impact of each choice: in this way conflicts can be more easily identified and solutions found. This system has been tested at the Belgian university hospital Gasthuisberg and, according to the authors, it was deemed to be very promising for the purpose of facilitating and improving the schedule generation process. The interface itself, again, could require some training to be used to its fullest. Among commercial systems, we examined Kappix’s DRoster [8] and PlanningPME [9] have been examined: while they both do not allow automatic generation of schedules, they offer several features that allow users to specify work shifts through an interface similar to calendar applications. In DRoster, an automated rule engine spots violations or inapplicable schedules. An interesting feature is the possibility of specifying “places” and authorize staff from this point of view. DRoster allows schedules to be viewed both in a daily and in a hourly resolution. PlanningPME visualizes schedules in several different resolutions: morning or afternoon shift and daily, weekly or monthly.
3 The Requirement Gathering Process With the aim of designing the scheduling system according to real users’ needs, we involved some physicians at the Pediatric Hospital “Giovanni XXIII” in Bari. They were happy to collaborate with us because they are open to new technologies. We have successfully collaborated with them also in the development of a mobile system for supporting physicians treating epilepsy [10] and a system supporting remote collaboration among physicians [11]. A field study with contextual interviews has been performed during the requirements analysis to better understand the medical domain, how the physicians operate and the main features of the application to be implemented. User observation methods and principles of participatory design have proven to be effective in user interface
442
C. Ardito et al.
design. The initial investigation was carried out by HCI researchers who performed contextual analysis at the hospital. They observed how physicians define the schedule and discussed the current process with them through interviews and focus groups. The familiar workplace helped the interviewed to recall certain work anecdotes that could have otherwise been neglected should the interview had taken place in another context. We discovered how the scheduling process was handled: despite technological advancement, schedules were still built manually via several paper forms, during a 30-day period. The different agreements that have to be made in order to guarantee the fulfilment of preferences and special requirements are the results of personal negotiations between the interested parties and the schedule manager. All the material was collected for later examination and for a preliminary description of the interaction possibilities that would have had to be managed. This material was used to formulate interaction scenarios that could depict the most frequent tasks users should be able to perform. The requirement gathering phase highlighted that the system to develop is very complex from the functional viewpoint. In fact the resulting schedules have to satisfy several constraints, some of which related to regulations, others to the history of previous schedules and others to individual physicians’ needs or preferences. For this reason, it was decided to rely on an expert system for the management of the schedule generation process, but leaving users the possibility of making changes, should special occasions arise that would make certain staff members unexpectedly unavailable; for instance, it could obviously happen that a physician cannot get to work for some days because s/he must attend a professional meeting or because s/he might be ill; it is thus necessary to insert this information and have the system recompute schedules. Therefore, it was necessary to design an intuitive interface that provided a) an overview representation of the overall situation and b) the possibility of interacting directly with visualized data.
4 The Scheduling System The scheduling main components are: the expert system module and the visualization module. The former is responsible of analyzing the facts provided in input by the schedule manager (in this paper we assume to be a male physician) and computing a solution that tries to satisfy the constraints. The latter visualizes the computed schedules and allows the operator to modify them. These modules are described in detail in the following sections. 4.1 The Expert System Module The expert system is developed in Java and JClips [12] and uses XML for data interoperability. It exploits one of the most commonly used techniques to develop expert systems, namely rule-based programming. With this paradigm, a set of rules (which represent heuristics) indicate how the system should behave in a variety of situations. Each rule is composed by an “if” and “then” portion. The former contains a series of patterns (or facts) that, if satisfied, will cause the rule to be applicable. The latter represents a set of actions that will be executed when the rule is applicable.
An Information Visualization Approach to Hospital Shifts Scheduling
443
To begin the scheduling process, the system must know some data that are inserted by the user such as: start and end date of the period to be scheduled, physician’s days off, etc. Other data that is needed is those which is computed internally by the system and is related to the amount of duty physicians and working and festive availabilities assigned to physicians in the Operative Unit. The insertion of facts in the knowledge base allows the expert system to activate the rules that will produce the final state, and hence the full schedules for the considered period of time. The expert system models these constraints that emerged during the requirement analysis phase, e.g.: • • • •
A physician can work only one shift per day. A physician assigned the night shift must not be assigned further shifts on the next day and the day after that. A physician who worked on a particular bank holiday (e.g.: New Year’s Eve) will not be expected to work on the same day of the following year. Some physicians may be exonerated from working night shifts
Among the different strategies that can be used to compute the schedules within CLIPS, our module uses the Simplicity and Random ones, that in the performed experiments produce better results in the distribution of physicians in the resulting schedule. Simplicity means that a rule is chosen depending on the number of facts necessary for it to be satisfied: less is better. Random, as the name implies, means that rules with the same salience value will be chosen randomly. The execution of a rule may change the list of applicable rules by adding or removing facts. Therefore, after a rule is executed, the environment selects the next applicable rule, until no more remain. 4.2 The Information Visualization Module The information visualization module is the result of a series of refinements which contributed to its overall improvement. This was obtained through various iterations of paper prototypes. Developing paper prototypes before any implementation phase is fundamental in the software life-cycle. Through paper prototypes it is possible for final users to experience the interface before it is actually implemented. Should problems arise, they can be fixed and the interface modified in a quick and cheap way, according to the user’s feedback and criticism. For our system, we built several paper prototypes before getting to the final interface. Each prototype was discussed with the physicians at the hospital in order to gain their feedback about our proposals. One of the main improvements was the introduction of a data entry interface (see Fig. 1.) that facilitated physicians in entering personal information and adjusting their preferences and requirements for the schedule generation. For example, in order to specify working preferences, the schedule manager simply has to click on the corresponding colored buttons. The feedback of this action is immediately observable visually in the interface. From the interviews conducted after examining paper prototypes, we were able to gather some interesting insights: physicians appreciated the feature which let them have an overview of schedules (i.e., yearly) rather than shorter time periods (i.e., monthly or weekly).
444
C. Ardito et al.
4.3 A Walkthrough in the System In this section we describe a walkthrough of the information visualization module in order to describe how the schedule manager inserts data for starting the schedule process (data-entry interface), and how he interacts with the scheduling interface. Data-entry interface As discussed earlier, the scheduling process starts by requiring some data from the schedule manager. The system shows the interface in Fig. 1. The user indicates which period he may want to generate schedules for. By using the two colored bars (referred to as “Preferenze Generali”, Italian for “General Preferences”) in the upper part of the interface, the operator may insert the preferences of each physician. General preferences concern which days of the week (and in which shifts) a physician would like to work or those to avoid working. The operator may assign different preference levels on each item by clicking the respective colored button. For example, for each day the operator can select the green button for expressing the best preference, orange for neutral, or violet to express that the physician dislikes to work in that day. Similarly, he can indicate the preferred shift among morning, evening and night. In Fig. 1 the physician, whose name is highlighted in the leftmost column, would like to work on Wednesday, Friday and Saturday (green colored), avoiding Tuesday, Thursday and Sunday (violet colored); about the shifts, he would like to work in the afternoon. The bottom of the screen (referred to as “Preferenze Dettagliate”, Italian for “Detailed Preferences”) allows the operator to assign to specific days preferences about the
Fig. 1. The data entry interface
An Information Visualization Approach to Hospital Shifts Scheduling
445
desired shift or time off. For example, the operator can specify that a physician would like to have a week off in this month. After having inserted the desired information about general and/or detailed preferences for the physicians to be considered, the schedule generation process may begin. Scheduling interface The monthly scheduling, computed by the expert system module, is visualized in Fig. 2. It allows the schedule manager to compare each physician’s schedule to those of colleagues of the same Operative Unit. The example shown in Fig. 2. refers to the October 2008 schedule. Each row refers to one of seven physicians belonging to the Operative Unit, while the columns represent days. By using colors and labels we are able to codify different activities the physician has to carry out during each day. For example, on October the 30th, the light blue icon with an “A” inside indicates that the first physician, whose name is Arcangelo Clemente, must work in the surgery unit of the hospital, while the fifth physician, Anna Mele, must work during the evening shift, as shown by the blue icon. In the monthly visualization, physicians are not interested in seeing the schedules belonging to physicians outside their unit in this interface; instead, they prefer to reserve this feature for the yearly overview, where the comparison between different Operative Units was felt to be important. In the yearly visualization, as it can be seen in Fig. 3, there is a row for each month and they are aligned depending on the first day of the month. In the shown example, August (abbreviated AGO, for Agosto, in Italian) starts on Tuesday (abbreviated MA, for Martedì, in Italian) while September (SET) starts on Friday (GIO) and so on. Each row is further divided in as many columns as there are days in that particular month. Each day is then divided in three further rows which represent the three different shifts there are in a given day: morning 8 a.m. – 2 p.m., evening 2 p.m. – 8 p.m., night
Fig. 2. Monthly overview
446
C. Ardito et al.
8 p.m. – 8 a.m.. Thanks to this visualization, holidays can be immediately spotted. In fact, physicians strongly stressed the need to highlight holidays and holiday eves. From our interviews it came to light the need of introducing the so called “totalizers”, that is, counters which represent the sum of the total amount of shifts performed by each operative unit. These are displayed as an optional sliding panel on the right, which displays the totals both as a number and as a proportional bar. If the schedule manager is not satisfied with the computed output, or needs to adjust some last minute details of the schedule, he may customize the results. For example a problem has occurred with the first physician: he cannot work on Wednesday. The operator chooses him and selects “Modify” from the menu. The data entry interface will be displayed showing the current schedule for the chosen physician. The operator can now insert the new preference (i.e.: avoid working on Wednesday) and the overall schedule will be recomputed to satisfy the updated situation.
Fig. 3. Yearly schedules overview
5 Conclusions In this paper we presented a system which supports staff shift scheduling. It is a challenging activity, because several aspects must be considered and it is even harder to satisfy all the constraints. Algorithms that automatically generate such schedules do not satisfy people in most cases, especially when last minute modifications are necessary, as it is often the case in hospitals. It is useful to employ a computer system able to generate schedules. The approach adopted in this paper combines an expert system
An Information Visualization Approach to Hospital Shifts Scheduling
447
that generates the schedules on the basis of rules specific for the hospital with Information Visualization techniques in order to give users the possibility to drive the decision making process and possibly perform the necessary adjustments. Since the early stages of system design, we have involved physicians of a paediatric hospital. Thanks to the feedback received during the various steps of the development, including the revisions of paper prototypes, the system resulted being usable by the schedule managers. Among the system features, the interactive overview was much appreciated because it represented an innovative idea, since users were accustomed to working with paper schedules. The rapidity with which users are able to deal with several unexpected situations gives the scheduling system an edge over old-fashioned manual approaches. These advantages come from the adopted visualization techniques, which enable users to quickly understand the status of a particular schedule, just by looking at the visualization. Acknowledgments. This work was partially supported by University of Bari (“ex60%” funding) and by EU and Regione Puglia under grant “DIPIS”. We like to thank Ass. Prof. Stefano Ferilli for his valuable collaboration and the students of the Computer Science curriculum who participated to the development of the system.
References 1. Aiken, L.A., Clarke, S.P., Sloane, D.M., Sochalski, J.A., Silber, J.H.: Hospital nurse staffing and patient mortality, nurse burnout, and job dissatisfaction. J. of the American Medical Association 288(16), 1987–1993 (2002) 2. ISO 13407, Human-centered design process for interactive systems. International Organization for Standardization (1998) 3. Burke, E.K., De Causmaecker, P., Van Den Berghe, G., Van Landeghem, H.: The State of the Art of Nurse Rostering. J. of Scheduling 7(6), 441–499 (2004) 4. Ernst, A.T., Jiang, H., Krishnamoorthy, M., Sier, D.: Staff scheduling and rostering: A review of applications, methods and models. European J. of Operational Research 153(1), 3–27 (2004) 5. Eveborn, P., Rönnqvist, M.: Scheduler – A System for Staff Planning. Annals of Operations Research 128(1), 21–45 (2004) 6. Ghosh, B., Cruz, G.: Nurse requirement planning: a computer-based model. J. of Nursing Management 13(4), 363–370 (2005) 7. Beliën, J., Demeulemeester, E., Cardoen, B.: Visualizing the Demand for Various Resources as a Function of the Master Surgery Schedule: A Case Study. J. of Medical Systems 30(5), 343–350 (2006) 8. DRoster, http://www.kappix.com/ 9. Planning PME, http://www.planningpme.com/ 10. Ardito, C., Buono, P., Costabile, M., Lanzilotti, R.: Two Different Interfaces to Visualize Patient Histories on a PDA. In: MobileHCI 2006, Espoo, Finland, pp. 37–40. ACM Press, New York (2006) 11. Costabile, M.F., Fogli, D., Lanzilotti, R., Mussio, P., Piccinno, A.: Supporting Work Practice through End-User Development Environments. J. of Organizational and End User Computing 18(4), 43–65 (2006) 12. JClips, http://sourceforge.net/projects/jclips/
Abstract. This paper uncovers issues in the design of camera-based technologies to support retail shopping in a physical store, specifically clothes shopping. An emerging class of technology is targeting the enhancement of retail shopping, including the trying on of clothing. Designing such systems requires careful considerations of physical and electronic design, as well as concerns about user privacy. We explore the entire design cycle using a technology concept called the Responsive Mirror through its conception, prototyping and evaluation. The Responsive Mirror is an implicitly controlled video technology for clothes fitting rooms that allows a shopper to directly compare a currently worn garment with images from the previously worn garment. The orientation of images from past trials is matched to the shopper’s pose as he moves. To explore the tension between privacy and publicity, the system also allows comparison to clothes that other people in the shoppers’ social network are wearing. A user study elicited a number of design tradeoffs regarding privacy, adoption, benefits to shoppers and merchants and user behaviors in fitting rooms. Keywords: Ubiquitous computing, pervasive, ambient intelligence, retail technologies, privacy, online social networks, fashion.
technologies are providing new capabilities to supplement the physical shopping experience. In addition to displaying conventional product information found on the web, these technologies also augment a human’s assessment of the more subjective, perceptual properties by capturing the shopper’s experience for use as a memory aid and as an artifact for sharing with friends and family. This paper explores the design considerations for creating interactive technologies for clothes fitting rooms using the design and evaluation of a prototype technology called the Responsive Mirror (Figure 1) [35, 36, 37], which is an implicitly controlled video technology for clothes fitting rooms that allows a shopper to directly compare a currently worn garment with images from the previously worn garment. The orientation of images from past trials is matched to the shopper’s pose as he moves. To explore the tension between privacy and publicity, the system also allows comparison to clothes that other people in the shoppers’ social network are wearing.
Fig. 1. The Responsive Mirror concept
2 Related Work Clothing stores often provide an area for trying on clothes. Generally, there is a private place in which to change clothing, which we refer to as a changing area, and an area with a mirror to view the fit, which we refer to as a fitting area. In this paper, we consider these areas separately, but they are sometimes combined in one place. The fitting-area mirror is a specific point of customer interaction with products where a shopper selects items to purchase. Marketers have recognized this “point of decision” as an opportunity to engage the customer with supplemental information.
450
B. Begole et al.
2.1 Clothes-Fitting Technologies Technologies in Fitting Rooms: A Prada boutique in New York contains a sophisticated dressing room designed and constructed by Ideo and IconNicholson [16]. A scanner identifies each garment as the shopper takes it in, providing additional information about the garment’s price and alternate colors and sizes – the kind of information shoppers can find when browsing products online. The fitting room also contains a motion-triggered video camera that records the shopper and plays back the video after a pause. One of the central components of the system is the Magic Mirror which allows a person trying on clothes to send video of himself to friends who can send back comments and vote (thumbs up/down) [21]. The Magic Mirror can also project a static image of an alternate garment onto the mirror allowing the person to see roughly how it might look on him. The Magic Mirror has also been deployed in Bloomingdales. Similar systems have been created and deployed for market evaluation in a number of other retailers [34]. The Gardeur Shop in Essen Germany created an RFID-based system that provides product information including price, available sizes and colors, material and care instructions. Warnaco created a system to show fitting information for women’s braziers. Poggi reports other retailers employing non-sensor technologies in fitting rooms, such as Metropark, Nordstrom, Kira Plastinina and Macy’s. Virtual Fittings: Some online services, such as Intellifit [14] and MyShape [19], take measurements of a customer’s body and suggest clothes that will fit and provide a flattering appearance for that body shape. These services have been deployed in stores such as Levi’s and Charming Shoppes in Ohio, Florida and Texas [34]. Other web services such as MyVirtualModel [20], Knicker Picker [17], and 3Dshopping [1] provide a set of predefined body types that allow a shopper to view the fit of clothes on their selected body type. It is also possible to use augmented reality techniques to map an image of a garment onto a captured image of the customer that will change shape in real time as the customer moves in a virtual mirror [8]. In essence, these virtual fitting technologies are comparable to seeing clothing on a personalized mannequin. However, just as one needs to try on clothes seen on a mannequin, a customer must try the garment on his own body to get the full experience of feel, drape and look. Although virtual fitting technologies continue to advance, Protopsaltou et al. report that reliance on such technology often results in unexpected fit or color, and unpleasant feel of the fabric [25]. Such technologies also do not cover the full range of ways in which a person can wear a garment. A buttoned down shirt, for example, can be worn buttoned or unbuttoned, tight or baggy, tucked or untucked, tied at the bottom, and sleeves rolled to different lengths. Exploring this variety of options ultimately requires physical control of the clothes. 2.2 Reactive Displays The ability to change information in reaction to the presence and motion of people has been demonstrated in several prior systems. Haritaoglu and Flickner describe a prototype that uses computer vision to count the people looking at a display and to infer demographic information [13]. There are some commercial advertising technologies
Designed to Fit: Challenges of Interaction Design
451
(Reactrix [26], Freeset Human Locator [10], P.O.P. ShelfAds [24]) that detect presence and motion of people and change the content of the display in reaction. These reactive displays do not display images that mirror the image or motion of the person looking at the display as the Responsive Mirror does. 2.3 Privacy Concerns from Cameras in a Fitting Room A common concern that sensor technologies face is how they affect a user’s sense of personal privacy. Will users accept a camera or other sensors into a traditionally semiprivate space such as a fitting area? What concerns would they have and what measures should the system design incorporate to mitigate such concerns? In the press articles, privacy concerns are sometimes raised but rarely explored in depth. People are subjected to video capture on a daily basis, sometimes happily (home movies, video conferencing, camera-based game controllers) and sometimes without much choice (store surveillance, toll collection, traffic cameras). Such applications have achieved at least some level of tolerance, yet debate continues about the overall effects of sensing technologies on the long-term health of our society. Grudin [12] observes that collecting and distributing such data electronically heightens concerns of being susceptible to undesirable uses. A basic framework of feedback and control of image capture was outlined by Bellotti and Sellen [5], which systems such as the Responsive Mirror commonly employ. Nevertheless, when a camera or other contextsensing technology is introduced into an unexpected situation, such as a fitting room, we need to examine the potential impact on privacy. Palen and Dourish [22], synthesizing Irwin Altman’s research of face-to-face interaction [2], describe the boundaries and tradeoffs between publicity and privacy that people navigate under technology-mediated interactions. Systems such as the Responsive Mirror introduce new capabilities that affect people’s expectations under the three boundaries of their framework, described next. Also relevant is the work of Petronio [23] who examines a multitude of publicity/privacy boundaries and the varying permeability of the boundaries. We describe below the potential encroachments across these boundaries that camera technologies create. Disclosure: People naturally have an expectation that a fitting room is a private place where they can change clothes without being seen. Although the Responsive Mirror cameras do not capture images of people changing clothes in the changing area itself, only in the fitting area outside, an individual may still have some concern of undesired disclosure of potentially embarrassing interactions in the mirror. In addition to the awkward poses that a shopper may take to test the fit of clothing, a shopper naturally expects that she can experiment with clothes styles in which she might not like to be seen for any number of reasons: fit, style, status, expressiveness, etc. She may be troubled if the images were seen by other people. On the other hand, a shopper interacting with Responsive Mirror may not desire that he fittings be completely undisclosed. She may want to show clothes to friends, or publish them to a social fashion network. Identity: The Responsive Mirror presents images of the shopper as well as others while making decisions about clothing that will become part of his presentation of self. The ability to see images of others provides a means for a person to test the fit of
452
B. Begole et al.
his presentation of self among the presentations of a community. He can examine styles that others are wearing and choose where he would like to fit. On the other hand, he may not desire to be seen as actively positioning himself along such dimensions, perhaps preferring to present a nonchalant attitude about fashions. Thus, a record of his examinations of the images of other people could be sensitive. Temporal: Shoppers today expect that an experimental fitting is transient, not lasting for others to see at another time. In addition to the obvious concern that a shopper might have about recoding images in an embarrassing outfit, images in flattering outfits could become a concern in the future as style and tastes change. While a flattering garment may be an excellent choice at the moment, it could come to represent something contrary to the shopper’s style in the future. This concern can be mitigated by providing the ability to delete regrettable images, but the damage cannot be wholly undone as the images may have been stored in an Internet archive. Furthermore, simple deletion is possibly more drastic than people would desire. The temporal boundary is one that has a varying permeability over time and people might prefer simply that it become more difficult, but not impossible, to see older images. Social Aspects of Fashion Decisions: Another under-explored consideration for such technologies is the social aspects of shopping and fashion. Only the Magic Mirror begins to provide support for social interactions by employing mechanisms for synchronous image sharing and voting by remote friends or family. These capabilities are only the tip of the iceberg regarding social dynamics of shopping and clothing. What kind of social impressions are shoppers concerned about? How do such systems support exploration of the “language” of clothing? When buying clothes, shoppers often assess how others would perceive a particular item, that is, the “fashion statement” she is making. Indeed, people often ask companions or store assistants how they look to get this kind of feedback. Although evaluating one’s self image is an important part of clothes-buying decisions, shoppers in a store are provided very little information about what other people are wearing, other than what is presented in advertisements. Outside of the store, people find inspiration from others and from fashion media, some of which today contain photographs of amateur people. Sociologists have long explored the multiple roles that fashion plays in society. In a synthesis of more than 200 sources, Davis [7] examines the social construction of meaning in fashion and how an individual’s choices communicate social status, gender identity, sexuality, and conformity among other characteristics. Clothing is undoubtedly a form of communication, as declared famously by Lurie [16] and examined in depth by Barnard [4], but Davis believes it is a code of “low semanticity” because the meanings differ with time, trends and among different individuals. Although technologies cannot interpret fashion, they can mediate human discourse on the topic. We see this happening today in online social fashion networks, such as ShareYourLook [28] and IQONS [11, 15]. These sites allow members to upload photographs of themselves wearing various outfits and they can categorize, tag and comment on the images. The services merely provide scaffolding; the members construct the semantics through tagging and commentary. Providing access to a shopper’s social fashion network during a shopping experience would provide access to the state of fashion as constructed by the shopper and her online community.
Designed to Fit: Challenges of Interaction Design
453
3 Formative Evaluation of Responsive Mirror Our prototype was designed to test assumptions about the usefulness of the new capabilities and to draw out lessons for future design. First, will shoppers benefit from being able to directly compare their clothing alternatives in a physical shopping experience? Second, will shoppers find value from images of other people wearing similar and different clothes? We also wanted to learn which, if either, feature would shoppers prefer, what uses they would make of them and what other capabilities they would desire. We considered conducting field observations in an actual store’s fitting room but determined that before doing so we needed to gain some understanding of the privacy implications of the camera. Therefore, in this initial study, we observed people’s behavior using the prototype in our lab. We recruited 12 male participants who wear medium to large men’s shirt size in the age range of 28-52 years old. We were only able to use male participants because the prototype’s body segmentation and clothes detection algorithms are initially targeted for men’s shirts because there is less variance among men’s shirt varieties than among women’s. We recognize that men are only a sub-population of the eventual user population but testing on this demographic allowed faster iteration of the technology-design cycle. A limitation of this study is that it used participants of only one gender (male). Thus, it is not clear to what extent the same phenomena would be observed with participants of the other gender (female), particularly regarding questions of privacy and social effects. The experiment consisted of three conditions. 1. Mirror: Conventional mirror alone. 2. Previous Outfit: Conventional mirror with a display on the left showing an image of the person from the most recent previous fitting. The orientation of the person in the image matches the orientation of the person in realtime. 3. Other People: Conventional mirror with a display on the right showing four images of other people. For each condition, participants put on six different shirts consisting of three collared polo-style shirts and three crew neck shirts of a mix of colors. The price of the shirt was removed. There were three sets of six shirts and there was a comparable mix of colors and patterns across the sets. The order of conditions and shirt sets was counter-balanced using a latin-square three-by-three design. 3.1 Task and Procedure Participants were told to imagine themselves in a scenario where they need to decide which, if any, of the six shirts they will buy. To eliminate price as a factor in their decision, participants were told to imagine that they can afford as many shirts as they like and would consider the prices to be reasonable in all cases. For each shirt, the participant stood approximately 250 cm (8.2 ft) in front of a fulllength mirror (165cm (65 inches) high by 73cm (29.5 inches) wide). Participants were asked to view the shirt from multiple angles in a natural way. After trying all six shirts per condition, the participants were asked to fill out a questionnaire measuring how appealing they found the shirt and how likely they are to buy the shirt on a 5 point
454
B. Begole et al.
scale. They were also asked questions about the condition they had just experienced. After all three conditions were complete, the participants filled out a questionnaire with questions spanning the conditions and potential privacy issues. 3.2 Results To examine whether the display conditions affected participants’ perception of the appeal or desire to purchase any of the shirts, we conducted two repeated-measures ANOVAs with Display Condition as the within-subject variable, and the appeal and purchase variables as dependent variables in turn. Neither the ratings of the desire to purchase nor appeal of the shirts were significantly different among the conditions (respectively F[2, 110] = .42, p = .66 and F[2, 110] = .32, p = .73). We asked participants to rank their preference for the three conditions plus a configuration using all three components (which they did not experience). To test whether participants ranked these four configurations differently, we performed a Friedman test on the ranked values within subjects. There was a significant difference in the rankings (χ2 = 9.10, p = .03), where the preference (from strongest to weakest) were: Previous Outfit plus Other People (M = 1.92) > Previous Outfit (M = 2.00) > Other People (M = 2.83) > Plain Mirror (M = 3.25). And finally, we asked participants how much the Previous Outfit and Other People conditions enhanced their shopping experience. Participants rated the Previous Outfit (M = 3.00, SD = .85, where 5 = Extremely Helpful and 1 = Not Helpful At All) as having enhanced their experience more than the Other People (M = 2.50, SD = .52) condition, which is in line with the ranked measures. A paired-samples t-test showed the ratings to be significantly different between the conditions (t[11] = -2.57, p = .03). 3.3 Results from Privacy Related Questions Regarding the issues of disclosure, we asked participants how much it would bother them if someone from same or opposite gender and specific social group (family, friends, co-workers, and stranger) viewed images of them that were captured by the system. They specified their response on a five point scale (5=bothers me a great deal, 1=doesn’t bother me at all). The responses were not significantly different according to the gender of who might see the image, so we calculated the mean for each social group across gender. The means were Family = 1.08 (SD =.19), Friends = 1.50 (SD = .71), Stranger = 2.08 (SD = 1.06) and Co-worker = 2.25 (SD = 1.14). We performed a repeated-measures ANOVA and found there was a significant difference among the social groups (F[3,33] = 5.76, p = .003). Post-hoc contrasts revealed that Family is not significantly different from Friends. and that CoWorkers is not significantly different from Strangers. Family and Friends are significantly different from CoWorkers and Strangers (p = .01). The frequency distributions for all groups have a mode of 1 (doesn’t bother me at all) except Coworkers which is bimodal with modes at 1 (doesn’t bother me at all) and 3 (bothers me somewhat). We asked participants how much it would bother them on the same 5 point scale if other people saw images of them wearing a shirt that looks good versus a shirt that looks bad on them. Participants rated their level of concern significantly higher for bad shirts (M=3.0) versus good shirts (M=1.42) (p = .001).
Designed to Fit: Challenges of Interaction Design
455
Regarding the aspects of personal and group identity, we asked participants how often (5=Always, Often, Sometimes, Seldom, Never=1) they think of someone else who might like the clothes they are trying on. The mean response was 2.67 (SD=0.98). We asked participants how often they think about how similar the clothes are to what other people they know and don’t know are wearing. The mean for people they know was 2.92 (SD=0.9) and for people they don’t know was 2.33 (SD=0.98) with no significant difference between them. Participants responded with a mean of 3.6 (SD=1.07) to the question of how often do they consciously consider how others will perceive them in the clothes they are trying. Regarding the issues of temporality, we asked participants if their tastes changed in the future, would they want to remove images of themselves wearing a contrary style. On a five point scale (5=Definitely, 1=Definitely Not), participants responded with a mean of 3.08 (SD=1.16) (closest to Possibly). We asked participants at what period of time would they consider removing images of themselves that they had allowed others to see. The highest number of responses was for 3 months (5 participants) and the distribution of the remaining responses was spread across times within 1 year (5 participants) and never (2 participants). These responses can guide the points in time at which to remind users of the existence of past images.
4 Lessons for Design Our experience with the Responsive Mirror uncovered some unexpected issues in the design of user experience in this domain. Because of the error rates of sensing technologies, designers need to anticipate variance in the user experience. Our prototype used a straightforward color-histogram comparison from a relatively small data set and yet we found that sometimes the system retrieved different matches for different participants wearing the same clothes. Although this inconsistency was not noticed by the individual participants, designers need to be aware of potential variance in system behavior and avoid designs that expect a deterministic result. Designers of pervasive technology must be aware of the ways that information systems such as these are affected by our notions of personal privacy. We already see that the increasing volumes of social content creation, sharing and retrieving are modifying societal norms. Designers of retail systems should shape the system to provide benefits that balance, or potentially outweigh, the risks to privacy in terms of disclosure, identity and temporal boundaries. Our experience indicates that some users are receptive to a system that bundles the capture of potentially sensitive images with the benefit of seeing images of self and others. Perhaps most important to merchants, our results showed no difference in participants’ perceptions of their likelihood to buy a shirt after using our prototype. The quality of the product matters more than the technologies used to evaluate it. Nonetheless, participants did perceive the system to have been “helpful” to their buying decision, providing value that merchants should be able to capitalize on. Pervasive technologies such as the Responsive Mirror change the clothes-buying experience by connecting information between our physical lives and online digital identities. Currently these realms are artificially separated in a false dichotomy of
456
B. Begole et al.
“virtual” versus “real” life. Context-sensing technologies enable the design of applications such as Responsive Mirror that not only augment the immediate physical experience but also integrate information across the multiple aspects of one’s social life. As we see increasing spread of such technologies, designers need to be informed of the unique considerations that must be made in these environments, particularly with respect to privacy and social goals.
References 1. 3DShopping.com, http://www.3dshopping.com 2. Altman, I.: The Environment and Social Behavior: Privacy, Personal Space, Territory and Crowding. Brooks/Cole Pub. Co., Inc., Monterey (1975) 3. Anguelov, D., Lee, K., Gokturk, S.B., Sumengen, B.: Contextual identity recognition in personal photo albums. Computer Vision and Pattern Recognition (CVPR) (2007) 4. Barnard, M.: Fashion as Communication, 2nd edn. Routledge (2002) 5. Bellotti, V., Sellen, A.: Design for privacy in ubiquitous computing environments. In: Proceedings of the European Conference on Computer-Supported Cooperative Work, ECSCW, pp. 77–92. Kluwer Academic Publishers, Norwell (1993) 6. Brown, J.: Prada Gets Personal, BusinessWeek. The McGraw-Hill Companies, March 18 (2002), http://www.businessweek.com/magazine/content/02_11/ b3774612.htm 7. Davis, F.: Fashion, Culture and Identity. University Of Chicago Press (1994) 8. Ehara, J., Saito, H.: Texture overlay onto deformable surface for virtual clothing. In: Proceedings of the 2005 international Conference on Augmented Tele-Existence ICAT 2005, vol. 157, pp. 172–179. ACM Press, New York (2005) 9. Ehrlich, S.F.: Strategies for encouraging successful adoption of office communication systems. ACM Trans. Inf. Syst. 5(4), 340–357 (1987), http://doi.acm.org/10.1145/42196.42198 10. Freeset Human Locator, http://www.freeset.ca/locator/ 11. Geller, M.: Friendly fashion — Cyber-style!, Reuters (April 28, 2007), http://blogs.reuters.com/2007/08/28/ friendly-fashion-cyber-style/ (Last accessed January 29, 2009) 12. Grudin, J.: Desituating Action: Digital representation of Context. Human-Computer Interaction 16(2-4), 269–286 (2001) 13. Haritaoglu, I., Flickner, M.: Attentive billboards. In: Proceedings of the Conference on Image Analysis and Processing 2001, September 26-28, 2001, pp. 162–167. IEEE Press, Los Alamitos (2001) 14. Hilsmann, A., Eisert, P.: Deformable object tracking using optical flow constraints. In: 4th European Conf. on Visual Media Production, 2007, November 27-28, 2007. IETCVMP, pp. 1–8 (2007) 15. Intellifit, http://www.intellifit.com/Intellifit/Home.aspx 16. IQONS, http://www.iqons.com/ 17. KnickerPicker.com, http://www.knickerpicker.com/ 18. Lurie, A.: The Language of Clothes, Random House (1981) 19. Like.com, http://www.like.com/ (Last accessed September 8, 2007) 20. Lou, H., Luo, W., Strong, D.: Perceived critical mass effect on groupware acceptance. Eur. J. Inf. Syst. 9, 91–103 (2000)
Designed to Fit: Challenges of Interaction Design
457
21. MyShape, http://www.myshape.com/ 22. My Virtual Model, http://www.mvm.com 23. Nanda, S.: Virtual Mirrors. Reuters, January 29 (2007), http://www.reuters.com/news/video/videoStory?videoId=5219 24. Palen, L., Dourish, P.: Unpacking "privacy" for a networked world. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI 2003, pp. 129–136. ACM Press, New York (2003), http://doi.acm.org/10.1145/642611.642635 25. Petronio, S.: Boundaries of Privacy: Dialectics of Disclosure. State University of New York Press, Albany (2002) 26. P.O.P. ShelfAds, http://www.popbroadcasting.com/main/intilashelf. html 27. Protopsaltou, D., Luible, C., Arevalo, M., Magnenat-Thalmann, N.: A body and garment creation method for an Internet based virtual fitting room. In: Computer Graphics International 2002 Conference Proceedings, pp. 105–122. Springer, Heidelberg (2002) 28. Reactrix, http://www.reactrix.com/advertisers_details.php?id=9 29. RockYou, http://www.rockyou.com/ (Last accessed September 8, 2007) 30. ShareYourLook, http://www.shareyourlook.com/ (Last accessed September 3, 2007) 31. Song, Y., Leung, T.: Context-aided human recognition – clustering. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3953, pp. 382–395. Springer, Heidelberg (2006) 32. Weiser, M.: The computer for the 21st century. Scientific American 265, 94–104 33. Staedter, T.: Smart’ Fitting Room Suggests What to Wear. Discovery News, January 23 (2008) 34. Poggi, J.: Dressing Rooms Of The Future, Forbes.com, July 22 (2008) 35. Chu, M., Begole, B.: Human-Centric Interfaces for Ambient Intelligence. In: Aghajan, H., Delgado, R.L.-C., Augusto, J.C. (eds.) Natural and implicit information seeking cues (to appear) 36. Zhang, W., Matsumoto, T., Liu, J., Chu, M., Begole, B.: An intelligent fitting room using multi-camera perception. In: Proceedings of the 13th international Conference on intelligent User interfaces (IUI 2008), pp. 60–69. ACM, New York (2008) 37. Zhang, W., Begole, B., Chu, M., Liu, J., Yee, N.: Real-Time Clothes Comparison Based on Multi-View Vision. In: Proceedings of ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC 2008), September 7-11 (2008)
Usability for Poll Workers: A Voting System Usability Test Protocol Dana Chisnell1, Karen Bachmann2, Sharon Laskowski1,3, and Svetlana Lowry3 1
UsabilityWorks, 453A Chestnut St., San Francisco, CA 94133 [email protected] 2 Seascape Consulting, Inc. 13911 W Hillsborough Ave, Tampa, FL 33635 [email protected] 3 National Institute of Standards and Technology 100 Bureau Drive, Gaithersburg, MD 20899 sharon.laskowski, [email protected]
Abstract. In this paper, we discuss our efforts to develop a repeatable test protocol for assessing usability for poll workers—temporary election officials who ensure secure and private voting in voting places. The research described in this paper is part of a larger effort to develop a standard for voting systems. This is the first time that formal and substantial usability requirements as part of a standard for voting systems have been established in the United States. The standard includes requirements for poll worker usability and associated test methods to assess whether a system meets these requirements. The test method described in this paper sets up a protocol and pass/fail criteria for assessing the usability of voting system documentation for poll workers. Keywords: usability testing, pass/fail criteria, elections.
1 Background While much research on elections has concentrated on voting and the voter experience, usability for poll workers is equally important. Poll workers spend extremely long hours for very little pay on Election Day setting up voting stations, making sure the equipment is operating properly, assisting voters, and shutting down the voting stations when the polls close. Along the way, they often encounter problems with voting systems that delay or even prevent voters from voting or cause votes to go uncounted. 1
This document describes research in support of test methods and materials for the Election Assistance Commission's next iteration of the Voluntary Voting System Guidelines (VVSG). It does not represent a consensus view or recommendation from National Institute of Standards and Technology (NIST), nor does it represent any policy positions of NIST. Certain commercial entities, equipment, or material may be identified in the document to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that these entities, materials, or equipment are necessarily the best available for the purpose.
Usability for Poll Workers: A Voting System Usability Test Protocol
459
The purpose of the research described in this paper is to explore how to perform usability testing to ensure that voting systems achieve an acceptable level of usability for poll workers, meaning that they can complete typical poll worker tasks by following instructions in voting system documentation. This work will form the basis of a repeatable test protocol to be employed for uniform testing of voting systems. Other research is being conducted to learn about usability for voters. In the United States, under the auspices of the U. S. Election Assistance Commission (EAC), a standard for voting systems has been developed called the Voluntary Voting System Guidelines (VVSG)2. The National Institute of Standards and Technology (NIST) is currently developing test methods for determining if a voting system complies with the requirements in the VVSG. A significant set of usability requirements is part of the VVSG and, in particular, there are requirements for usability for poll workers. Test laboratories have been designated to test voting systems to determine if they meet the standard and, if so, the EAC certifies the systems. Elections are run by the States. The States are encouraged to purchase systems that have been certified. For more information about the EAC, see http://www.eac.gov. For more information about NIST’s work related to voting, see http://vote.nist.gov. The VVSG can be found at http://vote.nist.gov/vvsg-report.htm. It is undergoing a public review process; its current status at http://www.eac.gov. This is the first time that formal and substantial usability requirements as part of a standard for voting systems have been created in the United States. It is critical that the test methods for determining conformance to the standard include usability testing in addition to checking for adherence to design requirements through review. It is important to confirm that poll worker interaction with the system is effective. (There is other research looking at voter interaction.) This can only be judged by observing and measuring the voting system interface performance with actual users performing typical election tasks. Further, because of the certification process, the test methods employed by the test laboratories must each have a pass/fail criterion: a voting system is certified if it meets all the VVSG requirements by passing all the associated test methods. The goal of the research described in this paper is to establish a test protocol to evaluate usability for poll workers. The challenges are that the protocol must y y y y y
apply to the testing of any voting system include clear pass/fail criteria be repeatable in any test laboratory be feasible in time and cost ensure that a successful outcome demonstrates an acceptable level of usability.
2 Poll Worker Usability Needs The usability of both the documentation and the voting system itself affect poll workers. The VVSG reflects this, saying that procedures should be “reasonably easy” to “learn, understand, and perform,” and that voting system documentation should include “clear, complete, and detailed instructions” for poll workers. 2
In this paper, “VVSG” refers to the VVSG Recommendations to the EAC, August 31, 2007.
460
D. Chisnell et al.
Clearly written instructions are essential to usability. Most poll workers get training provided by the election jurisdiction. That training, as well as any troubleshooting needed on Election Day, ultimately relies on information that comes from documentation supplied by voting system manufacturers. Even with training, poll workers have been reported to misinterpret messages from voting systems, thereby mistakenly invalidating ballots; have problems managing accessible voting systems, leaving people with disabilities unable to vote without assistance; or issue the wrong type of ballot— among other issues. The voting system itself, including system messages and troubleshooting documentation, must be usable by poll workers. Also, voting systems and their documentation must support the realistic situation of the polling place on Election Day. In our review of existing voting system documentation supplied by manufacturers, we found that much of it fell short of best practices for technical communication and information design. [1] We focused on evaluating these two aspects of usability of voting systems for poll workers: system documentation and poll workers using the system along with the documentation at the voting place.
3 Designing the Usability Test Protocol To meet the VVSG requirements and to determine whether voting systems and associated documentation are usable for poll workers, we had to design a test method that goes beyond typical usability testing: How do you design a usability test that someone else, whom you don’t know and can’t talk to, will have to conduct? How do you create a usability test that determines whether something meets requirements in the VVSG rather than generating data to use to improve the design of the system? How do you design a usability test for dozens of different systems that all do the same things in different ways? How do you design a usability test of the documentation in conjunction with the operation of those systems? How do you define success or failure for a requirement? To answer these questions, we constructed a usability test protocol that includes pass/fail criteria. We then designed and conducted an exploratory study in which we acted as the test administrators of the protocol. This helped us see how the protocol might work in practice, better understand the context in which the test would be used, and refine our methodology. 3.1 Creating the Initial Protocol We first created a test design. An essential element of the test design was determining how many participants to use. For the exploratory study, we decided on eight participants, who would work in teams of two in each session for four test sessions. While it is common to use individual participants in usability tests, poll workers work in teams on Election Day to do the tasks we wanted to study in our research. It was our hypothesis that four teams would provide enough evidence for a pass/fail determination. We assumed we were looking for usability problems that any usability expert would identify as severe—that is, based on the observed evidence during the test, poll
Usability for Poll Workers: A Voting System Usability Test Protocol
461
workers could not complete the task. We also assumed that this is one of many tests for the many requirements (including other usability requirements) in the VVSG so the cost of the test must be reasonable. We then created a proposed formal testing protocol. A test administrator at a voting system test laboratory will follow the protocol to conduct usability tests with poll workers on voting systems and their documentation. We developed clear criteria for evaluating system operation and its documentation based on the VVSG and best practice in technical communication and information design. If participant pairs were unable to complete tasks using instructions in the voting system documentation, we considered this a failure of the voting system. The hypothesis is that these criteria will ensure consensus among experts observing the same performance. The testing protocol included instructions and a script for the test administrator, and task scenarios for the participants to follow. In addition, we created a pass/fail checklist for the administrator to document the evaluation. These are described below. Script for the test administrator. The script for the test administrator includes specific, detailed instructions for the administrator to follow, so every test session is administered the same way. For example, it includes instructions to the test administrator about where to be in the room in relation to the test participants to get the maximum benefit from observing the participants without being obtrusive. The script includes specific wording for the test administrator to say at each step during the test. The script also describes what the participants should be doing and how to interact with the participants. Task scenarios. There are three tasks for participants to do in this usability test: y Set up the voting equipment to open the polls. y Conduct voting. y Close the polls to end voting. The test protocol includes task scenarios describing a task goal that participants must respond to by performing the tasks while following instructions in the voting system documentation provided. Here the protocol directs the test administrator what to say and do to introduce each task scenario, and shows what the participants should be doing in response to each step. The protocol also describes how to provide assistance to the participants if they cannot continue with a task. Pass/fail checklist. The pass/fail checklist is a one-page form for the test administrator to use during each test session to document the usability evaluation of the voting system operations and documentation. Because the test administrator at the test laboratory will have experience as a usability test moderator, will be familiar with the VVSG and voting systems, and will have knowledge of technical communication and information design, we felt it should take little training or instruction to use the checklist effectively. The instructions could be minimal. Figure 1 below shows an example of the checklist from the exploratory study. In this case, the main criteria may have been met (the X marks in the boxes for
462
D. Chisnell et al.
Fig. 1. The pass/fail checklist in use during a testing session
Task 1 and Task 2), but the test administrator, using usability and domain expertise, could determine that the documentation and system fail to support this team of poll worker test participants because a key element was missing. For example, here the system messages did not clearly convey that the voter may leave some contests unvoted and still have this ballot count. The form includes: • Evidence—Behaviors to look for. For example, test administrators observe for whether “Participants can use the documentation to find the information they need to complete tasks,” or “… read, understand, and react to system messages.” • Criteria—Operationalized principles on which to evaluate usability. For example, “Are the instructions easy to act on;” “Are poll workers able to respond to messages appropriately?” • End states—Descriptions of what the voting system should be doing, should not be doing, or should be showing that indicate that participants have completed each task successfully. For example, the end state of a successful Task 3 is that the machine cannot take any more votes and that system messages indicate the polls are closed. • Final determination—Based on observations of each pair of participants, and taking into account the criteria and end states for each task, the test administrator decides whether the voting system documentation has an acceptable level of usability for poll workers. Acceptable usability means that poll workers could complete the task using instructions in the documentation. This assessment figures into the larger evaluation taken across sessions. • Comments—An area where the test administrator can include reasons for the final determination, including specific usability problems that participants encountered during the test.
Usability for Poll Workers: A Voting System Usability Test Protocol
463
Attending to the combination of evidence, end states, and criteria for all the test sessions, a test administrator makes a determination about whether the usability of the documentation is acceptable and therefore passes. If a pair of participants is unable to complete tasks, this constitutes a failure. If a majority of participant pairs is unable to complete any one of the tasks, then the documentation, and hence the system, fails the test.
4 Testing the Proposed Test Protocol in an Exploratory Study Having drafted a protocol for test laboratories to use, we tested it in an exploratory study to further refine the script, task scenarios, and checklist. We conducted the exploratory study to answer these questions about the protocol: y y y y y
Do the scripts for the test administrator to use during the session work well? Do they produce appropriate behavior from participants? What are the appropriate performance criteria for determining the acceptable usability of the voting system instructions? What should the success/failure criteria be? What data gathering tools or forms should be in the final protocol? Can the voting system test laboratories conduct the sessions in a reasonable amount of time?
First, we conducted several dry runs of the protocol with different voting systems, refining it each time. Next, we conducted day-long sessions at a research facility, asking pairs of participants who were poll workers to work with two different voting systems, one system at a time. Each of four pairs of poll workers worked on a separate day. Each pair of poll workers worked with one system and its poll worker documentation in the morning and a different system and its documentation in the afternoon. We alternated the order of voting systems each day. All of the participants used the same two voting systems. The poll worker participants used the manufacturer-supplied documentation for each system to open the polls, conduct voting, and close the polls. The tasks did not include dealing with security of the systems (except as the systems are delivered by the vendor) because security procedures are different in each jurisdiction and the VVSG has other requirements specifically to evaluate security. If participants encountered problems for which they asked questions, we gave them hints. If they still could not proceed, the administrator gave them direct assistance. We gave participants hints and assistance because we wanted each pair to attempt each task so we could exercise the entire protocol and script. This helped us refine the script and the wording of the task scenarios, and clarify the pass/fail criteria. Assistance with completing tasks is not part of the proposed protocol for voting system test laboratories to use in certification evaluations. Based on interviews with election directors and our own experiences using voting systems while setting up polling, voting, and closing the polls, we estimated that it could take between one and three hours to complete all of the tasks on each voting system. We used two different voting systems in this study. One was an optical scan system in which voters fill in bubbles on a paper ballot and submit that to a ballot
464
D. Chisnell et al.
scanner. The other was a direct record electronic (DRE) voting system with a touch screen interface. Both are currently being used in elections in the United States. Participants were told that they must follow the instructions in the voting system documentation to do the tasks they were given. The documentation was printed on regular paper measuring 8 ½ by 11 inches. It was printed from files delivered electronically by the system manufacturers to NIST in color, double-sided. The pages were bound with large clips. The documentation for one system was 116 pages. The documentation for the other system was a group of short manuals that taken together was 123 pages.
5 Findings from the Exploratory Pilot Study We theorized that using pairs of poll workers as participants would both reflect a realistic situation for using the voting system and generate evidence on which to base pass/fail determinations. We also theorized that developing specific criteria and end states would ensure consensus among experts observing the same performance. In exploring these two issues, we also wanted to learn how many pairs of poll worker participants it should take to determine whether the voting system documentation passed or failed certification. While the draft version of the protocol that we went into the test with worked reasonably well, we iterated improvements to the test design and protocol as we learned from the sessions. We are reasonably confident that the final test kit would work well for test administrators and that ratings would be similar from administrator to administrator as they use the pass/fail checklist. However, the test must be repeated by others who were not involved in designing the test to verify that it is suitable for use by the test laboratories. 5.1 Tasks Took about 90 Minutes The timing of each session with a voting system ranged from 75 to 135 minutes. Most pairs were done with a system in about 90 minutes. There is one time-dependent activity. The first task, opening the polls, should be stopped after one hour. The usual amount of time scheduled for setting up a polling place is typically between 45 minutes and 90 minutes, with one hour being the most common amount of time allotted. The other tasks are not time-dependent in the same way as the first task. They both happen based on a schedule for Election Day that need not figure into this test. 5.2 Five Sessions Should Be Enough to Determine Pass/Fail In addition to the test administrator’s dry run of the protocol with the voting system and its documentation, we recommended five sessions with poll worker participants to: y lessen any effects for any particular pair of participants having issues with working together y minimize any effects for individual poll workers having a bad day y be confident in an expert analysis based on observation of poll worker performance whether the documentation supports poll workers.
Usability for Poll Workers: A Voting System Usability Test Protocol
465
In the exploratory study, we conducted four sessions each on two voting systems. Our experiences in the exploratory study suggest that as few as three sessions would generate enough evidence to pass or fail the voting system. However, conducting at least two additional sessions—for a total of five—should help the test administrator make a sound analysis that should agree with results if the testing was repeated. For example, if a majority of participants across sessions have the same type of problems with responding to messages (or the lack thereof) about undervoted ballots, the consensus across the test sessions would show that the voting system fails the test. 5.3 Using Pairs of Poll Workers Worked Well We hypothesized that because a polling place relies on a group of strangers working together for a day that using pairs of poll workers as participants would work well for this protocol. We expected that one participant would read the documentation while the other tried to perform the steps being read. This was what happened; participants regularly traded the documentation back and forth between them without prompting from the administrator. The arrangement made it easy to observe whether the instructions were supporting the tasks that the poll workers were trying to do. For example, one poll worker might begin reading the documentation in response to a task but not be able to find the information needed, and then they might switch roles. 5.4 Reaching Specific End States Constitutes Successful Completion In typical usability tests, the design team determines successful completion criteria as well as error conditions. While the test we designed does define successful completion criteria—participants must follow instructions in manuals to reach specific end states—it is not designed to gather error data to improve the design of the documentation or the user interface of the voting system. It is specifically designed to determine if poll worker requirements in the VVSG are met. Task completion here is clear: for Task 1, the system must have the polls open and be ready to receive the first ballot; for Task 2, the system must have ballots cast; for Task 3, the system must be unable to accept any more votes. 5.5 Documentation Failure Conditions are Based on Behavior and Performance Evidence We found there are two fundamental ways that the voting system can fail: y Performing the tasks, participants exceed the total allotted time of two hours for the entire test session. y Participants ask for assistance. Many issues may contribute to participants exceeding the time allotted for the test or having to ask for assistance. For example, for Task 1 opening the polls, for most DREs there are two parts: a voting station and a card activator (or a PIN (Personal Identification Number) generator). If either is not set up properly, the system fails to meet the VVSG requirement for that team of test participants for that task. Analysis
466
D. Chisnell et al.
by the usability expert might show, for example, that the instructions have been left out completely, are in the wrong place in the documentation, or are not easily found. For a laboratory test for specific VVSG requirements, we must assume that assistance is not available. If poll workers must get help from sources other than the documentation, it has failed to support poll worker tasks. If a majority of participants are unable to complete any one of the tasks in usability tests conducted by a voting system test laboratory, then the system documentation fails the test. However, results from our exploratory study suggest that additional analysis may be required for the decision to pass or fail the voting system.
6 Conclusions and Future Research Typically in a usability test, the purpose is to find and remedy design problems. However, testing for voting system certification occurs when a voting system is ready to be deployed. Systems are certified if they meet all the requirements of the standard. Conducting the exploratory study allowed us to refine a certification test protocol: we clarified the wording of the tasks for participants, adjusted the format of the protocol, and tightened the level of detail on the pass/fail checklist. The result is a usability test protocol that, based on our exploratory study, is suitable for use by voting system test laboratories to use to determine whether to certify voting systems for use in elections. However, there is some additional research needed to ensure that the test protocol is ready to be added to the set of VVSG test methods. It is critical to have other usability practitioners perform the tests to verify repeatability and confirm that all aspects of the test protocol are clearly defined. The pass/fail checklist was easy for us to use and we had high inter-rater reliability, but we designed it. We have some questions about the process of making the pass/fail determination. For example, does a failure of one task in one session imply failure of the system or should we require a majority of the sessions to show this same task failure to determine whether a voting system fails the test, and hence certification? Finally, while much research on elections has been focused on voting, usability for poll workers is equally important. Poll workers’ answers to questions and knowledge about voting systems (or lack thereof) have determined whether voters can vote. Our protocol for evaluating voting system documentation based on poll worker performance reveals issues that expert review alone does not. We encourage further research focused on poll worker interaction with voting systems. Acknowledgements. Dana Chisnell and Karen Bachmann were funded under NIST contract SB1341-05-Z-0023-67310. The entire team wishes to thank our many reviewers for their valuable feedback on this paper and the many deliverables generated for the research project.
References 1. Chisnell, D.E., Becker, S.C., Laskowski, S.J., Lowry, S.Z.: Style Guide for Voting System Documentation. Improving U.S Voting Systems, http://www.vote.nist.gov/NISTIR-7519.pdf (accessed November 21, 2008)
Usability for Poll Workers: A Voting System Usability Test Protocol
467
2. Dumas, J.S., Redish, J.C.: A practical guide to usability testing. Ablex, Norwood (1993); Rev. Sub. edn. Intellect Ltd. (January 1, 1999) 3. Laskowski, S.J., Redish, J.C.: Making ballot language understandable to voters. In: USENIX/ACCURATE Electronic Voting Technology (EVT) Workshop, Vancouver, BC, August 1 (2006), http://www.usenix.org/events/evt06/tech/tech.html (accessed on December 7, 2007) 4. PlainLanguage.gov. Federal plain language guidelines, http://www.plainlanguage.gov/howto/guidelines/ reader-friendly.cfm (accessed October 15, 2007) 5. Redish, J.C.: Guidelines for writing clear instructions and messages for voters and poll workers. NIST (National Institute of Standards and Technology) (2006), http://vote.nist.gov/032906PlainLanguageRpt.pdf (accessed September 17, 2007) 6. Rubin, J., Chisnell, D.E.: Handbook of Usability Testing. In: How to Plan, Design, and Conduct Effective Tests, 2nd edn. Wiley, Chichester (2008) 7. Skelton, T.M.: Testing the Usability of Usability Testing. Technical Communication, Third Quarter, 343–359 (1992)
CAD and Communicability: A System That Improves the Human-Computer Interaction Francisco V. Cipolla Ficarra1,2 and Rocío A. Rodríguez1 HCI Lab. – F&F Multimedia Communic@tions Corp. ALAIPO: Asociación Latina de Interacción Persona-Ordenador 2 AINCI: Asociación Internacional de la Comunicación Interactiva Via Pascoli, S. 15 – CP 7, 24121 Bergamo, Italy [email protected] 1
Abstract. We present an analysis of communicability methodology in CAD interactive systems, called DOQ (DObby Quality). This methodology has been under development between 2001 and 2008, obtaining excellent results in both educational and productive contexts. In studies where there is a bi-directional interrelation between usability and communicability of technical interfaces for the design of tissues in cotton, linen, etc., they ease the learning process of the designers who use a textile interactive system for the first time. We also present an educational prototype that can adapt easily to the real production of fabric: DobbyCAD. Keywords: Dobby, CAD, Computer Graphics, Interfaces, Communicability, Usability, Education.
CAD and Communicability: A System That Improves the Human-Computer Interaction
469
The excellent results obtained in heuristic assessment have allowed us to establish two modalities of presenting the same information on the screen. These two modalities are essential for the potential users since it allows them to simulate real fabrics in little time. This important option does not exist in the other assessed systems. Besides, the results of research with real users of CAD systems has made it possible to make a simulator where it is feasible to have on a same screen up to 12 models of fabrics on the same screen that occupies a A4 printer page. Consequently, another advantage of the methodology we present here is saving impression costs on the colour of textile simulations. The beta versions of the product have been experimented successfully in the industrial context and the current final version has been positively assessed by textile design studies, in university teaching centres, etc. The communicability assessment techniques have made it possible to develop an effective emulation system which adapts easily to different kinds of users. The compatibility with operative systems and hardware is 100%. The used methodology allows one to obtain an excellent quality product whose cost is tantamount to 10% of the price of the current commercial systems aimed at industrial textile design. In the current work a brief statement of the state of the art of the textile industrial situation is made, related to the users and responsible staff of textile design. Additionally, there is a summing-up of the methodology used in the realization of the current system. Later on the main requirements of software engineering are included such as the personalization of the different modules of a CAD/CAM system, the different interfaces of the interactive system. Finally, the importance of the current interactive system is stressed, not only in the production of tissues, but also to insert textures in virtual characters and to increase realism.
2 Design, Textil CAD and Users One of the main problems that is observed with the future industrial tissue designers is that in the technical teaching institutes aimed at textiles, the young users are forced to work with real CAD/CAM industrial systems. Obviously, the teachers from these institutes also teach in universities. However, this teaching staff does not make a differentiation by age, training or previous experience of the potential users of these systems. The main problem lies in the interface of the interactive systems. The commercial textile CAD systems for dobby tissues carry out the same functions more or less. However, through the communicability and usability studies that have been made of them it has been seen that the conception of the system, the mindset of the potential users , is practically equal to nil. That is to say, there is a total lack of communicability of the design, which may entail serious problems in usability. One consequence is the high rate of staff change in these design offices. We can see that educative training is aimed more at the industrial factor than the didactic factor (which is common in many textile provinces/regions of Northern Italy). Additionally, every manufacturer and industrialist wants a special personalization of the CAD software, with which the user requires a long learning period of the software in order to make his textile designs. In our case, the CAD system has been created for the different kinds of users based mainly on the educative aspect and simultaneously in the industrial sector. DobbyCAD is easy to use, but through it you can reach real
470
F.V. Cipolla Ficarra and R.A. Rodríguez
manufacturing of tissues. Currently, computer aided design systems have reached a stage of maturity and consolidation in the set of performances and functionalities they offer to potential users, whether they are experts or not. The industrial sector has a wide range of CAD commercial systems available in accordance with its productive activities and services. A user can assess at the moment of designing whether what he is virtually creating adjusts to and fulfills the intended functionalities. In the digital model it is possible to test, validate and verify a product. Here it is necessary to carry out a small anchoring of the notional product in 2D and/or 3D design [5]. When we talk about a product we refer to a house, a piece of an engine or a tissue, to mention some examples. The word "product" means that in some textile industries from Bergamo (for instance) they prefer that name instead of the CAD sector. Obviously, we are in the face of an ambiguous and opaque situation in internal organization those industries. Aside from this confusion situation, having a virtual prototype eliminates the need for the construction of a physical prototype. Simultaneously, the current CAD systems allow the interface to adapt to the needs of personalization of products and potential users [6]. This is a direct consequence of the flexibility of the software [7], [8]. To make changes, design variables in the virtual models have a lower cost than in the physical models. Additionlly, it allows users to instantly contribute improvements to a product that is being designed with a team who work on-line. The systems analysts, programmers and graphic informatics engineers are constantly moving towards a complete integration and interrelation with other programs that intervene in the development of a product. It allows for the prevention and minimisation of manufacturing mistakes and on top of that, production costs. Most of the CAD systems present a modular structure that can be widened in order to work in the different stages of the life cycle of a new development. This modular structure exists in the medium range CAD systems, such as the high rank systems. In the case of textile products, the different modules may refer to the type of tissues, that is to say, dobby. jacquard, etc. The presented educational prototype respects the notions of modularity and flexibility. 2.1 Software Development and Design: Personalization The users play a determining role in the incorporation of the new performances. In the same way that the context of the industrial products is characterized by a high personalization, in CAD systems this tendency is also detected, by incorporating new commands that are the results of requests from the different users. In the case of the textile, the hardware usually sets very important differences in the interface, especially if the Macintosh brand is compared with the other personal computers (or classically called ‘compatible with IBM PC’). In the Macintosh environment CAD systems are aimed at the realization of models, that is to say, one works with image desktop publishing programs (DTP), such as Photoshop or others related to vectorial images, such as is the case of the Corel Draw, for instance. That is to say, it is an environment that usually does not make issues. However, within the portability of the software it is necessary that the CAD system in an industrial environment should be able to function in both hardware platforms. In our educational prototype the compatibility of the software is guaranteed 100%. These kind of users are experts in desktop publishing and design. Consequently, the systems usually follow the style rules in the design of
CAD and Communicability: A System That Improves the Human-Computer Interaction
471
Macintosh interfaces [9]. In the industrial context the CAD systems usually generate their interfaces in accordance with the used programming language. The preponderance of the style of the Windows operating system in some components of the interface is observed, such as: menu bar, dialog boxes with display rectangles, labels for interface elements. etc. The requirements on the part of the users of the first group – Macintosh, are different to the second group– Windows. The first are related to the performances of the bi-dimensional drawing of human figures, the colouring of dresses, and exceptionally the inclusion of real textures of clothes in their creations. Each one of them may need simulation of the movements in these clothes in relation to the different raw materials used [10], that is to say, cotton, wool, linen, silk, etc., in virtual characters. An important factor of CAD is the possibility of extending the data to the production, that is to say, CAM. Having two or three systems in parallel is synonymous with a loss of financial resources and time, that is to say, a system for design (CAD), another for the management (administrative data, accountancy, staff, etc.) and the third for the CAM (production). Currently, these requirements are not necessary in the production sector who work in a Windows interface environment, because they are aimed at the real creation of tissues, that is to say, there is a symbiosis between CAD/CAM. The design is more related to technical issues, since in a later stage there lies the real production of these creations that originated on computer screens, which is mistakenly called in many Lombardy industries the ‘product department’. Obviously, it can be said that the degree of personalisation surpasses exponentially that of the first group. The origin of this phenomenon lies in the lack of graphical knowledge or computer design on the part of the people who run it, whether it is in the design department as in the data processing, and/or in the information technology department. Consequently, the costs of the second CAD systems widely surpass the first group, although the principles of applied graphical informatics are the same.
3 DobbyCAD: The Main Advantages and Characteristics DobbyCAD is a program for the design of textile tissues, which can be inserted in any kind of virtual characters, regardless of the modeling software or the animation that has been used. The executable program takes less than one Mb. Addtionally, the application is modular and does not require any hardware in particular. That is to say, from the point of view of the software and the hardware it is 100% compatible with the current computers. –PC or Mac– and operative systems (Windows XP, Vista, Linux, Mac OSX, etc.). Consequently, it does not need additional costs for its incorporation into the design departments or offices. (figure 1). The user, starting from the basic notions of textile design [11], will be able to create freely in a few hours those models he/she is most interested in so as to increase the realism of the scenes where there are human characters who use shirts, T-shirts, blouses, trousers, jeans, skirts, dresses, handkerchiefs, caps, etc. avoiding that metalized material in the clothes, which have so often been seen in computer animated films. Besides, the texture created with DobbyCAD, whether it is in a bitmap and vectorial format (jpg, tif, bmp, etc) [12], can be modified later on with any self-edition program: Corel Draw, Illustrator, Photopaint, Photoshop, etc. (figure 2).
472
F.V. Cipolla Ficarra and R.A. Rodríguez
Fig. 1. DobbyCAD –Easy design and simulation of real textile tissues
Fig. 2. DobbyCAD –Weave Design
In the same way as the CAM (computer.-aided-manufacturing) module makes it possible to transform the data of the graphical information, the management software for production, that is to say, the loom. In other words, it is feasible to create any kind of ligament and fold and weaving that the designer always has available in a database for its latter visualization and modification. DobbyCAD has demonstrated very well its ability in such environments, as it can be appreciated through the exquisite and demanding customers of the textile world, to which daily and through the years it is producing kilometers of tissues of the highest
CAD and Communicability: A System That Improves the Human-Computer Interaction
473
quality for the later elaboration of prestigious clothing made in Italy (Bergamo) – currently, factories and/or academic institutions in Argentina, Brazil, Britain, Canada, China, Costa Rica, France, Germany, India, Mexico, Portugal, Spain, Turkey, U.S.A., Venezuela, etc. The origin of the design of the real tissues lies in the notion of weaving and warping. The plot or weaving is the horizontal line of the clothes, and the warping is the vertical one. In the representation a square box system is used in which the horizontal and vertical threads crisscross (Figure 1), that is to say, the weaving and the warping. Starting from these thread combinations the tissue is born (see figure 3). With regard to the threads there is an interesting variety to be considered in the real world, such as chenilles, flamés, friezes, etc. Obviously, these threads can be generated through the use of commercial DTP programs or computer animation, such as Illustrator, Maya, etc. It is necessary to point out that the quality obtained through the tridimensional threads these days surpasses by far digital photography.
Fig. 3. DobbyCAD –Access to database and automatic simulation of textile tissues
These threads can be stored later on in a database. Another way to obtain them is through the use of a special scanner, working directly on threads or real tissues [13]. One can also resort to digital photography. Additionally, there are those CAD textile system programmers who maintain that photography of the threads microscope is enough in order to modify them and give them colour later on. Obviously, the option of making them manually with some desktop publishing program is a valid alternative, when other resources of commercial graphical computers are not available, such as can be 3D Studio Max, Rhinoceros 3D, Softimage, etc. As for colour, its selection is very easy with DobbyCAD and can be made through the keyboard or the scanner. In the first case, it is through the incorporation of the Pantone code, or giving a personalized code to each one of the colours that are used in the collection that is being created (figure 4). The second alternative is to pick up a thread or a skein, a cardboard or paper and get its colour. In both options, the human factor is always important because a mistake at that moment may entail significant loss of time and, indirectly, money.
474
F.V. Cipolla Ficarra and R.A. Rodríguez
Fig. 4. DobbyCAD –An easy selection of the colors for the yarns
The issue of the threads and the colours is very interesting because sometimes the prevailing ignorance in some working environments entails changing a whole CAD/CAM system, because according to the unaware bosses in the design department the CAD can not simulate/emulate colourless tissues, when in reality it is just simple bitmap files where it is necessary to simulate and emulate the ‘z’ coordinate, adding some other lighting effect. With programs such as 3D Studio Max, Cinema 4D, LightWave, etc. or even the classical ‘Paint’ of the Windows operating system, these effects can be achieved in a splendid way. The costs from the human, labour and economical points of view may be very high in the face of heads of the creative or design sectors who lack a minimal information in infography. Of course in these cases these software textile selling industries will invite you to change completely the CAD/CAM system. The management of the threads and the colours from DobbyCAD is very simple. Additionally, the changes made with the 2D or 3D commercial software on the threads can easily be seen on the computer screen or in the printer. In the triadic relationship between printer, screen and the real tissue textures, a whole calibration process of the colours has to be carried out, which entails several hours of tests, so that it adjusts as close as possible to the 100% of realism, between what is seen and what is printed. DobbyCAD admits the possibility of printing several tissue models on a same page: 1, 2, 4, 6, 8, 9, 12, etc. In the figure 6, six real simulations of tissues with different colours in the selection of the threads are depicted, and have been printed in a single folio. The philosophy followed in this software is ecological, also in the consumption of paper. Each one of the advantages presented in the DobbyCAD is the result of a long analysis process and perfecting of a series of heuristic techniques which have made up a methodology or method. The method used for the making of the current system has basically consisted of the heuristic analysis of the main CAD/CAM systems in the European market and work with a set of strategies (figure 5). To this end quality attributes have been used as well as the usability and communicability metrics of MECEM (Metrics for the Communication Evaluation in Multimedia) [14]. Aside from the heuristic assessment table (readers who are interested will find a wide bibliography in [14], [15], [16]).
CAD and Communicability: A System That Improves the Human-Computer Interaction
475
Fig. 5. DobbyCAD –Communicability, Design and Software Strategies
The results obtained with the application of that methodology have made it possible to develop the first beta versions of the system, which has been successfully adopted by real users, without the need to make tests in usability laboratories, for instance. Furthermore, the costs in the stage design of the system have been minimal thanks to the presence of a heuristic assessor who specializes in communicability and usability.
Fig. 6. DobbyCAD –Possibility of printing several tissue models on a same page
4 The Textures Increase the Realism of the Scene If we watch the evolution of computer made productions on the screen, we find that the first characters that mimicked human beings were tin or plastic puppets: the short film Tin Toy or the full-length film Toy Story [17]. However, already in the Woody
476
F.V. Cipolla Ficarra and R.A. Rodríguez
character from Toy Story the first shirt with its big checks appears. This type of shirt depicts the very history of textile industry, because the classical fashion made with stripes, squares or just the cloth without any design has endured until today. Now there is a tendency to emulate 3D with these two lines: warping and weaving. Now, we see too that there is a new tendency to solve certain problems which are inherent to the handling of textile textures at the moment of the animation. Nevertheless, in the computer graphics studies [18], the textures are still being obtained from databases with tissues that have been previously digitalized through a scanner or with a photography of them. This is the great Achilles heel of the bookshops in programs such as Quidam, Zbrush, etc. Through the DobbyCAD with a few hours and some basic notions of textile design the user can make real tissues for the virtual characters. An example of this is in the figure 7 of the named ‘Antaxel’. One of the problems that persists and which has been solved with DobbyCAD is to have real tissues available, which can be interactively modified through the simulator in the different shades of colours and even go down to the minimum texture detail.
Fig. 7. Antaxel –Virtual Character
Fig. 8. Textil Design and Visual Illusions
It shouldn't be forgotten either that through the tissues there is a series of images that from the point of view of perception can generate a series of illusions, making a synthetic character be accepted or not, as we can see in the texture of figure 8 (the combination of weavings and warpings are white, although we may perceive them as black).
5 Conclusions There is an excellent convergence of graphical engineering and person-computer interaction in the current work through the possibility of combining commercial programs of graphical computers to generate 2D and 3D animated characters and the novelty contributed from DobbyCAD (whose cost is 90% lower to those currently existing in the international market). This graphical software has evolved qualitatively, serving since 2002 the real production of wear clothes and world-renowned stylists. Through it, it is feasible to improve the quality of realism in animated graphical computers and very especially, in the virtual characters, thanks to its splendid interactive simulator.
CAD and Communicability: A System That Improves the Human-Computer Interaction
477
Acknowledgments. A special thanks to Emma Nicol (University of Strathclyde), Maria Ficarra (Alaipo & Ainci – Italy and Spain) and Carlos for their helps.
References 1. Kaldor, J., James, D., Marschner, S.: Simulating Knitted Cloth at the Yarn Level. ACM Transactions on Graphics 27, article 65 (2008) 2. Goldenthal, R., et al.: Efficient Simulation of Inextensible Cloth. ACM Transactions on Graphics 26, article 49 (2007) 3. Shneiderman, B.: Designing the User Interface –Srategies for Efective Human-Computer Interaction. Addison Wesley, Massachusetts (1998) 4. Paoluzzi, A.: Geometric Programming for Computer Aided Design. John Wiley, West Sussex (2003) 5. Mitchell, W., McCullough, M.: Digital Design Media. Van Nostrand Reinhold, New York (1995) 6. Ishii, H.: The Tangible User Interface and Its Evolution. Communications of the ACM 51, 32–36 (2008) 7. Ebert, C.: Open Source Software in Industry. IEEE Software 25, 52–53 (2008) 8. Ghezzi, C.: Fundamentals of Software Engineering. Prentice Hall, New Jersey (1991) 9. Apple: Macintosh Human Interface Guidelines. Addison-Wesley, Massachusetts (1992) 10. Kautz, J., Boulos, S., Durand, F.: Interactive Editing and Modelling of Bidirectional Texture Functions. ACM Transactions on Graphics 26, article 53 (2007) 11. Grana, C.: Tecnologia e Merceologia Tessile. San Marco, Bergamo (2003) 12. González-Jiménez, J.: Visión por Computador. Paraninfo, Madrid (2000) 13. White, R., Crane, K., Forsyth, D.: Capturing and Animating Occluded Cloth. ACM Transactions on Graphics 26, article 34 (2007) 14. Cipolla-Ficarra, F.: Communication Evaluation in Multimedia –Metrics and Methodology. LEA, Mahwah, vol. 3, pp. 567–571 (2001) 15. Cipolla-Ficarra, F.: Communicability design and evaluation in cultural and ecological multimedia systems. In: Proc. Workshop Communicability MS 2008, pp. 1–8. ACM Press, New York (2008) 16. Cipolla-Ficarra, F.: Table of Heuristic Evaluation for Communication of the Multimedia Systems. In: Proceedings of the HCI International, pp. 940–944. LEA, Crete (2003) 17. Smith, T.: Industrial Light & Magic: The Art of Special Effects. Del Rey Books, New York (1986) 18. Newman, W., Sproull, R.: Principles of Interactive Computer Graphics. McGraw-Hill, New York (1979)
A Novel Visualization Tool for Evaluating Medication Side-Effects in Multi-drug Regimens Jon Duke1,2, Anthony Faiola1, and Hadi Kharrazi1 1
Indiana University Purdue University Indianapolis, School of Informatics 535 W. Michigan St., Indianapolis, Indiana 46202 2 Regenstrief Institute, Medical Informatics, HITS 2000 550 W. 10th Street Indianapolis, IN 46209 [email protected], [email protected], [email protected]
Abstract. The evaluation and management of medication side-effects is a common and complex task for physicians. Information visualization has the potential to increase the efficiency and reduce the cognitive load involved in this process. We describe the design and development of Rxplore, a novel tool for assessing medication side-effects. Rxplore supports simultaneous lookup of multiple medications and an intuitive visual representation of query results. In a pilot study of Rxplore’s usability and utility, physicians rated the system highly for efficiency, intuitiveness, and clinical value. Keywords: information visualization, medical informatics, adverse reactions, medications, side-effects.
A Novel Visualization Tool for Evaluating Medication Side-Effects
479
with each medication and note how frequently these effects are reported in clinical trials. Yet even the most efficient systems require physicians to look up medications one-by-one. Furthermore, physicians must rely on memory or note-taking to organize the information they uncover while interacting with these systems. In this paper, we describe the development of a new visualization tool designed to address these issues and to expedite the review of side-effect data. We also present the results of a pilot study on the usability and clinical utility of this system.
2 Theory and Background: Information Visualization and Biomedical Informatics Information visualization is a mapping technique that utilizes visual graphics to represent non-spatial abstract datasets in physical space [6]. Visualization enables users to make discoveries and decisions regarding complex data, elucidating subtle patterns and making information-intensive tasks more manageable. In the following section, we explore the theoretical underpinnings of information visualization as a means to augment human cognition. Then we review some current applications of visualization in the healthcare setting. 2.1 Information Visualization and Augmented Cognition A prime challenge of modeling a meaningful visualization is identifying an appropriate visual mapping of the information that best supports correlations and relationships between the unstructured data; thereby enhancing new understanding. How people perceive, understand, and apply information visualization is essential to grasp the relationship between human-computer interaction and human cognition. As Heer et al. state, information visualization seeks to “augment human cognition by leveraging human visual capabilities to make sense of large collections of abstract information” [7]. The amplification of cognition through information visualization modeling, joined with data analysis, provides users interaction with a support for more sense-making of large and complex datasets. For example, amplifying cognition through information visualization both increases cognitive resources by expanding working memory and reduces search by representing large datasets in a small space. As Beale et al. state [8], cognitive amplification through the use of visualization methods “helps to shift the work load from the cognitive to the perceptual system; expands the working memory; and allows a high level of interaction… [thus aiding in the users’] confirmation and discovery of knowledge”. Furthermore, information visualization supports relationships of perceptual inference that would otherwise be quite difficult to reproduce. One of the greatest benefits of information visualization is that it (unlike static media) provides a means to manipulate and enable the exploration of a space of parameter values [8,9]. This is observed in the theory of external cognition as applied to design issues and an analysis of how graphical representations are used during cognitive activities such as problem-solving [10]. Specifically, external cognition refers to the “interaction between internal and external representations when performing cognitive tasks such as learning” [11]. The design dimensions of information visualizations are intended to help researchers and practitioners, through
480
J. Duke, A. Faiola, and H. Kharrazi
combinations of visualizations, to experience a type of “computational offloading, the extent to which different external representations vary the amount of cognitive effort required to carry out different activities” [11]. This concurs with Norman’s early work [12,13] in the psychology of humancomputer interaction, in which he argued for a fundamental paradigm shift in understanding the way interactive products are designed. For example, he suggested that well-designed artifacts should reduce the need for users to remember large amounts of information. In the section titled “The Conspiracy against Memory” [12], Norman highlights our inability to freshly retain many items, i.e., the way these items work and the way they relate to one another. He asserts that the human mind is limited in its ability to think deeply about any given topic, primarily because of the restricted capacity of working memory. For this reason, visual aids, such as information visualization, are necessary to support both cognition and an array of learning processes, as well as to reduce human error. In sum, researchers and system designers need to find ways to arrange complex systems that are visible to reduce information in memory and reflect human logic. Consequently, we can consider the visualization interface functions as the contact point for interaction, where information enters through the human sensory system, is organized, and finally recognized to ensure an error-free display of the information. When interacting with a visualization system, users may process the information into different forms, e.g., perception and transformation of understanding. On one hand, the computational modeling of data visualization is critical for an accurate display of information. On the other, interaction design and information design are at the heart of information visualization. Most importantly, empirical findings have shown that properly designed visualizations have the potential to give a clear picture at a glance. This is because the laws of semantic configuration must be learned, but images provide automatic comparisons and obvious relationships [14]. 2.2 Information Visualization in Healthcare Information retrieval is an important part of the daily routine of physicians and other healthcare providers. Yet the body of knowledge in medicine is increasing enormously, forcing clinicians to rely on various external sources of medical information [15]. These resources can range from textbooks, to external websites, or even to fully integrated clinical decision support (CDS) systems. When discussing these tools, it is important to differentiate between those designed purely for “data availability” and those which have extended this concept to include the notion of “information extraction” [16]. Resources that offer data availability alone (such as a textbook or patient chart) will provide the physician with accurate information but leave him with the burden of collecting, maintaining in working memory, and interpreting these data. Conversely, resources that move beyond availability and provide assistance with information extraction (such as context-sensitive clinical reminders or real-time cardiac telemetry) can reduce the physician’s cognitive load. Information visualization has been shown to be one such successful method of supporting extraction and reducing cognitive load in the medical setting [17,18]. There are currently a number of visualization applications in clinical use. One such application that is often overlooked is the electrocardiogram (EKG), which is
A Novel Visualization Tool for Evaluating Medication Side-Effects
481
essentially a visual representation of the heart’s direction and force throughout the cardiac cycle. More traditional forms of data visualization would be graphical monitoring of blood pressure, pulse, and other vital signs in the intensive care unit [19]. Graphical displays of laboratory results have also become a standard feature of electronic medical record systems in both the hospital and outpatient settings. More recently, work has been done to visually represent patient clinical histories, such as diagnoses, medications, surgeries, and so forth [20,21]. These clinical visualizations provide an integrated view of the temporal characteristics of a patient’s history, giving the physician an integrated high-level view of the patient’s symptoms and treatments over time. Despite these areas of progress, there are relatively few reports in the literature of visualization techniques being applied to the evaluation of medications and their sideeffects. One such application is DOPAMINE, a tool for visualizing drug therapeutic classes, but this work was primarily focused on ontology development rather than clinical practice [22]. Another application is called “Mister VMC,” a tool which implements an anatomic representation technique to portray side-effects, contraindications, and interactions in a real-time clinical context. A study showed improved accuracy and speed of obtaining drug information using this tool when compared with a standard text-based system [23]. However the major drawback of this system is its complex iconography, requiring prior training in a proprietary visual language to use the application effectively. 2.3 Adverse Drug Reactions and Clinical Decision Support While the availability of graphical decision support systems for drug information is quite limited, text-based decision support applications have been increasingly adopted by physicians. Two prime examples are Epocrates and UpToDate. These applications have been shown to be effective in reducing prescribing errors and assisting clinical decision-making [24,25]. Both solutions are popular, with 100,000 Epocrates users and over 300,000 UpToDate users visiting their websites every month [26,27]. Whether accessing the PDA or Desktop version, a user interacts with these applications by searching for a medication of interest and having the option to choose from several pertinent categories of prescribing information. Side-effect data are generally displayed in a slightly condensed (UpToDate) or significantly condensed (Epocrates) form. Users typically write down or just remember the information they need before closing the application and returning to the task at hand. There is no current method of looking up multiple medications simultaneously, or of searching across a set of medications for a particular side-effect. While Epocrates and UpToDate are both excellent information resources and have improved the efficiency of retrieving drug side-effect data, they have not significantly reduced the overall cognitive complexity of the process. In other words, their focus remains on data availability rather than information extraction. In the following section, we describe the creation of a novel drug information resource which aims to support both these elements in equal measure through effective data visualization methods.
482
J. Duke, A. Faiola, and H. Kharrazi
3 Methods 3.1 Development of the Rxplore Visualization Tool The development goals were to create a tool which would allow users to rapidly retrieve side-effect data on multiple medications and to deliver these results in an easily interpretable manner with minimal cognitive load. The development process comprised two stages: 1) creation of a database containing quantitative data on medication side-effects 2) construction of a web interface to query this database and display the results of multiple queries in graphical form. Creation of the Quantitative Adverse Reaction Knowledgebase (QuARK). Adverse reaction data were extracted from 250 FDA-approved medication labels by both manual and natural language processing extraction methods. These data were based on multiple sources (e.g., clinical trials, post-marking reports) and could be either quantitative or qualitative. In order to account for these differences, we constructed algorithms to simplify the relationship of each medication and side-effect to a singular value called the Rxplore Score (Table 1). Table 1. Example algorithms used in generating Rxplore Scores Reporting Format Drug and Placebo Frequency Range Qualitative Terms
Example Nausea seen in 27% on Drug X and 12% placebo Itching occurred in between 3% and 9% of patients Rash occurred “rarely”
Algorithm Drug % - Placebo %
Rxplore Score 27 – 12 = 15
LB+(UB–LB) / 3
3+(9-6)/3 = 5
Expert consensus + Matching terms in quantitative datasets
0.3
This scoring system was not designed to be an exact measurement of frequency, but rather to provide a conceptual gauge reflecting the association of each drug with a particular side-effect. The original data were retained in the knowledge-base to allow recalculation of the scores in case of an algorithm modification. The final Rxplore database contains information on over 16,000 medication and side-effect pairs. Interface and Visualization Design. A web-based interface to the Rxplore data was then developed using ASP.NET, Flash, and SQL Server. The requisite features were: 1) the ability to create a list of multiple medications 2) a means to search for a specific side-effect across this entire list and 3) an intuitive visual display of the search results. For the visualization, we selected a stacked-bar format in which each row represents a single side-effect, and each bar within the row reflects the Score of that effect for a given medication (Figure 1). Bars are differentiated by color to allow the user to see which medications are associated with a particular effect. The size of each bar is proportional to the calculated Rxplore Score, so that when seen as a collective, the graph conveys instantaneously which medications are most likely to cause a given effect.
A Novel Visualization Tool for Evaluating Medication Side-Effects
483
Fig. 1. A summary graph from Rxplore, depicting the most common side-effects reported with a regimen consisting of six different medications. Larger bars suggest a higher associated frequency between the drug and a side-effect.
Fig. 2. A summary graph of medication side-effects with detailed frequency data displayed. Moving the mouse over a bar on the graph retrieves the specifics for that medication and sideeffect combination.
An “overview plus detail” model was used to provide textual side-effect data on demand. For example, if the user wants to retrieve the details on the relationship between Zoloft and Insomnia, he can hover over the graph in the Insomnia row on the Zoloft bar and the desired information will be displayed (Figure 2). Hence, the visualization technique allows for rapid interpretation of the high-level data as well as expedient retrieval of lower level details.
484
J. Duke, A. Faiola, and H. Kharrazi
In addition to displaying the most common side-effects (Figure 2) for a medication regimen, the user can also retrieve the most common effects of a particular type (e.g. cardiac, gastrointestinal) or search for an individual side-effect of interest. Medication regimens and custom side-effects lists can be stored for future review and adjustment. 3.2 Evaluation Study Research Questions. In conducting a preliminary evaluation of Rxplore, we sought to answer two basic questions. First, how do health care providers rate the system’s usability? Specifically we were interested in its learnability, efficiency, and ease of use. Second, how do these providers rate the system’s clinical utility? Our metrics for utility were perceived clinical value and relevance to doctors’ daily work. Participants. We invited 63 physicians and nurses at the Indiana University Medical Group to participate in this evaluation study, of whom 10 enrolled. Treatment. The subjects were first shown an introductory video to explain the concept and features of the Rxplore tool. Then, they were presented with two patient scenarios, each containing a clinical description, current medication use, and recent history and physical exam findings. For each scenario, there were 5 questions regarding the patient’s symptoms and potential causal relationship of the patient’s medications (e.g. “Which of these medications is most likely to be causing his Nausea?”). Subjects worked through the questions independently using the Rxplore tool. Upon completion of the tasks, they were given an online questionnaire regarding the system’s usability and clinical utility. Follow-up interviews were conducted to gain further insight into the subject’s experiences with Rxplore.
4 Results 4.1 Demographics Participants consisted of 90% physicians while 10% were with nursing backgrounds. The subjects described themselves as 56% avid users of computers, 33% average users and 11% with poor computer skills. We explored their current drug information needs and found that 67% used an electronic resource to look up medication sideeffects on a daily basis, 22% did so a few times per week, and 11% accessed such a resource only a few times a month. 4.2 Utility Rxplore’s effectiveness and clinical utility were measured by task-oriented selfreported questions. For example, the users were asked to rate their experience when performing tasks such as adding a medication, deleting a medication, finding a specific side-effect or selecting a side-effect category. Rxplore was rated by average 94.5% effective in accomplishing the aforementioned tasks.
A Novel Visualization Tool for Evaluating Medication Side-Effects
485
4.3 Usability A number of questions focused on the usability of the system. Its Intuitiveness rated 4.7 out of 5 while Ease to Use rated 4.6 out of 5. Participants overall found the system appealing (4.6 out of 5) and also aesthetically pleasing (4.9 out of 5). Subjects rated Rxplore’s Speed and Time-Saving potential as 4.4 out of 5. Furthermore, when asked if it was faster than their current source of drug information, subjects rated Rxplore at 96% for retrieving all common side-effects and 100% for retrieving the frequency of a specific side-effect. Participants were also asked specific questions about the system’s visual concept and the meaning of its elements. For example, regarding the clarity of the bars in predicting the risks of experiencing a side-effect, users rated the system 100%. The same result was concluded regarding the relationship between multiple bars on the same side-effect and several medications causing the effects. One area of weakness was in recognizing Rxplore as a conceptual measure rather than as a precise statistical metric. This concept received only a 60% rating for clarity. 4.4 Usefulness Regarding the potential clinical value, subjects rated the system 4.5 out of 5 in terms of relevancy to their work and 4.9 out of 5 in terms of usability in clinical practice (Figure 8). Overall satisfaction was high, with 90% of subjects rating their experience with Rxplore as Excellent or Good; none had a Fair or Poor experience. Likelihood of using Rxplore again was high (4.6 out of 5) as was the willingness to recommend Rxplore to colleagues (4.6 out of 5).
5 Discussion The benefit of visualizing complex information has been established in multiple health care settings. In this paper, we have described a new use of information visualization for the purposes of evaluating medication side-effects. This visualization tool, known as Rxplore, has two main advantages over current methods of researching adverse drug reactions. First, it allows for retrieval of information on multiple medications simultaneously. Second, by quantifying the relationship between each medication and its associated effects, Rxplore can provide an immediately interpretable visual representation of this complex set of data. Although small in size, our pilot study’s cohort of physicians demonstrated considerable enthusiasm for the tool’s usability and potential value in the clinical setting. By using standard visualization methods, including stacked bar graphs and rollover text boxes for detailed data, Rxplore’s learning curve was kept to a minimum. No dedicated training was required to use the system, and participants rated it highly in terms of both ease of use and intuitiveness. While quantitative measures of cognitive load were not recorded in this study, subjects reported the system was efficient, timesaving, and faster than their current drug information resource. This increased efficiency may relate to decreased demands on working memory and the computational off-loading achieved by providing information on multiple medications in a single image. Typically, users would need to remember their findings for each individual
486
J. Duke, A. Faiola, and H. Kharrazi
drug as the data are looked up sequentially. With Rxplore’s simultaneous graphical retrieval technique, cognitive load previously spent on memory tasks can be shifted to the interpretation of the data. This shift in effort results in improved information extraction without additional work on the part of the physician. Despite its potential benefits, Rxplore does have several limitations as well. The most significant issue being the reliability of the underlying algorithms used to calculate the Rxplore Scores. Though validation studies are underway, these Scores are necessarily coarse reflections of the complex (and sometimes conflicting) data available regarding medication side-effect frequencies. In absence of a gold standard, physicians must rely on the body of available knowledge and their own clinical experience to judge potential adverse reactions in any given patient. Thus, the Rxplore Scores have been designed not as a precise statistical model of side-effect likelihood, but rather as a guide to encourage physicians to explore the underlying data in a more efficient manner. The challenge in any information visualization is to ensure that the viewer understands the full meaning of the displayed information. With Rxplore, avoiding hasty assumptions and encouraging exploration of the underlying data is absolutely essential. Based on the study results, several physicians did not fully understand the use of Rxplore Score as a conceptual rather than statistical metric. Thus, we must present the methodology to users more clearly to ensure clinical judgments are based on the full spectrum of available data. The major limitation of our pilot study was its small size and absence of a control resource for comparison. Upcoming work will include a comparative study between Rxplore and traditional drug information resources. We will also expand the number of participants as well as the acquisition of quantitative data for measuring usability, efficiency, and cognitive load. Finally, integration of Rxplore into an existing electronic medical record system is now underway and will provide a rich test bed for further investigation into the system’s use.
References 1. Lazarou, J., Pomeranz, B.H., Corey, P.N.: Incidence of Adverse Drug Reactions in Hospitalized Patients: A Meta-analysis of Prospective Studies. JAMA 279, 1200–1205 (1998) 2. Ernst, F.R., Grizzle, A.J.: Drug-related morbidity and mortality: updating the cost-ofillness model. J. Am. Pharm. Assoc. 41, 192–199 (2001) 3. DailyMed: About DailyMed, http://dailymed.nlm.nih.gov/dailymed/about.cfm 4. Veehof, L.J.G., et al.: Adverse drug reactions and polypharmacy in the elderly in general practice. European Journal of Clinical Pharmacology 55, 533–536 (1999) 5. Kaufman, D.W., et al.: Recent Patterns of Medication Use in the Ambulatory Adult Population of the United States: The Slone Survey. JAMA 287, 337–344 (2002) 6. Voigt, R.: An Extended Scatterplot Matrix and Case Studies in Information Visualization. Published as Diplomarbeit (2002), http://www.vrvis.at/via//resources/DA-RVoigt/DA.pdf 7. Heer, J., Card, S., Landay, J.P.: A Toolkit for Interactive Information Visualization. In: Proceeding of the ACM Conference on Human Factors in Computing Systems, pp. 421–430 (2005)
A Novel Visualization Tool for Evaluating Medication Side-Effects
487
8. Thomas, J.J., Cook, K.A.: Illuminating the Path. 200 (2005) 9. Card, S.K., Mackinlay, J.D., Shneiderman, B.: Readings in Information Visualization 686 (1999) 10. Scaife, M., et al.: External Cognition: How Do Graphical Representations Work. International Journal of Human-Computer Studies 45, 185–213 (1996) 11. Rogers, Y.: New theoretical approaches for human-computer interaction. Annual Review of Information Science and Technology 38, 87–143 (2004) 12. Norman, D.A.: The Design of Everyday Things 257 (1990) 13. Norman, D.A., Draper, S.W.: User Centered System Design 526 (1986) 14. Gregg, L.W., Bower, G.H., University, C.: Cognition in Learning and Memory, pp. 51–88 (1972) 15. Holzinger, A.: HCI and Usability for Medicine and Health Care 458 (2007) 16. Workman, M., Lesser, M.F., Kim, J.: An Exploratory Study of Cognitive Load in Diagnosing Patient Conditions. Int. J. Qual. Health Care 19, 127–133 (2007) 17. Shachak, A., et al.: Primary Care Physicians’ Use of an Electronic Medical Record System: A Cognitive Task Analysis. J. Gen. Intern. Med. (2009) 18. Stoicu-Tivadar, L., Stoicu-Tivadar, V.: Human-Computer Interaction Reflected in the Design of User Interfaces for General Practitioners. International Journal of Medical Informatics 75, 335–342 (2006) 19. Wenkebach, U., Pollwein, B., Finsterer, U.: Visualization of Large Datasets in Intensive Care. In: Proc. Annu. Symp. Comput. Appl. Med. Care, pp. 18–22 (1992) 20. Shahar, Y., Cheng, C.: Intelligent visualization and exploration of time-oriented clinical data. Top Health Inf Manage 20, 15–31 (1999) 21. Plaisant, C., Mushlin, R., Snyder, A., Li, J., Heller, D., Shneiderman, B.: LifeLines: Using Visualization to Enhance Navigation and Analysis of Patient Records. In: Proc. AMIA Symp., pp. 76–80 (1998) 22. Wroe, C.J., Solomon, W.D., Rector, A.L., Rogers, J.: Dopamine: A Tool for Visualizing Clinical Properties of Generic Drugs. In: Lavrac, B.K.N., Miksch, S. (eds.) International Workshop on Intelligent Data Analysis in Medicine and Pharmacology. The Fifth Workshop on Intelligent Data Analysis in Medicine and Pharmacology (2000) 23. Lamy, J., Venot, A., Bar-Hen, A., Ouvrard, P., Duclos, C.: Design of a Graphical and Interactive Interface for Facilitating Access to Drug Contraindications, Cautions for Use, Interactions and Adverse Effects. BMC Medical Informatics and Decision Making 8, 21 (2008) 24. Berner, E.S., et al.: Improving Ambulatory Prescribing Safety with a Handheld Decision Support System: A Randomized Controlled Trial. J. Am. Med. Inform. Assoc. 13, 171– 179 (2006) 25. Wilcox, R.A., Whitham, E.M.: Reduction of Medical Error at the Point-of-Care Using Electronic Clinical Information Delivery. Internal Medicine Journal 33, 537–540 (2003) 26. Quantcast UpToDate Profile, http://www.quantcast.com/uptodate.com 27. Quantcast Epocrates Profile, http://www.quantcast.com/epocrates.com
Design of a Web Intervention to Change Youth Smoking Habits Kim Nee Goh1, Yoke Yie Chen2, Emy Elyanee Mustapha3, Subarna Sivapalan4, and Sharina Nordin5 1,2,3
Department of Computer and Information Sciences 4,5 Department of Management and Humanities Universiti Teknologi PETRONAS Bandar Seri Iskandar, 31750 Tronoh, Perak Darul Ridzuan, Malaysia {gohkimnee,chenyokeyie,emyelyanee,subarna_s, shahrina_mnordin}@petronas.com.my
Abstract. Web interventions are gaining popularity in trying to change a person’s behavior. However, poorly designed intervention websites will affect the learning process of a person, what more to remember the content that they have learnt. The objective of this paper is to discuss the design of a web intervention using Gagne’s Condition of Learning Theory and cognitive dissonance. The usage of learning theory enables the designers to develop the intervention website based on users’ learning capabilities, thus placing users as priority. This ensures that users are able to maximize learning and recall of its content when faced with the decision to smoke or not. We are planning to recruit target users who are smokers, ranging from the 18 to 22 age range. Smokers will be categorized according to a baseline survey. Each category of users will go through the web intervention of different content. It is hoped that by combining the abovementioned theories, smokers are able to reduce their intake. Keywords: Smoking, Gagne’s Condition of Learning Theory, Cognitive Dissonance, Web Intervention, User Interface, HCI.
Design of a Web Intervention to Change Youth Smoking Habits
489
Bernhardt [1], the Internet is a new tool which can revolutionize health education. The web is flexible; hence delivery of health messages can be instantaneous and tailored to the specific needs, characteristics, and preferences of the user [1]. Because health education is not always a desirable topic to be discussed by the younger generation, health educators seem to embark on other/more advanced technological methods to gain students’ interest in learning about a health topic, specifically topics which could generate controversy, for instance alcohol consumption, sex education and even smoking. According to statistics, Malaysia had 3.6 million smokers in 2000 and this number is expected to increase to 4.6 million by 2025 [15]. The Malaysian government is trying to stem this increase by forbidding the sales of cigarettes to individuals less than 18 years of age. In addition, non-smoking campaign such as ‘TAK NAK’ (Say No to Smoking) has also been implemented nationwide. However, the effectiveness of non-smoking campaigns was never at a satisfactory level for the huge amount of money invested in promoting the campaign. [15]. The study reported in this paper therefore seeks to propose the use of a web-based educational approach as an alternative approach towards educating smokers on the adversities of smoking. Research indicates that information technology is a novel platform to reach out to the public. It has also been deemed as a cost-effective way to educate the public on issues concerning health. These traits are seen as instrumental in developing websites that are individualized and tailored to the needs to each patient, depending on their history, addiction patterns, age, race etc. Computers have made it possible to easily incorporate personalized feedback or tailored messages into a print or Internetgenerated intervention. [9]. For health educators, the practical implication of this research is that mass media channels are appropriate for creating awareness, but interpersonal interactions are essential for persuading individuals to adopt healthpromoting behaviors [3]. As this project is still in the preliminary stage of development, this paper will discuss the interface design and the experimental design of the proposed web intervention.
2 Related Works Information about smoking can be obtained from almost anywhere; Internet, mass media, brochures, cigarette packages; however most of the messages are being ignored as the medium used to disseminate this information is considered “old fashion”. Furthermore, the information is seen as repetitive and boring; hence people tend to be ignorant over time. According to [2] as cited in [3], mass media is an effective way to reach and inform large number of audience, but it was found that interpersonal channels are more successful in influencing attitudes and motivating behavior changes in a person. Rather, patients/users prefer information that is personalized to their individual needs and situations [11]. Previous research indicates that tailored information is more likely to be read, remembered, and experienced as personally relevant, which in turn has a greater impact in motivating patients to make a behavior change [4]. If tailored messages increase the attention span of users, which thus educate them, information seekers will then benefit from using tailored health education online [1]. Hence, a web-based tailored application on smoking may be used as an innovative
490
K.N. Goh et al.
intervention or prevention tool for teenagers who have different smoking patterns and personal characteristics (e.g. age, hometown, race, gender etc.). Many works have been done regarding intervention websites for smoking [19], alcohol [12], skin cancer prevention [1] and others. Different design methodologies have been adapted into these researches, for example, developing content which follows certain theory [20] or even creating websites that dynamically change the message content and design tailored for individual recipients to increase the user’s attention span [1]. Although the message content has been tailored, many individuals still do not pay enough attention to the health message they received [1]. Hence the focal point of this work will not only tailor the message to each individual, but also ensure that the screen is designed in a way that enables the individual to remember the information easily. This design is based on the learning theory described in section 2.1. Section 2.2 will then discusses how we can use cognitive dissonance, a behavioral theory, to develop our message. 2.1 Gagne’s Condition of Learning Theory A learning theory has been adapted to develop an intervention website that persuades changes in a person’s behavior. Gagne’s conditions of learning theory stresses on the different kinds of learning levels in individuals’ [7], namely (1) verbal information, (2) intellectual skills, (3) cognitive strategies, (4) motor skills and (5) attitudes. Hence, to cater to a wide group of learning styles, the 9 instructional events that Gagne outlines plays an important role for designing the intervention website [7]: 1) 2) 3) 4) 5) 6) 7) 8) 9)
Gaining attention Informing learners of the objective(s) Stimulating recall of prior learning Presenting the stimulus Providing learning guidance Eliciting performance Providing feedback Assessing performance Enhancing retention and transfer
An important component in Gagne’s theory emphasizes on selecting the right media for designing the instructional events. This is related to the concept of transfer of learning, which was introduced by Edward Thorndike and Robert S. Woodworth in 1901, where designers ought to design website which uses media elements that enable the transfer of knowledge easily from past experiences of the users [14]. For example, media elements that relate well to the target audience need to be used, where they have experienced those elements before and seeing them again in the intervention website will enable them to easily learn and respond to those media, when necessary. For example, cell phones are gadgets that every young adult owns nowadays, thus, if the web intervention contains an animated cell phone media, the user would easily transfer the knowledge from the real world to the virtual environment. For users of this intervention website to maximize their gain and share knowledge easily, low road transfer would be a better option as compared to a high road transfer when designing the website [14].
Design of a Web Intervention to Change Youth Smoking Habits
491
2.2 Cognitive Dissonance Cognitive dissonance theory was first introduced by Leon Festinger in 1957. This theory suggests that people are motivated to reduce their cognition dissonance by changing their attitudes [5]. A study done by Rachel D. Graham shows that people experience cognitive dissonance after making a difficult decision concerning morality [6]. A number of works done using cognitive dissonance theories [8, 13, 16], indicate that this theory has always been applied to making decision based on two options. Between these two options, there are self-debated consequences attached to oneself. Regardless, when someone makes a decision that is against their own judgment, cognitive dissonance will increase. A number of studies were done focusing on cognitive dissonance among smokers [13, 10, 17]. Cognitive dissonance theory is a theory which is synonym in explaining the complexity of an individual who smokes. In 1991, McMaster and Lee wrote regarding cognitive dissonance in tobacco smokers among Australian smokers. Based on their study, they conclude that smokers may experience cognitive dissonance because they use tobacco despite its well-publicized ill-effects. Therefore, they wrote that interventions targeting rationalizations for smoking will be useful in smoking cessation [13]. Meanwhile in 1994, Halpern wrote regarding the effect of smoking characteristics on cognitive dissonance in current and former smokers. He concluded that on overall, current smokers exhibited more cognitive dissonance than former smokers with regards to smoking-related belief [10].
3 Design 3.1 Interface Design As mentioned by Gagne, Briggs and Wager [8], the abovementioned instructional events should be satisfied, which serves as a basis for designing instructions and selecting appropriate media. An example of how the nine events are applied in the intervention website is shown in Table 1. Table 1. Example of events for learning theory Events 1. Gaining attention 2. Informing learners of the objective(s) 3. Stimulating recall of prior learning 4. Presenting the stimulus 5. Providing learning guidance 6. Eliciting performance 7. Providing feedback 8. Assessing performance 9. Enhancing retention and transfer
Example Ask users what are their most important values. Tell the users why it is important to reduce/eliminate smoking habits. Review what they have learnt previously. Give the users the facts of smoking. Give the importance / facts of each value they selected. Ask them why they smoke. Provide feedback based on their answers. Provide solutions to problems they face. Recall the importance of their values.
492
K.N. Goh et al.
Referring to Table 1, in the event of gaining attention, we can ask the users to rate their most important core values in life from choices like family, friends, finance, health, self-confidence and others. From these ratings, the web intervention is able to capture their important core values and reemphasize them throughout the web intervention and tailor the messages accordingly to remind them of the values they have chosen. This can also enhance their retention and transfer according to Gagne’s learning theory. In presenting stimulus, we could ask the user a series of questions that will reaffirm their knowledge of smoking, for example some questions to identify between fact and fiction. Some example of questions are “People gain weight when they quit smoking.”, “Quitting smoking will be hard.”, “I have too much stress in my life to quit right now” and others. 3.2 Experimental Design Fig. 1 shows the design process of this web intervention. Pre-screening questions will be given to purposefully sampled smokers to narrow down the data scope to be collected. Some examples of pre-screening questions are demographic questions, family history of smoking, what influenced them to start smoking and how many sticks of cigarettes they smoke in a day. The collected results will then be screened to categorize these students into three main categories: (1) heavy, (2) moderate and (3) low smokers based on certain criterias. For each category, the respondents will be given different sets of experiments, which will contain different approach and content. Experiment A, B and C will contain similar interface screens, but the content of message will be different as the tailoring approached will be used. Similar interfaces will ease in web development, but new interfaces development may be necessary to enhance delivery depending on content. The target respondents are undergraduate students in a local private university in Malaysia aged between 18-22 years. Students enrolled in the same program will be selected for easy tracking of these students where a follow-up study will be conducted after a period of 6 months. A total number of 30 students were chosen based on their smoking history rather than random sampling for better filtering. Students will be given a span of two weeks to complete the web session. Each session should last for about 30 minutes. By going through the series of web intervention sessions, it is able to educate the students about smoking habit and provide them with information about smoking. The web sessions will mostly provide information that is tailored to each student so
Fig. 1. Design Process
Design of a Web Intervention to Change Youth Smoking Habits
493
that information can be individualized. By this method, students will feel that the information is more relevant and suited to their needs, based on their response given during the web experiment [9]. After the implementation of this study, it therefore seeks to experiment how interface design can affect or influence change in a person’s behavior (from breaking a bad habit) without the use of external tools such as nicotine replacement therapy. This experiment may not be able to entirely eliminate the smoker’s smoking behavior but it reduces their amount of intake. For example, if they smoke one packet (24 sticks) a day, they can reduce their intake to 10 sticks a day.
4 Conclusions The number of smokers has not decreased for the past decades. Smoking relateddiseases kill one in ten adults globally, or cause four million deaths. If current trends continue, smoking will kill one in six people by 2030 [18]. It is believed that if individuals do not get into the habit of smoking in their youth, they would probably never smoke as adults [15]. There are many reasons why smoking starts so early among the youngsters. One of the main reasons is the promotion by tobacco companies that projects a positive and appealing image of smoking. Growing information about the enjoyment and satisfaction derived from smoking projected by the mass media has all contributed to obscure the fact that smoking endangers health. It is therefore believed that by combining both learning theory and cognitive dissonance in designing the user interface of a web-based tailored application on smoking, the young smokers would be able to reduce their amount of cigarette intake. The intervention study is currently underway (as of January 2009) and the effectiveness of the website in changing the students’ smoking behavior will be reported in a separate publication.
References 1. Bernhardt, J.M.: Tailoring Messages and Design in a Web-Based Skin Cancer Prevention Intervention. The International Electronic Journal of Health Education 4, 290–297 (2001) 2. Backer, T.T., Rogers, E.M., Sopory, P.: Designing health communication campaigns: What works? Health Communication on the Internet: An Effective Channel for Health Behavior Change? Journal of Health Communication 3, 71–79 (1992) 3. Cassell, M.M., Jackson, C., Cheuvront, B.: Health Communication on the Internet: An Effective Channel for Health Behavior Change? Journal of Health Communication 3, 71–79 (1998) 4. Davis, S., Abidi, S.S.R.: Adaptive Patient Education Framework Featuring Personalized Cardiovascular Risk Management Interventions. In: Wade, V.P., Ashman, H., Smyth, B. (eds.) Adaptive Hypermedia and Adaptive Web-Based Systems. Springer, Heidelberg (2006) 5. Festinger, L.: A Theory of Cognitive Dissonance. Stanford University Press, Stanford (1957) 6. Graham, D.R.: Theory of Cognitive Dissonance as it Pertains to Morality. Journal of Scientific Psychology, 20–23 (2007) 7. Gagne, R.M.: The Conditions of Learning Theory of Instruction. CBS College Publishing, New York (1985)
494
K.N. Goh et al.
8. Gagne, R., Briggs, L., Wager, W.: Principles of Instructional Design, 4th edn. HBJ College Publishers, Fort Worth (1992) 9. Hageman, P.A., Walker, S.N., Pullen, C.H.: Tailored Versus Standard Internet-delivered Interventions to Promote Physical Activity in Older Women. Journal of Geriatric Physical Therapy. Journal of Geriatric Physical Therapy 28(1:05), 28–32 (2007) 10. Halpern, T.M.: Effect of Smoking Characteristics on Cognitive Dissonance in Current and Former Smokers. Addictive Behaviours 19(2), 209–217 (1994) 11. Hoffman, T., Russell, T., McKenna, K.: Producing computer-generated tailored written information for stroke patients and their careers: system development and preliminary evaluation. International Journal of Medical Informatics 73, 752–758 (2004) 12. Linke, S., Brown, A., Wallace, P.: Down your drink: A web-based intervention for people with excessive alcohol consumption. Alcohol and Alcoholism 39(1), 29–32 (2004) 13. McMaster, C., Lee, C.: Cognitive Dissonance in Tobacco Smokers. Addictive Behaviours 16, 349–353 (1991) 14. Perkins, D., Solomon, G.: Transfer of Learning. In: Postlethwaite, T.N., Husén, T. (eds.) International Encyclopedia of Education, 2nd edn. Pergamon Press, Oxford (1992) 15. PROSTAR (Healthy Programme Without AIDS for Youth), http://www.prostar.gov.my/JR_Rokok_BI.htm 16. Radhakrishna, G., Saxena, A.: Application of Cognitive Dissonance Theory to Reduce Dropouts in Distance Education System. In: ICDE International Conference, pp. 1–5 (2005) 17. Simmons, V.N., Webb, M.S., Brandon, T.H.: College-student smoking: An initial Test of an Experiential Dissonance-enhancing Intervention. Addictive Behaviours 29, 1129–1136 (2004) 18. World Health Organization Smoking Statistics, http://www.wpro.who.int/media_centre/fact_sheets/ fs_20020528.htm 19. Lenert, L., Muñoz, R.F., Stoddard, J., Delucchi, K., Bansod, A., Skoczen, S., Pérez-Stable, E.J.: Design and Pilot Evaluation of an Internet Smoking Cessation Program. Journal of the American Medical Informatics Association 10(1), 16–20 (2003) 20. Lucero, A., Zuloaga, R., Mota, S., Muñoz, F.: Persuasive Technologies in Education: Improving Motivation to Read and Write for Children. In: Persuasive Technology, pp. 142–153. Springer, Heidelberg (2006) 21. Aggarwal, A.K., Bento, R.: Web-based education. In: Aggarwal, A.K. (ed.) Web-based learning and teaching technologies: Opportunities and Challenges. IGI Publishing, USA (2000)
Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application Eriko Iwabuchi, Maki Nakagawa, and Itiro Siio Ochanomizu University 2-1-1 Otsuka, Bunkyo-ku, Tokyo 112-8610, Japan {iwabuchi.eriko,nakagawa.maki}@is.ocha.ac.jp, [email protected]
Abstract. In this paper, we present a system that aids people in wearing makeup easily and make the process enjoyable. We call the proposed system the “Smart Makeup Mirror”, which is an electronic dressing table that facilitates the process of makeup application. In this system, we place a high-resolution camera above a computer display and have added some functions such as “Automatic zoom to a specific part of the face”, “Display the face from various angles”, “Simulation of lighting conditions”, and “Internet voting on better makeup results” to facilitate the makeup application process. People who use this device for applying makeup will obtain highly satisfactory results, while enjoying the process.
2 Smart Makeup Mirror Figure 1 shows the over view of the proposed system. As part of this system, we place a high-resolution camera (Point Grey Research, Grasshopper, with IEEE-1394b connection, 1624 x 1224 pixels, 30 fps) above the computer display. The camera is trained on the woman, and a mirror image of the woman is shown on the screen; thus, the device functions and assists the woman in applying makeup effectively while looking at her image. There are some researches that use a standard-definition TV camera and display images as a mirror would [2][3]. In the proposed system, we use a high-resolution camera in order to enable practicable makeup application. In order to realize the functions described later in the paper, we processed the images captured by the camera. We set up a low-resolution camera (Logicool, Qcam Pro for Notebooks, in a 320 x 240 pixel mode, 30 fps) for such image processing. We used OpenCV in the image data processing and FlyCapture for the controlling the high-resolution camera. These cameras are connected to a PC (Gateway GT5092j) with Windows Vista OS. Moreover, we set up an infra red range sensor and a proximity sensor for performing the following functions. These devices are controlled with a microcomputer, and communicate with the PC program through serial connection. We developed the following functions to support the makeup application process.
Fig. 1. Overview of Smart Makeup Mirror
3 Makeup Support Function Using this system, users apply makeup while looking at the computer display where an image of the user’s face is captured by the high-resolution camera is displayed. This configuration is slightly different from a realistic mirror because the user’s gaze
Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application
497
is not aligned with the image, because the camera takes pictures from the top of the display. The response of the image may be delayed because of computation. Further, the resolution is lower than that of a realistic mirror, and the image may be out of focus where a user moves considerably from the focusing position of the camera. These differences may obstruct the makeup application process. The purpose of this research is to implement an artificial mirror with functions that cannot be achieved by using a realistic mirror in order to compensate for the abovementioned differences. In order to estimate the influence of the abovementioned differences on the makeup application process, we applied makeup using an early-stage prototype of our electronic mirror. The following observations were made. y Satisfactory makeup application using the electronic mirror is generally possible. y However, a function that enables gaze alignment is required. When a user tries to see an enlarged view of her face and comes considerably close to the camera, the face image gets blurred as the camera does not have an auto-focus mechanism. From the above observations, we conclude that at least two additional functions are required to make the electronic mirror as effective as a realistic mirror in the makeup application process. One is to capture and display the user’s face from the front, and the other is to enlarge this image for ease of use. The details of these functions are explained in the following sections. 3.1 Automatic Zoom and Pan During the makeup application process, we usually bring our face close to the mirror in order to ensure that our makeup is satisfactory and to check the makeup near certain face parts such as the eyes and the mouth. This is a critical part of the makeup application process, is frequently needed, and is often the most troublesome as it requires a strenuous and awkward posture. For this purpose, we developed an automatic zoom function in the proposed system for zooming at a particular at part of the face. When a user brings the makeup tool, with an attached color marker (a small green sticky label), near her eyes, the camera recognizes the marker, zooms in, and switches to an enlarged image of the eye. In order to detect the color marker, it is not practical to use the high-resolution camera image because the processing time will increase significantly and the response will be unacceptable: moreover, the accuracy of position detection achieved by the high-resolution camera is not required. Therefore, we use an additional low-resolution camera for recognizing markers. This enables users to apply makeup without having to physically approach the display. 3.2 Intuitive Magnification Rate Control A professional makeup artist recommends having at least one magnifying glass at hand as it aids in carrying out a thorough checkup of the applied makeup [4]. We developed a “magnifying glass function” in order to provide finishing touches to the makeup application process, thereby improving the quality of the applied makeup. The proposed system uses an infrared range sensor to measure the distance between the camera and the face. Further, the magnification rate of the display image can be
498
E. Iwabuchi, M. Nakagawa, and I. Siio
increased/decreased by simply and slightly approaching/leaving the display. A similar technique was recently used for a document browser [5]. 3.3 Left and Right Reversing Mirror A professional makeup artist will always recommend checking a person’s appearance from the viewpoint of another person standing in front, with the help of a left and right reversing mirror (the reversal mirror) [4]. The impression of the hairstyle: the shape of the eyebrow, eyes, and lips: and the positions of moles are different, when the face is seen in a reversal mirror. Therefore, we added this functionality to the proposed system, enabling the system to as display the image captured by the camera as a left and right reversing mirror would. 3.4 Profile Check In everyday life, the face is observed not only from the front but also from the side, from the bottom, and in the diagonal direction. Thus, while applying makeup, we must inspect the face from various angles. In the proposed system, the camera captures the image of our face and displays it for several seconds. By looking at that picture, we can inspect the makeup on the face from various angles. As a result, it is possible for us to check our profile and our view from the back, which is not possible to observe in a conventional mirror. When a user captures her face while she is looking at the camera, she can check the gaze-aligned view of her face that a conventional mirror always provides. 3.5 Lighting Mode It is usual that to apply makeup in a place that is as bright as possible. However, the makeup of a user and hence her appearance may appear different under different lighting sources such as sunlight, fluorescent lamps, and incandescent lamps. This system simulates the lighting conditions depending on the surroundings. (Fig.2) The lighting conditions are makeup mode, office mode, cloudy mode, clear weather mode, red sunset mode, and candle mode. The selected lighting mode is displayed in the title bar of the interface and can be confirmed easily. 3.6 Makeup Log When we change cosmetics, tools, or the method of makeup, it is difficult to find differences in the face between how our face looked with the old makeup and how it looks with the new makeup. Moreover, it is also difficult to see our own face objectively in a conventional mirror, because we usually hold a good expression in front of the mirror. The smart makeup mirror allows us to save the image being displayed to a computer file with a particular file name and date. By using this function, we capture an image of our face after applying makeup. This will help us to accurately compare the color, brightness, or texture that suits our face by capturing an image every time in the same environment.
Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application
499
Fig. 2. Differences in the face appearance in different lighting modes
3.7 Noncontact Operation While applying makeup, our fingers become dirty and it is undesirable to touch anything with the dirty hand. All the operations of the abovementioned functions can be operated without touching any part of the system. Automatic zoom to a specific part of the face and intuitive magnification rate control have already been achieved by noncontact by using a color marker (makeup tool) and the range sensor (position of the body). We attached four proximity sensors to the right of the display. On bringing our hand close to the proximity sensor, we can operate the left and right reversing mirror, profile check, lighting mode, and makeup log functions.
500
E. Iwabuchi, M. Nakagawa, and I. Siio
4 Evaluation Test 4.1 Test by the Author One of the authors (a 23-year-old woman) applied makeup by using this system. Figure 3 shows the steps from “no makeup” to “full makeup and hair arrangement”. The author’s opinions are listed below: y It is easy to line the lower eyelid but not the upper eyelid. y For some makeup activities, the timing of zoom switching and position of color markers need to be adjusted. After considering these findings, we have improved the timing of zoom switching, and the position of the marker.
Fig. 3. Steps from “no makeup” to :full makeup and hair arrangement.”
4.2 Test by Ordinary Women We have carried out an evaluation test (Fig. 4). Two women (age: 23 and 25 years) participated in the test. They are students in our laboratory and familiar with how to operate a computer. They applied eye shadow, eye liner, eye blowing out, mascara, lipstick, and blush on. The results are given below: y The two women found the proposed system easier to use than they expected. y They did not like the fact that they could see spots and dullness more clearly in the electronic mirror than in a conventional mirror. y They liked the automatic zooming function. y They occasionally lost sight of the spot where they were applying make up when the system switched to an enlarged image. y They found the left and right reversing mirror function useful for checking of their appearance. y They also found the ability to change the lighting mode useful.
Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application
501
y They found it easy to apply makeup to the lower eyelid, but difficult to do so on the upper eyelid. y They found it difficult to use the eyelash curler because of the motion delay caused by the computation. Difficulty of applying makeup to the upper eyelid was pointed out by both the author and the two subjects. We will describe the improvement related to this issue in the following section.
Fig. 4. Evaluation tests by two women
5 Adjusting Position of Display In order to decrease the difficulty in applying makeup to the upper eyelid, we carefully observed users when they applied makeup to the eyelid. When they applied makeup to the upper eyelid using a mirror, they wanted to see their face from below. For this purpose, they raised their head and lowered their line of vision. (Fig. 5 left) Conversely, when they applied makeup to the lower eyelid, they lowered their head and glanced upward (Fig. 5 right). In the prototype system, the image is displayed at the upper part of the display: therefore, users felt that it was easier to apply makeup to the lower eyelid than to the upper eyelid.
Fig. 5. Application of makeup to upper eyelid (left) and lower eyelid (right)
502
E. Iwabuchi, M. Nakagawa, and I. Siio
This problem could be solved by changing the display position of the image according to the makeup activity to the lower and upper eyelids. Thus we have implemented a function that displays a magnified image of the eyes at the upper part of the display when the user applies makeup to the upper eyelid, and displays the magnified image at the lower part of the display when the user applies makeup to the lower eyelid. The user’s activity of applying makeup to the upper or lower eyelids is detected by means of computer vision through the low-resolution camera. Thus, users easily apply makeup not only to the lower eyelid but also to the upper eyelid.
6 Internet Voting By using the makeup log function, a user can capture an image of her face after applying makeup. Therefore, she can accurately compare the color, brightness, or texture that suits her face by capturing an image every time in the same environment. However, it is not sufficient to just find the makeup method suited to the user. Even if a user thinks that the one of the makeup results is suitable for her, other people may not think so. Then, we have developed a web page function that the user’s friends can use to judge whether the makeup suits the user (Fig. 6).
Fig. 6. Web page where users’ friends can vote
Smart Makeup Mirror: Computer-Augmented Mirror to Aid Makeup Application
503
The picture of a face after applying makeup is open to the public on a web page that is password protected. The page shows pictures of a user’s face in a favorable order. User’s friends who are given the password can change the order of pictures with the arrow buttons on the page. They can arrange better pictures above worse pictures. By using this web page, a user can get useful opinions from her friends in an anonymous manner. This page would provide objective opinions on her makeup.
7 Conclusions and Future Work We have proposed and implemented an electronic mirror that facilitates the process of applying makeup and makes it enjoyable. During the feasibility test with a female subject (age: 23 years), we have received favorable comments for the system. We carried out the evaluation tests with two women (age: 23 and 25 years). Further, we improved the system on the basis of the result of the tests. Additionally, we developed a web page function that allows friends to vote on a makeup result that suits the user the most. We are planning to verify the utility of this system by using it for the long term.
References 1. Takagi, S., Namikawa, C., Yoshimoto, F.: Advice System for Progress in Makeup Skill. The Journal of theSociety for Art and Science 2(4), 156–164 2. Kawauchi, H., Inoue, A., Ichimura, S.: Electronic make-up mirror that supports point make-up in real time. In: The 69th national convention of IPSJ, pp. 4-201–4-202 (March 2007) 3. Furukwa, T., Tsukada, A.: Magic Makeup Mirror – Makeup Simulation based on Real-time Face Recognition. Imagelab 13(10), 34–38 (2002) 4. Watarai, H.: The true basis of make-up. Studio Tac Creative Co., LTD, Tokyo (2006) 5. Harrison, C., Dey, A.K.: Lean and zoom: proximity-aware user interface and content magnification. In: Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing System, CHI 2008, Florence, Italy, April 05-10, 2008, pp. 507–510. ACM, New York (2008)
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work: The Case of IT Service Delivery Eser Kandogan, Eben M. Haber, John H. Bailey, and Paul P. Maglio IBM Almaden Research Center, 650 Harry Rd., San Jose, CA 95120 {eser,ehaber,baileyj,pmaglio}@us.ibm.com
Abstract. IT service delivery is challenging to study. It is characterized by interacting systems of technology, people, and organizations. The work is sometimes reactive, sometimes carefully planned, often risky, and always complex and collaborative. In this paper we describe how we’ve learned about IT work, using a variety of methods including naturalistic observations, contextual interviews, surveys, and diary studies. We provide examples of our study results, showing what we’ve learned with the different methods. We argue that to effectively study such systems, a variety of methods may be needed to complement insights and validate findings. We found that naturalistic observations were extremely time and labor intensive, yet offered us the time and space to observe unplanned events and long-lasting tasks, bringing out the full complexity and risks involved in real work. Contextual interviews and diary studies provided fewer details, yet gave a broader context to individual’s work. Surveys provided an even broader picture, going beyond individual differences, yet they were limited by details and issues of sampling.
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work
505
administration work, and they ignore the everyday constraints and issues that make the work dynamic and reactive to the unplanned events. We believe short lab studies offer only limited insight into use cases and fail to sufficiently explore the design space. We found ethnography a good fit for studying IT administration work. Naturalistic observations and contextual interviews conducted in the field allow inquiry of phenomenon within its real-life context, allowing time and space to see unplanned, transient and long-spanning events and tasks, all of which we found to be characteristic of system administration work. Field studies allow the observer to be situated in the work environment for extended periods, and they help facilitate in-depth study of the development, adoption, and use of new tools, practices, and organizations. We complemented field studies with diary studies and surveys to validate our findings across broader time scales and populations, though we achieved less detail with these methods. In this paper, we present examples from our findings to demonstrate the benefits and challenges in our study methods.
2 Studies of IT Services We conducted a series of studies in IT service delivery organizations employing methods including naturalistic observations, contextual interviews, a survey, and a diary study. In 16 site visits to large corporate, university, and government data centers across the United States, we observed and interviewed more than 30 IT workers. We observed the work practices people involved in management of security, databases, web sites, storage, operating systems and data center operations [1]. Our methods were as follows: y Contextual Interviews: 30-90 minute interviews with subjects at their desks, inquiring about their work and encouraging them to show us examples of important tools, techniques, and artifacts they use on a daily basis. y Naturalistic Observations: following, observing, and if possible recording all the work-related activities of an administrator for anywhere from a few hours to a week. This usually involved two researchers, one to operate audio/video recorders, and the other to take notes and occasionally ask questions (though trying to avoid interrupting the flow of work). We asked participants to speak aloud if possible, and to put phones on speaker. At the end of each day, we asked clarifying questions about the observations from that day. Additionally, we collected physical and electronic materials and took pictures their work environment. y Surveys. To validate our findings across a wider population, we administered one survey on tool use to 100 administrators. We have also worked with the SAGE organization which performs an annual survey on 1000-10,000 sysadmins, looking through their data for further correlations, and also adding a few new questions. y Diary Study. One of our observation subjects was willing to share his 10-month diary listing, for each day, all the work-related activities he had performed that day. Below we present samples from our findings to illustrate the results from each method. The first case shows how naturalistic observations were able to illuminate the extremely complex world of security administrators at a university computing center. In the second case, we employed contextual interviews to examine an organization
506
E. Kandogan et al.
delivering storage management: Centralized Storage Services (CSS1). Here, contextual interviews helped us gain insight into the workings of the larger organization through descriptions of work processes that could take as long as a month with multiple people across different organizations. In the third case we describe our findings from our survey of trust issues in the design of system administration tools [6], which helped validate and refine some of our observational findings. We also describe some of our experience working with the data from the SAGE survey. Finally, we discuss a diary study that provided long-term validation for our observational findings with respect to collaboration between administrators. Let’s describe each in more detail. 2.1 Case 1: A Game of Cat and Mouse… We conducted two separate week-long observations of the five-member data center security team at a large public university in the United States. We saw how their dayto-day work included continual monitoring for new security threats, policy formation, and helping remove security vulnerabilities. We also got to watch a notable episode, as they responded to a world-wide security incident, Stakkato. Stakkato had persistently attacked military, educational, and government sites across the United States and Europe. We observed the senior security administrator, Joe, handle this incident through both local work and also collaboration with other educational and government institutions. Collaboration was necessary because the attacks came through a complicated maze of computers in as many as 7 countries, making tracking very difficult. Joe was working with Aaron, a junior security administrator, and others to respond to the attacks. This was very much a game of cat-and-mouse: the attacker would use vulnerabilities to gain access to machines, while the security admins knew about vulnerabilities in the attacker’s tools that they would use to try and trace the attack back to its source. Often a compromised machine would be left vulnerable to allow more time to trace the attacker. While the center had not been attacked for couple of weeks, the attacks remained a primary concern for Joe, and appeared to have become a personal issue for him. Thus Joe seemed sad and frustrated when he received the news that a compromised machine at another institution was turned off – he wanted the machine to stay on so he could continue to use it to trace the attacks, yet the institution that owned the machine didn’t want to leave it vulnerable any longer. Tracing attackers was no easy task. It was technically challenging as security admins needed to exploit the particular techniques the hackers were using. On a social level, it was even more challenging as they needed to coordinate work with several sites and develop an understanding to share information that could be sensitive. Security staff at a number of institutions held weekly calls and exchanged information, tools, and strategies through email, phone, etc. Joe played an important role coordinating the effort and communicating their findings: Joe: Usually they [the attackers] come in from Europe to a machine in the US, and they make either one or two more hops before they start launching any attacks. We had that narrowed down to where they are coming in, but that site wasn’t able to help 1
To protect the identity of the people, organizations, and companies, we used fictitious names throughout the paper.
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work
507
us. So, we knew that machine, we knew the second machine they are hopping to, and there is a third machine that was here in town that they are using as well. I need to call them today, and I need to track down some of these other sites, the admins there. The particularly challenging issue was that the attackers kept coming back relentlessly using new malware. Upon reviewing session logs Joe noticed a number of new exploits and asked Aaron’s help to find more out about them. Joe and others at the center used an elaborate directory structure to collaborate on incidents. They had directories for each incident to keep scans and exploit code organized: Joe: They have two new things in there, that I didn’t notice. I told them about identity key there (pointing to ingresd.x.x.edu) and I am not sure what it is from. They are trying to figure out what it is from as well. […] But that mod_rootme, we need to find out what that is. […] The other thing they are using is that ussl thing. Those are two new tools, I haven't seen them used before. Aaron’s research on several security sites revealed that mod_rootme was a highthreat vulnerability of Apache Web Server. It allowed a remote user to get a root shell without being noticed as communication was disguised as normal web traffic. He also found a copy of the exploit source code and noticed a couple of identities there: printf("[*] named 8.2.x ( < 8.2.3-REL) remote root exploit by lucysoft, Ixix"); printf("[*] fixed by [email protected] and [email protected]\n\e");
A web search on these identities, i.e. lucysoft, [email protected] and [email protected], led him to the BIND exploit. Aaron explained that while it was very difficult to understand source code, references to identities often led him to further clues on the exploits: Aaron: […] and that is what I generally look for. Like most of them have assembly code or something else. Rather than spending time on what exactly the exploit is doing, first I try to look at the comments, explanations, and signature of the authors. Because generally all the hackers they tend to write their own signature handle. Aaron also noticed another source code (a.c) in the sessions which he discussed with Joe, later in his office: Joe: Okay. Where was that a.c? Was that in there?? Aaron: Uhh. Yeah, it is in canopy. Joe: So, this is a BIND one, huh? We should be able to build, put that together. Aaron: main.c and main.h Joe said they would often try these exploits in quarantine environments on virtual machines with limited connectivity and learn how they worked. When Aaron got back to this office and began working on other tasks, he got an email alert from the intrusion detection system, Bro. Aaron and his colleagues used several automated systems that scanned network traffic and computer activity for suspicious events. This alert was about new activity on a formerly-compromised host: Watchdog found the following alerts in tcpread. These seem to be coming from the known compromised hosts. Please take time to investigate. Non-XX IP (Once Compromised IP’s) x.x.31.10 #nyx.engine.xx.edu (6/8 from victor, used as login to XX) XX IP’s Connections > Jul 27 15:10:14 x.x.31.10 0.1kb > x.x.63.22/http 711kb 0.0b %77125
508
E. Kandogan et al.
The alert stated that there had been recent network activity, an HTTP transfer of a file of size 711kb with the log ID 77125, to a local machine. This was a machine reported as compromised by Victor, a fellow security administrator at another site, on 6/8. Aaron had customized the intrusion detection system to register certain hosts as compromised and added further contextual information such as who reported it and when and how it was compromised in the past. This alert was of utmost importance as the host in question was directly related to the Stakkato incident. When Aaron examined the HTTP logs he found out that a particular file had been downloaded. Aaron first identified the host name, and then searched for the owner of the machine on an online database. Aaron immediately did a search and found his home page. Examining web page he discovered that the user worked on high performance computing, making the file in question a legitimate download. Both Tom and Aaron were relieved as this potentially serious alert turned out to be a false alarm. Responding to a security attack required a range of activities, from minute-byminute monitoring of activities to long-term research, planning, and collaboration across multiple sites. Security administrators have a fascinating culture, always trying to learn about vulnerabilities of the enemy, while preventing the enemy from realizing what is known about them. Naturalistic observation was very helpful in enabling us to gain a detailed understanding of the full richness of this environment. 2.2 Case 2: Everybody Thinks That They Are Number One We conducted a three-day study with several contextual interviews in the Centralized Storage Services (CSS) group at a large IT Service company. CSS offers enterprisescale storage solutions centered at one location, with managed storage resources distributed over 10 sites across the world. With a novel offering that delivered ondemand storage capacity allocation, 24 by 7 monitoring, and management services, CSS grew its customer base from only 2 customers with 2 high-end “SHARK” storage systems to 27 customers with about 50 SHARKs, just within two years. We interviewed Henry, who worked in the Storage Operations Center (SOC) as the storage team “focal” (i.e., project lead) since CSS’s infancy. Henry’s group handled anything having to do with storage, from technical work such as updating microcode software to coordinating work such as negotiating change windows with customers. Henry explained that customers got more or less equal treatment from CSS, though certain tasks took a higher priority: Henry: Everybody thinks that they are number one, you know. So, our model is that nobody is given priority over anybody else. Yes, we set a priority over how it has to be done cause basically if there is an outage, lets say, filesystem has reached 99.99%, and it is about to crap out and cause the server to break down, we may have to push doing that allocation before that one. Nonetheless, with rapid customer growth they hit technical limits that required significant human intervention, as illustrated by one incident the previous week: Henry: The SHARK has a limitation that allows you so many port logins. We hit that limit. So, support came back and said that the only fix for that was to recycle [reboot] the host bays. So, now you gotta get approvals from the customer. That is a problem. We had like 50-60 servers, which has a different application for each one. You gotta get
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work
509
approval from all these people. So, we finally got approval. Problem was fixed but it took like a week and a half to two weeks away, just to get approvals. Customer is happy now. They see the storage, but we hit one of those limits that is in a book like that thick. Rapid growth not only brought technical limits but also organizational issues to deal with. A simple setting, maximum number of port logins, when exceeded required approvals from each customer. While customers were isolated from each other in terms of service quality, we saw that collective use of shared storage resources led to new issues that would not have risen otherwise. In CSS, they aimed for no outages and downtimes. This meant that operations needed to be performed concurrently, i.e. without taking the whole cluster down. One of the critical issues in concurrent operations was microcode updates. To address this issue Henry and others developed a 4-week process, depicting tasks and interaction with customer as a flowchart. The first two weeks were primarily for aligning requirements and schedules with the customer. Handshaking occurred from both a technical perspective and social perspective: Henry: It is a four week process from the point we approach the customer. We establish the [change] window, we tell them here are the servers we see you have, storage on your SHARK, and levels of SDDs you running on these servers. A lot of times what you see on the SHARK is not what you get. So, you have to make sure there is a handshake between the zoning and the SHARK. So, between weeks 2 and 3 we do all that handshaking making sure that SAs are aware. Is it compliant, if not we tell you, hey are you willing to bring down your server? The remaining two weeks were primarily for performing technical work. Week 3 was for upgrades to meet minimum requirements. Week 4 was final check with the customer detailing schedules, contact information and technical details to give a last chance to withdraw. Once SOC performed the change they closed the ticket with a final report to the customer. Synchronization and coordination of work among parties always presented a challenge. Henry mentioned that when storage work was completed in SOC, it might still take time to bring up the servers to verify the change. And he found it always frustrating when customers could not see their storage and he had already moved to another issue. Remembering changes from weeks before was a burden, especially as administrators were not devoted a specific customer. To help with this they developed a database to keep track: Henry: Do you really remember what you did four weeks ago? We have a database that can house that information so you can pull it up easily. Because if I am working on customer A today, three weeks from now, customer C says, well, you worked on my stuff a month ago. Customer A is on my mind right now. I am not thinking about customer C! Henry described how most of these challenges at CSS were manageable through careful planning and processes developed in the SOC. Henry and others developed spreadsheets and flowcharts to guide work and coordinate with customers. Lags within the organization presented a special problem. Differences in storage design, allocation, and server board times were particularly difficult to deal with. Solving these issues required invoking organizational memory to recall problems, participants, and solutions to revisit and resolve them with the right people and tools.
510
E. Kandogan et al.
2.3 Survey Studies In our field studies we observed numerous cases where the tools failed to support system administrators, reporting incorrect data, working unreliably, or simply being poorly aligned with the administrator’s work practices. Given the risk factors in the IT administration we expected that trust played a major role in the way administrators used and interacted with information, people, tools, and systems. To see if these findings were valid across a broader population, we performed a survey of about 100 system administrators recruited through online news groups, and local and national system administrator organizations. In the survey we asked specific questions about administrators’ comparative qualitative judgments of the CLIs and GUIs for system management they use regularly. One set of questions concerned the perceived speed, ease of use, reliability, robustness, accuracy, trustworthiness, and likeability of the CLIs vs. GUIs they used for system administration. Additional questions about both CLI and GUI tools were based on McAllister’s survey, using a 7-point Likert scale, which measures monitoring behavior and cognition-based trust for interpersonal relationships in organizations. The details of the survey results are included in [6]. In sum, we found statistically significant preferences for command-line tools over graphical tools in the areas of trustworthiness, reliability, robustness, accuracy, and speed. In response to the McAllister questions, we found that cognition-based trust ultimately plays a major role as opposed to seemingly affective factors. We received some comments, such as, “I know what I am doing. Please NO MORE GUI! If people need a GUI they are not qualified to be doing whatever they are trying to do.” In search of further statistically grounded data, we also worked with SAGE, the System Administrators Guild, which annually administers a survey intended to collect demographic and salary information. This survey is very large, with between 680 and 9600 respondents depending on the year. The survey includes questions about supervisory roles, and also administrators per site, from which estimates of administrator team size can be inferred. Further, we were able to add some explicit questions about collaboration into the 2007 survey. The results suggest that administrators work in teams regardless of company size, and that the majority of them spend a third or more of their time working with others. We did need to carefully formulate the questions to make them easy to answer, since subjects often failed to answer complex or openended questions. 2.4 Diary Study George was a web-application administrator in a large corporate service delivery center. We observed him in two separate week-long field studies conduct several tasks such as installing web servers, configuring authentication servers, trouble shooting problems, etc. For his own purposes, he kept a record of tasks performed daily, such as troubleshooting (e.g. “Continued with webseal problems on acp2”), and meetings (e.g. “RFS customer call - 16 new servers coming in”), etc. On a typical day, he had about five to ten records. He did not, however, attribute the amount of time spent on each task. George was willing to share his diary with us. We analyzed it by categorizing tasks in each record, and by noting what tools and people were mentioned. Our purpose was to get a break down of tasks over an extended periods and get a sense of the
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work
511
extent of collaboration in conducting tasks. Our analysis confirmed that collaboration was indeed practiced extensively with about 23% of the records referring to meetings with other system administrators. Of the remaining tasks such as planning (21%) and trouble-shooting (11%) , the diary mentioned other admins nearly half the time. Across all tasks he indicated collaborating with others about 45% of the time. This was a high number even though we suspected that collaboration could have been downplayed in self-reports. We were able to evaluate self-reporting by comparing a 3 hour troubleshooting session we video-recorded with the diary for the same day. In coding the session, we observed him working with one of ten other people for 90% of the time. His report, however, discounted the extensive collaboration simply saying, “Communication problem with PD server - we had port opened in wrong direction were able to use extra port 7236 - works fine now.” The only indication of teamwork was mentioning “We” as opposed to “I”.
3 Analysis: Multiple Methods for a Complex Topic System administration work is clearly a challenging area to study. The work is sometimes reactive, sometimes carefully planned, often risky, and always complex and collaborative. We have found that a variety of methods is necessary to generate a complete and accurate picture of administration work: ethnographic approaches such as naturalistic observation and contextual interviews, as well as surveys, and diary studies. Consider the case of the security administrators reacting to alerts about potentially suspicious activity. It is hard to imagine any way of capturing the intricate and coordinated response and evaluation of the alerts, short of watching them unfold. Consider also the complexity of the technological arms race and information warfare going on between the security administrators and their adversaries. The admins are continually researching vulnerabilities in their own systems. They must also search for the often subtle signs of intrusion, and try to find vulnerabilities in the intruders’ tools. Attacks, when detected, are often permitted to continue, in order to trace them back to their source. Any data left behind by the attackers must be analyzed, reverse engineered, or even decrypted to find clues about the attack’s origin. Furthermore, attacks often come from compromised machines at other institutions, requiring collaboration across the broader community. Observation seems to be the only way to get a clear picture of this very complex environment. Contextual interviews are an important complement to naturalistic observations, they provide the context, history, motivation, and descriptions of exceptional or rare events that might be difficult to tease out from observations alone. Our interviews with Henry about his storage administration work gave us insights into the history of his group, and the clearly defined workflow that formalized the interaction among the participants. For example, Henry explained the elaborate flowchart for the four-week long interactions with the customer when updating microcode software. A significant part of the interactions focused on synchronization and coordination. We learned that rapid growth in the customer base led to technical issues, which in turn led to a weeks-long process to get approvals from all the participants involved and perform changes. Dynamic growth also required them to perform flexible resource allocation. This reflected in the way CSS got organized over long periods of time.
512
E. Kandogan et al.
An ethnographic approach was extremely valuable in studying IT service delivery because it allowed us to capture work broadly and flexibly in time and space with all the interdependent variables of technology, people, and organizations as they emerged during the course of study. Thus, we argue that to understand fundamental issues and formulate hypothesis of work in the context of larger systems of people and technology an ethnographic approach is invaluable. The most common alternative to ethnography, in-lab user studies, begin by eliminating or controlling variables, thus they are more suited to achieve shorter term goals such as measuring usability or task performance. Another alternative is modeling usage in a computational way in terms of actions and affordances. This approach does help to get statistical data on performance, yet it may be neglecting important aspects of experience [3, 4]. Ethnography goes beyond actions of individuals and considers cognition also from a social perspective. Thus, ethnographic studies are better suited for verifying hypothesis. Expecting similar short term goals such as design guidelines from ethnographic approaches is not the best criterion for the relevance, utility, and quality of an ethnographic account of work, and such an expectation indeed underplays potentially radical gains that can be achieved by laying out a broader design space [2]. Ethnography does have its limitations or challenges, too. First, ethnographic methods are expensive. It takes considerable time and commitment to conduct long-term observations. It also takes time to establish the trust of the members of the study group, who must put up with being observed, followed, and questioned. We found it considerably easier to obtain permission to observe and record employees within our own company; obtaining the legal agreements required to do such work at other companies proved to be extremely challenging. Once the data was collected, it took even more time to analyze: an hour of video might require a day to a week to truly understand. We even developed our own video analysis tools to speed up the process of video analysis and annotation. Ethnographic studies are not ideal for developing low-level design guidelines that necessitate quantification of user needs and performance. Absence of quantification, and lack of control for variables are often used to question the validity and reliability of results though both issues are addressable when ethnography is used to derive hypotheses. Validity is addressed by collecting data from diverse situations and sources and considering all to expand hypotheses instead of discarding any data that did not fit a hypothesis. Reliability is addressed by triangulation where data is collected from multiple sources, across situations, and time. Last but not least, ethnographic accounts are descriptive rather than prescriptive thus making multiple interpretations a possibility, a quality that can in fact be considered as an advantage. We faced several challenges as we conducted our ethnographic field studies. First, the work of the system administrators was very complex, and outside of our own areas of expertise. It was nearly impossible for us to track and understand commands, errors, and discussions surrounding them as the work was unfolding. Likewise, a lot of the work was dynamic and concurrent, many issues were being addressed at the same time, through simultaneous conversations among staff and interactions with multiple systems. To address these issues we utilized video analysis to help us better understand the work. However, it still required extensive effort afterwards to make sense of all the technical issues and unfold multiple concurrent interactions into coherent parts, and establish cause and effect. All this required meticulous attention to every detail, on the computer screen and utterances of the participants, as well as
Studying Reactive, Risky, Complex, Long-Spanning, and Collaborative Work
513
technical documentation studied after the fact. Tracking issues from multiple perspectives of the participants also helped to the extent possible. Unlike traditional anthropologists, we were not able to spend months or years with our subjects. This meant that our observations and interviews drew from a limited sample. Data sources such as George’s diary were extremely valuable in showing that collaboration practice was not an isolated case but occurred on longer time spans across different types of tasks. The extremely labor-intensive nature of naturalistic observation also left us with a limited population sample. Surveys were one means to validate findings across a broader population, especially when we could frame very specific questions to ask.
4 Conclusion The purpose of this article is to discuss the strengths and limitations of various study methods as applied to the dynamic, risky, complex, and long-spanning work environment of IT service delivery centers. We argue that observational methods offer both time and space to observe short-term reactive work and long-lasting phenomena and tasks with its full complexity with all considerations of the interplay of technology, people, and organization systems through detailed illustrations of two cases from our field work. We also describe how contextual interviews can provide further background, details, and historical information. We tried to address some of the shortcomings of ethnographic approaches through surveys and diary studies that offered validity beyond time and individual differences. Finally, we discussed challenges we faced, and made recommendations on possible ways to address them.
References 1. Barrett, R., Kandogan, E., Maglio, P.P., Haber, E., Takayama, L., Prabaker, M.: Field Studies of Computer System Administrators: Analysis of System Management Tools and Practices. In: Proc. of the CSCW 2004, pp. 388–395. ACM, New York (2004) 2. Dourish, P.: Responsibilities and implications: further thoughts on ethnography and design. In: Proceedings of the 2007 Conference on Designing For User Experiences (2007) 3. Norman, D.: Emotional Design: Why We Love (or Hate) Everyday Things. Basic Books (2004) 4. Picard, R.: Affective Computing. MIT Press, Cambridge (1997) 5. Spohrer, J., Riecken, D.: Introduction. Communications of the ACM 49(7), 30–32 (2006) 6. Takayama, L., Kandogan, E.: Trust as an underlying factor of system administrator interface choice. In: CHI 2006 Extended Abstracts on Human Factors in Computing Systems, pp. 1391–1396. ACM, New York (2006)
Human Computer Interaction in Virtual Standardized Patient Systems Patrick G. Kenny, Thomas D. Parsons, and Albert A. Rizzo Institute for Creative Technologies University of Southern California 13274 Fiji Way Marina Del Rey, CA 90292, USA {kenny,tparsons,rizzo}@ict.usc.edu
Abstract. Interactive computer generated characters can be applied to the medical field as virtual patients for clinical training. The user interface for the virtual characters takes on the same appearance and behavior as a human. To assess if these virtual patients can be used to train skills such as interviewing and diagnosis they need to respond as a patient would. The primary goal of this study was to investigate if clinicians could elicit proper responses from questions relevant for an interview from a virtual patient. A secondary goal was to evaluate psychological variables such as openness and immersion on the question/response composites and the believability of the character as a patient. Keywords: Virtual Patients, Artificial Intelligence, Clinical Psychology.
Human Computer Interaction in Virtual Standardized Patient Systems
515
One of the main research questions of the virtual human work is in developing these characters so they appear and act like real humans without falling into the Uncanny Valley [3]. The Uncanny Valley was defined by Masahiro Mori in the 1970’s to describe robots and characters that look like humans, but don’t act like humans, or what we expect them to act like. As the realism of the appearance of the character approaches human appearance, for example the face, eyes or skin, while the actions and the behavior don’t, or are off just a little bit, for example blank stares, or lips that don’t move with the same muscle fidelity as humans, this causes an uncomfortable feeling amongst people interacting with them and destroys their believability. Our virtual human system, although they are realistic looking and acting, have not had any problems with falling into the uncanny valley. The focus of the virtual patient is applying these virtual humans to the medical domain to create virtual standardized patients (VP) that can be used to teach interview, diagnosis, and social-interaction skills. The primary goal of this research is to assess the technology and system in eliciting correct question/response pairs from novice clinicians in a clinical interview. A secondary goal is to investigate the impact of psychological variables such as the subjects’ state, current mood, and personality traits, openness to new experiences and immersion upon the resulting question/response composites and the overall believability of the characters. Medical students currently perform interview training with human actors acting as standardized patients. The actors portray some clinical problem in what is called an Objective Structured Clinical Examination (OSCE) [4]. These tests typically take from 20-30 minutes, a faculty member watches the student perform. The evaluation consists of self assessment rating along with faculty assessment. Although schools commonly make use of standardized patients to teach interview skills, the diversity of the scenarios that standardized patients can characterize is limited by the availability of human actors and their skills at portraying the condition. Additionally the actors most likely vary their performance from subject to subject and location to location. This is an even greater problem when the actor needs to be an adolescent, elder or portray a difficult condition. Our process is similar to an OSCE, but the actor is replaced with a virtual patient and an observer is replaced by video recording. Using virtual patients will allow standard performance assessments for all subjects. The virtual patient system was used in a series of subject testing experiments with novice clinicians and medical students. The role of the clinician was to ask appropriate and relevant questions to elicit correct responses from the virtual patient in a structured, yet free flowing, interview for history taking and diagnosis of the character. The results of the subject testing will be discussed as well as the human interaction issues with the system. Enabling rich and engaging interaction with virtual characters in a medical setting will ultimately allow powerful experiential learning engagements for new and experienced students on a continual basis for practice with a variety of patient cases they may get little or no training with. 1.1 Virtual Human Interaction Human to human interaction is very complex, enough so that many people in many fields devote a lot of time trying to understand it, and people find it so interesting that
516
P.G. Kenny, T.D. Parsons, and A.A. Rizzo
most movies produced are about human behavior and relationships. The interactions can vary based on numerous factors in both the person and the social setting. One on one interaction is different than multi-party interactions. Trying to re-create and model these interactions in virtual human systems is a great challenge. The assumptions and expectations that people have while interacting or engaged with other humans is brought over when people interact with the virtual humans [5], and these need to be replicated in the virtual character. For example users of the virtual human system expect to talk to the characters as they would with real humans, sometimes forgetting, or not knowing, the complexity and limitations of the underlying system software. Our VP system interaction is not based on the use of a traditional mouse and keyboard or pull down menus, but uses natural speech recognition. Speech interaction can cause confusion in the user if the character responds in a way that the user doesn’t expect. The user interacting with a person builds a mental model through dialog and non-verbal behavior of that person, if that representation is violated, then the user may lose engagement or be confused. The same confusion can happen with the virtual patient system and performance could suffer. For the medical domain we have some leeway as patients can, and usually do, act in non-traditional manners, thus if the patient responds in an off topic manner then this may not be thought of as incorrect. For our testing of the system we want to allow freedom of interaction and not constrain the user to specific bounds of what they can or can’t say or how they should say it. We usually do not know what kind of questions the clinicians may ask the patient, which makes it hard to design the domain. Additionally, clinicians have varied training and there are multiple approaches to Interviewing and Diagnosis [6] that the system will ultimately have to take into account. However, this freedom of exchange is a good thing to capture for it will allow us to evaluate the technology and character interaction to provide methods for improving or automating the system for a more natural human computer interaction in the future.
2 Virtual Patient System This virtual patient system consists of a computer generated 3D character that interacts with a human through natural speech. The character responds in kind through speech and gestures. Creating virtual patients that interact falls into two main areas; the technology and the domain. The technology needs to support what the character should do and how the user interacts with it, and in this case supporting the medical domain. In our virtual patients we have been concentrating on building characters with psychological problems in contrast to physical problems, they are more dialog based then motion or action based. One of the challenges of building characters for this domain is designing what needs to go into the patients in terms of the dialog, behaviors and actions to fit the patient profile. 2.1 Psychological Medical Domain The role of the clinician during an initial meeting and engagement with a patient is to capture a history of the person, find out what is going on, and try to narrow down the problem in what is called a differential diagnosis by ruling out issues and problems
Human Computer Interaction in Virtual Standardized Patient Systems
517
not relevant to the case. The virtual patient interaction should mimic the real patient interview as much as possible with the goal of allowing the clinician to ask appropriate and relevant questions to elicit correct responses from the virtual patient in a structured, yet free flowing, interview for history taking and diagnosis of the character.
Fig. 1. Justina Virtual Patient
The virtual patient for this research is an adolescent female character with PostTraumatic Stress Disorder (PTSD) called Justina, Figure 1. PTSD usually happens to people after some kind of traumatic event, such as a military engagement or assault and causes changes in behavior of the person. The effects of trauma exposure manifest themselves in a wide range of symptoms: anxiety, post-trauma stress, fear, and various behavior problems. New clinicians need to come up to speed on how to interact, diagnose and treat this trauma. According to the most recent revision to the American Psychiatric Association’s DSM Disorders, PTSD is divided into six major categories; refer to the DSM-IV category 309.81 [7] for a full description and subcategories; A. Past experience of a traumatic event and the response to the event. B. Re-experiencing of the event with dreams, flashbacks and exposure to cues. C. Persistent avoidance of trauma-related stimuli: thoughts, feelings, activities or places, and general numbing such as low affect and no sense of a future. D. Persistent symptoms of anxiety or increased arousal such as hyper vigilance or jumpy, irritability, sleep difficulties or can’t concentrate. E. Duration of the disturbance, how long have they been experiencing this. F. Effects on their life such as clinically significant distress or impairment in social or educational functioning or changes in mental states. Diagnostic criteria for PTSD includes a history of exposure to a traumatic event in category A and meeting two criteria and symptoms from each B, C, and D. The duration of E is usually greater than one month and the effects on F can vary based on severity of the trauma. Effective interviewing skills are a core competency for the clinicians, residents and developing psychotherapists who will be working with children and adolescents exposed to trauma. A clinician needs to ask questions in each of these categories to properly assess the patient’s condition.
518
P.G. Kenny, T.D. Parsons, and A.A. Rizzo
One of the challenges of designing and building these interactive virtual standardized patients has been in enabling the characters to act and carry on a dialog like a real patient that has the specific mental condition for the domain of interest. Additional issues involve the breadth and depth of expertise required in the psychological domain to generate the relevant material for the virtual character. The PTSD domain for the virtual patient allows exploration of the interaction with real end users while still being able to constrain the dialog and character behavior to a small and manageable corpus. The system needs to respond appropriately based on the six categories mentioned above and requires developing dialog for the virtual character to say around each of the categories. Table 1 is a set of example questions the clinician may ask and responses the virtual patient may say. Two additional categories were added for questions that involved building rapport or general questions, i.e. ‘what’s your name?’ and for technical issues, like accidental button presses while thinking about a question to ask. Table 1. Question / Response Categorization for PTSD Category (A) Trauma (B) Re-experience (C) Avoidance (D) Arousal
User Question So, what happened to you that night? Do you still think about what happened? Do you go out with your friends? Do you feel jumpy?
(E) Duration (F) Life Effect
How long has this been going on? Are you upset?
(G) Communication (H) Other
Hi Justina, I’m Doctor… ‘Button Press’
Virtual Patient Response Something really bad happened. Sometimes I feel like the attack is happening all over again I just stay away from everyone now. I feel like I have to watch my back all the time. A few months Sometimes I don’t do anything but stay in my room and cry. Hello I don’t get what you mean.
2.2 Technology The virtual patient system consists of a set of distributed modules of which only a few are interfaces to the user. The user interacts through speech recognition to talk to the virtual character and through the 3D graphics that shows the character’s animation in a virtual environment on a large monitor. The distributed set of components that make up the system form a pipeline that is the information flow from the input of the user to the output of the character. The main components can be divided into three areas: User Input • Speech Input – This component takes the user input from a microphone and translates that into a string of text to be used by the natural language system. The speech recognizer requires a speech and language model. Since everyone has a different voice, i.e. male, female, child, elderly, a different speech model is required and is changed for each user. In our case the users are male or female adults. A language model is required that defines the possible set of words in the domain that can be recognized. The corpus for the virtual patient consistes of 20K words.
Human Computer Interaction in Virtual Standardized Patient Systems
519
Processing • Natural Language – The text from the speech recognizer is sent to a statistical question/response module [8] that picks a response based on the input question. The question/response pairs are matched by hand prior to deployment. For the virtual patient PTSD domain there were 500 questions with 100 responses. If a question is asked that is not in the domain, an off topic response would be given such as; “I don’t know” or “I would rather not talk about that”. The questions and responses were acquired through expert knowledge, roleplaying and best guesses. • Behavior Generation – After a response is selected then gesture animations are applied based on a set of rules that govern the non-verbal behavior [9]. Gestures can be hand movement, body posture, gaze or the like and is only limited by available animations. Character Output • Speech Generation – The speech output can be generated with an automated speech generation system or with pre-recorded voice overs that match the response text. For the virtual patient pre-recorded voice was used. • Animation Output – The animations that were selected for the non-verbal behavior are combined with the generated speech to synchronize and play out together through a procedural animation system called Smartbody [10]. This drives the 3D character in a realistic fashion.
3 Subject Testing Subject testing of the virtual patient system was conducted with medical students to evaluate the interaction, systems usefulness, effectiveness and usability as a medium to communicate with the students in performing the interview task. The evaluation consisted of an assessment of the system as a whole through questionnaires and data collection during the interview. The human computer interaction factor evaluation examined the technology underlying the speech recognition, dialog interaction and behavior of the character for this task with the user. An important issue in the study of intelligent virtual agent interaction is to take into account the users openness to the interaction, new experiences and novel technologies. It has been suggested in a recent study [11] that physiological arousal appeared to be moderated by participant openness. High-absorption individuals may be more capable of imagining that the VP has PTSD when it is suggested. The users’ openness will be compared to their performance to assess if they did better on the interview task. 3.1 Participants Participants were asked to take part in a study where they would interact with an advanced prototype technical virtual patient system, in a similar matter to how they currently perform an OSCE. They were not told what kind of condition the VP had if any. Recruitment methods were by poster advertisements on the university medical campus, and classroom recruitment. A total of 15 people (6 females, 9 males; mean age = 29.80, SD 3.67) took part in the study. Ethnicity distribution was as follows:
520
P.G. Kenny, T.D. Parsons, and A.A. Rizzo
Caucasian = 67%; Indian = 13%; and Asian = 20%. The subject pool was made up of three groups: 1) Medical students (N=7); 2) Psychiatry Residents (N=4); 3) Psychiatry Fellows (N=4). For participation in the study, students were able to forgo certain medical round time with the time spent in the interview and questionnaires, which took approximately 45 minutes. 3.2 Method The subject testing was divided into three phases, a pre-test and pre-questionnaire, the interview and a post-questionnaire. The pre-questionnaire was performed in a separate room from the interview and took about 10 minutes. For the interview the participants were asked to perform a 15 minute interaction with the VP and assess any history or initial diagnosis of a condition of the character. The participants were seated in front of a large monitor that had the virtual patient sitting on a couch in the therapists’ room. The subjects used a head mounted microphone and were required to press the mouse button, talk, and then release the mouse button. The participants were asked to talk normally as they would to a standardized patient, but were informed that the system uses speech recognition and was a research prototype. They were free to ask any kind of question and the system would try to respond appropriately. At the end of the 15 minute exchange they would be sent to another room to take the post-questionnaire. Video, system logs and data from the various modules was logged as follows: • First, the user speech was recorded from the automated speech recognition (ASR) engine, the speech before and after the engine processed it was captured. The before speech was later transcribed to compare with the processed speech. • Second, the text from the natural language (NL) statistical question/response system was saved. The NL system records a transcript of the entire dialog session, this is used later to help analyze the question/response interaction. • Third, system log files of the messages between the modules were captured and could be used to reconstruct what happened during the interaction. • Fourth, cameras recorded participant’s facial expressions and body language during interaction with the virtual patient system to be used for future studies. 3.3 Measures The following measures were used to assess the impact of openness (absorption and immersiveness) upon the “believability” of the system. Prior to the experiment itself, the subjects were required to fill in the following questionnaires: 1. Tellegen Absorption Scale (TAS). The TAS standardized questionnaire aims to measure the subject’s openness to absorbing and self-altering experiences. The TAS is a 34-item measure of absorption [12]. 2. Immersive tendencies questionnaire (ITQ). The standardized ITQ measures individual differences in the tendencies of persons to experience “presence” in an immersive virtual environment (VE). The majority of the items relate to a person’s involvement in common activities. While some items measure immersive tendencies directly, others assess respondents’ current fitness or alertness, and others emphasize the user’s ability to focus or redirect his or her attention. The ITQ is comprised of 18 items, and each is rated on a 7-point scale [13].
Human Computer Interaction in Virtual Standardized Patient Systems
521
3. Virtual Patient Pre-Questionnaire (VPQ1). This unstandardized scale was developed to establish basic clinical competence for interaction with a person that is intended to be presented with PTSD, although no mention of PTSD is on the test. 4. Justina Pre-questionnaire (JPQ1). We developed this scale to gather basic demographics and ask questions related to the user’s openness to the environment and virtual reality user’s perception of the technology and how well they think the performance will be. There were 5 questions regarding the technology and how well they thought they might perform with the agent. After the experiment the subjects filled in the following questionnaires: 1. Presence questionnaire (PQ). The Presence Questionnaire is a common measure of presence in immersive virtual reality. Presence has been described of as comprising three particular characteristics: sense of being within the VE; extent that the VE becomes the dominant reality for users; and extent to which users view the VE as a place they experienced rather than simply images they observed. The PQ is a widely used questionnaire [12]. 2. Justina Post-questionnaire (JPQ2). We developed this unstandardized scale to survey the user’s perceptions related to their experience of the virtual environment in general and experience interacting with the virtual character, in particular the patient in terms of its condition, verbal and non-verbal behavior and how well the system understood them and if they could express what they wanted to the patient. Additionally there were questions on the interaction and if they found it frustrating or satisfying. There were 25 questions for this form. 3. Virtual Patient Post-questionnaire (VPQ2). This scale was exactly the same as the Virtual Patient Pre-questionnaire and will be used in the future for norming of a pre-post assessment of learning across multiple interactions with the VP.
4 Results The present focus is on effect sizes that describe the strength of association between question and response pair composites for a given PTSD category. An effect size of 0.20 was regarded as a small effect, 0.50 a moderate effect, and 0.80 a large effect.
Fig. 2. Effect Size on Question / Response Composites
522
P.G. Kenny, T.D. Parsons, and A.A. Rizzo
Figure 2 shows a chart of the data. Moderate effects existed for PTSD Category A,B,C,G, but only small effects were found for Category D and F. After controlling for the effects of the Tellegen Absorption Scale, increased effects were found for Category A, C, D, and F. To assess the impact of psychological characteristics such as absorption and immersiveness upon the “believability” of the VSP and student interaction we created a composite variable that included scores from the TAS and the ITQ. Strong effects existed between the ITQ and the PQ (0.78), and moderate effects existed between the ITQ and the VSP Post-questionnaire (0.40). These results showed that the users were able to ask question and elicit responses in each of the PTSD categories. Additionally the findings suggest that the presence and openness appears to moderate user reaction and perform better on the task. Future studies should make use of physiological data correlated with measures of immersion to augment and quantify the effects of virtual human scenarios.
5 Conclusion and Future Work Here we focused on effective interview skills—a core competency for psychiatry residents and developing psychotherapists. The keys aspects of the interview that we looked at were: interpersonal interaction; attention to the VP's vocal communications, as well as verbal and non-verbal behavior. Specifically, we wanted to assess whether the user (clinician in training) asked questions related to the reason for referral and received appropriate responses and also made attempts to gather information about the VP’s problems. Finally, we wanted to see if the user would attempt detailed inquiry to gain specific and detailed information from the VP, separating relevant from irrelevant information. The primary goal in this study was evaluative and the Question/response composites were developed to reflect the shared variance existing between the responses of the VP and the users Questions that are necessary for differential diagnosis. In future work we plan to compare the virtual patient system with live standard patient actors to assess if the technology constrains the communications or rapport between the clinician and patient. We will also compare a VP with PTSD against a VP that does not have PTSD to assess if a clinician can make a proper diagnosis. Additional improvements in the language and speech for the domain will allow for deeper and richer dialog. Building virtual characters is an iterative approach that improves the technology with feedback from real users to assess if these systems can be used as effecting teaching and training tools, which we believe is the case.
References 1. Kenny, P., Hartholt, A., Gratch, J., Swartout, W., Traum, D., Marsella, S.D.: Piepol Building Interactive Virtual Humans for Training Environments. In: Proceedings of I/ITSEC (November 2007) 2. Gratch, J., Rickel, J., André, E., Badler, N., Cassell, J., Petajan, E.: Creating Interactive Virtual Humans: Some Assembly Required. IEEE Intelligent Systems, 54–63 (July/August 2002)
Human Computer Interaction in Virtual Standardized Patient Systems
523
3. Mori, M.: On the Uncanny Valley. In: Proceedings of the Humanoids-2005 workshop: Views of the Uncanny Valley, Tsukuba, Japan, December 5 (2005) 4. Walters, K., Osborn, D., Raven, P.: The development, validity and reliability of a multimodality objective structure clinical examination in psychiatry. Medical Education 39, 292–298 (2005) 5. Gratch, J.N.W., Gerten, J., Fast, E., Duffy, R.: Creating Rapport with Virtual Agents. In: 7th International Conference on Intelligent Virtual Agents, Paris, France (2007) 6. Evans, D.H.M., Uhlemann, M., Lvey, A.: Essential Interviewing: A Programmed Approach to Effective Communication, 3rd edn. Brooks/Cole Publishing Company (1989) 7. DSM, American Psychiatric Association (DSM-IV-TR) Diagnostic and statistical manual of mental disorders, 4th edn., text revision. American Psychiatric Press, Inc., Washington (2000) 8. Leuski, A., Traum, D.: A Statistical Approach for Text Processing in Virtual Humans. In: Army Science Conference (2008) 9. Lee, J., Marsella, S.: Nonverbal Behavior Generator for Embodied Conversational Agents. In: 6th International Conference on Intelligent Virtual Agents, Marina del Rey, CA (2006) 10. Thiebaux, M., Marshall, A., et al.: SmartBody: Behavior Realization for Embodied Conversational Agents. In: International Conference on Autonomous Agents and Multi-Agent Systems, Portugul (2008) 11. Macedonio, M., Parsons, T.D., Rizzo, A.A.: Immersiveness and Physiological Arousal within Panoramic Video-based Virtual Reality. Cyberpsychology and Behavior 10, 508–516 (2007) 12. Tellegen, A., Atkinson, G.: Openness to absorbing and self-altering experiences (“absorption”), a trait related to hypnotic susceptibility. Journal of Abnormal Psychology 83, 268–277 (1974) 13. Witmer, B., Singer, M.: Measuring presence in virtual environments: a presence questionnaire. Presence: Teleoperators and Virtual Environments 7(3), 225–240 (1998)
Towards Standardized Pen-Based Annotation of Breast Cancer Findings Suzanne Kieffer1, Annabelle Gouze1, Ronald Moncarey1, Christian Van Brussel1, Jean-François De Wispelaere2, Françoise Kayser2, and Benoît Macq1 1
Communications and Remote Sensing Laboratory, Université catholique de Louvain Place du Levant 2, B-1348 Louvain-La-Neuve, Belgium 2 Cliniques universitaires UCL de Mont-Godinne Avenue G.Thérasse 1, B-5530 Yvoir, Belgium {suzanne.kieffer,annabelle.gouze,ronald.moncarey, christian.vanbrussel,jean-francois.dewispelaere, francoise.kayser,benoit.macq}@uclouvain.be
Abstract. The development of computer technologies provides a means to support and facilitate the daily activities of potentially all users. This may be of particular importance for experts in breast cancer imaging and diagnosis. While many research efforts have been carried out separately on the implementation of task-oriented systems, much less effort has been undertaken to design and develop technologies compliant with domain standards or in accordance with end-user needs and expectations. This further suggests the need to improve both the usefulness and the usability of breast cancer-dedicated systems. This paper reports the results of a development method combining the application of usercentered design together with usability development methods. At different time frames in the life-cycle, the development method employed knowledge elicitation interviews, scenario-focused questionnaires, paper mock-ups and usability tests. Owing to its naturalness and its convenience, pen-based interaction with a graphics tablet was chosen as the modality to interact with the system. Additional innovative solutions were designed and implemented in order to facilitate and improve the visualization and the manipulation of data during the lesion characterization: namely an icon framework, a star-menu and a semi-automatic lesion detection system. The resulting user interface is a penbased interactive tool supporting visualization, navigation, standardized lesion characterization and reporting. The usability tests suggest that it provides endusers with an efficient, reliable and usable system.
Towards Standardized Pen-Based Annotation of Breast Cancer Findings
525
tend to split their work between a growing number of interactive tools, workstations and media (e.g., screen, mouse, keyboard, and handheld recorder). In addition, interactive tools for BC screening and reporting have to be compliant with the BI-RADS standard [1], an approved system of descriptive terms and reporting guidelines. Such tools not only facilitate reporting, providing radiologists with structured and standardized reports, but lead to data accessibility as well (e.g., data exchange and storage, interpretation monitoring [13], retrieval of useful and interesting cases for teaching and research purposes [14]). Finally, interests in user-centered design together with usability stem from the goal to design and implement interactive systems supporting the activities of domainexpert users, who are not necessarily experts in computer science. As highlighted in [3], great care must be devoted to the study of the needs and the expectations of such domain-expert users. In particular, attention must be paid to usability throughout the software life-cycle in order to design and implement user-friendly and easy-to-use interfaces [9,6].
2 Objectives and Significance of the Work The ultimate objectives of the research presented here are to design, implement and evaluate a BC-oriented interactive system which integrates the interactive annotation of significant findings (i.e., lesion characterization and reporting) and the semiautomatic lesion detection. This paper focuses on the design and the evaluation of the annotation tool; refer to [5] for more details about the semi-automatic lesion detection tool. A prototype for lesion annotation, based on semantic web technologies, was presented during SPIE Medical Imaging 2007 [4]. Our work goes further mainly by providing BC-experts with a new interaction style to characterize findings: the penbased annotation with a graphics tablet. The significance of the work can be highlighted depending on three axes: • End-user: provide experts in BC screening with useful and usable tools; • Accessibility: increase breast imaging data accessibility thanks to standardization (data exchange and storage, interpretation monitoring [13], useful and interesting cases retrieval for teaching and research purposes [14]); • Usability: develop and promote support for designers of usable systems.
3 Annotation of Breast Cancer Finding The pen-based user interface (PUI) is an effective method to provide end-users with a natural, intuitive and convenient interaction [12]. Owing to its high naturalness and mainly to its convenience to satisfy the lesion characterization requirement, pen-based interaction with a graphics tablet was chosen as the modality to interact with the system: navigating in a clinical case (i.e., among the mammograms), navigating in a specific mammogram (i.e., zoom-in, zoom-out), sketching a region of interest (ROI), annotating findings and reporting (i.e., direct manipulation of menus, icons, widgets). The BI-RADS [1] provides a standardized terminology for the description of BC findings. Any finding is described according to a lesion type (i.e., mass, calcification,
526
S. Kieffer et al.
architectural distortion, special case or associated finding), and type-related characteristics. Beyond the specific characteristics related to a lesion type, the breast imaging report contains the finding location and the comparison to previous studies, whatever the type. An icon framework was created according to this standard in order to enable any finding to be fully described. Every single term of the BI-RADS lexicon is represented by a unique icon (altogether about 150 different icons), so that the lesion characterization is straightforward and unambiguous. A color code was adopted in order to facilitate the discrimination between the findings: masses in blue, calcifications in yellow, architectural distortions in green, special cases in violet, and associated findings in orange. The schemes on the icons related to the finding location and the comparison to previous studies are common to all the lesion types; only colors are different. The Table 1 presents the icons related to the specific characteristics of masses. Masses are characterized by basic shape (round, oval, lobular or irregular), margin (circumscribed, microlobulated, obscured, indistinct or spiculated) and density (high-density, equal density, low-density or fat-containing radiolucent). Table 1. Icons related to the specific characteristics of masses: shape, margin and density Shape
Round
Oval
Lobular
Irregular
Obscured
Indistinct
Margin
Circumscribed
Microlobulated
High-density
Equal Density
Spiculated
Density
Low-density
Fat-Containing Radiolucent
4 Development Method The development method combined user-centered design together with usability development methods, and employed: knowledge elicitation interviews, scenariofocused questionnaires, paper mock-ups and usability tests. Domain- and task-relevant knowledge was collected early in the life-cycle thanks to knowledge elicitation interviews. Five domain-expert users were questioned thoroughly about the BC domain, the task series involved in their activity, their needs and their expectations with the goal to implement the collected information in the system. The equipment used was paper notes and video recording. Scenario-focused development method was used to define and select the interaction scenarios which would best support and fit end-user activities. Thanks to
Towards Standardized Pen-Based Annotation of Breast Cancer Findings
527
Table 2. Scenario-focused questionnaire. The end-user activity is the description of the lesion type (column 1). There were two proposed interactive scenarios to support this activity: using an array of buttons or using a pie-menu (respectively, columns 2 and 3). Screening analysis activity Description of the lesion type: • • • • •
Scenario#1 Array of buttons:
Scenario#2 Pie menu:
Masses Calcifications (CA++) Architectural distortion Special cases Associated findings
Fig. 1. Paper mock-ups. On the left: the experimental material such as paper, glue, and pen. On the right: the device in use.
end-user involvement, such a design method proved to reduce both development time and development cost, and to improve usability [12]. Referring to [12], a written scenario-focused questionnaire (see Table 2) and paper mockups (see Fig. 1) were implemented complementarily. Both were elaborated from the information collected during knowledge elicitation interviews and were presented to six domain-expert users in order to evaluate the icon framework and the overall spatial organization of the interface, and to select the potential interactive scenarios. The questionnaire was used by the interviewer as visual aid during face-to-face interviews. Paper mock-ups were preferred to computer prototypes since the available prototyping tool did not support the scenarios applying for implementation, and since the evaluation should lead to a lot of drawings, direct manipulation of paper components and discussions between designers and domainexpert users [10]. The analysis of the data collected from scenario-focused questionnaire and paper mock-ups led to the expert validation of: the terminology and the iconic framework, the color code, and the preliminary expert validation of the Pie and the Star menus. Providing experts in BC imaging with an interactive tool supporting their activity is a tough problem of human-computer interaction considering the user requirements of usefulness and usability. The usefulness is ensured by the compliance of the system with the domain standard (see section 3), and by the integration within a single
528
S. Kieffer et al.
interactive tool of the functionalities of image visualization, finding characterization (i.e., annotation), semi-automatic lesion detection and reporting. In order to guarantee the usability of the system, especially during the lesion characterization task, sustained attention has been paid to the graphical representation of the mammographic BI-RADS terminology (by the design of an exhaustive set of icons), to the spatial organization of multimedia data (not only the overall spatial organization of the user interface, but the specific location of widgets as well), and to the design of new interactive solutions suited to the finding annotation with a pen on a graphics tablet. Therefore, two complementary menus were implemented to support the pen-based annotation of BC findings: a pie menu [2] for the pen-based selection of the lesion type, and a star menu for the pen-based description of the type-related characteristics of the lesion. The pie menu (see Fig. 2, left) was implemented in order to facilitate the pen-based annotation of the lesion type. This format was chosen because it reduces the target seek time and improves the accuracy of target selection [2,8]. The star menu (see Fig. 2, right) was implemented in order to facilitate the pen-based annotation of complementary characteristics by grouping icons related to the same characteristic on a single line. This format was chosen because such a display layout was proved to be very efficient and accurate for visual inspection or visual detection by comparison with matrix, elliptic and random spatial structures [11].
Fig. 2. Menus for the pen-based annotation of findings. Pie menu (left) and star menu (right).
5 Usability Evaluation of the Pie and the Star Menus The purpose of this investigation was to evaluate the usability of the pie and the star menus (pie-star menus) during the pen-based annotation of BC findings. The usability evaluation criteria were the system effectiveness, the system efficiency and the satisfaction of the users [9,6]. The effectiveness metric was the task completion, whereas the efficiency metrics were task completion times and the number of clicks. Participants were asked to annotate BC findings in clinical cases by using either the pie-star menus, or an array of icons located at the top of the user interface. The array icons and the star menu icons are exactly the same in terms of scheme, color and size. The array allows the sequential selection of finding characteristics, according to the following order: lesion type, type-related characteristics (each characteristic has to
Towards Standardized Pen-Based Annotation of Breast Cancer Findings
529
Fig. 3. Array of icons. Lesion type (top). Shape of a mass (bottom).
be described one at a time), comparison to previous studies and location. Fig. 3 presents the array of icons displayed to describe the lesion type (top) and the shape of any mass (bottom). Nine volunteers ranged in age from 43 to 58 and including 3 females and 6 males participated in this study. All participants were experimented breast radiologists practicing in different hospitals in Belgium. They were recruited regarding their experience in breast cancer screening. Computer skills were assessed thanks to a background questionnaire. All participants were familiar with computers and especially with medical computer-based applications and all were experienced in visual search and navigation activities on computer displays. They were also standard mouse and keypad users with similar quick motor reactions. The usability test employed a 2x5 factorial design, with two experimental conditions (pie-star menu versus array) and five medical cases to characterize (i.e., five tasks). Each participant carried out ten tasks: five per experimental condition. The order of the experimental conditions was counterbalanced between participants according to a 2x2 Latin Square design. Likewise, the order of the five medical cases per condition was randomized. Counterbalancing and randomization were used in order to neutralize possible task learning effects and to control inter-individual diversity. The tests were carried out in an isolated room in each hospital. Participants were seated approximately 40 cm from the graphics tablet. Pen-based annotation was used as input modalities whereas visual display was used as output modality. The computer system used in this study was a computer with an Intel Core2 Duo E8400 (3GHz) processor, 4 GB of DDR SDRAM and a 9600GT Nvidia graphic card. The screen was a WACOM CINTIQ 21UX. DICOM (Digital Imaging and Communications in Medicine) images were loaded into the viewer. The test sessions involved one volunteer at a time. First, participants were given an oral presentation of the project, an explanation of their role in the usability tests, and a demonstration of the functionalities of the tool. Then, they started the training session: one clinical case to annotate per experimental condition. Once they felt comfortable enough with the tool and got used to the manipulation of the pen, they were provided with the paper printed instructions, and the demographic and background questionnaires to fill prior to the effective test. After each condition, they were asked to fill a satisfaction questionnaire. After the two conditions, they were asked to fill the CSUQ [7], a 19-item questionnaire which aims at evaluating the usability of a system in terms of System Usefulness (SysUse), Information Quality (InfoQual) and Interface Quality (IntQual) on a 7-point Linkert scale. A debriefing ended the session. The effective tests lasted approximately 30 minutes.
530
S. Kieffer et al.
6 Results 6.1 Statistical Analysis The sample includes 219 entries (i.e., 219 findings were annotated in all). Analyses of variance (ANOVA) were used to examine the presence of significant differences in task performance, as measured by both annotation times in seconds and number of clicks: per conditions (pie-star and array), per view (CC1 and MLO2), and per finding type (mass, calcification, architectural distortion, special case and associated finding). Table 3. ANOVA Procedure. Factors: experimental condition, view, and finding type. DF stands for degree of freedom, AT for annotation time, and NC for number of clicks. The statistically differences are bold. Factors Condition View Finding type
Annotation times. The results presented in the Table 3 show no statistically significant difference between the experimental conditions, but a tendency (F=3.5605; p=0.0605). On the other hand, they show a significant view effect (F=5.6496; p=0.0183) and a significant finding type effect (F=2.7884; p=0.0274). Number of clicks. The results presented in the Table 3 show a highly significant finding type effect (F=9.0073; p<0.0001). The results from Table 4 show that the annotation of BC findings with the pie-star menus is faster than with the array of icons (pie-star: 17.5 sec versus array: 20.5 sec). Since the results highlight no significant effect of the number of clicks between the two conditions, this difference of speed may be a matter of visualization and visual perception: the star menu displays simultaneously all the icons related to a finding type, whereas the array displays the icons related to one characteristic at a time. In 1 2
The Cranial-Caudal view (CC) is taken from above. The MedioLateral Oblique view (MLO) is taken from an oblique view.
Towards Standardized Pen-Based Annotation of Breast Cancer Findings
531
opposition to the array, the star menu enables the users to anticipate their next clicks and, consequently, to be faster. In addition, the results from Table 4 show that the annotation of BC findings is faster in the CC view than in the MLO view (CC: 20.5 sec versus MLO: 16.34 sec). This difference may be explained by the combination of the following two reasons. First, in practice, breast radiologists start the diagnostic by the analysis and interpretation of the CC view. It may be natural to adopt the same task order with the interactive tool. Second, findings such as masses and calcifications need to be characterized in both views. Thus, a “duplicate” button was implemented in order to reduce the number of clicks necessary for the complete annotation of findings. Table 5. Means and standard deviations of number of clicks Finding type Mass Calcification Arch. Dist. Special Case Associated finding
N 69 89 39 5 17
Mean 5.46377 4.58427 4.02564 4.00000 6.58824
Standard deviation 0.21331 0.18782 0.28373 0.79241 0.42975
Finally, the results from Table 3, Table 4 and Table 5 show that the speed and the number of clicks to perform the annotation task depend on the type of the finding under annotation. This difference may be explained by the combination of the following two reasons. First, the number of characteristics differs from a finding to another (i.e., six characteristics for calcifications, five for masses, and only three for architectural distortions, special cases and associated findings). Second, the important amount of icons to memorize (i.e., about 150) necessarily involves a considerable learning time, and it sounds acceptable that the annotation of unusual findings such as special cases and associated findings requires more time and more clicks in comparison with masses, calcification and architectural distortion which are more frequent. 6.2 User Satisfaction and Preferences Through the questionnaires and during the interviews, participants considered the interaction with the system as natural, intuitive and reliable. A majority of participants (8) hesitated less than five times, and all participants were satisfied with the compliance with the BI-RADS. Five participants expressed very positive judgments on the star menu in terms of information visualization, speed and comfort. They preferred the star menu because: “it enables the parallel visualization of the items thanks to its spatial organization”, “it is more comfortable thanks to its position close to the center of the screen”, and “it is faster (than the array)”. Four participants preferred the array because “it is usual” and “the characteristics follow a logical sequence”. These results are consistent with the CSUQ results reported in the Table 6. The interface quality, especially, is assessed very positively by the participants.
532
S. Kieffer et al.
Table 6. Summary of the overall sample CSUQ. Each 19 item was score on a 7-point Linkert scale (1=totally disagree, 7=totally agree). Statistical indices are mean and standard deviation. Mean SYSUSE INFOQUAL INTERQUAL OVERALL
5.46 5.56 5.81 5.56
Standard deviation 0.96 1.11 0.88 1.89
7 Conclusion A pen-based interactive tool for standardized annotation of BC lesions was designed and implemented combining user-centered design and usability development methods. Our approach employed knowledge elicitation interviews, scenario-focused questionnaires, paper mock-ups, lab tests, field tests and post-test questionnaires. Additional complementary solutions were designed and implemented in order to facilitate and improve the manipulation of data during the BC finding annotation: the pie and the star menus. The emphasis of this approach is the attention paid to users and usability. The benefit of this approach is improved user satisfaction. The pie and the star menus lead to better user performances than with the array of icons. This is remarkable with respect to the fact that this unusual interaction style is brand-new and the users thus had no previous experience with it. Furthermore, participants to the usability tests expressed very positive judgments on the star menu in terms of information visualization, speed and comfort and on the user interface. In particular, the interface is judged easy-to-use and adapted to the human activity. However, as it requires a substantial amount of collected data and numerous individual interviews in great details, the availability of representative users can be an obstacle to the implementation of this novel interaction concept introduced here for other applications. Acknowledgments. The research presented in this paper has been financed by the Walloon Region within the WIST2-616446 VIGILE project. The authors would like to thank the radiology departments of the Cliniques universitaires UCL (MontGodinne), the Institut Jules Bordet (Brussels), the Clinique Saint-Pierre (Ottignies), the Clinique Sainte-Anne (Anderlecht), and the Clinique Notre-Dame (Tournai).
References 1. American College of Radiology, Breast imaging reporting and data system (BI-RADS®) atlas, 4th edn. American College of Radiology, Reston (2007) 2. Callahan, J., Hopkins, D., Weiser, M., Shneiderman, B.: An empirical comparison of pie vs. linear menus. In: Proc. ACM CHI Conference on Human Factors in Computing Systems, pp. 95–100 (1988)
Towards Standardized Pen-Based Annotation of Breast Cancer Findings
533
3. Costabile, M.F., Fogli, D., Fresta, G., Mussio, P., Piccinno, A.: Computer environments for improving end-user accessibility. In: Carbonell, N., Stephanidis, C. (eds.) UI4ALL 2002. LNCS, vol. 2615, pp. 129–140. Springer, Heidelberg (2003) 4. Gemo, M., Gouze, A., Debande, B., Grivegnée, A., Macq, B.: A versatile knowledgebased clinical imaging annotation system for breast cancer screening. In: Proc. SPIE Medical Imaging, pp. 6514–6565 (2007) 5. Gouze, A., Kieffer, S., Van Brussel, C., Moncarey, R., Macq, B.: Interactive breast cancer segmentation based on relevance feedback: from user-centered design to evaluation. In: SPIE Medical Imaging, vol. 7260, p. 726021 (2009) 6. ISO 9241, Ergonomics requirements for office work with visual display terminals (VDTs) (1998) 7. Lewis, J.R.: IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction 7(1), 57–78 (1995) 8. Moyle, M., Cockburn, A.: Analysing Mouse and Pen Flick Gestures. In: Proc. SIGCHI-NZ Symposium On Computer-Human Interaction, pp. 266–267 (2002) 9. Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers, San Francisco (1994) 10. Sefelin, R., Tscheligi, M., Giller, V.: Paper prototyping – what is it good for? A comparison of paper- and computer-based low-fidelity prototyping. In: Proc. CHI 2003 Extended abstracts on human factors in computing systems, pp. 778–779 (2003) 11. Simonin, J., Kieffer, S., Carbonell, N.: Effects of display layout on gaze activity during visual search. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 1054–1058. Springer, Heidelberg (2005) 12. Wang, D., Dai, G., Wang, H., Chiu, S.C.: Scenario-focused development method for a pen-based user interface: model and applications. The Journal of Supercomputing (Online First) 13. Wittenberg, T., Elter, M., Schulz-Wendtland, R.: Bildverarbeitung für die Medizin 2007. In: Complete digital iconic and textual annotation for mammography, pp. 91–95. Springer, Heidelberg (2007) 14. Zheng, Y., Wu, M., Cole, E., Pisano, E.D.: Online annotation tool for digital mammography. Academic Radiology 11(5), 566–572 (2004)
ImproV: A System for Improvisational Construction of Video Processing Flow Atsutomo Kobayashi, Buntarou Shizuki, and Jiro Tanaka Department of Computer Science, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki 305-8573, Japan {atsutomo,shizuki,jiro}@iplab.cs.tsukuba.ac.jp
Abstract. ImproV is a video compositing system for live video. It uses a dataflow diagram to represent the video processing flow and allows performers to edit the diagrams even while the video is running. Traditional live video is limited to video editing, but ImproV allows users to construct video processing flows on the fly. We present the design of ImproV and report on some actual live video performances using ImproV as preliminary evaluation in this paper. Keywords: live performance, visual music, visual performance, visual jockey, VJ, improvisation, dataflow, visual programming, video authoring, video compositing.
1 Introduction In live video performances, such as those at musical events or fashion shows, the video performer selects videos and switches between them according to the event’s atmosphere, the occasion, and the type of music used at the event. Therefore, while showing a video to the audience, the performer must select video and optionally adjust the cue point, playing speed, and effects parameters among others. Although each performer has their own style and method, their performance includes some improvisational attributes. We have developed a video compositing system for live video performances called ImproV. In Sec. 2, a standard live video performance and its requirements are explained. In Sec. 3, systems similar to ImproV and live video performance related researches are discussed. In Sec. 4, we report on our preliminary evaluation and results of this system. Our conclusion is given in Sec. 5.
ImproV: A System for Improvisational Construction of Video Processing Flow
535
In the step 1, the video creators create 2D/3D animation or shoot with a video camera. The animation can be character animation or moving abstract graphics, which are called motion graphics. The footage can be real world scenes or live-action. Then, in step 2, the video creators process the raw video footage. They correct the colors, apply some video effects, layer multiple video footage and many other things. Common video compositing software applications, e.g. Adobe After Effects, uses a layer model to represent the video processing flow. In step 3, the video compositions are arranged along a time axis. Editing software applications, e.g. Adobe Premier, use a timeline representation in contrast to what a layer model uses. For live video performances, switching videos corresponds to step 3 in video creation. The live video performers finish steps 1 and 2 before the live video performance is held. This means it is impossible for the live video performers to change the effect parameters and video processing flow of the compositions. Wires are used to connect the hardware devices necessary to construct a video processing flow when using a video mixer, such as a Roland V-4. The live video performer can find it very hard to change the video processing flow. Even with the proper software applications, most being modeled after a video mixer, the video processing flow is fixed such that typically two video sources are mixed using a two-channel video mixer with optional effectors. We thought this fixing of video processing flow limits the performer's improvisational possibility and believe a performance is more expressive with the improvisational construction of the video processing flow. 2.1 Design Requirement: Background Compositing One of the most common methods used by a performer during a live video performance is to prepare another video in the background of the current video the audience is watching, and then switch them. Live video performers use hardware video mixers or software applications based on a video mixer. M. Lew analyzed the method of editing video on the fly using a video mixer [1]. In the analysis, the following three steps are used. STEP 1: Media retrieval STEP 2: Preview and adjustment STEP 3: Live manipulation A performer browses the video clips and selects the next one (STEP 1). Then, the performer adjusts the cue point or playing speed of the video clip, applies any necessary effects, and adjusts the effect parameters by previewing them using a monitor display (STEP 2). Finally, the performer starts to switch the current video clip to the next video clip by manipulating the video mixer and optionally animates the effect parameters by directly manipulating the effectors (STEP 3). With traditional live video performances, STEP 2 is merely controlling the parameters and optionally selecting an effect. We think this severely limits expressiveness in traditional live video performances. To solve this problem, we thought that STEP 2 should include the constructing and changing of the video processing flow. That is to say, a performer can repeatedly use trial and error to create videos till they are satisfied. To prevent unintended videos from being shown to the audience, previewing is important in this method. We call this method background compositing.
536
A. Kobayashi, B. Shizuki, and J. Tanaka
2.2 Representation of Video Processing Flow Video creators mix several video materials in video compositing. The video mixer and video mixer representation in the software are not powerful enough to treat three or more videos concurrently. The systems with layer representation such as Adobe After Effects provide more flexible video processing flow than systems with a mixer. The systems with layers can overlay an arbitrary number of images. However, while showing a video, the user can't make and mix/switch to another video composed of a different processing flow from the one being shown. As mentioned above, the background compositing and previewing features are necessary for video compositing during live video performances. Moreover, a layer representation works only when all the videos are simply overlaid. However, when the video processing flow is complex such as when using a mask, in which a mask video is correlated to another video and overlaid with yet another video, the user carefully checks each layer, and sometimes opens or pulls down the layer property to see which layers are correlated to each other. Moreover, the applied effects are hidden in the layer representation. Most of these software feature an extra window and show only the effects applied to a selected layer. These modal interfaces force the user to search inside the video processing flow. During a live performance, glancability and accessibility should be preceded since it’s a realtime situation. For all the above-mentioned reasons, we decided to use a dataflow diagram to represent the video processing flow.
3 Related Work There are many systems that use a dataflow diagram to represent the video processing flow. Jitter [2] is one of most standard dataflow systems for videos in the fields of art and entertainment, but it only supports a workflow that separates the construction and execution of a dataflow. On the other hand, systems like vvvv [3] and Quartz Composer [4] support the dynamic construction of a dataflow. However, these systems, including Jitter, are designed as programming environments and the abstraction level of dataflow diagrams are too low to be used by video creators and live video performers. A similar dataflow system that is not for programmers but for end-users to ImproV is FindFlow [5], which aims at interactive retrieval from mass data. Some studies have been conducted on live video performances. EffecTV [6] is a real-time video effects framework and is aimed at live video performances. Novel controllers, such as SPATIAL POEM [7], Rhythmism [8] and video-organ [9], are used for controlling the visual attributes of live video performances. However, these controllers assume that the system settings or video processing flow are already set and fixed before the live video performance is started. Soundium [10] by Mueller et al. started from a similar concept to that we used for ImproV, which was to use impromptu video creation to create a dynamic image to improve the expressiveness of a live video performance. Their challenges were the technical issues of real-time media processing. In contrast to these researches, we worked on the exploration of the user interface designs for the interactive construction of video processing flows.
ImproV: A System for Improvisational Construction of Video Processing Flow
537
4 ImproV We have developed ImproV, a real-time video compositing system. To increase a performer's improvisational availability, we designed ImproV to allow a performer to construct a video processing flow on the fly. To this end, we decided to use a dataflow diagram to represent the video processing flow. Several nodes are connected to each other with the edges in the dataflow diagram. In the dataflow diagram in ImproV, each node represents a video source (e.g., a video file or a camera), an effector (e.g., a blur effector or a mixer), or an output screen. The video processing flow is constructed by connecting these nodes. 4.1 Understandability for Live Video Performers A performer can easily understand the concept of connecting nodes in the dataflow, because they are used to connecting video processing devices such as video tape players, video effectors, and video mixers. We carefully designed the nodes of ImproV so that the performer can use their knowledge about video processing devices. In the other words, we simplified the design barriers part of the six learning barriers of programming systems [11]. 4.2 Dataflow Diagram of ImproV Figure 1 shows the dataflow diagram of ImproV. The most basic node is shown as part a “Camera” in Fig. 1, which has a label to show the type of node and has an output port at the right. Many of the nodes have some input ports listed under the name label, as shown as part c “Mixer” in Fig. 1. The user uses the menu to create these nodes and connects the nodes by dragging and dropping from the output port of a node to the input port of another node. Figure 1 part b is the same node “Camera” as in part a, but it is connected to the “Mixer”. Some nodes have original user interfaces. Figure 1 part d “Output Screen” this type of user interface to show a preview of the output.
Fig. 1. a: a node that has no input, b: same node as a that is connected to another node, c: a node which has four inputs, and d: a node that has an input and custom user interface
There are two data types, video image and floating point number, for controlling the effect parameters, in an ImproV dataflow diagram. In Fig. 2, the video image is distorted by “Distortion”. The degree of distortion is decided by the value input to “Value” input port. The “Slider” outputs a floating point value of about 3.0 in Fig. 2.
538
A. Kobayashi, B. Shizuki, and J. Tanaka
Fig. 2. Dataflow diagram with two data of types
Fig. 3. Dataflow diagram mixes two videos
In Fig. 3, two videos, one is an image of a flower and the other is particle animation, are overlaid with additional blending. The video with the image of a flower is applied and gradated using a “Blur” effect. The degree of blur is decided by the “Slider”. This video processing flow is similar to traditional live video performance systems. 4.3 Supporting Complex Video Processing Flow Traditional live video performance systems mainly focus on mixing/switching wellpreprocessed videos, and their video processing flows are simply made. For example, one may want to copy a video to make eight, rotate them 45 degrees, and mix them with additional blending (this will result in a kaleidoscope-like effect). These systems do not support this idea without preprogrammed effects. In ImproV, the performer can create several edges from one output. Branching edges from one output is treated as copying the value from the output. Therefore, the performer can construct the video processing flow of this example on the fly. Figure 4 shows the video processing flow of this example in ImproV. 4.4 Supporting Background Compositing and Previewing Since constructing a video processing flow can cause unintended results, we designed ImproV to support background compositing and previewing. Figure 5 illustrates how to mix two video processing flows.
ImproV: A System for Improvisational Construction of Video Processing Flow
Fig. 4. Video copied, rotated, and overlaid
Fig. 5. Mixing two video processing flows
539
540
A. Kobayashi, B. Shizuki, and J. Tanaka
• Fig. 5 (a): After the example above, the performer makes a new video of another video processing flow, which consists of a video file and an effector “Transparency”. • Fig. 5 (b): The performer inserts a mixer before the output screen. • Fig. 5 (c): The performer makes the new video transparent by using the effector “Transparency”, and then overlays the new video over the old one by connecting the effector and the mixer. • Fig. 5 (d): The performer remakes the transparency, resulting in the new video fading-in. Note that we treat two video processing flows, the old complex one and the new simple video file, equally. ImproV allows users to edit video processing flows arbitrarily. The result is that users can independently create a video processing flow from the current main output, which the audience is watching. As mentioned above, previewing images is important for the background compositing method. A user can connect any nodes on the dataflow to output screens because of the nature of a dataflow diagram. Therefore, the user can preview videos at arbitrary points in the video processing flow. Figure 6 (b) shows the previewing from three points in the video processing flow in Fig. 6 (a).
Fig. 6. Performer can preview any points in video processing flow
ImproV: A System for Improvisational Construction of Video Processing Flow
541
5 Preliminary Evaluation The first author recorded a live video performance of a musical event for use as a preliminary evaluation. Several bands played jazz music at the event. The live video performance covered two hours of the whole event. Figure 7 shows the band and the display of the computer running ImproV at the event. We used a USB camera and some pre-rendered video clips. The projection was directed toward the screen behind the bands through the bands themselves. The computer running ImproV and the first author were behind the audience. From a stability aspect, ImproV worked continuously without any problems for two hours. In addition, the first author could keep the video playing by using background compositing for the entire two hours. At the most complex situation, there were over 20 nodes on the display. We confirmed that novel expression is practicable using ImproV. Figure 4 shows images that are the same as the ones of the video processing flows made in this evaluation. The first author tried to make videos expressing a different feeling for each song. We found that a video processing flow with 10-20 nodes could be made in 3-5 minutes. This is reasonable span of time to create video expressing different feelings for each song, which normally take around five minutes. Moreover, we found that a 17-inch display enabled the performer to edit a video processing flow with around 30 nodes. However, we also realized that the edge connecting operation is perplexing.
Fig. 7. Actual live video performance using ImproV
6 Conclusion We described ImproV, which is a system focused on the improvisational construction of video processing using a dataflow diagram. ImproV allows a live video performer
542
A. Kobayashi, B. Shizuki, and J. Tanaka
to make novel improvisational video compositing. ImproV supports background compositing and previewing to protect from showing an unexpected video to the audience. We tested ImproV in an actual live video performance environment and confirmed its capability. We also found the edge connecting operation problem while conducting this evaluation. We are planning to work on this problem and conduct a more formal evaluation.
References 1. 2. 3. 4.
5. 6.
7.
8.
9.
10.
11.
vvvv group: vvvv Apple Inc.: Quartz Composer Cycling ’74: Jitter Choi, J., Hong, H.: SPATIAL POEM: A New Type of Experimental Visual Interaction in 3D Virtual Environment. In: The 8th Asia-Pacific conference on Computer-Human Interaction, pp. 167–174 (2008) Bongers, B., Harris, Y.: A structured instrument design approach: the video-organ. In: the 2002 conference on New interfaces for musical expression, pp. 1–6 (2002) Fukuchi, K., Sam, M., Ed, T.: EffecTV: a real-time software video effect processor for entertainment. In: The international conference on entertainment computing, pp. 602–605 (2004) Hansaki, T., Shizuki, B., Misue, K., Tanaka, J.: FindFlow: visual interface for information search based on intermediate results. In: The 2006 Asia-Pacific Symposium on Information Visualisation, pp. 147–152 (2006) Ko, A., Myers, B., Aung, H.: Six Learning Barriers in End-User Programming Systems. In: 2004 IEEE Symposium on Visual Languages - Human Centric Computing, pp. 199–206 (2004) Lew, M.: Live Cinema: designing an instrument for cinema editing as a live performance. In: International Conference on Computer Graphics and Interactive Techniques, pp. 144–149 (2004) Mueller, P., Arisona, S., Schubiger-Banz, S., Specht, M.: Interactive media and design editing for live visuals applications. In: International Conference on Computer Graphics Theory and Applications, pp. 232–242 (2006) Tokuhisa, S., Iwata, Y., Inakage, M.: Rhythmism: a VJ performance system with maracas based devices. In: International conference on Advances in computer entertainment technology, pp. 204–207 (2007)
E-Assessment: A Suitable Alternative for Measuring Competences? Martin Kröll Ruhr-Universität Bochum, Institut für Arbeitswissenschaft, Universitätsstraße 150, 44780 Bochum, Germany
Abstract. More and more companies accomplish tests for the assortment of trainees to measure the aspirants competences and competence potentials. The article in hand is dedicated to the academic evaluation of the adoption of new E-Assessment methods. Thereby it has to be resolved in how far the expectations linked with the E-Assessment’s assignment for example seen from the company’s view or from the proband’s one, can be achieved. To adopt the computer-aided diagnostic professionally, its advantages and disadvantages are to be considered. Keywords: employability, e-assessment, self-organization, key qualification.
2 Understanding the E-Assessment Procedure E-assessment is commonly understood to be: “the use of internet-based tests (instruments) …, which serve to appraise and predict relevant job-related biographical and psychological variables and to assess the suitability of applicants” [7] (translated by the author). This paper deals not only with online processes, but also with computer-based diagnostics of suitability as e-assessment approaches (Ridder et al., 2004: 34). Such tests, carried out in the context of formal selection procedures, often take place on site. E-assessment can and is used in a range of organizations (e.g. businesses, public authorities, schools, universities) and applied to different functions (e.g. pre-selection, command of problematic situations, education controls). Furthermore, e-assessment can be used for various target groups such as (potential) apprentices, students, specialist workers and middle management. Empirical studies which deal with the prevalence of e-assessment are mostly of an exploratory character. The opacity of the multitude of tests and uncertainty as to their quality are considered by businesses to be an obstacle to their take-up of the eassessment procedure. In order to evaluate the quality of the procedure for a jobrelated appraisal of suitability, quality standards were developed at different levels. At the international level we have from AERA “Standards for educational and psychological testing”, “Principles for the validation and use of personnel selection procedures” from SIOP, and the standards from the “Task Force on Test User Qualifications”. In the German-speaking world we have DIN 33430, whose focus is on the entire process of suitability appraisal. The efforts concerning all aspects of DIN 33460 are to be critically reflected upon against the background of various approaches to evaluation (e.g. responsive evaluation). Further points of critique are adequacy and timeliness.
3 Diagnosis of the Construct ‘Competence’ In the context of e-assessment, bespoke procedures for competence measurement should be deployed and related to one another, in order to capture the competences of the individuals holistically in terms of occupational competence – the competence to act autonomously in the chosen occupation/work situation. Occupational competence is achieved when a relationship is present, or is established between the competence dimensions – technical, method-based, social and co-operative. Competences are not constituted by the accumulation of knowledge for testing. Rather, competences are to be interpreted as the “holistic development of capabilities, units of knowledge and lines of reasoning, as relevant to the action in question” [1] (translated by the author). What then is a holistic comprehension of competences? Two concepts can be differentiated: the first seeks a process which captures the individual’s capability for action as exhaustively as possible; the alternative seeks to analyze discrete competence dimensions through various means. It is, however, questionable whether a cumulative measure of various competence dimensions can satisfy the claim to capture results holistically. The relationship between the results is crucial.
E-Assessment: A Suitable Alternative for Measuring Competences?
545
Returning to the question as to which competences are necessary for the employment of an organization’s staff, the various competence dimensions – technical, method-based, social and co-operative competences – are given a particular value. There is disagreement in the research on competence development as to the role of the different dimensions, to do justice to the changes in working life. Great hope was and is attached to the concept of key qualification. The belief that a onesided focus on technical competence should be overcome gained currency. Technical competence alone was not sufficient to cope with the demands of the workplace, let alone to shape it. Within the competence development research, there is controversy surrounding which qualifications belong to the ‘real’ key qualifications. Nevertheless, almost all these approaches assume that the key qualifications concern superficial knowledge and capabilities. The great expectations vested in the concept of key qualifications, in accordance with Mertens, have not yet been met. A central assumption of the approaches to the key qualifications is that these cannot age and therefore their acquisition is more important that that of the technical competences. However, simply because technical competences become outdated with time, this does not mean that the technical competences fundamentally lose their value in the context of job-related activities. This is the first weakness in the debate about key qualifications. When it comes to meeting changing work requirements, not just the lack of technical competences, but in particular outdated technical competences become a decisive obstacle. The dominance of the discussion on key qualifications hides a second weakness of that debate: from the perspective of the individuals concerned it can be entirely reasonable to acquire competences in different technical areas in the context of an interdisciplinary focus. Furthermore, it is noteworthy that there are to date hardly any empirical comparative studies which prove that the possession of key qualifications significantly increases the likelihood of employment. In her empirical study, Müller [8] explores the extent to which the possession of key qualifications influences the likelihood of apprentices remaining in the same trade after completing their training. The ability to communicate and co-operate, self-regulated learning in accordance with PISA 2000, performance motivation as well as aspects of self-competence (such as self-effectiveness) were all included as key qualifications. She could not identify any favorable patterns in their progression through their chosen vocations as a result of the acquisition of key qualifications. A differentiated and critical analysis of approaches which deal with key qualifications is required; this is however complicated by the unclear and shifting use of terminology. It is then clear that the exclusive focus on key qualifications, whereby technical aspects (e.g. expertise as know-how imbued with experience) are diminished in their importance, proves inadequate. I now proceed to discuss the principles and models to which various approaches to competence research refer, and how they differ between competence dimensions (see Table 1). These ideas are the basis for judging the use of the e-assessment procedure. Bunk bases his definition of occupational competence on the construct of the occupation [3]. As a consequence of structural change, the boundaries between clearly defined occupations are increasingly blurred, which causes this definition to appear of limited suitability. A new model which does justice to the transformation on the internal and external labor markets is the concept of ‘employability’. This is defined as „the ability of a person, founded upon their technical and occupational
546
M. Kröll
competences and their ability to create value and to perform, to supply their labor and thus to enter into working life, to keep their job, or if necessary, to seek new gainful employment” [2] (translated by the author). The ideas surrounding the concept of employability are at least in part too unspecific to serve as a point of reference for the competence development of individuals, and should be put in concrete terms. Wunderer & Bruch [9] examine the distinguishing features of entrepreneurial competences, whose degree they consider to be dependent on five classes of coentrepreneurial behavior: (1) subcontractor, (2) co-entrepreneur, (3) entrepreneurially inclined staff, (4) staff with low co-entrepreneurial competence and (5) individuals who act as if sacked, are overburdened or procrastinate. Current approaches in the competence development research treat the competence to self-organize as being at the heart of the scientific debate [4]. In order to satisfy the requirements of the working environment (increasing complexity and insecurity), it is necessary for staff to possess the competences of self-organization and self-regulation. The actions of individuals must be increasingly self-directed. Competences are interpreted as a predisposition to self-organized action. They would escape direct scrutiny. Table 1. Points of reference for occupational competences Author
- technical competence (continuity: knowledge, skills and capabilities) - method-based competence (flexibility: procedural methods) - social competence (sociality: behavioral patterns) - co-operative competence (participation: ways of structuring) - ability to create value for the businesses, - ability to organize oneself and one’s career, - to market the abilities oneself which are in demand on the internal and external labor market.
Wunderer & Bruch 2000
Erpenbeck & Rosenstiel 2003, p. XV f.
Competence as a predisposition of self-organized action
- strategy-orientated problem-solving competence (deployment of value-creating innovations), - implementation competence (creating acceptance of innovative ideas) - co-entrepreneurial social competence (combination v. independent-autonomous action and co-operative-integrated behavior) - personal competences (attitudes, values system, self-perception) - activity-oriented competences (integration of emotions, motivations and experiences in drives of the will, etc.) - technical-methodical competence (solving objectiverepresentational problems) - social-communicative competences (developing group and relationship-orientated behavior, tasks and goals)
Occupational competences, relevant for future employment, are dependent on (1) the concrete work tasks, (2) the requirements of the work in terms of quality, and (3) the authorization which a work task attributes to individuals. In light of the view of competences as occupational competences of individuals (Ortmann, 2007), the
E-Assessment: A Suitable Alternative for Measuring Competences?
547
capacity to act (4), which they can develop and activate, is also relevant for their (continuing) employment. Furthermore the relevant occupational competencies (5) are subject to the repetitive interaction between the assignment of competences (in the sense of tasks) and the (potential) capacity to act of individuals. Clearly, selforganization and the corresponding (6) competence to self-organize play an evergreater role in working life. But without (7) a corresponding other-organization, e.g. from the side of the business, there is a risk that the efforts of self-organization will not have the desired results. In contrast to personality features, competences relate to a high degree to concrete situations which call for their application. The possession of competences can only be measured in context. In what manner will the e-assessment procedure be undertaken in order to capture the competences and/or personality features of individuals? It will either be based upon the established paper-pencil form, or new tests will be developed against the background of a firm’s specific understanding of competence. In the latter scenario, the firm would firstly determine precisely what its requirements were in terms of relevant competences which every staff member should possess [5]. A new requirement analysis would not need to be carried out for every new appointment. Corresponding approaches are based on extensive algorithms, whereby various competences are placed in hierarchic relationships with one another and a different weighting of the expression of competences follows. Such an approach that leads to the forming of statistical opinions is to be distinguished from concepts which seek to capture the entirety of a person (the clinical process of forming opinions).
4 Case Study of the Application of E-Assessment Based on an actual case study of a large energy trading company, the difficulties of the use of new e-assessment forms as an attribute-oriented selection process are demonstrated. Over the course of a project, running over several years, an eassessment tool was developed. In contrast to the classic online methods, this new form of e-assessment uses test landscapes and the development of short stories. Cover stories involve a gradual construction of drama and should leave a holistic impression on test persons, with potentially a meaningful impact. Various interrogation methods are assigned to particular objects of the test landscape, and the test persons are guided by a virtual test leader. Characteristic of this new form is a stronger use of mediaspecific conditions, manifesting in experience-driven question elements such as reaction tests or movement-related tests, together with multimedia extras in classic questioning. Through a real-time, standardised and self-explanatory reporting system, the test persons receive prompt feedback about their results. There is simultaneously an orientation towards DIN 334340. By means of this new concept, the firm strives to enhance its image through an applicant-orientated test. From the perspective of the test persons, these new forms are of interest due to their aim of reducing anxiety of the tests, increasing test motivation, individualising the length of the test, reducing the experience of strain, as well as enabling faster feedback. Firms pursue these goals in order to increase social acceptance of the test procedure.
548
M. Kröll
First the occupational orientation of the test persons is determined. When they move about in the test landscape and take a virtual tour of the firm or its departments, they can choose between the various test modules. For example: (1) Figural intelligence: the test persons proceed in a knowledge management course by inserting an object in a door lock. (2) Arithmetic intelligence: this form of intelligence is examined, for instance, when the test persons must find a particular poster in a poster display which reflects requested data. (3) Mechanical-technical understanding: the test persons must solve particular tasks in order to progress through an obstacle course. (4) Memory capacity: the test persons should note various paths within the building and between the buildings which are shown briefly. (5) Performance motivation: the LMI-test is used for this and a depiction along the lines of a paperpencil test follows (answers through clicks). (6) Concentration/ attention: the test person is confronted with dangerous situations and a role-playing adventure game in the test landscape (e.g. dust which shouldn’t be breathed in). (7) Learning potential: short stories are presented, for instance, about which test persons must later answer questions. (8) Fear of texts/exams: situations will be portrayed which typically trigger fear, ending with an evaluation via a fear thermometer “How scared are you at this moment?” (9) Stress management: short stories will be introduced as situational impulses. Questions are subsequently asked. It appears plausible that, through the implementation of this form of e-assessment, social acceptance rises compared with conventional forms of e-assessment, but this must be tested. Empirical studies generally indicate that the acceptance of internetdriven personality tests is comparatively high. It must furthermore be clarified whether the validity of the test results is influenced through test persons having a relatively large amount of freedom to choose their tests. There are no consistent research results as to whether paper-pencil tests or internet-supported tests lead to better results [6]. The test case has also highlighted the danger that primarily those competences will be rated as significant for competence measurement, in respect of which approved tests are already available. The existence of, for example, the competence for self-regulation, entrepreneurial competence or technical competence is not examined in this case study. On the other hand, it could be claimed that the points dealt with here mostly concern key qualifications. The results of Müller [8], which could not be confirmed, indicated that possessing key qualifications, such as performance motivation pursuant to LMI inevitably leads to improvement in the employment situation. Finally it should be noted that, in the context of the tests, personality features are predominantly measured. These features are comparatively stable over time and only change to a limited extent. The results of such tests offer little indication at present as to how the test persons may further develop their competences.
5 Conclusion and Outlook The advantages and disadvantages of computer-aided diagnostics should be taken into account for their professional application. The following disadvantages may arise: high purchase costs, additional expenditure through the creation of an infrastructure, software-ergonomic deficits, hardware and software non-compatibility, high costs
E-Assessment: A Suitable Alternative for Measuring Competences?
549
through long dial-in times, low quality of e-assessment concepts (e.g. often simply 1:1 replication of paper-pencil tests), out-of-date content and unsuitability of the norms (see DIN 33430), inflation of applicant numbers, unsecured access of the test persons with the computer, direct contact with the applicants taking place relatively late, and abuse or dissemination of model answers. The advantages include: savings of time, space, material and personnel, increased independency of time and place constraints in carrying out the tests, more precise recording of time, greater flexibility in the possibilities for constructing the test, valid results and better security against error, greater neutrality and objectivity due to removal of the test leader effect, image benefits as an attractive employer, real-time analysis of results and quicker feedback, as well as higher acceptance through greater consideration of the needs of the test persons. The degree to which the advantages and disadvantages of e-assessment are manifest is dependent upon how e-assessment is deployed and embedded organizationally. The latter point is central task for the future utilization of e-assessment. In light of the current level of experience it becomes clear that use of the eassessment procedure at present is only partially able to contribute to overcoming central challenges which arise in relation to the professionalization of competence development (e.g. competence transfer, evaluation of competence development). Against the background of the current usage of the e-assessment procedure, the preselection of applicants is in the foreground. The demands that will be made of staff in the future are for many reasons increasingly more difficult to predict with precision. At the same time, the scientific debate presumes that a precise determination of competences is only possible if the demands on the (potential) staff members are available in a specific form. This insight confronts the application of the e-assessment procedure with huge challenges. Moreover the e-assessment procedure at the level of the test persons can lead to an intrapersonal contradiction: people ‘slip’ into the role of an object under examination and/or they experience it as such. However later on in working life they will be expected to act entrepreneurially and on their own authority. The following research questions are for consideration in the further scientific debate: how can internet-based procedures in the strategy of personnel recruitment and human resource development be better integrated? How high is the unintentional drop-out rate of competent applicants during e-assessment processes, who would be of interest to the firms? Could the results of e-assessments provide a foundation for the construction of skill or competence databases in firms? To what extent can the results of e-assessments be relevant to the employees of the HR department, managers, directors and the individuals themselves? How can/is the connection between the (relevant) competences of individuals and the firm capabilities considered? Under what conditions could the results of e-assessment provide the foundation for organization and control of competence development at the level of the firm and at the level of the individual? Depending on the answers to these questions, the next step is to elaborate the requirements in respect of the use of the e-assessment procedure.
550
M. Kröll
References 1. Bernien, M.: Kompetenzen messen und bewerten. In: QUEM (Hrsg.): Kompetenzentwicklung 1997. Waxmann-Verlag. Münster (1997) 2. Blancke, S., Roth, C., Schmid, J.: Employability “Beschäftigungsfähigkeit” als Herausforderung für den Arbeitsmarkt dem Weg zur flexiblen Erwerbsgesellschaft. In: Akademie für Technikfolgenabschätzung in Baden-Württemberg (Hrsg.): Arbeitsbericht Nr. 157 Stuttgart (2000) 3. Bunk, G.P.: Kompetenzvermittlung in der beruflichen Aus- und Weiterbildung in Deutschland. In: Kompetenz: Begriff und Fakten. Europäische Zeitschrift Berufsbildung H. 1, S. 9–15 (1994) 4. Erpenbeck, J., Rosenstiel, L.v. (Hrsg.): Handbuch Kompetenzmessung. Schäffer-Poeschel Verlag, Stuttgart (2003) 5. Kirbach, C. et. al,: Recruting und Assessment im Internet. Vandenhoeck & RuprechtVerlag Göttingen (2004) 6. Klinck, D.: Computergestützte Diagnostik. Göttingen u.a. Hogrefe (2002) 7. Konradt, U., Sarges, W.: E-Recruictment und E-Assessment. Göttingen Hogrefe (2003) 8. Müller, K.: Schlüsselkompetenzen und Verbleib nach der Ausbildung. Dresden (unveröffentlichte Dissertation (2007) 9. Wunderer, R., Bruch, H.: Unternehmerische Umsetzungskompetenz. Diagnose und Förderung in Theorie und Praxis, Vahlen, München (2000)
Green Advocate in E-Commerce Ying-Lien Lee1, Fei-Hui Huang2, and Sheue-Ling Hwang3 1
Dept. of Industrial Engineering and Management, Chaoyang University of Technology, No. 168 Jifong E. Rd., Wufong Township Taichung County 41349, Taiwan [email protected] 2 Dept. of Marketing and Distribution Management, Oriental Institute of Technology, No. 58,Sec.2,Sihchuan Rd.,Pan-Chiao City,Taipei County 22061,Taiwan [email protected] 3 Dept. of Industrial Engineering and Engineering Management, National Tsing Hua University, No. 101, Section 2, Kuang-Fu Road, Hsinchu 30013, Taiwan [email protected]
Abstract. The continuous growth of e-commerce sees waves of information explosion. Online shoppers have to confront with more information than ever when they are making purchasing decisions. Among the tools that try to reduce this burden of information overload, recommender system is one of the widely employed techniques which can be seen in stores such as Amazon.com and iTunes Store. This paper presents an approach of the interaction design of recommender system in the context of green digital products. By cultivating the field of game design, elements that make game fun and engaging are borrowed and applied to the design of the recommender system to motivate shoppers to opt for greener choices. In addition, the idea of Kansei Engineering will be employed in the system to recommend according to the perceived characteristics of products. A framework of such system will be described, along with future extensions of the framework in the realm of e-commerce. Keywords: e-commerce, recommender system, green digital product, game design, interaction design, Kansei Engineering.
information overload. This research focuses on recommender systems among these shopping tools. Conventional recommender systems usually base their recommendation mechanism upon profit revenue or customer preference. Such systems recommend items so that the potential profit for the store can be higher, or that the customer is more likely to prefer if the customer actually buy items in the recommendation list. However, such profit or preference seeking systems may not be adequate when there is an agenda that the store wants to advocate, or when the preference of a customer is not easy to determine. Consider green consumer electronics. More and more companies are promoting their brand images or products by adopting green practices or green design. They utilize the trend of environmental awareness and set their green marketing strategies, but such trend is not reflected in the design of recommender systems. Moreover, greenness of a product may drive the price higher or lower. Customers need to be informed and encouraged to buy green products. Recommender system can take on the role of an advocator in this respect. As to the determination of the preference of a customer toward a product, some product fits the bill better than the other. Take movies for example. Each movie has a set of characteristics that can be directly related to customers’ preferences, such as casts, directors, and movie genre. It is the same with music or books, but not quite so with consumer electronics. Recommender systems for consumer electronics are usually specification-wise, which means they recommend products that are similar to each other in terms of specifications. Such design is useful for consumers who have a clear utility goal in mind, but it is only a specification-wise match between a product and a consumer’s functional needs. Although some products are perceived as fashionable and some others as professional, items are still categorized according to their specifications, not their perceived characteristics. In addition, for specification-based recommender systems, it is difficult in the regard of recommending across product categories. For example, a fashionable customer has bought a fashionable mobile phone and now he or she wants to buy a headset that is as stylish as the phone. Since the specification ontology of these two types of product is different, it is not easy to recommend products by specification only. Additional characteristics need to be considered for this purpose. To extend the field of recommender systems toward the directions mentioned above, this research proposes a framework that integrates an advocate module and a Kansei (“feeling” in Japanese) module in addition to a conventional recommender module, and the framework will be illustrated in the context of green consumer electronics. The task of the advocate module is to recommend greener products by a mechanism based upon a theory borrowed from game design called Flow Theory. This module tries to advocate green consumerism by advancing consumers’ green appetite in a progressive way. The task of the Kansei module is to recommend products according to the perceived characteristics of the products whose perceived characteristics are defined as a set of Kansei words. A Kansei scoring system will also be developed to encourage consumers to contribute Kansei words, much like the concept of Web 2.0. A fuzzy inference system is used to synthesize the inputs from these three modules.
Green Advocate in E-Commerce
553
2 Literature 2.1 Game Design Games have a lot to offer when it comes to making systems fun and captivating. Applying game-like interface or game-like mechanism is quite promising in the design of ordinary systems, such as process management tools and educational programs. A theory which game design bases heavily upon is the Flow Theory by Csíkszentmihályi [1]. Csíkszentmihályi studied extensively to find out what makes enjoyable experience, or flow experience, which is an experience “so gratifying that people are willing to do it for its own sake, with little concern for what they will get out of it, even when it is difficult or dangerous”. An important condition for the flow experience is the match between the challenge of a task and the skill of a user. 2.2 Kansei Engineering Kansei Engineering (KE) is a methodology established by Nagamachi in 1970 to identify the relationship between consumers’ feeling and design elements [2]. It has been successfully applied to areas such as automotive industry, construction machinery industry, office and home appliance, house construction, fashion industry, and so on. There are three types of procedure to apply KE [3]: Type I is category classification, Type II is computer-assisted system, and Type III is mathematical modeling. The Kansei module used in this research is closely related to that of Type II KE.
3 The Proposed Framework 3.1 General Structure and the Framework There are four major modules in the proposed framework: Recommender module, Advocate module, Kansei module, and Fuzzy Inference System. Recommender module is responsible for recommending products based upon product information and derived consumers’ needs, which works like most conventional recommender systems. Advocate module is responsible for deciding a suitable level of product greenness for a consumer. Kansei module is responsible for the determination of the match of users’ Kansei requirements and the perceived characteristics of products. Finally, Fuzzy Inference System is responsible for fusing together the outputs of the three modules and other user-defined criteria. The general structure of these modules is shown in Fig. 1, where the inputs (user model and product model) and outputs (recommended items) are also included. 3.2 Determination of Product Greenness Different eco-labels have different meaning. Some refers to the raw material while some refers to the energy efficiency. To determine the overall greenness of a product, the framework uses Analytical Hierarchical Process [4], or AHP, to integrate experts’
554
Y.-L. Lee, F.-H. Huang, and S.-L. Hwang
Fig. 1. General structure of the proposed framework
opinions on how important an eco-label is to the greenness of a product. The hierarchy can be one or two levels. Once the weight of each eco-label is determined, the approximation of product greenness can be calculated. Let PEi be the row matrix of eco-label status of product i. The j-th item in the row matrix is either 1 or 0 indicating whether product i has ecolabel j or not. Let EW be the column matrix of eco-label weights obtained from the AHP mentioned above. Then, the overall greenness of product i, or PGi, is PEi × EW . 3.3 Determination of Consumer Green Consciousness In the proposed framework, the green consciousness of a consumer is defined as the relative weight of product greenness among other subjective criteria. For example, in some products, greenness may drive the price higher or lower. It may also impede or facilitate some product features, such as slow computing speed for energy efficiency. AHP is also used in the elicitation of the weights, and to simplify the process, only one level is used. The weights of these factors are then stored in User Profiles and will be used in the Fuzzy Inference System. 3.4 Product Kansei Tagging To encourage users’ involvement in the product Kansei tagging effort, a user reputation system is developed which works like a product rating system. In a product rating system, a user can rate and comment on the product, while other users can rate his or her ratings and comments. If the ratings or comments are well-received by other users, the user who gives ratings or comments can build up his or her reputation in the website. The process of Kansei-tagging is very similar to that of rating a product. Once a user becomes involved in a Kansei tagging session, he or she can tag a product with a set of Kansei words that describe the perceived characteristics of the product in question. Then, another set of Kansei words of the product is randomly retrieved from the Product Kansei Tags database. If there is no previous data for this product, an initial set of Kansei words is used instead. This initial set can be supplied by the retailer or by the brand owner of the product. The matching score between these retrieved and current sets are calculated and stored in User Profiles along with the set of Kansei word. This step updates the part of the User Model that indicating what
Green Advocate in E-Commerce
555
Kansei words are more likely to be associated with a user. To update the set of Kansei words of the Product Model, words which are frequently used as tags and highly matched are appended to the set of Kansei words of the product in question. Less frequent and lowly matched tags can be removed. The process of calculating the matching score of the retrieved and current sessions can be described as a “gaming with a purpose” (GWAP) scenario. GWAP is a class of games “in which people, as a side effect of playing, perform tasks computers are unable to perform“ [5] . In our case, tagging a product with Kansei words that describe the perceived characteristics of a product is not easy for a computer to perform. To engage users in the tagging, the framework pairs the current user with a previous to create the sense of competition or social interaction, which is an output-agreement game according to Ahn & Dabbish [5]. 3.5 Advocatory Recommender
ha nn el C ow Fl
Product Greenness
The core part of the framework is the advocatory recommender, which consists of a green advocate module, a Kansei ranking module, a recommender module, and a fuzzy inference system. The advocatory recommender is activated explicitly or implicitly. By explicit activation of the advocatory recommender, a user actively seeks for advices on what he or she can buy. The user can choose to enter some Kansei words as the criteria of the perceived characteristics of product. Adjustments of the weights of the criteria can made at this moment, as well as the addition of extra criteria, such as brand or year of make. Those are user inputs. Then, the green advocate module is invoked to determine an appropriate range of greenness of products according to the user’s green consciousness. Finally, the fuzzy inference system is used to synthesize the outputs of the modules mentioned above according to the weights of user criteria.
Fig. 2. The flow channel of the green advocate module
The green advocate module adopts the concept of Flow Theory and provides a progressive challenge that encourages users to opt for greener choices over time. Consider the three dotted circles in Fig. 2. In the normal mode, this module recommends products whose greenness matches the green consciousness of a user, such as circle A in the figure. After some sessions of purchase, the module tries to recommend products whose greenness is a bit over the users’ green consciousness, as the
556
Y.-L. Lee, F.-H. Huang, and S.-L. Hwang
circle B in the figure indicates. If the user finds it acceptable and chooses it, then the module will give the user a “level up”, or moving up a notch, in the axis of green consciousness, as the circle C indicates. If the user doesn’t choose it, then the module should fall back to circle A. The advocatory recommender uses a fuzzy inference system to integrate multiple decision variables. The proposed rules for this inference system are summarized in Table 1. In our proposed form, only two decision variables are considered in each rule, and only two levels are considered in each variable. However, it is possible to extend to consider more variables and more levels. Table 1. The proposed rules of the fuzzy inference system Decision Variables Greenness
Customers with higher weight Feature-demanding
Price
Feature
High
Rich
N/A
High
Poor
N/A
Low
Rich
N/A
Low
Poor
N/A
Price-sensitive demanding Price-sensitive
High
N/A
High
Green-conscious
High
N/A
Low
Low
N/A
High
Low
N/A
Low
rule
Output Neutral Undesirable
and
feature-
Favorable Neutral Neutral Undesirable
Price-sensitive conscious Price-sensitive
and
green-
Favorable Neutral
4 Conclusions and Future Research The process of decision making of online shopping is inherently complex in this information age. On the one hand, information can be gathered easier and quicker than ever, promising to solve the problem of information imbalance. On the other, the volume of information is to vast for us to take advantage of, creating another problem of information overload. Several solutions to this dilemma are put forth, and recommender system is among the widely used. A recommender system mimics the interaction of people asking for advices from friends, gurus, or sales person. Some notable examples of recommender systems can be found on Amazon.com, Netflix, or iTunes Store. However, most recommender systems base their recommendation on profit or user preference. What if there is an agenda, such as green marketing, which a store wants to advocate? Given the rising concern of the environmental issues, this paper proposes a framework of recommender system that has a green advocate module that encourage users to opt for greener products in a progressive way. The green advocate
Green Advocate in E-Commerce
557
module is based upon a theory called Flow Theory borrowed from the field of game design. Products such as movies and music albums have a lot of characteristics that can be related to customers’ preference. Yet for products like consumer electronics, specifications and functions are usually used in recommender systems. To enrich the recommendation, the concept of Kansei Engineering from the field of Industrial Design is used in the proposed framework. Users are encouraged to tag products with Kansei words that describe the perceived characteristics of products. The tagging process is based upon the concept of gaming with a purpose, and the score of the tagging game can help the user build his or her reputation on the e-commerce website. Recommender system can take advantage of the Kansei tags of products and Kansei requirements of users to generate recommendations by determining the similarity of a product and user’s Kansei requirements. The integration of the conventional recommender module, the Kansei module, and the green advocate module is achieved via a fuzzy inference system. Acknowledgments. The authors would like to express their gratitude to National Science Council of Taiwan for the funding under the grant number NSC-97-2221-E324-018-MY3.
References 1. Csíkszentmihályi, M.: Flow: The Psychology of Optimal Experience. HarperCollins Publishers, New York (1991) 2. Nagamachi, M.: Kansei Engineering: A new ergonomic consumer-oriented technology for product development. International Journal of Industrial Ergonomics 15(1), 3–11 (1995) 3. Nagamachi, M.: Kansei Engineering: An ergonomic technology for product development. International Journal of Industrial Ergonomics 15(1), 1 (1995) 4. Saaty, T.: Risk-Its Priority and Probability: The Analytic Hierarchy Process. Risk Analysis 7(2), 159–172 (1987) 5. Ahn, L.v., Dabbish, L.: Designing games with a purpose. Commun. ACM 51(8), 58–67 (2008)
Gesture-Based Sharing of Documents in Face-to-Face Meetings Alexander Loob and Christian Rathke Hochschule der Medien (Stuttgart Media University), Wolframstraße 32, 70191 Stuttgart, Germany {loob,rathke}@hdm-stuttgart.de
Abstract. Many of the Electronic Meeting Systems on the market today only support sharing of electronic documents via email or a common user space. This kind of sharing enforces short work interruption and requires substantial mental effort than necessary. This contribution describes the development of a simple EMS, its shortcomings and the further development to a gesture-based EMS for sharing documents in face-to-face meetings. The system implements the concept of a “throwing” gesture for transferring documents to one or more participants. This gesture is explained and evaluated in further detail. Keywords: Gesture-based Interaction, Document Sharing, Electronic Meeting System, face-to-face meeting, reduction of mental effort and work interruption.
Gesture-Based Sharing of Documents in Face-to-Face Meetings
559
documents by dragging them to graphical representations of the participants. In a second step, the system is augmented by providing means for a gesture-based interaction which alleviates the necessity of the graphical representations. Our effort is embedded in our more general strive to design and develop theories, methods and tools to support cooperative work. We concentrate on face-to-face situations where people meet to work together as part of an overall purposeful collaborative process.
2 Meeting Support Infrastructure As a starting point for our research, we devised a simple EMS – called the Workshop Trader (Fig. 2 and Fig. 3) – as one of the software components of our Corporate Communication Lab [3]. In this Lab we pursue research in collaboration technology and explore improvements of common work situations such as collocated or distributed meetings. The Lab’s conference table is composed of several modular working places with touch-sensitive displays. It features one large wall-mounted display for moderation and presentation, and one large height-adjustable display, both of which are also touch-sensitive. The contents of any of the individual table displays may be mirrored to the two large displays (Fig.1).
Fig. 1. The Corporate Communication Lab at the Stuttgart Media University
The Workshop Trader software is capable of representing meetings and their respective participants. The clients of its client server architecture run on all of the Lab’s displays. They can be used for sharing electronic documents between the meeting’s participants. To support a wide spectrum of electronic devices and operating systems in addition to the Lab’s hardware, the entire Workshop Trader application is implemented in Java. The client application in particular does not require any additional software
560
A. Loob and C. Rathke
Fig. 2. The Workshop Trader
libraries. As long as a Java virtual machine (JVM) is available for the desired target system, the Workshop Trader client may run on the participant´s device. To start the collaboration process, each participant can choose a user and a meeting session for login to join the meeting. After joining a session each available recipient and additionally the group as a whole are represented by rectangular areas, respectively (Fig. 3). Using the client, any visible document on the screen may be dragged from its location and dropped on one of these areas.
Fig. 3. Representation of the participants of a meeting session
As a result, the electronic document is transmitted to the Workshop Trader server and it appears as a small icon within the sender’s area at the recipient’s client. From there, the document may be opened directly or copied to the recipient’s private environment. Due to security and (network) performance reasons the document will not be copied to the target system until the recipient accesses the document.
Gesture-Based Sharing of Documents in Face-to-Face Meetings
561
3 Shortcomings of the Trader System One of the shortcomings of interacting with the Trader or similar systems is the additional effort necessary to identify the area on the screen which is associated with the intended recipient. This can be interpreted as a more general issue when dealing with electronic representations of real world situations: The user is required to interrupt the intuitive flow of operation by mapping the real world situation to the electronic counterpart. According to Schön [4] professionals engage in situated work until their expectations are not met. Then a breakdown is experienced and they have to stop and reflect on how to overcome the current breakdown. In design activities, these breakdowns provide an opportunity to learn and construct new knowledge [5]. In face-to-face meetings, this breakdown diverts from the situated flow of activities because it forces the participant to realize and deal with procedures which are not an intrinsic part of the task at hand.
4 Our Approach Currently, the electronic document to be shared must be dragged from the desktop and dropped onto the application’s virtual representation of a meeting participant. This informs the application to transmit information about the shared document to the target participant. Compared to traditional approaches, the Workshop Trader effectively facilitates sharing of documents, but sharing activities requires explicit interaction with the application. Users need to be aware of its existence. They must know how to tell the system what they expect. Interacting with the application may be compared to asking a person to pass a document to someone else. In this case, communication would be done verbally, i.e. by using natural language speech. But communication with a third person is not necessary if the contributing participant has a way of directly addressing the intended recipient. For instance this is the case if the recipient is an immediate neighbor or the group is small enough for each member to be within direct proximity. In situations like this, the document is passed directly. No intermediate is involved. To transfer this direct mode to the electronic environment, we have developed a mechanism, which allows a person to use a “throwing” gesture for the purpose of passing an electronic document to another participant. Following this approach, we make use of an intuitively used gesture for transferring a physical object to another person “on the other side of the table”. No mental effort is required for becoming aware of a third party or some secondary representation of the recipient. A basic requirement to make use of the natural gesture for sharing a document is the spatial representation of all participants. The system must know each participant’s position and heading (Fig. 4). Additional information regarding position and heading in the physical environment is specified by the participant during the login process. The throwing gesture may be executed by using any of several pointing devices including fingers on touch sensitive surfaces, styluses on tablet PCs, or simple mouses. Also, the movement of the pointing device must be recognized as a throwing gesture and correctly interpreted as pointing to a unique target person.
562
A. Loob and C. Rathke
Fig. 4. Spatial representation of the participants of a meeting
Recognizing a movement as a gesture (not necessarily a throwing gesture) is simply done by measuring the velocity of the pointer. Any movement above a certain threshold is determined to be a gesture. The current implementation of this gesture recognition mechanism does not support different settings for the velocity threshold depending on the diagonal and the resolution of the input media. Prerequisite for the “throwing” gesture is the “grabbing” of the electronic document, which is synonymous with pressing of the left mouse button (resp. touching a touch-sensitive surface). Once this prerequisite is fulfilled, all movements of the pointing-device including the coordinates and the timestamp are recorded from this point of time. Following the completion of the “throwing” gesture by releasing the left button of the pointing-device, the recorded movement- and time-information is cleared of irrelevant information. For this purpose, the velocity in pixels per second is calculated continuously between two consecutive measuring points. Once the velocity is lower than the velocity threshold, the measuring point recorded earlier in the timeline will be removed. In the next step, the measuring points at the end of the potential gesture are inspected and the velocity continuously between two consecutive measuring points is calculated. In case the velocity value is lower than the given threshold the measuring point recorded later in the timeline is removed (Fig. 5). The red line in figure 5 shows the velocity curve between the measuring points remaining after the reduction mechanism. If there are a minimum of three measuring points or more remaining after the reduction of the measuring points the movement can be interpreted as a gesture (but not implicitly as a “throwing” gesture). In addition, a straight throw to one participant is separated from a round movement, which is interpreted as a throw to all participants.
Gesture-Based Sharing of Documents in Face-to-Face Meetings
563
Fig. 5. Example of a velocity curve of the pointer movements
In order to be able to distinguish between these different “throwing” gestures the absolute values of the angle-differences between two consecutive measuring points are calculated at each occurrence and added up. Afterwards this sum is divided by the amount of the measuring points. The result of this calculation is then compared with another given threshold which is used to differentiate between a straight “throwing” gesture and other gestures. In case the value is lower than this given threshold, the recently captured movement can be determined to describe a straight “throw” and so the gesture can then be interpreted as a document sharing offer to one single participant of the meeting. If the result of this calculation represents a higher value than the threshold the captured movement is inspected more closely. The goal is to be able to determine if the movement described a circular path and can be interpreted as a document sharing offer to the whole group of participants. Based on each participant’s position and on the throwing angle of the gesture refined using the heading of the source participant, the target person can be identified by the source person (Fig. 6). Special care is taken to distinguish between two or more potential targets and hidden participants. Comparable to the physical environment, participants could be covered up by other participants in the virtual representation, too. For example, a participant could be positioned behind another participant from the view of the “throwing” person. This possible cover-up-effect needs to be calculated by each client separately. In case participants are out of view, they are marked as non-reachable persons in the set of the potential targets. Given the calculated angle of the “throwing” gesture and the heading of the source participant it now becomes possible to determine a maximum of two participants extracted from all potential target participants. Here, one participant above and one participant below the angle of the “throwing” gesture are selected. Each participant shows the lowest difference between the angle from the view of source participant and the angle of the throw.
564
A. Loob and C. Rathke
Fig. 6. Identification of target
In Figure 6 the dotted line represents the angle of the real “throwing” gesture. The solid lines represent the angles to the two closest target participants from the view of the source participant. The participant with the smallest distance to the “throwing” angle receives the document from the sending participant. The real implementation of this gesture recognition mechanism is more complex than described here and several more specific cases are considered (e.g. zero degree pass through). However, discussing these details is not in the scope of this paper. Furthermore, it is possible to realize the recognition of other gestures with this gesture analysis method in the future. Due to several reasons, an individual analysis method for gesture recognition and interpretation was implemented. We decided not to use existing gesture recognition libraries (e.g. iGesture [6] or Mouse Gestures [7]). One of the reasons is that we plan to port our approach to the Java Mobile Edition at the next major release to integrate participants with touch-sensitive smart-phones in such meetings.
5 Evaluation / Testing For testing and evaluation the gesture controlled EMS described in the previous chapter we designed a multi staged testing and evaluation concept. All tests of these stages will be done with a special testing environment that allows us to record all important information for analysis afterwards in a very detailed way. This testing environment consists of a simple desktop with a dummy document placed in the center of it. In different sessions virtual participants can be placed in the surroundings to simulate several meeting situations. These sessions can be stored for later use. At the first stage of the testing and evaluation concept the testing environment is used to enhance the analysis method and adjust the used thresholds. Different input media requires different thresholds. Given touch-sensitive displays, the velocity
Gesture-Based Sharing of Documents in Face-to-Face Meetings
565
threshold depends on the size of the display and the used resolution. In this case it is possible to calculate the dots per inch (dpi) of the input media and vary the velocity threshold depending on the dpi. Given other input media like mouses or touchpads of notebooks, it is not as simple to calibrate this threshold because the required values cannot be generated automatically. This stage is also used to implement new gestures to control our EMS application. At the second stage of the testing and evaluation concept several test persons share the dummy document placed in the middle of the screen with a randomly selected virtual participant. This stage is divided in two phases. In the first phase the test person receives feedback if the submitted movement was recognized as a “throwing” gesture in addition to the time when it was recognized, if the gesture hits the right participant. The test person will have ten tries to input a gesture and hit the right participant. With this test we hope to measure the learning effect given by the visual feedback. In the second phase of this test the test person does the same test but without any feedback. In this phase the test person does also have ten tries. During both tests all movements are recorded and the differences between the actual gesture angle and the correct angle to the randomly selected participant are recorded, too. These tests will also be done with different input media. As reference media we use the mouse as a standard input device. At the third stage of the testing and evaluation concept we try to measure the differences of the mental effort between the use of the Workshop Trader and the use of the gesture based EMS. In this test several test persons share dummy documents. At first the documents are shared using the Workshop Trader. Then, same documents are shared using the gesture based EMS. This complete test will be recorded with an eye tracking application in one of our usability labs and evaluated afterwards. At the moment (February 2009) we are in the first stage of this testing and evaluation concept, but we strongly hope that we can present the final testing results of stage two and stage three in the second quarter of 2009.
6 Other Approaches to Sharing Documents in Face-to-Face Meetings An absolutely classic approach to sharing documents is via network file systems e.g. provided by Active Directory Services [8] or a samba server [9]. But these solutions have a high administrative effort to grant user access, so that this is inconvenient especially for including external persons. Another approach, pursued by some Electronic Meeting Systems, is the distribution of documents per (invitation) email before a meeting (like Meeting Sense Software Corporation MeetingSense [10]). Other systems [1, 2] have a common space, where documents may be placed before or during the meeting. The access to these systems is often granted by an invitation email to the participants of the meeting. Thereby roles and rights in the EMS can be assigned by the inviting person or an administrator of the EMS.
566
A. Loob and C. Rathke
7 Other Approaches of Gesture-Based Interaction Gesture-based interaction with electronic devices is actively pursuit in research and development. First outcomes of these activities are already commercially available such as the ART+COM Touchmaster [11], the Apple iPhone [12, 13], the Google Android Operating System [14], most of the HTC Phones [15], the HP TouchSmart PC [16], the Microsoft Surface [17], and the Nintendo Wii Remote controller [18].
References 1. GroupSystems, ThinkTank: The Leading Virtual Interactive Meeting, Brainstorming, Facilitation & Collaboration Technology (2009), http://www.groupsystems.com/technology/thinktank 2. Grouputer Solutions. Overview (2009), http://www.grouputer.com/grouputeroverview.html 3. Hochschule der Medien (2009), Corporate Communication Lab, http://www.hdmstuttgart.de/wi/is/laborwiese/ccl/index_html 4. Schön, D.A.: The Reflective Practitioner: How Professionals Think in Action. Basic Books, New-York (1983) 5. Fischer, G.: Turning Breakdowns into Opportunities for Creativity. Knowledge-Based Systems, Special Issue on Creativity and Cognition 7(4), 221–223 (1994) 6. Kurmann, U.: iGesture: A General Gesture Recognition Framework (2007), http://www.igesture.org/ 7. Smardec, Mouse Gestures (2008), http://www.smardec.com/products/mouse-gestures.html 8. Microsoft Corporation, Windows Server 2003 Active Directory (2009), http://www.microsoft.com/windowsserver2003/technologies/ directory/activedirectory/default.mspx 9. Hertel, C.: Samba: An Introduction (2001), http://us1.samba.org/samba/docs/SambaIntro.html 10. Meeting Sense Software Corporation (2008), How MeetingSense Works, http://www.meetingsense.com/meetingsense/ 11. ART+COM, A.G.: CeBIT 2009: ART+COM Technologies presents Touchmaster (2009), http://www.artcom.de/index.php?option=com_acnews&task=view& id=414&Itemid=136&page=0&lang=en 12. Apple, Inc. Apple-iPhone, http://www.apple.com/iphone/ 13. Hotelling, S., et al.: Gestures for touch sensitive input devices. United States Patent Application Ser. No. 12/118,659 (2008) 14. Google, Inc., What is Android? (2009), http://www.android.com/about/ 15. HTC Corporation. HTC Products (2009), http://www.htc.com/us/product.aspx 16. Hewlett-Packard Development Company, L.P, HP TouchSmart PC (2008), http://www.hp.com/united-states/campaigns/touchsmart 17. Microsoft Corporation, Microsoft Surface: The Power, The Magic, The Possibilities (2008), http://www.microsoft.com/surface/index.html 18. Nintendo, What is Wii? (2008), http://www.nintendo.com/wii/what/controllers
Developing, Deploying and Assessing Usage of a Movie Archive System among Students of Film Studies Nazlena Mohamad Ali1, Alan F. Smeaton2,1, Hyowon Lee2,1, and Pat Brereton3 1
Centre for Digital Video Processing and 2 CLARITY: Centre for Sensor Web Technologies Dublin City University, Ireland 3 School of Communications, Dublin City University, Ireland {nmohamadali,asmeaton,hlee}@computing.dcu.ie, [email protected]
Abstract. This paper describes our work in developing a movie browser application for students of Film Studies at our University. The aim of our work is to address the issues that arise when applying conventional user-centered design techniques from the usability engineering field to build a usable application when the system incorporates novel multimedia tools that could be potentially useful to the end-users but have not yet been practiced or deployed. We developed a webbased system that incorporates features as identified from the students and those features from our novel video analysis tools, including scene detection and classification. We deployed the system, monitored usage and gathered quantitative and qualitative data. Our findings show those expected patterns and highlighted issues that need to be further investigated in a novel application development. A mismatch between the users’ wishes at the interviews and their actual usage was noted. In general, students found most of the provided features were beneficial for their studies. Keywords: Video browsing, deployment effort, usage analysis.
but have not yet been practiced or deployed. The MovieBrowser2 system we developed incorporates automatic video analysis techniques, namely shot boundary detection, keyframe extraction and classification of scenes into action, dialogue or montage [3]. Our application domain is film studies where students need to study movie contents and to analyse sequences. We began with the identification of user needs through interviews and brainstorming, followed by sketching and prototyping an online system that incorporates the video analysis tools we offer as functional features. We then deployed this system, monitored its usage and gathered quantitative as well as qualitative data. In this paper we overview our developed system, discussing the findings from the user and usage analysis of our trial deployment.
2 Related Work Related deployment efforts of experimental video systems include Newsblaster at Columbia University [5], which automatically crawled news websites and summarised them for users. This was first deployed in 2001 and a number of user studies have been conducted; an Austrian interactive TV trial [1] deployed a novel TV application to a local cable TV provider in Salzburg, Austria, and ran for 4 months in 2004-5; Físchlár-News [4] incorporated a number of multimedia and recommendation techniques and was deployed within a University campus for 3 years, during which interaction logging and diary methods were used to capture its usage; SportsAnno [2], a video browsing system that allows users to make comments, share opinions and ideas on sports events, was deployed during the 6 weeks of World Cup 2006. Deploying experimental systems like these is always difficult because the underlying novel multimedia technologies, which they incorporate, are often beyond user expectations and the challenge is finding the right way to productively blend these novel functions into tools to support users’ tasks. Such efforts would show a growing awareness of the importance of user evaluation in realistic environments but studies which incorporate the end-user perspective from the conception stage of the project are rare; most technology trials start purely from a technical point of view and only after deployment do they try to get any form of feedback from real usage and users. Our work is similar to the other trial studies listed above, but instead of the technology being demonstrated dictating the design and deployment process, we based the development process from usability engineering, focusing on user-centred design and that is the significant differentiator of our work.
3 System Descriptions MovieBrowser2 was designed to support two undergraduate film studies course modules at Dublin City University (CM272 National and Ireland Cinema and CM135 Analyzing Media Content). In both modules, students were required to ‘read’ a variety of movie sequences in detail referring to the process of analysing a movie sequence closely in order to understand different levels of meanings intended by its creators and manifested from the elements like framing, music, plot, camera angle,
Developing, Deploying and Assessing Usage of a Movie Archive System
569
lighting and so on. Libraries of 30 movies were made available on MovieBrowser2. They include various genres (comedy, drama, romance, action, etc.), and ranged from contemporary Hollywood movies to old Irish movies. Fig. 1 shows examples of MovieBrowser2 screenshots. Fig. 1(a) shows a screenshot of the front page that displays movie posters and some movie information. Filtering by movie genre and director are provided by a drop down list at the top area. Fig. 1(b) shows a screenshot of the main viewing area with a visual timeline corresponding to events like dialogue, montage and exciting on the top area, shot keyframe view, the playback area and note-taking section under the playback area.
Fig. 1. Interface screenshots: (a) front page, (b) main view area
The note feature is public, similar to the common commenting features of blog sites. Each shot keyframe represents an automatically detected scene, clicking on a keyframe will play video from the scene. The similar colour schemes for event classifications were used in the timeline and the shot keyframe’s borderline and text (i.e. yellow colour for exciting events). The scenes also can be filtered using radio buttons. The playback from the keyframe list will play the sequences, while playing the whole movie can be carried out using the ‘PLAY’ button on top of radio buttons.
4 Usage and User Analysis 4.1 User Access MovieBrowser2 was deployed throughout the Spring semester (11 weeks) of 2007-8. Out of the total of 268 students in both classes, 107 students (40%) accessed MovieBrowser2. From the actual log data collection, we see that on average, almost all 107 students accessed MovieBrowser2 at least 2 times during the trial period. A vast majority of the students (84%) accessed the system for less than 2-hours. Only 17 users (16%) from the total group accessed between 2-8 hours in total. In thinking about the number of hours our students used the system over the semester, it is worth noting that:
570
N. Mohamad Ali et al.
• MovieBrowser2 was used as a complementary tool in the movie analysis classes where the lecturers encouraged the students to use the tool and there was a short introductory sessions conducted during the classes; • MovieBrowser2 featured some movies that are not available from the University library's collection especially for Irish movies; • MovieBrowser2 can be accessed only within campus. This means those students who want to work on their essay at home during the weekend or evening are unable to access the system; • MovieBrowser2 was developed for a specific technical environment in which it was deployed (computer labs in School of Communications, DCU), consisting of Microsoft Windows XP and Microsoft IE v6+. Thus compatibility with other machines and browsers when some of the students tried to use their own laptops was an issue (as found in post-trial questionnaires). The total access duration time was around 86 hours during the trial (CM272: 57 hours; CM135: 29 hours). Access duration time for CM272 was almost double than the other module. This may be because the assignment for the former class required students to use Irish movies as examples which were mostly not available in the university library, whereas the assignment for the latter class was not restricted to Irish movies thus much more accessible from conventional sources (library, DVD rental, cable TV, etc.). From the total movie collection, 23 were Irish movies with 7 contemporary Hollywood added to the collections. Our justification in having students from the CM272 and CM135 module was because these two modules were running during our trial semester and had a similar nature of textual analysis assignment. In this work, we are not focusing on comparing each module specifically but mainly to examine students’ general access patterns. All 7 Hollywood movies that were stored in the system library were accessed a total of 73 times (39%) with the movie Shrek (2001) mostly accessed by students, 24 times. Irish movies were accessed in total 116 times (61%) with the movie About Adam (2000) the most frequently accessed 21 times, followed by The Butcher Boy (1997) at 20 times accessed. A few short movies such as The Visit (1992) and Bent Out of Shape (1995) had no access at all by students. 4.2 Features Access We divided our movie collection into ‘Advanced’ and ‘Basic’ types for navigating movie content and the reason behind this idea was to see the pattern of user interactions when some added technology features are incorporated. The advanced type of browsing consists of some features that could enhance user browsing and navigating of movie content on top of standard playback features. These features are mainly designed to enhance film-reading based on the three events categorizations (i.e. montage, dialogue and exciting). The basic type incorporates only standard playback features (i.e. pause/stop/slider bar and etc). User interactions are captured and represented as in Fig. 2. The result shows that the percentage of interactions on the features: ‘Basic’ and ‘Advanced’ have a similar patterns in playback movie activities. ‘Click the whole movie’ has the highest interaction that shows activity in watching the whole movie, while ‘click play button’
Developing, Deploying and Assessing Usage of a Movie Archive System
571
Fig. 2. Types of interactions
denoted the activity of playing a movie after it being paused. User activities such as watching or playing a sequences or the whole movie were identified from user-action entries such as ‘click pause button’, ‘click stop button’, ‘click play button’, ‘click slider bar’ and so on, labelled in the chart as basic features. These are standard interactions that are mainly related to conventional movie playing activities as normally found in a video player (i.e. play, pause, stop, slider bar and volume adjustment). As for the advanced type of browsing, there are some ‘extra’ interactions on top of the standard playback activities. These extra features are provided in the advanced screen as well as standard movie playback (Fig. 1(b)). The result reveals that the amount of interaction of playback-related features was spread out into that of extra features in the advanced page as Fig. 2 shows. The spread of the interactions shows the tendency of the user to doing more exploration on the page. Users seem to appreciate the addon features provided in the advanced page with the increased number of interaction percentages and this result is also reflected by the increased hours spent on the advanced type with 45 hours as compared to 33 hours in the standard or basic interaction. This result indicates that the advanced features made the students stay longer on the system on the event segmentation that underlies the features. Users managed to jump from one point to another point easily on the movie using visual representations of a timeline or the shot keyframe view. Instead of browsing sequences from the normal playback interaction for example either using the pause button or slider bar, the playback of sequences shifted to playing from the shot keyframe view as depicted in the chart with the highest percentage (18%). The findings in the qualitative comments given by students also reveal a preference for these extra features (see Table 1) as well as some complaints from students about those movies with no advanced features.
572
N. Mohamad Ali et al.
It is worth noting that the interactions that have been represented in Fig. 2 are based on different movie watching. This analysis was not meant to compare between basic screen [A] vs. advanced screen [A+α] but mainly in monitoring the usage pattern of some value add-ons features in navigating and browsing of movie content and whether it would influence usage in a particular way. We also provided other features on how users make movie selections. Features included filtering movie collections based on film-director, film-genre or no filtering at all. In the percentage of log interactions, we noted only 11% of interactions filtered the movie collections based on film-director, 17% filtered based on film-genre with the rest (72%) no filtering at all. We listed all movie posters in the front page of the system by default (see Fig. 1(a)). From our observations, film genre seems an important factor in selecting a movie in the student’s actual textual analysis as the topics given by the lecturer are normally based on movie themes. Dealing with mismatch – A note-taking (or commenting) feature was incorporated as a result of initial requirements analysis but was underused and unappreciated (used by 3 students only), showing an interesting mismatch between what our users said would be beneficial and what they actually used (see Fig. 2). As this is a result that perhaps indicates how conventional usability engineering based on capturing user requirements/wishes is not sufficient in developing a novel interface, we want to analyse this point further. A follow-up email was sent to students asking questions regarding their use of notes. We asked them why they did not use the features during deployment. We got an email reply from 15 users. The reasons for not using notes features was calculated and grouped into several categories as concluded below: • • • •
“I don't want my colleagues to steal my ideas” --- Privacy issue “I like to write with pen and paper” --- Preference for conventional practice “I wanted to do it at home” --- Access limitation Interface design issue (e.g. notes button at the bottom of the screen, thus not emphasised enough)
A mismatch between students' initial wishes for a note or commenting feature and its actual usage during the trial triggered more question for us in how users' wishes collected at the requirement engineering stage should be interpreted in the context of usage rather than treated as an isolated feature in itself. 4.3 Usage over Time Students started using from the 10th Mar (week-6) of the semester. The system seems to have had quite heavy usage approaching the deadline of the assignment submission, which was on the 9th May 2008 (week-14) for both modules. It is generally believed that this pattern of usage corresponds with our previous observations that even though the topic was given early by the lecturer (i.e. week-6 for CM272), students tend to complete their assignment just before the deadline. The lecturer advises students not to do last minute assignments since reading and analysing movies cannot be done within a short duration, it needs longer time so that the skill of reading will evolve. Huge usage was found around the month of April until early May, 1-week before the deadline of the assignment. This pattern corresponds with our findings in
Developing, Deploying and Assessing Usage of a Movie Archive System
573
our user study that the process of reading and understanding movies starts by watching the movie many times before the essay can be written on paper. Another possibility from these usage patterns is that students were engaged with other assignments from other modules. Students have to follow priority deadlines. Some informal conversations with students indicated that this was strong possibility. A few email reminders were also sent to students regarding the deployment trial and we found that once an email was sent, there were some usages recorded.
Fig. 3. Student’s usage in the semester
4.4 User Opinion Out of the 107 students who accessed the system, only 60 students (56%) responded to the questionnaires that we administered within week-13 to -14. In identifying what are the features or functions students like or dislike most particularly in the system we developed, we gave students a qualitative question. Table 1 summarises several items mentioned most frequently by users on the system - both likes and dislikes. From the Table, it can be seen that note-taking was among the most frequently mentioned features that they liked, with 21 (30%) out of the total (71 mentioned) and this is followed by event categorization (24%), timeline (18%), shot keyframe view (11%) and playback of the movie (8%). These responses mainly correspond to the advanced features that had been adopted in the system. The idea of having the facility to take notes while playing a movie scene seems advantageous. We noticed that most important value of the system was simply the fact that it allowed easy access to movies in a non-linear fashion. The timeline and keyframe view which highlight where the action, dialogue and montage scenes overlap in a movie were praised as very useful, indicating that a strong temporal orientation with additional cues on the movie contents is useful as some comments show. “The timeline as it breaks down the film into the various sections - montage, action, dialogue, etc. --- this makes it easier to carry out a more in depth analysis of the movie” [P11].
574
N. Mohamad Ali et al.
“I found the combination of timeline and event categorization very useful since I can select those parts of the movie that contains the events of interest” [P2]. “I liked the way I could go directly to the exciting or montage parts” [P59]. Not only that, we also noticed some other features that our users liked from reading their comments, which we categorise under system design and access. For example, in the system design layout, comments were mainly on the well-designed layout and presentations, which makes it easy to navigate. The rest of the frequently mentioned items were on the convenience of access as an online-based application without having to borrow a DVD from somewhere else. Table 1. Frequency mentioned of system features. *convenience, **less coverage.
Features Note-taking Event categorization Visual timeline Shot keyframe view Movie playback Other: System design Access Limited movies Streaming problem Compatibility Total mentioned
System-likes 21 17 13 8 6 4 2* 71
System-dislikes 1 1 1 3 3** 10 9 3 31
Regarding those aspects that our users didn’t like (right column of Table 1), there are very few comments related to system features as there was only one mention found on each for the timeline, notes and shot keyframe view. We noticed system scalability among the issues in feedback on system dislikes. The highest frequently mentioned issue was about the limited number of movies that were stored in the library, which might restrict usage. Other comments we read were such as system compatibility (i.e. MAC user/Internet browsers). The system design (dislikes) was related to the lack of function, like not able to change the password and the advanced type of browsing did not apply for all movies. Ease of access in the ‘system-likes’ column in Table 1, was meant as a convenience factor by users, but in the ‘systemdislikes’ column, it was meant as less coverage of access. When comparing the frequency of mentioned, which were, either likes or dislikes, we noticed that no issues arose much on the “system design” and “features provided” aspects. Most of the disliked features were related to the system scalability issue. We also asked a question to the students about their overall experiences in using MovieBrowser2 after the semester had completed. Among these 60 students, 43 of them (72%) said they would use it in the future. We calculated the positive and negative expressions of their overall experiences and we estimate that 19 of them (32%) gave positive expressions and only 4 (7%) gave a negative tone of expression while
Developing, Deploying and Assessing Usage of a Movie Archive System
575
the rest of 37 (62%) did not express either positive or negative expressions. Examples of positive expressions include emotion (i.e. “I'm very happy/discovered …”) and feature usefulness (i.e. “I found it is useful/able to …”) and negative such as system limitation (i.e. “Not enough/database is too small…”). Finally, we compiled a ‘wish-lists’ from all user feedback. Some of these list elements appear due to the difficulties in the implementation and would not be expected during the development design stage. The list entries were categorized below: 1. 2. 3. 4.
Larger and varied type of movie database (i.e. Irish, Hollywood and Europe) System compatibility (i.e. Internet browsers and MAC users) Improved access (i.e. off-campus) Technicality constraints (i.e. high-speed access)
We also believe that these ‘wish-list’ elements contributed as the main reasons for low usage during the trial. Our users mainly want to access movie resources to be used in their textual essays. Having difficulties in the conventional way of assessing DVDs, means the tool is appreciated much by the students. We did not receive many complaints on the design aspects of the features we provided (i.e. navigation, colorcoding, page layout, buttons, graphics, ‘look and feel’ etc.) and these can be considered as a minor aspect. We believed that for the future, whatever the design for a similar system to this, it could be of potential benefit if these four ‘wish-list’ elements could be improved.
5 Conclusion The results presented in the trial deployment experiment highlighted some interesting patterns for students of film studies in browsing and playing movie content. User access and usage were found to be varied and influenced by many factors. In general, students found the features we provided were beneficial for their studies. Some issues or mismatches arose during the trial. A ‘wish-lists’ was drawn up that might be useful for the future system developer. Our big strength in this study was on the interactions among real users, namely students of the School of Communications, DCU throughout our 3-years of experiments. Acknowledgements. We would like to thank the School of Communications, DCU for the support given in this work. Also special thanks to Bart Lehane of CDVP for the automatic movie content analysis engine. The work was supported by the Ministry of Higher Education and University Kebangsaan Malaysia and by Science Foundation Ireland as part of the CLARITY CSET (07/CE/I1147).
References 1. Bernhaupt, R., Obrist, M., Tscheligi, M.: Usability and usage of iTV services: lessons learned in an Austrian field trial. ACM Computers in Entertainment 5(2) (2007) 2. Lanagan, J., Smeaton, A.F.: SportsAnno: What Do You Think? In: Proc. of Large-Scale Semantic Access to Content (Text, Image, Video and Sound), RIAO. Pittsburgh, PA, USA (2007)
576
N. Mohamad Ali et al.
3. Lehane, B., O’Connor, N., Smeaton, A.F., Lee, H.: A System for Event-Based Film Browsing. In: Göbel, S., Malkewitz, R., Iurgel, I. (eds.) TIDSE 2006. LNCS, vol. 4326, pp. 334– 345. Springer, Heidelberg (2006) 4. Lee, H., Smeaton, A.F., O’Connor, N., Smyth, B.: User Evaluation of Físchlár-News: An Automatic Broadcast News Delivery System. TOIS - ACM Transactions on Information Systems 24(2), 145–189 (2006) 5. McKeown, K., Barzilay, R., Evans, D., Hatzivassiloglou, V., Klavans, J., Nenkovo, A., Sable, C., Schiffman, B., Sigelman, S.: Tracking and summarazing news on a daily basis with Columbia’s Newsblaster. In: Proc. of the Human Language Technology Conference (2002) 6. Smeaton, A.F.: Indexing, Browsing and Searching of Digital Video. In: ARIST - Annual Review of Information Science and Technology, ch. 8, vol. 38, pp. 371–407 (2004)
Using Activity Descriptions to Generate User Interfaces for ERP Software Timothy O’Hear and Yassin Boudjenane Revelate SA, 38 ch. de Pré-Gentil, 1242 Satigny, Switzerland [email protected], [email protected]
Abstract. Delivering tailor-made ERP software requires automation of screen and printed report creation to be cost effective. Screens generated directly from data structures tend to have poor usability. An approach is considered using a domain specific language to describe use cases. Paper-prototyping and usability testing results define the usability characteristics the DSL portrays. The DSL is capable of defining a variety of screen types and user interface elements including forms, lists, pivot tables, Gantt charts, calendars and graphs. This approach is currently used in production to generate an interactive “AJAX” web user interface as well as HTML, PDF and Excel reports from descriptions stored in XML files. We believe that further research could extend our results to include non-ERP type software. Keywords: AJAX, domain specific language, DSL, ERP software, HTML, interaction design, paper-prototyping, usability, user interface.
1.2 Removing Dependence on Specific Rendering Technology A few years into the life of our company we had to deal with a rather traumatic event. The very first version of our application was developed with Visual Basic 6, a choice driven by a desire for maximum integration with the Windows platform. Well aware of VB6’s limitations we were betting on Microsoft’s promise of a much improved version in a couple of years. When .Net was finally released the upgrade process took a sledgehammer to every one of our screens and required months of work to fix. We realized then just how risky it is to store customer specifications inside a closed format, mixed up with platform specific code. 1.3 Improving Usability Though our windows client was generally liked by our clients, we felt that considerable improvements in both initial learning and everyday use were possible. We were particularly inspired by the work of Adam Cooper [1], Jakob Nielsen [2] and Don Norman [3]. Whatever new ideas we had would need to be prototyped and thoroughly user tested before making it into production. 1.4 Choosing a Starting Point When in 2006 we started to design the next version of Revelate it was clear that big changes were required for screen and report production. As a consequence of the above our brief included: • Screen and report layout must be fully automatic. • Screens and reports must be described independently from a specific rendering technology (e.g. Java’s Swing, .Net’s XAML, HTML or Flash). • Usability of the windows client must be improved upon. • Screen and report descriptions must be device independent; they can be rendered on anything from a mobile phone to a workstation. We simply couldn’t find much existing work even less an existent screen or report description language that matched these criteria. On the other hand Model Driven Architecture approaches appeared too abstract and generally focused on business logic rather then screens and reports. See [4], [5], [6], [7] for examples of these approaches. There are a number of user interfaces generators such [8] available that rely purely on a description of the underlying data structures. Typically, they use raw database tables or the business object domain. Our analysis of such generators is that, while the screens they generate are suitable for system administration tasks, they rarely meet reasonable usability requirements, and as such are uncompetitive with manually crafted user interfaces. This is due to the underlying data model rarely matching the user’s mental model of an application. Other generator approaches such [9],[10],[11] didn’t quite match our needs. Our existent approach required us to describe each of our clients’ use cases as part of the specification document. Hence we decided to consider the use case (or activity) as a container of screens and/or reports. We could then define the use cases with a
Using Activity Descriptions to Generate User Interfaces for ERP Software
579
domain specific language containing sufficient information to generate the actual screens and reports. This approach is examined below.
2 Hypothesis Design decisions resulting in a well crafted user interface can be systematically reproduced by an automatic generator. This is achieved by identifying individual tasks or activities and defining their key characteristics. These characteristics will not be related to a specific graphical user interface toolset, nor indeed a specific screen size or input method. Furthermore, similar characteristics can be defined for printed reports.
3 Methods We first manually “generated” a user interface according to a set of rules and user tested this with a paper prototype. This was not particularly successful so we switched to the design of a complete application module without initially concerning ourselves with how it would be generated. We built an interactive prototype to test our design assumptions and achieved excellent results. Reverse engineering the tested design provided the foundation for the first DSL. Finally analysis of existing applications provided further ideas to enhance our DSL. 3.1 Paper-Prototyping the First Approach Our first approach was based on the idea that a use case or activity could be split into a multitude of small steps. Each step would have its own screen and a navigation system would be generated to link each step together. Such interfaces, we felt, could easily be generated. We created a paper prototype of the concept and organized user testing as described in [12] for a simple task: entering the information contained in a business card. We decided to model the business card as 2 contacts (the employer and the employee) with addresses and phone numbers that could belong to one or the other. We were aware that it would not necessarily match the users’ mental model of a business card, but felt that it was representative of typical complexity in an ERP package. Testing revealed that we underestimated the impact of the mental model, furthermore the considerable amount of navigation required to accomplish this task rapidly led to loss of context. It became clear that preserving context had to be built into our user interface model. 3.2 Ex-nihilo Design of a Module and Constructing an Interactive Prototype As the results of the paper prototype were not encouraging our interaction director, who had not been all that happy with the “let’s test screens we know how to generate” approach, insisted on doing the opposite: a thorough ex-nihilo design of a complete project management module using the process described in About Face [1]. This
580
T. O’Hear and Y. Boudjenane
Fig. 1. A storyboard describing the planning module with complete lack of regard for the length of our developers’ work week
design was built up using personas (Cooper’s version of use cases and roles), storyboarding, paper prototyping (Fig. 1) and finally an interactive prototype. User testing was extremely successful thanks to the new design’s use of context preserving mechanisms such as in-situ editing, progressive disclosure (Fig. 2) and pop-up windows. By in-situ we mean that anywhere you can read information you can edit, delete or add to it. By progressive disclosure we mean that for a given screen element additional screen elements may be displayed either automatically when the user focuses on the screen element or at the user’s discretion. By pop-up window we mean both transient non-modal elements such as tooltips, contextual menus or warning messages as well as modal dialogs.
Fig. 2. Progressive disclosure used in a list. A panel containing a large comment field is displayed inside the currently selected line. Note also that fields appear editable only when selected.
Using Activity Descriptions to Generate User Interfaces for ERP Software
581
Another strength of the new design was its use of advanced user interface elements such as calendars and pivot tables to manipulate data. We opted to reverse engineer the simpler screens first and look after the advanced elements later. This led to two separate DSLs. Note that paper prototyping of such a highly interactive design didn’t work particularly well, hence the need to build an interactive prototype. 3.3 Iterative Evolution of the Domain Specific Language in Real World Situations To kick start the process, we designed the first DSL by listing the information required to render a simple set of screens and finding out ways to deduce anything that would have dependent on the rendering technology or the screen size. As we added more complex screens and client requirements the DSL evolved. The second DSL built on the lessons learned from the first one and had a much longer design gestation. It is capable of representing all screen elements of the first DSL as well as trees, tree grids, pivot tables, Gantt charts and calendars. Both of the DSL have now been used in multiple projects and have evolved accordingly. 3.4 Analysis of Existent Applications To further increase the potential of our DSLs a variety of existing applications renowned for their user interfaces were analyzed, their user interaction design patterns and layout rules identified and their design decisions deduced. Often similar activities were represented using completely different approaches. Discovering the forces causing these differences provided a number of the key characteristics that we’re currently integrating into the DSLs. We are also analyzing the numerous catalogues of user interaction patterns such as [13] in view of integrating this approach into the DSL.
4 Results Both domain specific languages are considered with the organization or layout of “controls” (sometimes called widgets) in screen space and the projection of data onto these controls. The first DSL focuses mainly on static screen layout, whereas the second DSL is able generate sophisticated data driven layouts. 4.1 The first DSL or “Activity Structure” The Activity Structure is based on a fairly static hierarchy of “controls” which are outlined in Fig. 3 and described in Fig. 4. An Activity contains one or more sections; each section contains one or more regions and each region contains one or more fields. Most of the action occurs at the Region and Field levels. Regions can be lists, forms or dashboards. Fields can be buttons, query fields or inputs (with a large variety of input types). Sections can be pages, drop-downs, sliding panels, tooltips and
582
T. O’Hear and Y. Boudjenane
Fig. 3. Our current application displaying a typical activity-section-region-field layout
dialogs. The variety of section types enables us to create the level of interactivity that our tests demonstrated was required. Each of the controls has a number of characteristics (e.g. its importance or whether it is editable or read-only). None of these characteristics define information that is specific to screen size or rendering technology.
Fig. 4. UML class diagram of the Activity Structure
The Activity Structure also describes object sets that are bound to regions and defines the portion of the object domain used in the activity. They are filled with objects either by default queries defined in the DSL or via user interaction. Object sets can sort and filter data. They can also be linked together to define parent-child relations between on screen objects such as invoice – line items. The major limitation of the Activity Structure is that additional regions or input types need to be developed explicitly; the language is not capable of describing them
Using Activity Descriptions to Generate User Interfaces for ERP Software
583
directly. Also the object sets used are one-dimensional so they can not directly describe hierarchies or pivot tables. 4.2 Motivation for the Second DSL The second DSL’s brief required the ability to describe trees, tree grids, pivot tables, Gantt charts, calendars and charts. In terms of our first DSL, the Activity Structure, it is a language for describing Regions or Fields. The underlying concept is that we are projecting an n-dimensional object space onto the x-y axis of screen space. Take for example work items produced by a given employee on a specific date, where the employee belongs to a department. There’s a number of different ways we could project this information on to the screen, for instance: • A list of all the work items with 4 columns: employee, department, date, units. • A form displaying a single work item at a time with a set of arrows to move back and forward in the list. Four fields display employee, department, date, units. • A pivot table with months along the x axis and employees along the y axis, with the total units displayed in each cell • A tree with department nodes containing employee nodes, containing work item leaves. • A bar chart with departments along the y axis and total units along the x axis. Though these screens appear to be completely different underneath they are really more or less the same. 4.3 The Second DSL or “Control Structure” An outline of the “Control Structure” is as follows: • Object set o Dimension Sort order • Control o Object set [link to a set defined above, used for display data] o Display Controls Control o Axis [0,1,2] Binding function [discrete or continuous] − Object set Dimension [optional] Coordinate object set [optional] Axis link − Child axis Generally speaking the Control Structure lays out controls along a Cartesian referential defined by 0, 1 or 2 Axes. Objects are positioned on each axis using a binding function then displayed using the Display Controls. For example an input form has no Axis, and as many Display Controls as inputs fields. A list has a single Axis and as many Display Controls as columns. A pivot table typically has a single Display Control and 2 axes.
584
T. O’Hear and Y. Boudjenane
Fig. 5. The Control Structure is used to define our in-house DSL editor
A tree or multi-level pivot table are created using a parent-child Axis link. This can be defined as recursive as the number of levels in a tree isn’t necessarily known before hand. Fig. 5. illustrates such trees used by our DSL editor, itself generated by a Control Structure description. In the cases above, the Axes are discrete, in the sense that they position the Display Controls in specific cells. Axes can also be continuous to position Display Controls outside the confines of cells. So for instance a calendar in day view has 2 Axes; the vertical Axis uses a continuous binding function. A Gantt chart also has 2 Axes; in this case the horizontal axis uses a continuous binding function. A line chart has 2 axes both of which use continuous binding functions. 4.4 Rendering the Control Structure The rendering pipeline is composed of 2 blocks: • Abstract rendering: projects objects into a 2-dimensional off screen representation of the control structure. This involves laying out child controls and making screen size related decisions. • Concrete rendering: the 2-dimensional abstract rendering is mapped to a specific rendering technology. For example, if using HTML as the rendering technology the concrete renderer will decide where to use tables or lists.
5 Discussion The first versions of both domain specific languages have been integrated into our ERP application and used successfully in client projects. We use three different
Using Activity Descriptions to Generate User Interfaces for ERP Software
585
concrete renderers. One renderer uses dynamic html (“AJAX”) to generate user interfaces with a level of interactivity matching the best desktop applications. Another renderer produces nicely laid out PDF reports. The third renderer produces partially interactive Excel reports. As both domain specific languages are stored as xml, they are combined in a single file (a *.ract file) that thus contains the full description of an activity or use case.
Fig. 6. An elaborate chart that would be a massive pain to recreate from scratch each time
The Control Structure DSL in particular is still evolving. For instance, we are working on producing the type of charts illustrated in Fig. 6. Though these can be described using the Control Structure the result is too verbose. Therefore the next step is to define Control Structure templates to enable reuse of the more complex controls.
6 Conclusion These results support the theory that elaborate user interfaces for ERP software can be generated with a simple domain specific language. They also demonstrate that the same description can generate user interfaces for different technologies. The current implementation relies on the fact that our ERP uses a different screen for each activity, whereas typical desktop productivity software (e.g. Office or Photoshop) allow a large number of activities on a single screen. Further work is required to handle this scenario, but it doesn’t seem conceptually orthogonal. Acknowledgments. The authors gratefully acknowledge and thank Ben O’Hear for his interaction design work and Jean-François Burdet for the implementation of the first DSL and many helpful comments and suggestions.
586
T. O’Hear and Y. Boudjenane
References 1. Cooper, A.: About Face 3: The Essentials of Interaction Design. Wiley, San Francisco (2007) 2. Nielsen, J.: Prioritizing Web Usability. New Riders Press, Indianapolis (2006) 3. Norman, D.: The Design of Everyday Things. Basic Books, New York (2002) 4. Abrams, M., Phanouriou, C., Batongbacal, A.L., Williams, S., Shuster, J.: An ApplianceIndependent XML User Interface Language. In: Proceeding of 8th International WorldWide Web Conference, 5. XUL tutorial, http://www.xulplanet.com/tutorials/xultu/ 6. Puerta. A.R., Eisenstein, J.: A Common Representation for Interaction Data 7. Azevedo, P., Merrick, R., Roberts, D.: OVID to AUIML - user-oriented interface modeling. In: Proceedings of 1st International Workshop, Towards a UML Profile for Interactive Systems Development, TUPIS 2000, York (2000) 8. Metawidget, http://www.metawidget.org 9. Sukaviriya, P., Foley, J., Griffith, T.: A Second Generation User Interface Design Environment, Bridges between Worlds. In: Proceedings InterCHI 1993, pp. 375–382. ACM Press, New York (1993) 10. Vanderdonckt, J.: Knowledge-Based Systems for Automated User Interface Generation: the TRIDENT Experience 11. Elwert, T., SchlungBaum, E.: Modeling and Generation of Graphical User Interfaces in the TADEUS Approach, Designing, Specification and Verification of Interactive Systems, pp. 193–208. Springer, Heidelberg (1995) 12. Snyder, C.: Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces. Morgan Kaufmann, San Francisco (2003) 13. Tidwell, J.: Designing Interfaces: Patterns for Effective Interaction Design. O’Reilly Media Inc., Sebastopol (2005)
Developing a Nomenclature for EMR Errors Win Phillips and Yang Gong Department of Health Management and Informatics, CE707 CS&E Bldg, School of Medicine, University of Missouri Columbia, MO 65212 USA [email protected], [email protected]
Abstract. Latent medical errors may occur in electronic medical record (EMR) systems. Analyses of medical errors, including the cognitive theory of action and the systems approach, are described. Key aspects of EMR systems are presented and examples are provided. A nomenclature is suggested to improve reporting and communication about EMR errors. The nomenclature uses concepts of an error state and a precipitating event. The error state comprises an error element, an error condition, and an error context. The precipitating event comprises an event agent, and event task, and an event context. The event task includes a task object, a task action, and task parameters. Keywords: medical errors, electronic medical records, electronic health records.
1 Introduction In healthcare contexts electronic medical record (EMR) systems are gradually replacing paper-based records. One hope was that EMR systems would reduce the incidence of medical errors. Unfortunately, the use of EMR systems has brought new kinds of error, “EMR errors.” Discussion of the nature and causes of medical errors has resulted in different perspectives and classification schemes for medical errors. One wonders how to go about understanding, classifying, and preventing EMR errors. As awareness of EMR errors increases, we suggest that clinicians and information systems professionals could benefit from a standard way of talking about EMR errors. In this paper we suggest such a nomenclature for discussing EMR errors. We start by discussing several different perspectives on medical errors. We then mention typical features of EMR systems and discuss EMR errors. After this we suggest a possible nomenclature for use in reporting and discussion of EMR errors.
During the last decade widespread attention to medical errors and their prevention has become part of the culture in most healthcare organizations [2], [3]. A variety of terms are involved in the general discussion of medical errors. The term “adverse event” is often used for an instance of patient harm not due to the natural course of a disease or illness. A “close call” or “near miss” is an event that could have resulted in an adverse event but fortunately did not, whether through sheer chance or being caught in time. Both adverse events and near misses can be seen as medical errors. Another distinction often used is between active and latent errors. Active medical errors are when the error event really happens – the patient gets the wrong medication, for instance. Latent errors are more in the way of conditions existing that under the right set of circumstances could give rise to active medical errors. An example would be incorrect or missing data in a patient’s medical record that could result in a faulty treatment decision. There has been much discussion within healthcare about how and why medical errors occur and how they should be classified. Two different perspectives on how they occur are represented by the cognitive theory of action and the systems approach. The cognitive theory of action seeks to understand errors in terms of the psychological processes of an individual agent or actor. Donald Norman famously divided errors into the two basic classes of mistakes and slips. Mistakes are errors in intention, while slips are errors in carrying out the intention [4]. Norman and others expanded the analysis to delineate specific psychological steps or stages in the action process. Zhang et al, expanding on Norman, claim any simple action proceeds through seven stages of execution and evaluation: establishing a goal, forming an intention, specifying an action, executing the action, perceiving the system state, interpreting the system state, and evaluating the system state [5]. A mistake or slip could occur through a failure to properly carry out any of these stages. For example, a mistake could occur through a nurse misunderstanding the meaning of an alert message on the computer screen (failure in interpreting the system state), while a slip could occur when a physician trying to delete a duplicate chart note accidentally selects the wrong button (failure in executing the action). Various analysts believe that error occurs during the action process due to a lack of attention or through degradation of the user’s internal representational model of the event [6]. A different perspective on error, though one not necessarily incompatible with the cognitive theory, is represented by the systems approach, which focuses not on psychological processes but on faulty systems. A famous proponent of this approach is James Reason. Reason claims the traditional “blame the person” approach to error overlooks the role of the larger context of systems. Error occurs when multiple systems fail, much like what happens when all the holes momentarily line up on adjacent slices of Swiss cheese [7]. Applying the systems approach to medical errors, Leape claims we must analyze proximate and ultimate causes to determine the multiple contributing factors or root causes that resulted in the error, and then we should redesign the system or systems that failed to make it harder for the error to occur [8]. For example, if someone makes an error after bring interrupted, don’t focus on blaming that person but instead on changing the environment to reduce interruptions. Instead of retraining an individual to more carefully distinguish between two look-alike medication packages, change the packages [9]. On this view, medication errors from drugdrug or drug-allergy interactions are not caused by careless individuals as much as faulty or nonexistent checking systems.
Developing a Nomenclature for EMR Errors
589
Besides attempting to understand medical errors, analysts have attempted to classify such errors. Given the variety of medical error classification systems available perhaps what anyone thinks of as the “best” system will be the one most suited to their use of it. In arguably the most comprehensive effort, JCAHO-sponsored investigators gleaned key insights from numerous previous attempts and from industry experts to arrive at five category types (called “root nodes”): impact (harm to the patient), type (processes that failed – such as communication, patient management, or clinical performance), domain (setting and persons involved – such as general hospital, emergency department, etc.), cause (factors and agents – such as system failures or human failures), and prevention and mitigation (measures to reduce occurrence and effects). These primary categories were further divided and subdivided, eventually resulting in over 200 categories of medical error [10].
3 EMR Systems Traditionally medical records consisted of paper-based charts, which are patientspecific folders or binders containing paper notes on diagnosis and treatment, lab test results, and related information. Physician orders for lab tests, medications, and procedures likewise traditionally are provided on paper forms. Records of physician orders are usually included in the patient chart. Paper-based records suffer from many inconveniences and limitations, such as the need for large storage space if there are many patients, the unfortunate ease with which a chart may be misplaced, and the fact that it is difficult for more than one individual at a time to access a paper chart. Paper-based records and orders are susceptible to other problems such as documentation errors and misinterpretation of (sometime almost illegible) handwritten records or orders. Gradually paper-based charts are being replaced by electronic medical record systems. Likewise, paper-based ordering is being replaced by computerized physician order entry (CPOE); CPOE functionality is often incorporated into or integrated with EMR systems. Within the U.S., many different EMR (and CPOE) systems have been developed by different hospitals or are offered by different vendors, and now professional organizations are playing an increasingly large role in supporting a common set of functions. The Institute of Medicine has defined important functions for any EMR system, including managing basic health information, results management, order entry and management, decision support, electronic communication, patient support, administrative processes, and population reporting [11]. To guide development of EMR systems, the Health Level 7 Standards Development Organization has created a standard of more than 125 EMR functions, including direct care functions such as health information capture and management, care plans, medication management, orders and results management, clinical decision support, and clinical communication. They also specified supportive functions such as provider directories, registry notification, and research and reporting, and they listed information infrastructure capabilities such as user access management, standards, and other administrative functions [12]. The Certification Commission for Healthcare Information Technology (CCHIT) has attempted to push a standard by developing a set of
590
W. Phillips and Y. Gong
functionality criteria required for any healthcare electronic record system to merit their certification [13]. Apart from any pressure created by such mandates, it stands to reason that while EMR systems may vary in appearance, navigation, and operation, they will all try to do mostly similar sorts of things because, within the U.S. anyway, healthcare services and medical records have become fairly standardized. There may be variation among configurations of EMR systems for certain specialties, such as pediatrics, and among different contexts, for example, hospitals as opposed to small ambulatory care private practices. But an EMR system would be worthless if it could not store documentation of clinical encounters between healthcare providers and patients or data about patient medications. So, as recognized by professional organizations, much of the functionality will be similar. Functionality might be thought of as grouped into EMR “modules.” A typical module might be “chart notes” or “chart note management” that allows users to create, view, and manage notes that describe clinician-patient encounters or visits. Examples of other EMR modules would be patient demographics, medications, allergies, problems (diagnoses), orders (CPOE), results (lab tests), clinical alerts, and clinical communication. An EMR system will also have supporting functions such as patient search capability, electronic signatures, and reporting. As is common practice in the use of other information systems, EMR systems may limit user access to functionality based on role. This allows users to be able to do what they need to do without having the ability to do what they don’t need to do. Examples of roles are shown in table 1. Table 1. Examples of EMR roles Role
Privileges
Physician
Create, electronically sign, and close chart notes, place orders, full viewing Nurse practitioner / Similar to physician but may need physician cophysician assistant signature on chart notes and orders Nurse Similar to physician but ability to create and sign notes and orders may be limited Scheduler/Clerk Limited “view-only” ability for notes System administra- Ability to correct and delete chart notes and data entry tor errors throughout system; manage users in system
As mentioned, EMR systems are used in a variety of clinical settings, both in hospital and in ambulatory clinics and practices. Computers can be located at nursing stations, physician offices, front reception desks, exam rooms, hospital patient rooms, and even on mobile carts or freestanding laptops, tablet computers, and PDAs. For configuration purposes, within a hospital or multi-branch clinic there may be the need to distinguish in the system among access from different offices. So, for example, a chart note created at the main clinic might need a different letterhead address than a note created at a branch office. Table 2 presents examples of clinical contexts.
Developing a Nomenclature for EMR Errors
591
Table 2. Examples of clinical contexts
Main office Cherry Street branch office Exam room A Nursing station, 4th floor Physician office, Dr. Smith Operating room #3 Patient room, #412 Mobile cart, surgical floor Roaming laptop, assigned to psychiatry PDA, assigned to Dr. Jones An important reason for the documentation in an EMR is to make information available for future decisions. From the perspective of distributed cognition theory, both paper-based and electronic medical record systems are external aids that greatly affect the decision-making processes of physicians and other clinicians. Consider a physician who, perusing a list of chart notes completed by various clinicians on a particular patient, desires to see the last chart note he composed on that patient so as to refresh his memory about the treatment plan he developed for a particular problem of the patient. To make finding this note easier he performs the task of sorting the chart note list alphabetically by physician name. To better understand such an action, one could analyze this task into components, using, for example, the concepts of event, agent, object, action, and parameter. The instance of performing the task is an event. The event agent (here a physician) performs an event task (sorting the chart note list alphabetically by physician name). The event task comprises a task object (chart note list), a task action (sorting the chart note list), and task parameters (alphabetically by physician name). These concepts will be useful later in this paper. Let’s turn to consider EMR errors. Medication errors and wrong-site surgery probably get the most headlines, but errors also occur in medical records and physician orders. Handwriting and the use of dictation and transcription can lead to data errors and misinterpretations in paper-based medical records and ordering systems. One hope for the adoption of EMR (and CPOE) systems was that the occurrence of medical errors of various kinds would be reduced, for example by streamlining documentation and ordering, by making important information more readily available, and by providing decision-support tools. Computerization may well have reduced some types of error, as well as increasing convenience of access, but unfortunately the use of computerized systems seems to have created new kinds of error. For example, now that medical records are on the computer, clinicians can inadvertently copy and paste sections of notes into the wrong place or even the wrong patient’s chart, accidentally delete unclosed chart notes, and draw the wrong conclusions about patient progress by misinterpreting lab results displayed on poorly designed summary screens. Several studies have shown various types of error or failure of expected outcomes upon deployment of ordering systems [14] [15]. Ash et al found instances in which “patient care information systems” (their term for systems such as EMR and CPOE
592
W. Phillips and Y. Gong
systems) “seemed to foster errors rather than reduce their likelihood.” Focusing on human-computer interaction, Ash et al found such systems resulted in problems in the process of information entry/retrieval and in the processes of communication and coordination. These problems are of various kinds. For example, user interfaces (screens, navigation) sometimes were ill-suited to busy healthcare work environments, documentation created via pre-supplied phrases and sentences reduced readability, overly busy screens and too-frequent intrusive alerts caused “cognitive overload,” and there was insufficient error-checking by the system [16]. We suggest use of the term “EMR error” for, within an EMR system, any incorrect data or faulty functionality, any aspect of the system not functioning according to design specification or user requirements, any incorrect design specification or user requirement, and any significantly suboptimal usability. We do not restrict EMR errors to problems from human-computer interaction; programming errors and hardware failure are included. Now of course a misplaced character in an obscure part of a chart note might be irrelevant to patient care, but significant EMR errors conceivably could lead to real instances of patient harm. In contrast to “active” medical errors such as wrong-site surgery, significant EMR errors might be seen as “latent” medical errors in that they create conditions that could result in active medical errors (e.g., diagnostic or treatment errors) that might compromise patient safety. Clinicians using an EMR system who notice (significant) faulty data or functionality will likely report the problem to technical support or system administrators in their institution or working for the EMR software vendor. This could occur by phone call, email message, or creating a “ticket” in a help desk reporting system. The tech support person will attempt to understand the nature of the problem, and when and how it occurs, and then they or someone else will work to determine the cause and the appropriate resolution or prevention measures. This kind of EMR error reporting, analysis, and resolution requires cooperation and effective communication among clinicians, technical support staff, and management, but communication may be hindered if there is no good way of describing or referring to the problem. The user may have a basic understanding of what happened, but to tech support the user description of a problem may consist of vague, incomplete, or confusing phrases such as “the menu is missing things,” “I can’t see the bottom of the screen,” or even simply “the system doesn’t work right.” What menu or screen is involved, what specifically is wrong with it, or in what way are things not right? Such vagueness complicates the work of tech support because lengthy phone conversation, observation, and attempts at recreation may be needed before the precise nature of the problem even can be understood. In an online system, tech support may have created drop down menus asking for choices from users when the ticket is first submitted, this to try to route the problem to the right person, but the menu items may be beyond the ability of a typical clinician to decipher or decide upon. How many typical users can say whether it is a hardware, software, application, operating system, or network error? Beyond such initial reporting from the user, even staff investigating the error may lack a convenient way of referring to the problem. The problem may be assigned a tracking number to uniquely identify it, but this number will fail to tell anyone whether and in what way it is related or similar to other problems.
Developing a Nomenclature for EMR Errors
593
4 A Suggested Nomenclature for EMR Errors To facilitate worthwhile discussion concerning EMR errors we suggest development of a more nearly standard nomenclature or vocabulary. The concept of a standard nomenclature or vocabulary should be nothing new to clinicians used to standardized clinical vocabularies and codes such as ICD-9 diagnosis codes (for diseases) and Current Procedural Terminology (CPT®) codes for office visits and procedures. Standardized terms and codes for diseases and procedures have been used for years to avoid situations in which every clinician has his or her own distinctive set of names. We are urging that a similar (though less extensive) standardization effort be undertaken for EMR error reporting and analysis to improve the quality of EMR error communication, discussion, and resolution. We are not suggesting a numerical classification scheme but rather merely the adoption of some standard way of talking of EMR errors. In light of the plethora of efforts expended by many parties during the last decade or so to try to develop the best classification system of medical errors, seemingly not stemmed by the comprehensive JCAHO effort [17], we are not here proposing an ultimate classification system or ontology for EMR errors. But such classification efforts immediately suggest two possible categories for use in such a nomenclature: “type” and “cause.” Our system does allow for possible use of the concept of an EMR error “type,” but it remains to be seen what the best options for type might be and whether this concept would be useful for any nomenclature attempting to be relevant to actual discussion of EMR errors in the field, i.e., clinical settings and conversations among clinical and information systems personnel trying to identify and resolve real EMR error instances. Medical error “types” mentioned in most academic discussions are likely too abstract or vague for feasible use by actual clinicians, whether or not such categories are helpful to academics trying to understand the nature and causes or such errors. Common EMR error types might be data (incorrect or missing data), functionality (some function that needs to work is not working or not working correctly), and performance (the system response is perceived as slow), though one also thinks of such possibilities as communication, data entry, screen layout, screen navigation, missing button or menu item, etc. Unfortunately the typical clinical user might have difficulty determining the proper type for their error with even a very simple set of options because they are not focused on classifying computer system errors, they are focused on performing clinical tasks. So we hesitate to pontificate about what options should be available for type or even whether it should be used. In light of the controversial topic of whether the “cause” of a medical error is to be found in psychological processes (depicted in the cognitive theory of action, for instance), faulty systems (the systems approach to error), or something else, we suggest that “cause” not be included as a basic category or element in our nomenclature. It is unrealistic anyway to expect the user who reports the problem or those who subsequently discuss it to be able to correctly tell you the ultimate or root cause or type of cause of the error prior to the investigation. Also, no matter what user options for elements or categories are chosen, the option terms used should, if possible, not
594
W. Phillips and Y. Gong
be unduly technical but rather something the average user can understand. These considerations are meant to try to ensure that any nomenclature developed for use in discussing EMR errors will be something useful in a first-pass, preliminary, and elementary discussion of the error. We distinguish between two aspects of the type of nomenclature that we have in mind. First are the concepts, categories, or elements of the nomenclature; these are the kinds of things we take note of or use to talk of the error. Second, within each of these elements are the values or options from which one might choose. Below we offer suggestions about the error elements or concepts that might be useful in this effort, as well as particular values or options within each element, but perhaps pilot studies using this scheme and analysis of its usefulness will be the ultimate arbiters of the optimal elements and options. Our suggested nomenclature includes the concepts of an error state, an associated precipitating event, and related subconcepts. To put it simply, the error state is what is wrong, and the precipitating event is what the user was doing when they noticed something wrong. Table 3 displays the concepts and subconcepts in the nomenclature. Table 3. EMR error nomenclature concepts and subconcepts Concept
Subconcepts
Error State (what is wrong)
Error Element (incorrect datum or function) Error Condition (what is wrong about element) Error Context (location of element in system)
Precipitating Event (what the user was doing)
Event Agent (user) Event Task (activity being performed in system) Task Object (involved in task) Task Action (what was being done) Task Parameters (modifiers) Event Context (clinical or admin. context)
The error state is what is wrong, incorrect, or malfunctioning in the system that makes the situation the occurrence of an error. The error state comprises an error element (the incorrect datum or function), an error condition (what is wrong or incorrect about the error element), and an error context (for example, the screen, textbox, button, report, EMR modules, etc. in which the error element is located). As an example consider the situation of a physician who notices that dates suddenly go missing after sorting a list. The error state would be “chart note dates missing from chart note list in chart note management module.” The error element would be “chart note dates,” the error condition would be “missing,” and the error context would be “chart note list in chart note management module.” Table 4 provides examples of error elements and error conditions. In that table a sample condition is associated with each element, though of course other conditions could be associated with that element.
Developing a Nomenclature for EMR Errors
595
Table 4. Examples of error elements and error conditions
Error Element Chart note Date Blood pressure Screen Summary screen Patient address Medication dosage Medication SIG menu Chief complaint Problem list Plan Drug allergy Food allergy Save button Edit chart note function Results display
Error Condition missing incorrect transposed misaligned poorly designed outdated inapplicable working incorrectly duplicate unsorted irrelevant expired confusing misleading not working ungrouped
Associated with the error state is a precipitating event, which is, from the user’s perspective, what gave rise to the error state or at least what the user was doing or attempting to do right before or when the error was observed. In fact this precipitating event could have played a causal role (triggering the error) or simply a revealing role (allowing the error to reveal itself). But the user will not necessarily know whether the precipitating event had a causal role, and as mentioned, the notion of the “cause” (as opposed to causal factors) may be controversial. In our nomenclature the precipitating event will feature an event agent (the user performing or attempting a task), an event task (an activity being performed or attempted), one or more task objects (involved in the task), a task action (what was being done), one or more task parameters (specifics about the way the task was done), and an event context (the clinical or administrative context of the precipitating event). Please refer to table 3 for clarification. Event agents are typical user roles correlated with different levels of access in the system; examples were listed in table 1. The event task, including task object, task action, and task parameters, were already discussed earlier in this paper in our discussion of EMR systems and are going to reflect the many tasks one can accomplish with an EMR, such as creating a chart note, entering a patient’s prescription, sorting, deleting, copying, etc. The event context could be “Main office exam room A”; additional examples of the event context were listed in table 2. Our suggested nomenclature permits post-coordination and is intended for use in documenting and reporting EMR errors, which would facilitate improved communication between users and issue solvers. Ultimately, using this type of nomenclature to group similar cases could allow trigger case-based reasoning and provide for timely solutions.
596
W. Phillips and Y. Gong
References 1. Kohn, L.T., Corrigan, J.M., Donaldson, M.S.: To Err Is Human. National Academy Press, Washington (1999) 2. Zhang, J., Patel, V.L., Johnson, T.R., Turley, J.P.: Health Informatics and Medical Error., pp. 34–35. U.S. Healthcare Strategies, Business Briefing (2005) 3. Zhan, C., Kelley, E., Yang, H., Keyes, M., Battles, J., Borotkanics, R., et al.: Assessing Patient Safety in the United States: Challenges and Opportunities. Medical Care 43(3), I-42I-7 (2005) 4. Norman, D.: Categorization of Action Slips. Psychological Review 88(1), 1–15 (1981) 5. Zhang, J., Patel, V.L., Johnson, T.R., Shortliffe, E.H.: A Cognitive Taxonomy of Medical Errors. Journal of Biomedical Informatics 37(3), 193–204 (2004) 6. Botvinick, M.M., Bylasma, L.M.: Distraction and Action Slips in an Everyday Task: Evidence for a Dynamic Representation of Task Context. Psychonomic Bulletin & Review 6(12), 1001–1017 (2005) 7. Reason, J.: Human Error: Models and Management. BMJ 320, 768–770 (2000) 8. Leape, L.L.: Ethical Issues in Patient Safety. Thoracic Surgery Clinics 15, 493–501 (2005) 9. Leape, L.L., Bates, D.W., Cullen, D.J., Cooper, J., Demonaco, H.J., Gallivan, T., et al.: Systems Analysis of Adverse Drug Events. JAMA 274(1), 35–43 (1995) 10. Chang, A., Schyve, P.M., Croteau, R.J., O’Leary, D.S., Loeb, J.M.: The JCAHO Patient Safety Event Taxonomy: A Standardized Terminology and Classification Scheme for Near Misses and Adverse Events. International Journal for Quality in Health Care 17(2), 95–105 (2005) 11. Board on Health Care Services, Institute of Medicine: Data Standards for Patient Safety, Key Capabilities of an Electronic Health Record System, Letter Report (2003) 12. Abdelhak, M., Grostick, S., Hanken, M.A., Jacobs, E.: Health Information: Management of a Strategic Resource, 3rd edn. Saunders Elsevier (2007) 13. Certification Commission for Healthcare Information Technology: CCHIT Certification for Ambulatory Electronic Health Records, http://www.cchit.org/certify/ambulatory/index.asp 14. Han, Y.Y., Carcillo, J.A., Venkataraman, S.T., Clark, R.S., Watson, R.S., Nguyen, T.C., Bayir, H., Orr, R.A.: Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System. Pediatrics 116(6), 1506–1512 (2005) 15. Koppel, R., Metlay, J.P., Cohen, A., Abaluck, B., Localio, A.R., Kimmel, S.E., Strom, B.L.: Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors. JAMA 293(10), 1197–1203 (2005) 16. Ash, J.S., Berg, M., Coiera, E.: Some Unintended Consequences of Information Technology in Health Care: The Nature of Patient Care Information System-Related Errors. J. Am. Med. Inform. Assoc. 11, 104–112 (2004) 17. Jacobs, S., O’Beirne, M., Derfiingher, L.P., Vlach, L., Rosser, W., Drummond, N.: Errors and Adverse Events in Family Medicine: Developing and Validating a Canadian Taxonomy of Errors. Canadian Family Physician 53(2), 270–276 (2007)
Mapping for Multi-source Visualization: Scientific Information Retrieval Service (SIRS) Dario Rodighiero1, Matina Halkia2,*, and Massimiliano Gusmini3 1
Arcadia S.I.T. Via Mondovì, 4 I-20132 Milano (MI), Italy [email protected] 2 Joint Research Centre of the European Commission, Institute for the Protection and Security of the Citizen Via E. Fermi, 2749 I-21027 Ispra (VA) Italy [email protected] 3 Reggiani Via Tonale, 133 I-21100 Varese (VA), Italy [email protected]
Abstract. This paper discusses the design process of a multi-index, multisource information retrieval system (SIRS). SIRS provides comprehensive visualization of different document types for the JRC working environment. The interface design is based on elastic window management and on the Focus+Context method to browse large amounts of information without losing its contextual relevance. Source integration was achieved by mapping techniques, on which we applied methods, degree-of-separation and closure, to provide advanced relational context for objects. Keywords: interface design, information visualization, mapping, multiple indexes, SIRS, adaptive interface.
1 Introduction In recent years, many organizations have invested in semantic integration of information systems. Examples include the Terminology Services by OCLC, the mapping experiments by FAO, the CrissCross by the German National Library and Vascoda by the German Research Foundation [1]. This project also investigates the potential of document retrieval based on multiple indexes. In this case however, indexes are integrated through an adaptive interface as to enable browsing across different types of documents, reflecting the complex work environment of a public research organization. The work was conducted at the Joint Research Centre (JRC), European Commission (2006-2009). The project aim was to establish a well-organized and comprehensive scientific information discovery system. The system is provided as a JRC Central Library service, and it is known as Scientific Information Retrieval Service, (SIRS). *
The first two authors equally contributed to the paper.
The technical development was completed in February 2009 while the content is still being expanded. We will discuss this work using the Löwgren and Stolterman principles [2]. According to their principles, the design process is composed by the Vision, Operative Image and Specification phases. Löwgren and Stolterman believe that the interaction design process starts earlier than the traditional methodologies allow in information system development and software engineering. According to them, the “design of the design process,” the questions chosen to be addressed at an early stage and requiring the “mind of a thoughtful designer,” (how much time and importance is placed to the organization of the project, the users, the clients, the choice of technology: old and tested, or innovative and creative, etc.), greatly impact the end result. The three Löwgren and Stolterman stages of design are: − The vision which emerges early in the design process. Basic elements take shape: an idea, a function or an infrastructure. Intuition plays an important role in this early stage; − The operative image is the first tangible expression of the vision. It takes the form of sketches, metaphors, analogies, scenarios and storyboards. By a process of cyclical refinement or an iterative dialectical process we obtain the final model; − The specification, or final model, where all interface details are well-defined and ready to be integrated in the product construction. To be sure, there is no clear division between design and construction, nor such definitive boundaries between the three phases. However, we have found the Löwgren and Stolterman approach to be very useful in organizing the development work on SIRS and in discussing it critically. Indeed, substantial time was spent in these three phases before construction started.
2 Designing SIRS 2.1 Vision The underlying idea behind SIRS is to provide JRC users with a novel browsing experience by permitting control over three different knowledge sources, EU legislation, JRC publications, and JRC Central Library holdings, separated in three different databases, but used interoperably in the working life of JRC scientists. Now, the SIRS system allows users to discover this information in one space – the SIRS interface – that accurately describes the complexity of the organization they serve. In this phase, which is the investigative part of the design process, we identified the types of users, which led us to their needs. Then, we identified the type of documents they use in their daily working life, which in turn directed us to the indexes these documents refer to. The design process started with these basic elements: The users: − The researcher, who should be able to retrieve the journal articles, the bibliographic references to books and journals, relevant scientific websites, and JRC publications in his specific field of interest;
Mapping for Multi-source Visualization: Scientific Information Retrieval Service
599
− The thematic programme leader, who should be able to retrieve information about scientific institutions active in a specific field, relevant EU policy documents, and descriptions of the main JRC research themes in the relevant policy area or in the specific scientific field; − The JRC Action leader, who should be able to retrieve European legislation, supported by JRC actions, and the internal publications output in support of the former. The investigation of users' needs led us to identify three information sources whose items were designated as documents: − The EU legislation, whose policy instruments indicate the present and future directions of the research in the JRC; − The JRC publications, which comprises all JRC publications produced in the past; − The JRC Central Library holdings, which includes books and both paper and digital journals, reflecting the scientific production outside JRC. Since each document is classified according to its source, referring to a different indexing structure, we have three indexes: − The JRC Actions, a taxonomy which reflects the internal JRC organization and provides the descriptors for profiling the JRC publications; − The Eurovoc Thesaurus [3], a thesaurus that comprises more than six thousands index terms, developed specifically for organizing EU legislation, by the European Parliament and the Office for Official Publications of the European Communities (OPOCE); − The Dewey Decimal Classification [4], an internationally applied decimal system of library classification with more than ten thousands captions, which index Library holdings. 2.2 Operative Image Once we gathered the basic design elements, the information was translated in scenarios. In the process of writing these scenarios, we realized the importance of integrating the three different types of indexes and documents. Thus, when we started sketching the interface, the Focus+Context paradigm [5] was introduced as a natural way to manage the quantity of information while keeping it contextually relevant. Two assumptions were made early on: the contextual relevance of information and the adaptability of the interface. On one hand, to keep the contextual relevance of information visible to the user, the SIRS interface space should include all indexes and all documents at all times. On the other hand, to provide an adaptive interface to the variable user focus, the screen organization should be elastic in order to accommodate the movement of users' attention. The first assumption implied a rational spatial organization of the screen in modules, and the second dictated the behaviour of the geometric characteristics of these modules, which we call frames. Each frame is rectangular and tagged by a label, which indexes the content. The first division reflects the source composition: index on the left and documents on the right, according to the direction of western writing. Then, each frame splits horizontally or vertically as needed for inserting three indexes and three document sets. Indexes are aligned vertically and documents horizontally.
600
D. Rodighiero, M. Halkia, and M. Gusmini
Fig. 1. Succession of frames accommodation
To make the interface space elastic, we enhanced the SIRS interface with a mechanism to enable frames' size customization: in three actions, the users could change the frames' size according to the movement of their focus area. During the change from an arrangement to another, the visual continuity is ensured by the use of a smooth movement of all frames. The three actions are: − Enlarge, which makes the focused frame double-sized when others are still present in the same area; − Maximize, which makes the focused frame as big as possible; − Normalize, which restores default size values.
Fig. 2. Frames customization on 1) index enlargement, 2) documents set enlargement, 3) index maximization and 4) documents maximization
Frames host objects. Objects belonging to an index are called terms. Vertical frames, which host indexes, display terms in two ways: as a multiple-level structure or as a simple list. Multiple-level structures allow browsing through index hierarchies. By default, the SIRS interface provides terms presented as a simple list. Horizontal frames display documents only as a list. The documents are listed chronologically, both for EU legislation, and JRC publications, while it is alphabetical for Library holdings. Due to the fact that the list of documents could be very long (in
Mapping for Multi-source Visualization: Scientific Information Retrieval Service
601
the thousands, in the case of EU legislation), we decided to limit the number of visible documents to less than forty. The complete list of documents is available by an appropriate link at the bottom of the frame. This accommodates needs for different user experience. 2.3 Specification Once the operative image has been discussed sufficiently we can move to the system specification. The first decision we made for the specification of the interface was to use Adobe Flash technology. It is a standard tool for cross-browser compatibility, and offers advanced capabilities in terms of user experience. Another important decision we took for managing the amount of data we had to manipulate was to access by query the remote data repositories through three different web services: EU legislation, JRC publications and Library holdings. We chose not to store the data locally using an ad hoc repository; only integrate information in the SIRS interface. Through web services, the system is able to retrieve documents’ metadata and to point to relative websites for documents’ reading. The next problem we faced and seemed to be insurmountable was that there were no tools available to manage multiple indexes. In fact we had two major technical issues to solve: the first one was the necessity to manage more than one index, and the second was the requirement to create cross-index relations. To respond to these needs, we decided to build the tool ourselves. We call it Thesaurus Management Service (TMS). TMS is an open-source application based on the free PHP technology. Oracle is used for implementation. In order to keep the tool compatibility with open platforms, the system is also available for MySQL. TMS is developed according to the British Standard for Thesauri Construction [6], which manages indexes as concepts and terms. Full TMS development being onerous, the JRC works in partnership with the FAO (Food and Agriculture Organization) and the BGS (British Geological Survey) to complete this project, which is also called Thesaurus Tool Code [7]. In order to enable multi-source browsing, we had to create a unique network composed of terms and documents. To unify different sets of indexes and documents in a single interface, we used the technique of mapping, which enables cross-index relations to connect terms located in different indexes. The interface is based on multiple window management, more specifically the concept of elastic windows, which allows a variable screen real estate to be used according to the role or task performed [8]. As discussed earlier (2.2) there is a basic left-vertical-index to right-horizontal-document screen division. Both indexes and documents are considered sources. EU legislation is the backbone among sources. In fact the Eurovoc is placed in the middle of the three indexes as a bridge. The other two are JRC Actions and Dewey. On the left side, the Eurovoc is mapped to the JRC Actions. Each JRC Action is classified by a set of Eurovoc terms. The relations created through the Eurovoc terms enable cross browsing from EU legislation to JRC publications and vice versa. On the right side, the Eurovoc is mapped with the Dewey. This mapping, which is based on semantic similarity defined by Martin Doerr [9], allows the navigation from EU legislation and Library holdings in both directions.
602
D. Rodighiero, M. Halkia, and M. Gusmini
Fig. 3. The SIRS mapping is based on two types of cross-relations: 1) one between Actions and Eurovoc terms and 2) one between Eurovoc and Dewey terms
Once the relations network was set up, we proposed a method for visualizing information based on graph navigation. For a selected term, we can navigate all relations in order to get the set of terms and documents at the first degree of separation. Moreover, we can use that set as input to receive a new set of terms and documents at the second degree of separation from the selected term. The degree of separation is the measure that we use to define the context. The methodology consists in getting indexes at once, then in retrieving related documents. By selecting a term in the middle index the user follows the shortest route to the objects. The contextual distance is not perceived by the user, but is used to populate the interface. For example, if a Eurovoc term is selected, two degrees of separation are in place. At the first degree, the resulting set includes EU legislation documents, JRC Actions and Dewey objects (Fig. 4.1). At the second degree, we use only the two indexes (JRC Actions and Dewey) as input for retrieving documents from JRC publications and the Library holdings (Fig. 4.2).
Fig. 4. SIRS composes the context from a Eurovoc term respectively using 1) the first and 2) the second degree of separation
On the contrary, when a term in one of the two lateral indexes (JRC Actions and Dewey) is selected, three degrees of separation are in place in the resulting context. For example, if a JRC Action is selected, the first set of related nodes is composed by JRC publications and Eurovoc terms (Fig. 5.1). The second set, which uses only Eurovoc terms as input, includes EU legislation documents and Dewey terms (Fig. 5.2). The third set is obtained using Dewey terms and contains Library documents (Fig. 5.3). Moreover, in the second example, a behavior we call closure [10] can be observed. Closure is a way to suggest similar objects. For example, when we select a JRC Action, we obtain a set of Eurovoc terms. As EU legislation documents are retrieved from the latter, we can also retrieve other JRC Actions (Fig. 5.4). This kind of operation - similar to a boomerang's flight path - suggests a group of JRC Actions, which
Mapping for Multi-source Visualization: Scientific Information Retrieval Service
603
Fig. 5. SIRS composes the context from a JRC Action term respectively using 1) the first, 2) the second and 3) the third degree of separation. 4) The last picture represents the relation used for closure.
Fig. 6. This picture shows how the JRC Action MASURE is connected to JRC Action FISHREG by the sharing of a Eurovoc terms. Closure makes this relation available to users.
share at least one Eurovoc term with the initially selected JRC Action. Closure provides an interface with advanced relational context. As seen in the previous examples the method has four types of parameters: − − − −
The input, in the form of an array of sets; The output, an array of sets too; The degree of separation, an integer; The selected term, an ID number.
To avoid inserting the wrong numbers of parameters, we have to remember that pairs of input and output sets must have the same numerical occurrence as the degree
604
D. Rodighiero, M. Halkia, and M. Gusmini
of separation. For each degree of separation, a pair of input and output sets has to be declared. Practically this means that for two degrees of separation there are two pairs of input and output sets, for three degrees, three pairs and so forth.
3 Discussion We have discussed multi-index, multi-source information retrieval in the context of a scientific knowledge management service. We have presented the design-process stages and discussed them according to the Löwgren and Stolterman levels of abstraction. Types of users, documents and indexes were introduced, as well as their contemporaneous organization on the screen area based on the Focus+Context paradigm by means of frames and objects. We then discussed the techniques of mapping, degreeof-separation and closure and how these were used to provide comprehensive, customized information visualization for the JRC working environment, as well as advanced relational context between objects. Finally, we hinted at the technical functions supporting the system. Although the Löwgren and Stolterman approach has proven to be extremely useful, in fact the construction process starts at the moment technical issues are addressed, which can happen even at the earliest stages of the design process, and vice versa: technical issues reformulate the design process. In the three abstraction levels of their theory, we would add an additional one: construction. Multi-index systems have been used for a while in information retrieval, including advanced mapping techniques. However, little has been done to translate these information structures in adaptive interfaces providing accessibility and transparency to the information networks underlying multi-index integration. We propose an interface for accommodating multiple sources of different type simultaneously on the screen, which enables information discovery by visualizing contextual relations between objects, by degree of separation and closure. Acknowledgements. Carlo Ferigato contributed in many ways on SIRS, especially on the mapping specification. Daniela Panfili contributed to the interface design. Giovanni Salvagno has been an enormous technical resource. Giuseppe Merlo, in the JRC Central Library, provided general background in information science. Finally, we wish to thank Marc Wilikens, under whose responsibility this project was undertaken and financed at the JRC.
References 1. Mayr, P., Petras, V.: Cross-concordances: terminology mapping and its effectiveness for information retrieval. In: IFLA World Library and Information Congress. Québec (2008), http://www.ifla.org/IV/ifla74/papers/129-Mayr_Petras-en.pdf (accessed on March 12, 2009) 2. Löwgren, J., Stolterman, E.: Thoughtful Interaction Design: A Design Perspective on Information Technology. The MIT Press, Cambridge (2004)
Mapping for Multi-source Visualization: Scientific Information Retrieval Service
605
3. Eurovoc thesaurus, vol. 1 Permuted alphabetical version (parts A and B), vol. 2 Subjectoriented version. Office for Official Publications of the European Communities, Luxembourg (2007) 4. Dewey, M.: Dewey Decimal Classification and Relative Index, 21st edn. Forest Press, Albany (1996) 5. Spence, R.: Information Visualization: Design for Interaction, 2nd edn. Pearson/Prentice Hall, Harlow (2007) 6. Official development website for BS 8723, http://schemas.bs8723.org/ (accessed on March 12, 2009) 7. Thesaurus Tool Code, https://www.assembla.com/wiki/show/thesaurusToolCode/ (accessed on March 12, 2009) 8. Baeza-Yates, R., Ribeiro-Neto, B.: Modern information retrieval. Addison-Wesley, Reading (1999) 9. Doerr, M.: Semantic Problems of Thesaurus Mapping. Journal of Digital Information, vol. 1(8) (2001), http://journals.tdl.org/jodi/article/viewArticle/31/32 (accessed on March 12, 2009) 10. Ferigato, C., Merlo, G., Panfili, D., Rodighiero, D.: Role of Thesauri in a Scientific Organization. In: Networks of Design, Design History Society Conference 2008, Brown Walker Press, Boca Raton (2009) (in press) 11. Arms, W.Y.: Digital Libraries. The MIT Press, Cambridge (2000)
Client-Side Visualization of Internet Forums for Information Retrieval Guangfeng Song Penn State Greate Allegheny, 4000 University Dr, McKeesport PA 15132, USA [email protected]
Abstract. This paper presented a method to visualize Internet forums on the client side. Enhancement to web browsing was proposed to solve problems in information retrieval from forums. Specifically, visualization of structural information provided useful overviews for web browsing. A conceptual model described forums as three-dimensional spaces with information objects as points and web browsing as movements in the spaces. A series of diagrams were proposed to provide overviews for users' web browsing movements. Implementation of the visualization system and examples of the diagrams were presented in the paper. Keywords: Visualization, Internet Forums.
Client-Side Visualization of Internet Forums for Information Retrieval
1. 2. 3. 4. 5.
607
Lack of effective quality control in forums Difficult to search information in forums by timestamp Rigidity of topics organization in forums Inability to search for communication and collaboration Deficiency of search engines in searching forums
Much of the above problems can be blamed on the inadequate utilization of attributes of the forum objects. A rigid organization by topics is often the only visible forum structure to users. While other attributes such as authors, timestamps, and size of posts are displayed along the posts, these attributes are not used to organize forums. Nor can they be used to quickly browse the forums. Search engines are limited because they do not utilize these attributes either. Moreover, search results as long list of web pages destroys the familiar look of forums. Navigation of the long list of search result can be slower than following the structure of forums. The alternative as proposed in this paper is to enhance the usability of browsing forums by utilizing all attributes of forum objects. A well-known weakness of web browsing is the loss of location awareness which is commonly referred as lost in cyberspace [1]. This occurs because users do not have an overview of cyberspace which is hard to generate for websites of pages without well-defined structures. A generic network graph with identical nodes of web pages will not be a useful overview for users. However, Internet forums do have structures that enable more useful overviews. The spatial relationships among entities in a graphical overview will be easy for human to understand. More broadly, many information visualization techniques can be used to enhance the ability of users to browse forums.
2 Information Visualization Shneiderman provided a taxonomy of visualization tasks that has been widely adopted [2]. Visualization tasks can be classified into the following aspects of information presentation: overview of information, zooming, filtering, details on demand, relationship among items, history of actions, and extraction of subset. Despite the theoretic benefit of information visualization, experimental studies have been mixed on the benefit of visualization in reducing task completion time [3]. This discrepancy is likely due to complex interaction between task factors, user characteristics, and visualization techniques. A mismatch of these items may be the cause of the problems [4]. Therefore, successful visualization depends on appropriate use of visual primitives to represent information. These primitives include: length, area size, volume, relative location, shape, density, color, and others. Intrinsic meanings have emerged for some attributes have developed due to their consistent use in people’ real life experience. For example, larger size (length, area, or volume) represents larger value, darker density or color intensity refers to larger values, leftmost position corresponds to first item, and rightmost position corresponds to last item [5]. Colors are most often used in visualization to distinguish one element from another [6]. There’s no intrinsic meaning for color hues but the difference of colors can be easily perceived. Hence, contrast of colors is useful to distinguish groups while similar colors aid grouping.
608
G. Song
However, overuse of colors for grouping can lead to clutters and should be avoided [6]. A classification of many visual primitives can be found in [7]. There are a large variety of metaphors to present data with selected visual primitives. They range from simple diagrams such as simple bar charts, to hierarchical bar charts [8], and to metaphors of real world such as People Garden [9]. A classification of simple visualization metaphors can be found in [10]. Successful metaphors require good mapping between data attributes and the visual primitives in a setup that easily perceivable by users. Constant exploration of new metaphors is part of progress researchers are making in the field of information visualization. Visualization of Internet forums is relevant to visualization of social networks. Freeman reviewed the roles of visualization in analyzing social networks [11]. Researchers often concentrated on the network structure emerged from specific user communities of collaboration groups [9][12]. Visualization studies have also explored the temporal dynamics of communication flows in social networks. In [12], Flows of communication were generated from a mailing-list by mining email log files Graphs over a period of time are merged into movies to highlight the temporal aspects. The visualization is similar to the weather map in weather forecast. The emphasis is on the progressive changes in shape and connectivity of graphs. The use of animation is common practice in visualization of temporal attributes [10]. Multi-dimensional information is a natural result of using multiple attributes of objects in visualization. This is exactly the case with Internet forums. However, high dimensionality must be reduced before visualization is possible. The dimensions are usually reduced to two or three by multidimensional scaling methods. Dwyer and Gallagher visualized a high dimensional portfolio holding data set with attributes that change over time. Their visualization was achieved with dimension reduction [13]. Other studies in visualization of multidimensional data include “Table Lens” for data in a table [14] and WebTOC [15] for multiple attributes of all web pages in a website.
3 Methods to Visualize Internet Forums The model of internet forums is a mix of hierarchy and linear arrangement. The basic objects in a forum are posts made by members. Posts of a topic are arranged by time in a linear sequence. The topics or threads are also arranged by time in a linear sequence and they are placed in a branch of the hierarchy of forums. Therefore, Internet forums have the hierarchy of posts-threads-forums and the linear sequence of posts and threads. The information in a forum has many natural attributes. A post has its author, post time, length, and other attributes. In addition, many aspects are often necessary to describe the text content of a post. Hence, the interface of a forum has the difficult task of presenting complex information of multiple attributes to users. In particular, the interface has to effectively convey these attributes under the limitation of 2-d display. Unfortunately, the roles of these attributes in facilitating information retrieval from Internet forums are often ignored in current forum interfaces. Conceptually, members, content (members’ posts), and time are three major "dimensions" of Internet forums. With this notion of three dimensions, each forum post becomes a point in a three-dimensional space. Browsing a forum is therefore modeled
Client-Side Visualization of Internet Forums for Information Retrieval
609
as a sequence of movements inside the three-dimensional space. A move from one point to another in the space can be one of the three types: moves along one dimension while keeping other dimensions constant (Type I), moves within a twodimensional plane while keeping the third dimension constant (Type II), and moves that involve changes along all three dimensions (Type III). In Figure 1, B,C, and D are Type I moves while A is a Type II move. Type III moves can be simulated by combining Type I and Type II moves such as E in Fig. 1. Member
B
C D
A E
Content
Time Fig. 1. Dimensions in Internet Forums
A list of visualization methods are presented in Table 1. Each method is a visualization of either a Type I move or a Type II move. Type III moves are not directly visualized, however, because they would require 3-d diagrams. We intentionally limit the choice of diagrams to two-dimensional for several reasons: Computer interfaces are inherently two-dimensional; the value of 3-d visualization has not been convincing [16][17]; and Type III moves can be effectively simulated by these 2-d diagrams. These interfaces serve to uncover relationships among dimensions that help the search for relevant information by users.
4 Implementation of a Client-Side Visualization System Fig. 2 illustrates a client-side visualization system for Internet forums. The system comprised of a proxy server that serves three purposes when users are browsing a forum: relays information from the Web, displays available local visualization of the forum, and instructs the crawler to retrieve more information from the forum. The crawler receives definition of the forums and a list of web addresses to process. A forum definition file enables the crawler to understand the content and structure of web pages in the forum so that structured data can be retrieved into a database. The visualization engine create diagrams discussed in previous section as requested, using the database and the forum definition.
610
G. Song Table 1. Visualization of Type I and II Browsing Moves
Dimension 1
Dimension 2 Content
Content
Time
Member Member Time
Member
Specific Information
Visualization
Hierarchy of forums
V0
Threads by threads similarity
V1
Posts by posts similarity
V2
Threads by Time
V3
Posts by Time Similarity of posts by members Threads started by members Participation of members in threads Members to Members quote Members’ participation by time
V4 V5 V6 V7 V8 V9
Table 2. Diagrams to implement the visualization for browsing moves Visualization V0 V1 V2 V3, V4
Current Interface Text links N/A N/A Text lines
V5 V6, V7
N/A Text pages
V8 V9
N/A N/A
Visualization Diagram Treemap of forums and sub-forums 2-D Scatter plot with threads as nodes 2-D Scatter plot with posts as nodes Multiple lines of bar charts along timeline (sparklines []) 2-D Scatter plot with posts as nodes Correspondence diagram between nodes of members and threads 2-D Scatter plot with members as nodes Multiple lines of bar charts along timeline
Web
Web Browser
Craw List
Proxy
Crawler
Forum Definition
Visualization Engine
Databas
Fig. 2. Implementation of the visualization system on the client-side
Client-Side Visualization of Internet Forums for Information Retrieval
611
The proposed system was tested on a popular forum avsforum.com. A number of visualization diagrams were generated. Examples are shown below. Hierarchy of forums. This is a Treemap diagram that describes forums and sub-forums (V0 in Table 2). Categories of forums are distinguished by different colors. Each rectangle is a forum and the area size of the rectangle represents number of threads in the forum. Tooltips of forum information are displayed while the mouse is hovering over rectangles. A click of a rectangle bring users to list of threads along timeline (Fig. 3).
Fig. 3. Treemap diagram for hierarchy of forums
Similarity diagrams. These are 2-D scatter plots as shown in Fig. 4. This diagram is similar in appearance to "Topic Islands" in [18]. Each circle represents a post. The size of circles corresponds to the size of posts. Closeness of the circles corresponds to similarity of among the posts. Mouse hovering over a circle reveals information of the corresponding post. Clicking a circle bring users to the web page of the post. The similarity diagrams may be used to implement V1,V2,V5 in Table 2, when colors are used to represent different meanings.
Fig. 4. Scatter plot for similarity among posts
612
G. Song
Timeline diagrams. Two types of timeline diagrams were created: one for visualizing multiple threads along the timeline (V3 in Table 2, shown in Figure 5) and one for visualizing posts inside a thread (V4 in Table 2, shown in Figure 6). In both diagrams, the height of bars represents the size of the posts and the location of the bars represents the time when the posts were created. Colors were used to distinguish among members who made most posts within a thread. The timeline diagrams was also used to implement the posts made by a member over time (V9 in Table 2) when color represented different members. These diagrams are similar to Sparklines in providing overviews [19].
Fig. 5. Timeline diagram for posts in a thread
Fig. 6. Connected timeline diagram for posts in a thread by pages
5 Limitations and Conclusion The main limitation of the proposed visualization approach is the forum definition of semantics that must be performed manually. Although software tools have been developed to ease the process, it is still a limiting factor to visualize large number of Internet forums and changes in the forums may require new definition files. Another limitation is the potential large amount of data that has to be downloaded and stored locally for the visualization to work. Therefore, the visualization method in this paper may be most appropriate for users who only browse a few forums frequently. On the other hand, server-based visualization solves these problems nicely as the data and data structure is readily available on servers. But server-based visualization is less flexible and may significantly increase the burden on servers because all visualization has to be generated by servers. Both client and server-based approaches shall be pursued in future work. In conclusion, this paper presents a client-side visualization solution to the problem of information retrieval from Internet forums. Forums are modeled as threedimensional space and browsing forums is equivalent of moving in the space. The proposed visualizations enhance the ability of users to quickly browse for information in forums. The client-side implementation is a highly flexible system that can easily support any forums with any number of visualizations.
Client-Side Visualization of Internet Forums for Information Retrieval
613
References 1. Brake, D.: Lost in cyberspace. New Scientist 154(2088), 12–13 (1997) 2. Shneiderman, B.: The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of IEEE Symposium on Visual Languages, pp. 336–343 (1996) 3. Zhu, Y.: Measuring Effective Data Visualization. In: Advances in Visual Computing, pp. 652–661. Springer, Berlin (2007) 4. Sebrechts, M.M., Cugini, J.V., Laskowski, S.J., Vasilakis, J., Miller, M.S.: Visualization of search results: a comparative evaluation of text, 2D, and 3D interfaces. In: Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, Berkeley, California, United States, pp. 3–10 (1999) 5. Senay, H., Ignatius, E.: Rules and principles of scientific data visualization. State of the art in data visualization (ACM) SIGGRAPH Course Notes (27) (1990) 6. Healey, C.G.: Choosing effective colours for data visualization. In: Proceedings of the 7th conference on Visualization 1996, p. 263. IEEE Computer Society Press, San Francisco (1996) 7. Wünsche, B.: A survey, classification and analysis of perceptual concepts and their application for the effective visualisation of complex information. In: Proceedings of the 2004 Australasian symposium on Information Visualisation, vol. 35, pp. 17–24 (2004) 8. Keim, D.A., Hao, M.C., Dayal, U.: Hierarchical Pixel Bar Charts. IEEE Transactions on Visualization and Computer Graphics 8(3), 255–269 (2002) 9. Xiong, R., Donath, J.: PeopleGarden: creating data portraits for users. In: Proceedings of the 12th annual ACM symposium on User interface software and technology, Asheville, North Carolina, United States, pp. 37–44 (1999) 10. Senay, H., Ignatius, E.: A knowledge-based system for visualization design. Computer Graphics and Applications, IEEE 14(6), 36–47 (1994) 11. Freeman, L.C.: Visualizing Social Networks. Journal of Social Structure 1(1), 4 (2000) 12. Gloor, P.A., Zhao, Y.: TeCFlow-A Temporal Communication Flow Visualizer for Social Networks Analysis. In: CSCW 2004 Workshop on Social Networks (2004) 13. Dwyer, T., Gallagher, D.R.: Visualising changes in fund manager holdings in two and a half-dimensions. Information Visualization 3(4), 227–244 (2004) 14. Rao, R., Card, S.K.: The table lens: merging graphical and symbolic representations in an interactive focus + context visualization for tabular information. In: Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence, Boston, Massachusetts, United States, pp. 318–322 (1994) 15. Nation, D.A., Plaisant, C., Marchionini, G., Komlodi, A.: Visualizing websites using a hierarchical table of contents browser: WebTOC. In: Proceedings of the 3rd Conference on Human Factors and the Web (1997) 16. Risden, K., Czerwinski, M.P., Munzner, T., Cook, D.B.: An initial examination of ease of use for 2D and 3D information visualizations of Web content. Int. J. Hum.-Comput. Stud. 53(5), 695–714 (2000) 17. Cockburn, A., McKenzie, B.: 3D or not 3D?: evaluating the effect of the third dimension in a document management system. In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 434–441. ACM Press, New York (2001) 18. Miller, N.E., Wong, P.C., Brewster, M., Foote, H.: TOPIC ISLANDS - A Wavelet-Based Text Visualization System. In: Ebert, D., Hagen, H., Rushmeier, H. (eds.) IEEE Visualization 1998, pp. 189–196 (1998) 19. Tufte, E.R.: Beautiful Evidence. Graphics Press (2006)
Social-Technical Tools for Collaborative Sensemaking and Sketching James Sullivan1, Meredith Banasiak2, Christopher Messick3, and Raymond Rimey4 1 Department of Computer Science College of Architecture and Planning 3 Department of Electrical and Computer Engineering Center for Lifelong Learning and Design University of Colorado Boulder, Colorado USA 80309-0430 {James.Sullivan,Meredith.Banasiak, Christopher.Messick}@colorado.edu 4 Lockheed-Martin Corporation Denver, Colorado USA [email protected] 2
Abstract. Sensemaking is a deliberate effort to understand events or information, and a sketch is an exploratory graphic composition of a concept or observation. Within the architecture domain, sketching is employed during pre-design phases to create a shared understanding among clients and stakeholders. While sensemaking is highly collaborative, sketching is usually a solitary activity. This paper describes the design and evaluation of two prototype social-technical tools to support collaborative “same time, same place” sketching and sensemaking: (1) a software environment (SketchBook) that allows users to quickly generate and capture ideas; and (2) a wireless, scalable, multi-user pen interface (FireFly). When used together, these tools support simultaneous sketching, diagramming, and annotation within the same work space without traditional bottlenecks of “turn taking” by passing a single pen. This paper presents the motivation for sensemaking and sketching, and findings from a preliminary evaluation involving a design charrette with architecture students. Keywords: design, sketching, architecture, collaborative interfaces, sensemaking.
Social-Technical Tools for Collaborative Sensemaking and Sketching
615
Sketching is a useful and powerful construct during sensemaking activities because it empowers individuals to collaboratively express and share concepts through informal drawing representations. A sketch also serves as a powerful tool of both thought and communication [5]. Sensemaking and sketching are used in a variety of domains: television sportscasters often sketch over “video replays” to help audiences understand what happened in an athletic competition or to describe nuances of team tactics that might otherwise go unnoticed; weather forecasters sketch on maps to explain how changing meteorological data could impact a geographical region. Law enforcement and military staffs likewise sketch on maps to collaborate and create a shared understanding of what has happened in assigned areas, summarize activities, and develop courses of action. 1.1 Sketching, Sensemaking and Design in Architecture Pre-design Architects use sketching to support sensemaking during pre-design activities as they explore and document issues, constraints and goals with client and stakeholders [6]. During the pre-design phase, architects iteratively work to synthesize information from clients and stakeholders and create meaningful spatial and conceptual relationships through sketches [7]. Pre-design processes. The pre-design phase is characterized by the following collaborative processes: • diagrammatic reasoning through sketching: informal representations are generated to facilitate flexible thinking and rapid conceptualization of abstract and global specifications. • generative brainstorming: there is a rapid production of multiple usage scenarios. • collaboration and social learning: there is an exchange of ideas and information between stakeholders, clients and designers. • record of salient information: values, priorities, goals, and design rationale are captured through a process of imaging and visualization. Architects use sketching to support sensemaking during predesign activities as they uncover and document client and stakeholder issues [6]. Through the sketches, designers create external artifacts to support a shared understanding about the problem space between themselves, clients and stakeholders [7]. According to Schön’s notion of “reflection-in-action”, professional practice is an interleaved process of thought and action, and sketching allows a designer to “have a reflective conversation with the materials of a design situation” [8]. By generating (sketching) and interpreting their representations, the designer(s) may change their view of the current design. This is a powerful duality, and it raises the question of what might be possible if design communities (e.g. designers, clients, stakeholders, etc.) could have an interface to easily interact in the sketching process. Design charrettes. An architectural practice for engaging in sensemaking activities in a democratic process with all constituency groups is a design charrette [9, 10, 11]. During a design charrette, stakeholders and designers come together to participate in an intensive workshop for the purpose of creating consensus about the vision and direction for a design process by creating graphic documentation of their ideas and
616
J. Sullivan et al.
explorations. A charrette is participatory in nature; all parties share their unique perspectives and information about project goals. 1.2 Other Computational Environments for Sketching Several computational sketching approaches have previously been developed and investigated including: (1) personal sketching and modeling environments, such as SketchUp [12] and Skencil [13]; and (2) on-line, multi-user sketching environments, such as iSketch [14] and SwarmSketch [15]. Other approaches for supporting collaborative sketching include computational whiteboard designs such as multi-touch table top environments [16], and electronic multi-touch whiteboards [17]. Personal sketching environments present collaborative limitations similar to those found in other “personal” applications, such as MS Word. When using “personal applications”, participation takes place in a solitary, parallel modality and products must be “merged” into a final product. This methodology is not supportive of the highly fluid and collaborative workflows used in fields such architecture. On-line, multi-user sketching environments (similar to SwarmSketch) may also be considered for “same time, same place” collaborations. However, when these systems are used in an on-line “groupware” modality, they may create problems with participant focus and communication as co-located collaborators attend to their personal computer while listening to others who are sketching, drawing, or presenting ideas on a shared space. 1.3 Research Focus Given the previous approaches toward on-line and personal sketching environments, our research focused on the design of computational sketching tools that support the creation of a shared understanding among co-located collaborators engaged in sensemaking activities. “Same time, same place” collaborations are very desirable for situations involving ambiguity and incomplete information since participants engage in face to face communications while presenting ideas. Within the architecture domain, analog “whiteboards” are often used to capture and organize collaborator concepts, sketches, photos and other related information about the project at hand. Computational whiteboard and tabletop systems share many affordances of analog systems – as well as limitations: • accessibility issues: when computational whiteboard and tabletop systems are used as collaboration environments, all output “real estate” must be visible and physically accessible to all participants. This creates obvious problems for people with physical or mobility disabilities who would desire to annotate, highlight or sketch to express their ideas. It also presents problems if there is a single user input device since a facilitator is needed to interpret and represent ideas on the system. • proximity issues: as with traditional analog whiteboards, digital whiteboard systems [Perceptive Pixel 2008] typically require that users work in close proximity to a projected image space. If the projected image is large, it may be difficult for the sketcher to obtain a holistic view of the entire design space, and the image can become hopelessly obscured if several people are interacting in front of a digital whiteboard.
Social-Technical Tools for Collaborative Sensemaking and Sketching
617
D
Key A – multiple FireFly pens, each with a light emitting diode (LED), on-board sensors, and wireless digital data communications with computer B – standard video camera to transmit video images to computer for tracking pen positions C – computer running FireFly interface software and SketchBook environment with wireless digital data communications to pen and data communications to video camera D – computer or projection screen display output from FireFly pens.
Fig. 1. Overview of FireFly and SketchBook architecture
2 System Overview Two key components were designed to support collaborative sketching and sensemaking: • a software environment that allows users to quickly generate and capture ideas (SketchBook), and • a wireless, scalable, pen interface (FireFly) that allows multiple users to simultaneously sketch, diagram, and annotate within the sketching environment (SketchBook). When used together, these tools support simultaneous sketching, diagramming, and annotation within the same work space without traditional bottlenecks of “turn taking” by passing a single pen (Figure 1). The software environment (SketchBook) and interface were designed as independent components to support comparative evaluations between different software environments (SketchBook vs. SketchUp, etc.), as well as different interface treatments (SketchBook with FireFly, tablet stylus, or mouse interface).
618
J. Sullivan et al.
Fig. 2. Exterior view of current FireFly prototype with light emitting diode, power and momentary switches. CPU, power supply, and wireless communication components fit into a dry marker package and a user “draws” by pressing the momentary switch while moving the pen.
2.1 Description of the Prototype Firefly Multi-user Interface FireFly is a wireless “smart pen” with a light emitting diode (LED) on the pen tip. Each FireFly pen communicates with the SketchBook environment via two communication channels: • optical tracking: A standard VGA video camera optically tracks multiple FireFly pens as users draw in a virtual space. Using the camera video image (see Figure 1), movements of FireFly pens are mapped to the SketchBook drawing space. • digital commands: each FireFly pen contains a CPU, switch sensors, and wireless communication technologies to detect and send commands to the SketchBook. Current pen commands include detecting “mouse down” and “mouse up” using a small momentary switch (see Figure 2) and recording personalized “gestures” that can be mapped to application menu commands. FireFly users sketch by moving the pen in space (A in Figure 1), while a video camera (B in Figure 1) located above the computer screen optically tracks each pen position. A standard VGA video camera with manual exposure controls (aperture, contrast, and shutter speed) provides a flexible way to detect the LED on a FireFly pen tip in a wide range of ambient room lighting conditions. Since each FireFly pen transmits a unique initialization “handshake” upon power up, multiple pens can be detected and simultaneously tracked using the video image data. When a FireFly user wants to draw “ink” in the SketchBook environment, the sketcher presses a small momentary switch, indicating a “mouse down” command. Since each FireFly pen contains a CPU to process on-board sensor data, the pen could also detect subtle user gestures that may not be visible to the video camera, such as pressure changes, or rotational changes. Once detected, pen commands are interpreted and wirelessly transmitted the computer running the SketchBook environment. Since each pen is tracked and analyzed in its own software thread, one FireFly pen can be drawing, while another is making a menu selection (changing line color, undo, etc.).
Social-Technical Tools for Collaborative Sensemaking and Sketching
619
2.2 Description of the Multi-user SketchBook Environment SketchBook, a multi-user sketching environment, currently supports two FireFly pens and a mouse, allowing multiple users to simultaneously engage in free-form sketching, annotation, and diagrammatic reasoning (Figure 3). SketchBook provides a minimal menu set to present a simple and informal “paper and pencil” interface motif. Menus choices include line color and thickness, as well as “clear”, “undo” and “erase”. Whenever a user performs a potentially destructive operation (e.g. “clear”, “undo” or “erase”), a “thumbnail sketch” appears to the right of the collaborative sketch space to preserve ideas that were under consideration (Figure 3). In this way, SketchBook allows users to quickly get their ideas out without the worry of losing previous work, and without the overhead of naming and saving files.
concept thumbnails
menus
collaborative sketch space
scroll controls
Fig. 3. SketchBook, a multi-user sketching environment
Fig. 4. A student design team sketching in a pilot design charrette [Lennertz et. al 2008] with FireFly and Sketchbook prototypes on a large display screen
620
J. Sullivan et al.
The SketchBook environment runs on a PC, Tablet PC, Macintosh, and Linux. The environment supports simultaneously drawing on a scalable range of output displays including notebook, large screen displays or projected images (see Figure 4). In addition to the prototype FireFly interface, the SketchBook application can also be used with a PC mouse, stylus pad, or track pad. This design permits evaluations of the software environment with a variety of interface treatments.
3 Evaluation Studies Evaluation studies were conducted to (1) understand how the prototype Sketchbook application and FireFly interface support collaborative sensemaking and pre-design processes within architecture design groups; and (2) identify potential improvements for the prototype application and interface. User studies were situated within a “design charrette”. The design charrette started with an open-ended design problem requiring group ideation, brainstorming, and establishment of design goals. Once a goal was established, the group created conceptual designs and prepared a presentation to communicate key design concepts and design rationale. Following group presentations, each participant completed an anonymous survey providing feedback about (1) the overall design experience and (2) the tools used. Data from presentation outputs and surveys were aggregated and analyzed to identify refinements in the technologies and/or the design problem. 3.1 Evaluation Task Twenty-one architecture student volunteers were divided into design teams of five or six members in a design studio style exercise [18]. Each team was provided background materials about a real-world design task involving a large studio space in a campus environmental design building (Figure 4). Teams could choose between focusing on macro-level issues (layout of the overall studio) or micro-level features (team work areas and desk space design). Each team was instructed to create a conceptual design and prepare a group presentation summarizing their findings and design recommendations. Each group had a laptop computer as well as paper and pencil. Students were instructed to focus on the design task, not critique the prototype tool. One member of each group was randomly designated as a team leader as responsible for overall time management and group participation, while another acted an internal “observer” and took notes anytime the team either reached an impasse or had a problem with the prototype tools. Since one large screen and two prototype FireFly pens were available, teams rotated between the following interface treatments and software: (1) the FireFly and SketchBook application on a large screen; (2) SketchBook on a tablet PC and a stylus; and (3) SketchBook on laptops with a mouse or track pad. Teams were also permitted to use paper and pencil to represent any conceptual sketches, diagrams, or annotations that were difficult to represent using the assigned tools.
Social-Technical Tools for Collaborative Sensemaking and Sketching
621
Fig. 5a (left) and 5b (right). Groups were provided a real world design task and asked create concepts focusing on macro-level floor plan design considerations or micro-level workspace issues.
3.2 Evaluation Data Complementary quantitative and qualitative data were obtained through final team presentations, sketch artifacts, post-task individual surveys, and observer notes. These data provided multiple ways to triangulate on system strengths, weaknesses, and potential improvements through collective and individual assessments. 3.3 Key Findings Concept sketches were analyzed to assess high level goals, and how well the FireFly interface and SketchBook application allowed users to draw and communicate their designs. User surveys also illuminated what interfaces were preferred, features subjects found most useful and usable, and recommendations for improvements. Support for high level goals and expressiveness. All groups elected to use the SketchBook application for sketching ideas during the design phases. Figure 5a highlights a design that focused on “macro-level issues” including floor space layout, exterior wall, and lighting issues, while Figure 5b illustrates a design created by a group that elected to focus on “micro-level” design issues including workspace, desk, and group cubicles. Application interfaces used. All four groups elected to use the FireFly interface to make design presentations, and all groups extensively used the “snapshot” functionality in SketchBook to present issues considered and the evolution of their designs. Two groups used the FireFly interface to sketch during their presentation and one also used a mouse interface as a pointer. One group elected to use the mouse as the primary interface, and used the FireFly interface as an augmentation to highlight key design features. Usability and usefulness. In post-task surveys, users were asked to rate the SketchBook applications with different interfaces (FireFly, tablet stylus, and mouse) vs. paper and pencil, a traditional sketching tool. In rating the usefulness in sketching and representing key aspects of the design, the SketchBook application with a tablet interface was rated as a close second to paper and pencil (9.0 vs. 9.2 on a subjective
622
J. Sullivan et al.
scale of 1 to 10, with 1=poor and 10=excellent). The mouse/track pad and FireFly interfaces were a distant third and fourth (5.8 and 4.1 respectively). In rating usability to represent key aspects of design, the SketchBook application with a tablet was again rated as just below pencil and paper (8.7 vs. 9.2 respectively), followed by mouse or track pad, and FireFly (5.4 and 3.8 respectively). Overall impressions. Users were asked to provide overall ratings about the SketchBook application and FireFly interface using the same 1 to 10 subjective interface. The most highly rated features in SketchBook included the ability to import external pictures (9.9), simplicity (9.4), and the ability to keep track of design revisions and changes (9.1). These ratings were followed by the ability to share ideas (8.7) and flexibility (7.8). The FireFly interface was most highly rated for its usefulness in presenting ideas (7.7), ability to change drawing colors/thickness (7.5), simplicity (6.8), and usefulness to simultaneously sketch with more than one person (6.4); less well rated was the ability to annotate and draw (3.9 and 3.0 respectively).
4 Summary and Future Work These preliminary studies illuminate both the potential and shortcomings of this prototype application and interface. While users exploited the simple yet powerful collaborative dimensions of the SketchBook application, the tracking resolution of FireFly prototype is currently not sufficiently accurate for creating detailed free hand sketching and annotations. Despite these shortcomings, it is remarkable that the FireFly prototype pens were selected by all teams during design presentations. When making presentations, members interactively “tag teamed” between speaking and highlighting sketch design features, often while another member was speaking. This type of collaborative presentation is not usually seen when there is a single speaker and mouse interface. Future plans for the FireFly interface will explore the use of high resolution tracking cameras and smoothing algorithms to improve drawing performance. We will also explore the use of advanced pen sensors to support the detection, interpretation, and mapping of user-defined gestures to application commands, a strategy for enabling additional application functionality, while preserving the highly rated simplicity of the SketchBook interface and supporting meta-design [19]. In response to survey suggestions, we will also consider extending the SketchBook application to support import and export of a broader range of graphics files, allowing the rough sketches created in the SketchBook application to be further used and refined in other drawing applications. In summary, multi-user environments offer the promise of greater collaboration and participation in domains where barriers to participation are restricted by system models that assume a single user. Sketching is a powerful and natural activity that occurs when people gather to communicate concepts and ideas, and computational systems for sketching and collaboration should likewise be designed to support simultaneous parallel communications and exchange of ideas. While additional work is needed to improve the tracking resolution of the FireFly wireless pen interface, this research provides a proof-of-concept that a multi-user interface and application can support creative group collaborations.
Social-Technical Tools for Collaborative Sensemaking and Sketching
623
References 1. Dervin, B.: Information as a User Construct: The Relevance of Perceived Information Needs to Synthesis and Interpretation. In: Glazier, J.D., Powell, R.R. (eds.) Qualitative Research in Information Management, Englewood, CO, Libraries Unlimited, pp. 61–84 (1983) 2. Weick, K.: Sensemaking in Organizations. Sage Publications, Thousand Oaks (1995) 3. Weick, K.E., Sutcliffe, K.M.: Organizing and the Process of Sensemaking. Organization Science 16(4), 409–421 (2005) 4. Hutton, R., Klein, G., Wiggins, S.: Designing for Sensemaking: a Macro-cognitive Approach. In: CHI 2008 Sensemaking Workshop, Florence, IT, April 6 (2008), http://dmrussell.googlepages.com/sensemakingworkshoppapers 5. Lawson, B.: How Designers Think: The Design Process Demystified. Architectural Press, Oxford (2007) 6. Pena, W.: Problem Seeking. Wiley, Hoboken (2001) 7. Zeisel, J.: Inquiry by Design. W.W. Norton, New York (2005) 8. Schön, D.A.: The Reflective Practitioner: How Professionals Think in Action. Temple Smith, London (1983) 9. Kohn, S.: Experiment in Planning an Urban High School: The Baltimore Charrette. Educational Facilities Laboratories, New York (1969) 10. Lennertz, B., Lutzenhiser, A., Failor, T.: An Introduction to Charrettes. National Charrette Institute, http://www.charretteinstitute.org/resources/files/ charrettes_pcj_article.pdf 11. National Charrette Institute (NCI) website, http://www.charretteinstitute.org 12. SketchUp website, http://www.sketchup.com 13. Skencil website, http://www.skencil.org 14. iSketch website, http://www.isketch.net 15. Swarmsketch website, http://www.swarmsketch.com 16. Mitsubishi Electronic Research Laboratory (MERL) UbiTable: http://www.merl.com/projects/UbiTable 17. Perceptive Pixel website, http://www.perceptivepixel.com 18. Schön, D.A.: The Design Studio: An Exploration of its Traditions and Potential. RIBA Publications, London (1985) 19. Giaccardi, E., Fischer, G.: Creativity and Evolution: A Metadesign Perspective. Digital Creativity 19(1), 19–32 (2008)
Developing Some User Interfaces of TV under Enormous Channels Environment Shumpei Tamaoki, Tomohiro Torikai, and Hirohiko Mori Musashi Institute of Technology 28-1, Tamadutsumi 1-chome, Setagaya-ku, Tokyo, 158-8557, Japan [email protected], [email protected], [email protected]
Abstract. Multi-channel digital broadcasting is very popular among people and offers us enormous programs and the increase of the number of the channels affects our behavior in watching TV. In the case of target-oriented watching, we have already some solutions. However, in the case of non-target watching, it is getting for us difficult to find some interesting programs in a relaxed attitude. In this study, we focus on the issue of EPG and we propose the improved systems to solve it. These systems facilitate users to shortening of program search time and grasping of program contents in the relaxed attitude. From the results of the experiment to evaluate them, we showed shortening of program search time. Keywords: Home Appliance, TV, Enormous Channels, Non-target Watching, EPG, Thumbnail.
Developing Some User Interfaces of TV under Enormous Channels Environment
625
targets in advance. The purpose of this study, therefore, is to develop systems which can improve the non-target watching under enormous channels environment.
2 Background In the non-target watching, we assumed that there are two approaches for users to find interesting programs. One is the channel surfing [2] where user browses TV programs in the sequence of the channel number and when they find a program interesting after zapping many programs they go back to it. The other is using EPG (Electronic Program Guide). In this behavior, they repeat searching programs on the EPG and trying a program when they find a seemingly interesting one until they find the interesting one. 2.1 Previous Work In the initial stage of this research, we focused on the channel surfing [2]. The result of the preliminary experiment showed that it is hard to return to the program which users were interested in. To solve this problem, we designed a prototype system. In this prototype system, we install the “bookmark button” to the wireless remote controller, and the user can manually bookmark the interesting programs. By doing so, we aimed that the user facilitate to go back to the channel easier and it would become facile to redisplay the interesting programs like an internet browser. In the experimental result, however, most of the user in relax attitudes did not make the bookmark on their interesting programs actively. It was observed that user could not decide whether a tying program is interesting or not among all programs at that time and they search more interesting programs by the short glance of many more programs, and, finally, they attempt to go back to the interesting one. This means the different kinds of the user interfaces from the target-oriented watching are required in relaxing time. Therefore, we redesigned the second prototype system. In this system, the interesting programs are automatically bookmarked and bookmarked programs are visualized by utilizing the Fish Eye View. This function is contributing the user to make easy to go back to the interesting programs in the relaxed attitude. Moreover, the improved prototype system displaying the thumbnail on the Fish Eye View buttons improved more. From these results, in the non-target watching, we showed that it is very effective to visualize information to return to the interesting programs and time of watching a program during program surfing can be used as a estimating the degree of interesting. 2.2 EPG There are the masking of the TV screen and extra information as problems of the EPG. Therefore search time increases, and it is difficult for the content of the program to grasp it. The user interface of the program search function (input keyword, category and performer) is a design of target-oriented watching in addition to their problems. Furthermore, there are the findings that TV Guide magazine is used than EPG.
626
S. Tamaoki, T. Torikai, and H. Mori
3 Evaluation of Epg's Display Position and Operation Method In the non-target watching, we assumed that the hiding the TV program streams caused by changing the EPG display affects us adversely. 3.1 Experiment 1 To solve the masking of the TV screen, we evaluate two types of EPG’s display position (Figure 1 and 2).
Fig. 1. Display position A Switch the TV screen, and display EPG
Fig. 2. Display position B Display EPG with the other screen (the tablet PC at hand)
3.1.1 Method We conducted the experiment according to the following procedure in enormous channels environment (50ch). In addition, five subjects were involved in this experiment and they were asked to perform the task in the relax attitude. 1. The subject performs browsing and zapping the EPG freely and chooses five programs which wants to watch most. At the same time, we measure program search time and the number of the programs which subject watched. 2. The subject conducts zapping it from 1 to 50ch and chooses five programs which wants to watch most. 3. We investigation how many programs which I chose by procedure 2 accord with procedure 1. 3.1.2 Results Table 1 indicates measurement data of experiment 1. In the result of Table 1, we compared B with A and indicated that average program search time was short though there was much number of the average watched programs. The congruent numbers of programs did not show significant difference in A and B. Moreover, all subjects answered that it was easy to use B.
Developing Some User Interfaces of TV under Enormous Channels Environment
627
Table 1. Measurement data of EPG’s display position Display Position A B
Number of Average Watched Programs (ch) 15.2 18.8
Average Program Search Time (sec) 661 546
Congruent Numbers of Programs 2.6 2.8
3.2 Experiment 2 In the experiment 1 about the EPG’s display positions, it is thought that different types of the display manners for EPG are suitable. Therefore, in the experiment 2, we display EPG to the other screen and evaluate an operation method by two patterns (figure 3 and 4). In addition, the experiment method is the same as experiment 1.
Fig. 3. Operation method C
Indirect operation that it used a wireless remote controller for
Fig. 4. Operation method D
Direct operation that it used a pen touch for
3.2.1 Results Table 2 indicates measurement data of experiment 2. In the result of Table 2, we compared D with C and indicated that program search time was short. Congruent numbers of programs did not show significant difference in C and D. Moreover, four subjects answered that it was easy to use D. Table 2. Measurement data of EPG’s operation method Operation Method C D
Number of Average Watched Programs (ch) 18.8 15.8
Average Program Search Time (sec) 546 426
Congruent Numbers of Programs 2.8 2.6
3.3 Considerations In the experiment 1 and 2, we found shortening of program search time by display EPG to the other screen and direct operation method. However, there was no
628
S. Tamaoki, T. Torikai, and H. Mori
substantial change in congruent numbers of programs. It can be considered to be happened by following two problems (Alpha and Beta). • Alpha: Difficult to grasp the contents of the program by caption text of the EPG, and cannot narrow down the program which I want to watch. • Beta: Difficult to check state (whether you watch) of the program during zapping and cannot narrow down the program which I want to watch. Therefore, to solve these problems, we design improved systems and conducted the experiment.
4 Improved Systems We designed improved systems based on the result of experiment 1 and 2. 4.1 The First Improved System Figure 5 is the first improved system. We aimed to facile narrowing down a program. Including display EPG to the other screen and direct operation method, to solve alpha problem, there are two features in this system. One is to add the thumbnail of the program to the text information (program name) of EPG's screen. The other is displaying on EPG only the program which can be broadcast by “3x3”. It is because the program which is not broadcast is unnecessary in non-target watching.
Fig. 5. First improved system
4.1.1 Experiment 3 To evaluate the first improved system, we investigate whether subject can grasp the contents of the program by three kinds of display methods (Figure 6, 7 and 8) precisely. In addition, the experiment method is the same as experiment 1.
Developing Some User Interfaces of TV under Enormous Channels Environment
Fig. 6.Only text
Fig. 7. Only thumbnail
629
Fig. 8. Text and thumbnail
4.1.2 Results Table 3 indicates measurement data of experiment 3. Table 3 shows that the condition with only texts and with thumbnail the search time of the programs are the shortest. However, when only a thumbnail was displayed, there are most congruent numbers of programs. Moreover, it did not show significant difference about each item. Table 3. Measurement data of first improved system
Display Method Only Text Only Thumbnail Text and Thumbnail
Number of Average Watched Programs (ch) 13.7 12.0 11.7
Average Program Search Time (sec) 394 375 374
Congruent Numbers of Programs 2.5 3.1 3.0
4.2 The Second Improved System To solve beta problem, we designed the following two display manners. • Figure 9: In this system, the user can switch the three types of lists: all channels, the programs which have watched the programs which do not have watched already. • Figure 10: This system changes the background color of the program title which the user has watched. Here, the background color of the current watching program is red and the ones of watched programs are red. We aimed, in these designs, to allow the users easily to find which programs have been watched and which ones does not have done. 4.2.1 Experiment 4 To evaluate the second improved system of two types, we investigate whether subject can grasp the contents of the program precisely. In addition, the experiment method is the same as experiment 1.
630
S. Tamaoki, T. Torikai, and H. Mori
Fig. 9. Second improved system
Fig. 10. Second improved system
(Change type)
(Color type)
4.2.2 Results Table 4 indicates measurement data of experiment 4. Table 4 indicates that the color type display was the shortest in the program search time. However, there was no substantial change in congruent numbers of programs. Table 4. Measurement data of second improved system
Display Method Second Improved System (Change Type) Second Improved System (Color Type)
Number of Average Watched Programs (ch)
Average Program Search Time (sec)
Congruent Numbers of Programs
12.7
436
2.7
13.4
340
2.5
4.3 Consideration Two Systems In the result of experiment 3 and 4, we could shorten program search time but there was no substantial change in congruent numbers of programs. At first, to browse the EPG that increased a thumbnail, we assumed that narrow down the program which we wanted to watch. However, as a result of having investigated non-congruent numbers of programs, it was the program which we did not watch. This means that such kinds of EPG systems do not work for the users to find the programs which they wanted to watch, but they are useful to eliminate the programs which do not want to watch from their focuses.
5 Conclusion In this paper, firstly, to solve increase of the program search time by the masking of the TV screen, we investigated display position and operation method of the EPG. In
Developing Some User Interfaces of TV under Enormous Channels Environment
631
the results, preparing independent display device for EPG and the direct manipulation of the target program on the EPG shorten the program search time. However, because it is difficult to grasp of the whole contents of the program, we developed the two types of the user interfaces for EPG that it was easy to recognize in a relaxed attitude, facilitating for the user shortening of program search time and grasping of program contents in enormous channels environment. Though these systems work to shorten program search time, there was no substantial change in congruent numbers of programs. These results was caused why such kinds of EPG systems do not work for the users to find the programs which they wanted to watch, but they are useful to eliminate the programs which do not want to watch from their focuses.
References 1. Shigeru, E.: Research on the relationship between affinity for television and television viewing behavior. The Japanese Society of Social Psychology 22(3), 267–273 (2007) 2. Shumpei, T., Hirohiko, M.: Developing User Interface for Watching TV under Enormous Channels Environment. In: IADIS International Conference IHCI 2008 (part of MCCSIS 2008), Japan, pp. 339–343 (2008) 3. Tomoyuki, Y.: Recognizing Actions for Automatically Acquiring Personal Preferences from TV Viewer’s Behaviors. The Institute of Electronics, Information and Communication Engineers 103(585), 65–70 (2004) 4. Yasushi, T.: Attitude and Sense-of-value in Watching Television and Requirements of Human Interface. Toward Human Content Interface Design. Japanese Society for the Science of Design 45(2), 63–72 (1998) 5. Yoshiharu, F.: Frequency-based Channel Repertoire and Information Orientation in the Multi-channel Environment. The Japan Society for Studies in Journalism and Mass Communication (64), 135–149 (2004)
Electronic Glassboard – Conception and Implementation of an Interactive Tele-presence Application Peter Thies1 and Benjamin Koehne2 1
Abstract. This work presents a conception of a novel tele-presence system with an integrated interactive component. Previous solutions in this research area mainly focus on communication and do not offer sufficient, intuitive cooperation support for distributed meeting members. The Electronic Glassboard fills this gap by integrating the video display and drawing area. This allows for cooperative sketching without losing direct eye-contact with the cooperation partner. Keywords: tele-presence, CSCW, EMS, electronic meeting support, transparency, sketching, distributed conferencing, videoconferencing.
1 Introduction Collaboration of distributed partners gains in importance in the economy. Communities of practice span around the globe and The Global Village [1] has already become reality. Extensive travel, combined with resource management (direct travel expenses, travel time), is necessary in order to maintain and support such dynamic networks. Lost work time caused by business trips represents an immense cost factor for companies in every industry sector. In order to reduce such costs, and serendipitously to have a positive effect on climate change, more and more solutions for distributed collaboration are coming into operation.
Electronic Glassboard – Conception and Implementation
633
Using the 3C–classification [3], in the context of Computer-Supported Cooperative Work (CSCW), groupware systems can be classified into 4 system classes with differential emphases on communication-, coordination- or cooperation-tasks. These system classes are communication systems, shared information spaces, workflowmanagement systems and workgroup computing. Following the space-time matrix [4], the class of communication systems can be structured into synchronous vs. asynchronous and collocated vs. distributed applications. In this context, Nunamaker et al. [5] shaped the term of the electronic meeting system. Bly et al. [6] and Streitz et al. [7] similarly framed the development of their respective systems. The Colab meeting room system [8] has been the basis for many studies in this area (e.g. [9]). Videoconferencing systems are synchronous in nature and have long been used by organizations in distributed situations. In the early years of videoconferencing, research work focused mainly on broadband difficulties in connection with transporting audio- and video- signals. Researchers were interested in dedicated transport channels such as ISDN-lines. However, with the massive increase in bandwidth available via the Internet and its increasing ubiquity in modern society, videoconferencing has become an increasingly popular service for individuals. The range of products reaches from simple systems, such as Skype1 and Microsoft Live Meeting2, to highly sophisticated systems with dedicated high-speed communication links and custom network hardware and software.
As videoconferencing systems have become more sophisticated in recent years, a new class of commodity systems has emerged in the market with the goal to optimize the quality of communication. Formerly reduced portraits of the communication partner evolved to life-size, high quality representations. These systems try to create the impression that distributed workgroups all appear to be in the in the same room: they appear to be tele-present. This impression is further enhanced by a specific room layout. The experience level improves in many ways. Perceived qualitative factors, such as facial expressions and gestures, lead to an improved awareness state. At present, the two most advanced products in this context 1 2
http://www.skype.com, access on: 02/25/2009. http://office.microsoft.com/en-us/livemeeting/default.aspx, access on: 02/25/2009.
634
P. Thies and B. Koehne
are the immersive solutions CISCO TelePresence3 and Hewlett Packard Halo4 (see Fig. 1). The use of high-quality components comes at a price. These solutions quite frequently require the planning of investments up to the medial 6-digit Euro range (depending on the number of users). Although the operation itself generates additional costs for dedicated data channels providing guaranteed quality of service5, many of the Fortune 500 companies use these products today to reduce both travel expenses and loss of work time accordingly. 2.1 Cooperation and Tele-presence Systems Systems available on the market today mostly focus on the communication between meeting participants. These systems offer similar solutions in terms of cooperation of the participants (see [7]). On the one hand, document cameras are applied to transport picture information of non-digitizable objects, like construction parts for example. Alongside, individual laptops can be connected with a conventional display cable in order to present screen content to all participants. Current systems also use an additional display to present picture information. The HP Halo system, for example, mounts a screen showing the digitized content above the displays showing the communication partner (see Fig. 1a, upper center image). The CISCO TelePresence 3000 system, similarly arranges an additional screen below the participant’s displays (see Fig. 1, b), center, below desk board). As the systems discussed so far are mainly focused on communication, both implementations lack advanced interaction support. Furthermore, the participants are not aware of each other’s state during their interaction with shared content. At this stage of development, we present the Electronic Glassboard approach, which we will explain in further detail throughout the course of this paper. 2.2 Room Layout for Tele-presence Applications Lewe [10] describes numerous parameters for the layout of electronic meeting rooms. His analysis concentrates on collocated meetings and evaluates their influence on the productivity of teams. He presents a series of design alternatives for the layout of such rooms ([10], p. 216 et sqq.). Amongst all layouts described by Lewe [10], the room layout “Classic Double” is the most significant model for tele-presence solutions. The “Classic Double” (see Fig. 2 a) emanates from a group located between two boards or interaction surfaces i1, i2 (i.e. whiteboard or flipchart) respectively. A conference table is arranged in the center. It supports separating two groups whose persons p can communicate with each other face-to-face. The interactive surfaces i1, i2 can be used for presenting digital content. Given a distributed context, the room shown in Fig. 2 a) can be seen as two single rooms, separated along line l (see Fig. 2 b). The setup of the systems Hewlett Packard Halo and CISCO Telepresence, discussed earlier, overcome this separation by placing 3
http://www.cisco.com/en/US/products/ps7060/index.html, access on 02/25/2009. http://www.hp.com/halo, access on 02/25/2009. 5 See also: IEEE Workshop on Quality of Service, http://iwqos07.ece.ucdavis.edu/, access on 02/25/2009. 4
Electronic Glassboard – Conception and Implementation
635
Fig. 2. Room layout, top view
large LCD displays along line l. Both of these systems use the interaction surfaces only for presentation functions and also place them near the separation line l (see chapter 2.1). Hence, the remote team could follow deictic gestures (e.g. pointing to a certain part of a presentation), making the arrangement shown in Fig. 2 b) beneficial to interaction. However, this room arrangement is not the best choice for the systems introduced earlier. On the one hand, for the viewer, the image caption is not big enough in order to record content situated behind the remote person. On the other hand, this arrangement would demand a constant re-orientation of the people in the room (constant turning of the body of approx. 180° around the vertical axis) from the presentation surface i1 or i2 respectively to the displays showing the communication partners v1, v2 and vice versa. Additionally, a camera with a very high depth of field would be necessary for focusing on both the participants sitting in the foreground and the illustration on the interaction surface situated behind them. Another challenge is to normalize the audio signal volume for both persons sitting at the table and, if applicable, for the persons standing close to the interaction surface. Therefore, looking at the most prominent tele-presence solutions on the market today, a realization of an interaction surface being separate from the participant’s displays is not the ideal solution. On the contrary, it creates additional problems regarding the communication channel. In the following section, we present a conceptual design that combines interaction surface and participant display into one integrated system. One advantage of the systems discussed earlier in this chapter is that identical installations are not necessary in both rooms. Given that r1 is installed as described in Fig. 1, a conventional web-conferencing solution could be used in r2. Here, however, a shared whiteboard component could be shown on v2. The video signal of k1 would be shown on v2 anyway. Given a web-based training environment, this would offer substantial advantages over conventional videoconferencing systems. During a lecture, for example, students can see the instructor but cannot be aware of his interaction with the shared whiteboard or other presentation tools in use.
3 An Interactive Video Wall for Tele-presence Applications Many CSCW systems that focus on dynamic collaboration processes across distance aim for the closest possible resemblance to face-to-face communication. Hollan and
636
P. Thies and B. Koehne
Stornetta [12] argue that face-to-face communication provides a way to work collaboratively in a seamless manner. The goal should be to develop systems that people would rather choose to use instead of face-to-face communication. Systems should add to the experience of “just being there” [12]. We follow this approach with the introduction of the Electronic Glassboard system. The approach of the Electronic Glassboard interprets the viewer’s screen as a combination of glass panel and video wall. Glass panels are put to use in the military sector, for example, in control centers for documentation and planning of tactical maneuvers. Their advantage is that they can be written on from both sides with light pens, as well as read from both sides. In military contexts, e.g. in control centers, all participants or viewers respectively are in the same room. It makes no difference on which side the viewer is situated as the viewer can project gestures in reference to the board’s content, thereby associating parts of the illustration to facilitate comprehension. The systems described in section 2.1 do barely support this interpretation of gestures. The Electronic Glassboard utilizes the concept of a glass panel and exchanges the backside from the viewer’s point of view with a video image. It thereby virtualizes the backside. By layering glass panel and video image (see Fig. 3), the user can create sketches in cooperation with a communication partner and discuss them while maintaining direct, eye-to-eye contact.
Fig. 3. Software prototype
3.1 Fundamental Layout The systems described in chapter 2.1 are based on a fixed distance between the participants and the video display. This distance is approximately 3 meters, according to the observation of the authors. The cameras used to record the persons sitting at the tables are focused accordingly. Additionally, they are pointed in such a way that the participants have to be situated in fixed positions in order to allow unproblematic recording. The goal of the Electronic Glassboard is to extend large-sized video screens with an interactive component. As opposed to the systems available on the market, we aim for a completely integrated solution. The user should be enabled to directly step
Electronic Glassboard – Conception and Implementation
637
towards the video image and to use an application, e.g. a sketching tool, layered above the video image in an intuitive way. A distance of 3 meters, as observed in conventional systems, cannot be bridged by the human arm. In order to interact with the screen, the user has to be able to stand within a distance of one arm’s length, at most, much like a blackboard. This leads to questions regarding suitable hardware for interactivity and an adequate camera position that allows recording the persons situated in front of the display. The Electronic Glassboard is more advanced than the ClearBoard approach of Ishii and Kobayashi [13] since it doesn’t have to deal with problems of not being able to erase the partner’s marks and double hand images. This is achieved through a novel design of hardware and software. Systems such as HP Halo or Cisco Telepresence use camera systems which are mounted directly above the video displays, at eye level (see Fig. 1). The Halo system features a separate camera above each video display; Telepresence 3000 arranges 3 cameras above the central video display. Given a distance of approximately 65cm (arm length), this camera position would not be suitable, as assumed in Fig. 4, to record a distortion-free image of the participant situated in front of the display. Given a distance of 3m between video display and participant, the camera position above the video displays is unproblematic as this would result in a very small angle between optic vector v1 generated by the camera k and actual optic vector v2 (parallel to the normal vector of the display area). Eye point a2 of the distant observer and eye point a1 resulting from the camera position are different. However, this only has an effect if the user is positioned directly in front of the screen, which is the case with the Electronic Glassboard. Therefore, the camera should be placed on vector v2, as close as possible to the eye point of the remote user. In other words, behind the display. Full-size LCD displays available on the market today offer so called “Touch Overlays” for implementing a touch-sensitive surface. However, they cannot be considered for a realization of the Electronic Glassboard because a camera positioned at a2 would not be able to “see” through it. Therefore, a touch-sensitive rear projection surface area is necessary. This screen shows the projected image only, while allowing the camera to record the viewer standing in front of it. a1 v1 v2
p1
a2
p2
Fig. 4. Camera position, side view, schematically
638
P. Thies and B. Koehne
3.2 Holographic Projection Surface To allow for a frontal recording of the actor at the Electronic Glassboard, a special projection component is used that combines both interactivity and display functionality while still remaining translucent. This combination becomes possible by the application of a holographic projection display. Companies offering such products include Sax 3d6 or G+B Pronova7. Holographic technology, also known from the field of photography, was adapted by some manufacturers in a novel way. Here, holograms that make up holographic projection surfaces are not used for saving and displaying three-dimensional objects any more, but for the exact redirection of projected light. These holographic elements can be embedded into glass, like plastic films for example. A holographic projection surface consists of many thousands holographic elements that transport the projector’s light to the viewer in an optimal way. This way, the projector’s beam is not directed perpendicular to the panel, but at an angle of 36° to the surface normal. The holographic projection panel is transparent. Fine structures can only be seen when the panel is observed from a very close proximity. Looking at the front, only areas of the panel with displayed content become opaque. Looking from the back, almost no projected image information can be perceived. The holographic elements reflect the projector’s light, for the most part, to the front. Given the good transparency level these panels provide, this technology is predestined for positioning the camera behind the panel in order to record persons in front of it. Initially, holographic projection panels represent pure output mediums. However, the manufacturers mentioned above also offer panels that are touch-sensitive. This allows panels to perform as input mediums as well. 3.3 Software Application Alongside the hardware architecture, the Electronic Glassboard’s application software is essential for cooperation between distributed communication partners. In our approach, the software primarily implements a special whiteboard component for cooperative sketching tasks. Fig. 3 shows the actual software prototype. Cooperative sketching describes the cooperative creation and annotation of drawings and sketches that support the information exchange during meetings. The concept is based on writable boards in conference rooms. Cooperation systems today offer shared whiteboards that rudimentary follow the WYSIWIS-principle8 [14]. Applied to the Electronic Glassboard, such a sketching area is layered directly on top of the video image of the remote partner. Sketches are drawn with a touch screen that is invisible to the user, which is layered on top of the transparent projection film. Additionally, free-hand drawing is implemented for creating sketches. The application allows the setup of line strength and -color. Actual drawings can be deleted to start a new sketching process. The two participating endpoints transmit the drawings as mouse events, which are captured by the software during drawing input. During that process the inputs from 6
http://www.sax3d.com, access on 02/25/2009. http://www.holopro.de/en/index/, access on 02/25/2009. 8 “What you see is what I see“. 7
Electronic Glassboard – Conception and Implementation
639
both sides are combined. The format of entries is also transmitted. Menu options give insight into real-time information regarding the data transmission and conversion. In the background of the drawing area, the video-stream of the remote partner is embedded. Control functions allow pausing and restarting the incoming video stream. This software follows the classic client-server model. A central server application administers the video data streams along with the user entries at the connected Electronic Glassboard clients. The software was implemented in Java. For multimedia handling, the Java Media Framework API (JMF), version 2.1.1e, from Sun Microsystems9 was used. The Java Media Framework is a library for processing video and audio signals. The actual JMF version 2 is an expansion of the software which was formerly built for playing multimedia data and its control only. Here, the most important addition is the possibility to process data from audio-visual capture devices using Java in real-time. For later versions of the application, the implementation of additional drawing tools and more advanced administrative functions is planned. This includes switching between different sketching areas as well as saving and loading of created sketches.
4 Conclusion Tele-presence solutions are increasingly appealing to distributed workgroups. Unfortunately, the products available on the market today do not offer sufficient interactivity for intuitive cooperation of meeting participants. Here, the Electronic Glassboard concretely contributes with a new approach. The video display becomes the drawing area, in a self explanatory way. The system is suitable for meetings that require a high degree of cooperation. It allows the cooperative creation of sketches without losing eye contact with the dialog partner during the process. The gestures involved in sketching content are consistent with the content visible to all participants. Acknowledgements. We are grateful to our colleagues, both at the Department of Informatics at the University of California, Irvine and at the research group for cooperation technology at Stuttgart Media University for their support. We would especially like to thank Gary Olson, David Redmiles, Steve Abrams and Erik Trainer for their insightful comments on earlier drafts of this paper.
References 1. McLuhan, M.: The Gutenberg Galaxy. The Making of Typographical Man. University of Toronto Press, Toronto (1962) 2. Olson, G., Olson, J.: Distance Matters. Human-Computer Interaction 15, 139–178 (2000) 3. Teufel, S., Sauter, C., Mühlherr, T., Bauknecht, K.: Computerunterstützung für die Gruppenarbeit. Addison-Wesley, Bonn (1995) 4. Johansen, R.: Groupware: Computer Support for Business Teams. The Free Press, New York (1988) 9
http://java.sun.com/products/java-media/jmf/, access on: 02/25/2009.
640
P. Thies and B. Koehne
5. Nunamaker, J.F., Dennis, A.R., Valacich, J.S., Vogel, D., George, J.F.: Electronic meeting systems. Commun. ACM 34(7), 40–61 (1991) 6. Bly, S.A., Harrison, S.R., Irwin, S.: Media spaces: bringing people together in a video, audio, and computing environment. Commun. ACM 36, 28–46 (1993) 7. Streitz, N.A., Geißler, J., Holmer, T., Konomi, S., Müller-Tomfelde, C., Reischl, W., Rexroth, P., Seitz, P., Steinmetz, R.: i-LAND: an interactive landscape for creativity and innovation. In: CHI 1999: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 120–127. ACM, New York (1999) 8. Stefik, M., Foster, G., Bobrow, D.G., Kahn, K., Lanning, S., Suchman, L.: Beyond the chalkboard: computer support for collaboration and problem solving in meetings. Commun. ACM 30, 32–47 (1987) 9. Moran, T.P., Chiu, P., Harrison, S., Kurtenbach, G., Minneman, S., van Melle, W.: Evolutionary engagement in an ongoing collaborative work process: a case study. In: Ackerman, M.S. (ed.) Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work, CSCW 1996, Boston, Massachusetts, United States, November 16-20, 1996, pp. 150–159. ACM, New York (1996) 10. Lewe, H.: Computer Aided Team und Produktivität: Einsatzmöglichkeiten und Erfolgspotentiale. Wiesbaden: Gabler. Zugl.: Hohenheim, Univ., Diss. (1994) (1995) 11. Ferwagner, T., Wang, Y., Lewe, H., Krcmar, H.: Experiences in Designing the Hohenheim CATeam Room. In: Benford, S.D., Bowers, J.M. (eds.) Studies in Computer Supported Cooperative Work, pp. 251–266. North-Holland, Amsterdam (1991) 12. Hollan, J., Stornetta, S.: Beyond being there. In: Bauersfeld, P., Bennett, J., Lynch, G. (eds.) Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1992, Monterey, California, United States, May 03-07, 1992, pp. 119–125. ACM Press, New York (1992) 13. Ishii, H., Kobayashi, M.: ClearBoard: a seamless medium for shared drawing and conversation with eye contact. In: CHI 1992: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 525–532. ACM Press, New York (1992) 14. Stefik, M., Bobrow, D.G., Foster, G., Lanning, S., Tatar, D.: WYSIWIS revised: early experiences with multiuser interfaces. ACM Trans. Inf. Syst. 5(2), 147–167 (1987)
A New Automatic Teller Machine (ATM) Proposal through the Analysis of ATMs of Three Banks Serdar Yarlikas Informatics Institute, Department of Information Systems, Middle East Technical University, 06531 Ankara, Turkey [email protected]
Abstract. This study tries to propose a new ATM through the analysis of automatic teller machines (ATMs) of different banks. To propose a new ATM, the ATMs of three banks in Turkey were investigated. These banks were Bank1, Bank-2 and Bank-3. The strengths and weaknesses of the ATMs of the three banks were tried to be determined by comparing the ATMs. To determine the strengths and weaknesses of these ATMs, transaction performance analysis and a questionnaire were applied to the participants. There were 30 participants in the study. Through the transaction performance analysis and the questionnaire, strength points of ATMs were determined. The strength points of ATMs were proposed in order to adapt into the new ATM. At the end of the study, the design properties and the features of the new ATM were stated. Keywords: ATMs, Transaction Performance Analysis, Questionnaire, New ATM.
1 Introduction Automatic Teller Machines (ATMs) are public technology devices which are located and used in public places [1]. ATMs are used to perform banking transactions. Generally, people perform cash withdrawal transactions via ATMs in many countries. ATMs are named as self-service machines and they are becoming increasingly important [2]. If ATMs are designed so as to meet the expectations of people from different social groups, the importance of ATMs increase and more people use ATMs. To solve the usability issue of ATMs, different interface designs are developed. One of the possible solution for the optimal use of ATM is to speak directly to ATM. A speech driven ATM may provide a more natural interaction between the user and the ATM [3], [4], [5]. Besides, physically and visually impaired users can perform banking transactions on their own through speech driven ATM [5]. However, speech driven ATM has various problems. For instance, speech varies between speakers of different gender, age, experience, language, preferences and expectations [6], [7]. In addition to this, stress may affect the speaker’s voice [8]. There are also other alternatives for interface design of ATMs. Especially, in a highly literate society, ATM interface may be made accessible through the provision of instructions and menus [9]. An icon-based visual interface is an other alternative. Icon-based interfaces are considered as an alternative to speech recognition interfaces
for illiterate populations [9], [10]. The literature related with ATM usability shows that the researchers generally deal with interface design of ATMs. Designers developed iconic and speech-based interfaces instead of text-based interfaces. The main aim of these designs is to make ATMs accessible to general population. The researchers generally deal with ATMs in terms of the usage of older adults. Besides, they try to show the differences between the illiterate and literate people in terms of the ATM usage. Generally, they applied questionnaires, performance tests related with banking transactions and used direct observation method while applying the performance tests. Their aim was to determine the design issues related with ATMs usage. By evaluating the results, they proposed new ATMs design or modifications to the existing ATMs. It is inferred from the literature that the main aim of the researches related with ATMs is to determine the design issues and proposing new ATMs design to solve these issues. Instead of designing a completely new ATM, a new ATM can be designed by comparing the strengths, weaknesses and similarities of existing ATMs of different banks. By combining the strengths of the ATMs of these different banks in the new ATM, a more proper ATM can be designed and constructed. To determine the strengths, weaknesses and similarities of these ATMs, usability tests related with banking transactions and questionnaires can be applied. Instead of determining the effects of individual differences on ATM usage, determining the design issues resulted from the design of ATMs machine can be a more proper approach .
2 Method In this section, the method used in the study was explained in a detailed way. Pilot study was conducted before beginning the main empirical study. The explanations and evaluations about the pilot study were presented in this section. The characteristics of the participants were stated. Tasks and the questionnaire applied in the study were also explained. In addition to these, a special section was made for the research question. The research question was stated and defined in a detailed way in this section. 2.1 The Pilot Study A pilot study was decided to be applied before accepting the method that would be used in the study. Before beginning the study, it was thought that there should be two main steps in the study. In the first step, a questionnaire, using a combination of open-and close-ended questions, should be used to collect data from participants. The questionnaire should include both demographic data items and the questions related with the design of ATMs. The ATMs of three banks were selected to evaluate. These banks were Bank-1, Bank-2 and Bank-3. The aim was to determine the strengths and weaknesses of these ATMs by comparing them in terms of their design. It was thought that a questionnaire and the performance results for banking transactions could be helpful in determining the strengths and the weaknesses. A questionnaire which included 37 questions was designed for the study. Some of the questions in the
A New Automatic Teller Machine (ATM) Proposal through the Analysis of ATMs
643
questionnaire were adapted from the literature directly. Besides, the other questions were adapted and added according to the investigation of the ATMs of these three banks. In the second step, the duration of transactions via ATMs for the participants would be collected as quantitative data. The time spent by users while performing specific banking transactions via ATMs would be determined. This quantitative data would be collected from participants separately. The chronometer would be used to determine the duration of transactions. The common tasks performed via these three ATMs were listed. There were five common tasks which could be applied to the participants, but one of them was related with password change. This task was eliminated since it was thought that the participants would not apply this task due to the security reasons. For these reasons, remaining four tasks could be applied in the study. The pilot study was conducted with three participants. There were one participant for Bank-1, one participant for Bank-2 and also one participant for Bank-3. All of the participants stated that the questions in the questionnaire were proper and they were not repetitive. They also stressed that they understood all of the questions. As a result of the pilot test, the questions in the questionnaire were not changed. All of the participants performed the four tasks. The tasks were not complicated for the participants. They understood easily what they should do with each of the four tasks. Consequently, the pilot study denoted that the method was appropriate for the study. 2.2 The Characteristics of the Participants There were 30 participants in the study. There were equal number of participants for each of the ATMs of three banks. There were 21 males and 9 females in the study. The percentage of male participants was 70% and the percentage of female participants was 30% for each of the ATMs of three banks. The mean age of participants for Bank-3 was 33 years, the mean age of participants for Bank-1 was 43.3 years. Besides, the mean age of participants for Bank-2 was 27.2. The mean age of all participants for the study was 34.47 years. The mean period of ATM usage for the participants for Bank-3 was 8.3 years. The mean period of ATM usage for the participants for Bank-2 was 7.6 years and the mean period of ATM usage for the participants for Bank-1 was 10.8 years. The mean period of ATM usage for all participants was 8.9 years. 80 % of the participants for the ATM of Bank-1 had university education level, on the other hand, 20 % of them had high school education level. 80 % of the participants for the ATM of Bank-2 had university education level and 10% of them had high school education level. Besides, 10% of them had primary education level. 60 % of the participants for the ATM of Bank-3 had university education level and 20% of them had high school education level. Besides, 20% of them had primary education level. 2.3 The Tasks In the study, the participants preferred performing banking transactions via the related ATM in the study before filling the questionnaire. The tasks were determined according to the investigation of ATMs of Bank-1, Bank-2 and Bank-3. The users of
644
S. Yarlikas
ATM of Bank-1 can perform 8 banking transactions via ATM of Bank-1. The users of ATM of Bank-2 can perform 14 banking transactions via ATM of Bank-2. On the other hand, the users of ATM of Bank-3 can perform 13 banking transactions via ATM of Bank-3. To compare the ATMs of these three banks and to evaluate them under the same circumstances, the common banking transactions for each of the ATMs of three banks should be determined. There were five common banking transactions for these three banks. These transactions were money withdrawal, display of the balance of the account, printing the abstract of the account, transfer (money order) and changing password. The changing password task was eliminated since it was thought that the participants would not apply this task due to the security reasons. For these reasons, remaining four banking transactions were applied in the study. The chronometer was used to determine the duration of these four transactions for each participant. The transaction time for money withdrawal was calculated from the time the participant inserted the card into the ATM to the time the participant took the receipt related with money withdrawal. I wanted from all participants to take the receipt related with this transaction. The transaction time for the display of the balance of the account was calculated from the time the participant inserted the card into the ATM to the time the participant saw the display of the balance of the account. The transaction time for printing the abstract of the account was calculated from the time the participant inserted the card into the ATM to the time the participant took the card. In this transaction, the participants took the printed abstract of account and then they took the card. The transaction time for transfer (money order) was calculated from the time the participant inserted the card into the ATM to the time the participant took the card. In this transaction, the participants took the receipt related with transfer and then the card at the end of this transaction. All of the 30 participants performed firstly money withdrawal transaction, secondly display of the balance of the account transaction, thirdly printing the abstract of the account transaction. Finally, they performed transfer (money order) transaction. 2.4 The Questionnaire The 30 participants filled the questionnaire after performing banking transactions. The questionnaire was applied in Ankara and in Kutahya. The questionnaire included 37 questions. The mean time of the questionnaire for a participant was 11 min 40 sec. The questionnaire lasted approximately 350 minutes (5 hours 50 minutes). The questionnaire was applied in holidays and also applied except the working hours. The participants were selected randomly from the people waiting in the ATM queue for the transaction. The participants filled the questionnaires near me. Since they filled the questionnaires near me, they could ask me about the questions in the questionnaire. Some of the questions in the questionnaire were adapted from the literature directly [11], [12]. Besides, the other questions were adapted and added according to the investigation of the ATMs of these three banks. First 4 questions in the questionnaire were for determining the individual characteristics of participants. Question 5 in the questionnaire was for determining
A New Automatic Teller Machine (ATM) Proposal through the Analysis of ATMs
645
how frequent the participants used the ATMs of their bank. Question 6 was related with the ATM preference of participants. Question 7, 8,10,11 were related with the ease of use in ATMs. Question 9,13 were related with lighting conditions in ATMs. Question 12 was related with the location of keys in ATMs. Question 14,24,32 were for evaluating the error correction, error elimination and emergency exit structures of ATMs. Question 15 was used to determine the usage of four banking transactions in the study according to three ATMs. Question 16,17 were related with money withdrawal transaction. The results of these two questions were evaluated with the transaction time for money withdrawal for the ATMs of three banks. Question 18,19 were related with transfer (money order) transaction. The results of these two questions were evaluated with the transaction time for transfer (money order) for the ATMs of three banks. Question 20, 21 were related with display of the balance of the account transaction. The results of these two questions were evaluated with the transaction time for the display of the balance of the account for the ATMs of three banks. Question 22,23 were related with printing the abstract of the account transaction. The results of these two questions were evaluated with the transaction time for printing the abstract of the account for the ATMs of three banks. Question 25 was to determine whether the participants were found usable the ATMs of their bank. Question 26,27 and 29 were related with overall satisfaction level related with ATM services, speed of banking transactions and visuality. Question 28 and 30 were related with security and password issues in ATMs. Question 31,35 were related with the receipt and money amount. Question 33,34 were related with alternate design interfaces for ATMs. Question 36 was related with the issues resulted from ATM. Question 37 was related with the proposals for the ATMs. 2.5 Research Questions This study aims to determine the strengths and weaknesses of the ATMs of three banks. By comparing the ATMs of these three banks through the analysis of the tasks and the questionnaire, the aim was to propose a new ATM design. The strength points of the ATMs of these three banks would be adapted and located into the new ATM design. For these reasons, the study was related with determining the best design features and best transaction properties of these three banks. Consequently, the aim was to propose a new ATM design with the analysis of these three banks.
3 Results Results of the study achieved through the task analysis and questionnaire were summarized and presented in this section. 3.1 Results for Yes or No Questions The results for yes or no questions in the questionnaire were summarized in the following table.
646
S. Yarlikas Table 1. Evaluation of yes or no questions in the questionnaire Bank-1
Bank-2
Bank-3
Question No
Yes (%)
No(%)
Yes (%)
No(%)
Yes (%)
No(%)
6
100
0
100
0
100
0
7
90
10
100
0
100
0
8
100
0
100
0
100
0
9
40
60
70
30
50
50
10
100
0
90
10
100
0
11
100
0
80
20
90
10
12
100
0
90
10
90
10
13
90
10
70
30
50
50
14
50
50
50
50
60
40
24
90
10
50
50
40
60
28
60
40
30
70
70
30
30
20
80
0
100
10
90
31
30
70
20
80
40
60
32
50
50
70
30
90
10
33
60
40
70
30
60
40
34
70
30
50
50
40
60
35
40
60
10
90
40
60
3.2 Results for Multiple Choice Questions The results for multiple choice questions in the questionnaire were summarized in the following table. Table 2. Evaluation of multiple choice questions in the questionnaire Question No 5 15 (Money Withdrawal) 15 (Transfer) 15 (Display of the balance of account) 15 (Printing the abstract of account) 26 27 29
Bank-1 Mean Scale Point 3 4.4 1.9 3 1.7 1.9 3.3 3.2
Bank-2 Mean Scale Point 3.3 3.9 1.9 2.3 2.4 2.1 3.8 3.3
Bank-3 Mean Scale Point 3.2 4.5 2.1 2.8 2.3 2.2 3.6 3.3
3.3 Results for Banking Transactions The results for banking transactions were summarized in the following table.
A New Automatic Teller Machine (ATM) Proposal through the Analysis of ATMs
647
Table 3. Evaluation of banking transactions in the study Bank-1 Question No 16
Yes (%)
17 18
10
19 20
30
21 22
10
23
10
No (%)
LP
Bank-2
3.7
MTT (sec) 50.51
3
92.8
4.2
31.08
3.6
39.11
90
10
70
40
90
90
Yes (%)
10
20
No (%)
LP
Bank-3
4.2
MTT (sec) 42.35
3.4
73.96
4.2
20.01
3.8
23.6
90
40
90
No (%)
LP 4.6
20
60
80
Yes (%)
20
40
MTT (sec) 39.6 2
80 3.5
86.5 6
4.3
19.0 7
4.1
24.9 5
60
80
60
In Table 3, LP states the Likert Point, besides, MTT states the Mean Transaction Time for the related transaction in the ATM of the related bank. 3.4 Results for Usability Situation of ATMs The 70% of users of ATM of Bank-1 were found their ATM usable. The 60% of users of ATM of Bank-2 were found their ATM usable. On other hand, the 80% of users of ATM of Bank-3 were found their ATM usable. The results showed that the users of Bank-3 found their ATM most usable. The mean number of ATM used by the participants of Bank-3 was 2.1. The mean number of ATM used by the participants of Bank-2 was 3.2. The mean number of ATM used by the participants of Bank-1 was 2.4. The results showed that all of the three group users use more than one ATM. As a result of this, it can be stated that the structure of participants in the study were suitable for making comparisons between ATMs. 3.5 The Issues Resulted from ATM Machine The common issues faced in the ATMs of these three banks were listed as below: 1. Out of Service Situation (Service not available)(%76,67). 2. The amount of money problem in ATMs. The users can not withdrawal the amount of money that they wanted (%60). 3. Running out of money ( %56.67). 4. Inconvenient physical location of ATM (%16.67). 5. Missing card (16,67%). 6. Unclear instructions on ATM (10.00%). 7. Not knowing if the ATM is operating (3.33%). The other issues which were stressed by some of the participants were stated as below: • All of the money withdrawal issue (Bank-1) • The fault in ATMs in holidays are postponed until working days. ( Bank-1)
648
S. Yarlikas
• Emergency exit issue (Bank-1) • The ATM machine gives the receipt related with the abstract of account except the request of the user. (Bank-2) 3.6 The Proposals for ATMs The proposals of the participants of the ATMs of three banks were stated in the following table. Table 4. Proposals for ATMs of the banks Proposals Bank-1
Bank-2
Bank-3
• Transactions without ATM card. • Telephone Payment through ATM • Money capacity of the ATM machine should be increased. • The number of ATM machines should be increased. • Money investing transaction should be adapted. • Credit card Payment system should be adapted. • Stock exchange transactions must be adapted.
• The instructions and explanations in the receipt of abstract of account should be clear and understandable. • There should not be any fee for the transactions performed via ATM.
• The number of ATM machines should be increased. • Transfer (money order) operation can be done except the working hours. • Money investing transaction should be adapted. • The instructions and explanations in the receipt of abstract of account should be clear and understandable.
4 Discussion The features and design properties of new ATM achieved through the analysis of ATMs of Bank-1, Bank-2 and Bank-3 were stated and explained in this section. In other words, main results of the study were clarified in this section. Suggestions for further research were also stated in this section. 4.1 The New ATM The features and design properties of ATM were achieved through the task analysis and the questionnaire related with the ATMs of Bank-1, Bank-2 and Bank-3. The features and design properties of new ATM according to the study were stated as below: • The ease of use structure of Bank-3 or Bank-1 should be adapted into the new ATM .
A New Automatic Teller Machine (ATM) Proposal through the Analysis of ATMs
649
• For lighting situation, the structure of Bank-2 should be preferred and the structure of Bank-3 for the sun light should be adapted into the new ATM. • The location of keys in the ATM machine of Bank-1 should be adapted into the new ATM. • Error correction, error elimination and emergency exit structures of Bank-3 should be adapted into the new ATM. • Money withdrawal transaction and money withdrawal process of Bank-3 should be adapted into the new ATM. • The transfer transaction and transfer process of Bank-2 should be adapted into the new ATM. • The display of balance of account transaction and display of balance of account process of Bank-3 should be adapted into the new ATM. • Printing the abstract of account transaction and printing the abstract of account process of Bank-2 should be adapted into the new ATM. • In terms of overall satisfaction level of ATM services, Bank-3 should be adapted into the new ATM. • In terms of the general speed of banking transactions, Bank-2 should be adapted into the new ATM. • Visuality of Bank-2 or Bank-3 should be adapted into the new ATM. • Security level of ATM of Bank-3 should be adapted into the new ATM and password structure of Bank-2 should be adapted into the new ATM. • None of these three ATMs should be adapted into the new ATM in terms of receipt situation and money amount in ATMs. A new structure should be constructed for these issues and then this new structure should be adapted into the new ATM. • Touch screen and voice based structures can be adapted into the new ATM. • New ATM should not be out of service. • The users can withdrawal the amount of money that they wanted. • Money should not be run out of in the new ATM. • Transactions can be done in the new ATM without ATM card. • Telephone payment can be done in the new ATM. • Money investing transaction, credit card payment and stock exchange transactions should be in the new ATM. • The instructions and explanations in the receipt of abstract of account should be clear and understandable thanks to the new ATM. • There should not be any fee for the transactions performed via the new ATM. • Transfer (money order) operation can be done except the working hours thanks to the new ATM. 4.2 Suggestions for Further Research The study can be applied by investigating the ATMs of more banks. More technical questions related with the design of ATM can be adapted into the questionnaire. The number of tasks can be increased in the new study. Preliminary investigation can be done about the ATMs of many banks. Detailed statistical analysis techniques can be used and applied for evaluating the results.
650
S. Yarlikas
References 1. Baber, C., Stanton, N.A., Johnson, G.I.: From Public Technology to Ubiquitous Computing: Implications for Ergonomics. Ergonomics 41, 921–926 (1998) 2. Manzke, J.M.: Adaptation of a Cash Dispenser to the Needs of Blind and Visually Impaired People. In: Third International ACM Conference on Assistive Technologies, pp. 116–123 (1998) 3. Rogers, W.A., Fisk, A.D.: ATM Design and Training issues. Ergonomics in Design 5, 4–9 (1997) 4. Honet, K.S., Graham, R., Maguire, M.C., Baber, C., Johnson, G.I.: Speech Technology for Automatic Teller Machines: An Investigation of User Attitude and Performance. Ergonomics 41, 962–981 (1998) 5. Manzke, J.M., Egan, D.H., Felix, D., Krueger, H.: What Makes an Automated Teller Machine Usable by Blind Users? Ergonomics 41, 982–999 (1998) 6. Khalid, H.M., Helander, M.G.: Automatic Speech Recognition. In: Karwowski, W. (ed.) International Encyclopedia of Ergonomics and Human Factors, pp. 631–635. Taylor & Francis, London (2000) 7. Kamm, C., Helander, M.: Design Issues for Interfaces Using Voice Input. In: Helander, M., Landauer, T.K., Prabhu, P.V. (eds.) Handbook of Human-Computer Interaction, 2nd edn., pp. 1043–1059. Elsevier Science, Amsterdam (1997) 8. Karis, D., Dobroth, K.M.: Psychological and Human Factors Issues in the Design of Speech Recognition System. In: Sydral, A., Bennett, R., Greenspan, S. (eds.) Applied Speech Technology, pp. 359–388. CRC Press, Boca Raton (1995) 9. Dix, A., Finlayson, J.E., Abowd, G.D., Beale, R.: Human-Computer Interaction, 2nd edn. Prentice Hall, London (1998) 10. Thatcher, A., Shaik, F., Zimmerman, C.: Attitudes of Semi-Literate and Literate Bank Account Holders to the Use of Automatic Teller Machines (ATMs). International Journal of Industrial Ergonomics 35, 115–130 (2005) 11. Rogers, W.A., Cabrera, E.F., Gilbert, D.K.: An Analysis of Automatic Teller Machine Usage by Older Adults: A Structured Interview Approach. Applied Ergonomics 28, 173–180 (1997) 12. Pereira, L.M., Cordeiro, A., Lopes, J.B., Espadinha, C., Ribeiro, M.: Case Study: An Example of How Evaluation Change the Design: ATM project in Portugal. In: Roe, P.R.W. (ed.) Towards an Inclusive Future: Impact and Wider Potential of Information and Communication Technologies, section 6.4, The role of evaluation of accessibility, COST219ter, Brussels, ch. 6, pp. 262–272 (2005)
Designing Usable Bio-information Architectures Davide Bolchini1, Anthony Finkestein2, and Paolo Paolini3 1
Indiana University, School of Informatics (IUPUI) [email protected] 2 University College London, Dept. of Computer Science [email protected] 3 Politecnico di Milano, Dept. of Information and Electronics [email protected]
Abstract. Bioinformatics websites offer to the life science large community repositories of information ranging from genes, genomes, proteins, experimental data and their integration, with the aim of supporting the elucidation of biological processes. As the bioinformatics community increasingly relies on the design, sharing and use of web-based resources, it is important to systematically address the usability of these applications and to deliver a more rewarding user experience to researchers. The bioinformatics community is also acknowledging the role that Human-Computer Interaction can play to improve the usability of these systems. In the context of a project aiming at improving the usability of large bioinformatics websites, we carried out an in-depth usability analysis and conceptual redesign of a well-known protein repository, with the aim of characterizing information architecture usability problems and providing corresponding design solutions to improve the user experience. This design has been validated and refined using interactive prototypes with users, usability experts and domain experts, and opens a new set of navigation opportunities which has the potential to improve the research work of bioinformaticians. Although being a preliminary study, the research reveals generic information architecture and navigation issues which have design implications for browsing-intensive bioinformatics repositories at large. Keywords: usability, information architecture, navigation design, bioinformatics.
aimed at understanding and characterizing general usability issues of web-based bioinformatics resources and provide design improvements, we have carried out an in-depth usability analysis and conceptual redesign of a well-known protein classification repository (CATH, http://www.cathdb.info/). Through a detailed usability inspection [3][6] of the information architecture currently supported, we have discovered that, being based on a hierarchical classification of the content, the navigation architecture follows a strict tree-based hierarchical paradigm, where each branch of the tree determines a sub-class of the previous branch, and where each element in the hierarchy has one and only one position in the tree. This hierarchical classification of the proteins is used as the main navigation paradigm to access and explore the protein collection. This situation has a potential negative impact on the overall usability of the application, as it hinders the possibility of exploring and navigating the classification and protein collection with efficiency and flexibility, and poses obstacles to the accomplishments of simple, basic exploratory user tasks (e.g. the possibility to browse proteins without being forced to specify beforehand 8 different parameters). To address these issues and propose an enhanced design solution which can be useful and applicable to a variety of bioinformatics resources using with similar design solutions, we have proposed a fundamental paradigm shift in the design of the browsing experience, by capitalizing on well-known design principles from the hypermedia and web communities. The basic navigation paradigm that we have proposed for CATH is based on the assumption that each classification criterion for the proteins (e.g. class, topology, architecture, homologous superfamily) can be modelled as a primary navigation dimension (facet or trail), to be browsed orthogonally to all the others, instead of being represented just as a level of the hierarchy. Using each classification criterion as an independent navigation driver, it is possible to make these criteria interact, enabling the users to visualize, explore and browse the relationships between them with a greater flexibility than the one currently offered by a hierarchical navigation paradigm. This design, which have been validated and iteratively refined on interactive prototypes with users, usability experts and domain experts (bioinformaticians), opens a whole set of new navigation possibilities which improve the quality of the overall user experience and which will be reported in this paper. The remainder of the paper is organized as follows. Section 2 overviews the basic related work and underlying body of knowledge that supports the redesign work described in the paper, and mainly the design principles that have been used. Section 3 shows the key results of the preliminary usability inspection of CATH; which identified a number of usability and the need for a conceptual redesign. Section 4 illustrates the conceptual remodeling of the CATH user experience, which led to the development of specific design proposals, briefly illustrated and discussed in Section 5. Section 6 summarizes the contribution of the work and points to relevant research outlooks.
2 Theoretical Background and Related Work There is evidence of an increasing awareness of the need of usability studies in the development of biomedical systems in general and of the benefits that a systematic user-centered design process can bring to the development of interactive systems in the
Designing Usable Bio-information Architectures
655
bio-related areas. There have also been some initial but notable efforts to address the usability of bioinformatics web-based applications [6]. The challenge of bringing an increased awareness of usability and user-centered design to the development of bioinformatics applications is currently tackled from different disciplinary perspectives. The Human-Centered Software Engineering (HCSE) at Concordia University has worked on developing integrated web-based interfaces to popular bioinformatics portals in order to provide integrated access to web resources relevant to a set of typical tasks [5]. Acknowledging the fact that web bioinformatics resources are so diversified and scattered around - thus forcing researchers to discover, locate these resources and then learn different interaction paradigms to access, search for data and complete tasks - this research explored the possibility of offering a unique one-stop access interface to a selected (limited) set of recurrent bioinformatics tasks. Approaching the complexity of bioinformatics resources from another perspective, the Human-Computer Interaction Lab at the University of Maryland is investigating advanced visualization techniques to access and manipulate large multimedia information sets in biological databases [4]. Usability challenges for complex data visualization and exploration in support to discovery and decision-making in bioinformatics are tackled with advanced visualization paradigms. Moving to a higher level of user activities, Joan Bartlett at McGill University has been investigating the daily activities of bioinformatics researchers in order to derive a list of typical information tasks that entail the use of web-based resources to complete [1]. Although these contributions cover important aspects of improving the user experience of biological databases little has been done to analyse the underlying design characteristics of web bioinformatics resources that can lead to potential usability problems. Tackling design issues captures the usability problems at their source, thus providing strategies to prevent the emergence of problems in current and future applications. Recently, a contribution in this direction has been elaborated by the authors in their preliminary work in collecting and characterizing general types of usability problems in web-based bioinformatics repositories [6]. Given the centrality of the attention to the quality of design as one important factor determining the quality of the user experience, the underlying theoretical and methodological foundation for this recent advance in the field, and also for the research presented in this paper, lies in the long-standing tradition of conceptual hypermedia and web modeling. Over the last decade, a rich body of knowledge aimed at providing the conceptual tools to design, analyze and describe the structure of complex hypertext and hypermedia applications (and consequently the ones available on the web) has been produced in hypertext, hypermedia and web engineering communities. A constant line of research that can be identified is the one related to design models, i.e. systematic and cohesive set of modeling abstractions useful to describe the complexity of an interactive application at the proper level of granularity, with the goal of making this complexity more understandable and tractable for various purposes (mainly design, analysis and evaluation). One of the key underpinning theoretical construct of these models is the distinction between the design of the content or information base (also called “hyperbase”) and the design of the access structures (navigational paths enabling the users to locate and reach the content of interest). Recently, one of the lasts heir of this research tradition is IDM (Interactive Dialogue Model), which offers a basic, lightweight set of modeling primitives to design the information and navigation patterns of a complex
656
D. Bolchini, A. Finkestein, and P. Paolini
interactive applications in terms of dialogue between the user and the system [2]. A common denominators of this rich (and growing) body of knowledge in the areas of conceptual design, design primitives, and modeling notations for complex interactive applications, is the assumption that the design of the user experience involves a set of subjective (although reasoned an grounded) design decisions driven by the application goals and the user needs and tasks, and not by a predefined organization of the information available in the knowledge domain. A quite recent and related line of work, ultimately stemming from the information architecture and library science tradition, but conceptually very similar to the early advancement of hypertext and hypermedia design models, is the definition of flexible navigation paradigms based on faceted classification [7], one of whose tenets is that multiple access paths can be designed to reach the same information object, and this provides more flexibility in effectively supporting user tasks. This fundamental principle led to the definition of methods and tools to design navigation design patterns, access structures to content and information architectures that might support a flexible set of user tasks and scenarios, and are not primarily driven by an a priori information structure defined independently from the user experience requirements. Given our recent research results, we claim that the adoption of these (and other) fundamental design principles used in the usability, hypermedia and web design research community can be applied to the design of bioinformatics web applications and can bring an enhanced level of usability and, ultimately, research productivity in this growing domain of web-based resources.
3 Usability Issues in Rigid Hierarchical Navigation In the context of a project aiming at improving the usability of large bioinformatics websites, we carried out an in-depth usability inspection and conceptual redesign of a well-known protein web-based repository (CATH, http://www.cathdb.info/), with the aim of characterizing the nature of the usability problems and providing corresponding design solutions to improve the user experience. CATH is the primary online resource for protein domain classification and is developed, maintained and hosted at University College London, Biochemistry Department. CATH classifies protein domains (currently ca. 93’000) into 8 levels, 4 related to the similarity concerning the structural characteristics (mainly the shape) of the proteins (levels are, for example, class, architecture, or topology), and 4 related to the similarities in the sequence of amino acids. In particular, CATH classifies protein domains according to the following hierarchical levels (adapted from http://www.cathdb.info/): Class, C-level is determined according to the secondary structure composition and packing within the structure. Architecture, A-level describes the overall shape of the domain structure as determined by the orientations of the secondary structures. Topology (Fold family), Tlevel are grouped into fold groups at this level depending on both the overall shape and connectivity of the secondary structures. Homologous Superfamily, H-level groups together protein domains which are thought to share a common ancestor. Additional 4 levels (S,O,L,I,D), grouping proteins according their similarity in sequence.
Designing Usable Bio-information Architectures
Fig. 1a. CATH hierarchical navigation
657
Fig. 1b. Interface over the hierarchy
This hierarchical classification of the protein domains is reflected by a tree-based hierarchical navigation structure as the only navigation paradigm offered by CATH to browse and explore the protein collection (Figure 1a). The information architecture designed is a classical 8-level hierarchy or tree, where each node has only one ancestor, i.e. it has a unique position in the hierarchy, and serves a mainly classification purpose, as the actual content (i.e. the detailed information about the protein domains) is contained in the leaf nodes (end points of the tree). We should note, however, that while the hierarchical classification of proteins based on their structure is per se a valuable knowledge asset for the life science community, the usability problems intrinsic to a large, multi-level tree-based navigation structure are several, and are here briefly summarized. At each level of the hierarchy, access is granted to nodes to the immediate next level; whereas nodes further down on the hierarchy tree are not directly accessible (i.e., it is not possible to skip levels, see Figure 1b). This design poses severe obstacles to users who need to explore the protein classification by a specific criterion (Topology), from a given point in the tree, and are not interested in exploring intermediate classifications. In other words, this design forces users to traverse all the levels of the hierarchy to reach a protein domain of interest (leaf node). A necessary access sequence is imposed to the user’s navigation within the hierarchy, which has the consequence of making the browsing flow very rigid, as potentially irrelevant steps are put on the way. This navigation design might be effective only in scenarios in which users are able to clearly specify upfront the values of all (8) parameters of the hierarchy, in order to locate a protein domain. The solution, however, is some way far from being effective and efficient when users have more ill-defined knowledge of the classification parameters, need exploring and iteratively refining the browsing scope. As additional usability problem, we should note that the rigidity imposed to the browsing mechanisms makes the user’s interaction flow even more inefficient in case a sequence of branches in the hierarchy is minimally populated (one node for each level): this flattened classification makes the content base (leaf nodes) even more desirable for quick access but still users have to traverse all these one-node populated branches to reach the protein domains of interest.
658
D. Bolchini, A. Finkestein, and P. Paolini
4 Remodelling Hierarchies through Flexible Navigation Paradigms To address these issues, we claim that the starting point to conceptualize the requirements for an enhanced design solution it to decouple two fundamental concerns during design: (a) the Information Architecture paradigm: a knowledge, domain-driven, purposeful representation of the information (e.g. hierarchical classification of the protein according to their structure), which can be useful as is for knowledge sharing, for scientific dissemination, or for very specific tasks; (b) the Navigation/Interaction paradigms, i.e. design strategies to support the user experience in terms of interaction and browsing possibilities on top of the existing information architecture. Many navigation styles can be supported on top of the same basic information architecture (including hierarchical ones). The design of the navigation paradigms should allow the users to browse and explore more efficiently and effectively the information architecture and the content offered, covering a wider range of potential tasks than the one supported by the constraints of the underlying information architecture. The basic navigation design paradigm that we have proposed for CATH is based on the assumption that each classification criterion (class, topology, architecture, homologous superfamily), instead of being represented as a level of the hierarchy, can be modelled as a primary navigation dimension (facet or trail), to be browsed orthogonally to all the others.
Fig. 2. Remodelling the Access Paths for the Protein Domains
Using each classification criterion as an independent navigation driver, it is possible to make these criteria interact, enabling the users to visualize, explore and browse the relationships between them with a greater flexibility than the one currently offered by a hierarchical navigation paradigm. With this new modeling of the navigation space, the user can choose to browse the protein classification by any desired criteria (one of the 8 available in CATH), visualize the corresponding protein domain collection, and use the other criteria to refine or filter the browsing scope. The first result of the redesign is that, while the underlying information architecture remains strongly hierarchical, the navigation is remodeled as a semi-flat structure composed by the set of all classification criteria (called “facets”,
Designing Usable Bio-information Architectures
659
Fig. 3. Remodelling Associative Navigation
according to faceted navigation, or “trail” or “access paths”, according to hypermedia design) always available to the user. A plastic representation of this conceptual remodeling is shown in Figure 2. As each protein domain can be labeled according to its structural characteristics (each characteristic corresponds to one of the 8 CATH levels), these same characteristics can be used to design the access structures to the protein domains. The selection of a specific value of a characteristic determines a collection of protein domains, which could be browsed and accessed as is, independently by the other characteristics. For example, users interested in protein domain of architecture “horse shoe” can visualize and access all protein domain of with horse shoe architectures, independently by their specific topology, homologous superfamily or sequence. As additional design opportunity, this hypertextual modeling enables supporting classic associative navigation from the protein domain. Independently by the specific access path users have chosen to reach a given protein domain, they can navigate from the details of that protein domain to other similar protein domain by architecture, topology, superfamily, or sequence. In other words, the same characteristics of the protein domains used for access purposes can be efficiently reused to offer an enriched navigational experience once a protein domain has been reached. This design can help continue the navigation through a richer exploration of the content available, thus paving the way for more serendipity in the user experience (e.g. discovering new content of interest).
5 Design and Exploratory Prototyping On the basis of the new modeling of the navigation paradigms to be supported (on top of the existing hierarchical information architecture and content), new requirements and design alternatives has been envisioned and discussed with CATH stakeholders, usability and domain experts. An interactive prototype has been produced mainly as the basis for the discussion, as a plastic vehicle to communicate design ideas and elicit new requirements for future development.
660
D. Bolchini, A. Finkestein, and P. Paolini
Fig. 4. Excerpts from the conceptual prototype design for advanced CATH navigation
Based on the conceptual modelling illustrated in the previous section, the main navigation console offers a semi-flat navigation structure, where users can select any CATH classification criteria to start browsing the protein domain collection. Each criterion can be expanded to include its values. For example, the criterion “architecture” includes all 40 architectures of proteins. The criterion topology includes all 1084 topologies identified, and so on. The notion of hierarchy is completely disappeared from the navigation perspective (the user can start browsing from any criterion); it remains, however, as the underlying architecture of the content. The design concept also enables to use a classification criterion at choice as primary navigation dimension to visualize the rest of the hierarchy by the values of that criterion. For example, users can choose to select the “architecture” of the protein as primary classification criterion, select a specific architecture, for example “alpha horseshoe”, and project this value over the rest of the classification, throughout all the levels down of the hierarchy. In this way, it is possible to (a) visualize the relative distribution of the cardinality of the protein domains among classification criteria; (b) efficiently skipping levels of the hierarchy; (c) get a clear idea of the cardinality of each classification being visualized. To illustrate the type of insights that can be gained by such a design, examples of possible questions that can be answered by browsing the classification are: How superfamilies are distributed among topologies? How superfamilies are distributed among architectures? Which homologous superfamilies have architecture “alpha horseshoe”? Which topologies have architecture “alpha horseshoe”? What are the class “alpha” topologies? What are the class “alpha” architectures? How many protein domains are there in each class? Overall, the potential of the introduction of this design concept is to enable researchers to reason about the classification of protein domains, and exploring the relationships in terms of relative distribution of the protein collection, before going into the details of a specific protein data. In terms of hypermedia design, one of the cores of the user experience that this solution can support is the use of the access structures as relevant content (where insights can be gained from). Besides supporting a more advanced navigation of the full protein collection and associative linking (Figure 5b), the new design also provides a way to superimpose multiple classifications over the previously selected protein collection (Figure 5a). This helps understand which the protein domains of a given type are within a
Designing Usable Bio-information Architectures
Fig. 5a. Advanced browsing in the protein collection.
661
Fig. 5b. Navigating to similar proteins
previously selected collection. These and other design features enabled by the remodeling and shown by the interactive prototype are the results of three evaluation sessions: two with experts in usability, information architecture and bioinformatics and one with biologists and bioinformaticians.
6 Conclusions and Future Work This work is an important step of a wider effort aiming at surveying a larger number of biological databases to collect and characterize the typical usability breakdowns and propose design solutions that can improve the quality of the user experience for the life science community. The ultimate goal is to make available proven design patterns (i.e. proven solutions that work) and conceptual tools in order to promote a more aware human-centered development process of bioinformatics applications. The research directions that this and other works of the authors have recently initiated in the same line open new research opportunities for the HCI and Web community to provide a rich contribution to the improvement of the user experience of bioinformatics applications, both in the area of requirements elicitation and analysis (understand the users and stakeholders involved and their goals), conceptual design (findings design solutions to effectively shape these information-intensive applications) and usability evaluation (applying and improving existing tools and techniques to cope with a domain of growing complexity).
Acknowledgements We thank Prof. Christine Orengo and the CATH team, most notably Ian Sillitoe, for their feedback and support to the wider project work aimed at exploring usability issues in CATH. This work has been funded by a grant from the Swiss National Science Foundation (SNSF) and the UCL Experimental Cancer Medicine Centre supported by Cancer Research UK and Department of Health. Anthony Finkelstein has been supported by Cancer Research UK and the National Cancer Research Informatics Initiative.
662
D. Bolchini, A. Finkestein, and P. Paolini
References 1. Bartlett, J.C., Toms, E.G.: Developing a protocol for bioinformatics analysis: an integrated information behavior and task analysis approach. Journal of the American Society for Information Science and Technology 56(5), 469–482 (2005) 2. Bolchini, D., Paolini, P.: Interactive Dialogue Model: a Design Technique for MultiChannel Applications. IEEE Transactions on Multimedia 8(3), 529–541 (2006) 3. Bolchini, D., Garzotto, F.: Quality of Web Usability Evaluation Methods: an Empirical Study on MILE+. In: Proceedings of WISE (Web Information Systems Engineering), Lille, France. Workshop on Web Usability and Accessibility. Springer, Heidelberg (2007) 4. Hochheiser, H., Baehrecke, E.H., Mount, S.M., Shneiderman, B.: Dynamic Querying for Pattern Identification in Microarray and Genomic Data. In: Proceedings of IEEE International Conference on Multimedia and Expo (2003) 5. Javahery, H., Seffah, A., Krishnan, S.: Beyond Power: Making Bioinformatics Tools UserCentric. Communications of the ACM - Special Issue on Bio-Informatics 47(11), 58–63 (2004) 6. Bolchini, D., Finkelstein, A., Perrone, V., Nagl, S.: Better Bioinformatics through Usability Analysis. Bioinformatics 25(3), 406–412 (2009) doi:10.1093/bioinformatics/btn633 7. Broughton, V.: Faceted classification as a basis for knowledge organization in a digital environment. The New Review of Hypermedia and Multimedia, 67–102 (2001)
Run-Time Adaptation of a Universal User Interface for Ambient Intelligent Production Environments Kai Breiner1, Daniel Görlich2, Oliver Maschino1, Gerrit Meixner3, and Detlef Zühlke2 1
Software Engineering Research Group, University of Kaiserslautern, 67663 Kaiserslautern, Germany {Breiner,Maschino}@cs.uni-kl.de 2 SmartFactoryKL, P.O. Box 3049, 67653 Kaiserslautern, Germany {Daniel.Goerlich,Detlef.Zuehlke}@DFKI.de 3 German Research Center for Artificial Intelligence (DFKI), 67663 Kaiserslautern, Germany [email protected]
Abstract. The SmartFactoryKL is an arbitrarily modifiable and expandable (flexible) intelligent production environment, connecting components from multiple manufacturers (networked), enabling its components to perform context-related tasks autonomously (self-organizing), and emphasizing userfriendliness (user-oriented). This paper presents the results of a research project focusing on the run-time generation and adaptation of a universal task-oriented user interface for such intelligent production environments. It employs a Roombased Use Model (RUM) developed in the context of a continuing research project series on universal remote control devices for intelligent production environments. The SmartFactoryKL is the first ambient intelligent production environment for demonstration and development purposes worldwide. After three years of research, a first prototype has been finished that allows for controlling the production line using a single remote user interface able to adapt to varying remote devices according to the actual context of use, in a complex, model-based approach. Keywords: MBUID, Model driven development, generating user interfaces, modeling, adaptable user interfaces.
vision of Mark Weiser [12] concerning ubiquitous computing – also in production environments – is becoming a reality. Besides the many different benefits offered by smart technologies, there are also drawbacks. One main drawback is the fact that the number and complexity of technical devices, their user interfaces, and their usage situations in industrial production environments are constantly growing. In today’s production environments, technical devices often stem from multiple vendors with different user interfaces differing in complexity, look&feel, and interaction styles. Such highly complex and networked technical devices or systems can provide any information at any time and in any place. This advantage can turn out to be a disadvantage when information is not presented properly according to the users’ needs. This leads to problems, especially concerning the usability of the user interface. The level of acceptance of a user interface largely depends on its ease and convenience of use. A user can work with a technical device more efficiently if the user interface is tailored to the users’ needs, on the one hand, and to their abilities on the other hand. Therefore, providing information in a context- and location-sensitive manner (depending on user, situation, machine, environmental conditions, etc.) has to be ensured. To reduce the usage complexity of user interfaces and improve their usability, one of our goals is to adequately support users in performing their tasks by interacting with a user interface. Therefore, the particular user interface has to be adaptable to different usage situations – definable, for example, by user, task, interaction device, and device functionality. The increasing complexity due to technological development will be reduced by using a model-based approach for the generation of user interfaces [9]. The core model of a model-based approach focusing on user-centered development is often the task model of a user interface. A task model describes the tasks a user wants to perform in a system. One comprehensive task model is the Use Model, which integrates detailed information about the tasks, e.g., temporal relationships, conditions, or task types [7]. The Use Model is formalized through the XMLbased Useware Markup Language (useML). For describing Use Models in ubiquitous environments, the Use Model needs to be extended to include the integration of spatial information, which leads to the Room-based Use Model. First evaluation results have been obtained in the SmartFactoryKL, our testbed for future production environments, which is located in Kaiserslautern, Germany. The remainder of this paper is structured as follows. Section 2 describes the Roombased Use Model on the basis of an enhanced version of useML as well as the function model. Section 3 introduces the model-driven generation process, the interpreter, the adaptation mechanisms, and the first prototype developed. In section 4, we conclude and provide an outlook to the future.
2 The Room-Based Use Model (RUM) The Room-based Use Model (RUM) is a partial model focusing on the tasks of users and the way they fulfill tasks using multiple devices in complex, highly instrumented environments. In the following, we will describe the enhancement of the original useML and the structure of the function model, which is necessary for the automatic generation process that we will elaborate later.
Run-Time Adaptation of a Universal User Interface
665
2.1 Enhancing useML useML was originally introduced by Reuther [11] (see Fig. 1) to formalize several user groups’ task models into a single model. The semantic of the first version of useML was later enhanced substantially [7]. Such a model stores all potential variations of all user groups’ approaches to achieving desired goals. This model can then be instantiated at any time to automatically adapt the respective device’s user interface to perfectly fit the current user’s tasks and needs.
Use Model
Use Object
Elementary Use Object
change
trigger
select
enter
inform
Fig. 1. Use Model structure according to Reuther. [11]
useML was restricted to static user interfaces and to single devices or device families only. It has been extended by a hierarchical structure of – logical (organizational) or physical – rooms containing device compounds that themselves can comprise other device compounds or devices (see Fig. 2.). Thereby, whole business processes can be represented – from a human-machine interaction perspective – in an RUM. While the RUM provides merely structural elements to specify spatial and device hierarchies, it further allows adding, for example, device profiles, coordinates, and interaction zones. It also provides means for modeling interactions between devices and for defining common Use Models for groups of devices, among other things. In addition to the hierarchical task structure common in task modeling languages (see [8]), the RUM also comprises modeling tools common in software engineering (activity diagrams) and provides support for the application of usability patterns. Still, complex tasks can be refined into less complex and finally elementary tasks (here: elementary use objects) in the classical, hierarchical way. RUMs can be extended by additional formal elements and sub-structures. When needed, they are supplemented with user models, usage situation models, or other (semi-)formal context representations. An RUM representing the SmartFactoryKL was complemented by a function model linking user tasks with data sets of the wireless communication protocols of the SmartFactoryKL development and demonstration facility. By using a wireless, mobile interaction device, we were able to automatically generate fully functional user interfaces. The used function model will be presented in detail in the next subsection. However, since [6] has shown that a model of human-machine interactions must consist of, at least, a task, a dialog, and a presentation model, we needed a mapping
666
K. Breiner et al.
Fig. 2. Integrated Room-based Use Model, containing contextual information about the entire environment, as well as all interactional information about the tasks to be performed by users. [3,4,5]
between tasks and user interface objects (see Section 3) for automated user interface generation and adaptation at run-time. Section 3 will show the feasibility of our approach of combining these models. 2.2 Function Model The central idea of the function model is to create a linkage between a user interface and the application logic based merely on a given RUM. One major challenge we explored in our previous work was to automatically interface the application services while generating the user interface [1]. For this, we formalized what is communicated between the interaction device and the device to be controlled and how. This extension makes it possible to bind the application logic to the individual graphical elements with an UI generator in a completely automated way. These models were elicited on the basis of the PROFIBUS implementation [10] used in the demonstration environment, as in the case of many current production environments. Therefore, it is important to mention that this kind of communication is bi-directional: Once the connection to the target device is established, communication frames can be exchanged cyclically.
Run-Time Adaptation of a Universal User Interface
667
Fig. 3. A device compound node provides a function model consisting of connection information as well as the structure of the data to be transferred
With respect to the structure of the RUM (see Fig. 2.), we created extensions on two different levels of abstraction. First, in order to establish communication between two devices, we needed information about the host to be addressed and the communication channel – this information is device-specific. Second, we needed to know how the content of the communication has to be structured in order to be understood by its receiver. Due to the fact that in our application domain, the type and channel of communication can vary depending on the type and manufacturer of the device, we attach this specific information to every device compound node. As shown in Fig. 3, the function model consists of the nodes connection and data. The structure of this model was elicited from several sample projects and implemented with respect to the uniform resource identifier (URI) standard [2]. Consisting of scheme, host, data-reference (data-structure, see below), device number, device type, and priority, this information is sufficient for an interpreter to establish a communication channel to this particular device.
Fig. 4. Connection is a description of the structure of information needed to establish the connection to the desired device
Additionally, it is important to know how to communicate with the respective target device and, therefore, how the transferred data needs to be structured in order to be understood by this device. The PROFIBUS protocol stipulates that communication between devices is, by definition, message-based. Therefore, the content of these messages is embedded into a clearly defined structure, which depends on what kind of information is to be communicated. Fig. 5 shows that the data node consists of distinct structures for incoming and outgoing datasets. Analogously to the protocol, one dataset is composed of a position (of the data within the sent/received frame), a length (of the information within this frame), a (unique) identifier, and a defined data format. Additional, but not compulsory, information might be the measurement unit (e.g., gallon, liter, Celsius, Fahrenheit), the conversion factor (if the data needs to be post-processed),the min/max (possible) range, significant digits, and a status message.
668
K. Breiner et al.
Fig. 5. According to the direction of the communication, the data node distinguishes between outgoing datasets and incoming datasets. A dataset consists of several necessary properties.
Fig. 6. Each instance of an elementary use object possesses an assignment node containing a link to a specific function
Using this structure, an interpreter will be able to encode the information received from a user’s interaction in a way the device can understand and vice versa. The RUM already supports extensions according to assignments on the level of elementary use objects. We equipped each assignment node with the ability to attach a function node pointing to a particular dataset structure, by using the unique identifier (see Fig. 6). Hence, if an elementary use object is activated by the user, the interpreter knows how to encode the given data according to the dataset structure. Since data structures are assigned to a particular device compound that provides a connection node, the interpreter can send this encoded information.
3 Generation and Adaptation of the User Interface As a result, the RUM contains information about the user interaction with all devices of the environment as well as information about how to interact with the
Run-Time Adaptation of a Universal User Interface
669
corresponding application services, if triggered by the user – which is sufficient for automatically generating a functional user interface. To demonstrate the feasibility of this model-driven approach, we set up an interpretation process and implemented a basic prototype generator. Prior to this, we formalized our demonstration environment – the SmartFactoryKL – as an RUM. From this model, we derived instances that comprise certain subsets of the demonstration environments or certain user roles such as a guest user role restricted to read-only access to the modeled devices. In the following sections, we will elaborate the process/interpreter, the prototype in the demonstration environment, and our idea of an adaptation mechanism at run-time supported by this model. 3.1 The Generation Process and Interpreter The aim of this process is to visualize the modeled environment in a single user interface, which will provide control over devices present in the environment. In case of the prototype, we use a simplified presentation model in which the available device compounds and devices will be displayed as a slim navigation structure in a canonical mapping and the use objects will group the functions of the underlying elementary use objects. To activate a certain device (and therefore a certain Use Model), the user selects the respective device in the navigation bar. Concerning the interpretation of the selected Use Model, let W be the set of available user interface widgets (such as labels, buttons, text fields, etc.) the interpreter will be able to use in order to compose the final user interface. On the other hand, let EUO be the set of elementary use objects and elementary use objects in compound configurations. The function m:EUO → W describes the relation of how the interaction objects are mapped onto real user interface widgets that are displayed and can be manipulated by the user. This function needs to be formalized in the implementation of the interpreter; in the case of our prototype, it has to be hard-coded. The interpreter will map all (elementary) use objects to a dedicated visual use object (graphical widget). In addition, the interpreter should not only be able to control devices that are present, but also be adaptive – e.g., hiding the user interfaces of devices that are described in the RUM, but are not really present in the environment. 3.2 Adaptation According to the RUM, let DM be the set of devices defined in this model – the devices (device types) the interpreter is able to cope with. Let DP be the set of devices present in the environment to be addressed. Available devices are found by scanning for Bluetooth devices, and by querying known Programmable Logic Controllers. Then, the interpreter has to create the intersection between these sets to provide a functional user interface. Hence, DD := DM ∩ DP is the set of devices that can actually be controlled with the generated user interface and consequently will be provided by the interpreter. At this moment, we have an adaptation mechanism for providing a usable user interface that is a direct result of the generation process. Violating this basic type of
670
K. Breiner et al.
adaptation might result in usage-related errors. Assuming DM/DP ≠ ∅ as well as DP/DM ≠ ∅, if only DM were to be displayed, the user interface would provide functionality for devices that are not present, and if only DP were to be displayed, the interpreter would have to generate user interfaces without knowing how to interface the application logic. Another adaptation feature supported by the RUM is adaptation according to the role of the user. Analogously to the function model, the user model provides user restriction definitions on the level of the Use Models, attached to each device compound. These definitions result in a restricted view on the Use Model and therefore on the user interface to be displayed. In our scenario, we currently assume that a certain role applies to the complete usage lifecycle of the generated user interface. Thus, DD' = view(DD, userrole) is the restricted view on the user interface to be displayed according to the actual user role. According to the mapping function defined earlier, the final user interface is the result of the function m(DD’). In the next iteration of our prototype, which is described below, we will integrate more and different implementations of these adaptation mechanisms in order to evaluate them. Due to the fact that this is still research in progress, we have only shown the feasibility so far. But it will be very interesting, of course, to evaluate the influence of different adaptation mechanism on the user experience. 3.3 The Prototype The prototype was implemented in Java and the generator composes the user interface using elements of the swing library. Xmlbeans was used to create a simple interface to the XML structure of the source files. The wireless communication infrastructure in the SmartFactoryKL mainly relies on Bluetooth connections, so it was necessary to include a Bluetooth stack in the implementation, which was BlueCove. According to the function model, we included Bluetooth-specific communication information in the connection node, enabling the generator to integrate Bluetooth calls in the action events of the interaction widgets.
Fig. 7. SmartFactoryKL – the comprehensive user interface running on the PaceBlade touch screen (foreground) is used to steer devices of the production environment (background)
Run-Time Adaptation of a Universal User Interface
671
The software runs on a PaceBlade Slimbook P110 TabletPC. It permanently queries the RUM file for changes, in order to let the generator adapt the user interfaces immediately whenever the model is being changed. Fig. 7 shows the generated user interface in the demonstrator environment.
4 Conclusion and Future Work In this paper, we elaborated the extension of an existing model-based approach to meet the requirements of user interface generation and adaptation in intelligent production environments at run-time. The newly developed Room-based Use Model (RUM) is capable of providing a description of the entire environment, comprising all devices present as well as their user task models. Additionally, we integrated a generic function model, which contains information about the type and way of communication with the application logic, if triggered by a user event. In order to deal with highly dynamic environments, adaptive user interfaces are necessary in order to prevent usage errors. These models were the basis of an automatic model-driven user interface generation process. In our sample production environment, the SmartFactoryKL, we showed that the information provided by these models is sufficient for creating a functional user interface at run-time. We were also able to integrate basic mechanisms to ensure the adaptability of the user interface to the environmental configuration as well as to the user – which means that the user interface always reflects the configuration of the devices present. A next step will be to conduct several evaluations using this first prototype. Since the intention was to support the user in interacting with a heterogeneous set of devices (a range of UIs from different vendors providing different user experiences, located in various places, etc.) and to reduce human errors while improving the users’ workflow, one goal would be to measure the effectiveness of interacting with our prototype compared to the previous situation of distributed heterogeneous user interfaces. It would be interesting to evaluate how easily users can handle adaptable user interfaces, even if those reflect the current physical configuration. What is the influence on the user experience? Should the adaptation be performed completely automatically or is it possible to integrate the user into this process, improving the level of acceptance? What is the adequate level of user integration: e.g., adaptation only, information, or mutual adaptation? Acknowledgments. This work was funded in part by the German Research Foundation (DFG).
References 1. Adam, S., Breiner, K., Mukasa, K., Trapp, M.: Challenges to the Model Driven Generation of User Interfaces at Runtime for Ambient Intelligent Systems. In: Proceedings of the Workshop on Model Driven Software Engineering for Ambient Intelligence Applications, European Conference on Ambient Intelligence, Darmstadt, Germany (2007) 2. Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifiers (URI): Generic Syntax. RFC. RFC Editor (1998)
672
K. Breiner et al.
3. Breiner, K., Maschino, O., Görlich, D., Meixner, G.: Towards automatically interfacing application services integrated in an automated model-based user interface generation process. In: Proceedings of the Workshop on Model Driven Development of Advanced User Interfaces, 14th international Conference on intelligent User interfaces IUI 2009, Sanibel Island, Florida, USA (2009) 4. Görlich, D., Breiner, K.: Useware modelling for ambient intelligent production environments. In: Engels, G., Opdyke, B., Schmidt, D.C., Weil, F. (eds.) MODELS 2007. LNCS, vol. 4735. Springer, Heidelberg (2007) 5. Görlich, D., Breiner, K.: Intelligent Task-oriented User Interfaces in Production Environments. In: Proceedings of the Workshop on Model-Driven User-Centric Design & Engineering, 10th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine-Systems, Seoul, Korea (2007) 6. Luyten, K.: Dynamic User Interface Generation for Mobile and Embedded Systems with Model-Based User Interface Development, Ph.D. thesis, Limburgs Universitair Centrum, Transnational University Limburg: School of Information Technology, October 21, Expertise Centre for Digital Media, Diepenbeek, Belgium (2004) 7. Meixner, G., Seissler, M., Nahler, M.: Udit – A Graphical Editor for Task Models. In: Proceedings of the Workshop on Model Driven Development of Advanced User Interfaces, 14th international Conference on intelligent User interfaces IUI 2009, Sanibel Island, Florida, USA (2009) 8. Meixner, G., Görlich, D.: Eine Taxonomie für Aufgabenmodelle. In: Proceedings of Software Engineering (SE 2009), Kaiserslautern, Germany (2009) 9. Myers, B., Hudson, S., Pausch, R.: Past, present, and future of user interface software tools. In: ACM Transactions on Computer-Human Interaction (TOCHI), pp. 3–28. ACM Press, New York (2000) 10. PROFIBUS Protocol, http://www.profibus.com/pb/ (last visited 02.02.09) 11. Reuther, A.: useML – Systematische Entwicklung von Maschinenbediensystemen mit XML (Ph.D. thesis) In: Fortschritt-Berichte pak. vol. 8, University of Kaiserslautern, Germany (2003) 12. Weiser, M.: The computer for the 21st century. Scientific American 265(3), 94–104 (1991)
Heuristic Evaluation of Mission-Critical Software Using a Large Team Tim Buxton, Alvin Tarrell, and Ann Fruhling University of Nebraska Omaha, Peter Kiewit Institute - Office 174C 6001 Dodge Street Omaha NE 68182 {ebuxton,atarrell,afruhling}@unomaha.edu
Abstract. Heuristic evaluation is a common technique for assessing usability, but is most often conducted using a team of 3-5 individuals. Our project involved a team of 16 stakeholders assessing usability of a mission-critical decision support system for the US military. Data collected from so many evaluators could easily become overwhelming, so we devised a method to first filter evaluations based on agreement between evaluators, and then further prioritize findings based on their individual Frequency, Impact, and Severity scores. We termed our methodology the ‘Integrated Stakeholder Usability Evaluation Process,’ and believe it will be useful for other researchers conducting similar research involving heuristic evaluations with large groups. Keywords: Usability Evaluation Methods, Heuristic Evaluation, Decision Support.
number of errors, enhanced accuracy, a more positive attitude on the part of the user toward the target system, and increased usage of the system by the user [3]. Cummings and Guerlain [4] describe concerns for cognitive load and time pressures in use of mission-essential and time-critical software, a telemedicine system. Usability problems which increase cognitive effort or increase time to complete tasks can lead to failures of system effectiveness. As a decision support tool with access to large volumes of current information, the system must also function well in Information Retrieval (IR) as described by Xie [5], but as she notes, strategies for IR may change when there is extreme urgency. The system being evaluated shared these characteristics, driving a change from standard usability testing methodologies. This unique environment necessitated going beyond standard usability testing. We wanted to utilize a User Interface Evaluation (UIE) tool that would contribute to finding usability problems in our situation, where cognitive load and urgency or operational tempo are such major issues Our research method, given below, draws on experiences of others with similar usability needs, and led to some significant results that we describe. The contribution of this study is the repeatable process that we developed to effectively capture feedback from various key stakeholders on the quality of the system we studied. We adapted the heuristic evaluation method and broadened the scope to include UI guidelines established by the CCDS development team. We felt it was the best method, and had in fact been found by Cummings and Guerlain to predict usage problems even for applications used under time pressure [4]. The process is more fully explained in the following sections.
2 Background Developers create applications providing the required information and functionality, but developers often don’t know how to present it in the most intuitive (to domain experts), usable way. Domain experts who use programs may be frustrated by what is in fact a usability problem, but don't realize how easily it could be remedied if identified [6]. Human-Computer Interaction (HCI) professionals have developed systematic ways to bridge this disconnect and improve application usability. When application usability is improved, substantial benefits result to the domain expert users [6]. These come from improved learnability, visibility, user control, error prevention and recovery, and speed of task completion (efficiency). The problem we set out to solve was: “How can domain experts, HCI professionals, and IT personnel collaborate to find usability problems and make an existing application more usable by applying collective knowledge that no one person possesses?” A second question, given that the evaluation team now consisted of 16 diverse stakeholders, was: “How can we accommodate the large amounts of data generated, how can we define agreement between reviewers, and how can we rank order our findings?” Our Integrated Stakeholder Usability Evaluation Process was designed to help provide a solution to these questions. The system studied will be referred to as the Command and Control Decision Support (CCDS) System. CCDS is a strategic information system intended to enhance
Heuristic Evaluation of Mission-Critical Software Using a Large Team
675
command and control of critical military units, providing national-level government leaders and military commanders the capability to monitor the status of critical forces, to make and implement decisions, and to be aware of potential threats. CCDS command and control capability relies on information made available by several reporting systems. CCDS-processed information forms the basis for decisionmaking and for implementing decisions about command and control functions. The primary CCDS users are the decision advisors; i.e. the members of the military operations team providing advice to senior decision makers. They obtain CCDS data and information, integrate those with relevant material from other sources, and present the resultant knowledge and recommendations to the key decision makers.
3 Research Method We selected the heuristic evaluation method as being most suitable because it can be used for existing systems, takes little user time, and is relatively quick and inexpensive. Tang et al. [7] have also demonstrated that heuristic evaluation is able to point to usability problems in a system that required task performance in a timely fashion, leading further credence to its use in this situation. Jeffries et al. [8] describe heuristic evaluation as having a user interface expert or a group of experts with knowledge of good user interface design principles study an interface and, based on experience and training, identifying potential areas of difficulty. The evaluators are generally experts in usability, although it is often desirable to use individuals who are both usability and domain experts [9]. They study the interface in depth and look for properties that they know, from experience, will lead them to problems. The idea is that while no one individual assessor will find all the violations of the heuristics, several expert evaluators working independently may be very effective. The end result of the evaluation is a list of problems or conflicts with the associated heuristics referenced [10]. When all the evaluators have finished their evaluation they will aggregate their list of problems [11]. The heuristic evaluation method is often selected because it is a cost effective method for an organization that does not have the facilities, time, and expertise necessary to do exhaustive usability engineering.
4 Heuristic Evaluation Process The Research Team followed a new framework for interface usability evaluation which consisted of six steps detailed in Figure 1 below, starting with the selection of the usability evaluation method and ending with a set of prioritized improvement recommendations. The support tools selected for the usability evaluation were simple tools that were used to manage the process and collect data, in effect extracting the consensus of the group. This allowed for easy and quick iterative refinement throughout the process. We discuss the six steps next.
676
T. Buxton, A. Tarrell, and A. Fruhling
Phase 1 – Selection of Usability Evaluation Method Phase 2 – Evaluation and Modification of the Heuristic Evaluation Tool
rod dP
Phase 5 – Development of General and Specific Findings
Re fin e
d ve ng pro di Im rstan de Un
Phase 4 – Data Reduction and Preliminary Analysis
uc t
Phase 3 – Heuristic Evaluation of CCDS
Phase 6 – Prioritization and Generation of Recommendations
Fig. 1. Heuristic Evaluation Framework
Phase 1 – Selection of Usability Evaluation Method We were asked to concentrate on the usability of the user interface, and the heuristic evaluation technique works well for that purpose. This approach also did not infringe significantly on end-users’ time commitments. However, it is still a thorough, comprehensive process, and it is cost effective and could be completed in a timely manner. Phase 2 – Evaluation and Modification of the Heuristic Evaluation Tool Nielsen’s Ten Usability Heuristics [12], derived by Jakob Nielsen from a factor analysis of 249 usability problems, were used as the basis of the study. We also included an additional three heuristics identified by Denise Pyrite, Xerox Corporation [13], adding 43 evaluation items. Finally, we also incorporated 37 CCDS-specific requirements contained in the Graphical User Interface guide written specifically for CCDS (CCDS Java User Interface Standards). The resulting thirteen heuristics contained a total of 329 individual evaluation items from a combination of these three sources. Modifications to the heuristics themselves were also made, including changing the ‘Yes/No’ structure of the questions to a more appropriate 7-point Likert scale to afford the evaluators some flexibility in determining not only whether system had met the requirements of a particular heuristic, but also how well the system met that requirement. This also provided the opportunity to do more quantitative analysis of the resulting data. Phase 3 – Heuristic Evaluation of CCDS This phase began by discussing the meaning of the heuristic and how it might apply to CCDS, then making a determination of whether or not the heuristic should be adopted and whether or not it should be specifically included in the CCDS Java User Interface Standards. If the checklist question was deemed applicable, each evaluator entered a rating on how well CCDS enforced the checklist question. Another column was added for comments by each evaluator that indicated where the system violated the question
Heuristic Evaluation of Mission-Critical Software Using a Large Team
677
and also where the system could be improved. The evaluation process was fully explained to the evaluation team at the start of the research, with a short reminder session provided at each meeting. We also created a comprehensive numbering system for items on the user interface menu tree, e.g. 1.1.2, that included all possible menu selections. This provided a “common format for documentation” as recommended by Koutsabasis, et al. [14]. Because of the mission-critical nature of the software, we wanted as many reviewers as possible. The team consisted of 16 individuals, although not all were able to complete all 13 heuristics. We were able to get 10 reviewers for many heuristics, which should be expected to detect 85% of usability problems, as opposed to 60% for 3 reviewers, according to Nielsen and Landauer [15]. The CCDS Usability Evaluation Team evaluated the system primarily in group sessions. This allowed us to discuss examples of success or failure, and in some cases decide that a heuristic did not apply. If the heuristic was adopted and applied, we navigated through the system evaluating how well the system complied. Each evaluator entered his or her rating independently and also added comments to explain the rating, especially if the system did not comply with the heuristic. The CCDS application was projected for all to view during most of these discussion sessions, and each person also had the CCDS application available at his or her desktop. The team also captured other enhancement ideas that were generated during the analysis and discussion. Sixteen individuals completed some portion of the evaluation, with six individuals completing all 13 heuristics and the remainder completing the heuristics to varying degrees. A total of 100 person-heuristics were ultimately completed, or an average of 7.7 evaluations per heuristic. The process we followed is shown in diagrammatic form in Figure 2. Phase 4 – Data Reduction and Preliminary Analysis After the evaluations were completed, the data were migrated from the Excel spreadsheets into a Microsoft Access database for query-based analysis and ease of report generation. Forty-six of the original 329 checklist items were rejected during the evaluation sessions as not being applicable to CCDS, so a final set of 283 items were analyzed. Preliminary evaluations for inconsistencies (standard deviations) were first conducted, and then items were first prioritized based on the mean scores. This research produced 2,612 data points (individual heuristic question items rated) and 944 separate comments from the evaluators. Average heuristic ratings and standard deviations for each of the 283 items were calculated and various reports were generated for analysis. Some of the reports utilized were: Adopted Heuristics – Complete listing of the heuristics adopted by the research team, sorted by category and item number taken from the original Xerox tool. Additional CCDS User Interface Guidelines were added in under the appropriate heuristic category, on the same line as the Xerox heuristic if it coincided, and on a new line if it did not map to a Xerox heuristic. Average Ratings by Heuristic – The Adopted Heuristics report, with the addition of average CCDS ratings as assigned by the research team
678
T. Buxton, A. Tarrell, and A. Fruhling
Adopted Heuristics
Average Rating by Heuristic
Low Ratings High Agreement
Low Ratings Low Agreement
Discussion and Clarification
Frequency, Impact, Severity Assessment
Prioritization of Heuristics
Draft Recommendations
Final “Top 21” Usability Improvement Recommendations
Fig. 2. Steps in the Integrated Stakeholder Usability Evaluation Process (ISUEP)
Low Ratings, High Agreement – A subset of the Average Ratings by Heuristic report, filtered to show only those items that received an average score of less than five on the seven point rating scale with sufficiently low standard deviation as to indicate agreement between the raters. Disagreement Between Raters – A listing of adopted design heuristics, regardless of average rating, where the standard deviation between the ratings was greater than two. This report was used as a review and revision tool by the research team to identify areas where there might be differences in interpretation or experience with the CCDS system.
Heuristic Evaluation of Mission-Critical Software Using a Large Team
679
Phase 5 – Development of General and Specific Findings Using the “Low Ratings, High Agreement” report as a starting point, the team identified 70 items that warranted further examination. These were, in general, items that scored an average score of ‘5’ or less on the 7-point Likert rating scale. A more quantitative analysis was then undertaken to help identify priority items within the group of 70. Each of these items was scored based on how often it occurred (Frequency), how many users were affected (Impact), and the difficulty of recovering from or overcoming the problem (Severity) as described by Nielsen [16]. This discussion compensated for the tendency to produce false positives by HE noted by Hornbaek and Frokjaer [17] by dismissing deviations which had only minimal impact. Relative rankings of the 70 items of interest were developed based on these scores (with a double weighting of the Severity factor), and a prioritized list of potential action items began to take shape. The team was reduced at this point to the three key researchers and the development team representative; we felt this was necessary to allow the group to better focus on the rankings without facing debate among 16 separate evaluators. Recommendations for corrections and improvements were then developed, often using the original heuristic as a style guide, as well as incorporating CCDS-specific language and ideas. The team also reexamined those heuristics which were just "below the line" for initial consideration, either by average score or by standard deviation. Phase 6 – Prioritization and Generation of Recommendations Based on the process described above, the evaluation team identified a prioritized list of twenty-one recommendations to present to the CCDS managers and development team. That resulted in the generation of a summary report: Final Usability Improvement Recommendations – A “Top-21” list of recommendations presented in order of importance based on of the highest average perception of positive impact to CCDS system usability.
5 Results This research produced 2,612 data points or individual heuristic ratings and 944 comments. Ranking the least followed heuristics by percentage of items in the Top 70 by Heuristic Category, we found the most problematic heuristic was Consistency and Standards, followed by Flexibility, Error Prevention, Flexibility, and Error Recognition and Recovery. We observed a number of surprising benefits in our project that can be expected in any heuristic evaluation. Usability professionals initially constitute novice users, and so can be useful in finding misleading parts of the UI. Landauer [18] points out that testing with novices is important to good user centered design, and in fact we confirmed this by several catches of usability problems by novices that had escaped experienced users. For example, new users noticed that the function of icons was not clear to new users, and they needed to indicate their function more intuitively and make them more distinct. Some boxes which were shaded as if not selectable were in fact selectable – new users were good at noticing this sort of usability problem.
680
T. Buxton, A. Tarrell, and A. Fruhling
We also observed that our sessions led to general comments and ideas which were captured with a list of Extra Comments not tied to specific heuristics details. Evaluation sessions often led to serendipitously finding functional errors in the new revisions of the software as well, leading to generation of software deficiency reports. For example, when all fields were left blank on a report design form, the database crashed. So it is reasonable to point out these bonuses to the organization as costs of the method are assessed. Formal results provided to CCDS included a prioritized list of the ‘Top-21’ usability items that the evaluation team thought should be addressed. The report also included usability scores and comments for all 283 evaluation items, allowing the development team to use this information as they saw fit. The developer in our team showed increased awareness of usability best practices as a result of his participation. We would expect that this leads to better usability being built in, as was also observed by Tang, et al. [7]. Most importantly, however, we developed and documented a repeatable process that can be used for usability evaluations using large numbers of evaluators. Our methodology allowed us to quickly assess inconsistencies in evaluations among the different inspectors, look for evaluations with simultaneous low scores and high agreement among the evaluators, and then quickly prioritize those findings based on frequency, impact, and severity.
6 Implications and Conclusion Our first question of how to apply collective knowledge that on one person possesses to improve usability, we solved in two ways. One was the rating data collection system, including the comments. The small group reading the comments and ratings had access to a lot of pooled collective knowledge to use in usability improvement. An unexpected result of doing the evaluations in group sessions was that discussion always led to participants hearing about problems or perspectives that were “new” to them, so the collective knowledge was increased. We recommend the group sessions for just that reason, over evaluation by individuals asynchronously, if that is possible. Heuristic evaluators can produce divergent and varied results. Our contribution was to create analytic reports to extract the consensus of the large group that we had engaged to evaluate usability. These reports were then very productive in guiding the discussion of a small team tasked with finding the low-hanging fruit in the usability improvement process. Our assignment of a menu tree numbering system to the application was found to be vital in clarifying discussion of which exact parts of the large application showed specific usability problems, and in communicating to the development team exactly what needed to be done to improve usability. Heuristic evaluation is just one method for user interface usability evaluation. We developed the Integrated Stakeholder Usability Evaluation Process as a means of dealing with a large group of diverse evaluators; this process is designed to quickly identify items with strong user agreement and with high priority for correction. We recommend that additional usability evaluations - cognitive walkthroughs, task analysis, and user observations – also be considered. These future evaluations could be conducted as stand-alone tests, or could possibly be undertaken during normal CDSS
Heuristic Evaluation of Mission-Critical Software Using a Large Team
681
operations. This would require a minimal commitment of user time, and allow “real life” evaluations, potentially leading to valuable improvements. Heuristic evaluation is just one strategy for user interface usability evaluation. We recommend that additional usability evaluations be considered such as cognitive walkthroughs, tasks analysis, and user observations. Task completion times, user errors and other evaluations and suggestions could be potentially built in to ongoing routine exercises and after-action evaluations of performance. Applications are complex, and we found the menu tree numbering system was important to add to clarify both our discussions in evaluating the software, and later for software developers to find and fix the exact problem, without having to relocate it. Future directions for research might include evaluating the measures of usability problem impact. Are frequency, impact and severity the best scales to use, and how is it best to weight them against each other? Are there other measures of impact for heuristic evaluation that would be of more value in time-critical command and control applications, perhaps prioritizing serious errors, and how likely they are and how difficult to recover from quickly? Also, in large applications, coverage is difficult to assess – even a larger number of evaluators have a limited time, and some areas of the application may still not get tested. There may be ways to improve coverage that could be found in future work. We applied all of the heuristics for thoroughness. However, we would expect that in a time-critical decision support system such as this, certain heuristics would be found to be more likely to find usability problems that show up during time-critical operation, and time could be saved by applying only those found to be most critical. These would likely be heuristics that add to intuitive “information scent” as described by Spool et al. [19] and “information foraging” as described by Pirolli [20], as being crucial to quickly giving users intuitive guides to where to go to accomplish their intended tasks. Nielsen’s Match Between Systems and the Real World heuristic might be expected to be key here. Improvements in intuitive usability, in addition to providing more efficient and accurate task completion, might even lead to less training being necessary. Feedback from students during training could be used to improve the intuitiveness of the design, and would be an avenue of further research. Acknowledgements. We would like to thank the participants in our heuristic evaluation sessions for CCDS at USSTRATCOM. Without their commitment the analysis would not have been as comprehensive or complete.
References 1. Nielsen, J., Mack, R.: Usability inspection methods. John Wiley & Sons, San Francisco (1994) 2. Shneiderman, B.: Designing the User Interface Strategies for Effective Human-Computer Interaction. Addison-Wesley, Reading (1998) 3. Lecerof, A., Paterno, F.: Automatic support for usability evaluation. IEE Transactions on Software Engineering 24, 863–888 (1998) 4. Cummings, M.L., Guerlain, S.: An Interactive Decision Support Tool for Real-time Inflight Replanning of Autonomous Vehicles. In: AIAA 3rd “Unmanned Unlimited” Technical Conference, Workshop and Exhibit, September 20-23 (2004)
682
T. Buxton, A. Tarrell, and A. Fruhling
5. Xie, H.: Interactive Information Retrieval in Digital Environments. IGI Publishing, Hershey (2008) 6. Nielsen, J.: Usability engineering at a discount. In: Smith, G., Salvendy, M.J. (eds.) Designing and Using Human-Computer Interfaces and Knowledge Based Systems, Elsevier Science Publishers, Amsterdam (1989) 7. Tang, Z., Johnson, T.R., Tindall, R.D., Zhang, J.: Applying Heuristic Evaluation to Improve the Usability of a Telemedicine System.1. Telemedicine and e-Health 12, 24–34 (2006) 8. Jeffries, R., Miller, J.R., Wharton, C., Uyeda, K.M.: User interface evaluation in the real world: a comparison of four techniques. Communications of the ACM, 119–124 (March 1991) 9. Kantner, L., Rosenbaum, S.: Usability Studies of WWW Sites: Heuristic Evaluation vs. Laboratory Testing. In: Proceedings SIGDOC, pp. 153–160 (1997) 10. Leventhal, L., Barnes, J.: Usability Engineering, Process, Products, and Examples, p. 216 (2008) 11. Preece, J., Rogers, Y., Benyon, D., Holland, S., Carey, T.: Human-Computer Interaction, p. 676. Addison-Wesley, Reading (1994) 12. Nielsen, J., Molich, R.: Heuristic Evaluation of User Interfaces. In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 249–256 (1990) 13. Weiss, E.: Making Computers-People Literate. Jossey Bass (1993) 14. Koutsabasis, P., Spyrou, T., Darzentas, J.: Evaluating Usability Evaluation Methods: Criteria, Method and a Case Study. In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4550, pp. 569– 578. Springer, Heidelberg (2007) 15. Nielsen, J., Landauer, T.K.: A mathematical model of the finding of usability problems.Amsterdam. In: The Netherlands Proceedings ACM/IFIP INTERCHI 1993 Conference (1993) 16. Nielsen, J.: Severity Ratings. Jakob Nielsen’s Website, http://www.useit.com/papers/heuristic/severityrating.html 17. Hornbaek, K., Frokjaer, E.: Usability Inspection by Metaphors of Human Thinking Compared to Heuristic Evaluation. International Journal of Human-Computer Interaction 17, 357–374 (2004) 18. Landauer, T.K.: The Trouble With Computers: Usefulness, Usability and Productivity, pp. 219–221. M.I.T. Press, Boston (1995) 19. Spool, J.M., Perfetti, C., Brittan, D.: Designing for the scent of information. User Interface Engineering, Middletown, MA (2004) 20. Pirolli, P.: Information Foraging Theory. Oxford University Press, New York (2007)
Interface Development for Early Notification Warning System: Full Windshield Head-Up Display Case Study Vassilis Charissis1, Stylianos Papanastasiou2, and George Vlachos3 1
University of Glasgow, Digital Design Studio, 10 Dumbreck Road, G41 5BW, Glasgow, UK 2 Chalmers University of Technology, Communication Systems and Information Theory, SE-412 96 Goteborg, Sweden 3 University of Glasgow, Department of Management, Main Building, G41 5BW, Glasgow, UK [email protected]
Abstract. This paper elaborates on the development of a prototype Head-Up Display (HUD) system designed to offer crucial navigation information to the driver, under adverse weather conditions. In particular the paper presents the implementation process and evaluation of the sharp turn notification and traffic warning cues which reflect some of the most common risks that may be encountered in a collision in a motorway environment under low visibility. Additionally, emphasis was placed on the prioritisation and effective presentation of information available through vehicular sensors, which would assist, without distracting, the driver in successfully navigating the vehicle under low visibility conditions. This information which appear in the form of symbolic representations of real objects, are projected in the vehicle’s windscreen and superimposed onto the real scenery. Overall the paper examines the potential benefits and occurring issues of the proposed HUD interface and presents the results of a large scale evaluation of the system on a group of forty users, as performed using a driving simulator. Keywords: HUD, HMI, Warning systems, Simulator, Driver’s Behaviour.
neighbouring vehicles and other objects are veiled from view and become unnoticeable. Under such unfavourable driving conditions the inability of in-vehicle notifications to effectively portray information increases the difficulty of the driving task. Furthermore, due to the attention seeking notifications of the various infotainment devices present inside the vehicle, the driver’s attention can be dispersed on fruitless gazing at the instrumentation panel dials, in the case of Head-Down Displays (HDD), as well as on discerning the hazy external scene. Hence if, for instance, one of the lead vehicles breaks abruptly, the driver does not have the required time and situational awareness to proceed in a collision avoidance braking manoeuvre. It would be, thus, fair to conclude that in some cases, notifications compromise rather than enhance the driver’s safety [2]. In the framework of human situational awareness, various studies have supported experimental Human-Machine Interfaces (HMIs) in an effort to tackle different issues by utilising customised Head-Up Display interfaces (HUD). In particular, previous observations have suggested that under low driving-load the attention required of the driver is considerably less, which in turn may reduce the driver’s overall awareness and subsequently encourage careless or reckless driving [3]. Based on our previous experience with regard to the design and evaluation of automotive HUDs as well as being aware of contemporary technological and costrelated constrains, we developed a series of interface components which enable the driver to anticipate potential hazards [4,5]. This paper introduces a novel design for an automotive HUD interface, which aims to improve the driver’s spatial awareness and response times under low visibility conditions with particular emphasis placed in early notification warnings of motorway hazards such as traffic congestion and out-of-view sharp turns. A working prototype of a Human-Machine Interface has been designed and implemented to fulfil these requirements.
2 Automotive Head-Up Displays Contemporary interface design efforts have targeted the dashboard (or instrument panel) as such an information conduit and have enriched its functionality with visual and audio warning cues from proximity systems [6]. Interestingly, a particular area of intense research focus has been the design and utilization of visual cues embedded in the vehicle's windshield, which effectively becomes a head-up display (HUD). Live trials have convincingly demonstrated that superimposing useful information on a fully operational HUD results in more rapid and stable driving responses compared to traditional instrument panels or Head-Down Displays (HDDs) [7,8,9,10]. Nonetheless, early examples of HUD devices and interfaces have somewhat failed to exploit in full the potential offered by the large-scale projection area of the windshield. Arguably, such issues are not the immediate consequence of limitations in the technology used but mainly derive as a result of the designers approach. Evidently, the focus of early research has been the development of technological features which would improve the performance of the “machine” element. The driver (or the human element) was largely delegated as a secondary consideration in those designs which resulted in an unbalanced, ineffective interface.
Interface Development for Early Notification Warning System: Full Windshield HUD
685
In other efforts, HMI designs have produced visually cluttered interfaces that aim to entail every available infotainment cue. These sporadic efforts have shown that there is significant utility in such devices when used with interface HUD projection on the vehicle’s windshield [8, 11]. Significantly, HUD projection has currently returned in the focus of automotive manufacturers, as the excessively burdened dashboards seem incapable of accommodating any future infotainment and navigation systems.
3 Proposed HUD Interface The proposed HMI design aims to identify the needs of the user in a potentially unsafe driving situation under adverse weather conditions. To this end it has been deemed necessary to categorise the incoming information according to significance for each given moment. Opting for minimalistic depictions of the incoming information, we have developed a group of symbols which are instantly recognisable by the drivers [12, 13]. Evidently, the collaboration between human and machine, could offer remarkable results as the machine can rapidly categorise the bulk of information and offer to the driver options between which to decide. In this paper, we are focusing in the development, implementation and evaluation of a specific group of early warning notifications related mainly with the traffic congestion and the sharp turns typically camouflaged in the terrain as depicted in Figure 1. The traffic congestion symbol aims to warn in advance for potential collisions that occur when leading vehicles rapidly decelerate perhaps as a response to traffic congestion along the road. The sharp turn symbol highlights certain parts of the motorway, such as junctions, intersections and hairpin turns, which can be exceptionally difficult to negotiate, particularly under adverse weather conditions.
Fig. 1. Traffic congestion and sharp turn situations (second scenario)
A real system implementation enabling the extraction of raw data which would make both notifications realisable, is currently under development with the employment of Vehicular Ad-Hoc Network systems (VANETs), GPS and road mapping software [14, 15].
4 Simulation Set-Up and Experimental Rationale The complete proposed HMI system has been evaluated in an Open Source Driving Simulator developed explicitly to measure drivers’ performance with the proposed
686
V. Charissis, S. Papanastasiou, and G. Vlachos
HUD interface and compare its effectiveness to traditional instrumentation techniques [16]. Notably, the preliminary user trials have demonstrated that the system conveys crucial information efficiently and in a non-distracting manner, thereby minimising the accident risk. A screenshot of the actual driving simulator illustrates the HUD interface appearance and the overall simulation environment. The driving simulator comprised a rear-projection screen 1.8m width by 1.2m height, driven from a single PC with two Intel Xeon 3.6GHz processors and a high-end graphics card (nVidia Quadro FX4400), a driver’s seat and Logitech steering wheel, gearbox and pedals.
Fig. 2. Screenshot from the actual driving simulation
4.1 Traffic Congestion and Sharp Turn Scenario The scenario designed for the evaluation of the aforementioned interface components recreated a traffic-congestion scene with 20 participating vehicles. Moreover, a traffic “bottleneck” was positioned in a blind turn under a bridge, which presented a significant accident risk as illustrated in the diagram of Figure 3. Our consequent evaluation of the proposed HUD interface aimed to determine the actual response time benefits derived through its usage and subsequently the real impact in the decrease of accident propensity. Additionally the number of potential collision avoidance manoeuvres could be utilised as another indicator of the system’s success or failure. 4.2 Users The driving simulation trials attracted 40 users which participated voluntarily. All the participants had driving licence and have been randomly selected in order to cover the widest possible array in professions and age and evenly distributed between males and females. This wide variety of drivers aimed to be in accordance with the “average
Interface Development for Early Notification Warning System: Full Windshield HUD
687
user” (or average driver) norm which has been coined [17] to describe the generic characteristics of a contemporary driver. The definition and clustering of this information aimed to develop a framework of European standards, which need to be addressed by any automotive system designed for operation by the driver. In fact, the proposed human-factors design guidelines [17] present a group of provisions for the rudimentary operation of any automotive system by any user that holds a driving license and falls approximately in the categories described in this nomenclature. Adhering to the above automotive industry dictum, the prototype HUD interface had to be compliant with the majority of the users for validation purposes. As such it was of outmost importance to define not only the structural characteristics of the system (i.e. projection distance) but mainly to identify how the “average user” would utilise the HUD interface and how such system might affect his/her driving under low visibility conditions. All the analysis within the main document followed the above guidelines.
5 Driving Pattern Analysis In this work, the proposed HUD interface presents the driver with vital information for collision avoidance. Notably, the specific scenario was designed to re-construct typical real-life accidents. Hence the “average user” was expected to misjudge the headway (HW) distance from the traffic congestion and perform last moment panic braking or collision avoidance manoeuvre [5, 16]. Even though this result was highly anticipated, as real life paradigms have unfortunately demonstrated in previous occasions [18], it was unclear how the average user would react with the use of the HUD interface. In this simulation the recorded video and data are suggestive of a typical reaction to the “traffic congestion and unexpected road turn” scenario as shown in Figure 3. Although, broadly, the drivers did not perform inadequately in comparison to the first scenario, the number of collisions that occurred was alarming. Again, analysis of the collision data brought to light a driver’s pattern. A particular user was sampled as he performed with typical reactions to the unexpected sharp turn and to the traffic congestion. Notably, the driver had driven through the second scenario using the instrumentation panel (HDD) and in turn with the assistance of the HUD. However, as mentioned above, the first and second scenarios with their variations (with and without HUD) were presented in a random order to the user in order to avoid any potential detection of similarities of the events. The graphs presented below focus on the last 10 seconds before the driver arrives at the traffic congestion.
Fig. 3. Traffic congestion and sharp turn situations (second scenario)
688
V. Charissis, S. Papanastasiou, and G. Vlachos
In particular Figure 4(a) illustrates the driver’s performance regarding the speed maintenance with and without the use of the HUD. From the graph it becomes apparent that the user maintained a higher speed than expected. The reason is that the neighbouring traffic in this scenario was sparse and gave little indication of the final traffic congestion 5km ahead. In close proximity and prior to the traffic congestion a sharp turn underneath a road-bridge formed another challenging element of the simulation. In the fifth second before the potential collision the driver approached the turn with considerably higher speed than ideal for effectively negotiating the curve. Realising that the vehicle could not follow the desired trajectory, the user braked instantly and immediately tried to steer the vehicle clear from danger.
Fig. 4. Graphs showing (a) the distance from the leading vehicle and (b) the driving speed, with and without the HUD
As a result the driver lost control of the vehicle and drifted vertically in the flowdirection. In a desperate second attempt to avoid the collision with the wall, the driver over-steered and headed, this time, towards the congested traffic, which he approached as he performed these different manoeuvres, as depicted by the curve increase between the ninth and tenth seconds before the collision with the traffic. Eventually the user’s vehicle stopped as it collided with the bridge wall and in turn with the motionless traffic vehicles. Observing the curve derived from the same user when the HUD was enabled, we can see that the user has a more orderly style of
Interface Development for Early Notification Warning System: Full Windshield HUD
689
driving. However the user is persistently driving at higher speeds than would be ideal under such conditions. Hence this repeated driving pattern characterises his driving behaviour and is not an “invented” attitude applied solely in the simulation environment. In fact, speeding of this sort is quite common; consider that accident statistics provide estimates that 70-85% of drivers exceed daily the speed limits [19, 20]. Critically, it was made clear to the participants that their driving behaviour or performance would not be graded during the trials. The evaluation focussed explicitly on the impact of the HUD on their driving performance. Further examination of this “average” user behaviour with the assistance of the HUD interface makes it evident that the user decelerates gradually at the first second mark. Possibly, the user is alarmed by the colour changing of the turn symbol that indicated a rapid approach to the sharp turn. Momentarily, the user then increases speed in order to approach faster the highlighted events. Notably the two indications (traffic and turn symbols) have a direct effect on the user’s behaviour resulting in a minor but constant deceleration. Importantly, the user completes the second simulation with the use of the HUD interface without colliding either with the side barriers or the congested vehicles. The interface’s contribution to this successful performance can also be detected in the driver’s unruffled reactions regarding the potential hazards that lay ahead. The HUD interface contribution was also highlighted by the percentage of collisions occurred with and without the use of the proposed system as Figure 5 illustrates. Notably the system reduced the collisions by 32.5%.
Fig. 5. Number of collisions recorded with and without the HUD interface
Crosschecking between metrics and videos had to be carried out in order to detect a driver’s common reactions and their impact on the vehicle’s manipulation and trajectory. As mentioned previously and as was pointed out by traffic police officers of the city of Strathclyde (whom we consulted during the course of this work), the above “average” user matches the profile of drivers that typically get involved in similar accidents. Hence it was crucial for this work to investigate the reactions, body posture, facial expressions, accelerating metrics, braking, manoeuvring and lane changing of every user individually. The majority of these data were graphically represented particularly
690
V. Charissis, S. Papanastasiou, and G. Vlachos
Fig. 6. A sample of graphical representations developed during the evaluation process
for the moments just prior to a potential collision as Figure 6 illustrates. The most popular actions repeated by the majority of subjects were clustered in groups of similar behaviours. In turn, the user with the most clear and common reaction was selected for further commenting and analysis. This process enabled us to understand how a typical driver negotiates with adverse weather and driving conditions with the use of contemporary equipment provided by the instrumentation panel. The HUD’s assistance resulted in measurable improvements in the driver’s behaviour and minimized substantially any panic reactions.
6 Conclusions Overall, this study has outlined the evolution of the HUD interface design, elaborated on its interface design philosophy and presented the outcome of the user trials that contrasted the use of the proposed HUD against a typical Head-Down Display (HDD). This paper has elaborated on the development process used for the design and evaluation of the sharp turn notification and traffic warning cues which reflected some of the most common risks present in a motorway environment under low
Interface Development for Early Notification Warning System: Full Windshield HUD
691
visibility. Throughout this work emphasis was placed on the prioritisation and effective presentation of information available through vehicular sensors in view of aiding the driver in successfully navigating the vehicle under low visibility conditions. The harmonic collaboration between the human (driver) and machine (vehicle) elements has been supported by utilizing the machine as a collecting and distilling hub of information. In turn, the human agent has been urged towards improved decision making through careful consideration of user characteristics and needs. This positive effect on the driver has been achieved by conveying the distilled information through carefully placed visual cues on the HUD interface. In future work, we aim to improve on the HMI characteristics of the HUD and expand the number and quality of the acquired data necessary for the HUD functionality. To achieve this goal, we aim to make use extensive use of Vehicular Networks functionality and consider the use of more diverse in-vehicle sensors.
References 1. Whiffen, B., Delannoy, P., Siok, S.: Impact on Road Transportation & Mitigation. In: Options presented in National Highway Visibility Conference, Madison Wisconsin, USA, May 18-19 (2004) 2. Ward, N.J., Parkes, A.M.: Head-Up Displays and Their Automotive Application: An overview of Human Factors Issues Affecting Safety. International Journal of Accident Analysis and Prevention 26(6), 703–717 (1994) 3. Tinker, P., Azuma, R., Hein, C., Daily, M.: Driving Simulation for Crash Avoidance Warning Evaluation. In: Proceedings of the 29th ISATA Dedicated Conference on Simulation, Diagnosis and Virtual Reality in the Automotive Industry, Florence, Italy, pp. 367–374 (1996) 4. Charissis, V., Papanastasiou, S., Vlachos, G.: Comparative Study of Prototype Automotive HUD vs. HDD: Collision Avoidance Simulation and Results. In: Proceedings of the Society of Automotive Engineers World Congress 2008, Detroit, Michigan, USA (2008) 5. Charissis, V., Papanastasiou, S.: Human-Machine Collaboration Through Vehicle HeadUp Display Interface, appearing in the Cognition. In: Hollnagel, E., Cacciabue, P.C. (eds.) Technology and Work Journal. Springer, London (2008) 6. Lee, J.D., Hoffman, J.D., Hayes, E.: Collision Warning Design to Mitigate Driver Distraction. In: Proceedings of CHI Conference 2004. ACM, Vienna (2004) 7. Kiefer, R.J.: Effect of a Head-Up Versus Head-Down Digital Speedometer on Visual Sampling Behavior and Speed Control Performance During Daytime Automobile Driving, SAE Technical Report Paper No. 910111, Society of Automotive Engineers New, USA (1991) 8. Kiefer, R.J.: Defining the HUD Benefit Time Window. In: Gale, A.G., et al. (eds.) Vision in Vehicles-VI, pp. 133–142. North-Holland/Elsevier, Amsterdam (1995) 9. Hooey, B.L., Gore, B.F.: Advanced Traveller Information Systems and Commercial Vehicle Operations Components of the Intelligent Transportation Systems: Head-Up Displays and Driver Attention for Navigation Information, Report for Federal Highway Administration (Technical Report No FHWA-RD-96-153), VA, USA (1998) 10. Gish, K.W., Staplin, L.: Human Factors Aspects of Using Head UP Displays in Automotbiles: A Review of the Literature, US Department of Transport, National Highway Traffic Safety Administration (NHTSA), Report No DOT HS 808 320 Washington, DC.USA (1995)
692
V. Charissis, S. Papanastasiou, and G. Vlachos
11. Sato, A., Kitahara, I., Kameda, Y., Ohta, Y.: Visual Navigation System on Windshield Head-Up Display. In: Proceedings of the 13th World Congress on Intelligent Transportation Systems (ITS), London, UK (2006) 12. Charissis, V., Patera, M.: Symbolic vs Alphanumeric Representations in Human Machine Interface Design. In: Proceedings of The 9th World Congress of the International Association for Semiotic Studies (IASS-AIS 07) Helsinki/Imatra, Finland (2007) 13. Green, P.: Design and evaluation of symbols for automotive controls and display. In: Peacock, Karwowski, W. (eds.) Automotive ergonomics. Taylor and Francis, London (1993) 14. Charissis, V., Papanastasiou, S.: Exploring the Ad Hoc Network Requirements of an Automotive Head-Up Display Interface. In: Proceedings of: 5th International Conference on Communication Systems, Networks and Digital Signal Processing, CSNDSP 2006 (IEEE), Patras, Greece (2006) 15. Papanastasiou, S., Ould-Khaoua, M., Charissis, V.: A survey of MAC Protocols for Mobile Ad Hoc Networks. In: Pan, G.M.Y., Fan, P. (eds.) Advances in Wireless Networks: Performance Modelling, Analysis and Enhancement. Analysis, Evaluation and Enhancement of QoS for Wireless Multimedia (2007) ISBN: 1-60021-713-3 16. Charissis, V., Arafat, S., Chan, W., Christomanos, C.: Driving Simulator for Head-Up Display Evaluation: Driver’s Response Time on Accident Simulation Cases. In: Proceedings of the Driving Simulation Conference DSC 2006, Asia/Pacific, Tsukuba/Tokyo, Japan (2006) 17. Ross, T., et al.: HARDIE Design Guidelines Handbook: Human Factors Guidelines for Information Presentation by ATT Systems. In: Harmonisation of ATT Roadside and Driver Information in Europe (HARDIE), Commission of the European Communities, R&D Programme: Telematics Systems in the Area of Transport, DRIVE II (1996) 18. SPD, Strathclyde Police Department: Accident Statistics 2001-2004, Glasgow, UK (2004) 19. Webster, D.C., Wells, P.A.: The Characteristics of Speeders, TRL Report 440, Crowthorne, UK (2000) 20. Stradling, S.G.: The Speeding Driver: Who, How and Why? Scottish Executive Research Findings 170/2003, Edinburgh, UK (2003)
Reflections on the Interdisciplinary Collaborative Design of Mapping the Universe Chaomei Chen1, Jian Zhang1, and Michael S. Vogeley2 1
College of Information Science and Technology, Drexel University 2 Department of Physics, Drexel University {cc345,jz85,msv23}@drexel.edu
Abstract. We describe our experience of developing scientific software in an ongoing multidisciplinary research project participated by information scientists and astronomers. In particular, we reflect on how the interdisciplinary collaboration is facilitated by the development of a unique boundary object – a hybrid map of the Universe and scientific discoveries. With reference to theories of interdisciplinary communication, we underline that information visualization serves a fundamental role in scientific software development. The impact of information visualization can go far beyond the content representation, and can facilitate communications across distinct disciplines. Keywords: Interdisciplinary collaboration, boundary objects.
1 Introduction Our scientific software development is part of a three-year research project funded by the Science and Engineering Information Integration and Informatics (SEIII) program of the National Science Foundation (NSF). The project is entitled “Coordinated Visualizations of Astronomical Data and Literature.” Its goal is to establish a generic, coordinated analytic environment so that astronomers and information scientists can access, analyze, and synthesize the large volume of scientific data and scientific knowledge seamlessly and cohesively. In particular, the project focuses on astronomical data collected by the Sloan Digital Sky Survey1 (SDSS) and scientific publications related to SDSS in various ways. 1.1 Sloan Digital Sky Survey (SDSS) The SDSS survey is an ambitious digital sky survey, directly participated by a consortium of 300 scientists and 23 institutions. Many current and upcoming astronomical missions are closely tied to SDSS. It is designed to collect astronomical data systematically so that astronomers can investigate critical questions in cosmology and study the origin and evolution of the Universe. Specific goals are to map the large-scale structure of the universe seen in the distribution of galaxies, to characterize galaxy properties, and to study the properties of quasars and their evolution. The survey data 1
are also invaluable for studies of our own galaxy and the stars within it. Some of the recent discoveries were only possible with the amount of data collected by the SDSS survey. For example, observations of 13 million galaxies and 200,000 quasars from the SDSS allow astronomers to detect cosmic magnification caused by the gravitational effect of dark matter throughout the2 universe. The SDSS survey produces images, spectra, a photometric catalog, and a spectroscopic catalog, through a dedicated 2.5-meter wide-field telescope at Apache Point Observatory in New Mexico. The resulting data stream is processed at Fermilab. Images, spectra, and measured parameters of the detected objects (galaxies, quasars, and stars) are then stored in a Microsoft SQLServer database (SkyServer), from which the data may be accessed by SDSS investigators and the public to do science. Over one hundred parameters are measured for each object. For astronomers, these data are a goldmine of cosmological information. 1.2 The Challenges SDSS generates a vast volume of astronomical data of large-scale structures, galaxies, quasars, and stars through a series of data releases to the public. The website for access to publicly-released data (skyserver.sdss.org) receives 4 million hits per month. The provision of the SDSS data has led to a rapidly growing body of scientific literature. According to the NASA ADS database of astronomy and astrophysics literature, currently there are more than 2,200 publications in ADS and they have been cited more than 77,000 times! The large volume of astronomical data and the fast-growing literature are also leading to challenges. Astronomers need tools that will enable them to troubleshoot systematic errors in large data sets, optimize databases for most productive use, identify highly utilized dimensions of the data space, and design future surveys. Information scientists, on the other hand, are primarily concerned with the growth of scientific domains and the impact of such digital sky surveys on scientific discovery. They need not only efficient access to the rich survey data or the fast-growing body of new discoveries in the literature, but also effective and cohesive access to the connections across different types of data. Until recently, scientific data and scientific knowledge are typically treated in isolation from one another. Scientific publishers and government agencies are increasingly making scientific data available alongside scientific publications. Navigating from scientific publications to data in general is not adequately supported by existing tools and it is still at an earlier stage of becoming a widely accepted practice, although the difficulty is not primarily due to technical barriers. More fundamental and potentially critical challenges, however, are associated with navigating from scientific data to existing scientific knowledge about them. The SDSS has amplified the magnitude of these challenges because of the scale of the SDSS data and the rapid growth of scientific publications resulted from the SDSS data. The core objective of the interdisciplinary project is to develop a virtual environment in which astronomers and information scientists can both access not only the rich data and publications resulted from the SDSS survey but also the most critical and valuable information that can lead to further scientific discoveries. 2
Reflections on the Interdisciplinary Collaborative Design of Mapping the Universe
695
2 Boundary Objects The intrinsic complexity of multidisciplinary development projects is largely due to the need to synthesize different perspectives and communicate across disciplinary boundaries to share knowledge. The diversity of perspectives is a double-edged sword. Diversity is often an important source of inspiration and creativity [2]. On the other hand, diversity can be a barrier for effective communication. Fischer [4] identifies four types of collaborative design: spatial, temporal, technological, and conceptual distributions. Complexities due to conceptual discrepancies are more fundamental for knowledge sharing and communication. An important understanding of how conceptually distributed collaborative design work is that the overall system, including designers, developers, users, design artifacts, and the functionality, must have the ability to evolve [5]. The design itself serves as a mediator between people who may have different perspectives and from different disciplines. Boundary objects represent information that is interpreted differently by people from different disciplines or practices [7-12]. Boundary objects may be abstract or concrete, and “such objects have different meanings in different social worlds but their structure is common enough to more than one world to make them recognizable, a means of translation. The creation and management of boundary objects is a key process in developing and maintaining coherence across intersecting communities." Boundary objects are externalized ideas so that one can point to in their communication, which is critical for conceptually distributed collaborative design. Agile software development is characterized by several properties that can be found in our project. For example, scientific software development involves scientists from different communities. Scholars in the literature often distinguish communities of practices and communities of interest when referring to homogeneous and heterogeneous communities of participants. Agile software development models would be recommended based on the levels of novelty, interactivity, and commitment involved [3]. In our case, astronomers and information scientists are clearly from different communities of practice.
3 Conceptual Gulfs Across Disciplinary Boundaries Some assumptions we take for granted in one discipline may become invalid across disciplinary boundaries. We will first identify what we agree at the disciplinary level and then where the validity of our assumptions begins to change. 3.1 Shared Perspectives Astronomers need to obtain a conceptual roadmap of how the SDSS has contributed to the advances of astrophysics in terms of successful and failed techniques. Such strategic understanding is important for planning future surveys. An example of current interest is the ongoing GALEX (Galaxy Explorer) satellite that is obtaining images and spectra of galaxies at ultraviolet wavelengths that cannot be observed from
696
C. Chen, J. Zhang, and M.S. Vogeley
the surface of the Earth. Lessons learned from the SDSS will provide insightful inputs, but current tools are not optimal to give us such roadmaps readily. Like in many scientific fields, astronomers need to identify emerging trends. Relating the astronomical literature to the three-dimensional spatial distribution of objects is a potentially rewarding approach. For example, are new discoveries being made at a faster rate in stellar, galactic, extragalactic, or cosmological research? Are clusters, voids, filaments, or some other feature of structure in the universe being explored? Astrophysical theorists would like to gauge which new discoveries deserve attention. In what areas is the most progress being made from the new observations? Further, relating the literature to the original data will allow searches for useful methods. For example, an astronomer who works on features in the surface brightness profiles of galaxies could trace the literature based on that specific type of data in SDSS and learn new methods for analyzing his/her own data, thus aiding dissemination of novel techniques and avoiding duplication of effort. There is an increasingly strong trend in science that massive scientific data are being collected by one group of scientists and being analyzed by another group of scientists [8]. Examples include the SDSS project in astrophysics and the human genome project in biomedicine. Therefore, it becomes vital for the two groups of scientists to better understand each other and better communicate with each other at an increasingly large scale. 3.2 Invalid Assumptions Researchers in information science are increasingly challenged by the tension between the overwhelming volume of scientific literature and the lack of tools that can help them to uncover hidden structures across the boundaries of individual articles, to reveal how such structures evolve over time, and to understand what role is played by such structures in the advances of science [9]. In information science, an extensively used approach to understanding the dynamics of scientific knowledge and scholarly communication is to study the structure and dynamics of scientific literature. Citation analysis focuses on emergent patterns associated with references made by scientists in their publications. Citation indexing capitalizes on the potential intellectual value attributed to a referential link made by a scientist. A fundamental assumption is that such links are similar to a voting system by nature and they reflect a collective and contemporary view of many scientists on an intellectual association. One of the widely used bibliometric analysis methods is called author co-citation analysis (ACA) [13]. The unit of analysis in ACA is an author of a cited reference. ACA is a useful method to generate an overview of the scientific community of a domain in terms of how various researchers’ work is interconnected in the eyes of authors who subsequently publish in this domain. Traditionally, the use of ACA follows an implicit assumption that the lead author of a paper is the most representative contributor to the work. In practice, ACA takes into account the first author of a paper but omits all other co-authors. Although information scientists have pointed out that considering all authors of a paper may provide a more accurate picture, the underlying assumption that the first author is the best representative of the underlying work has remained unchallenged. However, according to our astronomer collaborator, this
Reflections on the Interdisciplinary Collaborative Design of Mapping the Universe
697
assumption is not valid in the context of the SDSS literature because the ways the authorship are determined. The average number of authors per paper in the SDSS literature has been gradually decreasing since its peak in 2000 (about 18 co-authors on average) to the average of 7 co-authors per paper in 2006. It is not uncommon to see an SDSS paper with hundreds of coauthors. More importantly, in many cases, the first author is not necessarily the one who has made the most scientific contributions directly represented by the paper. Because research in astronomy involves large amounts of data, data collection is a significant science in its own right. The SDSS survey is one of such examples. In addition to papers on astronomical discoveries, the SDSS literature also includes papers on data collection and processing techniques, notably including papers describing individual data releases. Because of the complex eco-system in astronomical research, information scientists have seen for the first time some of the practices that would never occur to them, but taken for granted by astronomers. For example, astronomers can use a website to sign up or claim their authorship on a paper that they don’t need to write due to their roles such as being principle investigators of the development of a telescope. 3.3 Sources of Discoveries The second example of the mismatch perspectives from different disciplines is about the role of scientific literature in making scientific discoveries. In information science, one pays special attention to the role of scientific literature. There is even a research specialty that focuses on literature-based discovery [10-11]. Astronomers, in contrast, place a substantial amount of their emphasis on observational data. Numerical analysis and computer simulations are key components of their research activities. We increasingly realized through our interdisciplinary collaboration that our perspectives are considerably different as far as the role of scientific literature in scientific discovery is concerned.
4 The Design Iterations of Mapping the Universe We have the common goal of constructing a virtual environment for navigating through the domain knowledge and making scientific discoveries. The challenge was to find or build a common ground that would give us a reference framework to integrate information from different disciplines. The first inspiration came as we (information scientists) encountered the Logarithmic Map of the Universe [7]. The logarithmic map depicts the entire visible Universe in a rectangular shape with the Earth as the bottom line of the map and the Big Bang as the top of the map. The rectangular map includes SDSS galaxies and quasars as well as astronomical objects that one can see from the Earth, such as the Sun, the moon, and stars in famous constellations. A computer printout of the map stretches from the floor all the way to the height of an office door. A point on the celestial sphere can be identified by its right ascension and declination degrees. The rectangular map contains the positions of SDSS galaxies and quasars in terms of right ascension and their distances measured from the Earth. With the
698
C. Chen, J. Zhang, and M.S. Vogeley
rectangular map, viewers can easily tell how far away an astronomic object or structure is from us. 4.1 Initial Design Our initial design was to reconstruct the rectangular map and overlay two additional types of literature related information: (1) the dates of discoveries of major astronomical objects such as the discoveries of 3C 273, the Sloan Great Wall, and the Hubble Ultra Deep Field; and (2) indicators of research interests in terms of citation bursts, which measure the speed of change in citations received by a paper over time. The dates were manually compiled, whereas the citation bursts were computed.
Fig. 1. The initial design (Left): A time spiral of discoveries around the rectangular map of the Universe. Each discovery event on the time spiral is linked to the corresponding astronomic object in the rectangular map. The circular design (Right).
The rectangular map has a long but narrow shape. Its aspect ratio is 0.28:1, which is about 1:4. In comparison, computer monitors and the standard TV have an aspect ratio of 4:3 (1.33:1). Since our goal is to develop an interactive virtual environment through computers, a design with an aspect ratio close to the standard 4:3 will have an advantage; for example, viewers would be able to examine details more easily. This is part of the reason we designed a circular map which has an aspect ratio of 1.35:1, very close to the 4:3 ratio for computer monitors and the standard TV. The width of the rectangular map measures the right ascension of an astronomical object in the map. The left and right sides of the map are in fact identical. In other words, each horizontal line in the map represents a full circle determined by right ascension and the height of the line is the distance away from the Earth. However, for non-astronomer viewers, this may not be obvious. The continuity of the full circle is interrupted by the artificial and arbitrary boundaries. Furthermore, although lines near the top and those near the bottom are depicted with the same length (i.e. the width), they are in tremendously different scales of distance. The top of the map is measured in the scale of gigparsecs (gpc), whereas the bottom of the map is on the scale of kilometers (km). The rectangular map served as the first boundary object that facilitated our interdisciplinary communication between astronomers and information scientists. Another important factor from the scientific software development point of view is that the creators of the original logarithmic map of the universe are kind enough to make the
Reflections on the Interdisciplinary Collaborative Design of Mapping the Universe
699
data and the original computer program available for us. The original rectangular map was generated by a set of macros written in an interactive plotting programming package called SM3. For information scientists, learning how these macros work together to plot the rectangular map is much easier than learning what astronomers do with their research topics. In addition, as we were learning the SM macros, we became familiar with the data that were depicted on the map. Therefore, focusing on the rectangular map was a significant step in our experience with scientific software development, and the underlying computing programming language, although it is different from the language we used in the final design, Java, provides a valuable common ground to expand our understanding of the creation of the map. 4.2 New Design Given that the two sides of the rectangular map are identical, we decided to join the two sides and make a circular map of the Universe. The circular design also changed the aspect ratio from the original 1:4 to about 4:3. The circular design preserves the relative positions of objects on the original rectangular map, but transforms the upwards view to a 360° view originating from the Earth. The transformation can be seen as we stretch a rectangular shape to a fan shape and then keep stretching until the edges of the fan meet.
Figure 1 (Right) also shows the use of a polar coordinate system. Each data point on the map is positioned in terms of its right ascension and a logarithmically transformed distance from the center of the Earth. The right ascension increases counterclockwise. Five scales of distances are used in the map, namely, kilometer (km), astronomical unit (au), parsec (pc), kilioparsec (kpc), megaparsec (mpc), and gigparsec (gpc). The mpc and gpc scales are enlarged for clarity because the SDSS galaxies and quasars fall into these areas. Figure 2 shows the final design of Mapping the Universe. It is currently a still image, but we intend to use it as a stepping stone to design our virtual environment for scientists to access astronomical data and literature. The design features a circular map of the entire visible Universe, with the Earth at the center. The blue band of data points depicts galaxies identified in the SDSS survey data, whereas the red band depicts quasars found by the SDSS survey.
5 What We Have Learned At the beginning of the collaboration, information scientists were expecting to find common and typical ways in which astronomers do their research, how they would deal with data, and how they would be guided by the existing literature. On the hindsight, this expectation was naïve, and now we realize that there is no standard or conventional procedure for scientific activities. In fact, we have subsequently found that this seems to be a common character in other disciplines as well. Although there is a sizable body of literature on scientific discoveries from philosophical and sociological perspectives, we have not found works in the literature that are generic enough to suit the needs for software development or interdisciplinary collaboration. Therefore, an important lesson in our experience is that understanding the needs of scientists is a crucial element in scientific software development; however, it is not realistic or practical to expect that one can find a ‘typical’ scientist. The second lesson is that the traditional assumption of author co-citation analysis is no longer valid in astronomy. Many bibliometric studies represented the concerns and interests of information scientists and not necessarily that of domain scientists. More importantly, since the validity of analytical methods and interpretations may not be thoroughly cross-examined by domain scientists, the practical values of such studies can be considerably limited. Therefore, we have come to the conclusion that future bibliometric studies should aim to go beyond the disciplinary boundary of information science and take into account qualitative as well as quantitative information from researchers in scientific domains. The third lesson is about finding common artifacts to build a common ground, specifically, about the role of earlier computer-generated boundary objects. The design of the original rectangular map of the Universe and the computer program used to generate the map provided valuable boundary objects between astronomers and information scientists. They provided not only concrete examples of a data representation framework in astronomy, but also demonstrated how it was done. As soon as we were able to re-construct the rectangular map on a new platform, it becomes much easier to transform the original design into new designs. Communication and knowledge sharing across disciplinary boundaries become constructive and externalized because both
Reflections on the Interdisciplinary Collaborative Design of Mapping the Universe
701
sides now refer to specific items on the map. Communications become specific with much reduced ambiguity. The fourth lesson is that interdisciplinary collaborative design and software development involves a substantial degree of exploration and uncertainty. The project has demonstrated several characteristics of agile software development: 1) A short development lifecycle. The design of the Mapping the Universe visualization started from the second half of May 2007. The first prototype was produced by June 11, 2007. A number of iterations took place in the second half of July and the first week of August. The duration of the specific project is less than 3 months. 2) A large number of prototypes produced within the short period of time. Visualized representations provided an effective vehicle for communication. We maintained intensive communications via frequent brief but specific email and weekly face-to-face meetings. 3) The intermediate role of visualization prototypes as boundary objects. The original rectangular map was an important starting point for knowledge sharing. Its design provided a valuable reference point for the subsequent design. Multiple iterations of subsequent prototypes facilitated the construction of a common ground. Finally, once the visualization as a boundary object is in place, it has been surprisingly effective in terms of learning disciplinary knowledge. Once we have the basic circular map of the Universe, our astronomer collaborator can easily advise developers to add significant objects as shown in the following quote: “Again on the subject of interesting solar system objects, if you could make the label and discovery date of Quaor visible, that would be good. This is the object whose discovery by Mike Brown at Caltech got all the Pluto controversy rolling.”
6 Conclusions Scientific software development faces inherited challenges in terms of crossdisciplinary communication and knowledge sharing. The notion of boundary objects and Agile software development appear to provide a useful framework. Visualizations of domain-specific information can play an instrumental role in facilitating multidisciplinary collaboration and they have demonstrated many characteristics of boundary objects. We recommend that future scientific software development should pay particular attention to the use and evolution of boundary objects in collaborative design and knowledge sharing.
Acknowledgements The work is supported by the National Science Foundation under the grant IIS0612129. Special thanks to J. R. Gott III and M. Juric for providing the data and computer programs used for the original rectangular map of the Universe.
References 1. Bowker, G.C., Star, S.L.: Sorting Things Out — Classification and Its Consequences. MIT Press, Cambridge (2000) 2. Burt, R.S.: Structural holes and good ideas. American Journal of Sociology 110, 349–399 (2004)
702
C. Chen, J. Zhang, and M.S. Vogeley
3. Datta, S.: Agility measurement index: a metric for the crossroads of software development methodologies. In: ACM Southeast Regional Conference 2006, Melbourne, FL, pp. 271–273 (2006) 4. Fischer, G.: Distances and Diversity: Sources for social creativity. In: Cognition and Creativity 2005, London, UK, pp. 128–136 (2005) 5. Fischer, G., Giaccardi, E., Ye, Y., Sutcliffe, A.G., Mehandjiev, N.: Meta-Design: A Manifesto for End-User Development. Communications of the ACM 47, 33–37 (2004) 6. Gordon, M.D., Lindsay, R.K.: Toward discovery support systems: A replication, reexamination, and extension of Swanson’s work on literature-based discovery of a connection between Raynaud’s and fish oil. Journal of the American Society for Information Science 47, 116–128 (1996) 7. Gott, J.R., Juric, M., Schlegel, D., Hoyle, F., Vogeley, M.S., Tegmark, M., Bahcall, N., Brinkmann, J.: A map of the Universe. Astrophysical Journal 624, 463 (2005) 8. Gray, J., Szalay, A.: Where the rubber meets the sky: Bridging the gap between databases and science. Microsoft Research (2004) 9. Hargens, L.L.: Using the Literature: Reference Networks, Reference Contexts, and the Social Structure of Scholarship. American Sociological Review 65, 846–865 (2000) 10. Lindsay, R.K., Gordon, M.D.: Literature-based discovery by lexical statistics. Journal of the American Society for Information Science 50, 574–587 (1999) 11. Star, S.L.: The Structure of Ill-structured Solutions: Boundary Objects and Heterogeneous Distributed Problem Solving. In: Huhs, M., Gasser, L. (eds.) Readings in Distributed Artificial Intelligence, vol. 3, pp. 37–54. Kaufmann, Menlo Park (1989); Swanson, D.R.: Undiscovered public knowledge: Library Quarterly 56, 103–118 (1986) 12. Wenger, E.: Communities of Practice — Learning, Meaning, and Identity. Cambridge University Press, Cambridge (1998) 13. White, H.D., Griffith, B.C.: Author co-citation: A literature measure of intellectual structure. Journal of the American Society for Information Science 32, 163–172 (1981)
Distilling Support Opportunities to Improve Urban Search and Rescue Missions Tjerk de Greef1,2, A.H.J. Oomes1, and Mark A. Neerincx1,2 1
Delft University of Technology, Man-Machine Interaction Group, Mekelweg 4, 2628 GA Delft, the Netherlands 2 TNO Defence, Security and Safety, Department of Human Factors, Kampweg 5, 3769 ZG Soesterberg, the Netherlands [email protected], [email protected], [email protected]
Abstract. Current USAR missions are challenged by many factors leading to a study on how human computer interaction can provide support in this domain. Using data from a two-day observation in combination with mission reports, we applied a situated cognitive engineering design methodology to distill the operational demands, the human factors challenges, and the current and future technological design space. The operational demands result in a set of core functions that were explained in various parts of the USAR mission organization. Furthermore, an exemplary support scenario and prototype was provided in combination with claims on the envisioned effect. Keywords: urban search and rescue, situated cognitive engineering, usercentered design, work domain analysis.
1.1 Cognitive Engineering This study applied the situated cognitive engineering (sCE) methodology to establish a theoretically sound and empirically proven requirements database. The sCE methodology [1] was developed corresponding to the “classical” cognitive engineering methods [2-4] that consists of an iterative process of generation, evaluation, and refinement. In addition, the sCE method combined the classical human-centered perspective with a technology-centered perspective to systematically address the nature of both human and synthetic actors with their reciprocal dependencies as expressed in the joint cognitive systems paradigm. Furthermore, the sCE method includes an explicit transfer and refinement of general state-of-the-art theories and models—which include accepted features of human cognitive, affective, and social processes— into situated support concepts for the specific operational contexts [5]. Application of the sCE method results in a sound rule base (i.e., core functions, claims, and scenarios) with corresponding best practices for the application domain. The baseline possibly includes design patterns, software frameworks, and algorithms for core support functions. The process of specification, refinement and validation is based on three information or feedback sources (see Fig 1). First a work domain and support analysis identifies operational, human factors, and technological challenges. Secondly an expert and task-analytical review assesses the rule base itself (i.e., scenarios, claims and core functions), and the third source describes scenario-based prototype evaluation of claims and core functions.
Fig. 1. The iterative process of requirements analysis (adapted from [1]) distills core functions, claims, and scenarios from operational demands, human factors knowledge, and a technological design space. These three aspects provide a requirements baseline that is evaluated using review and prototype evaluations.
Distilling Support Opportunities to Improve Urban Search and Rescue Missions
705
For the specification of the requirements baseline, we distinguish three steps that should be followed both from top and bottom. First, the core functions of the system are derived from the work domain and support analysis. Second, for each core function, one or more testable claims on its operational effects have to be specified; such a claim can be assessed unambiguously in review or evaluation processes. Both positive and negative claims can be specified. The claims consist of standard usability measures (i.e. effectiveness, efficiency, and satisfaction), and can easily be extended with other measures. Third, scenarios have to be specified describing coherent and situated stories about how an actor or team of actors will behave in specific circumstances with the operational consequences.
2 Work Domain and Support Analysis 2.1 Operational Demands We conducted an operational analysis consisting of a cognitive work analysis (CWA) based on operational reports, lessons learned from previous missions, and a two-day observation at an USAR training facility. According to the ecological approach, a field description is fundamental to understand the domain in order to distill support concepts. CWA [6] is a work-centered conceptual framework designed to analyze cognitive work processes. A CWA recognizes five concepts to describe the domain and we will utilize these concepts to describe a typical USAR operation. Work Domain. The system that an USAR organization or mission is controlling is the environment that is hit by a natural or a man made disaster, for example an earthquake. The work domain is characterized as an unstructured chaotic environment where possibly all critical social and governmental infrastructure facilities (e.g. power supplies, hospitals) have broken down and require a lot of effort to be operational. Depending on the size and type of the disaster, governmental problems of security and responsibility might arise. A typical mission constructs a base camp that provides a resting facility and a command post. The actual search and rescue activities take place in so-called worksites. Control Task. The goal of an urban search and rescue is to rescue trapped people after a natural or a man-made disaster. Besides rescuing trapped victims additional medical, reconstructive, and organizational tasks are executed by an USAR operation supportive to the additional goal in terms of flag planting. Every mission is politically determined based on considerations that often have limited correlation with humanitarian necessity. Furthermore, an USAR organization has to protect itself from inward and outward threats. Outward threats are defined by attacks or robberies on the organization while inward threat deals with the safety conditions of the organization itself (e.g., personal health, sanitation). Strategies. An USAR organization has two general mechanisms to achieve their goals. First by putting the right action on the right place at the right moment provides the organization the largest chance to find survivors. Secondly, the safety of the own organization is guaranteed by integrating several information sources.
706
T. de Greef, A.H.J. Oomes, and M.A. Neerincx
Social Organization and Cooperation. An USAR mission recognizes a command group, a staff group, a support group, and a number of rescue groups (see Fig. 2). The command group is in charge of the operation while the staff and support groups have respectively delegated responsibilities and support functions for the overall management. Each search and rescue group consists of about eight to ten people of whom some have a specialized function like a group leader, a dog handler, a technical searcher, and a medic. Furthermore, additional organizations are important. An USAR operation has to work in cooperation with the United Nation’s Office for the Coordination of Humanitarian Affairs (OCHA), the home country’s local operational team (LOT), and the local emergency management authority (LEMA). The OCHA organization is mandated to coordinate international assistance in disasters and humanitarian crises exceeding the capacity of the affected country. The LEMA has the ultimate authority for the overall command, coordination, and management of the response operation and can refer to national, regional, or local authorities (or combinations thereof) that are collectively responsible for the disaster response operation.
Fig. 2. A typical construction of an USAR mission deals with a United Nations office, a local emergency management authority (LEMA), a local operational team (LOT), a command group, a staff group, a support group, and four search and rescue groups. These groups coordinate joint activity using six coordination loops.
Distilling Support Opportunities to Improve Urban Search and Rescue Missions
707
In order to analyze the cooperation between the various groups we utilize the concept of coordination loops [7]. A coordination loop describes a process of coordination among actors where common ground is monitored and actions are mutually observed and (re)directed. A typical USAR mission acknowledges six coordination loops. The first loop (see Fig. 2) describes some sort of initialization loop. In order to have USAR teams respond, the troubled country needs to request assistance trough OCHA. OCHA will respond by asking available USAR teams to mobilize through each country’s LOT. The second coordination loop describes the effort of the LOT to mobilize and transport an USAR team. Once an USAR team is available and present at the disaster site, coordination with the LEMA is required as the LEMA bears the ultimate authority (loop 3). The fourth loop reports a coordination effort between the command, staff, and support groups in order to optimize joint activity. The fifth and sixth coordination loop respectively describe an effort to maintain a common understanding between the search and rescue groups and the staff group and between the command group and the OCHA. Worker Competencies. The USAR domain imposes some special requirements on the workers involved. Each member is required to have an excellent physical condition because the initial phase of a mission deprives each member of normal sleep. After this initial phase, a rotation system is put into place. In addition to these physical requirements, the workers must be emotionally fit as these emotions are challenged by the possible massive destruction caused by the disaster. To minimize post-traumatic stress experiences, post-mission debriefing sessions are mandated on a personal and a group level. The higher ranks of the USAR team require some diplomacy skills as goals of a LEMA might conflict with goals defined by either the UN or by the USAR team’s government. A part of the team requires special technical skills to for example operate equipment or handle dogs. 2.2 Human Factors In addition to the operations analysis, a review of human factors related issues in complex high demanding task environments resulted a number of key issues to be addressed by a support system. First, situation awareness [8] and sense-making [9] relate respectively to the perception, comprehension, and projection of the elements in the environment and the process of making sense of the situation. Klein, Moon, and Hoffman [10] describe the latter as a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively. The USAR domain is characterized by chaos in every detail: an unclear governmental structure, loss of all infrastructural connections, who has done what are typical chaotic examples within a typical mission. Secondly, collaboration [11] refers to a coordinated attempt to obtain mutual benefits by sharing or partitioning work to achieve collective results that participants would be incapable of when working alone. The chaotic conditions after a disaster makes gaps and overlap in work a true risk for optimal joint activity and support helps to avoid these gaps and overlap in individuals' assigned work (i.e. support coordination). Furthermore, adding capability to support the generation and maintenance of a shared mental model for interdependent actors working at
708
T. de Greef, A.H.J. Oomes, and M.A. Neerincx
geographical different locations promises a useful capability. By mediating between actors, insight will be provided into the other actors’ goals, intentions, behavior, and needs. Third, crew resource management [12] is a combination of techniques used to train a crew to operate in complex environments, aiming to minimize the effects of errors related with human factors (including communication and cultural aspects) and to maximize effectiveness. A support system should manage the skills and task performance of the crew, and plan and support training to keep performance to a level required by operational demands. Support for personal task load management such as emotion and physical task load come to surface as a forth opportunity. Each worker experiences physical demanding working conditions as set by the short mobilization period and continuous deployment. Besides these physical inconveniences, the environment is emotionally challenging as bizarre, unreal, and cruel sights dominate the mission potentially increasing the risk for post-traumatic stress experiences. 2.3 Envisioned Technology In addition to the operations and human-factors analysis, we conducted a technology assessment that distinguished a number of developments that have potential to be applied in the USAR domain. First, developments in sensing equipment technology show increasing capabilities to have technology available to have an inside view of collapsed structures helping rescue worker to improve assessing whether survivors are present. Secondly, a continuous push in the area of robotics lead to the application of unmanned (aerial) vehicles in the domain of USAR [13] enabling rescue workers a better visual overview with less related costs. These human robot interaction issues involve multiple people working with one robot while being in a different location. Third, various human computer interaction techniques, such as digital sketching pads and gesture recognition, show to have potential to be applied in the USAR field. Fourth, new sensor technology shows potential to be integrated in working outfits in order to sense, interpret, and anticipate individual human conditions and behaviors (e.g. to improve safety and health). Last, new developments in ad-hoc network technology facilitate means to be connected by one another in areas where limited connectivity exists.
3 Requirements Baseline: First Iteration The previous section describes the first components of the sCE methodology. This section describes a first iteration of the construction of the requirements database thereby defining the core functions. Based on the work domain and support analysis we distilled eight core functions (see Table 2 for explanation). Core functions are high-level tasks to be applied in a different part of the organization.
Distilling Support Opportunities to Improve Urban Search and Rescue Missions
709
Table 1. Core Functions within an urban search and rescue mission Core Function Manage Resources Situation Assessment Human Resource Management Planning Search & Rescue Locate Victims Determine Attack Route
Manage Information
Manage Contacts
Explanation Relates to the organization of resources whether they be personnel, equipment, or others like water supplies This core function describes the current and future state of the elements (and their relation) Despite the definition by [12] (crew resources management), this core function relates to the location and physical and emotional state of every living being (thus personnel and dogs) Describes the activities required to have an optimal deployment of the search & rescue activities including all related interdependent tasks Illustrates the undertaking to locate the victims by the utilization of dogs, sensor equipment, and video footage After locating a victim, the expertise required to rescue a victim requires a so called attack route that describes the plan to rescue a victim in relation to the victim’s health, the fitness of the team, and other pending tasks Managing information to the media and family is sometimes a delicate matter while other information might flow to the LOT or the OCHA An important aspect is to have local knowledge & contacts
Table 2. Breakdown of core functions subdivided by organizational part Local Organization
Search & Rescue
External communication
Manage Resources x Personnel x Equipment x Resources Situation Assessment x Environment x Base Camp Human Resource management x Check-In-Check-Out x Team Fitness Planning Search & Rescue
Locate Victims Manage Resources x Personnel x Equipment x Resources Situation Assessment x Action areas x Buildings x Attack route management x Locals Determine Attack Route x Medical Victim Assessment x Rescue/salvage x Team fitness x Communication with C.C.
Manage Information x x x
Media Family Update LOT centre x Update (v)OSSOC Manage Contacts x x
LEMA Local Embassy
In addition, we acknowledge three distinct organizational parts: local, search & rescue, and external communication. The local organizational part refers to core functions required for the base camp to operate effectively and efficiently. The search & rescue organizational parts relate to the core functions that needs to be executed by every search and rescue team. A totally different aspect of the organization concerns
710
T. de Greef, A.H.J. Oomes, and M.A. Neerincx
Table 3. This scenario describes how the system supports the planning process leading to the claim that it improves effectiveness of the planning process and improves satisfaction of the search and rescue teams. Scenario
Claims Effectiveness + Efficiency 0 Satisfaction + Currently, search and rescue team Alpha is deployed in worksite 10 and has located four victims. Assessment of the attack route indicated a prolonged rescue operation at this site to rescue all victims leading to the plan to replace team Alpha by team Charlie. At some point in time, team Alpha makes huge progress reaching the goal of rescuing all four victims. The support system makes the staff group aware of the rapid progress of the Alpha group. The staff group has new information telling them that action area 5 has a high chance to find survivors and they reallocate team Charlie to be deployed at worksite 5 instead of having them active at site 10 where their presence is probably of limited use.
external communication that manages all official information flow to, for example, the media. Each core function is positioned in an organizational part and possible leading to different division of subtasks (see Table 3). Second, for supporting a core function, one or more testable claims on its operational effects have to be specified. Such a claim can be assessed unambiguously in a review or a prototype evaluation. Both positive and negative claims can be specified. The claims all relate to effectiveness, efficiency, and satisfaction. Third, scenarios have to be specified. Scenarios are coherent and situated stories about how specific actors behave or will behave in specific circumstances.
4 Prototype for Core Function Planning While the previous sections applied the sCE methodology on the USAR domain, this section provides a prototype that supports a core function in the local organization part of a mission: planning. First we provide an exemplary scenario including claims, and subsequently we clarify our support system using a prototype. Table 3 reports a scenario offering support to the core function planning that materialized as results of observational data gathered during a two- day training mission. Currently, the planning staff has limited insight into the progress of task execution of search & rescue groups leading to a diminished capability to observe deviations to the plan. Having such information aids the staff to identify significant deviations to the plan and enables them to update and improve the plan thereby maximizing effectiveness. The support therefore aims to offer support by making actions of (a team of) actors working at a different location observable thereby aiding a local actor in the observation, comprehension, and projection of the progress. Fig. 3 displays an initial prototype offering insight into the progress towards goal accomplishment of team Alpha all in order to make predictions and anticipate deviations from the plan. Anticipation is the result of combining multiple dimensions. A navigation system, for example, anticipates the arrival time based on previous progress (e.g. 25 miles in 30 minutes) onto the remaining distance (e.g. 50 miles). This example shows that the temporal dimension and the distance dimension provide a two-dimensional space that is the basis for the anticipative act.
Distilling Support Opportunities to Improve Urban Search and Rescue Missions
711
Fig. 3. A prototype display improves the insight of the progress of activities of a search & rescue team
Anticipation in the USAR domain is based on a similar idea where progress on one dimension provides an indication on the progress on another dimension. Consider drilling activities of a search & rescue team. Frequently these activities can last for hours and previous experience in combination with an estimate of the thickness and type of material provide a prediction in the temporal dimension. Consider Fig. 3 where drilling through a structure is taken as an example. It is anticipated that the team requires 3 hours and 40 minutes of drilling to make a hole through one meter fortified concrete. The team starts at 9.35 leading to the prognoses that the drilling is finished at 13:15. At 12:05 the display indicates that the team has drilled through 80 cm and a simple calculation shows that they probably finish early as they have more than one hour left to drill through 20 cm of concrete while the previous 80 cm were covered in 2:30. The prototype fortifies the rapid progress prior to 12:05 by an increasing distance between lines indicating an increased drilled distance between fixed timestamps.
5 Conclusion Current USAR missions are challenged by many factors leading to a study on how human computer interaction can provide support in this domain. We applied a situated cognitive engineering design methodology describing the operational demands, the human factors challenges, and the current and future technological design space leading to a scenario-based prototype. Future research will focus on the validation of these types of displays.
Acknowledgements We gratefully acknowledge funding for our work from the IOP-MMI Programme that is run by SenterNovem, an agency of Dutch Ministry of Economic Affairs.
References 1. Neerincx, M.A., Lindenberg, J.: Situated Cognitive Engineering for complex task environments. In: Schraagen, J.M., Militello, J.M.C., Ormerod, L., Lipshitz, R. (eds.) Naturalistic Decision Making and Macrocognition, pp. 373–390. Ashgate Publishing Limited, Aldershot (2008)
712
T. de Greef, A.H.J. Oomes, and M.A. Neerincx
2. Hollnagel, E., Woods, D.D.: Cognitive systems engineering: New wine in new bottles. International Journal of Man-Machine Studies 18, 583–600 (1983) 3. Norman, D.A.: Cognitive Engineering. In: Norman, D.A., Draper, S.W. (eds.) UserCentered System Design: New perspectives on human-computer interaction, pp. 31–62. Lawrence Erlbaum Associates, Hillsdalte (1986) 4. Rasmussen, J.: Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. North-Holland, Amsterdam (1986) 5. Neerincx, M.A.: Cognitive task load design: model, methods and examples. In: Hollnagel, E. (ed.) Handbook of Cognitive Task Design, pp. 283–305. Lawrence Erlbaum Associates, Mahwah (2003) 6. Vicente, K.J.: Cognitive Work Analysis. Lawrence Erlbaum Associates, Mahwah (1999) 7. Voshell, M.G., Woods, D.D., Prue, B., Fern, L.C.: Coordination Loops: a new unit of analysis for distrubuted work. In: Mosier, K., Fischer, U. (eds.) Proceedings of the Eighth International NDM Conference, Pacific Grove, CA (2007) 8. Endsley, M.: Design and evaluation for situation awareness enhancement. In: Proceedings of the Human Factor Society 32nd Annual Meeting, pp. 97–101. Human Factors Society, Santa Monica (1988) 9. Weick, K.E.: Sense-Making in organizations. Sage, Thousand Oaks (1995) 10. Klein, G., Moon, B., Hoffman, R.F.: Making sense of sensemaking I: alternative perspectives. IEEE Intelligent Systems 21, 70–73 (2006) 11. Mohammed, S., Dumville, B.C.: Team mental models in a team knowledge framework: Expanding theory and measurement across dsciplinary boundaries. Journal of Organizational Behaviour 22, 89–106 (2001) 12. Helmreich, R.L., Merritt, A.C., Wilhelm, J.A.: The evolution of Crew Resource Management training in commercial aviation. International Journal of Aviation Psychology 9, 19–32 (1999) 13. Murphy, R.R.: Human–Robot Interaction in Rescue Robotics. IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews 34 (2004)
A New Approach to Design an Interactive System for Molecular Analysis Mouna Essabbah1, Samir Otmane1, Joan Hérisson2, and Malik Mallem1 1
IBISC laboratory, CNRS, University of Evry, Evry, France 2 Epigenomics Project, Genopole, Evry, France {mouna.essabbah,samir.otmane,malik.mallem}@ibisc.fr, [email protected]
Abstract. The rapid evolution of molecule’s imaging and observation’s techniques has caused a growing interest in studying molecular structures. Naturally, scientists have turned to simulation and 3D modeling in order to better understand biological phenomena. Thus, several 3D modeling systems have emerged. Some of these systems are dedicated to 3D visualization, and others are interested in 3D handling. However, we observed that these systems use classical 3D interaction techniques, frequently used in virtual reality (VR). On the other hand, the biological environment is very complex and binding as well. Thus, to remain faithful to the constraintes of the environment and be closer to natural behavior of molecules, we have tried to propose a 3D manipulation adapted to the domain, a bio-supervised 3D manipulation. Keywords: 3D Manipulation, Complex Systems, Biological Constraints, Adaptability, Bio-supervisor.
interactions at the same level as the actions undertaken by the system. The interaction is no longer an independent bloc in the system architecture, because it must be able to adapt to the environment in which it will be integrated. The modality of interaction defined by a document is not necessarily the same as the applications context. In our research, we develop an assistance system for molecular analysis while proposing a more credible and global genome's representation. Thus, the handled object is not simple. Moreover, beyond the 3D environment constraints, it is subjected to other very strong constraints, those of physics and biology. It seems obvious that the interactive activity of this application must be adapted to the domain. In this paper, we present a new approach to design bio-supervised 3D manipulation techniques that is aware of the environment. First, we define the context as well as the raised issues, particularly, the constraints of 3D and biological environments. Then we describe previous work in the assistance for molecular analysis field. The last section will include a description of the proposed approach, where we explain the global architecture; afterward we denote the interest of each component. We conclude this paper with some discussions and future work.
2 Context and Issues 2.1 3D Environment Constraints The choice of representation and metaphors do not respect a set of fixed rules. If the choice of representation is realistic, as in our case, interactions will also work with the constraints that we find in the real world (i.e. biological environment). In contrast, if the choice of representation is moving towards a non-realistic environment, simple metaphors will be used to represent complex tasks. The choice of these metaphors is dependent of devices that we decided to use and consistency that we want to keep for the global application. For a given task, there is non-fixed number of possible interaction’s metaphors. To take into account these constraints, we orient our platform to enable the creation of various elements of the application. The devices used will be very varied depending on the application, ranging from simple keyboard and mouse to more specific devices such as data gloves, tracking head, etc. This diversity is due to the complexity of the task to realize in the virtual environment that can claim a many degrees of freedom. Furthermore, the choice of devices becomes more delicate when the tasks are heterogeneous, it must be done through compromise among all tasks. This involves the development of interaction techniques to achieve the task with a device, although it is less suitable (Eg trackball or ray-casting that allow interaction for the selection and manipulation of 3D objects with a simple 2D device). 2.2 Biological Environment Constraints The interest of a part of our research is to obtain a 3D model of the molecule's structure, which is closer to the reality. Hence, we rely on natural elements, as the molecule in its environment. Indeed, researches around the genome's organization have identified some biophysical data and biological models, each one represent a
A New Approach to Design an Interactive System for Molecular Analysis
715
part of this spatial organization. We are looking particularly for DNA’s organization, at the chromatin level. Architecural Constraints: Hence, the physical data that we have been based on are: the chromatin's diameter (which is of 30nm [2]), the chromatin’s persistence length an intrinsic feature of the chromatin fiber that represents the rigidity of the chromatin, and that is 150nm for the chromatin in vitro [3] -, the confined volume of in which the DNA evolve (the cell nucleus and the chrmosome’s territory [4]), and finally, the curvature energy of the chromatin. Functional Constraints: The architectural constraints represent only the physical aspect of the chromatin, disregarding its functional aspect. Thus, we are interested in biological models. Each model has different physico-chemical and biological implications that will affect the molecule’s 3D model. Among the models that will be exploited, we quote: The 3C method (Capturing Chromosome Conformation) [5], the location of transcription factories [6], the formation of loops [7] and the solenoid principle [8]. The challenge that we have fixed is to provide an assistance system for molecular analysis while proposing a credible and global genome's representation. After identifying data and models to exploit, we have worked on building a simple first structure of the molecule that is only based on biophysical constraints. The four biophysical constraints that we have mentioned above, allow us to establish a simple chromatin’s 3D model, which is a "worm-like" model based on a sequence of identical cylinders (Fig. 1 shows the conceptual model).
(a)
(b)
(c)
Fig. 1. (a) Basic cylinder: The diameter = chromatin's diameter (30 nm) and the length = 1/5 chromatin's persistence length (150 nm), (b) Angle determining the curvature energy, (c) The confined volume in the cell's nucleus.
3 Assistance to DNA’s Analysis: Related Work Initially scientists were interested in sequence analysis for the study of its rich syntax. Gradually (between 1984 and 1987) molecular biology lived a fulgurating progress of its technical means, which leads to automation and an increasingly advanced and refined miniaturization. On the other hand, genomics allowed nowadays a global and complete approach for sequence analysis. It is no longer limited to the molecules study, but it also includes the study of the relationships between these elements, which causes the dynamic aspect of the whole. DNA can be represented by its textual sequence and also by its spatial distribution. The main interest for the threedimensional visualization of this molecule is the study of genes. But this can be done simply by textual analysis of the sequence. However, the 3D model showed that two patterns of the genome might have spatial similarities, and have very different textual
716
M. Essabbah et al.
sequences. It is also possible, thanks to the 3D model, to seek a pattern throughout the molecule not only by its sequence but also by its spatial form. This could deduct the influence of the entire genome in certain areas and vice versa. On the other hand, DNA spatial visualization provides an opportunity to study its interaction with other molecules, especially proteins. More generally, 3D visualization provides an overall view of the studied molecule. It allows modeling phenomenon or simulates a biological mechanism. Currently, molecular modeling is an essential research area. It helps scientists to develop new drugs against diseases in general and particularly serious ones such as AIDS and cancer. This sector assists also the genomes analysis. As the interest that was accorded is extremely important, many researchers became specialized in molecular modeling. The assistance in analyzing the DNA’s spatial structure was done in two stages: first by a completely automated method and then a more interactive approach. 3.1 Automatic Approach DNA’s 3D visualization is based mainly on two approaches. First, the observatory approach consists in reproducing the molecule’s 3D model from crystallographic data, which is not fairly well accepted in the case of DNA, because current techniques for the experimental study of DNA’s 3D structure (crystallography, cryo-electron microscopy, AFM microscopy, etc.) present many limitations (size, molecule’s deformations, etc..) and are very costly in time and price. Dispates, several threedimensional molecular viewers, simple and freely available, have emerged (gOpenMol, MoleKel, ViewMol, MolMol, PyMol, RasMol (Fig. 2), etc.). It is possible to have more examples in this study that we have conducted [9]. Besides, most modelisation concerns especially proteins thanks to a great 3D data bank (“PDB” format, Protein Data Bank). That's why 3D modeling techniques have been developed. Hence, the predictive approach aim to predict approximately the DNA’s 3D structure from its textual sequence (Fig. 3) thanks to spatial conformation model of DNA (obtained by statistical methods on small fragments of DNA), which is considerably easier to implement, cheaper and faster than the experimental methods in vitro.
Fig. 2. RasMol: free software for viewing PDB files
Fig. 3. 3DNA: local DNA’s 3D modeling by a predictive method
Fig. 4. DNA’s 3D model based on biophysical data
A New Approach to Design an Interactive System for Molecular Analysis
717
In our case, a part of the research was to propose a new DNA’s 3D modeling (Fig. 4) based on biophysical data (described in section 2.2). However, the backtracking algorithm that we use is also costly in computing time, in order to cover a large search space [1]. Therefore, the automatic processes of DNA’s 3D modeling and visualization must overcome an important difficulty to reach relevant results, which is the computation time of research algorithms. Hence, the DNA’s 3D modeling and analysis assisted by multi-modal VR-technology is a relevent alternative to enhance the result. In the past, the most interesting molecular 3D modeling/visualizing systems have been either too large or too complex to model in real time, but now, many molecular 3D simulation programs run efficiently on parallel computers. Thus, a new opportunity was offered to biologists who may now have an interactive control on those applications during their execution. 3.2 Interactive Approach Gradually the visualization interest no longer stops at the molecules observation, but it extends to the structure analysis and functionality interpretation of molecules by their forms. Moreover, the limitations of conventional 3D modeling algorithms and the new possibilities offered by Virtual Reality (VR), brought some research teams to look on problems related to interactive and immersive molecular modeling [10]. Early works in this area have focused on identification of technical needs and technological limitations. It was research in an area totally unknown by biology, at the time. Cyrus Levinthal, at MIT, built the first system for the interactive display of molecular structures in the mid-1960s. This system allowed a user to study interaction between atoms and the online manipulation of molecular structures. The speed and orientation of a molecular structure can be controlled by a globe-shaped device (a kind of trackball) (Fig. 5 shows the VR interface used)[11]. Moreover, it was possible to select from a menu, choose an object, or zoom into important parts of the molecule using a light pen on the computer display.
Fig. 5. The display terminal shows the molecule structure in wireframe fashion
Fig. 6. Stereoscopic visualization of chromosomes and immersive navigation with ADN-Viewer
718
M. Essabbah et al.
Subsequently, new interactive viewers appear in order to study the molecule dynamics through its three-dimensional structure. These specific applications offer more complex modeling like molecules functionality. The immersive system AMMPVis uses a virtual environment for collaborative molecular modeling [12]. It offers to biologists and chemists the possibility of manipulating molecular models through a natural gesture. It allows receiving and displaying real-time molecular dynamics simulation results. It allows also adapted views sharing and provides support for local and remote collaborative research. The highly immersive molecular modeling (HIMM) environment is built from computer aided molecular modeling tools (already done) and virtual reality (VR) engines [13]. The proposed architecture tends to provide with minimal effort a flexible HIMM environment. Other research has highlighted the importance of using VR for education, combining immersive environment (efficient visual representation) and haptic devises (interactive manipulation) to offer a good instructional aid for students [14]. Another system, ADN-Viewer [15] is particularly interested in DNA’s spatial distribution. The software offers 3D reconstruction of DNA’s structure through several 3D representations. It also provides the opportunity to explore chromosomes’ structures thanks to stereoscopic visualization (Fig. 6). In the same principle of stereoscopic navigation, a stereographic table has been built by Marchese and al. [16] to support molecular visualization. Scientists have developed many others interactive molecular visualization software even with collaborative technologies [17, 18]. However, they are often limited in only a part of the interface with the virtual reality (either navigation, or selection, or manipulation, etc.) and this is insufficient to really explore the ability of multimodal interaction in VR to improve the molecular analysis.
4 A Hybrid and Multimodal Approach 4.1 Hybrid Approach Our approach aims to combine the contribution of multimodal rendering and the experts’ knowledge in certain phases of the study. In fact, multimodal vitual environments succed better tha single-sensory technologies in creating a sense of presence [19]. In allowing the user to interact earlier in the process rather than post or by setting the single system, we hope to reduce the computing time and also the risks of wrong 3D models. More Specifically, the protocol that we develop is user-centred. It describes a cycle that alternates successively using the automatic and the interactive approach of modeling by the user’s command (illustrated in Fig. 7). The first step is related to the automatic procedure of modeling, and will generate an initial 3D structure, with the geometrical computing engine. Then, in the immersive environment, the biologist will be able to improve the 3D model in the configurations that seems interesting through visual, auditory and haptic renderings. During this stage, the user has, as reference, a set of biological models at its disposal. This first step will quickly reduce the search space of possible conformations, being based on the expert’s capacity of analysing 3D patterns and his knowledge of the specific molecule and biological models. Finally, the expert can choose to restart automatic generation of 3D model locally or globally-, but taking into account the current changes.
A New Approach to Design an Interactive System for Molecular Analysis
719
Fig. 7. Hybrid approach for the assistance to DNA’s analysis
4.2 Hardware Architecture The hardware architecture of our molecular modeling environment is based on a semi-immersive VR/AR multimodal platform called EVR@1 (see Fig. 9). It permits stereoscopic display, wireless hand/head gesture tracking and force feedback. We have ART optical tracking system composed of two ARTTrack1 infrared cameras, two wireless Flystick, one head marker and one hand marker. We also have a 6D force feedback device SPIDAR-G2 [20] and 5DT Data Gloves. Each device is associated to a specific server, which is accessed via the C++ VRPN library [21] by clients. The interactivity between the user and the VE is done by using Virtools™ 4.0 [22] as a front-end. 4.3 Software Architecture To take into account these constraints, we orient our approach to a modular architecture (Fig. 8). The modular approach, beside the advantage of reusability, is also able to modify or adapt more easily an application. We will refine the components through the design phase. The architecture of a classical 3D interaction is often divided into two parts: the user and the different devices, and the application, which try to communicate with the user through these devices. So, we propose the design of a new 3D interaction paradigm, using what we call a bio-supervisor, by adding this new component between the user and the VR application. The supervisor concept has already been used for 3D multimodal interaction by Bouyer and al. [23] in order to make the 3D interaction more intuitive for the user, but without taking into account the domain (ie. the environment). 1 2
http://evra.ibisc.univ-evry.fr Space Interface Device for Artificial Reality.
720
M. Essabbah et al.
Fig. 8. An assistance system for molecular analysis based on bio-supervised 3D interaction
In our case, the supervisor role will be to control the user's actions according to the biological constraints that we have already mentioned. When interacting, if the user exceeds any of these constraints, the bio-supervisor must let him know while limiting its action. Therefore, a multimodal interaction must be set up so that the user becomes aware of the limits imposed by the supervisor by a visual, auditory and haptic feedback. For instance, the task to perform is to curve the chromatin (molecule), however, the chromatin's energy restrains the curvature, imposing a physical limit. In this case, the user can get a visual and auditory feedback through a classical workstation, however, he can exceed the limits and continue to twist the chromatin whereas the 3D model does not follow the command. Thus, there is a contradiction between user's action and the 3D model rendering. In contrast, in addition to the known benefits (a real 3D rendering, a human scale, an immersion, etc.), a virtual reality platform (see Fig. 9) will provide coherence between the user's action and the
Fig. 9. Handling a 3D molecule through our semi-immersive virtual/augmented reality platform: Evr@
A New Approach to Design an Interactive System for Molecular Analysis
721
limits imposed on the 3D model. Hence, through a multimodal feedback system (visual, auditory, haptic, etc.), the user will become aware of the 3D model limit, but it also will feel it and will be restricted on it.
5 Conclusion We have described in this paper a new approach to design 3D interaction techniques context-aware, that is an important goal in our work in order to propose a 3D molecular modeling system, more natural and intuitive. Our objective is to set up a framework of 3D interaction favorable to the complex and constraining biological environment. Indeed, through the 3D interaction, we want to provide an active assistance for the manipulation and the analysis of DNA in a very intuitive way. Moreover, we can take advantage of the multi-modality to offer to the user the maximum of information that could assist him, for example, use the force feedback through haptic devices in order to represent more naturally the constraints and limits of biology. The next work is the evaluation of the proposed assistance system with subjective and objective assessments.
References 1. Essabbah, M., Hérisson, J., Otmane, S., Mallem, M.: Towards a biophysical 3D model of the DNA. In: 1th IEEE International Workshops on Image Processing Theory, Tools and Applications, Sousse, pp. 1–6 (2008) 2. Robinson, P.J.J., Fairall, L., Huynh, V.A.T., Rhodes, D.: EM measurements define the dimensions of the “30-nm” chromatin fiber: Evidence for a compact, interdigitated structure. Proc. Natl. Acad. Sci. USA 103(17), 6506–6511 (2006) 3. Bystricky, K., Heun, P., Gehlen, L., Langowski, J., Gasser, S.M.: Long-range compaction and flexibility of interphace chromatin in budding yeast analyzed by high-resolution imagining techniques. Proc. Natl. Acad. Sci. USA 101(47), 16495–16500 (2004) 4. Cremer, T., Cremer, M., Dietzel, S., Muller, S., Solovei, I., Fakan, S.: Chromosome territories – a functional nuclear landscape. Current Opinion in Cell Biology 18, 307–316 (2006) 5. Dekker, J., Rippe, K., Dekker, M., Kleckner, N.: Capturing Chromosome Conformation. Science 295, 1306 (2002) 6. Cook, P.R.: Predicting three-dimensional genome structure from transcriptional activity. Nature genetics 32, 347–352 (2002) 7. Marenduzzo, D., Faro-Trindade, I., Cook, P.R.: What are the molecular ties that maintain genomic loops? TRENDS in Genetics 23(3), 126–133 (2006) 8. Képès, F., Vaillant, C.: Transcription-BasedSolenoidalModel of Chromosomes. Complexus 1, 171–180 (2004) 9. Essabbah, M., Otmane, S., Mallem, M.: 3D molecular modeling: from theory to applications. In: IEEE Conference on Human System Interaction (HSI 2008), Krakow, Poland, pp. 350–355 (2008) 10. Marchese, F.T.: Molecular Visualization at the Interface. In: Trends in Interactive Visualisation / Advanced Information and Knowledge Processing Series, pp. 251–268. Springer, London (2009) 11. Levinthal, C.: Molecular model-building by computer. Sci. Am. 214(6), 42–52 (1966)
722
M. Essabbah et al.
12. Chastine, J.W., Brooks, J.C., Zhu, Y., Owen, G., Harrison, R., Weber, I.T.: AMMP-Vis: a collaborative virtual environment for molecular modeling. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST 2005, Monterey, CA, USA, pp. 8–15. ACM Press, New York (2005) 13. Drees, R.C., Pleiss, J., Schmid, R.D., Roller, D.: Integrating molecular modeling tools and virtual reality engines: an architecture for a highly immersive molecular modeling (HIMM) environment. In: Proceedings of the computer graphics international, pp. 391– 392. IEEE Computer Society, Washington (1998) 14. Sato, M., Liu, X., Murayama, J., Akahane, K., Isshiki, M.: A Haptic Virtual Environment for Molecular Chemistry Education. T. Edutainment 1, 28–39 (2008) 15. Hérisson, J., Gherbi, R.: Model-Based Prediction of the 3D Trajectory of Huge DNA Sequences Interactive Visualization and Exploration. In: 2nd IEEE International Symposium on Bioinformatics and Bioengineering (BIBE 2001), pp. 263–270 (2001) 16. Marchese, F.T.: A stereographic table for biomolecular visualization. In: Proceedings of the sixth international conference on information visualization: IV 2002, pp. 603–607. IEEE Computer Society, Washington (2002) 17. Marchese, F.T., Brajkovska, N.: Fostering asynchronous collaborative visualization. In: Proceedings of the 11th international conference information visualization: IV 2007, pp. 185–190. IEEE Computer Society, Washington (2007) 18. Su, S., Loftin, R., Chen, D., Fang, Y., Lin, C.H.: Distributed collaborative virtual environment: PaulingWorld. In: Proceedings of 10th international conference on artificial reality and telexistence, pp. 112–117 (2000) 19. Hecht, D., Reiner, M., Halevy, G.: Multimodal virtual environments: response times, attention, and presence. Presence: Teleoper. Virtual Environ. 15, 515–523 (2006) 20. Kim, et al.: Development of tension based haptic interface and possibility of its application to virtual reality. In: Proceedings of the ACM symposium on Virtual reality software and technology, pp. 199–205 (2000) 21. Taylor, et al.: VRPN: a device-independent, network-transparent VR peripheral system. In: VRST 2001: Proceedings of the ACM symposium on Virtual reality software and technology, pp. 55–61 (2001) 22. Dassault Systemes, http://www.virtools.com/ 23. Bouyer, G., Bourdot, P.: Supervision of 3d multimodal rendering for protein-protein virtual docking. In: Eurographics Symposium on Virtual Environments (EGVE), vol. 28, pp. 49–56 (2008)
The Differences of Aviation Human Factors between Individualism and Collectivism Culture Wen-Chin Li1, Don Harris2, Lon-Wen Li3, and Thomas Wang4 1
Psychology Department, National Defense University, Taiwan, R.O.C. [email protected] 2 Human Factors Department, Cranfield University, United Kingdom 3 Training Centre, National Defense University, Taiwan, R.O.C. 4 Flight Safety Division, Aviation Safety Council, Taiwan, R.O.C.
Abstract. Culture is at the root of action; it underlies the manner by which people communicate and develop attitudes towards life. This research examined statistical differences in the 18 categories of Human factors Analysis and Classification System (HFACS, Shappell & Wiegmann, 2003) across 523 aviation accidents in the Republic of China (a collective culture) and 119 aviation accidents in the USA (an individual culture) . The result suggests that the culture of individualism seems to be superior for promoting aviation safety compared to collectivist cultures, however, factors such as the design of the aircraft, the management procedures and the nature of safety regulation all have a strong Western influence from the individualist culture. All of these factors are culturally congruent with the USA. It is essential to identify the potential causal roots for these differences from the underlying factors in these aviation mishaps, and identify what kind of factors drive people to act or react to dynamic situations that either lead to an accident help to develop an effective accident prevention strategy. Keywords: Accident Investigation, Aviation Safety, Cross-culture, Human Factors.
different paths in developing government and in developing science and technology. Whereas the Romans spread the principle of 'government by law', the main continuous principle of Chinese was 'government by man' [3, 4]. Soeters and Boer [13] found that more individualist cultures showed a lower probability of total loss accidents. On the other hand, collectivist cultures exhibited a greater chance of accidents. As aircraft have become increasingly more reliable, human performance has played a proportionately increasing role in the causation of accidents. In recent years, in accident investigation the scientific focus has shifted away from psychomotor skill deficiencies and emphasis is now more placed upon inadequacies in decision-making, attitude, supervisory factors and organizational culture as being the primary causal factors [1, 5, 8]. Based upon Reason’s model [12] of human error in which active failures are associated with the performance of front-line operators in complex systems and latent failures are characterized as inadequacies or mis-specifications which might lie dormant within a system for a long time and are only triggered when combined with other factors to breach the system’s defenses, the Human Factors Analysis and Classification System (HFACS) was developed as an analytical framework for the investigation of the role of human factors in aviation accidents. The HFACS was originally designed and developed as a generic human error framework for investigating and analyzing human error accidents in US military aviation operations. HFACS has been shown to be useful within the context for civil and military aviation, as both an effective data analysis framework and a reliable accident investigation tool for over twenty years [16]. Recently, research comparing the underlying patterns of causal factors in accidents comparing Eastern and Western cultures has suggested underlying differences attributable to culture. Using the Human Factors Analysis and Classification System, it was observed that issues concerning inadequate supervision at higher managerial levels and sub-optimal organizational process were more likely to be implicated in accidents involving aircraft from Eastern cultures [11]. It was suggested that small power-distance cultures with a high degree of individualism seemed to be superior to collective with high power-distance cultures for promoting aviation safety, especially in terms of the processes and procedures at the higher organizational levels. Such an analysis may provide additional explanatory power to elucidate why national differences in accident rates occur, however, it provides no explanatory power to explain why individualist cultures were safer than collective cultures in the aviation industry. The power of culture often goes unrecognized since it represents 'the way we do things here'. There have been several studies investigating the relationship between culture and accident causal patterns [2, 7, 13]. However, no research has investigated specifically the relationship between collectivist cultures and individualist cultures to the underlying causes of accidents. There is an increasing need for investigating the relationship between Chinese culture and safety of aviation operations for the both of Chinese population and the South Eastern Asian market for aviation industry.
2 Method Data: There were 523 accidents with 1,762 instances of human error categorized using the HFACS framework from data collected by the Taiwan Air Force between 1978 and
The Differences of Aviation Human Factors
725
2002 [10]; and 119 accidents with 319 of categorized instanced of human error in US data recorded between 1990 and 1996 [15]. According to Hofstede’s [4] cultural dimension of Individualism versus Collectivism, the score of Taiwan is 17, the score of US is 91; the world average is 43. It is clear that Taiwan is a collective culture, the US is an individualist culture. It is hypothesized that these different cultures will show different patterns in the underlying causal factors in aircraft accidents. Classification Framework: This study based on the HFACS framework as described in Wiegmann & Shappell [14 - 16]. The first level of HFACS categorizes is ‘unsafe acts of operators’ that can lead to an accident including four sub-categories of 'decision errors'; 'skill-based errors'; 'perceptual errors' and 'violations' . The second level of HFACS concerns 'preconditions for unsafe acts' which has a further seven sub-categories of 'adverse mental states'; 'adverse physiological states'; 'physical/mental limitations'; 'crew resource management'; 'personal readiness'; 'physical environment', and 'technological environment'. The third level of HFACS is ‘unsafe supervision’ including 'inadequate supervision'; 'planned inappropriate operation'; 'failure to correct known problem', and 'supervisory violation'. The fourth and highest level of HFACS is ‘organizational influences’ and comprises of the sub-categories of 'resource management'; 'organizational climate' and 'organizational process'. To avoid over-representation from any single accident, each HFACS category was counted a maximum of only once per accident. These counts acted simply as an indicator of presence or absence of each of the 18 categories in any given accident. These data were then subject to chi-square (χ2) analyses to measure the statistical strength of association between HFACS category and country. Reliability of HFACS Framework: Inter-rater reliabilities for the data from Taiwan, calculated as a simple percentage rate of agreement, the reliability figures for the 18 categories of HFACS between 72.3% and 96.4% [10]. The average of the inter-rater reliabilities of the data gathered from the US data was 76% [15].
3 Results and Discussion There were six HFACS categories that exhibited significant differences in the frequency of underlying causes in aviation accidents between Taiwan and US (Table 1). The category of ‘resource management’ (χ2= 50.09, df=1, p=.000) at level-4 was over-represented in Taiwan and was under-represented in the USA. The category of ‘inadequate supervision’ (χ2= 39.45, df=1, p=.000) at level-3 was over-represented in the Taiwan sample and under-represented in the USA. There were two categories with significant differences in frequency of occurrence between Taiwan and the USA at level-2: ‘personal readiness’ (χ2= 6.91, df=1, p=.008) was over-represented in Taiwan, and under-represented in US accidents; ‘adverse mental states’ (χ2= 21.35, df=1, p=.000) was over-represented in the Taiwan, and under-represented in the USA sample; Finally, there were two HFACS categories which showed differences in the frequency of occurrence between the two regions at level-1: ‘decision errors’ (χ2= 7.99, df=1, p=.004) was over-represented in Taiwanese sample and under-represented in the US sample; and ‘skilled-based errors’ (χ2= 11.65, df=1, p=.000) were over-represent in US accidents but under-represented in the sample from Taiwan.
726
W.-C. Li et al. Table 1. The Frequency of HFACS Categories in Taiwan and USA HFACS Categories
Taiwan
USA
Yes
No
Yes
No
Organizational process
76 80
447 443
10 18
109 100
Organizational climate
4 7
519 516
0 2
119 117
Resource management
184 156
339 366
3 36
116 83
Supervisory violation
8 9
515 514
2 2
117 117
Failed correct a known problem
12 12
511 511
2 3
117 116
Planned inadequate operations
24 22
499 501
1 5
118 114
Inadequate supervision
177 144
346 379
6 33
113 86
Technology environment
44 na
479 na
na na
na na
Physical environment
74 na
449 na
na na
na na
Personal readiness
29 25
494 498
0 6
119 113
Crew resource management
146 142
377 381
35 32
84 87
Physical/mental limitation
73 77
450 446
13 17
106 102
Adverse physiological states
2 5
521 518
2 1
117 118
Adverse mental states
184 156
339 367
16 36
103 83
Violations
160 158
363 365
32 36
87 83
Perceptual errors
116 106
407 417
17 24
102 95
Skilled-based errors
226 245
297 278
72 56
47 63
Decision errors
223 202
300 321
34 46
85 73
The Differences of Aviation Human Factors
727
Hofstede [4] defined the dimension of Individualism (IDV) as:
‘A High Individualism ranking indicates that individuality and individual rights are paramount within the society. Individuals in these societies may tend to form a larger number of looser relationships. A Low Individualism ranking typifies societies of a more collectivist nature with close ties between individuals. These cultures reinforce extended families and collectives where everyone takes responsibility for fellow members of their group’. There are fundamental difference between Chinese minds and Western minds. Through their different logics, the main continuous principle of the Chinese was 'government by man', focusing on the inter-relationship in the environment, which resulted in very different ways of making inferences about the world. Westerners followed different paths in developing science and technology. American emphasis on identity is based in the individual and the same rules should apply to everyone as justice should be blind. The category of ‘Resource management’ (level-4) includes the selection, staffing and training of human resources at an organizational level, excessive cost cutting, providing unsuitable equipment, and a failure to remedy design flaws. It is clear for over-represented in Taiwan and under-represented in US, as Chinese society is relationship-orientation which means people need to have a connection with the person in charge for getting the necessary resources easily. Resources were managed more unevenly (or unfairly) in a collectivist culture than individualist culture. As a result, collectivist cultures exhibit a greater likelihood of accidents than individualist culture [13]. The category of ‘inadequate supervision’ (level-3) includes factors such as a failure to provide proper training, a lack of accountability, failure to track qualifications and performance, using untrained supervisors and loss of situation awareness at the supervisory level. It was over-represented in the Taiwanese sample and underrepresented in the US sample. In the Chinese culture emphasis is on 'government by man' and 'harmony priority' to keep face for each other. As a result, the principles and regulations for flight operations were applied flexibly. This can also be supported from the frequency of violation of SOPs. There is a famous Chinese saying ‘open one eye, close the other eye’ for different regulations applying to different people with different relationships. Western culture believes in absolute guidelines about right and wrong, Chinese culture believes what is right and wrong depends on the circumstances. This may be illustrated from the US data which has a lower accident rate in the category of ‘inadequate supervision’ than Taiwan. The supervisory levels in Taiwan were not following strict principles when performing their duties which caused problems. There were two categories with significant differences between Taiwan and the US at ‘Preconditions for unsafe act’ (level-2). Both were over-represented in their frequency of occurrence in Taiwan, and under-represented in US accidents. ’Adverse mental states’, which includes issues such as stress, loss of situational awareness, distraction and task saturation; and ‘Personal readiness’ which encompassed issues associated with inadequate training, self-medication, poor diet, and overexertion while off duty. In patterns of attention and perception, Chinese attend more to the whole
728
W.-C. Li et al.
environment: Americans attending more to individual objects. Westerners prefer the use of formal logical rules to understand events. As a result, Westerners more precisely identify the problems in front of them than Easterners do. There were two categories with significant differences between Taiwan and US at the level of ‘Unsafe act of operators’ (level-1). The category of ‘Decision errors’, which includes issues such as selecting inappropriate strategies to perform a mission; improper in-flight planning; making an inappropriate decision to abort a take-off or landing; or using improper remedial actions in an emergency, was over-represented in the Taiwanese sample and under-represented in the US sample. In the development of science and technology, American culture stimulates analytic thinking, whereas Chinese is inclined to synthetic thinking [4]. This cultural characteristic is illustrated by the Chinese having a higher instance of ‘decision errors’ than the US sample, perhaps because the cockpit designs are based upon a Western approach. It is not necessary true that the analytic thinking approach is a safer approach than the synthetic thinking in aviation domain. However, there was only one category over-represented in US accidents but under-represented in the sample from Taiwan, that was ‘Skill-based errors’, which includes actions such as inappropriate stick and rudder coordination; excessive use of flight controls; glide path not maintained, and adopting an improper airspeed or altitude. It may be suggested that the explanation for these observations is that the US has a culture which prefers individual decision making and responsibility for the self, they believe more in the controllability of situation than Chinese. In Hofstede’s [4] terms the USA is an ego-oriented society. The difference between individualist and collectivist cultures was found to be based on the ways of communication. High-context communication fits the collectivist society, and low-context communication is more typical for individualist cultures [4]. High-context communication implies that little has to be said because most of the information is either in the physical environment or internalized in the person. On the other hand, low-context communication implies that the mass of information is made explicit. The US culture has strong desire searching for truth and governed by ‘law’, the regulations are clearly specific to follow. The Chinese tradition does not hold laws and abstract principles in high regard and is governed by ‘man’, only a small part is in the coded. This could possibly explain the Taiwanese higher accident rate involving ‘resource management’ (level-4) as Chinese culture is relationship orientated. Value standards differ for in-groups and out-groups. Furthermore, Western culture believes in absolute guidelines about good and evil, Chinese culture believes what is good and evil depends upon the circumstances. It might be illustrated from the US data which has a lower accident rate in the category of ‘inadequate supervision’ (level-3) than the Taiwanese sample. The supervisory levels in Taiwan were not following strict principles to perform their duties which caused problems. The cultural difference which dictates that ‘ability most important for career of Westerners’ with ‘employees responsible for themselves’ might explain the lower US rate of ‘adverse mental states’ (level-2) being involved in accidents than in the Taiwanese sample. Also, the ‘knowing the right person most important for career’ in collectivist cultures underlay the precondition for unsafe acts and caused a higher ‘personal readiness’ problem (level-2) in Taiwan than in the US.
The Differences of Aviation Human Factors
729
4 Conclusions People from different nations differ in their cognition in ways that result in dissimilar perceptions, judgments and decision-making [9]. National culture provides a fundamental basis for a group member's behavior, social roles and cognitive processes. It also provides underlying rules about safety, effective communication, and provides the basis for verbal and nonverbal interactions. This research, using the HFACS framework suggests that there are six categories having significant differences in the relative frequencies of the underlying human factors causes in aviation mishaps between Taiwan and US. The underlying cultural causes of these differences are postulated. It should be noted, the individualist culture seems to be superior for promoting aviation safety compared to the collectivist cultures. However, factors such as the design of the aircraft, the management procedures and the nature of safety regulation all have a strong Western influence from an individualist culture. All of these factors are culturally congruent with the USA. It is essential to identify the potential causal roots for these differences in relative frequency of the underlying factors in these aviation mishaps, and identify what kind of factors drive people to act or react in the dynamic situations that lead to an accident to develop effective accident prevention strategies
References 1. Diehl, A.: The Effectiveness of Training Programs for Preventing Aircrew Error. In: Sixth International Symposium on Aviation Psychology, pp. 640–655. The Ohio State University (1991) 2. Helmreich, R.L., Merritt, A.C.: Culture at Work in Aviation and Medicine: National, Organizational and Professional Influences, Ashgate, Aldershot, England (1998) 3. Hofstede, G.: Cultures and Organizations: Software of the Mind. McGraw-Hill Book Company, London (1991) 4. Hofstede, G.: Culture’s Consequences: Comparing Values, Behaviors, Institutions, and Organizations Across Nations. Sage Publications, California (2001) 5. Jensen, R.S.: The Boundaries of Aviation Psychology, Human Factors, Aeronautical Decision Making, Situation Awareness, and Crew Resource Management. The International Journal of Aviation Psychology 7(4), 259–268 (1997) 6. Jing, H.S., Lu, C.J., Peng, S.J.: Culture, Authoritarianism and Commercial Aircraft Accidents. Human Factors and Aerospace Safety 1, 341–359 (2001) 7. Johnston, N.: CRM: Cross-Cultural Perspectives. In: Wiener, E.L., Kanki, B.G., Helmreich, R.L. (eds.) Cockpit Resource Management, pp. 367–398. Academic Press, San Diego (1993) 8. Klein, G.: Analysis of Situation Awareness from Critical Incident Reports. In: Endsley, M.R., Garland, D.J. (eds.) Situation Awareness Analysis and Measurement. Lawrence Erlbaum, London (2000) 9. Klein, H.A.: Cognition in Natural Settings: The Cultural Lens Model. In: Kaplan, M. (ed.) Cultural Ergonomics, pp. 249–280. Elsevier, San Diego (2004) 10. Li, W.-C., Harris, D.: Pilot error and its relationship with higher organizational levels: HFACS analysis of 523 accidents. Aviation Space Environmental Medicine 77(10), 1056–1061 (2006)
730
W.-C. Li et al.
11. Li, W.-C., Harris, D., Chen, A.: Eastern Minds in Western Cockpits: Meta-analysis of human factors in mishaps from three nations. Aviation Space and Environmental Medicine 78(4), 420–425 (2007) 12. Reason, J.: Human Error. Cambridge University, New York (1990) 13. Soeters, J.L., Boer, P.C.: Culture and Flight Safety in Military Aviation. International Journal of Aviation Psychology 10, 111–134 (2000) 14. Wiegmann, D.A., Shappell, S.A.: Human Factors Analysis of Postaccident Data: Applying Theoretical Taxonomies of Human Error. The International Journal of Aviation Psychology 7(1), 67–81 (1997) 15. Wiegmann, D.A., Shappell, S.A.: Human Error Perspectives in Aviation. The International Journal of Aviation Psychology 11(4), 341–357 (2001) 16. Wiegmann, D.A., Shappell, S.A.: A Human Error Approach to Aviation Accident Analysis: The Human Factors Analysis and Classification System, Ashgate, Aldershot, England (2003)
Web-Based Training System for Improving Aviation Maintenance Performance Guo-Feng Liang1, Jhih-Tsong Lin1, Sheue-Ling Hwang1, Eric Min-yang Wang1, Patrick Patterson2, and Jiun-Fa Li3 1 Department of Industrial Engineering and Engineering Management, National Tsing Hua University, 101, Section 2, Kuang-Fu Road, Hsinchu 300, Taiwan 2 Department of Industrial Engineering, Texas Tech University, Box 43061, Lubbock, TX 79409-3061 3 Department of Aviation Mechanical Engineering, China Institute of Technology, No.200, Zhonghua St., Hengshan Shiang, Hsinchu County 312, Taiwan [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Abstract. To increase aviation maintenance and inspection safety, we propose a web-based training system (WBTS) for technician training and performing maintenance tasks. Toward this goal, the risks of human errors were considered for each procedure from the perspectives of performance shaping factors (PSFs). WBTS functions include English and Chinese explanations, human error effects on human-machine system, human errors relative to serious rankings and frequency, and graphic information aid in each component removal and installation procedure. To verify the proposed platform, experiments were conducted on a JT8D engine during the inaugural flight of Boeing’s 727 to compare traditional workcard and proposed WBTS in two complex teamwork tasks. The results revealed that teams’ risk cognition, situation awareness, and performance have been increased by proposed WBTS comparing to that by the traditional work-card instructions. Keywords: Human error, performance shaping factor, maintenance incidents, WBTS, situation awareness, workload.
replaced by a much younger and less experienced workforce [10]. Therefore, potential human errors exist in the maintenance jobs. International Air Transport Association (IATA) in 2003 reported that maintenance factors initiated the accident chain in 26% of 92 accidents. Maintenance errors are responsible for an estimated 20% to 30% of engine and aircraft equipment in-flight shutdowns [6]. These are usually latent errors whose actions have a delayed effect [12]. For instance, maintenance errors may require air turn-backs, delays in aircraft availability, gate returns, in-flight shutdowns, maintenance rework, damage to maintenance equipment, and injury to maintenance personnel. Thus, human errors in maintenance need to be addressed and prevented. Since the 1980s, Rasmussen’s widely used skill-, rule-, and knowledge-based (SRK) framework has been applied in a wide variety of settings [13]. Hobbs and Williamson [7] investigated aircraft maintenance errors based on the SRK classification model. Their results showed that 60.7% of errors were skill-based, 27.1% were rule-based errors, and 11.3% were knowledge-based errors (the remaining 0.9% of errors were unclassifiable). To reduce human errors and increase team performance, as well as system and human safety, intervention design for error reduction is most effectively implemented when used in combination with computer-based training and aiding systems [11]. A workcard, which provides specific instructions on the task to be accomplished, is the primary job aid for aircraft maintenance and inspection. It describes the inspection procedures of the aircraft, the warning signals about aircraft and personal safety, and the details of tools and equipment. However, the typical workcard does not conform well to the principles of good information design and does not meet user expectations [3]. Computer-based training (CBT) and aiding programs have also been developed for aviation inspection and maintenance technicians, with a good deal of research has been done on the computer-based approach. Chandler [2] traced a sophisticated computer-based training program, the System for Training Aviation Regulations (STAR), into an on-the job training program aid for aviation safety inspectors. It was found that (1) the system needs to be highly modular, (2) applying a browser approach to interface design increases the options for how the system can be used, and (3) electronic publishing may offer some solutions for training systems that rely on living documents as the basis of their training. Andresen et al. [1] performed a series of simulator experiments on computer-based procedure (CBP) systems in which crews of licensed operators engaged in safety-related scenarios while operating from CBP systems with varying degrees of automation. The results were used as a general framework for obtaining input to human reliability analysis (HRA) in simulator experiments. Drury et al. [4] applied human-factor guidelines to the design of aircraft maintenance documentation intended to replace paper-based workcards with a portable computer system. It was found that the computer-based system was easier to understand, increased the organization and consistency of information, and increased overall workcard usability. Previous studies have proposed various computer-based procedure (CBP) systems, and computer-based training (CBT) and aiding programs to ensure safety. However, few studies have included detailed analysis of potential human error relative to systems and individuals, or looked at team impact on each maintenance procedure. In fact, the system needs a more effective pre-warning mechanism to remind technicians before the maintenance tasks are started. Thus, the purposes of this study are (1) to
Web-Based Training System for Improving Aviation Maintenance Performance
733
design a web-based training system (WBTS) for improving aviation training performance that includes a pre-warning model and risk evaluation of the procedure, (2) to conduct an engine assembly and disassembly experiment comparing the current workcard with the aircraft maintenance manual (AMM) and the proposed WBTS in real aircraft, (3) to quantify the human error impact risk index (IRI), and (4) to evaluate WBTS performance using subjective and objective measures.
2 Case Study - X Airline Company To design a human-error preventive system for aircraft maintenance tasks, 40 maintenance events and their cause relative to human errors of X Airline Company from 2006 to 2007 were analyzed. The causality of contributing factors can be identified and analyzed to develop improvement strategies and prevent failure caused by maintenance tasks. The investigation reports contain three categories for reporting error occurrences including (1) events- the event frequencies were accumulated over 24 months. The highest-frequency event was operational process event (45%), (2) maintenance error - maintenance errors included installation error, fault isolation/test/inspection error, foreign object damage error, airplane/equipment damage error, and personal injury error. The highest-frequency maintenance error was incomplete installation and tools/equipment used improperly, and (3) contributing factors checklist which jobs/tasks (21%) in repetitive/monotonous jobs/tasks design, complacency, and work process/procedure not followed were critical contributing factors to maintenance errors. From these results, it is clear that prevention of maintenance errors requires focus on the issues of (1) repetitive/monotonous jobs/tasks, (2) complacency, and (3) work process/procedure not followed. Therefore, the WBTS was designed for this study. An experiment was then conducted to evaluate the effects of the proposed WBTS.
3 Method 3.1 Research Platform The web-based training system (WBTS) was developed to embed the maintenance risk for each technician. The platform is similar to an expert system, which continuously collects expert knowledge and experience into a database. From the case study results, the critical item and its procedure were first investigated. Procedure analysis was conducted by managers, senior technicians, and experts (e.g., training instructors). Then, the decomposition of each procedure included potential human errors, their effect on the system and humans, and the human error impact risk index (IRI) relative to the seriousness rank and frequency of human error. Key functions of the WBTS included: 1. Explanation of each procedure in English/Chinese language The platform extended the procedure of Aircraft Maintenance Manual (AMM) to provide the language aid in Chinese, as well as English.
734
G.-F. Liang et al.
2. Potential human error analysis in each procedure Human error could exist in each procedure when technicians made errors of omission, commission or violation [14]. Knowledge about potential human errors may be acquired from the experiments or lessons and, after discussions with managers, senior technicians and experts, the initial database of potential human error was established. 3. The effect of each procedure’s potential human error on the system and human Error effects on systems and humans were analyzed to form a database. 4. Human error impact risk index (IRI) of each procedure The human error impact risk index (IRI) considers the effects of human error on both systems and humans and weights those effects. The scales of the effect on systems and the effect on humans are given a numerical value from 0 to 4 (0=no effect, 1=minor effect, 2=middle effect, 3=major effect, and 4=fatal effect) on the x axis and y axis. The point (x, y, z) considers the effect of error on the systems (x), the effect of the error on humans (y), and the weight of the effect (z). The degree of effect on each procedure j can be obtained as follows:
IRI j = z j × f j where IRI j represents the impact risk index (IRI) of the
(1)
jth procedure; z j is the weight
jth procedure; and f j is the error frequency in the evaluation stage. Thus, the potential error effect of task i can be accumulated by the sum of j procedures and be expressed by of error effect on systems and humans of the
IRIi = ∑ IRI j = ∑ z j × f j j
(2)
j
5. Graphic information aid in each removal and installation procedure Graphic information can reduce display density and show all necessary spatial, numeric, and temporal information [4]. In this training platform, the graph of each maintenance procedure was used to assist the technicians in the maintenance tasks. The next section reviews an experiment conducted to evaluate the performance of the proposed WBTS in reducing maintenance errors caused by work procedures not being followed and for repetitive/monotonous jobs/tasks. 3.2 Participants
Forty-nine participants on fourteen teams took part in the experiment and were paid NT$1000 (around $33) for their participation. Participants consisted of (1) 28 nonexperts who had just finished the China Institute of Technology Aviation Mechanical Engineering School with one year of training and had received the Mechanic License Test Requirement issued by the Council of Labor Affairs Executive Yuan Taiwan, (2) 20 experts who had been employed with X Airline Company in Taiwan for 2-17 years (mean=4.73 years, S.D.=3.51 years), and (3) one judge who held the certificate of Civil Aviation Authority and had been in the aircraft maintenance and research field for more than 30 years. Before running the formal experiment, six non-experts in two teams participated in a pilot experiment. The remaining forty-two participants engaged in the formal experiment.
Web-Based Training System for Improving Aviation Maintenance Performance
735
3.3 Equipment and Apparatus
The equipment used in this study were (1) the JT8D engine of the inaugural flight of Boeing’s 727, (2) a human-computer interface, WBTS, with Aircraft Maintenance Manual (AMM) for training purposes, (3) a typical workcard with Aircraft Maintenance Manual (AMM), and (4) repair tool kits. 3.4 Experimental Procedures
At the beginning of the experiment, the experimental task and procedure were explained to the participants, and they signed an informed consent to participate. Then the complex maintenance procedure for the Engine Driven AC Generator – removal/ installation tasks and the Generator Constant Speed Drive – removal/installation tasks were run. Because the Engine Driven AC Generator was located in front of the Generator Constant Speed Drive, the sequences of four maintenance tasks were (1) remove Engine Driven AC Generator, (2) remove Generator Constant Speed Drive, (3) install Generator Constant Speed Drive, and (4) install Engine Driven AC Generator. For these four tasks, the procedures of each task were discussed by the team members while each team leader was instructed on how to perform the removal/installation procedures. Then the team implemented the maintenance tasks by using either the WBTS or workcard, depending on the assigned training instruction method. Once the team was assigned to work by using the WBTS, team members were required to announce the potential human errors, error effect, and error impact risk on each procedure. At the end of the Engine Driven AC Generator removal, workers took a 10-minute rest during which situation awareness (SA) questions were given to each team member. The second time, situation awareness (SA) questions were asked at the end of the Generator Constant Speed Drive installation. Finally, a subjective questionnaire that included the NASA task load index (NASA-TLX) was filled out by the team members after the experiment. Each team took about 3 hours to complete the experiment. 3.5 Experimental Variables
The independent variables for this study were two training instructions, WBTS and workcard, for the maintenance and inspection tasks in the aircraft’s engine components. The three dependent variables in this experiment were: 1. Subjective questionnaire: The subjective questionnaires were surveyed according to the PSFs in the maintenance tasks. Five PSFs relative to two different maintenance training instructions were classified as: communication and coordination, information display, software/hardware, mental workload evaluation of NASA-TLX, and subjective situation awareness (SA). 2. Objective performance measures: To evaluate the objective performance of each team, an aircraft maintenance expert observed and recorded the performance, during the entire maintenance and inspection tasks.
736
G.-F. Liang et al.
4 Results 4.1 Subjective Questionnaire Analysis
To compare the technician’s risk knowledge between traditional workcard and WBTS training instructions, the importance of each PSF was evaluated. A paired t-test indicated the technicians had higher subjective situational awareness using the WBTS than using the workcard (p<.001 for all items). The results of communication and coordination were significantly different in the two instructions (p<.001), suggesting that team members can get more assistance, a more common view, greater efficiency discussion, and more error warning signal from WBTS instruction. The two methods of instruction were also significantly different (p<.001), in terms of the information, where text displays, graph display, context, error effect and error risk message, and procedures assisted team members more in WBTS instruction than in workcard instruction. The result for software/hardware design showed a significant difference between workcard and WBTS (p<.001) revealing that traditional workcard design had less extendable flexibility, harder-to-read interface, and ineffective reference search. The rating scales of NASA-TLX for each item were from 0 to 100 (0=lowest workload, 100= highest workload). Paired t-test (p<.001) indicated the mental workload was lower in WBTS (mean = 42.57) than in workcard (mean = 63.1). 4.2 Objective Performance Analysis Expert Grades and Evaluations An aircraft maintenance expert observed and recorded (1) the frequency of human errors, (2) human errors from which recovery was achieved (if participants conducted the recovery, then the expert recorded “Yes”; otherwise, “No”), and (3) the scores of each procedure from 0 to 5, as follows.
0=omission of procedure; 1=commission of procedure and unawareness; 2=commission of procedure and awareness but no recovery; 3=commission of procedure and awareness and recovery but recovery failed; 4=commission of procedure and awareness and recovery and recovery was correct; 5=neither omission nor commission occurred. The performance of each team or individual can be represented as N
Pi = (
∑X
ij
) / 5N
s.t . X ij = 0,1, 2, 3, 4, 5 ; j = 1, 2,..., N ; 0 ≤ Pi ≤ 1
(3)
j =1
where X ij is the score of the ith task for its
jth procedure, and Pi is the total score for
each team after performing N procedures. From the result of a paired t-test (t-value=6.82), the performance from WBTS team training (mean=0.98; SD=0.02) was better than that of workcard approach (mean=0.72; SD=0.13), and the t-value=6.82 was significant (t(13) > 4.22 , P< .001).
Web-Based Training System for Improving Aviation Maintenance Performance
737
Objective situational awareness The results of the questions related to objective situational awareness (SA) were coded as either 0 or 1, referring to an incorrect answer or a correct answer, respectively. The sum of the correct answers was defined as the individual SA score, while the team SA score was the sum of the scores of the three members, representing the total operations knowledge of the crew [5].The paired t-test result indicated that the SA scores for those using the WBTS team training were significantly better than those of the workers using the workcard training (P<.01). 4.3 Human Error Impact Risk Index (IRI) Analysis
The human error impact risk index (IRI) of Engine Driven AC Generator – Removal/Installation and Generator Constant Speed Drive – Removal/Installation was calculated using Eqs. (1) and (2). The scores of IRI in Engine Driven AC Generator and Generator Constant Speed Drive using workcard were 535 and 275 while the corresponding scores for WBTS were 118 and 126. To compare the efficiency of the two tasks between the two sets of instructions, the risk-decreasing efficiency was (1) Engine Driven AC Generator= (535-118)/535×100% =77.94%, (2) Generator Constant Speed Drive= (275-126)/275×100%=54.18%.
5 Discussion 5.1 Maintenance Behavior and Workcard Issues on the Real Case Survey
In fatal aviation accidents, the possibility of maintenance and inspection error is a critical aspect of the subsequent investigation and quantification ([12], [6]). After analyzing data from 40 events from X Airline Company over two years, we found that the highest percentage of key factors contributing to errors were repetitive and monotonous jobs, work procedure not followed, and complacency. From the questionnaire survey, some of these errors could be attributed to technician maintenance behavior and the design of the current workcard. Although both nonexperts and experts previewed the procedures before the maintenance tasks and, while noticing the warning signals on the workcard, they failed to check the procedures systematically during the maintenance tasks. One issue may stem from the technicians’ language ability, which prevented them from easily understanding the maintenance context in the AMM or workcard. Another is that technicians could not understand the risk concepts relative to human errors that affect the systems and humans, or the human error impact risk in each procedure. This study found that workcard design did not conform to existing human factors guidelines for information design. This finding is similar to that of Drury [3] and Drury et al. [4], who studied aircraft inspection. 5.2 Team Performance Assessment
Computer-based design has been used to aid the maintenance and inspection training tasks in the aircraft industry and has been compared to instructor-based training ([2], [4], [10], [1]). This study further considered the decomposition of each procedure in
738
G.-F. Liang et al.
designing the web-based training system (WBTS) for aviation maintenance. To evaluate the proposed human error risk concepts to be included in the platform, two complex tasks were used to assess team performance by both a subjective questionnaire and an objective performance measure. The subjective questionnaire of PSFs, suggested by Toriizuka [15] for industrial plant maintenance tasks, included subjective situational awareness, communication and coordination, information, software/hardware, and mental workload. The results revealed that the proposed platform, which considers human error risk concepts and their effect on each procedure, increases task satisfaction. A new expert grade and evaluation method was developed and applied to the objective performance analysis in this study. The results, using expert observation, recordings of (1) omissions or commissions, (2) awareness or unawareness, and (3) recovery accuracy from maintenance human errors, showed that teams performed better using the WBTS than they did using the traditional workcard design. Technicians with more years of work experience had more errors (e.g., errors of commission and omission) because of complacency and not following work procedures. This result was consistent with the finding of previous studies [11]. 5.3 Advantages and Limitations of the WBTS
Computer-based training to enhance teamwork skills and improve performance has advantages over workcard-based training. Kraus and Gramopadhye [10] established that the Aircraft Maintenance Team Training software obtained advantages of (1) standardization, (2) adaptability, (3) record-keeping, and (4) cost-effectiveness. The proposed WBTS preserves the workcard advantages (e.g., standards of procedure, warning signals) while also providing the same advantages that Kraus and Gramopadhye [10] obtained, as well as (1) dynamically inputting expert knowledge (e.g., native language assistance, potential error, error effect, error impact, and graphic information), (2) transmitting human error case studies in the network provided by the supervisors or managers, (3) recording turnover and training information for each technician, (4) providing pre-warning risk signals for each procedure, and (5) dispatching the tasks to each technician before performing the tasks. However, the proposed WBTS has some technical limitations. First, it is difficult to read the instructions on the monitor or portable device while working. In a future study, we will develop a larger electronic billboard to show the maintenance information and set a touch panel device near the workplace to provide technicians a way to check each procedure in real time. Second, in this study, the proposed WBTS provided for maintenance and inspection reference during the working periods but, in practice, technicians should follow the legal procedure document (e.g., workcard and aircraft maintenance manual) and sign their name when finishing the tasks. Therefore, the proposed platform must address the legality of the I-signature.
6 Conclusions Errors in aviation maintenance operations may result in serious safety problems and accidents. This study used the case of X Airline Company to demonstrate that
Web-Based Training System for Improving Aviation Maintenance Performance
739
performance shaping factors (PSFs) affect the occurrence of human error in this industry. Data analysis showed that key factors contributing to errors were repetitive and monotonous jobs, work procedures not followed, and complacency. Therefore, a web-based training system (WBTS) was proposed to mitigate these contributing factors in aviation maintenance. Human error risk knowledge in aviation maintenance tasks has always been important because it relates to accidents and the ground damage incidents. The proposed network platform allows technicians to link to a database containing the dispatched tasks on the company server. Technicians can learn more about human error risk knowledge during the training stages, as well as during the performing stage. The platform can contribute to safety as well as human reliability. In future studies, more high-complexity and critical tasks will be analyzed. The proposed platform will also be used for formal training instruction to assist the technicians in the airplane company and the students in the airplane maintenance training school. Acknowledgments. This study was supported in part by National Science Council (NSC) of Taiwan Grant (NSC96 2221 E007 083 MY3).
－ －
－ －
References 1. Andresen, G., Collier, S., Nilsen, S.: Experimental studies of potential safety-significant factors in the application of computer-based procedure systems. In: IEEE 7th Human Factors Meeting, Scottsdale Arizona (2002) 2. Chandler, T.N.: Keeping current in a changing work environment: design issues in repurposing computer-based training for on-the-job training. Int J. Ind. Ergonom. 26, 285–299 (2000) 3. Drury, C.G.: Case study: error rates and paperwork design. Appl. Ergon. 29(3), 213–216 (1998) 4. Drury, C.G., Patel, S.C., Prabhu, P.V.: Relative advantage of portable computer-based workcards for aircraft inspection. Int J. Ind. Ergonom. 26, 163–176 (2000) 5. Endsley, M.R., Robertson, M.M.: Situation awareness in aircraft maintenance teams. Int J. Ind. Ergonom. 26, 301–325 (2000) 6. Hackworth, C., Holcomb, K., Dennis, M., Goldman, S., Bates, C., Schroeder, D., Johnson, W.: An international survey of maintenance human factors programs. In: Federal Aviation Administration, Washington, DC 20591 (2007) 7. Hobbs, A., Williamson, A.: Skills, rules and knowledge in aircraft maintenance: errors in context. Ergonomics 45(4), 290–308 (2002) 8. Joint Aviation Authority (JAA) CAP 718: Human Factors in Aircraft Maintenance and Inspection (previously ICAO Digest No. 12), January 24 (2002) 9. Kraus, D., Gramopadhye, A.K.: Effect of team training on aircraft maintenance technicians: computer-based training versus instructor-based training. Int J. Ind. Ergonom. 27, 141–157 (2001) 10. Kraus, D., Gramopadhye, A.K.: Team training: role of computers in the aircaft maintenance environment. Comput. Ind. Eng. 36, 653–654 (1999) 11. Latorella, K.A., Prabhu, P.V.: A review of human error in aviation maintenance and inspection. Int J. Ind. Ergonom. 26(2), 133–161 (2000)
740
G.-F. Liang et al.
12. Reason, J.: Human Error. Cambridge University Press, Cambridge (1990) 13. Rasmussen, J.: Human errors: a taxonomy for describing human malfunction in industrial installations. Occup. Accid. 4(2-4), 311–333 (1982) 14. Reinach, S., Viale, A.: Application of a human error framework to conduct train accident/incident investigations. Accident Anal. Prev. 38, 396–406 (2006) 15. Toriizuka, T.: Application of performance shaping factor (PSF) for work improvement in industrial plant maintenance tasks. Int J. Ind. Ergonom. 28, 225–236 (2001)
Allocating Human-System Interfaces Functions by Levels of Automation in an Advanced Control Room Chiuhsiang Joe Lin1, Chih-Wei Yang2, Tzu-Chung Yenn2, and Lai-Yu Cheng1 1
Department of Industrial and Systems Engineering, Chung-Yuan Christian University, 200, Chung Pei Rd., Chung Li, Taiwan 32023, R.O.C. 2 Nuclear Instrumentation Division, Institute of Nuclear Energy Research, 1000, Wenhua Rd., Jiaan Village, Longtan Township, Taoyuan, Taiwan 32546, R.O.C. [email protected]
Abstract. Human factors engineering (HFE) focuses on the design of humansystem interfaces (HSIs). The HSIs, those NPPs parts that personnel interact with in performing their tasks, included control switches, red, green, amber, and white indictor lights, mimic displays, lighted annunciator panels, and handwritten status boards. The advanced technology has introduced the capability of integrating information from numerous plant systems and supplying needed information to operations personnel in a timely manner. Challenges of the wellintegrated computerized control room include ensuring reduced staffing does not treat with increased task complexity, achieving a consistent user interface, ensuring increased automation does not adversely affect the operator’s mental model of the plant, and systems actually support the operator. This study investigated the process of the HSI functions allocation by considering which functions should be automated and to what extent, which is also called the level of automation (LOA). Keywords: Human factors engineering, human-system interface, nuclear power plants, type of automation, level of automation.
1.1 Functional Allocation Functional allocation, which is also called allocation of function (AOF), is one of these critical HFE PRM elements. AOF is the process of allocating tasks to humans and machines in a way that adds value to the capability in terms of cost, safety and performance. Decisions of AOF provide an essential foundation for the identification of system requirements, and for design process (e.g. workstation and HSI design). Human factor inputs in the element are required to increase the understanding of human capabilities and constraints, workload limitations and human-computer interaction issues. All of these factors need to be identified and traded-off against each other to arrive at the optimal allocation of function between the humans and machines. The AOF decision-making process should first consider all possible ways that functions could be implemented. These possibilities should be documented and tabulated in a way that shows three alternative solutions: Human, Machine, and Human-Machine. Criteria should be established in terms of cost, performance, reliability, maintainability, personnel requirements, safety, user preference, limitations, workload, etc [3]. These criteria should be used to determine the optimal allocation of each function. Methods and tools include: job or flow process charts, function allocation evaluation matrix, role of the person, function allocation tool, roles of humans & automation, integrated computer-aided manufacturing definition. Successful AOFs are realized when the human and machine are given best allocation consideration. While the AOF change, such as automation, the roles of human operators in the system have also been shifted from a direct manual controller to a supervisory controller and system monitor who are largely removed from direct control. As described in variety of guidance, automation should be used to protect society from the fallibility and variability of humans. This requires a detailed task analysis that is proposed for a human, including the possible errors and the possible consequences. Further, automation should be used to reduce human cognitive overload. Humans can be ill with from information overload and consequent mental overload. This can occur from high information rates, competing tasks, or task complexity. For example, system operators working with automation have been found to have a diminished ability both to detect system errors and subsequently to perform tasks manually in the face of automation failures, compared with operators who manually perform the same tasks [4]. The above situation is named as the ‘out-of-theloop’ (OOTL) performance problems. 1.2 Level of Automation Human-centered automation, as proposed by Billings [5]; [6], is a famous approach for avoiding OOTL performance problems by optimizing the human-automation function allocation. Billings [5]; [6] considered human-centered automation a philosophy that facilitates a cooperative relationship in control and management with potential performance benefits. Some researchers proposed their concepts to realize this philosophy, such as the level of automation (LOA). The LOA can be defined as the level of task planning and performance interaction maintained between a human operator and computer in controlling a complex system
Allocating Human-System Interfaces Functions by Levels of Automation
743
[7]. These studies on the LOA [8], [9] have developed theoretical frameworks for HIP functions regarding what complex system functions should be automated, and to what extent. Building on the taxonomy of Endsley and Kaber [8], Kaber and Endsley [10] assessed the performance, situational awareness (SA), and workload effects of low, intermediate, and high LOAs using a simulated control task. Their results demonstrate that LOA is a crucial determinant of primary task performance and SA. Furthermore, they also found that low-level automation produced superior performance and intermediate LOAs facilitated higher SA. Their research was performed in a controlled laboratory setting using a simulated task to develop general results applying to numerous domains. Without a doubt, automation may contribute to reducing operator workload and fatigue, improving safety, and facilitating faster and more accurate control of multiple simultaneous tasks, it can also lead to problems in the interaction between operators and automated systems [11]. These problems include reduced operator system awareness, increased monitoring workload, and reduced manual skills. Although considerable body of literature exists on the effects of automation, there is a surprising lack of information regarding the influences of LOAs on the HSI design in the ACR and providing a framework for preventing human errors. Therefore, this study proposed a process to investigate how to allocate HSI functions by appropriate LOA in an ACR.
2 Related Issues of the HSI Function Allocation in an ACR 2.1 Regulatory Requirements and Guidelines The HFE-PRM [1] used by the staff of the US Nuclear Regulatory Commission to review the HFE programs of applicants for construction permits, operating licenses, standard design certifications, combined operating licenses, and for license amendments. The purpose of these reviews is to verify that accepted HFE practices and guidelines are incorporated into the applicant’s HFE program. In this section, this study discussed the related issues of HFE-PRM. Functional requirements analysis is the identification of those functions which must be performed to satisfy the plant’s safety objectives, i.e., to prevent or mitigate the consequences of postulated accidents that could cause undue risk to the health and safety of the public. Function allocation is the analysis of the requirement for plant control and the assignment of control functions to (1) personnel (e.g., manual control), (2) system elements (e.g., automatic control and passive, self-controlling phenomena), and (3) combinations of the two (e.g., shared control and automatic systems with manual backup). Task analysis is the identification of task requirements for accomplishing the functions allocated to plant personnel, such as (1) provide one of the bases for making decisions on design, (2) verify that human-performance requirements do not exceed human capabilities, and (3) from the basis for specifying the design requirements for the displays, data processing, and controls needed to carry out tasks. The HSI should be designed using a structured methodology that should guide designers appropriately translating functional and task requirements to the detailed design of alarms, displays, controls, and other aspects of the HSI.
744
C.J. Lin et al.
Except the HFE-PRM, the NRC staff will use the methods described in this Interim Staff Guidance (ISG) [12] to evaluate licensee compliance with USNRC requirements as presented in submittals in connection with applications for standard plant design certifications and combined licenses. This ISG provides acceptable methods for addressing the highly-integrated control room-human factors issues (HICR-HF) in the digital I&C system designs. Minimal inventory of the HSIs is an important topic and should be considered in the HSI design. The minimal inventory of HSIs (i.e., alarms, displays, controls) needed to implement the plant’s emergency operating procedures, bring the plant to a safe condition, and to carry out those operator actions shown to be risk important should be described. Due to the regulatory requirements and guidelines, to allocate HSIs functions by the LOA in an ACR is a critical way to achieve the plant’s safety objectives. While accomplishing the excellent HSIs functions allocation, the detailed design of alarms, displays, controls will meet the task requirements. Then, these task requirements can achieve the safety functions assigned to humans, as shown in Figure 1. 2.2 Allocating HSI Functions by LOA in an ACR (1) Functional Requirements Analysis and Function Allocation In this stage, those functions which must be performed to satisfy the plant’s safety objectives should be identified. Then the initial analysis of the requirement for plant control and the assignment of control functions to personnel, system elements, or combinations of the two by considering cost, performance, reliability, maintainability, personnel requirements, safety, user preference, limitations, and workload should be made. (2) Type of Automaton and Level of Automation Parasuraman et al. [9] proposed a model for types and levels of automation. The theoretical basis for classifying types of automation (TOAs) is offered by the four stage human information processing (HIP) model [13]. The model proposed by Parasuraman et al. [9] can cover the automation of different types of functions in a human machine system, including information acquisition, information analysis, decision-making and action selection, and action implementation. Endsley and Kaber [7], [8] addressed the classification of automation into four cognitive and psychomotor aspects of HIP, including monitoring display, generation of processing options, selection of an ‘optimal’ option and the implementation for this option. The taxonomy of Endsley and Kaber’s TOAs provided a wide range allocation of system functions to human, computer, and human/computer combinations. (3) Initial Task Analysis The aim of the initial stage of task analysis is to collect and organize the information (data) in a meaningful way, such that subsequent to analysis, the information is easily and efficiently used for a variety of purposes (e.g., training requirements, training content, design and design review, etc.). Specifically, the goal of this task analysis is design; therefore, information management is structured toward that end. In order to define the optimum man-machine interface based on the requirements made evident by the inherent predictive nature of this task analysis.
Allocating Human-System Interfaces Functions by Levels of Automation
745
(4) Detailed Task Descriptions In the engineering design process, a task is defined as the collection of activities performed by a person or by a machine directed toward achieving a single subfunction. The product resulting from the task analysis applied to those functions allocated to humans is basic for developing detailed task descriptions that address: information requirements; decision-making requirements; response requirements; feedback requirements; associated task support requirements; workplace factors; staffing and communications requirements; hazard identification; personnel workload.
Fig. 1. Relationships between safety objectives and the HSI design
(5) Human System Interface Design The HSI meet the technical requirement (i.e., reliability, operating experience, etc.) as required by the task analysis design requirements, operator evaluation, and applicable plant procedures (e.g., operating, abnormal, emergency, etc.). The design evaluation is based on the objectives of the systems design. What should the system do, who will use it, where will it be used and when will it be used. If the objectives are clear, the evaluation of the results will be made simpler.
746
C.J. Lin et al.
3 A Case Example of the HSI Function Allocation in an ACR 3.1 System Design Description The target system discussed here is about rod control and information. The detailed system descriptions are explained as the following. (1) Controls changes in the core reactivity, power and power shape. (2) Displays summary information to the plant operator about positions and status of the control rods. (3) Provides control rod position and status information to other systems in the plant. (4) Provides for both manual and automatic insertion of all control rods, by an alternate and diverse method. (5) Provides for both manual and automatic insertion of selected control rods for core stability control. (6) Prevents potentially unsafe rod movements by automatically enforcing rod movement blocks. (7) Provides for performing planned surveillance tests. (8) Prevents any further rod withdrawal movement in the presence of a rod withdrawal block signal. (9) Provides part of the controls and protection features to assure that the single rod drop event is an incredible event that does not need to be analyzed or tested. 3.2 Functional Requirements Analysis This functional requirements analysis has been performed to define the system functions, system processes, system process elements, system performance requirements and system support requirements for the target system. None of the functions analyzed are safety related. Three modes of operating modes include in the system: (1) Automatic rod movement. The automatic mode provides for automatic ganged rod selections and movements. (2) Semi-Automatic rod movement. The semi-automatic mode allows the operator to automatically select and move the next gang or rods (as appropriate). Rod movements can be performed manually carried out by the plant operator. (3) Manual rod movement. The manual mode provides for manual rod selection and movements under the direct command of the plant operator. The operator’s selection of any specific rod in the gang automatically results in the selection of all other associated gang members of that rod. 3.3 Allocation of Functions This AOF analysis for the target system supports the design of the HSI. Its conclusion is that the control actions allocated to the human can be properly performed by humans, considering that the machine performs the actions allocated to it, as shown in Table 1 and Table 2.
Allocating Human-System Interfaces Functions by Levels of Automation
747
(1) Initiation. To take the decision and/or to initiate the performance of the function by the machine or the human or both. Assigned to the machine. It is the function initiation through automatic or interlock signal coming either from this system or another. Assigned to the human. It is the function initiation accomplished by the operator through push-button, key switch or another similar device in the main control room. (MCR man machine interface device). Assigned to the combination of human and machine. Automatic and manual operations combined in the initiation of a function. (2) Performance: This is the accomplishment of the actions to achieve the alignment allowing the fulfillment of the function. Assigned to the machine. It involves the automatic fulfillment of the actions, on the components, in order to accomplish the function. Assigned to the human. It involves the operator manual fulfillment, in the MCR, of the necessary actions on the components, in order to accomplish the function. Assigned to the combination of human and machine. Automatic and manual operations combined for the fulfillment of a function. (3) Verification: Set of actions performed by the machine or the human in order to verify that the function is achieving its purpose or, on the contrary, if it is no longer required. This is the system response checking. Assigned to the machine. It involves the automatic verification of components and parameters for all control actions related to a function or function segment. Assigned to the human. It involves manual verification of components and parameters for all control actions related to a function or function segment. Assigned to the combination of human and machine. Automatic and manual operations combined for verification of components and parameters for all control actions related to a function or function segment. (4) Terminate: Set of actions performed by the machine or the human to finish the function performance. Assigned to the machine. It involves the automatic fulfillment of the actions on the components, for the conclusion of the function. Assigned to the human. It involves the operator manual fulfillment, in the MCR, of the necessary actions on the components, for the conclusion of the function. Assigned to the combination of human and machine. Automatic and manual operations combined for the conclusion of a function. Table 1. Part of the system functions and operating modes Function Identification Core reactivity changes control
OM01 OM02 OM03
Operating Mode Identification Automatic rod movement mode Semi-automatic rod movement Manual rod movement
748
C.J. Lin et al. Table 2. Hypothetical allocation of system overall control actions
Roles Planning Selecting Human Human Human Human Human Human Human/Computer Human Human/Computer Human Human/Computer Human/Computer Computer Human Human/Computer Computer Computer Computer Computer Computer
3.4 Level of Automation At this stage, one can ask what LOA should be applied. The 10-level taxonomy of LOA was implemented here that is intended to have applicability to a wide array of cognitive and psychomotor tasks requiring real time control [8]. As shown in Table 3, multiple levels of automation can be considered for the combination of four TOAs. 3.5 Task Analysis By performing the task analysis, the following goals are achieved: (1) Develop operational sequence diagrams for the tasks to be performed by the operators when interacting with the system, in order to achieve the control functions allocated to them and estimate operator workloads. (2) Identify critical tasks and risk-important human actions (3) Identify the general inventory and minimum inventory of alarms, displays and controls (hardware and software) necessary to perform control room tasks, paying special attention to those required to perform critical task and riskimportant human actions.
Allocating Human-System Interfaces Functions by Levels of Automation
749
(4) Identify those tasks which require, during their performance, operator communications with personnel outside the main control room (MCR). (5) Identify operator aids that could be needed by the operators when performing their job. 3.6 Human System Interface Design This human-system interface design has been performed to define the information, controls and alarms that must be contained in the MCR video display units, for controlling and monitoring the target system. In addition, all the fixed information control and alarms that the target system has available in panels and consoles have also been identified.
4 Discussion and Conclusions As existing plants undergo modernization and new plants are designed, modern control and information system technologies are being employed. However, some uncertain problems, such as the roles of human and automation, existing in instrumentation, control systems, and control rooms are continuous investigated by researchers. To solve the above problem, this study defined requirements for the HSI functions allocation in an advanced control room for nuclear power plants and investigated the process of the HSI functions allocation by considering which functions should be automated and to what extent, which is also called the level of automation (LOA). Further, a case example of the HSI function allocation in an ACR was used to describe the process. The process explained by this study can provide a direction for the HSI designer in the stages of HSI plan, analysis, and design. It is expected the process may improve operational safety of HSIs in an ACR. Due to the limitation of techniques, this study does not evaluate the performance for allocating HSI functions by LOA using a simulated experiment at the present time. For ensuring operating safety of the ACR in NPPs, it would be critical and valuable to study the effects of allocating HSI functions by LOA in the future study.
Acknowledgement This paper is financially supported by a project from the National Science Council of Taiwan under contract No. NSC- 97-2221-E-033 -033 -MY2.
References 1. U.S. Nuclear Regulatory Commission (USNRC), NUREG-0711, Rev. 2, Human factors engineering program review model, USNRC, Washington D.C (2004) 2. Brookhaven National Laboratory, http://www.bnl.gov/humanfactors/default.asp
750
C.J. Lin et al.
3. U.K. Ministry of Defense, http://www.ams.mod.uk/aofcontent/tactical/hfi/index.htm 4. Endsley, M.R., Kiris, E.O.: The out-of-the-loop performance problem and level of control in automation. Human Factors 37(2), 381–394 (1995) 5. Billings, C.E.: Human-centred aircraft automation: A concept and guidelines (NASA Tech. Memo. No. 103885), NASA-Ames Research Center, Moffet Field, CA (1991) 6. Billings, C.E.: Aviation Automation: The Search for a Human-Centered Approach. Lawrence Erlbaum Assoc., Mahwah (1997) 7. Kaber, D.B.: The Effect of Level of Automation and Adaptive Automation on Performance in Dynamic Control Environments, Tech. Work. Doc. No. ANRCPNGITWD-97-01. Amarillo National Resource Center for Plutonium, Amarillo, TX (1997) 8. Endsley, M.R., Kaber, D.B.: Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 42, 462–492 (1999) 9. Parasuraman, R., Sheridan, T.B., Wickens, C.D.: A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans 30(3), 286–297 (2000) 10. Kaber, D.B., Endsley, M.R.: The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science 5(2), 113–153 (2004) 11. Singh, I.L., Molloy, R., Parasuraman, R.: Automation-induced monitoring inefficiency: role of display location. International Journal of Human-Computer Studies 46(1), 17–30 (1997) 12. U.S. Nuclear Regulatory Commission (USNRC), Digital Instrumentation & Controls (DI&C-ISG-05) Task Working Group #5: Highly-Integrated Control Rooms—Human Factors Issues (HICR—HF) Interim Staff Guidance Rev. 1, USNRC, Washington D.C (2008) 13. Wickens, C.D., Hollands, J.G.: Engineering Psychology and Human Performance, 3rd edn. Prentice Hall, Upper Saddle River (2000)
Development of an Expert System as a User Interface for an RFID Application Deok Hee Nam 1055 N. Bickett Road Engineering and Computer Science Wilberforce University Wilberforce, OH 45384 [email protected]
Abstract. The paper presents developing an expert system as a user interface program to decode radio frequency identification codes for simulation and modeling of natural disasters. The entire developed environment for the expert system is intended to integrate all subtasks as a common user interface program to simulate and report the damages due to the catastrophic disasters. To perform the simulation as a part of the entire system, the proposed expert system reads in RFID codes in order to provide the desired information about the damages due to the catastrophic disasters based upon the available fields. Keywords: intelligent decoder, expert system, Radio Frequency Identification, user interface program.
due to the catastrophic disasters, locations, and etc. Then, the expert system as a user interface program can read the RFID codes in order to represent the information after decoding the input RFID codes.
2 Overview of Expert System [1] The expert system has been an important research field for developing the intelligent system as an application of artificial intelligence which can be used for the strong robust method in order to replace the human expert actions based upon the developed and collected knowledge base. The main concept of developing an expert system focuses on techniques to incorporate detailed domain expert knowledge into suitable software systems. In other words, an expert system is a reasoning and computing system which simulates or interprets the human expert actions within some restricted knowledge base engine. Comparing to other artificial intelligence programs, the expert system can perform more capabilities as an intelligent program such as dealing with subject matter of realistic various complexity that normally requires a considerable amount of human expertise, exhibiting high performance for being a more efficient tool, explaining and justifying the required solutions and suitable suggestions to convince the users with the developed knowledge base engine. As examples, there are a lot of applications for the expert system around the environment. In earlier days, the applications of the expert system are like the relatively simple game playing or puzzle solving to provide the theoretical knowledge bases. As the computers are developed, people are interesting to develop the intelligent machines which can understand the human actions or natural languages such as human dialogs or stories. It causes that an intelligent expert system is able to replace the roles of real human tasks. In modern days, the expert system is characterized by more selfconsciousness and self-criticism with involving the psychological aspects simultaneously.
3 Expert System as Intelligent Decoder 3.1 Scope of Proposed System The scope of the proposed system is designed to read Radio Frequency Identification (RFID) codes intelligently to provide the information about states, counties, disaster types, impact factors, disaster events types, disaster consequence (damage) categories, affected human damages, building damages, and economic loss due to the damages for simulation and modeling natural activities or events after attacked by the catastrophic disasters. 3.2 Logic Flow of Proposed System The proposed system is mainly consisting of three parts, which are the interface with the RFID reader, the implementation of RFID codes based upon the knowledge base
Development of an Expert System as a User Interface for an RFID Application
753
Expert System: Decoding RFID codes Implementation of RFID Codes
Radio Frequency Identification Application RFID codes
Knowledge Base System
Implemented RFID codes
Displaying Information Of RFID Codes
Fig. 1. System Overview for intelligent RFID decoder
engine, and displaying the results about the implementation of RFID codes. First, the interface part receives the codes from the resources like RFID readers or sensors. Using the input codes, the implementation part analyzes the RFID codes based upon the information in each field based upon the collected knowledge base engine. Finally, the results from the implementation part display the resolved information.
Logic Flow for Implementation of Decoder Start
Identify Disaster types
Identify Disaster Consequence Category
Read RFID code
Identify Impact factors
Identify damage of Human Affected
Identify RFID ID field
Identify natural disaster events
Identify damage of Buildings
Identify state field
Identify man-made disaster events
Identify county field
Identify hybrid disaster events
Identify Economic Loss due to Damages
Fig. 2. Logic Flow Diagram of Decoder
End
754
D.H. Nam
4 Information of RFID Codes 4.1 Fields of RFID Coding System As Table 1 is shown, 12 fields of RFID codes are described. Table 1. Fields of RFID Coding System [2, 3, 4, 5] NO.
Names
No. of Digits for Coding 8
1
2
RFID Code Identification NO. States
3 4
Counties Disaster Types
3 1
5
Impact Factors
1
2
Corresponding Codes
Codes will be varied based upon the roles or locations of installed sensors in the specific area. 01 – ALABAMA 02 – ALASKA 03 – AMERICAN SAMOA 04 – ARIZONA 05 – ARKANSAS 06 – CALIFORNIA 07 – COLORADO 08 – CONNECTICUT 09 – DELAWARE 10 – DISTRICT OF COLUMBIA 11 – FEDERATED STATES OF MICRONESIA 12 – FLORIDA 13 – GEORGIA 14 – GUAM 15 – HAWAII 16 – IDAHO 17 – ILLINOIS 18 – INDIANA 19 – IOWA 20 – KANSAS 21 – KENTUCKY 22 – LOUISIANA 23 – MAINE 24 – MARSHALL ISLANDS 25 – MARYLAND 26 – MASSACHUSETTS 27 – MICHIGAN 28 – MINNESOTA 29 – MISSISSIPPI 30 – MISSOURI 31 – MONTANA 32 – NEBRASKA 33 – NEVADA 34 – NEW HAMPSHIRE 35 – NEW JERSEY 36 – NEW MEXICO 37 – NEW YORK 38 – NORTH CAROLINA 39 – NORTH DAKOTA 40 – NORTHERN MARIANA ISLANDS 41 – OHIO 42 – OKLAHOMA 43 – OREGON 44 – PALAU 45– PENNSYLVANIA 46– PUERTO RICO 47 – RHODE ISLAND 48 – SOUTH CAROLINA 49 – SOUTH DAKOTA 50 – TENNESSEE 51 – TEXAS 52 – UTAH 53 – VERMONT 54 – VIRGIN ISLANDS 55 – VIRGINIA 56 – WASHINGTON 57 – WEST VIRGINIA 58 – WISCONSIN 59 – WYOMING Codes will be varied based upon states 1 – Natural Disasters 2 – Man-Made Disasters 3 – Hybrid disaster 1 – air 2 – road 3 – rail 4 – flood 5 – nuclear 6 – chemical 7 – plane
Development of an Expert System as a User Interface for an RFID Application
6
Natural Disaster Events
1
7
Man-made Disaster Events
2
8
Hybrid Disaster Events
2
9
Disaster Consequence (damage) Category
1
10
Damage of Human Affected
1
11
Damage Buildings
of
1
12
Economic Loss due to Damages
1
8 – wind 0 – not applied 1 – Tropical cyclones 2 – Hurricane 3 – Typhoon 4 – Earthquake 5 – Tornado 00 – not applied 01 – Dam Failure 02 – Accidental release 03 – Structural explosions 04 – Chemical explosions 05 – Nuclear explosion 06 – Thermonuclear explosion 07 – Mine explosions pollution 08 – Acid rain 09 – Chemical pollution 10 – Atmospheric pollution 11 – Bridge Collapse 12 – Black-Out 13 – Power Breakage 14 – Highway Collapse 00 – not applied 01 – Dam Failure 02 – Accidental release 03 – Structural explosions 04 – Chemical explosions 05 – Nuclear explosion 06 – Thermonuclear explosion 07 – Mine explosions pollution 08 – Acid rain 09 – Chemical pollution 10 – Atmospheric pollution 11 – Bridge Collapse 12 – Black-Out 13 – Power Breakage 14 – Highway Collapse 1 – Minor (Category 1) 2 – Significant (Category 2) 3 – Severe (Category 3) 4 – Major (Category 4) 5 – Catastrophic (Category 5) 1 – 10 or less people affected 2 – 10 or more and less than 100 people affected 3 – More than 100 people or less than 1000 people affected 4 – More than 1000 people affected 1 – 10 or less buildings are damaged 2 – More than 10 buildings and 100 or fewer buildings are damaged. 3 – More than 100 buildings are damaged 1 – Less than $10,000 2 – Between $10,000 and $ 100,000 3 – Between $100,000 and $ 1,000,000 4 – Between $1,000,000 and $ 5,000,000 5 – Between $5,000,000and $ 10,000,000 6 – Between $10,000,000 or more
755
756
D.H. Nam
Field 1: RFID ID This field is about the sensor ID to communicate with other sensors intelligently through the wireless network system. The required number of digits for the coding is 8 bits. Field 2: States This field contains 59 states including all US states, District of Columbia, Federated States of Micronesia, Guam, Northern Mariana Islands, Palau, Puerto Rico, and Virgin Islands. It requires two digits for the RFID codes. Field 3: Counties All counties are varied based upon the states. It requires three digits for the RFID codes. Field 4: Disaster types There are three different disaster types, natural disasters, man-made disasters, and hybrid disasters. It requires one digit for the RFID codes. Field 5: Impact factors There are eight different types of the disaster factors. It requires one digit for the codes. Field 6, 7, and 8: Disaster events types Fields 6, 7, and 8 are disaster events based upon the disaster types from Field 4. Field 6 is about natural disaster events. Field 7 is about man-made disaster events. Field 8 is about hybrid disaster events. Each field requires one digit for the RFID codes. Field 9: Disaster consequence (damage) categories There are five different damage categories. It requires one digit for the RFID codes. Field 10: Affected human damages There are four different categories depending upon the number of damaged buildings due to the disasters. It requires one digit for the RFID codes. Field 11: Building damages There are three different categories depending upon the number of affected people due to the disasters. It requires one digit for the RFID codes. Field 12: Economic loss due to the damages There are six different categories depending upon the number of economic damage assessments due to the disasters. It requires one digit for the RFID codes.
5 Proposed Expert System 5.1 Structure of RFID Codes In the structure of Radio Frequency Identification (RFID) Codes [6, 7, 8], there are 12 fields including RFID id, states, counties, disaster types, impact factors, disaster events types, disaster consequence (damage) categories, affected human damages, building damages, and economic loss due to the damages. The intelligent decoder of RFID codes has been implemented through an expert system as a user interface. Based upon RFID reader’s input codes, the intelligent decoder can convert the codes into the appropriate information to simulate the
Development of an Expert System as a User Interface for an RFID Application
757
Structure of RFID codes 0
0
1
0
1
1
RFID ID
1
1
0
0 0
0
0
State Counties
0
1
0
0
0
Natural Disaster Events
0
0
0
0
Hybrid Disaster Events
1
1
0
0
1
0
0
1
0
2 5 Economic Loss due to Damages Damage of Buildings
Man-made Disaster Events
Damage of Human Affected Disaster Consequence
Impact Factors
(damage) Category
Disaster Types
Fig. 3. Structure of RFID codes
damage of disasters using the expert system. The design of the code system was based on 24 digit system. Each field has the unique number of digits to implement the required information. All 12 fields are displayed after analyzed based on the knowledge base system for each field. 5.2 Expert System to Decode Radio Identification Codes The expert system receives the RFID codes from the communicated sensors up to 24 digits. Then, each code can be implemented based upon each field’s knowledge base engine. Finally, the recognized information based upon all 12 fields is displayed as a result. Fig. 4, shows the initial interface to start the expert system. Fig. 5 captures the codes and Fig. 6 displays the translated information based upon the RFID code: 000000012200218200005325.
Fig. 4. Initial Interface Window for Intelligent Decoder
758
D.H. Nam
Fig. 5. RFID codes, 000000012200218200005325, are entered
Fig. 6. Information of RFID codes entered by RFID reader
6 Conclusion The expert system for the intelligent decoder of RFID codes has been implemented. Based upon RFID input codes, the intelligent decoder can convert the codes into the appropriate information to simulate the damage of disasters using the expert system. The design of the code system was based on 24 digit system. Each field has the
Development of an Expert System as a User Interface for an RFID Application
759
unique number of digits to implement the required information. All 12 fields are displayed after analyzed based on the knowledge base system for each field. Acknowledgments. This material is based upon work supported by the National Nuclear Security Administration of the Department of Energy under Award number, DE-FG52-06NA27579.
References 1. Jackson, P.: Introduction to Expert Systems. Addison-Wesley Publishers, Wokingham (1986) 2. Shaluf, I.M., Ahmadun, F., Mustapha, S.: Technological disaster’s criteria and models. Disaster Prevention and Management 12(4), 305–311 (2003) 3. Shaluf, I.M., Ahmadun, F., Mustapha, S.: A review of disaster and crisis. Disaster Prevention and Management 12(1), 24–32 (2003) 4. Shaluf, I.M.: An overview on the technological disasters. Disaster Prevention and Management 16(3), 380–390 (2007) 5. Combs, D., Quenemoen, L., Parrish, R., Davis, J.: Assessing disaster-attributed mortality: development and application of a definition and classification matrix. International Journal of Epidemiology 28, 1124–1129 (1999) 6. Chang, T.C.: Lecture notes: Group Technology. Purdue University 7. Sing, N.: Design of cellular manufacturing systems: An invited review. European Journal of Operation Research 69, 284–291 (1993) 8. Girdhar, A.: Expansion of group technology part coding based on functionality, Master’s thesis, Univ. of Cincinnati (2001)
Developing a Validation Methodology for Educational Driving Simulators and a Case Study Hatice Sancar1, Kursat Cagiltay2, Veysi Isler3, Gizem Tamer4, Neslihan Ozmen5, and Utkan Eryilmaz6 1
Mersin University, Computer Education and Instructional Technology Department, 06531 Mersin, Turkey 2 Middle East Technical University, Computer Education and Instructional Technology Department, 06531 Ankara, Turkey {hsancar,kursat}@metu.edu.tr 3 Middle East Technical University, Computer Engineering Department, 06531 Ankara, Turkey [email protected] 4 Middle East Technical University, Software Management Department, 06531 Ankara, Turkey [email protected] 5 Middle East Technical University, Institute of Applied Mathematics, 06531 Ankara, Turkey [email protected] 6 Middle East Technical University, Department of Information Systems, 06531 Ankara, Turkey [email protected]
Abstract. The aim of this study is to develop a methodology for validating driving simulators in terms of simulator usability. The methodology was used to validate a truck simulator designed for training truck drivers about economic fuel consumption. The participants were eight truck drivers. Interview and observation methods were used to gather data. The results of the study showed that drivers did not have difficulty to recognize the parts of the driving simulator. Also, they stated that driving the simulator was easy. However, they said that they had some difficulties to use some systems of the simulator. Keywords: Driving Simulators, Validation, Usability.
Developing a Validation Methodology for Educational Driving Simulators
761
Control” [4]. Also, the driving simulators provide safe and economical testing driving performance [5]. The driving simulator research studies include studies of simulator validation [6] [5] [7], effects of simulators to learning [8] [9] [10], usability of simulators [11] [2] simulator sickness [12], evaluating drivers’ performances by driving license bureaus [1] and driver workload [13]. However, there is limited study that investigated the validation of driving simulators although the validation of a simulator is important for designing good simulators. Also, simulator validation should be taken into account because "The success of a man-in-the -loop real time simulator depends upon how well it meets the design goals which is determined via a process called Validation.” (p. 1) [14]. There is confusion on types and methods used for the simulator validation [6] [15]. For example, validation types in Psychology as “construct, content and criterion validity” are used for simulator validation [6]. Physical Correspondence and Behavioral correspondence are the two aspects of simulator validity which are generally focused in the literature [4]. On the other hand, validity can be divided as internal and external validity [16]. Some literature review studies which focus on different validity concepts also show the confusion on simulator validation concepts, types and methods. Simulator validation concepts are summarized as “Accuracy, Algorithmic validity, Believability, Conceptual (face) validity, Construct validity, Content validity, Criterion (predictive) validity, Educational validity, Empirical validity, Empiricism (objectivism, foundationalism, verificationalism), Event validity, External validity, Hermeneuticism, Internal validity, Operational validity, Plausibility, Positive Economics, Rationalism, Realism, Relativism (Conventionalism), Representational validity, Validation, Verification and Verisimilitude.” [16] In the scope of this study, the reviewed literature on the validity concept and classification used for simulators shows that there is not a precise validity classification for the simulators which include usability issues. For that reason, the aim of this study is to develop a validation model which includes simulator usability can be used for driving simulators. To do this, the validation concepts and classifications as well as driving simulator validation and usability studies were reviewed. So, the important parameters to validate a driving simulator were defined. Moreover, the developed validation model was used to validate a truck simulator designed for economic fuel consumption and safe driving trainings in terms of usability issues. How the developed model can be used was shown with a research based approach. So, the important parameters that were taken into account while choosing the appropriate methods and how the simulator validation study can be conducted was shown in a practical way. 1.1 The Proposed Simulator Usability Validation Model In the literature, there is an agreement that there are not precise simulator validation definition and simulator validation models which propose usability types and methods for each type. Also, there are many simulator usability studies in the reviewed literature, usability are not mentioned in the validation studies [11] [2]. However, it should be taken into account because it may affect the behavior of the users as well as success of the simulation. The driving simulators’ properties should be considered in the usability validation studies because they are different than other computer based
762
H. Sancar et al.
applications in that they represent a real life model and provide a real life experiences although they are designed on computer and have a virtual environment. For that reason, the representation of real life experiences, modeling, people’s behaviors while using the simulators, people’s perceptions on them, simulator design models used and purposes of simulators designed are the main concepts and issues related to the simulator validation. Some of them are connected with the simulators’ own physical characteristic parallel to simulator design purpose while some of them connected with user behaviors and perceptions. Therefore, the data collection methods used for simulator usability are defined as interview and observation. So, the information about the users’ perception on if they have difficulty in recognizing and/or using any parts of the simulator can be obtained.
Design Purpose Level of Driving Simulators Level of Driving Simulators Designed
Fig. 1. Simulator Usability Model
Moreover, driving simulators have different parts as steering gear, gas pedal, gearshift, break, speed indicator, gasoline indicator, turn signal indicator lamp and so on. The researchers should focus on important parts parallel to driving simulator aim and their usability issues. Also, they should prepare the instruments for each part. Since the parts of the driving simulators are important, our model has two components as “Use of the Simulator Parts” and “Recognition of the Simulator Parts”. The factors that may affect the results of validation research should be considered while conducting a driving simulator usability model. For that reason, the defined factors are placed in our model. One of the factors was purpose of designing the simulator. The purposes of designing driving simulators are defined as Research, Training, and Screening [1]. Moreover, it is stated that simulator validity should be determined by taking into account of design purpose [17]. The other factor is that level of the simulator in a simulator classification structure wanted to be validated. Although there is no implication about the importance of defining validation by considering level of the simulator in the reviewed literature, level of the simulators are important
Developing a Validation Methodology for Educational Driving Simulators
763
in the simulator validation model proposed in this study because identification of the level of a driving simulator is important in order to know which skill, knowledge and attitude task elements can be best fit to it. Also, it shows the capabilities of the simulators. The level of the simulators is connected with the quality of them [18]. For that reason, while defining the validation methods, one should consider the capabilities of the simulator. The other factor that may affect the results of the validation research is users’ demographic characteristics such as age, gender, being novice or expert and so on [6]. For that reason, the proposed simulator validation model has also this factor as a component.
2 Methodology of the Study The study aimed to develop a methodology and validate a truck simulator related to usability issues. The qualitative approach was applied to gather data. Eight volunteer drivers were participated to the study. While four of them are truck drivers, four of them are semi-trailer drivers in real life. All of them stated that they had not used a simulator before. All drivers navigated through the same scenarios and they used the simulator in two areas in the simulation environment: 1) at hill 2) in city. The following scenario was developed for driving simulator sessions: 1. Selecting the starting point: a. from beginning of the hill to end of the hill (3,5 km); b. from the beginning of the city center to the end of the city center (2,5 km) 2. Selecting vehicle load: no load 3. Selecting traffic density: medium vehicle density 4. Selecting the weather: sunny The methodology provides a step by step system for the researchers who aim to investigate validation of a simulator. The methodology has 5 steps:
Define the need for validation Define the scope of validation
Define the methods for validation
Define the aim of using the simulator
Fig. 2. The steps of the validation methodology
Implement the method and Report results
764
H. Sancar et al.
Firstly, before stating the validation investigation, the aim of using the truck simulator was taken into account: the truck simulator would be used for educational purposes and accordance with this purpose which parts usability should be investigated was defined. Secondly, the scope of the validation is categorized as “User perception on simulator”, “reality of the simulator”, “usability of the simulator”, “simulator sickness” and in this study, the usability of the truck simulator was chosen as study scope. Next, the methods of simulator use validity which is defined process assessing how much the driving simulators are user friendly regarding to reality and user perception were applied. According to the chosen method, the instruments were prepared. Finally, the data were gathered and results were reported. 2.1 Materials Used in the Study Interview Form. Interview form consisted of 10 questions on usability of the simulator. The questions were prepared by benefiting from the literature on simulator validation studies and revised by an expert. Observation Form. Observation form was prepared to enable researchers to focus on important actions of the drivers while they were using the simulator. Two researchers observed the drivers while they were using the simulator according to observation form. They focused mainly on usability issues. The Truck Simulator. The truck simulator is developed to train drivers on economic fuel consumption. The parts such as steering gear, gas pedal, gearshift, break, speed indicator, gasoline indicator, turn signal indicator lamp and so on in the simulator are the same as those in real trucks. The simulator visual system consists of 3-LCD screen, it has 120-degree horizontal field of view and 2 degree of motion platform (Figure 3.)
Fig. 3. The Truck Simulator
Developing a Validation Methodology for Educational Driving Simulators
765
3 Results The interview questions were prepared to investigate if the drivers have difficulty in recognizing and using the parts of the truck simulator. The drivers were asked that whether they had difficulty while using the truck simulator and if they did what the reasons were. The observation data consisted of two researchers’ observation notes according to the observation form. Also, there were three cameras used to validate the observation notes taken by researchers. One camera was placed to record the simulator screen. Another one was placed to record the drivers’ feet actions. The last one was placed to record the drivers’ facial expressions. 3.1 Recognition of the Simulator Parts To investigate how much the truck simulator is a user friendly technology, the drivers were asked if they have difficulty in recognizing the steering gear, gas pedal, gearshift, break, signals, lights, speed indicator and rpm indicator of the truck simulator. All of the drivers stated that they could easily recognize parts of the simulator. They stated that these parts were the same as the ones in real trucks. One driver stated that “The parts of the simulator are exactly the same ones of the real truck. Anyone can easily recognize them.” The observation data also supported the result of the interview results. According to the observation results, the drivers could easily find the parts of the simulator that they needed to use. Only, they could not understand the type of the gearshift. For that reason, the trainer had to explain the type of the gearshift and how it works. Moreover, the drivers were asked how driving the truck simulator is similar to driving on real truck. All the drivers stated that driving the truck simulator is similar to driving a real truck in about 80 percent. They also pointed out that the differences were stemmed from the computer environment of the simulator and they were not getting used to that environment. They stated that the similarities were stemmed from their using steering wheel, gas pedal, gearshift, break and signals while driving the truck simulator as a real truck. One driver stated: “Driving the simulator is similar to driving a real truck because I used steering wheel, gas pedal, gearshift, break and signals to driving it but I does not use computer. Maybe for that reason, I cannot say that the driving experiences in the simulator was the same as driving a real truck” Five drivers stated that the view field of the truck simulator was perfect and two drivers pointed out that it was much better than real trucks. All the drivers stated that people, buildings, vehicles, roads and traffic signs in the simulation environment were the same with ones in the real environment. The observation results also support the interview results in that the drivers obeyed the traffic rules, used signals while passing other vehicles and waited people while crossing the street.
766
H. Sancar et al.
3.2 Use of the Simulator Parts The drivers were asked if they had any difficulty while using the gear, gas pedal, gearshift, break, signals, lights, speed indicator and rpm indicator of the simulator. So, the parts of the truck simulator that the drivers had difficulty while using and what type of difficulty they had met were defined. According to the interview results, the drivers had difficulty in using steering wheel, gas pedal, gearshift and break. Also, according to the observation results, all of the drivers had difficulty while using steering wheel and seven drivers had difficulty while using gearshift. Five drivers stated that the steering wheel was so sensitive to their actions and they had difficulty in keeping the truck simulator on the roadway. According to the observation results, the drivers had to rotate the steering wheel at the left and right direction too much. For that reason, they lost the simulator control and went on the opposite side of the road. One driver stated: “Driving the simulator is similar to driving a real truck. However, the steering wheel of the simulator is so sensitive. I lost the control of the simulator. I think it happens because it is a computer environment” The gearshift of the simulator was also difficult to use according to four drivers. While three drives stated that they could not change to second gear, one driver stated that he could not change to fifth gear. Also, none of the drivers could understand the type of the gearshift. However, all the drivers who pointed out that they had difficulty while using gearshift stated that it was stemmed from their not getting used the computer environment. Moreover, three drivers stated that although they could easily keep the real truck rpm at an appropriate level, they could not do this while using the simulator. The drivers pointed out that this stemmed from the lack of adjustment in the gas pedal. They said that when they stepped on the gas, the simulator rpm suddenly jumped. Another problem that the drivers stated was in the brake system. Three drivers pointed out that the brake was giving late response to their actions. For that reason, they had difficulty in controlling the truck simulator. One driver stated: “There is a break system, however, it does not give a feeling that the vehicle is stopping. You do not understand that you are slowing down without looking at the speed indicator” Also, the drivers stated that they did not meet any problem due to the people, other vehicles, traffic signs and buildings in the simulation environment. According to the observation notes, only one driver had an accident because a pedestrian hit the truck while the driver waited at the traffic lights.
4 Conclusion There are many sectors such as automotive industry, military, academic research fields, government, space, recreational computer markets and medical sector which
Developing a Validation Methodology for Educational Driving Simulators
767
use simulators today [1]. They use the simulators, especially, for training purposes [2] because simulators decrease the cost of training as well as provide a safe environment for the users [4] [3]. But, the success of the training with simulator depends on how well the simulator meets the design goals and it is determined through validation studies [14]. Moreover, simulators are different than the real life tests or other computer applications and for that reason, validation models and types used for real life tests or computer applications should not be used for validating simulators [6]. Therefore, this study aimed to propose a validation model in terms of usability of the driving simulators and validate a truck simulator according to the proposed model. The simulator usability model consists of two parts as “Use of the Simulator Parts” and “Recognition of the Simulator Parts”. The factors that may affect the results of the validation process are defined. These factors are design purpose of the driving simulators [17], level of the simulators and users’ demographics. According to the Simulator Usability Validation model defining the parts of the driving simulator and taking the user perception on its usability are important. A truck simulator was validated by using the Simulator Usability Validation model. It is a high level simulator. The researchers followed a step by step system to conduct the study as “Define the need for validation”, “Define the aim of using the simulator”, “Define the scope of validation”, “Define the methods for validation” and “Implement the method and Report the results”. Firstly, before conducting the study, the design aim of the truck simulator had been taken into account: the truck simulator was designed to train the truck drivers on economic and safe driving. Secondly, reality and usability were chosen as the scope of the validation study. Thirdly, face validity which is defined as what is the target population perception on the resemblance degree between the truck simulator and the real truck. The instruments which were prepared by taking into account the chosen method were a questionnaire, an interview form and an observation form. Finally, the data were gathered and results were reported. Eight volunteer truck drivers participated to the study and all drivers drove the truck simulator twice. The same scenario was used through their first driving and second driving. Firstly, they drove it by their own. Secondly, a trainer helped them while driving the truck simulator. Two researchers observed the truck drivers through the drivers’ first and second driving in addition to three cameras. After driving, the drivers were interviewed. According to the interview results, the drivers found driving the truck simulator was similar to driving a real truck since they used the steering gear, gas pedal, gearshift, break, signals, lights, speed indicator and rpm indicator through driving. Moreover, they stated that the steering gear, gas pedal, gearshift, break, signals, lights, speed indicator and rpm indicator of the simulator were the same ones in real trucks. However, they pointed out that they had difficulty in using steering wheel, gas pedal, gearshift and break parts. According to five drivers the steering wheel was so sensitive and for that reason, they could not control the truck simulator. Moreover, three drivers pointed out that the rpm indicator showed high levels due to the gas pedal. Three drivers stated that there was a problem in break system because it was giving late response to their actions. Moreover, four drivers found difficult to use gearshift of the simulator. However, according to them it was stemmed from their not getting used the computer environment.
768
H. Sancar et al.
In the future work, we plan to implement this model to other driving simulators. Also, we plan to replicate this study with a new and large samples.
Acknowledgments This work was supported by British Petroleum Company under the Philosopher project. Also, we thank Ercan Sevim who is a trainer of HED Academy.
References 1. Straus, S.: New, Improved, Comprehensive and Automated Driver’s License Test and Vision Screening System. Department of Transportation and Federal Highway Administration, FHWA-AZ-04-559(1). Presented to Arizona Department of Transportation Motor Vehicle Division (2005) 2. Uke, H.: The Usability Study of the Trafikent Driver Training Simulator. In: Proceedings of TRODSA, Traffic and Road Safety Third International Congress/Exhibition, vol. 2, pp. 545–553 (2006) 3. Deyo, R., Briggs, A., Doenges, P.: Getting Graphics in Gear: Graphics and Dynamics in Driving Simulation. SIG GRAPH Proceedings, Computer Graphics, 317–326 (1988) 4. Reed, M.P., Green, P.: Validation of a Low-Cost Driving Simulator Using a Telephone Dialling Task. UMTRI, 95-19 (1995) 5. Lee, H.C.: The Validity of Driving Simulator to Measure on-Road Driving Performance of Older Drivers. Journal of Transport Engineering in Australia 8(2), 89–100 (2003) 6. Blana, E.: Driving Simulator Validation Studies: A Literature Review. Institute of Transport Studies, University of Leeds, Working Paper 480 (1996) 7. Allen, R.W., Mitchell, D.G., Stein, A.C., Hogue, J.R.: Validation of real-time man-in-theloop simulation. VTI Raport. No 372A, Part 4, 18-31 (1991) 8. West, C., Snellen, J.: A Report on the Research and Development of Instructional Simulation. ERIC Document Reproduction Service, No. ED 340362 (1991) 9. Wallace, P.R., Regan, M.A.: Case study: Converting Human Factors Research into Design Specifications for Instructional Simulation. In: Proceedings of the Third International SimTect Conference. Adelaide, Australia, 117–121 (1998) 10. De Winter, J.C.F., Wieringa, P.A., Dankelman, J., Mulder, M., Van Paassen, M.M.: Driving simulator fidelity and training effectiveness. In: Proceedings of the 26th European Annual Conference on Human Decision Making and Manual Control, Lyngby, Denmark (2007) 11. Kuhl, J., Evans, D., Papelis, Y., Romano, R., Watson, G.: The Iowa Driving Simulator: An Immersive Research Environment. Computer 28(7), 35–41 (1995) 12. Durdu, O.P., Cagiltay, K.: Investigation of Simulator Sickness in a Driving Simulator. In: Proceedings of TRODSA, Traffic and Road Safety Third International Congress/Exhibition, vol. 2, pp. 554–563 (2006) 13. Green, P., Lin, B., Bagian, T.: Driver Workload as a Function of Road Geometry: A Pilot Experiment. Technical Report UMTRI-93-39. University of Michigan Transportation Research Institute, Ann Arbor, Michigan (1993) 14. Galloway, R.T.: Model Validation Topics for Real Time Simulator Design Courses. In: Summer Computer Simulation Conference, Orlando FL, United States (2001)
Developing a Validation Methodology for Educational Driving Simulators
769
15. Eskandarian, A., Delaigue, P., Sayed, R., Mortazavi, A.: Development and Verification of a Truck Driving Simulator for Driver Drowsiness Studies, http://www.cisr.gwu.edu/truck_sim.pdf 16. Feinstein, A.H., Cannon, H.M.: Fidelity, Verifiability, and Validity of Simulation: Constructs for Evaluation. Developments in Business Simulation and Experiential Learning 28, 57–67 (2001) 17. Sargent, R.G.: Verification and Validation of Simulation Models. In: Proceedings of the 30th Conference on Winter Simulation, Washington, D.C., United States, pp. 121–130 (1998) 18. Longridge, T., Ray, P., Boothe, E., Bürki-Cohen, J.: Initiative towards more affordable flight simulators for U.S. commuter airline training. In: Proceedings of the Royal Aeronautical Society Conference on Training - Lowering the Cost, Maintaining the Fidelity. London, UK (1996)
Developing a Usable Mobile Flight Case Learning System in Air Traffic Control Miscommunications Kuo-Wei Su, Keh-Yeu Lee, Po-Hsin Huang, and I-Tsun Chen Dept. of Information Management, National Kaohsiung First University of Science and Technology, 811 Jhuoyue Rd., Nanzih, Kaohsiung City, Taiwan, R.O.C. [email protected]
Abstract. Aviation’s highest priority is safety. The primary risk to safety derives not from automated systems but from human factors, most notably pilots and air traffic control. We present results from development of a flight case learning system designed and aviation regulation retrieve system to operate on mobile phones used by pilots and air traffic controllers. Our system takes advantage of key ontology concepts, human-centered design strategies, and appropriate small-screen interface design protocols. A questionnaire to assess user interaction satisfaction (QUIS) was deployed for subsequent usability testing and to verify acceptance of, and satisfaction with the system. Twelve students participated in the questionnaire-based evaluation of subjective satisfaction. In addition, two flight experts served on a review panel for domain knowledge verification and acceptance of the interface design. Our results confirm that MFCLS is a suitably designed mobile learning system that can accelerate selflearning for both pilots and controllers. Keywords: ATC communications, aviation regulation, ontology, HCI (Human Computer Interaction).
1 Introduction Safety” is the minimum requirement for a human life, and is also the most important key for aviation enterprises. To prevent pilot and ATC communication errors, this research would start to carry them out from “crew errors.” And the procedures and regulations error is always in the aviation incident report. The regulation error is an important factor in the human error that caused the aviation accident. In view of the serious problem of aviation safety, the study aims at developing a mobile flight case learning system (MFCLS) imitating cases of ATC verbal communication errors and aviation regulation retrieve system. It could help pilots and ATCS further their experience and situation alertness at anytime and anywhere.
Developing a Usable MFCLS in Air Traffic Control Miscommunications
771
influence the communication process between a controller and a pilot. These factors can be analyzed using the well-known SHELL model [3],[4]. This SHELL model provides a conceptual framework to help to understand human factors. 2.2 Aviation Regulation The Table 1 is this study collection and coordination the aviation regulations of our country and the other countries and their management units. Table 1. The major countries in this world with their aviation regulations and the management units. (Classified by this study). Country
Major Aviation Regulation
Management Unit
Republic of China
Civil Aviation Regulation
Civil Aeronautics Administration (CAA)
China
Civil Aviation Regulation of the People's Republic of China
Civil Aviation Administrator of China (CAAC)
United States of America (USA) Canada
Federal Aviation Regulations (FAR) Canadian Aviation Regulations (CAR)
Britain Australia
Civil Aviation Article (CAA) Australia Civil Aviation Act
Japan
Aviation Constitution
Federal Aviation Administration (FAA) 1. Canadian Aviation Regulation Advisory Council (CARAC) 2. Air Navigation Services and Airspace Civil Aviation Authority Civil Aviation Safety Authority (CASA) Japanese Civil Aviation Bureau
2.3 Ontology In the last decade, the word ‘‘ontology’’ has become a fashionable word inside the Knowledge Engineering Community [5]. An ontology contains a set of concepts and relationship between concepts, and can be applied into information retrieval to deal with user queries [6]. And ontology web language (OWL), are the foundation of semantic-web efforts to use ontologies for web services [7],[8],[9]. This study will use the conceptual of the ontology with semantic web to develop the system. One of these most important technologies is the ontologies which brought a new spirit to the knowledge management systems. Ontology is a method of conceptualization on a specific domain [10]. The ontologies have emerged as one of the most popular modeling approaches for taxonomies, classifications, and other structures used in intelligent systems [5]. By a philosopher, ontologies could describe facts or
772
K.-W. Su et al.
entities in the real world. That is to say, ontologies could express types and relations between things according to the circumstances of the real world [11]. Semantic Web was initially proposed by Tim Berners-Lee as an evolution of the current World Wide Web (WWW), in which information and services can be understandable and usable both by human beings and by computers [12]. In that sense, the Semantic Web goal is not to make computers understand the human language, but to define an universal model for the information expression and a set of inference rules that machines can easily use in order to process and to relate the information as if they really understand it [13]. 2.4 Mobile Human-Computer Interaction Design The current mobile environment has already achieved portability for many characteristics; The impact of screen size on reading and comprehension was investigated mostly during the 1980s and early 1990s (before the advent of the Web). To present information effectively on a small-screen interface, designers must minimize the inherent limitations in that interface. Researchers have suggested that users did not want to use the conventional page-to-page navigation as it was interactively very costly on the small screen [14]. 2.5 Usability Engineering The study of HCI for mobile devices was a relatively young research field in which commercially successful devices had only been available for less than a decade and leading conferences had only a few years of history [15]. Usability engineering became the banner under which diverse methodological endeavors were carried out in the 1980s (Nielsen, 1993). A major portion of usability engineering and thus usability testing was the HCI, “the study of how people interact with computer technology and how to make this interaction effective” [16].
3 Research Methodology 3.1 Construction of ATCM Knowledge Base This study attempted to form a knowledge structure using the above taxonomies. Figure 1 represents Figure 2 is the relation schema of knowledge content of ATCM, and there are four kinds of relationships, which are “By Reason Of”, “Made”, “Resulted In”, and “Happened At”. This schema is our knowledge structure which illustrates courses of event of ATCM occurrences. And then one of the cases would be used to interpret its meaning as an instance (see Figure 3). This study used Protégé which is an ontology editor to establish the domain ontology and the case base, generally called the knowledge base of ATCM.
Developing a Usable MFCLS in Air Traffic Control Miscommunications
Fig. 1. The knowledge composition of ATCM
Fig. 2. The relation schema content of ATCM
Fig. 3. The semantic content of instance case
773
774
K.-W. Su et al.
3.2 Retrieve System Architecture Figure is this study ontology-based system architecture. The system used the ontology tools (protégé 2000) to build the aviation regulation knowledgebase.
Fig. 4. Aviation regulation Retrieve System Architecture
3.3 Retrieve System Process Figure 5 is about this study retrieval system retrieval process architecture. The architecture included three parts. The left is the well-designed user interface. The middle is the aviation regulation ontology. The right part is the aviation regulation database.
Regulation Retrieval System Interface
Fig. 5. System Retrieval Process Architecture
3.4 Experimental Design The methodology of the experimental design adopts two approaches to evaluate usability and utility of our systems. One is the objective evaluation to measure effectiveness and the other is a subjective evaluation that can help to realize user
Developing a Usable MFCLS in Air Traffic Control Miscommunications
775
preference and behavior in more detail. The participants were twelve undergraduates, one pilot and one controller in the experiment. The experimental procedure has three phases (1) The participants first is given a short practice in 3 minutes on system with a PDA. (2) Each participant was requested to perform the seven tasks following the experimental scenario in the flight learning system. (3) Then the participants were asked to fill out a subjective user interaction satisfaction questionnaire.
：
3.5 Development of System The development of system was based on experience and knowledge of experts to build a knowledge base of ATCM by Protégé, to create a web application by java server page (JSP) and to redesign the web interfaces for small screen using the interface design guidelines based on M-HCI. The main function of system is the search by data entry with pull-down menus. The Figure 6 is showed the example of our system, the case search function in terms of selection lists.
Fig. 6. The example of our system
4 Results 4.1 Experimental Results The Table 2 showed the means of the constructs results. The results were above 4 points expect the construct of symbol investigation whose mean was 3.95 points and odds with 4 points was just 0.05 points. This meant the result represented high satisfaction for our systems. 4.2 Discussion Based on the results of the subjective assessment, the questionnaires show that the all constructs of the screen display layout, symbol investigation, system information and feedback, learning, system reliability, and overall user reactions gained good approval from the students. And the pilots also gave positive responses regarding the interface
776
K.-W. Su et al. Table 2. The validity results
design and system contents. So, this study ascertained that small-screen interface design based on the guidelines have its definite advantages.
5 Conclusion This study developed a mobile flight case-based learning system and aviation regulation retrieve system. In the internal side, the experts’ knowledge and experiences were reserved in the knowledge base. In the external side, the knowledge content would be represented by the form of small screen. In other words, pilots and controllers can acquire and learn the knowledge by themselves at any time and anywhere through an intellectual mobile device, such as a PDA. Therefore, there expressed such a mobile flight learning system based on human-computer interaction (HCI) is worthy to be weeded through the old to bring forth the new in the future. Further, we’ll implement the experimental design for confirming the system usability also. This work is in progress.
Acknowledgments The authors express sincere thanks to the National Science Council of TAIWAN, ROC for the financial support under the grant number NSC97-2221-E-327 -020 MY2. And Dr. Li-Ji Ho who is a pilot and Mr. Wei Hsu who is the senior specialist gave us many assistance, thanks to them.
References 1. Hopkin, V.D.: Human factor in air traffic control. Taylor & Francis, Bristol (1995) 2. Rantanen, E.M., Kokayeff, N.K.: Pilot Error In Copying Air Traffic Control Clearances. In: The Proceedings of the 46th Annual Meeting of the Human Factors and Ergonomics Society, Santa Monica, Human Factors and Ergonomics Society (2002)
Developing a Usable MFCLS in Air Traffic Control Miscommunications
777
3. ICAO.: Human Factors Training Manual. International Civil Aviation Organization, ICAO Doc 9683-AN/950 (1998) 4. Hawkins, F.H.: Human factors in fligh. Ashgate, Aldershot (1987) 5. Eriksson, H.: The semantic-document approach to combining documents and ontologies. Int. J. Human-Computer Studies 65, 624–639 (2007) 6. Han, L., Berry, D.: Semantic-supported and agent-based decentralized grid resource discovery. Future Generation Computer Systems 24, 806–812 (2008) 7. World Wide Web Consortium, OWL web Ontology Language Reference. W3C recommendation (2004a), http://www.w3.org/TR/owl-ref/i 8. World Wide Web Consortium, RDF Vocabulary Description Language 1.0: RDF Schema. W3C recommendation (2004b), http://ww.w3.org/TR/rdf-schema/i 9. World Wide Web Consortium, Resource Description Framework (RDF): Concepts and Abstract Syntax. W3C recommendation (2004c), http://www.w3.org/TR/rdf-concepts/i 10. Noy, N.F., Hafner, C.D.: The state of the art in ontology design. AI Magazine 18(3), 53–74 (1997) 11. Sugumaran, V., Storey, V.C.: Ontologies for conceptual modeling: their creation, use, and management. Data & Knowledge Engineering 42, 251–271 (2002) 12. Pereira, R.G., Freire, M.M.: SWedt: A Semantic Web Editor Integrating Ontologies and Semantic Annotations with Resource Description Framework. AICT/ICIW. 0-7695-25229/06 (2006) 13. Berners-Lee, T.: What the Semantic Web can represent (1998), http://www.w3.org/DesignIssues/RDFnot.html 14. Jones, M., Marsden, G., Mohd-Nasir, N., Boone, K., Buchanan, G.: Improving Web interaction on small displays. Computer Networks 31, 1129–1137 (1999) 15. Jesper, K., Connor, Ġ.: A Review of Mobile HCI Research Methods. In: Chittaro, L. (ed.) Mobile HCI 2003. LNCS, vol. 2795, pp. 317–335. Springer, Heidelberg (2003) 16. Battleson, B., Booth, A., Weintrop, J.: Usability testing of an academic library web site: A case study. The Journal of Academic Librarianship 27, 188–198 (2001)
Author Index
AlFedaghi, Hadlaa 3 Al-Nanih, Reem 429 Al-Nuaim, Hana 429 Al-Osaimi, Asma 3 Alsumait, Asmaa 3 Andreou, Lefkothea 266 Ardito, Carmelo 439 Bachmann, Karen 458 Bailey, John H. 504 Baldiris, Silvia 12 Banasiak, Meredith 614 Begole, Bo 448 Bergstom, Ilias 266 Bernaten´e, Silvia 101 Bolchini, Davide 653 Boudjenane, Yassin 577 Breiner, Kai 663 Brereton, Pat 567 Brown, David 111 Buono, Paolo 439 Buxton, Tim 673 Buzzi, Maria Claudia 21 Buzzi, Marina 21 Cagiltay, Kursat 94, 760 Carri¸co, Lu´ıs 186, 217 Casas, Ignacio 31 Chalfoun, Pierre 39 Chan, Susy 304 Chao, Chih-Yi 353 Charissis, Vassilis 683 Chen, Chaomei 693 Chen, Elvis Chih-Hsien 325 Chen, I-Tsun 770 Chen, Sherry 156 Chen, Yan 49 Chen, Yoke Yie 488 Cheng, Lai-Yu 741 Chisnell, Dana 458 Choi, Myungil 387 Chu, Maurice 448 Chung, Donghun 279 Cipolla Ficarra, Francisco V. 58, 68, 78, 101, 468
Cipolla-Ficarra, Miguel 68, 78 Costabile, Maria F. 439 Dai, Guozhong 256 Damon, Jeffrey 111 de Antonio, Angelica 315 Defazio, Joseph 335 de Greef, Tjerk 703 Deicke, Benedikt 88, 196 de S´ a, Marco 217 De Wispelaere, Jean-Fran¸cois Di Loreto, Ines 287 Do, Trien V. 296 Dogusoy, Berrin 94 Duarte, Carlos 186 Duke, Jon 478
524
Eryilmaz, Utkan 760 Essabbah, Mouna 713 Evans, Andrew 49 Fabregat, Ram´ on 12 Faiola, Anthony 335, 478 Fang, Xiaowen 304 Fardoun, Habib 236 Fern´ andez Niello, Jorge 101 Fern´ andez-Ziegler, Rodolfo 101 Ferre, Xavier 315 Finkestein, Anthony 653 Frasson, Claude 39 Fruhling, Ann 673 Gale, Alastair 49 Gallud, Jos´e 236 Ghinea, Gheorghita 156 Go, Kentaro 176 Goh, Kim Nee 488 G´ omez, Sergio 12 Gong, Yang 587 G¨ orlich, Daniel 663 Gouze, Annabelle 524 Greco, Mario 101 Gusmini, Massimiliano 597 Haber, Eben M. 504 Halkia, Matina 597
780
Author Index
Harris, Don 723 He, Zheng 246 H´erisson, Joan 713 Higuchi, Yuki 176 Hsu, Chun-Cheng 325 Huang, Fei-Hui 551 Huang, Po-Hsin 770 Hwang, Ha Sung 378, 387 Hwang, Sheue-Ling 551, 731 Imbert, Ricardo 315 Inoue, Tomoo 416 Isler, Veysi 760 Iwabuchi, Eriko 495 James, Jonathan 49 Jelin, Martin 111 Kandogan, Eser 504 Kang, Min 119 Kashiwagi, Harumi 119 Kayser, Fran¸coise 524 Kenny, Patrick G. 514 Kharrazi, Hadi 335, 478 Kieffer, Suzanne 524 Kim, Chae-Hwan 279 Kim, Kyujung 368 Kiriyama, Takashi 345 Kobayashi, Atsutomo 534 Koehne, Benjamin 632 Konno, Fumiko 176 Kosunen, Ilkka 406 Kr¨ oll, Martin 543 Kuikkaniemi, Kai 406 Lane, H. Chad 129 Lanzilotti, Rosa 439 Laskowski, Sharon 458 Lee, Hyowon 567 Lee, Jiunde 353 Lee, Jong-Weon 296 Lee, Keh-Yeu 770 Lee, Ying-Lien 551 Leporini, Barbara 21 Li, Jiun-Fa 731 Li, Lon-Wen 723 Li, Wen-Chin 723 Liang, Guo-Feng 731 Lilley, Mariana 140 Lin, Chiuhsiang Joe 741
Lin, Jhih-Tsong 731 Liu, Juan 448 Loob, Alexander 558 Lotto, Beau 266 Lowry, Svetlana 458 Lozano, Mar´ıa 236 Macq, Benoˆıt 524 Maekawa, Yasuko 149 Maglio, Paul P. 504 Majima, Yukie 149 Makita, Yuki 204 Mallem, Malik 713 Mampadi, Freddy 156 Maschino, Oliver 663 Matsumoto, Takashi 448 Mazza, Riccardo 166 Mazzola, Luca 166 McCadden, Douglas 111 Medinilla, Nelson 315 Meixner, Gerrit 663 Mej´ıa, Carolina 12 Messick, Christopher 614 Milde, Jan-Torsten 88, 196 Mitsuishi, Takashi 176 Miyadera, Youzou 266 Mohamad Ali, Nazlena 567 Moncarey, Ronald 524 Mori, Hirohiko 624 Mori, Taketoshi 359 Mustapha, Emy Elyanee 488 Nair, Chitra 304 Nakagawa, Maki 495 Nakamura, Yumiko 149 Nam, Deok Hee 751 Neerincx, Mark A. 703 Noguchi, Hiroshi 359 Nordin, Sharina 488 O’Hear, Timothy 577 Ochoa, Sergio F. 31 Ohtsuki, Kazuhiro 119 Okada, Ken-ichi 416 Oomes, A.H.J. 703 Ormandjieva, Olga 429 Otmane, Samir 713 Ozmen, Neslihan 760 Paolini, Paolo 653 Papanastasiou, Stylianos
683
Author Index Park, Myunjin 368 Park, SungBok 378, 387 Parsons, Thomas D. 514 Patterson, Patrick 731 Penichet, Victor 236 Pereira, Joana 186 Peters, Kirsten A. 396 Phillips, Win 587 Pohl, Hans-Martin 88, 196 Puente, Jaime 31 Pyper, Andrew 140
