Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbruecken, Germany
6772
Gavriel Salvendy Michael J. Smith (Eds.)
Human Interface and the Management of Information Interacting with Information Symposium on Human Interface 2011 Held as Part of HCI International 2011 Orlando, FL, USA, July 9-14, 2011 Proceedings, Part II
13
Volume Editors Gavriel Salvendy Purdue University School of Industrial Engineering West Lafayette, IN, USA and Tsinghua University Department of Industrial Engineering Beijing, P.R. China E-mail:
[email protected] Michael J. Smith University of Wisconsin-Madison Department of Industrial and Systems Engineering Center for Quality and Productivity Improvement Madison, WI, USA E-mail:
[email protected]
ISSN 0302-9743 e-ISSN 1611-3349 ISBN 978-3-642-21668-8 e-ISBN 978-3-642-21669-5 DOI 10.1007/978-3-642-21669-5 Springer Heidelberg Dordrecht London New York Library of Congress Control Number: 2009928850 CR Subject Classification (1998): H.5, K.6, H.3-4, C.2, H.4.2, J.1 LNCS Sublibrary: SL 3 – Information Systems and Application, incl. Internet/Web and HCI
© Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Foreword
The 14th International Conference on Human–Computer Interaction, HCI International 2011, was held in Orlando, Florida, USA, July 9–14, 2011, jointly with the Symposium on Human Interface (Japan) 2011, the 9th International Conference on Engineering Psychology and Cognitive Ergonomics, the 6th International Conference on Universal Access in Human–Computer Interaction, the 4th International Conference on Virtual and Mixed Reality, the 4th International Conference on Internationalization, Design and Global Development, the 4th International Conference on Online Communities and Social Computing, the 6th International Conference on Augmented Cognition, the Third International Conference on Digital Human Modeling, the Second International Conference on Human-Centered Design, and the First International Conference on Design, User Experience, and Usability. A total of 4,039 individuals from academia, research institutes, industry and governmental agencies from 67 countries submitted contributions, and 1,318 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human–computer interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. This volume, edited by Gavriel Salvendy and Michael J. Smith, contains papers in the thematic area of human interface and the management of information (HIMI), addressing the following major topics: • • • • •
Access to information Supporting communication Supporting work, collaboration, decision-making and business Mobile and ubiquitous information Information in aviation
The remaining volumes of the HCI International 2011 Proceedings are: • Volume 1, LNCS 6761, Human–Computer Interaction—Design and Development Approaches (Part I), edited by Julie A. Jacko • Volume 2, LNCS 6762, Human–Computer Interaction—Interaction Techniques and Environments (Part II), edited by Julie A. Jacko • Volume 3, LNCS 6763, Human–Computer Interaction—Towards Mobile and Intelligent Interaction Environments (Part III), edited by Julie A. Jacko • Volume 4, LNCS 6764, Human–Computer Interaction—Users and Applications (Part IV), edited by Julie A. Jacko • Volume 5, LNCS 6765, Universal Access in Human–Computer Interaction— Design for All and eInclusion (Part I), edited by Constantine Stephanidis
VI
Foreword
• Volume 6, LNCS 6766, Universal Access in Human–Computer Interaction— Users Diversity (Part II), edited by Constantine Stephanidis • Volume 7, LNCS 6767, Universal Access in Human–Computer Interaction— Context Diversity (Part III), edited by Constantine Stephanidis • Volume 8, LNCS 6768, Universal Access in Human–Computer Interaction— Applications and Services (Part IV), edited by Constantine Stephanidis • Volume 9, LNCS 6769, Design, User Experience, and Usability—Theory, Methods, Tools and Practice (Part I), edited by Aaron Marcus • Volume 10, LNCS 6770, Design, User Experience, and Usability— Understanding the User Experience (Part II), edited by Aaron Marcus • Volume 11, LNCS 6771, Human Interface and the Management of Information—Design and Interaction (Part I), edited by Michael J. Smith and Gavriel Salvendy • Volume 13, LNCS 6773, Virtual and Mixed Reality—New Trends (Part I), edited by Randall Shumaker • Volume 14, LNCS 6774, Virtual and Mixed Reality—Systems and Applications (Part II), edited by Randall Shumaker • Volume 15, LNCS 6775, Internationalization, Design and Global Development, edited by P.L. Patrick Rau • Volume 16, LNCS 6776, Human-Centered Design, edited by Masaaki Kurosu • Volume 17, LNCS 6777, Digital Human Modeling, edited by Vincent G. Duffy • Volume 18, LNCS 6778, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris • Volume 19, LNCS 6779, Ergonomics and Health Aspects of Work with Computers, edited by Michelle M. Robertson • Volume 20, LNAI 6780, Foundations of Augmented Cognition: Directing the Future of Adaptive Systems, edited by Dylan D. Schmorrow and Cali M. Fidopiastis • Volume 21, LNAI 6781, Engineering Psychology and Cognitive Ergonomics, edited by Don Harris • Volume 22, CCIS 173, HCI International 2011 Posters Proceedings (Part I), edited by Constantine Stephanidis • Volume 23, CCIS 174, HCI International 2011 Posters Proceedings (Part II), edited by Constantine Stephanidis I would like to thank the Program Chairs and the members of the Program Boards of all Thematic Areas, listed herein, for their contribution to the highest scientific quality and the overall success of the HCI International 2011 Conference. In addition to the members of the Program Boards, I also wish to thank the following volunteer external reviewers: Roman Vilimek from Germany, Ramalingam Ponnusamy from India, Si Jung “Jun” Kim from the USA, and Ilia Adami, Iosif Klironomos, Vassilis Kouroumalis, George Margetis, and Stavroula Ntoa from Greece.
Foreword
VII
This conference would not have been possible without the continuous support and advice of the Conference Scientific Advisor, Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications and Exhibition Chair and Editor of HCI International News, Abbas Moallem. I would also like to thank for their contribution toward the organization of the HCI International 2011 Conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, Maria Bouhli and George Kapnas. July 2011
Constantine Stephanidis
Organization
Ergonomics and Health Aspects of Work with Computers Program Chair: Michelle M. Robertson Arne Aar˚ as, Norway Pascale Carayon, USA Jason Devereux, UK Wolfgang Friesdorf, Germany Martin Helander, Singapore Ed Israelski, USA Ben-Tzion Karsh, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Nancy Larson, USA Kari Lindstr¨om, Finland
Brenda Lobb, New Zealand Holger Luczak, Germany William S. Marras, USA Aura C. Matias, Philippines Matthias R¨ otting, Germany Michelle L. Rogers, USA Dominique L. Scapin, France Lawrence M. Schleifer, USA Michael J. Smith, USA Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK
Human Interface and the Management of Information Program Chair: Michael J. Smith Hans-J¨ org Bullinger, Germany Alan Chan, Hong Kong Shin’ichi Fukuzumi, Japan Jon R. Gunderson, USA Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Hirohiko Mori, Japan Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA
Youngho Rhee, Korea Anxo Cereijo Roib´ as, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P. R. China
X
Organization
Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven A. Landry, USA
Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China
Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA Pietro Carlo Cacciabue, Italy John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Wen-Chin Li, Taiwan James T. Luxhøj, USA Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands
Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa Matthew J.W. Thomas, Australia Mark Young, UK Rolf Zon, The Netherlands
Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth Andr´e, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian B¨ uhler, Germany Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy
Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA Patrick M. Langdon, UK Seongil Lee, Korea
Organization
Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria
Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Panayiotis Zaphiris, Cyprus
Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA Simon Julier, UK David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA Gordon McK Mair, UK
David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Jose San Martin, Spain Dieter Schmalstieg, Austria Dylan Schmorrow, USA Kay Stanney, USA Janet Weisenford, USA Mark Wiederhold, USA
Internationalization, Design and Global Development Program Chair: P.L. Patrick Rau Michael L. Best, USA Alan Chan, Hong Kong Lin-Lin Chen, Taiwan Andy M. Dearden, UK Susan M. Dray, USA Henry Been-Lirn Duh, Singapore Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA Sung H. Han, Korea Veikko Ikonen, Finland Toshikazu Kato, Japan Esin Kiris, USA Apala Lahiri Chavan, India
James R. Lewis, USA James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA Katsuhiko Ogawa, Japan Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin R¨ ose, Germany Supriya Singh, Australia Alvin W. Yeo, Malaysia Hsiu-Ping Yueh, Taiwan
XI
XII
Organization
Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Peter Day, UK Fiorella De Cindio, Italy Heidi Feng, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Andrew Laghos, Cyprus Stefanie Lindstaedt, Austria Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan
Anthony F. Norcio, USA Ulrike Pfeil, UK Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Laura Slaughter, Norway Sergei Stafeev, Russia Asimina Vasalou, UK June Wei, USA Haibin Zhu, Canada
Augmented Cognition Program Chairs: Dylan D. Schmorrow, Cali M. Fidopiastis Monique Beaudoin, USA Chris Berka, USA Joseph Cohn, USA Martha E. Crosby, USA Julie Drexler, USA Ivy Estabrooke, USA Chris Forsythe, USA Wai Tat Fu, USA Marc Grootjen, The Netherlands Jefferson Grubb, USA Santosh Mathan, USA
Rob Matthews, Australia Dennis McBride, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Banu Onaral, USA Kay Stanney, USA Roy Stripling, USA Rob Taylor, UK Karl van Orden, USA
Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Giuseppe Andreoni, Italy Thomas J. Armstrong, USA Norman I. Badler, USA Fethi Calisir, Turkey Daniel Carruth, USA Keith Case, UK Julie Charland, Canada
Yaobin Chen, USA Kathryn Cormican, Ireland Daniel A. DeLaurentis, USA Yingzi Du, USA Okan Ersoy, USA Enda Fallon, Ireland Yan Fu, P.R. China Afzal Godil, USA
Organization
Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Bo Hoege, Germany Hongwei Hsiao, USA Tianzi Jiang, P.R. China Nan Kong, USA Steven A. Landry, USA Kang Li, USA Zhizhong Li, P.R. China Tim Marler, USA
XIII
Ahmet F. Ozok, Turkey Srinivas Peeta, USA Sudhakar Rajulu, USA Matthias R¨ otting, Germany Matthew Reed, USA Johan Stahre, Sweden Mao-Jiun Wang, Taiwan Xuguang Wang, France Jingzhou (James) Yang, USA Gulcin Yucel, Turkey Tingshao Zhu, P.R. China
Human-Centered Design Program Chair: Masaaki Kurosu Julio Abascal, Spain Simone Barbosa, Brazil Tomas Berns, Sweden Nigel Bevan, UK Torkil Clemmensen, Denmark Susan M. Dray, USA Vanessa Evers, The Netherlands Xiaolan Fu, P.R. China Yasuhiro Horibe, Japan Jason Huang, P.R. China Minna Isomursu, Finland Timo Jokela, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan Seongil Lee, Korea Kee Yong Lim, Singapore
Zhengjie Liu, P.R. China Lo¨ıc Mart´ınez-Normand, Spain Monique Noirhomme-Fraiture, Belgium Philippe Palanque, France Annelise Mark Pejtersen, Denmark Kerstin R¨ ose, Germany Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Janet Wesson, South Africa Toshiki Yamaoka, Japan Kazuhiko Yamazaki, Japan Silvia Zimmermann, Switzerland
Design, User Experience, and Usability Program Chair: Aaron Marcus Ronald Baecker, Canada Barbara Ballard, USA Konrad Baumann, Austria Arne Berger, Germany Randolph Bias, USA Jamie Blustein, Canada
Ana Boa-Ventura, USA Lorenzo Cantoni, Switzerland Sameer Chavan, Korea Wei Ding, USA Maximilian Eibl, Germany Zelda Harrison, USA
XIV
Organization
R¨ udiger Heimg¨artner, Germany Brigitte Herrmann, Germany Sabine Kabel-Eckes, USA Kaleem Khan, Canada Jonathan Kies, USA Jon Kolko, USA Helga Letowt-Vorbek, South Africa James Lin, USA Frazer McKimm, Ireland Michael Renner, Switzerland
Christine Ronnewinkel, Germany Elizabeth Rosenzweig, USA Paul Sherman, USA Ben Shneiderman, USA Christian Sturm, Germany Brian Sullivan, USA Jaakko Villa, Finland Michele Visciola, Italy Susan Weinschenk, USA
HCI International 2013
The 15th International Conference on Human–Computer Interaction, HCI International 2013, will be held jointly with the affiliated conferences in the summer of 2013. It will cover a broad spectrum of themes related to human–computer interaction (HCI), including theoretical issues, methods, tools, processes and case studies in HCI design, as well as novel interaction techniques, interfaces and applications. The proceedings will be published by Springer. More information about the topics, as well as the venue and dates of the conference, will be announced through the HCI International Conference series website: http://www.hci-international.org/ General Chair Professor Constantine Stephanidis University of Crete and ICS-FORTH Heraklion, Crete, Greece Email:
[email protected]
Table of Contents – Part II
Part I: Access to Information Developing Optimum Interface Design for On-screen Chinese Proofreading Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alan H.S. Chan, Joey C.Y. So, and Steve N.H. Tsang “Life Portal”: An Information Access Scheme Based on Life Logs . . . . . . Shin-ichiro Eitoku, Manabu Motegi, Rika Mochizuki, Takashi Yagi, Shin-yo Muto, and Masanobu Abe
3 11
Proposal of the Kawaii Search System Based on the First Sight of Impression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyoko Hashiguchi and Katsuhiko Ogawa
21
Development of a Tracking Sound Game for Exercise Support of Visually Impaired . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoshikazu Ikegami, Keita Ito, Hironaga Ishii, and Michiko Ohkura
31
From Personal to Collaborative Information Management: A Design Science’s Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mei Lu and Philip Corriveau
36
A Classification Scheme for Characterizing Visual Mining . . . . . . . . . . . . . Elaheh Mozaffari and Sudhir Mudur
46
Transforming a Standard Lecture into a Hybrid Learning Scenario . . . . . Hans-Martin Pohl, Jan-Torsten Milde, and Jan Lingelbach
55
Designing Web Sites and Interfaces to Optimize Successful User Interactions: Symposium Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Robert W. Proctor and Kim-Phuong L. Vu
62
Petimo: Sharing Experiences through Physically Extended Social Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nimesha Ranasinghe, Owen Noel Newton Fernando, and Adrian David Cheok Comparison Analysis for Text Data by Using FACT-Graph . . . . . . . . . . . . Ryosuke Saga, Seiko Takamizawa, Kodai Kitami, Hiroshi Tsuji, and Kazunori Matsumoto A Comparison between Single and Dual Monitor Productivity and the Effects of Window Management Styles on Performance . . . . . . . . . . . . . . . Alex Stegman, Chen Ling, and Randa Shehab
66
75
84
XVIII
Table of Contents – Part II
Interface Evaluation of Web-Based e-Picture Books in Taiwan . . . . . . . . . Pei-shiuan Tsai and Man-lai You
94
A Digital Archive System for Preserving Audio and Visual Space . . . . . . Makoto Uesaka, Yusuke Ikegaya, and Tomohito Yamamoto
103
Experience Explorer: Context-Based Browsing of Personal Media . . . . . . Tuomas Vaittinen, Tuula K¨ arkk¨ ainen, and Kimmo Roimela
111
Part II: Supporting Communication Service Science Method to Create Pictograms Referring to Sign Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Naotsune Hosono, Hiromitsu Inoue, Hiroyuki Miki, Michio Suzuki, Yuji Nagashima, Yutaka Tomita, and Sakae Yamamoto
123
MoPaCo: Pseudo 3D Video Communication System . . . . . . . . . . . . . . . . . . Ryo Ishii, Shiro Ozawa, Takafumi Mukouchi, and Norihiko Matsuura
131
Analysis on Relationship between Smiley and Emotional Word Included in Chat Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junko Itou, Tomoyasu Ogaki, and Jun Munemori
141
Designing Peripheral Communication Services for Families Living-Apart: Elderly Persons and Family . . . . . . . . . . . . . . . . . . . . . . . . . . . Yosuke Kinoe and Mihoko Noda
147
Visual Feedback to Reduce Influence of Delay on Video Chatting . . . . . . Kazuyoshi Murata, Masatsugu Hattori, and Yu Shibuya
157
Research on the Relationships between Visual Entertainment Factor and Chat Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tomoyasu Ogaki, Junko Itou, and Jun Munemori
165
Multimodal Conversation Scene Analysis for Understanding People’s Communicative Behaviors in Face-to-Face Meetings . . . . . . . . . . . . . . . . . . Kazuhiro Otsuka
171
A Virtual Audience System for Enhancing Embodied Interaction Based on Conversational Activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoshihiro Sejima, Yutaka Ishii, and Tomio Watanabe
180
VizKid: A Behavior Capture and Visualization System of Adult-Child Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Grace Shin, Taeil Choi, Agata Rozga, and Mario Romero
190
Interactive e-Hon as Parent-Child Communication Tool . . . . . . . . . . . . . . . Kaoru Sumi and Mizue Nagata
199
Table of Contents – Part II
SAM: A Spatial Interactive Platform for Studying Family Communication Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Guo-Jhen Yu, Teng-Wen Chang, and Ying-Chong Wang
XIX
207
Part III: Supporting Work, Collaboration, Decision-Making and Business The Effects Visual Feedback on Social Behavior during Decision Making Meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Merel Brandon, Simon Epskamp, Thomas de Groot, Tim Franssen, Bart van Gennep, and Thomas Visser Co-Creation of Value through Social Network Marketing: A Field Experiment Using a Facebook Campaign to Increase Conversion Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Asle Fagerstrøm and Gheorghita Ghinea Towards Argument Representational Tools for Hybrid Argumentation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mar´ıa Paula Gonz´ alez, Sebastian Gottifredi, Alejandro J. Garc´ıa, and Guillermo R. Simari
219
229
236
Development of a Price Promotion Model for Online Store Selection . . . . Shintaro Hotta, Syohei Ishizu, and Yoshimitsu Nagai
246
Design Effective Voluntary Medical Incident Reporting Systems: A Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lei Hua and Yang Gong
253
Technology-Based Decision-Making Support System . . . . . . . . . . . . . . . . . . Hanmin Jung, Mikyoung Lee, Pyung Kim, and Won-Kyung Sung
262
Economic Analysis of SON-Enabled Mobile WiMAX . . . . . . . . . . . . . . . . . Seungjin Kwack, Jahwan Koo, and Jinwook Chung
268
ICT-Enabled Business Process Re-engineering: International Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ya-Ching Lee, Pin-Yu Chu, and Hsien-Lee Tseng
278
A Methodology to Develop a Clinical Ontology for Healthcare Business . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mario Macedo and Pedro Isa´ıas
285
Advances in E-commerce User Interface Design . . . . . . . . . . . . . . . . . . . . . . Lawrence J. Najjar
292
Information Technology Services Indutry and Job Design . . . . . . . . . . . . . Yoshihiko Saitoh
301
XX
Table of Contents – Part II
Dodging Window Interference to Freely Share Any Off-the-Shelf Application among Multiple Users in Co-located Collaboration . . . . . . . . Shinichiro Sakamoto, Makoto Nakashima, and Tetsuro Ito
305
Process in Establishing Communication in Collaborative Creation . . . . . . Mamiko Sakata and Keita Miyamoto
315
Real-World User-Centered Design: The Michigan Workforce Background Check System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sarah J. Swierenga, Fuad Abujarad, Toni A. Dennis, and Lori A. Post What Kinds of Human Negotiation Skill Can Be Acquired by Changing Negotiation Order of Bargaining Agents? . . . . . . . . . . . . . . . . . . . . . . . . . . . Keiki Takadama, Atsushi Otaki, Keiji Sato, Hiroyasu Matsushima, Masayuki Otani, Yoshihiro Ichikawa, Kiyohiko Hattori, and Hiroyoki Sato An Efficient and Scalable Meeting Minutes Generation and Presentation Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Berk Taner, Can Yildizli, Ahmet Ozcan Nergiz, and Selim Balcisoy
325
335
345
Part IV: Mobile and Ubiquitous Information Object and Scene Recognition Using Color Descriptors and Adaptive Color KLT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Volkan H. Bagci, Mariofanna Milanova, Roumen Kountchev, Roumiana Kountcheva, and Vladimir Todorov What Maps and What Displays for Remote Situation Awareness and ROV Localization? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ryad Chellali and Khelifa Baizid Evaluation of Disaster Information Management System Using Tabletop User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hidemi Fukada, Kazue Kobayashi, Aki Katsuki, and Naotake Hirasawa Relationality-Oriented Systems Design for Emergence, Growth, and Operation of Relationality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takuya Kajio, Manami Watanabe, Ivan Tanev, and Katsunori Shimohara
355
364
373
381
Real-Time and Interactive Rendering for Translucent Materials Such as Human Skin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hiroyuki Kubo, Yoshinori Dobashi, and Shigeo Morishima
388
Local Communication Media Based on Concept of Media Biotope . . . . . . Hidetsugu Suto and Makiba Sakamoto
396
Table of Contents – Part II
XXI
Big Fat Wand: A Laser Projection System for Information Sharing in a Workspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Toru Takahashi and Takao Terano
403
Disaster Information Collecting/Providing Service for Local Residents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuichi Takahashi, Daiji Kobayashi, and Sakae Yamamoto
411
Comfortable Design of Task-Related Information Displayed Using Optical See-Through Head-Mounted Display . . . . . . . . . . . . . . . . . . . . . . . . Kazuhiro Tanuma, Tomohiro Sato, Makoto Nomura, and Miwa Nakanishi
419
Usability Issues in Introducing Capacitive Interaction into Mobile Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shuang Xu and Keith Bradburn
430
Balance Ball Interface for Performing Arts . . . . . . . . . . . . . . . . . . . . . . . . . . Tomoyuki Yamaguchi, Tsukasa Kobayashi, and Shuji Hashimoto
440
Study on Accessibility of Urgent Message Transmission Service in a Disaster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shunichi Yonemura and Kazuo Kamata
446
Part V: Information in Aviation Is ACARS and FANS-1A Just Another Data Link to the Controller? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Vernol Battiste, Joel Lachter, Sarah V. Ligda, Jimmy H. Nguyen, L. Paige Bacon, Robert W. Koteskey, and Walter W. Johnson Flight Deck Workload and Acceptability of Verbal and Digital Communication Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summer L. Brandt, Joel Lachter, Arik-Quang V. Dao, Vernol Battiste, and Walter W. Johnson Conflict Resolution Automation and Pilot Situation Awareness . . . . . . . . Arik-Quang V. Dao, Summer L. Brandt, L. Paige Bacon, Joshua M. Kraut, Jimmy Nguyen, Katsumi Minakata, Hamzah Raza, and Walter W. Johnson Effect of ATC Training with NextGen Tools and Online Situation Awareness and Workload Probes on Operator Performance . . . . . . . . . . . . Ariana Kiken, R. Conrad Rorie, L. Paige Bacon, Sabrina Billinghurst, Joshua M. Kraut, Thomas Z. Strybel, Kim-Phuong L. Vu, and Vernol Battiste
453
463
473
483
XXII
Table of Contents – Part II
Effects of Data Communications Failure on Air Traffic Controller Sector Management Effectiveness, Situation Awareness, and Workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joshua M. Kraut, Ariana Kiken, Sabrina Billinghurst, Corey A. Morgan, Thomas Z. Strybel, Dan Chiappe, and Kim-Phuong L. Vu Pilot Information Presentation on the Flight Deck: An Application of Synthetic Speech and Visual Digital Displays . . . . . . . . . . . . . . . . . . . . . . . . Nickolas D. Macchiarella, Jason P. Kring, Michael S. Coman, Tom Haritos, and Zoubair Entezari How Data Comm Methods and Multi-dimensional Traffic Displays Influence Pilot Workload under Trajectory Based Operations . . . . . . . . . . Jimmy H. Nguyen, L. Paige Bacon, R. Conrad Rorie, Meghann Herron, Kim-Phuong L. Vu, Thomas Z. Strybel, and Vernol Battiste Macroergonomics in Air Traffic Control – The Approach of a New System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Luiza Helena Boueri Rebello A Preliminary Investigation of Training Order for Introducing NextGen Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . R. Conrad Rorie, Ariana Kiken, Corey Morgan, Sabrina Billinghurst, Gregory Morales, Kevin Monk, Kim-Phuong L. Vu, Thomas Strybel, and Vernol Battiste Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
493
500
507
516
526
535
Table of Contents – Part I
Part I: Design and Development Methods and Tools Visual Programming of Location-Based Services . . . . . . . . . . . . . . . . . . . . . Antonio Bottaro, Enrico Marino, Franco Milicchio, Alberto Paoluzzi, Maurizio Rosina, and Federico Spini
3
Connecting Envisioning Process to User Interface Design Process . . . . . . Naotake Hirasawa, Shinya Ogata, and Kiko Yamada-Kawai
13
Learner-Centered Methodology for Designing and Developing Multimedia Simulation for Biology Education . . . . . . . . . . . . . . . . . . . . . . . Chi-Cheng Lin, Mark Bergland, and Karen Klyczek
20
User Interface and Information Management of Scenarios . . . . . . . . . . . . . Robert Louden, Matt Fontaine, Glenn A. Martin, Jason Daly, and Sae Schatz
30
Giving UI Developers the Power of UI Design Patterns . . . . . . . . . . . . . . . Jocelyn Richard, Jean-Marc Robert, S´ebastien Malo, and Jo¨el Migneault
40
The Cultural Integration of Knowledge Management into Interactive Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Javed Anjum Sheikh, Bob Fields, and Elke Duncker
48
Supporting of Requirements Elicitation for Ensuring Services of Information Systems Used for Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuki Terawaki
58
Visualizing Programs on Different Levels of Abstractions . . . . . . . . . . . . . . Jo-Han Wu and Jan Stelovsky
66
Measurement and Evaluation in Service Engineering . . . . . . . . . . . . . . . . . . Sakae Yamamoto, Miki Hiroyuki, and Hirohiko Mori
76
A Human Interface Toolkit for Developing Operation Support System of Complex Industrial Systems with IVI-COM Technology . . . . . . . . . . . . Yangping Zhou, Yujie Dong, Xiaojing Huang, and Hidekazu Yoshikawa
82
Part II: Information and User Interfaces Design A Conceptual Model of the Axiomatic Usability Evaluation Method . . . . Yinni Guo, Robert W. Proctor, and Gavriel Salvendy
93
XXIV
Table of Contents – Part I
Study on Evaluation of Kawaii Colors Using Visual Analog Scale . . . . . . . Tsuyoshi Komatsu and Michiko Ohkura
103
Representation of Decision Making Process in Music Composition Based on Hypernetwork Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tetsuya Maeshiro, Shin-ichi Nakayama, and Midori Maeshiro
109
Some Issues toward Creating Human-Centric Services . . . . . . . . . . . . . . . . Hirohiko Mori
118
A User-Centric Metadata Creation Tool for Preserving the Nation’s Ecological Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fatma Nasoz, Renee C. Bryce, Craig J. Palmer, and David J. Rugg
122
Comparison between Mathematical Complexity and Human Feeling . . . . Masashi Okubo, Akiya Togo, and Shogo Takahashi
132
How Do Real or Virtual Agent’s Body and Instructions Contribute to Task Achievement? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yugo Takeuchi and Hisashi Naito
142
Interaction Mediate Agent Based on User Interruptibility Estimation . . . Takahiro Tanaka and Kinya Fujita
152
Ontological Approach to Aesthetic Feelings: A Multilingual Case of Cutism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akifumi Tokosumi and Fumina Teng
161
Constructing Phylogenetic Trees Based on Intra-group Analysis of Human Mitochondrial DNA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ivan Vogel, Frantiˇsek Zedek, and Pavel Oˇcen´ aˇsek
165
A Qualitative Study of Similarity Measures in Event-Based Data . . . . . . . Katerina Vrotsou and Camilla Forsell
170
Feasibility Study of Predictive Human Performance Modeling Technique in Field Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Naomi Yano, Toshiyuki Asahi, Shin’ichi Fukuzumi, and Bonnie E. John Surprise Generator for Virtual KANSEI Based on Human Surprise Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masaki Zenkoyoh and Ken Tomiyama
180
190
Table of Contents – Part I
XXV
Part III: Visualisation Techniques and Applications Explicit Modeling and Visualization of Imperfect Information in the Context of Decision Support for Tsunami Early Warning in Indonesia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Monika Friedemann, Ulrich Raape, Sven Tessmann, Thorsten Schoeckel, and Christian Strobl
201
Kansei Stroll Map: Walking around a City Using Visualized Impressions of Streetscapes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuichiro Kinoshita, Satoshi Tsukanaka, and Takumi Nakama
211
Multivariate Data Visualization: A Review from the Perception Aspect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yan Liu
221
Methods for Service Sciences from Visualization Points . . . . . . . . . . . . . . . Hiroyuki Miki, Naotsune Hosono, and Sakae Yamamoto
231
Interacting with Semantics: A User-Centered Visualization Adaptation Based on Semantics Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kawa Nazemi, Matthias Breyer, Jeanette Forster, Dirk Burkhardt, and Arjan Kuijper
239
Riding the Technology Wave: Effective Dashboard Data Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lisa Pappas and Lisa Whitman
249
Peer-to-Peer File Sharing Communication Detection Using Spherical SOM Visualization for Network Management . . . . . . . . . . . . . . . . . . . . . . . . Satoshi Togawa, Kazuhide Kanenishi, and Yoneo Yano
259
Visualizing Stakeholder Concerns with Anchored Map . . . . . . . . . . . . . . . . Takanori Ugai
268
VICPAM: A Visualization Tool for Examining Interaction Data in Multiple Display Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Roshanak Zilouchian Moghaddam and Brian Bailey
278
Part IV: Security and Privacy Privacy Concern in Ubiquitous Society and Research on Consumer Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yumi Asahi
291
Modelling Social Cognitive Theory to Explain Software Piracy Intention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ameetha Garbharran and Andrew Thatcher
301
XXVI
Table of Contents – Part I
A Practical Analysis of Smartphone Security . . . . . . . . . . . . . . . . . . . . . . . . Woongryul Jeon, Jeeyeon Kim, and Youngsook Lee
311
Cryptanalysis to a Remote User Authentication Scheme Using Smart Cards for Multi-server Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Youngsook Lee, Jeeyeon Kim, and Dongho Won
321
Exploring Informational Privacy Perceptions in the Context of Online Social Networks: A Phenomenology Perspective . . . . . . . . . . . . . . . . . . . . . . Emma Nuraihan Mior Ibrahim
330
Server-Aided Password-Authenticated Key Exchange: From 3-Party to Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junghyun Nam, Juryon Paik, Jeeyeon Kim, Youngsook Lee, and Dongho Won
339
Does Privacy Information Influence Users’ Online Purchasing Behavior? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jimmy H. Nguyen and Kim-Phuong L. Vu
349
Analysis of Authentication Protocols with Scyter: Case Study . . . . . . . . . Oˇcen´ aˇsek Pavel
359
Routing Functionality in the Logic Approach for Authentication Protocol Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ocenasek Pavel and Hranac Jakub
366
An Approach for Security Protocol Design Based on Zero-Knowledge Primitives Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oˇcen´ aˇsek Pavel
374
Part V: Touch and Gesture Interfaces Effects of Joint Acceleration on Rod’s Length Perception by Dynamic Touch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takafumi Asao, Yuta Kumazaki, and Kentaro Kotani
381
ERACLE: Electromyography System for Gesture Interaction . . . . . . . . . . Paolo Belluco, Monica Bordegoni, and Umberto Cugini
391
Development of Tactile and Haptic Systems for U.S. Infantry Navigation and Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Linda R. Elliott, Elmar T. Schmeisser, and Elizabeth S. Redden
399
Utilization of Shadow Media - Supporting Co-creation of Bodily Expression Activity in a Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Koji Iida, Shiroh Itai, Hiroko Nishi, and Yoshiyuki Miwa
408
Table of Contents – Part I
XXVII
Virtual Interaction between Human and Fabric . . . . . . . . . . . . . . . . . . . . . . Shigeru Inui, Akihiro Yoneyama, and Yosuke Horiba
418
Hand Gesture-Based Manipulation of a Personalized Avatar Robot in Remote Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Teruaki Ito
425
Vector Keyboard for Android Platform-Based Devices . . . . . . . . . . . . . . . . Martin Klima and Pavel Slavik
435
Study on Haptic Interaction with Digital Map on Mobile Device . . . . . . . Daiji Kobayashi, Yoshitaka Asami, and Sakae Yamamoto
443
Characteristics of Information Transmission Rates Using Noncontact Tactile Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kentaro Kotani, Masayoshi Hayashi, Nobuki Kido, and Takafumi Asao Multimodal Threat Cueing in Simulated Combat Vehicle with Tactile Information Switching between Threat and Waypoint Indication . . . . . . . Patrik Lif, Per-Anders Oskarsson, Bj¨ orn Lindahl, Johan Hedstr¨ om, and Jonathan Svensson
450
454
Design of Vibration Alert Interface Based on Tactile Adaptation Model to Vibration Stimulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yuki Mori, Takayuki Tanaka, and Shun’ichi Kaneko
462
Applicability of Touch Sense Controllers Using Warm and Cold Sensations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Miwa Nakanishi and Sakae Yamamoto
470
Information Processing for Constructing Tactile Perception of Motion: A MEG Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ayumi Nasu, Kentaro Kotani, Takafumi Asao, and Seiji Nakagawa
478
A Study on Selection Ability in the 3D Space by the Finger . . . . . . . . . . . Makoto Oka, Yutaro Ooba, Hidetaka Kuriiwa, Ryuta Yamada, and Hirohiko Mori Characteristics of Comfortable Sheet Switches on Control Panels of Electrical Appliances: Comparison Using Older and Younger Users . . . . . Yasuhiro Tanaka, Yuka Yamazaki, Masahiko Sakata, and Miwa Nakanishi Support for Generation of Sympathetic Embodied Awareness: Measurement of Hand Contact Improvisation under Load Fluctuation Stress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Takabumi Watanabe, Yoshiyuki Miwa, Go Naito, Norikazu Matsushima, and Hiroko Nishi
488
498
508
XXVIII
Table of Contents – Part I
Part VI: Adaptation and Personalisation Different People Different Styles: Impact of Personality Style in Web Sites Credibility Judgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rahayu Ahmad, Jieyu Wang, Karoly Hercegfi, and Anita Komlodi A Comprehensive Reference Model for Personalized Recommender Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Breyer, Kawa Nazemi, Christian Stab, Dirk Burkhardt, and Arjan Kuijper Dynamic Interface Reconfiguration Based on Different Ontological Relations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elisa Chiabrando, Roberto Furnari, Pierluigi Grillo, Silvia Likavec, and Ilaria Lombardi
521
528
538
Analysis of Content Filtering Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Woongryul Jeon, Youngsook Lee, and Dongho Won
548
A Smart Movie Recommendation System . . . . . . . . . . . . . . . . . . . . . . . . . . . Sang-Ki Ko, Sang-Min Choi, Hae-Sung Eom, Jeong-Won Cha, Hyunchul Cho, Laehyum Kim, and Yo-Sub Han
558
Interactive Personalization of Ambient Assisted Living Environments . . . Alexander Marinc, Carsten Stockl¨ ow, Anreas Braun, Carsten Limberger, Cristian Hofmann, and Arjan Kuijper
567
Development of a System for Proactive Information Service . . . . . . . . . . . Myon-Woong Park, Soo-Hong Lee, Young-Tae Sohn, Jae Kwan Kim, Ilju Bae, and Jae-Kwon Lim
577
My Personal User Interface: A Semantic User-Centric Approach to Manage and Share User Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Till Plumbaum, Katja Schulz, Martin Kurze, and Sahin Albayrak
585
Part VII: Measuring and Recognising Human Behaviour Effect of Menstrual Distress on Task Performance . . . . . . . . . . . . . . . . . . . . Keiko Kasamatsu, Mi Kyong Park, and Seiko Taki A Study of Attention Control by Using Eye Communication with an Anthropomorphic Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tatsuya Mita, Ryo Wada, Noriaki Kuwahara, and Kazunari Morimoto Auditory Feature Parameters for Music Based on Human Auditory Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Masashi Murakami and Toshikazu Kato
597
603
612
Table of Contents – Part I
Construction of a Model for Discriminating between Electroencephalographic Patterns at the Time of Incorrect Inputs Based on Sensitivity Spectrum Analysis . . . . . . . . . . . . . . . . . . . . . . Raita Ohori, Daiki Shinkai, Yoshimitsu Nagai, and Syohei Ishizu Basic Study of Analysis of Human Brain Activities during Car Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Noboru Takahashi, Shunji Shimizu, Yukihiro Hirata, Hiroyuki Nara, Hiroaki Inoue, Nobuhide Hirai, Senichiro Kikuchi, Eiju Watanabe, and Satoshi Kato Bereitschaftspotential Modeling by DBNM and Its Application to BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shino Takata, Toshimasa Yamazaki, Maiko Sakamoto, Takahiro Yamanoi, and Kenichi Kamijo Emotional Human-Machine Interaction: Cues from Facial Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tessa-Karina Tews, Michael Oehl, Felix W. Siebert, Rainer H¨ oger, and Helmut Faasch Development of an Eye-Tracking Pen Display for Analyzing Embodied Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michiya Yamamoto, Hiroshi Sato, Keisuke Yoshida, Takashi Nagamatsu, and Tomio Watanabe
XXIX
618
627
636
641
651
Care Giving System Based on Consciousness Recognition . . . . . . . . . . . . . Noriko Yokoyama, Tomoyuki Yamaguchi, and Shuji Hashimoto
659
Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
669
Developing Optimum Interface Design for On-Screen Chinese Proofreading Tasks Alan H.S. Chan, Joey C.Y. So, and Steve N.H. Tsang Department of Manufacturing Engineering and Engineering Management, City University of Hong Kong, Kowloon Tong, Hong Kong
[email protected]
Abstract. This paper includes a review of some related empirical studies concerning display factors that may contribute to the on-screen Chinese proofreading performance. The effects of typeface, font size, number of text lines, text direction, and copy placement on speed, accuracy, and subjective preferences in past reading tasks are discussed. This paper, in particular, introduces the development of a Chinese Proofreading System for proofreading experiments and delineates some research ideas for identifying the optimum interface design settings for on-screen Chinese proofreading tasks. The results of this research work are expected to provide useful design recommendations to assist in determining the display factor settings and text display layout that would improve work performance and satisfaction in Chinese comparison proofreading tasks. Keywords: Proofreading, Chinese Reading, Chinese Proofreading System, Usability Evaluation, Subjective preference.
1 Introduction Proofreading is an important task for reducing input and printing errors to ensure accurate information presentation to readers. The two main ways of proofreading are comparison proofreading and noncomparison proofreading. Comparison proofreading is a critical process in electronic book production and it involves careful reading of the dead copy (original copy) with a live copy and marking deviations (e.g. typeface, page and column breaks) from specifications, misspellings, nonstandard grammar, and other errors in language on the live copy. Noncomparison proofreading is usually done when there is no true dead copy or when the dead copy is only referred to if the proofreader sees something puzzling. An overview of the display factors of typeface, font size, number of lines of text, text direction, and copy placement that are likely to affect speed, accuracy, and subjective preferences in proofreading tasks is presented below. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 3–10, 2011. © Springer-Verlag Berlin Heidelberg 2011
4
A.H.S. Chan, J.C.Y. So, and S.N.H. Tsang
1.1 Typeface The display factors of typeface and font size have been shown to affect text readability and reading task efficiency on computer screens [1] - [3]. Research studies on Chinese characters with regard to legibility assessment [4], reading performance [5], and character identification task [6] have been reported. Chi et al. [4] found that Hei characters are the most legible, followed by Ming, Kai, and Li characters. In reading a variety of message signs, Lai [5] found that participants achieved higher accuracy for Hei and Ming styles than for Kai style. Given the diverse results concerning the effects of font type on different perceptual tasks, it is of great interest to attempt to determine the best font type for the proofreading process. 1.2 Font Size In measuring readability on computer screens as a function of character size, Snyder and Maddox [7] examined the effects of dot element size and character size and showed that smaller characters produced faster English reading speed than larger ones. However, in the same study, larger characters were found to produce faster search times in a search task. Larger characters are generally considered more readable than smaller ones, but it seems likely that there may be an optimum size such that changes in either direction (smaller or larger) will reduce reading performance [1, 8]. For reading or proofreading Chinese characters, the effects of font size are still largely unknown, though it was shown that the legibility of Chinese characters was enhanced by increasing font size [9]. Therefore, it is expected that on the basis of previous studies, larger font size should be preferred and result in better Chinese proofreading performance. 1.3 Number of Lines of Text, Line Spacing, and Line Length Past research examining the number of lines of text in paged view has produced varied results in reading tasks [10]. Ling and van Schaik [11] investigated the effects of text presentation on information search tasks and found that wider line spacing resulted in better accuracy and faster search times. Previous work on line length was conducted for English reading and the results revealed that line spacing and number of text lines are two opposing factors for determining the optimum line length for reading [12] - [14]. However, the optimum line length for reading Chinese is not yet known. There is thus a need to find out the optimum number of lines, line spacing, and line length for presenting large amount of materials in windowed view in Chinese proofreading task. 1.4 Text Direction Chinese text has been traditionally written and printed vertically from top to bottom of a page, with columns moving from right to left. However, since the early 20th
Developing Optimum Interface Design for On-Screen Chinese Proofreading Tasks
5
century this traditional convention has been largely westernized and, in a lot of printed material today, Chinese characters appear horizontally (usually starting in the top left corner). Sun et al. [15] found that horizontal Chinese characters were read twice as fast as vertical ones. Shih and Goonetilleke [16] revealed that for the Chinese population, a one-row word menu in Chinese was more effective than a one-column word menu in terms of search time and accuracy. Also, in a visual search study, Goonetilleke et al. [17] found that Chinese people used predominantly horizontal search patterns whilst searching for a Chinese target character in a background of distracting characters. Thus, the horizontal reading direction is expected to be more preferred and result in better Chinese proofreading performance than the vertical direction. However, it is believed that in a proofreading task, the text direction factor may interact with the copy placement factor. 1.5 Copy Placement Copy placement refers to the way in which the live and dead copies are arranged. Anderson [18] recommended that the live copy is always placed nearest to proofreader’s pen position. For the common left-right copy placement setting, the dead and live copies are under the left and right hands, respectively for the righthanders. An above-below setting can also be used and a common practice is that the dead copy is placed at the top and the live copy at the bottom [19], such that the live copy is also close to the proofreader’s hand. For Chinese characters particularly, a study on the effects of copy placement and its interaction with text direction on proofreading is necessary to help designers create better interfaces for proofreading tasks on computer screens.
2 Methodology 2.1 Experimental Design This study will examine the ways of improving proofreading performance and proofreader subjective preferences in two phases of research work. In view of the possible influence of the above mentioned display factors on proofreading Chinese, the effects of typeface, font size, text direction, and copy placement on speed, accuracy, and subjective preferences in proofreading will be examined in the first phase of this research. While the first phase is aimed to finding the optimum values or settings for display factors to present printed material on a page, the second phase will be directed at investigating how a large amount of text can be displayed and read effectively using scrolling in the window view method when the material cannot be accommodated within a single viewing page. The proofreading time and performance (number of hits, misses, false alarms and other errors.) will be measured and analyzed. To evaluate subjective preferences, a 10-point Likert scale is used to obtain participants’ opinions on attributes of proofreading comfort, ease and fatigue [3].
6
A.H.S. Chan, J.C.Y. So, and S.N.H. Tsang
2.2 Participants Native Chinese university students with more than five years of computer experience will be invited to participate in each experiment. They will all be right handed in operation of a computer mouse and for handwriting in daily life. Participants will be tested to ensure normal or corrected-to-normal vision with an Orthorator. 2.3 Materials A number of unrelated Chinese prose passages will be extracted from past examination papers of the Chinese Language subject of the Hong Kong Certificate of Education Examination. Mean passage length, including punctuation marks, will be around 550 characters. An average of five errors will be embedded per passage of a live copy. The errors will include extra word, missing word, wrong word, wrong order, and extra spacing, etc. Both dead and live copies will be presented in a positive polarity condition with black text on a white background. 2.4 Apparatus and Software A personal computer (Intel Xeon E5506 2.13 GHz) with a 24-inch liquid crystal display monitor and an application program prepared with Microsoft Visual Studio 2008 Professional will be used for stimulus presentation and response capture. At this stage of study, a Visual Basic program (2008 Express Edition) was used to develop the interface of Chinese Proofreading System. An experimenter can adjust the typeface, font size, line length, line spacing, text direction and copy placement according to specific test conditions. The system operation flow is shown in Fig. 1. An illustration of the proofreading interface is shown in Fig. 2. The proofreading system is embedded with five different check functions, which corresponds to the errors of extra word, missing word, wrong word, wrong order and extra spacing. Those check functions were assigned with different colors to stand out their representing errors. The dead copy and live copy are shown on the screen at the same time. The page number of the passage is shown on the top right hand corner to tell the progress of the test. The Next and Undo buttons are for proceeding to next passage and returning to last step respectively. Proofreading time and hitting accuracy for each passage are calculated by default formulas and those data will be stored for analysis. For detail operation of the proofreading system, readers can refer to the section 2.5. Generally, the Chinese Proofreading System contains three main components: User Maintenance, Participant Maintenance and Passage Maintenance. The User Maintenance allows the experimenter to assign experimental conditions to the participants. The Participant Maintenance allows participants to input their personal information. Their test performance is recorded. The Passage Maintenance allows experimenter to add or edit the passages.
Developing Optimum Interface Design for On-Screen Chinese Proofreading Tasks
Start
Start the Program
Enter Personal Information
Test Trial
Trial
Subjective Evaluation No
End of the Passages?
Yes End Fig. 1. System Operation Flow
7
8
A.H.S. Chan, J.C.Y. So, and S.N.H. Tsang
Fig. 2. An illustration of a developed proofreading interface
2.5 Chinese Proofreading System The purpose of this section is to show a proposed Chinese Proofreading System which will be used for the planned proofreading experiments. During proofreading, participants are allowed to scroll or navigate the two documents independently or jointly together by sliding the scroll wheel in the corresponding document frame areas. An illustration of the proofreading interface is shown in Figure 2. Participants will be asked to compare the live copy against the dead copy and to highlight the errors in the live copy as fast and accurately as they can by using the computer mouse to click the corresponding error type button shown at the bottom of the screen. Then the participants will be asked to evaluate subjective preferences for that condition using a 10-point Likert scale on attributes like comfort (1 = least comfortable, 10 = most comfortable), ease (1 = least easy, 10 = most easy) and fatigue (1 = least fatiguing, 10 = most fatiguing). This process will be repeated until the testing of all experimental conditions is finished. To avoid mental or visual fatigue, a two-minute rest will be given after a certain number of passages are proofread. Participants will take around one to two hours to complete the experiment.
3 Summary In view of the likely influence of the display factors discussed above on Chinese proofreading, the effects of typeface, font size, text direction, and copy placement on
Developing Optimum Interface Design for On-Screen Chinese Proofreading Tasks
9
speed, accuracy, and subjective preferences in proofreading materials displayed in a single page will be examined in the first experiment of this study. A natural extension of this work is to investigate how a large amount of text can be effectively displayed on screen so as to maximize performance and satisfaction of proofreaders when it is not possible to accommodate and present the material within a page of view. In each stage, the design details of an experiment will be carefully considered and confirmed. The experiments will then be conducted to collect participant task times, accuracy, and preference ratings for the different test conditions. The experimental data will be analyzed and the results will be useful for formulating design recommendations and determining the display factor and screen design settings that will improve comparison proofreading performance for Chinese. Acknowledgments. The work described in this paper was fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CityU 110410). The authors thank for the technical support provided by SM Wong.
References 1. Bernard, M.L., Chaparro, B.S., Mills, M.M., Halcomb, C.G.: Comparing the effects of text size and format on the readability of computer-displayed Times New Roman and Arial text. International Journal of Human-Computer Studies 59, 823–835 (2003) 2. Wang, A.H., Chen, C.H.: Effects of screen type, Chinese typography, text/background color combination, speed, and jump length for VDT leading display on users’ reading performance. International Journal of Industrial Ergonomics 31, 249–261 (2003) 3. Chan, A.H.S., Lee, P.S.K.: Effect of Display Factors on Chinese Reading Times, Comprehension Scores and Preferences. Behaviour & Information Technology 24, 81–91 (2005) 4. Chi, C.F., Cai, D., You, M.: Applying image descriptors to the assessment of legibility in Chinese characters. Ergonomics 46, 825–841 (2003) 5. Lai, C.J.: An ergonomic study of Chinese font and color display on variable message signs. Journal of the Chinese Institute of Industrial Engineers 25, 306–331 (2008) 6. Yau, Y.J., Chao, C.J., Hwang, S.L.: Optimization of Chinese interface design in motion environments. Displays 29, 308–315 (2008) 7. Snyder, H.L., Maddox, M.E.: Optimal element size-shape spacing combinations for a 567 matrix in information transfer from computer-generated dot-matrix displays.Tech. Rep. HFL-78-3, ARO-78-1 (1978) 8. Mills, C.B., Weldon, L.J.: Reading text from computer screens. Computing Surveys. Computing Surveys 19, 329–358 (1987) 9. Cai, D., Chi, C.F., You, M.: The legibility threshold of Chinese characters in three-type styles. International Journal of Industrial Ergonomics 27, 9–17 (2001) 10. Dyson, M.C.: How physical text layout affects reading from screen. Behaviour & Information Technology 23, 377–393 (2004) 11. Ling, J., van Schaikm, P.: The influence of line spacing and text alignment on visual search of web pages. Displays 28, 60–70 (2007) 12. Duchnicky, R.L., Kolers, P.A.: Readability of text scrolled on visual display terminals as a function of window size. Human Factors 25, 683–692 (1983)
10
A.H.S. Chan, J.C.Y. So, and S.N.H. Tsang
13. Dyson, M.C., Kipping, G.J.: The effects of line length and method of movement on patterns of reading from screen. Visible Language 32, 150–181 (1998) 14. Rayner, K., Reichle, E.D., Pollatsek, A.: Eye movement control in reading: an overview and model. In: Underwood, G. (ed.) Eye guidance in reading and scene perception, pp. 243–268. Elsevier, Oxford, UK (1998) 15. Sun, F.C., Morita, M., Stark, L.W.: Comparative patterns of reading eye movement in Chinese and English. Perception & Psychophysics 37, 502–506 (1985) 16. Shih, H.M., Goonetilleke, R.S.: Effectiveness of menu orientation in Chinese. Human Factors 40, 569–576 (1998) 17. Goonetilleke, R.S., Lau, W.C., Shih, H.M.: Visual search strategies and eye movements when searching Chinese character screens. International Journal of Human-Computer Studies 57, 447–468 (2002) 18. Anderson, L.K.: Handbook for Proofreading. NTC Business Books, Lincolnwood (1990) 19. Newby, G.B., Franks, C.: Distributed proofreading. In: Proceedings of the 3rd ACM/IEEECS Joint Conference on Digital Libraries, pp. 361–363. IEEE Computer Society, Washington (2003)
“Life Portal”: An Information Access Scheme Based on Life Logs Shin-ichiro Eitoku1, Manabu Motegi1, Rika Mochizuki1, Takashi Yagi1, Shin-yo Muto1, and Masanobu Abe2 1 Nippon Telegraph and Telephone Corporation, 1-1 Hikari-no-oka, Yokosuka-Shi, Kanagawa, 239-0847 Japan {eitoku.shinichiro,motegi.manabu,mochizuki.rika, yagi.takashi,muto.shinyo}@lab.ntt.co.jp 2 Okayama University, 1-1-1 Tsushima-naka Kita-ku Okayama-Shi Okayama, 700-8530 Japan
[email protected]
Abstract. In this paper, we propose a life log viewer that gives the users new findings from life logs in daily life, and provides seamless integration with external information. We classify life logs into two types. "Unintended life logs" are recorded automatically without the user's direct input; "intended life logs" are recorded purposefully by the user. Based on this classification, we implement a prototype that has two characteristics. First, it can visualize a user's unintended life log from long-term and multi-dimensional points of view. Second, its user interface is designed to visualize the transitions from the analysis results of the unintended life log to event data in the intended life log, and from event data in the intended life log to search results that provide information about the events. We believe that this viewer is a stepping-stone to the "Life Portal" that integrates existing portals with life log analysis to create a stimulus for search initiation. Keywords: Life log, Visualization, Scheduler, E-mail, GPS.
1 Introduction Thanks to the rapid growth of the Internet, we can now easily access enormous amounts of information. That includes niche information that we could not imagine accessing decades ago. Needless to say, search engines play a key role. However, they fail to support the user most effectively in that they do not recognize the user's interests or come up with ideas or things that the user wants to know; the user needs to be inspired by acquiring the desire to initiate searches. The miniaturization of mobile terminals with various sensors (e.g. GPS devices and acceleration sensors) has made it possible to continuously collect some kinds of life log over long periods without the user's direct intervention (e.g. position from GPS devices, operation from remote controller). In the MyLifeBits project, for example, Gemmell et al. proposed a platform to manage the personal life log extracted from many information sources [1]. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 11–20, 2011. © Springer-Verlag Berlin Heidelberg 2011
12
S. Eitoku et al.
Our idea is to use recollections as cues to initiate new searches; the result is that all searches start with life log analysis. We call this new search framework, "Life Portal." In this paper, we propose a life log viewer that can extract, for the user, new findings as search cues from the life log of daily life activities, and provides seamless integrated access to external information. For this purpose, we classify life logs into the types of “unintended life logs” and “intended life log”. "Unintended life logs" are recorded automatically without the user's intervention. "Intended life logs" are recorded manually for some specific purpose. Based on this classification, we implemented a prototype that has the following characteristics. Point 1: it visualizes the user's unintended life log gathered over the long-term with multi-dimensional viewpoints. Point 2: its user interface is designed to visualize transitions from the analysis results of the unintended life log to event data in the intended life log, and from event data in the intended life log to search results that provide information about the events. We conduct a simple experiment and confirm that these features of the prototype can give the user new viewpoints (motivation) to access other information.
2 Classification of Life Logs and Usages of Life Log Generally speaking, we are happier, feel more satisfaction or become excited when we find something that we did not expect. From this point of view, in terms of life log and its usage, we introduce the classification hypothesis shown in Figure 1. There are two types of life log. The intended life log covers blogs, photos, schedules, and so on, all of which are created by the user or intentionally recorded by the user for later use. The unintended life log includes locations obtained by GPS, motions as identified by acceleration sensors, actions obtained by a remote controller, and so on. These are automatically and continuously recorded with no direct user intervention over a long period. Our idea is that the user is more surprised when we get information from less aware things. Also, the user is more surprised when we can get information that is not included in the life logs. Therefore, user surprise increases in the following order, expected usage of intended life log, unexpected usage of intended life log, and unexpected usage of unintended life log. For example, reviewing old photos (intended life log) is an expected usage, because the users can get no more information than the one included old photos, so the degree of surprise is small. On the other hand, recommendation using user’s life log is one of the most effective usages at this moment, because the user will get unexpected information from unintended life log. The recommendation system of Amazon.com, Inc.[2], for example, proposes several Life log
Intended record
Expected usage
Add value
- Blogs - Photo taken by the user - Schedule etc. Unintended record
- Location (by GPS devices) - Motion (by acceleration sensors) - Action (by remote controller) etc.
- (Active) recall etc.
Unexpected usage - Recommendation - Information access (discussed in this paper) etc.
Fig. 1. Value in Life Log Usage
“Life Portal”: An Information Access Scheme Based on Life Logs
13
items that are related to an item selected by a particular user. The recommendation is generated based on the tendencies of other users that bought the selected and the tendency is calculated from unintended life log data; i.e., purchase history or URL click logs of other users. The use can get an item information that is not included his/her life log (unexpected usage), and purchase history or URL click logs are recorded automatically without the user's intervention (unintended life log). Our conviction is that new findings or ideas inspired by life log (especially, unintended life log) observations will invoke new searches, another unexpected usage of life log.
3 Concept of “Life Portal” The dotted frame in Figure 2 shows the current information access framework. The motivations for information access include user’s interest or curiosity and are invoked by stimuli from the external world. Users can get information by applying search engines to the Internet or from the feedback to questions proposed to Social Networking Service (SNS). Here, it should be noted that the user is expected to be aware of what he/she wants in the first place; i.e., even if there is a theme that will prove interesting to the user, nothing happens unless the user initiates the search by himself/herself. Another limitation of the framework is that the stimuli that cause user’s interest of curiosity mainly come from the external world. Much information in external world is recorded in the form of web page (e.g. news site and shop site) and we can access and use them easily in anytime. Although, human’s memory has limitation, so, it is more difficult to access and use his/her memories. Against these limitations, life log makes it possible to add a new aspect to this framework. First, life log can provide "personal" events that can invoke new searches, because it records what the user, and only the user, experienced. Therefore, the recall of a forgotten event raises the chance that the user will become curious about the event. Second, life log analysis can provide the user with new viewpoints or tendencies, because life log can continuously collect data over long periods from several information sources. By observing the data from long-term or multidimensional points of view, the user may reach a finding that differs from those he reached at the time. For example, if the user feels that he is busy today, his understanding can be improved by reference to activities over the last few weeks. Again, the new findings can invoke new searches. The integration of all components, shown by the solid frame in Figure 2, yields the "Life Portal". The visualization of unintended life logs and combination with the current information access framework Support techniques - Recommendation - Visualization etc.
Life Log
Awareness, New viewpoint Tendency, Event recalled
External world
Stimulus
Life Portal
Stimulus
Interest Curiosity
Current information access framework Portal Site
Search
Browse information
SNS Site
Post opinion or Call for information
Feedback from other users
Fig. 2. “Life Portal” and Current Informational Access Framework
14
S. Eitoku et al.
will create, in effect, a new information source that can invoke other interests or raise the user's curiosity. Just recording the user’s life logs, the user can find such a interests or curiosity and access related information easily.
4 Related Works Olguin et al. analyzed the frequency of face-to-face communication and the unrecognized behaviors of organizations [3]. They extracted the relationships between the subjects and estimated the subjects' condition. Ushiama et al. studied a method to extract the relationships between life logs collected from different information sources [4]. However, none of these papers discussed approaches to visualizing such life logs. On the other hand, there are some studies to visualize life logs for recall. Major approaches to the visualization of different kinds of life log include time lines and mapping. Rekimoto proposed an approach that uses time lines to visualize the files in a PC [5]. Aizawa et al. studied the interface needed to access life log holding data of outdoor actions [6]. De Silva et al. proposed a method to visualize and search the life log collected from peoples' activities in a house [7]. The method proposed by Ringel et al. shortens the time taken to search a life log by showing multiple search results on the same timeline [8]. Kim et al. use two axes for displaying a life log; a timeline and a map [9]. According to the result of user's operation on one axis, the life logs displayed on the other axis are automatically changed. Eagle et al. collected life logs of 100 subjects for 9 months by using the sensors on their cellular phone; GPS, Bluetooth, and so on. They visualize the data on a map [10]. As another approach to help the user to understand past events easily, methods that use the comic style have been proposed [11] [12]. However, none of these papers discussed how the user could be given new viewpoints that are one of important functions for “Life Portal”. In terms of combination of life logs for visualization, Ueoka et al. proposed a method in that the kinds of life log are changed dynamically to match the time granularity [13]. Kapler et al. proposed a method to visualize the activities on a timeline and a map at the same time [14]. However, the main usage of these methods is recall and they do not support integration with current information access frameworks, like having a function to access other sources of information.
5 Prototype System of “Life Portal” The prototype system was designed to support the following functions: - Visualization to find new information from unintended life logs - Seamless integration with the current information access framework The visualization of unintended life log induces new viewpoints. This raises new interests or curiosity that may trigger the access of other information. By linking the new viewpoints, those the user already recognizes, and external information, the user can access and browse external information through Life Portal.
“Life Portal”: An Information Access Scheme Based on Life Logs
15
As a basic function for supporting access to information through the motivation of the stimulus induced by the visualization, we introduce a scheduler. Note that the user enters the 4W attributes (who, what, when, and where) by himself/herself. Although it is not always true that the scheduler items are actually entered, the information is still a good reference for knowing what the user was interested in. Life log sources used by our prototype are the information registered in the scheduler, e-mails sent and received, and GPS data. The following sections introduce "Visualization of unintended life log" and "Access of Information" as provided by this prototype. 5.1 Panoramic View: Visualization to Find New Information from Unintended Life Log The first main function is "Panoramic View"; its aim is to show the frequencies with which the user contacted each person in each period. The frequencies of contact are calculated from the number of e-mails sent and received and the number of scheduled meetings or contacts. Visualization is implemented as color density (Figure 3). Stronger colors mean higher contact frequencies. Vertical axis plots the time line, and horizontal axis plots the names of the people contacted by the user. In this prototype, frequency is calculated by the following equation (1).
f [i ] = u[i ] / max u[ j ] j =0... n
i, j : ID allocated to each person the user contacted f[i] : Frequency concerning User i for a certain period u[i]: Sum of the number of schedule information and e-mails sent and received for a certain period, concerning the person whose ID is i
(1)
The first characteristic is to show long-terms trends and to permit comparisons among other people at the same time. Both attributes are important in finding new viewpoints. For example, a manager can find any imbalance in his business communication by comparing the contact frequencies of his subordinates. That is, we visualize these frequencies on two axis; time and people, not just timeline. The second characteristic is that data calculated on different granularities are displayed at the same time. For example, though a user got 100 e-mails from a certain person, receiving 100 e-mails on just one day in a month has quite different meaning from receiving 3 or 4 per day for a month. Therefore, we consider that it is important to display multiple levels of data at the same time. 5.2 Scheduler View: Seamless Integration with Current Information Access Framework The second main function is "Scheduler View"; its aim is seamless combination with the current information access framework. This is achieved by using the information in the user’s scheduler (Figure 4 (A)). The characteristic is that the user can access related information and browse it in the "Scheduler View" since the schedule information is automatically linked to the visit location extracted from GPS data.
16
S. Eitoku et al.
27/02/2009(Fri.) – 22/02/2010(Mon.) 2009/02 03
Contact frequencies for Meaning of color and color intensity
04 05
Time line
Blue color part: Frequency calculated per week (Strong color: high frequency)
06 07 08
1mont
1week
09
1week
10
1week
11
1week 1week
12
Mr. A Mr. B Mr. C Mr. D Mr. E Ms. F Mr. G Mr. H Mr. I Mr. J Mr. K Mr. L Mr. M Mr. N Mr. O Mr. P Mr. Q Mr. R Mr. S Mr. T
Contacted persons’ names
Frequenc y
…
02
Frequency for Ms.F
2010/01
…
Red color part: Frequency calculated per month (Strong color: high frequency)
Fig. 3. Overview of “Panoramic View”
This prototype provides two kinds of information through the "Scheduler View" according to the date of the schedule information; - Actual visited place and time corresponding to the selected schedule information (An example of the other kind of the user’s life log) - Information of shops near the place scheduled to visit (An example of information other than the user’s life log) When the user selects the schedule information of past events, actual visited addresses and times are displayed on a map (Figure 4 (B)). There is sometimes a difference between the information entered in the scheduler and the real action in terms of time. Also, it is sometimes difficult to identify the address from just the information in the scheduler, because the place names sometimes consists of just a part of an address (e.g. "Tokyo", "Osaka") or alternate terms (e.g. "Office") from which it is difficult to identify the address of visited place (e.g. "Yokosuka Kanagawa, Japan"). Therefore, this prototype determines the visited place as that which corresponds to the selected schedule information using the time attribute. At first, this system extracts visited place and time from GPS data using the method proposed in [15], and calculates the similarity defined by equation (2) for each extracted visited place. We assume that the visited place that had the highest similarity s(u,v,y,z) to the real action corresponded to the schedule information. The resulting matched pairs (e.g. Place name entered in scheduler: "Office" --- Corresponding address: "Yokosuka Kanagawa, Japan") are stored in a database, and this is used for estimating the place at which a future event will occur (mentioned below).
“Life Portal”: An Information Access Scheme Based on Life Logs
s (u , v, y, z ) = e
−(
17
z − y v −u ( v + u ) − ( z + y ) + − ) v −u z − y 2⋅( v −u )
(2)
u: Planned start time registered in scheduler,v: Planned end time registered in scheduler y: Start time the user stayed in the place, z: End time the user stayed in the place
When the user selects the information about a future event as entered in the scheduler, this system shows on map information of the shops near the place where the user will be staying (Figure 4 (C)); this information is an example of information that supports the user's future action. The system uses the place attribute (address) of selected schedule entry in the database to conduct a search. In the example mentioned above, when the user registers the schedule information whose place name is "Office", the system presumes that the scheduled event will be conducted at "Yokosuka Kanagawa, Japan". The prototype then displays the shop information near the address. (A) Overview
Information registered in scheduler Date (When)
Summary (What)
Attendee Place (Whom) (Where) Mr.E Mr.Y
Meeting
Musashino
(B) For past event
Visited place extracted from GPS data
(C) For future event
Place the user will Shop name
Musashino-Shi, Tokyo, Japan 12/02/2010 14:31-17:02
1-14-11 Musashino, Tokyo
12/02/2010 14:31-17:02 Musashino, Tokyo
Visited duration extracted from GPS Shop place
Shop address
Fig. 4. Overview of “Scheduler View”
5.3 Example of Usage Flow Figure 5 shows an example of usage flow. The user browses the frequency of contact in "Panoramic View" in Figure 5 (i). When the user clicks the contact frequency for certain person in a certain period (Figure 5 (A)), the frequency in the selected term is displayed in numerical value (Figure 5 (B)). Its range is from 0.0 to 1.0, 1.0 is the highest frequency. By clicking the button in Figure 5 (C), the prototype displays a summary of schedule information for the selected term. When the user selects a certain summary of schedule information (Figure 5 (D)), details of the selected item (date, summary, attendee and place information) are shown (Figure 5 (iv)). By clicking the place associated with this schedule information (Figure 5 (E)), the user
18
S. Eitoku et al. (A)
Visualization for giving awareness / new viewpoints
Ms. A
Mr. B
Mr. C
Mr. D
Mr. G
Mr. E Mr. F
Mr. Y
Display information to support user in acquiring details to find viewpoints
(i) Visualization of frequency of contact number calculated from e-mails sent and received and schedule information
(C)
Click (A)
Mr.J
(ii) Displaying contact frequency in numerical value Mr.Y
(B) Contact frequency
Mr.E
Click (C)
(D)
Mr.M
Schedule related to Mr. Y 2/12
Exhibition of N Corp.
2/23
Meeting for N Corp.
(iii) Displaying title of emails sent and received and summary of schedule information related to the selected person
Click (D) 12/02/2010 13:30-17:00 Exhibition of N.Corp.
(iv) Displaying the date, summary, attendee and place of selected schedule information
Mr. E, Mr.Y Musashino
Click (E) [For past event]
(E)
Musashino, Tokyo, Japan 12/02/2010 14:31-17:02 02/12/2010 14:31-17:02 Musashino, Tokyo
(v) Displaying the other life log as related to the schedule information.
(F) Access to external information
Click (E) [For future event] 1-14-11, Musashino, Tokyo
(vi) Displaying external information (e.g. recommended shops in the area where user will stay) using schedule information and GPS data
(G)
Fig. 5. Example of Flow in Usage of Prototype
“Life Portal”: An Information Access Scheme Based on Life Logs
19
can access other life log or information other than the user's life log. From the information about the past event, actual visited place, and time corresponding to the selected schedule information are shown; see Figure 5 (F). On the other hand, from the information of future events, some information about shops near the scheduled meeting spot is displayed (Figure 5 (G)).
6 Visualization of Actual Life Log We entered one subject's life logs (scheduler entries, e-mails received and sent for business, and GPS data) gathered over a 2-year period into the prototype (The subject is one of the authors). The subject then used this system and reported his experiences in using the prototype. The subject pointed out that "Panoramic View" allowed him to more clearly recognize the key turning points, which had been recognized only vaguely before. We can see that the subject exchanged e-mails and had meetings mainly with Mr. Q until October 2009 (Figure 6 (A)); however, from October 2009, the main contact person changed from Mr. Q to Mr. M and Ms. F (Figure 6 (B)). His work content actually changed around this time. He also pointed out the difference between the completion time of a project between the information in the schedule (intended life log) and that in the contact frequencies (unintended life log). He discovered that although the scheduler indicated that the project had already been finished, he continued to frequently contact the person in charge for some time (Figure 6, right). These results indicate that "Panoramic View" can stimulate new interests or satisfy curiosity by accessing other information. 27/02/2009(Fri.) – 23/02/2010(Tue.) 2009/0
Schedule Information
(A)
26/05/2009 11:00-12:00 Meeting concerned with “Event A”
03 04 05
02/06/2009 09:30-11:00 “Event A”
06 07
(B) 2009/05
08 09
2009/06
10 11 12
Mr. A
2010/0 02 Mr. A Mr. B Mr. C Mr. D Mr. E Ms. F Mr. G Mr. H Mr. I Mr. J Mr. K Mr. L Mr. M Mr. N Mr. O Mr. P Mr. Q Mr. R Mr. S Mr. T
Fig. 6. Example of Frequency of E-mails Sent and Received
20
S. Eitoku et al.
7 Conclusion and Future Works This paper introduced the framework of "Life Portal" as a new value of "Unexpected usage" of life logs. It allows life logs to yield new motivation for accessing information. In the "Life Portal" framework, we classify life logs in two types. "Unintended life logs" are recorded automatically without the user's intervention; "intended life logs" are deliberately recorded by the user. Based on this classification, a prototype was developed that could visualize the unintended life log in panoramic view, combine unintended and intended life logs, and provide seamless access to related information through schedule information. We conducted a simple experiment and confirmed that the prototype could give the subject new viewpoints that motivated him to access other information. In the future, we will conduct more detailed experiments to investigate the influence of visualization in finding new viewpoints. We will also develop a function to access information other than shop information.
References 1. Gemmell, J., et al.: MyLifeBits: fulfilling the Memex vision. In: Proc. of the tenth ACM international conference on Multimedia (MULTIMEDIA 2002), pp. 235–238 (2002) 2. Amazon.co.jp, http://www.amazon.co.jp/ 3. Olguin, D.O., et al.: Sensible Organizations: Technology and Methodology for Automatically Measuring Organizational Behavior. IEEE Trans. on Systems, Man, and Cybernetics Part B: Cybernetics 39(1), 43–54 (2009) 4. Ushiama, T., et al.: A Life-Log Search Model Based on Bayesian Network. In: Proc. of IEEE 6th International Symposium on Multimedia Software Engineering, ISMSE 2004 (2004) 5. Rekimoto, J.: Time-Machine Computing: A Time-centric Approach for the Information Environment. In: Proc. of UIST 1999, pp. 45–54 (1999) 6. Aizawa, K., et al.: Efficient Retrieval of Life Log Based on Context and Content. In: Proc. of CARPE 2004, pp. 22–30 (2004) 7. De Silva, G.C., et al.: An Interactive Multimedia Diary for the Home. IEEE Computer, Special Issue on Human Centered Computing 40(5), 52–59 (2007) 8. Ringel, M., et al.: Milestones in Time: The Value of Landmarks in Retrieving Information from Personal Stores. In: Proc. of INTERACT (2003) 9. Kim, I., et al.: PERSONE: Personalized Experience Recording and Searching On Networked Environment. In: Proc. of CARPE 2006, pp. 49–53 (2006) 10. Eagle, N., et al.: Reality Mining: Sensing Complex Social Systems. Personal and Ubiquitous Computing 10(4), 255–268 (2006) 11. Sumi, Y., Sakamoto, R., Nakao, K., Mase, K.: ComicDiary: Representing Individual Experiences in a Comics Style. In: Borriello, G., Holmquist, L.E. (eds.) UbiComp 2002. LNCS, vol. 2498, pp. 16–32. Springer, Heidelberg (2002) 12. Cho, S.B., et al.: AniDiary: Daily Cartoon-Style Diary Exploits Bayesian Networks. IEEE Pervasive Computing 6(3), 66–75 (2007) 13. Ueoka, R., et al.: Virtual Time Machine. In: Proc. of 11th International Conference on Human-Computer Interaction, HCII 2005 (2005) 14. Kapler, T., et al.: GeoTime Information Visualization. Proc. of IEEE Information Visualization 2004, 25–32 (2005) 15. Nishino, M., et al.: A place prediction algorithm based on frequent time-sensitive patterns. In: Proc. of Pervasive 2009 (2009)
Proposal of the Kawaii Search System Based on the First Sight of Impression Kyoko Hashiguchi and Katsuhiko Ogawa Faculty of Environment and Information Studies, Keio University, 5322 Endo Fujisawa-shi, Kanagawa-ken, 252-0882, Japan {t07624kh,ogw}@sfc.keio.ac.jp
Abstract. We propose a blog search engine called “Kawaii Search” (where Kawaii means pretty) to search blogs based on the impression of their text on a printing surface, considering factors such as the format and layout of text and density of words. Particularly in Japan, blogs reveal the personality characteristics of users depending on how they place their text. For example, some writers leave more space between lines or use hieroglyphics and “Gal words[1],” which consist of slang or abbreviations. Further, words can be categorized using four types of characters: kanji, hiragana, katakana, and alphabet. Each results in a different impression that reveals a writer’s personality. Given this approach, blog readers can not only read blog, but also interpret each writer’s personality. By focusing on impression differences, we propose a new search algorithm specialized for Japanese blogs. To show that these differences can act as the base of our search algorithm, we conducted an experiment that successfully verified the algorithm applied to the following three blog patterns: “kawaii” (pretty or lovely), “majime” (seriousness or industrious), and “futsu” (normal). The results show that in terms of the accuracy of the algorithm, our study categorized “kawaii” well; however, “majime” and “futsu” did not show good results. Keywords: Impression, Blog search engine, text formatting, Japanese blogosphere, information retrieval.
1 Introduction Blog search systems are generally based on the statistical and structural information in the blog text, including the frequency or relationships of words [2]. These systems search for blog articles that suit user requirements based on the content and keywordbased search techniques, including page rank and TFIDF [3]. With increase in the variety of blog writers’ styles and readers searching blogs for different reasons, a more sophisticated search system is required. For example, users not only want to read an article that matches the content they are searching for, but also want to find a blog that meets their aesthetic requirements. Particularly in Japanese blogs, pictographs and emoticons are frequently used to express a blog writer’s individuality. Conventional search systems such as Google do not reveal the atmosphere or personality of its text; however, when people read blogs and diaries, they often look G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 21–30, 2011. © Springer-Verlag Berlin Heidelberg 2011
22
K. Hashiguchi and K. Ogawa
not only at the words, but also the design and layout of the text. The Kawaii Search system analyzes the qualitative information such as impression and layout of blogs quantitatively. We therefore propose this search system to find blogs which are visually preferred by users.
2 Kawaii Search Kawaii Search is a system that searches blogs based on their appearance. In this section, we reveal typical differences found in blog appearances. We also describe the concept and vision of Kawaii Search. 2.1 Blogs in Japan When we compare blogs in Japan to those in other countries, blogs in Japan have more character sets, including kanji, hiragana, katakana, and Roman. Further, Japanese blog writers use more spacing, symbols, pictograms, and emoticons than blog writers in other countries. These various styles give readers different impressions for each individual blog. As an example, a blog discussing “Ichiro” and consisting of more kanji, less blank space, and no pictograms may give an impression of a serious blog, as shown in Figure 1(a). Conversely, even though the topic is the same, a blog that consists of less kanji, more blank space, and many of pictograms will have a pretty (or light) impression, as shown in Figure 1(b). Different writing patterns create different impressions.
(a)
(b)
Fig. 1. Two blogs discussing “Ichiro”; (a) consists of more kanji, less blank space, and no pictograms; (b) consists of less kanji, more blank space, and many emoticons
In Japan, blogs written celebrities are popular among the public [4]. Each individual writer has a distinct writing pattern. Since writers know that many people read their articles, they may consider ways to increase readership through the blog’s textual layout. A writer may arrange the article by using effective spacing and emoticons so that it appears prettier, or use kanji for a smarter appearance. In other words, a blog’s
Proposal of the Kawaii Search System Based on the First Sight of Impression
23
appearance indicates the impression of the article, and the style of writing reflects the writer’s personality and character. Blog’s readers not only understand the meaning of the blogs, but also read writer’s personality and character. In this way, because a blog’s appearance is important for reader and writer, we propose the search system based on the impression of their text on a printing surface, considering factors such as the format and layout of text and density of words. 2.2 Concept The Kawaii Search concept is described in this section. In addition to keyword-based blog search, Kawaii Search shows blogs similar to those of celebrities in terms of appearance. As illustrated in Figure 2, three celebrity blogs are shown at the top of the Kawaii Search site. These blogs are categorized by the following icons: Majime (serious), Kawaii (pretty), and Futsu (normal). Above these icons is a textbox in which users can enter their search keywords. At the bottom of the Kawaii Search interface, search results are listed and the site shows the titles and thumbnails of each “hit.” Instructions are as follows: 1. Enter the keyword and click one of the three icons shown (i.e., Majime, Kawaii, or Futsu). 2. The site will show blog articles similar to the one that the user selected. Figure 3 shows example results from a search request.
Fig. 2. Kawaii Search interface
In Figure 3, the first article is a kawaii celebrity blog, and the second article is a search result that is the top hits of kawaii. Since both writers use many pictograms and spaces, those articles have similar impression. Conversely, the third article is a majime celebrity blog, and the fourth article is the search result that is the top hits of majime. Both articles consist of more words and no pictograms or emoticons. As is evident from these examples, Kawaii Search can successfully match blogs that have a similar appearance to the base blog.
24
K. Hashiguchi and K. Ogawa
a. Kawaii celebrity blog
b. Search result similar to c. Majime celebrity blog kawaii celebrity blog
d. Search result similar to the majime celebrity blog.
Fig. 3. Example Kawaii Search results
3 The Kawaii Search Algorithm 3.1 Kawaii Value It is difficult to quantify judgmental standards used when a person reads a blog article; however, in many cases, we may judge external characteristics of the blog using the overall impressions of sentences. This includes measures such as line row, character arrangement, condition of sentences, and so on. Kawaii Search focuses on these constituents, using the following six variables: Conspicuous Value, Words Value, Vertical Space Value, Hiragana Value, Emoticon Value, and Pictogram Value. An explanation of each of these follows. Conspicuous Value. Conspicuous Value is based on how obvious Japanese characters are to the human eye. As illustrated in Figures 4(a) and 4(b), the conspicuousness of a word is based on the word itself and the words surrounding it. In Figure 4(a), when we compare (phoenix) and (one), seems bolder, whereas in Figure 4(b), in is not as prominent. As described in Figure 4(a), when we compare “ ” (phoenix) and ” ” (one), “ ” seems bolder. On the other hand, as shown in Figure 4(b), ” ” in “ ” is not as bold as “ ” in Figure 4(a). We define conspicuousness as the Conspicuous Value based on the strokes of a character. The calculation method is as follows. As shown in Figure 5, the red boxes identify the groups of words. The blue numbers are the number of strokes in each word. For example, to calculate the Conspicuous Value of the word (sunny) in the sentence shown, (it is sunny today), we first add 12 and 3, the number of strokes for . Second, if we have multiple characters in the word, we calculate the average by dividing the resulting sum by the number of characters in the word (i.e., 15/2 = 7.5). Third, we add the number of strokes in words next to this word; in the example, the neighboring words are and (i.e., 4 + 4 + 3 = 11). Fourth, we divide this sum by the number of characters in those surrounding words (i.e., 11/3 = 3.67). Finally, we divide the average number of the word by the
鸞 鳥鸞鳥
鸞
鸞
一
鸞
鸞
鸞
一 鳥鸞 鳥
晴れ
今日は晴れです 晴れ
は
です
晴れ
鸞
Proposal of the Kawaii Search System Based on the First Sight of Impression
25
average number of the surrounding words (i.e., 7.5/3.67 = 2.04). The Conspicuous Value for is therefore 2.04. By using these steps, all words are assigned Conspicuous Values.
晴れ
(a)
(b)
Fig. 4. Conspicuousness: (a) high conspicuousness; (b) low conspicuousness
Fig. 5. Example words and corresponding strokes for calculation of the Conspicuous Value of each word
Other Values. In this section, we describe the other proposed variables, i.e., the Words Value, Vertical Space Value, Emoticon Value, Pictogram Value, and Hiragana Value. Figure 6 illustrates these values using an example. Words Value is the number of words used in the given blog article. This value is used to measure the length of the article.
Fig. 6. Example blog excerpt with Words Value, Space Value, Emoticon Value, Pictogram Value, and Hiragana Value shown
Vertical Space Value, Emoticon Value, and Pictogram Value are the frequencies of appearance of those types of elements in the given blog article. Vertical Space Value corresponds to vertical space created by
tags; Emoticon Value corresponds to emoticons; and Pictogram value refers to pictograms. Figure 7 shows examples of emoticons. These values quantify the characteristics of the article as follows:
26
K. Hashiguchi and K. Ogawa
Emoticon Value = Emoticon / Words
(1)
Vertical Space Value = Space / Words
(2)
Pictogram Value = Pictogram / Words
(3)
In each of the above equations, words refers to the number of words in the given blog. ᧭
᧮
᧯
᧰
⅙㡴ቒ椇ₙቑ傃剡ትሺቂ
⅙㡴ቒ AYA
⅙㡴ቒ AYA
⅙㡴ቒᅀAAᅀ
ᇭ
椇ₙቑ傃剡ትሺቂᇭ
椇ₙAA
椇ₙቑAA
AA
ቑAA
傃剡! !
傃剡ትAA
AYA
Fig. 7. Example blog excerpt (“I practiced track and field today”) showing emoticons; in the first blog article, the Emoticon Value is zero; progressing left to right, the Emoticon Value. rise.
The Hiragana Value is the appearance frequency of hiragana in the given blog article, as illustrated in Figure 8. This value can be interpreted as the degree of softness, as expressed in the formula below (where letters do not include pictograms): Hiragana Value = hiragana / letters
(4)
Fig. 8. Example blog excerpts illustrating hiragana; progressing left to right, the Hiragana Value rises
3.2 Blog Templates: Kawaii, Normal, Serious We identified three blog templates as patterns for kawaii, futsu, and majime [5]. Each pattern’s values are detailed below. Majime Value =0.36 ×Conscious Value + 0.33 × Words Value - 0.51 ×Vertical Space Value
(5)
Kawaii Value = 0.31 × Vertical Space Value + 0.26 ×Pictogram Value+0.22×Emoticon Value+ 0.16 ×Words Value + 0.16×Conscious Value - 0.25×Hiragana Value
(6)
Futsu Value =0.28×Vertical Space Value + 0.23×Emoticon Value + 0.16×Hiragana Value - 0.35× Words Value - 0.06×Pictogram Value
(7)
Proposal of the Kawaii Search System Based on the First Sight of Impression
27
We set the Majime Value which can search the blog articles including a lot of numbers of characters, and a few spaces. Kawaii Value can search the blog articles including in a lot of spaces, pictograms and emoticons and words, and Futsu Value can pick out the blog articles including spaces, emoticons, and hiragana. We did scoring from the blog articles that the score is high by using these three values (Table 1). Table 1. The characteristic of each Value Majime Value
Kawaii Value
Futsu Value
High score
Conscious Value Words Value
Vertical Space Value Emoticon Value Hiragana Value
Low score
Vertical Space Value
Vertical Space Value Pictogram Value Emoticon Value Words Value Conscious Value Hiragana Value
Pictogram Value Words Value
3.3 System Structure In this section we describe the Kawaii Search system, which is composed of three building blocks. Overall, the Kawaii Search system is implemented in PHP and MySQL. The first component is the crawler, which downloads the blog articles. The second component is the indexer, which receives blog articles from the crawler and isolates the text. Next, the indexer analyzes the text using Mecab[6], which splits the text into its individual morphemes. In many cases, images, links, and advertisements are included in the given blog article. Our system does not accept any images, except for the pictographs, as determined by the algorithm. The indexer saves only the text, pictographs, emoticons, and
(line breaks) in the underlying database, and it calculates the six values described above and stores them in the database. The third component is the searcher. Users enter keywords and click on an icon categorized as Kawaii, Futsu, or Majime. The searcher obtains the six parameters for the blog article that the user clicked. At the same time, the searcher obtains blog articles in which the keywords match and retrieves the corresponding six parameters; this action occurs via the database. Next, the searcher calculates the difference between the acquired values and the values of the blog article that the user clicked. Scoring is done based on the number of differences (the fewer differences, the better). Finally, the searcher sorts in descending order and displays the results.
4 Evaluation 4.1 Experimental Method By using our Majime, Kawaii, and Futsu indices, we conducted an evaluation experiment to assess whether the searched blog articles represent the personality of the writers. The subjects in this evaluation were 12 college students in their 20s (6 women and 6 men); each participant reported reading blog articles before. In this time we set
28
K. Hashiguchi and K. Ogawa
searching words “Ukeru (where Ukeru means many kinds of meaning such as interesting, receive, catch, fun and so on ),” “Aho(where Aho means cluck),” “Rikujyo (Where Rikujo means track and field, land and so on.)” which are able to search valorous blogs in terms of impression. In the experiment, the top 10 (of 1200) blog articles that were searched using the Majime Value, Kawaii Value, and Futsu Value were used. Subjects were shown the blog articles which present blog articles that have been evaluated by many to be cute, normal, and serious. For each of the top 10 blog articles, subjects were asked to evaluate their similarity with the cute blog article, the normal blog article, and the serious blog article; the following 5-point scale was used: (5) very similar; (4) a little similar; (3) cannot say either; (2) not very similar; and (1) not similar at all. 4.2 Experimental Result and Discuss Experimental Result. Figure 9 shows the mean values of the scores reported by the subjects. The upper side of the table lists the keywords used in the experiment. For example, searching the blog articles that contained the keyword “Aho” by using the Kawaii Value returned articles that many that people found cute, but only a few articles that were found serious or normal. In contrast, searching the articles by using the Futsu Value returned articles that were found to be normal and serious to almost the same extent. .H\ZRUG$KRcluck
.DZDLL
PDMLPH
)XWVX
Keyword : Ukeru (interesting, receive, catch and fun) Keyword :Rikujo(track and field, land and so on)
Fig. 9. Mean Kawaii Value, Majime Value, and Fujitsu Value reported by subjects
Proposal of the Kawaii Search System Based on the First Sight of Impression
29
Discuss. As is evident in Figure 9, for any keyword, the Kawaii Value can be used to successfully select blogs that people find cute. The Majime Value and Futsu Value could not be used as they were similar. The Kawaii value is the indices that a blog article with many pictograms, spaces, emoticons is easy to be picked. Such blog articles are easy to understand the difference of the appearance in comparison with Majime and Futsu blog article. Therefore a blog article that user feel pretty is easy to be searched. In the Majime value that picked out the articles including much number of words and little space between the lines. For search keyword “Ukeru,” results were successful. Because the word “Ukeru” has several possible meanings (i.e. receive, get, and fun), the crawler downloads many types of blogs. However, the other keywords could not be used as they were similar. It is thought that there was a problem for the articles that the crawl downloaded. For search keyword "aho" and “Rikujo”, space tends to become wide if the blog articles have much number of words. For example, in "Rikujo", there are many blog articles written by records of the time, and such a blog article has a lot of words and much space. In addition, with a “aho”, there are many articles transferred from the other sites such as twitter [7] or 2channel[8]. These blog articles have a lot of words and much space between the lines. Therefore, the blog articles with a little space between the lines are hard to be chosen. Futsu Value could not be used as they were similar. There was a problem for the Futsu Value itself. The blog articles that are included many hiragana letters, and have much space are many kinds of appearances. So the Futsu value searched the blog article of various appearances. We need to adjust a blog article doing the crawl and Futsu value.
5 Results and Considerations In this paper, we proposed Kawaii Search to search blogs based on the impression formed by their text on human readers. We performed experiments to verify the utility of the Kawaii Search algorithm. By using experiments, we found that the Kawaii Value produce good results for selecting pretty. However, the Majime Value and Futsu Value did not produce good results. For our future work, we first need to correct the Majime Value and Futsu Value. Second, in addition to the six values described above, there are many factors that affect the impression of a word such as font and color. We need to expand the search algorithm by adding these types of factors. Third, when users read blog articles, the hardware used (for e.g., PC, iPad, smartphone) may have varying screen sizes, which affects the impression of the text. In this paper, we considered a standard PC or laptop screen; we need to consider the size of the screen in which the reader actually reads the blog article. Fourth, we analyzed the precision of our system based on an evaluation by 12 college students; in future, we plan to improve the search results by increasing the number of reviewers. Finally, we plan to solve these problems and improve search efficiency.
30
K. Hashiguchi and K. Ogawa
References 1. Tanabe, K.: Speech Patterns of Japanese Girls or Gals ‒Symbol of Identity and Opposition to Power, OPAL 3. Queen Mary, Univ. of London, London (2005) 2. Lindahl, C., Blount, E.: Weblogs: Simplifying Web Publishing. IEEE, Computer 36(11), 114–116 (2003) 3. Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, NY (1983) 4. ameba, http://official.ameba.jp/ 5. Kyoko, H., Katsuhiko, O.: MENKUI SEARCH: Search System Based on the First Sight of Impression Keio University, graduation thesis (2011) 6. mecab, http://mecab.sourceforge.net/ 7. twitter, http://twitter.com/ 8. 2channel, http://www.2ch.net/
Development of a Tracking Sound Game for Exercise Support of Visually Impaired Yoshikazu Ikegami, Keita Ito, Hironaga Ishii, and Michiko Ohkura Shibaura Institute of Technology, 3-7-5 Toyosu Koto-ku, Tokyo, Japan {m110007,m108017,m105005,ohkura}@sic.shibaura-it.ac.jp
Abstract. We developed an exercise support system that the visually impaired can use alone at home. Our system used the entertainment characteristics of games to encourage users to continue exercising. We focused on continuity, fun, and system usability and performed and improved our system by conducting experiments repeatedly. Keywords: visually impaired, system, sports.
1 Introduction Many visually impaired people have the desire to exercise [1]. However, they often cannot exercise for reasons that include no time and no facilities available for visually impaired. Based on this background, we developed an exercise support system that the visually impaired can use alone at home. Exercising must be continued to be effective. Therefore we developed an exercise support game that utilizes fun to motivate continued exercise. After development, we improved the fun of our system based on the results of evaluation experiments. In addition, we focused on its usability, and improved it so that the visually impaired can use it even when they are alone.
2 Development 2.1 Outline Figure 1 is an overview of the system. We employed a bicycle-type device for the use at home because it needs relatively small space. The system was controlled by a PC. The device sends such data as the degrees of leaning of the steering wheel and the numbers of pedal rotations to a PC, which sends sound data to the headphones. 2.2 Outline Because our previously developed system for the visually impaired that employed sound localization got favorable comments [2], we also employed sound localization for our new system. Fig. 2 shows an image of the game. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 31–35, 2011. © Springer-Verlag Berlin Heidelberg 2011
32
Y. Ikegami et al.
PC Sound Headphones
Data (degree of leaning)
Steering wheel
Pedal Data (pedal rotation) User
Fig. 1. Overview
Users
Target
Fig. 2. Image of game
The game flow is as follows: 1. Users wear the headphone and ride the bicycle-type device. 2. The PC outputs a target sound that is assumed to be located around the users.
Development of a Tracking Sound Game for Exercise Support of Visually Impaired
33
3. Users pedal the bicycle to catch the target to which they get closer to the target by handling the steering wheel and pedaling. 4. When users reach the target, it disappears, and a score is given. Then, a new target appears. 5. After a certain number of targets have appeared, one stage is finished, and the next stage begins. 6. When three stages have finished, the users receive by voice their total score, ranking, and mileage. We employed a lion’s roar and the sound of maracas as the target sounds.
3 Evaluation 3.1 Experiment We experimentally evaluated our prototype system with 20 subjects: seven were totally blind, five had amblyopia, and eight were unimpaired. We recorded the log data of the system and performed five-point scale questionnaires. The questionnaire results that focused on fun are shown in Fig. 3. The game was evaluated as enjoyable by both the visually impaired and unimpaired.
Totally⏷䦁劔 blind
Amblyopia ㇀尥劔
㤃䧋劔 Unimpaired
Fig. 3. Questionnaire results about system’s fun
34
Y. Ikegami et al.
3.2 Improvement of System Based on the questionnaire results, we made the following improvements. • • • • •
Added a ranking function. Added a function to raise the degree of difficulty when a user plays well. Added a function for pedaling backward. Added various game modes and target sounds. Simplified the game’s start.
For raising the difficulty, targets appeared at the positions that are difficult to localize and the time limit was shortened. We also added a new game mode with new target sounds such as sheep bleats and dog barks. 3.3 Evaluation Experiment of Improved System We re-evaluated our system after the above improvements. We set it up in a school for visually impaired students, and 17 students freely used it for two months. As in previous experiment, we recorded log data and performed questionnaires whose main items were the following. • Which parts of the system should be improved? • Which elements of the system were enjoyable? Figure 4 shows how many times each student used the system. Although some used it more than ten times, many students only used it a few times. This indicates that our current system needs more improvement to increase continuity. Figure 5 shows the questionnaire result that focused on the enjoyable system elements. Riding a bicycle-type device and getting scores were considered fun.
Students Times system used Fig. 4. Frequency that system was used
Development of a Tracking Sound Game for Exercise Support of Visually Impaired
35
(1) Catching sounds
(2) Operating bike
(1)
(3) (4)
(2)
(5) (3) Exercising
(4) Getting scores 0%
20%
40%
60%
80%
100% (5) Other wise
Fig. 5. Questionnaire results: enjoyable elements
4
Conclusions
In this study, we developed a system that encourages the visually impaired to continue exercising at home. The evaluation experiment results suggest that our system provided enjoyable support for continued exercise for the visually impaired. Data from an evaluation experiment of our improved system, showed advancements of continuity and fun and what elements were considered fun. We will implement a competition function because of such a strong demand from the young visually impaired students.
References 1. National rehabilitation center for persons with disabilities: A report of survey about exercise and sports for graduates of national rehabilitation support center for visually impaired (in Japanese), http://www.rehab.go.jp/achievements/japanese/19th/paper20.html 2. Ishii, H., Inde, M., Ohkura, M.: Development of a game for the visually impaired. In: 16th World Congress on Ergonomics, IEA2006, CD-ROM, Maastricht (July 2006)
From Personal to Collaborative Information Management: A Design Science’s Approach* Mei Lu¹ and Philip Corriveau² 1 Samsung Information Systems America 75 West Plumeria Drive, San Jose, California, USA 2 Intel Corporation, 5200 NE Elam Young Parkway, Hillsboro, Oregon, USA
[email protected],
[email protected]
Abstract. This paper reports findings from evaluation of five solution concepts aimed to address challenges in managing projects, tasks, and different modes of work. Both users and Information Technology (IT) departments appeared to best resonate with the concept of “project workspace,” which was conceptualized as a persistent space that allowed users to organize, track, and resume active project work. Even though users agreed that multi-tasking and interruptions were characteristics of their daily jobs, they did not desire mechanisms to block or alter information flows. Instead, users wanted information management to be streamlined in the contexts of collaboration and teamwork. The most desirable scenarios included spontaneous retrieval of information related to a customer or colleague, quick information assembly for different phases of a project, effective management of team tasks, and seamless connection from personal to team workspaces. Keywords: Information management, search, tasks, projects, collaboration.
1 Introduction Abundant research has studied business users’ challenges in personal information management. Three main areas of issues are documented: 1) Interruptions and multitasking -- Users typically handle multiple tasks on their computer, and work with frequent interruptions from both internal and external sources; 2) Source and tool fragmentation -Users’ work or decision-making often relies on information from disconnected sources, devices, or applications; 3) Challenges with information organization -- Users spend minimum effort to organize information thus later have challenges in finding and re-using useful information. Two main gaps exist in the literature: 1) Most research focuses on end user perspectives; few has examined IT departments or overall business’ views; 2) While a number of design implications have been generated from different studies, fewer projects have evaluated multiple potential solutions in business settings, and compared benefits and priorities. This research adopts a design science research [11]’s approach, which creates and evaluates IT artifacts intended to solve identified organizational problems. The authors provide seven guidelines for an effective design *
The research was conducted when the author was working in Intel Corporation.
G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 36–45, 2011. © Springer-Verlag Berlin Heidelberg 2011
From Personal to Collaborative Information Management: A Design Science’s Approach
37
science research –“design-science research requires the creation of an innovative, purposeful artifact (Guideline 1) for a specified problem domain (Guideline 2).” (p.8) It requires thorough evaluation of the artifact (Guideline 3), and innovative solving of known or unknown problems (Guideline 4). The artifact must be rigorously defined (Guideline 5) based on the construction of the problem space (Guideline 6). “Finally, the results of the design-science research must be communicated effectively” (Guideline 7). These seven guidelines can be summarized into four major milestones: 1) A thorough understanding of the problem space; 2) Creation of innovative artifacts; 3) Evaluation of the effectiveness of the artifacts; 4) Communication of results. In this project, we first tried to understand challenges business users face in personal information management through literature reviews. Secondly, with team brainstorming and ideation, five potential solution concepts were proposed. Afterward, to evaluate the concepts and further explore ideas for desired solutions, indepth interviews were conducted with 28 participants from 14 US companies. The remainder of the paper is organized according to the major steps of design science research whereby we start with a definition of the problem space, then discuss designs or proposed solution concepts. Further, we report findings from the evaluation research, and provide recommendations for design and development.
2 The Problem Space: Challenges in Information Management Personal information management refers to individual user activities to acquire, organize, or retrieve information on personal computers. The dynamic and complexity of personal information management in business environments have increased dramatically in the past decades, as individual work, collaboration, and business operations are all becoming more computerized. Challenges are well documented. They can be summarized into three main areas. Multi-tasking and interruptions: Users typically handle multiple tasks on computers, and work with frequent interruptions from both internal and external sources [2, 7, 10, 12, 16]. Interruptions cause distraction from the current work, redundant effort, forgotten tasks, and cognitive burden and delays when users try to resume previous work [12, 16]. Current information organization tools do not provide adequate support for frequent task suspensions and resumptions. Long-term projects are more complex and harder to return to than short-term activities. Further, as pointed out by Chudoba, et al. [6], the issue of multi-tasking and interruptions can be potentially worsened by the situation that employees in large companies typically work simultaneously in multiple teams and projects. Information source and tools fragmentation: Users’ work creation or decisionmaking often depends on information from disconnected sources or applications [3, 13, 15]. Lu, et al. [15] demonstrate that variety and fragmentation of tools and information sources is detrimental to teams’ performance. Further, the tools people use are often inadequate for their goals [14]. For example, Bellotti, et al. [2] report that users embed extensive task management activities into e-mail communication, which leads to ineffectiveness in managing priority, deadlines, and workload. Several other studies have observed that e-mail is overly and inadequately used for multiple purposes, such as task and contact management, and personal archiving [8, 18, 19].
38
M. Lu and P. Corriveau
Burdens of information organization and re-use: Most users devote little time to organize information on computers [5]. For those who do organize, they have to spend scattered efforts to classify different types of information, such as e-mail, documents and web bookmarks [5]. Storing information in different formats and structures leads to challenges in retrieving and re-using information [9]. To summarize, key user needs for personal information management include: 1) Ability to easily organize information and preserve contexts for different task and projects, so that users can resume tasks or work after temporary suspension [7,12,16]; 2) Assistance with tracking, organization and reminder of short term activities; 3) Integration of information from different sources so that users can easily access all information related to different topics of work [3, 13, 17]; 4) Tools to contextualize interruptions -- As observed by Mark, et al. [16], collocated employees work longer uninterrupted sessions than distributed employees, suggesting collocated employees are more likely to be interrupted at natural breaks. Tools may help users better communicate work contexts so that interruptions can be more appropriately timed.
3 Design Artifacts: The Solution Concepts A team of 19 people in Intel gathered for a 6-hour brainstorming and ideation session. The participants included user researchers or human factors engineers, designers, market researchers, marketing and strategic planners, platform architects, software architects from Intel’s IT department, and a researcher from an external research firm who facilitated the discussions. The brainstorming session started with a review of literature and research data on user needs compiled by a user researcher and market researcher. The meeting was followed by a team discussion on information assistance, or how computers or intelligent agents can assist users with information management. In order to stimulate innovative ideas, the team first tried to generate metaphors and real world analogies of “assistance.” Examples included: a golden retriever helps his or her owner to fetch newspapers in the morning; an employee at Home Depot helps a disoriented customer at a store; a tunnel that helps travelers to get to the destination quicker; animals leave traces along the way they have walked by; a person’s frustration when he or she cannot find a car key at home, and actions he or she may take to prevent that. The team was then divided into four sub-groups to brainstorm on solutions. The brainstorming concluded with a consolidation of ideas from subgroups. Afterward, a smaller team of researchers and designers continued to refine the ideas into five solution concepts, which are summarized in Table 1.
4 Evaluation of the Artifacts: The In-Depth User Research Semi-structured in-depth interviews were conducted with a total of 28 employees from 14 businesses in two US cities: the New York City and Kansas City. Of the businesses, three were small businesses with fewer than 100 employees, five were mid-sized (100-1000 employees) and six were large enterprises (>1000 employees). They came from a variety of industry verticals. We interviewed two people separately from each business – a knowledge worker and an IT decision maker, who might be the chief information officer (CIO), directors/managers of IT departments, or those who served as IT people in small businesses.
From Personal to Collaborative Information Management: A Design Science’s Approach
39
Table 1. Proposed solution concepts and targeted user needs Solution concepts Project workspace
Easy search
Proactive search Task handler
Mode selector
Summary
Targeted needs
A persistent space that organizes and tracks all relevant information for each project. Users no longer need to switch from one application to another to view different elements for a project – all elements can be viewed in one “workspace.” Previous contexts are preserved if interruption occurs. Users can easily search across both internal (e.g. a PC’s hard-drive or company network) and external (e.g. the Internet) information sources. Searches can be extended across different file types, and can be initiated from any application. It learns about users’ interests and activities, and proactively seeks and compiles information relevant to a user’ job or tasks. It enables users to use handhelds or computers to create and track tasks. Users use handhelds to (e.g., through voice commands) communicate to their computers to create or update tasks. Further, the program will automatically analyze and structure tasks to track due dates, reminders, and identify bottlenecks in workload. Users can select different work modes (e.g., “In a Meeting,” and “Presenting”). Depending on the mode, the solution reconfigures a user’s desktop and applications, and communicates the user’s status. Non-essential elements are faded into the background, and non-critical incoming communications may be blocked or routed.
Integration of information from different sources; preservation of work contexts; easy work resumption after interruption Reduced needs for manual information organization. Integration of information from different sources. Integration and notification of useful information. Assistance with tracking, organization and reminder of short-term tasks and activities.
Interruption reduction or contextualization; Better access to tools and information needed for different modes of work.
The two-hour interviews were conducted in participants’ offices, and had two parts. The first part was about users’ current practices in information management. In the second part, we evaluated the five solution concepts described with visual storyboards and scenarios. According to Hevner et al. [11], IS research “must address the interplay of business strategy, IT strategy, organizational infrastructure and IS infrastructure.” (p. 4). Thus in the interviews, we asked participants to assess the usefulness and uniqueness of the proposed solutions from different perspectives. IT decision makers were invited to discuss perceived benefits of the solution concepts and major adoption hurdles from IT and their company’s business perspectives. With knowledge workers, we evaluated how the solutions might be relevant to work effectiveness and efficiency. At the end, the participants were invited to provide ratings along a five-point scale on the usefulness and uniqueness of the concepts. In this qualitative research, the quantitative rating was used as a way to stimulate deeper thinking on why a participant liked or disliked a concept, and the extent of its relevancy for his or her job. Participants were encouraged to support their viewpoints with real examples, or demonstrate on their computer on issues they had encountered.
40
M. Lu and P. Corriveau
4.1 Information Flows and Personas We identified two kinds of information flows that were both critical to users’ daily jobs or business operations: structured and unstructured. Structured information was data stored in databases with consistent formats, and could be computed and analyzed by computers. Examples included data in manufacturing production, inventory, financial, customer relationship management, and business intelligence systems. Unstructured information was content that could not be easily interpreted and analyzed by machines, such as e-mail messages, web sites, and documents. For both structured and unstructured information, users’ daily information flows involved internal colleagues and external collaborators, for example, customers, suppliers and partners. Personas are archetypical representation of major categories or segments of users [1]. By creating personas, we sought to typify user needs, values, and behavior patterns, and assist designers and architects to envision solutions against those needs. We observed four major business personas who had different patterns of priorities. Senior management: His or her workdays were highly unstructured. E-mail and meetings were the two most important communication mechanisms. Newly received information often determined what they needed to do next. Constant access to e-mail and business information (e.g., budget information in the financial database) was critical. They had a strong preference for a device that was wearable in a pocket and allowed them to be always connected with business information and key staff. Road warriors: They worked with both structured and unstructured information while on the road. They typically had job roles such as sales, account managers, and customer services. Primary goals were to be instantaneously prepared for their customer meetings, quickly solve problems, or easily relay information back to their company on customer orders, issues, or market trends. Office workers: They were mainly knowledge workers or middle management. They typically worked in offices more with unstructured information. They had challenges in getting information organized and shared, quickly finding related information for work creation or completion. Special workforces: A large portion of the workforces performed business critical functions with structured data from thin clients, which were light-weight computing devices mainly served as connecting points between servers and monitor displays. IT described a number of advantages for using such devices, including ease of security management; ease of device deployment and replacement without the needs to install the operating system, applications and data; and ease of update and maintenance. 4.2 Acceptance of the Solution Concepts In this section, we summarize user feedbacks to the five solution concepts. Project Workspace. This concept appears to be best received. It appealed more to knowledge workers/middle management than to other personas -- senior management and road warriors’ work tended to be less structured around projects; special workforces typically worked with information in structured databases. The concept received an average usefulness rating of 4.2 out of 5, and a uniqueness rating of 4.1.
From Personal to Collaborative Information Management: A Design Science’s Approach
41
Users could identify the benefits we had hypothesized for this concept, including, one location for different types of information for a project; easy retrieval of information related to a particular topic; easy focus or resumption of tasks after interruptions; easy accumulation of project history and reservation of work contexts; and potentials for easy sharing of projects information and more effective collaboration. The main question users asked was how easy it might be to learn to use it. IT was particularly concerned with the potential learning curve, and regarded it highly detrimental to IT’s reputation if users were unwilling to adopt a solution IT deployed. In addition, IT questioned whether the solution would create multiple layers of information organization and duplication of files. The concern was about the potential extra burden for data storage management and file backup. Task Handler. This concept appealed more to senior management and road warriors. It received a usefulness rating of 3.9, and uniqueness of 3.7. It was hypothesized to allow more effective management of tasks that requires immediate or near term actions. However, users identified more with benefits that were initially not in the core of hypothesis, such as • Voice inputs into PC: Several participants, especially senior management or people with a support staff, viewed voice inputs as more efficient than typing. For example, in a law firm, lawyers often used audio dictation as a way to communicate with assistants. Participants mentioned that they could use such a tool to create voice mails or delegate tasks to other people. Participants also perceived benefits in using a voice tool to create inputs into structured databases. For example, a sale person could use this tool to record customer requests or feedbacks in a customer relationship management system, immediately after he or she left the customer’s office; or a railroad operator could use it to easily update the status of a cargo car in databases. • Spontaneous capture of thoughts: such a tool allowed users to capture thoughts which they might forgot later on, as the operation manager in a large retail company said, “I am kind of tangent thinker. I might be walking down the street and thinking of something. This would enable me to kind of document it and get it active.” IT expressed more concerns on this concept than knowledge workers. One was security. If such a device would constantly sync up with their computers or enterprise’s networks, IT questioned the cost and burden to implement the virtual private network (VPN) for the device. IT and senior management were not convinced of the justification of such a potentially expensive device only for the purposes of tracking and managing daily tasks. They regarded task management a basic skill from their employees. They expected such a device to perform more business-critical functions, for example, for users to use voice to create inputs into business operation databases, particularly at places without computer access or network connection; for users to manage team tasks or quickly relay tasks to other people; or for senior management to access and manage business critical tasks on operation databases. Easy Internal and External Search. This concept received an average usefulness rating of 3.9, and uniqueness of 3.3. An easy search tool was regarded a must-have, however, it would not satisfy all needs for information retrieval and organization.
42
M. Lu and P. Corriveau
Participants mentioned that they still would like to browse first, and use search as the last resort. Search was especially important for finding historical documents either created by self or others in the company. Key needs include speed of search; ability to search across own hard drives and shared network repositories; and search within different file types, including documents, e-mails, calendars, PDF files, and binary files (for example, computer aided design documents). Several IT managers said that they did not see the needs to combine both internal and external searches – ‘They serves different purposes.” And “there might be security issues.” In addition, IT managers were concerned with the burden of search indexing, storage maintenance, backup, and access right control; as well as potential impact on the performance of document management systems. Proactive Search. This concept received an average usefulness rating of 3.7 and uniqueness of 3.4. It appealed more to senior management or those people who worked with time-sensitive information. For example, a person who managed investor relationships mentioned that it was extremely important to stay current with any thing people might say about the company. The participants could identify the potential values of proactive and tailored information delivery, which would integrate and track information from different sources, and allow them to be constantly informed about topics of interests. However, they worried that the tool would deliver more irrelevant information and thus cause overload or distraction. Some participants mentioned that they would like to retain the control on when and how to find information, as one participant said, “I would rather search when I think I need it.” Or as another user puts it, “I don’t want my computer to be too smart. It is just a tool.” In addition, several participants asked about their privacy – “will I be always tracked on what I read?” Mode Selector. The participants typically didn’t find this concept appealing. It received an average usefulness rating of 3.0 and uniqueness of 3.7. They didn’t consider themselves working in different modes. More importantly, they seemed to be proud of the dynamic of their workdays created by different information flows, and regarded it an indicator as being successful in workplaces. Even if they might need to work without interruption, for example, when in a meeting with executives, they didn’t see the needs of any sophisticated applications to block or alter incoming communication flows, or help to rearrange their information or applications. We sensed a strong sentiment that users didn’t like something that implied that they needed help organizing their work or remembering their tasks. “If you have something important to do, you really should be just doing it yourself. You shouldn’t need this, all this stuff to help you ... And this seems like it’s something for people who can’t concentrate...”
5 Discussion and Recommendations Even though personal information management is an individual action that happens on personal computers, we find that the most important needs can be best described within the contexts of users’ collaboration with internal and external teams. From the participants’ discussions about their current practices of information management,
From Personal to Collaborative Information Management: A Design Science’s Approach
43
and their reactions to the potential solution concepts, their information management priorities are anchored around mainly two notions: projects and people. Both knowledge workers and IT regard effective management of information related to projects and customers as most critical to their work and business objectives. They desire solutions that integrate both structured and unstructured information and streamline the flows in different phases of project activities. They desire solutions that support quick information retrieval and re-use for better services or support to either internal colleagues or external customers. User activities and needs appear to evolve as a project advances through its life cycle. Three phases of project activities are identified from the study: exploration, creation, and conclusion. Project-based information management is intertwined with management related to three main groups of people: customers, experts and team members. Exploration and Initiation. As one participant said, “a project often becomes a project after a fair amount of discussions.” A good search tool is critical to identify 1) existing documents or information that can be re-used for a potential project; 2) expertise in the company that is relevant for project decision or formation of teams. As a Vice President in a legal firm said, “quick (document) turn around (to a customer) is critical.” A desirable scenario is for users to be able to use a search tool to pull information from different sources, such as shared document repositories, email messages, and structured databases. Afterwards, users can easily identify reusable information (for example, budget, scope and duration of a similar project), quickly form a proposal or project plan for a potential customer. As the project is formally established, a user can gather all the relevant information from earlier exploration and discussion, create a formal workspace for next phases of the project. Work Creation. Consistent with several studies [3, 7, 15, 17], users need to have better ways to organize active work and information related to projects. Currently, a user may find active information related to a project scattered in different applications, such as e-mail, word processing, spreadsheet, and presentation applications. There is no effective mechanism to easily bring different information related to a topic in one view. Over the time, users need to easily classify and accumulate information from different sources or of different types, e.g., e-mail, documents, instant messaging, notes, web pages, and work history, in a persistent space. While the concept of “project workspace” was presented as workspace for individuals, the participants wanted their personal workspace to be seamlessly connected to a team space thus individual work can be shared with teams. During the course of work, users will often need spontaneous information retrieval for unplanned events. As described by one participant, “everything is planned, but it could change or shift around depending on what priorities take place and what new information is discovered.” Examples of useful scenarios can be: when a user receives a phone call from a customer inquiring about project status, he or she can use one interface to retrieve all information related to the customer, including e-mail messages, customer account information in a database, or attachments this customer has sent via e-mail, but has been saved to a hard drive. As a participant said, “When a phone call comes in, if it (software) automatically pulls up the files, that would be a
44
M. Lu and P. Corriveau
fabulous thing for a sales person.” Users may use similar tools to quickly gather information in a meeting about meeting topics, presenters or attendees. Users need mechanisms that unobtrusively help them capture and track their tasks. Task management needs to be embedded in applications or other information management tools. Blandford and Green [4] identify important factors determining people’s choices of tools for time and task management, as how portable, accessible, shareable, and updateable tools are. From this study, the majority of users informally manage short-term tasks and commitments with tools that they are already familiar with, for example, e-mail, documents, calendars, and sticky notes. Very few of them spend time setting up a formal task lists using a dedicated tool. Users seem unwilling to invest time to learn a new tool; IT or senior managers are not eager to invest for extra hardware or software. Management of shared or delegated tasks is regarded as more important than that of own individual tasks. An example of desired scenarios is creating a task list for a team can be as easy as creating a table in a document or email message. Once the list is sent or saved in the team space, the tasks are automatically updated in different owners’ individual task lists. A user can also easily “tag” information embedded in different sources to update own task list, for example, highlight a sentence in an e-mail, and select to tag it as “my task”. Project Conclusion. Upon successful completion of a project, users and IT want to easily archive all related information. IT wants to manage the content or information according to document retention policies, for example, in compliance with legal requirements. For end users, the primary goal is to ensure easier retrieval for future projects or references. A project workspace as described above can help users to accumulate all related information over the course of creation so that the burden is not huge at the end of the project. As a conclusion, while reporting findings on user needs and desired scenarios for collaborative information management, we have described a methodology designed to gather in-depth feedbacks from business users on values of potential technologies at the early stage of a product or strategic planning cycle. With this method, we seek to gather information from three perspectives -- 1) End users: whether a solution can fit into and improve users’ daily jobs, and help them better fulfill their primary goals; 2) IT: whether a solution can be easily managed and supported by the IT departments; 3) Business, in this study, represented by IT whose decision on IT investment is tied to business needs: whether a solution is tied into overall business objectives or strategies. Within Intel, these data have been used, in conjunction with quantitative data, to inform decision-making on platform feature prioritization or ecosystem enablement. These data are also used to create detailed user scenarios to inform technical capability design or gap analyses.
References 1. Anderson, G., Bramlett, B.W., Gao, J., Palmer, R., Marsh, D.: Intel Usage-to-Platform Requirements Process. Intel Technology Journal 11(1), 23–34 (2007) 2. Bellotti, V., Ducheneaut, N., Howard, M., Smith, I.: Taking Email to Task: the Design and Evaluation of a Task Management Centered Email Tool. In: Proc. CHI 2003, pp. 345–352 (2003)
From Personal to Collaborative Information Management: A Design Science’s Approach
45
3. Bergman, O., Beyth-Marom, R., Nachmias, R.: The Project Fragmentation Problem in Personal Information Management. In: Proc. CHI 2006, pp. 271–274 (2006) 4. Blandford, A.E., Green, T.R.G.: Group and Individual Time Management Tools: What You Get Is Not What You Need. Personal and Ubiquitous Computing 5, 213–230 (2001) 5. Boardman, R., Sasse, M.A.: Stuff Goes into the Computer and Doesn’t Come OutA CrossTool Study of Personal Information Management. In: Proc. CHI 2004, pp. 583–590 (2004) 6. Chudoba, K., Wynn, E., Lu, M., Watson-Manheim, M.B.: How Virtual Are We? Measuring Virtuality in a Global Organization. Information Systems Journal 15(4), 279– 306 (2005) 7. Czerwinski, M., Horvitz, E., Wilhite, S.: A Diary Study of Task Switching and Interruptions. In: Proc. CHI 2004, pp. 175–182 (2004) 8. Ducheneaut, N., Bellotti, V.: E-Mail as Habitat: An Exploration of Embedded Personal Information Management. Interactions, 30-38 (September – October 2001) 9. Dumais, S., Cutrell, E., Cadiz, J.J., Jancke, G., Sarin, R., Robbins, D.C.: Stuff I’ve Seen: A System for Personal Information Retrieval and Re-Use. In: Proc. SIGIR 2003, pp. 72–79 (2003) 10. Gonzalez, V.M., Mark, G.: Constant, Constant, Multi-tasking Craziness: Managing Multiple Working Spheres. In: Proc CHI 2004, pp. 113–120 (2004) 11. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in Information Systems Research. MIS Quarterly 28(1), 75–105 (2004) 12. Iqbal, S.T., Horvitz, E.: Disruption and Recovery of Computing Tasks: Field Study, Analysis, and Directions. In: Proc. CHI 2007, pp. 677–686 (2007) 13. Karger, D.R., Jones, W.: Data Unification in Personal Information Management. Communication of the ACM 49(1), 77–82 (2006) 14. Kratkiewicz, G., Mitchell, G.: An Adaptive Semantic Approach to Personal Information Management. In: Proc. IEEE International Conference on Systems, Man and Cybernetics, pp. 1395–1400 (2004) 15. Lu, M., Watson-Manheim, M.B., Chudoba, K.M., Wynn, E.: Virtuality and Team Performance: Understanding the Impact of Variety of Practices. Journal of Global Information Technology Management 9(1), 4–23 (2006) 16. Mark, G., Gonzalez, V.M., Harris, J.: No Task Left Behind? Examining the Nature of Fragmented Work. In: Proc. CHI 2005, pp. 321–330 (2005) 17. Pickering, C., Wynn, E.: An Architecture and Business Process Framework for Global Team Collaboration. Intel Technology Journal 8(4), 373–382 (2004) 18. Whittaker, S., Sidner, C.: Email Overload: Exploring Personal Information Management of Email. In: Proc. CHI 1996, pp. 276–283 (1996) 19. Whittaker, S., Bellotti, V., Gwizdka, J.: Email in Personal Information Management. Communication of the ACM 49(1), 68–73 (2006)
A Classification Scheme for Characterizing Visual Mining Elaheh Mozaffari and Sudhir Mudur Concordia University, Computer Science & Software Engineering, 1515 St. Catherine St. West, Montreal, Quebec, Canada {e_mozafa,mudur}@cs.concordia.ca
Abstract. Visual mining refers to the cognitive process which integrates the human in analysis of information when using interactive visualization systems. This paper presents a classification scheme which provides user-centered representation of goals and actions that a user performs during the visual mining process. The classification scheme has been developed using contentanalysis of published literature containing precise descriptions of different visual mining tasks in multiple fields of study. There were two stages in the development. First, we defined all the sub-processes of visual mining process. Then we used these sub-processes as a template to develop the initial coding scheme prior to utilizing specific data from each of the publications. As analysis proceeded, additional codes were developed and the initial coding scheme was refined. The results of the analysis were represented in the form of a classification scheme of the visual mining process. The naturalistic methods recommended by Lincoln and Guba have been applied to ensure that the content analysis is credible, transferable, dependable and confirmable. Keywords: Visual mining, large dataset analysis, human information behaviour.
1 Introduction In today’s applications, data are becoming available in large quantities. Fields as diverse as bioinformatics, geophysics, astronomy, medicine, engineering, meteorology and particle physics are faced with the problems of making sense out of exponentially increasing volumes of available data [1]. Therefore, one of our greatest challenges is to take advantage of this flood of information and turn raw data into information that is more easy to grasp and analyze. Over the years, a large number of interactive visualization systems have been developed, all claiming to help users analyze, understand and gain insight into the large quantity of available data through appropriate transformations of the raw data into visual representations. We refer to the human analytical process that uses such visually represented information as being the Visual Mining (VM) process. It concerns the cognitive process which integrates the human factor during the course of mining and analyzing information through the visual medium. It contributes to the visual discovery of patterns which form the knowledge required for informed decision making. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 46–54, 2011. © Springer-Verlag Berlin Heidelberg 2011
A Classification Scheme for Characterizing Visual Mining
47
Purely from a technology perspective there are many studies which have focused on techniques and tools for building up visualization systems. Also, the importance of understanding users’ workflows and practices has been recognized by many researches [2-4] . Jim Gray points out that without integration of users’ workflows and interactions with the information, even the best system will fail to gain widespread use [5]. However, there are to date no reports on studies from the perspective of user behavior in visual mining of large data sets. To understand users of large datasets while performing visual mining, studies about users’ information behaviours are critical. Information behavior is defined as: “The totality of human behavior in relation to sources and channels of information, including both active and passive information-seeking, and information use” [6]. However, as previously mentioned, studies with specific focus on scientists’s information behaviour (how they look for required information and actually use them) in the visual mining process are rare. The study that is reported in this paper answers this call, and aims to improve our understanding of information behaviors of users activities during the process of visual mining of large datasets. The rest of the paper is organized as follows. In section 2 we review the related studies and identify the problem. In section 3 we describe and justify our choice for the methodology used for addressing this problem. Section 4 includes description and justification of our method of chosing visual mining case study samples and the analysis of user information behavior in visual mining in these samples. Results of this research are presented in section in section 5. Section 6 concludes this paper and discusses potential for future work.
2 Background Information behaviour has been the focus of many researches in the few last decades in the field of library and information science. The highlights of studies on information behaviour include Wilson's (1981) model of information-seeking behaviour [7] , Dervin's (1983) sense-making theory [8] , Ellis's (1989 and 1993) behavioural model of information seeking strategies [9,10] , Kuhlthau's (1991) model of the stages of information seeking behaviour [11], Belkin’s (1993) characterization scheme of information-seeking [12] and Wilson's (1997) problem solving model [13]. The studies presented above are however inadequate with regard to their suitability for representing user’s information behavior in VM process. These studies are cannot completely model user information behavior in visual mining. Their sole goal is to describe the information-seeking activity, the causes and consequences of that activity, or the relationships among stages in information-seeking behavior [14] . For example the model proposed by Belkin et al. represents dimensions of information seeking behaviors in information retrieval system. Information seeking is one of the sub-process of VM process (as we shall explain in more details in section 3) therefore it can not completely describe VM process. In addition, the studies done with library patrons focus on the user’s tasks that are perhaps learned behaviors due to their prior knowledge of how libraries work. They tend to ask questions that they know can be answered. Visualizations might support a different way of asking questions and getting answers [15].
48
E. Mozaffari and S. Mudur
This paper presents a study that began with the aim of extending our understanding of the interdisciplinary process of visual mining, and in doing so looked to strengthen and improve our understanding of user’s information behavior in the visual mining process. The study has yielded a classification scheme for characterizing visual mining. Such a classification scheme has many different applications: support requirement analysis in system engineering of interactive visualization software, studying and assessing different tasks which typically occur in visual mining, improving the functionality and interface design of newer interactive visualization systems and by providing a system-independent taxonomy of tasks, it can be used for evaluating and classifying existing interactive visual mining systems based on what they support.
3 The Sub-processes of Visual Mining Today, many different groups around the world are undertaking research on visualization for data mining and analysis in order to effectively understand and make use of the vast amount of data being produced. Different terms are used to describe this process, visual data mining [16, 17], visual exploration [18] and visual analytics [19] to name but a few. In addition, there appears to be some variation in understanding that people have of the process even under the same term. Niggemann [20] defined visual data mining as visual representation of data close to the mental model. Ankerst [21] considered visual data mining as a step in the knowledge discovery process which utilizes visualization as a communication channel between the computer and the user. Visual analytics is defined as an iterative process that involves information gathering, data preprocessing, knowledge representation, interaction and decision-making. In order to gain further insight, it integrates methods from knowledge discovery in databases, statistics and mathematics, together with human capabilities of perception and cognition [22] . From these definitions, one common theme that can be recognized is that they all rely on the human visual channel and take advantage of human cognition. They also emphasize on the three aspects: task, visualization and process. From the above approaches it may be noted that in VM, data visualization takes place either before or after a data mining algorithm, or perhaps even independently. For the purposes of our research, however, we will focus on the human involvement in the visual data exploration process which utilizes human capabilities to perceive, relate and conclude. We consider VM as a hypothetical formation process that primarily uses the visual medium. Such visualization allows the user to interact directly with visually represented aspects of the data, analyze, gain insight and perhaps even formulate a new hypothesis. Later on, the user can evaluate the best possible hypotheses and make a judgment based upon it. In fact, this visual information exploration process helps to form the knowledge for informed decision making. In order to identify analysis tasks and human information interactions in the VM process, we first look at how analysis works and then extend its sub-processes to the context of VM. The analytical process itself is both structured and disciplined. Usually analysts are asked to perform several different types of tasks such as assessing, forecasting and developing options. Assessing requires the analyst to
A Classification Scheme for Characterizing Visual Mining
49
describe their understanding of the present world around them and explain the past. Forecasting requires that they estimate future capabilities, threats, vulnerabilities, and opportunities. Finally, developing options in order to establish different optional reactions to potential events and assess their effectiveness and implications. The process begins with planning. The analyst must determine how to address the issue, what resources to use, and how to allocate time to various parts of the process. Then, they must gather relevant information and evidence in order to relate them to their existing knowledge. Next, they are required to generate multiple candidate explanations in the form of hypotheses, based on the evidence and assumptions, in order to reach a judgment about which hypothesis can be considered to be the most likely. Once conclusions have been reached, the analyst broadens their way of thinking to include other explanations that were not previously considered and provide a summary of the judgments they had made [19]. Building upon the above description of the analytical process, we defined the subprocesses of VM as follows: 1. The user initiates the VM process by planning how to address the issue, what resources to use and how to allocate time to various parts of the process to meet deadlines. The next step is to gather all relevant information by seeking information through searching, browsing, monitoring and generally being aware [23]. 2. Searching refers to active attempts to answer questions, look for a specific item or develop understanding around a question or topic area. Browsing is an active yet undirected form of behavior. For example, when performing physical acts such as 3D navigation tasks or scrolling/panning, the user has no special information need or interest, but becomes actively exposed to possible new information. Monitoring is a direct and passive behavior. The user does not feel such a pressing need to engage in an active effort to gather any information but may be motivated to take note of any expected or unexpected information. Also, when the user has a question in mind, and may not be specifically acting to find an answer, they would take note of any relevant information that appears. Being aware is a passive undirected behavior and is similar to browsing except that the user could locate information or data unexpectedly. 3. The next step in the VM process is to relate the findings with the knowledge that is hidden in the expert's mind. 4. Based on the findings, the user then generates multiple candidate explanations in the form of hypotheses. By applying analytical reasoning the user can use their prerogative to either confirm or reject any hypothesis and formulate a judgment about which is the most relevant. 5. Once conclusions have been reached, the user will be engaged in the act of broadening their thinking to include other possible explanations that were not previously considered. Then the user summarizes analytical judgment either as assessment, estimation or evaluation of options depending on the goal. 6. As the concluding step, the user usually creates a product to include the analytical judgment in the form of reports, presentations or whatever other form of communication is deemed appropriate.
50
E. Mozaffari and S. Mudur
4 Method In this section, we describe our methodology for characterizing visual mining process. But first, we provide the justification for choice of our methodology. Surveys and interviews are the most common research methods for studying users’ information behavior [24]. As we know, in a typical user study or survey, user’s motivation, knowledge and expertise considerably influence user performance and thus the final conclusions. Of course, using domain experts provides more realistic results [25]. However, it is not easy to employ enough participants (domain experts) for interviews and surveys in this type of study, nor is it possible to have access to them for any extended period of time because most of the experts are distributed across different external institutions. Therefore we turned to scientific publications which, in general, clearly record the behavior of experts while being engaged in the visual mining process and equally importantly are also peer reviewed. We adopted the qualitative direct-content analysis approach [26], to reveal the visual mining behavior of scientists from such publications. Qualitative content analysis is an unobtrusive method which uses nonliving form of data, generally categorized as texts. And it is well established that one kind of text that can be used for qualitative data inquiry in content analysis is official publications [27, 28]. The advantages of working with prior published works are: 1. The data are stable and non-changing, 2. The data exists independent of research. This is because the data is not influenced through researcher’s interaction as is the case with interviews. They already exist in the world regardless of the research currently being done [29] and 3. They provide information about procedures and past decisions that could not be observed by researchers [27]. We obtained the information on work practices through analysis of end-results of researches as they were described in published scientific literature. The naturalistic methods recommended by Lincoln and Guba were applied to ensure that the content analysis was (to the extent possible) credible, transferable, dependable and confirmable.
5 Case Study Samples For our content we initially chose sixty one published papers primarily concerned with reports on effective use of visualization for analysis and mining of large datasets. The chosen papers were from four different domains, namely, medicine, bioinformatics, epidemiology and geoscience. Each paper was studied and those which did not report actual case studies by experts were excluded from further consideration. The final numbers of papers which contained cases studies that described interaction with visual information in each domain are given in Table 1. Every one of these papers was analyzed and used in the information interaction coding process described next.
A Classification Scheme for Characterizing Visual Mining
51
Table 1. Numbers of papers analyzed in each domain for qualitative content analysis
6
Context
Number of papers
Bioinformatics
8
Medical
9
Geoscience
11
Epidemiology
3
Analysis of Case Studies
The Sub-Processes of Visual Mining explained in section 3 were used to develop the initial coding scheme prior to data analysis [26]. As analysis proceeded, additional codes were developed and the initial coding scheme was revised and refined. Coding of the data took place in multiple iterations. (1) Initial coding of each paper began with manual annotation of paper by reading case studies line by line, to highlight each relevant concept of human interaction and label it. Subsequent iterations of reading and coding of each paper in a constant comparison with previous paper and coding allowed emergence of categories and themes. We used NVivo 9 software that helps work with unstructured information such as documents, surveys, audio, video and pictures in order to assist in better decision-making [30]. NVivo 9 allowed us to code relevant concepts of VM in the articles and assign them to nodes which can be as hierarchical (tree nodes) or un-hierarchical (free nodes) as required. The relevant concepts of visualization were first coded as free nodes. Then, after coding a few articles and comparing them with previous ones, were either modified to tree nodes, renamed or deleted as required. Coding with NVivo 9 was convenient since it allowed adding, renaming, deleting or merging of codes as required but it did not, however, automate the coding process. (2) The consistent coding was addressed by including several iterations of coding around a period of a year. (3) Peer debriefing technique was used to confirm the interpretations and coding decisions. Peerdebriefer, a disinterested observer, analyzed the research materials and questioned the data, meaning, and interpretation. She was a colleague and had a PhD in computer Science, was not involved in the study. She had knowledge about qualitative research and phenomenon under investigation. The interactions between researcher and the peer-debriefer also included in the audit trail. She also acted as the auditor. (4) The coding changes were maintained by creating static models in NVivo 9 for future reference. In addition ideas, discussions, interpretations and decisions were recorded in the memos in NVivo 9 to keep tracking of the development of analysis. These allowed an audit trail to be maintained. (5) An external auditor examined the audit trail. (6) The dynamic models illustrating code relationships were used to visualize explore and present connections and patterns in the data. (7) At the end, member checking which is most important action in a naturalistic inquiry [31] was conducted to test the result of analysis with a geographers and a research fellow in biomedical engineering. They confirmed the results and verified the interpretations.
52
E. Mozaffari and S. Mudur
7 Results The above mentioned process led us to formulate a set of criteria which characterizes the VM process. Table 2 presents these criteria and their possible values as the task model for visual mining. Table 2. Classification Scheme of VM Criteria
Values
Goal
assess, estimate, develop options
Information seeking
searching, browsing, monitoring, being aware
Retrieval
pattern, hypothesis, judgment
In the resulting classification scheme, the user’s goal of visual mining requires an understanding of the current situation and explaining the past (assess), estimating future capabilities (estimate) and developing different possible options (develop options). In order to accomplish these goals, the user must gather relevant information and evidence through active or passive information-seeking activities which, as already described, are classified as searching, browsing, monitoring and being aware. The retrieved item(s) during these activities can be a pattern, hypothesis or final analytical judgment. Finally, in order to further validate the classification scheme, typical real-world visual mining tasks were extracted and listed from the reviewed literature. All extracted tasks were re-described using the VM classification scheme in order to validate the model. Finally to ensure that further refinement is not needed, visual interaction tasks were extracted from ten new papers all containing reports of visualization case studies. All these tasks were comprehensively described by the VM task model. This process was repeated again with an additional five papers. Since no changes were required in the classification scheme, we concluded that our final classification scheme was stable and no further refinements were needed.
8 Conclusion and Future Work To understand users of large datasets while performing VM, studies about users’ information behaviours are critical. However, studies that focus on scientists’s information behaviour in the visual mining process are rare. To his end, this paper has presented a summary of a study concerned with human interactions with visually represented information which aimed to improve our understanding of information behaviors of users of large datasets. By carrying out a trustworthy qualitative content analysis procedure using published papers reporting visual information interaction tasks, we have derived that user behaviours in this context can be differentiated along
A Classification Scheme for Characterizing Visual Mining
53
a small set of three criteria. These three criteria were represented in the form of a classification scheme of the visual mining process. This classification scheme allows to describe real world visual mining tasks which play an important role in analysis of large datasets. In our future work we plan to use these criteria in modelling user behavior through behavioral strategies, validating these strategies against known case studies in different domains and applying it in comparative evaluation of visualization systems and in the design of newer systems.
References 1. Mann, B., Williams, R., Atkinson, M., Brodlie, K., Williams, C.: Scientific Data Mining, Integration and Visualization. In: Integration, and Visualization Report of the workshop held at the e-Science Institute (2002), http://www.nesc.ac.uk/talks/sdmiv/report.pdf 2. Atkinson, M., De Roure, D.: Data-intensive Reseach: Making best use of research data. eScience Institute (2009) 3. Van de Sompel, H., Lagoze, C.: All Aboard: Toward a Machine-Friendly Scholarly Communication System. In: Hey, A.J.G., Tansley, S., Tolle, K. (eds.) The Fourth Paradigm: Data-intensive Scientific Discovery, Microsoft Research: Redmong, pp. 193– 199 (2009) 4. Borgman, C.L.: Scholarship in the Digital Age: Information, Infrastructure, and the Internet. MIT Press, Cambridge, MA (2007) 5. Gray, J.: Scientific Data Management in the Coming Decade. SIGMOD 34(4), 34–41 (2005) 6. Wilson, T.D.: Human information behavior. Informing Science 3, 49–55 (2000) 7. Wilson, T.D.: On user studies and information needs. Journal of Documentation 37(1), 3– 15 (1981) 8. Dervin, B.: An overview of sense-making research: concepts, methods and results to date. International Communications Association Annual Meeting, Dallas, Texas (1983) 9. Ellis, D.: A behavioural approach to information retrieval design. Journal of Documentation 46, 318–338 (1989) 10. Ellis, D., Cox, D., Hall, K.: A comparison of the information seeking patterns of researchers in the physical and social sciences. Journal of Documentation 49, 356–369 (1993) 11. Kuhlthau, C.C.: Inside the search process: information seeking from the user’s perspective. Journal of the American Society for Information Science 42, 361–371 (1991) 12. Belkin, N.J., Marchetti, P.G., Cool, C.: Braque: Design of an Interface to Support User Interaction in Information Retrieval. Information Processing and Management 29, 325– 344 (1993) 13. Wilson, P.: Information behavior: An inter-disciplinary perspective. In: Vakkari, P., Savolainen, R., Dervin, B. (eds.) Information Seeking in Context, pp. 39–50. Taylor Graham, London (1997) 14. Wilson, T.D.: Models in information behaviour research. Journal of Documentation, 55, 249–270 (1999) 15. Morse, E. L.: Evaluation of Visual Information Browsing Displays. PhD Thesis, University of Pittsburgh (1999)
54
E. Mozaffari and S. Mudur
16. Keim, D.A.: Information Visualization and Visual Data Mining. IEEE Transaction on Visualization and Computer Graphics 8, 1–8 (2002) 17. Simoff, S.J., Michael, H., Böhlen, M.H., Mazeika, A.: Visual Data Mining - Theory, Techniques and Tools for Visual Analytics. Springer, Heidelberg (2008) 18. Tominski, C. Event-Based Visualization for User-Centered Visual Analysis. Ph.D. thesis, University of Rostock, Rostock, Germany (2006) 19. Thomas, J.J., Cook, K.A.: Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE press, New York (2005) 20. Niggemann, O.: Visual Data Mining of Graph-Based Data. Ph.D. Thesis, University of Paderborn (2001) 21. Ankerst, M.: Visual Data Mining. Dissertation (Ph.D. thesis). Faculty of Mathematics and Computer Science, University of Munich (2000) 22. Keim, D.A., Mansmann, F., Schneidewind, J., Thomas, J., Ziegler, H.: Visual Analytics: Scope and Challenges. LNCS. Springer, Heidelberg (2008) 23. Bates, M.J.: Toward an Integrated Model of Information Seeking and Searching. In: Fourth international Conference on Information Needs, Seeking and Use in Different Contexts, vol. 3, pp. 1–15 (2002) 24. McKechine, L.E.F., Baker, L., Greenwood, M., Julien, H.: Research method trends in human information literature. New Review of Information Behaviour Research, 3, 113– 125 (2002) 25. Plaisant, C.: The challenge of information visualization evaluation. In: Proc. of the Conference on Advanced Visual Interfaces (AVI). ACM, NY (2004) 26. Kyngas, H., Vanhanen, L.: Content analysis (Finnish). Hoitotiede 11, 3–12 (1999) 27. Patton, M.Q.: Qualitative research and evaluation methods, 3rd edn. Sage Publications, Thousand Oaks (2002) 28. Bhowmick, T., Griffin, A.L., MacEachren, A.M., Kluhsmann, B., Lengerich, E.: Informing Geospatial Toolset Design: Understanding the Process of Cancer Data Exploration and Analysis. Health & Place 14, 576–607 (2008) 29. Hesse-Biber, S.N., Leavy, P.: The practice of qualitative research. Sage publications, Thousand Oaks (2006) 30. QSR International, http://www.qsrinternational.com/ news_whats-new_detail.aspx?view=367 31. Lincoln, Y.S., Guba, E.G.: Naturalistic inquiry. Sage Publications, Inc., Beverly Hills (1985)
Transforming a Standard Lecture into a Hybrid Learning Scenario Hans-Martin Pohl1, Jan-Torsten Milde2, and Jan Lingelbach2 1 University of Applied Sciences Heinrich-von-Bibra-Platz 1b, 36037 Fulda, Germany 2 University of Applied Sciences Marquardstrasse 35, 36039 Fulda, Germany
[email protected], {jan-torsten.milde,Jan.lingelbach}@informatik.hs-fuda.de
Abstract. We describe the successful transformation of a traditional learning setting of a standard lecture into a hybrid learning arrangement. Based on the 3C-Modell of Kerres and de Witt, the lecture has been extended to integrate exercises. Students are motivated to work in smaller groups. In order to allow students to work according their own work speed and motivation, the learning material is distributed using the e-Learning platform. This material includes video recordings of the lecture in a high quality. The evaluation of this transformation process shows evidence, those students take profit from the extended hybrid learning arrangement. Keywords: Hybrid learning scenario, interactive slide presentation, blended learning.
1 From Standard Lecture to a Hybrid Learning Scenario E-Leaning has been very successful over the last years. This is especially true in the context of higher education, where many universities have set up e-Learning systems to support their teachers and students. While the systems provide a rich set of possibilities to the teachers, it can be seen, that many (if not most) of them do not integrate e-Learning into their teaching settings. At this point in time the didactics of e-Learning scenarios needs to get more attention. Using e-Learning or blended learning as a central part of the general teaching methodology has substantial effect onto the design of both teaching and learning style. In this paper we would like to focus onto the transformation problems and outcomes, that arise when using new e-Learning based teaching methods and integrating new media into higher education teaching (see [Ker05]). The transformation process leads to a more open learning setting that we expect to lead to a higher learning motivation and better learning outcomes. (see [Sch04], [Nie04], [Sch09]) We will provide observations how the didactics have been influenced by each of the transformation steps taken (see [Ojs06], [Car02], [Bro89], [Lav91]). G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 55–61, 2011. © Springer-Verlag Berlin Heidelberg 2011
56
H.-M. Pohl, J.-T. Milde, and J. Lingelbach
In our example a traditional introductory course on bachelor level had been transformed into a blended learning arrangement. The course “Introduction to Electronical Engineering” of the winter term 2009/2010 is a first years course (see [AISOP]). It is structured into a lecture and additional exercises, which were taught separately from the lecture. The students in this introductory course have a very inhomogeneous background. Especially their knowledge in mathematics and physics vary substantially. This leads to a number of problems. While it was possible for most of the students to keep up with the content of the lecture, many had problems when it came to doing exercises, which in turn formed the basis for the final written exam. Some students also had problems with the complexity of the lecture and were unable to structure the content to their needs, making it hard for them to solve the exercises. 1.1 The 3C-Modell as a Didactical Framework for the Design of Blended Learning For most blended learning arrangements lectures or seminars are combined with the application of new media. This technical support is often based strongly to quite available learning-management-systems (LMS). The combination is obtained in the optimum case, the best of both worlds. On the one hand there is direct contact between teachers and students with the opportunity to interact directly in focus. On the other side the rather learner-centered view is stressed. By the offer of electronic materials the self-determination of the students increases. They get the possibility to themselves to choose her learning paths and to determine the learning speed independently. [Dem07] About test scenarios and learning success excrement roles the teacher can intervene steering and accompanying. Hence, it corresponds to often demand "shift from teaching to learning". However, the main question with Blended Learning attempts is always after the right mixture of on-line study and presence study. Also the variety of the electronic support must be used carefully. The pure supply of digital materials is a really successful didactic conversion of a draught in the rarest cases. [Bac01] For Driscoll [Dri02] blended learning scenarios are a good entrance in the learning with new media. Besides, in particular must be taken on them, mostly still to be developed, media competence of the learning consideration. The arrangement of these teaching arrangements is always a question to the concept of the teacher. So Kerres und de Witt wrote: “In our interpretation, blended learning basically refers to (at least) the mix of different: − didactical methods (expository presentations, discovery learning, cooperative learning, …) and − delivery formats (personal communication, publishing, broadcasting, …)” [Ker03] The 3C-model by Kerres and de Witt [Ker03] underlying to this article creates exactly for these questions a framework to describe the components of the learning arrangement and their weighting and to determine the suitable methods and formats in a second step.
Transforming a Standard Lecture into a Hybrid Learning Scenario
57
It is based on three components: content, construction and communication. Content is all material the learner should be able to recall. All types of media, e.g. literature, scripts, slides, pictures, animations, podcasts, video podcasts, lecture recordings, are possible. Thereby the content is necessary to allow the learner to internalize the externalized knowledge and construct its own knowledge. The construction component is necessary when the information from the material should be available for actions. The learner must reflect the learned facts and fit it in the whole issue. For an integrated learning arrangement the communication component is essential to e.g. discuss learning results with other students, to reflect the facts with the practical reuse in contact with the teacher or got a different view by interaction with others. All three components could be realised in a number of different media types including synchronous and asynchronous online presentations. In a traditional teaching setting, content will be presented by the teacher (e.g. as a slide presentation), in a non-communicative setting (teacher is presenting, students are listening) with a very low fraction of construction time (students will repeat content for themselves after the presentation). In our transformation process we attempted to change this situation to make the teaching situation more communicative and constructive. The slide presentations in the lecture have been changed to a more interactive way of presentation to support the content component. Exercises have been integrated into the lecture, thus closing the temporal gap between content presentation and practical appliance to enhance the construction component. The complete lecture has been video recorded and is provided as an online video stream to the students. In order to support the students with their exercises, sample solutions have been worked out and have also been video recorded. The complete learning material of the lecture is located on the central e-Learning platform of the university and can be used by students asynchronously to the lectures and exercises. Furthermore communication tools were provided there to promote the communication component.
2 Interactive Presentation The interactive slides presentation has been realised using the “Oxford Paper Show” system (see [papershow]). This system allows to couple traditional slide projection with a computer based presentation. The underlying technology is based on a special paper with extremely tiny control marks printed onto it. These marks are recorded and processed by a camera, which is integrated into a special pen, which in turn is wirelessly connected to the computer using Bluetooth. In our arrangement we copied all slides onto the paper and created scans of these slides. A physical version of the slides was given to the teacher and a virtual version of the slides existed on the computer. As an effect of this technology, teachers do not have to change their style of presentation. The slides can be selected with the digital pen and will be projected by the computer. It is possible to write onto the (paper) slides. These comments will be shown on the virtual slides in real time. Every slides transaction is therefore captured
58
H.-M. Pohl, J.-T. Milde, and J. Lingelbach
and can be recorded. This allows an asynchronous playback of the slide presentation. In addition, all comments on the slides will be stored and can later be distributed electronically. A noticeable effect of the integration of this new technology into teaching is the calm presentation style. The teacher explains the content to the student and simultaneously writes down comments. As such, he remains seated and is not moving across the class room. The students seemed to be much more focused and kept listening to the teacher. Instead of writing down everything the teacher explained, more individual notes were taken.
3 Integration of Exercises In order to consolidate the lecture's content, exercises were integrated into the lecture. The exercises were given to the students at the end of each lecture. After 20 Minutes a sample solution was presented by the teacher using the paper show technology. Further exercises were given, which should be solved as part of the self-study phase of the students. If requested, the solution for this exercises were presented in the following lecture. Integrating the exercises resulted in a deeper understanding of the lecture's content. Students took part in the lecture more actively and misunderstandings were identified immediately. Another effect of the exercise integration is the speed reduction of the lecture. The teacher focused on the most relevant topics, thus reducing the complexity of the lecture. The students profited from the interactive presentation style of the sample solution. The exercises are relatively complex and the stepwise explanation helped them to comprehend the solutions. If needed, the presentation could be replayed multiple times. To interact and communicate with other students outside the lectures the students uses very often asynchronous methods like threated discussion. It was interesting to see that for this interaction not the standard learning management system was used. Instead a server outside of the university supported by student representatives of the department Applied Computer Science.
4 Video Recordings of the Lectures In order to facilitate an intensive self-study phase, all lectures have been video recorded and were put onto the e-Learning platform for individual downloading (see [AISOP]). In addition a streaming server was set up. This made it possible to watch the lecture even on a slow bandwidth connection. It only takes about two hours of post processing (mainly automatic transcoding) until the lectures are available to the students. For the recording of the lectures a mobile recording system had been designed and set up by the central e-Learning laboratory of the university. The system has been built around a “Sony AnyCast Station”, a mobile video recording system, allowing to recording and mixing down up to six video channels. We attached two cameras to the
Transforming a Standard Lecture into a Hybrid Learning Scenario
59
system. These cameras are fully remote controllable. Using two cameras provided us with a higher flexibility for the art work of the recording, resulting in a more “interesting” video providing a higher level of immersion. In order to capture the signal of the presentation computer, a splitter had been attached to the system, thus making it possible to both show the signal on the data projector and to record it in a high quality. The sound has been recorded using wireless microphones. The recording, video mixing and camera control is done in real time by a single person. We trained a couple of student tutors, who are now able to record lectures on their own. As the recording system is designed to be mobile, it can be used in any standard lecture class. Setting up the system takes about 15 minutes for two people. The students used the video recordings very intensively. The video were watched almost around the clock. Quite often only smaller parts were selected and have been watched multiple times. The recordings also had positive effects onto the lectures. Students explained that it was comforting to them, that they could repeat topics, even when they did not understand the topic during lecture time. The attendance rate did not drop, so students still wanted to take part in the “live” lecture. A similar effect was noticed by the teacher. It took him little time to get used to the recording situation. Knowing that the content was available for replay, he was able to refer to the video, if needed. That reduced the need to repeat things during the lecture, leaving more time for the exercises.
5 Presentation on the e-Learning Platform Additional material to the video recordings and the slide presentation was put onto the central e-Learning platform of the university. This material included the texts of the exercises and a large number of exercises of past years. The students rarely used this additional material and rather stuck to the content which was of immediate interest to them. The e-Learning platform provided means of communication, such as forums and chats. These were used by the students for organisational purposes only. A content related discussion did not take place. The e-Learning platform formed the basis for the self-paced study style of the students. In earlier years students tended to learn “on demand”, starting to learn about two weeks before the final exam was due. With the platform online and the material available, the students started to work in a more continuous way.
6 Conclusions At the end of the term, the students were asked to fill out a questionnaire. A total number of 78 persons attended the course. Of these 55 persons took part in the evaluation. The questionnaire is standardised which makes the automatic processing of the data possible. The evaluation is archived for documentation purposes. Therefore long term comparison becomes possible and will be performed during the next years.
60
H.-M. Pohl, J.-T. Milde, and J. Lingelbach
The analysis of the evaluation showed a high acceptance rate of the used methodologies and technologies. More than 55% of the participants took advantage of the new media. Almost 70% observed positive learning effects when using the elearning material for their course preparation and course repetition (see Fig. 1).
Fig. 1. Evaluation results
Switching from a standard lecture to a hybrid learning scenario has resulted in a number of positive effects. Most of the participants explained, that e-Learning played an important role in their learning success. The students liked the online support and would like to see it extended in the future. As shown in figure 2 especially online exercises and audio and video recordings were requested. This is a very positive outcome, as the self-activation of the students was one of the central targets of this transformation.
Fig. 2. Requested forms of online support
Offering an open learning situation to the students leads to a more intensive learning experience and results in a deeper understanding of the content. The integration of exercises into the lectures provided a transfer to the practical appliance of the theoretical content. The used technologies allowed to preserving a classical teaching/presentation style, while transforming the material into the digital world. The teacher was able to focus onto the lecture and was not distracted by technological problems.
Transforming a Standard Lecture into a Hybrid Learning Scenario
61
References [Ker05] Kerres, M.: Didaktisches Design und E-Learning. In: Miller, D. (Hrsg.) E-Learning Eine multiperspektivische Standortbestimmung, pp. 156–182. Haupt Verlag (2005) [Sch04] Schulmeister, R.: Didaktisches Design aus hochschuldidaktischer Sicht - Ein Plädoyer für offene Lernsituationen, Stand: (March 4, 2010), http://www.zhw.uni-hamburg.de/pdfs/Didaktisches_Design.pdf [Sch09] Schulmeister, R.: Studierende, Internet, E-Learning und Web 2.0. In: Apostolopoulus, N., et al. (Hrsg.) E-Learning 2009 - Lernen im digitalen Zeitalter, pp. 129–140. Waxmann (2009) [Ojs06] Ojstersek, N., Heller, I., Kerres, M.: E-Tutoring. Zur Organisation von Betreuung beim E-Learning. In: Arnold, R., Lermen, M. (Hrsg.) eLearning-Didaktik, pp. 107–116. Schneider Verlag, Hohengeren (2006) [Nie04] Niegemann, H.M., et al.: Kompendium E-Learning. Springer, Heidelberg (2004) [papershow] http://www.papershow.com/de/index.asp (July 5, 2010) [AISOP] E-Learning Plattform der Hochschule Fulda (July 5, 2010), http://elearning.hs-fulda.de/aisop/ [Bro89] Brown, J.S., Collins, A., Duguid, P.: Situated Cognition and the Culture of Learning. Educaltion Researcher 18, 32–42 (1989) [Lav91] Lave, J., Wenger, F.: Situated Learning: Legitimate Peripheral Participation. Cambridge Press, New York (1991) [Car02] Carman, J.M.: Blended Learning Design: Five Key Ingredients (July 5, 2010), http://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.95.3197&rep=rep1&type=pdf [Pel01] Pellegrino, J.W. (Hrsg.): Knowing What Students Know: The Science and Design of Educational Assessment. National Academy Press, Washington, D.C (2001), ISBN 9780309072724 [Bac01] Bachmann, G., Dittler, M., Lehmann, T., Glatz, D., Rösel, F.: Das Internetportal LearnTechNet der Uni Basel: Ein Online Supportsystem für Hochschuldozierende im Rahmen der Integration von E-Learning in der Präsenzuniversität. In: Haefeli, O., Bachmann, G., Kindt, M. (Hrsg.) Campus 2002 – Die Virtuelle Hochschule in der Konsolidierungsphase. Münster, pp. 87–97. Waxmann (2002), ISBN 978-3830911913 [Ate04] Attewell, J., Savill-Smith, C. (Hrsg.): learning with mobile devices – research and development. Published by the Learning and Skills Development Agency (2004), ISBN 1 85338 833 5 [Dem07] Demetriadis, S., Pombortsis, A.: E-Lectures for Flexible Learning: a Study on their Learning Efficiency. Educational Technology & Society 10(2), 147–157 (2007) [Foe02] Foertsch, J., Moses, G., Strikwerda, J., Litzkow, M.: Reversing the lecture/homework paradigm using eTEACH web–based streaming video software. Journal of Engineering Education 91(3), 267–274 (2002) [Wie10] Wiethäuper, H.: E-Learning: Integration von Mediendidaktik und Lerntechnologie in Bildungsprozesse (2010), http://www.uni-marburg.de/fb21/sportwiss/mitarbeiter_seiten/ wiethaeuper/elearning/zw_medien_technologie.pdf, Stand: (January 14, 2011) [Dri02] Driscoll, M.: Blended Learning: Let’s Get Beyond the Hype, Stand: (January 20, 2011), http://www-07.ibm.com/services/pdf/blended_learning.pdf
Designing Web Sites and Interfaces to Optimize Successful User Interactions: Symposium Overview Robert W. Proctor1 and Kim-Phuong L. Vu2 2
1 Department of Psychological Sciences, Purdue Univerity, West Lafayette, IN, USA Department of Psychology, California State Univerity Long Beach, Long Beach, CA, USA
[email protected],
[email protected]
Abstract. Since the Web became widely available in the mid 1990s, it has come to be used by a range of people for many purposes. Effective user interactions are required for a Web site or product to accomplish its intended goals. Given the user-intensive nature of the Web and the many usability issues associated with performing tasks on the Web and with mobile devices, it is important for designers and researchers to understand issues that relate to how to optimize interfaces for the Web design and other systems involving humancomputer interaction. This symposium is devoted to issues involved in the design of Web sites and interfaces to promote successful user interactions. Keywords: Information Display, Input Devices, Mobile Devices, Organization of Information, Web Design.
1 Introduction Since the World Wide Web became widely available in the mid 1990s, it has come to be used by a variety of people for many different purposes, including e-commerce, social networking, data display, information sharing and collaboration, and mobile transactions. Effective user interactions are required for a Web site or product to accomplish its intended goals. Given the user-intensive nature of the Web and the numerous usability issues associated with performing tasks on the Web and with mobile devices, designers and researchers need to understand issues relating to Web design and Web usability. There is often a tendency to pit academicians against practitioners, basic research against applied research, and theoretical knowledge against experiential knowledge. Yet, from our experience, we have found that an approach that emphasizes multiple perspectives and multiple methods is most beneficial for acquiring knowledge and advancing technology [1]. Communication among individuals with various backgrounds, interests, and training is essential for facilitating the development and transfer of knowledge between researchers and practitioners in the domain of human–computer interaction (HCI), among other applied domains. Because the most complete understanding of usability problems arises from combining the insights of practitioners and in industry and government with the knowledge of academicians gained from controlled research, we consistently strive to encourage interaction among experts from the different communities. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 62–65, 2011. © Springer-Verlag Berlin Heidelberg 2011
Designing Web Sites and Interfaces to Optimize Successful User Interactions
63
To that end, in 2005, we edited the Handbook of Human Factors in Web Design [2], in which we articulated the general view described above specifically for Web design, stating: It is our opinion that the handbook should reflect the varied backgrounds and interests of individuals involved in all aspects of human factors and Web design. Consequently, we have made a concerted effort to obtain contributions from a diverse group of researchers and practitioners. The contributors are an international ensemble of individuals from academia, industry, and research institutes. Furthermore, the contributors have expertise in a variety of disciplines. We think that awareness of the wide range of views and concerns across the field is essential for usability specialists and Web designers, as well as for researchers investigating theoretical and applied problems concerning Web use and design. (p. xv) Consistent with this view, the book covered a full range of topics relevant to human factors in Web design, from historical developments and human factors principles in general to specific human factors applications in Web design. These content areas included content preparation for Web sites, search engines and interface agents, issues of universal accessibility, usability engineering, Web applications in academic and industrial settings, information security, and emerging technologies such as wireless communication and e-health. That same year, we also organized a symposium for HCII 2005, “Human Factors Considerations in Web and Application Design”, which highlighted many of the then current issues concerning usability in Web design. Papers presented in the symposium included: “Web-based Presentation of Information: The Top Ten Mistakes and Why They Are Mistakes” [3], “User Search Strategy in Web Searching” [4], “Cross Cultural Web Design” [5], “Understanding Online Consumers: A More Complete Picture” [6], “Web-based Programs and Applications for Business” [7], and “User Interface Design Guidelines” [8]. All of the topics covered in the 2005 handbook and symposium are still relevant today, but technological and societal developments have fueled many changes in Web use since that time. Social networking was in its infancy, mobile computing was being developed, and 4G wireless was not available. Rapid developments in these areas, as well as others, have greatly changed the computing landscape, and all of the developments involve new human factors issues associated with use of the technology. To capture these developments, we have edited a second edition of the handbook, which was recently published [9]. Because most of the topics that were relevant in 2005 continue to be relevant today, many chapters of the second edition provide updated information on those topics. New chapters are devoted to topics that have emerged as important since the first edition. They include: Organization of Information for Concept Sharing and Web Collaboration, Web-Based Organization Models, Web Portals, Human Factors of Online Games, Accessibility Guidelines and the ISO Standards, Use of Avatars, and Mobile Interface Design for M-Commerce. Much as the 2005 symposium was intended as an adjunct to the first edition of the handbook, this 2011 symposium is intended to be a companion to the recently published second edition, highlighting a subset of usability topics of interest to
64
R.W. Proctor and K.-P.L. Vu
designers and researchers interested in the Web and HCI. Fitting with our emphasis on communication among academicians and practitioners, the papers represent contributions from persons working in industry and academia.
2 Overview of Symposium This symposium is devoted to issues involved in organization and display of information for HCI in general and Web design in particular. The first two papers address user interactions on e-commerce sites. Najjar presents “Advances in Ecommerce User Interface Design,” in which he emphasizes that interface design is not a static field but a dynamic one in which possible interface features to incorporate into designs are continually changing. He describes new user interface features of which ecommerce designers may want to take advantage when designing Web sites. Examples are provided for many features, including social media connections, automated product recommendations, contextual product visualization, flash sales, and mobile commerce. When performing e-commerce transactions, personal information is transmitted through the Web. The topic of privacy is addressed by Nguyen and Vu in their paper, “Does Privacy Information Influence Users’ Online Purchasing Behavior?” In their study, users indicated whether they would make a purchase of an inexpensive or expensive item at different Web sites. Privacy information was made salient by a search engine called PrivacyFinder, but this information did not appear to influence users’ purchasing behaviors. Pappas and Whitman, in their paper “Riding the Technology Wave: Effective Dashboard Data Visualization,” discuss how to optimize data dashboard displays, identifying the types of data that are best represented in particular formats and techniques for displaying multiple visualizations. They emphasize that the choice of what to display and how to display it depends on the needs of the particular user. In “A Conceptual Model of Using Axiomatic Evaluation Method for Usability of the Consumer Electronics,” Guo, Proctor, and Salvendy describe concepts from Axiomatic Design theory that are based in information theory, and they discuss how it can be applied as a usability evaluation method for mobile consumer electronics. An experiment is described in which participants identified more usability problems associated with a cell phone when employing axiomatic evaluation than when using a more traditional usability method. Mobile devices are also of concern to Xu and Bradburn, whose paper, “Usability Issues in Introducing Capacitive Interaction into Mobile Navigation,” focuses on user interactions with mobile devices. They present an experiment that evaluates the use of capacitive touch sensors that are able to distinguish light and forceful touches, as a possible option for interface design. Xu and Bradburn discuss issues associated with implementation of capacitive touch devices and propose initial guidelines for their use. In the final paper, “Movement Time for Different Input Devices,” Bacon and Vu describe an experiment showing movement times for three input devices commonly used in HCI tasks. Movement time was shortest when the input modality was a button press on a response panel, intermediate when it was a computer mouse, and slowest when it was a touch screen. The authors discuss implications for the design of display-control configurations using these input devices.
Designing Web Sites and Interfaces to Optimize Successful User Interactions
65
This symposium contains both basic and applied knowledge derived from experiments and design experience. Each paper provides a unique contribution to understanding issues for optimizing interfaces and Web sites for human use.
References 1. Proctor, R.W., Vu, K.-P.L.: Complementary Contributions of Basic and Applied Research in Human Factors and Ergonomics. Theor. Iss. in Erg. Sci. (in press) 2. Proctor, R.W., Vu, K.-P.L. (eds.): Handbook of Human Factors in Web Design. Lawrence Erlbaum Associates, Mahwah (2005) 3. Tullis, T.: Web-based Presentation of Information: The Top Ten Mistakes and Why They Are Mistakes. In: HCI International 2005, Human-Computer Interfaces: Concepts, New Ideas, Better Usability, and Applications, vol. 3. Lawrence Erlbaum Associates, Mahwah, (2005) 4. Fang, X.: User Search Strategy in Web Searching. In: HCI International 2005, HumanComputer Interfaces: Concepts, New Ideas, Better Usability, and Applications., vol. 3. Lawrence Erlbaum Associates, Mahwah (2005) 5. Rau, P.-P.P., Choong, Y.-Y., Plocher, T.: Cross Cultural Web Design. In: HCI International 2005, Human-Computer Interfaces: Concepts, New Ideas, Better Usability, and Applications, vol. 3. Lawrence Erlbaum Associates, Mahwah (2005) 6. Volk, F., Kraft, F.: Understanding Online Consumers: A More Complete Picture. In: HCL 2005, Human-Computer Interfaces: Concepts, New Ideas, Better Usability, and Applications, vol. 3. Lawrence Erlbaum Associates, Mahwah (2005) 7. Vaughan, M., Dumas, J.: Web-based Programs and Applications for Business. In: HCL 2005, Human-Computer Interfaces: Concepts, New Ideas, Better Usability, and Applications, vol. 3. Lawrence Erlbaum Associates, Mahwah (2005) 8. Najjar, L.: Accessible Java-Application User Interface Design Guidelines. In: International, H.C.I. (ed.) HCI International 2005, Human-Computer Interfaces: Concepts, New Ideas, Better Usability, and Applications, vol. 3. Lawrence Erlbaum Associates, Mahwah (2005) 9. Vu, K.-P.L., Proctor, R.W. (eds.): Handbook of Human Factors in Web Design, 2nd edn. CRC Press, Boca Raton (2011)
Petimo: Sharing Experiences through Physically Extended Social Networking Nimesha Ranasinghe1, Owen Noel Newton Fernando1, and Adrian David Cheok1,2 1
Keio-NUS CUTE Center, IDM Institute, National University of Singapore, 119613, Singapore 2 Keio University, Hiyoshi, Kohoku-ku, Yokohama City, Kanagawa, Japan {nimesha,newtonfernando,adriancheok}@mixedrealitylab.org
Abstract. This paper presents an experience-sharing platform, Petimo, which consists of two modules, Petimo-World and Petimo-Robot. This system extends the traditional social networking concept into the physical world by incorporating a child friendly soft robotic toy for easy and safe social experience. It adds a new physical dimension to social computing and provides extra safety in making friends by physically touching each other’s robots. Petimo system can be connected to any social network and it provides safety and security for children. Petimo-World demonstrates many basic features with traditional online social networks in order to share personal experiences. Petimo-World stands out from all other virtual worlds with its interesting and sophisticated interactions such as the visualization of friends’ relationships through spatial distribution in the 3D space to clearly understand the closeness of the friendship, personalized avatars and sending of special gifts/emoticons.
1 Introduction Most of the times people are able to use full range of expressions in face to face communications: language, expressions, gestures, all the senses (hearing, sight, touch, smell, and taste), and interaction with the artifacts and space. However, remote communication has to rely on a more limited range at present: text, sound, image, and video alone or in any combinations [1]. Thus the main motivation of proposed research is to share the experience in remote communication. However, we have a little understanding and knowledge on feeling, emotion, or mood provoked in users, though it is a basic element in the human-mind [6]. The most common definition for Experience is “the knowledge or skill which comes from practice rather than from books or something that happens to one and has an effect on the mind and feelings” as explained [3]. Noticeably, in Japanese culture there is a field of study known as ‘Kansei’. It is a process, which expresses the feelings gathered through all the senses (i.e. hearing, sight, touch, smell, and taste). This process has a broad interpretation including sense, sensibility, emotion, feeling, and experience. Furthermore, when developing the prototype system we have incorporated several steps to address the main motivation defined as in Kansei process [4, 5]. In the present society, social networks have become the latest trend for online experience sharing and online communications especially among young children. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 66–74, 2011. © Springer-Verlag Berlin Heidelberg 2011
Petimo: Sharing Experiences through Physically Extended Social Networking
67
Social networks facilitate for making new friends while keeping old friends in close contact as well as for expressing its users themselves or their personal experiences to friends. The users in social networks such as facebook, myspace, and twitter are using one or more means of communication such as texts, images, audios, or videos to communicate their experiences. Especially, with the expansion of digital media, the attraction of teenagers and younger children to social networks and other activities in the cyber-world is growing. Among many of the related issues in current social networking services two main problems have identified to answer through this research. Firstly, although the social networking services greatly improved during the last couple of years, the traditional methods of using social networking services have not changed. The users still have to use social networking services through either a computer or a mobile phone interface. There have been many researches conducted on the importance of touch for children as well as adults, especially for childhood developments and infants' developments [2, 7]. Because of this reason people are more physically isolated though they have enough tools to connect remotely [8]. The lack of physicality in these modes of communications is one of the main reasons and that is one motivation for this research. Secondly, cyberspace is increasingly becoming an unsafe and more victimized environment, especially for children [9]. This results in conflicting messages between parent and child, social isolation, cyber connectivity with unknown people with unverified identities. Psychologists have theorized about the meaning of online relationships during adolescence and warned about the dangers of sexually exploitative online relationships [10]. With these motivations we introduced “Petimo”, which is aimed towards providing a novel means of physical interaction to social networks as well as a novel platform for children family interactions. As finer inspirations to the described research theme, Petimo provides an experience communication platform through social networking and physically extending the social networking concepts towards children and family communications. “Petimo” and the “Petimo-World” are two main components in this research where Petimo is a soft robotic companion and Petimo-World is a 3D representation of a virtual world designed for children. Both the Petimo and PetimoWorld are influenced by the popular Japanese “Kawaii” (cute) culture [11], as shown in Figure 1. The Petimo-World is a 3D virtual world with added social networking capabilities along with the soft robotic toy named Petimo. Petimo extends the virtual social network into the real world and provides physical interactions and direct communication with the characters in Petimo-World. When children play with their friends using Petimo, the virtual characters in Petimo-World interact with each other accordingly. This research is also motivated by Japanese “Kawaii” values and aiming for designing an experience-sharing platform with this insight of cuteness. From this perspective, we decided to focus on designing a robot with a warm feeling and a tender image of personality. By using feminine colors and a smooth surface, we aim to reduce mechanical feeling and increase human kind sensation into Petimo robot. With a spherical outer and a curve shaped display, it will be more like a pet, which could play with the children with its lovely cute eyes. In addition, based on Japanese manga (comic) [12] culture, characters are designed with big eyes and egg-shaped
68
N. Ranasinghe, O.N.N. Fernando, and A.D. Cheok
faces that bring a soft feeling to children and the ‘chubby cheeks’ are similar to a baby face [13]. Without limiting to just online interaction, Petimo is expected to further the social interaction into the real physical world by providing such similar features in the typical social networking services. We embed interaction modes such as shaking, touching, squeezing etc. in the robot with the objective of allowing natural human communication through the device. By introducing physical face-to-face friend adding in social networks, it was aimed to provide more close coherence between online and real worlds in addition to providing security. Finally, we believe that this approach will introduce a new paradigm for remote communication by introducing experience communication to the existing social networking concept thus increasing the natural qualities of lives.
Fig. 1. Petimo Robot and Petiomo-World
2 Related Works Modern online social networks have been enhanced with lots of interesting features as the worldwide user attractions are expected to rise boundlessly. MySpace and Facebook can be considered as some of the common online social networks for adults. Safe social networking cannot be expected through these networks, especially for children as they may provide unsafe methodologies in socializing. Conversely, social networks, which are specially designed for children like Hello Kitty Online [15] and Club Penguin [14], could be categorized as similar social networks to Petimo-World. They provide messaging and social networking services like email, emoticons (emotion icons), actions such as waving or dancing, discussion boards, online video sharing etc. This may create certain security lapse for child-safety, especially by exposing children to abuse by strangers. More importantly these are purely virtual worlds that do not have the advantage of having physical interaction and safe friend making features like Petimo. Poken [16] allows users to connect through a small tangible device in the shape of a palm with four fingers. In this way, users make friends and exchange social information based on the time and place that they meet. While this may be effective for adults to interact and socialize with one another, there are potential problems for young children in using Poken. The physical device itself is relatively small, enclosed in a hard casing. Currently, as the Poken user interface has a quite simple contact adding mechanism, emotional and experience based communication is difficult to
Petimo: Sharing Experiences through Physically Extended Social Networking
69
articulate among users. Petimo is designed for children, with a soft, squeezable enclosure, cute design, and color display integrated to allow children to perform bidirectional emotional communication such as sending emoticons and gifts. Tangible and physical objects have rich affordances which users can learn simply by grasping and manipulating them [17]. Previous generation of children before the explosive growth of computers and the Internet learn by exploring and manipulating physical objects. The power of information and the Internet mean that computers have taken over any other toys or natural physical environment as the tool for learning. It is hard to deny that computer hold immense power for children to learn from it. However, there still exist a gap between the digital computer and the physical world. Learning using the computer as a tool neglects the lessons we can learn from interacting with real physical objects. Therefore, this approach supports traditional exploratory play with physical objects, extend, and enhance by the interactive power of digital technology.
3 System Description The software architecture of the system is depicted in Figure 2. The Petimo-World client side comprises of two software components, Petimo-World Client and PetimoInterface Client. Petimo-World Client is an extension to Multiverse Client. PetimoInterface is the software component that implements the communication between the Petimos. Petimo-Interface connects directly with the Petimo-World server while the communication between two Petimo-World browsers is done through Multiverse online gaming platform. Petimo-World server is a centralized server that stores the data related to Petimo-World users and coordinates the communication in PetimoWorld. 3.1 Petimo-World Features This section presents a detailed technical description of the two levels in the PetimoWorld, known as the Macro and Micro worlds. Petimo-World has designed as two levels named macro world and micro world. Macro world has a novel 3D friend list arrangement while micro world is a garden like environment where other friends can visit and play. As explained, Macro level was developed to provide the user management functionality on the Petimo-World. The main user is represented as a character named Seedar that has a shape very similar to the Petimo robot. The friends are arranged around the seedar in Spherical orbits where the whole set of Seedars are immersed in a pink color galaxy. When the user logs in to the Petimo-World, he or she is directed to the Macro level. The user's seedar appears on the screen with friends arranged in spherical orbits in the galaxy. The user can navigate through the galaxy and reach the friend seeders. Macro level provides interactions such as visiting a friend's micro level, removing a friend through right clicking on the seedar character. As the arrangement of the friends in the Macro world is based on the concept of spherical orbits, the Perlin noise [18] based approach was chosen for the algorithmic
70
N. Ranasinghe, O.N.N. Fernando, and A.D. Cheok
base because it renders a more natural arrangement of friends in a spherical orbit. The friends are scattered into spheres based on the grouping created by a grouping algorithm as shown in Figure 3.
Fig. 2. Software architecture of Petimo
Fig. 3. Spatial arrangement in macro world
Petimo: Sharing Experiences through Physically Extended Social Networking
71
By clicking on friends' Petimo characters, users can visit their friends' micro world, which lies below the macro world. Micro world is a garden like environment, as shown in Figure 4, representing the world inside the Petimo planet.
Fig. 4. Overview of micro world
3.2 Petimo-Robot Features Petimo includes a friend adding function using close proximity radio frequency identification (RFID) technology. As shown in Figure 5 children can add friends by
Fig. 5. Friend adding function
72
N. Ranasinghe, O.N.N. Fernando, and A.D. Cheok
activating the “Add Friend” option on the Petimo menu and physically touching their friends' Petimo. This internally results in exchanging of unique 64-bit identification keys between two Petimos and sending this event to the online user verification system for authentication, after which the relationship is created. The user input sensing includes a smooth scrolling enabled resistive touch-sensing pad primarily for child-friendly menu navigation. Pressure activated squeeze areas of the robot surface facilitates exchange of special gifts and emoticons online. To ensure the rich content and personal experience sharing, a vibrotactile effect generator, sound output module, and a display module have also been used for actuation. A mini, low cost, energy saving color Organic Light Emitting Diode (OLED) [19] display has been used in Petimo as the primary media for interactive feedback as in Figure 6. The unique RFID key exchanging mechanism extends the communication bandwidth comprehensively without additional complexity associated with tangible interfaces.
Fig. 6. Petimo Robots and OLED display (Emoticon and Gift sending)
4 Communication Module Communication module is the heart of the Petimo platform and the communication module bundled with the Petimo-World. However, it has the ability of performing tasks independently with the Petimo server. This allows Petimo users to use robot and interact with the system without interacting with the Petimo world. The ability to configure more than one communication module in one PC provides the ability to configure several Petimo robots for one user thus to map different characters in virtual world. This module has two sub modules: Robot to PC- and PC to servercommunication. Robot to PC communication is implemented through Bluetooth protocol while PC to server communication is implemented through TCP/IP sockets.
5 Conclusion In this paper, the importance of the multi-sensory communication mediums along with social networking as well as the importance of sharing personal experiences is
Petimo: Sharing Experiences through Physically Extended Social Networking
73
considered. We have extensively described Petimo as a revolutionary, interactive, and friendly soft robotic device, extending its capabilities to change social networks fundamentally providing a novel approach for children to make friends easily in a more protected and safe social networking environment. Petimo together with PetimoWorld, encourages the building of real social networks through interactions as they interact by squeezing, touching and sending gifts or emoticons to their friends, family, and parents. This will dramatically change the younger generation's tendency of being disconnected from family and loved ones by bridging the gaps of existing social network security issues and acting as a powerful means to support a child's safe path toward a secured and personally enriching social networking experience. The individual concepts gleaned from this can be widely used future works with new interfaces which could not have been imagined before extending its capabilities to fundamentally change social networks and providing a novel approach to helping children make friends easily in a more protected and safe social networking environment. Additional Authors. Kening Zhu, Dilrukshi Abeyrathne, Kasun Karunanayaka, Chamari Priyange Edirisinghe, Roshan Lalintha Peiris, and James Keng Soon Teh, are from National University of Singapore. Yukihiro Morisawa, Charith Fernando, Miyuru Dayarathna, Anusha Indrajith Withana, Nancy Lan-Lan Ma, and Makoto Danjo are from Keio University Japan. Acknowledgement. This research is carried out under CUTE Project No. WBS R7050000-100-279, partially funded by a grant from the National Research Foundation (NRF) administered by the Media Development Authority (MDA) of Singapore.
References 1. Hertenstein, M.J.: Touch: Its Communicative Functions in Infancy. Human Development 45(2), 70–94 (2002) 2. Field, T.: Touch. MIT Press, Cambridge (2003) 3. Longman Dictionary of Contemporary English 4. Levy, L., Yamanaka: On Kansei and Kansei Design- A Description of Japanese Design Approach. In: International Association of Societies of Design Research (IASDR 2007) conferences, Hong Kong (2007) 5. Elokla, N., Morita, Y., Hirai, Y.: Using the Philosophy of Kansei: Happiness with Universal Design Product. In: International DesignEd Asia Conference, Hong Kong (2008) 6. Nagashima, T., Tanaka, H., Uozumi, T.: An overview of Kansei engineering: a proposal of Kansei informatics toward realising safety and pleasantness of individuals in information network society. International Journal of Biometrics 01(01), 3–19 (2008), ISSN: 17558301 7. Hertenstein, M.J.: Touch: Its Communicative Functions in Infancy. Human Developement 45(2), 70–94 (2002) 8. Eriksen, T.H.: Tyranny of the moment: Fast and slow time in the information age. Pluto Press (2001)
74
N. Ranasinghe, O.N.N. Fernando, and A.D. Cheok
9. Cho, C.H., Cheon, H.J.: Children’s Exposure to Negative Internet Content: Effects of Family Context. Journal of Broadcasting & Electronic Media 49(4), 488–509 (2005) 10. Wolak, J., Mitchell, K.J., Finkelhor, D.: Escaping or connecting? Characteristics of youth who form close online relationships. Journal of Adolescence 26(1), 105–119 (2003) 11. Lee, D.: Inside look at japanese cute culture (September 2005), http://uniorb.com/ATREND/Japanwatch/cute.htm 12. Schodt, L., Frederik.: Manga! Manga!: The World of Japanese Comics (Manga). Kodansha America (March 1986), http://www.amazon.com/exec/obidos/ redirect?tag=citeulike07-20&path=ASIN/0870117521 13. Hatch, Joshua, Rasinski, Tinothy, V.: Comic Books: From Superheroes to Manga. Red Brick Learning (2005) 14. Club penguin online(cpo) (2008), http://www.clubpenguin.com 15. Hello kitty online, hko (2008), http://www.sanriotown.com/main/index.php?lang=us 16. Welcome to poken (2008), http://www.doyoupoken.com 17. Ishii, H., Ullmer, B.: Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of the SIGCHI conference on Human factors in computing systems, Atlanta, Georgia, United States, pp. 234–241. ACM, New York (1997) 18. Perlin, K.: An image synthesizer. In: Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 287–296. ACM, New York (1985) 19. Oled (2008), http://www.4dsystems.com.au/prod.php?id=29
Comparison Analysis for Text Data by Using FACT-Graph Ryosuke Saga1, Seiko Takamizawa2, Kodai Kitami2, Hiroshi Tsuji3, and Kazunori Matsumoto2 1
Kanagawa Institute of Technology, Facuty of Information and Computer Science 1030 Shimo-ogino, Atsugi, Kanagawa, 243-0292, Japan 2 Kanagawa Institute of Technology, Graduate School of Engineering, 1030 Shimo-ogino, Atsugi, Kanagawa, 243-0292, Japan 3 Osaka Prefecture University, Graduate School of Engineering, 1-1 Gakuen-cho, Nakaku, Sakai, 559-8531, Japan {saga,matumoto}@ic.kanagawa-it.ac.jp, {seiko.takamizawa,kodai.kitami}@gmail.com,
[email protected]
Abstract. This paper describes a method to apply the Frequency and Cooccurrence Trend (FACT)-Graph to comparison analysis. FACT-Graph is the method to visualize the changes in keyword trends and relationships between terms over two time periods. The usefulness of FACT-Graph has been shown in tracking trends in politics and crime. To apply FACT-Graph to compare information, we use class transition analysis and separate analysis periods into categories that are the target of comparisons, and collate the features in each comparison target. For the comparison analysis by using 138 articles from two newspapers, we compare topics such as politics and events in them by using the relationships between terms found in the FACT-Graph results. Keywords: Comparison Analysis, Visualization, FACT-Graph, Text Mining, Knowledge Management.
1 Introduction As information systems progress, several business organizations have begun to focus on knowledge management to create business value and sustain competitive advantage by using data in data-warehouses [1][2]. To make these data-warehouses work to their advantage, they have to recognize their strong points, develop a strategy, and make effective investments. To recognize advantages, comparison analysis is often done by using crosstabulation and visualization analysis. Comparison analysis is relatively easy when the comparative data are expressed quantitatively. However, most significant data often occur in text data and are difficult to obtain from pre-defined attributes. Therefore, text data in questionnaires, reports, and so on must be analyzed. Text mining is useful for analyzing text data to obtain new knowledge [3]. In text mining, the applicable areas are wide-ranging such as visualization, keyword extraction, G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 75–83, 2011. © Springer-Verlag Berlin Heidelberg 2011
76
R. Saga et al.
summarization of text, and so on. We have developed the Frequency and Co-occurrence Trend (FACT)-Graph for trend visualization of time-series text data[4]. FACT-Graph is used to visualize the trends in politics and crimes to extract important keywords that look unimportant at a glance. This paper describes a method to compare two targets by using FACT-Graph. However, FACT-Graph targets time-series data, so we cannot apply it for comparison analysis. Therefore, we change data for analysis on the basis of class transition analysis to enable FACT-Graph to carry out comparison analysis. The rest of this paper is organized as follows: Section 2 describes the overview and underlying technologies of FACT-Graph. Next, Section 3 describes how to apply FACT-Graph for comparison analysis. After that, Section 4 performs a case study of two Japanese newspapers. Finally, we conclude this paper.
2 FACT-Graph FACT-Graph is a method to create visualized graphs of large-scale trends[4]. It is shown as a graph embedded co-occurrence graph and the information of keyword class transition. FACT-Graph enables us to see the hints of trends, which have been used for analyzing trends in politics and crime by using analysis tool [5][6]. FACT-Graph uses nodes and links. It embeds the change in a keyword’s class transition and co-occurrence in nodes and edges. It has two essential technologies: class transition analysis and co-occurrence transition. 2.1 Class Transition Analysis and Co-occurrence Transition Class transition analysis shows the transition of keyword class between two periods [7]. This analysis separates keywords into four classes (Class A to D) on the basis of term frequency (TF) and document frequency (DF) [8]. The results of the analysis detail the transition of keywords between two time-periods (before and after) as shown in Table 1. For example, if a term belongs to Class A in a certain time period and moves into Class D in the next time period, then the trend regarding that term is referred to as “fadeout”. FACT-Graph identifies these trends by the node’s color. For example, red means fashionable, blue unfashionable, and white unchanged. In convenience, we call the fashionable patterns Pattern1, the unchanged patterns Pattern 2, and unfashionable patterns Pattern 3. Additionally, a FACT-Graph visualizes relationships between keywords by using co-occurrence information to show and analyze the topics that consist of multiple terms. As a result, useful keywords can be obtained from their relationship with other keywords, even though that keyword seems to be unimportant at a glance, and the analyst can extract such keywords by using FACT-Graph. Moreover, from the results of the class-transition analysis, the analyst can comprehend trends in keywords and in topics (consisting of several keywords) by using FACT-Graph. Also, FACT-Graph pays attention to the transition of the co-occurrence relationship between the keywords. This transition is classified into the following types. (a) Co-occurrence relation continues in both analytical periods. (b) Co-occurrence relation occurs in later analytical period. (c) Co-occurrence relation goes off in later analytical period.
Comparison Analysis for Text Data by Using FACT-Graph
77
The relationship in type (a) indicates that these words are very close together, so we can consider them to be essential elements of the topic. On the other hand, relationships in types (b) or (c) indicate temporary topical changing. Table 1. Transition of Keyword Classes; Class A (TF: High, DF: High), Class B (TF: High, DF: Low), Class C (TF: Low, DF: High), and Class D (TF: Low, DF: Low) After
Before
Class A Class B Class C Class D
Class A Hot Common Broaden New
Class B Cooling Universal Widely New
Class C Bipolar Locally Active Locally New
Class D Fade Fade Fade Negligible
2.2 Output of FACT-Graph Figure 1 overviews the steps for outputting FACT-Graph. Firstly, the text data must be morphologically analyzed in order to output FACT-Graph. A morpheme is the smallest unit that gives meaning to a sentence (a, the, -ed, etc.). Text data is divided into morphemes, and the parts of speech are also judged by using these tools. This step also extracts the attributes of each document, such as date of issue, length of document, category of document, and so on. Then, the term database is built. The user sets up the parameters such as analysis span, the filter of documents/terms, thresholds used in the analysis, and so on. Then, the term database is divided into two databases (first and second half periods) in accordance with the analysis span. Each term’s frequency is aggregated in respective databases, and keywords are extracted from terms under the established conditions. These keywords go through procedures concerning the transition of keyword’s classes and co-occurrence. The output chart that reflects the respective processing results is a FACT-Graph.
E e
Time-series Text Data
Term DB
Transition Analysis (Keywords and Links)
F
B
D d G g
t1 ? t 2
A a
t2? t3
C
FACT-Graph 2-span database Parameter Analysis period
Thresholds (TF,DF, co-occurrence)
Number of keywords
User
Fig. 1. Overview of Outputting FACT-Graph
78
R. Saga et al.
3 Comparison Analysis by FACT-Graph 3.1 Approach To apply FACT-Graph to compare information, we pay attention to class transition analysis in FACT-Graph. As we mentioned before, class transition analysis is carried out on the basis of two time periods, and FACT-Graph shows the changes between them. The other side of the coin is that FACT-Graph shows the results of the comparison between the periods, and the periods are simply regarded as the categories “Before” and “After”. In other words, we can comprehend that FACT-Graph performs a comparison analysis between two categories “Before” and “After” although it treats time-series text data. By replacing the periods with targets for comparison, we can compare them by using FACT-Graph. However, applying FACT-Graph to comparison analysis has three problems: processing target data, explaining class transition analysis in comparison analysis, and how to express co-occurrence relationships. 3.2 Converting Target Data To apply FACT-Graph to comparison analysis, we need to convert date data. In FACT-Graph, the time data must be included in target data because the data are necessary and help to separate all target data into two periods. On the other hand, for comparison analysis the time data are not necessarily, and the time data do not exist in target data from the very first. Therefore, we attach pseudo time data to target data as a category that belongs to either a period between t1 and t2 like the “Before” period or a period between t2 and t3 like the “After” period. Therefore, it is possible to perform comparison analysis by using FACT-Graph. 3.3 Explanation of Class Transition Analysis and Co-occurrence The interpretation of comparison analysis by FACT-Graph is different from that of trend analysis, but the essential idea is same. The concept of the comparison between two targets is the same as that of class transition analysis although the meanings of a FACT-Graph change, and we can compare the two targets in the same way we analyze FACT-Graph. For example, if two targets have the same keywords that belong to Class A (high TF and high DF), these targets have the equivalent features about topics that the keywords indicate. Let one target have Class A keywords and another Class B, Class C, Class D keywords. Then the former target is characteristic of the topic. FACT-Graph has three types of co-occurrence. For comparison analysis, the cooccurrence means that one or more target uses the terms together. That is, the cooccurrence of type (a) means that a co-occurrence relationship exists in both targets. The other types mean that a co-occurrence relationships exists in alternative targets. By the way, for trend analysis by using FACT-Graph, flux and reflux of the tides of terms are important, so we classify classes into four classes, Class A to D, by the height of TF and DF. However, for comparison analysis, knowing whether a term
Comparison Analysis for Text Data by Using FACT-Graph
79
exists or not is necessary to find features of comparison targets in comparison analysis. Therefore, we add a new class, Class E, which expresses a term existing in only one side of comparison targets. Additionally, this class is expressed by a circular broken line in a visualized graph. Summarizing the above discussion, the FACT-Graph for comparison analysis visualizes terms as shown in Figure. 2.
Class A: High TF and DF Class B: Low TF and High DF Class C: High TF and Low DF
E e
F Class D: Low TF and Low DF
B
D d G g A a
C
Class E: TF = DF = 0 Pattern 1 and Type (a) : Be characteristic terms to both targets and exist links in both targets Pattern 2 and Type (b) : Be characteristic terms to Target B and exist links in Target B only. Pattern 3 and Type (c) : Be characteristic terms to Target A and exist links in Target A only.
Fig. 2. FACT-Graph for Comparison Analysis
4 Experiment 4.1 Data Set By using FACT-Graph, we carried out an experiment to verify whether comparison analysis can be performed. In this study, we used editorials published in The Mainichi and The Asahi newspapers, two of Japan’s major newspapers, between 2006 and 2008. Editorials are used because they pick up on important issues and are often written on the basis of interviews or opinions. Generally, these articles are written from several viewpoints, and the assertions are characteristic of and different for each publisher. Note that we regard few frequent words as unnecessary terms because there is a probability that they are noise and error words. Therefore, we removed the terms for which TF is less than 2 and DF is equal to 1. In this case study, we limited articles to those on the topic of the Olympic Games. Mainichi had 64 editorials and Asahi 74. We apply Jaccard coefficient as cooccurrence and adopt the relationships whose co-occurrence is over 0.3. To carry out class transition analysis in FACT-Graph, we configure the threshold into the top 20% ranked termed on the basis of Zipf's law[9], which is often called 20-80 rules.
80
R. Saga et al.
4.2 Result of Analysis Figure 3 shows the results of FACT-Graph in these conditions. In this graph, blue nodes and links indicate the features in The Mainichi and red nodes and links The Asahi. When we take a global view of FACT-Graph, the term “Olympic”, which is the most important word, is bigger than other nodes and belongs to Class A and Pattern 1 in this graph. Also, there are “Beijing”, “Japan”, and “China”, which have much the same pattern and class as “Olympic”. Therefore, in this analysis period, the biggest topic in this graph concerns the Beijing Olympics. Also, in the central parts of Figure 3, the nodes of Pattern 3 connected by type (c) have been closed up. These nodes are a lot of words that are relevant to the games themselves, such as “Kitajima” (a Japanese gold medal winning swimmer), “Judo”, and “Skating”. For this reason, we can say that The Mainichi describes the Olympic Games without referencing anything else. Figure 4 scales up the lower right of Figure 3. There are a lot of Pattern 2 nodes about China in this area, and all the nodes are connected to each other with type (b). From this area, we can conclude that The Asahi wrote the articles about the Olympics that reference the government of China. On the other hand, Figure 5 scales up the upper left of Figure 3. There are some topics about not only Tokyo’s bid for the 2016 Summer Olympic Games but also the Turin Winter Olympics and Beijing Summer Olympics in the target data. Specifically, the 2016 Summer Olympic Games is deeply relevant to the Tokyo gubernatorial election. In fact, this area has nodes such as “Ishihara” or “governor of Tokyo”, and The Asahi used these nodes extensively even though the relationships between terms often occurred in The Mainichi. In the same way, The Asahi also mentioned “Taiwan”, "Nationalist Party”, and so on. That is, The Mainichi writes separately about elections and the Olympics. On the other hand, The Asahi describes topics politically because it writes about the Olympics while referring to a political party of a foreign country. As a result of the analysis limited to articles about the Olympics, we found that the articles in The Asahi have a stronger political tone than those in The Mainichi. 4.3 Consideration As we mentioned above, comparison analysis for text data by using FACT-Graph is possible. However, there are two analytic issues. 1. FACT-Graph for trend analysis attaches more importance to the features in the latter period than those in first period. That is to say, the FACT-Graph represents the trends from the views of the latter period. Actually, Figure 3 shows the features of The Asahi more strongly. However, it is better that each period is observed equally. Therefore, we have to analyze text data by interchanging Target A and Target B and analyze two graphs. 2. For setting the threshold of class transition analysis, we use Zipf's law in this analysis. However, we should pick out the value so as to apprehend the terms’ features depending on the characteristics of the data.
Comparison Analysis for Text Data by Using FACT-Graph
Fig. 3. Result of Visualization with Mainichi Newspaper and Asahi Newspaper
81
82
R. Saga et al.
Fig. 4. Broaden Lower Right of Figure 3
Fig. 5. Broaden upper left of Figure 3
5 Conclusion This paper described a method to compare two targets by using FACT-Graph, which can visualize the trends for time series text data. To apply FACT-Graph to comparison analysis, we interchanged target data with time series on the basis of class transition analysis. Also, we explained the two essential technologies for comparison analysis and performed comparison analysis. To validate the usability of FACT-Graph, we compared the features of The Asahi and The Mainichi newspapers by using editorials from both. From the results of comparison analysis targeting the word “Olympic”, we found that The Asahi tended
Comparison Analysis for Text Data by Using FACT-Graph
83
to write more political articles than The Mainichi and showed that the proposed method could be used for comparison analysis between two targets. We have three future works. First is visualizing both features of the analysis target. To perform comparison analysis more accurately, we have to regard all features of target data as equal. Therefore, we should integrate figures that express features of both targets. Second is integrating graphs among three targets. In this paper, we integrated graphs between two targets. However, comparing three targets may possibly be necessary, so we should integrate graphs among three targets by considering the explanation of class transition analysis. Finally, we should verify the optimal threshold for TF and DF. Acknowledgement. This research was supported by The Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan Society for the Promotion of Science (JSPS), Grant-in-Aid for Young Scientists (B), 21760305, 2009.4-2011.3.
References 1. Tiwana, A.: The Knowledge Management Toolkit: Orchestrating IT, Strategy, and Knowledge Platforms. Prentice-Hall, Englewood Cliffs (2002) 2. Inmon, W.H.: Building the Data Warehouse. John Wiley & Sons Inc., Chichester (2005) 3. Feldman, R., Sanger, J.: The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data. Cambridge University Press, Cambridge (2007) 4. Saga, R., Terachi, M., Sheng, Z., Tsuji, H.: FACT-Graph: Trend Visualization by Frequency and Co-occurrence. In: Proceedings of the 31st annual German conference on Advances in Artificial Intelligence, Kaiserslautern, Germany, pp. 308–315. Springer, Heidelberg (2008) 5. Saga, R., Tsuji, H., Tabata, K.: Loopo: Integrated Text Miner for FACT-Graph-Based Trend Analysis. In: Proceedings of the Symposium on Human Interface 2009 on Human Interface and the Management of Information. Information and Interaction. Part II: Held as part of HCI International 2009, pp. 192–200. Springer, Heidelberg (2009) 6. Saga, R., Tsuji, H., Miyamoto, T., Tabata, K.: Development and case study of trend analysis software based on FACT-Graph. Artificial Life and Robotics 15, 234–238 (2010) 7. Terachi, M., Saga, R., Tsuji, H.: Trends Trends Recognition. In: IEEE International Conference on Systems, Man & Cybernetics (IEEE/SMC 2006), pp. 4784–4789 (2006) 8. Salton, G. (ed.): Automatic text processing. Addison-Wesley Longman Publishing Co., Inc. (1988) 9. Baayen, R.H.: Word Frequency Distributions. Springer, Heidelberg (2002)
A Comparison between Single and Dual Monitor Productivity and the Effects of Window Management Styles on Performance Alex Stegman, Chen Ling, and Randa Shehab College of Industrial Engineering, University of Oklahoma, Norman. 200 W. Boyd St., Room 124 Norman, Oklahoma 73019
[email protected]
Abstract. Several research studies have been published on user opinion and productivity of using dual monitor systems. These studies found that users typically enjoy using multiple monitors, but none found a strong increase in performance and productivity. Other researchers have focused on improving multiple monitor usability, but often without any statistical framework. This study compared single and dual monitor productivity measures: task time, cursor movement, and number of window switches. Additionally, window management styles (WMS) were studied in order to help designers understand user behavior better. WMS were broken into two categories, toggler and resizer, and then compared to the WMS created by Kang and Stasko (2008). The results of the research showed a significant difference between the number of open applications and a significant difference between single and dual monitors for the number of window switches. The only significant difference between the toggler and resizer WMS was the number of window switches, which was an interaction between the styles and the tasks. Keywords: Dual Monitors, Window Management Style, Productivity.
1 Introduction Research on computer monitors varies greatly. Examples of the research topics include monitor size, the benefits of using LCD rather than CRT, and visual fatigue. Research on the use of multiple monitors in a work environment indicates that productivity increases when dual monitors are used in lieu of single monitors. [13]. The present study was performed to describe the differences in the work patterns between single and dual monitor configurations in a controlled experiment simulating an engineering work environment. The study provided a comparison of task time, cursor movement, and the number of toggles between applications for both single and dual monitor configurations. The study also outlined how users with different work patterns designate applications to monitors and how they utilize the virtual workspace. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 84–93, 2011. © Springer-Verlag Berlin Heidelberg 2011
A Comparison between Single and Dual Monitor Productivity
85
2 Literature Review Over the past decade, several important pieces of literature have been authored on using multiple monitors and the associated problems. Productivity continues to be a relevant topic because businesses and people are interested in understanding the possible benefits of using additional monitors. Through a better understanding of this new technology, new operating systems and applications can be designed to take full advantage of the virtual real estate gained when multiple monitors are utilized. Research has shown that when a person uses a single monitor, they will often use it as a single space; if they were using multiple monitors, they consciously treat the space on each monitor separately. [6]. The author also noted user tasks were split into primary and secondary tasks, and that most of the secondary tasks were for communication, such as instant messaging or email, and personal resources, such as calendars and web browsers. In addition, multiple monitor users often use the secondary monitor(s) in direct support of a primary task on a primary monitor. While increasing display space does not resolve the limitation on information management, such as the placement of windows and task bars, [6] went on to say that it does help by allowing users to spread information out and organize their windows more effectively. However, this study discovered that not all information is treated equally; people prefer to allocate their attention to one task while having additional information readily available. Research on time saved when using multiple monitors showed no significant difference in the amount of time spent switching between windows or the number of visible windows when comparing single and dual monitor configurations. The research also showed that the participants often hid specific window content that was determined to be distracting or unnecessary. [7] Researchers and software developers researched multiple monitor usage, tracking who used software to gather data on each participant’s work PC over an extended time period. [8] The results of the research showed that single monitor users were nearly twice as likely to have their active window fully visible, and much less likely to have their email fully visible. The researchers also found that the amount of unused space increased as the amount of space and monitors increased, meaning that participants used the space more efficiently. In conclusion, [8] reflected that users often interact with windows simultaneously to complete a task. The authors explained that multiple monitors help users view more windows at any given moment, but they still switch between windows as frequently. They speculate multiple monitor users may use larger window sizes, and believe that this pattern may not actually be related to monitor size. The research done also showed that multiple monitor users do not rely on the taskbar as much as single monitor users, as well information being spread across many monitors. Usage patterns with multiple monitors were more dynamic compared to single monitor usage. The research, however, remained unable to statistically test these patterns because the data were collected from an uncontrolled source. Focusing more on multiple monitor usage patterns, [9] identified three main usage strategies when interacting with the dialog boxes. They found that each participant
86
A. Stegman, C. Ling, and R. Shehab
used one of the basic strategies, but only a small fraction used the same strategy throughout the entire experiment. These findings indicate that a sole solution to problems encountered when using multiple monitors may not fit all users or fit all types of tasks. Given that participants exercise multiple strategies, software and operation system designers must think of robust solutions. The researchers indicated that further study of multiple monitor usage patterns would be beneficial to designers, but may not provide clear solutions to increase usability. Research on the effects that multiple monitors have on work performance and patterns for light weight computer tasks was completed by [13]. In a controlled experiment, the researchers compared how the participants performed on both single and dual monitor computers while working with word processors, web browsers, and spreadsheets. During the tasks, cognitive load, task time, and the window operations were recorded. Tracking software also collected the opening and closing of windows, locations of windows, as well as moving and resizing of windows. The cognitive workload measure indicated no significant differences between the single and dual monitors, or the sequence of which they were used. The results, however, did show a trend of multiple monitor users feeling less workload than single monitor users. An analysis on the different behaviors of single and dual monitor users on window management styles (WMS) was also performed by [13]. The categories were formed based on how the participants accessed windows and for the methods by which they organized their screen space. Participants accessed their windows by either using Alt+tab to switch the window in view, or by moving and resizing their windows. Three additional categories were formed for organizing screen space: “Maximizers”, “Near Maximizers”, and “Careful Coordinators”. Maximizers kept windows at the maximum size, while using Alt+Tab to switch between windows. Near Maximizers resized their windows to occupy most of the screen space, while Careful Coordinators resized their windows so that several windows were visible simultaneously. These categories, however, did not significantly affect task time or workload when compared to the number of monitors being used or the sequence of use. Even though the differences were not significant, when multiple monitors were used during the first session, Alt+Tab users and Move/Resizers both benefited. During the second session, the Alt+Tab users performed faster when using a single monitor, which could have been attributed to the learning curve. During the two sessions, the researchers observed that Alt+Tab users were less likely to use Alt+Tab with multiple monitors during the first task assignment, and were probably led to act more like Move/Resizers due to unfamiliarity of the task. In contrast, the Move/Resizers were able to coordinate their windows more rapidly and less frequently due to the increased screen space. The workload measures also showed that the Move/Resizers felt less workload during the multiple monitor setting in the second session. A higher workload may have been experienced when multiple monitors where first used, since the user had to manage more screen space and more windows. As the users gained experience using the additional monitors, they were able to develop strategies that allow them to efficiently utilize the increased screen space with less workload.
A Comparison between Single and Dual Monitor Productivity
87
3 Problem Definitions and Hypothesis The primary objective of the study was to compare the differences in work behaviors and usage patterns when participants used a single monitor and used dual monitors. Differences were examined for three metrics. Task time was measured as the number of seconds it took to complete each individual task. Cursor movement was quantified as the number of pixels the cursor moved during each of the computer tasks that the participant completed. Finally, the total number of switches between windows was recorded as a sum of the number of times a participant switched from one application to activate another application. This metric was chosen because it measures a loss in productivity due to cognitive processes being interrupted when users must seek out information, as well as the time lost in activating windows. An explanation of how productivity is affected by interrupting tasks can be found in [2]. Each of the three metrics was examined with respect to WMSs formulated within this research, as well as for the WMSs developed by [13]. Because the user’s WMS and work patterns may vary as the complexity of the task changes, altering the number of open applications may provide insight into how work patterns change when workload varies. Therefore the number of applications used was varied between 2, 3, and 4. The first set of hypotheses focused on the benefit of using dual monitors over single monitors in terms of time, cursor movement, and how many times participants had to switch between windows. The hypotheses for each of these dependent variables were that dual monitors would allow participants to perform faster with less cursor movement and less window switches. This notion was founded on participants being able to spread the windows across the larger virtual workspace when using dual monitors. Since the participants will have more viewable information, the amount of cursor movement to the task bar, as well as the number of window switches, would decrease as a result. The second set of hypotheses focused on the effects of different WMSs on performance. Two WMSs were created and tested for differences, toggler (TOG) and resizer (RES). A TOG was defined as a user who primarily sizes their windows to occupy most of the screen, relying on the task bar more heavily when activating windows. A RES was defined as a user who sized multiple windows to allow them to simultaneously view multiple windows. It was hypothesized that RESs would show a decrease for all three dependent variables, based on the thinking that since RESs are able to view more information at once, they would not have to search for windows using the task bar or activate applications by clicking overlapping windows. The RES did not have to activate windows to see pertinent information. The results of these three TOG and RES hypothesis tests were then compared to the WMS of [13] in order to identify any advantages either set of WMS may have over the other.
4 Methodology The computer tasks used for the experiment were designed to simulate normal engineering computer work that would be completed in a typical office setting while
88
A. Stegman, C. Ling, and R. Shehab
using multiple windows simultaneously on the computer monitor(s). Software packages were chosen to simulate the typical software used in an office environment. These software simulated drafting, data entry, spreadsheet analysis, email, and gathering data. A total of six tasks were designed. Each task was categorized by how many active windows were needed during the task. Three categories of total number active windows were used, two, three, and four, and there were two iterations of each. The task with two active windows used Excel and Outlook. The three active window task added Internet Explorer. Finally, the four active window task added AutoCAD to the suite. Each participant performed all 6 tasks during the experiment, 3 on both a single monitor and dual monitors. Prior to the experiment, the participants completed a short questionnaire and training. After each monitor setting, the participants completed a questionnaire. Finally, a post completion survey was administered. Overall, 36 participants were recruited, mostly from the School of Industrial Engineering at University of Oklahoma. The range of participant age was 18 to 39 years, with an average of 23 years of age, and a standard deviation of 3.5 years. The gender distribution of participants in this study was 67% males and 33% females. The primary equipment used by the participants was a DELL Optiplex 745 PC with two 19 inch monitors. The computer was fit with a standard keyboard and an optical mouse. The dual monitors were placed equidistant from the midsagittal plane. Three independent variables were studied during the experiment: 1) the number of screens (single or dual), 2) the number of active windows open during the tasks (2, 3, and 4), and the WMSs (TOG and RES). WMS was also researched in [13]; however, the categories have been defined differently in this research. Instead of having multiple categories for both sizing and accessing application windows, the categories used in this research combine the two. “Alt-Tabbing” was prohibited by experimental instruction in order to control the behavior, unlike [13]. This research focused on whether or not users are viewing information simultaneously, with the thought that by simplifying the categories, significant differences between the different WMSs would be found.
5 Data Validation, Correlations, and Results For task time, the fitted value plot showed a troublesome trend, as the fitted value increases the residual value also increases. The residual plots for cursor movement showed that the assumption of normality may have been violated due to the slight curvature of the line and the presence of several outliers. The fitted value plot showed a curve trend and an increasing fitted value. The violation to a zero mean for the error term could clearly be seen in the histogram, where the negative residuals showed a much higher frequency than the positive terms. The plots for window switches showed a violation occurred due to the increasing trend of the residual values as the fitted value increased. The histogram, however, did show a defined bell shaped curve, while the order plot also showed no violation. A correlation between cursor movement and task time was found (r2=0.7466, p <0.0001) indicating that as cursor movement increases, task time also increases. The correlation between task time and number of window switches showed that as the
A Comparison between Single and Dual Monitor Productivity
89
number of switches increases, task time also increases (r2= 0.5330, p<0.0001). Finally, the correlation between cursor movement and the number of window switches showed a positive trend for cursor movement as the number of window switches increases (r2=0.5191, p<0.0001). Since cursor movement and number of window switches were correlated to task time, they can both be considered indicators of productivity. Table 1. Means and standard deviations off all data points Window Switches Standard Deviation
Mean # of Window Switches
Mouse Movement Standard Deviation
Mean Mouse Movement
Task Time Standard Deviation
Mean Task Time
4
Sample Size
3
# of Monitors
# of Windows 2
1
34
461.06
168.55
92806.99
39233.39
9.59
7.21
2
34
484.41
154.48 101314.65
50355.19
4.06
3.09
1
34
687.91
267.78 144077.49
66694.41
34.88
10.79
2
34
643.97
253.18 124435.74
63203.57
30.79
12.49
1
34
1314.82
456.51 224794.09
96363.69
32.68
8.72
2
34
1310.09
414.08 225310.30
93169.08
32.06
9.55
As the means suggest, there were no significant differences for the number of monitors used or interactions. The number of windows did, however, produce a significant difference (p<0.0001). Tukey analysis showed that all three tasks were different from each other, although the two and three window task times were similar. Despite the mean mouse movement being higher for the 2 and 4 window tasks, no significant results were found. There was a significant difference in mouse movement for number of windows being used (p<0.0001). Though there were no significant interactions, the 3 window task exhibited a decreasing trend between the single and dual monitor tasks. Tukey analysis confirmed that all three tasks were different. There was both a significant difference in the number of windows and in the number of monitors (p<0.0001 and p=0.0028, respectively). Unlike task time and mouse movement, the Tukey showed that the 3 and 4 window tasks were similar, while the 2 window task was different from both. A statistical analysis was performed on the WMS in two ways. The first analysis was performed only on participants that exhibited one style of window management, called pure TOGs (PT) and pure RESs (PR). Of the 34 total participants, three were classified as PRs and 15 were determined to be PTs. The second analysis examined the patterns of all 34 participants. Task time for PTs and PRs was found to be significantly different (p<.0001). Tukey analysis revealed that the 3 window task requires less mouse movement that the 4 window task. Mouse movement only revealed significant differences for the number of windows being used. Tukey analysis revealed that the 3 window task requires less mouse movement than the 4 window task. The number of windows switches did not produce a significant difference (p=0.420). Only the interaction between the number of windows and the WMSs was significant for number of window switches (p=0.040).
90
A. Stegman, C. Ling, and R. Shehab
An additional statistical analysis was performed of all of the participants considering their respective WMS for each task. However, the results showed significant results only for the number of windows used for both measures of task time and mouse movement. The means for each type of WMS can be found below in table 2. Table 2. Means of TOGs and RESs with respect to number of monitors, Pure users are identified by bold italics # of Monitors Style
Mean T ask T ime
Mean Mouse Movement
Mean # of Switches
1
R
952.59
1028.83 172859.61 176445.11 35.77 32.00
1
T
1024.70
939.70
2
R
845.00
1088.17 164926.70 182445.29 32.88 29.50
2
T
1021.04 1032.23 178188.46 170398.93 30.94 32.63
189972.23 182445.29 32.88 31.93
6 Conclusions, Recommendations, and Future Work The experiment showed mixed results on the benefit of using dual monitors instead of single monitors. Unlike [13], no significant difference was found between single and dual monitors with respect to task time. It was surprising that despite including more complicated tasks than what was found in the literature, no significant differences for task time were found. Contrary to what the actual task times reveal, participants felt they had performed faster in the dual monitor configuration. When trying to determine the reasoning behind the results, two logical interpretations were found. Either there truly were no differences in task time due to the lack of operating system and software compatibility with multiple monitors, or problems existed within the experiment obstructing any significant differences from being found. It could have been possible that resizing windows and locating the windows on the additional monitor mitigated the productivity benefits of using dual monitors because of the time spent moving windows. When looking at the results for cursor movement, the differences between the single and dual monitor configurations were minimal, especially for the four window tasks. Cursor movement was a dependent variable that went unexamined by previous research. Even though no significant differences were found in this study, the survey revealed that the participants thought that they were moving the mouse less during the dual monitor configuration. The benefit of using dual monitors was seen through the decreased number of window switches required to complete a task with the dual monitor. However, time savings associated with reduced switching of windows were most likely mitigated by relocating and positioning windows. Other possible reasons behind the lack of difference between the single and dual monitor configuration results could have been limitations within the experiment, such as lack of experience with AutoCAD or lack of exposure to dual monitor computers. Drafting software experience was required of each participant in the experiment, but
A Comparison between Single and Dual Monitor Productivity
91
participants were not required to be familiar with AutoCAD. Due to the simple nature of the drafting required, proficiency with just AutoCAD was deemed to be unnecessary. Even though training was provided to each participant on the exact functions that were used during the drafting task, the lack of experience of those participants may have adversely affected their task times, mouse movement, and number of window switches. Also, there was no control for dual monitor experience, thus the results of the study may indeed have been impacted by user experience. When examining these variables no strong differences were seen between inexperienced and experienced AutoCAD users or multiple monitor users. The third possible explanation of why the experiment did not find differences between single and dual monitors was that the four window task could be completed using three windows at a time. Studying the four window tasks more closely, the tasks closely resemble a set of two separate tasks, rather than one task that requires all four windows to be used simultaneously. Therefore, only three windows were used simultaneously. Had the instructions stated that the user should perform both subtasks simultaneously, the results may have been different. Alternatively, if a between subject experimental design was utilized, equal tasks would not have been required. Another issue that may have affected the data arose during the design of the experiment. The researchers determined that it was necessary to create varying tasks of the same number of windows with equivalent task times. Using a GOMS analysis, two tasks for each of the 2, 3, and 4 window levels were created. However, during the experimental trials the researchers noticed that the task times for the variations in each of the 2, 3, and 4 window sets were not equal. The researchers determined that the cause of the disparities was due to the varying amounts of reading and comprehension that took place between the tasks. The tasks were then adjusted to equalize the task times, and the GOMS analysis results were adjusted to reflect the changes. It is possible that the adjustments to the tasks could have caused these differences. However, without a formal task analysis that incorporates a key stroke analysis and considers mental activity, it was impossible to form tasks that are truly equivalent. Therefore, it is recommended that future researchers consider developing a formal analysis that can be used to accurately form tasks that are equal in complexity. In conclusion to the dual monitor analysis, there may still be advantages to having multiple monitors despite the lack of evidence that they increase productivity in terms of time savings. The post-experimental survey revealed that people enjoyed using dual monitors, and believed that they were more productive when using them. If users are more satisfied with using a piece of equipment, job fatigue may be reduced while motivation and work morale may be increased. It would then be logical that providing multiple monitors to a work force would increase productivity for drafting, data mining, programming, and for gathering and recoding large amounts of information. Multiple monitors are especially advantageous when dynamic information, data that requires the user to frequently check, is being displayed by the user. Using a single monitor system would cause the user to allocate prime screen real estate to be able to view the information instantaneously, or cause them to search for the windows more often, thus reducing productivity. With a multiple monitor system, a user is able to designate an area or monitor to view secondary information.
92
A. Stegman, C. Ling, and R. Shehab
The results for three WMS analyses were surprising, no significant differences were found for any of the WMS. The trends of the means generally showed that there were some differences; therefore, perhaps an additional study with larger sample size would be able to show a statistically different result. Although, Kang and Stasko’s styles showed larger differences in performance for each WMS, compared to the TOG and RES WMS, there was still criticism of their categorization method. Kang and Stasko’s styles are based on both window size and how much information can be viewed. While the styles worked for single monitors, it did not effectively categorize dual monitor users. This was due to the fact that the participants could be classified as multiple styles simultaneously since two different styles could be utilized on different monitors. During the analysis, the process of categorizing each participant was difficult, due to the parameters set for each style. This occurrence was largely due to window size being used as a factor in determining what WMS a participant used. For example, during a four window task, a participant may have two maximized windows on one monitor, toggling between them, while on the second monitor having two windows sized so that both are fully visible at one time. Therefore, that user would be a Maximizer on one monitor and a Careful Coordinator on the second. Using the TOG and RES styles, the same participant would be identified as a RES, because they are generally trying to view as many windows as they can simultaneously. The reason that the TOG and RES categorization method was determined to have an advantage was because it focused on how a participant was gathering information. The most important factor on how windows were managed was how the participant was retrieving information, and how much information they were viewing at one time. Thus, the potential improvement to performance from being able to view more information at once would drive a user to manage their windows differently. While Kang and Stasko’s method considered how the windows are positioned, when they considered window size as a factor they confounded the process of categorization. The exact size of the windows does not matter; what actually matters is whether the participant can view multiple sources of information. In future studies multiple monitor productivity should examine the influence by many different factors such as: monitor size, monitor separation, integration with laptops, software, operating systems, and WMS. From the results of this research, it is recommended that more attention be placed on WMS and the amount of windows being used simultaneously. Alternatively, increasing task complexity by forcing participants to reference windows more often should also be examined. For both methods of increasing task complexity, it is also important to measure how the different styles of window management compare to each other when task complexity is increased. Finally, future research should also place more focus on recruiting an equal sample size of each of the WMS and longer experiment times.
References 1. Ashdown, M., Oka, K., Sato, Y.: Combining Head Tracking and Mouse Input for a GUI on Multiple Monitors. Computer Human Interatcion- Extended Abstracts (2005) 2. Bailey, B., Konstan, J., Carlis, J.: Measuring the Effects of Interruptions on Task Performance in the User Interface. IEEE, 757–762 (2000)
A Comparison between Single and Dual Monitor Productivity
93
3. Card, S., Moran, T., Newell, A.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum, Hillsdale (1983) 4. Covin, J., Tobler, N., Anderson, J.A.: Productivity and multi-screen displays. Rocky Mountain Comm. Review, Dept. Comm. Univ. Utah 2(1), 31–53 (2004) 5. Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G., Starkweather, G.: Toward characterizing the productivity benefits of very large displays. In: Proceedings of Interact 2003, pp. 9–16 (2003) 6. Grudin, J.: Partitioning Digital Worlds: Focal and Peripheral Awareneness in Multiple Monitor Use. In: Proceedings of Computer Human Interaction 2002, pp. 458–465 (2003) 7. Hutchings, D., Stasko, J.: New operations for display space management and window management. GVU Techinical Report GIT-GVU, pp. 2-18 (2002) 8. Hutchings, D., Czerwinski, M., Smith, G., Meyers, B., Robetson, G.: Display space usage and window management operation comparisons between single monitor and multiple monitor users. In: Proceedings of AMI 2004, pp. 32–39 (2004) 9. Hutchings, D.R., Stasko, J.: Mudibo: Multiple Dialog Boxes for Multiple Monitors. In: Proceedings of Computer-Human Interaction 2005 Extended Abstracts, pp. 1471–1474 (2005) 10. Hutchings, D. R., Stasko, J.: Revisited display space management: understanding current practice to inform next generation design. In: Proceedings of Graphics Interface, pp. 127– 134 (2004) 11. Jaschinski, W., Heuer, H., Kylian, H.: Preferred position of visual displays relative to the eyes: A field study of visual strain and individual differences. Ergonomics 41(7), 1034– 1049 (1998) 12. Jon Peddie Research (2008), 2008 CAD report, http://www.jonpeddie.com/publications/cad_report/ 13. Kang, Y., Stasko, J.: Lightweight task/application performance using Single versus Multiple monitors: A comparative study. In: Proceedings of the Graphics Interface (2008), pp. 17–24 (2008) 14. Mackinlay, J.D., Heer, J.: Wideband Displays: Mitigating Multiple Monitor Seams. In: Proceedings of Computer-Human Interaction Conference (2004) 15. Ringel, M.: When one isn’t enough: an analysis of virtual desktop usage strategies and their implication for design. In: Proceedings of Computer Human Interaction Extended Abstracts 2003, pp. 762–763 (2003) 16. Robertson, G., Czerwinski, M., Baudisch, P., Meyers, B., Robbins, D., Smith, G., Tan, D.: The Large-Display User Experience, pp. 44–51. IEEE Computer Society, Los Alamitos (2005) 17. John St., M., Harris, W., Osga, G.A.: Designing for multitasking environments: Multiple monitors versus multiple windows. Proceedings of Human Factors and Ergonomics Society. 41, 1313–1317 (1997) 18. St. John, M., Manes, D.I., Oonk, H.M., Ko, H.: Workspace Control Diagrams and HeadMounted Displays as Alternative to Multiple Monitors in Information-Rich Environments. In: Proceedings of Human Factors and Ergonomics Society, 43 (1999) 19. Tan, D.S., Czerwinski, M.: Effects of visual separation and physical continuities when distributing information across multiple displays. In: Proceedings of INTERACT, pp. 252– 265 (2003) 20. Tullis, T., Albert, B.: Measuring the user experience. Morgan Kaufmann, San Francisco (2008) 21. Valleta, R.: Computer use and the U.S. Wage Distribution, 1984-2003. FRBSF Working Paper 2006-34, pp. 32-39 (October 2006)
Interface Evaluation of Web-Based e-Picture Books in Taiwan Pei-shiuan Tsai1,2 and Man-lai You1 1 Graduate School of Design National Yunlin University of Science & Technology, Taiwan 2 Department of Early Childhood Education Taipei Municipal University of Education, Taiwan
[email protected],
[email protected]
Abstract. Web-based e-Picture books can integrate the elements of multimedia and offer the special reading experience that is different from printed picture books. The research aims to understand the development status of three epicture book websites in Taiwan and give recommendations for improvement. Through 1) 12 adults who filled out the questionnaires about browing e-Picture books; and 2) 10 surveyed (two teachers, two parents and six children) who were observed in their operation and given in-depth interviews, the research analyzed three interface designs of e-picture websites in Taiwan: “Guru Bear Parent-Child Common Reading Network: Dear Bear Reading Room”, “Kiddo Book” and “CCA (Council for Cultural Affairs) Children Cultural Center: Picture book Garden”. The results of analysis were: 1) most of them were flipping pattern; 2) primarily linear development; 3) less interactive; 4) most of them were adapted from physical printed books; 5) it is difficult for children to register additional account and install browsing software. The recommendations for future publishers and designers were: 1) increase the interaction of the story; 2) Make good use of multimedia interactive design elements; 3) Enhance user control; 4) Integrate e-Picture Book platform; and 5) create all-new e-Picture books. Keywords: e-Picture book, e-Storybook, e-Book, Usability.
1 Introduction After the launch of iPad by Apple, the e-Book market has reach climax. The e-Books in the current markets could mainly refer to the e-Book readers that every manufacturer has their own specifications. But most of the e-Books present themselves on the reader in the form of static image and text, or even only the electronization of printed books. So strictly speaking, there are few e-Books that use multimedia elements and interactive design to design exclusive contents for the eBook. Being not able to recognize the words, pre-school children usually look at the colors and patterns with high novelty. E-Picture book is drawing-based, supplemented G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 94–102, 2011. © Springer-Verlag Berlin Heidelberg 2011
Interface Evaluation of Web-Based e-Picture Books in Taiwan
95
by a small amount of text and could be a helper for cultivating reading habits. It combines multimedia elements plus convenient online platform to design e-Picture book. From the aspect of creation, production and browsing, it is a new try and challenge that could transmit and distribute by an economic and convenient way and provides reading contents that have different media expression and sensory stimulation. The research selected three e-Picture books websites in Taiwan to conduct interface design, evaluation and analysis and understand the current situation of development before providing recommendations for improvement.
2 Literature Review 2.1 Electronic Book An electronic book is a text and image-based publication in digital form produced on, published by, and readable on computers or other digital devices (Wikipedia, n.d.). The Chinese term “electronic book” is directly translated from English. In literature, Van Dam mentioned electronic books for the first time. In a broad sense, it means the media that stores and transmits the characters and pictures information through electronic channel (Lwo, 1995). Barker (1992) argued that the electronic book was used to describe new type of books that was different from traditional paper books. But like paper books, they were composed of pages. The difference was that each page of an electronic book was designed and dynamic electronic information. Electronic book could be considered an aggregation of multi-pages, responsive and lively multi-media (includes information of characters, picture or voice). A picture book is an art form that combines visual and verbal narratives in a book format. A true picture book tells the story both with words and pictures. Electronic picture book (or e-Picture book, EPB) is to present picture books in the electronic form including CD-ROM, WWW. The applied multi-media elements include characters, pictures, animations, voice, sound effects and music. It mainly operates through mouse and keyboard in user control (interactive operation pattern). The manipulation of mouse includes drag and click whereas the manipulation of keyboard I includes character enter and key enter. The source of story materials includes adaptation and creation. The e-Picture book of the present invention includes a plurality of pages graphically depicting or telling a story. The e-Picture book in the research means the web-based e-Picture book. 2.2 Usability Usability is the ease of use and learnability of a human-made object. Lazar (2006) highlights ease-of-use as an equally important usability consideration he also advocates for a balanced approach to Web design that allows for the appropriate use of media elements such as graphics, plug-ins, and animation. Schneiderman (1993) emphasizes consistency and predictability in interface design that provides for a high
96
P. Tsai and M. You
level of user control. Usability means that the people who use the product can do so quickly and easily to accomplish their own tasks. This definition rests on four points: (1) Usability means focusing on users; (2) people use products to be productive; (3) are busy people trying to accomplish tasks; and (4) users decide when a product is easy to use. (Dumas & Redish, 1999) Usability is the quality of attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process. Usability is defined by five quality components: (1) Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design? (2) Efficiency: Once users have learned the design, how quickly can they perform tasks? (3) Memorability: When the users return to the design after a period of not using it, how easily can they reestablish proficiency? (4) Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors? (5) Satisfaction: How pleasant is it to use the design? (Nielsen, 2003) In conclusion, a usability analysis can tell us which part of page users view first, how long they stay and where they go next. Usability is about analysing how a user interacts with a website and using that information to make the website as userfriendly as possible.
3 Method 3.1 Procedure 1. Browse separately assigned e-Picture books (two for each type of each website). 2. Conduct questionnaire surveys and interviews through five-point Likert items: a. strongly disagree; b. disagree; c: neither agree or disagree; d. agree; e. strongly disagree. The contents of questionnaire include the satisfaction with the items such as overall page design, convenient to find e-Picture books, easy to operate, appropriate size of footage of story, animation design of story, suitable for children on their own operations. The primary contents of interviews focused on the questions on the e-Picture operations in different websites to understand the reason of satisfaction or dissatisfaction. 3.2 Subjects 1. 12 adults filled out the questionnaires (their education background included ten graduated from the children education related departments, two from the department of the Chinese language and literature; the occupations included three mothers who had young children, two kindergarten teachers, two elementary school teachers, five early childhood education educators). 2. 10 people included six children (one 10-year-old boy, one 10-year-old girl, two 7year-old boys and two 7-year-old girls), two kindergarten teachers, two parents were received one-to-one in-depth interviews on the e-Picture books they had browsed. Among them, four adults were picked up from the adults who had filled out the questionnaires.
Interface Evaluation of Web-Based e-Picture Books in Taiwan
97
Table 1. Comparison of three e-Picture book websites in Taiwan Guru Bear Parent - Child Read Together Net
Kiddo book
Website
http://www.gurubear.com.tw/
http://www.kiddobook.com/
http://children.cca.gov.tw/garde n/
Establishment Time
2008.07
2006.12
1999.09
Membership Registration
9
9
咕嚕熊親子共讀網:熊熊閱讀室
Install Browsing Software Fee Trial Subscription
奇豆線上書房
CCA Children Culture Hall Picturebook Garden
兒童文化館:繪本花園
9 9 Need credit for reading and earn credit for publishing article 9 3 trial subscriptions per category Inactive English Picture book
E-Live Picture book
9 NT$290/month
Free
9 6 books trial subscription
The contents of trial subscription is not complete version
Electronic Book
Story Amount
24
27
35
111
144
Story Voice
9
9
9
9
9
Story Animation
Good
Better
None
Limited
Better
yGame zone yThree-dimensional play zone yCreation zone
yGame (each story) yReading guide (each story) yDiscussion (each story)
Extension Activities
yInteractive Learning
2011.01.31 Update
Fig. 1. Example of Inactive English Picturebook
Fig. 2. Example of Kiddo book
Fig. 3. Example of CCA Children Culture Hall - Picturebook Garden
98
P. Tsai and M. You
4 Results and Conclusions Summing up the results of questionnaire and interviews and list as follows: 1. The Design of Website Homepage. The three websites get not bad evaluation. The average scores are above 3.6. 2. The Design of Interface The e-Picture book of “Kiddo Book” needs extra installment of browsing software and has more options of interactive functions, the average score of manipulation design is the lowest in all websites. The interface design of “Kiddo Book” is considered by users as the best one. It provides switching of previous/next pages, options of different captions (English/Chinese/Chinese plus phonetic notation), options of text location, play/pause, automatic/manual, etc., that is, it provide more user’s control. 3. the Screen Size of e-Picture book The screen size of “Children Culture Hall” and “Guru Bear Parent - Children Read Together Net - Interactive English Picture book” are the smallest and their average scores are the lower. The e-Picture book of “Kiddo Book” is playing in almost full screen and get highest score 4.7. 4. the Design of Story Animation The story animation of “Guru Bear Parent - Children Read Together Net – E-Live Picture Book” and “Children Culture Hall” has more camera effects and dynamic expression of characters and gets higher scores. 5. Story Voice “Guru Bear Parent - Children Read Together Net – E-Live Picture Book” and “Kiddo Book” are most highly praised by interviewers. The dubbing of different roles is different. 6. Whether suitable for children to self-manipulate “Kiddo Book” needs extra installment of browsing software, so it obtains the lowest score 2.4. “Children Culture Hall” need not to register user’s name and most easy to manipulation, so it obtains the highest score 4.2. Four 7-year-old interviewee children indicated that it is difficult for them to register user’s name and had no such experiences. It is also difficult for all children to install software for viewing the picture books. As a whole, there were following common points in the designs of e-Picturebooks: a) flipping pattern; b) linear story; c) insufficient interaction; d) most of them were adapted from printed books; e) needed extra registration account number and installment of browsing software that was difficult for children. So far, there have not yet been the most satisfactory design for web e-Picture book and there still are much space for improvement.
Interface Evaluation of Web-Based e-Picture Books in Taiwan
99
Table 2. Analysis of of Questionnaire Statistics N=12
Mean
Q1-1
4.0
Q1-2
3.6
Q1-3 Q2-1
S.D.
Min
Max
0.603
3
5
0.900
2
5
3.8
0.622
3
5
3.4
0.996
2
5
Q2-2
3.3
0.754
2
4
Q2-3
3.8
0.577
3
5
Q3-1
3.3
0.622
2
4
Q3-2
4.1
0.515
3
5
Q3-3
3.8
0.452
3
4
Q3-4
3.0
0.739
2
4
Q3-5
4.0
0.739
3
5
Q4-1
2.5
0.798
1
3
Q4-2
3.0
0.853
1
4
Q4-3
3.8
0.754
2
5
Q4-4
4.7
0.492
4
5
Q4-5
2.6
0.900
1
4
Q5-1
3.1
0.669
2
4
Q5-2
4.0
0.953
2
5
Q5-3
2.7
0.888
1
4
Q5-4
3.6
0.996
2
5
Q5-5
4.0
0.739
3
5
Q6-1
3.2
0.937
1
5
Q6-2
3.8
0.866
2
5
Q6-3
3.9
0.515
3
5
Q6-4
2.4
0.996
1
4
Q6-5
4.1
0.793
3
5
5 Suggestions 1. Increase the interaction of story The e-Picture books in the three websites were adapted from physical printed books that the publishers operated them for nothing more than the added-values of original publications. Most of story contents are linear developments that lack of interaction. In future, they should bring the advantages of multimedia and web into the full play and increase double or multi-routes of story development to strengthen the interactions between the readers and story contents.
100
P. Tsai and M. You
2. Make good use of multimedia interactive design elements The e-Picture book should emphasize more on the use of multimedia interactive design and allow the readers have the reading experiences that are different from the printed books. Unfortunately, the performance on the multi-media by existing ePicture books in three websites was not, on the contrary, good as Living Books series. So they should make good use of the advantages of web version to interact with their readers to extend reading activities. 3. Enhance user control “Kido Book” provides the function options including page up and down switch, different subtitle options (English/Chinese/Chinese plus phonetic notations), options of text location, play/pause, auto/manual play. It provides the most user controls in the three websites. Most of the surveyed recommend that it increase the control options of the size of screens and subtitles that the users could decide by themselves the size of picture and subtitle, that is, it enhances the users’ control power. 4. Integrate e-Picture Book platform At present time, most of e-Picture Books websites in Taiwan are operated by the publishers, some even need to install browsing software for reading that is not convenient to the readers. In future, they should construct an integrated platform that the e-Picture Books from different publishers could be read by the users at the same time. It would not need to set different accounts and could further to reach the goal of mutual exchange and healthy competition. 5. Creative All-new e-Picture book Most of the existed e-Picture books came from the adaptation of physical printed picture books that is the digitalization of existed publications. They usually use scanning method to process the images or part of images were processed through simple animation and nothing else. It is pity that they did not consider the creation of e-Picture books from the angle of multimedia elements and interaction. So the creation and publication of all-new e-Picture books should be encouraged.
References 1. Barker, P.: Electronic books and libraries of the future. The Electronic Library 10(3), 139–149 (1992) 2. Dumas, J.S., Redish, J.C.: A practical guide to usability testing. Intellectual books, Portland (1999) 3. Lazar, J.: Web usability: A user-centered design approach. Pearson Education, Inc., Boston (2006) 4. Lwo, L.-S.: Electronic books and the new communication era. Instructional technology and media, 21, 13-16 (1995) (Chinese) 5. Nielsen, J.: Usability 101: Introduction to Usability (2003), http://www.useit.com/alertbox/20030825.html (Retrieved December 20, 2008) 6. Schneiderman, B.: Leonardo’s laptop: Human needs and the new computing technologies. The MIT Press, Cambridge (2002) 7. Wikipedia. E-book (n.d.). Retrieved, from http://en.wikipedia.org/wiki/E-book
Interface Evaluation of Web-Based e-Picture Books in Taiwan
101
Appendix: The Results of Questionnaire Statistics Questions I like the website design of “Guru Bear Parent-Child Read Together Net”
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 Ave S.D. 4
4
4
4
3
3
5
4
4
4
4
4
4.0 0.603
I like the website design of “Kiddo Book”
4
4
5
2
3
3
4
5
3
4
3
4
3.6 0.900
I like the website design of “Children Culture Hall” I can find the e-Picture book of “Guru Bear Parent-Child Read Together Net” easily I can find the e-Picture book in “Kiddo Book” easily I can find the e-Picture book in “Children Culture Hall easily I feel easy to manipulate “Bear Reading Room –– Interactive English Picture book” in “ Guru Bear Parent-Child Read Together Net” I feel easy to manipulate “Bear Reading Room –– E Live Picture book” in “Guru Bear Parent-Child Read Together Net” I feel easy to manipulate “Bear Reading Room – Electronic Picture Book” in “Guru Bear Parent-Child Read Together Net” I feel easy to manipulate “Kiddo Study Room” in “Kiddo Book” I feel easy to manipulate “Picture book Garden” in “Children Culture Hall” I feel the screen size of “Bear Reading Room –– Interactive English Picture book” in “Guru Bear Parent-Child Read Together Net” is appropriate I feel the screen size of “Bear Reading Room –– E Live Picture book” in “Guru Bear Parent-Child Read Together Net” is appropriate. I feel the screen size of “Bear Reading Room—Electronic Picture book” in “Guru Bear Parent-Child Read Together Net” is appropriate I feel the screen size of “Kiddo Study Room” in “Kiddo Book” is appropriate I feel the screen size of “Picture book Garden in “Children Culture Hall” is appropriate I like the story animation of “Bear Reading Room –– Interactive English Picture book” in “Guru Bear Parent-Child Read Together Net” I like the story animation of “ Bear Reading Room –– E Live Picture book” in “Guru Bear Parent-Child Read Together Net” I like the story animation of “Bear Reading Room—Electronic Picture book” in “Guru Bear Parent-Child Read Together Net” I like the story animation of “Kiddo Study Room” in “Kiddo Book” I like the story animation of “Picture book Garden” in “Children Culture Hall”
4
4
4
4
3
5
3
3
4
3
4
4
3.8 0.622
2
4
4
4
2
4
5
2
4
4
3
3
3.4 0.996
4
3
4
2
2
3
4
4
3
3
3
4
3.3 0.754
4
4
3
5
4
4
3
4
4
4
4
3
3.8 0.577
4
3
4
3
3
4
3
2
3
4
3
3
3.3 0.622
4
5
4
4
3
4
4
5
4
4
4
4
4.1 0.515
4
4
4
3
3
4
4
4
4
4
4
3
3.8 0.452
4
3
4
2
3
3
2
2
3
4
3
3
3.0 0.739
4
4
3
5
4
5
3
4
4
5
3
4
4.0 0.739
2
3
4
3
3
2
2
1
2
3
2
3
2.5 0.798
3
3
4
4
3
2
4
1
3
3
3
3
3.0 0.853
2
4
4
5
4
3
4
4
4
4
3
4
3.8 0.754
5
5
5
4
4
5
5
4
5
5
4
5
4.7 0.492
4
3
3
3
2
2
2
1
2
4
2
3
2.6 0.900
4
3
3
4
3
4
2
2
3
3
3
3
3.1 0.669
4
5
5
5
4
2
5
4
4
4
3
3
4.0 0.953
3
4
4
3
2
3
3
1
2
2
3
2
2.7 0.888
5
5
4
3
2
5
3
3
3
3
4
3
3.6 0.996
4
3
3
4
4
4
4
5
5
5
4
3
4.0 0.739
102
P. Tsai and M. You
Questions I think it is suitable for the elementary school children to self-manipulate “Bear Reading Room – Interactive English Picture book” in “Guru Bear Parent-Child Read Together Net” I think it is suitable for the elementary school children to self-manipulate “Bear Reading Room –– E Live Picture book” in “Guru Bear Parent-Child Read Together Net” I think it is suitable for the elementary school children to self-manipulate “Bear Reading Room—Electronic Picture book” in “Guru Bear Parent-Child Read Together Net” I think it is suitable for the elementary school children to self-manipulate “Kiddo Study Room –– Electronic Picture book” in “ Kiddo Book” I think it is suitable for the elementary school children to self-manipulate “Picture book Garden –– Electronic Picture book” in “ Children Culture Hall”
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 Ave S.D. 3
4
4
5
3
3
3
1
3
3
3
3
3.2 0.937
2
4
4
5
3
4
4
5
4
4
3
3
3.8 0.866
3
4
4
5
3
4
4
4
4
4
4
4
3.9 0.515
2
4
4
2
3
2
1
1
2
3
3
2
2.4 0.996
5
4
4
5
3
5
3
4
4
5
3
4
4.1 0.793
A Digital Archive System for Preserving Audio and Visual Space Makoto Uesaka, Yusuke Ikegaya, and Tomohito Yamamoto College of Information Science and Human Communication, Kanazawa Institute of Technology 7-1 Oogigaoka, Nonoichi, Ishikawa, 921-8501 Japan
[email protected]
Abstract. A digital archive system has been widespread in various fields because it can preserve precious cultural heritage, books, pictures or videos without any deterioration. Moreover, preserving its information on the web, a digital archive system can share a lot of things between general users, and can pass them down new generation easily. In this research, we focus on spatial information of a place or an event which can provide high presence and retrieve personal memories, and develop a digital archive system which can preserve such kind of spatial information. Keywords: Digital Archive, Omnidirectional image, Multi-channel audio, Spatial information.
1 Introduction A digital archive system has been widespread in various fields because it can preserve historical and cultural heritage such as paintings, pictures or sculptures without any deterioration [1]-[4]. Moreover, by providing its data on the web, a digital archive system realizes to share a lot of things between general users, and pass them down next generation. However these digital archives, especially archives provided on the web are likely to be composed of a flat picture and stereo sound. As a result, users can not get as same reality or presence as real things. In the field of Virtual Reality (VR), to solve this problem, some researches preserve historical heritage by high realistic way and display them by special display system. For example, Abe et. al. have preserved “Maijishan Grotto” in China by stereo camera and displayed them by stereo graphics [5]. In addition to such the research, not only precious historical heritage but also daily life of all around the world has been archived. For example, Watanabe et. al. have archived personal data of people in Tuvalu and reported daily life of Tuvalu where has suffered from sea surface elevation by global warming [6]. In the research of the digital archive of historical heritage, it is possible to preserve and display them in high realistic way. However such the system tends to need expensive equipments for measuring and displaying. Moreover, the high realistic G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 103–110, 2011. © Springer-Verlag Berlin Heidelberg 2011
104
M. Uesaka, Y. Ikegaya, and T. Yamamoto
display tends to need large space for setting up and does not have mobility. As a result, only precious historical heritage can be archived in the system. On the other hand, in the research of the digital archive of daily life, it provides only photograph, video or text on the web. Therefore it is difficult to feel the reality of peoples’ daily life deeply. In this research, we develop the digital archive system which provides high realistic information without any expensive equipments but mobile devices. In our system, archived data are provided from website, and which can display an omnidirectional image using WebGL and stereo sound using HTML 5. Users also can download the archived contents into our audio-visual display [7], and enjoy them in high realistic way. Moreover, for the archive system, we shoot some contents which are not historical heritage but personally or locally important scenery and space.
2 A Digital Archive System 2.1 System Overview Our digital archive system is composed of two components (Fig.1). One is web system which provides archived contents. The other is spatial audio-visual display which realizes to reproduce downloaded contents. Users can enjoy archived contents by the procedure below; 1. Users access our web site from local PC. 2. Selecting an archive from lists, and users preview it on the browser. 3. Downloading a preferred archive to local PC, and users set up archived data into our display system and enjoy them. Server side of the web system generates a web page dynamically to client side. Client side realizes to preview and download contents and so on. The spatial audio-visual display is composed of head mount display (HMD) and multiple mobile devices, and which can reproduce downloaded contents by high realistic representation.
Fig. 1. System Overview
A Digital Archive System for Preserving Audio and Visual Space
105
2.2 Client and Server Side of the Web System Fig.2 shows a top page of a web site. The top page has a function that realizes to search contents from map of Hokuriku area in Japan, or by title, tags and created date. Fig.3(a),(b) show preview images of archived contents. An image of Fig.3(a),(b) derived from a same omnidirectional image but different view angle. In this preview mode, users can see archived space and listen to archived surround sound easily. In this system, archived image is mapped into sphere model, and virtual camera is arranged in center of sphere by using WebGL. If users want to see omnidirectional view, users just click and drag on the displayed image. Moreover, sound data correspond to movement of omnidirectional view (sound data have its own position data in the sphere). If user moves the direction of view, direction and volume of sound also are changed by using API of HTML5.
Fig. 2. Top page of web site
(a)
(b) Fig. 3. Preview page
106
M. Uesaka, Y. Ikegaya, and T. Yamamoto
Server side of web system is composed of Fedora for the OS, MySQL for the database server, Apache for HTTP server. In server side, generating dynamical page, uploading and downloading process and searching function are implemented using PHP. 2.3 Spatial Audio-Visual Display System To enjoy archived contents in high realistic way, users download contents in local PC, and set up them into our audio-visual display system [7]. Our system is composed of multiple terminals such as a PC or an iPhone as shown in Fig.4. Each terminal is connected to LAN and these are divided into one server, one visual client and multiple sound clients. In this system, at first, the server simulates an archived space. After simulation, the server transmits the information of the virtual space to terminals by UDP multicast. Then, terminals present the high presence to a user using received information. To reproduce spatial sound, sound clients calculate distance decrement of sound using three positions (a listener’s position, a sound source’s position and a sound client’s position). Using these positions, the system calculates the volume in real time and reproduces a spatial sound by the volume difference of each client. To represent spatial view, visual client uses a head mount display and a webcamera. For presenting spatial view, the visual client detects user’s posture by head tracking sensor, and reflects it to viewing angle in the virtual space. This system is basically implemented as a software level, and allows any device to become the terminal if the device can be connected to network. Therefore, using devices such as desktop, laptop, or smart phone which have been widespread, our system can be built relatively inexpensive.
Fig. 4. Spatial audio-visual display
A Digital Archive System for Preserving Audio and Visual Space
107
3 Contents Digital archive systems developed so far have mostly focused on highly valuable heritage such as historical sculptures or fine arts. One reason why such heritage has been archived is that it has been very expensive to create such archives. However in the world, there are a lot of precious things to archive such as rural beautiful scenery, local festival or events, or personally valuable space. Some of these scenery, events tend to exist in a moment, or tend to fade away if population of its area decrease in the future. In this research, we focus on such things and archive them in low cost way, and provide in highly realistic way. Concretely, in this research, we have archived beautiful rural scenery of Hakusan area in Ishikawa, and daily student life of Kanazawa Institute of Technology. In Hakusan area, a population decreases year by year, because there are few jobs in the area. Therefore some villages will be vanished in the future, and beautiful rural scenery called “Satoyama (a very special type of natural environment that cannot exist without moderate intervention of human beings)” also will be vanished. To archive such rural part contributes to preserve the area itself, and if the scenery will be lost, it is possible to tell how this area was to next generation. Students have a lot of enjoyable, miserable or emotional experience in their student life. Ordinary, they can remember fragments of such memories by photograph or video. If our system provides where they were or how they were in high realistic way, they can experience as if they came back to their student life. Moreover, after student became a parent, they can tell children their student life. 3.1 Making an Omnidirectional Image and Video Fig.5(a) shows all of shooting kit of an omnidirectional image (a general digital camera, a panoramic camera mount and a tripod). To make an omnidirectional image, at first, it is necessary to take pictures by rotating a camera mount by 30 degree. Next, to angle a camera mount up and downward, and take pictures by rotating a mount. Finally, to take a picture of the top. By using these taken pictures (about 40 pictures), an omnidirectional image is created by editing soft (Panoweaver). Fig.6 shows pictures before editing and Fig.7 shows a created omnidirectional image. To make an omnidirectional video, Ladybug2 (PointGreyResearch) is used. Fig.5(b) shows shooting kit. This 6 lens camera can create omnidirectional video at real time (4.7mega pixel per image). 3.2 Making Spatial Sound In this research, R-09HR (Roland) and a general headphone were used to record sound. However, if it is possible to record sound source clearly in WAVE format, any device is acceptable. Recorded sound sources are placed in virtual sphere in which a created omnidirectional image is mapped. After that, audio data and visual data integrated into one file set (contents information is described in XML format). When reproducing contents, our audio-visual display reads its XML data and simulates archived space.
108
M. Uesaka, Y. Ikegaya, and T. Yamamoto
(a)
(b)
Fig. 5. Shooting equipments of an omnidirectional image and video
Fig. 6. Unidirectional pictures
Fig. 7. An omnidirectional Image
A Digital Archive System for Preserving Audio and Visual Space
109
3.3 Examples of a Digital Archive Fig.8-10 show examples of archived contents. Fig.8 shows old Japanese houses called “Gasshou zukuri” which are composed of woods and straw. It is necessary a lot of cost to maintain these houses. However, in this area, young generation go out cities to get jobs, and only elder generation stay. Therefore it is very difficult to keep this scenery in the next generation. Fig.9 shows Satoyama in Hakusan. Satoyama is very popular Japanese scenery in the old days. In Satoyama, famers cultivated their vegetables and cut firewood for stove in winter. However such famers are rare nowadays and Satoyama vanishes year by year.
Fig. 8. An archive of “Gasshou zukuri” houses in Gokayama
Fig. 9. An archive of “Satoyama” in Hakusan
110
M. Uesaka, Y. Ikegaya, and T. Yamamoto
Fig.10 shows students’ project “Tukimi Kouro” of Kanazawa Institute of Technology. In this project, students light up various places in Kanazawa city by handmade lumps. All of students made a lot of effort in this project. Therefore, this archive would remember their students’ life vividly.
Fig. 10. An archive of “Tsukimi Kouro” at Ishiura Shrine in Kanazawa
4 Conclusion In this research, we developed a digital archive system which realize to preserve high realistic spatial information and to provide it at low cost. Moreover we created some local and personal archived contents. In future works, we will increase contents and release the web sites in public. Moreover we will develop the web system to create an omnidirectional view and surround audio for general users to create high realistic spatial archives.
References 1. 2. 3. 4.
Nagasaki Archive, http://nagasaki.mapping.jp/ World Digital Library, http://www.wdl.org/en/ National Archives of Japan, http://www.digital.archives.go.jp/ Daijyouji Temple Digital Museum of the Maruyama School, http://museum.daijishan.or.jp 5. Abe, N., Kawai, T., Ohya, J., Zha, H., Ando, M.: Digital Archiving of Maijishan and Stereoscopic VR Content. TVRSJ 4(3), 275–282 (2009) (in Japanese) 6. Suzuki, M., Watanabe, Y., Endo, S., Watanabe, H.: Tuvalu Visualization Project. SIGGRAPH ASIA 2009, Sketches, Article No.259 (2009) 7. Takahashi, K., Yamamoto, T.: 3D Audio-Visual Display Using Mobile Devices. In: ACM SIGGRAPH (2010) (Posters)
Experience Explorer: Context-Based Browsing of Personal Media Tuomas Vaittinen1, Tuula Kärkkäinen2, and Kimmo Roimela3 1
Nokia Research Center, P.O. Box 407, FI-00045 Nokia Group, Finland
[email protected] 2 Tampere University of Technology, Unit of Human-Centered Technology, P.O. Box 589, 33101 Tampere, Finland
[email protected] 3 Nokia Research Center, P.O. Box 1000, FI-33721 Tampere, Finland
[email protected]
Abstract. We designed and built a system for browsing digital content and activity data created and gathered with mobile phones. We evaluated the system with 13 users to study the value of the context-based visualizations in real life. In addition to supporting reminiscing, content aggregated on the map revealed life patterns supporting reflection. Aggregation of items from several people also revealed common interests among friends. Keywords: Personal content, context, lifelogging, self reflection, user study.
1 Introduction People want to be able to reminisce and reflect on the important events and every day incidents in their lives. Modern technology supports recording of a wide variety of content types. Furthermore, a mobile phone can be used for recording the user's context as well. In addition to private reminiscing, the social uses of lifelogs have been identified as important [1], but little research has been carried out on them. In this paper we present a system for browsing personal media, enhanced by using context information recorded by a mobile phone for organizing the content items. The system is designed for browsing the user’s own content as well as content that the user's friends have shared with him or her. In addition, we describe a user trial that studied what kind of value the gathered content and one’s recorded context, visualized in relation to friends, time and location, would provide for users.
2 Related Work 2.1 Previous Lifelogging Systems The ability to store all the relevant documents, photos and messages during one's life has enticed people from the days of Vannevar Bush [2]. His iconic article described a vision of Memex, a desk-sized tool for storing documents on microfilm and searching G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 111–120, 2011. © Springer-Verlag Berlin Heidelberg 2011
112
T. Vaittinen, T. Kärkkäinen, and K. Roimela
them effectively. Since then, several “memex”-like approaches have also been presented for desktop computers; MyLifeBits, for example, stores all the viewed photos, read documents and listened audio [3]. Typically, lifelogging systems focus on browsing the user’s private content, and do not include features for sharing the content. Furthermore, early lifelogging systems tend to make only limited use of the information about one’s social interactions. Current mobile phones are exceptionally well suited for recording the owner’s life events. In addition to being almost always carried, a mobile phone includes a considerable amount of information about the owner’s communications and social connections. In addition to content and activities, a mobile phone suits well for recording the context of the user. Context can be defined as any information that characterizes the situation of, for example, a person [4]. As an example, a system called Affective Diary records the user's movement, level of arousal and names of Bluetooth devices around the user [5]. It combines this context information with content, like photos and SMS messages, to create visualizations of the user’s activity [5]. 2.2 Browsing the Lifelogs While storing digital memories automatically, the vast amount of the content quickly becomes a challenge. The items valuable for reminiscing become easily obscured with the irrelevant items [6]. When looking for means to cope with that, people and location have been identified as important memory cues in searching photos [7]. Furthermore, people tend to group their photos based on events and later on they rely on the event grouping when browsing the photos [8]. The strong tie between the content and the event in which it has been captured suggests that browsing should be supported simply by automatically grouping the content by events [9]. Timeline view for presenting stored content about one's life is a natural and well established approach used in many research systems such as MyLifeBits [3], LifeLines [10], Lifestreams [11] and Stuff I’ve Seen [12]. While the basic idea of the implementations is the same, different visualization methods and additional information have been used to increase the value of the view. LifeLines uses the thickness and color of the line to indicate the significance of the events presented on the lines [10]. Stuff I've Seen displays search results with a preview of each item in chronological order, indexing the results with landmark events from, for example, the user's personal calendar [12], [13]. 2.3 Value of Lifelogs Photos are a common trigger for reminiscing and lately one of the most popular research approaches to lifelogging is the continuous visual capturing of one’s life with a wearable camera [14], [15], [16]. The findings of Kalnikaité et al. [16] suggest, however, that different data types and views to the data support different types of remembering, from detailed recalling to inferring the past events and habitual patterns. They suggest that image data should be the cornerstone of lifelogs, but richer recordings about the past should be collected. Lifelogs can support general recalling of emotionally valuable events, but they have been shown to help in self-reflection as well [17].
Experience Explorer: Context-Based Browsing of Personal Media
113
Digital memories are often shared with the people who were present at the time of capture [17]. Hence, the information should be shown and provided as a preset when selecting recipients for sharing. Recently, popular photo browsing applications like iPhoto and Picasa have incorporated face recognition to enhance the photos with information about who appear in them. Although the described type of content analysis works for photos, it requires some manual work from the user, and with a variety of media types, different methods are needed for determining who was there.
3 Experience Explorer We built a system called Experience Explorer (ExEx) for recording and browsing digital memories and visualizing one’s life history in relation to friends. The system is designed for browsing the user’s own content and content shared to him or her by friends. The information about the user’s context and interactions with friends is used to support event-based finding of content, to invite reflecting on one’s past actions, and to provide an overview of one’s social life. The possibilities to follow other users’ life too closely are, however, minimized. Discussion on the general privacy issues of the logging system has been published separately [18]. 3.1 System Components The system consists of three main components: (1) a persistent context logging client on the mobile phone, (2) a central server hosting the data collected with the logging client, and (3) a desktop web application for browsing the collected data. For logging the context data, we use Nokia Simple Context Collector. Its client runs as a background task in the user’s S60 phone and periodically samples context data such as GPS location, network cell ID, and the surrounding Bluetooth (BT) and Wireless Local Area Network (WLAN) device environment. In addition, it collects music listening data, call log information and sent and received SMSs. All the data is uploaded to a network server, from where our central server fetches it and stores the resulting context information in a database. In addition to the context data, our server scans selected third-party services, such as Flickr, for personal digital media uploaded by the users. The metadata about each content item is harvested and stored into the same database as the context data. The original content remains on the third-party servers, but details such as the title, description, and thumbnails for visual content are stored in our database for indexing. A more detailed description of the platform has been published separately [19]. Having the joint dataset of context and content metadata, we apply a number of algorithms to determine relations between the users and the content items. 3.2 User Interface The UI of ExEx is implemented in Adobe Flex and has three main views: the lifethreads, map, and item info views. The lifethreads and map views are used for browsing content previews and for selecting items to be viewed individually in the item info view or to be shared within the system. The content item types supported by
114
T. Vaittinen, T. Kärkkäinen, and K. Roimela
the system are photos, videos, listened-to music tracks, text messages, phone calls and location tracks (i.e. sequences of stored GPS coordinates). Lifethreads View. The lifethreads view presents items in chronological order on user’s own and his or her friends’ lifethreads. User's own lifethread is drawn vertically on the left, with his or her items laid on the line (see Fig. 1). The more there are items created with similar timestamps, the wider the user’s lifethread is at that point. Friends' lifethreads (in the middle and on the right in the Fig. 1) are displayed next to the user’s lifethread, with items in similar vertical chronological order. The horizontal position of the friends’ lifethreads at each point of time depends on the friend’s distance from the user at that time. The user has several options for selecting the distance criteria used in the visualization: physical distance visualizes how far the user’s friends have been from the user geographically, communication activity shows how much the user has communicated with each of the friends, and music taste shows how similar the user’s music taste has been with the friends during the viewed time. When using the physical distance as the closeness criteria, the approach aims to visually group together all the items created in a same event, providing a more complete picture of a shared experience than a single user’s items would.
Fig. 1. Lifethreads view
In the case of photos and videos, the items are presented as thumbnails, and for the rest of the media types, as icons. By holding the mouse cursor over an item, a tooltip is displayed. The items can be selected for closer viewing in the item info view by double clicking them. Map View. The map view lays the items out on top of map graphics to the locations they were created. The location is induced from recorded context data, matching it to the creation time of the item. As the user’s context is continuously recorded, the location can often be deduced on the basis of the recorded GPS data at or around the
Experience Explorer: Context-Based Browsing of Personal Media
115
creation time. When the GPS signal is blocked, the system reverts to the WLAN access points and the mobile network cell IDs. For both of these, it determines the location of each base station based on the GPS data recorded, by any user, while within the range of that base station. Items are displayed as thumbnails or icons similarly as in the lifethreads view. To avoid filling the map with hundreds of overlapping paths, GPS tracks are shown as icons, instead of drawing the full path on the map. Item Info View. The item info view displays a larger version of the item in question and the metadata related to it. The location of the item is shown on a small map below the item. Also, a list of friends who were present at the time of creation is displayed. Next to the friends, thumbnails and icons of related items from oneself and others are displayed. The content shown in the related content is selected using multiple criteria: geographical proximity, similarity of creation time, similarity of surrounding Bluetooth context, and the number of similar tags or keywords. Each item is scored based on the sum of these individual criteria, and the results are presented with the highest-scoring item first. Privacy Considerations. The intention was to provide as rich information about one’s social connections as possible within the acceptable limits of privacy. A user’s exact location at a certain time is visible only for those friends with whom the user has explicitly shared a GPS track or an item with location information. The location of a user is, however, used for the visualization of the lifethreads showing the user’s distance to one's friends, and for listing the friends present at an item creation.
4 Evaluation A user trial was conducted in Finland to evaluate the system. It focused on the reminiscing and self-reflection value that the context-related UI components provided users as well as on gaining insight to the optimal ways to present the information. 4.1 Method The study consisted of two phases: 6 to 9 weeks of gathering content for browsing and 8 to 12 days of using the full system including the ExEx web application for browsing the content. To provide the users with enough content for reminiscing and reflecting on their lives, the content collection and logging started early. The moment the implementation of the web application reached the level of quality to deploy to the users, the second phase started. During the study, the users had Nokia Simple Context Collector running on their mobile phones. They also used ShoZu (www.shozu.com) for tagging the photos and videos with GPS coordinates and uploading them to Flickr, which was used for storing the users’ photos. During the first phase, participants could browse their photos and videos only in Flickr, but in the second phase all their stored content was automatically visible to them in ExEx as well The users were lent Nokia N95 8GB
116
T. Vaittinen, T. Kärkkäinen, and K. Roimela
multimedia phones and compensated for flat rate data plans for the duration of the study. Participants paid the rest of their phone bill, i.e. calls and SMSs, normally. The first phase started with a group interview. After the participants had their data plans activated, we met them again and gave the phones. The instructions were to use the phone as they would normally do, but upload the photos that they took to Flickr. In addition, if they were listening to music while on the move, they were asked to use the phone as their primary player. In the end of the phase, participants were met individually to introduce the desktop web application for browsing the content. They signed into ExEx with a laptop and were able to see their data for the first time. After describing how they interpret the different elements in the UI, small tasks were given to them. For example, they were asked to check with whom they had communicated most around the time the study started. The aim was to find out usability problems and to ensure that all users familiarize themselves with the core functionalities of the system. During the tasks, the users were asked to think aloud as much as they could. In the second phase, the users used ExEx individually for 8 to 12 days. The usage of the different UI components was logged. The participants filled in a diary about their usage and experiences. After this, the users were met individually for the last time for interviewing them on their experience on using the web application. The interviews were recorded and transcribed for analysis. The observations from the usage and diaries, as well as the interview quotes from the sessions were grouped and conclusions were made by reviewing the appearing salient themes. 4.2 Participants The participants were recruited via mailing lists. From the groups who signed up, we selected three groups of friends to participate in the study. Two groups had five participants and one had three so the total was 13 users. 11 of the users were male and two were female (the skew mostly due to a last minute cancellation by one of the groups). Their ages varied from 19 to 32 years (average of 23). 10 of the users were students and three participants were in working life (factory worker, IT consultant and surveying engineer). Participants were all active media consumers and familiar with social media sites. Two of them were active Flickr users already before the study and three had Flickr accounts but had just tried the service a couple of times. 4.3 Results By the end of the trial, the logging had been running 7.5 to 11 weeks depending on the user (including both phases of the trial). During that time, on average 1997 items were logged for each user. The differences between the users were quite large, as 3 of the users had less than 1000 items and one had more than 6000 items to browse. Differences were mainly caused by the varying music listening habits of the participants. Some users listened only to a couple of albums during the trial, whereas some listened to music many hours a day. Each user had on average 427 calls, 320 SMSs, 70 photos, 1.3 videos, 1087 music tracks and 91 location tracks to browse from themselves. In addition, they had some items their friends had made visible to them. The shared items were mainly photos, although some sharing experiments with the other media types were done as well.
Experience Explorer: Context-Based Browsing of Personal Media
117
According to the diary data, the participants used ExEx 2 to 5 times during the individual usage period. The sessions lasted from 5 to 20 minutes and each user's total usage time was between 21 and 70 minutes. In addition, the users used the system in the both interviews for about 45 minutes each. The actual usage was not as active as we had hoped for but the users were well familiar with all the features of the system and everyone had enough experience to reflect on their own needs for such systems. The considered benefits of the logging not only included utilitarian uses of one’s personal log, like reducing memory load, or personal reflection related issues like seeing where one has been and how one has been using the phone, but the information related to one’s friends was considered equally important. The following sections discuss in more detail the interplay of reminiscing, personal reflection and information about friends. In addition, the roles of individual items and aggregated visualizations are considered. The changes that the logging caused to the participants’ every day behavior is discussed separately in another paper [18]. Reminiscing. Lifelogging tools naturally lend themselves to reminiscing past events. The most evocative item types in our study were reported to be the photos and SMSs. Photos were mentioned by two thirds of the users and the SMSs by one third. The special value of the photos was highlighted by the usage logs as well. For example, 42% of clicks on related items in the item info view were done on images, twice as many as on the second most popular type: messages (20%). The chosen presentation style naturally had an effect on the value of the individual items. Seeing only the time and the name of the caller of a phone call did not trigger many memories, whereas some information about the contents of the call might have. In everyday life, content items and phone interactions clustered on the map were more intriguing for the participants than the pure recorded location tracks. However, when the location track recorded a memorable event, for example when taking a tour outside of one’s hometown and away from daily routines, the pure track also provided value for reminiscing. Having made the actions outside of one’s daily routines and circles added to the value of content as well. For example, users who had traveled abroad were especially interested in their actions from the trip shown on the map. ”I was so looking forward to seeing the phone calls and text messages there (on the map of the service). The ones that I made when I was abroad.” (female, 22 years) Self-reflection. Another potential benefit of lifelogging tools is the reflection of one’s habits they might enable. A single music track may have been listened to many times on many occasions, so displaying the plain item in the UI does not pinpoint a certain event for reminiscing as precisely as photos and messages do. However, due to the amount of music listening events logged in our study, they started to reveal participants’ daily life patterns valuable for reflection. The music and other routine actions like phone calls formed interesting and meaningful information when aggregated on map. For example, the frequent paths formed by these icons provided participants with an outsider’s view of their actions: ”This is really intriguing. You can actually see my route from my home to the city center. On my way to work, there are all these, music, some text messages, phone calls, a line is formed by these, what I’ve done. This is really nice.” (female, 22 years)
118
T. Vaittinen, T. Kärkkäinen, and K. Roimela
Friends’ Activities and Social Context. ExEx was designed not only to collect and show the user’s own actions but also to show the social context of one’s life and information about one’s friends’ lives. The overall importance of seeing friends' content in the system came up frequently in the interviews but was highlighted also by the number of clicks users made on friends' photos. 72% of times a user clicked a photo in the lifethreads or map views it was a photo from someone else than the user oneself. The lifethreads view with the physical distance criteria and the related items in the item info view both gathered the user’s and his or her friends’ items from the same event together. The features were valued by the users: “If there are photos from some event you get all of them conveniently since friends' items are attached as well” (male, 25 years) Two participants mentioned in the interview that they found something previously unknown that was common among them. For example, the lifethreads view with the music track items from friends helped noticing listened-to artists common to one’s friends. Again, aggregated visualization supported making the individual items meaningful in the web application. Lifethreads vs. Map. The lifethreads and map views were seen to support each other well. Most of the users considered the lifethreads view to be the most important view of the service. The map view was used about every time the service was used, but still less than the lifethreads view. According to the logs, 63 percent of the time spent in one of the two modes was spent in lifethreads view. It was described as an easy way to get a general view of the recent activity. According to diaries and interviews, users were checking their own activity (especially music listening), whether their friends had been close, and if someone had shared new photos with them. “Well, I start always from checking a couple of last days, has anyone been near me or published some photos recently or something” (female, 21 years) 4.4 Discussion and Future Work Although we were able to collect data about the value the logged data and ExEx visualizations provided users, many issues would benefit from further studies. Sharing happened mostly on photos so a long-term study would be needed to get a better view of, for example, issues in location sharing. Sharing of tracks happened especially while traveling abroad, suggesting self-presentational value, but more data should be collected to get a more complete picture of social uses of lifelog data. The question of real-time sharing also came up, since one participant noticed he had been waiting at the railway station on the same evening as his friend. Information about the past supports reminiscing, but viewing and sharing of live information would support social activities. The slow response time of the lifethreads view and the occasional unpredictable changes in the closeness visualization were criticized and had reduced the motivation of some of the participants to use the system more extensively. However, the view to one's social activity clearly provided value, and insights on implementing similar visualizations were gained.
Experience Explorer: Context-Based Browsing of Personal Media
119
5 Conclusions In this paper, we discuss how automatically captured context data can be used in the UI for organizing personal content in a meaningful way, as well as in visualizing one’s life history in relation to friends to support reminiscing and self reflection. In our study, items with evocative contents, like photos and SMSs, were inherently valuable for reminiscing of singular events. Furthermore the aggregation and locationbased presentation of the content increased also the value of logged actions, like phone calls and music tracks. When aggregated on the map they revealed life patterns supporting reflection. Since the system allowed sharing of items with friends, the reminiscing of joint experiences could be based on items from several participants. The results show that lifelog information recordable with current mobile phones visualized in appropriate ways would not only support reminiscing, but also assist users to realize new things about their own life patterns and themselves. Moreover, as social encounters and communication with others play an essential role in people’s lives, the information about the user’s friends was also highly appreciated in the study. Hence, taking into account the privacy needs, lifelogging systems should support sharing and showing content from friends as well. Acknowledgments. We thank Minna Wäljas,Joel Hakulinen, Tero Hakala, Joonas Itäranta, Ville-Veikko Mattila, Rich Hankins and the C3 team from NRC as well as Mahtava development for their contribution to the study. We would also like to thank Thomas Olsson and David Murphy for their insightful review of the paper.
References 1. Peesapati, S.T., Schwanda, V., Schultz, J., Lepage, M., Jeong, S.-y., Cosley, D.: Pensieve: supporting everyday reminiscence. In: 28th ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 2027–2036. ACM Press, New York (2010) 2. Bush, V.: As We Think. Atlantic Monthly 1, 101–108 (1945) 3. Gemmell, J., Bell, G., Lueder, R., Drucker, S., Wong, C.: MyLifeBits: Fulfilling the Memex Vision. In: 10th ACM International Conference on Multimedia, pp. 235–238. ACM Press, New York (2002) 4. Dey, A.: Understanding and Using Context. Pers. Ubiquit. Comput. 5(1), 4–7 (2001) 5. Ståhl, A., Höök, K., Svensson, M., Taylor, A.S., Combetto, M.: Experiencing the Affective Diary. Pers. Ubiquit. Comput. 13(5), 365–378 (2009) 6. Czerwinski, M., Gage, D.W., Gemmell, J., Marshall, C., Pérez-Quiñones, M., Skeels, M., Catarci, T.: Digital memories in an era of ubiquitous computing and abundant storage. Commun. ACM 49(1), 44–50 (2006) 7. Naaman, M., Harada, S., Wang, Q., Garcia-Molina, H.: Paepcke. A.: Context Data in Georeferenced Digital Photo Collections. In: 12th ACM International Conference on Multimedia, pp. 196–203. ACM Press, New York (2004) 8. Gargi, U., Deng, Y., Tretter, D.: Managing and Searching Personal Photo Collections. Technical report HPL-2002-67, HP Labs (2002), http://www.hpl.hp.com/techreports/2002/HPL-2002-67.pdf
120
T. Vaittinen, T. Kärkkäinen, and K. Roimela
9. Bentley, F., Metcalf, C., Harboe, G.: Personal vs. Commercial Content: The Similarities Between Consumer Use of Photos and Music. In: 24th ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 667–676. ACM Press, New York (2006) 10. Plaisant, C., Milash, B., Rose, A., Widoff, S., Shneiderman, B.: LifeLines: Visualizing Personal Histories. In: 14th ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 221–227. ACM Press, New York (1996) 11. Fertig, S., Freeman, E., Gelernter, D.: Finding and Reminding. SIGCHI Bull. 28(1), 66–69 (1996) 12. Dumais, S., Cutrell, E., Cadiz, J., Jancke, G., Sarin, R., Robbins, D.: Stuff I’ve Seen: a System for Personal Information Retrieval and Re-use. In: 26th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 72–79. ACM Press, New York (2003) 13. Ringel, M., Cutrell, E., Dumais, S., Horvitz, E.: Milestones in Time: The Value of Landmarks in Retrieving Information from Personal Stores. In: Ninth IFIP TC13 International Conference on Human-Computer Interaction, pp. 184–191. IOS Press, Amsterdam (2003) 14. Hodges, S., Williams, L., Berry, E., Izadi, S., Srinivasan, J., Butler, A., Smyth, G., Kapur, N., Wood, K.: SenseCam: A Retrospective Memory Aid. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 177–193. Springer, Heidelberg (2006) 15. Mann, S., Fung, J., Aimone, C., Sehgal, A., Chen, D.: Designing EyeTap Digital Eyeglasses for Continuous Lifelong Capture and Sharing of Personal Experiences. In: CHI 2005 Extended Abstracts on Human Factors in Computing Systems, pp. 2002–2006. ACM Press, New York (2005) 16. Kalnikaité, V., Sellen, A., Whittaker, S., Kirk, D.: Now Let Me See Where I Was: Understanding How Lifelogs Mediate Memory. In: 28th ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 2045–2054. ACM Press, New York (2010) 17. Olsson, T., Soronen, H., Väänänen-Vainio-Mattila, K.: User Needs and Design Guidelines for Mobile Services for Sharing Digital Life Memories. In: 11th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 273–282. ACM Press, New York (2008) 18. Kärkkäinen, T., Vaittinen, T., Väänänen-Vainio-Mattila, K.: I don’t mind being logged, but want to remain in control: a field study of mobile activity and context logging. In: 28th ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 163–172. ACM Press, New York (2010) 19. Belimpasakis, P., Roimela, K., You, Y.: Experience Explorer: a Life-Logging Platform Based on Mobile Context Collection. In: 3rd IEEE Conference on Next Generation Mobile Applications, Services and Technologies, pp. 77–82. IEEE Press, New York (2009)
Service Science Method to Create Pictograms Referring to Sign Languages Naotsune Hosono1, Hiromitsu Inoue2, Hiroyuki Miki3, Michio Suzuki4, Yuji Nagashima5, Yutaka Tomita6, and Sakae Yamamoto7 1 Oki Consulting Solutions Co., Ltd., 4-11-15 Shibaura, Minato-ku, Tokyo 108-8551, Japan 2 Chiba Prefectural University of Health Sciences 3 Oki Electric Ind. Co., Ltd. 4 Association of Architectural of Japanese DEAF 5 kogakuin University 6 Keio Univercity 7 Tokyo University of Science
[email protected]
Abstract. This paper discusses a method to create pictograms referring to several local sign languages with applying the concept of Service Science with Multivariate Analysis (MVA). Since pictograms are universal communication tools, human centred design (HCD) and context analysis by Persona model are applied. The experiments consist of two steps. Through the proposed method, the relationship between selected words and local sign languages are initially explained by sensory evaluation of the subjects. Under the cycle of HCD, the pictogram designer will perform to summarize the expression of several local sign languages by this method. The acquisition of user experience is to include it as a design guideline for context of emergency and traveling situations. Considering the results of the second experienced phase to prove the outcome design, the proposed method is one of the guidelines to create pictograms referring to several sign languages. Keywords: Service Sciences, Human Centred Design, Pictogram, Universal Communication, Sensory Evaluation.
1 Introduction This paper discusses a method to create pictograms or icons referring to several local sign languages with the concept of service science and Multivariate Analysis (MVA) [1]. Since pictograms or icons are universal communication tools, Human Centred Design (HCD) [2] and context analysis by Persona model by Alan Cooper [3] are applied in this research. This research was started in order to investigate the context of universal communication through local sign languages. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 123–130, 2011. © Springer-Verlag Berlin Heidelberg 2011
124
N. Hosono et al.
HCD is based on the context of use which is organized by four factors as user, product, task and environment in use (Figure 1). The research scope covers not only linguistic studies of sign language but also HCD with context of use [4].
Foreign People Impaired People
Travelling Emergency
Input
Users
Tasks
Equipment
Environment
Output Satisfaction Efficiency Effectiveness
Communication Service Collaboration Service
In use at Station, Airport Ambulance, Hospital
Fig. 1. Context of use of Guidance on usability
2 Research Purpose and Issues The purpose of this research is to figure out a method to create meaningful pictograms or icons referring to several local sign languages [5]. The sign language (SL) is basically a communication method from one person to the other. The main factors of sign language consist of the hand shape, location and movement. There is a dilemma that SL is a language with motion whereas pictograms or icons are still ones. There was quite a discussion among researchers. Then hand shapes and locations are drawn by an animation and movements are done by arrows referring to a snapshot of the related local sign languages.
3 Research Procedures Considering such research purpose and issues, the following research procedures are prepared; − − − − − − −
Phase 1: Phase 2: Phase 3: Phase 4: Phase 5: Phase 6: Phase 7:
Determine a concept Create Persona Model and Scenario Extract key words extraction on emergency and travelling situations Conduct first Sensory Evaluation with 7 local sign languages Design summarized pictograms Conduct second Sensory Evaluation with 8 local sign languages Conclude a method
Service Science Method to Create Pictograms Referring to Sign Languages
125
3.1 The Context Determination Based on the concept described above, two context situations have initially been chosen [6]. Alan Cooper proposed the Persona Model related to HCD where several situation representing Personas are imaginably created in order to simulate and find how they will behave under a certain context. This method is highly accepted by the manufacturers in creating new product plans and has been applied to service science as well. 3.2 Persona Model and Scenario Creation The first step is to create two Personas with applying the Persona Model under HCD [3]. The first Persona is a deaf person in a situation where he suffers a sudden illness while commuting in the morning, and is carried to the hospital by an ambulance. The second one is an office lady who lives in Hong Kong and has to visit Tokyo on business and then pleasure (Figure 2). Diary like scenarios underlying Personas are described from discussions with three colleagues utilizing the Brain Storming Method. These scenarios mainly pay attention to the dialogues between the Persona and those people surrounding [7]. The first scenario of the deaf person in an emergency consists of about 600 words (equivalent to 3000 Japanese characters) and second with the traveling woman about 1700 words (equivalent to 8500 characters).
First visit to Japan Sensitivity for apparel fashion Middle income class
Yie Ling (依林) “Interested in city living of Japan”
Profile
Personal Profile
•
• • • •
This time a package tour is used however normally independent tour is preferable to reserve transportation and accommodation. Dislike to act in a group Usually the trip schedule is not fix up to the final stage. Cancelled quite often. Interested in a big city lives of Tokyo or Osaka and night life. Interested in advanced technologies and modern concepts. Try to get on Shinkansen. Dislike a bus tour of package tour. Try to Shinkansen even though extra payment. Interested in Japanese famous view point such as cherry blossoms or Mt. Fuji. Wish to find Japanese friends and start communication. Shopping is essential when traveling. Always care for souvenirs for friends and colleagues. This time plan to purchase cosmetics and home electric appliances by TAX free and Japanese foods. Eat relatively much and warm dishes are preferable. Buffet style, needles and beef. Interested in Japanese cousin. Preferred a single room of JPY6,000-7,000. Interested in hot spa and wish to stay Japanese style Ryokan. Preconception of Japan Little chance to use Chinese language, Expensive price, Good service, Safe and secured –
• • •
–
•
–
•
–
•
–
Goal • • • • •
Wish to experience to live in a big city, advanced technologies and good services. Visit famous view points in Japan. Buy cosmetics, home electric appliances by Tax free and Japanese typical souvenirs. Try to take exotic foods Try to save travel expenses
Occupation:OL of overseas company Nationality: Chinese(Hong Kong) Age:27 Characters:,Cheerful, Friendly, Challenged minded, independent and established, Easy to get lost Spoken languages: Chinese and English
Trip information in advance
About Japan: enough Trip budget: JPE200K Trip days: One week(Business and pleasure)
Mobile Usage Situation
Mobile Phones ‒ Daily used ‒ Mobile mail and Internet ‒ Internet sites are games and music. Other used functions ‒ Music play ‒ Digital camera
Fig. 2. An example of a Persona model
126
N. Hosono et al.
3.3 Keywords Extraction on Emergency and Traveling Situations This research is focused upon dialogues with several participants and referring to observations from the view point of the provider and the receiver under service science principle. The next phase is to extract words that are fundamentally essential to the dialogues of the scenarios. 37 words were selected and categorized by discussions with three colleagues. Looking at the dialogues in the scenarios under the selected context, the hardest process is initiating the dialogue to a stranger. In modern times, people are worried about security. They are extremely cautious when approached by an unfamiliar person. Several interjections are included to assist the initiation of dialogues. 3.4 First Sensory Evaluation with 7 Local Sign Languages The research is initially focused upon creating pictograms or icons to make dialogues since the fundamentals of sign language are hand shape, location and movement. This research references to a collection of animation figures consists of seven local sign languages whose author is a deaf architect, gave overwhelming support to the research by supplying and permitting reference to the database. The seven local sign languages are of American, British, Chinese, French, Korean, Japanese, and Spanish [8].
S
A
J
F
C
U
K
E
Fig. 3. An example of a voting sheet of “Expensive”
Service Science Method to Create Pictograms Referring to Sign Languages
127
In the experiment, subjects are first shown an expression with the collection of animation figures consists of seven local sign languages. After then subjects are informed of the sign meaning, they are requested to vote with 19 tokens which of the seven different local sign language expressions (samples) best coincides with the informed image. They are asked to put all 19 tokens on the condition that they are permitted eventually zero voting on some samples. This sensory evaluation method can easily make relative comparisons between the seven expressions of local sign languages and is more applicable than the ordering method or pair comparison method. An example of voting sheet of “Expensive” with local sign languages is shown in Figure 3 Then the correspondence analysis of Multivariate Analysis (MVA) by statistic Software, Statistical Package for Social Science (SPSS) [9, 10] is applied. The outcome is plotted as similar local sign languages are to be plotted closely on a plane. In the characteristics of correspondence analysis, the subjects who have general and standard ideas are positioned in the centre, whereas those who have extreme or specialized ideas are positioned away from the centre. The center crossing point of the first and second Eigenvalues is gravity point or average point. The first experiment subjects are 13 people in their 20’s including nine science course students, four humanity course students. Some have experience living overseas and sign language interpreting. After voting by the tokens, all the subjects are asked of their confidence level with Semantic Differential (SD) method. Figure 4 is an example of outcome chart where “Expensive” is plotted.
Dimension 2
Dimension 1
Fig. 4. A plot chart of “Expensive” with seven sign languages
3.5 Summarized Pictograms Design Through the experience of the first Sensory Evaluation with seven local sign languages of 37 words, many sign language expressions are identified by representing
128
N. Hosono et al.
the meaning. Among them the most converged seven words of “when?”, “good-by”, “painful”, “thank you”, “where?”, “toilet”, and “expensive” among 37 are selected by means of brain storming. Following to the cycle process of HCD, the original designer is asked to summarize and design an animation like pictogram referring to the outcome of several local sign languages by the sensory evaluation mentioned above. The newly designed pictogram is added to seven local sign languages with American, British, Chinese, French, Korean, Japanese, and Spanish. 3.6 Second Sensory Evaluation with 8 Local Sign Languages The next procedure is the same manner as the first experiment of Phase 4. After subjects are informed of the sign meaning, this time they are requested to vote with 23 tokens which of the eight different local sign language expressions including newly designed one will be the best coincides with their image. The procedure was the same manner as the first step, and the correspondence analysis of Multivariate Analysis (MVA) by SPSS is once again performed. The outcome including the newly designed pictogram is plotted with other seven local sign languages in order to measure whether the newly created pictogram represents of the cluster.
Dimension 2
Dimension 1
Fig. 5. A plot chart of “Expensive” with eight sign languages
The second experiment subjects are 20 engineering department students in their 20’s including two female students. Almost all except three are different subjects from the first experience. After voting by the tokens, all the subjects are again asked of their confidence level with Semantic Differential (SD) method.
Service Science Method to Create Pictograms Referring to Sign Languages
129
Figure 5 is an example of outcome chart where “Expensive” is plotted. The newly designed one is plotted close to Japanese, Korean, Chinese and French sign languages. Whereas American, British and Spanish plotted further down. These deployments of the plots are simular in seven and eight sign languages experiments. 3.7 Conclude the Method Comparing two outcomes of Phase 4 with seven local sign languages and 6 with eight local ones, followings are concluded. • All seven newly designed animation pictograms are positioned in the centre of the related local sign languages cluster. • Even though almost of the subjects are different at the first and second experiment, the general outcome plot patterns hold similar patterns in space. • In oriental sign languages of Japanese, Korean and Chinese tend to be plotted closely together.
4 Conclusion and Discussions This paper discusses a method to extract the summarized expression of several local sign languages in order to draw pictograms by applying the sensory evaluation with MVA. The experiments consist of two steps. The first step is to find out a pictogram is a majority common expression upon a word among seven local sigh languages. Looking at the first step, this method looks valid in practice since Japanese, Korean and Chinese sign languages are similar by historical background, and in fact and they are plotted close to each other. The second step is to prove the characteristics of the pictogram represent the meaning of the word. Almost all of the newly designed pictograms positioned in the centre of the cluster then it is representative of the clusters. Through the proposed method, the relationship between selected words and local sign languages are initially explained by sensory evaluation by the subjects. Under the cycle of HCD, the pictogram designer will perform to summarize the expression of several local sign languages by this method. The acquisition of user experience is to include it as a design guideline for instance of the context of emergency and traveling situations. The issues are that the quality of the newly designed pictogram depends on the designer’s ability to summarize several ones. The newly designed pictograms are still biased by sign languages in this research in order to become much easier communication tool, and require further improvement to simplify and easily to understand for everybody. Considering the results of the second experienced phase to prove the outcome design, the proposed method is one of the guidelines to create pictograms by referring to several sign languages. Acknowledgements. This research is supported and funded by SCOPE (Strategic Information and Communications R&D Promotion Programme) project organized by
130
N. Hosono et al.
Ministry of Internal Affairs and Communications (MIC) of Japan. A collection of local sign language database is supplied and permitted for research use by Mr. M. Akatsuka of Architectural Association of Japanese DEAF (AAJD). Gratitude to Dr. K. Nakazono of NTT who gave some comments to the research.
References 1. Hosono, N., Inoue, H., Tomita, Y.: Sensory analysis method applied to develop initial machine specification. Measurement 32, 7–13 (2002) 2. International Organization for Standardization: ISO9241-210 (former 13407:1999), Ergonomics Human-centred design processes for interactive systems (2010). 3. Cooper A.: About Face 3, Wiley (2007) 4. Miki H., Hosono, N.: Universal Design with Information Technology (Japanese version), Maruzen (2005) 5. Horton W.: The Icon Book, John Wiley & Sons, Inc., New York (1994) 6. International Organization for Standardization: ISO9241-11, Ergonomic requirements for office work with visual display terminals (VDTs), Guidance on usability (1998) 7. International Organization for Standardization: ISO9241-110, Ergonomic requirements for office work with visual display terminals (VDTs), Dialogue principles (2006) 8. Akatsuka, M.: Seven sign languages for tourists: Useful words and expressions, ChineseJapanese-American Working Group (2005) 9. Field, A.: Discovering Statistics Using SPSS, 3rd edn. Sage Publications, Thousand Oaks (2009) 10. SPSS. Categories in Statistical Package for Social Science ver.18, SPSS (2009)
MoPaCo: Pseudo 3D Video Communication System Ryo Ishii, Shiro Ozawa, Takafumi Mukouchi, and Norihiko Matsuura NTT Cyber Space laboratories, NTT Corporation, 1-1, Hikari-no-oka, Yokosuka-Shi, Kanagawa 239-0847, Japan {ishii.ryo,ozawa.shiro,mukouchi.takafumi, matsuura.norihiko}@lab.ntt.co.jp
Abstract. We propose a pseudo 3D video communication system that imparts motion parallax which adjusts to the viewpoint position of a user and enables the user to view video pictures in which depth can be perceived with an ordinary equipment setup, namely a monocular camera and a 2D display. We have implemented the system and evaluation experiment results with it showed that its imparting of motion parallax allows it to represent distances that closely reflect actual face-to-face situations better than 2D video can. In addition, subjective evaluations confirmed that motion parallax gives users the feeling that the conversational partner is actually present and makes it easier for them to comprehend the positional relationship of the conversational partner in space. Keywords: Video communication, motion parallax, depth perception, interpersonal distance.
1 Introduction Our aim is to implement a video communication system which not only enables ordinary conversation, but which also provides a highly realistic feeling that enables users to work together in a natural manner while sharing a mutual space and observing each other, in a manner that is equivalent to actually being face-to-face. For this reason, it is important to give the user the feeling that the conversational partner is in front of him and have natural transmission of nonverbal information such as inter-personal distances, gaze, and pointing gestures, using a body that shares a mutual space with the conversational partner. An ordinary video communication system that uses a 2D display to show video footage which has been captured with a monocular camera lacks most of the depth information about the conversational partner and the space around him. In addition, the visibility of the conversational partner and background does not change to match the changing viewpoint when the user moves. For this reason, in addition to the problems of a feeling of spatial separation from the conversational partner [1, 2] and a lack of feeling of the presence of the conversational partner [3], there is a huge problem in that nonverbal information associated with depth (such as inter-personal distance and pointing gestures) cannot be transmitted correctly. In order to transmit such information, it is necessary to impart depth information within the video footage, and there are currently many research projects into tackling these challenges. It is known that G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 131–140, 2011. © Springer-Verlag Berlin Heidelberg 2011
132
R. Ishii et al.
motion parallax and binocular parallax are major cues that enable humans to sense depth at close distances [4]. Of these, binocular parallax can be used comparatively simply, so several methods have been proposed for using binocular parallax to represent depth by providing a stereoscopic view of the conversational partner [5]. To present correct depth information naturally in video communication, however, it is absolutely essential to present video that has motion parallax that corresponds to the observation position of the user. In this study, we propose a method that implements video communication that is associated with motion parallax with a simple setup. Since this method is intended for systems that are generally available, it was implemented with a configuration similar to that of an existing video communication system, namely a monocular camera and a 2D display. In this paper, we also report on evaluation experiments using the thus-implemented system, by which we proved experimentally the perception of depth towards the conversational partner due to motion parallax, impressions relating to the conversation video, and the accuracy with which pointing actions that follow pointing gestures are transmitted, and thus report on the effectiveness of motion parallax.
2 Related Work A video expression method that enables users to depth perspective using motion parallax alone has been proposed. Suenaga [6] proposed a 3-dimension perspective display with motion parallax. The method of display is to project video pictures corresponding to the user’s viewpoint position tracked by stereo cameras onto a 2D display. However, the system provides only CG model contents. That is, the system cannot generate photographed video in real time. The objectives of this study are to propose a new method of implementing motion parallax by using only a monocular camera in real time, and also to verify the effect on the transmission of information about nonverbal behavior by motion parallax and impressions of the resultant video.
3 Proposed MoPaCo System 3.1 Overview of MoPaCo System In this study, we propose a real-time video communication system called MoPaCo (which is a contraction of “Motion Parallax Communication”) which implements motion parallax although using only a monocular camera. Fig. 1 shows images of motion parallax video representations by MoPaCo of a conversational partner which correspond to the viewpoint positions of different users. The display for a user who is at some distance from the conversational partner in the video can give the user the feeling they are linked as if through a window. It is thought that this motion parallax video representation will eliminate spatial separation, improve the feeling of presence
MoPaCo: Pseudo 3D Video Communication System
133
of the conversational partner, and enable the transmission of nonverbal information that is associated with depth by imparting depth information to video pictures. To present a motion parallax video of a conversational partner on a 2D display so as to correspond to the viewpoint positions of different users, the following processes are necessary: (1) Measurement of each user’s viewpoint position (2) Construction of a 3D space having information on the dimensions and positional relationships of the people and the background, based on information obtained from a camera or other means (3) Rendering of the 3D space constructed in step (2) on a 2D display, to correspond to each user’s viewpoint position that was obtained in step (1). We propose the technique to perform the steps (1) and (2) using only a monocular camera and implemented the MoPaCo system using a monocular camera. In this section, we describe the details for each process.
Background Conversational partner
2D display
Scene from right side User
Scene from right side
Scene from front side Fig. 1. Concept images of video representations caused by motion parallax
3.2 Measurement of User’s Viewpoint We propose the use of only a monocular camera to detect each user’s viewpoint. Before calculating the 3D position from parts information (coordinate position) of each face in the 2D image, the system measures the eye separation distance of each user, as preprocessing. It then acquires the distance of each user from the camera, using the Depth from Focus function that is used for achieving focus in ordinary cameras. During this process, template matching is performed on the image captured from the camera to measure the positions of both eyes (2D coordinates within the image) and the orientation of the head. The system calculates the eye separation distance of each user from the distance from the user to the camera, information measured from the image, and the angle of view and resolution of the camera.
134
R. Ishii et al.
Based on this information, real-time capture is started and the system obtains the positions of both eyes (2D coordinates within the image) and the orientation of the head from the captured image, and calculates the viewpoint position z of that user from the camera from there at that time. Note that the x- and y-coordinates are calculated from the 2D coordinates within the image and the pixel pitch of the image. Based on this information, real-time capture is started and the system obtains the positions of both eyes (2D coordinates within the image) and the orientation of the head from the captured image, and calculates the viewpoint position z of that user from the camera from there at that time. Note that the x- and y-coordinates are calculated from the 2D coordinates within the image and the pixel pitch of the image. 3.3 Construction of a 3D Space In this study we propose a method of constructing 3D information with respect to an image captured from a single camera. This is done by performing background difference processing using background information that was acquired beforehand, maintaining the 2D plane and dividing it into a personal area and a background area, and creating a multi-layer structure with those areas arranged as layers in accordance with their depth-wise positions. The use of 2D images ensures that a high-resolution display is possible. In addition, if only the background difference is subjected to image processing, the processing costs are low and thus real-time processing is possible. Using the person and background images, the system then generates multiple layers having a distance relationship from the camera, at full size (in the rest of this paper, the layers that use the person image and the background image are called the person layer and the background layer). The method of calculating distance information measures for the background layer beforehand is by the depth-from-focus method provided in the autofocus function of the camera, when the background difference image is acquired. For the person layer, the user viewpoint position is used. These distances become distance information from the camera of the person layer and the background layer, respectively. Based on this distance information, the system uses Equation (1) to calculate the full size (width wi height hi) of each layer i from the thus-acquired distance information di and the camera’s angle of view (width θw, height θh). This procedure configures a 3D space having full size and position information.
×
wi = 2*di*tanθw/2, hi = 2*dh*tanθh/2
(1)
3.4 Rendering of 3D Space Based on User’s Viewpoint As shown in Fig. 2, the person layer and background layer generated by the 3D spatial information module are projected in perspective to match the viewpoint position of the user, using the 2D display as a projection surface. Thus motion parallax video is implemented.
MoPaCo: Pseudo 3D Video Communication System
2D display
135
Layers of image
Viewpoint
Conversational partner
Fig. 2. The person layer and background layer are projected
3.5 Results of Implementation Using the proposed method described above, we implemented the MoPaCo system which enables bidirectional viewing of motion parallax video in real time. The development environment was a computer with Intel Core i7 Extreme 980X as the CPU, 12 GB of memory, and a NVIDA GeForce GTX480 graphics board. The results of this implementation are shown in Table 1. The “Reflection of conversational partner’s image” in this table is that the conversational partner’s video appears in the user’s display from the captured video, and “Reflection of motion parallax” motion parallax is the appearance in the video from a movement of the user’s viewpoint position. The table also shows frame rate and response (time lag) in each part. The frame rate is sufficiently high. In contrast, response is somewhat late in arriving. Table 1. Implementation results Captured image size
1280×780 (HD)
1920×1080 (Full HD)
Reflection of conversational partner’s image
Frame rate
30 fps
18 fps
Response
260 ms
330 ms
Reflection of motion parallax
Frame rate
30 fps
Response
300 ms
Fig. 3 shows a scene as observed in 2D video, motion parallax video with MoPaCo, and actual face-to-face (REAL) situations. In comparison with the face-toface condition, there was no parallax in the video in the 2D condition, even if the user’s head moved, so the human dimensions and positional relationships did not match. With the MoPaCo condition, in contrast, the dimensions and positional relationship between the person and background were reproduced in the video.
136
R. Ishii et al. REAL
2D
Glass window
MoPaCo
2D display
Fig. 3. Scenes for three observational conditions: actual face-to-face (REAL), 2D video, and motion parallax video with MoPaCo
4 Experiments 4.1 Experiment Procedure Using the MoPaCo system, we conducted experiments with test subjects, with the objective of verifying the transmission by motion parallax video representation of inter-personal distances, which are particularly important in nonverbal information, and the effect of motion parallax on increasing the feeling of a partner’s presence, the positional awareness of the partner’s space and spatial comprehension, etc. In the experiments, the position of the conversational partner was varied in face-to-face conversational situations, conventional 2D video, and video with motion parallax. After each subject had observed a situation, they gave their evaluations. There were a total of 18 experimental conditions, consisting of combinations of three observation methods and six distances of the conversational partner, as shown in Table 2. Table 2. Implementation results Observation conditions (3 conditions) • REAL condition: Observing the conversational partner through a glass window • 2D condition: Observing the conversational partner in an image displayed on a 2D display (in this case, the user’s viewpoint position is where the image is displayed at a position when the user is sitting straight in the chair) • Motion Parallax with MoPaCo (MP) condition: Observing the conversational partner in an image having motion parallax, which is shown on a 2D display Distance to the observer (6 conditions) • The mannequin’s position is set to 0, 20, 40, 80, 120, and 200 cm from the partition (display plane), in other words, 150, 170, 190, 230, 270, and 350 cm from the subject.
To measure the perceived distance to the conversational partner in the verification of the effect of inter-personal distance representation, we adopted a subjective evaluation by which subjects gave the distance from their chest to the chest of the conversational partner by a scale-less string length. Asking the subjects to answer
MoPaCo: Pseudo 3D Video Communication System
137
with the length of a piece of string is designed to eliminate any personal differences in the subjective criterion, in comparison with methods in which answers are numerical values. In this case, if the string length given as the answer under the REAL condition is the same as the length under the MP condition, we can say that motion parallax video makes it possible to perceive depth to the conversational partner in a similar manner to actual face-to-face conversation. We obtained the subjects’ impressions of motion parallax by asking them to evaluate the seven items listed in Table 3, such as the feeling that the conversational partner is present, using a six-step Likert scale (1-6 points, six being best) on a questionnaire sheet for the 2D and MP conditions.
Table 3. Subjective evaluation items • Ease of viewing video: Was the video of the conversational partner easy to see? • Stereoscopic effect: Did you feel that the conversational partner’s space was in 3D? • Intuition of distance comprehension: Could you intuitively grasp the feeling of depth between you and the conversational partner? • Presence of conversational partner: Did you feel that the conversational partner was present? • Face-to-face feeling: Did you feel that you had actually met the conversational partner? • Ease of spatial comprehension: Was it easy to understand the positional relationship between the conversational partner and the background? • Feeling through window: Did you feel that you had met the conversational partner through a window via the display?
<Subjective space>
<Partner’s space>
Partition
Background 340 cm
Glass window or 2D display
Mannequin
Subject 46 cm 150 cm
~200 cm
0
94 cm
Fig. 4. Experimental equipment
The experimental setup was such that the subject was seated on a chair at a position 150 cm from a partition in which a glass window was installed, as shown in Fig. 4, and was able to observe the space of the conversational partner through the glass window and display. Note that with the 2D condition and the MP condition, the display was installed directly behind the glass window. The size of the glass window (46 cm high x 80 cm wide) was less than that of the display, so the edges of the display were not visible from the subjects. Note that we used a humanoid mannequin
138
R. Ishii et al.
in place of an actual person, in order to ensure that the visual stimulus of the conversational partner was uniform. Before the experiment, each subject observed the actual mannequin and verified its size. Taking the effect of the order of trials into consideration, the sequence of trials was performed under randomized conditions. Subjects were asked to sit in the chair and raise their head upon hearing the start signal, then observe the mannequin while remaining seated in the chair but moving their head freely. No time limits were set, but all the subjects finished their observations within ten seconds and then made their evaluations. They made their perceived distance evaluations after undergoing a trial for each of the experimental conditions. In contrast, they made their video impression evaluations after going through all the experimental conditions. 4.2 Experiment Results for Perceived Distance Experiments were conducted using ten subjects (nine males, one female, age range 20-60). We first show the results obtained for perspective distance in Fig. 5. The figure shows the average values of string lengths measured for all subjects. The actual distance from the subject to the conversational partner is plotted along the horizontal axis and the average value of the string length (perceived distance) is plotted along the vertical axis. The dotted line shows the values when the actual position of the conversational partner matches the string distance. It was clear that perceived distances given as string lengths were much smaller than the actual distances (the dotted line). Significant differences were seen overall, in that the string lengths became shorter in the sequence for the REAL → MP → 2D condition. In other words, it was confirmed that the MP condition is better able than the 2D condition to represent inter-personal distance in a manner close to that of the REAL condition. However, because a difference in depth perception exists between the MP and REAL conditions, we think that it is necessary to define the relationship between depth perception and motion parallax video, and to determine how distance in the MP condition can be represented accurately in the same way as it is in the REAL condition.
Average string length (Perceived distance) (cm) Actual distance from subject to conversational partner (cm) Fig. 5. Experiment results
MoPaCo: Pseudo 3D Video Communication System
139
4.3 Experiment Results for Video Impression Fig. 6 shows the average values of the subjective evaluation scores for the questions that 10 subjects were asked to answer on the questionnaire. In this experiment, we performed paired t-testing for each evaluation item. These results are also shown in the graphs. The “Ease of viewing video” in the figure caused us some concern, in that since movements under the MP conditions result in large changes in the video, drunkenness or other abnormalities might not be noticed. However, the existence of such a problem could not be confirmed. We consider that the MoPaCo feature of imparting motion parallax to video pictures in accordance with viewpoint position will give humans a natural visual effect. This is supported by results showing that MP was evaluated significantly higher than 2D in two paired T-tests for the six evaluation factors other than “Ease of viewing video” (Stereoscopic effect : t(9)=2.68, .10
Fig. 6. Impression evaluation results for motion parallax video
140
R. Ishii et al.
5 Conclusion We have proposed a pseudo 3D video communication system that imparts motion parallax which adjusts to the viewpoint position of a user and enables the user to view video pictures in which depth can be perceived with an ordinary equipment setup, namely a monocular camera and a 2D display. The system was implemented and evaluation experiment results with it showed that its imparting of motion parallax enables it to represent distances that closely reflect actual face-to-face situations better than 2D video can. In addition, it has been confirmed from subjective evaluations that motion parallax gives the user a feeling as if the conversational partner is actually present and makes it easier for him/her to comprehend the positional relationship of the conversational partner in space. In the future, we plan to conduct real-time conversations to further confirm the effectiveness of using motion parallax in video communication.
References 1. Williams, E.: Coalition formation over telecommunications media. European Journal of Social Psychology 5, 503–507 (1975) 2. Strickland, L.H., Guild, P.D., Barefoot, J., Paterson, S.A.: Teleconferencing and leadership emergence. Human Relation 31, 583–596 (1978) 3. Heath, C., et al.: Disembodied conduct: communication through video in a multimedia environment. In: Proc. CHI 1991, pp. 99–103. ACM Press, NY (1991) 4. Cutting, J., Vishton, P.: Perceiving Layout and Knowing Distances. In: Epstein, W., Rogers, S. (eds.) Perception of Space and Motion, pp. 69–117. Academic Press, New York (1995) 5. Towles, H., Chen, W.C., Yang, R., Kum, S.U., Fuchs, H., Kelshikar, N., Mulligan, J., Daniilidis, K., Holden, L., Zeleznik, B., Sadagic, A., Lanier, J.: 3DTele-Immersion Over Internet 2. In: ITP 2002 (2002) 6. Tsuyoshi, S., Yoshio, M., Tsukasa, O.: 3D Display Based on Motion Parallax Using Noncontact 3D Measurement of Head Position. In: Proceedings of OZCHI 2005 (2005)
Analysis on Relationship between Smiley and Emotional Word Included in Chat Text Junko Itou1, Tomoyasu Ogaki2, and Jun Munemori1 1
Faculty of systems Engineering, Wakayama University, 930, Sakaedani, Wakayama 640-8510, Japan {itou,munemori}@sys.wakayama-u.ac.jp 2 Graduate School of systems Engineering, Wakayama University, 930, Sakaedani, Wakayama 640-8510, Japan
[email protected]
Abstract. In this research, we analyze the relationships between smileys and emotional words in chat text aiming to apply these relationships to an embodied character chat system. Smileys add various meanings, especially mental information to plain chat texts to make our text communication successful. We focus on the way to use smileys and emotional words so that we can estimate the chat atmospheres. We performed an experiment to investigate the relationships between smileys and emotional words in chat dialogue.
1 Introduction Online communication system such as e-mail, a chat system, a remote meeting system, and a distance learning, becomes a common tool for people. Now we can communicate with people using a graphical chat system that agents talk in virtual 3D space in place of users. Users obtain messages by watching embodied character's actions as well as by reading plain texts in a chat system that employs a embodied character. A character in the chat systems plays a role as an agent of a user not only to express the user's emotional states or intentions which cannot be displayed by a chat message, but also to make the chat alive. As the result, it becomes clear what meaning the user implies for the chat messages. In order to control actions of embodied characters, users need to input texts or click a button consciously at the each timing. It is a troublesome task for a chat user. Therefore, a character chat system is required that the nonverbal expressions are automatically controlled according to the dialogue atmosphere and input chat messages. Previous works on controlling nonverbal expressions of embodied characters mainly discuss the consistency of nonverbal expressions with the speech utterances or the goal of conversation for each agent [1][2]. However, when we consider dialogues between a pair of embodied characters, we need to consider the interdependences between nonverbal expressions displayed by those two characters. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 141–146, 2011. © Springer-Verlag Berlin Heidelberg 2011
142
J. Itou, T. Ogaki, and J. Munemori
Emotional words and smileys add nonverbal information variously to text messages and express the user's emotional state just as embodied characters add. So, we focus on the relationships between emotional words and smileys aiming to apply these relationships to an embodied character chat system in the remainder of this article.
2 Related Researches on a Chat System with Embodied Characters and Nonverbal Expressions In most current graphical communication systems, a user should choose and specify the character's action explicitly, or a system focuses on the consistency of nonverbal expressions with the speech utterances. Cassell[1] proposed an agent system which interpreted user's inputs and generated responses automatically with nonverbal expressions to be fulfilled according to the desired function. This research focused on the relation between spoken words of users and responses of the agent which show words or actions. However, it is reported in the field of social psychology that there are some interdependences between nonverbal expressions displayed by “each person” in daily conversations [3]-[6]. It has been investigated in social psychology what features are found in nonverbal expressions given by humans during their conversation. In the experiments by Matarazzo, noddings by the listeners in a conversation encouraged utterances of the speakers. As the result, animated conversation between the speakers and the listeners is realized [5]. In another experiments by Dimberg, facial expressions of the test subjects were affected by those of their partners [6]. The subjects smiled when their partners gave smiles to them, whereas they gave expressions of tension when their partners had angry faces. These results imply there are positive correlations or synchronicities between the nonverbal expressions given by the conversation partners. Table. 1 illustrates these positive correlations by *. Table 1. Interdependences between nonverbal expressions of different persons
By employing the knowledge in social psychology we aim to realize automatic actions of embodied characters. People substitute smiley for nonverbal expressions in a text communication. That means we can estimate mental states of a user by the usage of smileys and emotional words.
Analysis on Relationship between Smiley and Emotional Word Included in Chat Text
143
3 Nonverbal Expressions Shown by Smiley and Emotional Words 3.1 Goal In the most current graphical chat systems, users should choose and specify the character's actions explicitly. By employing the knowledge described in the previous section, we aim to realize automatic actions and reactions of embodied characters in order to make users' chat alive. It is preferable that character's actions and reactions can be produced by users’ messages because users attend to only their chat and it is very troublesome for users to set manually all actions and reactions at each message during their chat. We will apply the interdependences between nonverbal expressions described in section 2 to the actions of embodied agents in chat system. 3.2 Emotional Words and Frequently Used Smileys People can interpret a smiley variously so they use and receive smileys in many ways. In order to investigate the relationship between smileys and emotional words, we collected smileys which were generally used in a text communication and the emotional words remembered by the smileys. At first, we collected in total 151 kinds of smileys from a smiley list registered with IME2007. Fig. 1 shows a part of the smileys. We asked 7 college students who ordinarily used smileys in text communications to answer whether they knew the meanings of the smileys and used those. It was 9 kinds of smileys that more than 5 subjects replied they often used the smileys and it was 48 kinds of smileys they had not used at all. I show the 9 kinds of smileys mentioned above at Table. 2.
Fig. 1. Example of smiley Table 2. Frequently-selected smileys
Next, we indicated the subjects to write the words associated with the 9 smileys when they converted from Kanji to the 9 smileys and to categorize the 9 smileys into 35 emotional word categories [7] in Table.3 and into 8 basic emotions [8]. The 8 basic emotions include ‘‘delight’’, ‘‘grief’’, ‘‘affection’’, ‘‘aversive’’, ‘‘fear’’, ‘‘rage’’, ‘‘amazement’’, and ‘‘caution’’.
144
J. Itou, T. Ogaki, and J. Munemori Table 3. Emotional word category
4 Experimental Results 4.1 Categorized Result The subjects had a tendency to choose “delight’’ and “affection’’ among 8 basic emotions for the smiley (1) - (5). They also associated the smiley (1) - (3) with the emotional words expressing friendly such as pleasure, joy and satisfaction. As for the smiley (4) and (5), joy, pleasure and satisfaction were given as the smiley (1) - (3), but consideration and warmth were chosen, too. It became clear that these emotional categories are indispensable in the chat. The smiley (4) and (5) showed approximately the same result, but it is revealed that the expression of emotion was stronger in (5). If a smiley expresses a same mental state, we should reflect the strength of the emotion in an embodied character of a chat system according to the situation. Many subjects categorized the smiley (6) into “grief’’ of 8 basic emotions but there was some subjects chose uneasiness, fear, pity, pain and unpleasant. It is thought that this smiley depends on context greatly. As the smiley (6), information except the smiley is necessary for estimate of feelings. The smiley (7) shows a person sheds tears. Most of subjects got hold of a same image for the smiley (7) and they selected ‘‘grief’’ and sorrow. As for the smiley (8), ‘‘grief’’ was mainly selected however some subjects chose ‘‘affection’’ and ‘‘aversive’’. It was the smiley (9) that answers was divided most. They selected ‘‘delight’’, ‘‘affection’’, ‘‘grief’’ and ‘‘aversive’’ among 8 basic feelings. They tend to choose sorrow, loneliness and regret for the smiley (8), and to choose disappointment, uneasy, hesitation, regret and pain for the smiley (9). The words associated with the smiley (8) are “I'm sorry’’ and “I apologize’’ such as words to express a feeling of the apology, “Thank you in advance’’ such as words to make a request, and, “Thank you’’ such as words to be given thanks to. 4.2 Smileys in Chat Texts We performed second examination to investigate how the smiley used in real chat conversations and what kind of smiley appeared. Experimental subjects were 10 college students: they were divided into 5 pairs. They connected to the chat server from different rooms. We instructed them to exchange chat messages freely for 15 minutes. They can only use plain text and
Analysis on Relationship between Smiley and Emotional Word Included in Chat Text
145
smileys in the chat system. We show them the list of smileys which more than 3 subjects answered they frequently used, in previous subsection’s experiment. Add to the above direction, we announced they had not to use smiley when they think it was not necessary to input smileys. As a result, we obtain 234 sentences and 55 smileys. 67.3% of appeared smileys can be categorized (1) – (5) in Table. 2, 12.7% smileys are members of (9) and 7.3% smileys are (7). A total of 87.3% of the smileys that were appeared in the chat texts, so these smileys and the basic emotions associated with the smileys are essential feature to chat smoothly and emotionally. 4.3 Future Work As a future work, we will add the above direction to a chat system that employs embodied characters. As discussed in section 2, the actions of characters are not only determined by messages of a user but also the actions of the conversation partner character. For example when one character laughs by a message of the user that employs the character, the partner character should smile back without a text reply of the chat partner.
Fig. 2. Overview of a chat system that employs embodied characters
We have proposed a chat system that has embodied characters. Fig. 2 is the overview of our chat system. The main window shows information about login users. In the chat window, a log of users’ chat is shown. A user's agent character and the chat partner’s agent character variously act according to the users' input chat texts. When a message includes the keywords corresponding to "think", the partner character shows a reaction to stimulate their dialogue according to the research by
146
J. Itou, T. Ogaki, and J. Munemori
Matarazzo. From the indication mentioned in section 2, we intend to reflect these relationships between an action and a reaction.
5 Conclusions In this article, we investigated the relationships between smileys and emotional words to apply to an embodied character chat system. In the experiments, it is revealed that the basic emotions “delight’’, “affection’’ and ‘‘grief’’ are most frequently appeared in daily chat conversations so we should make the embodied character act to satisfy the relationships among nonverbal expressions, and relationships between emotional words and smileys. Finally, we should plan to investigate whether atmospheres produced by actions of embodied characters based on smileys are accordance with real atmospheres of chat conversations.
References [1] Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Ilhjalmsson, H., Yan, H.: Embodiment in Conversational Interfaces: Rea. In: CHI 1999, pp. 520–527 (1999) [2] De Carolis, B., Pelachaud, C., Poggi, I., De Rosis, F.: Behavior Planning for a Reflexive Agent. In: Proc. of International Joint Conference on Artificial Intelligence (IJCAI 2001), pp. 1059–1066 (2001) [3] Beattie, G.W.: Sequential patterns of speech and gaze in dialogue. Semiotica 23, 29–52 (1978) [4] Kendon, A.: Some functions of gaze direction in social interaction. Acta Psychologica 26, 22–63 (1967) [5] Matarazzo, J.D., Saslow, G., Wiens, A.N., Weitman, M., Allen, B.V.: Interviewer Head Nodding and Interviewee Speech Durations. Psychotberapy:Theory, Research and Practice 1, 54–63 (1964) [6] Dimberg, U.: Facial Reactions to Facial Expressions. Psychophysiology 19(6), 643–647 (1982) [7] Yoshida, M., Kinase, R., et al.: Multidimensional scaling of emotion. Japanese Psychological Research 12(2), 45–72 (1970) [8] Plutchik, R.: The Multifactor-Analytic Theory of Emotion. The Journal of Psychology 50, 154–171 (1960)
Designing Peripheral Communication Services for Families Living-Apart: Elderly Persons and Family Yosuke Kinoe and Mihoko Noda Graduate School of Intercultural Communication, Hosei University 2-17-1, Fujimi, Chiyoda-ku, Tokyo 102-8160, Japan
[email protected],
[email protected]
Abstract. We developed a new augmented communications environment which aims to engender a greater sense of social proximity to geographically distributed family members and improve their emotional well-being. First, a field study was conducted to determine important peripheral communication cues for sensing presence and mood of family members as well as memory triggers. Secondly, the design principles were extracted from the study results to guide the development of a first-of-a-kind prototype, the “SharedEpisodes” that delivers visual peripheral cues along the line with the story of family episodes. Finally, an initial field evaluation was conducted. Overall responses of the participants indicated positive for this type of communications environment that supports the awareness of the presence and state of family members, and the exchange of peripheral communication cues based on family episodes. Future works involve methodological improvement and prototype enhancements including the choice of alternative modality. Keywords: telecommunication, periphery, peripheral cues, distributed family.
1 Introduction This paper describes an effort to develop a new augmented communications environment which aims to engender a greater sense of social proximity to geographically distributed family members. We focus on the issues of supporting the exchange of peripheral communication cues individual family cultivated while they lived together [1] and the improvement of emotional well-being of families that live apart. The present research consists of a field study of peripheral communication cues in family relationships, development of an augmented communications environment based on the field study and its initial field evaluation. 1.1 Social Background The recent change of the social structure [2] and family composition steadily influences the transformation of family functioning [3] and view of the family [4][5]. The number of families that live apart, either by choice or necessity, has been increasing due to G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 147–156, 2011. © Springer-Verlag Berlin Heidelberg 2011
148
Y. Kinoe and M. Noda
various social circumstances. Additional trends, in part due to greater longevity, indicate a growing elderly population, often living in isolation from the rest of their families. In Japan, the percentage of people living alone including elderly people who live solitary lives increased by a staggering 28.6% between 1995 and 2005, while there was a drop of 2.5% in the percentage of nuclear family households [6]. This seems to be the trend for the next decades. Similarly, in the United States, the proportion of nuclear family households (two married parents and a child) dropped from 40% of all households in 1970 to 23% in 2005, while in the same period, the number of single adult households climbed from 16% to 28% [7]. These dramatic shifts in household composition entail significant changes in the nature of family relationships and, we argue, place increased importance on the role of communications technology for social benefit. 1.2 Related Works: Supporting Families Who Live-Apart Field studies of technology use in the home have been conducted for a variety of purposes include natural observational studies of family awareness [8] and assist family members living apart, in particular, seniors. Such technologies include a jacket for “hug over a distance” [9], family portraits [10], an augmented “planter” [11], a synchronized décor [12], information organizing systems [13], and an interactive installation supporting touch over a distance [14]. Social and emotional factors have also been considered within the eldercare experience of “aging in place” [15]. However, there have been comparatively few studies on the details of background communication among close individuals [11].
2 Peripheral Communications We believe that the reflection of individual family values in their life-world is important for developing communications support for family. Those values are associated with family episodes, memorabilia, emotional bond or attachments [16], habits, communication styles and special objects they keep in their homes [17]. People who live together consciously or unconsciously convey, perceive, and share various peripheral information in their everyday lives (Fig. 1). Examples include the cues of tone of voice, singing in the kitchen or shower, the pace of footsteps, doors being opened or slammed shut, light or music leaking through a door, movement of personal belongings such as keys, hats and bags of family members, and the aroma of coffee brewing or cookies baking. Each family has individual style of using these cues to gain awareness of the mood or physical presence of other family members. When family members move apart, these cues are no longer shared, which, we believe, diminishes the sense of close contact the family previously enjoyed. Our research project is investigating how technology can help convey these subtle, but significant elements of peripheral information to family members or partners living apart, so as to increase peripheral awareness of the state of loved ones without the technology becoming intrusive or overly demanding of foreground attention.
Designing Peripheral Communication Services for Families Living-Apart
149
Fig. 1. Family who live together convey, perceive, and exchange various peripheral communication cues among family members. They sometimes use these cues to gain awareness of the mood or physical presence of other family members.
3 Field Study: Seniors and Their Families In the design of a communications environment for close individuals who live apart, we believe that it is important to understand meaningful peripheral cues and family episodes, and how those cues are used by these individuals while living together. We conducted a field study consisting of a series of empirical sessions by combining multiple qualitative research methods. The aims were to collect and determine important peripheral cues for sensing presence and mood of family members, family episodes, memory triggers that evoke feelings of missing one another, and verify that our assumptions concerning the use of peripheral communications were valid. 3.1 Method and Research Settings The session of the field study involves interactive semi-structured interviews, completion of a questionnaire, in-situ contextual inquiry, and open-ended discussion. Eight respondents (one male and seven female), ranging between 48 and 83 years of age, participated. We also conducted complementary interviews with two care personnel of an assisted-living facility to understand background of everyday lives of elderly residents staying in the facility. These respondents were classified into three classes based on the current resident status; namely an elderly couple, pairs of elderly solitary parent and an independent child, and residents living in an assisted-living facility. Table 1 summarizes the profile of these individuals. The field study took place in the suburbs of Tokyo, Japan, between July and December of 2008. The sessions, spread over several weeks, were conducted on an individual basis to assure participants’ privacy and divided into two or three components. Table 2 describes overview of the sessions of our field study. Prior to the interviews, the introductory session facilitated for participants a better understanding and awareness of their everyday background communications with family members as well as the establishment of an appropriate rapport. The typical topics of the sessions included (a) profile information including family structure and overview of family
150
Y. Kinoe and M. Noda
history, conversational partner, topics, frequency, and method of communication, (b) cues for sensing presence of other family member, (c) cues for sensing mood of other family member, (d) memory triggers that evoke feelings of missing, (e) family memorabilia while living together and living-apart, (f) topics they wish to convey, and reality of their everyday situations of family communications. Table 1. Summary of profiles of the respondents
Class / Relation
# 1 2 3 4 5 6 7 8
Gender Living apart from Family Duration / Age
mother F daughter (married) F mother F daughter (married) F husband (in hospital) M elderly couple wife F F resident living in an assisted-living facility F
elderly solitary parent & independent child
83 58 79 48 83 74 81 83
daughter
parent daughter
parent partner partner son sister
30 years 20 years 4 years 30 years 13 years
Living with n/a husband, son n/a husband, son n/a son n/a n/a
Table 2. Overview of a structure and topics for the field study session
#
Module
1
• informed consent, rapport building • questionnaire, • semi-structured interview.
2, • • (3) • •
Detail
introduction, profile information including family structure and family history, conversational partner, topics, frequency, method of communication, cues for sensing presence and mood of other family member, memory triggers that evoke feelings of missing, cues for sensing presence and mood of other family member informed consent, (cont’d), questionnaire, semi-structured interview, 7. memory triggers that evoke feelings of missing (cont’d), in-situ contextual inquiry, 8. family memorabilia (while living together and living-apart), open-ended discussion. 9. topics you wish to convey, a gap between the wish and reality in their everyday family communications, 10. debriefing. 1. 2. 3. 4. 5. 6.
3.2 Analysis Approximately sixteen hours of verbal reports were obtained from the field interviews over sixteen sessions in total and contained a broad range of topics relating to communication practice among the families. All the verbal reports obtained from the field interviews were transcribed and analyzed according to a standardized analysis procedure of the VPA Method [18]. In this method, analysts first clarify the analysis viewpoints and define an analytic model that consists of a set of specific terms for characterizing the data. Then, analysts consistently characterize bulk of verbal reports using a standardized procedure of the “Segmentation & Tagging.” All the transcripts are divided into segments. Each segment is encoded with a tag, a combination of the attributes selected from each analysis viewpoints of the analytic model. In this study, the following viewpoints
Designing Peripheral Communication Services for Families Living-Apart
151
were established for encoding: (a) topical themes (e.g. recent news, memorable episode), (b) situation (Five Ws), (c) relationship with the person mentioned, (d) primary modality and (e) emotional response to the topic. 3.3 Results The field interviews involved a very broad topics such as recent joyful news from a grandchild, a regrettable incident with a married son, as well as family memorabilia during a few decades of family histories, what kind of cues were used, and how those cues were used for gaining the awareness of the moods of family members. Use of Conventional Communications Media. Several shortcomings of conventional communications media were noted. In general, emotional characteristics of communications, including one's overall mood and expressions of sarcasm or humor, were felt to be not as easily conveyed through communications media as in person. In particular, senior participants were hesitant to initiate contact using telecommunications technologies even when they wish to speak with their loved ones. Peripheral Cues. A total of 54 distinct peripheral communication cues were identified. These were analyzed according to the primary modality of individual cues (Table 3). Reference data obtained from our previous study using the same research framework for young adults in Montreal [1] are also presented (Table 3, right). The results indicate that visual (Tokyo:68.5%; Montreal:43.5%) and auditory (13.0%; 46.0%) were the two dominant modalities of peripheral communication cues with family. Very few cues were reported for olfactory (5.6%; 3.1%). A small number of cues involved the other modalities of somatosensory (0%; 1.9%), taste (0%; 0%), and multiple modalities over a particular interval, or did not involve any particular modality information. These were classified in “others” (13.0%; 7.5%). Table 3. Summary of peripheral communication cues obtained from the field study for elderly persons and their families in Tokyo (left), and reference data from our previous field study using the same research framework for young adults and their families in Montreal (right)
# Gender /Age 1 2 3 4 5 6 7 8
F F F F M F F F
83 58 79 48 83 74 81 83
(a) Tokyo Living Peripheral cues apart Vis Aud Olfact Oth from ual itory ory er daughter 4 0 0 2 parent 6 2 0 2 daughter 4 0 0 1 parent 6 0 0 0 partner 6 1 2 1 partner 8 3 0 1 son 3 1 1 0 sister 0 0 0 0
#
1 2 3 4 5 6 7
(b) Montreal (Kinoe & Cooperstock, 2007) Peripheral cues Gender/ Living Age apart Vis Aud Olfact Oth from ual itory ory er M 26 brother 14 14 0 0 M 23 brother 17 16 1 0 F 19 parent 9 16 0 3 F 22 partner 4 3 1 0 F 22 partner 13 9 0 4 F 23 partner 9 13 3 2 F 23 partner 4 3 0 0
Unforgettable Episodes of Family. Interestingly, significant portion of peripheral communication cues were mentioned in association with their family episodes. Those episodes concretely describe how they used the cues in a particular situation (i.e. Five
152
Y. Kinoe and M. Noda
Ws) with a particular affect. For example, the participant #4 in Tokyo genially smiled and described that a mother’s cane staying put in its holder was an explicit cue intelligibly indicating her mother was in good condition. Family episodes seem essential and sometimes work as a kind of symbol of their close relationships among the families that live-apart. Their semantics of the cues within family members have been developed through a long period of their life history even though these were not so meaningful to others.
4 Designing Peripheral Communication Services Based on the analysis results of the field study, the design principles were established to guide the development of a first-of-a-kind prototype for peripheral communication services. The principles were continuously refined and played a role of the cornerstone throughout the development. 4.1 Design Principles The following principles were established for designing a prototype which aims to support geographically distributed families to maintain and cultivate their relationships in a natural way. • Principle #1: Allow family members to exchange their unique peripheral communications cues at the adequate level of fidelity. • Principle #2: Snuggle up to family members’ everyday lives without their noticing. No additional attentions and actions are required for initiating and terminating their peripheral communications. • Principle #3: Deliver their cues along the line with the story of family episodes. 4.2 The “SharedEpisodes”: A Prototype for Peripheral Communications We developed several types of design prototype for facilitating peripheral communications among family members by utilizing different modality of peripheral communication cues such as visual, auditory and cutaneous sensation. The “SharedEpisodes” is a first-of-a-kind prototype which emphasizes on visual cues. First, based on the analysis of field study, several distinctive family episodes were selected from the field interviews and decomposed into elements of the target situation. Secondly, core features of the target family episode were incorporated into three levels of the layers of a prototype, namely basic, common and episode-specific. The “SharedEpisodes” senses specific events and peripheral communication cues caused by a family member’s behavior and visually delivers those cues to the remote party. Fig. 2 shows an example of its application featuring (a) the episode#1 of an elderly mother and her daughter and (b) the episode #2 of an elderly couple. Focus Episode #1: between elderly mother and her married daughter. A married daughter living with her husband and children often concerns health condition of her elderly mother who lives-apart. Her mother often enjoys going for a walk and sometimes leaves a cane when being in a good condition. While she lived with her mother, she reasoned about her mother’s condition by seeing a cane left in its holder
Designing Peripheral Communication Services for Families Living-Apart
153
near the front door. The “SharedEpisodes” supports their peripheral communications based on the episode. It senses contact state between a mother’s cane and its holder by using RFID tags/reader and then delivers the scene of her mother’s favorite cane resting in its holder to a digital picture frame in her daughter’s house (Fig. 2-a). This aims to help the daughter imagine her mother’s condition through visual imagery of personal relevance, without any intentional foreground communication. Focus Episode #2: between wife and her husband staying in a hospital. The husband of an elderly couple had been stayed in a hospital for several years. His wife hesitates making a call as it’s limited in a hospital. They engendered various heartwarming memories concerning gardening while they lived together. The husband well knows his wife enjoys gardening habitually at the turn of the year and remembers a beautiful hanging basket she made for him. The “SharedEpisodes” thus senses the movement of her trowel used in gardening and contact state between a trowel and hanging basket in the garden by using RFID tags/reader and then visually delivers the cues to a digital picture frame in a hospital room (Fig. 2-b). This aims to help the husband imagine his wife’s everyday life especially gardening through visual imagery of personal relevance, without any intentional foreground communication.
(a) Elderly mother and her married daughter
(b) Elderly couple – wife and her husband who is hospitalized for a long time
Fig. 2. The “SharedEpisodes” is a prototype for peripheral communications services. It senses meaningful events and peripheral communication cues caused by a family member’s specific behavior and visually delivers those cues to the remote party.
5 Evaluation of the “SharedEpisodes” We conducted a quick field evaluation of the “SharedEpisodes.” The aims were to evaluate its effects on family communications, their relationships and emotional wellbeing, and validate our design principles previously established.
154
Y. Kinoe and M. Noda
5.1 Methods and Research Settings The field evaluation took place in the suburbs of Tokyo, Japan, between December of 2008 and January of 2009. Four respondents including a pair of elderly mother and her daughter (participants #3, #4 in Table 1) and an elderly couple (participants #5, #6 in Table 1) who joined our previous field study, ranging between 48 and 83 years of age, again participated. It involves interactive semi-structured interviews, completion of a questionnaire and an open-ended discussion. The “SharedEpisodes” was installed in their living spaces. Due to the network limitation, the tool was modified to deliver the cues previously recorded off-line. The participants were asked to evaluate a total communications environment including the tool after they used it for seven days. The interview sessions were conducted on an individual basis and also asked to answer the same questionnaire at the timing of start and end of the deployment. 5.2 Results Three participants excluding the participant #3 (an elderly mother) claimed that the “SharedEpisodes” facilitated increasing opportunities for remembering family members and gaining the awareness of their presences. However, their responses were not always naively positive, but also contained complicated and sensitive feelings for gaining the awareness of the presence of their family members. Several changes between pre and post deployment of the prototype were found in the responses from the participants #4 (daughter of an elderly mother) and #6 (wife of an elderly couple) to a part of questionnaires on the status of their well-being. The participant #6 changed her answer to “completely agree” (post) from “agree” (pre) for the question “I feel a warm bond with my family member.” She claimed she could foster various aspects of her husband staying in a hospital while using the prototype. On the other hand, the participant #4 changed her answer to “completely agree” (post) from “somewhat agree” (pre) for the question “I get anxious about my life in the future.” She explained that after she began to use the prototype she paused to realize a great influence of the presence of her mother on her and became anxious with a possible pain of the loss of the presence of her mother in the future. 5.3 Discussions Contextual Use of Family Episodes. The participants’ overall responses indicated positive for communication support that aims to facilitate peripheral communications among family living-apart, along the line with the story of family-specific episodes. Mixed Feelings. One of the expected effects was to allay anxieties of the audience; however, the responses were more complex. For example, the participant #4’s response obviously involved mixed feelings of gratefulness and anxieties for her mother while gaining the awareness of the presence of the mother. It is suggested that a process of maintaining a close relationship of family involves not only delightful but also essentially mixed, sensitive, and sometimes anxious feelings for the family. Enhancing the Design Principles. Our strategy is to deploy the design principles throughout the development. The design principles #1, #2 and #3 provided a useful framework but are insufficient for realizing successful peripheral communication
Designing Peripheral Communication Services for Families Living-Apart
155
services. The results suggested the importance of allowing users to overlook or be unaware of the cues by obscuring them in the background when users don’t need them. This requirement closely relates to the key concept “periphery” [19]. The Choice of Modality and Presentation Method. A digital picture frame was very helpful for presenting the cues visually in a household setting, yet it sometimes attracted a certain degree of unexpected attentions of users. Ambient display technology or “peripheral audio” [1] will be a considerable alternative. Methodology and Process for Developing Peripheral Communications Services. Recently, various service engineering processes were presented. Anaby-Tavor et al. proposed a model-driven process consisting of a sequence of activities: analysis, strategy design, development, and deployment [20]. Our methodology adopted in this study provides a sub process that consists of (a) qualitative research in the field, (b) qualitative analysis and derivation of family values including cues, episodes, memorabilia of family, (c) establishment of design principles, and (d) development of peripheral communications environment and its field evaluation. These correspond to the analysis, design and development phases of the process.
6 Conclusion We developed a peripheral communications environment which aims to engender a greater sense of social proximity to distributed family members by facilitating the exchange of peripheral communication cues among family living-apart. A field study was conducted to determine important peripheral communication cues and episodes in family relationships. The design principles were established to guide the development of the “SharedEpisodes,” a prototype that delivers visual cues along the line with the story of family episodes. An initial field evaluation was also conducted. The participants’ overall responses indicated positive for the tool, but also seemed to involve mixed feelings. We are assessing the effectiveness of peripheral technologies as related to various design factors such as the choice of modalities, use of high- or low-fidelity media, and the contextual use of family-specific episodes, and evaluate their influence on users’ emotional responses. Our future works involve methodological enhancements, refinement of the design principles and prototype including the choice of alternative modality. We hope that this study, and the associated technologies being developed, will stimulate further research into the use of peripheral communications, helping distributed family members maintain awareness of each other’s state and emotional well-being. Potential audience of our peripheral technology seems to include close individuals also in different conditions of separation such as diaspora and immigrants. Acknowledgments. This work was partially supported by Grant-in-Aid for Scientific Research (20500675, 23300263). We thank our study participants and nursing-care staffs who volunteered their time to provide us with insight into the role of peripheral cues in family communication. We thank Dr. Jeremy R. Cooperstock who jointly conducted our previous studies in Montreal, and C.Honda, C.Ojima and Kinoe Lab. members of Hosei University who devotedly supported our field studies in Tokyo.
156
Y. Kinoe and M. Noda
References 1. Kinoe, Y., Cooperstock, J.R.: Peripheral telecommunications: Supporting distributed awareness and seamless transitions to the foreground. In: Okadome, T., Yamazaki, T., Makhtari, M. (eds.) ICOST. LNCS, vol. 4541, pp. 81–89. Springer, Heidelberg (2007) 2. Murdock, G.P.: Social Structure. Free Press (1965) 3. Parsons, T., Bales, R.F.: Family socialization and interaction process. Routledge (1956) 4. Holstein, J.A., Gubrium, J.F.: Constructing the Life Course, 2nd edn. General Hall (2000) 5. Fineman, M.A.: The Autonomy Myth: A Theory of Dependency. New Press, NY (2005) 6. Population Census of Japan, Statistics Bureau, Ministry of Internal Affairs and Communications of Japan, http://www.stat.go.jp/english/data/kokusei/2005/nihon/ index.htm (translated) 7. US Census Bureau datasets, http://factfinder.census.gov 8. Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B.B., Druin, A., Plaisant, C., et al.: Technology Probes: Inspiring Design for and with Families. In: Proc. ACM CHI 2003 (2003) 9. Vetere1, F., Gibbs, M.R., Kjeldskov, J., Howard, S., Mueller, F., et al.: Mediating Intimacy: Designing Technologies to Support Strong-Tie Relationships. In: Proc. ACM CHI 2005 (2005) 10. Vetere1, F., Gibbs, M.R., Kjeldskov, J., Howard, S., Mueller, F., et al.: Mediating Intimacy: Designing Technologies to Support Strong-Tie Relationships. In: Proc. ACM CHI 2005 (2005) 11. Miyajima, A., Itoh, Y., Itoh, M., Watanabe, T.: “Tsunagari-kan” Communication: Design of a New Telecommunication Environment and a Field Test with Family Members Living Apart. Intl. J. of Human-Computer Interaction 19(22), 253–276 (2005) 12. Tsujita, H., Tsukada, K., Siio, I.: SyncDecor: Appliances for Sharing Mutual Awareness between Lovers Separated by Distance. In: Extended Abstracts ACM CHI 2007, pp. 2699– 2704 (2007) 13. Taylor, A.S., Swan, L.: Artful Systems in the Home. In: Proc. ACM CHI 2005 (2005) 14. Hayashi T.: Mutsugoto 2007, http://www.tomokohayashi.com/mutsugoto2007.html 15. Hirsch, T., Forlizzi, J., Hyder, E., Goetz, J., Stroback, J., Kurtz, C.: The ELDer Project: Social and Emotional Factors in the Design of Eldercare Technologies. In: Proc. CUU 2000 (2000) 16. Bowlby, J.: A Secure Base: Parent-Child Attachment and Healthy Human Development. Basic Books, New York (1988) 17. Csikszentmihalyi, M., Rochberg-Halton, E.: The Meaning of Things: Domestic Symbols and the Self. Cambridge University Press, Cambridge (1981) 18. Kinoe, Y., Mori, H.: Discovering Latent Relationships among Ideas: A Methodology for Facilitating New Idea Creation. In: Bullinger, H., Ziegler, J. (eds.) Human–Computer Interaction: Ergonomics and User Interfaces, pp. 1242–1246. LEA (1999) 19. Weiser, M., Brown, J.S.: The Coming Age of Calm Technology (1996), http://www.ubiq.com/hypertext/weiser/acmfuture2endnote.htm 20. Anaby-Tavor, A., Amid, D., Sela, A., Fisher, A., Zhang, K., Tie Jun, O.: Towards a Model Driven Service Engineering Process. In: 2008 IEEE Congress on Services - I, pp. 503–510 (2008)
Visual Feedback to Reduce Influence of Delay on Video Chatting Kazuyoshi Murata, Masatsugu Hattori, and Yu Shibuya Kyoto Institute of Technology Matsugaski, Sakyo-ku, Kyoto 606-8585 Japan {kmurata,shibuya}@kit.ac.jp,
[email protected]
Abstract. When there is a certain delay in video chatting, participants often misunderstand other partners’ response and make unintended interruptions. In this paper, to overcome these problems, we present two kinds of visual feedback: Scroll Wave Indicator and Afterimage Indicator. An experiment was conducted to confirm effectiveness of these indicators. The result of experiment showed that the Scroll Wave Indicator helped participants to understand the remote participant’s response timing and decreased unintended interruptions even if there was 2 [sec] round-trip delay. Keywords: visual feedback, delay time, video chatting.
1 Introduction Video chat systems have been popular to support communication among spatially distributed friends, couples, or families. However, there are unavoidable problems caused by a delay in such video systems. For example, Ruhleder and Jordan [1] showed unintended interruptions as one of the particular problems due to delay for video-based remote communication. Fig.1 illustrates an example of unintended interruptions in delayed situation. If there is not a delay, an utterance of participant A is transmitted to participant B’s site right away and participant B can receive the utterance immediately. However, if there is a certain delay, each participant receives delayed utterance of other participant. In this example, participant A utters something (A1), and then waits responses from participant B. However, there is no response for a while because of delay. After a certain time, participant A starts another utterance (A2) because he/she assumes that participant B does not want to respond. Then participant A is interrupted by utterance of participant B (B1’) suddenly and he/she is confused. At participant B’s site, participant B receives a delayed utterance (A1’). Participant B starts responding to participant A (B1) after a certain thinking time, and then he/she is interrupted by next utterance of participant A suddenly (A2’). Participant B is also confused because the utterance of A2 is not related to A1’ and B1. These interruptions are called unintended interruptions. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 157–164, 2011. © Springer-Verlag Berlin Heidelberg 2011
158
K. Murata, M. Hattori, and Y. Shibuya
No response…
A’s waiting time A1 A2
A’s site
B1’
time B’s site
A2’ A1’ B1
Unintended Interruptions occured.
B’s thinking time
Fig. 1. An example of unintended interruption
Thus, when there is a certain delay in video chatting, participants often misunderstand partners’ response and make unintended interruptions. These misunderstandings and unintended interruptions disrupt conversation and make participants confused. In this situation, because participant A cannot know if there is some delay or not, he/she cannot distinguish the waiting time due to the delay from the thinking time of participant B. However, if the participant can recognize the relationship between the timing of the playback of his/her utterance at remote site and the timing of other participant’s response to it, he/she can understand the response timing of other participant and avoid these unintended interruptions. In this paper, we propose a visual feedback method to overcome these problems. The proposed method consists of two kinds of indicators: Scroll Wave Indicator and Afterimage Indicator. There are many studies aimed to reduce delay of video-based telecommunication. However, it is impossible to get rid of delay completely. The purpose of our proposed method is to reduce the influence of delay by improving user interface of video chat system instead of removing the delay itself.
2 Proposed Method Our proposed method provides visual feedbacks which show the status of voice and video playback of remote site. Fig.2 illustrates an example of display timing of these feedbacks. The display timings of these feedbacks are delayed for a length of a oneway delay from actual start timing and stop timing of the voice and video playback. That is, every start timing and stop timing of the visual feedbacks is delayed for the length of a round-trip delay from these timings of the participant’s own utterances. The length of his/her partner’s response timing at remote site (TR) is equal to the length of it at local site (TL) consequently.
Visual Feedback to Reduce Influence of Delay on Video Chatting
159
The participant who sees these feedbacks can recognize when his/her partner responded to his/her utterance. Consequently, misunderstandings of the response timing of his/her partner and occurrences of unintended interruptions are prevented.
B is listening to my voice, so I’ll wait until the playback stops.
The visual feedback is displayed after a round-trip delay from actual timing of A’s utterance.
TL ( = TR = B’s thinking time)
A’s site
time
B’s site
TR ( = B’s thinking time) Fig. 2. An example of display timing of proposed feedbacks
We introduced the Progress Bar Indicator to decrease unintended interruptions of voice chatting in our previous study [2]. However, it is difficult to apply this indicator to video chatting. Kawashima et al. proposed Visual Filler to solve these problems [3]. However, the Visual Filler requires inferring the end timing of the user’s utterance automatically. This inferring process is not needed for our proposed method. 2.1 Scroll Wave Indicator Fig. 3 shows an appearance of our proposed method. The Scroll Wave Indicator shows a state of a voice playback of a user and his/her partner. A waveform of upper area means an amplitude of the partner’s voice playback. This waveform appears from left end of a black rectangle area and scrolls toward left side of the indicator at a constant rate as the voice of the partner is played at the user’s site. That is, the waveform is displayed after a length of a one-way delay from actual start and end timing of an utterance of the partner. A waveform of lower area means an amplitude of the user’s voice. The waveform also displayed after a length of a round-trip delay. The waveform appears from the black line, and then it scrolls toward left side of this indicator at a constant rate. This black line is shifted to the right from the center of the lower area. At the center of the lower area, a dashed line is drawn. The distance between the black line and the dashed line means the length of the round-trip delay. That is, when the waveform reaches the halfway between the black line and the dashed line after it scrolled for the length of a
160
K. Murata, M. Hattori, and Y. Shibuya
one-way delay, the user’s voice starts to play at the partner’s site. Subsequently the waveform reaches the dashed line after it scrolled for the length of a round-trip delay. The waveform over the dashed line means the voice of the user which was being played at his/her partner’s site when the voice of the partner which is being played at the user’s site was uttered by the partner. By comparing the positional relation between the partner’s waveform and the user’s waveform, the user is able to understand the timing when the partner’s utterance which is being played at the present moment was uttered at his/her partner’s site. In addition, the user is able to infer which part of his/her utterance was played at the partner’s site from the figure of the waveform. The user is able to refrain from unnecessary utterance, e.g. “Can you hear me?”
9LGHRLPDJHRIDSDUWQHU
7KH6FUROO:DYH,QGLFDWRU 7KH$IWHULPDJH,QGLFDWRU $GHOD\HGVHOIYLGHRLPDJHLV FRPELQHGZLWKUHDOWLPHVHOI YLGHRLPDJH 7KH6FUROO:DYH,QGLFDWRUIRUYRLFHVXSSRUW
Speech
waveform of a Speech waveform of a participant.
Fig. 3. An example of display timing of proposed feedbacks
2.2 Afterimage Indicator Through video chatting, participants often communicate not only verbal messages but nonverbal messages, e.g. notations or gestures. Accordingly we propose the Afterimage Indicator as a visual feedback which presents the timing when such nonverbal messages are played through a video image at a remote site. An appearance of the Afterimage Indicator is also shown in Fig.3. In the Afterimage Indicator, a video image of a user which is delayed for a length of a round-trip delay is overlapped with a real time video image of the user. This overlapped image helps the user to
Visual Feedback to Reduce Influence of Delay on Video Chatting
161
understand the timing when the user’s nonverbal messages are played at his/her partner’s site.
3 Experiment We conducted an experiment to confirm the following effectiveness of our proposed method. • The proposed method helps a user to understand the timing of his/her partner’s response. • The proposed method decreases unintended interruptions. 3.1 Procedure Six pairs of volunteers, ranging in age 22 to 25 years, were participated in this experiment. They were four students, seven graduate students and one alumnus of our university. They had been known well each other and made daily face-to-face communication. Each participant of a pair was in separate rooms and both of them wore headsets. They talked to each other through the headsets. They were asked to make three minutes discussion. The discussion topic was to line up three candidates for certain goal. For example, “You are going to have lunch. What kind of lunch will you have?” They were asked to make the discussion with our proposed method. Moreover, they were asked to do it with usual video chat system (hereafter, “conventional method”). The video chat system had two video windows of the partner and the participant himself/herself, and they talked each other through headsets. This experiment was performed under three kinds of round-trip delay conditions: 0, 1 and 2 [sec]. 3.2 Measures We measured rates of unintended interruptions in this experiment. In addition, participants were asked to answer a questionnaire for subjective evaluation after each discussion. Rate of unintended interruptions. The rate of unintended interruptions (hereafter, “RUI”) was defined as a following equation. RUI = NIU / NUP. NIU: Number of utterances interrupted by a partner that were uttered by a participant successively. NUP: Number of utterances of a participant.
Participants were able to communicate by using both of voice and a user’s body motion through headset and video image on PC in this experiment. Therefore we considered that there were two kinds of unintended interruptions. One was unintended interruptions by a partner’s voice through a headset, and the other was unintended interruptions by a partner’s body motion through a video image. We defined the former as RUI by voice, and the latter as RUI by video. The Scroll Wave Indicator was expected to decrease RUI by voice and the Afterimage Indicator was expected to decrease RUI by video.
162
K. Murata, M. Hattori, and Y. Shibuya
Questionnaire. The participants were asked to answer a questionnaire including following questions. 1. You could understand the timing when your partner responded to your utterance. (1: disagree – 5: Agree ) 2. How often did you watch the self-video image on PC? (1: didn’t watch it – 5: always watched it)
4 Result and Discussion Fig. 4 shows the result of RUI by voice. In the case of 2 [sec] delay condition, a t-test revealed a significant difference between the RUI by voice with proposed method and that with conventional method (p<.05). There was no significant difference in the case of 1 [sec] condition. Fig. 5 shows the average score of question #1. An ANOVA showed a significant main effect of delay condition in the case of using the conventional method (F2,11=4.696, p<.05). A post-hoc test revealed that the score of 2 [sec] delay condition was significantly lower than that of 0 [sec] delay condition. However, there was no significant difference in the case of using the proposed method.
Conventional method
Proposed method
Rate of unintended interruptions
0.4
*: p<.05
0.3 0.2 0.1 0 1 [sec] delay
2 [sec] delay
Average score of question #1
Conventional Proposed Fig. 4. Themethod result of RUI by voice.method *: p<.05 5 4 3 2 1 0 [sec] delay
1 [sec] delay
2 [sec] delay
Fig. 4. The average score of question #1
Visual Feedback to Reduce Influence of Delay on Video Chatting
Conventional method
163
Proposed method
Average score of question #2
5 4 3 2 1 0 [sec] delay
1 [sec] delay
2 [sec] delay
Fig. 5. The average score of question #2
Thus, as the length of the delay was extended, it became hard to understand the timing of a partner’s response with the conventional method. On the other hand, even if the delay was 2 [sec], the participant with the proposed method was able to understand the timing easily. These results indicate that the Scroll Wave Indicator helps a participant to understand the timing of his/her partner’s response. Furthermore, these results suggest that the Scroll Wave Indicator decreases unintended interruptions. With regard to the Afterimage Indicator, the effect of the Afterimage Indicator was unable to be analyzed because there were few unintended interruptions by a participant’s body motion. In this experiment, the task which performed by participants did not require them to use nonverbal expressions by body motion. They therefore needed not to use a video image on PC. The score of question #2 is shown in Fig. 6. There was no difference between the score of the proposed method and that of the conventional method. This result indicates that participants barely watch their self-video image. Some participants commented that the overlapped delayed selfvideo image was obstacle to see the usual self-video image. To solve this problem, we need to improve the Afterimage Indicator which combines the effectiveness of the indicator and visibility of the self-video image.
5 Conclusion In this paper, the Scroll Wave Indicator and the Afterimage Indicator are proposed and evaluated experimentally. The result of evaluation showed that the Scroll Wave Indicator made participants understand the remote participant’s response timing and decreased unintended interruptions even if there was 2 [sec] round-trip delay. However, the effectiveness of the Afterimage Indicator was not shown. Future work will involve an experiment using tasks that should be performed with both of verbal and nonverbal communications.
164
K. Murata, M. Hattori, and Y. Shibuya
References 1. Ruhleder, K., Jordan, B.: Co-Constructing Non-Mutual Realities: Delay-Generated Trouble in Distributed Interaction. Computer Supported Cooperative Work 10(1), 113–138 (2000) 2. Murata, K., Nakamura, M., Shibuya, Y., Kuramoto, I., Tsujino, Y.: Visual feedback to reduce the negative effects of message transfer delay on voice chatting. In: Smith, M.J., Salvendy, G. (eds.) HCII 2007. LNCS, vol. 4558, pp. 95–101. Springer, Heidelberg (2007) 3. Kawashima, H., Nishikawa, T., Matsuyama, T.: Visual filler: Facilitating Smooth Turntaking in Video Conferencing with Transmission Delay. In: CHI 2008 extended abstracts on Human factors in computing systems, pp. 3585–3590 (2008)
Research on the Relationships between Visual Entertainment Factor and Chat Communication Tomoyasu Ogaki, Junko Itou, and Jun Munemori Graduate School of systems Engineering, Wakayama University, 930, Sakaedani, Wakayama 640-8510, Japan Faculty of systems Engineering, Wakayama University, 930, Sakaedani, Wakayama 640-8510, Japan
[email protected], {itou,munemori}@sys.wakayama-u.ac.jp
Abstract. In this article, we analyze the effects of visual entertainment factors included in visual cues on chat communication aiming to enliven exchanging chat messages. Visual cues such as smileys, avatars and pictograms are essential to make our communication successful. However, visual cues can be used as only a substitution to express user's intentions. Therefore we propose a chat system with characters which change different forms according to chat messages input bye users with visual cues and we investigate the effects on exchanging chat messages.
1 Introduction The use of online communication is becoming widespread and the diversity of online communication tools has increased in recent years. For example, several online communication tools such as e-mail, chat system, and distance learning are currently in use. Further, varieties of chat systems provide features, from conventional textbased ones to those that use a graphical interface in which virtual 3D agents represent users in a virtual 3D space. Using visual cues including smileys, avatars and pictograms, users can visually express their emotions or intentions in their chat messages. Visual cues facilitate smooth communication between chat system users. Many people use visual cues in their daily electronic text-based conversations. Over 70 % of cellular phone users use a pictograph in their communication according to an investigation that took into account people of all age groups from senior citizens to young people [1]. According to another research, approximately 90% of university students use emoticons in their daily e-mail messages [2]. Moreover, an investigation revealed that among young people aware of an avatar, approximately 50% used an avatar on the Internet and approximately 70% wanted to use an avatar [3]. Therefore, it can be deduced that the visual cues mentioned above are used by several people from different generations in their daily communication, including chat communication, and that visual cues are useful for everyday conversations. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 165–170, 2011. © Springer-Verlag Berlin Heidelberg 2011
166
T. Ogaki, J. Itou, and J. Munemori
Previous works on chat systems mainly discuss how visual cues can be used for smooth communication. In this work, we focus on the use of visual cues for creating a sense of fulfillment as well as for entertainment. We propose a chat system with visual entertainment factors and visual entertainment features that users can bring up to recreate their original characters through chat, and we investigate how visual entertainment features affect chat communication.
2 Application of Visual Cues to Communication Systems 2.1 Situations Where Visual Cues Are Used Visual cues include emoticons, pictographs, and avatars. Emoticons are a combination of a character and the sign that can be employed to express a use’s emotion. Emoticons are commonly used at the end of a sentence in e-mail, chat conversations, and even bulletin boards. Pictographs consist of texts. It is used as a picture on cellular phones and instant messengers. Pictographs can be of various types, such as those expressing emotional states, or those depicting animals, plants, and buildings. User’s employ an avatar is as an agent chat, instant messengers and online games. An avatar is a human-like character that can easily express the user’s emotions as well as emoticons and pictographs can do. From the above explanation, it is clear that these visual cues play an important role in expressing a user’s emotions and intentions more accurately in an e-mail or chat system. 2.2 Communication Systems Many visual cues have already incorporate communication systems. Instant messengers are applications that enable the members logged into chat with one another. Latest instant messengers allow users to exchange messages containing pictographs, animations and sounds. Koda proposed a communication tool that can write messages on a picture of a character displayed on the screen of the user’s personal computer [4]. The user can choose from many different types of character templates according to the message content. “MEDIAC Messenger” is a chat system that users can use 3D characters as their agents [5]. In this messenger, 3D characters perform various actions in the window on the desktop and react to the words typed in chat messages. This chat system focuses on the relationships among emotional words, the actions of the characters, and the reactions of the characters. This chat system also allows visual cues such as pictographs, smileys, and avatars. However, the main purpose of this system is to convey the user’s emotions and intentions. Our aim is to use visual cue to obtain an active feedback and a passive response in a chat system.
Relationships between Visual Entertainment Factor and Chat Communication
167
3 Chat Systems with Visual Entertainment Features 3.1 Goal Our goal is to evaluate the proposed chat system on the basis of its visual entertainment features. We used visual cues for not only smooth communication but also humorous communication: therefore, we introduced “evolving of a character” into our chat system as a visual entertainment features. We designed characters that can induce positive feelings in users by changing their aspects in order to confirm how the character affects users and whether it triggers chat communications. 3.2 Evolving of a Character The designed character evolves into newer forms according to the words, smileys, and emoticons used in chat messages. The overview of the evolving of a character is shown in Fig.1. The characters change their forms when the number of input chat messages exceeds a threshold value. When a user inputs “positive” words or smileys often, the state of the character changes to that on the left side in the second step with a high probability, as shown Fig.1. On the other hand, when a user inputs more “negative” words or smileys, the state of character tends to change to that shown on the right side in the second step. The “Positive” words includes “happy”, “delight”, “smile” and so on. The “Negative” words includes “angry”, “tired”, “sad”, “bored”, and so on. This system accepts twenty types of keywords and seven types of characters.
Fig. 1. Evolving of a character
168
T. Ogaki, J. Itou, and J. Munemori
3.3 System Structure Our chat system consists of a server and multiple clients. When users start the client system, some windows open up on their screen as shown in Fig.2. The right window displays characters or the users’ avatars, chat logs, input forms and a list of emoticons. In the top portion of this window, users’ avatars perform various actins and evolve according to the chat text input bye users. After user A inputs a message in the field provided in the chat window and sends it to the server, the server extracts a keyword from the sent message by morphological analysis to determine the action that character A should perform. The instructions for the action that the character should perform and user A’s message are sent from the server to each client. When a user selects an emotion icon from the list as an input, a balloon containing that emotion icon is displayed. The left window displays a list of smileys. Smiles are divided into five categories: “delight”, “angry”, “sorrow”, “surprising” and “others”. Users can easily input smileys into chat messages from this list.
Fig. 2. Overview of proposed system
4 Experimental Results To investigate the role of visual entertainment features in triggering chat communication, we performed an evaluation experiment using three types of chat systems: (a) text-based chat, (b) avatar chat, and (c) chat with an evolving avatar. In chat system (a), users can input only text. In chat system (b), they can use text as well as an avatar that doesn’t change its form. Chat system (c) is the proposed chat system.
Relationships between Visual Entertainment Factor and Chat Communication
169
The experimental subjects were ten college students: they were divided into five pairs. They were made to sit away from each other to prevent them from knowing their partner’s emotions or reactions. We instructed each subject to freely chat online for 15 min using smileys and/ or emotion icons. They used the chat systems in order of chat system (a) to chat system (c). The result is summarized in Table.1. Averages were calculated for each type of data. The table entry “visual cue” indicates that the corresponding average is for those users that selected smileys and emotion icons. From Table.1, it is clear that the participants had a tendency to use visual cues actively and the number of messages and wordage increased significantly when they used chat system (c). This result shows that use of visual entertainment features triggers user’s conversations and does not disturb the exchange of messages between users. Table 1. Averages of different kinds of chat-related data
number of messages total wordage visual cues
(a) 17.88 273.50 0.13
(b) 20.25 319.25 4.00
(c) 23.00 369.63 9.74
The results of a questionnaire filled out by the subjects are provided in Tables.2 and 3. From table 2, it can be understood how chat system (c) affects chat conversation. From the evaluated values listed, it can be concluded that the character and other emoticons did not disturb the chat communication between users and that the participants could communicate as smoothly using chat system (c) as they did using a traditional chat system such as system (a). Table 2. Effect of the chat system with an evolving character on chat conversations
Questionnaire Item ( i ) Use of the chat system with an evolving character disturb your chat conversation (1: yes - 5: not at all) ( ii ) Use of the chat system with an evolving character makes you input messages in a proactive (1: not at all - 5: yes) (iii) Emoticons are useful in chat conversation (1: not at all - 5: yes)
Average 4.2
3.9 4.2
According to Table.3, the evaluated value for “ease in conversing” for chat system (c) was 4.1 and the decentralized value was 0.1. On the other hand, the decentralized value of the same item for chat system (a) was 0.8. This implies that most subjects tend to feel that they can easily exchange messages using chat system (c). From these results, we deduce that our proposed chat system enlivens chat conversations with the
170
T. Ogaki, J. Itou, and J. Munemori
help of visual entertainment features that help trigger a dialogue between subjects and that the subjects enjoy their chat. Table 3. Evaluation of each chat system
Questionnaire Item Ease in conversing Fun in conversing Triggering of conversations
(a) 3.0 2.8 2.3
(b) 3.5 3.7 3.5
(c) 4.1 4.6 4.4
5 Conclusions In this article, we discussed the effects of visual entertainment features on the chat conversations of users in a chat system. By a comparison test of three types of chat systems, we were able to confirm that visual entertainment features trigger the exchange of chat messages. In this experiment, we used each chat system only once and for a short term. Therefore, we could not observe how the continual use of our chat system affects chat conversation. In the future, we plan to consider this point and improve our chat system.
References 1. MyVoice Communications, Inc. The research on commonly-used function of cellar phone (2008), http://www.myvoice.co.jp/biz/surveys/11601/ 2. Okamoto, E.: Research on transition of emotion by face mark, Sonoda Women’s University (2002) 3. NTT ADVERTISING, Inc. The research on utilization and recognition of ‘avatar’ in internet sites, NEWS RELEASE (October 17, 2007), http://www.ntt-ad.co.jp/news/20071017/20071017.pdf 4. Koda, T.: Development and Analysis of an Emotionally Expressive Communication Tool "Petaro". The Transaction of Human Interface Society 8(1), 101–108 (2006) 5. Itou, J., Hoshio, K., Munemori, J.: A Prototype of a Chat System Using Message Driven and Interactive Actions Character. In: 10th International Conference on Knowledge-based Intelligent Information & Engineering Systems, KES 2006 (2006)
Multimodal Conversation Scene Analysis for Understanding People’s Communicative Behaviors in Face-to-Face Meetings Kazuhiro Otsuka NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corp., 3-1 Morinosato-Wakamiya, Atsugi-shi, Kanagawa-pref., 243-0198 Japan
[email protected] http://www.brl.ntt.co.jp/people/otsuka/
Abstract. This presentation overviews our recent progress in multimodal conversation scene analysis, and discusses its future in terms of designing better human-to-human communication systems. Conversation scene analysis aims to provide the automatic description of conversation scenes from the multimodal nonverbal behaviors of participants as captured by cameras and microphones. So far, the author’s group has proposed a research framework based on the probabilistic modeling of conversation phenomena for solving several basic problems including speaker diarization, i.e. “who is speaking when”, addressee identification, i.e. “who is talking to whom”, interaction structure, i.e. “who is responding to whom”, the estimation of visual focus of attention (VFOA), i.e. “who is looking at whom”, and the inference of interpersonal emotion such as “who has empathy/antipathy with whom”, from observed multimodal behaviors including utterances, head pose, head gestures, eye-gaze, and facial expressions. This paper overviews our approach and discusses how conversation scene analysis can be extended to enhance the design process of computer-mediated communication systems. Keywords: Conversation scene analysis, meeting analysis, multimodal interaction, nonverbal behavior.
1 Introduction Face-to-face conversation is one of the most basic forms of communication in daily life and group meetings are used for conveying/sharing information, understanding others’ intention/emotion, and making decisions. Despite the rapid progress in network infrastructure, mobile devices, and social media, visual communication among spatially separated people is still impeded by unnatural and uncomfortable usability, it remains quite different from face-to-face conversations. To design better communication systems, the author believes that it is critical to elucidate the mechanism of human communication, i.e. how people communicate/interact with each other. In recent years, the analysis of human conversation such as group meetings has been acknowledged as an emerging research field, and a growing number of researchers are becoming engaged in this field [1]. G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 171–179, 2011. © Springer-Verlag Berlin Heidelberg 2011
172
K. Otsuka
In face-to-face conversations, people exchange not only verbal messages, but also nonverbal messages, such as eye-gaze, head and body gestures, facial expression, and prosody [2]. Therefore, it is expected that conversation structures and dynamics can be largely understood by observing the nonverbal behaviors of the conversing participants. Conversation scene analysis aims to provide the automatic description of conversation scenes from such multimodal nonverbal behaviors of participants as captured by cameras and microphones. It ranges from low-level (e.g. physical and/or behavioral) to high-level attributes (e.g. mental and/or social). So far, the author’s group has proposed a research framework that probabilistically models conversation phenomena for solving several basic problems including speaker diarization, i.e. “who is speaking when” [3, 4], addressee identification, i.e. “who is talking to whom” [5, 6, 3], the estimation of visual focus of attention (VFOA), i.e. “who is looking at whom” [5, 6, 7], interaction structure, i.e. “who is responding to whom” [8, 9], and interpersonal emotion such as “who has empathy/antipathy with whom” [10], from observed multimodal behaviors including utterance, head pose, head gestures, eyegaze, and facial expressions. This paper overviews our approach to multimodal conversation scene analysis. In Section 2, we introduce our concept of the probabilistic modeling of conversation phenomena and several examples. Section 3 details our sensing approach including a real-time system for conversation scene analysis. In section 4, I will introduce our latest effort toward understanding interpersonal emotions such as empathy and antipathy during conversations. Finally, in Section 5, we will discuss how conversation scene analysis can advance the design process of human-human communication systems.
2 Conversation Scene Analysis Based on Probabilistic Conversation Models The main components of our approach to conversation scene analysis involve probabilistic conversation models and multimodal sensing of human nonverbal behaviors. This section sheds light on the modeling part and later, in section 4, the sensing part is presented. Our approach to conversation modeling features i) probabilistic modeling, ii) a hierarchical generative model, and iii) unsupervised inference. This approach assumes that conversation phenomena can be modeled as generative hierarchical models, where the higher layer governs the dynamics of the lower layer. Based on this idea, the author has developed a dynamic Bayesian network (DBN) with three layers. In the following subsections, we introduce our methods for estimating the joint visual focus of attention (VFOA) of meeting participants, i.e. “who is looking at whom”, and addressee identification, “who is talking to whom” [5, 6]. Next, section 2.2 introduce the problem of estimating an interaction structure, i.e. “who is responding to whom” [8, 9]. Section 2.3 shows our recent approach for estimating VFOA by joint modeling of head pose and imagebased gaze detection [7].
Multimodal Conversation Scene Analysis
173
Fig. 1. Conversation Model. (a) Concept of layered structure, (b) Basic model for estimating gaze patterns and conversation regimes. (c) Enhanced model for inferring interaction structures involving utterance and head gestures.
Fig. 2. Gaze pattern and indicated types of conversation, called conversation regimes. (a) most frequent gaze patterns in a 4-person conversation (left to right). Nodes represent people and arrows indicate interpersonal gaze directions. No arrow indicates looking at no one. (b) The three conversation regimes are convergence, dyad-link, and divergence. The typical topology of gaze patterns is shown for each regime. A regime can indicate participants’ roles such as speaker, addressees, and side-participants during a conversation.
2.1 Joint Estimation of Visual Focus of Attention and Conversation Structures Gaze, also called visual focus of attention (VFOA), has been of interest to psychologists for many years. They have advanced several functions of gaze such as monitoring others, expressing one’s attitudes/intentions, and regulating turn-taking [11, 12]. In 2005 [5], authors targeted the problem of estimating visual focus of attention (VFOA), i.e. “who is looking at whom”, in multiparty conversations, and proposed “conversation structure”, the attempt at first joint modeling of multiple people’s gaze, in addition to conversational context. The structure indicates “who is talking to whom, and who is listening to whom”. It also addresses the subsequent
174
K. Otsuka
problem of addressee identification. Fig. 1 illustrates the main concept of the model. The top layer, called the conversation regime, represents the global status of conversation, especially the patterns of message exchange among people, e.g. “who is talking to whom and who is listening to whom”. The middle layer, the interaction layer, represents how people interact with each other, e.g. gaze interaction. The bottom layer is behavior layer, which includes the various nonverbal behaviors of meeting participants such as head pose, presence/absence of utterance, head gestures, and facial expressions. These human behaviors can be measured with sensors [5], computer vision techniques [3, 6, 9], and audio signal processing [3, 4]. The resulting model can be represented as a Bayesian network as shown in Fig. 1(b) and (c). The model in Fig. 1(b) is directed at the problem raised in this section, while the one in Fig. 1(c) is used in the next section. In the model in Fig. 1(b), the conversation regime controls the dynamics of gaze patterns and utterances. The gaze pattern is a hidden variable and is estimated from head-pose measurements in the bottom observation layer. The key idea of this model is the hypothesis that the topology of gaze pattern (a set of gaze directions of all participants) is strongly related to the basic conversation structures, i.e. “who is talking to whom, and who is listening to whom”, we call these conversation regimes. Fig. 2(a) shows typical gaze patterns and our conversation regimes. This model is based on the common observation that “speaker orients his/her gaze to addressees, who tend to look at speaker”. With this model, the regimes and the gaze patterns are jointly estimated from the utterances and the head directions measured with sensors or face tracking in videos; the estimation is implemented using a MCMC (Markov chain Monte Carlo) method. Author’s contributions include shedding light on the joint gaze directions of people and rendering the problem suitable for computational analysis. 2.2 Estimation of Multimodal Interaction Structures, i.e. “Who Responds to Whom, When, and How?” In addition, an extended model has been proposed for analyzing cross-modal nonverbal interactions, i.e. “who responds to whom, when, and how”, from multimodal cues including utterances, gaze, head gestures [8, 9]. Fig. 1(c) shows the structure of this model. This model focuses in particular on head gestures such as nodding, shaking, and tilting, which play important roles in conversations. Speaker’s head gestures appear as visible signs of addressing, questioning, and stressing, and hearer’s gestures can be interpreted as signs of listening, acknowledgement, and agreement/disagreement [13]. Their gestures are used to regulate various interactions such as question/answer, addressing/back-channel response, and yielding the floor. As an additional hidden variable to be estimated, this model incorporates a causal relationship among behaviors, called interaction structure, i.e. “which behavior triggers which behavior”, where behavior refers to utterances and head gestures. Fig. 3 shows two ways of visualizing the estimation results. In Fig. 3 (a), the wave from the rightmost person (P4) to leftmost person (P1) indicates that P4's nod is a reaction toward P1. In Fig. 3 (b), reactions are indicated by arrows starting from responder to the triggering behavior of the other.
Multimodal Conversation Scene Analysis
175
Fig. 3. Visualization of interaction structures. (a) spatial diagram, (b) temporal diagram. In (a), arrows indicate estimated gaze directions, and wave starts from responder to person who triggered the response. Fig. 3 (b) shows detected utterance intervals (gray rectangles), detected head gestures (lines below utterance), the estimated response (white arrows started from responder to the triggering person).
2.3 Estimation of Visual Focus of Attention from Head Pose and Image-Based Gaze Detection So far, head pose has been considered as a reasonable indicator of gaze direction (VFOA) in the field of meeting analysis. However, it can be ambiguous and the determination of correct gaze direction can often fail because a single head pose affords multiple gaze directions. For more reliable estimation, the new conversation model incorporates an image-based gaze detector, which can model of joint head and eye-gaze dynamics [7]. Its superiority over head-only models is particularly strong when dealing with complex attention shifts and gaze aversion dynamics.
3 Real-Time Meeting Analysis System Sensing people's behavior is another important part of conversation scene analysis. People's nonverbal behaviors are mainly exchanged through audio and visual channels. Therefore, audio signal processing and computer vision techniques are expected to be useful for sensing and measuring the nonverbal behaviors. To advance sensing technology, the author’s group developed a real-time system for analyzing group conversations using an omnidirectional camera-microphone sensor [3], which is combined with face position/pose tracking and voice detection with arrival direction, to determine “who is talking to whom” and “who attracts most attention”, as shown in Fig. 4. This system targets a round table meeting with 4 to 8 people, and captures the meeting scenes with omnidirectional camera-microphones, as shown in Fig 4(c). Of particular note, we developed a GPU-based face tracker, called STCTracker [14], that uses Memory-based particle filtering [15], which can realize both real-time speed and robustness. Real-time response is the key to realizing practical applications such as computer-mediated teleconferencing and conversational robot/agents.
176
K. Otsuka
Fig. 4. Real-time system for group meeting analysis, (a) meeting scenes, (b) real-time 3D visualization, (c) omnidirectional camera-microphone
Fig. 5. Model of emotional interaction. The model links the emotional interaction (empathy/ antipathy/unconcern), gaze patterns, and facial expressions between two people, based on cooccurrence nature of facial expressions when people interact to confirm/exchange/share their emotion with others.
4 Inferring Emotional Interactions: Who Feels Empathy with Whom? Beyond the structural aspect of conversations, the next challenge is discovering people's emotions, i.e. “what kind of emotion is raised by others” and “how human emotions change with interactions in conversations”. Recently, the author’s group has focused on interpersonal emotion developed in the course of conversational interactions. We pay particular attention to empathy and antipathy, and aim to automatically infer “who shared empathy/antipathy with whom and when” [10]. To that end, we focus on the facial expressions of the participants as a useful cue, and hypothesize that the inter-play of facial expressions between two people is strongly related to their emotional interactions, which shape empathy/antipathy between the two. For example, a typical interaction in conversations is for the recipient of a smile (a facial expression) to returns it. Here, if the recipient feels empathy with the first
Multimodal Conversation Scene Analysis
177
person in terms of his/her opinion and/or his/her personality, the recipient shares the feeling of the first person. Note that empathy sharing can be evidenced by sad-sad exchange in addition to smile-smile exchange, i.e. when talking about sad events. On the other hand, responding with an opposite expression, e.g. anger from smile, can indicate antipathy. No response means unconcern or sometimes a sign of antipathy to the person. Based on our hypothesis, we have developed a probabilistic model that links the emotional interaction (empathy/antipathy), co-concurrency of facial expressions between the pair of people, their gaze directions, and utterances. MCMC (Markov chain Monte Carlo) simulation yielded the posterior distribution of emotional interaction that well replicated the judgment of human external observers who watched the video. Here, we introduce a new scheme for evaluating human emotions. Instead of assuming that there is one solid ground truth in the mind of the targeted person that can be reliably measured, we focus on the statistical nature of human discernment of others’ emotion from the observer’s viewpoint, i.e. human observation/interpretation inherently varies from person to person. From this point of view, we evaluated the estimation results by measuring the distance between two distributions, one is the posterior distribution from the model computation, and the other is the histogram obtained from annotations by multiple human observers.
Fig. 6. Design framework for developing human-to-human communication system based on conversation scene analysis
5 Discussion of Communication System Design The author believes that conversation scene analysis has the potential to yield enhanced communication system designs for spatially separated people. Fig. 6 illustrates our research and design framework based on conversation scene analysis for developing such systems. In Fig. 6, the group meeting on the left can be analyzed by the techniques mentioned in this paper, and the result of the analysis can be used for synthesizing/visualizing meeting scenes for remote viewer(s), see the right hand of Fig. 6. This flow is denoted by the arrow (a) in Fig. 6. We have developed to date a simple visualization scheme as mentioned in the earlier sections. Further sophistication is an important research topic for better understanding of remote meetings by remote users. Moreover, for bi-directional communication, the return flow, which also involves analysis and synthesis as indicated as arrow (b), is required
178
K. Otsuka
to send the remote user’s image/audio back to the meeting participants on the left. Moreover, to evaluate such computer-mediate communication systems ((c) in Fig. 6), the idea and techniques of conversation scene analysis are useful. The analysis is expected to yield several quantitative measures for assessing how naturally people behave and conduct communication through the system, based on their behaviors including gaze, utterances, gestures, and facial expressions. A comparison of the behaviors between face-to-face setting and remote communication systems would provide important clues for assessing naturalness in remote communication sessions. Such quantitative assessments and related insights will enable designers to improve remote systems in terms of both analysis and synthesis components (indicated as (d) in Fig. 6). For example, a remote system might degenerate the impact of some nonverbal behaviors because of the limited visual communication channel, and the analysis could tell system designers what nonverbal behavior was weakened and must be reinforced. In addition, this framework would also be useful for developing agent/robot communication systems.
6 Conclusion This paper introduced our approach to multimodal conversation scene analysis, and discussed its future in terms of designing better human-to-human communication systems. So far, the author’s group has proposed a research framework based on probabilistic modeling of conversation phenomena to solve several basic problems including speaker diarization, addressee identification, an interaction structure, the estimation of visual focus of attention (VFOA), and the inference of interpersonal emotion, from observed multimodal behaviors including utterance, head pose, head gestures, eye-gaze, and facial expressions. In addition, this paper discusses how the conversation scene analysis can be extended to and can enhance the design process of computer-mediated communication systems. Multimodal analysis of people's communication enables a deeper understanding of the human-human relationship in all aspects of human life, not only in the online world, but also in real life. Future works include following. First, it is necessary to target more realistic environments, which include not only human-human conversation, but various activities involving objects, tools, and environments. Therefore, continuous effort is needed to develop more robust sensing technologies for both vision and audio processing that can deal with low quality image and audio captured in highly cluttered and noisy environments. In addition, full multimodal integration and modeling is essential for understanding high-level aspects of human communication such as cultural and language differences. Third, it will contribute to social computing such as social network analysis. Finally, it would be interesting to explore its possible application in psychology and sociology research.
References 1. Gatica-Perez, D.: Automatic Nonverbal Analysis of Social Interaction in Small Groups: A Review. Image and Vision Computing 27, 1775–1787 (2009) 2. Argyle, M.: Bodily Communication, 2nd edn. Routledge, London and New York (1988)
Multimodal Conversation Scene Analysis
179
3. Otsuka, K., Araki, S., Ishizuka, K., Fujimoto, F., Heinrich, M., Yamato, J.: A Realtime Multimodal System for Analyzing Group Meetings by Combining Face Pose Tracking and Speaker Diarization. In: ACM ICMI 2008, pp. 257–264 (2008) 4. Ishizuka, K., Araki, S., Otsuka, K., Nakatani, T., Fujimoto, M.: A Speaker Diarization Method based on the Probabilistic Fusion of Audio-visual Location Information. In: ICMI 2009, pp. 55–62 (2009) 5. Otsuka, K., Takemae, Y., Yamato, J., Murase, H.: A Probabilistic Inference of MultipartyConversation Structure based on Markov-Switching Models of Gaze Patterns, Head Directions, and Utterances. In: ACM ICMI 2005, pp. 191–198 (2005) 6. Otsuka, K., Yamato, J., Murase, H.: Conversation Scene Analysis with Dynamic Bayesian Network based on Visual Head Tracking. In: ICME 2006, pp. 949–952 (2006) 7. Gorga, S., Otsuka, K.: Conversation Scene Analysis based on Dynamic Bayesian Network and Image-based Gaze Detection. In: ACM ICMI-MLMI 2010 (2010) 8. Otsuka, K., Sawada, H., Yamato, J.: Automatic Inference of Cross-modal Nonverbal Interactions in Multiparty Conversations. In: ACM ICMI 2007, pp. 255–262 (2007) 9. Otsuka, K., Yamato, J.: Fast and Robust Face Tracking for Analyzing Multiparty Face-toFace Meetings. In: MLM 2008 (2008) 10. Kumano, S., Otsuka, K., Mikami, D., Yamato, J.: Analyzing Empathetic Interactions based on the Probabilistic Modeling of the Co-occurrence Patterns of Facial Expressions in Group Meetings. In: 9th IEEE Conference on Automatic Face and Gesture Recognition, FG 2011 (2011) 11. Kendon, A.: Some Functions of Gaze-direction in Social Interaction. Acta Psychologica 26, 22–63 (1967) 12. Goodwin, C.: Conversational Organization: Interaction Between Speakers and Hearers. Academic Press, London (1981) 13. Maynard, S.K.: Interactional Functions of a Nonverbal Sign: Head Movement in Japanese Dyadic Casual Conversation. J. Pragmatics 11, 589–606 (1987) 14. Mateo Lozano, O., Otsuka, K.: Real-time Visual Tracker by Stream Processing. Journal of Signal Processing Systems, 285–295 (2008) 15. Mikami, D., Otsuka, K., Yamato, J.: Memory-based Particle Filter for Face Pose Tracking Robust under Complex Dynamics. In: IEEE CVPR 2009, pp. 999–1006 (2009)
A Virtual Audience System for Enhancing Embodied Interaction Based on Conversational Activity Yoshihiro Sejima1, Yutaka Ishii2, and Tomio Watanabe3 1
Graduate School of Science and Engineering, Yamaguchi University, 2-16-1 Tokiwadai, Ube, Yamaguchi, Japan 2 Information Science and Technology Center, Kobe University, 1-1 Rokkodai, Nada, Kobe, Japan 3 Faculty of Computer Science and System Engineering, Okayama Prefectural University 111, Kuboki, Soja, Okayama, Japan
[email protected],
[email protected],
[email protected]
Abstract. In this paper, we propose a model for estimating conversational activity based on the analysis of enhanced embodied interaction, and develop a virtual audience system. The proposed model is applied to a speech-driven embodied entrainment wall picture, which is a part of the virtual audience system, for promoting enhanced embodied interaction. This system generates activated movements based on the estimated value of conversational activity in enhanced interaction and provides a communication environment wherein embodied interaction is promoted by the virtual audience. The effectiveness of the system was demonstrated by means of sensory evaluations and behavioral analysis of 20 pairs of subjects involved in avatar-mediated communication. Keywords: Human Interaction, Nonverbal Communication, Virtual Communication, Enhanced Interaction, Virtual Audience.
1 Introduction Advances in information technology have enabled humans to communicate through computer-generated (CG) characters called avatars in virtual worlds such as Second Life and online games [1]. Many studies on the development of communication systems using avatars [2]-[4] and evaluation of the embodied conversational agent in such communication systems [5],[6] have been performed. However, it is difficult to enhance the embodied interaction in these systems and to create an environment where the talker can share the sense of unity because the characteristics that enhance such embodied interaction has not been introduced thus far. Therefore, the development of an embodied interaction support system that enhances embodied interaction and increases talker involvement is essential. In a previous study, we developed a speech-driven embodied entrainment system called InterWall wherein interactive CG sunflowers act as listeners [7]. This system G. Salvendy, M.J. Smith (Eds.): Human Interface, Part II, HCII 2011, LNCS 6772, pp. 180–189, 2011. © Springer-Verlag Berlin Heidelberg 2011
A Virtual Audience System for Enhancing Embodied Interaction
181
can support human interaction and communication by generating embodied entrained movements such as nodding and body movements based on the speech input of a talker. We confirmed the importance of providing a communication environment in which not only the avatars but also CG objects placed around the avatars are related to virtual communication. In this study, in order to promote enhanced embodied interaction, we analyzed the enhanced interaction t