Complete Proceedings of the NordiCHI 2010 Conference

Proceedings of the Sixth Nordic Conference on Human-Computer Interaction

NordiCHI 2010 Extending boundaries

16-20 October, 2010, Reykjavik, Iceland

NordiCHI 2010 Extending boundaries Proceedings of the 6th Nordic Conference on Human-Computer Interaction, Reykjavik, Iceland, October 16–20, 2010 Editors Ann Blandford, Jan Gulliksen Ebba Thora Hvannberg, Marta Kristin Larusdottir Effie L-C Law, Hannes Hogni Vilhjalmsson

In cooperation with ACM SIGCHI www.nordichi.org

ACM Conference Proceedings Series ACM Press

The Association for Computing Machinery, Inc. 2 Penn Plaza, Suite 701 New York, NY 10121-0701

ACM COPYRIGHT NOTICE. Copyright © 2010 by the Association for Computing Machinery, Inc (ACM). Permission to make digital or hard copies of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantages and that copies bear this notice and thefull citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permission to republish from: Publications Dept. ACM, Inc. Fax +1 (212) 869-0481 or E-mail [email protected]. For other copying of articles that carry a code at the bottom of the first or last page, copying is permitted provided that the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, +1-978-750-8400, +1-978-750-4470 (fax). Notice to Past Authors of ACM-Published Articles ACM intends to create a complete electronic archive of all articles and/or other material previously published by ACM. If you have written a work that was previously published by ACM in any journal or conference proceedings prior to 1978, or any SIG Newsletter at any time, and you do NOT want this work to appear in the ACM Digital Library, please inform permissions@ acm.org, stating the title of the work, the author(s), and where and when published.

ACM ISBN 978-1-60558-934-3

© 2010 by the Association for Computing Machinery, Inc.

Cover design and typesetting by Einar Þ. Samúelsson; contributions camera-ready by authors.

Proceedings: NordiCHI 2010, October 16–20, 2010

Preface This volume contains the proceedings of the 6th NordiCHI conference on Human-Computer Interaction. NordiCHI is the main forum for human-computer interaction research in the Nordic countries. The first five NordiCHI conferences were held in Stockholm (2000), Aarhus (2002), Tampere (2004), Oslo (2006) and Lund (2008). This year’s conference, held on October 16-20 in Reykjavik, is hosted by Reykjavik University and the University of Iceland. The theme of the conference is Extending Boundaries. Understanding, respecting, crossing and expanding boundaries is a part of our daily lives. Boundaries underpin not only geographical and cultural differences between countries, but also differences between sectors and scientific disciplines. Understanding boundaries is an integral part of human computer interaction. It is a part of understanding how people behave and what they value, of designing technologies, of learning how people express their needs, intentions and feelings, of designing HCI technologies and of evaluating them. Selecting Extending boundaries as the theme of NordiCHI 2010, organizers want to encourage contributors and participants to consider HCI’s boundaries, staying within them, or to think beyond them to address grand challenges. The call for participation created an excellent response and we are happy to offer a high-quality programme. The programme combines five different categories of submissions: full papers, short papers, design cases, industrial experience presentations and interactive demonstrations. The industrial experience presentation category follows the initiative from NordiCHI 2006 and the design case category follows the initiative from NordiCHI 2008. This year, 218 full papers were submitted, of which 60 were accepted for presentation at the conference. A further 188 short papers, 5 design cases and 3 interactive demos were submitted. The acceptance rate for full papers was 28% and 31% for short papers. The submitted papers represent 31 countries, with the accepted ones representing 22. All papers were reviewed by at least three independent reviewers; most had four reviews. All papers were discussed at the Programme Committee meeting; those for which reviewers were unanimous in their recommendations were accepted or rejected in accordance with those recommendations. Where there were divergent views amongst reviewers, the substantive details of the reviews were discussed in order to reach a decision; in a few cases, a further review of a paper was commissioned to ensure that decisions were based on the best information possible. We are grateful to all reviewers, the vast majority of whom delivered high quality reviews in a timely way. We are also grateful to the PC members who gave their time to attend the meeting and contributed constructively to the decision making process. Furthermore we would like to thank the organizing committee for their great work. In particular, we want to thank Margret Dora Ragnarsdottir for all her organizing and public relation work. Last but not least, we thank our cooperating and sponsoring organizations for their support: NordiCHI past chairs, ACM, and our main sponsor Nokia. You are very welcome to NordiCHI 2010 Ann Blandford, Jan Gulliksen, Ebba Thora Hvannberg, Marta Kristin Larusdottir, Effie L-C Law and Hannes Hogni Vilhjalmsson

Proceedings: NordiCHI 2010, October 16–20, 2010

Sponsors

Proceedings: NordiCHI 2010, October 16–20, 2010

Organizing Committee

General chairs: Program chairs:

Ebba Þóra Hvannberg, University of Iceland, Iceland Marta Kristín Lárusdóttir, Reykjavik University, Iceland Ann Blandford, University College London Jan Gulliksen, Royal Institute of Technology, Sweden

Short paper & poster chairs:

Effie Law, University of Leicester, UK/ ETH Zürich, Switzerland Hannes Högni Vilhjálmsson, Reykjavik University, Iceland

Doctoral consortium chairs:

Ann Lantz, Royal Institute of Technology, Sweden Dag Svanæs, Norwegian University of Science and Technology, Norway Kaisa Väänänen-Vainio-Mattila, Tampere University of Technology and Nokia Research Center, Finland

Workshops chairs: Tutorials chair: Public relations chair:

Klaus Marius Hansen, University of Iceland, Iceland Kasper Hornbæk, Copenhagen University, Denmark Janet Read, University of Central Lancashire, UK Margrét Dóra Ragnarsdóttir, Síminn, Iceland

Interactive experience and design cases chairs:

Arnold P.O.S. Vermeeren, Delft University of Technology, The Netherlands Gunnar Grímsson, Web design consultant, Iceland Sigríður Sigurjónsdóttir, Iceland Academy of the Arts, Iceland

Industrial and Government Experience chairs:

Margrét Dóra Ragnarsdóttir, Síminn, Iceland Áslaug María Friðriksdóttir, SJÁ ehf., Iceland Birna Íris Jónsdóttir, Betware Gautam Ghosh, Norwegian Tax Authority, Norway Sigrún Eva Ármannsdóttir, Skýrr, Iceland

Sponsor chairs

Marta Kristín Lárusdóttir, Reykjavik University, Iceland Ebba Þóra Hvannberg, University of Iceland, Iceland

Student volunteer chair

Martha Dís Brandt, Reykjavik University, Iceland

Local arrangements

Inga Sólnes, Gestamóttakan, Your Host in Iceland

Proceedings: NordiCHI 2010, October 16–20, 2010

Programme Committee for Full and Short Papers Silvia Abrahao, Universidad Politecnica de Valencia, Spain

Steinar Kristoffersen, Østfold Regional College, Norway

Liam Bannon, University of Limerick, Ireland

Kari, Kuutti, University of Oulu, Finland

Olav Bertelsen, Aarhus University, Denmark

Ann Lantz, KTH Royal Institute of Technology, Sweden

Staffan Björk, Chalmers University of Technology, Sweden

Effie L-C Law, ETH Zurich / University of Leicester, United Kingdom

Ann Blandford, University College London, United Kingdom

Marta Kristín Lárusdóttir, Reykjavik University, Iceland

Tone Bratteteig, University of Oslo, Norway

Scott MacKenzie, York University, Canada

Susanne Bødker, Aarhus University, Denmark

Lone Malmborg, IT University of Copenhagen, Denmark

Torkil Clemmensen, Copenhagen Business School, Denmark

Andrew Morrison, University of Oslo, Norway

Gilbert Cockton, School of Design, Northumbria University, United Kingdom

Erik G. Nilsson, SINTEF, Norway

Anders Mørch, University of Oslo, Norway

Håkan Eftring, Lund University, Sweden

Margrét Dóra Ragnarsdóttir, Síminn / University of Iceland, Iceland

Morten Fjeld, Chalmers University of Technology, Sweden

Roope Raisamo, University of Tampere, Finland

Gautam Ghosh, Norwegian Tax Authority, Norway

Kari-Jouko Räihä, University of Tampere, Finland

Jan Gulliksen,KTH Royal Institute of Technology, Sweden

Janet Read, University of Central Lancashire, United Kingdom

Agneta Gulz, Lund University / Linköping University, Sweden

Virpi Roto, Nokia Research Center, Finland

Klaus Marius Hansen, University of Iceland, Iceland

Frode Eika Sandnes, Oslo University College, Norway

Michael Herczeg, University of Luebeck, IMIS, Germany

Jan Stage, Aalborg University, Denmark

Jo Herstad, University of Oslo, Norway

Erik Stolterman, Indiana University, USA

Morten Hertzum, Roskilde University, Denmark Kasper Hornbaek, University of Copenhagen, Denmark

Dag Svanæs, Norwegian University of Science and Technology, Norway

Ebba Thora Hvannberg, University of Iceland, Iceland

Loren Terveen, University of Minnesota, USA

Kristina Höök, Mobile Life @ Stockholm University, Sweden

Konrad Tollmar, KTH Royal Institute of Technology, Sweden

Giulio Jacucci, Aalto University, Finland

Kaisa Väänänen-Vainio-Mattila, Tampere University of Technology / Nokia Research Center, Finland

Lars-Erik Janlert, Umeå University, Sweden Victor Kaptelinin, Umeå University, Sweden Jofish Kaye, Nokia, USA

Arnold P.O.S. Vermeeren, Delft University of Technology, The Netherlands Hannes Högni Vilhjálmsson, Reykjavik University, Iceland

Proceedings: NordiCHI 2010, October 16–20, 2010

Additional Reviewers of Full and Short Papers Sisira Adikari, University of Canberra, Australia Abdullah Al Mahmud, Eindhoven University of Technology (TU/e), The Netherlands

Henry BL Duh, National University of Singapore, Singapore Mark Dunlop, University of Strathclyde, United Kingdom

Daniela Andrei, Romania

Helen Edwards, University of Sunderland, United Kingdom

Jenni Anttonen, University of Tampere, Finland

Magy Seif El-Nasr, Simon Fraser University, Canada

Carmelo Ardito, University of Bari, Italy

Asbjorn Folstad, SINTEF, Norway

Henrik Artman, KTH Royal Institute of Technology, Sweden

Tim French, University of Bedfordshire, United Kingdom

Pelin Atasoy, The Netherlands

David Geerts, K.U.Leuven, Belgium

Simon Attfield, Middlesex University, United Kingdom

Jens Gerken, University of Konstanz, Germany

Nikos Avouris, University of Patras, Greece

Marco Gillies, University of London, United Kingdom

Cedric Bach, University of Toulouse, France

Peter Gorny, Uni Oldenburg, Germany

Olle Bälter, KTH Royal Institute of Technology, Sweden

Solza Grceva, FON University, Republic of Macedonia

Regina Bernhaupt, University Salzburg, Austria

Anders Green, KTH Royal Institute of Technology, Sweden

Nigel Bevan, Professional Usability Services, United Kingdom

Erik Grönvall, Aarhus University Denmark

Franca Garzotto, Politecnico di Milano, Italy

Timothy Bickmore, Northeastern University, USA

Nils-Erik Gustafsson, KTH Royal Institute of Technology, Sweden

Mark Bilandzic, Queensland University of Technology, Australia

Simen Hagen, Oslo University College, Norway

Amy Bruckman, Georgia Institute of Technology, USA Anders Bruun, Aalborg University, Denmark Hrönn Brynjarsdóttir, Cornell University, USA Asa Cajander, Uppsala University, Sweden Georgios Christou, European University Cyprus, Cyprus Dan Cosley, Cornell University, USA Teresa C-Pargman, Stockholm University, Sweden Stefan Cronholm, Linköping University, Sweden Alma Leora Culén, University of Oslo, Norway Sergio Roberto da Silva, Universidade Estadual de Maringá, Brazil Alexander De Luca, University of Munich, Germany

Göran Hagert, Malmö University, Sweden Tabin Hasan, University of Trento, Italy Per-Olof Hedvall, Lund University, Sweden Ingi Helgason, Napier University, United Kingdom Bo Helgeson, Blekinge Institute of Technology, Sweden Niels Henze, University of Oldenburg, Germany Tobias Hesselmann, OFFIS, Germany Wilko Heuten, OFFIS, Germany Clint Heyer, ABB AB, Corporate Research, Norway Dirk Heylen, University of Twente, The Netherlands Rune Thaarup Hoeegh, Aalborg University, Danmark Harald Holone, Högskolen i Östfold, Norway Jettie Hoonhout, Philips, The Netherlands

Proceedings: NordiCHI 2010, October 16–20, 2010

Netta Iivari, Oulu University, Finland

Stacy Marsella, University of Southern California, USA

Emilio Insfran, Universidad Politecnica de Valencia, Spain

Stefan Marti, Samsung, USA

Poika Isokoski, University of Tampere, Finland

Maristella Matera, Politecnico di Milano, Italy

Mattias Jacobsson, Swedish Institute of Computer Science, Sweden

Emanuela Mazzone, University of Central Lancashire, United Kingdom

Mikkel R. Jakobsen, Copenhagen University, Denmark

Tom McEwan, Edinburgh Napier University, United Kingdom

Anders Jansson, Uppsala University, Sweden Janne Jul Jensen, Aalborg University, Denmark Anne Kaikkonen, Nokia, Finland Karrie Karahalios, University of Illinois, USA

Martin Mihajlov, Faculty of Economics - Skopje, Republic of Macedonia Tonja Molin-Juustila, Oulu University, Finland

Amela Karahasanovic, SINTEF, Norway

Francisco Montero, University of Castilla-La Mancha, Spain

Kristiina Karvonen, Aalto University, Finland

Camille Moussette, Umeå Institute of Design, Sweden

Jinsook Kim, Jacksonville State University, USA

Christina Mörtberg, University of Oslo, Norway

Clemens Nylandsted Klokmose, Aarhus University, Denmark

Stina Nylander, Swedish Institute of Computer Science, Sweden

Anders Kluge, University of Oslo, Norway

Marianna Obrist, University of Salzburg, Austria

Tomoko Koda, Osaka Institute of Technology, Japan

Lars Oestreicher, Uppsala University, Sweden

Andreas Komninos, Glasgow Caledonian University, United Kingdom

Alex Olwal, KTH, Sweden

Liisa Kuparinen, University of Jyväskylä, Finland

Benoît Otjacques, Centre de Recherche Public - Gabriel Lippmann, Luxemburg

Jarmo Laaksolahti, Swedish Institute of Computer Science, Sweden

Saila Ovaska, University of Tampere, Finland

Rosa Lanzilotti, Universita di Bari, Italy

Jacob Palme, Stockholm University / KTH Royal Institute of Technology, Sweden

Astrid Twenebowa Larssen, Ashesi University College, Ghana

Ignacio Panach, Technical University of Valencia, Spain

Pierre Leclercq, University of Liege, Belgium Henry Lieberman, MIT Media Laboratory, USA

Fabio Paterno, ISTI, Italy Thomas Pederson, IT University of Copenhagen, Denmark

Sara Ljungblad, Stockholm University, Sweden

Samuli Pekkola, Tampere University of Technology, Finland

Andres Lucero, Nokia, Finland

Mark Perry, Brunel University, United Kingdom

Martin Ludvigsen, Aarhus School of Architecture, Denmark

John Sören Pettersson, Karlstad University, Sweden

Sigrun Lurås, DNV, Norway

Greg Phillips, Royal Military College of Canada, Canada

Charlotte Magnusson, Lund University, Sweden

Martin Pielot, OFFIS, Germany

Päivi Majaranta, University of Tampere, Finland

Proceedings: NordiCHI 2010, October 16–20, 2010

Volkmar Pipek, IISI - International Institute for SocioInformatics, Germany

Anna-Liisa Syrjänen, Oulu University, Finland

Costin Pribeanu, ICI Bucharest, Romania

Jakob Tholander, Mobile Life, Stockholm University, Sweden

Taina Rajanti, University of Art and Design, Finland

Manfred Tscheligi, University of Salzburg, Austria

Tim Regan, Microsoft Research, United Kingdom

Brygg Ullmer, Louisiana State University, USA

Hayley Davison Reynolds, MIT, USA

Anna Vallgårda, Swedish School of Textiles, Sweden

Rosamelia Ribeiro, Universidade Federal de Goiás, Brazil

Jan Van den Bergh, University of Hasselt, Belgium

Mikko Rissanen, ABB AB, Corporate Research, Sweden

Konstantina Vassilopoulou, Harokopio University, Greece

Armanda Rodrigues, Universidade Nova de Lisboa, Portugal

Olli Vilkki, Laurea University of Applied Sciences, Finland

Mattias Rost, Swedish Institute of Computer Science, Sweden

Chris Vincent, University College London, United Kingdom Mattias Wallergård, Lund University, Sweden

Fredrik Rutz, Stockholm University, Sweden

Alexandra Weilenmann, Göteborg University, Sweden

Antti Salovaara, Aalto University, Finland

Håvard Wiik, Bouvet, Norway

Carmen Santoro, ISTI, Italy

Marco Winckler, University Paul Sabatier, France

Corina Sas, Lancaster University, United Kingdom

Yan Xu, Georgia Institute of Technology, USA

Martin Schmettow, University of Twente, The Netherlands

Jim Youll, RocketSync, USA

Tomas Sokoler, IT University of Copenhagen, Denmark

John Zimmerman, Carnegie Mellon University, USA

William, Soukoreff, University of Toronto, Canada

Tihomir Zoranovic, University of Novi Sad, Serbia

Mark Springett, Middlesex University, United Kingdom

Valdís Þorsteinsdóttir, Vinnuvernd ehf, Iceland

Chris Stary, Johannes Kepler University Linz, Austria

Rikke Ørngreen, Aarhus University, Denmark

Petra Sundström, Stockholm University, Sweden

Proceedings: NordiCHI 2010, October 16–20, 2010

Reviewers of Design Cases Mathilde Bekker, , Eindhoven University of Technology, The Netherlands

Sigríður Sigurjónsdóttir, Iceland Academy of the Arts, Iceland

Gunnar Grímsson, Skuggaþing, Iceland

Arnold P.O.S. Vermeeren, Delft University of Technology, The Netherlands

Marc Hassenzahl, Folkwang University of the Arts, Germany

Reviewers of Industrial Experinece Reports Sigrún Eva Ármannsdóttir, Skýrr, Iceland

Birna Íris Jónsdóttir, Betware, Iceland

Áslaug María Friðriksdóttir, SJÁ ehf., Iceland

Margrét Dóra Ragnarsdóttir, Síminn, Iceland

Gautam Ghosh, Norwegian Tax Authority, Norway

Table of Contents

Proceedings: NordiCHI 2010, October 16–20, 2010

Keynote Speakers Tone Bratteteig Changing Boundaries of Design and Use...................................................................................................................................... 1

Ari Kristinn Jónsson Developing Interactive Artificial Intelligence Software for NASA Mission Control.................................................................... 1

David Merrill The New Era of Physical/Digital Play........................................................................................................................................... 2

Full Papers Let’s All Get Up and Walk to the North Pole: Design and Evaluation of a Mobile Wellness Application Aino Ahtinen, Pertti Huuskonen and Jonna Häkkilä..................................................................................................................... 3

Location-based Crowdsourcing: Extending Crowdsourcing to the Real World Florian Alt, Alireza Sahami Shirazi, Albrecht Schmidt, Urs Kramer and Zahid Nawaz............................................................. 13

Analysis of Precedent Designs: Competitive Analysis Meets Genre Analysis Mattias Arvola, Jonas Lundberg and Stefan Holmlid.................................................................................................................. 23

Exploring Distance Encodings with a Tactile Display to Convey Turn by Turn Information in Automobiles Amna Asif, Wilko Heuten and Susanne Boll............................................................................................................................... 32

Facilitating Continuity: Exploring the Role of Digital Technology in Physical Rehabilitation Naveen Bagalkot, Elena Nazzi and Tomas Sokoler..................................................................................................................... 42

4Photos: A Collaborative Photo Sharing Experience Martijn ten Bhömer, John Helmes, Kenton O’Hara and Elise van den Hoven........................................................................... 52

Two-Handed Input in a Standard Configuration of Notebook with External Mouse Florian Block and Hans Gellersen............................................................................................................................................... 62

Tenori-on Stage: YouTube as Performance Space Mark Blythe and Paul Cairns....................................................................................................................................................... 72

Training Software Developers in Usability Engineering: A Literature Review Anders Bruun............................................................................................................................................................................... 82

Design and Evaluation of Player Experience of a Location-Based Mobile Game Tara Carrigy, Katsiaryna Naliuka, Natasa Paterson and Mads Haahr......................................................................................... 92

Understanding the Everyday Use of Images on the Web Boon Chew, Jennifer Rode and Abigail Sellen.......................................................................................................................... 102

HCI & Sustainable Food Culture: A Design Framework for Engagement Jaz Hee-jeong Choi and Eli Blevis............................................................................................................................................ 112

Distributed Usability Evaluation: Enabling Large-scale Usability Evaluation with User-controlled Instrumentation Lars Christensen and Erik Frøkjær............................................................................................................................................ 118

A Study of Mobile Mood Awareness and Communication through MobiMood Karen Church, Eve Hoggan and Nuria Oliver........................................................................................................................... 128

Engaging Spect-actors with Multimodal Digital Puppetry Céline Coutrix, Giulio Jacucci, Anna Spagnolli, Lingyi Ma, Matti Helin, Gabriela Richard, Lorenza Parisi, Stefano Roveda and Prayag Narula........................................................................................................................................... 138

Proceedings: NordiCHI 2010, October 16–20, 2010

Table of Contents

Comparing User Interaction with Low and High Fidelity Prototypes of Tabletop Surfaces Jan Derboven, Dries De Roeck, Mathijs Verstraete, David Geerts, Jan Schneider Barnes and Kris Luyten............................ 148

Drop-and-Drag: Easier Drag&Drop on Large Touchscreen Displays Sebastian Döweling and Urs Glaubitt........................................................................................................................................ 158

Extending Boundaries with Meta-Design and Cultures of Participation Gerhard Fischer.......................................................................................................................................................................... 168

Wattsup?: Motivating Reductions in Domestic Energy Consumption Using Social Networks Derek Foster, Shaun Lawson, Mark Blythe and Paul Cairns..................................................................................................... 178

Empirical Investigation of Web Design Attributes Affecting Brand Perception Franca Garzotto, Davide Bolchini, Tao Yang and Fabio Sorce................................................................................................. 188

“Luckily, I don’t need it”: Elderly and the Use of Artifacts for Time Management Leonardo Giusti, Eleonora Mencarini and Massimo Zancanaro............................................................................................... 198

Studying Mobile Context-aware Social Services in the Wild Paul Holleis, Sebastian Boehm, Johan Koolwaaij and Matthias Wagner.................................................................................. 207

Negotiating Privacy Boundaries in Social Applications for Accessibility Mapping Harald Holone and Jo Herstad................................................................................................................................................... 217

Transferring Qualities from Horseback Riding to Design Kristina Höök............................................................................................................................................................................. 226

Controlling the Use of Collaboration Tools in Open Source Software Development Heli Ikonen, Netta Iivari and Henrik Hedberg........................................................................................................................... 236

Piles, Tabs and Overlaps in Navigation among Documents Mikkel Rønne Jakobsen and Kasper Hornbæk.......................................................................................................................... 246

Determining Usability Requirements into a Call-for-tenders. A Case Study on the Development of a Healthcare System Timo Jokela................................................................................................................................................................................ 256

Descriptive Quality of Experience for Mobile 3D Video Satu Jumisko-Pyykkö, Dominik Strohmeier, Timo Utriainen and Kristina Kunze................................................................... 266

Unobtrusively Controlling and Linking Information and Services in Smart Environments Bastian Kriesten, René Tünnermann, Christian Mertes and Thomas Hermann........................................................................ 276

Evaluating Multimodal Systems - A Comparison of Established Questionnaires and Interaction Parameters Christine Kühnel, Tilo Westermann, Benjamin Weiss and Sebastian Möller............................................................................ 286

Rise of the Expert Amateur: DIY Projects, Communities, and Cultures Stacey Kuznetsov and Eric Paulos............................................................................................................................................. 295

Do People Say What They Think: Social Conformity Behavior in Varying Degrees of Online Social Presence Lieve Laporte, Christof van Nimwegen and Alex J. Uyttendaele............................................................................................. 305

Using the Hybrid Simulation for Early User Evaluations of Pervasive Interactions Karin Leichtenstern, Elisabeth André and Matthias Rehm........................................................................................................ 315

XTag: Designing an Experience Capturing and Sharing Tool for Persons with Aphasia Abdullah Al Mahmud, Rikkert Gerits and Jean-Bernard Martens............................................................................................ 325

Pipet: A Design Concept Supporting Photo Sharing Bernt Meerbeek, Peter Bingley, Wil Rijnen and Elise van den Hoven...................................................................................... 335

Table of Contents

Proceedings: NordiCHI 2010, October 16–20, 2010

User Experience (UX) Patterns for Audio-Visual Networked Applications: Inspirations for Design Marianna Obrist, Daniela Wurhofer, Elke Beck, Amela Karahasanovic and Manfred Tscheligi.............................................. 343

Collective Interaction by Design - Collective Controllers for Social Navigation on Digital Photos Marianne Graves Petersen, Peter Gall Krogh, Morten Boye Mortensen, Thomas Møller Lassen and Ditte Hvas Mortensen..... 353

Privacy-Awareness Information for Web Forums: Results from an Empirical Study Stefanie Pötzsch, Peter Wolkerstorfer and Cornelia Graf.......................................................................................................... 363

Tactile Camera vs. Tangible Camera: Taking Advantage of Small Physical Artefacts to Navigate into Large Data Collection Mathieu Raynal, Guillaume Gauffre, Cédric Bach, Bénédicte Schmitt and Emmanuel Dubois............................................... 373

Five Design Challenges for Human Computation Stuart Reeves and Scott Sherwood............................................................................................................................................ 383

Designing for Crowds Stuart Reeves, Scott Sherwood and Barry Brown..................................................................................................................... 393

Mobile Interaction with Real-Time Geospatial Data by Pointing Through Transparent Earth Erika Reponen and Jaakko Keränen.......................................................................................................................................... 403

Lightweight Personal Sensemaking Tools for the Web Brendan Ryder and Terry Anderson........................................................................................................................................... 413

The Effect of Aesthetically Pleasing Composition on Visual Search Performance Carolyn Salimun, Helen C Purchase, David R Simmons and Stephen Brewster...................................................................... 422

HandsDown: Hand-contour-based User Identification for Interactive Surfaces Dominik Schmidt, Ming Ki Chong and Hans Gellersen........................................................................................................... 432

ExoBuilding - Breathing Life into Architecture Holger Schnädelbach, Kevin Glover and Ainojie Alexander Irune........................................................................................... 442

Sustainable Energy Practices at Work: Understanding the Role of Workers in Energy Conservation Tobias Schwartz, Matthias Betz, Leonardo Ramirez and Gunnar Stevens................................................................................ 452

Body-Centric Interaction Techniques for Very Large Wall Displays Garth Shoemaker, Takayuki Tsukitani, Yoshifumi Kitamura and Kellogg S. Booth................................................................. 463

From Product Concept to User Experience: Exploring UX Potentials at Early Product Stages Sandra Sproll, Matthias Peissner and Christina Sturm.............................................................................................................. 473

Electronic Resource Discovery Systems: From User Behaviour to Design Hanna Stelmaszewska, B.L. William Wong, Simon Attfield and Raymond Chen.................................................................... 483

Design Qualities for Whole Body Interaction - Learning from Golf, Skateboarding and BodyBugging Jakob Tholander and Carolina Johansson.................................................................................................................................. 493

A Case Study: The Impact of Using Location-Based Services with a Behaviour-Disordered Child Lisa Thomas, Pam Briggs and Linda Little............................................................................................................................... 503

Eyes-free Text Entry with Error Correction on Touchscreen Mobile Devices Hussain Tinwala and Ian Scott MacKenzie............................................................................................................................... 511

User Experience Evaluation Methods: Current State and Development Needs Arnold P.O.S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout and Kaisa Väänänen-Vainio-Mattila.......................................................................................................................................... 521

Family Storytelling for Grandparents and Grandchildren Living Apart René Vutborg, Jesper Kjeldskov, Sonja Pedell and Frank Vetere.............................................................................................. 531

Proceedings: NordiCHI 2010, October 16–20, 2010

Table of Contents

Supporting Cooperative Design through “Living” Artefacts Dhaval Vyas, Anton Nijholt and Gerrit van der Veer................................................................................................................. 541

Crowdsourcing Human-Based Computation Doug Wightman......................................................................................................................................................................... 551

Curve: Revisiting the Digital Desk Raphael Wimmer, Fabian Hennecke, Florian Schulz, Sebastian Boring, Andreas Butz and Heinrich Hußmann..................... 561

How to Stay in the Emotional Rollercoaster: Lessons Learnt from Designing EmRoll Farnaz Zangouei, Mohammad Ali Babazadeh Gashti, Kristina Höök, Tim Tijs, Gert-Jan de Vries and Joyce Westerink....... 571

Keep Talking: An Analysis of Participant Utterances Gathered Using Two Concurrent Think-aloud Methods Tingting Zhao and Sharon McDonald....................................................................................................................................... 581

Short Papers tacTiles - A Low-Cost Modular Tactile Sensing System for Floor Interactions Jan Anlauff, Tobias Grosshauser and Thomas Hermann........................................................................................................... 591

Interaction Design Qualities: Theory and Practice Mattias Arvola............................................................................................................................................................................ 595

Virtual Fashion and Avatar Design: A Survey of Consumers and Designers Jeffrey Bardzell, Tyler Pace and Jennifer Terrell....................................................................................................................... 599

Space-Multiplexed Input on Mouse-Extended Notebooks Florian Block and Hans Gellersen............................................................................................................................................. 603

The Impact of Cognitive Load on the Perception of Time Florian Block and Hans Gellersen............................................................................................................................................. 607

Experiencing the Non-Sensuous: On Measurement, Representation and Conception in Urban Art Installations Morten Breinbjerg, Morten S. Riis, Tobias Ebsen and Rasmus B. Lunding............................................................................. 611

DisQo: A User Needs Analysis Method for Smart Home Joëlle Coutaz, Emeric Fontaine, Nadine Mandran and Alexandre Demeure............................................................................ 615

User Perception of Interruptions in Multimedia Annotation Tasks Chris Creed, Chris Bowers, Robert Hendley and Russell Beale............................................................................................... 619

Seeking a Theoretical Foundation for Design of In Sitro Usability Assessments Yngve Dahl................................................................................................................................................................................ 623

Why do Users Communicate via such or such Media? Some Insights from Users’ Daily Experiences Françoise Détienne, Béatrice Cahour and Liv Lefebvre............................................................................................................ 627

The Impact of Concept (Re)presentation on Users’ Evaluation and Perception Sarah Diefenbach, Marc Hassenzahl, Kai Eckoldt and Matthias Laschke................................................................................ 631

Exploration of Direct Bi-manual Interaction in Digitally Mediated Stop-motion Animation Mariza Dima, John Lee and Mark Wright................................................................................................................................. 635

Choosing Field Methods: A Reflection on a RepGrid Study Helen M. Edwards, Sharon McDonald and S. Michelle Young................................................................................................ 639

Mementos: A Tangible Interface Supporting Travel Augusto Esteves and Ian Oakley............................................................................................................................................... 643

Table of Contents

Proceedings: NordiCHI 2010, October 16–20, 2010

Analysis in Usability Evaluations: an Exploratory Study Asbjørn Følstad, Effie Lai-Chong Law and Kasper Hornbæk................................................................................................... 647

Pocket Bee - A Multi-modal Diary for Field Research Jens Gerken, Stefan Dierdorf, Patric Schmid, Alexandra Sautner and Harald Reiterer............................................................ 651

SmarterPhone – Supporting Meetings in a Mobile Context Humberto Glória, Tiago Guerreiro and Daniel Gonçalves........................................................................................................ 655

Evaluating Game Genres for Tagging Images Dion H. Goh, Rebecca P. Ang, Alton Y.K. Chua and Chei Sian Lee......................................................................................... 659

Designing a Personal Information Visualization Tool Paulo Gomes, Sandra Gama and Daniel Gonçalves.................................................................................................................. 663

The City at Hand: Media Installations as Urban Information Systems Roland Haring, Hideaki Ogawa, Christopher Lindinger, Horst Hörtner, Shervin Afshar and David Stolarsky....................... 667

Take me by the Hand: Haptic Compasses in Mobile Devices through Shape Change and Weight Shift Fabian Hemmert, Susann Hamann, Matthias Löwe, Anne Wohlauf, Josefine Zeipelt and Gesche Joost................................. 671

Experiments in the Wild: Public Evaluation of Off-Screen Visualizations in the Android Market Niels Henze, Benjamin Poppinga and Susanne Boll................................................................................................................. 675

TilePix: An Exploratory Interface to Astronomical Image Databases Avon Huxor and Steve Phillipps................................................................................................................................................ 679

Motor Efficiency of Text Entry in a Combination of a Soft Keyboard and Unistrokes Poika Isokoski, Benoît Martin, Paul Gandouly and Thomas Stephanov................................................................................... 683

Prototyping iPhone Apps: Realistic Experiences on the Device Anders P. Jørgensen, Matthijs Collard and Christian Koch....................................................................................................... 687

Onto-Frogger: The Making Of Amalia Kallergi and Fons J. Verbeek......................................................................................................................................... 691

Politics at the Interface - A Foucauldian Power Analysis Gopinaath Kannabiran and Marianne Petersen.......................................................................................................................... 695

Thumbs-Up Scale and Frequency of Use Scale for Use in Self Reporting of Children’s Computer Experience Akiyo Kano, Matthew Horton and Janet C Read...................................................................................................................... 699

EcoIsland: A Persuasive Application to Motivate Sustainable Behavior in Collectivist Cultures Hiroaki Kimura and Tatsuo Nakajima....................................................................................................................................... 703

Comprehending Parametric CAD Models: An Evaluation of Two Graphical User Interfaces Siniša Kolarić, Halil Erhan, Robert Woodbury and Bernhard E. Riecke.................................................................................. 707

Are Human-Computer Interaction Design Patterns Really Used? Christian Kruschitz and Martin Hitz.......................................................................................................................................... 711

Where are the Ionians of User Experience Research? Kari Kuutti................................................................................................................................................................................. 715

To what Extent Usability Truly Matters? A Study on Usability Requirements in Call-for-tenders of Software Systems Issued by Public Authorities Taina Lehtonen, Juha Kumpulainen, Timo Jokela and Tapani Liukkonen................................................................................ 719

Blue-Sky and Down-to-Earth: How Analogous Practices can Support the User-centred Design Process Sara Ljungblad and Clint Heyer................................................................................................................................................ 723

Proceedings: NordiCHI 2010, October 16–20, 2010

Table of Contents

GappaGoshti™ – A Social Networking Platform for Information Dissemination in the Rural World Sylvan Lobo, Pankaj Doke and Sanjay Kimbahune.................................................................................................................. 727

PINwI - Pedestrian Indoor Navigation without Infrastructure Markus Löchtefeld, Sven Gehring, Johannes Schöning and Antonio Krüger........................................................................... 731

Pointing for Non-visual Orientation and Navigation Charlotte Magnusson, Miguel Molina, Kirsten Rassmus-Gröhn and Delphine Szymczak....................................................... 735

Mobile or Desktop Websites? Website Usage on Multitouch Devices Max-Emanuel Maurer, Doris Hausen, Alexander De Luca and Heinrich Hussmann............................................................... 739

Venice Unfolding: A Tangible User Interface for Exploring Faceted Data in a Geographical Context Till Nagel, Frank Heidmann, Massimiliano Condotta and Erik Duval..................................................................................... 743

DESIGNi – A Workbench for Supporting Interaction Design Claudia Nass, Kerstin Klöckner, Sarah Diefenbach and Marc Hassenzahl............................................................................... 747

What makes a Butlerbot a Butlerbot? A Discussion of a Robot Classification System Mie Nørgaard............................................................................................................................................................................. 751

Towards a Model for Egocentric Interaction with Physical and Virtual Objects Thomas Pederson, Lars-Erik Janlert and Dipak Surie............................................................................................................... 755

Visualizing the Text of Philip Pullman’s Trilogy “His Dark Materials” Tim Regan and Linda Becker.................................................................................................................................................... 759

Teaching UbiComp with Sense Mike Richards and Neil Smith................................................................................................................................................... 765

Introducing Co-design for Digital Technologies in Rural Areas Fausto Sainz, Nacho Madrid and Jaisiel Madrid....................................................................................................................... 769

PyGmI – Creation and Evaluation of a Portable Gestural Interface Matthias Schwaller, Denis Lalanne and Omar Abou Khaled.................................................................................................... 773

Capital Music – Personal Expression with a Public Display of Song Choice Jan Seeburger, Marcus Foth and Dian Tjondronegoro.............................................................................................................. 777

Research Methods for Involving Hearing Impaired Children in IT Innovation Karin Slegers, Pieter Duysburgh and An Jacobs....................................................................................................................... 781

Evaluation of Adaptive SpringLens -- a Multi-focus Interface for Exploring Multimedia Collections Sebastian Stober, Christian Hentschel and Andreas Nuernberger............................................................................................. 785

Measuring Effects of Private and Shared Displays in Small-Group Knowledge Sharing Processes Sara Streng, Karsten Stegmann, Sebastian Boring, Sonja Böhm, Frank Fischer and Heinrich Hussmann.............................. 789

Mobile Device Interaction Gestures for Older Users Christian Stößel and Lucienne Blessing.................................................................................................................................... 793

Messenger Visual: A Pictogram-based IM Service to Improve Communications Among Disabled People Pere Tuset, Pere Barberán, Léonard Janer, Esteve Buscà, Sandra Delgado and Nuria Vilà...................................................... 797

From Magical Experience to Effortlessness: An Exploration of the Components of Intuitive Interaction Daniel Ullrich and Sarah Diefenbach........................................................................................................................................ 801

Role Playing with Fire Fighters – Using a Worst Case Scenario and Verbal Re-enactment in the Role Play Paula Valkonen and Marja Liinasuo.......................................................................................................................................... 805

Gesture Based Interaction for Visually-Impaired People Sylvie Vidal and Grégoire Lefebvre.......................................................................................................................................... 809

Table of Contents

Proceedings: NordiCHI 2010, October 16–20, 2010

Bridging Gaps with Pointer Warping in Multi-Display Environments Manuela Waldner, Ernst Kruijff and Dieter Schmalstieg.......................................................................................................... 813

Detecting the “Point of Originality” in Student Writing Brandon White and Johann Ari Larusson.................................................................................................................................. 817

Architectures of Interaction: An Architectural Perspective on Digital Experience Heather Wiltse and Erik Stolterman.......................................................................................................................................... 821

Measuring the Dynamics of User Experience in Short Interaction Sequences Benjamin Wimmer, Bernhard Wöckl, Michael Leitner and Manfred Tscheligi........................................................................ 825

Design and Interactive Demonstrations UCD Method Selection with Usability Planner Xavier Ferre, Nigel Bevan and Tomás Antón Escobar.............................................................................................................. 829

HawkEye: a novel process automation interface Kristoffer Husøy and Torgeir Enkerud...................................................................................................................................... 831

Temporal Relations in Affective Health Elsa Kosmack-Vaara, Iuliana Claudia Silvăşan, Anna Ståhl and Kristina Höök....................................................................... 833

linked. – A Relatedness Experience for Boys Matthias Laschke, Marc Hassenzahl and Kurt Mehnert............................................................................................................ 839

Exploratory Study of a Touch-based Gestural Interface for Elderly Chiara Leonardi, Adriano Albertini, Fabio Pianesi and Massimo Zancanaro........................................................................... 845

Bringing Playfulness to Disabilities Patrizia Marti............................................................................................................................................................................. 851

Untangling the Mess - A Redesign of a Technical Environment for a Telecommunication Company Call Centre. Part II: Capturing a Valid Order for a Telecommunication Service Margret Dora Ragnarsdottir....................................................................................................................................................... 857

Panel Grand Challenges for Future HCI Research: Cultures of Participation, Interfaces Supporting Learning, and Expansive Learning Yrjö Engeström, Annalisa Sannino, Gerhard Fischer and Anders I. Mörch.............................................................................. 863

Author index ................................................................................................................................................................................................... 867

Keynote Speakers

Proceedings: NordiCHI 2010, October 16–20, 2010

Tone Bratteteig

Associate Professor, Department of Informatics, University of Oslo

Changing Boundaries of Design and Use Biography Tone Bratteteig holds a Ph.D. from The University of Oslo, addressing design and use of IT as an interdisciplinary research field. She is associate professor at Department of Informatics, U of Oslo, where she leads a research group on Design of Information Systems. She is an experienced teacher, and her CV includes co-teaching a course at Stanford’s d.school. Tone was part of the first Participatory Design research project in Norway, and her current research addresses how to reframe PD for contemporary society.

Abstract Human-Computer Interaction concerns both humans and computers and involves a number of different disciplines and interdisciplinary areas. In this talk I discuss how recent technological and socio-cultural developments challenge the established categories of design and use of IT. I suggest seeing use as work, included in the work knowledge and constituting the conditions for the work. Drawing on traditional design disciplines design of IT can be conceptualized as a combination of ideas and materials, where the particular qualities of IT as a design material influence both the design process as well as the design result. In the talk I discuss these conceptualizations and how they can help us discuss how the changing relations between design and use of IT. I draw on both old and new research projects to illustrate how the concepts can be applied.

Ari Kristinn Jónsson

Rector of Reykjavik Unversity

Developing Interactive Artificial Intelligence Software for NASA Mission Control Biography Ari Kristinn Jonsson is the rector of Reykjavik University. He holds a Ph.D. and MS in Computer Science from Stanford University. Before joining the University of Reykjavik he was a senior research scientist at NASA Ames Research Center working on the Mars Exploration Rovers. For details, see his curriculum vitae (http://www.hr.is/starfsfolk/ari).

1

Abstract Over the last decade, artificial intelligence technology has moved from being an obscure research project within NASA to being an important tool for NASA mission controllers who operate spacecraft such as the Mars Exploration Rovers and the International Space Station. This achievement is in part due to advances in artificial intelligence, but a critical part is due to the development of a good understanding of mission controllers needs and how they interact with computer software. This talk presents the development of these interactive software tools, with focus on user involvement and how lessons learned were applied to improve the technology.

Keynote Speakers

Proceedings: NordiCHI 2010, October 16–20, 2010

David Merrill Co-Founder of Sifteo

The New Era of Physical/Digital Play Biography David Merrill is co-founder and president of Sifteo, (http:// sifteo.com/) a company based in San Francisco building the future of play. Sifteo’s first product is Siftables: a tabletop user interface for learning and play made of active physical +graphical tiles. David is a graduate of the Fluid Interfaces Group at the MIT Media Lab, where he studied with professor Pattie Maes and developed the first prototype of Siftables. His work explores how human interactions with computers can leave the limitations of the desktop interface behind, through the development of physical-digital tools that operate comfortably in our real-world environment to enable new forms of play, expressivity, problem-solving and collaboration. He has lectured in computer science at Stanford University and led music controller design workshops at the MIT Media Lab. David holds a Ph.D. and MS from the MIT Media Lab, and an MS in Computer Science and BS in Symbolic Systems from Stanford University. He was a Mayfield fellow in 2001, and was awarded graduate fellowships from Motorola and Samsung at MIT.

Abstract The landscape of human-computer-interface possibilities has changed dramatically in recent years. Rising popularity of mobile phones has driven down the cost of microprocessors and other electronic components, and driven their quality up. Sensors have become tiny and embeddable, and graphical displays can be found on more and more of our personal devices. We are now entering the era of ubiquitous computing that Mark Weiser famously wrote about, where these technologies are disappearing into the fabric of our everyday lives. How will these new capabilities affect our experience of being human? One domain already experiencing great change is play. In this talk I will discuss some changes to the ways we play that are arriving as a result of technological advancement. I will explore implications of these changes for human-computer and human-human interaction, and how technology-based play in the future may look more familiar to us than we may think.

2

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Let’s All Get Up and Walk to the North Pole: Design and Evaluation of a Mobile Wellness Application Aino Ahtinen, Pertti Huuskonen Nokia Research Center Visiokatu 1, 33720 Tampere, Finland {firstname.lastname}@nokia.com

Jonna Häkkilä Nokia Research Center Yrttipellontie1, 90230 Oulu, Finland [email protected]

ABSTRACT

people to exercise more (e.g. heart rate monitors, pedometers, fitness games), eat healthy (e.g. mobile journaling tools for diet tracking, web-portals for peer support) and relax (e.g. mobile relaxation applications, wrist-watch-type stress measurement devices). Mobile phones provide a promising wellness application platform [5],[13] because of their ubiquitous presence, communication channels, sensor data recording and computing capabilities, combined with their perceived role as personal and trusted technology [28].

Mobile wellness applications can motivate people to exercise more because of their ubiquitous presence and suitable technological possibilities. Wellness applications utilize several factors that can increase physical activity levels of users, e.g., data tracking, social sharing and playfulness. We describe the design and evaluation of a mobile-phone based wellness application addressing social sharing and playfulness. In our user study with 37 participants we focus on presentation of physical activity data as a virtual trip on a map-based game world. The findings reveal design implications that support users to be motivated in doing the physical exercise, including setting departure and destination places and viewing up-to-date progress between them. The virtual trip was considered an understandable, concrete and interesting analogy. The familiarity of the map of the home country as the game world was liked because it added concreteness and understandability to the achievements. Author Keywords

Mobile applications, wellness, playfulness, social sharing, metaphor, analogy, user experience, user studies, evaluation.

Figure 1. Into, a wellness application on the S60 platform.

ACM Classification Keywords

H.1.2 User/Machine Systems: Human factors. H.5.2 User Interfaces: Evaluation/methodology, User-centered design.

This paper explores the design space of mobile wellness applications to support motivation towards physical exercise, which is one of the most important factors in treating and preventing health problems. We have developed a mobile social and playful wellness application called Into (Figure 1). The idea behind it traces back to our former user studies around the topic of mobile wellness applications, in which some participants used an analogy of distance travelled on a map to represent the amount of their physical activities. Those participants recorded the number of kilometers they had been walking, running and biking, marked it down on a map as a line and imagined how far from home they would have travelled and in which place they would be at the moment. Similar analogies have also been used by some organizations in exercise campaigns that aim at increasing the physical activity level of people [20],[26]. We set out to develop this idea into a mobile wellness application with the help of a usercentered design process.

INTRODUCTION

Motivating people to maintain their health through preventive actions has become a global concern that calls for urgent actions. A growing number of health problems due to the modern lifestyle is well-known – people around the world suffer increasingly from obesity, type 2 diabetes, and cardiovascular diseases. The impact of these problems extends to both personal and societal levels. Technologies have been harnessed for wellness support to motivate Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

3

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

A design that provides the user enjoyable and playful experiences leads to positive emotional and motivational states [10]. Therefore, in the domain of wellness applications, the so called ‘fun factor’ or a playful approach has often been chosen to engage and persuade people into wellness activities, especially physical activity. Several activating game concepts and prototypes have been presented in the past years. Fish'n'Steps mobile application in Lin et al. [22] visualizes the level of physical activity with a bowl of growing fish, adding the ‘fun-factor’ and playfulness to the design. In Neat-o-Games [15], a wearable accelerometer provides data that is used to control an avatar representing the player in a virtual race game. Multiple players can participate in the game, and winners are declared on a daily basis. The players can use activity points that they have gained to receive hints in mental games such as Sudoku that are included in the application. In the Play, Mate! design Berkovsky et al. [7] aim at raising the players' motivation by increasing the difficulty of the game and introducing physical activity as a part of the game. One of the most well-known wellness games is Nintendo Wii Fit [37], which includes several types of exercise activities (aerobics, yoga, etc.) in the form of a game.

In this paper we describe, along with the most important related work, a user study of our wellness application. To the best of our knowledge, visualizing physical activity results as a virtual trip on a map has not been studied and analyzed in the user experience studies before, although the analogy has been used in some exercise campaigns, e.g., [20],[26] and commercial games [25]. We present the setting and main findings of our user trial with 37 participants. CURRENT PRACTICES AND BEYOND Applications to Support Physical Activity

The features of current technical applications and solutions that are designed to support physical activity can be roughly categorized to, for instance, four categories [1], but in many cases the categories overlap: • Loggers • Personal trainers • Playful applications and games • Social applications Logger is a simple approach for supporting physical exercise. They are able to measure one or several parameters related to the exercise, and store measurement information into logs. Pedometers offer easy-to-use and inexpensive tools for assessing physical activity. The basic models of the heart rate monitors also act as loggers, as they save data about the pulse, but do not offer sophisticated tools for the analysis or interpretation, or give much guidance. Mobile journaling tools can also act as loggers. An example is provided by the Nokia Wellness Diary [36] for manual tracking of exercise activities and other wellness-related parameters. Tracking and storing physical exercise data in a diary or log can motivate people to increase physical activity levels [29],[31].

Social sharing of the exercise-related data is an important motivator to do more physical activity [21],[34]. Many studies show that social applications can be effective in supporting motivation towards physical activity [5],[12],[35]. Exercising with others provides social support, a feeling of team spirit, an opportunity for social learning as well as pressure to begin and continue with the activities. The often recommended design guideline to “support social influence” [12],[17] has been taken into account in some existing wellness concepts and products. For instance, encouraging results have been gained with mobile phone prototype applications that allow the sharing of activity data, such as step count, within a community of friends [5],[12],[35]. In these studies, sharing the data with peers increased the average of weekly activity. In a different concept, auditory cues of a friend “jogging over distance,” in another geographical location, was found to be encouraging and successful in supporting the need for socializing [27]. Bickmore et al. [9] reported that even sharing information of physical activity with a PC-based animated conversational agent that acted as an exercise advisor increased the motivation to exercise more.

The personal-trainer approach usually includes a logger feature and, in addition, an analysis functionality. Moreover, they can provide feedback on performance. In [11], Buttussi et al. focus on the ability of GPS-enabled mobile devices to offer location-aware feedback. Their solution employs GPS for location information, and the PDA is used for viewing maps and an animated, humanfigured 3D fitness trainer. The trainer offers real-time feedback about how the user is proceeding during the exercise in graphical and audible format, and gives directions for the navigation and assistance of performing fitness tasks. Another example is presented by Asselin et al. [6]. They have developed a wearable Personal Wellness Coach to support a variety of activities such as health data collection, interpretation, feedback and self-monitoring. The Mobile Coach [23] is a mobile application that assists users by giving training programs based on their current fitness and activity level. It analyzes how effective the workouts are and tunes the training program for the user when the schedules change.

Into can be regarded as a playful and social wellness application, as it uses some design elements from both categories (see section “The Into Application”). Trying to reach destinations on the game world can be seen as a playful element, as well as the animal icons representing the team’s speed and the postcard that is received as a reward when reaching the destination. The social elements appear in the team formation, setting a team challenge, progressing towards the challenge with the team and sharing of step count within the team.

4

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Adventure is launched yearly by the Nuori Suomi organization and it takes place in Finnish primary schools. The school class collect daily exercise points to their personal exercise cards. The points are transformed to combined kilometers of the class, and the imaginary trip progresses from country to country. The progress is viewed on a web portal and a poster placed in the classroom. The campaign has proved to be a liked and effective – for example, in 2009 almost 10 000 school classes joined the campaign and the average time of exercise during the campaign was about 2,5 hours per pupil per day. The same idea is also used by Walking Campaign (Kävelykampanja) arranged by Finnish Kuntoliikuntaliitto [20] for working age citizens. The participants carry a pedometer, and the recorded exercises are manually transferred to the webportal, where the progress can be viewed as a trip on a map.

Our research seeks to articulate the earlier findings and guidelines into practical design decisions, and contributes to the state-of-the-art in further developing and validating the design rationale through research questions we state in section “Research Setting”. Usage of Analogies

The use of analogies and metaphors is common and reasonable in interaction design. Analogies can help in understanding the abstract and hard to imagine computerbased concepts or tasks, because “they provide a structure that is similar in some way to aspects of a familiar entity” [30]. The ‘match between system and real world’ principle has been one of the usability heuristics for a long time by now – the system should speak the user’s language, with words, phrases and concepts familiar to the user [24]. One of the most familiar metaphors is the desktop metaphor. The usage of interface metaphors has proven to be highly successful in helping users to use technological systems [30]. However, sometimes the metaphors cause confusion due to, for example, users’ cultural backgrounds [14].

A similar analogy is also in use in the Nintendo Personal Trainer Walking. Step info from the activity sensors is used to draw pictures of the earth, and the more the user walks, the more new pictures are drawn on the terrain. In another game from the same company, the users can post their step counts to combine them with other users’ data, to plot a virtual walk across the solar system. [25]

In the domain of wellness applications, the data (e.g. amount of exercise or burned calories) has traditionally been presented with numbers and graphs. However, new analogies of visualizing the wellness data have been proposed. Instead of numbers and graphs, Ståhl et al. [33] used animated human-like figures to visualize the wellness data. In their Affective Diary concept, the data was shown in a way that reminded users of their own bodies. Consolvo et al. [13] visualized the progress with a growing garden including butterflies and flowers and they reported that their system was effective in motivating to physical activity. Similarly, Albaina et al. [3] used an emotional flower in their photo frame concept to visualize walking activities. In addition, Lin et al. [22] presented the level of physical activity with a bowl of growing fish.

The analogy of presenting physical activity data as a virtual trip on a map differs from the approach where the physical activity data is recorded with GPS-based technology. The analogy should not be confused with GPS-based applications, like Nokia Sports Tracker [32] or Garmin Forerunner [16]. In the analogy, the map and trips are used as a metaphor and visualization, and they do not represent real routes conducted by the users in any other sense than the distance progressed. Even though the analogy of a virtual trip has already been successfully used in the exercise campaigns and is visible in some games, we are not aware of any mobile phone applications that utilize it, or user trials that have studied and analyzed the user experiences and perceptions of the approach. Our platform of choice, the mobile phone, offers capabilities that are superior to stand-alone diaries, game consoles or GPS devices—the phone is a device that is omnipresent but personal, yet gives natural access to social networks and networked services.

Compared to these delightful but abstract solutions, we explicitly seek presentations that are more concrete and measurable. We chose maps and distances travelled on them as the key presentational principles, since we hypothesised that these concepts would be very concrete for humans and provide a meaningful mapping from metaphors to physical activity. Visualizing Physical Activities as a Trip

RESEARCH SETTING Pre-study: Testing the Concept Design

Showing step-based physical activity as a distance is a familiar feature on pedometers. Even the simplest pedometers present the activity data both in steps and kilometers. A phone-based pedometer provides possibilities to design applications to support physical activity. For example, Apple provides several applications to track the activity with the accelerometer-based pedometer and view the distance and burned calories [18].

In our user-centered design process we first evaluated the Into concept with users in a simulated setup. A total of 12 people in 4 teams participated in a pre-study, where the idea of the game was evaluated with a step counter and paper prototype. We used pedometers on mobile phones to track the exercise, real world team formation, a paper map in which the progress was plotted (see Figure 2), and printed postcards as rewards when the team reached their target destinations (see Figure 3).

The analogy of presenting visual activity data as a trip has been used in a Finnish fitness campaign called “Exercise Adventure” (Liikuntaseikkailu) [26]. The Exercise

5

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2. Initial concept evaluation. A team gathered around a paper map where departure and destination points and the progress of teams were plotted.

Figure 3. Initial concept evaluation. A printed postcard from the destination was delivered to the team members when they completed a challenge.

The results of the prestudy helped to refine the concept and highlighted the essential design aspects that should be taken into account when transferring the design to a mobile application. The salient findings of the pre-study revealed that the game idea was found compelling. The participants reported example situations where they had deliberately increased their amount of exercise to get more steps (e.g. walked to work instead of taking a bus) and thus better results for the team. The familiar environment the map was representing, the support of team play, and the need for constant feedback surfaced as essential design factors. Also, implementing the game board as a mobile phone application was perceived as a good idea – the paper prototype could of course not provide automated status updates, and the pedometer was already integrated with a phone anyway.

mobile phone. The users “proceed” (travel) on a map based on their step count. First, the user creates a challenge, such as “Proceed from Helsinki to Tampere.” The application then displays the length of the challenge in steps and kilometers. The user can either play alone, or more interestingly, invite members to the team. The team members’ step counts are synchronized to a server and sent back to the mobile phones of all team members. The team’s combined progress is currently shown as a trip on a map and as numerical information (see Figures 4A & 6B). Based on the speed that the team progresses, the application shows an animal figure, e.g., a snail, a galloping horse or a flying eagle (see Figure 4B). In addition, the progress of each team member (steps, kilometers) is displayed separately as a list, which can be viewed by all team members (see Figure 5A). When the target destination is reached, the team receives a reward – an electronic postcard from the target city. The collection of received postcards can be viewed in the application (see Figure 5B). The postcards can be opened as full screen images.

The second step of the process was the iterative concept and UI design with several usability tests and iteration rounds both in the laboratory and field settings. The participants of the iterative user evaluations were content with the concept of visualizing physical activity results as a virtual trip on a map. The findings from the early evaluations led us to formulate the following research questions: (Q1) How do the users perceive the visualization of physical activity results as a virtual trip on a map? (Q2) What do the users think about the physical activity challenges that are presented as trips between places, for example cities or villages? (Q3) How do the users react to a familiar map, i.e. the map of their home country as the game world? The Into Application

Into is a mobile java application currently available for the Symbian S60 platform. It combines social team play with other playful elements to increase motivation towards physical activity. Into records the user’s steps with an accelerometer-based pedometer, which is in-built in the

Figure 4. Into application. (A) The team’s combined progress in numbers and as a line on the map, (B) An animal figure showing the speed of the team’s progress.

6

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

appealing the participants found it; and whether they would recommend it to an interested friend. The responses were given with a 5-point Likert scale. The web link to the online questionnaire was sent to the participants’ email addresses after one week of use of Into, and the response rate was 100%. Responding to the questionnaire took about 5-10 minutes.

Methodology

We carried out a trial with 37 participants to examine the participants’ subjective responses towards Into, focusing especially to their perceptions of visualizing physical activity results as a virtual trip. The trial lasted one week, and it was conducted in Finland in late 2009. The recruited participants were existing social groups, i.e. groups of friends, colleagues, neighbors and families. In total, nine Into teams consisting of 2 to 6 members were formed. Most teams had 4 members. Kick-off sessions were arranged with each team separately, and all members of the team were present in the session. The functionality and features of Into were demonstrated to the participants, as well as the phases and tasks related to the study period. The participants were handed out Nokia 6210 mobile phones, with Into installed, as well as pre-paid SIM cards. One of the team members was named as the team leader and a team challenge was created for the team, to be visible in all members’ mobile phones. The participants were advised to keep the mobile phone with them all the time and have the application running on the background. The users were educated for the technical idiosyncrasies of the application used in the study. The team leader was advised to create a new challenge whenever they completed the previous challenge. For testing purposes, the application included an additional feature, a step multiplier function. The multiplier was set to 5 for this evaluation due to the relatively short duration of the period and long travel distances. This meant that the participants step count achievements were multiplied by 5, and to get the real step count they needed to divide it by 5.

The open-ended questions of the interviews dealt with, e.g., the following themes: how did the participants find the idea of representing physical activity results as a virtual trip on a map; how did they like the challenges that were formulated as trips between cities on a map; and what were their reactions towards the map of their home country as the game world. The single-person 15-20 minute recorded interviews were carried out by phone after the trial period. Most of the participants responded first to the online questionnaire; all participated to the interview. The interview data was analyzed with the qualitative thematic classification method, which is similar to the Affinity wall technique [8]. First the recordings were transcribed word by word. Secondly, the raw data was classified under the main research themes, e.g., physical activity as a virtual trip; challenges presented as trips between places; and familiar game world. The ideas for the further design were the fourth theme to be discussed in this paper. There were also other main themes formed, e.g., social competition, social cooperation, postcards as rewards and icons reflecting the team’s speed, but they are out of the scope of this article. Inside each main theme, the raw data was further classified to form subcategories. The classification was done on computer by using spreadsheets, and not on the wall as is done in the Affinity wall technique. The “Findings” section describes the main findings from the four main themes on focus.

Online questionnaires and semi-structured interviews were used as data collection methods after one week of use. In the online questionnaire, we asked questions about, e.g., the motivating role of Into; how suitable its features are; how

Participant Profile

The 37 participants (31 females, 6 males) from 20 to 55 years (80% were 25-39), were from different backgrounds, e.g., medical doctor, entrepreneur, salesperson, nurse and researcher. The majority were sedentary workers. Most of them had no technical background, but they were experienced mobile phone users: 75% of them had used mobile phones for over 10 years. Five of the participants had no previous experience in using wellness-related technologies, while the rest had tried or used actively gadgets or applications such as heart rate monitors, pedometers, or Wii Fit. The participants had varying backgrounds in their physical activity levels. The four most popular activities among the participants were walking, jogging, gym and ball games. Most participants (70%) reported that they did physical exercises several times a week, 16% exercised once a week and 13% more rarely.

Figure 5. Into application. (A) The list view of the team members and their current achievements. (B) The list view of received postcards from the reached destinations.

7

Full Papers

Some negative effect

No effect

Some positive effect

Very positive effect

50,0

18,4

Competition between the team members

0,0

2,6

15,8

52,6

23,7

13,5

Challenges (A to B)

0,0

0,0

31,6

60,5

7,9

13,5

Viewing progress as a trip on the map

0,0

0,0

18,4

60,5

15,8

Map of home country as a game world

0,0

2,6

42,1

42,1

7,9

Step multiplier

10,5

23,7

31,6

31,6

2,6

Rewards (postcards)

0,0

2,6

60,5

28,9

0,0

Icons reflecting the speed (animals)

0,0

5,3

47,4

31,6

7,9

Totally agree

21,1

Somewha t agree

0,0

Neither disagree or agree

0,0

Somewha t disagree

Cooperation between the team members

Totally disagree

Very negative effect

Proceedings: NordiCHI 2010, October 16–20, 2010

Encourages me to do more physical activity

0,0

13,5

2,7

67,5

16,2

Includes features & functionalities suitable for me

0,0

Is an appealing application

0,0

5,4

5,4

75,6

I would recommend Into to a friend

0,0

13,5

2,7

70,2

15,8

7,9

71,1

5,3

Table 1. Responses to the question: “Rate the following statements according to your opinion”. The distribution of the responses in percentages, n=37. “Don’t know” responses not included in the table. FINDINGS General Findings

In the online questionnaire we asked questions about the motivating role of Into; how suitable its features were; how appealing the participants found it; and whether they would recommend it to an interested friend. The questions and the distribution of the responses in percentages can be found in Table 1. In all questions, most of the responses range from “somewhat agree” to “totally agree”. This clear trend indicates that the participants perceived Into as a motivating and appealing application, which has a suitable set of features, and they would also like to recommend it to others.

Table 2. Responses to the question: “What aspects affected your usage motivation of Into?” The distribution of the responses in percentages, n=37. “Don’t know” responses not included in the table.

the usage motivation of the application. About 76% of the participants rated that viewing the progress as a trip on the map had some or very positive effect on the usage motivation of the application. Almost 70% of the participants rated that challenges had some or very positive effect. The map of the home country as the game world was rated as having some or very positive effect by half of the participants.

We also asked the participants to rate how the different aspects of Into affected the usage motivation of the application. The distribution of the responses in percentages is shown in Table 2.

The step multiplier (5) used in the study was the only aspect that seemed to affect the usage motivation of the application somewhat negatively: almost 35% rated it as having very or some negative effect on the usage motivation. The participants commented on the interviews that they would rather have used the application without any multiplier because they wanted to see easily the realistic progress, i.e. real steps taken and kilometers travelled. The rewards and icons reflecting the speed of the team had some positive effect on the usage motivation: about 30% and 40% of the respondents considered them as having some or very positive effect.

The responses reveal that social aspects (cooperation and competition between the team members) were rated as having a positive effect on the usage motivation of the application: more than half of the participants had the opinion that the social aspects affected positively to the usage motivation of Into (see Table 2, marked as dark grey). We are satisfied with the overall finding from the interviews that Into enabled both competition and cooperation at the same time, and they were perceived as motivating factors by the participants. The users could make their selection between mentally competing against the other team members by comparing the results in the Team view (see Figure 5A), or just cooperating with them by contributing as well as possible to reach the team challenge together.

Next, we will cover in detail (1) how the analogy of presenting physical activity results as a virtual trip worked in the Into application and (2) the related user perceptions from the user study. The user study findings are based on the qualitative data from the semi-structured interviews with the 37 participants. The findings will be presented with the authentic user comments from the interviews.

Marked as light grey in Table 2, the aspects related to visualizing physical activity results as a virtual trip (challenges reflecting trips between places; viewing the progress between the places on a map; and the map of the home country as the game world) had a positive effect on

Visualizing Physical Activity Results as a Virtual Trip

In Into, the physical activity results, i.e. steps, were visualized for the users with an analogy of a virtual trip. The users travelled was on a map of their home country,

8

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

achievements more concrete than viewing the achievements as kilometers or steps, which was an alternative way to view the achievements in Into. The following examples of participant comments reveal how positively the participants reacted to the concept of visualizing physical activity achievements as a virtual trip, and how they emphasized the concreteness of the analogy: “It is good to have a destination as a target where one needs to get. I believe it motivates more than just seeing how long distance one has walked or run.” (male, team of 6 neighbors) “The concreteness is a good factor.” (female, team of 6 neighbors) “To view the progress on the map makes the progress much more concrete. And it makes it more interesting when compared to seeing only kilometers.” (female, team of 4 friends)

Figure 6. Into application. (A) Setting a departure point. (B) The team’s combined progress visualized as a trip on the map.

Finland. The departure and destination places could be selected from a list including 15 cities or smaller towns. As the team members took steps, the application combined the achievements of each team member and visualized the combined progress as a trip between the departure and destination places. The more they took steps, the quicker the red line between the places turned to green. The line between the departure and destination places reflected the true distance between the places in the physical world, and respectively the users needed to take as many steps together as the real distance required.

Following the progress on a map made viewing the physical activity achievement interesting for the participants. Many users liked to view the map often to see if they were already on the half way or if they soon reached the destination. Between the departure place and the destination they could think which places they had already passed along the way and by which place they were at the moment: “I thought about by which city we were at the moment. For example, when travelling to Helsinki we were by Hämeenlinna at the certain moment.” (female, team of 5 friends) “I think it is great to be able to see where we are going at the moment, that’s the best thing. To see the up-to-date situation and how much there is still left.” (female, team of 6 neighbors)

Concrete, Understandable and Interesting Analogy

The following open-ended interview questions contributed to the findings of this section: • What was your general impression of the usage of Into? • What were the factors or aspects that motivated the use? Why?

In addition to being concrete and understandable, the participants also experienced positive feelings (fun, enjoyment) when looking at the map and the progress on it: ”The longer the green line, the better mood I got.” (female, team of 4 colleagues) “It is fun! I mean the idea that we are walking towards some destination on that map. The map made it more concrete.” (female, team of 4 family members)

• What do you think about presenting the achievements of the team as a trip on a map? • How do you feel about the current challenges? • What kind of challenges would you like to have in the application? Why? Visualizing physical activity achievements as a virtual trip between cities or villages was liked by the participants. They emphasized the concreteness and understandability of the approach, and also commented that it made viewing the achievements interesting and fun. They explained that the target felt more concrete because there was a clear goal of travelling virtually from one place to another and realistic departure and destination places. The participants liked the idea of travelling between real-life cities, towns and villages, as can be seen on the following comments: “Nice idea to be able to see that we are progressing from city to city. It is a realistic alternative.” (female, team of 4 colleagues) “As a mental image a trip between cities is very nice.” (female, team of 6 neighbors)

The Familiar Game World

The following open-ended questions on the interviews contributed to the findings of this section: • What do you think of the current game world of Into? • What kind of a game world would you like to have in the application? Why? • How do you feel about the current challenges? • What kind of challenges would you like to have in the application? Why? The game world where the teams were travelling was a map of Finland, their home country. Thus, the departure and destination places were Finnish cities, towns and villages. Naturally, the map of Finland was quite familiar

For most of the participants, travelling virtually between cities or villages made viewing the physical activity

9

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

“On the way it could tell that now you are in Hämeenlinna, for example. It should include that kind of intermediate things.” (male, team of 3 family members) “Sometimes it would be nice to get information like now you are proceeding by the Toijala church or Ideapark shopping complex, or some similar small feedback to the users.” (female, team of 4 colleaques)

for the participants as they were all native Finnish citizens. The participants emphasized that the familiarity of the game world was a very good aspect, as they knew in advance approximately how long the distances were between the places. They had former experience in travelling between places in the real world and thus they had a mental picture of the distances and how much time and effort those would require. The familiarity of the game world was an important factor to make the challenges feel concrete: “The challenges from one city to another are great. They are concrete because one knows those cities.” (male, team of 3 family members) “For me map of Finland worked very well. One has a point of comparison for real life. I thought that in some phase when Finland would be roamed through why not to expand to abroad but initially I would not need anything else than Finland.” (female, team of 4 colleagues)

Clearly connected to the previous wish, the participants wanted to be able to zoom in on the map to view their current location in more detail. In the current version there was no zooming option and it included only one, general map level. Alone this one map level was not considered to be enough, but the participants wanted to zoom in to see a more detailed part of the trip. This function would enable users to get more feedback about the progress. This related also to keeping up users’ interest and usage motivation. “It takes some time to see the progress in the beginning of the challenge, because the whole map of Finland is there. It would be great to be able to zoom in to the ongoing trip so that one could see the progress better. Now one needs to walk quite long before seeing any achievement.” (female, team of 4 colleagues) “The map increased the usage motivation of the application. But it could also be better in a way that one could zoom in closer to see more detailed map.” (female, team of 4 colleagues)

Most participants commented that travelling between the provided cities and villages was very clear and concrete way to view the progress. However, a minority of the participants said that they could not perceive the distances between the provided places, because they were not so familiar with them. These participants wished to have a possibility to set the challenge between the places that were even more familiar to them, for example one participant wanted set the route to grandmother’s house as a challenge, and another wished to use the trip to the summer cottage: “I’m so weak in geography that I have no idea what it means to travel from, for example, Mikkeli to somewhere.” (female, team of 6 neighbors) “It would be nice to create own, personal challenges, because not everybody live in bigger towns. Smaller places would be nice. I think I would like to travel to our own summer cottage. That would be more concrete and comparable then.” (female, team of 4 friends)

Some participants felt that even though a familiar game world was welcome, travelling the same routes on and on again might be less interesting in the long run. They wished that after Finland would be roamed through, the game world would expand to contain other countries as well. Some participants also wished to be able to travel between famous points of interest or along a well-known travel path: ”If one used the application for a long time, going same routes again and again could become boring. For the sake of meaningfulness and variability it would be good to be able to get the maps of other countries. One would learn geography as well.” (female, team of 4 colleagues) “It would be really great if there were some famous routes, for example the pilgrimage in Spain, so that you could go walking virtually to Santiago de Compostela or some other route. Or you could decide to walk a virtual tour in Rome this weekend.” (male, team of 3 family members)

Ideas for Further Design

The interviews revealed three main wishes from the participants for the future design of the application related to the virtual trips, challenges and the game world. They were intermediate points, map zooming and more versatile types of challenges, to be discussed in this section. As could be seen on the user comments and perceptions on the previous sections, many participants liked to view and think about where they were going at the certain moment on the map. The current version did not show any intermediate points on the way from the departure to the destination, so the users needed to guess what there would be along the way. To maintain the user interest in the longer time period the challenges could be divided into shorter stages, and reaching these stages could reveal some content on the map, for example, the name of the city, village, castle, or other landmark that the team has passed by, or some visual material from those places:

Some participants lamented the fact that accelerometerbased pedometers can not measure all exercise types (e.g. biking or swimming), and thus the users did not get any credit out of such efforts. Into shares this problem with other step counter based applications. The participants also mentioned that they did not want to keep the phone in the pocket at home, and consequently missed steps. Moreover, sometimes when exercising they did not have suitable pockets in which to keep the mobile phone. Although mobile phones are nowadays carried along in almost all situations, clearly there are some occasions when it is not convenient to carry one. Similar limitations apply to most wellness technology, however.

10

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

related lifestyle changes usually do not happen immediately. The participants wished for intermediate points when travelling between places, and a zoom infeature to view the game world and route in more detail. Those would enable better possibilities for exploring the game world, which we think is one of the factors to keep up the long term user interest.

DISCUSSION & CONCLUSIONS

We have presented the background and design of a mobile application for supporting physical exercise activities. The Into application utilizes two known motivating factors towards increasing the level of physical activity: social sharing and playfulness. In the application, the physical activity progress of the team is presented with an analogy of a virtual trip. The main findings (detailed in Section “Findings”) related to the use of the analogy can be summarized as follows:

More versatile challenges and routes, maybe also an expanding game world, might increase the motivation towards the use of the application and conducting physical activities over longer time periods. For this aim we would like to explore the future use of real-world maps that are already included in many high-end phones for navigational purposes. Alternatively, artificial yet familiar wellness game worlds might be inviting for players of non-wellness games. For instance people who spend substantial fractions of their waking hours in the realms of World of Warcraft might prefer to map their physical activity into the same familiar universe.

• Users liked the idea of presenting the team’s physical activity results as a virtual trip, including setting departure and destination places and viewing an up-todate progress between them. • A virtual trip was considered an understandable, concrete and interesting analogy. • According to the users, the familiarity of the game world was an important factor to make the challenges feel concrete.

As our present study was qualitative in nature, and the trial period of the application was relatively short, broad generalizations based on the results are not relevant. Moreover, in the existing social groups available for the study, gender balance could not be achieved. In addition, the participants all came from a nation with a strong culture of outdoor activities [19] and a pervasive use of mobile devices for communication and exercise tracking.

Our study showed that the familiarity of the game world, in this case a map of the participants’ home country, provided a successful mental model for perceiving and understanding the physical activity data presentation. The participants knew in advance approximately how long the distances were between the places as they had former experience in travelling between them. So they had a mental picture of the distances and how much time and effort it would require to travel between them. Our trip analogy proved successful in this experiment.

Interesting research questions for the future research do arise. We would like to study further the use of the application over a longer time period and in different contexts. Dix [14] warns about the cultural bias that the metaphors may portray. That is why we would like to compare users from different cultures to know how the analogy of a virtual trip for presenting physical activity data works globally. We would also like to study more the familiarity of the game world, whether we can generalize the results to the representatives from other cultures as well.

The same analogy has also been successfully used by some exercise campaigns [20], [26] though without mobile phone technology. Mobile phone as a platform for the application enables effortless data measuring with an inbuilt accelerometer, seamless and real-time data transfer between the team members, and up-to-date follow-up of the progress, when compared to the manual tracking and data transfer techniques used by the above mentioned exercise campaigns. Our study did remind us, however, that a mobile phone is not always the ideal tool for tracking the exercise, because in some situations it is not convenient to carry along. People may not want to carry a phone when they spend time at home, and sometimes when exercising they do not have suitable pockets for the phone. A very small wearable sensor could solve the problem, but on the other hand it would be another separate piece to remember. Sensors that are integrated into fitness clothing and able to communicate with mobile devices offer a partial solution to this problem. However, of the contemporary devices that are widely accepted among the consumers, the mobile phone was the best platform for our application.

We were satisfied with the application in relation to our research focus, as it allowed us to evaluate it in the field settings in real use by people without technical background. The application concept seems to be a good one, as the user comments were so positive in spite of the limitations. All participants used the application until the end of the period and a majority of them liked it a lot. Future work includes longer term experiments to provide quantifiable measures on the application’s ability to improve exercise motivation. REFERENCES [1] Ahtinen, A., Isomursu, M. Huhtala, Y., Kaasinen, J., Salminen, J. and Häkkilä, J., 2008. Tracking Outdoor Sports – User Experience Perspective. In: E. Aarts, ed. European Conference Ambient Intelligence, Springer-Verlag, 192-209. [2] Ahtinen, A., Isomursu, M., Mukhtar, M., Mäntyjärvi, J., Häkkilä, J., and Blom, J. 2009. Designing social features for mobile and ubiquitous wellness applications. In Proc. 8th

Encouraged by the positive feedback from the participants, we will continue with the development of the concept. Maintaining long term motivation and rewarding should be addressed in Into, like in any application meant for supporting and motivating physical activity [12]. Wellness

11

Full Papers

[3]

[4]

[5]

[6]

[7] [8] [9]

[10] [11]

[12] [13]

[14] [15] [16] [17]

Proceedings: NordiCHI 2010, October 16–20, 2010

intl. conf. on Mobile and Ubiquitous Multimedia (MUM '09). ACM, New York, NY, 1-10. Ahtinen, A., Ramiah, S., Blom, J. and Isomursu, M., 2008. Design of Mobile Wellness Applications: Identifying CrossCultural Factors. In Proc. 20th Australasian Conf. on Computer-Human Interaction. Albaina, I. M., Visser, T., Van der Mast, C. and Vastenburg, M. H., 2009. Flowie: A Persuasive Virtual Coach to Motivate Elderly Individuals to Walk. In 3rd International ICST Conf. on Pervasive Computing Technologies for Healthcare, 1-3 April 2009 London, UK. Anderson, I., Maitland, J., Sherwood, S., Barkhuus, L., Chalmers, M., Hall, M., Brown, B. and Muller, H. Shakra: Tracking and Sharing Daily Activity Levels with Unaugmented Mobile Phones. Mobile Network Applications, 12, 2-3 (2007), 185-199. Asselin, R., Ortiz, G., Pui, J., Smailagic, A., and Kissling, C. Implementation and Evaluation of the Personal Wellness Coach. In Proc. 25th IEEE Intl. Conf. on Distributed Computing Systems Workshops, pp. 529-535 (2005). Berkovsky, S., Bhandari, D., Kimani, S., Colineau, N., and Paris, C. 2009. Designing games to motivate physical activity. In Proc. Persuasive 2009, ACM Press (2009), 1-4. Beyer, H. and Holtzblatt, K. Contextual Design – Defining Customer-Centered Systems. Morgan Kauffman, 1997. Bickmore, T. W., Caruso, L., and Clough-Gorr, K. Acceptance and Usability of a Relational Agent Interface by Urban older Adults. Extended abstracts of the Conf. on Human Factors in Computing Systems, pp. 1212-215 (2005). Blythe, M.A., Overbeeke, K., Monk, A.F. and Wright, P.C. (eds.) Funology, From Usability to Enjoyment. Kluwer Academic Publishers, MA, USA, 2003. Buttussi, F., Chittaro, L., and Nadalutti, D. Bringing Mobile Guides and Fitness Activities Together: A Solution based on an embodied virtual trainer. In Proc. 8th Conf. on Humancomputer interaction with mobile devices and services, pp. 29—36 (2006) Consolvo, S. Everitt, K., Smith, I. and Landay, J.A. Design Requirements for Technologies that Encourage Physical Activity. In Proc. CHI 2006, ACM Press (2006), 457-466. Consolvo, S., McDonald, D. W., Toscos, T., Chen, M. Y., Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., LeGrand, L., Libby, R., Smith, I. and Landay, J.A., 2008. Activity Sensing in the Wild: A Field Trial of UbiFit Garden. In Proc. CHI 2008, ACM Press (2008), 1797-1806. Dix, A., Finlay, J., Abowd, G. and Beale, R. HumanComputer Interaction. 2nd edition. Pearson Education Limited, UK. Fujiki, Y., Kazakos, K., Puri, C., Buddharaju, P., Pavlidia, I., and Levine, J. NEAT-o-games: blending physical activity and fun in the daily routine. Comput. Entertain. 6, 2 (2008). Garmin. http://www.garmin.com Gockley, R., Marotta, M., Rogoff, C. and Tang, A. AVIVA: A Health and Fitness Monitor for Young Women. Ext. Abstracts CHI 2006, ACM Press (2006).

[18] iPhone pedometer applications. http://www.iphoneappsfinder.com/blog/15-best-pedometerapplications-for-iphone/ [19] Kafatos, A., Manios, Y., Markatji, I., Giachetti, I., Vaz de Almeida, M. D. and Engstrom, L. M. Regional, demographic and national influences on attitudes and beliefs with regard to physical activity, body weight and health in a nationally representative sample in the European Union. Public Health Nutrition: 2, 1a (1999). [20] Kuntoliikuntaliitto Kävelykampanja. http://195.17.28.12/kuntostart/index.asp [21] Laverie, D.A. Motivations for Ongoing Participation in Fitness Activity. Leisure Sciences 20 (1998). [22] Lin, J. J., Mamykina, L., Lindtner, S. Delajoux, G. and Strub, H. B. Fish’n’Steps: Encouraging Physical Activity with an Interactive Computer Game. In Proc. Ubicomp 2006, Springer (2006), 261-278. [23] Mobile Coach. http://www.firstbeattechnologies.com [24] Nielsen, J. Usability Engineering. 1993. Academic Press, US. [25] Nintendo. http://www.personaltrainerwalking.com/walk-theworld/ [26] Nuori Suomi Liikuntaseikkailu. http://maapalloseikkailu.nuorisuomi.fi/fi-FI/ [27] O’Brian, S. and Mueller, F. Jogging the Distance. In Proc. CHI 2007, ACM Press (2007), 523-526. [28] Patrick, K., Griswold, W.G., Raab, F. and Intille, S.S. Health and the Mobile Phone. American Journal of Preventive Medicine 35, 2 (2008), 177-181. [29] Schlenk, E. A., Dunbar-Jacob, J., Sereika, S., Starz, T., Okifuji, A. and Turk, D. Comparability of daily diaries and accelerometers in exercise adherence in fibromyalgia syndrome. Measurement in Physical Education & Exercise Science 4, 2 (2000). [30] Sharp, H., Rogers, Y. and Preece, J. Interaction Design: Beyond Human-computer Interaction. 2nd Ed. 2007. John Wiley & Sons, UK. [31] Speck, B. J. and Looney, S. W. Effects of a Minimal Intervention to Increase Physical Activity in Women: Daily Activity Records. Nursing Research 50, 6 (2001). [32] Sports Tracker. http://sportstracker.nokia.com/nts/main/index.do [33] Ståhl, A., Höök, K., Svensson, M., Taylor, A.S. and Combetto, M., 2009. Experiencing the Affective Diary. Personal and Ubiquitous Computing 13 (5), 365-378. [34] Ståhl, T., Rütten, A., Nutbeam, D., Bauman, A., Kannas, L., Abel, T., Lüschen, G., Rodriquez, D., Vinck, J. and van der Zee, J. The importance of the social environment for physically active lifestyle – results from an international study. Social Science and Medicine 52 (2001). [35] Toscos, T., Faber, A., Connelly, K. and Upoma, A.M. Encouraging Physical Activity in Teens – Can technology help reduce barriers to physical activity in adolescent girls? In Proc. Pervasive Health 2008. [36] Wellness Diary. http://betalabs.nokia.com/apps/wellnessdiary [37] Wii Fit. http://wiifit.com/

12

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Location-based Crowdsourcing: Extending Crowdsourcing to the Real World Florian Alt, Alireza Sahami Shirazi, Albrecht Schmidt, Urs Kramer, Zahid Nawaz University of Duisburg-Essen Pervasive Computing and User Interface Engineering Schützenbahn 70, 45117 Essen, Germany {alireza.sahami, florian.alt, albrecht.schmidt}@uni-due.de, {urs.kramer, zahid.nawaz}@stud.uni-due.de ABSTRACT

Many projects emerged over the past years whose success is based on the contributions of a huge number of people. Wikipedia is a prominent example, which utilizes the broad knowledge of a massive number of people on the Internet. OpenStreetMap is another example where many users, living in different geographical regions, contribute, share, and process their location tracks to make a comprehensive online map. These are just two of many examples where a large number of people, who are often part of a community, make small contributions, which led to a completely new type of applications that would have been hardly imaginable before the pervasive availability of the WWW.

The WWW and the mobile phone have become an essential means for sharing implicitly and explicitly generated information and a communication platform for many people. With the increasing ubiquity of location sensing included in mobile devices we investigate the arising opportunities for mobile crowdsourcing making use of the real world context. In this paper we assess how the idea of user-generated content, web-based crowdsourcing, and mobile electronic coordination can be combined to extend crowdsourcing beyond the digital domain and link it to tasks in the real world. To explore our concept we implemented a crowdsourcing platform that integrates location as a parameter for distributing tasks to workers. In the paper we describe the concept and design of the platform and discuss the results of two user studies. Overall the findings show that integrating tasks in the physical world is useful and feasible. We observed that (1) mobile workers prefer to pull tasks rather than getting them pushed, (2) requests for pictures were the most favored tasks, and (3) users tended to solve tasks mainly in close proximity to their homes. Based on this, we discuss issues that should be considered during designing mobile crowdsourcing applications.

With the ubiquity of interactive mobile devices providing location awareness and network connectivity we expect this trend to accelerate. People carry their phones with them the entire day, providing them the opportunity to contribute at any time. We imagine that new forms of contributions (e.g., real-time media and tasks that require physical presence) will become accessible similar to knowledge work and information sharing in the WWW. Smart mobs [15] and dynamic ride sharing services are current examples that involve physical presence in order to complete a task. One specific form of harvesting wisdom of the crowd and contributions from users is crowdsourcing, as introduced by Jeff Howe [6]. The concept describes a distributed problem-solving and product model, in which small tasks are broadcasted to a crowd in the form of open calls for solutions. As a strategic model, crowdsourcing tries to attract interested and motivated crowds capable of providing the required solutions in return for incentives (mainly small amounts of money). Often, such so-called crowd workers gather in online communities consisting of experts, small businesses, and other volunteers working in their spare time. As a result, problems can be addressed very quickly, at little cost, and the task provider might exploit a wider range of talents [9]. Tasks are normally initiated by a client and are open either to anyone or to particular communities. The solution may be submitted by individuals as well as by a group. In comparison with ordinary “outsourcing”, a task or problem is outsourced to an undefined public rather than to a specific body. Crowdsourcing is effective in areas where the task can be easily described to humans and where these tasks are easier to do for humans than for computers, e.g., perception tasks and tasks involving creativity.

Authors Keywords

crowdsourcing, mobile phone, context, location Categories and Subject Descriptors

H.5.3 [Group and Organization Interfaces]: Collaborative computing, Evaluation/Methodology, Organizational design; H.5.2 [User Interfaces] INTRODUCTION

Over the years the World Wild Web (WWW) has evolved beyond being a platform for retrieving information only but has become a ubiquitous medium supporting various forms of communication, peer-to-peer interactions, shared collaboration, and the creation of user-generated content.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

13

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

We argue in this paper that mobile crowdsourcing offers great potential and new qualities when considering and exploiting the context of the user, e.g., his location. Mobile phones are ubiquitous in many parts of the world. Nowadays most devices provide not only means for communication and interaction, but they typically are enhanced with a range of different sensors (e.g., camera, GPS, accelerometer), hence making it possible to easily extract context information. When additionally considering that the WWW and data services are becoming more common on mobile phones, we envision such devices being the upcoming platform for crowdsourcing. We believe that mobile, and especially location-based, crowdsourcing has the potential to go beyond what is commonly referred to as “traditional” (digital) crowdsourcing by bringing it to the real world. Therefore we exploit both the seekers’ and the solvers’ physical location. We focus especially on tasks that go beyond the provision of digital content with no clear limitation on how they are being solved.

rely on the active contribution of the Web community are the center of the Web 2.0 phenomena. The contribution to services happens in different ways. In the following we discuss user-generated content and crowdsourcing in more details, as they have been the inspiration for the implemented platform. User-Generated Content

The creation of content can be discriminated in explicit and implicit content generation. It can be generated on the initiative of the contributor (e.g., adding a new entry in Wikipedia), based on a coordinated call (e.g., someone asks for the clean-up of an article, someone initiates a call to read chapters in librivox1), or on request from a potential web user (e.g., a request in a forum). Explicit content generation describes the process in which a number of web users individually produce content. The content production may be carried out independently or as a part of a coordinated effort. In both cases the central value is in the collective result. Wikipedia, an online encyclopedia, is an example, which is created based on entries added by a large number of web users. Similar examples are product reviews, experience reports, and recommendations provided by customers for others in online shopping platforms. Such collections are sometimes seen as making use of the wisdom of the crowd. There have been recent researches that assess how to best harness the wisdom of the crowds [9] [4]. Explicitly generated content requires effort by the user and typical incentives are peer recognition or immaterial or material benefits, such as payments or vouchers. In [12] the authors investigate how financial incentives impact the performance.

We conducted two field studies that focus on the evaluation of constraints and challenges that affect the crowd workers’ behavior. We found out that location (and hence the opportunity to retrieve tasks in the vicinity) has a crucial impact when it comes to assigning tasks by the crowd workers. Nowadays time is scarce. Thus most users preferred quickto-solve tasks in close proximity, which required minimal effort. This is backed up by the fact that crowd workers in the study tended to choose tasks, which could be solved through physical interaction, e.g., taking a photo. The contributions of this paper are as follows: (1) We introduce the architecture and implementation of a prototype system, which supports mobile crowdsourcing based on location information. (2) In a qualitative user study among 18 participants we explore novel aspects of crowd working and how location-awareness may facilitate and impact on the crowd working process. (3) We present design guidelines, helping developers of context-aware crowdsourcing applications to enhance functionality and uptake among potential crowd workers.

In contrast, implicit user-generated content describes content that is generated by implicit human computer interaction [16]. A prime example is a news website that provides a category “most popular articles” or an online shop with a top 10 of sold articles. Here users generate content (in these cases recommendations) by their actions (reading, downloading, and buying). What is interesting with regard to implicit user-generated content is that there is no extra effort required for the user in order to contribute this content, nevertheless there might be a cost associated (e.g., the loss of privacy). Looking to mobile technologies an example of implicitly generated content is the route someone takes from one location to another. If this is tracked, then the data can become a resource for others, as evident in OpenStreetMap 2. While collecting the tracks happens implicitly, the post-processing is an explicit wiki-based creation of street maps with meta-information. In this case, both an implicit and an explicit approach are used. For us these examples highlight the power of mobile crowds combined

This paper is structured as follows: we start with a brief overview of related works before presenting our concept in more detail. Then we explain the implementation and architecture of our prototype system. We describe the evaluation and report the results of our user study before finally outlining and discussing our design suggestions. BACKGROUND AND RELATED WORK

The Internet has become an essential platform for seeking and sharing information, communication, presentation, and collaborating for many users. This is facilitated by many applications, platforms, and services that are provided on the Internet. For many of these systems it is essential that Web users actively participate in generating content and providing services. Such applications and platforms that

1

www.librivox.org (accessed January 2010)

2

14

www.openstreetmap.org (accessed January 2010)

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

with the WWW to create new resources. Commercial systems like HD-traffic3 information use coarse location data from cell phones to provide high quality traffic information.

difficult to solve by computers and easily accomplished by humans. Examples of such tasks are the tagging of images, e.g., images of garments for an online catalog. Here the aim is to get a representative set of keywords so that users can find what they are looking for. Other domains are natural language processing, summarization, and translation. There is no clear limitation to what type of tasks can be solved through crowdsourcing, as long as they can be described in the system and the answer can be provided over the Internet. In most cases small amounts of money as compensation are provided to the users.

In our work we have looked at different types of usergenerated content and aimed at designing the platform to include as many types as possible. We see many crowdsourcing tasks as explicit requests for specific usergenerated content. Games and Content Generation There are several examples where games are successfully

Crowdsourcing on the World Wide Web

used to create content. Von Ahn et al. have shown that labeling images can be packed and provided to the users in a playful way [19] [18]. A side effect of playing the game is then the assignment of tags and labels to images. In this approach the game itself is already the incentive for contributing the content. We can imagine that it is feasible to create mobile games where users on the move through physical space would create meaningful and valuable information. An example could be running from location A to B with the condition to cross as few roads as possible.

Currently there are several websites available that are based on the concept of crowdsourcing. Amazon’s Mechanical Turk4 is a web-based marketplace for works requiring human intelligence in which anybody can post their tasks and specify prices for completing them. iStockPhoto5 is a webbased company offering huge collections of images uploaded and sold by photographers. Clients seeking stock images purchase credits and start buying the stock images they want. Another example is Innocentive6, which allows companies with specific R&D needs to share their challenges and specify awards among scientists dispersed all over the world. The solvers can submit their solutions through the Web, which go under review by the seeker. Also CambrainHouse7, built on crowdsourcing foundations, collects, filters, and develops the software ideas coming from the crowds. Artists or anyone with spare creativity can submit their T-shirt designs in Threadsless8, a clothing company collecting votes from the community and producing the top rated designs. In [7] it is explained how the power of Web 2.0 technologies and crowdsourcing approach are used to create new approaches to collecting, mapping, and sharing geocoded data.

Another popular type of game is Geocaching [13] [5]. Many of the motivations are not conflicting with the idea of exploiting the information people create while looking for Geocaches. In many cases players already provide photos of the location where they found the cache. This also coincides with our results discussed later where people favored photo-taking tasks. Smart Mobs and Ridesharing

Content generation and crowdsourcing tasks are so far restricted to the digital domain. From our perspective coordinated actions in the physical world such as Smart Mobs [15] or ride sharing supported by digital technologies hint a further direction of location and context-based crowdsourcing. The idea of a flash mob is that people use digital technologies and coordinate an action. If the action has a clear goal this is then considered as a smart mob. By bringing a number of people at a specific point in time to a certain location a political statement can be made, a street can be blocked, or an advertising campaign can be started.

Furthermore, there are researches that investigated various features of crowdsourcing systems. In [2] essential features of a crowdsourcing system and the precise relationship between incentives and participation in such systems are discussed. The authors reported that rewards yield logarithmically diminishing returns with respect to participation levels. In [1] authors studied Google Answer and found out that questions offering more money received longer answers. Yang et al. [20] explored the usage of the site “Taskcn”, a Chinese site where users submit solutions for various tasks and the winner earns a monetary reward. They found out that while new users are choosing unrealis-

With current mobile devices a new generation of ride sharing systems is investigated [8]. We see that ride sharing is essentially a crowdsourcing task in the physical world. It is context dependent (time and location) and may have a number of side conditions (e.g., only travelling with a person with more than 5 years driving experiences). Crowdsourcing

4

www.mturk.com (accessed January 2010)

Crowdsourcing on the WWW has gained popularity over recent years. There are several websites available serving as a platform to distribute crowdsourcing tasks. The characteristic of crowdsourcing tasks is that they are typically

5

www.istockphoto.com (accessed January 2010)

6

www.innocentive.com (accessed January 2010)

7

www.cambrianhouse.com (accessed January 2010)

8

www.threadsless.com (accessed January 2010)

3

www.tomtom.com/hdtraffic/ (accessed January 2010)

15

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

tically, those who are used to the site pursue a more profitable strategy by better balancing the magnitude of the rewards with the likelihood of success. Also Manson and Watts [12] investigated the effect of compensation and performance on Amazon’s Mechanical Turk platform and reported that increasing financial incentives increases the quantity of works done by participants but not necessarily the quality of them. In [11] a crowd translator is demonstrated that collects speech data from the crowd through the mobile phones, which is used to build a high-quality speech recognition system. In the described projects, tasks are location-independent and can be performed on any PC or mobile phone with Internet connectivity. However, there are certain situations where the problems are location-based and physical presence of a person for solving them is required. In our work we focus on location-based problems and on how crowdsourcing can be used to share and solve tasks that are inherently contextual. We fill in the gap between the seekers and a mobile crowd with a location-aware crowdsourcing platform and share tasks based on the solvers’ location. Mobile Crowdsourcing

Various research papers explored crowdsourcing based on the use of mobile phones. Eagle [3] developed txteagle, a mobile crowdsourcing system that enables people to earn small amounts of money by completing simple tasks such as doing translation, transcription, and filling out surveys by using their mobile phones. Askus is a mobile platform for supporting networked actions [10] that allows specifying tasks, which are then matched by the system to specific persons based on profiles. Such profiles may include geographic location. In contrast to our platform, Askus is pushing tasks actively based on a positive match rather than providing an open call addressed to crowd workers present at this location. Fashism9 is an online community that uses phones as a bridge between the physical and digital world. It provides an easy way for customers to get comments on their fashion style while doing shopping by sending a dressing-room photo to the community and getting votes and comments back from the crowds in real time. Google uses crowdsourcing to accumulate the road congestion data and provide the traffic conditions. A user’s phone running Google Maps10 for mobile phones sends bits of data back to Google anonymously, describing how fast he is currently moving. The combination of the data provided by the crowds supply a good overview of live traffic conditions. Ushahidi is an open-source platform from Kenya, which allows for crowdsourcing crisis information by letting par-

9

www.fashism.com (accessed January 2010)

10

www.google.com/mobile/products/maps.html (accessed January 2010)

16

ticipants submit information on violence through text messaging using a mobile phone, email, and the Web [14]. In contrast to these examples we deliberately aim at exploiting the users’ physical location and context. Additionally we investigate how the results of a crowdsourcing task can go beyond the provision of digital content. CROWDSOURCING BEYOND THE DIGITAL

When offering tasks to be solved by crowd workers, the person offering the task (in the following also referred to as the seeker) defines the product or result that is requested. Typically the seeker has little or no information about the potential crowd workers (solvers) that will carry out the task. Hence the description of the task is most critical to achieve the desired result. If tasks are beyond the digital domain and can only be solved in a certain situation it is crucial that the tasks are suggested to seekers who are likely to find themselves in such a situation. The system design needs to include means for matching potential crowd workers in the right location and at the right time with a described task. We believe that certain niches might evolve where crowd sourcing becomes remarkably relevant and successful, especially among certain communities. In the following we describe several scenarios that focus on tasks, which should be performed in a certain context in the real world and away from the desktop. Such tasks are characterized by the need to be performed in a specific location only, or require the presence of a certain context to solve it. Scenario 1: Recommendation on demand John is on his way home from work. On the train he sees an advertisement for a point-and-shoot camera he is interested in. The shop is on his way home but would require a short detour, so he uses mobile crowdsourcing to get information about the availability of the favored camera in this specific store. He is interested in some specific tangible qualities that can be easily assessed while holding the camera in his hands but would be hard to retrieve from a photo. As he trusts the judgment of a fellow customer more than the information he would get from the sales personal, he puts up the task for a crowd worker who just happens to be there. Scenario 2: Recording on demand Mary has a very important lecture today at university. The lecturer will discuss the sample questions for the final exam with the students. In the morning she sprains her ankle and hence she cannot go to university. As she knows that many students will attend the lecture, she puts out the task of recording the lecture into the crowdsourcing platform. She specifies the lecture, time, and location of what she would like to have recorded. A few minutes later she gets a positive feedback from Alex who has taken on the task and has a good reputation (completed many tasks and has been highly rated for them). Later, she receives audio and video files of the lecture as well as copies of Alex’s notes.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Scenario 3: Remotely looking around Sarah lives in New York and she is excited about a new job found in Munich. She will be there for six months and an estate agent has provided her some offers. On the Internet she finds the description of an apartment with some photos of its interior. Unfortunately, the offers did not include any photos and further information about the surrounding area. Using mobile crowdsourcing, she specifies a task and asks for more information on the area including photos. A nearby crowd worker who takes the task provides her a couple of pictures (captured with his mobile phone) of the street, some shops as well as a nice café. Scenario 4: Real-time weather information: Bob lives an hour’s drive from some great skiing resorts. As his meeting finishes before noon he decides to take the afternoon off in order to go skiing. He is not sure where to go – on the website all resorts state great conditions and perfect snow. From the webcams he cannot really see many differences. He decides to ask the crowds in his three most favorite places about the skiing conditions and crowdedness. Within a few minutes he gets back information from other skiers that provide him with the information.

Figure 1: System architecture Server

The server side is a PHP-based platform consisting of a MySQL database and several web services used by the web platform and the mobile client. The database includes a collection of all tasks defined in the system through the web platform. Each task is specified by a set of properties, which is then used to distribute it among the crowd. In Table 1 we provide a brief overview of the most important properties, which can be specified for each task. All tasks are associated with geographical information (Location), which is reused in the matching process to assign tasks to workers based on their current location.

Scenario 5: Translations on demand John is in China for two weeks. He decides to buy a very traditional souvenir for his parents. So he goes to downtown and finally finds one. But unfortunately the guys in the store cannot speak English and John does not speak Chinese and so he needs help. With the mobile crowdsourcing platform he searches for someone nearby who can help him in translating English to Chinese.

The Priority field of a task may be assigned to one of the three following priorities: priority 1 (which is the highest one) means that the task is time-critical and solutions have to be submitted within a pre-defined time period. This amount is being specified in the Expired property. Further, tasks with priority 1 are reserved for one solver only. Priority 2 tasks are not time critical (meaning that there is no specified deadline for submitting a solution), but the task is reserved to only one solver. Priority 3 tasks cannot be reserved and users should submit the solution when they sing up for them.

These scenarios show that crowdsourcing in the real world may offer a new quality. All tasks mentioned above are location-dependent and finding a person in close proximity is a crucial prerequisite. However, the tasks differ fundamentally in their time constraints, task duration, and in the way they are solved by the crowd worker. The time constraints range from minutes up to a day and similarly the duration of the tasks from seconds to several minutes. A LOCATION-BASED CROWDSOURCING PLATFORM

Web Platform

To investigate the potential of location-aware crowdsourcing we developed a prototype platform to easily create and distribute tasks among crowd workers. The platform consists of three components: (1) A web interface where seekers can upload arbitrary tasks associated with geographical information, (2) a server including a database for storing the tasks, which is responsible for distributing the tasks to potential crowd workers, and (3) a client application on mobile phones for the crowd workers, which pulls available tasks from the database based on the given location. Figure 1 depicts the system architecture.

To distribute tasks to the crowd we provide an AJAX-based web platform, which on one hand allows seekers to upload arbitrary tasks and distribute them in the community and on the other hand allows solvers to search and download tasks. Hence, we provide two different areas for seekers and solvers. The seekers’ area includes an overview of all tasks they previously specified (both solved and unsolved) where they can easily track their tasks and find out if a task was downloaded and when. We opted not to provide the names of the solvers in the prototype for privacy reasons – however we plan to integrate synonyms in the future to be able to provide a recommendation system. Further, seekers can create new tasks to be solved. All properties of the tasks

By providing such a platform we aim at bringing together all kinds of people regardless of their background, skills, and place of residence. So we are able to virtually offer seekers requesting any task at any time and anywhere.

17

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Description

A detailed description of the required task sent to the crowd worker

Location

The geographical location (longitude, latitude) for the task, e.g., a shop

Vicinity

Specifies a radius around the task location in km. This indirectly influences the amount of crowd workers receiving this task

Reward

Specifies the incentive if a crowd worker accepts to solve this task

Priority

Specifies the priority of the task

Expired

Allows for specifying a time interval in which a crowd worker has to submit the task solution. This is important for time-critical tasks

Assigned

The time a user accepted to work on the task

Solution

The solution of the task as entered by the crowd worker

Submission

The time the solution was submitted by the crowd worker

have to be provided completely before they are stored in the database and made available to potential solvers. For simplicity, seekers are not required to enter the geographic coordinates of a task but can use a nearby postal address based on which the correct coordinates are calculated. It is also possible that seekers specify the task’s location on Google Maps from where the geographical data are extracted (Figure 2). The solvers’ area includes an overview of all tasks they assigned themselves (both solved and not yet solved tasks). Besides submitting a task via the mobile client, solvers may also enter the solution to their task via the web interface. However this feature was not used during the evaluation. Mobile Client

We developed a JME-based mobile application, which can be used both by seekers and solvers. For solvers, the application lets users retrieve tasks from the database based on given locations. The location can be retrieved either by using an (internal or external) GPS receiver or by providing a postal address (street and zip code/city), which is then converted into geographical coordinates on the server. Additionally a Google Maps view is provided to the users to specify their location and retrieve tasks. This setup allows for retrieving tasks not only based on the current location but also based on other locations, which crowd workers potentially might visit in the near future (e.g., on the way home). Additionally, users can use this feature indoors where GPS devices do not work properly.

Table 1: Task’s properties specified by the seeker

When it comes to working on a task, two different modes of the mobile application can be distinguished: the assignment mode and the solution mode. In the assignment mode the crowd worker can browse and preview available (location-based) tasks based on the given location. Once he decided to work on a task, he may assign himself the task. Based on the Priority property, the task may be unavailable to other workers for the period of time specified in the Expired property. If the user does not submit his task during this period, the task is released and become available again to all crowd workers. It is also possible to assign more than one task at a time (however we limited the amount of parallel tasks to two for the study). In the solution mode, workers can submit their solutions for assigned tasks. In the current version, the submission can be based on texts and pictures. During the design process of the user interface, we focused on keeping the interface simple. Hence, we use a wizard interface to guide users through the different steps of assigning, solving, and submitting the tasks. Figure 3 shows screenshots from the mobile clients.

Figure 2: The web client: seekers can define new tasks in database and use Google Maps to specify the location.

(a)

When using the mobile client as a seeker, the user may create new tasks and publish them to solvers by uploading them onto the server. This allows seekers to create tasks at any time, e.g., while being in transit. The location of the task can be set independent of the current location.

(b)

Figure 3: The mobile client screenshots: (a) Main menu where users can search tasks. (b) A sample task retrieved from the database.

18

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

USER STUDY

To gather data on the users’ crowdsourcing behavior, we conducted two user studies. To obtain reliable results we wanted users to act in their natural environment. Hence, we opted to do field studies in a familiar setting that is at home, at work, and in the city center of users’ hometowns. Initial Study

Due to the complex setup und amount of influencing factors, we decided to run a pre-study for gaining initial insight in potential challenges as well as a better understanding of the users’ crowdsourcing behavior.

Photo Task

Informative Task

Action Task

Take a photo of the closest mailbox

Check the price for a 8GB iPod Nano in the nearby electronic store

Send an email to the user study instructor telling him how many bottles of wine you have at home

Take a photo of your refrigerator

Check how many of the laptops on sale are left at the discounter market

Buy a small bottle of coke from the store around the corner

Take a photo of the coffee machine

Check the departure time of the next bus

Bring a cup of coffee to your colleague next door

Take a photo of your desktop

Count the number of colleagues currently at their desks

Call the office of the user study instructor

Take a photo of the central post office

Check the opening hours of the Apple store

Buy some milk chocolate

Home

Demographics

For the study we recruited nine volunteers via mailing lists, from our courses, and friends. In total, four males and five females participated in the study. Participants were employees (3), students (4), and interns (2), with an average age of 27 years. Prerequisites to participate were that the subjects went to office or the university campus minimum once per working day, and had to own a mobile phone.

Office

Study Setup

To simulate location-based crowdsourcing we asked the participants during the recruiting process to provide us their home and office address, which we used to define different tasks in the database. We divided the tasks into two different dimensions: location and type of tasks. For the location we had (1) tasks at/nearby the users’ home location, (2) tasks at/nearby the users’ office locations, and (3) tasks in the city center of their hometown. For the types of tasks, we distinguished between (1) photo tasks, which required using the phone’s camera, (2) informative tasks, which required the user to type in some information into the application, and (3) action tasks, which required the user to perform a certain kind of action (see Table 2).

City center

Table 2: A task table for one participant on the 3rd day

groups changed the retrieving mode, which could be verified based on the log data. After the two-week study, participants were asked to fill in a final questionnaire. Results

The experience we gathered during the initial study showed that location-based crowdsourcing has potential for many application areas. From 50 tasks in the database 30 have been accomplished in the end. Based on the study, the questionnaire, and the analysis of our log data we extracted the following key findings:

For each participant we then created two different tasks per location and per task type, resulting in an initial set of 6 tasks per user (54 altogether). Then we “refilled” the task list daily based on the tasks the participants opted to complete. There was no restriction for the participants solving tasks and they were free to solve as many as they wanted. We assigned geographical coordinates and a vicinity of 2km to each task based on the location where we wanted the task to be solved. Table 2 shows a sample set of tasks for one participant for a day. Based on the current location maximum just six tasks were visible at the same time.

Users prefer address-based task selection: Our questionnaire revealed that more than three quarters of the participants preferred to retrieve tasks using postal addresses. There are three potential reasons for this: First, both connecting an external as well as activating an internal GPS device puts a burden that many users are not willing to accept. Second, users were often indoors or in public transportations when selecting tasks using the GPS receiver does not work properly. Third, tasks were mainly situated in places people lived in and were familiar with, which made address-based selection an easier option.

For those participants who had a mobile phone equipped with GPS and compatible with our application we helped them with the installation and testing. The other participants were provided with Nokia N73 phones where we had preinstalled the application. Since our application allows for extracting tasks based on both geo-graphical coordinates and postal addresses, we divided the participants into two groups. We asked the first group to only retrieve tasks using the GPS receiver in the first week whereas the other group started with address-based selection. After one week,

Picture tasks are most popular: Interestingly taking pictures was the most popular task among the participants. Obviously most participants could easily handle using the camera of the mobile phone and preferred this task against

19

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

more time-consuming informative tasks (due to the need to type in the solution) and the action tasks.

Study Setup

The study ran for a week with a similar setup as the initial study, however we made minor changes. To provide an even more natural environment, we asked the participants to use their own mobile phone. We invited participants to our lab and after explaining the study to them we installed and tested the application on all phones.

Tasks were mainly solved at or close to home: Based on the results from the questionnaire and an analysis of the solved tasks we found out that the favorite location where users worked on their tasks was at or close to their homes (45% of the tasks).

The tasks used over the course of the study were similar to the initial study. However, since we found out that users completed most tasks in close proximity of their daily whereabouts, we asked the participants to provide us a frequent visited address in addition to their home and office addresses, e.g., their parents’ or friends’ addresses to better simulate a real-world scenario. The given addresses were used to define different tasks in the database (see Table 2). Since most of the tasks in the first study were solved after 3pm, we decided to refill the task tables for each participant in the evenings. As compensation we paid each participant 20 Euros.

Tasks are solved after work: When analyzing the times users assigned themselves tasks, we found out that there are peaks in the morning (25% of the tasks were assigned between 6am and 9am, most likely on the way to work), during lunch time (21%), and after 5pm (40%). Interestingly, tasks assigned in the morning were not solved immediately, but mainly after work (81% of all solutions were submitted after 3pm). Thus, we conclude that people tend to download tasks whenever they are free (on the way to work, during breaks, on the way home) and potentially plan to solve the tasks on their way home. This is also supported by the fact that tasks close to home are mainly solved in the study. Further, this might also explain why users favored addressbased selection, since retrieving tasks at locations different from the current one is only possible with address-based selection in our implementation.

In the first study we found out that users were hardly interested in using GPS data for retrieving tasks, hence we disabled this feature for this study and asked all the users to search for tasks by postal address only. Similar to the first study, there was no limitation on solving tasks and they could decide whenever they wanted to use the system. After a week we asked the users to fill in an online questionnaire, which included questions related to the crowdsourcing platform and a System Usability Scale (SUS) test.

Response times vary: When analyzing the response times (the time between assigning a task and submitting the solution), we found that 40% of the tasks were solved within the first 3 hours, 70% within 15 hours, and 90% within 20 hours. This implies that mobile crowdsourcing works very well within a time frame of one day – however for timecritical tasks, further investigation is necessary.

Results

During this study 55 out of 110 tasks we provided in the system beforehand were completed successfully (average per participant = 6.1, SD = 2.4). Based on qualitative and quantitative feedbacks from the questionnaire and the log files we derived the following results (results are based on a 5-Point Likert scale, 1 = don’t agree at all, 5 = fully agree; given percentages are based on ratings of 4 or 5):

Second Field Study

Based on the result of the pilot study where we mainly gathered qualitative feedback, we conducted a second field study with 9 participants, aiming at providing empirical evidence for our findings. The participants were recruited from mailing lists and none had participated in the first study. The participants used their own mobile phones and SIM cards. Their phones had an integrated camera (hence all participants could solve picture tasks), and we made sure that the application was compatible with each phone.

Informative tasks are as popular as Picture tasks: from all accomplished tasks, 23 were Picture tasks, 21 were Informative tasks, and 11 were Action tasks. The popularity of those two types of tasks is also verified by the participants’ answers in the questionnaire: 89% of all users agreed or fully agreed to prefer Picture tasks, 89% answered to prefer the Informative task, and 11% to prefer Action tasks. This shows that Informative tasks were equally interesting for the users even though they had to enter texts as solutions making those tasks more complex than Picture tasks.

Demographics

The participants of the second study were three females and six males with an average age of 26 years. Six participants were university students with various majors (computer engineering, economics, applied informatics, education science) and the other three were employees. Further, five participants had surfed the WWW via their mobile phones before and five had at least once installed an application on their mobile phones. Only one of the participants had experience with crowdsourcing platforms (Amazon Mechanical Turk).

Time-critical tasks are out of interest: from 55 completed tasks, 30 tasks had priority 3, 20 tasks had priority 2, and just one task had priority 1. This indicates that solving time-critical problems through the location-based crowdsourcing platform cannot be achieved easily because crowd workers prefer tasks without temporal constraints.

20

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Solution should be achievable in 10 minutes: based on the questionnaire results the preferred amount of effort users were willing to take for solving a task is up to 10 minutes (88%). This also supports the previous finding since Picture and Informative tasks can, in general, be considered to be less time consuming than Action tasks. Put in other words, this is an indicator for time intensive tasks being less popular and it might be hard to find crowd workers for solving such tasks. Tasks are still solved after work: when it comes to the time the tasks’ solutions were submitted, we realized that 64% of the tasks were solved after 1pm (most likely after work). Additionally, based on the questionnaires’ results, 55% of the participants preferred to use the system in the afternoon and 11% at night (see Figure 4).

Figure 4: Tasks submitted over the course of a day. Challenges

We discovered the following challenges:

Midday breaks are good times to search for task: based on the questionnaire, 45% users tended to search for tasks during midday breaks such as lunchtime or between the lectures and 33% on the way home.

Users prefer to retrieve tasks by themselves: Users are willing to enter locations by themselves and having control over addresses used while interacting with the system. One main reason can be privacy issues, which is an important aspect in location-based systems.

Solving a task can take up to one day: the analysis of the response time (the time between assigning a task and submitting the solution) revealed that 57% of the tasks were solved within 10 minutes, 66% within two hours, and 95% within one day. This supports the finding of the initial study that mobile crowdsourcing works well within a maximum time frame of one day.

Provide means for easily choosing the task location: In many cases the locations where users assign themselves tasks are not necessarily the places where they solve them. Hence, an application should enable the user to choose any location, preferably close to their home. Here it might be helpful to provide a map of the surrounding area where users could easily click on a location. Additionally, the frequent locations users visit (e.g., parents’ or friends’ home, clubs...) have potential for distributing tasks. Hence, history of locations where users used the system and searched for tasks should be taken into account for dealing out tasks. Being able to assign oneself tasks over the course of a day seems to be a helpful feature with good uptake.

Home and surrounding areas are the most favorite places for solving tasks: interestingly, based on the results 66% of the accomplished tasks were retrieved at home and 61% of the solutions were submitted at home. Based on the questionnaire, 77% of the participants tend to solve tasks at home or close to it, 55% around the locations they visited frequently (e.g., downtown, clubs), and 44% around the location they daily went to for shopping.

Provide means for specifying priorities: Users prefer working on tasks in the after hours, although they tend to search for tasks during the midday breaks or on the way home. Hence, this is where the response time is likely to be short. However this means that seekers setting up tasks in the morning might have to wait the entire day for a result. We suggest using priorities and timer mechanisms for timecritical tasks.

Voluntary tasks have lower chance: the questionnaire revealed that 77% of the participants had done the task just because of the money, only 22% did it for having fun. Users search for tasks in their current location: we asked users if they searched for tasks in their current locations or locations which they plan to visit during a day. Based on the results, 88% wanted to search tasks in their current location and 22% also wanted to search tasks in locations where they are going to visit during the day.

Minimal effort for the crowd worker: We realized in the user studies that tasks requiring minimal efforts are in favor among users. Users mostly want to download a task and solve it afterwards and they tend to send up to 10 minutes to solve a task. Taking pictures was most popular, most likely due to the fact that no typing was required. Nevertheless, Informative tasks were also in the users’ favor, since they needed to type very short text. The same might be true for audio and video recording, when the application’s interface provides an easy way of using those features. Hence, it is a good strategy to break up tasks into minimal parts with short solutions.

The SUS score from the survey was 79.44, which indicates that users were comfortable with the application. DISCUSSION

Based on the user studies and the survey, the findings indicate that the design choices made for the types of tasks as well as for the means and time of delivery will impact how well location-based crowdsourcing will work. One shortcoming of the study was that tasks were not user-generated but self-designed. Yet, we envision only a minor impact on the solver’s behavior even for different tasks.

21

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Privacy

[3] Eagle, N.: txteagle: Mobile Crowdsourcing. In Proc. of

Working on location-based tasks raises severe privacy concerns, since from both the location where a user assigns himself a task (e.g., an address) as well as from the task description (e.g., check the opening hours of the post office) the current location of the user can be derived. However, this is not an implementation issue but rather a conceptual problem, which cannot be entirely solved on the system side. Possible options are not to associate the solution submission-time with a task (which only allows to derive an interval in which a user was at a certain location) and to provide users a way to manually enter their current location in an abstract way, such as a zip code or an address in the proximity.

Human-Computer Interaction International (HCII), vol. 5623, pp.447-456, San Diego, CA, July 2009. [4] Fuxman, A., Tsaparas, P., Achan, K., and Agrawal, R.

2008. Using the wisdom of the crowds for keyword generation. Proc. of WWW, 2008, 61-70. [5] Gram-Hansen, L. B. 2009. Geocaching in a persuasive

perspective. Proc. Persuasive '09, vol. 350. [6] Howe, J. Crowdsourcing: Why the power of the crowd

is driving the future of business. Crown Business, 1st Edition August, 2008 [7] Hudson-Smith, A., Batty, M., et al. Mapping for the

Masses: Accessing Web 2.0 Through Crowdsourcing. Social Science Computer Review 27, 4 (2009), 524.

CONCLUSION

In this paper we investigated how crowdsourcing can be extended beyond the digital domain. Based on a discussion of different approaches for content generation, that is explicitly and implicitly, we have proposed an approach for location-based crowdsourcing. To explore the opportunities we created a prototype for location-based mobile crowdsourcing consisting of a web and a mobile client. Through these clients, people of the crowd (solvers) can search for tasks and submit solutions that have a link to the real world.

[8] Kamar, E. and Horvitz, E. Collaboration and shared

plans in the open world: studies of ridesharing. Proc. AI 2009 [9] Kittur, A. and Kraut, R. E. 2008. Harnessing the wis-

dom of crowds in wikipedia: quality through coordination. Proc. CSCW '08, 37-46. [10] Konomi, S., Thepvilojana, N., Suzuki, R., Pirttikangas,

S., Sezaki, K., Tobe, Y.: Askus Amplifying Mobile Actions. Proc. Pervasive 2009, pp. 202-219.

We evaluated the system in two field studies with 18 participants. The results show the feasibility of location-based crowdsourcing and highlight important aspects. In the discussion we addressed discovered issues and presented recommendations for design and improvement of a mobile crowdsourcing platform. When designing location based crowdsourcing systems and mechanisms for distributing tasks among the crowd the following aspects are crucial for its success and should be supported by the system: chosen location (at best close to the crowd workers home), the type of task (most favorite tasks are taking photos), and the time of day (preferably after work).

[11] Ledlie, J., et al. Crowd Translator: On Building Local-

ized Speech Recognizers through Micropayments. [12] Mason, W. and Watts, D. J. 2009. Financial incentives

and the "performance of crowds". Proc. HCOMP '09, [13] O'Hara, K. 2008. Understanding geocaching practices

and motivations. Proc. CHI '08, 1177-1186. [14] Okolloh, O.: Ushahidi, or'testimony': Web 2.0 tools for

crowdsourcing crisis information, published in Participatory Learning and Action, No. 59, 2009 [15] Rheingold H, 2002 Smart Mobs. The Next Social Rev-

olution (Basic Books, Cambridge, MA)

As a future work we plan to enhance our applications with video and audio features. Further, it might be interesting to take additional types of users’ context into account. We believe that “targeting” tasks might increase the potential of crowdsourcing if the client application is able to learn, e.g., routes the user takes as well as types of tasks and locations he prefers or frequently visit. Finally, we plan to explore how the crowdsourcing application impacts on the uptake and the user behavior among communities (e.g., people might agree to solve tasks without incentives, or provide higher quality results).

[16] Schmidt, A. 2000. Implicit human computer interac-

tion through context. Personal and Ubiquitous Computing. 4 (2) Springer. June 2000. pp 191-199. [17] Surowiecki, J. The Wisdom of Crowds: Why the Many

are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations., Doubleday [18] von Ahn, L. and Dabbish, L. 2004. Labeling images

with a computer game. Proc. CHI '04. 319-326. [19] von Ahn, L., Liu, R., and Blum, M. 2006. Peekaboom:

REFERENCES

a game for locating objects in images. Proc. CHI '06

[1] Chen, Y., Ho, T.H., and Kim, Y.M. Knowledge market

[20] Yang, J., Adamic, L.A., and Ackerman, M.S. Crowd-

design: A field experiment at Google Answers. Working Paper, November 2008.

sourcing and knowledge sharing: Strategic user behavior on Taskcn. In Proc. of Electronic commerce,

[2] DiPalantino, D., Vojnovic, M. Crowdsourcing and all-

pay auctions. Proc. of Electronic commerce, 2009

22

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Analysis of Precedent Designs: Competitive Analysis Meets Genre Analysis Mattias Arvola Department of Computer and Information Science Linköping University SE-581 83 Linköping [email protected] +46 13 285626

Jonas Lundberg Department of Science and Technology Linköping University SE-581 83 Linköping [email protected] +46 11 363452

ABSTRACT

Stefan Holmlid Department of Computer and Information Science Linköping University SE-581 83 Linköping [email protected] +46 13 285633

the landscape of other offerings. Numerous papers have reported on using competitive analysis in interaction design, but none of them provide any details on procedures of analysis [2, 3, 7, 9, 10, 16, 24, 25, 38, 44]. This paper will present two procedures for conducting analysis of precedent designs. The first procedure builds on a method presented by Brown [8] and the second procedure builds on genre analysis as it has been conceived in literature and film studies [26, 45]. The procedures are then put to test with students, and finally further developments of them are discussed.

Designers need to survey the competition and analyze precedent designs, but methods for that purpose have not been evaluated in earlier research. This paper makes a comparative evaluation between competitive analysis and genre analysis. A randomized between-group experiment was conducted where graphic design students were conducted one of the two analysis methods. There were 13 students in one group and 16 in the other. The results show that genre analysis produced more detailed descriptions of precedent designs, but its process was more difficult to understand. It is concluded that genre analysis can be integrated into competitive analysis, to make use of the strengths of both methods in the analysis of precedents.

Competitive Analysis

Brown [8] has given a practical account of how competitive analysis can be conducted in web design. He uses the example of pet-related websites that can be compared on their range of products, the design of search boxes, navigation, contact information, and shopping functions. The basic idea of competitive analysis is to line up competitors side-by-side and highlight similarities and differences on selected points of comparisons. This will disclose expectations from users who are used to other sites, and best practices in everything from interface design to offered features. Inconsistencies may indicate that some sites have innovative solutions, or that industry has not settled on any single approach. The result from competitive analysis is a strategy document that helps define the general direction for design without defining the design itself. Surveying the competition is, according to Brown, a good way to get ideas and establish a baseline.

Author Keywords

Analysis of precedents, Genre analysis, Competitive analysis, Interaction design. ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

It is well known that designers learn from existing examples and precedent designs by surveying competing designs and seeking inspiration [23, 41, 46, 47, 48]. Experienced designers can also abstract from the specific cases to see design principles, patterns and schemes that are instantiated repeatedly [4, 5, 11, 27, 31]. Competitive analysis helps designers position their design in relation to

Procedure for Competitive Analysis

A competitive analysis can be broad and give a feel for the product landscape and identify best practices, or it can be narrow and identify how competitors have solved a particular design problem. The following steps are involved in a competitive analysis according to Brown: (1) Identify and describe the purpose of the analysis; (2) set up the competitive framework; (3) gather and present data; and (4) document the conclusions.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

23

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

1. Identify and describe the purpose of the analysis. A competitive analysis basically consists of two dimensions: a number of competitors and a set of criteria for comparison. This can often be visualized using a simple table where the competitors can be compared side-by-side and similarities and differences be highlighted. The difficult part of a competitive analysis is deciding what set of competitors to compare and what criteria of comparison to use. It is good to be clear on the purpose of the analysis to facilitate these decisions.

way is to score the competitors on every criterion, much like a restaurant review based on criteria like the quality of the food, the ambience, the service, and the expense. For such comparisons, you need to give an account of what, for instance, a high score on the service means. Descriptions is the most common form of data in competitive analyses and they can be explicit on how competitors stack up against each other, without risking potentially skewed numbers. 4. Document the conclusions. Data needs to be interpreted in relation to the client and/or the project at hand. The competitive analysis should project a direction for the future design work by stipulating conclusions based on data. Referring to clear design problems and a narrow analysis make it easier to draw explicit conclusions. If the purpose instead is to give a broad analysis and an overview of the product landscape, the analysis should provide a handful of consequences for design.

2. Set up the competitive framework. The framework, which consists of the two dimensions competitors and criteria for comparison, is often set up in a table. The competitors run along the top row and the criteria along the leftmost column. The criteria for comparison may be specific or general. Specific criteria for an online shop could be “Additional Item Categories”, “Account Management Links”, and “Offline Shopping Links”. A more general criterion for comparison could be “Home Page Navigation”. Another way to set up a competitive framework is to use a two-by-two, where competitors are placed along two dimensions. The criteria for comparison tends in such frameworks to be general since the number of criteria only are two. This type of visualization can be useful for identifying gaps in the product landscape that your product can fill. A third way to do a framework is to make a series of small images of the competitors that, for example, visualize their page layouts.

Genre Analysis

We will in this paper connect competitive analysis to the notion of genres. In, for example, literature and film studies it is common to use genre analysis in comparative analyses. We believe that this also may be a worthwhile approach in competitive analysis for interaction design. A common coding scheme in genre analysis consists of three points of comparison: form, content and purpose [45]. Think of a telephone bill, which can be described according to its layout (the form) that displays the phone calls and their respective cost (the content), to inform the recipient of how much to pay, when to pay, and what exactly it is that he or she is paying for (the purpose).

Very deep or very broad analyses risk becoming difficult to read. There are two ways to manage this: one way is to structure the document by criteria for comparison. This provides a collection of smaller competitions. You may, for example, compare a handful of sites focusing five groups of criteria for comparison: home page design, interior page design, search functionality, features offered, and navigation. This kind of organization makes it easy to identify the best competitors for a specific criterion and across all criteria, but it makes it difficult to get an overall picture of the user experience for each competitor.

A genre analysis describes the common features and differences between many similar objects, and sometimes how they change over time. Genre theory suggests that recurring forms of communication (e.g. the telephone bill) and interaction (e.g. how the telephone bill usually is delivered and paid) help participants understand the purpose of the interaction. The recurring and familiar forms also create expectations on form, content, and purpose of the interaction, and on the actors involved (e.g. for the telephone bill, that the sender is actually the telephone company and that the recipient is the one responsible for paying) [12].

The other way to manage larger analyses is to structure the document by competitor. That means there is one section for each competitor which gives each and one of them a profile. Every profile describes the criteria for comparison. This makes it easy to get an overview picture of the user experience for each competitor, but it makes it harder to compare the competitors with each other.

Genres are central to human cognition, and are based on the human ability to generalize common aspects of recurring situations and objects [35, 36, 37]. For people in everyday situations, genres help to find more of something one has experienced before, and to recognize what kind of situation one is in, for instance an argumentation or a negotiation [1, 12]. It is though, not obvious that more of the same means good design. For a design to be, for example, enchanting it must also evoke a sense of wonder at the newness and unexpectedness of the design [33]. Competitive analysis and genre analysis do, however, not drive design. They only provide a benchmark or basic understanding of the design domain at hand.

3. Gather and present data. When the framework is completed the analyst fills it with data from the analysis. The purpose of the analysis is to describe similarities and differences between products, and it is therefore important to find ways of highlighting them in the presentation of data. The data can be represented in different ways, with different fidelity. Yes/No values are good for comparing if competitors have or does not have a set of features. The drawback is that more subtle differences are lost. Another

24

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

For different genres of communication or interaction, people have expectations on the purpose, on the actors involved, and on the form of the genre object. If the form changes, for instance, by someone starting to fight during a negotiation, or by using inappropriate language, the deviation from the genre of negotiations re-defines the situation. Also, if the actors are seen as inappropriate, for instance a newspaper that apparently is not produced by a newspaper agency, then that deviation from the genre norm also re-defines the situation [12]. People draw upon genre rules, to re-create new situations according to their experiences and expectations [49]. Deviations from the norm can also change the genre, by altering the rules. Thus, genre is not just about repetition, but also about changes. Changes to a genre can divide it into sub-genres, appropriate for somewhat different purposes. Genres analysis can accordingly be used to describe and categorize phenomena into genres, sub-genres, and mix-genres, and to analyze what genre is appropriate for different ends. This has for instance been done for Swedish municipality websites [15]. Genre analysis can also be used to describe changes to a genre over time, which for instance has been done on cyber genres [43], on memos [49], and on web design styles [40]. Some designs are more generic, or more central to the genre, which has been shown in an analysis of personal homepages [13], and cultural aspects have also in earlier research been used as a basis for comparison [42]. Genre analysis has also been used to understand why some designs are better than other designs, by comparing the most successful genre variations to the less successful genre variations. Such analyses have been done on academic texts [45] and on online newspapers [21, 22].

of interaction design. In particular, when discussing aesthetics of interactive products and services we have to include not only characteristics of the artefact and user experiences, but also the characteristics of the interaction that takes place between artefact and user, and between users mediated by the artefact [28]. Procedure for Genre Analysis

The following description of how to conduct a genre analysis builds on our earlier work [14, 15, 19, 21, 22, 29, 30, 6], but is in this paper more comprehensively described. The following steps are involved: (1) Map out the contents of the products, what elements the contents consist of, and what the elements consist of; (2) identify the purposes of the whole products and the purposes of the elements; (3) describe the form of the whole product and of its elements; and (4) identify shared characteristics and differences between the products. 1. Map out the contents of the products, what elements the contents consist of, and what the elements consist of. The first step is to identify the most common elements of a product in the genre. This will provide a repertoire of design elements that can be used to compose genre typical products and to pinpoint characteristics that are unique to some products. Every design element can in turn be described in terms of its elements. This hierarchical breakdown will end at some level and the atoms of the design elements are, for example, lines, textures or specific behaviors. The contents of a product also have a structure: There are different ways to label elements and different ways to organize them in different products. It may also be relevant to analyze what design elements that appear together.

On a general level, different genre analyses largely share the same coding scheme, using form, content and purpose as points of comparison. There are, however, variations. Devitt [12], for instance, describes the characteristics of genres using a coding scheme consisting of a) characteristics of the communicators, b) situation with social and personal motives, c) form and content of communication. Swales [45] uses purpose, content, positioning, and form. Although some analysts use these abstract schemes, it is common to be more detailed. For instance, more detailed points of comparison for film analysis could be setting, characters, narrative, iconography and style [26]. For computer games a detailed set of points of comparison could instead be visual style, cut scenes, interface metaphor, pace, and control schemes [50]. Analyses made in information systems often concern digital media, its form in terms of positioning on screens, its contents in terms of services and information, and the purposes that the information system can fill for different actors [20, 32].

2. Identify the purposes of the whole product and the purposes of the elements. The next step is to find what purpose the genre as a whole fulfils, if particular products has partly other purposes (and may form sub-genres), and what different design elements are used for. You may speculate about the purpose, but the analysis needs to be complemented by interviews or surveys with users and producers in order to be really trustworthy. Such studies may show how important every element is to fulfill different stakeholders’ overarching motive. The composition of form, content and purpose of a product must be in line with the context in which it will be used. This is particularly important, when it comes to different stakeholders’ and users’ motives and experiences. Motives of different actors should match the purpose of the genre in order for the genre chosen to be appropriate. To give an example, a customer in an online store may want a specific item at a low price, while the owner of the store may want to sell as many items as possible to the same customer. The purpose of the online store should meet both these motives in order to be said to have an appropriate design. Experienced qualities like orientation, satisfaction, confusion or flow in the interaction may make the motives

The points of comparison we use in our approach have been developed over a few years. We have embarked from the notions of form, content and purpose, but have realized that they need to be interpreted somewhat differently in the area

25

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

easier or more difficult to achieve. It can, for example, be difficult for a user to reach the motive ‘shop food quickly’ in a confusing store.

A fifth form aspect is the sound that a product or design element has. Sound catches people’s attention, carries emotional content and provides an ambience.

3. Describe the form of the whole product and of its elements. When discussing form elements in interaction design we need to consider movement, space, time, appearance, texture, and sound [39].

4. Identify shared characteristics and differences between the products. Comparisons between the products are made by placing the analyses side-by-side and searching for shared characteristics and differences in terms of form, content and purpose of the products as a whole and of the design elements. Questions to ask include: What characteristics are necessary for a user to recognize the product as belonging to the genre? What characteristics are necessary for a user to know what to do with a product in this genre? What are the consequences of the variations in the genre for how the products are experienced and how they can meet peoples’ motives in different situations? Are there characteristics that make any product better adapted to various motives, experiences and situations? Are there mixgenres or sub-genres that meet partly different purposes and give partly different experiences? If so, what are the characteristics that differentiate the mix-genres or subgenres?

The first step of describing the form is to identify the positions of the design elements. The first form aspect is therefore space. A virtual space can also be described as a network of nodes where the user navigates using links. The concept of users’ space for action can be used to indicate actions that are close to hand and actions that are further away. Different kinds of spaces (physical, screen based, virtual) can also be mixed and combined. The spatial aspects of the form can be said to form a layout. When the layout has been described, the following step is to describe the second form aspect: movement. All interaction involves movement. It is about the movement of people and products, and how products and their design elements are transformed. Some elements are static, while other can change dynamically independently of the user’s actions. They can be passive, reactive interactive or proactive [17, 18]. Proactive elements perform actions based on predictions about the future. When users interact with an element its state also changes. An example is how the state of a web site changes for each step from page to page that the user make. This movement forms a history of where the user has been and where he or she is headed. The movement can be initiated by the user, the system, or be continuous. Continuous movement requires steering (as for example in a snowboard game where you ride down the slope). Movement is also effective for catching people’s attention.

Research Problem

Our practical experience of using genre analysis in interaction design research and education has indicated to us that it facilitates an awareness of the details in a design. The question for this study is whether this intuition is correct. Competitive analysis in Brown’s version may be equally effective. The aim is also to investigate what benefits and difficulties design students experience in competitive analysis and genre analysis. METHOD

A class of second-year graphic design and communication students (average age 23 years, 32% male and 62% female) were randomized into two groups to investigate if there is any difference in the level of detail between design descriptions produced using competitive analysis and descriptions produced using genre analysis. One group was assigned to do a genre analysis and the other group was assigned to do a competitive analysis. They had recently started on their first course in interaction design, after a year of graphic design studies. They were given written instructions in accordance with the earlier described procedures for the two methods. The assignment was to analyze and compare the start page, and the navigation to a specific movie using four web-TV applications (SVT Play, TV4 Play, TV3 Play and Reuters Video). They were told that they would later use their analysis as a pre-study for the design of a mobile web-TV application. They had one week to complete the assignment.

A third form aspect is the appearance of the product and its design elements. It gives clues to the behavior of the elements and how you interact with them. The appearance structures the interaction by presenting opportunities for interaction (i.e. affordances, [34]): If you look upon a door handle you may perceive that you can pull it or push it depending on your earlier interaction with the world. The presented structure may be static insofar that the same opportunities for interaction always are presented in the same way, or the opportunities for interaction may change continuously. A door handle cannot change its appearance but digital products can change their appearance completely from one moment to the next. Appearance also communicates emotional content. A fourth form aspect is the texture of the product or design element. It includes how a surface feels and looks. Just like appearance it communicates opportunities for interaction and emotional content. Other physical properties like weight also affect experienced qualities [31].

The alternative hypothesis was that there would be a difference in level of detail between the genre analysis and the competitive analysis. The null hypothesis was that there would be no difference. One could argue that a onedirectional hypothesis should be used given the literature

26

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

review and our previous work, but this would increase the risk of a type I error (rejecting a true null hypothesis).

Genre Analysis Benefit

The level of detail was operationalized as the number of observed features, where a feature was defined as a design element, or a description of a design element, alternatively an experience of a design element or the design in its entirety.

Gives interesting details Provides generally applicable criteria

13 completed genre analyses (3 male and 10 female) and 16 competitive analyses (5 male and 11 female) were handed in. Two uncompleted genre analyses were also handed in, but were not included in this study. The Mann-Whitney (Utest), which is a non-parametric test for independent samples, was chosen since the assumptions underlying the t-test (normality and homogeneity of variance) could not be met.

Number of features

Rank 3

102

1

160

7

125

2

168

9

146

4

194

11

147

5

256

16

159

6

271

17

166

8

324

19

177

10

325

20

202

12

370

25

227

13

420

26

229

14

430

27.5

239

15

430

27.5

283

18

576

28

327

21

338

22

351

23

360

24

R1 =

236

R2 =

198

Median for Group 1 =

324

Median for Group 2 =

214.5

Median for both groups combined =

Difficulty

Easy to understand procedure

Difficult to set up criteria

Gives broad picture

Uncertain what you miss when you use the criteria you have set up

their experiences and reflections was held. That discussion was documented by taking notes that subsequently were sorted into categories based on benefits and drawbacks. RESULTS

The median for the genre analyses was 324 features, and for the competitive analyses it was 214,5 features. The collected data is presented in Table 1. The study showed that there was a significant difference, to the advantage of the genre analysis, between the two methods (MannWhitney U(13, 16) = 63, p < 0.05, two tailed).

Rank

138

Difficult to know where to start and what to look for

Benefit

Table 1. Results from group discussion on benefits and difficulties of methods for design descriptions.

Competitive Analysis (Group 2, n2 = 16) Number of features

Unclear procedure

Difficult to know when to stop

The participants were, after the assignment, grouped in pairs or groups of three, to discuss what they had done and seen in their analyses. They were also instructed to discuss benefits and drawbacks of the methods, and what was easy and difficult in using them. Finally, a joint discussion of Genre Analysis (Group 1, n1 = 13)

Difficulty

Competitive Analysis

The group discussion with the participants after the assignment provided insights to strengths and weaknesses of the methods. These results are summarized in Table 2. The participants reported that they were unclear about how they would use the information in their future design of the mobile web-TV application. They thought it would have been easier to focus their description if they had more insight into that project. They were also asked to reflect on what would have happened if they had not been given any procedure and only the assignment to analyze precedents and competitors to get inspiration for their design. They answered that it would not had been as thorough and that there would had been a risk that they would only have clicked and played around. One participant thought that would have been better since he/she then only would collect all the good things and annoyances from different sites to know what to build on and what to avoid. Looking at what the participants actually wrote about in their analyses we can see that the genre analyses were in line with the framework given to them to follow. This means that they reported the elements that the web sites contained, and described their form and purposes. The competitive analyses varied more in the their focus, and could for example highlight more technical aspects and did

269.25

Table 2. Data on the level of detail in design descriptions.

27

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

more often line up features that were either present or not present among the competitors. They were also less exhaustive on form aspects.

4.

This procedure needs to be iteratively developed and resulting descriptions of precedent designs as well as impacts on the design process need to be evaluated. This was a study with second year graphic design students, and future research also needs to test the method with experienced design students and professional designers.

DISCUSSION

The results show that it is likely that genre analysis provides more detailed descriptions than competitive analysis using Brown’s [8] procedure. There were two methodological issues in the study. Firstly, it was difficult to decide what should count as a feature. Secondly, there were repetitions of the same feature in every description. These potential sources for error are, however, assumed to be evenly distributed in the genre analyses and the competitive analyses.

Conclusion

Interaction designers need to conduct analysis of precedent designs, but methods for that purpose have not been evaluated in earlier research. The results of this study indicate that genre analysis produces more detailed descriptions of precedent designs than competitive analysis does, but it is also more difficult to understand. Genre analysis can however be integrated with competitive analysis to make use of the strengths of both methods.

A plausible explanation to the higher level of detail in the genre analysis is that it offers points for comparison in an observation protocol (i.e. form, content and purpose), and that the analysis is hierarchically conducted. Such a framework is completely lacking in the competitive analyses. This means that the product in its entirety, its elements, and the elements of the elements, are analyzed in the genre analysis. This also means that the analysis may continue ad infinitum, which must be considered a risk.

ACKNOWLEDGMENTS

We wish thank the students who participated in this study. The research has been sponsored by Östersjöstiftelsen, and Santa Anna IT Research Institute.

The advantage of the competitive analysis in Browns version is that the instructions are easy to understand, and the importance of the purpose as a delimiter of the analysis is stressed. It can, in a procedure for analysis of precedents, be beneficial to merge the two methods.

REFERENCES

1. Agre, P. E. 1998. Designing genres for new media: Social, economic and political contexts. In CyberSociety 2.0: Revisiting Computer-Mediated Communication and Community, S. G. Jones, Ed. Sage Publications, Thousand Oaks, CA, 69-99.

Procedure for Analysis of Precedents

2. Alben, L. and Rigsby, M. 2000. Envisioning the equarium: strategic design planning for the Monterey Bay aquarium's web site. In Proceedings of the 3rd Conference on Designing interactive Systems: Processes, Practices, Methods, and Techniques (New York City, New York, United States, August 17 - 19, 2000). D. Boyarski and W. A. Kellogg, Eds. DIS '00. ACM, New York, NY, 452-454. DOI= http://doi.acm.org/10.1145/347642.347813

The genre analysis can in fact be integrated with a competitive analysis as in the procedure described below. This will not make the genre analysis any simpler to perform, but it will provide a context and purpose, which can make it easier to delimit the genre analysis. 1.

Identify and describe the purpose of the analysis (step 1 of competitive analysis). Decide also what usage task to analyze to limit your analysis (e.g. ’go from start page to specific movie clip’).

2.

Set up the competitive framework by placing the competing products side by side, and analyzing their form, content, and purpose:

3.

Document the conclusions (step 4 of competitive analysis).

3. Armitage, J. 2003. "And another thing..the current site is in German": the final project in an international digital business consultancy. In Proceedings of the 2003 Conference on Designing For User Experiences (San Francisco, California, June 06 - 07, 2003). DUX '03. ACM, New York, NY, 1-15. DOI= http://doi.acm.org/10.1145/997078.997107

a.

Map out the contents of the products, what elements the contents consist of, and what the elements consist of (step 1 of genre analysis).

b.

Identify the purposes of the product and the purposes of its elements (step 2 of genre analysis).

c.

Describe the form of the whole product and of its elements (step 3 of genre analysis).

d.

Identify shared characteristics and differences between the products (step 4 of genre analysis).

5. Ball, L, Ormerod, T., and Morley, N. 2004. Spontaneous analogising in engineering design: A comparative analysis of experts and novices. Design Studies 25, 5, 495-508.

Gather and present data (step 3 of competitive analysis).

6. Blomkvist, J., and Holmlid, S. 2009. Examples in Service Design: Tools for Communication and Setting

4. Arvola, M. 2006. Interaction Design Patterns for Computers in Sociable Use. International Journal of Computer Applications in Technology, 25 2/3, 128-139.

28

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

for Stories. In Proceedings of the First Nordic Conference on Service Design and Innovation (Oslo, Norway, November 24 - 26, 2009). ServDes ’09. Arkitektur- och designhøgskolen, Oslo, Norway.

16.Forrest, M., Forlizzi, J., and Zimmerman, J. 2008. Driving the family: empowering the family technology lead. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 2913-2918. DOI= http://doi.acm.org/10.1145/1358628.1358783

7. Brinck, T., Ha, S. S., Pritula, N., Lock, K., Speredelozzi, A., and Monan, M. 2003. Making an iMpact: redesigning a business school Web site around performance metrics. In Proceedings of the 2003 Conference on Designing For User Experiences (San Francisco, California, June 06 - 07, 2003). DUX '03. ACM, New York, NY, 1-15. DOI= http://doi.acm.org/10.1145/997078.997084

17.Holmlid, S. 2000. Adapting Users: Towards a Theory of Use Quality. Doctoral Thesis. Linköping Studies in Science and Technology, Dissertation No. 765. Linköpings universitet, Linköping, Sweden. 18.Holmlid, S. 2002. Selecting the abstract: When you get what you want. In Proceedings of the Ericsson Conference on Usability Engineering (Copenhagen, Denmark, 2002). ECUE ‘02.

8. Brown, D. M. 2007. Communicating Design: Developing Web Site Documentation for Design and Planning. New Riders, Berkeley, CA.

19.Holmlid, S., Arvola, M., and Ampler, F. 2000. Genres and Design Considerations of iTV Cases. In Proceedings of the 1st Nordic Conference on HumanComputer Interaction (Stockholm, Sweden, October 23 - 25, 2000). NordiCHI 2000. STIMDI, Stockholm, Sweden, CD-ROM.

9. Burke, P., Nguyen, S., Sun, P., Evenson, S., Kim, J., Wright, L., Ahmed, N., and Patel, A. 2005. Writing the BoK: designing for the networked learning environment of college students. In Proceedings of the 2005 Conference on Designing For User Experience (San Francisco, California, November 03 - 05, 2005). Designing For User Experiences, vol. 135. AIGA: American Institute of Graphic Arts, New York, NY, 32.

20.Holmlid, S., and Lantz, A. 2006. Developing e-services in a government authority: Different views on design in procurement and system development. Position paper at the NordiCHI 2006 Workshop on User Involvement and Representation in e-Government Projects (Oslo, Norway, October 14 - 18, 2006).

10.Chand, A. and Dey, A. K. 2006. Jadoo: a paper user interface for users unfamiliar with computers. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM, New York, NY, 1625-1630. DOI= http://doi.acm.org/10.1145/1125451.1125747

21.Ihlström, C. and Lundberg, J. 2003. The Online News Genre through the User Perspective. In Proceedings of the 36th Annual Hawaii international Conference on System Sciences (Hicss'03) - Track 4 - Volume 4 (January 06 - 09, 2003). HICSS. IEEE Computer Society, Washington, DC, 103.1.

11.Cross, N. 2004. Expertise in design: An overview. Design Studies 25, 5, 427-44. 12.Devitt, A. J. 1993. Generalizing about genre: New conceptions of an old concept. College Composition and Communication 44, 4 (Dec, 1993), 573-586.

22.Ihlström, C., and Lundberg, J. 2004. A genre perspective on online newspaper front page design. Journal of Web Engineering 3, 1, 50-74.

13.Dillon, A. and Gushrowski, B. 2000. Genres and the Web: Is the personal home page the first uniquely digital genre?. Journal of the American Society for Information Science, 51, 2, 202-205.

23.Keller, A. I., Pasman, G. J., and Stappers, P. J. 2006. Collections designers keep: Collecting visual material for inspiration and reference. CoDesign 2, 1, 17-33.

14.Edvardsson, A., and Lundin, J. 2003. Fotboll med Internet som spelplan: en metod för stt skapa designmönster för ett digitalt medium [Football with Internet as Game Field: A Method for Creating Design Patterns for a Digital Medium]. Masters Thesis. Department of Computer and Information Science, Linköpings universitet, Linköping, Sweden.

24.Kim, E., Koh, B., Ng, J., and Su, R. 2006. myPyramid: increasing nutritional awareness. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM, New York, NY, 1843-1848. DOI= http://doi.acm.org/10.1145/1125451.1125800 25.Kim, J. and Zimmerman, J. 2006. Cherish: smart digital photo frames for sharing social narratives at home. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). CHI '06. ACM, New York, NY, 953958. DOI= http://doi.acm.org/10.1145/1125451.1125635

15.Eliason, E. and Lundberg, J. 2006. The appropriateness of Swedish municipality web site designs. In Proceedings of the 4th Nordic Conference on HumanComputer interaction: Changing Roles (Oslo, Norway, October 14 - 18, 2006). A. Mørch, K. Morgan, T. Bratteteig, G. Ghosh, and D. Svanaes, Eds. NordiCHI '06, vol. 189. ACM, New York, NY, 48-57. DOI= http://doi.acm.org/10.1145/1182475.1182481

26.Lacey, N. 2000. Narrative and Genre: Key concepts in media studies. St. Martin's Press.

29

Full Papers

27.Lawson, B. 1980. How designers Architectural Press, London, UK.

Proceedings: NordiCHI 2010, October 16–20, 2010

think.

New York, NY, 898-903. http://doi.acm.org/10.1145/506443.506649

The

DOI=

39.Saffer, D. 2007. Designing for Interaction: Creating Smart Applications and Clever Devices. New Riders, Berkeley, CA.

28.Lim, Y., Stolterman, E., Jung, H., and Donaldson, J. 2007. Interaction gestalt and the design of aesthetic interactions. In Proceedings of the 2007 Conference on Designing Pleasurable Products and interfaces (Helsinki, Finland, August 22 - 25, 2007). DPPI '07. ACM, New York, NY, 239-254. DOI= http://doi.acm.org/10.1145/1314161.1314183

40.Schmid-Isler, S. 2000. The Language of Digital Genres. A Semiotic Investigation of Style and Iconology on the World Wide Web. In Proceedings of the 33rd Hawaii international Conference on System Sciences-Volume 3 - Volume 3 (January 04 - 07, 2000). HICSS. IEEE Computer Society, Washington, DC, 3012.

29.Lundberg, J. 2004. Shaping online news: Genre perspectives on interaction design. Linköping Studies in Science and Technology Dissertation No. 918. Linköping University, Linköping, Sweden.

41.Schön, D. A. 1987. Educating the Reflective Practitioner. Jossey-Bass Publishers, San Fransisco, CA.

30.Lundberg, J., Arvola, M., and Holmlid, S. 2003. Genres, Use Qualities and Interactive Artifacts. In Proceedings Volume 2 of the Conference HCI 2003: Designing for Society (Bath, UK, September 8 - 12, 2003). HCI 2003. Research Press International on behalf of British HCI Group, Bristol, UK, 165-166.

42.Segelström, F. and Holmlid, S. 2009. Online services and cultural diversity: Applying Hofstede’s framework to predict potential issues. In Proceedings The 11th International Research Symposium on Service Excellence in Management (QUIS) (Wolfsburg, Germany, June 11 - 14, 2009). Print Services, Volkswagen AG, Wolfsburg, 604-612.

31.Löwgren, J. 2007. Inspirational patterns for embodied interaction. Journal of Knowledge, Technology & Policy 20, 3, 165-177.

43.Shepherd, M. and Watters, C. 1998. The Evolution of Cybergenres. In Proceedings of the Thirty-First Annual Hawaii international Conference on System SciencesVolume 2 - Volume 2 (January 06 - 09, 1998). HICSS. IEEE Computer Society, Washington, DC, 97. DOI= http://dx.doi.org/10.1109/HICSS.1998.651688

32.Marriott, K., Moulder, P., and Hurst, N. 2007. Automatic float placement in multi-column documents. In Proceedings of the 2007 ACM Symposium on Document Engineering (Winnipeg, Manitoba, Canada, August 28 - 31, 2007). DocEng '07. ACM, New York, NY, 125-134. DOI= http://doi.acm.org/10.1145/1284420.1284455

44.Sood, S., Gilligan, R., Chandler, C., and Slack, S. 2004. Disneyworld.com redesign. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 869-884. DOI= http://doi.acm.org/10.1145/985921.985946

33.McCarthy, J., Wright, P., Wallace, J., and Dearden, A. 2006. The experience of enchantment in human– computer interaction. Personal Ubiquitous Comput. 10, 6 (Sep. 2006), 369-378. DOI= http://dx.doi.org/10.1007/s00779-005-0055-2

45.Swales, J., M. 1990. Genre Analysis: English in academic and research settings. Cambridge University Press, Cambridge, UK.

34.Norman, D. A. 1999. Affordance, conventions, and design. interactions 6, 3 (May. 1999), 38-43. DOI= http://doi.acm.org/10.1145/301153.301168

46.Sökjer, P. 2007. Interaction designers' use of their repertoire in meetings with clients. In Proceedings of the 21st British HCI Group Annual Conference on HCI 2008: People and Computers Xxi: HCI...But Not As We Know It - Volume 2 (University of Lancaster, United Kingdom, September 03 - 07, 2007). British Computer Society Conference on Human-Computer Interaction. British Computer Society, Swindon, UK, 257-258.

35.Paltridge, B. 1995. Working with genre: A pragmatic perspective. Journal of Pragmatics 24, 4, 393-406. 36.Rosch, E. 1973. On the internal structure of perceptual and semantic categories. In Cognitive Development and the Acquisition of Language, T. E. Moore, Ed. Academic Press, New York, NY, 111-144. 37.Rosch, E. 1978. Principles of categorization. In Cognition and Categorization, E. Rosch and B. Lloyd, Eds. Erlbaum, Hillsdale, NJ, 27-48.

47.Sökjer, P., Holmlid, S., Tholander, J., and Lantz, A. 2007. The dynamics of objects in client-designer communication. In Proceedings of The Virtual ‘07, (Almåsa, Sweden, September 20 - 22, 2007). M3, Södertörn University, Haninge, Sweden.

38.Rosenbaum, S., Wilson, C. E., Jokela, T., Rohn, J. A., Smith, T. B., and Vredenburg, K. 2002. Usability in Practice: user experience lifecycle - evolution and revolution. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI '02. ACM,

48.Winograd, T., Bennett, J., and Young, L. D., Eds. 1996. Bringing Design to Software. Addison-Wesley, Reading, MA.

30

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 773-774. DOI= http://doi.acm.org/10.1145/985921.985929

49.Yates, J., and Orlikowski, W. J. 1992. Genres of organizational communication: A structurational approach to studying communication and media, The Academy of Management Review 17, 2 (Apr. 1992), 299-326. 50.Ye, Z. 2004. Genres as a tool for understanding and analyzing user experience in games. In CHI '04

31

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Exploring Distance Encodings with a Tactile Display to Convey Turn by Turn Information in Automobiles Amna Asif1 , Wilko Heuten2 , Susanne Boll1 University of Oldenburg Escherweg 2 26121 Oldenburg, Germany {amna.asif—susanne.boll}@informatik.uni oldenburg.de 1

OFFIS Escherweg 2 26121 Oldenburg, Germany [email protected] 2

systems to support the driver. A car navigation system is one of the promoted and preferred information systems for cars. Visual and auditory displays in the present car navigation systems are successful to provide spatial information. However, the car navigation system increases a demand of the driver’s visual and acoustic attention while driving and are subject to distract the driver. The visual attention of the driver is essential in the primary task of driving, which is for example steering the car, using brakes and controls as well as observing the traffic. Pauzie and Marin [10] investigated that aging drivers spent 6.3% and young drivers spent 3.5% of their driving time glancing at the screen. Auditory displays on the other hand are challenging in a noisy environment. The driver performs a multiple number of primary and secondary tasks on visual and auditory displays that can impose mental workload [4] and distraction [12], which is harmful for safety of driving.

ABSTRACT

Visual and auditory displays successfully complement each other presenting information in car navigation systems. However, they distract the visual and auditory attention of the driver, which is needed in many primary driving tasks, such as maneuvering the car or observing the traffic. Tactile interfaces can form an alternative way to display spatial information. The way of how exactly information should be presented in a vibro-tactile way is explored rarely. In this paper we investigate three different designs of vibro-tactile stimulation to convey distance information to the driver using a tactile waist belt. We explore the tactile parameters intensity, rhythm, duration, and body location for encoding the distance information. We conduct a comparative experiment on a real navigation scenario in an urban environment to evaluate our designs. In our study we discovered that rhythm and duration are suitable parameters to generate tactile stimulation for encoding distance information. In this way the driver perceives countable vibro-tactile pulses, which indicate the distance in turn by turn instructions. The approach is found be simple way of encoding complex navigational information.

A tactile display can be used as an alternative interface for the car navigation system to reduce the mental workload and the distraction, following the Multiple Resource Theory (MRT) [21]. Two important parameters of the turn by turn navigation are direction and distance [2, 19]. In previous studies car simulators have been used to evaluate the vibro-tactile distance encoding. However, the investigation of the precise approach of vibro-tactile distance presentation in a real environment is missing. We investigated a number of the encodings based on rhythm, intensity and duration to discover an appropriate approach for the vibro-tactile distance encoding. We evaluated three vibro-tactile distance encodings in an experiment: (1) Only rhythm based distance encoding (2) Rhythm and intensity based distance encoding (3) Rhythm and duration based distance encoding. In our study, we focused on comparing different methods of conveying the distance with the vibro-tactile feedback in real urban environments. The study shows success of duration in combination with rhythm based distance encoding in the car navigation systems.

ACM Classification Keywords

H.5.2 User Interfaces: Haptic I/O; 1.3.6 Methodology and Techniques: Interaction techniques General Terms

Human Factors, Experimentation Author Keywords

Car navigation system, Tactile interface INTRODUCTION AND MOTIVATION

The activity of driving is multi-tasking and complex [12]. Vehicles are equipped with a number of distinct information

In the remainder of this paper, Section 2 introduces the reader with the state-of-the-art tactile interfaces and related approaches of the vibro-tactile distance and direction encoding in automobiles. The design space and the vibro-tactile distance encodings are described in Section 3. The experiment details are presented in Section 4. In Section 5 we report the find-

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16-20, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

32

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ings. In Section 6 we discuss the answers to our research question and further findings. We close the paper with a conclusion in Section 7.

vibrators in an 8 x 8 matrix [6] is used to evaluate the ability of drivers to distinguish up to eight different directions. The distance information is presented by van Erp and van Veen [17] with vibrocon (vibro-tactile icon) for three distance steps of 250 m, 150 m, and 50 m. The information is presented to the users by activating the four tactile actuators either under the left or right leg of the driver. The results of an evaluation in the simulator show that the tactile interface helps to reduce the visual burden of the drivers. We take the opportunity to explore the tactile encoding for spatial information in the real driving environment. van Erp and van Veen encoded a distance information in the form of rhythm patterns though it is possible to explore more methods of distance encoding. The proof-of-concept study shows that using vibro-tactile displays in the car is useful for presenting distance information.

RELATED WORK

Previous research has shown that the tactile displays were effectively used to provide navigation aids to the pedestrians and blind users. McDaniel et al. [9] present a scheme for using tactile rhythms to convey intimate, personal, social interpersonal distances to blind users. ActiveBelt [16] consists of a number of vibration components integrated into a belt or a vest. The vibration components are equally distributed around the person’s body and activated for showing him a direction. Tactile Wayfinder [5] was evaluated for the task of pedestrian route guidance. The tactile Wayfinder supported the pedestrian in orientation, choice of route, keeping through track, and recognition of destination. PeTaNa [20] a torso based wearable system was evaluated to present direction and distance information to soldiers in the field. The direction information was presented on the respective location of torso of the soldier. Distance was coded with the temporal rhythm on the vibration. van Erp et al. [19] investigated different distance encoding schemes with pedestrian participants. The vibration rhythm was used to code the distance and the body location was used to code the direction. An additional experiment investigated usefulness of tactile display with a helicopter and a fast boat. Straub et al. [14] used a vibro-tactile waist belt to encode distance for the pedestrian. They used four distance encodings based on the parameters of intensity, frequency, position (which tactor), and patterns. Pielot et al. [11] presented a position and spatial distance of several people with the help a tactile torso based display in fast paced 3D game. The results showed that the location of the team members can be effectively processed with the help of the tactile display in high cognitive demand. The team showed a better team play and higher situation awareness. The findings of the previous studies encourage the fact that it is possible to encode distance with tactile displays. The effectiveness of the tactile display for presenting direction and distance information to blind and pedestrians motivate the idea to discover the approach to code distance in vibrotactile signals in car navigation systems.

In the previous research, torso based tactile displays are successfully employed to present navigation information to the blind users and pedestrians but parameters still require exploration in the automobiles. In the previous studies [9, 11, 14, 19], a number of vibro-tactile encoding schemes are compared to display distance information to the team players, pedestrian and blind users. Besides previous proof-ofconcept studies [17] the tactile parameters are still required to explore to discover an appropriate approach of encoding distance with vibro-tactile signals in the car navigation system. In this paper a similar comparative approach is used to explore the tactile parameters to encode distance information in the car navigation system. EXPLORING DISTANCE ENCODING WITH VIBRO-TACTILE SIGNALS

Car navigation systems provide route guidance to the driver towards a destination [15]. In our study, we used a tactile waist belt to provide turn by turn directions and distance information to the driver. Figure 1 presents example cues that Distance Turn-now

Furthermore, the tactile interface is effectively used in Advanced Driver Assistance Systems on commercial scale e.g. in Citroen1 and Audi2 in the seat and steering wheel, respectively. Similarly, the previous studies investigated the feasibility of the tactile channel in vehicle information systems e.g. in car navigation systems. A vibro-tactile seat is developed with the characteristics of a matrix with 6 x 6 elements with interaction area of a 430 by 430 mm with dynamically modifying haptic feedback based on the driving sitting posture [13]. A prototype steering wheel [7] is integrated with six tactile actuators. The steering wheel is used to display the direction in a car simulator. The best driving performance is attained by combining tactile display with a visual display or an auditory display. A seat fitted with 24 1 http://www.citroen.com.hk/tech/sec

Direction Left

Near

Left

Far

Left

Very-far

Left

Figure 1. The direction and distance instruction presented by the tactile feedback, as the car approaches the crossing.

04.htm

2 http://www.audiworld.com/news/05/naias/aaqc/content5.shtml

are conveyed to the driver through the tactile belt while ap-

33

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

proaching the actual turn. In order to convey the direction left, the tactile belt indicates left by activating vibrators on the left side. The vibro-tactile signals provide information about the four categories of distance as proposed by Fukuda [3]: (1) Very-far, (2) Far, (3) Near, and (4) Turn-now. The driver make turning decisions by following the vibro-tactile instructions concerning the direction and distance while approaching the crossing. This section describes our vibrotactile distance encoding in the car navigation system and the practical approach that we adopted to explore the design space.

the study helped us in the design of vibro-tactile distance encodings for a further comparative evaluation. In addition, the pilot study was conducted to provide the proof-of-concept of displaying distance information with only tactile feedback on the real road. Participants, apparatus and Procedure

One female participated in the pilot study. She had 25 years of driving experience. The vibro-tactile signals are used to present distance information to the participant. The participant tested six options of vibro-tactile distance encoding in seven design solutions. We adapted the approach of thinking aloud to collect the participant’s comments and observations. We made videos for data collection. We measured the vibro-tactile signal perception and distance categories.

Vibro-tactile Distance Encoding

There are many potential ways to use vibration for encoding distance. According to [1, 18] the tactile parameters that can be modified to encode information in general are frequency, amplitude, waveform, duration, rhythm and body location. The frequency is the number of cycles per second [8]. Vibration stimuli will be detected when the amplitude exceeds a threshold which is dependent on several parameters including frequency [18]. The parameters of frequency and amplitude are not well suited to encode information [20]. The change in voltages results in change in frequency and amplitude simultaneously. The change is almost linear given in Precision Microdrive3 . So we can treat the frequency and amplitude as one parameter of intensity by following the approach in [11]. Furthermore, we cannot modify the waveform of the signal as its manipulation would require specific hardware [18]. Further parameters for encoding distances are rhythm and duration. The rhythm is created by grouping a number of pulses in a temporal pattern. Varying the duration of the vibro-tactile signal means to vary the length of a single pulse.

The distance signals were controlled by an experimenter on the backseat. The participant was trained on all the designs before going to the driving sessions. The participant drove in multiple sessions. The participant drove at least two times for each design. The participant was speaking whatever she thinks about the vibro-tactile signal perception and categories of distance. The videos were analyzed post-studies to collect the results. Results and discussions

In summary, we found that the following designs are acceptable according to the participant’s comments: rhythm based distance encoding, intensity and rhythm based distance encoding, duration and rhythm based distance encoding. The respective category of distance relative to the quantitative range of distance is given in Table 1. In the following, we will discuss our observations and the driver’s comments of the pilot study in detail.

We can design different vibration patterns by modifying the intensity, rhythm, duration, and the body location. In this study the body location of vibro-tactile cues is already used for presenting direction as the vibro-tactile waist belt uses the left and right vibrators to encode the direction of the upcoming turn. Thus, it cannot be used further for distance encoding.

Categories of distance: In distance encoding, the signal of turn-now is required to be comparatively intense and easy to identify to make sure that the driver has to take turn at that moment. According to observations, the vibration signal for turn-now will be few meters (i.e. 10m) before the next crossing while driving on the street.

In summary, encoding distance by vibration can be conducted by altering the parameters intensity, rhythm, and duration. These parameters can be altered individually or in combination. Following options are available for the adjustment: intensity only, rhythm only, duration of the signal, intensity and rhythm, intensity and duration, and rhythm and duration. In order to validate, whether all of these options were easy to perceive, to distinguish and to interpret by the driver, we investigated them in a first pilot study.

Vibration patterns in automobiles: The driver commented that ”I get strong sensations with less number of pulses, and while increasing the number of pulses feeling-wise it is less intense”. The driver gets smooth sensations with a higher number of pulses in a small time interval (e.g. minimum six pulses per second). The vibration pattern sensed intense and distinct if less pulses take place in a longer time interval (e.g. 2 pulses per second). The participant commented that ”The vibration is not easy to feel and I am undecided between different levels of intensity” for only intensity-based distance encoding. We observed that the participant annoyed by the intensity only vibro-tactile signal. Furthermore, it was difficult for the participant to understand the distance information. We discovered that rhythm is an important parameter for vibro-tactile information encoding in the car navigation system. We discovered that it is possible to get appropriate rhythm by changing a number of pulses while keeping the constant level of the intensity. Consequently, it is difficult to

Pilot study

The overall aim of the pilot study was to explore the usefulness of the different ways of encoding distance information with vibro-tactile signals. We conducted the pilot study to select the possible tactile parameters for encoding the distance information in the real driving scenarios. The results of 3 https://secure.precisionmicrodrives.com/product info.php? products id=92

34

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Category of Distance Very Far Far Near Turn Now

Rhythm based encoding Rhythm & intensity based encoding

Distance (meters) 200-150 100-80 50-30 10

Rhythm & duration based encoding

Very-far

Far

Table 1. The quantitative categories of four distances.

select among only rhythm, intensity and rhythm, and duration and rhythm based vibro-tactile distance encoding. We concluded that all designs in combination with rhythm are successful for vibro-tactile distance encoding.

Near

Turn-now

Although we conducted this study with only one participant, the observations and the driver’s comments helped us a lot to improve vibro-tactile distance encodings for further comparative experiment. The pilot study provided a proof-ofconcept that the tactile feedback is successful to present the distance and direction information to the driver on the real road without support of visual and auditory modalities.

Figure 2. Rhythm based, rhythm & intensity based, and rhythm & duration based distance encoding consists of different patterns of pulses with different intensity levels.

In the following duration, intensity and rhythm are used to encode distance. The driver recognizes the information in the form of countable pulses, the distinct rhythm patterns. Distance encoding in vibro-tactile cues

We discovered three promising vibro-tactile distance encodings based upon our pilot study and observations. The designs are composed of vibration duration, intensity, and rhythm. In the three designs we encode the categories of distances (see Table 1). The first design utilized the parameter rhythm, the second one utilized the parameters rhythm and intensity, and the third design made use of rhythm and duration. In the first two designs, the driver perceives a very smooth vibration at the distance of very-far and a very intense vibration at the distance of turn-now. The third design – rythm and duration encoding – conveys countable pulses to the driver. The three designs are explained with the help of graphical representation (see Figures 2), where the vertical axis presents the level of intensity and the horizontal axis presents time in milliseconds (ms). Furthermore, the four categories of distance are presented from the top to the bottom in the diagrams. The technical characteristics of the three designs are outlined in the following: Rhythm based distance encoding:

encoded in pulses per second with variable intensity. A pulse is composed of its activation and sleeping states. If we look from top to bottom in the diagram an event of very-far is composed of 25 pulses with 70% of intensity level. On the second place the event of far is composed of 10 pulses with 70% of intensity. On the third place the event of near is composed of 6 pulses with 80% of intensity. And in the bottom the turn-now event consists of 2 pulses with maximum intensity. Rhythm and duration based distance encoding:

The graphical visualization of the duration and rhythm based distance encoding is presented in Figure 2. In this design the pulses are more intense and less in a number. The distance is presented with a number of pulses in the given time along with its intensity level. A pulse is shown with its activation and deactivation states. The first event is composed of 4 pulses of the length of 2.5 seconds. The inter-stimulus interval between the pulses is 313 ms. The second event consists of 3 pulses of the length of 2 seconds. The inter-stimulus interval between the pulses is 333ms. The third event is encoded as 2 pulses of length of 1.5 seconds. The interstimulus interval between the pulses is 375ms. The fourth event is encoded as one pulse in a second with 500 ms interstimulus interval. On the basis of the pilot study, we proposed the three different methods of distance encoding based on: only rhythm, intensity and rhythm and duration and rhythm. In the following section, we present the design of the experiment in which we compared the methods of distance encoding.

The graphical visualization of only rhythm based distance encoding is shown in Figure 2. The encoding of a distance signal is presented with its intensity and length. The event is composed of pulses per second. A pulse is composed of its activation and deactivation states. The intensity is same for all the four events. If we observe from the top to the bottom the design consists of 25 pulses for the very-far. The far, near and turn-now consist of 10 pulses, 6 pulses, and 2 pulses respectively.

EVALUATION

The goal of the study is to compare the different approaches to encode distance information with vibro-tactile signals. We aim to investigate the most simple vibro-tactile distance encoding for the driver that leads to successful task completion. The comparative evaluation investigates the specified questions:

Rhythm and intensity based distance encoding:

The graphical visualization of the intensity and rhythm based distance encoding is presented in Figure 2. The distance is

35

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Q1: Will the participants perceive turn by turn distance information without support of visual and auditory modalities in the car?

Apparatus

We used a fully automatic Volkswagen Touran car in all sessions. The tactile belt is used to provide navigational aid in all three driving sessions (see Figure 3). The tactile belt is integrated to be worn around the waist of the driver. The fixed circumference of 90 cm is made up of a fabric in the shape of a flexible tube. The tactors are sewed into the fabric tube. These tactors are vibrating components of Samsung SGH A400, size of 11 mm. The six tactors are distributed equally around the waist of the participant. We have used the questionnaires, video recording, and screen captures for the collection of data.

Q2: Which one of the encodings – rhythm, rhythm and intensity, rhythm and duration – is the most usable encoding? Q3: Which distance encoding helps the driver to make the least number of information perception / interpretation errors? Q4: What are common errors that participants make to interpret distance information by using the tactile display? Experiment design

Our experiment consists of one independent variable. The vibro-tactile signal presents the direction and distance information to the participants. We compared the three vibrotactile distance encodings while driving in real urban environments. The approach of the experiment was within subject, with 3 different routes and 3 different vibro-tactile encodings of distance. In the within subject approach the different experimental conditions are the three vibro-tactile distance encodings and the same participants have been used in all the conditions. Dependent measures

Figure 3. The driver wearing the tactile belt.

Considering the questions for the evaluation Q1 to Q4, we selected the measures of usability of the designs, information perception errors and cognitive workload as dependent measures.

Questionnaire

Usability of the designs: We measure usability in the terms of information perception, learnability, length of vibro-tactile signal, ease of use, and user’s judgment rating on the design. We asked the participants to rate the design’s usability on a questionnaire after completing each route.

The questionnaire consists of questions regarding dependent measures like information perception and usability of the designs. The users’ responses are gathered on a 10 point scale. The scale is numbered from 1 to 10 and presents negative to positive response respectively.

Information perception errors: Information perception errors are the number of errors made by the participant in identifying the category of distance. We measure the number of participants’ wrong responses on the categories of distance for quantifying how many errors they made in perceiving the distance information. Furthermore, we analyzed how they interpreted the categories of distance.

Interviews

Distraction: We analyzed the drivers’ responses and videos to measure the distraction.

Screen captures and videos

In the end of the driving sessions, the participants were interviewed. They were asked about their impression about the vibro-tactile distance encodings, all driving sessions, tactile display preference over other displays, distraction and any further comments.

Video recordings are used for the data collection. Considering ethics, we have taken participants’ signatures for their willingness to record their videos. The open source software CamStudio is used for capturing the laptop screen of the experimenter and the voice of the participant. The participants were asked to announce the category of distance every time when they feel vibration. Different categories of distance pointed by experimenter are captured from laptop screen. Afterwards, the participant’s oral response on category of distance is compared with screen captures. A custom built application is used to analyze data from the videos and CamStudio. In the custom built application the keys of keyboard contained meaningful information related to correct

Participants

We evaluated a total of 13 participants, 1 female and 12 male in our experiment. The participants were between 2040 years old. All participants voluntarily participated in the experiment. Each participant completed all three sessions of the experiment. The participants have in the average 12 years of driving experience. We conducted (3*13) driving sessions with all participants in the whole study. We evaluated 3 participants after sunset and 10 in daylight. All participants completed assigned tasks easily in both conditions.

36

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and incorrect responses of the driver e.g. the participant’s response on very-far as far can be considered as meaningful information. On sensation of the vibration signal the driver guesses the category of distance by thinking aloud. Later, an analyzer watched the screen capturing segment with video. The analyzer presses the keys according to the response of the participant about the information signal. The information produced in the procedure is saved in a text file. The videos are observed two times by the analyzer.

Route-1 Route-2 Route-3

Procedure

The distance signals are controlled by the experimenter. The experimenter is seated next to the driver. During the driving session the four categories of distances are presented to the next crossing. We presented the categories of distance according to the length between two crossings e.g. all four categories of distance are displayed to the participant, if the length (in meters) is in the range of very-far distance (see Table 1). Furthermore, we displayed the 3 categories of distances if the length is in the range of the far distance and so on. We have created random sequences of the designs to balance any learning effect.

Figure 4. Three routes are used for driving in the experiment.

ilarly, we collected additional observations and the participants’ comments regarding the vibro-tactile distance encodings in the qualitative measures. In the following, we will discuss the quantitative results of our evaluation. Quantitative results

Before going to the venue, we explained the whole process of the experiment and trained each participant on all designs. Each participant was trained on the distance designs in five minutes sessions before leaving for real driving. The training was repeated before the beginning of every session.

In the following we present the quantitative results of our evaluation, structured with respect to the dependent measures (see Section ). We applied a non-parametric Friedman’s ANOVA test on data collected from the questionnaire to measure usability of the vibro-tactile distance encodings. Furthermore, to know the difference between individual conditions we applied Wilcoxon tests on each pair of conditions. We applied a simple descriptive statistics on the quantitative data collected from the videos and screen captures to measure information perception error. The results are presented with the help of boxplot and simple bar charts.

The participants were relying on the tactile belt to drive from origin to the destination. Turn by turn direction is displayed by tactile feedback. Each participant drives according to a random sequence of designs in the experiment. The participant is again trained on the first design in the sequence until he will be able to confidently recognize the signals presenting the categories of distance. In addition, the graphical presentation of the different designs – as shown in Figures 2– have been provided. After completing the first route the driver is asked to park the car on a safe place and to fill out the questionnaires. Then the driver is trained on the second design in the sequence until he will be able to confidently recognize the signals presenting the categories of distance and so on.

Usability of the distance encodings

The participants ranked the designs as most preferred, preferred, and least preferred. Figure 5 presents the participants’ ratings for the three vibro-tactile designs. The rhythm and duration based distance encoding is the most preferred design according to the statistics. The rhythm and intensity based distance encoding is rated as preferred. The rhythm based distance encoding is rated as the least preferred design by the participants.

Figure 4 presents the three different routes the participants had to drive. The destination of a route represents the start of the next route. The lengths of route 1, route 2, and route 3 are 1.3 km, 1.2 km and 1.4 km respectively. In the whole experiment the participants drive on every route at a minimum of 4 times for each design. The order of the designs was changed amongst the participants. However, we kept the same order of the driving routes.

Figure 6 presents the participants’ response on the learnability of the designs. The Friedman’s test shows that learnability of vibro-tactile distance designs are significant different (Chi-Square=9.19, p<.05). Further, the Wilcoxon text depicts that the learnability score of duration and rhythm based distance encoding is significantly higher than intensity and rhythm based distance encoding, p<.025. Similarly, learnability score of duration and rhythm based distance encoding is significantly higher than rhythm based distance encoding, p<.025.

RESULTS

All the participants were able to complete all the driving sessions. In the following, the results are divided into quantitative and qualitative results. The quantitative results provide the dependent measures like usability of the vibro-tactile distance encodings, and information perception errors. We applied statistical test to analyze the quantitative results. Sim-

In Figure 7 the boxplot presents the participants’ responses about information perception. The Friedman’s test shows that the participants perceive information significantly different in all designs (Chi-Square=10.286, p<.05). Further,

37

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 7. Ease of information perception results

Figure 5. Preference ratings for the different tactile designs.

the Wilcoxon test shows that the information perception in duration and rhythm based distance encoding ranked significantly positive than intensity and rhythm based distance encoding, p<.025. There is no difference in information perception of duration and rhythm based distance encoding and rhythm based distance encoding, p=.044. Figure 8 present the participants’ responses on the length of vibro-tactile signal. The participants’ acceptance of the length of vibro-tactile signals in the three designs is significantly different (Chi-Square=6.884, p<.05). The Wilcoxon test shows no significant difference exists in participants’ preferences on the length of the vibro-tactile signal between duration and rhythm based distance encoding and intensity and rhythm based distance encoding, p=.034. Similarly, no significant difference exists in participants’ preferences on the length of the vibro-tactile signal between duration and intensity based distance encoding and rhythm based distance encoding, p=.067.

Figure 8. Time lengths of signals results

Information perception errors

The three designs have been compared by the number of errors made by the drivers in recognizing the category of disThe Friedman’s test does not reveal any significant differtance. The percentages in Figure 9 show that the participants ence in the rating of the design judgment in all three vibromade the least number of mistakes in recognizing all catetactile distance encodings, (Chi-Square=4.478, p=.107). gories of distance in the duration and rhythm based distance encoding. In the rhythm based distance encoding the drivers According to the Friedman’s test there is no significant difmade most of the errors in perceiving a very-far distance, ference in the ease of use of the three designs, (Chi-Square=2.600, 35.7%. We observed the maximum error rate in recognizp=.273). ing far, near, and turn-now distances in the intensity and rhythm based distance encoding, 25.5%, 26%, and 7.2% respectively. Figure 10 shows that regarding the rhythm based distance encoding the drivers made most of mistakes in recognizing very-far as far, 33.3%. Furthermore, the drivers made equal amount of mistakes in recognizing far as very-far and near, 7.7%. The participants made mistakes in recognizing near as far, 17.5%. Figure 11 illustrates that most of the times the drivers made mistake in guessing the very-far distance as far, 12.8% in the rhythm and intensity based distance encoding. The drivers made higher number of errors in recognizing near as far, 26%.

Figure 6. Learnability results

A few mistakes of the participants were observed in rhythm

38

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and duration based distance encoding according to Figure 12. Turn-now Near Far Very-far

1.5

3.4

Observations and drivers’ comments

The comments and responses have been collected with the help of video recordings. In general most of the participants liked the vibro-tactile display in the car navigation system. The participants stated that the tactile vibration is sensed more effectively in sitting position inside the car. In the following we structured qualitative findings by the aspects of engagement with the environment, visual and auditory modalities, driver distraction and categories of distance.

7.2

0.0

19.3

0.0

15.4 4.7 12.8

26.0 25.5 35.7

Engagement with the environment Duration and rhythm

Only rhythm

We observed from the videos that the vibro-tactile feedback did not cause any visual distraction. In each driving session the participants were freely commenting about the houses and shops on the way. The participants stand no obstruction in thinking aloud and using the tactile display simultaneously. The drivers discovered the vibro-tactile feedback to be quite helpful while talking to the other passengers and concurrently searching a specific location in the city. The participants do not need to slow down the speed of the car to recognize and understand the vibro-tactile feedback. In total the participants missed out 3 out of 221 turns in the whole experiment. A participant commented that ”I have enjoyed this drive, because the system was not grabbing my attention at all”.

Intensity and rhythm

Figure 9. Error rates in recognizing distance categories in relation to all three designs

Visual and auditory modalities

In the responses to questions regarding the overall impression of the prototype 85% of the participants told that they like the vibro-tactile signals for displaying direction and distance. The idea of replacing the auditory feedback with the tactile feedback is preferred by 93% of participants. The participants commented that the tactile feedback is less irritating than the auditory feedback. The tactile interface is preferred for directional information by the participants. Two participants told that they would prefer to have a visual display with the tactile display. So, the visual displays can help in such situations when participants are not confident about the tactile signal.

Figure 10. Drivers’ recognition of distance categories in the rhythm based distance encoding

Driver’s distraction

No visual distraction is evident from the video analysis and the experimenter observations. We observed that in the rhythm based distance encoding the maximum intensity used to present all categories of distance was irritating for the drivers of the car. In the rhythm and intensity based distance encoding a lower intensity to indicate vary-far and far is liked by the participants. One participant told ”I will prefer lower intensity to present longer distances because it’s not disturbing”. Another participant commented ”It is good to present veryfar and far with low intensity signal to get warning before the next turn”.

Figure 11. Drivers’ recognition of distance categories in the rhythm and intensity based distance encoding

Categories of distance

The participants told that they can easily differentiate the vibro-tactile signals of very-far and far with turn-now. The participants preferred to get the signal of turn-now very close to the turn. We observed that the smooth signals for the distance very-far and far are less irritating for the participants. One participant did not felt the very-far signal in the rhythm

Figure 12. Drivers’ recognition of distance categories in the rhythm and duration based distance encoding

39

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and intensity based distance encoding twice. The participants told that they are certainly interested in the distances near and turn-now. In addition, the participants responded that it is good to have very-far and far signals in advance, so they will get attentive for upcoming turn.

drivers to easily recognize the meaning of the signal. The two designs, rhythm based distance encoding and rhythm and intensity based distance encoding, are composed of similar rhythm patterns with different levels of intensity. In both designs the results differ insignificantly in terms of mistakes of participants in perceiving distances. So, we conclude that modification in the level of intensity does not impact significantly on perception of distance in the designs. We also observed from our qualitative results that the lower intensity is less irritating for longer distance. In summary the approach of rhythm and intensity based distance encoding can be beneficial in some aspects (e.g. for longer distances lower frequency is not disturbing) but the approach of intensity based distance encoding is not mainly contributing. The results support the concept of quiet communication in the car proposed by [17]. The results encourage the use of tactile display for complex information in the car navigation systems besides the fact that the drivers need backup support of visual and auditory modalities [7].

DISCUSSION

The tactile interface is certainly helpful in the situations when it is very risky to put eyes off the road. The participants found the tactile feedback useful, less irritating and safe for the car navigation systems. The rhythm and duration based distance encoding, which supports the concept of countable pulses, is most successful among all tested designs. The results show that the rhythm and duration design is significantly more usable than the other two designs. The driver made considerably less errors in perceiving distance information. Therefore, it is discovered as an appropriate approach to encode distances through vibro-tactile signals. In the following the results regarding our research questions for this study (Section ) are discussed.

Q4: What are common errors that participants make to interpret distance information by using the tactile display? The drivers tended to make the least number of errors while they get opportunity of counting the number of pulses. In general, we can observe from the quantitative and qualitative results that most of the times, the participants face difficultly in distinguishing vibro-tactile signals for the two adjacent categories of distance. Most of the errors are evident in the rhythm based distance encoding. Similarly, the qualitative results show that the participant was unable to feel the vibro-tactile signal with lower intensity. The participants sometimes do not perceive the vibro-tactile signals of lower intensity because of vibration of the car.

Q1: Will the participants perceive turn by turn distance information without support of visual and auditory modalities in the car? The participants missed the crossing only 3 times in the whole experiment which supports the argument that vibro-tactile signals can be used to present distance information for the whole route while driving. We concluded from the qualitative results that the participants asked for visual display in case they were not sure regarding the meaning of the vibro-tactile signal. The findings like engagement with environment and the participants’ preferences to the tactile display support the fact that tactile feedback with no visual and auditory attention is beneficial for providing navigation information in many situations. So we determined that the vibro-tactile signals can be qualified to present spatial information in the car navigation system.

From qualitative results we can conclude that the tactile display does not imply any visual distraction, thus, the drivers are able to fully concentrate on the primary driving task. Our findings support the fact that the tactile displays in the car navigation systems cause less amount of distraction to the driver [17].

Q2: Which one of the encodings – rhythm, rhythm and intensity, rhythm and duration – is the most usable encoding? The results show that the rhythm and duration based distance encoding is proved to be more usable among all tested designs. In addition, it is easier for the driver to learn the distance information in rhythm and duration based distance encoding as compared to the other encodings. The concept of countable pulses for encoding distance information is quite distinct from previously proposed schemes [17, 19]. The results demonstrated that the countable pulses encoding is usable in the car navigation system. Furthermore, the participants believe that afterwards they can also gain confidence on the vibro-tactile rhythm based distance encoding approach with more practice.

The results of the experiment can only be applied to the street scenarios. The distances are decided for categories of very-far, far, near, and turn-now while driving in a residential area. The vibro-tactile signals were controlled by the experimenter, so we cannot neglect the chance of human error in the experiment. The gender aspect is also limitation of the study because mostly male participants have taken part. The results cannot be generalized for old drivers.

CONCLUSION

Q3: Which distance encoding helps the driver to make the least number of information perception / interpretation errors? The participants made the least number of errors on the rhythm and duration based distance encoding compared to the other two designs. Besides rhythm, the duration of the vibro-tactile signal plays an important role in the vibrotactile distance encoding in the car. The simplicity of the vibro-tactile information encoding information allowed the

From previous studies [6, 17] we may assume that tactile feedback can be used as an alternative way to present distance information to the driver without support of the visual or auditory displays. We investigated in this paper three vibro-tactile distance encodings based on rhythm, intensity and duration. To investigate the designs we conducted an evaluation on a real navigation scenario in an urban environment.

40

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The study shows that the tactile feedback is quite helpful for the navigation in the environment without support of visual or auditory displays. The tactile design which modifies the parameters rhythm and duration of the signal is proved to be most successful among the three tested designs. By this design the driver receives a tactile stimulation through countable vibration pulses, which encode the distance information. We can conclude that the approach can be applied to encode distance information in the most complex road scenarios. However, the present study is carried out in the urban environments and it is not clear, whether the results can be generalized for other environments, such as motorways or highways and how the distance categories have to be adapted to other scenarios. This issue remains for following analysis.

6. J. H. Hogema, S. C. D. Vries, J. B. F. V. Erp, and R. J. Kiefer. A tactile seat for direction coding in car driving: Field evaluation. IEEE Transactions on Haptics, 2 (4):181–188, 2009. 7. D. Kern, P. Marshall, E. Hornecker, Y. Rogers, and A. Schmidt. Enhancing navigation information with tactile output embedded into the steering wheel. Pervasive’09, 7:42–58, 2009. 8. N. J. Mansfield. Human Response To Vibration. CRC Press, 2005. 9. T. McDaniel, S. Kirshna, and D. Colbry. Using tactile rhythm to convey interpersonal distances to individuals who are blind. In CHI-2009, 4669-4674.

In the near future, further investigations will be conducted, in particular, to find out if there are any gender or age differences regarding the acceptance and usability of tactile presentations while driving. For our long term research, we have identified several research questions regarding tactile presentations for people on the move, such as:

10. A. Pauzie and C. Marin-Lamellet. Analysis of aging drivers behaviors navigating with in-vehicle visual display systems. In In-vehicle Navigation and Information Systems, 1991. 11. M. Pielot, O. Krull, and S. Boll. Where is my team? Supporting collaboration and situation awareness with tactile displays. In CHI ’10, 2010.

1. What other information besides turn by turn instructions can be presented with tactile feedback to the user, in order to improve safety and comfort?

12. M. A. Regan, J. D. Lee, and K. L. Young. Driver Distraction: Theory, Effects, and Mitigation. CRC Press Taylor & Francis Group, 2009.

2. Is it possible to transfer the findings also to cyclists, motorcyclists, and pedestrians?

13. A. Riener and A. Ferscha. Reconfiguration of vibro-tactile feedback based on drivers’ sitting attitude. In International Conferences on Advances in Computer-Human Interactions, 2009.

Information presentation in cars is currently a topic of high interest in the public and transportation industry. On the one hand, there is a huge demand from the end users’ point of view to ease the ability to move from one location to another with more safety and comfort. On the other hand, new technical developments allow advanced and innovative interaction techniques in cars, which will be an important argument for buying cars in the future. Tactile displays will have a great impact on future developments in human machine interface design of automobiles.

14. M. Straub, A. Riener, and A. Ferscha. Distance encoding in vibro-tactile guidance cues. In MobiQuitous 2009, 2009. 15. K. Tanaka, Y. Kishino, T. Terada, and S. Nishio. A destination prediction method using driving contexts and trajectory for car navigation systems. In ACM symposium on Applied Computing, 2009. 16. K. Tsukada and M. Yasumura. ActiveBelt: Belt-type wearable tactile display for directional navigation. In Ubiquitous Computing, 2004.

REFERENCES

1. S. Brewster and L. M. Brown. Tactons: structured tactile messages for non-visual information display. In AUIC ’04, 2004.

17. J. van Erp and H. van Veen. Vibro-tactile information presentation in automobiles. In Eurohaptics 2001, 2001.

2. G. E. Burntt and J. M. Porter. An empirical comparison of the use of distance versus landmark information within the human-machine interface for vehicle navigation systems. In Human Factors in Transportation, Communication, Health, and the Workplace, 2002.

18. J. B. F. van Erp. Guidelines for the use of vibro-tactile displays in human computer interaction. In Eurohaptics 2002, 2002. 19. J. B. F. van Erp, H. A. H. C. van Veen, C. Jansen, and T. Dobbins. Waypoint navigation with a vibrotactile waist belt. ACM Transactions on Applied Perception, 2:106–117, 2005.

3. H. Fukuda, T. Inoue, Y. Sato, and Y. Hayashi. Study on level crossing design and evaluation method based on cognitive model. Quarterly Report of RTRI, 40:26–31, 1999. 4. M. R. Grech, T. Horberry, and T. Koester. Human Factors in the Maritime Domain. CRC Press Inc, 2008.

20. H.-J. van Veen, M. Spape, and J. V. Erp. Waypoint navigation on land: Different ways of coding distance to next waypoint. In EuroHaptics 2004, 2004.

5. W. Heuten, N. Henze, M. Pielot, and S. Boll. Tactile Wayfinder: A non-visual support system for wayfinding. In NordiCHI 2008, 2008.

21. C. D. Wickens. Processing resources in attention. In Processing Resources in Attention. London:Academic, 1984.

41

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Facilitating Continuity: Exploring the Role of Digital Technology in Physical Rehabilitation Naveen Bagalkot, Elena Nazzi, Tomas Sokoler IT-University of Copenhagen Rued Langgaards Vej 07, Copenhagen nlba, elna, sokoler, @itu.dk ABSTRACT

INTRODUCTION

In this paper we report our early experiences on exploring if, and how, digital technology can help facilitate a stronger sense of continuity in the physical rehabilitation process, as experienced by the therapists and the senior citizens. We recognize four aspects of the design space offered by the notion of continuity, and present two design explorations: MagicMirror, exploring the design for self-monitoring and collaborative articulation; and Walky, exploring the design for self-monitoring and the integration of rehab activities with other everyday activities. Taking a research-throughdesign approach we have used sketching in a co-design setting with senior citizens and therapists, as the main vehicle for our research. With our MagicMirror and Walky design explorations, we aim to contribute on two levels: firstly, by providing two proof-of-concepts of designing digital technology for the physical rehabilitation of senior citizens. Secondly, and more importantly, we provide a pointer towards a new practice of physical rehabilitation: a practice where the role of digital technology is to support, through the facilitation of continuity, the collaboration between therapist and senior citizens across the private home and the rehab clinic. We speculate that, digital technology, in this way, can become a constructive part of the general move towards successful and more efficient rehabilitation processes for the ever-growing number of senior citizens in need of rehab.

Most of the developed world is facing a demographic challenge, of providing better welfare for an increasing population of senior citizens. This has triggered a host of national and pan-national initiatives in exploring the use of digital technology, aiming to enable assisted but independent living for senior citizens in their homes. A white paper from the Danish Government [8] states how a successful rehabilitation of senior citizens is central in facing this demographic challenge and identifies four basic aspects of a successful rehabilitation. In this paper we take the definition of successful rehabilitation from the Danish whitepaper as our starting point to explore the role of digital technology when aiming for more successful and efficient rehabilitation processes. In general, the Danish white paper [8] recommends more active involvement of senior citizens in the rehab process emphasizing that this process should be one of collaboration between therapist and senior citizens. Also, the white paper suggest the need for a rehabilitation process where the rehabilitation activities (i.e. physical exercises) are to be better integrated with the everyday mundane activities that constitute everyday life of senior citizens. This includes considering the rehab process as an everyday social phenomenon opening up opportunities for social wellbeing through social interaction within groups of senior citizens engaged in rehabilitation.

Author Keywords

Rehabilitation, interaction design, aging, continuity, everyday activities, physical sketches, research through design, self-monitoring

Put together, we read these recommendations as the overall need for a stronger sense of continuity that will render the rehabilitation process more coherent and more conducive towards collaboration involving all parties. We take this as our point of departure and present our explorations as part of negotiating the design space that emerges as we move forward.

ACM Classification Keywords

H.5.m: Information interfaces and presentation (e.g. HCI): miscellaneous, J.3. Life and Medical Science.

In particular, we present two such explorations: MagicMirror, exploring the design for self-monitoring and collaborative articulation; and Walky, exploring the design for self-monitoring and the integration of the rehabilitation activities with other everyday activities. Through these explorations we argue for a new role of digital technology in the physical rehabilitation of senior citizens: of enabling a successful and more efficient rehabilitation process by

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

42

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

facilitating a stronger sense of continuity in this process, as experienced by senior citizens as well as therapists.

technology we design to become part of physical rehabilitation processes:

In the following section we describe briefly the background and the motivation of our explorations. Before describing each of our explorations, we emphasize our methodology of researching through design. We then describe each of our explorations emphasizing the core scenario, the process of co-exploring in-situ, and the underlying aspects of continuity they explore. We then conclude by summarizing the explorations, highlighting our two core contributions and discussing future work.

Self-monitoring

Our design should help the citizen become more aware of his/her own condition and the effects of rehabilitation in general reinforcing the role as being an expert on his/her own condition. Collaborative articulation

Our design should encourage a process of collaborative articulation between citizens and professionals in the continuous negotiation and re-negotiation of rehab goals that drives the rehab process forward.

BACKGROUND

The national Danish whitepaper [8] defines a successful rehabilitation as follows:

Integration with everyday activities

Our design should support the integration of the rehabilitation process with the citizen’s everyday activities beyond the clinic for an independent and meaningful life.

"A goal-oriented, cooperative process involving a member of the public, his/her relatives, and professionals over a certain period of time. The aim of this process is to ensure that the person in question, who has, or is at risk of having, seriously diminished physical, mental and social functions, can achieve independence and a meaningful life. Rehabilitation takes account of the person's situation as a whole and the decisions he or she must make, and comprises coordinated, coherent, and knowledge- based measures."

Rehabilitation as social phenomenon

Our design should incorporate the notion that a rehabilitation process, though anchored within physical exercises, can be viewed as an everyday social phenomenon with inherent openings for social interaction across citizens, their relatives, professionals and the local community. We find that these four points encourage us to address an overall notion of facilitating a sense of continuity in the rehabilitation process, as experienced by the therapists and the senior citizens. We will talk about this as exploring four aspects of this same notion.

We read this definition as bringing forward four points for reflection when considering the role of the digital

Figure 1: Map showing the Design space offered by the notion of facilitating Continuity in Rehab process

43

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 1 is a map showing examples of the way the four aspects may play out as part of a rehab process. Also, the map shows how the type of rehabilitation processes, that we seek to design in support of, cuts across time and place— across exercising at home and exercising at the rehab center.

rehab activities like exercising at home or walking in the park have the potential to act as ‘tickets-to-talk’ providing openings for social interaction. Digital technology, in turn, may help make an individuals rehab activities noticeable to a group of relevant friends/rehabbuddies. Piggybacking on everyday activities as openings for social interaction has been suggested as a way to circumvent the stigma of loneliness when making initial contact [10]. Hence, though the rehab exercise primarily was meant to increase physical wellbeing it may, as a positive side effect, be a catalyst for social interaction possibly leading to increased social wellbeing within a group of senior citizens undergoing rehabilitation.

• Enabling senior citizens to bring information back and forth, between the home and rehab center, may induce a process of collaborative articulation during therapists-tocitizen meetings: a two way consultation between experts of each their kind. The senior citizens can let expertise on their own everyday lives meet with the expert knowledge on rehabilitation held by the professional therapist. This will inform and help to drive the negotiation process of planning the particular rehab activities relevant for the senior citizen in question. Previously, Sokoler et al [9] explored this aspect of collaborative articulation within the world of hand rehabilitation.

Hence, as the Figure 1 lays-out, these are the four different aspects of the notion of continuity, which become more prominent when the rehab process is explored beyond the boundaries of the rehabilitation centre. The overall objective of our research initiative is to explore how to design digital technologies for the rehabilitation by exploring the design space offered by the notion of continuity in the physical rehabilitation process. However, in this paper we limit our selves to explore the design space offered by three aspects: self-monitoring, collaborative articulation and integration with everyday activities. While, we believe that, the social interaction aspect is an important and promising avenue to explore, it is left for future work.

• Walking in the park or going to a nearby shopping mall are instances of everyday activities of senior citizens that may offer opportunities to integrate rehab activity in everyday activities, contributing to an ongoing rehab process. Providing senior citizens with their therapist’s recommendations of the ‘correct’ way to walk, as they walk is an example of this integration. Furthermore, enabling the senior citizen to bring information about everyday walking activities to the next meeting with the therapist may (as described above) inform the planning of rehab activities.

In the remaining sections we present the two explorations of the overall design space that we have conducted so far. The two explorations in question are: “MagicMirror” and “Walky”. The MagicMirror explores designing for the aspects of self-monitoring and collaborative articulation while “Walky” explores designing for the aspects of selfmonitoring and integration of the rehabilitation process with everyday activities.

• Inherent to the above two aspects is the need for the senior citizens to monitor the progress of their rehab process, as they are performing dedicated rehab exercises at home or going about their everyday activities. Thereby, self-monitoring is an integral part of the above two aspects.

In the next section we highlight our methodology of researching through design and then detail out the two explorations.

• Finally, as the rehab process is taken beyond the rehab clinic to the realm of the everyday life of senior citizens, the social dimension becomes more prominent. Everyday

Figure 2: Research through Design as a methodology of MagicMirror and Walky explorations.

44

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Here we have the opportunity to work with an interesting group of occupational therapists, very much involved into exploring new solutions for enhancing the rehabilitation process. Our collaboration started 2 years ago and includes different projects, among others, the use of the Wii platform and physical gaming for rehabilitation.

METHODOLOGY

We follow a research through design approach as a methodology for doing research. Research through design has gained importance in the recent years within the fields of HCI and Interaction Design research. As a still evolving form of research methodology, there have been many definitions of what it means to contribute to research through design [6, 13, 15, etc.]. However, the common understanding of researching through design is the ability to contribute to ongoing research discourses through designing for a particular design situation. As prerequisites of exemplary research through design, there has to be an evident expansion of the conceptual and philosophical framework [3], brought about through a reflective discourse built around a set of design exemplars [5]. The works presented in this paper follow this emergent way of contributing to research.

Citizens living in the local neighborhood visit the center. They undergo rehabilitation from simple aging problems, to mild or more severe impairments. We had the opportunity to work with two elderly women during our two explorations. While the first lady has problems with her back, the second lady has a balance problem and uses a four-wheeled walking aid since four years. EXAMPLE SKETCH: MAGIC_MIRROR

MagicMirror is a sketch inspired from the settings of a gym, where wall-wall mirrors provide a self-referential image of ones progress of muscle building. Here the MagicMirror takes multiple forms in different situations: a big screen at the rehab centre, and a TV at home, for instance.

As mentioned above, the larger framework explored with MagicMirror and Walky is, if and how digital technology can facilitate a sense of continuity in the physical rehabilitation process. This investigation is driven by the two concrete design explorations, where we explore the design space offered by three aspects of continuity: selfmonitoring, collaborative articulation, and integration of rehab activities in everyday life (see Figure 2). Thereby, we contribute to the fields of HCI and Interaction Design research, through these two design explorations, on two levels: firstly, by providing concrete proof-of-concepts of digital technology designed for the specific physical rehabilitation activities of senior citizens. Secondly, and more importantly, we provide a pointer towards a new practice of continuous, ongoing and collaborative physical rehabilitation process, informed and facilitated by digital technology.

Recently initiatives in ‘WiiHabilitation’ have gained ground in the world of physical rehabilitation [11]. With the Wii platform introducing a more physical gaming experience, the idea of using this technology as part of physical rehabilitation is straightforward. Hence, experiments with ‘WiiHabilitation’ are gaining ground throughout the world of geriatrics and physical rehabilitation [for instance, 2, 7, 11]. The Wii platform allows us to look into the general notion of integrating play, and the experiential qualities belonging to the gaming universe, with the tough everyday challenges of going through physical rehabilitation. With MagicMirror, we explore beyond what is offered by the immediate features of Wii and games for specific kinds of rehabilitation. Here, we move towards exploring the design of digital technology for facilitating the aspects of self-monitoring and collaborative articulation in the rehab process, beyond the rehab centre.

The MagicMirror and Walky design explorations lie in the design exploration vertex of the Interaction Design Research triangle [6], with sketching in a co-design setting as the central driving engine. Inspired from Bill Buxton [4], by sketch we mean early manifestations of design ideas in different forms, not the least in software/hardware, video and photomontages, etc., that are available for people to experience, and hence be part of the design exploration in early stages. The process of sketching and negotiation of the design space was situated in-situ, with the therapists and the senior citizens taking part in the process through enacting and experiencing. Thereby the evaluation of the sketches and the specific features of the design concepts was an ongoing and inherent part of the process. Additionally, the co-design process facilitated a move forward in co-articulating what it means to design, and use, digital technology to facilitate three aspects of continuity in rehab process, as experienced by the therapists and the senior citizens.

Specifically, we explore the possibilities of tracking and video recording the exercises the senior citizen performs under the supervision of the therapist, at the rehab centre. The senior citizen takes home this recorded video and uses it as the ‘reference’ exercise to monitor self while exercising at home, as it plays out on the TV. Simultaneously, the MagicMirror tracks the body movements, records and overlays it on the instructional video, thus giving the senior citizen a self-referential video for exercising. This recorded video is then taken back to the clinic where the senior citizen and therapist discuss the progress in detail, thus enabling a more collaborative rehabilitation process. A Scenario

Ole is a 70-year-old man living alone. He has developed knee pain due to a biking accident and is undergoing a rehab process at the rehab centre. He meets his therapist

SETTING: TÅRNBY REHABILITATION CENTER

The setting of our explorations is Tårnby Sundhedscenter, a local rehabilitation center of the Copenhagen Municipality.

45

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and they record the video of Ole doing the exercises at the rehab centre. This video is carried onto Ole’s cell phone to his home, where he plays the video on his television, while exercising. The MagicMirror tracks his movements and overlays this feed with that of the recorded version, thus giving Ole a direct reference to exercise correctly. On his next visit to the rehab centre, the therapist looks up the recorded home videos of Ole’s exercises and both of them discuss his progress and next steps.

being captured. Our sketch of MagicMirror is still work-inprogress and a little away from this technological possibility. Here, we simulated the tracking and overlapping by using a combination of Adobe Premier and Flash. The exercises were captured from the Adobe Premier suite through the webcam of a MacBook Pro, which were then taken to Adobe Flash and the body joints were tracked frame by frame (see Figure 3). This video was taken to the home of the senior citizen and played as the reference video. The new exercises were again recorded through Adobe Premier and played alongside the reference video during the exercise at home. This home video was taken back to the design studio, to be tracked and laid over the reference video. Finally we took the overlaid video to the rehab centre again to simulate and enact the discussion of progress.

Sketching and co-exploring MagicMirror

Currently in a physical rehabilitation process, the therapists prescribe exercises for the senior citizens to do at home, but don’t have much information of how the senior citizens managed to do these exercises. At the same time, unlike while exercising with their therapists, the senior citizens don’t have much clearer instructions during exercising at home. A case in point is an 82-year-old lady doing pushups at home on a soft mattress, while her therapist having no clue that his instructions will be carried out in such manner. This was observed during one of the field studies done by a master student working on the project. The MagicMirror scenario is an example of a way digital technology can facilitate rehabilitation process to go beyond the rehab centre by enabling information exchange between the rehab centre and home. As the scenario depicts, the recorded exercises at the rehab centre become the ideal exercise while exercising at home. The live tracking and overlaying of the current exercise over the ideal exercise gives a self-referential monitoring of the progress to the senior citizen. With this recorded home exercise available at the rehab centre, the senior citizen can discuss with the therapist the various aspects of the progress of the rehabilitation process. As the senior citizen and therapists manage this exchange of rehab information, they prepare themselves for a more collaborative articulation of the rehab process.

Figure 3: The body joints tracked for the particular exercise Co-exploring through enactments

For this initial exploration, we selected one form of exercise for back training: lifting both the hands from side of the body to the top and back. The vertical body position, and the straightness of the joints were the main features to be tracked in this exercise. As shown in figure 3, the MagicMirror sketch tracks the vertical position of the spine with respect to the hand movements. Also the senior citizen was asked to wear wrist, arm and head bands with IR emitting LEDs, which were tracked by a webcam ensemble, for later processing in an image processing software. For the purpose of this sketch, the black bands augmented the visual reference provided by the lines.

Sketching MagicMirror

The intended technology behind the MagicMirror is motion tracking through an Infrared enabled webcam, which will then be processed by an image processing software for tracking and overlay. This process happens in real-time, with the processing done by the software as the video is

Figure 4: A Three stage Co-designing of MagicMirror - at rehab center, at home and back at rehab center

46

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

During, and after, the process of sketching we involved two therapists from the Tårnby rehabilitation centre and a senior citizen with back problems. As an early exploration of the possibilities, we enacted a three stage cycle with the senior citizen and the therapists: recording of the reference video at the rehab centre, exercising at home using this reference video, and finally, the meeting back at the rehab centre with the therapist and senior citizen discussing over the home video (see Figure 4).

recording and tracking of exercises done, both at the rehab centre and at home. This feature addresses facilitating a more collaborative articulation by enabling both the therapists and the citizens to meet with richer information and discuss the progress of the rehab process. Facilitating the Aspect of Self-monitoring

• The senior citizens themselves are recorded as they exercise with the therapist at the rehab centre. While exercising at home looking at this reference video, the senior citizens themselves are their own ideal, rather than an animated avatar. This provides a more self-referential and immediate monitoring of the rehab progress.

Co-discussing the enactments

During the discussion after the enactments, the therapists immediately realized the importance of the video material, as any discrepancy was visually evident with the tracking of the body joints.

• MagicMirror intends to track the relevant body joints and draws a line-diagram (see Figure 3) in real-time as the exercises are being recorded. As the current video with this information is overlapped over the reference video, the senior citizen gets an immediate visual feedback of the correctness of the exercise.

Meanwhile, the senior citizen was enthusiastic of seeing herself as the focus of the reference exercise video. She also mentioned that the overlaying of her home exercise video with the reference video was helpful to encourage her, as she gets to see the extent she stretched her arms today, compared to her previous day’s effort.

These two features address specifically the possibility of self-monitoring as, and when, the senior citizens exercise at home.

During the third stage in the cycle, the senior citizen and therapist looked at the video of the exercise done at home on a big screen at the rehab center (see Figure 4). Here the line-diagram and overlaying of the videos gave a clear indication of both, the degree of progress made by the senior citizen and the discrepancies in the exercise. Though the senior citizen was well advanced in her rehab process, both the therapist and the senior citizen immediately sensed the possibilities of tracking and overlaying, in facilitating self-monitoring of the rehab process. The overlaid videos gave a direct visual evidence of to what extent the senior citizen could stretch her arms on this day as compared to the effort she put in the reference video.

EXAMPLE SKETCH: WALKY

Walky is a sketch that explores walking, as a rehab activity as well as an activity of daily living. It is driven by the idea of augmenting everyday objects of rehabilitation with digital technology to provide feedback for self-monitoring and to communicate with other objects of daily living. Walky is an augmented four-wheeled rollator providing the senior citizens opportunities of self-monitoring of their walking as they go about their everyday activities. Recently, assistive technologies have been developed to support the walking activity of senior citizens with mild and severe impairments. Passive robotics and intelligent environments have been studied to provide the support for autonomous object detection and avoidance, for navigation, and for breaks and speed control [e.g.13]. Sensors, for instance pedometers and accelerometers, are largely used to monitor walking activity. Flowie [1] is a recent example of self-monitoring applied to the walking exercise. It connects a pedometer and a digital picture frame in the home and creates a persuasive virtual coach that aims to motivate people to walk more.

During the enactment and later, the therapists were of the opinion that the video material from the home of the senior citizen was valuable information on the everyday realities of the world of the senior citizen, which will help them in fine-tuning the rehab process better for the senior citizen. Furthermore, the senior citizen, when discussing with the therapists, imagined that with this video material she could have a more central role in articulating the next stages in the processes. Design Rationales: Facilitating Self-monitoring and Collaborative articulation

With Walky we explore beyond passive robotics and concentrate on the design of digital technology for facilitating the aspect of integrating rehabilitation activity into everyday life of senior citizens, and the aspect of selfmonitoring. We specifically, concentrate on walking as an activity of everyday life.

In the next paragraphs we reflect on how MagicMirror facilitates the aspects of collaborative articulation and selfmonitoring. Facilitating the Aspect of Collaborative Articulation

• MagicMirror provides both, the therapist with the knowledge of how the exercise happened at home, and the senior citizen with the knowledge of how the exercises are done correctly as the senior citizen is exercising at home. It enables information exchange between the rehab centre and the home, through video

Walky consists of an augmented four-wheeled rollator, combined with a small movable device, called PocketWalky. The rollator is augmented with sensing, actuating and communication technologies. PocketWalky

47

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

communicates with the rollator, displays information about walking, and allows senior citizens to exchange progress information with the therapists in order to set and reset together the rehabilitation goals.

As a starting point we formulated three questions about walking: “how much I walk” to understand, and keep track of, the distance covered while walking; “how I walk” to provide clues about posture and balance for correct usage of the rollator; and “where I walk”, to get a sense of being part of ones rehabilitation friends through visual notifications of others’ activities. The third question reflects the fourth aspect of continuity: of facilitating the opportunity for social interaction through making rehabilitation exercises visible for within the local community of senior citizens. It has been just briefly touched during this first sketch of Walky and will be part of its future explorations.

Specifically, we explore the possibility to make the use of everyday objects a sources for: self-monitoring, and communication with the therapists and with the local community. In doing so, we aim to integrate rehabilitation exercises for better walking into other everyday activities of senior citizens involving walking. A Scenario

Anne is an 80-year-old woman living alone for 15 years. She has been using a four-wheeled rollator for 2 years due to a balance problem, and she is going to the rehabilitation center twice a week. The therapist introduces her to Walky and PocketWalky, a rollator that gives her feedback about her balance while walking and its companion movable device. This is done through mild vibrations on the handle of the side she is exerting more pressure. Anne and her therapist set her walking goal of the week in terms of 6 times the distance to the local shop from her home, and 3 times the distance to the park near her home (corresponding approximately to 3 km). During the week, after going to the market she sees her progress on PocketWalky, of how much she has been walking. During the next visit to the rehab center, her therapist takes a look at her progress and together they discuss to reset or maintain the goals.

In summary, with Walky we explore the design of an augmented walking aid that: • Enables senior citizens to self-monitor their walking activity (“how I walk” and “how much I walk”); • Makes the walking goals visible as the senior citizens go about their everyday activities; • And enables the therapist to view the report of how the walking went in everyday life when the citizens come back to the rehabilitation center. Sketching Walky

We sketched Walky using the Arduino1 platform. We augmented a common rollator used in Denmark with simple sensors: a pressure sensor on each handle, an ultrasonic distance sensor, and a switch to count the number of rotations of the rollator’s wheel. The first sketch included tactile (a vibration motor on each of the handles) and visual actuators (a couple of LED lights on the rollator frame). The sketch gives feedback on posture, balance, distance, and speed. We added a ZigBee2 module for the communication with PocketWalky, the movable device, to transmit information about walking activity, and to receive commands and calibration details. To sketch PocketWalky we used simple cardboard sketches and a more realistic mock up realized with a 3D printer. In this early exploration we gathered the information that PocketWalky would display with another Arduino+ZigBee sketch connected to a portable pc used to displaying the sensors data. Finally we used other cardboard sketches of digital picture frames to represent one of the everyday objects with which the movable device can communicate in the home environment. Figure 5 shows these sketches at work in our co-design sessions.

Sketching and co-exploring Walky

Currently the senior citizens undergoing mobility rehabilitation, with the help of walking aids, visit the rehabilitation center once or twice a week. They undergo an hour of exercises with the supervision of their therapists. An example of a common exercise is to walk as much as they can, back and forth, on a 20-meter stretch of corridor. The therapists correct small details of the usage of the aid; depending on the specific problem and severity, the therapists remind the senior citizens to keep their back straight, or to concentrate on keeping their balance in the center. The therapists suggest the senior citizens to exercise by going out for a walk everyday, when not visiting the rehab center. Some of the citizens fix their goals in terms of meters to walk in a week, others in terms of going to the nearby shop everyday. However, neither the therapists have any means to figure out how the senior citizens walked, nor the senior citizens have any means of verifying if, and how much, their everyday walking is helping them towards progress, or whether their posture and balance is right while walking.

These sketches were realized to answer the questions of how much I walk and how I walk. • “How much I walk?” While walking with Walky, the senior citizen can access and monitor the distance

As any other aid, rollators have a proper way to be used effectively: walk within the line joining the front two wheels (closer to the person walking), the back has to be upright, and the balance has to be kept by maintaining the body weight in the center of the imaginary line that divides the rollator in two symmetrical parts.

1

Arduino is an open-source electronics prototyping platform. See http://arduino.cc. 2

ZigBee is a standard-based communication technology. See http://en.wikipedia.org/wiki/ZigBee.

48

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

covered, and the duration of the current walking activity. Walky gives feedback of distance covered in terms of meters, which can be translated in terms of landmarks reached (e.g. how many times this distance equals the distance from home to my everyday shop). During the walk, this feedback is displayed on PocketWalky. Through this device the senior citizen can visualize the walking goal, track the progress made, and reset the weekly or daily goals.

walk. During the walk we gave her feedback on how much she was walking in meters. She liked the idea and actually discovered that she had walked much more than she thought she could. We used the cardboard sketches of PocketWalky to visualize the feedback, and to show her possible interactions. We prepared some paper based transparent screens to overlay to simulate the updating of the information. Co-discussing the experiences

• “How I walk?” Walky reminds the senior citizen to keep the proper posture and the right balance while walking. If the senior citizen is walking too far from the structure of Walky, the PocketWalky displays either graphical animations or colored LED lights. This will remind the senior citizen to walk within the rollator frame (closer to it) and keep the back upright, re-establishing the right posture. If the senior citizen is exerting more weight on the rollator handles, and/or the weight is not evenly distributed between the two handles, a mild vibration— on either handle—indicates the side on which the senior citizen is putting more weight.

In general the senior citizen felt the haptic feedback more direct, and discreet, as compared to the acoustic or visual one. Which highlights the fact that, while integrating rehabilitation as part of everyday life, we need to design the sensing and actuating technologies that don’t provoke embarrassment or invade privacy. For example, providing tactile feedback showed to be non intrusive, effective, and intuitively preferable to audio or visual feedback. The senior citizen said that she was engaged in keeping her correct balance by trying to make the rollator’s handle not vibrate. The therapists recognized the opportunities in receiving direct feedback from the aid. They recognized how a direct haptic feedback can communicate what is wrong, and produce an effective body reaction for correcting the usage of Walky.

Co-exploring through Experiencing the Sketch

During, and after, the process of sketching, we involved the therapists and an elderly woman to co-explore the possibilities offered by Walky. To begin, we organized two sessions: an indoor session at the rehabilitation center, and an outdoor session in the local neighborhood.

The senior citizen experienced direct feedback of selfmonitoring in the rehabilitation environment and she recognized its usefulness in this setting. During the first visit the senior citizen mentioned she wouldn’t need a direct feedback during her everyday walking outside the rehab center. However during the second visit outdoors, she appreciated how the feedback, about how much she has walked, could give her a more precise translation from her perception of distance to the actual meters covered. After this observation we could imagine a future scenario where self-monitoring has different levels and each of these levels can be turned on and off by the senior citizens according to the current stage of their rehab process. The therapists also recognized a good potential in PocketWalky, as a dedicated device to link rehabilitation and everyday activities, and the possibility it offers them to get informed about the progress made by the senior citizens outside the rehab center.

In the rehabilitation center, we introduced the senior citizen to the project through the Walky sketches with the collaboration of the therapists (see Figure 5a). The senior citizen was asked to walk with Walky, and to experience the different direct feedbacks about posture and balance. Both the senior citizen and the therapists had the chance to walk with the sketch and comment on the perception of the feedbacks. In the second outdoors visit (see Figure 5b) we focused on the questions of “how much I walk?” We organized a short walk outside the rehabilitation center. The rehab center is part of the senior citizen’s neighborhood and hence familiar for her. We let her decide the direction and the distance, and she indicated a white house, whose distance she thought was a good approximation of how much she could

Figure 5: Co-exploring Walky indoors (a), outdoors (b), Sketches of PocketWalky (c,d).

49

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The indirect feedback accessible from PocketWalky (e.g. distance covered) offered a more objective measurement of “how much” the senior citizen has walked. We realized that absolute feedback in terms of meters loses meaningfulness in relation to everyday life. Being able to walk four times the distance from home to the nearby shop in two days has much more meaning than walking 800 meters. Together we recognized that the possibility of setting goals, and being able to self-monitor the progress of these goals, in terms of local landmarks and icons encountered in everyday activities (shop, park, friend’s house, post office), could be very helpful. Finally, the therapists recognized the possibility that Walky offers in motivating senior citizens towards getting better, by offering them real-time feedback on how they are walking, and also by the possibility of treating their walk to a nearby shop as a rehabilitation activity.

for the senior citizens to exercise the correct way of walking, improving their rehabilitation. • These sketches work on everyday rehabilitation tools and everyday objects: PocketWalky, for example, enters the home and combines with common objects. It may take a form of a digital picture frame, a TV screen, or simply stay on the fridge, and support the visibility of goals for the rehabilitation of walking. CONCLUDING REMARKS.

In the above sections we detailed how we explore the design space offered by the 3 aspects of continuity in the physical rehabilitation process of senior citizens. We took our motivation from the definition of the successful rehabilitation by the Danish white paper [8]. We discussed how the recommendations of the white paper represent a call for facilitating the notion of continuity in the rehab process, as experienced by the therapists and the senior citizens. Then we detailed, and discussed, our early experiences in co-exploring MagicMirror and Walky, highlighting how the designs facilitate the aspects of selfmonitoring, collaborative articulation of the rehab process, and integration of the rehab process as part of everyday activities. We finally conclude in reflecting on our contributions: firstly, by providing concrete proof-ofconcepts in the forms of MagicMirror and Walky; and secondly, by opening up the design space for facilitating continuity and articulating a new way of digital technology in support of physical rehabilitation.

Design Rationales: Facilitating Self-monitoring and Integration of Rehab with everyday activities

In the following paragraphs we reflect on the features of Walky that facilitate self-monitoring, and integration of the rehab process with other everyday activities. Facilitating the Aspect of Self-monitoring

Walky provides, both direct and indirect, feedback for facilitating self-monitoring. Because of the ability to monitor their own walking activity, senior citizens can learn to be experts of their walking outside the rehabilitation center. • The haptic feedback from Walky initiates an immediate reaction from the senior citizen and correction of the walking posture. Here we piggyback on everyday artifacts (the walker handle) to give immediate feedback as facilitating self-monitoring.

MagicMirror and Walky: Two Proof-of-Concepts

As described above, we went about developing the sketches in-situ, with the therapists and the senior citizens as stakeholders and co-designers. The design of the sketches evolved from paper sketches to hardware manifestations in a co-design setting, where senior citizens and therapists evaluated the ideas and specific features as the process moved forward. In addition, the process was set in the actual site of possible usages: the rehab centre, the living rooms of the senior citizens, and outdoor parks, thereby bringing forward a more direct and relevant evaluation of the concepts. Thus we contribute to the field of HCI and interaction design research with two proof-of-concepts, in the forms of MagicMirror and Walky. They represent ways of digital technology facilitating concrete rehab exercises of upper body rehabilitation, and walking, respectively.

• Walky offers an opportunity for the senior citizens to set their rehabilitation goals in a more relevant manner, and together with the therapists, reflect and decide how much they would like to walk per day to achieve this goal. Having a visual reminder of their goal can be a stimulus and a motivation. The walking progress can be quantified using the progress towards the goal as a tool to exchange information between the therapist and the user. Through facilitating the aspect of self-monitoring in the above ways, Walky aims to add meaning while keeping the original meaning of walking to the nearby shop, for instance, intact.

Digital Technology facilitating continuity in rehab process

Furthermore, the MagicMirror and Walky both point towards a new role of digital technology in the rehabilitation of senior citizens. By facilitating information exchange between home and clinic, MagicMirror enables both the therapist and the senior citizens to engage in a more collaborative articulation of the rehab process. The Walky enables information exchange between everyday activities of walking, the senior citizen, and the therapist, thereby, integrating the rehab activity as part of everyday activities involving walking. Both the explorations facilitate the aspect of self-monitoring by providing immediate

Facilitating the Aspect of Integration of the rehab process with other everyday activities

Walking is in itself an activity of everyday life. Age and impairments often reduce the confidence and the ability of the walking activity. Walky and PocketWalky are focused on integrating the rehabilitation exercises of walking as part of the walking one does in everyday life, outside the rehab center. • While walking to the shop, or going to the park, Walky, by providing real-time feedback, gives the opportunity

50

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

possibilities for the senior citizens to monitor their progress, giving them a more direct control and active role in their rehab process. Thereby with these explorations we move towards articulating a new role for digital technology in the rehab process: to facilitate the notion of continuity in the rehabilitation process, as experienced by the therapists and senior citizens. This is realized by designing digital technology to address the facilitating the aspects of: selfmonitoring, collaborative articulation of the rehab process, and integration of rehab activities with everyday activities.

4. Buxton, B. (2007) Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc , USA 5. Ehn, P. (2002) Neither Bauhäusler nor Nerd: Educating the Interaction Designer. In Proceedings of the 4th Conf. On Designing interactive Systems: Processes, Practices, Methods, and Techniques (London, England, June 25 28, 2002). DIS '02. ACM, New York, NY, 19-23. 6. Fallman, D. (2008). The Interaction Design Research Triangle of Design Practice, Design Exploration, and Design Studies, Design Issues, Vol. 24, No. 3, p. 4-18, MIT Press.

Future Work

While in this paper, we specifically addressed how digital technology can facilitate the first three aspects of continuity, we in future, plan to address the fourth aspect of facilitating social interaction. Further explorations are planned to explore the role of technology in providing openings for social interaction amongst a local group of senior citizens, by taking advantage of the social character of rehabilitation activities, both, within and beyond the rehab centre.

7. Fergus, P. Kafiyat, K. Merabati, M., Taleb-bendiab, A. El Rhalibi, A. (2009) Remote Physiotherapy Treatments using Wireless Body Sensor Networks. IWCMC2009, Leipzig, Germany, June 21-24, 1191-1197. ACM, USA. 8. National whitepaper on rehabilitation: http://www.marselisborgcentret.dk/fileadmin/filer/hvidb og/hvidbog.pdf. 9. Sokoler, T., Löwgren, J., Eriksen, M. A., Linde, P., and Olofsson, S. (2007). Explicit interaction for surgical rehabilitation. In Proceedings of the 1st international Conference on Tangible and Embedded Interaction, Baton Rouge, Louisiana, February 15 - 17, pp117-124, TEI '07. ACM, USA.

With Walky, and “where I walk?”, we have started exploring the possibilities of “notice and be noticed”: have the feeling of what’s happening in the local community of senior citizens, by receiving notifications when there is a friend out walking, for instance, and sharing small bits of information about walking activity with the local community. With MagicMirror, we aim to explore the idea of MagicWindow, where senior citizens can notify their friends when they start exercising using the MagicMirror, thereby inviting implicitly their friends to be part of it.

10. Svensson, M. S. and Sokoler, T. (2008). Ticket-to-talktelevision: designing for the circumstantial nature of everyday social interaction. In Proc. of the 5th Nordic Conference on Human-Computer interaction: Building Bridges (Lund, Sweden, October 20 - 22, 2008). NordiCHI '08, vol. 358. ACM, New York, NY, 334-343

ACKNOWLEDGEMENTS

We would like to thank the therapists of Tårnby rehabilitation center, specifically, Hannes and Majbritt, and the senior citizens, Borgild and Anita, for their time, enthusiasm, and precious contributions during the coexplorations. We thank two master students, Louise Miller (IT University of Copenhagen) and Pragati Kapur (IDC, IIT Bombay), for their contributions in designing and sketching Walky. We also thank the reviewers for their constructive criticisms on the paper.

11. Tanner, L. (2008) Doctors use Wii games for rehab therapy. Associated Press, 2 2008 12. Weiss, P. Rand, D., Katz, N. Kizony, R. (2004) Video Capture Virtual Reality as a Flexible and Effective Rehabilitation Tool. Journal of Neuro Engineering and Rehabilitation. Volume 1, issue 12. 13. Wolf, T. V., Rode, J. A., Sussman, J., and Kellogg, W. A. (2006) Dispelling "design" as the black art of CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 - 27, 2006). R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, and G. Olson, Eds. CHI '06. ACM, New York, NY, 521-530.

REFERENCES

1. Albaina, T. Visser, C. A.P.G. van der Mast, M. H. Vastenburg (2009). Flowie: A Persuasive Virtual Coach to Motivate Elderly Individuals to Walk. In Proc. Of 3rd Int. Conf. On Pervasive Computing Technologies for Healthcare (PervasiveHealth). London, UK, April 1-3, 2009.

14. Yu, H., Spenko, M., and Dubowsky, S. (2003). An Adaptive Shared Control System for an Intelligent Mobility Aid for the Elderly. Auton. Robots 15, 1 (Jul. 2003), 53-66.

2. Alankus, G. Lazar, A. May, M. Kelleher, C. (2010) Towards Customizable Games for Stroke Rehabilitation. CHI2010, Atlanta, USA, April 10-15, 2113-2122. ACM, USA.

15. Zimmerman, J., Forlizzi, J., and Evenson, S. (2007) Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI Conf. on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI '07. ACM, New York, NY, 493-502.

3. Binder, T, and Redström, J. (2006) Exemplary Design Research. Accessed on February 17, 2010, from: http://eprints.sics.se/920/

51

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

4Photos: A Collaborative Photo Sharing Experience Martijn ten Bhömer1, John Helmes2, Kenton O'Hara2, Elise van den Hoven1 Department of Industrial Design1 Microsoft Research Ltd.2 Eindhoven University of Technology 7 JJ Thomson Avenue Den Dolech 2, 5600 MB, Eindhoven, CB3 0FB, Cambridge, the Netherlands United Kingdom [email protected], [email protected] [email protected], [email protected] ABSTRACT

offer new ways for people to store, organise, tag and manipulate photographic content. Digital photo frames and photo printers create different opportunities for the way photographic content is presented in domestic settings. Of significance too are the increased possibilities for sharing our digital photos with others through established CMC channels such as email and perhaps more importantly through the appropriation of online social networking sites such as Facebook. Alongside these shifts, there has been a growing interest among the HCI and design community in emergent practices and values arising from the adoption of these new photographic technologies within domestic settings (see Lindley et al. [13] for an overview). Curiously though, as a number of authors observe, there is a disparity in the focus of this work whereby technologies and behavioural practices around photo displays have been relatively underexplored compared to other aspects of photowork [5,19,20]. Of particular significance to their arguments are concerns with the material properties of photo displays that form the essential components through which social practices, values and meanings are realized. Understanding this relationship between material properties and behavioural practices with respect to photo displays can then provide us with a resource for design. By manipulating the material characteristics of photo display and their setting we should be able to introduce new dynamics into the discussions and other behaviours that happen around these.

In this paper, we describe the iterative design and user study of “4Photos”, a multi-screen table centrepiece allowing media content to be shared and enjoyed in a social setting. It was our intention to design an object with the purpose to gather qualitative data concerning the social effects of new ways of democratic, serendipitous and playful photo sharing. To facilitate this we used online photo repository content that most often gets experienced in an individual setting. Using 4Photos we positioned this content within a social setting and observed how the presentation of these images enabled new ways of „phototalk‟ to arise. We describe the design process, the final concept and reflect upon observed practices that emerged from people‟s usage of 4Photos. We then present several design implications and discuss future directions for continuation of this research. Author Keywords

Information interfaces, Photo sharing, Interaction, Serendipity, Collaboration, Screens, Iterative design. ACM Classification Keywords

C.0. Computer Systems Organization: General – Hardware/software interfaces. H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. H.5.2 [Information Interfaces and Presentation (e.g. HCI)]: User Interfaces - Input devices and strategies (e.g., mouse, touchscreen), User-centered design. INTRODUCTION

With the major proliferation of digital photo technologies, we have seen gradual and significant shifts in domestic photographic practices. Digital capture technologies in the form of dedicated devices and mobile phone based capture have opened up photographic capture to a much wider range of the population [5,18]. Other forms of photoware [6]

With this in mind we present an example of a new form of photo display, 4Photos, designed to be the centrepiece for a dinner table. In designing for this context, we make a number of design choices intended to introduce different social dynamics into the talk constructed around displayed photos. In particular, through our manipulations of display form factor, interaction access points and mechanisms for sourcing content, we hope to affect some of the subtle control dynamics of phototalk achieved with other forms of photographic presentation (e.g. paper or laptop). In this paper we present a discussion of the 4Photos design and then examine the impact of these factors through behavioural analysis of the system in use.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

52

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

new display technologies. Most commercial digital photo frames, for example, cycle through a selection of images meaning that particular location for display within the home need not be so constrained as by their physical counterparts.

Related work

Before discussing the design of the system, we first wish to build up the context for our arguments with reference to the literature. Work by Chalfen [2] has highlighted ways that domestic photography is a socially engaged practice through which family norms, traditions and values can be expressed and maintained. Such practices are reflected in activities such as the display of photographs in the home and the talk that is performed around photographs. These ideas are seen too in some of the foundational work in HCI and CSCW on photographic practices such as the work of Crabtree et al. [4] and Frohlich [6] on phototalk during the collocated sharing of photographs within domestic settings. Crabtree‟s work is particularly important in demonstrating the interactionally embodied nature of phototalk around printed photographs as revealed through ethnographic observations and analyses of photo sharing practices. Significant to our concerns in this paper is how Crabtree demonstrates the collaborative construction and coordination of phototalk as a practical achievement, realised through the way social groups gather round the photographic representation, the way the physical printouts are oriented to group members, passed around and gestured towards. While this study focused on the paper-based photographic representations predominant at the time, it is nevertheless significant in the ways it relates articulation and interaction work taking to the material properties of the representational artefacts. A key theme from this work is the issue of conversational control and its relationship to the representational artefact. For example, ownership of the physical photograph bestows a prominent role within the phototalk that occurs around it. In part this is about content ownership but also in the timing of photo distribution and physical orientation of the photos to others in the conversation [4].

Related to this and also the issues of control discussed above are notions of curatorial control over what is displayed. Durrant et al, among others, argue that curatorial control over placement of photos has traditionally resided with the family member in charge of the photographic technology [2,5,16]. However, with the apparent democratization of domestic photography in terms of capture and online sharing we again see shifting dynamics of curatorial control throughout the rest of the family. While there is increased democratization in display activity here, content of the photos displayed in the home remain closely bound to the owners of the space – namely the immediate family members that reside there. This too affects the dynamics with which such photos come to be used in phototalk on particular social occasions. Authors such as Kim and Zimmerman have highlighted alternative scenarios that may be of interest too [9]. For example, their work considers the scenario of visitors coming into the household. In response to this, they discuss a display concept that enables context sensitive presentation of photographic materials in response to particular visitors leading to potentially new dynamics in phototalk on such occasions of a visit. In this paper we want to explore other ways of shifting curatorial control over content to be more democratic (e.g. among a group of visitors to the home). Where we have seen interesting innovations in this area of photo display e.g. [9,20] what is curious is the relative lack of exploration in terms of their emergent behavioural properties arising from deployment in real world setting. A notable exception here is the recent work of Leong [11] which explores the use of a digital photo frame within a domestic setting. The particular concern of Leong was with the notions of serendipity arising from the random presentation of multiple images from a domestic photo collection. Leong demonstrated that abdicating choice of photographic presentation in this way, and the uncertainties associated to this, leads to new values and meanings relating to photographic display. We revisit some of these themes of choice abdication, uncertainty and serendipity in the discussion of our own fieldwork later in the paper and in particular how it relates to collaborative photo sharing experiences as opposed to the individual experiences that were the focus of Leong‟s enquiry.

The material properties of photo display in relation to behavioural practice and values are considered further in [19,20]. Again while this work was conducted with more traditional paper based photographic displays, it draws out key themes useful to our arguments. First is how social meaning is created through the placement of photographs in particular spatial locations around the home (a theme that is also developed in Crabtree‟s discussion of coordinate displays within the home [3]). Different meaning becomes associated with different household locations. Different locations too affect how photos are available to use in particular social circumstances, shaping the opportunities for use in photo talk – their availability to hand is bound up in the material qualities of their display. Critical to this meaning making with printed photographic materials is that physical display space is a finite resource. If a printed photo occupies a particular location, that location is no longer available for the display of another photo. As a consequence, the choice of photo for a particular location within the home acquires particular significance. Yet such dynamics potentially are shifting with the introduction of

Generally speaking, where innovation has occurred in the area of photo displays, there is often a strong grounding in the form factor of traditional displays. Many are designed to be positioned on wall, sideboards, shelves and mantelpieces in much the way that traditional photo frames are. The aim here is to augment the traditional with digital capabilities in terms of the way content is sent to be managed and displayed. As such, while location is a strong theme to emerge from the photographic literature little attention is

53

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

given over to consideration of alternative locations of display in design. This then is something we aim to address in this paper where we present a concept specifically for a different location, namely the dinner table.

fits neatly to a group defined by a location where the photographic display is placed. Indeed in light of different public private boundaries in different user segments raised by Miller and Edwards [14] with respect to online photosharing practices with sites such as Flickr and Facebook, it is likely that related concerns might be apparent in the linking of photo displays to online photo repositories in social networking sites. This is a question we consider further through our design and fieldwork with 4Photos.

This leads us to consider explorations of tabletop displays for presenting and managing photographs e.g. [1,10,17]. While these have yet to be adopted in any meaningful sense in domestic settings, they at least point to new ways of configuring people around photographic arrangements that can impact on the social context in which phototalk is conducted. This is due to the way the table arranges people around the photo display but also in the way that it opens up access points to all around the table through the use of multi-touch input mechanisms. These factors relate to our design concerns in this paper though we take a somewhat different approach to these issues. Tabletop computing typically treats the whole surface as the focal point for interaction and in this respect the photowork conducted is something that is a focal activity rather than just a component in a social gathering. Our approach, then, is to create a photo display to be situated on a tabletop where it is but one component in the assembly of artefacts making up the social setting, such as a dinner party. That is, it is designed as a tabletop ornament. In this respect, our aims are to exploit the ways a table physically and socially configures people for a particular social gathering but without requiring photowork to become dominant over other aspects of the social occasion and over other artefacts assembled on the table to support these occasions. This approach is in line with arguments about appropriate balancing of interactive and non-interactive aspects of tabletop assemblies [15].

Having discussed some of the factors shaping phototalk and display behaviour such as form factor, interaction access points, location, curatorial control, and mechanisms for sourcing content we now move on to a discussion of the 4Photos concept and ways in which we have manipulated these factors to achieve a more democratic phototalk appropriate to a social gathering. DESIGN APPROACH AND CONCEPTS

As mentioned within the previous sections it was our intention to design a research vehicle for exploring photoviewing activities in social contexts and locations not currently supported by traditional displaying devices. To prevent ourselves from exploring an unlimited amount of different social settings we decided to specifically focus at the social context of having dinner. We were particularly interested in this social context as it offers family and friends to come together for social interaction and presents interesting challenges in relation to social dynamics of talk. To this end, we initially explored a variety of different ideas and concepts. For example, we explored a concept called “media jar” which stores pictures and sound clips captured by mobile devices carried around by family members. This media is triggered on lifting the lid providing a resource for conversation. In a second example concept, the “media vase” mobile phones are docked in the “vase” to create a multi-view synchronised slideshow on the phones‟ displays. A third related concept we explored was called “photo centrepiece” in which photo viewing takes place on a dedicated device in the middle of the dinner table (see Figure 1).

A final line of discussion we want to introduce concerns the sourcing of content and how this potentially relates to shifting dynamics of phototalk. We have already seen how new photowork mechanisms such as online sharing are altering the dynamics of participation in the domestic use of photographic representation [5]. Given this more democratic participation enabled through these mechanisms (e.g., social networking sites such as Facebook or dedicated photo sharing sites such as Flickr), of interest for innovation and behavioural understanding concerns the relationship between these new resources and repositories for photowork and subsequent display behaviour. Indeed how can such democratic participation in online photo sharing be exploited in more democratic photo displays within particular social contexts, as repositories for digital photo displays. That is how can people‟s online photo resources be combined together for presentation on a photo display to shift dynamics of curatorial control and phototalk. There are of course now commercially available photo display systems that can be linked to online photo sharing and social networking. But we have very little understanding of the behavioural consequences and social values of this. For example, it is yet to be determined whether content designed for a particular online audience

Figure 1. The photo centrepiece concept. Apart from presenting media content it could also have a decorative function.

54

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Design goals

within our iterative design process. After several design iterations, mainly focused on its interaction features and form, the process led us to the final prototype; “4Photos”. In the following sections we describe the iterative design process from the photo centrepiece towards 4Photos.

Through these conceptual explorations we were able to identify and refine particular design goals pertinent to our focal scenario. In light of these goals, it was felt most suitable to pursue the photo centrepiece concept in more detail. We discuss this further below.

Photo centrepiece

Non-obtrusiveness

The photo centrepiece prototype was designed to be positioned on a dinner table. A dinner table is designed to naturally configure people to gather for social occasions like dinner and drinks. The aim of the photo centrepiece was to exploit this natural configuration and provide a way for everybody around the table to view and interact with the artefact while maintaining natural gaze and interpersonal awareness (following recommendations of Lindley and Monk [12] whereby displays for photo viewing should afford good interpersonal awareness). To support this, the photo centrepiece prototype was composed of 4 vertical sides, each side framing a small portrait oriented display and an IR distance sensor (see Figure 2).

The aim was to let the technology not become dominant in the social context, but rather blend in the social dynamics of the moment. Especially given the fact that we focused on the social context of people having dinner, it was important to carefully take family values and dinner rituals in consideration. It was important too not to interfere with the physical assembly of artefacts found in this scenario (e.g. food dishes, plates). In fact, it was our intention to create new possibilities in order to enrich the social experience and band between people. Democratic control and participation

Embedding the artefact in new social contexts required a different approach to the form and interaction design in contrast to traditional photo-displaying products. We wanted the artefact to enable multi-user viewing and participation for family and friends and provide focus on the social enjoyment of the experience. As mentioned before, to offer new ways for democratic participation.

The artefact is to be placed in the middle of the table as one component in the artefact assembly making up a dinner table – much in the same way one might use a candelabra or flower arrangement as a dinner centrepiece. In this respect it needed to be a light and mobile artefact that would integrate easily with this artefact assembly. With this in mind we adopted a vertical display arrangement, which is more natural for people‟s gaze and mutual awareness during the social context of a dinner compared to a horizontal tabletop setup in which people look further down. Bearing in mind the additional criteria for flexible and rapid iteration, the photo centrepiece was developed as shown in Figure 2.

Easy access to new content

Based on insights from Kirk et al. [10] it can be concluded that people like to use recent content to spark conversation and storytelling. However, this work also reflects issues concerning the high threshold for downloading new content from a capture device. By focusing on an existing media repository, in our case Facebook, we prevented the introduction of yet another object to „maintain‟ or collect content for.

Interaction mechanisms

We implemented two interaction mechanisms in the prototype. The first used IR proximity sensors. Being positioned on all four sides of the object, these sensors could detect gesture and movement towards the displays from each side. The second interaction mechanism was a rotatable part on top of the prototype, the “head” (see Figure 2).

Random but related

Using online photo repositories we did not merely want to present content in a similar fashion as pre-arranged on these services. For this reason the object had to stimulate a more serendipitous experience, as also described in the work of Helmes et al. [7] and Leong [11]; wherein serendipity contributes to the quality of the experience. In order to achieve this experience but still maintain the possibility for „story telling‟, the artefact had to pick random sets of related images from any album within the repository. DESIGNING 4PHOTOS

Based upon the design goals and early physical explorations we iteratively designed towards a prototype for the photo centrepiece concept. This initial prototype was meant for exploration of the interface and a basic interaction with materials chosen to speed up prototyping rather than focussing on the necessary aesthetics for fit within the domestic environment. It maintained some flexibility to support exploration of different sensors and several interfaces and was robust enough for early user evaluation and allowed us to gather valuable input for the next steps

Figure 2. Picture of the photo centrepiece prototype with a rotatable part on top, called the “head”. Approx. size: 12x12x25. cm

55

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

spark curiosity and conversation amongst users (see Figure 5). Interaction was limited to a zoom function triggered by moving a hand towards the IR sensors (the spinning top part was not used). In an early test, during a family dinner, the object was perceived as very unobtrusive and passive as desired. At the same time though it was considered little more than a picture frame with moving images. One articulated reason being the fact that the random presentation of photos lacked any coherence to facilitate story telling or meaningfulness. A second reason was that the zooming function was not perceived as very useful - the size of all photos on each side of the object increased simultaneously, which simply made it impossible to highlight or talk about one particular photo. Finally, it was evident users wanted control in order to refresh the object‟s content.

Figure 3. Viewing angles of the object, top view 1 shows the original layout of the photo centrepiece. View 2 shows the new layout in which the object is stretched and displays rotated. 4PHOTOS FINAL DESIGN

Through initial testing of the photo centrepiece in tabletop configurations we discovered viewing angles of the displays were not optimal from a number of different directions. The unsymmetrical distribution of the viewing angle, did not lend itself well to typical configurations of people around the dinner table. To overcome this we changed the shape of the prototype (by slightly stretching the original shape) which led to the final form factor of 4Photos. We also rotated and slightly tilted the displays to create a homogenous viewing field (see Figure 3). In iterating towards a new prototype we also wanted to create a more suitable aesthetic for the domestic environment, drawing inspiration from other products and concepts fitting within this setting. This resulted in the final form factor prototype displayed in Figure 4.

Figure 5. Illustration of the first interface. Random photos move around across the displays Iteration 2: Moving and horizontal photostrip

With these issues in mind we introduced several changes. First, in order to allow users to select between content from different people they were able to rotate the head of the device to change the album from which photos were drawn. Second, we changed the zooming function to result in the image, on the side of the triggered IR sensor, to zoom in and push the same image to the other three screens– providing users with a coordinating function to highlight and talk about a photo. Third, we introduced an additional feature by maintaining a more structured and meaningful presentation of the photos within a “photostrip”, clustering multiple images from a single person. With this photostrip the four displays create a linked display surface which the photostrip moves around, facilitating storytelling by automatic and continuous cueing [8] (see Figure 6).

Figure 4. Photo of the 4Photos prototype with the rearranged displays, domestic aesthetic and rotatable top part. Interface design

We designed the interface of 4Photos in parallel with the physical form. The next sections describe the iterations and choices we made that lead to the “photostrip interface”. Iteration 1: Random photos floating around

The goal of this initial interface was to create a serendipitous photo viewing experience, displaying random photos of people gathered around 4Photos. These random photos appeared on the sides of the object and started to float around – the photos being different on all four sides to

Figure 6. The photostrip variation of the interface. The photostrip rotates around the 4 sides of the object.

56

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Iteration 3: Final improvements for testing

Group 4: Gathering of three female friends aged 40-50. The daughter and son of the host are also present, aged 12-14. The meal is held at the host‟s home.

After implementing the previously described features and adding the „spinning‟ interaction (using the rotating top part), we again tested the object within a similar social setting as during our first user involvement. Based on the results from the user test we decided to maintain the interaction with the photostrip. We tweaked the interface to facilitate a better photo viewing experience. For example, we changed the order of the photos in the photostrip. In the previous iteration, after a spin, the photostrip would start again from the beginning of an album. We observed that people simply did not want to wait each time for 4Photos to go through the same images again before seeing new ones. In the improved version, the photostrip comprises a new set of photos each time a different person is selected creating a more random selection of new images.

Group 5: Group of student friends aged in their 20s. One female and four males having lunch at their university department.

Furthermore, as we so far tested with pre-uploaded images; which quite often lead to a lack of content, we decided to source the object with content from Facebook. Apart from having a large collection of available photos from this source, the Facebook API also allowed us to setup a special friends list to allocate the object to the photo repositories of the participants in the evaluation. As mentioned in the beginning of this paper, using Facebook to source content into 4Photos seemed a plausible implementation. Not solely to prevent people from having an additional device to populate with media, but also given the fact that currently Facebook content often gets viewed in an individual setting. We were quite interested as well in the social implications when projecting this content back into a social setting. Amongst other findings we will go into more depth about this topic within the next section.

Figure 7. Group 2 gathered around the 4photos at mealtime.

For each occasion, one of the attendees would create a Facebook friends list consisting of those people who would be present at that particular meal. The online albums of these people from this friends list would source the device with photos for that meal. The device would be placed on the dinner table among the other dinner paraphernalia and food. After a brief instruction on the basic interaction mechanisms, the groups would simply get on with their meal. All sessions were video recorded for later analysis. Results

Based on our analysis of the video recordings and guided by the design goals described earlier, we divided the results into six themes. We will illustrate these themes using examples and quotes.

USER EVALUATION

In order to further our understanding of behavioural practices around the device we conducted a series of qualitative tests in real world settings and contexts. The device was deployed on five occasions with different groups who were getting together socially for a meal. We deliberately chose to conduct these tests with different groups where different social relationships might play a role, such as, for example friends, couples, work colleagues, parents and children.

Display as a conversational resource

Situating the display within the social context of these social gatherings indeed provided a significant conversational resource for those present. SY “It‟s such a conversation starter.” The nature of these conversations varied and was dependent upon the make-up of social relationships in the group. We saw examples of shared reminiscing among the close groups of friends and examples where the photographs were used as a means to get-to-know about less well-known members, e.g. the work colleagues on the following conversation snippet:

Group 1: Colleagues from a research department having a meal together after work. Five males and one female all aged between 20 and 30. Meal is held in the kitchen area at the office. Group 2: Gathering of friends all aged 20-30. Three are males and two females. Two of the group are married and live in a shared house with another of the males. The other two are a couple who are visiting. The meal is in the home of the married couple and third male (see Figure 7).

P “Ah – there’s my daughter” [one of P‟s pictures is displayed up] J “Oh I didn’t realize you had a daughter.” P “Yes I have two. X is one and a half and Y is three and a half.”

Group 3: Family meal with husband, wife and two young children plus one female visitor. The husband and wife are in their 30s, the children are aged four and six. The visitor is in her 20s. The meal is held at the family home.

In some sense, these are fairly typical examples of phototalk that might be elicited through other forms of

57

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

SY “you pick up on random things like fish – you would never talk about fish in normal conversations.”

photographic display and presentation. However, there are subtly different properties and values that arise from the particular form factor and contextual situation in which the object is immersed that seemed to facilitate this. Of particular importance to people was the fact that the photo pool comprised photos from all present and not just the host.

As well as using individual photographs as triggers, there were times too when conversations made reference to multiple photographs from one particular photo strip. The clustering of the photos from a single album, then, allowed a more linked storytelling to take place

T “It’s taking it to a different level – it’s about everyone rather than one person’s photo.”

Coordinating views: physically, verbally, interactionally

While having different views on the screens provided more opportunity for triggering conversation, there were times too when participants wanted to see the same thing. A number of compensatory behaviours were observed to support this and allow smoother conversation. First, people would adjust their bodily position and gaze to view an adjacent screen. These movements would be in response to conversational or gestural based references to particular photographs on other screens. Typically, the nature of the viewing angles meant that people could only view at most two screens through small adjustments in body orientation (e.g. through leaning over); though occasionally more exaggerated body orientations were used to see additional screens. When bodily adjustments were difficult, participants also engaged in verbal based coordination behaviours.

This shifts the dynamics of control in significant ways, distributing both responsibility and opportunity for conversational control away from a single person. The photographs offered a means for everyone at certain points to make contributions to the conversation. But it also offered a means for people to invite others into the conversation through their questioning about the photographs – such as those who may have been shy and less able to get involved. Intrusiveness of the display

It is important to understand that conversation at the table was not simply about the photo content. Often, what started as a particular reference to a photograph would then evolve into its own conversational strand. People could exchange similar experiences and anecdotes with both weak and strong associative ties to the content. Conversations took on their own momentum where attention would shift from looking at the display to more “normal” face-to-face conversational gazes. That is, the device shifted in and out of conversational context throughout the meals, which shows that it supports storytelling. These dynamics are an important consideration for assessing whether the device is intrusive to the setting or shifts dynamics to facilitate the setting. In our observations, there were times when the displayed photos were very much a focal point of conversation – in particular when the phototalk had strong links to the content. But, equally, there were times when the conversation was less closely tied to the displayed photograph – either because of loose initial associative triggers or because of the natural progression of the conversational thread. In these instances the device seemed more unobtrusive and merged within the background.

C “Oh look at the turtle.” CO “Oh I haven’t got the turtle.” C “The turtle is coming round now.” CO “I’m not on the turtle. I’m on the bubbles.” Here, the group work around the “what you see is what I see” principle, using verbally mediated pauses in more of a “What you are about to see is what I see” principle. People also used the zoom function to coordinate conversation and shared views. This paused the content momentarily allowing more elaborated stories or closer inspection and reference to specific photographic. But it was also done explicitly to simultaneously push a particular piece of content to all four displays. Typically, the initiator of the conversation, being in front of the relevant display when the triggering photo was visible, performed this action. The timing of the interaction was used to draw attention to a particular photograph. But, it was also conducted in response to “questioning” looks or bodily gestures (e.g. leaning) from other participants.

People‟s orientation to the acceptability of this conversational influence was not consistent among groups or the different social contexts. All groups enjoyed conversations arising from the device placement and felt it was a great social facilitator. A couple of participants, though, made reference to the potential of the device to dominate talk. For informal settings and relationships this was not typically an issue but one participant expressed it may be too distracting for a more formal dinner party because of over distraction. Similarly, there were times in the family meal where the children were felt to be paying too much attention to the device and not focusing on meal time. Also valued was how the device created opportunities for conversational topics that would otherwise be unlikely.

Another factor that facilitated coordination of conversation across multiple views was that photos shown at any one time were from the same Facebook album. This provided a common contextual relationship for the photos imposed within the particular Facebook profile (e.g. from the same event or holiday). So, even when a conversation referred to a photo not immediately visible to all parties, this was mitigated by the common context of the photostrip.

58

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Playing with visibility and invisibility

men to continue looking at the same picture set. A few minutes later when the men appear to have finished their discussion of the bike-related photographs SY asked: “Can we spin it or are you still…?”. M replied, “No no I think it’s done now”. SY responded, “Are you sure?”.

While having different images on each screen led to a need for coordination work, there were also playful and humorous behaviours arising from this. For example in Group 2, all the males were arranged on one side of the table with two females on the opposite side. Participant M said “This is where it pulls in content you didn’t want to show.” A picture of M then came up and male participant, T, pretended it was showing embarrassing content of M M‟s girlfriend, SY, being seated opposite. The men started teasing M and SY playing on the fact that SY couldn‟t see what they are viewing. “Who’s that you are with M – who’s that girl... Is that a strip joint you are in?”

In most cases, coordination of these multiple access points was successfully achieved and negotiated. Where it did present particular problems was when younger children were present. For example, the two young children in Group 4 were particularly taken with the interaction mechanisms themselves. As a consequence they continually tried to interact with the device with little awareness of its role within the conversation. Consider the following intervention by L, the mother of participant O:

In a second example from group 5, we see humour arising from ambiguity of conversational reference in relation to the displayed content. In this example, participant C pointed at the screen facing her, displaying a picture of her brother and said “Ah that’s my brother. Oh this is a really crap photograph of him really.” At this point, participant CO bursts out laughing and makes a reference to the image on her own screen saying “We’ve got Hugh Jackman round here. I should point out that’s not her brother.”

L “Don’t touch it because we haven’t seen these ones yet… You’ve got to stop!” O “But I like it” L“I know you like doing it but we want to see the photos… Was that a glass of cream or something?” K “It was a coconut.” We see here the repair work arising from O‟s inappropriate interaction, with references being made to an object that has gone by. So while the parents work around these issues through verbal references, there is some suggestion that more support for this repair work here could have been useful, the ability to bring a photo back that has passed by.

In these social contexts, then, issues of shared visibility go beyond matters of coordination, highlighting alternative sources of value and meaning making to consider in design. Managing multiple access points

The ability to interact with the device from all directions, be that spinning the head or the zooming sensors, opened up control to all. In all groups we saw interactions with the device by all present. The device was more inclusive in terms of availability of actions and resources participants had to steer and control the conversation. This differs from photo slideshows viewed on a laptop where affordances of the interaction mechanisms lend themselves to control by individuals in charge of the technology. Within this context of shared interaction and of concern to us here, were the ways access and control was managed through social protocols and also the conditions under which this broke down. With different images being viewed by multiple people at any one time, any change of view had to be negotiated with others for whom the photos were potentially a resource. Cues in conversational content and tone and attentional orientation to or from the device provided important cues in the timing of these interactions such that they could be performed with minimal social disruption. What was striking is how this mutual access and control was often conducted with seemingly minimal effort. But there were times, too, when more explicit attempts were made to negotiate access and control when there was conflict or insufficiently unambiguous cues from others. For example, in Group 2, the men were discussing a photo from a mountain bike-centric photo album. While looking at the photos, one of the women, SY, moved her hand to interact with the device. As she approached the device, her intentions became apparent prompting T to say „whoah whoah‟ - to stop her moving the photo on, as he wanted the

Serendipity and goal driven control

The random aspects of photo presentation provided important user value. People expressed joy and surprise as new photos came up. The uncertainty about which and whose photos would be next created anticipation among participants that was a source of fun. Indeed, a number of the groups compared the interaction to the game of “spin the bottle [a truth or dare type game]” and playful purposes. SY “I like the way it’s a lottery – who’s it going to land on. You could actually have drinking games with it couldn’t you. Every time it lands on you, you have to take a drink” Handing over some control over content to the system provided the kinds of values of serendipitous presentation highlighted in the works of Leong [10]. People were reminded of things they had forgotten: CO “It makes me think of all those albums that I forgot were on there.” Or things they had forgotten to tell people about: CO “It reminds you of things you forgot to tell each other” Giving control to the device also allowed the device to remain active during periods when participants were engaged in other conversation threads or activities such as eating. There were times, too, when people were more deliberate and purposeful in their interactions. In contrast to the large playful spins used to achieve random content

59

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

selection, slower controlled rotation of the device head was used to select specific people‟s photos or even a specific photo.

For example, in Group 2, one of the men had an album of mountain biking related photos. While these were of interest to the men, they were not for the women present. SY commented: “Why does it end up on bike porn?” Again, of significance is not that this caused any observed problems or frustrations, but how it reveals potential differences in uses and purposes of content in Facebook versus in the collocated setting. As such, Facebook needs to be considered in more subtle ways than simply a low effort way of sourcing content for the device.

ST “Can I find the one with the Octopus?” At other times, this deliberate control was to create more even distribution of content if it was felt someone‟s content was represented enough – thereby trying to include him or her in the conversation more. In one episode, SY spun it and it comes up with the same photo again – SY says, “Oh S you are popular” – then one of the men on the other side intervened and spun the top again to get new content. A short while later, S‟s photos appeared again so she nudged the device head to move to the next album.

At times the relationship between Facebook and the device became two-way. In general, people were happy with the low effort option of utilising existing Facebook albums. However, there were instances where participants put new content on their Facebook accounts specifically because they were going to be using the device. The device and scenario, then, created a new context of usage for Facebook content that shaped Facebook content management. People also spoke of content they would have liked to appear. One couple mentioned they would like to have seen their wedding photos on there as that would have been interesting for the gathered group. These early indicators of preparation activities for these settings suggest more imaginative and creative uses of Facebook might arise with longer-term usage. Of interest here was how people spoke of tagged photographs of themselves or the albums of others as a resource for their photographs. In this sense, people‟s photograph collections within Facebook were not seen as simply their own immediate collections. Rather Facebook was more of a distributed repository of photographs. As a work around, some participants suggested adding other people to the Facebook “dinner” friends list who were not present but who had relevant photo collections for those attending. For example, T said, “we should have put Millsy up there.”, making reference to the fact that “Millsy” had lots of good photos of their wedding.

Facebook content

In using Facebook photo albums as a photo source, our aims were to provide a low effort way of providing content; exploiting and re-using people‟s existing photo organisation work. While this was born out, a number of additional issues became apparent about the link with Facebook. One issue of note here is how the device changed the setting of consumption and the potential audience. These issues came back in user behaviour and conversation. For example, there were comments relating to uncertainty about which photos might pop up and if they would be appropriate. J “I hope there is nothing embarrassing on there.” The concerns arise, in part, because control settings placed on photos in Facebook do not always neatly map onto the collocated setting. The device is set up using a host member to whom the other guests are connected to on Facebook. But other members of the collocated setting may not necessarily be connected to each other via Facebook. Therefore, the audience for the photos in the collocated setting is potentially outside the bounds of control settings used for a particular Facebook friends list. These concerns of course are dependent upon the particular relationships that make up the social setting. With the above quote, the social gathering was with work colleagues where identity management may differ from that with well-known friends. We saw these concerns raised, too, in social groups with young children. The father in that Group 4 expressed a similar hope that no “dodgy” content would come up. The young children were not Facebook users - it is entirely plausible this would not be an uncommon feature of other social gatherings around the device. While there was little in the way of actual incidents of embarrassment during our observations, what is significant is how people revealed their orientation to these concerns.

CONCLUSIONS

In this paper, we described the iterative design and use of 4Photos, a collaborative photosharing device. Our aims with the device were to explore a new scenario of use of photo displays beyond the more traditional “wall and mantle piece” approaches of many current digital photo frames. We focussed on the creation of a table centrepiece that would be used during social gatherings around mealtimes. Within the context of this particular scenario, we wanted to explore a different set of social dynamics around phototalk by manipulating the material properties of the display device. In essence, we wanted to democratise the way that photos could be used in the construction of phototalk as part of a social gathering. We approached this issue in several key ways: making the device visible from multiple directions; by making interactions with the device available from all directions and available to all present at the table; and finally by making curatorial control over the content on the device more of a shared experience. Furthermore we wanted a device that augmented the

A related issue is with interest in particular content presented in the collocated setting. Within Facebook, content is sometimes posted that is of interest to a limited subset of people rather than a more general audience. As a solitary viewing experience, people choose which albums to look at in detail according to interest. However, the dynamics of such use changes in the collocated viewing.

60

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

mealtime experience and that fitted both with the natural assembly of other artefacts on the table and the natural configuration of people around the table. With these issues in mind in the design of 4Photos, we iterated through several different physical instantiations exploring different key aspects of the form factor. These aspects included the object‟s aesthetic qualities, its size, mobility and screen orientation. Similarly, we iterated through several implementations of the interface, focusing on key issues such as levels of control (active, passive use), structured vs. random presentation, playfulness, and the extent to which views were coordinated across the device. We further explored implications of sourcing content in different ways, in particular by exploiting photos from Facebook.

situated displays: Social and Interactional Aspects of Shared Display Technologies. Kluwer Academic Publishers, 2003, 422. 4. Crabtree, A., Rodden, T., and Mariani, J. Collaborating around Collections: Informing the Continued Development of Photoware. In Proc. CSCW 2004, ACM Press (2004), 396. 5. Durrant, A., Taylor, A.S., Frohlich, D., Sellen, A., and Uzzell, D. Photo displays and intergenerational relationships in the family home. In Proc. HCI 2009, British Computer Society (2009), 10. 6. Frohlich, D., Kuchinsky, A., Pering, C., Don, A., and Ariss, S. Requirements for photoware. In Proc. CSCW 2002, ACM Press (2002), 166. 7. Helmes, J., Hummels, C., and Sellen, A. The other brother: Re-experiencing spontaneous moments from domestic life. In Proc. TEI 2009, ACM Press (2009), 233. 8. Hoven, E. and Eggen, B. Informing augmented memory system design through autobiographical memory theory. Personal and Ubiquitous Computing 12, 6 (2008), 433. 9. Kim, J. and Zimmerman, J. Cherish: Smart digital photo frames for sharing social narratives at home. Work in progress CHI 2006, (2006), 953. 10. Kirk, D., Sellen, A., Rother, C., and Wood, K. Understanding photowork. In Proc. CHI 2006, ACM Press (2006), 761. 11. Leong, T. Understanding Serendipitous Experiences when Interacting with Personal Digital Content. PhD Thesis, University of Melbourne, 2009. 12. Lindley, S. and Monk, A. Social enjoyment with electronic photograph displays: Awareness and control. International Journal of Human-Computer Studies 66, 8 (2008), 587. 13. Lindley, S., Durrant, A., Kirk, D., and Taylor, A. Collocated social practices surrounding photos. In Proc. CHI 2008, ACM Press (2008), 3921. 14. Miller, A.D. and Edwards, W.K. Give and take: A study of consumer photo-sharing culture and practice. In Proc. CHI 2007, ACM Press (2007), 347. 15. O'Hara, K. Interactivity and non-interactivity on tabletops. In Proc. CHI 2010, ACM Press (2010), 2611. 16. Rose, G. Family photographs and domestic spacings: a case study. Transactions of the Institute of British Geographers 28, 1 (2003), 5. 17. Shen, C., Lesh, N., and Vernier, F. Personal digital historian: story sharing around the table. Interactions 10, 2 (2003), 15. 18. Shove, E., Watson, M., Hand, M., and Ingram, J. The Design of Everyday Life. Berg Publishers, Oxford, 2007. 19. Swan, L. and Taylor, A.S. Photo displays in the home. In Proc. DIS 2008, ACM Press (2008), 261. 20. Taylor, A., Swan, L., and Durrant, A. Designing Family Photo Displays. In Proc. ECSCW 2007, Springer London (2007), 7.

We conducted a qualitative user test with five different groups that provided more insight into how 4Photos was used in a variety of different social settings. It was evident that 4Photos functioned as a conversation starter, either because of the enthusiasm it evoked for people to tell about photos or the curiosity of others to find out more about them. It prompted people to find out about each other and exchange stories and also encouraged shared reminiscing among friends and family. Often the content also inspired people to talk about things they would never have talked about before; we ob-served how very mundane content sparked fun conversations. Apart from functioning as a conversational resource its interaction possibilities enabled a shared responsibility. By distributing its control and content sourcing, 4Photos shifted away from a more individualistic, single person interaction setup. This was further emphasized by the verbal and physical interactions that emerged around the different content on each display leading to alternative sources of value and meaning for the participants (e.g. jokes referencing content about to appear on other people‟s screens). The serendipitous presentation of content was also a source of fun for participants. We also started to identify interesting social issues regarding the relationship with Facebook as a source for content. It was apparent that using Facebook raised more issues beyond being a low-effort photo source. There were some interesting differences in the audience and context of use that emerged and that affected the way that 4Photos and Facebook was used. ACKNOWLEDGEMENTS

We would like to thank our colleagues from the CML group and in particular Abigail Sellen and Richard Banks. REFERENCES

1. Apted, T., Kay, J., and Quigley, A. Tabletop Sharing of Digital Photographs for the Elderly. In Proc. CHI 2006, ACM Press (2006), 781. 2. Chalfen, R. Snapshot Versions of Life. Bowling Green State University Press, Bowling Green, 1987. 3. Crabtree, A., Hemmings, T., and Rodden, T. The Social Construction of Displays: Coordinate Displays and Ecologically Distributed Networks. In Public and

61

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Two-Handed Input in a Standard Configuration of Notebook with External Mouse Florian Block and Hans Gellersen Lancaster University InfoLab21, South Drive, Lancaster UK {block, hwg}@comp.lancs.ac.uk ABSTRACT

notebook computers, motivated by the observation that many users use these with an external mouse instead of the built-in pointing device. The result is a configuration of notebook and mouse (“N+M configuration” hereafter) in which two spatial input devices are available, in principle, although in current practice the built-in device is left unused once an external mouse is attached.

This paper discusses two-handed input for interaction with notebooks, motivated by the observation that notebooks are often used with an external mouse. We present results of a survey of 905 notebook users, of which 63.8% reported occasional, and 47.0% regular use of a mouse instead of the built-in pointing device (a touchpad in 95.8% of the reported configurations). Based on this finding, we propose use of the built-in touchpad with the non-dominant hand when an external mouse is used as primary pointing device. We provide a systematic analysis of the input space of such a configuration, and contribute a set of techniques that specifically exploit touchpad properties for input with the non-dominant hand. These techniques include flick, scale and rotate gestures; absolute positioning with tokens; and touchpad use as key modifier. The techniques are demonstrated in a variety of GUI applications in a standard environment of notebook with external mouse.

The observed use of an external mouse with notebook computers indicates a preference of many users for a mouse over the built-in device, for primary spatial input performed with their dominant hand. We propose that the built-in device can then be used beneficially with the non-dominant hand. A first benefit is that the built-in device and the mouse can be used simultaneously for two-handed input techniques, with asymmetric roles that naturally follow user preference of mouse over built-in device. A second benefit is that the built-in device can be used for customised input techniques, complementary to “general-purpose pointing” covered by the mouse in an N+M configuration. Based on these considerations, we make three main contributions in this paper: empirical work surveying notebook use to provide evidence that N+M configurations are common; theoretical work analysing the input space of such configurations to identify interface design opportunities; and design work contributing and demonstrating novel touchpad interaction techniques for notebooks used with an external mouse.

Author Keywords

Input Device, Touchpad, Laptop, External Mouse, TwoHanded Interaction, GUI. ACM Classification Keywords

H5.2. User interfaces: Input devices and strategies; Graphical user interfaces. INTRODUCTION

Two-handed input for spatial interaction with graphical user interfaces has been widely studied, with experimental evidence of performance advantages and cognitive benefits [5,15]. In practice, however, most interfaces for desktop interaction continue to be operated with a single spatial input device, most commonly a mouse, alongside a keyboard. In this paper, we consider two-handed input on

We have conducted a survey of notebook use with over 900 participants. The survey confirms our observation that external mouse use is common and widespread, with about two-thirds of notebooks occasionally used with an additional mouse, and half of notebooks regularly used in an N+M configuration. The built-in pointing device is most commonly a touchpad, available in 95.8% of the reported N+M configurations, and consequently the focus of our consideration of complementary input with the nondominant hand. The survey also provides qualitative insight into user preference of mouse over touchpad or other builtin pointing device. Understanding why a touchpad is less preferred as primary pointing device is relevant for our consideration of their use in an alternative role, complementary to the mouse.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

62

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Based on the survey, we contribute an analysis of typical N+M configurations in which the input space is defined by the combination of mouse and touchpad, in addition to the keyboard. A pairing of mouse and touchpad has a precedent in work of Hinckley et al., where a mouse and an external touchpad were combined for two-handed tracking and dragging tasks [12]. However, in this work we look at the input design space of N+M configurations more generally, and contribute a detailed characterisation of N+M interfaces in terms of Card et. al.’s taxonomy of input devices [6].

inherently asymmetric, first in that the two devices have different and distinctive properties [4,12], and secondly in that users have an a priori preference for the mouse as primary pointing device. Guiard’s Kinematic Chain model is therefore of key relevance to our work as it provides theoretical insight into principles governing asymmetric two-handed interaction [8]. The model describes a distinct role for the non-dominant hand, in setting the frame of reference in which the dominant hand operates (“Right-toleft Spatial Reference in Manual Motion”). It further proposes that the non-dominant hand performs less finegrained movement (“Left-Right Contrast in the Spatial Temporal Scale of Motion”). In particular the latter point is significant for our purposes. Notebook users who prefer a mouse over a touchpad tend to do this because of (perceived) finer-grain control over movement (as evident from our survey, see below). Arguably, a (perceived) limitation in accuracy and speed will be less evident when two-handed techniques are designed such that the touchpadoperating hand performs less, and less fine-grained, movement than the mouse-operating hand. This corresponds with experimentally observed left-right-hand input performance, finding that the non-dominant hand is well suited for tasks that do not require precise action, such as scrolling [13].

Our theoretical analysis of the N+M input space gives rise to a number of novel interaction techniques that we contribute in the final part of this paper. The techniques we present are based on customisation of the touchpad, which becomes possible when its original function, cursor control in a graphical interface, is taken over by an external mouse. The techniques include flick, scale and rotate gestures, absolute input with a token on the touchpad, and touchpad use as an input modifier. These techniques are known in principle from other forms of touch interface (e.g., tablets, touchscreen, interactive tabletops) but contribute distinct new interaction styles when they are provided on built-in notebook touchpads. The utility of the techniques is demonstrated in a variety of applications that we have implemented in a standard GUI environment.

Two-handed techniques reported in the literature in which the dominant and non-dominant hand perform complementary roles can be broadly classified as: • Setting context: various studies of two-handed input directly apply Guiards’s reference principle by using the non-dominant hand for context-setting of graphical workspaces while the dominant hand performs finegrained actions (e.g., scrolling/selecting text [5], panning/rotating viewports for painting [15], and navigation in 3D design [22,24]). • Toolglasses: another use for the non-dominant hand in GUI environments is to move toolbars, palettes or menus to a point of interest, to bring them closer to where the mouse operates, or to resolve occlusions [14,15,17]. • Gestures: two-handed techniques have been demonstrated in which the non-dominant hands performs gesture input to select commands or modifiers in conjunction with spatial positioning [1].

RELATED WORK

According to sales trends, notebooks are on the brink of being more widely adopted for desktop interaction than traditional PCs, but the input configurations in which they are used have received only limited research attention. Prior studies include surveys of different user groups with respect to ergonomics of notebook use [10,21], comparative evaluation of built-in pointing devices [7,20] and ethnographies of notebook usage in the home [23], but we are first to present a survey that provides quantitative and qualitative data on adoption of an external mouse for notebook interaction. In spite of the evidently widespread use of an additional mouse for notebook interaction there has been no prior work considering use of the built-in pointing device for complementary or alternative input alongside the mouse. It is evident from other work on capacitive touch sensing surfaces over the last decades that touchpads can in principle support other forms of input, distinct from mousestyle pointing [4] and including gestures [1,18]. However, this potential has not previously been explored in the concrete context of notebook-integrated touchpads. An example though of providing a new technique based on a notebook touchpad is ThumbSense, where touchpadsensing is used as an implicit key modifier for emulation of mouse buttons on the keyboard [19].

In all of these cases, the dominant hand is used for finegrained positioning, which naturally maps to mouse operation in an N+M scenario, while the non-dominant hand performs actions that can be realised on notebook touchpads. The combination of touchpad and mouse has been previously investigated, however with a touchpad as external device [12], rather than integrated to be close to hand from the keyboard. It is important to note conceptual differences between mouse and touchpad in such a configuration, and that the devices operate in different kinaesthetic reference frames. However general techniques such as context setting and moving toolglasses can be adapted to work with relative input from both devices, or

The proposed “activation” of notebook touchpads complementary to an external mouse facilitates interaction with two continuous input devices, leveraging a large body of research on two-handed input. The configuration is

63

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

alternatively with the touchpad as absolute and the mouse as relative input device. Research also suggests that separation of kinaesthetic reference frames in two-handed interaction can be compensated with visual feedback [2].

N

%

1. What built-in pointing devices does you notebook have? Touchpad 867 95.8 Pointing Stick 51 5.6 Track Ball 16 1.8 Touch-screen 14 1.6 2. Do you ever use an additional external mouse with your notebook?

Two-handed desktop interaction in practice involves the use of keyboard shortcuts or hotkeys. Single hotkeys or a combination of modifiers and hotkeys (such as CTRL+C for copy) can execute application commands as shortcut for graphical widgets, and are known to generally save time compared to clicking GUI widgets [16,17], particularly when invoking functions from a menu [9]. Such interactions differ from the above examples of two-handed input as the keyboard is not a continuous input device, but they inspire consideration of how touchpads in an N+M configuration can support hotkey techniques.

Yes

577

63.8

No

328

36.2

3. How often do you use your external mouse with your notebook in the office / at home / while traveling? *

A SURVEY OF NOTEBOOK USE

Regularly

We have conducted a survey of notebook use with the primary aim to corroborate our observation that many users regularly use an external mouse in extension of their notebook configuration. In addition to quantitative data we sought to gain qualitative insight into preference for an external mouse over the build-in device, to inform the development of novel techniques that leverage N+M configurations.

Occasionally Never

338 / 393 / 83

58.6 / 68.1 / 14.4

110 / 165 / 134

19.1 / 28.6 / 23.2

115 / 46 / 359

19.9 / 8.0 / 62.2

* percentages based on number of users of an external mouse Table 1: Survey results on built-in poiting devices and external mouse usage (contains only selected items of the actual questionnaire).

The survey was conducted with an anonymous online questionnaire, to obtain data from a larger population. The questionnaire was designed to be short to maximize response rates. It contained 11 questions in 3 sections. The first section asked for basic demographic data. The second section assessed the user’s technical equipment (notebook, docking-station, built-in input) and their qualitative rating of built-in devices. The third section concerned the external mouse (where applicable), asked for a qualitative rating in comparison with the built-in input, assessed in which applications/situations the mouse was used, and also collected data on use of mouse features (mouse wheel, additional buttons etc.). The questionnaire contained multiple-choice questions and scaled questions, but users also had the option to enter comments.

questionnaire. Pressing submit would add the anonymous questionnaire data to our database. With a separate database we made sure, that users of shared IP addresses could not complete the questionnaire several times. We received a total of 989 submissions. Fifteen submissions were identified as invalid (either empty or corrupted). Another 68 participants were filtered out since they did not use a notebook computer (control question answered with ‘no’). There were no duplicate entries. 14 submissions were missing age, and 7 missed information on gender. However, all these surveys were otherwise complete, and thus included in the analysis. After filtering, we had 905 valid responses (10.1 % response rate). The total number of responses came from 964 different IP numbers. After an inspection of all datasets with a counter greater 1 we came to the conclusion that the datasets were unique, which is realistic given the fact that we expected shared computers / IP-addresses. Consequently we included these datasets into our analysis.

Sampling

Our sampling frame were students of a local university. Convenience sampling was then used to determine the final sample. A call for voluntary participation for the online questionnaire was advertised on the university mailing list, reaching a total of 8975 students, of which 5900 were undergraduate students, 1900 postgraduate master students and 1175 research students (MPhil/PhD). Ethical approval was granted by the university ethics committee.

Results

Demographics. Of the 905 participants 333 were male and 565 were female (7 participants did not provide their gender), with an average age of 23.0 years (s.d. 7.3y, youngest 17y, oldest 65y).

Data Collection

A campus-wide mailing was sent out to call for participation. The email was explicitly recruiting notebook users. Volunteers followed an anonymous link to the web questionnaire. The questionnaire was fitted on a single html page, with minor scrolling necessary to complete the

Assessing Built-In Input Devices. The by far most frequently found built-in input device is the touchpad (95.8%) followed by pointing stick (5.6%), track ball (1.8%) and touch-screen (1.6%, cf. Table 1). 2.2% of all

64

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

participants used a docking station and 3.0% use a notebook stand either in the office or at home. This is an important issue to consider, since both docking stations and notebook stands could render the built-in input impossible or difficult to use. Participants were also asked to rate their built-in pointing device in terms of efficiency, accuracy and comfort on a 5-point Likert-scale. In all three categories, the built-in pointing device received close to neutral ratings: efficiency scored 3.4 (s.d. 1.1), accuracy 3.6 (s.d. 1.0) and comfort 3.6 (s.d. 1.1).

This observation was confirmed by the explicit feedback users gave. Users were asked why they preferred the external mouse over their built-in input. They could choose either one of the preset choices (‘mouse is faster’, ‘more accurate’, ‘more comfortable’) and had the option to enter comments. 66.72% of all mouse users selected ‘faster’, 75.2% selected ‘more accurate’ and 76.1% selected ‘more comfortable’. The comments will be discussed in the next paragraph. Another item in the questionnaire assessed applicationspecific preference (“For which applications do you prefer to use an external mouse”). Participants had multiple choices between three basic categories (‘office’, ‘communication & information retrieval’, ‘entertainment’) and the option to enter specific applications in a text box. Of all external mouse users, 80.9 users selected ‘Office’, 61.5% ‘Communications/Information Retrieval’ and 77.0% ‘entertainment’. Users entered a variety of additional applications, mostly graphics software (such as Photoshop or CAD) but also video editing and sound composing.

External Mouse Usage. Of the 905 notebook users who participated in the survey, 577 (63.8 %) use an external mouse at least occasionally. An overview of mouse usage can be found in Table 1, items 2 and 3. Users were asked whether they made regular use of mouse in any three situations (in the office, at home, or while travelling). Of our total of notebook users, 47.0% are using an external mouse regularly in at least one of these settings. Of our subset of “N+M users”, 58.6% reported that they use the mouse regularly at the office, and 68.1% at home. Unsurprisingly, regular use is low for the mobile setting with only 9.2% (when travelling). Occasional use can be observed across all three settings (office: 19.1%, home: 28.6%, travelling 23.2%), including the mobile setting. This confirms that external mouse usage in a mobile context is less common, but it also shows that a considerable proportion of users carry a mouse with them when they travel.

User Comments. The comments provided by the participants of our survey gave more detailed insight into qualitative aspects of mouse and built-in input. There were 63 comments in response to ‘Why do you use an external mouse, instead of the built-in pointing device?’ Within these comments we found 24 occurrences of ergonomic aspects. Strain to the wrist was mentioned, as well as general fatigue over longer work-sessions. 4 users gave a more detailed response, stating the advantage of the external position of the mouse over the fixed position of the built-in input device. A conceptual disadvantage of the touchpad was mentioned, namely that ‘clicks’ could be accidentally triggered whilst typing. 20 participants mentioned the general ‘ease of use’ of the external mouse in comparison with the built-in input. Many of these comments included implied references to ergonomic factors (‘easier for the hand’, ‘mouse is bigger’) but also to accuracy (‘generally easier and more accurate’). 4 participants explicitly mentioned an improved efficiency with the mouse (‘quicker use’, ‘faster in certain situations’).

Mouse Features. External mouse users were asked which additional features they used. The multiple choices were mouse-wheel, additional buttons (such as the ones facing the thumb) and thumb-wheel (some mice have an additional jog-dial where the thumb rests). The most common feature used was the mouse-wheel (90.1% of all mouse users), followed by additional buttons (38.1%) and thumbwheel (16.6%). Qualitative Assessment of Input Devices. First, we analysed the user rating of the built-in pointing device. We grouped users into those who use an external mouse and those who do not. Our hypothesis was that users who use an external mouse would rate their built-in input device significantly worse than those users who are not using an external mouse. This would give us implicit evidence for the users’ motivation to use a mouse, which could later be matched with the explicit feedback we gathered. A Kruskal-Wallis Test was used to analyse differences of ratings for efficiency, accuracy and comfort. It confirmed our hypothesis, showing that there is a significant difference between our two groups, with p < .01 for all items, efficiency (χ2(1) = 68.2), accuracy (χ2(1) = 45.6) and comfort (χ2(1) = 80.2). Ranking in all three categories was lower (worse) for participants who use an external mouse. This indicates that by choosing an external mouse, users seek to improve all three aspects for pointer input.

A second set of comments (64 in total) concerned application-specific aspects. 24 participants stated that they use their external mouse across all applications. 20 participants highlighted the benefits of the mouse specifically for the graphics software, such as Photoshop, CAD or other design tools. Some users gave a more generalized explanation, related to the better ‘moveability’ or ‘manoeuvrability’. One participant stated: ‘[I use the mouse] for […] tasks which require more control of the cursor than the touchpad allows’. This was also supported by several statements that certain tasks are very difficult to perform with the built-in input, such as ‘creating diagrams’, ‘drawing pictures’, ‘creating graphics’ or ‘dragging and dropping’.

65

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Discussion

The survey confirms that the proportion of notebook users who use an external mouse is indeed large, with 63.8% of surveyed notebook users using a mouse at least occasionally, and 47.0% using a mouse regularly. Dockingstations or notebook-stands, which could compromise or impede use of built-in input devices for novel techniques, are used only by a small fraction of users (2.2% and 3.0%). The survey method (convenience sampling) and population (university students) imply that we need to be careful in drawing conclusion with respect to notebook use in general. However the sample size is clearly significant and the results are strongly indicative of N+M configurations being commonplace as a distinct interface for desktop interaction.

Figure 1: Input space of notebook configurations with touchpad and external mouse (using Card et al.’s taxonomy [6])

The results also show that the touchpad is by far the most common built-in input device in notebooks (95.8%). Consequently, many of our considerations in this paper are specifically focussed on interaction with touchpad and mouse.

absolute position of one finger on the surface (the number “2” within the squares applies only to multi-touch sensing touchpads). Additionally, touchpads can determine proximity to a certain degree and thus give an estimate about the distance between finger and touchpad (while this input is unreliable for higher distance, it is essential for determining the point of contact). Most touchpads also have two co-located buttons, similar to a mouse. Using connect compositions (double arrows) one can derive also relative movement from the absolute positions in x and y.

A consistent motive for using an external mouse was a higher quality of input, in addition to ergonomics. This has many factors, most importantly accuracy, speed and expressiveness. These factors are specifically relevant for tasks performed with the dominant hand, where tasks performed by the non-dominant hand, such as context setting and toolglass placement, require less accuracy and speed.

Newer generation notebooks (e.g., the Macbook Air) have touchpads that can sense at least two fingers at a time. Even though this is not standard yet, we include these capabilities as they are likely to become commonplace. With two points of contact some of the mentioned primitives double (symbolised by “2” within the squares). Additionally, several new primitives can be derived (pentagons). The angle between two fingers can be mapped to rotation (rZ) and the distance between two fingers can be mapped to a new linear value. Since the described taxonomy does not accommodate for distances or scalars we interpreted the distance as a new linear value (the absolute relation between two positions). Its place along the X column was chosen arbitrarily. Both absolute distance and angle themselves can be connect-combined to their appropriate delta-values (e.g., taking the delta between distances could be mapped to scaling an object).

THE INPUT SPACE OF NOTEBOOK+MOUSE

Our survey has established that N+M configurations are common, and that in most of these cases the ‘neglected’ built-in device is a touchpad. Our input space is thus characterised as follows: 1.

The mouse is the preferred input device for the dominant hand, in the role of pointer input

2.

The keyboard is used for text entry and hotkey access

3.

The built-in input device is unused and can be treated as third independent input device

Figure 1 shows the input space of N+M configuration using the taxonomy of Card et al. [6]. Each input device is represented by a number of input primitives that are connected with different composition operators (merge, connect and layout). The mouse consists out of three input primitives (symbolised by circles): relative movement across X and Y, and 3 buttons (the number in the primitive reflects multiple primitives of the same kind). Based on our survey results, we also include the mouse wheel (triangle), which delivers a delta angle around the X axis.

Note that in the original design space, connect compositions were described as very basic mappings from one domain to another. The conversion from two points of touch to distance / angle is more complex than originally described, since it combines multiple primitives of different kinds and maps them to a single primitive in another domain (aggregation). However, we feel this is still within the spirit of the taxonomy, as it helps capture the complete set of input primitives that a device, in this case the touchpad, can support.

The keyboard provides a set of buttons (~90) and is mounted on the same panel as the touchpad (described by a layout compositor, symbolised by a dotted line).

The description of the N+M input space provides a baseline for understanding and exploring interaction opportunities. For our specific purposes, it allows us to systematically

The touchpad (squares) provides a range of primitives. On the most basic level, a conventional touchpad delivers

66

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

analyse potential use of the touchpad, in itself, and in conjunction with mouse or keyboard. TOUCHPAD TECHNIQUES FOR N+M CONFIGURATIONS

In the following we introduce a number of interaction techniques designed to leverage the touchpad in N+M configurations. The techniques exploit that general purpose pointing is performed with the mouse, allowing use of the touchpad to be customised.

Figure 4: A coin can act as physical token for absolute positioning, for instance of a palette or toolglass [3].

operate the touchpad while the fingers are in the home row for typing, eliminating the necessity to acquire and use the mouse for scrolling (Figure 3). This can be useful for example for looking up information in another part of the document before continuing to write (e.g., when crossreferencing). It can also be used to simply move the current page so that the cursor is in the centre of the screen (context setting for convenience was observed in [15]). With a common mapping (Y movement to scrolling) navigating across large text documents causes significant strain for the thumb. However with flicking, we found that we could comfortably use the thumbs to navigate within large documents. This example shows that the fixed location of the touchpad is not necessarily a constraint, but can in fact be advantageous.

Flick Gestures for Context Setting

The touchpad is well suited for context setting, both for technical and conceptual reasons. Technically, it provides highly responsive data on absolute finger position and proximity as well as relative movement in two dimensions, allowing expressive panning across virtual workspaces. Its input capabilities are supported by the touchpad’s flat character, affording unique interaction styles, such as flick gestures that have become popular with the iPhone. Figure 2 illustrates how flick-gestures can be applied to context setting in map navigation. In this example, the touchpad acts as direct handle for Figure 2: Flick gestures used for an underlying map: once the finger touches the navigating maps. touchpad, any movement is mapped directly to the panning of the map. When the finger is flicked off the touchpad, momentum is induced into the virtual map causing it to start spinning in the direction the finger has flicked into. The motion stops either by applying virtual friction over time or by pinning the finger down on the touchpad, once the map is at the desired location.

Absolute Input with Touchpads

Various examples of two-handed interaction such as context switching or for moving Toolglasses use absolute input. While the mouse is not able to sense absolute position, the touchpad can provide appropriate input primitives by sensing the finger’s absolute location relative to its boundaries. When applied to a practical scenario, such as moving Toolglasses, several observations can be made. On the one hand, absolut input gives direct and fast control over the Toolglass since it eliminates the continuous ‘pushing and re-orienting’ of the finger as is typical for relative interaction with the touchpad when used as pointer. On the other hand, using the touchpad’s absolute input felt counter-intuitive for several reasons. First, the touchpad’s established role for pointer input suggests mainly relative movement across the flat surface, not pointing to absolute locations. Secondly, it is also conceptually different from other absolute input devices, such as the puck used in [15], in that the touchpad lacks a physical token that would reflect the current absolute position.

This example demonstrates how the touchpad – once we can specialise its mapping – can make use of inherent affordances when it is mapped to a dedicated task (“spinning the globe”). In our example, the user might even prefer the touchpad for panning the map over the dominant hand, since it provides a more specialised interface for the given task than the mouse. Flicking is also efficient for interaction in that it can reduce the amount of finger activity necessary to overcome long panning distances. Within text-editors, flicking can be applied to horizontal scrolling, enabling new interaction styles. The touchpad’s position close to the keyboard generally allows the thumb to

Figure 4 shows, how this problem can be overcome in a simple manner, by adding a coin on the touchpad. A coin amplifies the capacitive fingerprint and a threshold can be set such that the controlled object (e.g., a toolglass, palette, or menu) would only be moved when moving the coin. The use of a coin (or other token) extends the touchpad with an important affordance when used as absolute input device, as the token is intuitive to use and gives a permanent physical indication of where the object is in respect to the screen. An added benefits is passive tactile feedback when the coin hits the borders of the touchpad. Moreover, it is possible to

Figure 3: The touchpad’s location allows the thumbs to flick through documents while leaving the fingers on the home row.

67

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 5: Scale and rotate gestures on standard touchpads using the button as second point of contact.

distinguish tokens of different size by their capacitive footprint. Scale and Rotate Gestures on Single-Point Touchpads

New generations of touchpads allow the sensing of multiple points of contact and thus support gestures for rotating and scaling (cf. Input Space). This can be benefitial for tasks such context setting (zooming in and out, rotating the view) and object manipulation. However, most existing touchpads do not support this, and can only track a single point of touch. To address this limitation, we contribute a technique for scale and rotate gestures with a single point of touch combined with a touchpad button.

Figure7: Simultaneous context setting with the non-dominant hand and free-form selection with the dominant hand.

(e.g., CTRL, ALT) is more centrally placed and larger, and thus easier to acquire. Standard hotkeys are binary triggers, however in combination with the touchpad new possibilities arise. For example, when a hotkey has been selected, movement on the touchpad can be used to parametrise the selected command. Figure 6 illustrates this with selection of a Brush tool by pressing touchpad and “B”, followed by thumb movement on the pad to manipulate brush size. As this example shows, touchpads can support entirely new types of hotkeys, that can be invoked and parametrised by one coherent action.

Figure 5 illustrates these techniques, in the context of map navigation. By taking a button as position of the thumb when clicking, the finger can move around the thumb either to perform scale or rotate gestures. The modes can be separated, clicking either left or right button for exclusive scaling/rotating, or holding both buttons together to achieve a combined effect. While this is a workaround for the missing second point of touch on the pad, it also has distinct advantages as it provides more control about what transformation the user intents to perform (e.g., achieving only rotation without scaling is difficult to perform with the dual touchpad, because the distance between two fingers would have to be kept constant).

APPLICATIONS

We have implemented several applications and applets that demonstrate the interaction techniques we designed for N+M configurations. The first one, a Virtual Desk application, is custom-built and directly taps into raw pointer input from the operating system, with global mouse hook to suppress all mouse messages and then implement our own GUI that can separately process touchpad and mouse input. The other applications extend existing software (such as Microsoft Word and Adobe Photoshop), with a background process that uses an API (Synaptics) to receive touchpad input separated from the primary pointer. Touchpad events are processed partly via injecting simulated mouse and keyboard events (e.g., simulating the mouse wheel for scrolling in Word), and partly via application specific API (e.g., the Photoshop COM API).

Hotkey Invocation and Parametrisation

Hotkeys are binary actions triggers and can be used as shortcuts for invoking applications via graphical interface elements such as icons or menus. Hotkeys often consist of a modifier and key, such as CTRL+C, allowing overloading of character keys with added functionality. The touchpad can add interesting aspects to hotkey interaction, as shown in Figure 6. When it is conceptualised as a single button (which is either touched or not touched), then it can be used as a modifier, that in contrast to conventional modifiers

Virtual Desk

This application was developed to provide a simple interface for distributing and organising large amounts of digital documents on a virtually infinite desk. Documents, such as PDF files can be simply dragged onto the virtual desk and are displayed by a stack of paper depicting a snapshot of its cover page. The large size of the virtual desk allows a user to spatially arrange and organise documents across large areas of virtual screen real-estate without introducing occlusions. Since as a result the virtual desktop

Figure 6: The touchpad can be used as a key modifier for hotkey selection combined with parametrisation in one action.

68

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

significantly exceeds the actual screen, frequent adjustment of the viewport is necessary, to navigate between distant documents, and when documents are selected and moved between distant locations on the virtual desk. For this purpose, the touchpad is mapped as a dedicated interface for controlling the viewport, while the mouse can simultaneously interact with objects, such as performing selection or dragging.

playback without leaving their typing position. As an alternative we have mapped audio control to the touchpad with a thumb tap for ‘pause’, a double tap for ‘play’ and slide with the thumb to ‘rewind’. The advantage of this technique is that it allows accurate playback-control, keeps fingers in typing position and does not require an additional device to be attached to the system. PhotoshopMagicLense

The touchpad interaction enables flick gestures for rapid panning across the table (cf. Fig. 2) as well as shrink / stretch gesture to set the zoom level of the viewport (cf. Fig. 5). This gives distinct advantages over the alternative of mouse and keyboard input. Figure 7 shows a typical problem that arises within virtual viewports: the user starts a free-form selection realising that not all documents that should be included in the selection are shown in the current viewport. Normally, once the mouse reaches the boundaries of the window, automatic scrolling is applied. However, since the mouse is in an active dragging state, scrolling automatically causes the mouse to leave a trace, which in our scenario could lead to the inclusion of unwanted elements that unexpectedly come into view. With the touchpad, the viewport can be independently adjusted with the non-dominant hand while the mouse can select revealed items in a controlled way. This is also superior to view adjustment via the keyboard, since it offers continuous spatial input instead of only binary controls.

We have implemented a toolglass for photoshop that can be positioned with the non-dominant hand. Positioning was realised in absolute terms by mapping touchpad input directly to a corresponding screen position. However the natural inclination of users for movement of the toolglass was to apply relative movement as familiar from pointing. To address this, we introduced use of a physical token into this scenario as described above. The token is intuitively perceived by users as representing the toolglass, and accordingly positioned and moved on the touchpad. DISCUSSION

A number of specific design issues were identified during the implementation of our techniques. First of all, the mapping of sub-task to either touchpad or mouse is not always clear. For instance, in the VirtualDesk application we mapped panning and zooming to the touchpad. On the one hand this enables the user to adjust the zoom level of the viewport while dragging or selecting objects. On the other hand, it is then impossible to zoom and pan at the same time, which might be a preferable mapping, for instance when searching for a particular object. An alternative mapping could use the mouse-wheel for zooming, which would enable simultaneous zooming with the mouse and panning with the touchpad. In general, this motivates a more systematic analysis of how the presented input primitives can be mapped to different subtasks. It is also interesting to consider, switching of mappings during usage and how the user could actively chose bimanual mappings that are most suitable for their current task.

EarthPad

Earthpad is a plug-in for Google Earth that uses the touchpad as direct handle for exploring the globe with flick&spin and zoom&rotate gestures. This applet demonstrates how application-specific mappings can be implemented on a touchpad to provide intuitive interaction styles to existing applications. ThumbScroll

ThumbScroll was implemented as an applet that enables touchpad scrolling for text editors. Similar to Virtual Desk, this technique can make selection of text across several pages easier, by performing scrolling with the nondominant hand while simultaneously selecting text with the dominant hand. Thumb-scrolling can also be used to navigate the document without leaving the home position for text input. The applet demonstrates customised use of the touchpad in the context of text editing, taking advantage of its fixed location with respect to the keyboard.

The implementation of concrete applications also brought out issues that we had not seen covered in prior work on two-handed input. Specifically, viewport adjustment with the non-dominant hand during a freehand selection with the right hand (as shown in Figure 7) affects the selection directly. The effect can be likened to moving a piece of paper while drawing on it, and in our application we found that this sometimes led to the selection of unwanted objects, One solution would be to move the mouse pointer during panning, so that it would retain its absolute location on the virtual canvas but this would have the disadvantage that the mouse pointer can get lost out of view. This example demonstrates that there are potentially unresolved issues when applying two-handed interaction to concrete tasks in practise.

TranscribePad

For transcribing it is necessary to listen and control recorded audio while typing textual content into a word processor. These sub-tasks are closely entwined, since the user often has to interrupt typing and pause or rewind the audio-stream. Traditionally, the function keys are used to provide access to audio playback, but this requires the fingers to leave their typing position. External devices, e.g. foot pedals, are common to allow users to control audio

An interesting general insight from this work was that the touchpad, while found to be only second best for its original purpose (i.e., pointing/dragging), exhibits many beneficial

69

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

aspects when it is customised for tasks it has not originally been designed for (i.e., gestures, absolute input, mode control). The techniques we have demonstrated are effective in leveraging distinctive properties of touchpads. This includes the responsiveness of the transducer to gestures, the flat physical layout enabling certain styles such as flicking, and the comprehensive set of input primitives, such as absolute input.

we have shown how the touchpad, in such a configuration, can be ‘activated’ for alternative input, complementary to pointing. We have contributed techniques that facilitate two-handed touchpad+mouse techniques on a ready platform of notebooks routinely used with external mouse, and in addition introduced distinctly novel uses of the touchpad for thumb interactions, absolute input, and parametrised hotkeys. The utility of the contributed techniques has been demonstrated through concrete applications.

Our survey had brought out qualitative issues concerning the use of touchpad’s as primary pointing device. However concerns related to accuracy and efficiency become less pronounced in the context of tasks performed with the nondominant hand, such as context setting and toolglass positioning. Ergonomic issues would also be expected to be factor in less, as interactions with the non-dominant hand would be less demanding in asymmetric tasks. Furthermore, while the touchpad’s fixed position has been highlighted as negative aspect for pointer input, some of our applications demonstrate that this can be advantageous for other forms of interaction. In this context, we see particular potential in thumb interaction since it provides access to the touchpad while the hand is in the common posture of resting on the keyboard. The thumb has the touchpad in immediate reach and can realistically perform a wide array of tasks. This is interesting, considering that both thumbs commonly serve the sole purpose of pressing the space bar.

REFERENCES

1. Balakrishnan, R. and Patel, P. 1998. The PadMouse: facilitating selection and spatial positioning for the nondominant hand. Proc. CHI’98, 9-16. 2. Balakrishnan, R. and Hinckley, K. 1999. The role of kinesthetic reference frames in two-handed input performance. Proc. UIST '99, 171-178. 3. Bier, E. A., Stone, M. C., Pier, K., Buxton, W., & DeRose, T.D. (1993). Toolglass and magic lenses: The see-through interface. Computer Graphics Proceedings, Annual Conference Series, 73-80, New York: ACM SIGGRAPH. 4. Buxton, W., Hill, R. & Rowley, P. (1985). Issues and techniques in touch-sensitive tablet input, Computer Graphics, 19(3), Proc. SIGGRAPH’85, 215-223. 5. Buxton, W., Myers, B., A Study in Two-Handed Input, Proceedings of CHI’86: ACM Conference on Human Factors in Computing Systems, 1986, 321-326. 6. Card, S. K., Mackinlay, J. D., and Robertson, G. G. 1991. A morphological analysis of the design space of input devices. ACM Trans. Inf. Syst. 9, 2 (Apr. 1991), 99-122. 7. Fallot-Burghardt, W., Fjeld, M., Speirs, C., Ziegenspeck, S., Krueger, H., and Läubli, T. 2006. Touch&Type: a novel pointing device for notebook computers. In Proc. NordiCHI’06: 4th Nordic Conference on Human-Computer interaction (NordiCHI’06), 465-468. 8. Guiard, Y. (1987). Asymmetric division of labour in human skilled bimanual action: The kinematic chain as a model. Journal of Motor Behaviour, 19, 486-517. 9. Grossman, T., Dragicevic, P., Balakrishnan, R. Strategies for accelerating on-line learning of hotkeys. In Proc. CHI 2007. 10. Harris, C. and Straker, L. Survey of physical ergonomics issues associated with school childrens’ use of laptop computers. Industrial Ergonomics, Vol. 26, Issue 3, September 2000, 337-346 11. Hinckley, K., Pausch, R., Proffitt, D., Patten, J., & Kassell, N. (1997). Cooperative Bimanual Action. Proceedings of the CHI’97, 27-34, New York: ACM. 12. Hinckley, K., Czerwinski, M., Sinclair, M., Interaction and Modeling Techniques for Desktop Two-Handed

We have focused on interactions with the touchpad, since it was found to be the most common built-in input device in our sample of N+M users. However, other input devices, such as pointing sticks or trackballs can also accommodate new interaction styles. Trackballs, for instance, have unique affordances that could be applied for instance to rotating objects (such as the globe). Also, our survey found notebook configurations that comprise both a touchpad and a pointing stick. This can give rise to new interaction styles that combine both devices (e.g., thumb on touchpad and finger on the pointing stick for rotate and scale gestures); or that allow switching between navigation modes (absolute navigation with the touchpad, and relative navigation with the pointing stick). Also, emerging multi-touch pads have the potential to support further interaction styles. CONCLUSION

Two-handed input and touch-based interaction techniques have been the subject of HCI research for a long time. The significance of this work is that it considers two-handed input in the context of a concrete interface configuration of notebook with external mouse and built-in touchpad. As foundation for our work, we have contributed quantitative and qualitative data collected through an online survey of notebook use. Through this survey we have evidence for the widespread use of notebooks in the above configuration. The touchpad is a highly expressive input device in terms of the input primitives it provides. Intriguingly though, the touchpad is left completely unused in configurations where an external mouse is attached to a notebook. In this paper

70

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Input, Proc. UIST’98 Symposium on User Interface Software & Technology, 49-58. 13. Kabbash, P., MacKenzie, I.S. & Buxton, W. (1993). Human performance using computer input devices in the preferred and non-preferred hands. Proc. InterCHI ‘93, 474-481. 14. Kabbash, P., Buxton, W.& Sellen, A. (1994). TwoHanded Input in a Compound Task. Proc. of CHI '94, 417-423. 15. Kurtenbach, G., Fitzmaurice, G., Baudel, T., and Buxton, W. (1997). The design of a GUI paradigm based on tablets, two-hands, and transparency. Proc. CHI’97 Conference on Human Factors in Computing Systems, 35-42. 16. McLoone, H., Hinckley, K. and Cutrell, E. Binamual Interaction on the Microsoft Office Keyboard. In Proc. Interact, 2003. 17. Odell, D.L., Davis, R.C., Smith, A. and Wright, P.K. Toolglasses, marking menus, and hotkeys: A comparison of one and two-handed command selection techniques. In Proc. Graphics Interface 2004, 17–24 18. Rekimoto, J. 2002. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proc.

of CHI’02 Conference on Human Factors in Computing Systems, 113-120. 19. Rekimoto. ThumbSense: Automatic Input Mode Sensing for Touchpad-based Interactions. In Proc. CHI’03 Conference on Human Factors in Computing Systems, 852-853. 20. Sawin, D. A., Stewart, A.M. and Calcaterra, J. A. (2002): Pointing Stick versus Touch Pad: Working Together. Proc. HFES 2002 Annual Meeting, CSTG Bulletin, Vol. 29-1 21. Sommerich, C. A Survey of Desktop and Notebook Computer Use by Professionals. Proc. HFES 2007 Annual Meeting, Industrial Ergonomics, pp. 1124-1128 22. SpacePilot. 3Dconnexion. http://www.3dconnexion.com/3dmouse/spacepilot.php 23. Woodruff, A., Anderson, K., Mainwaring, S. and Aipperspach, R. Portable, But Not Mobile: A Study of Wireless Laptops in the Home, Proc. Pervasive 2007, Toronto, Ontario, May 2007, 216-233. 24. Zeleznik, R. C., Forsberg, A. S., & Strauss, P. S. (1997). Two pointer input for 3D interaction. Proceedings of 1997 Symposium on Interactive 3D Graphics, 115-120.

71

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Tenori-on Stage: YouTube As Performance Space Mark Blythe School of Design Northumbria University, UK [email protected]

Paul Cairns Department of Computer Science University of York, UK [email protected]

ABSTRACT

created in different modes and users can play the device by flipping between blocks or performing other manipulations such as key transpositions.

This paper reports findings from four related studies of the “Tenori-on” as it appears on YouTube in order to consider Web 2.0 as a performance space. A quantitative analysis of returns for “Tenori-on” attempts to model how posts achieve and maintain popularity. This analysis suggests sustained posting and engagement amongst users rather than initial product launch enthusiasm. A content analysis of the videos returned demonstrates a very different response to the launch of other technologies like the iPhone 3G. A grounded theory explores comments to the most viewed video returned which was a post by the artist Little Boots. A range of comments indicate virtual applause and suggest that YouTube has been appropriated here as a space for performance. Finally perspectives from critical theory are drawn on to consider the meanings of the Tenori-on in this user generated context and the ways users creatively resist the most obvious affordances of the device.

Figure 1: Tenori-on. Photo by Gabriel “Gab” Pinto. Creative Commons

The designers were particularly concerned with the beauty of the light and sound, the ease of performance and the quality of the product as a whole [29]. There was an explicit aim to unite form and function in a digital instrument:

Author Keywords

Tenori-on. YouTube. User Criticism, Critical Theory

Experience,

Interaction

“In days gone by, a musical instrument had to have a beauty, of shape as well as of sound, and had to fit the player almost organically. Modern electronic instruments don't have this inevitable relationship between the shape, the sound, and the player. What I have done is to try to bring back these [...] elements and build them in to a true musical instrument for the digital age.” [29]

ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

The Tenori-on is a musical instrument designed by Toshio Iwai and Yamaha. It is part sequencer and part synthesizer with an interface made up of a grid of LED buttons which the user activates to make loops of sound (see figure 1). A line of lights pulses across the grid indicating bars of musical time. When one of the buttons on the grid is pressed it lights up and a note plays as the pulse passes over it. The user can create patterns of recurring notes to make loops that form single blocks. Up to sixteen blocks can be

The Tenori-on was first made commercially available in Britain in 2008 to test the market before a wider global launch [36]. This study began as an attempt gauge responses to the instrument through the videos returned for a search on the term “Tenori-on” on YouTube. Although the returns offered numerous insights on the user experience of the device they also indicated how YouTube is being used as a performance space. When this study began the most viewed return was a video of Iwai himself demonstrating the device at a launch event. During the study it became a video posted by Little Boots. The returns for “Tenori-on” remain relatively dynamic as the next section will indicate.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

Modeling the Data

Results for the search term “Tenori-on” on 7th July, 2009 produced a typical outcome. The results page suggests that there are over 1,100 hits for this search term. However,

72

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

when the results pages are trawled for the view counts, it becomes clear also that the results are no longer sorted by view count with view counts in the 360th to 380th videos varying wildly from several tens of thousand to zero. Thus we only model hits retrieved up to the 360 th video.

or secondly, an existing video is viewed in proportion to how much it has already been viewed. This latter behavior leads to the power law demonstrating a “rich get richer” process. In many ways, this process makes a lot of sense in the YouTube context. People may search for videos across a wide range of criteria but generally they will view the more popular ones. Thus the rich videos do get richer. But also, popular topics are also likely to get more videos on that topic and hence there is a steady growth in the number of videos to be viewed.

When sorted by view count, it is clear that whilst the top few videos have view counts of several hundreds of thousands, this rapidly tails off but in such a way that there is still a “long tail” of viewed videos. There are several well understood long tail distributions that produce this shape, the two simplest and most common being the exponential distribution and the power law distribution.

Why then is there not a proper power law distribution for view counts? One possibility is that though topics do grow, there is also a “shelf-life” to the topics and interest in adding videos to a topic falls away. Whilst popular videos continue to be viewed, there are diminishing returns in putting up a new video as the likelihood of getting viewed under the Yule-Simon process becomes vanishingly small. There is some evidence that YouTube viewing behavior follows a Yule-Simon process where the richer get richer but the uploaders get bored.

The log-log plot of view count against the rank when ordered by view count is shown in Figure 5. Whilst the initial part of the plot is linear, it is clear that there is a tailing off from a straight line with a particular down turn just as the search results run out. This suggests that a power law distribution is a good starting point to consider view counts. Very similar results have been seen in searches over other terms such as “Shadow of the Colossus” [12] and “iPhone 3G” [8] where a rapid drop off or knee is more readily apparent.

Evidence against this comes from what might be termed the degree of churn in YouTube videos. With a Yule-Simon process, there is little churn – once a video hits the top, it stays at the top. However, in many other power law situations there is a lot of churn [5] where the top 10 or top 100 ranked items vary enormously and that is exactly what is seen in YouTube. In a rich gets richer process, how do the richest ever get supplanted? Although many topics such as iPhone 3G or Shadow of the Colossus could be expected to have a shelf life strictly related to the release date, the Tenori-on is an instrument which might have appeal beyond the initial novelty. Sorting returns by the date of the upload in the week beginning 14 th September, 2009, there were 15 new videos. Sorting returns by the previous month showed that there had been 52 new uploads. This is comparable to the overall monthly average of 1,200 Tenori-on hits in the two years since its launch in Britain. This indicates a community of users continuing to keep a topic active and dynamic long after an initial launch. What then is this community doing?

Figure 2. A log-log plot of view count against rank for the hits from search “Tenori-on” with best fit line for a power law.

This shape of distribution in YouTube returns has been noted before by Cheng, Dale and Liu [14] also with a much more distinctive knee. The videos considered there were not the results of a particular search but rather obtained from a substantial trawl of all YouTube content. They propose, as others have, that the tail better fits a Weibull or Gamma distribution, each of which have a distinctive knee shape. Whilst this may be the case, this does not provide much insight into what processes drive viewing behavior. However starting with the power law distribution it is possible to consider possible processes that might lead to the observed view count distributions.

CONTENT ANALYSIS OF THE FIVE HUNDRED MOST VIEWED RETURNS

A content analysis [28] was performed on the first five hundred returns for the search “Tenori-on” sorted by most views on the 11th June 2009. Content analysis involves categorizing each video against pre-existing codes and counts the number of videos in each category. Figure 2 shows percentage returns for each of the categories.

The power law distribution was first accounted for by Yule [41] and then simplified by Simon [33]. It corresponds to a steady state situation where the rate of increase in view counts is proportional to the number of view counts each video has already achieved. This can be generated by a step by step process where at each step two things can happen. First, there is a small probability that a new video is added

73

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Coulter then attempts to play it and a long silence follows ended by audience laughter. In a post of the British singer Little Boots playing Glastonbury there is a similarly awkward silence as she picks up the device “hang on…” she says to laughter. After a technician arrives and fails to fix the connection she decides to play something else and turns to her synthesizer [23]. When Little Boots plays a keyboard the Tenori-on is sometimes placed on top of it and remains in shot as she sings. In band settings however it is raised up on a stand like a music stand. Its lights are lost in the stage lighting and it is not an integral part of the performance [26]. Similarly, a Bjork performance featuring the device begins with a close up of one of the backing musician holding it but it is mostly not visible in the wider stage shots [7].

Figure 3: Content analysis on first 500 returns for “Tenori-on”. Composition

Songs

The vast majority of returns featured people playing the device but there are distinctions to be made in terms of how they use it. Almost forty percent of the returns featured an original composition. Here users had composed pieces of music on the device and either set it to play a sequence of loops and blocks or directly manipulated the instrument to change blocks manually. Occasionally these Tenori-on compositions were accompanied by video effects or lightshows created with movie making software like iMovie. These were entirely instrumental pieces but often featured more than one instrument e.g. drum machines, synthesisers, compressors, effects units and other electronic instruments. Interestingly there was only one video featuring the Tenori-on and a non-electronic device; “Tenori-on meets sax” [27].

Although there are a great many videos featuring original compositions on the Tenori-on, they vary dramatically in quality as the comments below them readily illustrate. “Com4jai” like many others posted a video called “my first tenori-on song”. “Fewasdr” comments: “I could play that better by randomly slapping my penis on it” [18]. More kindly “utlewis” notes “you suck at this, no offence”. “Lionomega” asks “yes but where is the song?” (Ibid). The sub category of songs represents a wide variety of material and quality. Little Boots is a singer and song writer who came to prominence partly through the popularity of her videos on YouTube and MySpace. Unlike the majority of posts featuring instrumental compositions Little Boots is among the minority of Tenori-on users on YouTube to post songs. Just 4% of the returns featured anyone singing a song with the device. However these posts were amongst the most popular returns of all. Of these the most popular, and indeed the most popular “Tenori-on” returns of any kind at the time of this analysis were the posts by Little Boots. The Little Boots songs featured innovative cover versions as well as original compositions. The videos are usually static shots of the singer sitting in a bedroom and playing either the Tenori-on alone or a combination of Tenori-on with other instruments e.g. piano and stylophone. These will be returned to in later sections.

Performance

The performance category often also featured original compositions but these were performed in public rather than filmed in a bedroom or studio. Some of these performances featured well known performers like Bjork using the device, these were a small minority however. Most of the perfornances were by DJs or musicians in small venues. Many of these videos were shot at launch events or conferences and some feature Toshio Iwai himself playing the device before an audience of academics. It is interesting to note that the reception of such videos changed over time. The early comments on Iwai‟s performance are largely positive but later posts declare that he “sucks” and people should go and listen to “Little Boots”.

Words and Pictures

A small number of posts used the device primarily to create graphic displays of words like “Truth” or “Happy New Year”. Others used it to create simple pictures like an apple or animations of Eygptian hieroglyphic people dancing. A small percentage of the videos were demos of rival products such as iPhone apps with similar grid like interfaces and functionality (e.g. PakSound, Soundgrid, EasyBeats or Sequence). These offer similar grid-based interfaces but cost one or two dollars rather than sevaral hundred. Other minority posts were videos on a possible recall of the device because of an electrical fault, there were also two interviews with Iwai.

A small number of videos feature the device in a concert setting where it was not the primary focus. Musicians were playing a set and produced the instrument for one or two songs. Typically the performers apologized and joked about it. “I have to activate the doomsday device, hold on one second… You’re all going to feel a stinging sensation and then you’re going to be blind” [16]. At another concert the same artist tells the crowd “You may have noticed a flashing device, it’s a Tenori-on, it’s a …well, it’s a flashing device, I’m going to flash you with my device”.

74

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Review and demos

pizza box. The only graphic on it is a very simple line drawing of the device on the front. It is certainly not going to win the sort of awards that Apple‟s packaging designers are accustomed to receiving. But perhaps the reason for the lack of unboxing videos is that the users were thinking of it primarily as an instrument rather than a new gadget.

Although the device was played in demonstrations the focus was in showcasing different modes rather than composing music. Some of these featured musicians who had been given the device to play something on it for the purpose of promotion. One series of videos made by Yamaha specifically demos each mode of the device in a step by step guide.

The overall shape of the data then is quite different to that of returns on searches like iPhone. There are very few reviews, there are almost no unboxing moments. The vast majority of posts are musical performances of one kind or another.

There were remarkably few reviews, indeed only one in the first five hundred returns. This is extraordinary considering that returns for searches on other new devices like the Apple iPhone return such large quantities of reviews [8]. Devices like the iPhone and even individual apps like twitter clients receive detailed reviews which often focus on aspects of usability (ibid). Only one such review was returned for the Tenori-on and it was posted by a specialist technology webzine publication. It is possible then that the YouTubers who are posting are reading the device more as an expressive instrument rather than a piece of functional technology. Don Norman famously notes that nobody would expect a violin to be usable [30] perhaps this is also the case for electronic instruments like the Tenori-on.

The next section will consider in more detail the most popular return at the time of writing: the Little Boots video “ready for the fun” [24]. ANALYSING YOUTUBE’S NUMBERS

YouTube now provides audience demographics based on the information gathered from users with YouTube accounts who have logged on. According to the site statistics at the time of writing the “Ready for the fun” video was most popular with males from 35 to 44, followed by males aged between 45-54 and males aged between 2534. It was viewed most often from Britain, where Little Boots is from and where the Tenori-on was first made commercially available. That the video should be most popular in Britain is perhaps no surprise, that the video should be so much more popular with men than women is perhaps less predictable. Other Little Boots videos, such as some of her performances at Glastonbury are most popular with 13-17 year old girls.

Satire

There were also remarkably few humorous or satirical posts. Of the few that were returned some might be considered fairly unsophisticated forms of humour, one simply features a Tenori-on spelling out in lights the word “cock”. But others were more subtle, one, for instance, was a clip from the IT crowd where IT support worker Roy is shown wearing headphones and enthusiastically nodding along to the Tenori-on he is playing. When another character walks in he puts it away without comment and the audience laughs at this clear demonstration of Roy‟s essential geekiness. An even more direct satirical attack on the device comes in a demonstration of an “unplugged” Tenori-on. Kentaro Fukuchi‟s video begins with a sign noting that the Tenori-on is very cool but also rather expensive [21]. It goes on to show him making an “unplugged” version consisting of four vinyl pipes and a piece of bubble wrap. Fukuchi then plays the instrument for our listening pleasure by popping the bubbles on the bubble wrap attacking not only the price of the device but the sound quality.

The timeline in Figure 4 also supplied in “statistics and data” section goes some way to explaining what is happening:

Unboxing

Of the first five hundred videos only one featured a Tenorion unboxing. Unboxing videos accounted for over twenty percent of the first hundred most viewed videos for a search on the iPhone [8]. Unboxing videos are a very interesting YouTube phenomenon, it has been claimed that they provide an empirical justification for theoretical constructs of user experience which emphasise the importance of anticipation (Ibid). Why then should there be so few unboxing videos for the Tenori-on? Were the users not excited about opening it? The Tenori-on packaging consists of a fairly rough cardboard box something like a take away

Figure 4: YouTube “Statistics and Data” for little boots READY FOR THE FUN!!!. The letters in Figure 4 show the dates at which the video was embedded in sites like MySpace or referred from Google searches. It is likely that after being embedded in

75

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the MySpace site other demographics than the middle aged males searching for “Tenori-on” would have viewed the video on YouTube although they may not have logged in as users.

clearly been removed although traces of flames wars about them still remained. Figure 5 shows the percentages of comments assigned to each code.

YouTube claims that there are more or less equal numbers of males and females on their site [40]. There are several Tenori-on groups on Facebook and the most popular one has four hundred and ninety one members. Of these only around fifty are female. New technologies are often marketed specifically at males so it is possible that this explains the overwhelming gender bias in the audience for the most popular “Tenori-on” return. GROUNDED THEORY

On 28th July 2009, the most viewed return for the search “Tenori-on” was “little boots READY FOR THE FUN hotchip Tenori-on cover” with 399,627 views, 1,520 ratings with an average of five stars and 808 text comments [24]. The video is a single take performance of a song shot from a fixed camera. An attractive young woman in a black vest is sitting in a darkened room next to a keyboard with some cluttered shelves just visible behind her. She is lit by a strong light from screen left creating a chiaroscuro effect and also providing a strong visual platform for the white lights of the Tenori-on. In real time we see the artist setting a bass line in one block. She then creates another block with a synth pulse and another with a minimal drum pattern. Further blocks are added with synth pad sounds making up a chord sequence. Finally she adds a single voice triplet of lead notes. It is a virtuosic and clearly carefully rehearsed performance. There are almost no mistakes or mis-steps. Rather than begin the video with the Tenori-on part already created she performs its creation from scratch.

Figure 5: Comments on Little Boots video The praise was most often a textual form of applause “Bravooo” “A-MAZE-ING!!” “wow”. These comments often contained onomatapeic attempts to represent the sound of cheering “yay!”, “Whoo!”. These brief sound based responses indicate the ways in which the YouTube audience respond to the post as a performance. Just as they would clap at a live event they use text to applaud. These types of praise occasionally became direct address “whoop whoop herd u on radio 4 womans hour !” some followed praise with a question for the artist “What do you call the sound effect at 4:31 ? Bare foot slapping on bedroom oak effect wood floors?”. Although there are no responses from Little Boots there is a sense of direct communication with the artist in these posts. Some included requests for her to play their home town or recommendations of music or other technologies she might like. Many of theses posts were genral forms of approabation “you rock!” but others focussed on specifics such as her skill using the device, her voice, or her hair. Her hair was mentioned specifically in 12 posts, many more commented on her general beauty.

When the Tenori-on part is complete she begins singing and uses the transpose function to shift the entire sequence to create a different progression. By transposing the blocks and moving between them she creates a very complex and dynamic backing for her vocal which is showcased at one point by dropping the Tenori-on out altogether while she sings acapella. She ends the track by clearing each of the blocks until only one two note pattern remains. Finally she swirls her hand over the Tenori-on making a pattern and a random fall of notes.

Beauty

Many of the comments regarding her beauty simply noted that she was a “hot babe” . Others commented on the combination of looks and talent “what a babe. what a electronicmusician...”. Two of the comments were explictly sexual “oh shit i snapped my boner!!”. Many more were more emotional responses. “Simplely....I in love” Some of these declarations of love were followed by optimistic requests like “be my girlfriend”. Or “can you be my internet girlfriend”. Other emotional responses included more direct accounts of how the video made them feel, for example “sweet. made me feel good” or “goosebumps all over”. Often comments related to “sweetness” and cuteness “aww”.

A grounded theory analysis was performed on the comments for this video. Grounded theory begins with data, rather than pre-existing categories. Open codes are developed to summarise the data, these are then grouped together and linked in axial coding, the final stage of selective coding involves the selection of typical quotes to illustrate the “theory”. Theory here may refer merely to a broad description or set of categories rather than a fully worked predictive schema [13].

Technology

Praise

The comments that were not some form of praise for the artist‟s looks or talent were often praise of the Tenori-on.

The vast majority of comments on the video were in praise of Little Boots, indeed, some negative comments had

76

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

“intimate” peformance then funcioned very well as a marketing device for the more fully produced commodity in production.

Seventeen percent of the comments specifically referenced the technology. Some of these were questions about what the device was often followed by explanations from other viewers. Some were comments about the innovative way in which Little Boots was using the Tenori-on by people apparently familiar with the device “It must be in the choice of Tenori-On functions; key changes most important, the beepy-boopy Bounce mode probably least important.” This comment is interesting because it picks up on the innovative use of key change in the performance. Of the five hundred videos sampled very few make use of the key change button in this way and Little Boots is the only performer to use it to structure verse / chorus changes.

Intimacy on YouTube: the limits of ratings

The seeming intimacy of the close up performance allows the users to feel engaged in an emotional dialogue with the artist. The intimacy of the setting was occasionally contrasted to other media “Amazing. These solo versions on YouTube actually sound better than the mega-overproduced tracks on your EP. Your minimalist instrumentation lets us hear your great tunes and voice better, I think. Rock on girl” The video appears somehow more authentic, it is described in another comment as “acoustic”, another writes “Good on you for releasing wee home recorded vids like this, theres something very intimate about it, and that is never a bad thing”. The setting and the performance are taken as modest and somehow genuine.

A number of comments specifically reference the other Tenori-on performances on the site “You're the only one I have found so far who knows how to use the thing a write music with it. Everybody else seems like random noise.” One viewer noted that this was the best piece of advertising so far for the Tenori-on. Two viewers, however, posted jokes about it “cool...pong is back!!!” similarly “I like how she sings and can play space invaders at the same time :)” The slightly dismissive tone of these posts indicate a pleasure in “geeky” technology being used in this way “it's nice to see a girl so into synth-pop geekery doing so well.” Part of the orgininality being enjoyed here is the innovative use of the instrument which is in marked contrast to the way it is usually used, even by its inventor.

The star rating system offers a strictly numerical form of expression. It is interesting that the text boxes are used to emulate sound, noise and more intimate forms of expression such as the fan letter. The responses indicate that automated features such as rating are important but perhaps limited as forms of expression. FINDINGS SO FAR

Even if models could offer a compelling account of patterns of popularity (rich get richer plus decay over time) this would not tell us why one video rather than another is more popular. If it were simply the case that the rich got richer this still does not explain how they get rich in the first place. Clearly “Ready for the Fun” is returned for other searches, e.g. “Little Boots” and picks up more views and achieves a most viewed ranking, But this does not entirely explain how an artist and a technology can capture the public imagination in the way that Little Boots did. The next sections draw on critical theory to consider the meanings of the Tenori-on in the Little Boots video.

Criticism

Although abusive comments had been removed their traces remained in the flames sent by fans defending Little Boots against her detractors. “I think we all feel sorry for you” [the person who had been criticising Little Boots] because you are obviously unhappy with life. Don't you know what pop music is about? Its about trends and styles and the latest thing so do yourself a favour an chill.” It seems that the removed comment had slated Little Boots as trivial or trendy. Some direct criticism remained however. Some of this criticism, like much of the comments on YouTube in general is homophobic and mysoginistic “she is actually like all other chicks, she went on x factor and didn't get through.” Other posts dismissed the video as a “gimic” or expressed resistance to coverage in other media announcing her as “the next big thing.” Three of the posts expressed fears that the authenticity on display in the video would be swallowed by the music industry “I will still like your art after you've been over exploited by the music Industry!” Others contrasted this “amateur” performance with the artificiality of music industry products: “I really like it. Is it now the trend to go against what the industry says.” One viewer on the other hand felt that she could be great with some help “from the pros.”

CRITICAL THEORY

Critical theory is a field which encompasses many perspectives. It draws on literary studies, psychoanalysis, linguistics, cultural and media studies as well as philosophy [20]. As computing technology penetrates still deeper into everyday culture, perspectives from critical theory are increasingly relevant to studies of HCI [e.g. e.g. 1, 6, 19, 31, 32, 3, 8]. Critical theory is primarily concerned with interpretation and meaning. But critical “readings” are always open. There could never be a final reading of a poem, play, or any other cultural artifact. Nobody could plausibly claim to have finally discovered what Hamlet means. The meanings of cultural artefacts are never singular, they are always multiple. For this reason critical theory includes perspectives which are not only different from one another but sometimes completely opposed. Readings such as those by the cultural critic and philosopher Slavoj Zizek are “imagined hermeneutic

The references to other media were primarliy discussions of appearances by the artist either in concert or on television. A large number also referenced her debut CD “Hands” with promises that they would buy it. The “acoustic” and

77

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

understandings” [44] or provocative interpretations and make no empirical claim. For example Zizek claims that British toilets combine both inspection (water) and fast disposal (flush) indicating inclinations towards empiricism and pragmatism. This is not to claim that this is what their toilet designers were aiming at nor is to claim that this is what most Britons think while they are on the loo. The following sections then are likewise interpretations which make no empirical claim but rather, supplement the more empirical studies which precede it.

is skillfully composed. The sound quality is particularly good and the mix of sound levels between voice and instrument has clearly been given serious attention. Although the bedroom setting suggests an artless or naïve performance Little Boots is clearly very conscious of the way she is presenting herself.

Tenori-on as Commodity Fetish

Perhaps the most obvious interpretation of the Tenori-on on YouTube is that it exemplifies perfectly commodity fetishism. When Marx first formulated the notion of commodity fetishism he was concerned to indicate the ways in which commodities are figured as mysterious or magical objects “endowed with special powers” [44 p. 300]. This can be seen in the YouTube responses where delight and wonder are mixed equally with speculation about how much the device costs. This wonder explains perhaps the impulse to post videos of the first composition: look at this magical and mysterious object which I own.

Figure 6: Still from Little Boots MEDDLE bedroom version

That our technologies are now as much about fashion as our clothes is such a commonplace that it is increasingly challenged, for example Charlie Brooker, “anyone who thinks their phone is an expression of their personality hasn‟t got one” [11]. Indeed the adoption of a fetish is not about simple identification. Stonewashed jeans are supposedly associated with a working class lifestyle but they have always also been worn by the upper class. For Zizek it is in the gap between what something is supposed to represent and how it is actually appropriated where meaning is made; for this reason it is appropriate then that today‟s most popular clothing stores are called “the gap” [42]. In order for us to adopt a technology there must also be a gap. As previously noted many of the videos featuring the Tenori-on in performance show the performers joking about the device and their own geekiness: “I know very well that this is ridiculous, nevertheless…”

It is possible that she usually sits around in her bedroom wearing hot pants like the ones in Figure 6 but it is more likely that the performance is very thoughtfully staged. It is also clear that these performances are far from spontaneous. They are not the improvised patterns submitted by the majority of Tenori-on users posting to YouTube. Rather they are carefully planned and rehearsed performances. In other words the Tenori-on is far from the only reason this video is popular. In Lacanian terms the Tenori-on might be thought of as “object small a” [42]. This is a difficult Lacanian concept that Zizek illustrates with reference to the films of Alfred Hitchcock. Every Hitchcock film features a “MacGuffin” some plot element that motivates the characters. What the MacGuffin is does not matter, it serves merely to drive the characters forward, it might be a stolen microfilm, a plot by enemy spies, a lost love, a murderer – it could also be ambiguous [38].

However there were almost no unboxing videos in marked contrast to most other technology releases. This indicates perhaps that the device is either something more or something less than a commodity fetish.

The Tenori-on in Little Boots videos also functions in this way. It literally drives the video forward signalling both beginning and end, it provides a motivation for making and watching the video. Here is Little Boots playing this strange new instrument. The MacGuffin is a surplus of meaning, an excuse for self presentation.

Tenori-on as MacGuffin

The comments on the Little Boots video indicate that the technology is not the only thing being admired and desired. Of equal if not much more interest is Little Boots herself. It is likely that the Little Boots videos are not as artless as they may appear. John Bowers has remarked that “unplugged” performances are in fact always plugged. If an acoustic performance is to be recorded and broadcast it must be miked up and amplified just as carefully as an electronic instrument would have to be (Bowers, personal communication). In each of the Little Boots posts the shot

The Little Boots videos are far from typical of most of the Tenori-on returns. They are not the improvised patterns submitted by the majority of Tenori-on users posting to YouTube. They are quite unlike the compositions where users create spontaneous, solely instrumental patterns by exploiting the most obvious affordances of the device. What then can be said about the particular use of the device

78

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

in the music rather than the video? The next section turns to another perspective suggested by critical theory: technological determinism.

instrumental compositions and perhaps explains some of its power and popularity. TENORI-ON STAGE

As with previous studies drawing on YouTube, the medium is as important as the context of the posts [8] The majority of posts featured users making music of one kind or another with the device. In this sense, these users appropriate YouTube as a venue for performance. While Iwai may have partially achieved his goal of making an instrument for the digital age, challenges remain in terms of providing a space for performance. As previously noted, when the device is used in a concert setting (e.g. 7, 26) it is lost in the space and stage lights. It is at its most arresting and beautiful in a small setting like a bedroom, or indeed a TV studio, with a close focus on the device.

Tenori-on and Technological Determinism

The ease with which a complete beginner can create a pleasing sound with a Tenori-on was one of the features that were heavily promoted when the device was launched [36]. The affordances of the Tenori-on are towards patterns which even first time users create unwittingly as soon as they light up one of the buttons. The resulting patterns of music that might appear intrinsically modern or at least twentieth century. The music of Philip Glass is intensely patterned and repetitive. Minor variations in patterns lead to very complex pieces but for the uninitiated listener they can sound unbearably repetitive if not mechanically looped. Similarly dance music is a form which in the last twenty years has made an art out of manipulating loops of sound. Again, to those who are not familiar with the genre much of it can sound alike: “Call that a beat, they had proper bass in my day.” Likewise, the music which accompanies video games is often deeply patterned. Of course the current generation of video games can play any form of music but early games had to generate their themes “in house” and with very limited computational resources. Early video games like Sonic the Hedgehog featured soundtracks composed and played entirely on computers. They were as patterned and repetitive as any avante garde piece or dance track.

In some ways YouTube is a better performance space for the Tenori-on than a concert hall. In the bedroom videos both device and performers can be seen and appreciated. Beyond texting crude substitutes for applause the audience can, to a degree interact. There are several Facebook groups for the Tenori-on, and its members exchange tips about the device. It is not difficult to imagine a dedicated Tenori-on social networking site which would facilitate not only sharing recordings and videos but also perhaps collaborations. The ongoing uploads suggest a community of users who are interested in each others‟ work as well as their own. Such a Tenori-onTube might also provide more imaginative ways for viewers to applaud than the basic YouTube textbox, e.g. sound or graphic presets. As jazz clubs provided a space to develop new forms of music for new instruments like the saxophone perhaps a dedicated site might help the development of a Charlie Parker for the Tenori-on.

Wikipedia offers a range of definitions of technological determinism, here is one; „The idea that technological development determines social change' [39]. Although it is an extreme position it is hard to deny in every respect. The act of turning to Wikipedia for a definition rather than a book is evidence of the almost immediate impact of new technologies. But in a sense the Tenori-on provides a counter-example to technological determinism. The patterned music it produces existed long before it did. The patterns of music in Bach which could theoretically repeat ad infinitum were the dominant form signaling religious concerns with eternity. It was not until the Enlightenment that composers like Beethoven began to write linear music with identifiable beginnings and endings. Patterned music then is nothing new. In this sense the message preceded the medium.

Such a site might rank uploaded performances by views as with YouTube but it might also be interesting to allow users to categorise the submissions themselves. User-based categorization might be constructed as a content analysis where users could assign pre-existing categories (e.g. genre). More radically an open coding such as that practiced in grounded theory might allow for more creative and surprising patterns of user categorization to emerge. DISCUSSION: ALL THE WEB’S A STAGE

Many musicians are beginning to use YouTube as an intimate performance space with a range of instruments. In Reggie Watts‟ video “I just want to” the artist uses only a microphone with a sampler and mixer to produce an entire song on the fly. He begins by recording samples of “beat box” vocal percussion and sung bass lines which he then loops and mixes with instantaneous recordings of backing vocals and live lead vocals. Watts creates every part of the song in a single take and performs the whole piece in real time [37]. The virtuosity is astonishing and would be lost outside this intimate YouTube “bedroom” staging. But YouTube is a stage for other kinds of digital performance.

And as Little Boots demonstrates, there is more than one way to adopt a new technology. Part of the pleasure of the “ready for the fun” video is the virtuosity of Little Boots‟ performance and her mastery of the instrument. Little Boots‟ use of the device to produce a recognizable song “ready for the fun” by the electro-pop band Hot Chip. Little Boots‟ careful, controlled and skilful use of features like the transposition function illustrates the ways in which the most obvious affordances of the device can be creatively resisted. It is this creative resistance which, amongst other things, sets the video apart from the more general slew of patterned

79

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Players of games like Shadow of the Colossus archive particular performances of game play [12]. And there is increasing attention to amateur multi-media such as machinima and mash ups [2] where YouTube and other host domains are performance spaces.

demonstrated models of popularity based on the rich getting richer within a limited shelf life did not hold. Further regular uploads demonstrated a community of users that continue to be engaged with the device. The qualitative content analysis demonstrated that responses to the Tenorion were quite different to those to gadgets like the iPhone indicating perhaps that the device is being taken seriously as a musical instrument. The grounded theory of the most popular return also indicated that the community found the device almost as compelling and beautiful as Iwai hoped. Finally perspectives from critical theory helped articulate the multiple meanings of the device as well as the notion of YouTube as performance space. YouTube offers unprecedented glimpses into the lives of users and the ways in which they adopt and adapt technology. Observation and description of these data are not enough, there must also be interpretation. Of course, in HCI there will always be base level questions of functionality, usability and acceptability but increasingly there is the further problem of meaning.

The British theatre director Peter Hall argued that all that is necessary for an act of theatre to occur is for someone to walk across an empty stage while someone else is watching [10]. It is clear that many, if not all, of the postings to YouTube are performances but the interface is seldom considered as a stage. The posts are also films yet film theory is seldom referenced in HCI. Zizek is perhaps the best known cultural critic and theorist of film alive today and yet his appearance in an article on YouTube may seem bewildering. As HCI begins to concern itself with issues like aesthetics it is essential that the field engage with the arts and humanities [15]. As Jeffrey Bardzell has argued it is simply not the case that there are no definitions of what we mean by “aesthetics” as is sometimes claimed, there are standard textbooks on it. As Web 2.0 technologies are appropriated for creativity and performance, traditions of thought more readily associated with literary and cultural studies become increasingly relevant. There is then a growing recognition that researchers who ignore relevant work in the humanities and social science risk repeating previous work or revisiting previous philosophical dead ends [15, 9]

REFERENCES

1. Agre P. Toward a Critical Technical Practice: Lessons Learned in Trying to Reform AI. In Bowker G., Star S., Turner W., and Gasser L., eds, Social Science, Technical Systems and Cooperative Work, Erlbaum, 1997. 2. Bardzell J. (2007) Creativity in Amateur Multi-Media: Popular Culture, Critical Theory and HCI. Human Technology Volume 3 (1), February 2007, 12–33

This paper combines very different forms of analysis from social science and the humanities. Of the various forms of analysis directed at the Tenori-on on YouTube in this paper the critical theory will appear most alien and disreputable to a traditional HCI audience. George Sokal famously hoaxed a journal of cultural studies into publishing errant pretentious nonsense [34]. Since then it has been easy to sneer at the humanities. But scientists should not forget how often their own journals are duped by charlatans [22]. There is a lot of bad critical theory which is easy to laugh at but then there is also a fair amount of bad science which can be equally amusing [22].

3. Bardzell, J. Interaction criticism and aesthetics. In Proc. CHI 2009. ACM Press (2009), 2463-2472 4. Bardzell, S. 2010. Feminist HCI: taking stock and outlining an agenda for design. CHI '10. ACM, New York, NY, 1301-131 5. Batty, M. Rank clocks. Nature, 444 (2006), 592-596 6. Bertelsen, O., Pold, S. Criticism as an Approach to Interface Aesthetics. In Proc. NordiCHI 04, ACM Press (2004), 23-32 7. Bjork. Bjork ~ Who Is It? (Live in Paris ~ from Voltaic) (feat. Tenori-On). http://www.YouTube.com/watch?v=dFsHuu0ah_g

Critical theory is an umbrella term for a collection of sometimes mutually exclusive approaches to interpretation. The perspectives available within it are not methods but styles of thinking. The interpretations of the Tenori-on on YouTube here are examples of one perspective – psychoanalysis. They not suggested as definitive readings, there could be no such thing. There are a range of other perspectives (feminism, dramatism, performance theory) that would also be relevant. As interactive technology penetrates further into every aspect of our lives and culture the value of these different styles of thinking is becoming clear e.g. [4].

8. Blythe, M. and Cairns, P. Critical methods and user generated content: the iPhone on YouTube. In Proc. CHI '09. ACM, New York, NY, 1467-1476 9. Blythe, M., McCarthy, J., Light, A., Bardzell, S., Wright, P., Bardzell, J., and Blackwell, A. 2010. Critical dialogue: interaction, experience and cultural theory. In CHI EA '10. ACM, New York, NY, 4521-4524 10. Brook P. (1984) The Empty Space. Atheneum. New York 11. Brooker C., (2007) Dawn of the Dumb: Dispatches from the Idiotic Front Line. Faber and Faber. London

CONCLUSION

12. Cairns, P., Blythe, M. Research Methods 2.0: doing research using virtual communities. In Zaphiris, P., Ang,

This paper has reported findings from four related studies of the Tenori-on on YouTube. The quantitative analysis

80

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

J. (eds) Social Computing and Virtual Communities, Chapman & Hall, 2009.

28. Mayring P. Qualitative Content Analysis in Flickr. In Kardorff, U., Steinke, E. eds. A Companion to Qualitative Research. Sage, London, 2004.

13. Charmaz K. Constructing Grounded Theory. Sage, London, 2006.

29. Nishibori, Y. and Iwai, T. TENORI-ON. In Proc. SIGGRAPH 2005, ACM Press, 2005.

14. Cheng, X., Dale, C., Liu, J. Statistics and social network of YouTube videos. 16th Int. Workshop on Quality of Service, IEEE London (2008), 229-238.

30. Norman, D. A. Human-centered design considered harmful. interactions 12, 4 (2005), 14-19

15. Cockton, G., Bardzell, S., Blythe, M., and Bardzell, J. 2010. Can we all stand under our umbrella: the arts and design research in HCI. CHI EA '10. ACM, New York, NY, 3163-3166

31. Satchell C. Cultural Theory and Real World Design: Dystopian and Utopian Outcomes. In Proc. CHI 2008, ACM Press, 2009, 1593-1602 32. Sengers P., Boehner K., Shay D., Kaye J., Reflective Design (2005) Critical Computing Aarhus Denmark pp 49-58

16. Coulton, J. My Monkey w Tenori-on Pt 8. http://www.YouTube.com/watch?v=diu875wE8Ls 17. Coulton, J Jonathan Coulton plays Code Monkey live at Dingwalls London. http://www.YouTube.com/watch?v=hCZeb83i2DE

33. Simon, H. (1955) On a class of skew distribution functions. Biometrika, 42(3-4), 425-440

18. com4jai My First tenori-On song. http://www.YouTube.com/watch?v=eWV46e8Z4Wk

34. Sokal, G. (1996), Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity. Social Text 46/47 217-252.

19. De Souza C. The Semiotic Engineering of HumanComputer Interaction. MIT Press, Cambridge, MA, 2005.

35. Strauss A. and Corbin J. (1998) Basics of Qualitative Research: techniques and procedures for developing grounded theory. Sage Publications, London.

20. Easthope, A., McGowan K., eds. A Critical and Cultural Theory Reader, OU Press, Milton Keynes, 1992.

36. Walker, T. (2008) This gadget rocks! The world's newest musical instrument. The Independent 5 March, 2008

21. Fukuchi, K. Tenori Off http://www.YouTube.com/watch?v=ZNrV8LjRcRU

37. Watts R. (2009) I just want to. YouTube http://www.youtube.com/watch?v=344OpaQCAQI

22. Goldacre, B. Bad Science, 4th Estate, London, 2008.

38. Wikipedia, MacGuffin http://en.wikipedia.org/wiki/MacGuffin

23. Little Boots Little Boots at Glastonbury Festival Pt 1. http://www.YouTube.com/watch?v=YCdQriYrXA8

39. Wikipedia Technological Determinism http://en.wikipedia.org/wiki/Technological_determinism

24. Little Boots little boots READY FOR THE FUN!!! hot chip tenorion cover. http://www.YouTube.com/watch?v=N6tLRCDqJ2c

40. YouTube YouTube Fact Sheet http://www.YouTube.com/t/fact_sheet

25. Little Boots little boots MEDDLE bedroom version acoustic on piano, tenorion and stylophone oo la la. http://www.YouTube.com/watch?v=Zcc8gE54Md8

41. Yule, G. U. A mathematical theory of evolution, based on the conclusions of Dr. JC Willis, FRS . Phil. Trans. of the Royal Society, Ser. B, 213 (1925), 21-87.

26. Little Boots Little Boots performs 'Remedy' at the John Peel Stage Glastonbury Festival 2009 http://www.YouTube.com/watch?v=drFeVGVJ3Zs

42. Zizek S. Looking Awry: An Introduction to Jacques Lacan through Popular Culture. MIT Press Cambridge MA, 1992.

27. LustigAkustik Tenori-On Meets Sax On A Friday Night http://www.YouTube.com/watch?v=kff9b4x921I

43. Zizek S. The Fragile Absolute – or Why is the Christian Legacy Worth Fighting For? Verso London, 2000. 44. Zizek S. In Defence of Lost Causes. Verso London, 2008.

81

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Training Software Developers in Usability Engineering: A Literature Review Anders Bruun Aalborg University, Department of Computer Science Selma Lagerlöfs Vej 300, DK-9220, Aalborg Oest, Denmark [email protected] ABSTRACT

However, small companies do not even have the privilege of staffing usability specialists and these have to cope with issues such as the constraint of low budgets. In practice this means that small software companies do not have the funds to pay for comprehensive consultancy or staffing of usability specialists [13, 18, 28] as they are expensive to hire [23]. A survey conducted by Gulliksen et al. acknowledges this by showing that usability specialists are primarily employed by medium-sized or large companies [12]. In their study 70 % of the 194 respondents, are employed usability specialists in companies with at least 50 employees, which are categorized as medium-sized [9]. This is also supported by Rosenbaum et al. that shows that most usability specialists are employed in large companies consisting of 250 or more employees [25].

Software companies focusing on Usability Engineering face two major challenges, the first being the sheer lack of usability specialists leading to missing competences in the industry and the second, which regards small companies suffering from the constraint of low budgets, thus not being able to fund usability specialists or comprehensive consultancy. Training of non-usability personnel in critical usability engineering methods has the potential of easing these challenges. It is, however, unknown how much and what kind of research that has been committed to novice training in UE methods. This paper presents a comprehensive literature study of research conducted in this area, where 129 papers are analyzed in terms of research focus, empirical basis, types of training participants and training costs. Findings show a need for further empirical research regarding long term effects of training, training costs and training in user based evaluation methods.

The fact that small companies usually do not have staff with usability competences is expressed as one of the main barriers for incorporating UE in the development process [22]. This is also supported in the survey presented in [25] that shows that 17.3 % of 134 respondents mentioned missing competences as one of the main obstacles.

Author Keywords

Training, developers, usability engineering, literature review.

One way of solving the problems of incorporating UE in software companies is to increase knowledge of the subject across the set of stakeholders [12]. Thus, training nonusability personnel in critical usability engineering methods has the potential of easing problems regarding the lack of usability specialists in the industry, which is experienced in large companies. Training also provides an opportunity for small companies to apply UE methods as the developers themselves are driving the UE process, thus lessening the need to staff usability specialists, which cannot be funded.

ACM Classification Keywords

H5.2 Information interfaces and presentation (e.g.,HCI): User Interfaces -- User-centered design, Training, help and documentation. INTRODUCTION

During the last decade large software companies have increased their focus on introducing usability engineering (UE) methods into the development processes. One challenge for these companies is the sheer lack of usability specialists in the industry, which leads to missing competences across the board and hence, problems with incorporating UE in the development processes [16, 24].

Some studies provide promising insights regarding training of usability novices in UE methods. Metzker and Offergeld describe a software project in which developers participated in contextual task analysis, which motivated the participants to produce components with a high level of usability [22]. Another study presented in [17] describes how a nonusability specialist learned to apply a usability evaluation method. The participant stated that only little experience with cognitive psychology was needed in order to apply the method. Seffah et al. describes the opposite case in which usability specialists having sufficient technical knowhow eased integration of UE methods in the development process and gained wide acceptance from developers [29].

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

82

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

A reason for this could be that their perspectives are more closely aligned with that of the developers.

Papers were not selected if they had a purely pedagogical focus and did not describe training in particular UE methods or rendered too few details to uncover whether the training regarded analysis, design or evaluation methods. This was also the case if papers provided descriptions of particular methods aimed at novices or focused on skills needed by novices but no training was mentioned or given, for instance if novices were used as participants in experiments but details of their training was left unmentioned. Another constraint was if usability experts acted as participants or the methods used were dependent on expert knowledge, in which case the papers were excluded. The same goes for papers describing surveys of current use in software companies and papers focusing on description of courses aimed at UE lecturers. The final constraint regarded cases in which the same study by the same authors was described in different papers. In this case only one of these was chosen where journal versions were prioritized over conference papers, otherwise the latest version would be selected for inclusion.

Although seeming to be a promising approach for solving the above mentioned problems, it is still unknown how much and what kind of research that has been committed to novice training in UE methods. This paper presents a comprehensive literature study focusing on what research has been conducted in the area of training novices in UE methods. The following section describes the method used for identifying relevant research papers. Then an overview of the identified papers will be presented and analyzed in terms of research focus, empirical basis, types of training participants and cost. After this the findings are discussed according to future research needs and the final section presents conclusions. METHOD

This section describes the strategy undertaken to cover the body of knowledge regarding training of novices in UE methods. The overall strategy was based on the iterative process of identifying and reading papers, then identifying and reading potentially relevant papers from references or citations.

Identification process

In order to identify the first set of relevant papers a preliminary screening was performed. This was done using Google Scholar, as this search engine covers scientific papers from a broad set of publishers and proceedings. The search criteria were based on a full-text search in which all the words “training”, “developers” and “usability” were required and resulted in 33,800 records (search conducted December 1st 2009). As it would be too tedious a task to read abstracts from all these papers the first 200 abstracts were read of which 23 potentially relevant papers were selected and read in full. Eight papers were selected as relevant and defined the result of the screening process.

Selection of papers - criteria

Papers were selected as relevant if they described or focused on training of novices in UE methods. Here “novices” are defined according to Bonnardel et al.’s definition: “Lay-designers are people with little or no formal training in either web design specifically or its attendant skills (e.g., database design, graphic design, user interface design, etc.)” [5].

Papers referenced in the selected 8 papers from the screening were marked as potentially relevant. In addition the relevant 8 papers were looked up on Google Scholar which provides a utility to identify which papers are citing these. All citations were also marked as potentially relevant. Subsequently abstracts from all referenced and cited papers were read and papers fitting the selection criteria were read in full. This process continued until closure was reached after 8 iterations.

This definition does not, however, define what “little training” means exactly. For this the definition of Howarth et al. is used: “Additionally, all the evaluators selected for the study had less than one year of job experience related to usability engineering, thereby qualifying them as novices” [15]. Thus, this paper defines novices as persons with less than one year of job experience and no formal training in Usability Engineering methods.

Table 1 presents the statistics from the overall reading process and shows that 4155 abstracts were read and that 286 papers were read in full ending up with 129 actually relevant papers. Note that because of the length of the references these 129 papers are listed on a website [1] and are cited using parentheses and not square brackets, e.g. “(46)”.

Conference and journal papers were selected as relevant sources as these have gone through scientific peer reviews in order to be accepted at conference proceedings or journals. Ph.D. and master thesis’s were also included as they have been reviewed at higher exams. Books were not considered relevant in this study since it is uncertain whether or not these have gone through the same level of review. Table 2 shows how the selected papers are distributed according to publication type.

Table 2 presents an overview of the 129 papers distributed according to publication type. From the table it can be seen that the level of quality of the majority of papers adhere to conference proceedings and journal standards.

83

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Potentially relevant

Read in full

Actually relevant

Screening

200

23

8

References / Citations

3955

263

121

Total

4155

286

129

Table 1: Statistics from the overall reading process.

Conference proceeding

73

Journal

42

Workshop

7

Thesis

7

Total

129

Table 2: Distribution of papers according to publication type.

Figure 1: Number of papers distributed according to research focus and empirical basis (n = 129).

OVERVIEW OF CURRENT RESEARCH

This section provides an overview of the body of research focusing on training of novices in UE methods.

Practice – Field studies

The remaining categories of research focus consist of 59 papers with the purpose of describing, developing or evaluating UE methods for use in industrial practice. These are labeled as “Practice” in Figure 1 and are divided in 2 categories. Thirteen papers (10 %) have a field focus in which institutionalization of UE methods is done through practitioner training while considering the organizational context. These papers consider company size, staffing and development process and 3 are empirical studies where papers (1) and (36) present studies of integrating UE methods into small-sized companies through developer training and (66) uses a large company as their case. The remaining 10 papers describe non-empirical research by presenting theoretical frameworks for integration of various UE methods into development processes and present suggestions for industry training programs (12, 26, 27, 28, 52, 54, 67, 81, 86, 104).

Research focus

By reading the research questions of the individual papers 3 categories of research focus were identified and are described in further detail below. Figure 1 provides an overview of the 129 relevant papers categorized according to their research focus and empirical basis. University Focus

Seventy papers (54 %) have a university focus in which the main purpose is description, development or evaluation of UE contents for university curricula or pedagogical approaches applied in teaching UE methods to university students. The identified pedagogical papers do not focus on the curriculum contents, but on the way contents are taught, which differentiates this set of papers from those considering contents. Twenty-four are empirical studies of which 14 describe and evaluate courses with respect to student performance and perceived usefulness (6, 8, 9, 14, 24, 25, 55, 65, 90, 91, 93, 94, 120, 123). Ten are pedagogical papers evaluating the comprehensibility or effectiveness of the approach (15, 31, 35, 57, 58, 60, 83, 92, 98, 100), of which 2 are comparative studies of training materials comparing the effect of varying training conditions (15, 31). The remaining 46 papers are nonempirical of which 39 describe development of new curricula contents and course outlines (10, 11, 13, 17, 18, 20, 23, 29, 32, 33, 39, 41, 72, 74, 76, 78, 79, 80, 82, 84, 87, 89, 95, 96, 97, 99, 101, 102, 103, 105, 108, 113, 117, 118, 119, 121, 122, 125, 126, 128). Seven are studies describing new or previously applied pedagogical methods and brief descriptions of lessons learned (7, 34, 37, 38, 56, 69, 71).

Practice – Laboratory studies

Forty-six papers (36 %) focus on description, development or evaluation of UE methods or tools isolated from the organizational context and is thus labeled as “laboratory”. Six of these papers conduct empirical studies of novice performance when applying UE methods of which 5 present comparative studies of differences between usability specialists and novices (3, 85, 109, 111, 129). The sixth paper (46) evaluates the comprehensibility of a novel UE method when used by novices without comparing novices and specialists. Nine of the 46 “laboratory” papers focus on development or descriptions of software tools to support novices in applying UE heuristics or methods. Four of these are empirical studies that describe development of tools and evaluation of these using novices (50, 70, 106, 115) and 5

84

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Non-empirical

are non-empirical describing the development of specific software applications to support novices (4, 42, 59, 75, 114). Seventeen of the 46 “laboratory” papers have the main purpose of comparing two or more UE methods empirically using novices (2, 19, 21, 22, 30, 43, 44, 48, 49, 53, 61, 62, 63, 64, 68, 77, 107), while 14 papers are empirical evaluations of single methods applied by novices (5, 16, 40, 45, 47, 51, 73, 88, 110, 112, 113, 116, 124, 127).

The table shows that 12 papers of the 59 (20 %) describe non-empirical research targeted at practitioners. The majority of these discuss field issues of institutionalizing UE into software companies and the remaining describes development of tools to support novices. Students

Thirty-two papers (54 %) are empirically based using students for observations on training in methods targeted for use by practitioners in isolation from the organizational context, i.e. laboratory experiments. Most of these papers concern evaluation of a single or multiple UE methods and a few regard studies of pedagogical approaches and tool support.

Summary – Research focus

The above observations show that the majority of papers focus on development, description and evaluation of university curricula content or pedagogical approaches (54 %) and UE methods in isolation from organizational context (36 %). Few of the identified papers (10 %) focus on institutionalization issues in the field by considering the organizational context and that only 3 of these (2 %) are empirical studies on the subject. As this paper seeks to identify the body of research regarding training of novices in UE methods targeted for use in industrial practice, the remaining sections focus on the papers doing so, i.e. the 59 papers in the “Practice” categories shown in Figure 1.

Practitioners

Thirteen papers (22 %) describe empirical studies based on observations of novice practitioners trained in UE methods, which are also aimed for use by practitioners. The papers described in this set have a field focus on institutionalization via training of practitioners and others evaluate UE methods and supporting tools in laboratory conditions isolated from organizational contexts.

Participant types

Table 3 presents an overview of the 59 papers distributed according to their empirical basis and training participants. The matrix is divided in columns representing the empirical basis of the relevant papers, i.e. non-empirical and empirical, where empirical studies may base findings on experiments using either students, practitioners or both as subjects. Note that it was not possible to determine whether the subjects in the empirical studies of papers (15, 73) were students or practitioners, thus an additional category labeled “unclear” was necessary.

Both

A single paper (2 %) uses students and practitioners as the empirical basis. That paper describes a comparative laboratory study where the purpose is to evaluate a UE method using 4 different experimental conditions, of which 1 includes practitioners and the others include students. Summary – Participant types

Summarizing on the above it is shown that most of the current empirical research focusing on training in UE methods, for use by practitioners, is actually done using students as the empirical basis. Additionally, few of the identified empirical studies report of training novice practitioners in UE methods targeted for use in industry practice. Another observation is the fact that only 1 paper includes students and practitioners in the same study.

Empirical basis Nonempirical

Empirical Students

Practitioners

4, 12, 26, 27, 28, 42, 52, 59, 67, 81, 104, 114

2, 5, 16, 19, 21, 30, 31, 35, 40, 43, 44, 45, 46, 49, 50, 51, 61, 62, 63, 64, 68, 77, 85, 106, 107, 109, 110, 111, 112, 113, 127, 129

1, 3, 22, 36, 47, 48, 53, 54, 66, 70, 86, 115, 124

n = 12

n = 32

n = 13

Both 88

Unclear 15, 73

Training contents

n=1

Figure 2 provides an overview of the UE methods which are described and taught in the 59 papers targeting practitioners. For simplicity the methods are categorized according to the usability engineering activities in which they are to be applied [26]. The first activity category is “analysis”, which covers survey or task analysis methods used for needs analysis or user profiling. The second category is “design”, which regards design principles, patterns or prototyping methods applied during creation of the user interface. The third activity category is “evaluation”, which covers usability evaluation methods such as heuristic inspection or usability testing with users.

n=1

Table 3: Papers distributed according to empirical basis and participant type (n = 59).

85

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2: Number of papers targeted at practitioners distributed according to training contents (n = 59).

Figure 2 shows that 4 papers report on training in analysis methods exclusively, where 2 are empirical studies (1, 51) and 2 are non-empirical (52, 104 ). It can also be seen that 16 papers focus on training in design methods only, of which 4 are non-empirical (4, 42, 59, 114) and 13 are empirical (3, 16, 19, 31, 35, 61, 62, 66, 70, 86, 106, 115). The majority of papers focus on training novices in evaluation methods solely where a single paper is nonempirical (67) and 29 are based on empirical observations (2, 5, 21, 22, 30, 40, 43, 44, 45, 47, 48, 49, 50, 53, 63, 64, 68, 77, 85, 88, 107, 109, 110, 111, 112, 113, 124, 127, 129).

Figure 3: Number of papers focusing on training cost distributed according to research focus (n = 21). Training costs and laboratory studies

Fifteen of the papers considering training costs have description, development or evaluation of UE methods or supporting tools as a focal point. None of these consider organizational contexts, thus being categorized as laboratory studies. One of the 15 papers is a non-empirical study and describes the development of a tool to support novice practitioners in design tasks (42). The last 14 are empirically based of which 5 use practitioners as participants and 1 uses students and practitioners. The papers evaluate novice use of UE methods and supporting tools, where training contents regard either evaluation or design methods exclusively (47, 48, 53, 70, 88, 115). The remaining 8 papers use students as the empirical basis of which 1 describe development and evaluation of a tool to support practitioners in design activities (106) and the final 7 compare evaluation methods (21, 45, 68, 77, 85, 109, 127).

The remaining 8 papers describe training in several categories of methods where 3 are empirical (46, 54, 36) and 5 are non-empirical (26, 27, 28, 81, 12). Summarizing Figure 2 it can be seen that 11 papers describe training in analysis methods, 24 in design methods and 35 in evaluation methods targeted for use by practitioners. Training costs

This section provides an overview of the reported training costs in the 59 papers targeting practitioners of which 21 papers (36 %) mention training costs as a focal point. Training costs and field studies

Differences in training hours

Figure 3 provides an overview showing that 6 of these 21 papers consider the organizational context, thus having a field focus on institutionalization. The 2 empirical studies are using practitioners as the empirical basis. One of these 2 papers train participants in analysis methods (1) while the other describes training in analysis, design and evaluation methods (36). The remaining 4 papers having this research focus are non-empirical and present theoretical frameworks for integration of various UE methods into development processes. Two of these recommend training in several methods (27, 28) while the final 2 suggest training in design or evaluation methods exclusively (67, 86).

Fifteen of the 59 papers mention training costs in number of hours as the measuring unit of which 7 focus on cost (21, 36, 45, 47, 48, 53, 109) and 8 do not (2, 5, 30, 40, 49, 62, 107, 111). Figure 4 shows a box plot of these 15 papers distributed according to their focus on cost and training duration in hours.

86

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

cost focus. Another, less apparent, indicator is the medians, which are 7.5 for papers with a non-cost focus and 5 for those considering cost. However, these indications are inconclusive due to the few papers describing training costs in hours and due to the overlaps in training hours between papers focusing on cost and those that do not, especially in the first and second quartiles. The fact that there are considerable variations in the number of hours, even if training contents are similar, is also adding to the inconclusiveness. Thus, the needed amount of training in, e.g. evaluation methods is still an open question. Summary – Training costs

In sum only 2 empirical studies focus on institutionalization and organizational contexts in conjunction with training costs. Figure 4 also indicate disagreements on what constitutes a necessary amount of training to educate novices in UE methods, even in the first 3 quartiles, in which all papers describe similar training contents (evaluation methods exclusively). Variations in the fourth quartiles may be explained through differences in training contents, i.e. one paper covers analysis, design and evaluation where others consider evaluation or design methods only.

Figure 4: Box plot of empirical based papers targeted at practitioners (n = 15 papers mention training cost in hours).

The first quartiles (0 – 25 %) show a lower bound of 1 – 3.5 hours for papers not focusing on training cost where the 2 papers contained herein focus on training students in usability evaluation methods exclusively (2, 40). In case of cost focus the lower bound is between 1 – 2.5 hours and includes 2 papers describing training novice students (45) and practitioners in evaluation methods solely (53).

Evaluation Methods

This section focuses on papers reporting of training in usability evaluation methods. Methods related to analysis and design activities are of course also important training areas, however, results from usability evaluation methods have proven to be effective in creating the wake-up calls necessary for companies to start focusing on UE or to increase the awareness of developers [14, 27], which is why this section is dedicated to evaluation methods. From Figure 2 it can be extracted that 30 of the 59 papers (51 %) describe empirical studies of evaluation methods.

The second quartile (25 – 50 %) shows that the number of training hours is between 3.5 – 7.5 and 2.5 – 5 hours for non-cost and cost focus respectively. Papers (5, 107) do not focus on training costs and report on training students in evaluation methods exclusively, which is also the case for paper (21) focusing on cost. The third quartiles show larger differences spanning from 7.5 – 20.9 hours for papers not having training costs as a focal point and 5 – 7.5 for papers that do. These 2 sets contain 4 papers which describe training of novice practitioners focusing on cost (47, 48) and training of students with no cost focus (30, 49). The 4 papers all describe training in evaluation methods solely. The fourth quartiles span from 20.9 – 40 hours in the case of papers without a training cost focus and 7.5 – 40 for the papers mentioning low training costs. The 2 papers not mentioning costs describe educating students in design or evaluation methods solely (62, 111). One of the 2 papers considering low training costs educate novice students in evaluation methods exclusively (109) and the other train novice practitioners in all aspects of UE, i.e. analysis, design and evaluation methods (36).

Figure 5: Number of empirical studies targeted for use in practice distributed according to participant type (n = 30).

Thus, from Figure 4 it can be seen that papers focusing on cost overall spend less hours training participants in UE methods. This is in part indicated by the upper bounds of quartiles 1, 2 and 3, which are lower for papers having a

87

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 5 provides an overview of the specific evaluation methods taught in the 30 papers distributed according to the participant types. Paper (88) is represented as “both” as the empirical base consists of students and practitioners.

considering the context, in which the UE methods are to be used. This, however, is a crucial area of research, as methods isolated from the real world context potentially are of little practical relevance [31]. In software companies there exist several constraints such as team buy-in, resources, change management, political environments, technology and personnel [19, 29, 31], all of which pose challenges not experienced in studies using university students as participants or evaluation of methods in isolation. Three of the papers focusing on institutionalization (2 %) present empirical studies of integrating UE methods into companies through developer training (1, 36, 66). Results from all 3 studies acknowledge the potential of training novice practitioners in UE methods. Post-project interviews conducted in (1) revealed that participants perceived the training as being satisfying and important to future usability work, an observation similar to the findings described in (36), where the 13 participants graded the training course as 4.25 on a 1 – 5 scale (5 = very good). (66) report of lessons learned statements which indicates that 80 % liked the training course and 20 % did not and (1) and (36) indicate similar reactions where the practitioners wanted more training in UE. (36) reveals more detailed results by showing that the novice practitioners, on the other hand, experienced problems in categorizing the severity of identified usability problems when asked to conduct a heuristic inspection. A design exercise conducted before the training is also compared to one conducted after and shows some minor improvements in proposed designs. Thus, these studies have shown promising short term results, however, there is still a need for empirical research of long term effects on institutionalizing UE via training of novice practitioners. This is important because we need to include and cater for various organizational constraints.

Usability testing

Ten of the 30 papers report of training participants in user based methods and are mentioned as either think-aloud, usability test or user testing in the various studies. Taken together these are labeled as usability testing. Figure 5 shows that a single paper report on training in usability testing using novice practitioners as the empirical base (36) and 9 papers report on student training in usability testing (2, 30, 45, 50, 64, 109, 110, 111, 127). Non-user based evaluation methods

A total of 26 papers describe training in non-user based methods. Ten of these consider training in Cognitive Walkthrough (CW) where 4 are based on observations of novice practitioners (22, 47, 48, 124) and 6 on students (30, 40, 43, 49, 64, 107). The majority of papers report using the Heuristic Inspection (HI) method of which 4 are based on training of practitioners (22, 36, 53, 88) and 10 on students (21, 30, 44, 63, 68, 77, 85, 88, 112, 113). Seven papers describe various non-user based methods such as Metaphors of Human Thinking, Barrier Walkthrough, Abstract Tasks or Programmable User Model, which are categorized collectively as “other non-user based” evaluation methods in Figure 5. This figure shows a single paper describing training of practitioners in a method from the “other non-user based” category method (66) while the 6 papers are student based (2, 5, 21, 43, 44, 129). Summary – Evaluation methods

As mentioned in the introduction of this paper resource demands is one of the main obstacles companies face when wanting to incorporate UE into the development process. For this reason it is not only important to focus on training costs but also the costs of using the particular methods. Considering the 30 papers describing training of usability evaluation methods, 26 (87 %) report of training in the Cognitive Walkthrough, Heuristic Inspection and other non-user based methods, all of which are considered lowcost as no laboratory or end-users are required. There also exist varieties of user based usability testing methods focusing on low-cost, i.e. Instant Data Analysis or Rapid Iterative Testing and Evaluation [20, 21]. However, none of the above 30 papers focus on training participants in a resource saving user based evaluation method.

As mentioned above, it is important to focus on practitioners in industry. Research in UE methods targeted for use by practitioners is reported in 59 of the identified papers (46 %), of which 12 are non-empirical studies and 32 and 13 papers apply novice students and practitioners respectively, while a single paper uses both. Thus, current empirical research mainly focus on using students as participants, which in turn indicates a future need for empirical studies using practitioners employed in the software industry. In total 10 % of the 129 identified papers consider this type of participants. One of the reasons for the focus on using students as the empirical basis is that this approach is less demanding in terms of planning and external involvement. Nevertheless it is still important to get more focus on using practitioners as the empirical basis in order to include constraints such as team buy-in, political environments etc.

DISCUSSION

The above observations show that the current body of literature primarily focuses on development, description or evaluation of curricula contents or pedagogical approaches aimed at university students (54 %). Another large research area is description, development or evaluation of UE methods isolated from organizational contexts (36 %). Less focus (10 %) is devoted to institutionalization issues

As argued in the introduction high resource demands is one of the largest obstacles of introducing UE into software companies, which especially is the case for small software companies that have insufficient funds to pay for comprehensive consultancy or staffing of usability

88

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

specialists [4, 13, 18, 25, 28]. According to Seffah et al. practitioners want to produce software with a high level of usability, but as they work under time and budget restrictions they do not want UE to consume too much of their time [29]. For this reason a considerable amount of research has been committed to developing resource saving methods, mainly in the area of usability testing [20, 21, 23]. Resource saving methods are especially relevant for small companies that do not have the funding to incorporate UE. It also implies that not only should the taught UE methods be resource saving, but this must also apply to the training itself as there is no point in teaching resource saving methods if small companies cannot afford the training. This is also supported in [16] and [30], where the latter shows that most organizations tend to view training as a cost rather than an investment. Although cheap training is important, this review shows that 21 (36 %) of the 59 papers describing methods for practitioner use, focus on training costs. Additionally, 2 of these 21 papers are empirical studies of institutionalization issues in which the organizational context and training of practitioners are considered. Results show that although papers focusing on training costs report a lower number of training hours compared to papers without cost as a focal point, there are still disagreements on what constitutes a necessary amount of training to educate novices in UE methods. In this regard there is a need for obtaining knowledge of how much training is actually needed in order to apply UE methods satisfactorily. There is also a need for further empirical research considering cost and institutionalization issues.

ease the institutionalization of UE, a notion supported in [7]. This study shows that 10 papers describe training in user based evaluation methods, of which 1 uses practitioners as the empirical base. Thus, there is a need for empirical studies of training novice practitioners in user based evaluation methods. Considering the importance of user based methods and previous discussions on training costs, focus on organizational contexts and practitioners, this paper shows that a single empirical based paper is focusing on these crucial issues (36). That paper shows promising first results in integrating UE into a small software company through training in user based evaluations. The training provided positive attitudes towards UE from the developers who afterwards were further motivated in learning about usability aspects in software development. The user based method taught was, however, not reported as being resource saving, thus in part conflicting with the needed aim of introducing resource saving methods via cheap training to small companies. This also constitutes a future research need. CONCLUSIONS AND FUTURE RESEARCH NEEDS

This paper has presented a comprehensive literature study of the research conducted in the area of training novices in Usability Engineering (UE) methods. The 129 identified papers have been analyzed in terms of research focus, empirical basis, types of training participants and training costs. Five key areas for future research needs are identified and listed below.

Usability evaluation methods have proven to be valuable in generating the wake-up calls needed to make companies focus on UE and in increasing the awareness of developers [14, 27]. Thus, evaluation methods could be the potential best way to start a series of UE training sessions. Fonseca et al. [10] and Edwards et al. [8] also describe university curricula in which they begin the courses with letting students evaluate user interfaces, as this increases their awareness of interface problems. The survey described in [25] also states that usability testing in and without a lab are the most preferred methods by software companies. Furthermore several studies indicate that user based testing is superior to inspection or walkthrough methods in this regard. User based tests provide valuable first hand observations of the problems experienced by real users, which in turn increases motivation for making adjustments to the user interface [14]. The reason for this may be located in the fact the user testing provides empirical evidence of the problems at hand compared to theoretical inspections [6]. Additionally, the study presented in [11] indicates that a majority of evaluators prefer user based methods over the inspection methods Metaphors of HumanThinking and Cognitive Walkthrough. A similar study described in [2] shows that user based methods was rated above inspection methods regarding pleasantness of use. Thus, several studies indicate that user based testing potentially increases practitioner buy-in, which in turn may

Firstly 13 papers (10 %) focus on institutionalization issues by considering the organizational contexts, of which 3 are empirical based studies. The 3 empirical studies in this area have focused on measurements of short term effects of introducing UE into the organizations, thus indicating a need for further empirical research of long term effects. Secondly there is also a future need for focus on empirical studies using practitioners as few of the identified papers (10 %) consider this type of training participants. Third, due to disagreements in the current body of literature, this review also shows a need for further empirical research focusing on training costs and the amount of training necessary to obtain a satisfactory level of method usage. Fourth, in addition to training costs, few (2) of the identified papers are empirical studies in which institutionalization and cost issues are considered at the same time, hereby indicating another need. Fifth, as discussed previously, user based evaluation methods seem to provide the best wake-up call for software companies, and this review shows that a single empirical based paper is focusing on training novice practitioners in such a method. This leaves room for future research in this area.

89

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

REFERENCES

Human-Computer Interaction 21, 2 (2006), 173-196. Taylor & Francis (2006).

1. List of selected papers analyzed in this literature review. http://www.cs.aau.dk/~bruun/references.html.

15. Howarth, J., Andre, T. S. and Hartson, R. A Structured Process for Transforming Usability Data into Usability Information. Journal of Usability Studies 3, 1, 7-23 (2007).

2. Ardito, C., Costabile, MF, De Angeli, A. and Lanzilotti, R. Systematic evaluation of e-learning systems: an experimental validation. In Proc. NordiCHI 2006, ACM Press (2006).

16. Ji, Y. G., Yun, M. H. Enhancing the minority discipline in the IT industry: A survey of usability and UserCentered design practice. Int. Journal of HumanComputer Interaction 20, 2 (2006), 117-134. Taylor & Francis (2006).

3. Bailey, G. S. Iterative methodology and designer training in human-computer interface design. In Proc. INTERACT 1993, ACM Press (1993). 4. Bak, J. O., Nguyen, K., Risgaard, P. and Stage, J. Obstacles to usability evaluation in practice: a survey of software development organizations. In Proc. NordiCHI 2008, ACM Press (2008).

17. John, B.E. and Mashyna, M.M. Evaluating a Multimedia Authoring Tool with Cognitive Walkthrough and ThinkAloud User Studies. Journal of the American Society of Information Science 48, 9 (1995), 1004–1022.

5. Bonnardel, N., Lanzone, L. and Sumner, T. Designing web sites: The cognitive processes of lay-designers. Cognitive Science Quarterly 3, 1 (2003), 25-56.

18. Juristo, N., Moreno, AM and Sanchez-Segura, M. I. Guidelines for eliciting usability functionalities. IEEE Transactions on Software Engineering 33, 11 (2007), 744-758. IEEE Computer Society Press (2007).

6. Brown, C., Pastel, R. Combining distinct graduate and undergraduate HCI courses: an experiential and interactive approach. In Proc. SIGCSE 2009, ACM Press (2009).

19. Karat, C. M., Campbell, R. and Fiegel, T. Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proc. CHI 1992, ACM Press (1992).

7. Dayton, T. Skills needed by user-centered design practitioners in real software development environments: report on the CHI'92 workshop. ACM SIGCHI Bulletin 25, 3 (1993), 16-31. ACM Press (1993).

20. Kjeldskov, J., Skov, M. B. and Stage, J. Instant data analysis: conducting usability evaluations in a day. In Proc. NordiCHI 2004, ACM Press (2004).

8. Edwards, A., Wright, P. and Petrie, H. HCI education: We are failing–why? In Proc. HCIEd 2006, Springer (2006). Springer (2009). 9. European Commissions definition Medium-sized Enterprises:

of

Small

21. Medlock, M. C., Wixon, D., Terrano, M., Romero, R. and Fulton, B. Using the RITE method to improve products: A definition and a case study. In Usability Professional's Association Annual Conference 2002.

and

22. Metzker, E., Offergeld, M. An Interdisciplinary Approach for Successfully Integrating Human-Centered Design Methods Into Development Processes Practiced by Industrial Software Development Organizations. In Proc. IFIP International Conference on Engineering for Human-Computer Interaction 2001, Springer-Verlag (2001).

http://ec.europa.eu/enterprise/policies/sme/ facts-figures-analysis/sme-definition/index_en.htm 10. Fonseca, M., Jorge, J., Gomes, M., Gonçalves, D. and Vala, M. Conceptual design and prototyping to explore creativity. IFIP International Federation for Information Processing 289 (2009), 203-217. Springer (2009).

23. Nielsen, J. Usability inspection methods. In Proc. CHI 1994, ACM Press (1994).

11. Frøkjær, E., Hornbæk, K. Metaphors of human thinking for usability inspection and design. TOCHI 14, 4 (2008). ACM Press (2008).

24. Nielsen, J. Finding usability problems through heuristic evaluation. In Proc. CHI 1992, ACM Press (1992). 25. Rosenbaum, S., Rohn, J. A. and Humburg, J. A toolkit for strategic usability: results from workshops, panels, and surveys. In Proc. CHI 2000, ACM Press (2000).

12. Gulliksen, J., Boivie, I., Persson, J., Hektor, A. and Herulf, L. Making a difference: a survey of the usability profession in Sweden. In Proc. NordiCHI 2004, ACM Press (2004).

26. Rubin, J. Handbook of usability testing. John Wiley & Sons Inc., New York, NY, USA, 1994.

13. Häkli, A. Introducing user-centered design in a smallsize software development organization. Helsinki University of Technology (2005).

27. Schaffer, E. Institutionalization of Usability: A Step-ByStep Guide. Addison Wesley, Redwood City, CA, USA, 2007.

14. Høegh, R. T., Nielsen, C. M., Overgaard, M., Pedersen, M. B. and Stage, J. The Impact of Usability Reports and User Test Observations on Developers’ Understanding of Usability Data: An Exploratory Study. Int. Journal of

28. Scholtz, J., Laskowski, S. and Downey, L. Developing usability tools and techniques for designing and testing web sites. In Proc. Conference on Human-Factors & the Web 1998.

90

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

29. Seffah, A., Gulliksen, J. and Desmarais, M. An Introduction to Human-Centered Software Engineering. In Seffah, A., Gulliksen, J., Desmarais, M.C. (eds.) vol. 1, 59-70. Springer, Netherlands (2005).

31. Wixon, D. Evaluating usability methods: why the current literature fails the practitioner. Interactions 10, 4 (2003), 28-34. ACM Press (2003 ).

30. Taylor, M. J., England, D. and Gresty, D. Knowledge for Web site development. Internet Research 11, 5 (2001), 451-461. MCB UP Ltd. (2001).

91

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Design and Evaluation of Player Experience of a Location-Based Mobile Game Tara Carrigy, Katsiaryna Naliuka, Natasa Paterson, Mads Haahr Trinity College Dublin College Green, Dublin 2, Ireland

{carrigyt, naliukak, patersn, mads.haahr}@tcd.ie ABSTRACT

Augmented Reality (AR) Games and Location-Based Mobile Games (LBMGs) continue to evolve, game designers creating immersive gaming experiences will face unique and unprecedented challenges.

This paper reports on the design and evaluation of player experience of a Location-Based Mobile Game set in Dublin, Ireland in which players act as paranormal investigators hunting for ghosts and gathering evidence of paranormal activity. The paper focuses on players’ experience of engagement and immersion, which was evaluated through a qualitative user study undertaken over a three-day period with the participation of 19 subjects. We first discuss the concept of immersion in gaming and then review related work before presenting the design and implementation of our prototype and the results of our user study. The results show that the experience succeeds in creating a high level of immersion at several stages in the game and that this immersion can be influenced by several factors including usability, control, modes of interaction, aesthetics, flow and, perhaps most significantly, choice of location.

Engagement, Immersion, Location-Based Mobile Gaming

In this paper, we report on a study of player experience of a working prototype of a LBMG. Our primary goal was to test our assumptions about factors that contribute to the creation of engaging, compelling and immersive gameplay. Another motivation was to expand our understanding of players’ engagement and to develop strategies for future development of this game. We begin by exploring established notions of engagement and immersion in gaming. Next we survey related work, focusing on research projects that evaluate player engagement, most specifically in the context of LBMG and more broadly in the context of Pervasive and AR Gaming. We then describe the design and implementation of the prototype and present our set-up and methodology for the qualitative evaluation study. Finally, we present the results of our study and discuss our findings with some strategies for further development.

ACM Classification Keywords

ENGAGEMENT AND IMMERSION

Author Keywords

In the general context of gaming, gamers, developers and researchers describe the players’ experience of engagement or involvement in the game as immersion [3]. The term immersion is often used to describe the experience of losing track of the outside world and the boundaries of the magic circle, the imaginative space in which the game is played [1]. As we have already alluded to, initial understanding of the term immersion in gaming referred to the context of video and computer gaming and is therefore inevitably associated with VR technologies where the player enters or plunges into the virtual game world [5,19]. By stating that immersion is ‘the sensation of being surrounded by a completely other reality’ and ‘the experience of being transported to an elaborate simulated place’, Janet Murray’s definition of immersion reflects this association [19]. In the VR and Virtual Environments (VE) research community immersion is commonly accepted as the qualities of a media that create sensory impact by surrounding the user [9, 29] and for decades this community has defined a set of ‘immersive factors’ almost exclusively in relation to the technology. For example, Slater proposes that ‘immersion can be objectively assessed as the characteristics of a

H5.m. Information interfaces and presentation: User Interfaces—User-centred design; Evaluation INTRODUCTION

Evaluation of players’ experience in any form of digitally mediated gaming almost always refers to engagement, and specifically immersion, as desirable aspects of gameplay. In the general context of gaming, the term immersion is used to loosely describe the degree to which the player becomes involved or engaged in the game. Until recently understanding of immersion in computer gaming has mainly been developed in relation to PC and Console Gaming. As the new contexts of Pervasive Gaming, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

92

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

technology, and has dimensions such as the extent to which a display system can deliver an inclusive, extensive, surrounding, and vivid illusion of virtual environment to a participant’ [29]. In contrast, however, Witmer suggests that immersion is a psychological state that can be influenced by the immersive tendencies of the individual participant [34]. In VE research, immersion is often measured by presence, ‘the subjective experience of being in one place or environment, even when one is physically situated in another’ [34]. In this context it is common to assume that presence can only be experienced through immersion in the VE [29,34] and that greater presence leads to greater engagement.

Players expect to be rewarded for the time, effort and attention spent learning, what Lindley refers to as, the ‘gameplay gestalt’ [17], the pattern of player interaction with the game system. Cairns and Brown’s grounded theory of immersion suggests that as players pass through the barriers associated with the preliminary stages of psychological engagement, they are rewarded by a more imaginative experience that Cairns and Brown describe as total immersion [3,28]. Engagement and immersion therefore are part of a continuum of player experience, where initial engagement is associated with learning the game mechanics and is a prerequisite for deeper total immersion, which is the performance of the game mechanics in order to experience the gameplay. In this sense immersion is similar to flow, ‘the state in which individuals are so involved in an activity that nothing else seems to matter’ [6] and where attention becomes so focused that ‘sense of time is altered and sense of self is lost’ [3]. Flow is the ‘state at the boundary between engagement and immersion, of being totally absorbed in meeting the constantly unfolding challenge’, which Lindley also refers to as immersion in performance [17]. For the purposes of this paper, we take the essential difference between experiences at either end of this continuum as follows: engagement is more active than immersion, and total immersion is closer to flow, the immersive state achieved when this activity is balanced by the players’ skills and becomes so transparent that the player can experience deep but effortless involvement in the game [3,25].

Game theorists Salen and Zimmermann offer another perspective on immersion and presence when they observe that equating the experience of immersion with being ‘sensually’ transported to another separate, simulated reality so convincing that ‘the player truly believes that he or she is part of an imaginary world’ [27] is a fallacy that assumes the technologically mediated experience can become so sophisticated that it will be capable of ‘fully illusionistic experiences indistinguishable from the real world’ [27]. They propose that, in the context of gaming, the immersive fallacy of engagement through simulation is a one-dimensional approach to player engagement that ‘misrepresents how play functions’. In contrast Salen and Zimmermann suggest that engagement ‘occurs through play itself ’ in a process of double consciousness [27], whereby the players become engrossed in the game while simultaneously aware of the medium and the ‘artificiality of the play situation’ and thus perceiving play as something separate from, but nevertheless connected to the real world. This understanding of immersion does not disregard the power of simulation and the potential for engagement as a result of presence but it does present player engagement as a more complex and active process and this approach is especially relevant in the context of LBMG where the player is constantly shifting focus between the media and the physical environment and where gameplay moves fluidly between mediated and directly felt experience.

Immersion in gaming is considered to be more than the objective characteristics of the technology but also the subjective experience of the player as s/he plays the game. Recognising the different levels and stages of engagement and immersion in gaming, many researchers have emphasised that the experience of immersion is a transient and often fleeting state [3,10,23]. Part of the focus of this paper is to establish the factors that influence the players’ experience of engagement and immersion as they move through the game, and in particular highlight those specific to the context of LBMG.

Player engagement and immersion in games is essentially performative and participatory and occurs as a direct result of active involvement, attention and interaction. In games, players are actively rather than passively engaged in the gameplay experience and therefore the quality of the players’ interaction with the game system, through the game mechanics, is a key factor influencing immersion. Players enter the game through the game controls [3] and mastery of the game mechanics (the interface between the player and the game world) is the ‘price of admission’ [25] to the gameplay; as players learn how to use the game controls to play the game, their focus switches from conscious attention to internalised or tacit knowledge. This state of deep involvement shortens the distance between player and environment and results in immersion [4].

RELATED WORK

Researchers evaluating player experience in LBMGs have explored several factors that impact on player engagement and immersion, the most common focus being on the significance of location. Some LBMG blur the boundary between fact and fiction and leverage a combination of local legend and history to engage players, heighten their awareness and change their perception of the everyday places, where the games are located [2,22,34]. By evaluating REXplorer, a LBMG set in the Medieval city of Regensburg, researchers found that interweaving real world landmarks into the game’s narrative, which they based on a mixture of local folklore and medieval history, added an authenticity to the gameplay that increased players’ immersion [2]. Similarly, Visby Under and Frequency

93

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

1550, are both LBMGs that use the immersive qualities of location to bring history to life and enhance player engagement [30]. Likewise Nisi et al, in their evaluation of Media Portrait of the Liberties, also report achieving deeper story immersion and an enhanced sense of place by overlapping and connecting narrative with location [20]. Riot! 1831, a locative interactive play, used audio to recreate the riots of 1831 and GPS technology to situate this media at the exact location in Bristol where the riots occurred [24]. Participants reported empathy with the people involved in the riots and a sense of ‘walking in their footsteps’. Research revealed a high level of immersion as direct result of mapping historic narrative onto relevant locations but also exposed that immersion is a transient state that can be affected by environmental factors associated with that location. Evaluation of LBMGs created using the Mediascapes toolkit highlights how making use of aspects in the physical environment can make the games more engaging. In recognising the difficulty but nevertheless high potential for immersion associated with a tight coupling of game content and environment, the Mediascapes researchers propose a strategy for ‘designing for coincidence’ whereby environment features or the behaviour of other people become aligned with the content of the game and are perceived by the player as highly engaging ‘magic moments’ [24]. Finally in relation to choice of location, researchers evaluating LOCUNET found that locating LBMGs in public spaces could have a negative impact on immersion if players felt inhibited or threatened by the presence of passers-by [7].

may increase immersion in Locative Games [5]. In Songs of the North, researchers used an audio interface to create an engaging location-aware mobile game because they felt that audio was a more appropriate medium for creating an immersive hybrid gameworld [15]. Related work has also shown that the mode of interaction in LBMG can affect player engagement. For example, REXplorer [2] uses gestural based interaction to physically engage players, while the iPerg [32] researchers recommend that the use of authentic or real world interaction models, such as walking around to discover elements of the game, can increase player immersion. In addition to evaluations of LBMGs that focus on immersion and engagement, other related work includes research that has focused on methodology and models for evaluating player experience in the context of Pervasive Gaming. In the Pervasive GameFlow (PGF) model [13,14], Jegers develops a model of player enjoyment of Pervasive Gaming based on the GameFlow model originally developed by Sweetser and Wyeth [31] and following Csikszentmihalyi’s concept of flow as described in the previous section. With regard to immersion the PGF model recommends that pervasive games should enable players to shift focus from virtual and physical parts of the game and seamlessly transition between the gameworld and everyday contexts [13]. Ermi and Mäyrä’s SCI model of immersion [10] is also useful for evaluation in the context of LBMG because it opens up the possibility of measuring player immersion from the perspective of the player. The SCI model of immersion considers player immersion from three perspectives: Sensory, Challenge-Based and Imaginative and in doing so encompasses key elements of gaming and is closely aligned to game models proposed by the gaming and game theory community, for example Lindley’s Simulation, Gameplay and Narrative taxonomy [16,17].

As well as transforming the players’ perception of space with story, some researchers in the field of LBMG have explored techniques for augmenting space with visuals and sounds, which players will perceive as part of the hybrid gameworld, and have evaluated how this impacts on player engagement and immersion. For example, researchers compared player engagement in AR Façade to the previous PC version of the interactive narrative game Façade, and found that while the players of the AR version felt an increased sense of presence this did not necessarily result in increased player engagement [9]. Early user evaluation of TimeWarp, an outdoor augmented reality game played in the city of Cologne using a HMD, revealed that GPS tracking issues combined with the graphical realism of the AR objects, limited players’ sense of presence because the objects floated around too much or did not appear to be real [28]. Evaluation of Interference, a LBMG using AR technology to superimpose 3D models on the mobile’s live video stream, revealed that the aesthetics, which helped to blend the physical world into the game world, were the most significant factor contributing to player immersion in the game [33]. In relation to acoustic augmentation, research in VE has shown that accurately synthesized spatialised sound can increase presence [11] and researchers have shown how these techniques can be used to create realistic localised sound and a 3D audio soundscape that can be easily navigated by players and thus

DESIGN AND IMPLEMENTATION

Viking Ghost Hunt (VGH) is a location-aware adventure game, based on a Gothic ghost story set in Viking Dublin (800-1169). In this game the player assumes the role of a paranormal investigator and moves around the city hunting for ghosts, collecting evidence and solving the mysteries of haunted Viking Dublin. The game is designed as a singleplayer, immersive gameplay experience, in which the player is an active character in an unfolding drama. In order to progress the game, the player must unlock a sequence of location-specific narrative fragments, by completing a series of challenges and missions. Throughout the VGH game design, we focused on two design goals: to give meaning to play by maintaining the aesthetics of role-play and to exploit the characteristics of the locations by integrating them into the game to create an engaging gameworld.

94

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Role-Play

the gameworld using a combination of sound, sight and touch. Collecting evidence by photographing ghosts using the camera function is also an intuitive and familiar mode of interaction.

In order to preserve the illusion or magic circle the players’ assumed role as a paranormal investigator must persist at all times. We set this up by presenting all of the aspects of gameplay, including the aesthetics, challenges and modes of interaction, in the context of paranormal investigation. Equipped with a paranormal investigation device and an audio headset, the players’ goal is to conduct investigations by hunting down ghosts and gathering evidence, by making audio recordings or taking photographs of ghostly apparitions. Players use Google maps to search for the location of an alleged haunting in Viking Dublin. Players begin their investigation by using the Radar (see Figure 1) to locate paranormal activity in the region. This Radar also includes an Electromagnetic Frequency (EMF) meter, which informs players of the levels of paranormal activity and the strength of the manifestations. A dowsing meter (based on the metaphor of dowsing rods used to divine water or paranormal activity) helps to guide the player in the direction of the haunting. Once the player has located the paranormal activity, s/he can begin to collect evidence that will substantiate the alleged haunting. Players use the Frequency Scanner (see Figure 1) to tune in and decode the static, in order to reveal the ghostly voices in the white noise, in a process reminiscent of Electronic Voice Phenomenon. Ghost View can be used to photograph visual manifestations. Players’ can review their progress in Casebook, a database that stores players’ evidence and contains extra narrative about the ghosts.

Figure 1. Radar (left) and Frequency Scanner (right). Use of Location

We set the game around St. Audeon’s, a Medieval church and public parkland situated within the Old Viking City Walls of Dublin because of its historical and thematic relevance and potential for a varied player experience. With the intention of blurring the distinction between fact and fiction, we based the game’s narrative on a mixture of local ghost stories and urban myths as well as factual history. We included contrasting locations in the mission design in order to test the impact of location on gameplay. The main location was ideal for the theme and atmosphere of the game and we were able to integrate features such as the Medieval style church, windy ramparts, high stone walls, dark archways and damp and eerie laneways; all classic motifs of the Gothic ghost story genre. Narrative was used to draw player’s attention to physical features in the environment and integrate them as part of the gameworld. Dialogue, in combination with sound effects, helped players imagine a scene from the past. For example, in the Viking era, the River Liffey was much broader than it is today and hence the grassy area outside the old city walls was once a beach; we recreated this scene using sound effects and descriptive narrative. Similarly, the narrow laneway and medieval staircase at the side of the church was transformed into the ‘Lane of Hell’ by a dense interplay of atmospheric dialogue enhanced by realistic sound effects. Exploiting the characteristics of the locations is depends on presenting relevant content to the player in relation to their position in the gameworld. We set this up by specifying regions where the assets would be triggered and then used GPS to detect when players had entered these regions. Since GPS technology is prone to lagging and errors, efforts were made to minimize inaccuracies by creating large regions

User Interface Design

The User Interface (UI) design was guided by the aesthetics of role-play, as described above, and the intention of facilitating embodied interaction [8] with the game as is appropriate for situated and location-based gaming. To these ends, we designed the UI to look and feel like a tool for paranormal investigation and to evoke a similar cognitive process as someone investigating a radioactive area with a Geiger counter. By employing a visual style of presentation reminiscent of mechanical devices of the past, we established a set of affordances that suggested the appropriate mode of interaction to the user while simultaneously reinforcing the aesthetics of role-play and atmosphere of the game. By way of engaging the player in the processes of searching for and collecting evidence, we applied the principles of direct manipulation by enabling phenomenological interaction via touch screen gestures and multimodal feedback in response to players’ actions and movements. As much as possible, the intention was to reference real world actions as interaction metaphors so that players’ could easily master the gameplay gestalt by leveraging their existing tacit knowledge [4,17]. For example, navigating the gameworld is achieved through the activity of searching for ghosts by walking around the location and using the Radar to detect paranormal activity. The players’ proximity to the ghost is indicated using a combination of visual representation and a fluctuation in audio and tactile feedback, hence the players can navigate

95

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and positioning them at locations with good GPS reception (e.g. open areas not shadowed by trees and buildings). In situations where we needed to place content close to buildings, for example in the ‘Lane of Hell’ scenario, we used the physical properties of the space to compensate for the lack of GPS updates in this region. In this example, the assets were triggered by regions at either end of the lane thus limiting the experience of this content to the confines of the lane. Despite this approach, the resolution remained fairly low in precision and therefore the assets could only be loosely mapped to specific physical features.

enough apart to give the device time to re-load each of the individual assets as required. EVALUATION

We evaluated the VGH game by running a series of field trials over a three-day period. The 19 participants who took part had varying degrees of prior experience of gaming and ranged in age from 18-48. We evaluated the participants’ response to the game using a post-game qualitative questionnaire loosely based on a combination of established models of player engagement and immersion [10,14,31,34]. The questionnaire included both open ended and bounded questions designed to obtain feedback on engagement, control, usability, presence and impact of location. While an in-game evaluation technique such as ‘think-aloud’ may have captured a more immediate and visceral response to the game, this was not practical due to resource limitations. We also captured data during the game using the system logs. While this data did support the information gathered by the retrospective questionnaires it did not offer anymore insight into the player experience and therefore we will not present this data as part of our evaluation.

Figure 2. Using Radar’s touch screen zoom (left) and Ghost View (right). Augmented Reality

RESULTS AND OBSERVATIONS

Various Augmented Reality (AR) techniques were used to transform and augment the players’ visual and auditory perceptions of the location. With the aim of creating the illusion that the ghost is actually present in the player's surroundings, the players’ view of the scene is augmented by animations displayed on top of the live video stream in the viewfinder of the camera. The effect is perceived as more realistic when the ghost is anchored to a point in the scene and therefore appears to remain stationary in the physical surroundings even when the player moves the camera. Due to the impracticalities of placing optical markers into a public space and the expensive overhead of performing real-time image analysis, it was not possible to employ computer vision techniques to anchor the ghost overlay within the scene. Instead we tracked the direction of the viewfinder using the device’s orientation sensors and although (due to the unreliable nature of the sensor data) the ghost representation was not completely fixed within the scene, we speculated that the visual floating effect would be credible in the context of ghostly apparitions. Acoustic augmentation of the location was achieved by using spatialisation and reverberation techniques [19] to realistically match the game’s sound effects to those naturally occurring in the environment and as a way of deliberately creating ambiguity between the game and the real world. In addition a localised soundscape was created by varying the volume of the sound effects in response to the players’ movements around the hybrid gameworld. Audio and visual augmentations were constrained by the resource limitations of the Android development platform which requires the assets to be loaded into memory on demand. The implication of this constraint is that the file size needed to be very small thus hampering the creation of realistic effects and also the assets had to be positioned far

General Overview

Evaluation of the user study results reveals that overall participants were engaged in the game. Feedback from three of the bounded questions illustrate that the majority of participants (79%), agreed or strongly agreed, that it was a fun experience, time passed quickly and they felt engaged by the game. All participants reported varying levels of engagement at different stages of the game. In general, the findings support previous research that indicates immersion is a variable and transient state, which progressively deepens as players became more skillfully engaged in the gameplay [3,4,17,23]. Many players describe becoming more immersed as the game progressed and as they became more comfortable using the UI or accomplishing the challenges and as the story began to unfold. “It was more immersive once I got used to the UI and the story started to link one sighting to the next” “I didn’t feel immersed until near the end, the beginning isn’t so engaging, as I was trying to get used to how to control the game and find the evidence at the same time” Interaction and immersion in gameplay

The user study findings show that by far the most engaging aspects of gameplay were connected to the mastery of the game mechanics. For the most part, players found the search mechanic satisfying and enjoyed the process of physically moving around the environment hunting for evidence of paranormal activity and the sense of achievement when they accomplished the challenge. “The gameplay was lovely, the different modes and the physical moving around to track things down.”

96

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Overall players used the multimodal features of the Radar interface as a guide when searching for ghosts; 79% of participants mainly used the graphical interface but also used audio and tactile feedback to support the visual feedback. Since information about the position of the ghost was presented in different ways, players could choose their preferred method or tools to help them find the ghosts. For example, testers might focus on visual, tactile or audio feedback or a combination all features.

“The most difficult part of the game was the device reliability: sometimes the GPS ‘ghost’ would point me in the wrong direction.” Usability issues occurred due to latency and unresponsiveness of the touch screen and sensors as a result of resource limitations while new assets were being loaded. For example sometimes the zoom function of the Radar became slow to react and therefore participants found it awkward to use and often “required moving slowly and stopping regularly for re-calibration”

“I used the proximity meter (i.e. number readout) to determine my distance.”

Impact of location

“The sound effects and the pulsing of the handset helped point me in the right direction.”

A significant finding of the user study was the impact that the location had on player engagement and immersion. Almost all the participants agree (84%) that the location contributed to the overall game experience because the historical significance of the main site gave context to the narrative, while the atmosphere of the place enhanced the players’ perception of the supernatural theme of the game.

Many players described collecting evidence as the most immersive part of the game because they were simultaneously engaged in the activity of recording evidence while also receiving a reward for progressing in the game in the form of a story fragment.

“The historical significance of the place influenced my perception of the game” Most participants had not previously visited this site and commented that this added a sense of intrigue to the location. “The church had walls, nooks and crannies; also, the alleyway by the side of the church, the high walls, the archway all added to a kind of ‘mysterious’ atmosphere.” Participants confirmed that their experience of the game varied in relation to the ambience of the locations; the aesthetics of the main site were much more conducive to the atmosphere of the game than the commercial streets and housing estates at the periphery of the historic site. The relative remoteness of the church grounds also enhanced player immersion because the location was quiet and less populated than the urban streets and therefore afforded uninterrupted engagement.

Figure 3. Player interacting with ghosts in St.Audeons.

“It felt satisfying. The ability to find audio and images was a good mix. I never got bored searching. The most immersive part was after I’d located a ghost or artifact, and am trying to record it”

“The isolated lane around the side of the church was more atmospheric and hence I felt more immersed”

“Felt most immersed with dialogue and with use of camera, felt engaged when I caught the floating apparitions. Dialogue gave more direction and hence more interactive.” Impact of control

The study shows that GPS and usability issues negatively impacted on the player experience with the result that only 63% of players said they felt in control of the game. In the main, efforts to reduce the negative impact of GPS errors were successful. In particular the strategy implemented in the ‘Lane of Hell’ region, as described in the design section, resulted in the delivery of relevant content at appropriate locations despite lack of GPS coverage. Nevertheless some players did experience bad GPS service and this caused a high degree of distraction and frustration.

Figure 4. Players at St. Audeon’s Arch and the ‘Lane of Hell’.

Players’ noted that the use of headphones helped to engage them in the game by reducing distractions without completely disconnecting them from the ambient sounds of

97

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the location, which were still hear audible with the headphones on.

this helped to engage the player in the game because it communicated the players’ progress.

“The only thing that distracted me was a passing ambulance as I caught a sound ghost”.

“Felt kind of like a personal experience – like it was tailormade for me and the ghost was egging me on to the next clue”

Players found the locations on the busy urban streets outside of the church afforded a less immersion experience due to distractions such as noise levels, passersby and traffic safety issues. It was difficult to hear the audio at the points located near to the roadside due to the traffic noise and some felt that safety was an issue in this scenario. While a few players felt self-conscious playing the game in public, especially if they were in the way of other pedestrians when they stopped in the middle of the pathway to listen to the narrative, for the most part players took less notice of other people in areas such as the park as they felt these areas were already designated for play and one participant commented: “I was fully immersed and didn’t take notice of tourists and workmen around me”.

79% participants felt the game audio sounded natural in the environment and some testers refer to incidences when the game sound blended with the ambient sounds of the environment in a manner that created ambiguity as to which was virtual and which was real. “The natural sounds of the city were kind of a nice extra layer of atmosphere rather than a distraction, there were children playing and a church bell ringing and a little bit of wind.” Players also felt rewarded and satisfied when they encountered and photographed visual assets and their remarks reveal the cumulative success of this game mechanic.

Role-play and aesthetics

“Increasing use of visual (rather than auditory) clues heightened the experience as the game unfolded.”

In general, participants agreed (84%) that the visual appearance and sounds of the UI made the device feel more like a tool for paranormal investigation than a phone. In comparison less participants (68%) reported feeling like a paranormal investigator when they were playing the game and therefore we can conclude that while the audio and visual aesthetics supported the impression that the device was real, the game mechanics did not support role-play to the same extent. Nevertheless, participants described several features of the UI that they associated with paranormal investigation and felt that the headset helped make the role-play feel genuine.

Many participants reported feeling ‘engaged’, ‘entertained’, ‘excited’, ‘surprised’ or ‘satisfied’ when they encountered a visual manifestation of a ghost, others described the visual ghosts as ‘scary’, ‘realistic’, ‘cool’ and ‘good fun’ and one player remarked ‘the effect of it floating there was good, especially the first time when I wasn’t expecting it’. In general, the visual assets, were not as convincing as the audio ghosts. It seems that the limited animation sequence negatively impacted on the user experience and some participants’ remarks reveal that the animations were barely perceptible.

“I did feel like a paranormal investigator, hunting things down was great and twiddling with the knobs to tune in”

“Didn’t seem real. Could have made the ghost appear like it was more than just a picture and part of the scenery.”

“I had the device to find the object and I felt like the ghost hunters from the film”

Narrative Immersion

In addition, and reflecting the cumulative nature of immersion, many participants reported feeling more engaged in the role-play as the game progressed.

The most diverse feedback we received was in relation to narrative; while some players remained highly engaged with the game mechanics and less involved by the content of the story, others found that performing these mechanics gave players access to immersion in the narrative and “when clues began, the game was more exciting”

“After a few discoveries it was easy to get into the game and to feel more like an investigator.” Overall the findings show that participants perceived the game assets, and in particular the audio assets, as realistic and this contributed to player engagement. Participants reacted emotively to the dialogue spoken by the audio ghosts and many described the experience of listening to these ghosts as ‘scary’, ‘spooky’, ‘haunting’, ‘atmospheric’, ‘realistic’, ‘believable’ and as a result ‘engaging’, ‘immersive’ and ‘entertaining’.

“I wanted to figure out how to get the evidence… but I was not really interested in what the evidence was” “Story interests me most. I just like the ‘adventure’ idea of tracking down the information.”

“I found it [the ghost speaking] one of the most immersive features of the game – really enjoyed it.”

Some players noted that gameplay was a more essential aspect to the game but had they had more time they would have explored the narrative elements in more depth, while others stated that they were not that interested in using the casebook because they were more interested in gameplay.

Many players commented that when the ghosts spoke to them, the experience felt very personal and rewarding and

“The interaction with the game was more about achievement ….I would like to play the game again and

98

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

focus more on the story and not be distracted figuring out the equipment.”

focus on the feedback which they found most intuitive. Comparing user response to our two modes of collecting evidence (recording audio and taking photographs) also supports the argument that more familiar, ‘everyday’ interactions are easier to perform in this context because they are more immediately intuitive and serve as more expedient modes of interaction because they rely on players’ existing tacit knowledge [4,17]. These findings compliment the evaluation of REXplorer where players faced difficulties performing gestural interactions due to their novelty and associated learning curve [2]. As with previous research [7] our study reveals that players were less likely to experience immersion when they felt conspicuous playing the game in a public space however we also found that players were less inhibited when performing more ‘everyday’ interactions in public spaces designated for leisure activities. This confirms Jegers recommendation that LBMG should support a ‘seamless transition between the gameworld and everyday contexts’ in a manner that does not break the social norms associated with these contexts [13].

Most players felt they did not have time to use the casebook to find out more about ghosts but some players remarked that if they were playing under normal circumstances they would have used casebook more. Many players admitted to skimming the game and only did what they had to do to get to the end. “I skipped this bit [reading casebook] as I was lagging behind and felt I needed to move on to the next find.” The players who did use casebook reported that the backstories were interesting and increased their empathy with the characters. As with the other game features, immersion in the story increased as the game progressed. “As the game progressed, I certainly did get more immersed. As you find the different clues and items and realise you are following a particular ghost character it becomes more engaging” DISCUSSION

Control

Challenge-based Immersion

Since players enter the game through the game controls, usability issues can have a negative effect on immersion. At times GPS errors disrupted players’ enjoyment of the game and this well documented issue remains for anyone designing a LBMG. However in our user study, despite the fact that many participants did not always feel fully in control of the game many reported high levels of engagement and immersion and this seems to confirm Norman’s theory [3,17] that if the overall experience is interesting enough then users will be prepared to put up with minor usability issues. While some participants experienced difficulties at the beginning of the game, the findings of the study show that as players progressed, for the most part control ceased to be an issue. Having said that our recommendation for UI design for a LBMG is that the interface be made as simple and easy to control as possible because players have less tolerance for learning new interactions in this scenario and the inevitable GPS issues already create a difficulty level that need not be further extended by a complicated interface. Our study confirms Jegers’ recommendation that in relation to control, players in the context of LBMG need to be able to easily pick up the game and begin playing [13].

The results of the study show that engagement in play itself and immersion in performance were experienced by the players as the most immersive aspects of the VGH game. Furthermore the study shows that engagement in the game mechanics is cumulative, not only because as players’ skills increase they become absorbed in a state of flow, but also because by engaging in these mechanics, players become exposed to other gameplay elements such as reward and narrative. We found that players entered the game through the game mechanics and in this sense our study of immersion supports the research we reviewed at the beginning of this paper [3,10,25,27]. Reflecting Ermi and Mäyrä’s evaluation of PC games, we found that when we applied their SCI model of immersion [27] in the context of LBMGs, challenge-based immersion was also the type of immersion most frequently experienced by the players. For many players, recording audio or photographing ghosts was the most immersive part of the game because as well as creating atmosphere and communicated elements of the narrative, collecting evidence helped the players advance in the game. The study also indicates that this form of immersion is more easily achieved in the context of LBMGs when challenges are presented to the player in the form of easily controlled real world interactions.

Augmented Reality

The results of the user study show that choice of location can impact dramatically on sensory immersion in the context of LBMGs and by enhancing and transforming the players’ perception of their surroundings, atmosphere can become a unifying and highly engaging aspect of the game. The findings reveal that the aesthetics of the assets most effectively contribute to the players’ sensory immersion when they become subtly blended with the real world; for example, when the ambient sounds of the game merged with the environmental sounds of the location. The results

Real World Interaction

The results of the study confirm previous research indicating that in the context of LBMGs the most successful game mechanics are those that combine real world activities as part of the core interaction [2, 32]. For example, the search mechanic involves players walking around hunting for ghosts and the game supports a phenomenological approach to this challenge by providing multimodal feedback to players’ movements thus enabling players to

99

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

also reveal that if this juxtaposition is not smooth enough, as was the case with the AR visuals, then the players will perceive it as jarring and unconvincing and hence a disruptive rather than engaging factor. This supports the findings of the evaluation of TimeWarp [28] as reviewed in the related work section. While to some degree the floating effect of the AR visuals was perceived as acceptable within the ghostly theme and context of the VGH game, it is clear that the visuals need to be more integrated into the surroundings in order to achieve a credible visual effect. Stabilising the visual overlays within the scene would also enable designers to employ the cinematic technique of ‘mise-en-scene’ whereby explicit visual attention could be drawn to specific landmarks as the player views them through the lens of the device. This could also address the fact that some players felt the game did not facilitate enough engagement in the surrounding environment.

participants to achieve imaginative immersion to the extent that they almost had to close their eyes in order to block out their physical surroundings [23]. Our study reveals that while many aspects of the gameplay contributed to imaginative immersion, for example roleplay and the atmosphere of the location, the story itself was not a hugely engaging aspect of the game. While players enjoyed the activities associated with discovering the evidence and piecing together the clues in order to proceed to the next find, very few participants spent time exploring the stories behind the evidence, which together made up the over-arching narrative of the game. There are a number of reasons for this outcome, the first being that, as described above, players needed to master the game controls in order to gain access to the unfolding narrative. However the main reason was that this mode of interaction, reading text on a small screen, is inconvenient in an outdoor location-based context. In future developments of the VGH game we intend to focus on conveying all of the narrative through short audio and visual assets and in doing so take advantage of the more accessible descriptive potential of these mediums.

“Paradoxically the game encourages looking at the screen more than the surroundings.” These findings highlight one of the unique challenges of creating presence in the context of LBMGs using the processing power of a mobile phone. Given the limited technical resources of the device and the significant potential of leveraging the location, it seems that at the moment the greatest potential for creating presence and sensory immersion in this context lies in the power to subtly transform the existing characteristics of the location into a sensually engaging gameworld by enabling players to simultaneously focus on virtual and physical parts of the game.

CONCLUSION

In this paper we have observed that player experiences are cumulative. As players learned to gain control of the game mechanics they achieved deeper immersion in the narrative and imaginative aspects of gameplay. Supporting engagement and immersion in LBMGs is also dependent on utilising appropriate real world interactions and carefully selecting locations in relation to thematic relevance, atmosphere, aesthetics, safety issues, lack of potential distractions and social context. By leveraging the existing characteristics of the location, Augmented Reality techniques can be used to transform the surroundings in order to create a hybrid gameworld where the player can become immersed in the flow of play while simultaneously remaining connected to the real world.

Imaginative Immersion

Overall we found that players needed to achieve a degree of effortless engagement before they could fully experience imaginative immersion and therefore this type of immersion is not dissimilar to immersion in the general gaming context [3,10]. The choice of location can influence the level of imaginative immersion not only by supporting the atmosphere and narrative but by also ensuring safety and lack of distraction. In some instances the narrative and audio completely transformed the players perception of the landscape, for example, one participant referred to the ‘wistful description of the River Liffey’ and the soundscape that described the original course of the river.

ACKNOWLEDGMENTS

We wish to thank the rest of the VGH team (Roisin Cotton, Daniel Crowley, Alan Duggan, Tina Hedayet, Soren Kristian Jensen and Sean O’Reilly), the NDRC and all those who participated in the user study. REFERENCES

“The church and surroundings were smashing, really wonderful moment standing outside it and being told about the shores of the Liffey once being underfoot.”

1. Adams, E. and A. Rollings. Fundamentals of Game Design. Prentice Hall, Upper Saddle River, NJ, USA, 2006.

We speculate that part of the reason it was easy for players to imagine this alternate scene was the fact that the location where the game was situated was not that dissimilar to the one being described and was conducive to imaginative immersion due to the aesthetics of the location and lack of distractions. These findings contrasts with the evaluation of Riot! 1831 where researchers found that lack of visible structures that related to the experience made it difficult for

2. Ballagas, R., Kuntze, A. and Steffen P. W. Gaming Tourism: Lessons from Evaluating REXplorer, a Pervasive Game for Tourists. In Pervasive Computing 2008, LNCS vol 5013, Springer (2008), 244-261 3. Brown, E., & Cairns, P. A grounded investigation of game immersion. In Proc. of CHI 2004, ACM Press (2004), 1279-1300.

100

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

4. Calleja, G. Revising Immersion: A Conceptual Model for the Analysis of Digital Game Involvement In Situated Play. In Proc. of DiGRA Conference (2007).

19. Murray, J. H. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. Free Press, New York, 1997. 20. Nisi, V., Oakley, I. and Haahr, M. Location-Aware Multimedia Stories: Bringing Together Real and Virtual Spaces. In Proc. of ArTech, ACM Press (2008).

5. Cater, K., Hull, R., Melamed, T., and Hutchings, R. An investigation into the use of spatialised sound in locative games. In Ext. Abstracts CHI 2007, ACM Press (2007), 2315 – 2320.

21. Norman, D. A. The design of everyday things. New York: Doubleday, 1988.

6. Csikszentmihalyi, M. Flow: The Psychology of Optimal Experience. New York: Harper and Row, 1990.

22. Paterson, N., Naliuka, K., Jensen, S.K., Carrigy, T., Haahr, M., Conway, F. Design, Implementation and Evaluation of Audio for a Location Based Augmented Reality Game. In Proc. of AMC Fun and Games, ACM Press (2010).

7. Diamantaki, K., Rizopoulos, C., Charitos, D & Tsianos, D. “LOCUNET: Location-based games as media communicating social meaning”. In Souza e Silva, Α. & Sutko, D. (eds.), Digital Cityscapes: Merging Digital and Urban Playspaces, Peter Lang Pub, 2009.

23. Reid, J., Geelhoed, E., Hull, R., Cater, K., Clayton, B. Parallel Worlds: Immersion in location-based experiences. In Proc. of CHI 2005, ACM Press (2005).

8. Dourish, P. Where the Action Is: The Foundations of Embodied Interaction. Cambridge: MIT Press, 2001.

24. Reid, J. Design for coincidence: incorporating realworld artifacts in location-based games. In Proc. of ACM DIMEA, ACM Press (2008).

9. Dow, S., Mehta, M., Harmon, E., Mateas, M., Presence and Engagement in an Interactive Drama. In Proc. of CHI 2007, ACM Press (2007).

25. Rigby, S., Ryan, R., The Player Experience of Need Satisfaction ftp://ftp.immersyve.com/PENS_Sept07.pdf

10. Ermi, L. and Mäyrä, F. Fundamental Components of the Gameplay Experience: Analysing Immersion. In Proc. of the DiGRA, (2005) 15-27.

26. Rouse, R. Game Design: Theory and Practice. Wordware Publishing, Texas, 2005.

11. Hendrix, C., & Barfield, W. The sense of presence within auditory virtual environments. Presence: Teleoperators and Virtual Environments, 5(3) (1996), 290-301.

27. Salen, K. and Zimmerman, E. Rules of play: game design fundamentals. MIT Press, Cambridge, Massachusetts, 2004. 28. Seah, M. and Cairns, P. From Immersion to addiction in videogames. In Proc. of HCI 2008, vol 1 BCS, 55-63.

12. Herbst, I., Braun, A. K., Mccall, R., Broll, W., Timewarp: interactive time travel with a mobile mixed reality game. In Proc. MobileHCI '08, ACM Press (2008), 235-244.

29. Slater, M. and Wilbur, S. A framework for immersive virtual environments (FIVE): speculations on the role of presence in virtual environments. Presence: Teleoperators & Virtual Environments, Vol. 6 Issue 6, (1997) 603-617

13. Jegers, K. Pervasive Game Flow: Understanding Player Enjoyment in Pervasive Gaming. Computers in Entertainment. ACM Press (2007).

30. Souza e Silva, A., and Sutko, D. M. Digital cityscapes: merging digital and urban playspaces. Peter Lang Publishing, 2009.

14. Jegers, K. Pervasive GameFlow: Identifying and Exploring the Mechanisms of Player Enjoyment in Pervasive Games. Ph. D. Thesis, Dept of Informatics, Umeå University (2009).

31. Sweetser, P. & Wyeth, P. GameFlow: a model for evaluating player enjoyment in games. In ACM Computers in Entertainment, 3(3) ACM Press (2005), 124.

15. Lankoski, P.; Heliö, S.; Nummela, J.; Lahti, J.; Mäyrä, F.; Ermi, L. A Case Study in Pervasive Game Design: The Songs of North. In Proc. of NordiCHI, ACM Press (2004), 413–416.

32. Waern, A., Montola, M., Stenros, J. "The three-sixty illusion: designing for immersion in pervasive games". In Proc. of CHI 2009, ACM (2009)

16. Lindley C. Game Taxonomies: A High Level Framework for Game Analysis and Design. Gamasutra (2003) http://www.gamasutra.com/view/feature/2796/game_tax onomies_a_high_level_.php

33. Wetzel R., McCall R., Braun A-K., Broll W. Guidelines for Designing Augmented Reality Games. In Proc. of the 2008 Conference on Future Play, ACM Press (2008).

17. Lindley, C. A. 2004. Ludic Engagement and Immersion as a Generic Paradigm for Human-Computer Interaction Design. In Proc. of ICEC2004, Springer (2004), 3- 13.

34. Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence-Teleoperators and Virtual Environments, 7(3), 225-240.

18. McMahon, A. “Immersion, Engagement and Presence: A Method for Analysing 3-D Video Games.” In Wolf, M.J.P and Perron, B. (eds.), The Video Game Theory Reader. London: Routledge, 67-86, 2003.

101

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Understanding the Everyday Use of Images on the Web Boon Chew*, Jennifer A. Rode+, Abigail Sellen^ *University College London, +Drexel University, Microsoft Research Cambridge UCL Interaction Centre, 8th floor Malet Place Engineering Building, London WC1E 7JE [email protected] ABSTRACT

aspect of our personal lives.

This paper presents a qualitative study of domestic Webbased image use, and specifically asks why users access images online. This work is not limited to image search per se, but instead aims to understand holistically the circumstances in which images are accessed through Webbased tools. As such, we move beyond the existing information seeking literature, and instead provide contextual examples of image use as well as an analysis of both how and why images are used. The paper concludes with design recommendations that take into account this wider range of activities.

For some time now, researchers in Information Retrieval (IR) have argued that we need to look beyond simple models of search to support what people really do on the Web. They have argued that a model which assumes that users formulate goals, specify queries, and then look for results to satisfy those goals oversimplifies what people do with the Web [9, 15]. This is sometimes called “directed search” or “look-up”. A more recent emphasis on “exploratory search” stresses the ways in which users are often opportunistic when they use the Web, and that their strategies can be heavily reliant on browsing and serendipity [9, 15]. Others have gone one step further pointing out that “searching” or even the broader term “information seeking” assumes that Web use is purposeful, even though research shows that Web users may sometimes have no explicit goals at all [23].

Author Keywords

Image search, domestic, photowork, Web use, field study, diary study ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

Despite this, the simple look-up model continues to dominate in terms of driving the underlying design not only in text search, but also in specialized image search tools: users enter a set of keywords, and a rank ordered array of images is returned. Of course there are many specialized sites which focus on images (such as Flickr and Facebook) and also innovations in the way images are searched for (including searching by color [26, 10], or other visual features [10, 19]), but the fact remains that mainstream image search, at least in terms of the user experience, is not significantly different from any other kind of keyword search on the Web.

INTRODUCTION

Despite widespread use of images on the Internet [1, 7, 20], little is known about people’s behavior or motivations when accessing images via the Web and in the course of everyday life. There are studies on image search, but these tend to focus on innovations in the technology itself [e.g. 8, 19] or the performance of different techniques as assessed by various kinds of objective measures such as the relevance and ranking of returned images given a set of keywords [e.g. 2, 20]. Further, in characterizing the behavior in question as image “search”, this, in itself, makes implicit assumptions about what users are doing. In this paper, we want to show how a deeper exploration of what people do with images, and why they access them from the Web, goes beyond simple notions of search. We want, instead, to put on-line image use in the broader context of users’ activities and intentions. We choose a domestic context here to reflect the fact that Web use, including image use, is now firmly entwined in every

This study asks whether, like general Web search, this view of what users do with images on the Web is, in fact, too narrow, and thus whether we are missing opportunities to deliver more compelling kinds of Web-based tools. It does this by looking at the range of behaviors and motivations for engaging with images online from a user’s perspective. We limit the scope of this study in the following ways: First, we focus on the everyday ways in which people search for images on the Web from home, as opposed to purely professional use. We also constrain our enquiry to images which are not within people’s own personal collections, such as home photo and video collections. Largely this is because this topic has been covered extensively elsewhere [e.g. 12, 22] and the issues raised are likely to be quite different from searching for images for which one has little familiarity. Finally, there is no doubt that one of the main uses of images in the domestic sphere

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

102

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

is pornographic [3, 24]. Because of the highly personal and sensitive nature of this aspect of image use, we chose not to include this in our work.

in-depth interviews with users to delve into the strategies, motivations, and contexts where individuals engage with images on the Web. Our approach was to conduct in-depth interviews, but also to ground these using a careful record of participants’ activities over an extended period of time— a method used previously to investigate more general Web use behavior [23]. This way, one can capture a realistic “slice” of image search activities in the context of everyday life, along with the rich descriptions and stories that accompany them. This approach generally means using a relatively small sample of participants, but gathering a detailed set of data about each person in question.

Our contribution is to provide deeper insight into the ways in which people engage with images on-line. We will refer to this behavior as image “access” or “use”, to reflect the fact that, as we will show, these behaviors go beyond searching and browsing. In addition to providing this deeper understanding through contextual examples, a second contribution is to use this to show how the design of Web-based tools might better support the ways in which people want to use images from the Web. Accordingly, we develop a framework for classifying the various motivations for image use, and reflect on the implications of the range and diversity of these activities.

METHOD Participants

With this aim we undertook a study of nine individuals’ domestic image use practices, who were recruited through snowball sampling. Our participants included five men and four women ranging in age from 24 to 59: one participant was under 25, four were aged between 26-35, three between 36 and 45, and one over 46. With the exception of one person (a graphic designer), we recruited participants with professions that were not image-intensive. This included: a bookshop assistant, an accountant, a civil servant, an eco-blogger, a recruitment agent, a student, a homemaker, and a retiree who actively volunteered for charities. Our main criterion for selection was also that they reported using images from the Web at least 5 times a week. All participants used the internet for at least 30 minutes a day and were familiar with general Internet activities such as searching for content on the Web, browsing image galleries, and manipulating images on the Web (e.g. downloading, enlarging, and viewing).

RELATED WORK

Much of the literature on image search is technical in nature, focusing on new ways in which to increase the relevance of retrieved images through, for example, better matching of queries to images [5], better tagging of images with meta-data [27], or ways to do content-based image retrieval [13, 29]. New features and interfaces too, form a considerable body of work, exploring concepts such as new ways to cluster and present results, [see 1 for a review]. New and specialized image search tools are also appearing [19] but most are specialized to particular communities. In terms of understanding users’ intentions or behaviors when using images on the Web, there is much less pertinent research, and most of this concerns image search behavior in organizations and professional contexts [1, 2, 6, 16, 30]. Here, though, there are some interesting findings, pointing to the fact that, in contrast to general Web use, image professionals [16, 30] are more heavily reliant on browsing. More general analyses of image search logs [1, 10] also reveal that, in comparison to general Web search, image search relies on rapid browsing of many more pages of search results per query, involves the exploration of many more paths per query, and much more iterative modification of queries per session. One might reasonably conclude from this that image search is either more reliant on flexible navigation and exploration than mainstream Web searching, or that the tools are less successful for helping users find what they need.

Procedure

The study employed a diary study method [21] similar to the one used previously in Sellen et al. [23]. A diary study was chosen over an ethnographic approach given the likelihood of the experimenter’s presence influencing the environmental validity of findings combined with the impracticality gaining access for 24/7 ethnography. On a simple diary form, participants were asked to log a short description of each Internet activity involving images. Participants were told to log all Web-based activities in which images (such as illustrations, photos, and graphics) were deemed to be important in the accomplishment of their tasks. This could include using images in order to accomplish a goal, finding an image as an end goal in itself, or engaging with images in other ways that might not involve explicit goals. When in doubt, participants were encouraged to include rather than exclude an activity. They were also asked to exclude activities which involved images from their own personal photo collections. In addition to a short description of each activity, they were also asked to log the activity start time. They were told that these entries only needed to be sufficient to trigger a longer discussion of each activity in the ensuing interview. When necessary, participants’ browser histories were used to fill in gaps in the data and to validate the diary entries.

In terms of more in-depth studies of users’ behaviors and intentions, most studies of the everyday use of images outside the workplace focus on people’s use of their own personal photo collections. For example, Kirk et al. [12] provided an overview of “photowork”, or activities people perform with their digital photos after capture but prior to end use such as sharing, focusing on practices in the home. Other studies have addressed other aspects of personal photo use in the home, such as browsing [11], archiving [22] and sharing [17], particularly in the use of photo sharing Websites such as Flickr [14, 18]. Taken together, whether in professional contexts or otherwise, there are very few studies which have conducted

103

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Participants recorded an average of approximately 3 activities per day (see Table 1).

understanding of the value of images and the activities surrounding it.

Initial interviews introduced participants to the purpose of the study and the diary study method, and were used to gather demographic information about the participants and their households. After initial interviews, participants were contacted via phone or in-person for follow-up interviews approximately twice a week, each interview lasting between 30 to 60 minutes. The frequency of interviews depended on how many activities had been logged in a day. If fewer than 4, then the interview would be delayed until more activities were recorded. However, the interval between interviews was never less than 3 days, or more than 9 days.

Participant Pseudonym Elle Sue Florence Gracia Colin Elizabeth Andy Tom William

Overall Total 18 17 18 22 20 21 17 16 30

No. of days per person 3 9 12 8 10 8 9 9 8

Average No. of Imagebased Activities per day 6 2 1.5 3 2 2.5 2 2 4

Table 1. Overview of number of image-based activities logged per participant. (Participants’ names have been changed to protect their identities.)

During these interviews, participants were asked questions to explore both the “why” and the “how” of each activity logged, including why they were looking for specific images, how often the activities occurred, the process of the activity, internal or external influences on that activity (e.g. an email reminder or a high-level goal), and other contextual aspects (e.g. the environment or situation as the activity was taking place). Interviews were audio-recorded. All participants were given a shopping voucher for their participation on completion of the study. .

Table 1 provides a summary of the number of activities collected for each participant over the course of the study. The total number of image use activities in our corpus was 179, ranging from 16 to 30 activities per person over the course of the study. The average number of image use activities across different participants ranged from 1.5 to 6 per day. Strategies and Tools for Image Access

The data show that participants used one of four main ways to access and engage with images on the Web: they either used image-specific search tools, used social networking sites, visited domain-focused Websites, used Websites featuring maps, or used some combination of the four. The most frequent method was the use of domain-specific Websites that contained images (e.g. news, blogs, ecommerce Websites). Image-specific search tools (in this case exclusively Google Image Search) were used less than 10% of the time, despite its straightforward interface and relative popularity.

FINDINGS

We begin by providing an overview of the activities we examined, including the operational definition of image use we adopted for the purpose of the analysis. Next we look at the “how” of the activities we explored, highlighting the technologies and strategies used by participants in accessing images, and summarizing key observations regarding commonly used Internet sites. Third, and most importantly, we examine the “why” of these image use activities, and present an analytical framework (a taxonomy) for understanding users’ motivations for domestic image use.

Image-Specific Search Tools

Google Images is a general image search tool that retrieves images based on keyword queries, and the only such tool we saw used in the course of the study. Only five participants reported use of Google Images. They used it infrequently (less than 10% of the time) and mostly to support quick searches for specific images that were easily identifiable via a keyword (e.g. “fitflops”, “Van Gogh”). One participant, Gracia, used Google Images more frequently compared to other participants. She spoke English as her second language, and sometimes used Google Images to learn new English words. In one example, she used several keyword combinations to physically describe an outdoor cooking stove in order to find recognizable images and subsequently “guess” the correct term. Another participant, William, attempted several keyword combinations on Google Images to discover humorous pig photos. While this is a small sample, this suggests that tools such as Google Images are appropriate for goal-directed searches where users have ideas for a specific keyword and expected search result, occasionally supporting exploratory searching involving

Overview of Image Use Activities

In our instructions to the participants, and in our analysis, we sought to gather information on any activities in which the Web was used to obtain image-based content of significant importance or value to the participants during the activity. For example, if an image was present on a Website but was ignored, we did not count this as an activity of interest. Obviously, these judgments are subjective. However, we relied on participants to make those judgments, taking at face value the explanations for each activity that they offered up. This means that the analysis that follows includes activities where finding an image may not necessarily be the end goal. It also means that many of the activities considered here may not necessarily be “searching” or “browsing” tasks--images may be accessed for other purposes. This broader definition of image use rather than search, per se, aims to embrace a larger spectrum of image-related activities in order to get a more grounded and realistic

104

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

keyword variations. However, the fact that this is done infrequently either means that there are other effective ways of answering such queries, or that the need to do keyword search on images happens relatively infrequently.

engaged in more focused search. For example, William routinely used eBay’s advanced search feature to routinely locate new listings of items to bid on, followed by scanning thumbnails that matched his specific requirements.

Social Networking Sites

The examples above show how domain-specific Websites can be more effective than general image search tools (e.g. Google Images) due to features that are designed to support search strategies that are relevant to users of that site and the tasks that they engage in (e.g. searching for new items to bid on eBay).

Facebook, while primarily designed for social networking, provides a comprehensive photo gallery feature that members can use to upload images. Some participants made use of Facebook photo galleries to see what their friends were up to and posted comments on certain photos as a means of informing them that they had seen the photos or to amuse them. There were elements of serendipitous discovery as participants often visited photo galleries by clicking on a thumbnail of that gallery located on their Facebook home page. The activities involving Facebook showed how behavior is motivated by participants’ goals to connect with friends [28], and how the thumbnails and photo galleries catered to this need.

Maps

Maps are a unique type of image use, as they are visual in nature and afford various visual tasks such as exploration and recognition. The study observed the use of interactive maps (e.g. Google Maps) as well as illustrated (static) maps. They are distinguished from domain-specific Websites because they were used to provide confirmation and assistance for geographical and spatial navigation. For example, maps were used to confirm the location of a particular landmark (e.g. the location of a real estate property) or to obtaining the general direction to walk from a specific venue.

Only two participants used Flickr during their image related activities. One participant visited Flickr to view photos from her friends and sister about once a week to find out news about them, while another participant browsed Flickr mainly for amusement or inspiration.

Summary

Domain-specific Websites Containing Images

In summary, we can see that there were many different ways in which the participants in the study obtained images they were interested in, or otherwise used them in the context of larger activities. More than half of these were interwoven with routine Web use, often leading to serendipitous browsing and exploration. Others involved specific kinds of search strategies tailored to particular tasks, and thus making use of specific Websites designed for those tasks. Interestingly, fewer than 10% of the activities we examined actually involved the use of specially designed tools for image search. In order to understand this more deeply, we turn now to examining users’ motivations behind these activities.

The majority of image use activities involved domainspecific Websites such as news sites, company Websites and e-commerce sites (on average 42% of the time). Most of these were routine activities, such as reading the news or checking the weather, sometimes leading to other activities through serendipitous discovery of images. Some activities were occasional, but focused around a topic of interest such as property hunting, whereby participants were searching for specific types of images such as property floor plans. In all cases, images were important or even critical parts of the activities in question. It became clear that one reason that many of the image use examples were linked to domain-specific Websites was that they were interwoven into participants’ normal routines when using the Web. For instance one participant routinely visited the BBC Website for news and the weather. Another looked at Glastonbury event photos which led to deeper exploration of articles and more images relating to artists and bands.

Motivations behind Image use

The primary focus of this research was to understand the underlying motivations when people engage with images on the Internet. An analytical framework consisting of thirteen search motivation themes subsumed under four higher-level categories was constructed using grounded theory. These themes are concepts that were based on codes we assigned to each activity based on the various motivations behind how images were used. Some activities fall under multiple themes, and thus are not mutually exclusive. Table 2 summarises these categories and shows the overall number of examples in the corpus falling into each high level category. We now describe each of these categories, along with their sub-categories, in more detail.

Other times, participants were observed making routine visits to certain Websites in the course of more directed search. Elizabeth’s use of a property agent’s Website to examine properties for purchase, which was characterized by her search for floorplans and photos of the property, is an example of this. In this case, she commented on the importance of images: “The photos I usually look for are to get a better feel for the house, and the floor plans are very important as well–especially for knowing how to decorate the place”.

Learning and Research

The most frequent motivation for image use can be summarized as for the purpose of “learning and research”. Here we found a whole range of activities involving images that were undertaken in order to gain new insight and ideas on particular themes, topics or concepts. However, within

Some participants also performed routine tasks involving ecommerce sites (e.g. Amazon, eBay) and, once there,

105

Full Papers

High-level Categories

Proceedings: NordiCHI 2010, October 16–20, 2010

Frequency of Corpus Overall

Learning and Research

123

Image Access as Secondary goals

52

Recreating or Connecting to Remote Experiences

77

Images as the Objects of Communication

20

The key to these activities was that they were specific and specialized to the participants’ particular interests, and could be gleaned from sites which provided additional contextual details along with the images. This tended to happen as a matter of routine, where sites that were considered both trusted and familiar were visited regularly to check for new content.

Subcategories Images to support ongoing interests or research Images to satisfy curiosity Images for visual discovery Images as ideas Images as alternative answers Images as aids for geographical orientation Images as indexes Images for connecting to remote places Images for re-living past experiences Images for connecting to people and their lives Images to connect to personalities Images to do the work of communication Images for social interaction

Images to Satisfy Curiosity

Participants occasionally engaged in more focused and directed search for images to satisfy their curiosity about a particular issue or question, which can be considered another way in which images were part of learning. Fieldnotes for Tom: Tom was reading that people were claiming that TV presenter Christine Bleakley was older than her reported age. He did a search for her photos out of curiosity on various Websites through Google text search, but he couldn't find anything conclusive. He said in response that he often engages in random browsing on a daily basis–“it’s quite a nice thing to do before you go to sleep”. Fieldnotes for Andy: There was a picture of Nick Grinthorpe on the rotating banner at the top of the BBC Website, and Victoria Hesketh was next to him (of whom he was a fan). He clicked on the image which led to an article, and there were about 7 to 8 photos of her playing for a show, and he looked at them all.

Table 2. Framework for motivation for image use including the total number of instances of the corpus falling into each high level category. (Note that any one activity can fall into more than one category). this larger category, we found four main reasons why this occurred, as we shall describe. Images to Support Ongoing Interests or Research

Both Andy and Tom explained that their curiosity led them to look for images, confirming previous work which found that image search queries were often motivated by “curiosity or pursuit of entertainment” [7].

Similar to Marchionini’s concept of “searching to learn” [15] or Sellen et al’s “information gathering” [23], many image-based activities were in support of deepening or updating one’s knowledge with respect to existing interests, hobbies or research topics. These activities were characterized by regular, repeated activity around specific topics or domains which represented a significant part of the overall corpus (40% overall). Whether it be about parenting, modeling, sports, family history or politics, the use of images to support ongoing interests formed anywhere from half to three quarters of the examples of image use for learning and research.

Unlike the previous category, where images were entwined with regular visits to familiar sites, this kind of image use tended to be more focused around particular queries. This was reflected in the way this was accomplished, namely, mostly through visiting various Websites found through regular text search, or by exploring further on the site where the object of curiosity was found. Images for Visual Discovery

Moving on from supporting particular themes or topics, sometimes learning derived from the viewing of large collections of images where similarities and differences in visual features played an important role. This kind of learning was characterized by novelty for participants, where they specifically commented that they learned something new.

For example, some participants who were looking to purchase property consistently examined thumbnails for details such as the condition of an apartment and the external area of the location from apartment specialty sites. Elle, who was a part-time model, regularly followed the work of models and designers, and Gracia often searched illustrators’ portfolio Websites.

Fieldnotes for Sue: Sue was searching on Amazon for "meat mincers" but was unsure of what meat mincer she needed to get. As she was browsing the search results, she realised that there were two types of meat mincing devices – a clamp-type and a table-top type. In the scenario above, it was obvious from the images that there were two types of meat mincers from the product photos returned, as they were visually distinguishable.

Fieldnotes for Elle: Elle received an email from [model A], so she went to modelmayhem.com to see [model A’s] photos - she wanted to see the new work she had been doing since the last time she chatted with her. Later, on Facebook, she viewed photos of another model, [model B], after posting a comment on her wall. She looked at her photos because "she's gorgeous and a very nice person" and that she looked like a celebrity. Fieldnotes for Gracia: Gracia explained how she was looking for new work from [photographer A] on his Website. She found some new art there, then started searching for more of his work on Google Images, which she said was easier as it catches her attention.

In another example, as mentioned earlier, William used Google Images to search for humorous images of pigs by attempting different keyword combinations. Here, he used the phrase “weird pig”, and discovered an image of a pig with tusks:

106

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the customer, who was in front of him. The whole process took about 5 minutes.

Fieldnotes for William: He went to Google image search, tried to think of a variation of pigs that he hadn't used before, and entered "weird pig" and got a lot of results… He selected these photos because he'd never seen anything like them before - one was of a pig with tusks – a very striking image, "a very exciting pig".

These activities show how images can sometimes be more efficient than text in delivering answers or offering up information. Having the choice of text or image in both of the cases described allowed for greater flexibility and efficiency.

The element of discovery in both scenarios depended on the visual similarity or dissimilarity of the images (meat mincers and pigs) within the search results. Keyword queries also allowed participants to experiment with different search outcomes in order to explore these features along various dimensions such as shape, type, size, and so on.

Images as Aids for Geographical Orientation

Images can act as navigational aids, especially in combination with maps, to confirm the location of a particular landmark or for orientating oneself within a map. Although participants were already familiar with the location, they used images to obtain visual cues (e.g. landmarks, directions) to aid the physical navigation of a place.

Images as Ideas

A final way in which images were used for learning and research was in a more inspirational capacity. Rather than looking for information updates, or accruing additional knowledge about familiar topics, on the contrary, a few participants browsed images to be inspired and get new ideas for creative projects:

Fieldnotes for Florence: Florence thought she would drop by the Japan Centre to buy things on the way home. On her Web browser, she searched for "google maps japan centre london", which led her to the Japan Centre Website. On the “location” page of the Japan Centre site, there was an embedded Google Maps widget, which she used to find out where the Piccadilly Tube station was to get her bearings right. She also checked the address of Japan Centre, and noted the photo of the front entrance of Japan Centre at the top of the page because “it [was] easier to recognize the place”. As she closed the browser window, she made a mental note of the address, the photo of the entrance, and that it was on Piccadilly Street.

Fieldnotes for Gracia: Gracia likes to look at other illustrators’ work to learn from them or to see how far she's grown as a graphic designer. She called this constant ogling at other people’s work a "fixation". Fieldnotes for Sue: There were a lot of beautiful images on this blog — she would look at the images before reading the content, as the images were very important to her. She visits the blog regularly because she wants to update herself about DIY, recycling and design.

The above illustrates Florence’s use of the shop’s entrance as a way to identify the correct place. She also visually used the map to identify direction, match the address with the geographical space, and presumably the distance of travel as well. It is also interesting to note despite amount of information, she felt confident not to write it all down for future reference, as though she was depending solely on her mental model of the journey, made up of visual parts such as the shop’s entrance and pre-existing knowledge of the area. This shows how images can be useful placeholders for things that are useful for judging geographical placement, orientation, distance and meaning.

Here, the important aspect of these activities was finding large collections of images which would surprise and be stimulating. Key here was the ability to browse quickly through many different high quality images around very high level topics, rather than particular kinds of objects. Image Access as Secondary Goals

Images are sometimes used to facilitate other activities. This is characterized by an activity where a participant has a primary goal, but performs sub-tasks where images are used. There were three distinct types of scenarios where this pattern was observed, as we shall describe.

Images as Indexes

Because images are often easier to visually scan, participants made use of images as indexes to locate an item on lists or grids, such as in search results, product listings and catalogues. For example, Elizabeth’s search for a radio program on the BBC iPlayer Website returned search results with illustrations or artwork representing each program. Because she recognized the artwork belonging to the radio program she wanted, it was relatively easy for her to locate that program.

Images as Alternative Answers

In some activities, images were used as alternative ways of quickly finding an answer to a query. For example, Gracia’s recipe search for an authentic Mexican dish returned American versions of the recipe, which she didn’t want: Fieldnotes for Gracia: Gracia first looked for recipes for “arroz a la mexicana” via the main Google search but the results returned inauthentic versions of the dish. She then used Google Images and looked for images of the dish that looked authentic, and clicked on the image that led to the correct recipe.

A similar pattern was observed with Tom and Colin who identified specific DVDs they wanted on sites like play.com and find-dvd.co.uk just by looking at the cover art. This activity of visual browsing was essential as each cover art is unique, and subtle differences in the cover art can make a difference in actual product features such as whether it is a special edition DVD or a boxed set. Again, this represents an economical alternative to scanning

Another example was William’s search for a specific book for a customer waiting directly in front of him: Fieldnotes for William: He was searching for a specific bible for a customer, and the book supplier had a catalog online. He went to the Website and searched for "pocket bible". He clicked on a few options, and the picture was there, so he showed it to

107

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

headline and descriptive text for entries in a sequence, especially when it involves visually scanning for a known image or visual pattern.

These examples show how the images can provide a sense of nostalgia. It seemed that seeing details such as “bits of grass” were critical to the experience.

Recreating or Connecting to Remote Experiences

Images for Connecting to People and their Lives

Participants also used images to understand, connect to or imagine experiences that were otherwise remote, either spatially or temporally. Here, the rich visual nature of images allowed participants to experience Web-based information in ways that text-based content could not. Here, quality and fidelity of the images was important as we will discuss.

Images also represented a way for participants to get to know their friends better, or keep abreast of their whereabouts and activities. These activities often occurred regularly, repeating on a daily to weekly basis. An example of this was Florence’s regular visits to her sister’s photos on Flickr. Fieldnotes for Florence: Once a week, Florence checks her Flickr account to check on new photos from friends. She decided to visit Flickr because her sister was involved in a fashion show recently and so might have uploaded some photos… [However], she didn't see anything new… and left. She uses Flickr as a way of checking up with her sister.

Images for Connecting to Remote Places

Images were sometimes used by participants to “get a feel for a place”, referring to the sense of wanting to experience a specific physical location not previously visited. In one activity, a participant was looking at his friend’s photos of Wembley Stadium on Facebook:

The scenario above illustrates the habitual process participants often went through to get news. In this case, the images represented events or activities pertaining to the participant’s friends or family members. This was a common observation with participants using Facebook as well, which was often accompanied by social interactions such as comments or tagging [4].

Fieldnotes for Colin: Colin found it interesting how his friend managed to capture Michael Owen taking a goal at Wembley Stadium, as well as photos of players celebrating the goal. He had never been to the stadium before so it was nice to see inside. There were very close shots of the player and other players celebrating. He didn’t think you could get so close to the station. It made it feel very exciting.

Images for Connecting to Personalities

This involved multiple photos of a given location showing various perspectives of the space, particularly from a perspective seen from someone else (his friend). Details like the seat numbers, close-up photos of the players, and scoreboard helped Colin to visualize the experience of being at the stadium.

Some participants used images in order to connect to people they didn’t know personally, such as celebrities— not as gossip, but to somehow feel closer to them, or understand them. For instance, during our study Michael Jackson passed away, and several participants used viewing photos as a way of understanding his personality and the stories or situations surrounding his life.

Images for Re-living Past Experiences

We also found that images helped as a way of reconnecting people to a past experiences for familiar places from their past. This was the case even though they were not personal photos as most work on reminiscing assumes [25]. For example, Florence was informed of a snowstorm that hit the town of Dunedin, New Zealand, where she had previously lived:

Fieldnotes for Sue: She found that Michael Jackson died when everyone was talking about the news on Facebook… While she read the news, she looked at the pictures that accompanied the photos, all the while asking – “Did he really die? How did he die?”, reflecting that she was learning about this personality through the photos as well… So the photos helped her see his condition just before his death… Finally, she said she was satisfied from piecing together a believable story after browsing through all those photos.

Fieldnotes for Florence: Florence had heard from a friend that there was a snowstorm in Dunedin, New Zealand, where she used to live. She enlarged every thumbnail because it was a better experience, saying that it was more fun and satisfying and that she could see more details, such as bits of grass, and being able to identify which building that was. She missed Dunedin, and the photos helped her re-live the experience.

The above illustrates how some images have narrative qualities that are also valuable in helping users connect to the experiences of people. Here the participant was looking at the details of Jackson’s face to understand the discussions surrounding his plastic surgery and skin condition—fidelity was critical to the task at hand. As users sometimes hypothesize about personalities’ lives, these photos help users build believable stories surrounding these figures.

Another example showed how Elle was browsing through photos of a hotel she stayed in once: Fieldnotes for Elle: Elle loved looking at the photos of the hotel she stayed at one time, saying how it was “such a beautiful place”. She says she keeps looking at photos like these to remind herself of her goals in life.

Images as the Objects of Communication

The final way in which images were used was to facilitate communication and social interaction. This occurred in two ways—one where images did the work of communication, and the other where images facilitated social interaction.

Colin browsed through his friend’s photos of a concert he also attended to connect with that experience again: Fieldnotes for Colin: He was viewing other's people's Blur concert photos. He joyfully reminisced about being there as well, and the images helped him connect back with that experience.

Images to Do the Work of Communication

Images can sometimes be more effective than words in communicating. This example of a conversation between

108

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Florence and her flatmate involved learning that Van Gogh resembled someone they both knew:

to speculate on why this might be. The fact that image search tools are designed to direct users to unfamiliar sites via keyword queries may go some way to explaining why they were used so infrequently during our study. Most of the image-based activities we logged tended to involve routine and familiar activities (61% overall). When seeking out particular images, users may be more adept at finding them quickly through familiar sites. In addition, given that images were often used in the course of other activities, rather than as end goals in themselves, suggests that a model of “search” is simply not a good fit in most cases where images are used on the Web. This leads on to a second main finding, which is that domain-specific Websites seemed to be the most important way that the images were engaged with (42% overall). Here it was clear that what was important was the whole array of features that these Websites deliver, often specialized for the kinds of activities users want to carry out where images happen to play a role. The use of retail sites, property sites, social networking and map sites are all examples of where images most effectively accessed in the context of other kinds of activities.

Fieldnotes for Florence: Florence’s flatmate found out that Florence didn’t know who Van Gogh was, and said that Van Gogh bore a striking resemblance to someone they both knew and asked her to do a search for the artist. Florence did a quick scan at the Google Image search results and she pointed to a Van Gogh self-portrait and asked her flatmate, "is it that one?" and the flatmate said yes and they both laughed at it because it did remind her of someone that they both knew.

Here, the fact that no explanation was needed of who the image resembled points to the effectiveness of images for some kinds of communication. Of course, the goal was purely to amuse, but the example shows how images can sometimes speak differently than words. The example of William’s search for pigs (previously discussed) also illustrates this pattern, as his intent was to email the pig photos to his colleague to humor her. Fieldnotes for William: He found two images which he planned to send over the next few days... Another picture [he found] was one of a pig bikini - he knew this would make his friend laugh. It was very amusing because it fit "her sense of humor".

This, in turn, is why understanding specific needs, tasks, or motivations is so important if we want to understand image use, leading us to discover a whole diversity of reasons why people used Web-based images. Some of these did involve finding answers to specific questions via unfamiliar sites, but most did not. Rather than question-answering, we found it more useful to think about how images were used to build on ongoing learning or interests, how they could spark discovery or inspire, how they could connect people to remote experiences, or could be used for quick navigation or communication. While prior work has aimed to create better tools for searching [13] and browsing [5, 19, 29], these types of interactions reported in this study, we would argue, point to more diverse kinds of tools, and highlight other kinds of important criteria for assessing these tools beyond keyword relevance or ranking accuracy. We now briefly sketch out some other directions we might explore on the basis of our framework.

In these last examples, the funny images sent were intended as messages in and of themselves, because of their shared meaning between two friends. Images for Social Interaction

As a final sub-theme, images were seen to be used by participants in social games or social features online. Besides common examples like Facebook comments and wall posts, some scenarios showed social interactions taking place in online forums, such as this guessing game that involved locating interesting images of personalities and sharing them to encourage a response: Fieldnotes for Tom: The object of the game, played twice weekly, is to identify an obscure person based only on a photo. The person with the correct guess then uploads the next picture. While Tom was on the site, he clicked on other thumbnails of obscure people that looked interesting. There was one of Mae Busch, which looked interesting because you could barely recognize the person. In fact, he enlarged the photo so he could identify the person better.

Designing for Learning and Research

There were various ways in which we found that images were used to support learning and research. The main way was via familiar sites that were routinely visited, and often the goal was to see whether there was anything new to look at. Important here were not just new images, but the surrounding content, helping participants to keep abreast of any news, to be entertained, and to add to their already considerable knowledge around a topic. Such behavior suggests not necessarily the need for better keyword search, but the support of quick navigation to familiar sites, perhaps triggered by new content appearing on those sites. The analogy is almost like channel-surfing on TV, where interesting content can grab attention. It suggests that Web-based tools might be better at alerting users to changes in content (whether image-based or not) on sites that are frequently visited.

The social interaction here was indirect—the participant was entertained by other members’ interactions around images, with no direct communication between them. This shows how images can spark interest in unique ways beyond starting points for conversation. DISCUSSION

This study has shown that the term “image search” may hide the rich and interesting complexity of reasons why users engage with images on the Web, and how they go about these activities. The results of this study derive a number of interesting findings. First amongst these was the fact that image-specific search tools (e.g. Google Images) were not the main way that images were accessed and used (less than 10% of the time). While it is difficult to make too much of this on the basis of a small sample, it is interesting

109

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The desire to amass new knowledge on particular topics (such as new artists, or new work by a particular model) also suggests that more flexible tools for clipping and saving the results of Web browsing would be valuable. Tools that act as digital scrapbooks or repositories for information around a theme or topic could be effective ways of not just visiting sites, but collecting information from multiple sites, an issue which has been commented on before when looking at the predominance of the use of the Web for ongoing research activities [23]. A good example here would be the use of such “scrapbook” tools for putting together sources of ideas and inspiration, much like designers do when they keep notebooks.

Users also analyzed images in these scenarios from spatial and historical perspectives, thus we can speculate that the ability to position images in those dimensions can improve this experience. For example, geotagging images, allowing for panoramic views, rendering images in virtual 3D and being able to place images within a wider geographic context may be important. Likewise, re-connecting to the past might benefit from linking images to a timeline, for example, personal history. In both cases the ability to juxtapose multiple high fidelity views or perspectives is desirable, as participants often viewed all photos within a given set to get a better sense of the observed space.

Finally, learning and research sometimes did involve keyword search and the emergence of discovery through scanning large collection of images. Here, playing with different keywords, and analyzing images visually supported learning. Even though not a frequent activity, this suggests that tools that flexibly sort along different visual and semantic dimensions might be both fun and valuable to inspire and create new insights.

Images as understanding people or personalities both require related contextual information, such as timelines or links to more information. Activities involving image access to re-live past experiences would benefit from a feature that allows users to keep track of meaningful images for future reference (e.g. in Elle’s hotel scenario). Unlike traditional bookmarks, which facilitate information retrieval, here bookmarks are more akin to a digital keepsake box that allows the images to be re-experienced.

Designing for Image Access as Secondary Goals

Designing for Communication

When designing to support image access as a secondary goal, efficiency, accuracy and consistency are critical dimensions to consider. For example, one interesting finding was the way in which images could serve as an additional way of finding answers during keyword search. This suggests that, rather than having to always choose between a text search (with a ranked lists of text) or an image search (with a matrix of images), a mix of text and images in the returns might offer even more efficient ways of finding information or navigating to certain sites.

Images used in social interaction often occur over social platforms such as blogs, online forums, instant messaging, and social networking Websites. While some features exist (e.g. commenting, tagging) these could be extended, such as allowing users to be more creative with images in the course of interaction. For example, editing tools to modify images, create cartoons and storyboards, or simply to personalize an image for someone else are suggested. Further, general image search tools might more seamlessly connect with communication tools and social networking sites, recognizing that these are common follow-on activities.

Additionally, it is important to consider whether the image is part of the end goal. For instance, in the case of Gracia’s recipe search, the photo of the dish was a simply a step along the way to obtaining the real goal –the recipe. In other cases, images might be used more systematically as navigational indexes, such as the case of William locating a product by visually browsing thumbnails of DVD album covers on Play.com.

CONCLUSION & FUTURE WORK

Research going forward will include looking at the relative frequency of these new forms of image use, the context in which they are preformed, and the implications for search. However, this study has extended our understanding of the contextual and behavioral details of users’ interactions with Web-based images as they occur in the course of everyday life for a small but diverse sample of users. In doing so, we have shown how both the “how” and the “why” of these activities rarely conform to the model of user interaction which appears to inform most image search tools. Users do sometimes want to issue queries in terms of keywords and receive answers in terms of images, but of interest is both why this would be so, and the myriad of other ways in which images play into people’s experience of the Web. We have found that images are often intertwined with the ongoing process of learning about the world, that they provide a compelling source of entertainment and inspiration, that they help create connections to other people and remote places, that they help us reminisce about our personal past, and that they provide a way of navigating both the Web and the physical world.

Both cases suggest that rather than treating images as end goals, such as is implied in search tools, they might also be treated as steps toward better, more flexible forms of navigation. Not only could images be more integrated with text in the presentation of search results, but clicking on images might offer up new navigational pathways that help users to explore sites or products that they are connected to Designing for Recreating or Connecting to Remote Experiences

In this third category of image use, participants often remarked on how they valued the detail in photos they observed during tasks to recreate or connect to remote experiences. Thus, higher fidelity photos can be used to help users in the task of analyzing visual detail and contextual relevance; aiding recall, recognition and stimulating emotions.

110

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

12.Kirk, D., Sellen, A., Rother, C., & Wood, K. Understanding photowork. In Proc. CHI 2006, ACM P. (2006), 761-770.

Our hope is that a small but in-depth study of this sort can serve to complement and deepen our understanding of larger scale, quantitative clickstream and query data analyses in bringing to bear an alternative set of findings more firmly from the user’s perspective. Integrating these approaches, we believe, would be a valuable next step. Meanwhile, by beginning to look at the rich array of human values expressed within these findings, we hope to have shown that we can open up new opportunities for design to encourage reflection, connection, socialization and play with images.

13.La Cascia, M., Sethi, S., & Sclaroff, S. (1998). Combining textual and visual cues for content-based image retrieval on the world wide Web. Tech. rep., Boston, MA, USA. 14.Lerman, K., & Jones, L. Social browsing on Flickr. (2006) URL http://arxiv.org/abs/cs.HC/0612047 15.Marchionini, G. Information Seeking in Electronic Environnments. (1995.) Cambridge, Cambridge UP. 16.Markkula, M. & Sormunen, E. (2000). End-User Searching challenges indexing practices in the digital newspaper photo archive,. Information retrieval, 1, 4, 259-285.

ACKNOWLEDGMENTS

We would like to thank Xiang Cao, Darren Smith and Richard Harper for their comments throughout this project, and the Socio-Digital Systems group at Microsoft Research Cambridge for funding this research. We also would like to thank George Buchanan and Xia Lin for comments on drafts.

17.Miller, A.D. & Edwards, W.K. Give and take: A study of consumer photo-sharing culture and practice. In Proc. CHI 2007, ACM P. (2007), 347-356. 18.Negoescu, R. A., & Perez, D. G. Analyzing Flickr groups. In Proc. CIVR 2008, ACM P. (2008), 417-426. 19.Porta, M. Browsing large collections of images through unconventional visualization techniques. In Proc. AVI 2006, ACM P. (2006), 440-444.

REFERENCES 1.Paul André, Edward Cutrell, Desney S. Tan, Greg Smith, Designing Novel Image Search Interfaces by Understanding Unique Characteristics and Usage. In Proc. INTERACT 2009, ACM P. (2009), 340-353.

20.Pu, H. T. An analysis of failed queries for Web image retrieval. Journal of Info. Science 34, 3 (2008), 275-289. 21.Rieman, J. The diary study: a workplace-oriented research tool to guide laboratory efforts. In Proc. INTERCHI 1993, ACM P. (1993), 321-326.

2.Armitage, L. H, & Enser, P. G. B. Analysis of user needs in image archives. Journal of Info. Science 23, 4 (1997), 287-299. 3.Brewer, J., Kaye, J., Williams, A., & Wyche, S. (2006). Sexual interactions: why we should talk about sex in HCI. In CHI '06: CHI '06 extended abstracts on Human factors in computing systems, (pp. 1695-1698). New York, NY, USA: ACM.

22.Rodden, K., & Wood, K. R. How do people manage their digital photographs? In Proc. CHI 2003, ACM P. (2003), 409416. 23.Sellen, A.J., Murphy, R. M., & Shaw, K. (2002). How knowledge workers use the Web. Proceedings of CHI 2002, Minneapolis, MN. New York: ACM Press, pp. 227-234.

4.Burke, M., Marlow, C., & Lento, T. (2009). Feed me: motivating newcomer contribution in social network sites. In CHI '09: Proceedings of the 27th international conference on Human factors in computing systems, (pp. 945-954). New York, NY, USA: ACM.

24.Spink, A. (2004). Web searching for sexual information: an exploratory study. Information Processing & Management, 40 (1), 113-123.

5.Cai, D., He, X., Li, Z., Ma, W. Y., & Wen, J. R. (2004). Hierarchical clustering of www image search results using visual, textual and link information. In Proc. MULTIMEDIA '04, (pp. 952-959). New York, NY, USA: ACM.

25.Stevens, M. M., Abowd, G. D., Truong, K. N., & Vollmer, F. (2003). Getting into the living memory box: Family archives & holistic design. Personal and Ubiquitous Computing, 7 (3), 210216.

6.Choi, Y., & Rasmussen, E. M. Searching for images: the analysis of users' queries for image retrieval in American history. Journal of the American Society for Info. Science and Technology 54, 6 (2003), 498-511.

26.Süsstrunk, S., & Winkler, S. (2004). Color image quality on the internet. In Proc SPIE, vol. 5304, (pp. 118-131). 27.Toyama, K., Logan, R., & Roseway, A. (2003). Geographic location tags on digital images. In MULTIMEDIA '03: Proceedings of the eleventh ACM international conference on Multimedia, (pp. 156-166). New York, NY, USA: ACM.

7.Cunningham, S. J., & Masoodian, M. Looking for a picture: an analysis of everyday image information searching. In Proc. JCDL 2006, ACM P. (2006), 198-9. 8.Datta, R., Li, J., & Wang, J. Z. Content-based image retrieval: approaches and trends of the new age. In Proc. MIR 2005, ACM P. (2005), 253-262.

28.Van House, N., Davis, M., Ames, M., Finn, M., & Viswanathan, V. (2005). The uses of personal networked digital imaging: an empirical study of cameraphone photos and sharing. Ext. Abstracts CHI 2005, ACM P. (2005), 1853-1856.

9. Dumais, March, Schraefeumais, S. T. 2009. An interdisciplinary perspective on information retrieval. In Proc. of SIGIR '09. ACM, New York, NY, 1-2.

29.Wang, C., Zhang, L., & Zhang, H. J. (2008). Learning to reduce the semantic gap in Web image retrieval and annotation. In Proc SIGIR '08, (pp. 355-362). New York, NY, USA: ACM.

10.Goodrum, A., & Spink, A. Image searching on the excite Web search engine. Info. Processing and Management 37, 2 (2001), 295-311.

30.Westman, S. & Oittinen, P. (2006). Image retrieval by endusers and intermediaries in a journalistic work context. Information Interaction in Context, IIiX, 102-110.

11.Graham, A., Molina, H. G., Paepcke, A., & Winograd, T. Time as essence for photo browsing through personal digital libraries. In Proc. JCDL 2002, ACM P. (2002), 326-335.

111

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

HCI & Sustainable Food Culture: A Design Framework for Engagement Jaz Hee-jeong Choi Queensland University of Technology Brisbane QLD 4059, Australia [email protected]

Eli Blevis Indiana University Bloomington, IN, USA [email protected]

ABSTRACT

sustenance [17]. Humans simply cannot exist without food intake. Essentially, sustainability in this sense is based on ‘securing’ food: having access to stable availability and use of quality food [18]. However, the current food production and consumption practices do not ensure food security for the future but rather seriously threaten it [17]. One of the identified causes for this situation is the widening divide between the urban and rural environments. While people living in urban environments – including urban, suburban, and peri-urban (all of which are generally referred to as urban/metropolitan/city in this paper) regions – increasingly see food as objects of consumption or the ‘final products’ to be consumed. This situation deteriorates with the unprecedented scale of urban growth in recent years, broadening the segregation between the rural and urban environments, thereby further limiting knowledge about and access to fresh produce. Global urban population has been increasing rapidly, which suggests that availability of food resources in urban areas is also increasingly becoming scarce. The UN Population Fund [37] predicts that the urban population will grow further to reach 60% of the entire global population by 2030. Therefore, as seen in recent global economic and ecological turmoils, change is necessary in how we live our daily lives, in order to create a positive outlook for global food security. Now is a crucial time for HCI researchers to create actionable knowledge through identifying, testing, and building on technical opportunities that can be augmented and realised to cultivate urban food cultures that are environmentally, socially, and health-wise sustainable.

The current food practices around the world raises concerns for food insecurity in the future. Urban / suburban / and peri-urban environments are particularly problematic in their segregation from rural areas where the natural food sources are grown and harvested. Soaring urban population growth only deteriorates the lack of understanding in and access to fresh produce for the people who live, work, and play in the city. This paper explores the role of HumanComputer Interaction (HCI) design in encouraging individual users to participate in creating sustainable food cultures in urban environments. The paper takes a disciplinary perspective of urban informatics and presents five core constituents of the HCI design framework to encourage sustainable food culture in the city via ubiquitous technologies: the perspective of transdisciplinarity; the domains of interest of people, place, and technology; and the perspective of design. Author Keywords

Food, Sustainability, HCI, Urban Informatics, Design, Transdisciplinarity ACM Classification Keywords

H5.0. Information interfaces and presentation (e.g., HCI): General. K.4.2 Social Issues. INTRODUCTION

Assuring a sustainable future has become one of the main concerns for HCI researchers in recent years [14]. While sustainability as a term warrants multi-fold interpretations, recent years have seen a particular emphasis on environmental sustainability as the ‘climate change’ became part of the global political and social consciousness. One imminent consequences of the global warming is related to food, the vital foundation of human

In considering ‘how’ to design such HCI experience, we argue that the answer is ‘engagement’: engagement across disciplines, engagement with and amongst users (and nonusers), and engagement for sustainability, or rather, ‘sustained usability.’ In this paper, we will discuss each of these three main aspects of engagement: • Engagement across disciplines: The perspective of transdisciplinarity – transcending disciplinarity and values orientation

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

• Engagement with and amongst users/non-users: The domain of urban informatics at the intersection of people, place, and technology [19] – participatory, context-aware, and interactive networks

112

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• Engagement for sustained usability: The perspective of design – design criticism and critical design

reductionism/holism, diversity/unity. This duality is transgressed by the open unity that encompasses both the universe and the human being [29].

More specifically, two perspectives and three domains of interest are introduced, which together constitute the main five sections of this paper: perspective of transdisciplinarity, domain of interest of people, domain of interest of places, domain of interest of technology, and perspective of design. While these five facets form a frame that is general and applicable to any inquiry about or practice within a values-oriented design space, we intend to use it specifically to serve as a theoretical underpinning for our exploration of HCI design for sustainable food culture. This frame is not intended to be complete. We hope to expand and refine this frame as a theoretical foundation and also apply it to our study of sustainable food culture (http://www.urbaninformatics.net/projects/food) as a matter of theoretically informed practice. Similarly, our practice will further inform this theoretical frame, making the development an iterative and incremental process. The goal of this paper is to initiate this process by defining the theoretical frame in the context of sustainable food cultures as a basis to apply to future study.

Transdisciplinarity differs from similar integrative research approaches such as multi- and interdisciplinarity. While multidisciplinary research refers to a study in which several partners representing various disciplinary perspectives share their respective methods and domains as outputs and inputs one-with-another, interdisciplinary research refers to a study in which several partners representing various disciplinary perspectives pool their methods and domain expertise. Both approaches ‘overflow’ the discipline and remain within the framework of disciplinary research; transdisciplinary research is ‘at once between the disciplines, across the different disciplines, and beyond all discipline’ [29]. Therefore, transdisciplinarity is an orthogonal concept to interdisciplinarity and multidisciplinarity, and helps transcend disciplinarity by focusing on goals that emerge from a focus on values, issues, as well as ontological and epistemological perspectives, appealing to disciplinary notions of method and domains of expertise as needed and in the service of these larger goals. Methodologically, transdisciplinarity allows the researcher to flexibly yet rigorously explore and integrate research methods from various disciplines to result in coherent knowledge rather than united knowledge across disciplines [33]. Emerging research fields such as urban informatics [19] reflect a need to create such holistic body of knowledge in order to understand and tackle challenging global issues such as the climate change, governance, and education. Given the diversity of role that food plays in everyday life and its impact that is also diverse in type and scale, designing HCI for urban food sustainability thus must seek ways to utilise ubiquitous technology’s flexibility in scale of application (for example, on the continuum of individual / collective, private / public, and local / global) to improve the health, social, and environmental bottom-lines of everyday human-food interaction at the intersection of people, place, and technology. Transdisciplinarity for food sustainability research is an essential perspective as it provides dialogic knowledge development that can tackle real life problems that are inherently multi-faceted.

TRANSDISCIPLINARITY: TRANSCENDING DISCIPLINARITY & VALUES ORIENTATION

As mentioned earlier, sustainable HCI has been growing as a unique—nevertheless broad—domain of study in recent years. However, despite the complexity of what constitutes the notion of sustainability, the majority of studies remain within specific disciplines – for example, environment [2, 11], health [10, 30, 31], and social [28]. As DiSalvo et al. [14] argue, there is an opportunity and necessity to add to these efforts through HCI design by recognising and embracing the innate interconnecting threads across various scholarly disciplines, domains of research and practices. Recently emerging movements such as Take a Bite Out of Climate Change (www.takeabite.cc) highlight the confluence of domains that are germane to sustainability, thereby reasserting the need for creating a broader, holistic understanding of current issues in food culture. From the scholarly perspective, engage the issue of food sustainability through HCI design therefore must extend beyond disciplinary compartmentalisation or didactic information dissemination.

THE DOMAIN OF INTEREST OF PEOPLE: PARTICIPATORY CULTURE

Since early 1970s when Jean Piaget coined the term, transdisciplinarity has evolved into a field or more pragmatically, a conceptual tool that help researchers create ‘innovative research strategies and methods tackling complex objects and contexts’ [25]. Transdisciplinarity is not a method, but a principle or approach that seeks to create new overarching knowledge through integrative research [25, 5]. Nicolescu maintains:

In many parts of the world, the means to engage in participatory culture are no longer limited to the technically versed or the civically inclined. Scholars such as Jenkins [21] and Hartley [20] have identified socio-technical trends towards participation in digital culture shaped by individual expressions of creativity. Participatory or DIY culture is fast becoming a core element of contemporary society (at least in developed nations) in which citizenship is construed through ‘practice’ of self identification by individuals rather than a ‘contract between state and subject’ [20]. As evident in many grassroots initiatives such as the Local Food

Transdisciplinarity transgresses the duality of opposing binary pairs: subject/object, subjectivity/objectivity, matter/consciousness, nature/divine, simplicity/complexity,

113

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Movement—a collaborative effort to build more locally based, self-reliant food economies [15] and Slow Food International—a non-profit group focusing on preservation of the cultural, culinary, and artistic local traditions [22], significant transformations arise from large-scale consensual participation of individuals identifying with the value of a sustainable lifestyle both conceptually and pragmatically. For such transformations to have sustainable impact, they need to be incremental and self-initiated as opposed to abrupt and externally enforced [8, 7].

too-familiar places—via mass media such as print and TV, or micro-communications such as sensor networks—does not necessarily trigger sufficient motivation for behavioural change towards an ongoing health- and earth-friendly lifestyle. Instead, it is necessary to develop a better understanding about how to go beyond just informing and into motivating and encouraging positive changes in action and perception. As such, innovations in understanding of what makes technology persuasive and motivational [13, 23, 16, 1] for each user at any one given moment are likely to bring about sustained behavioural changes amongst individuals, and further increase the ideological impetus to enhance societal values. However, maintaining a food practice that upholds health, environmental, and social ideals is a time- and energy-demanding task. It currently requires an advanced understanding of a variety of sometimes conflicting information sources and a determination to commit the necessary time and effort to always know where to buy health- and earth-friendly produce, according to seasonality as well as to one’s differing tastes and lifestyle needs. While existing media channels such as television shows, magazines, and blogs on the topic of eating, cooking, and growing food may offer educational information—including nutritional, cultural, aesthetic—that could assist in this regard [24, 12], we can find additional and perhaps more effective benefit in ubiquitous technology’s capacity to streamline the vigorous management process particularly through real-time context awareness.

Therefore, usable and efficient human-computer interaction (HCI) design to bring about changes needs to be innately supportive and persuasive in guiding user’s actions rather than exerting control and leverage. Such design must be scalable in context awareness (from micro to macro – from the individual to community context, for example) and adaptive to provide the optimal solution according to the given context. Many examples of how participatory culture is enabled by recent technological innovation rely on socalled Web 2.0 applications such as Wikipedia, YouTube, Flickr, and social networking sites, which are arguably more customisable and open (though in many cases they impose regulatory boundaries but nevertheless allow some ways for interaction and modification though external means). In this regard, Burgess [9] has introduced a useful notion of vernacular creativity, which refers to ‘a productive articulation of consumer practices and knowledges (of, say, television genre codes) with older popular traditions and communicative practices (storytelling, family photography, scrapbooking, collecting)’ reducing the cultural distance between production and everyday experiences. Can the features that give rise to participatory culture on the web and significant changes in economy, politics, and socio-cultural domains— as seen in crowd sourcing phenomenon as notably explained by Shirky [34]—be harnessed to encourage people to further their sustainability awareness and actions in the context of food practices? The question for HCI designers is to find efficient and engaging ways to utilise technical resources to allow for collaborative information sharing, knowledge production, and user-led innovation. Some current popular services such as Urbanspoon (www.urbanspoon.com), Beanhunter (www.beanhunter. com), and Foursquare (foursquare.com) function as repositories of crowd-sourced recommendations for places to eat, coffee shops, and general places of interest by urban location. The users transform the virtual ‘space’ into ‘places’ by embedding values and meanings through social interaction [36].

The contexts of food culture vary widely from one place to another. Just as each continent and country may have its own values and practices associated with food, each place however small it may be, has its own atmosphere and rules of engagement for each entity in order for it to be counted as a valid constituent of the place. A brief observation around eaters in streets of any metropolis such as London, Tokyo, and New York would effortlessly show how, for example, an Italian pizzeria differs from a Korean barbeque restaurant that may be located nearby. As Steel asserts [35], ‘[f]ood shapes cities, and through them, it moulds us’ and the city is thus ‘partly shaped by food.’ Therefore, the opportunities that network technology imparts for cultivating sustainable urban food culture is essentially twofold: first, assistance in understanding and navigating through the food-layer of the given place; and second, encouraging sustainable food practices, which in turn shapes the city. Building a food culture toward a more sustainable future is thus an iterative and evolutionary process involving interactions amongst people, place, and technology.

THE DOMAIN OF INTEREST OF PLACES: CONTEXT AWARENESS & CONTEXT SPECIALISATION

An example of such system is Bonanni et al.’s [6] SourceMap (http://www.sourcemap.org). Both as an issue of food safety and minimizing food miles as a matter of sustainability, SourceMap is an example of a system that promotes geospatial context awareness in the domain of food. The system is targeted at providing small business

Prior to the rise of network technologies, likely places for placing advertisements for healthy lifestyle were television, billboards, and on the back of the cereal box. However, as evident in food culture today, simply providing people with environmental data and educational information in now all-

114

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

owners with the means to calculate the distances that the supplies they use need to travel and the means to make this information available to their clientele as a matter of demonstrating conscientious, socially responsible business practice. The system produces ‘receipts’ that denote the carbon footprint of a small business’s products, showing total carbon footprint and carbon footprints of various stages of transport of various constituent product elements.

There are a number of possible interactive systems that may play a positive role here. For example: (i) Food production and source tracking: As an issue of both sustainable supply and food safety, interactive systems can allow various groups of people to adapt to changing suitability of particular regions for growing particular crops and other forms of food production. (ii) Dashboard earth: As an issue of preparation and adaptation to changing climate, interactive systems can help ensure planning and preparation for how people in urban environments can respond to changes in food and water supplies and even to threats to the inhabitability of— particularly—coastal urban and other environments.

THE DOMAIN OF INTEREST OF TECHNOLOGY: NETWORKS & INTERACTIVITY

With respect to the domain of interest of technology, there are many forces in place, which create likelihood for changes with implications for food cultures that are yet to be understood. For each of these changes, there are opportunities for technologies—particularly in terms of networks and interactivity—to play a role with both positive and negative implications.

(iii) Orderly absorption: Urban environments are in some sense dynamically changing places—interactive systems can offer infrastructural support that allows urban policy makers to provide for the orderly immigration to or emigration from urban environments in the face of the effects of climate change.

The Intergovernmental Panel on Climate Change (IPCC) most recent complete report in 2007 predicts that climate change will have massive implications for global food production and conditions of production, as well as for water, coastal inhabitations, health, and ecosystems [32 as reported in 4]. For food production, ‘complex, localised negative impacts on small holders, subsistence farmers, and fishers’ is virtually certain, and the latitudes at which cereal production is viable will change depending on the amount of warming. Another salient effect is the likelihood of ‘increased damage from floods and storms’ and the possibility of coastal flooding and wetlands loss, depending on the amount of warming. These two factors and others in the IPCC report have implications for the many urban areas that are located in coastal environments, not just in terms of food, but also potentially in terms of habitability.

(iv) Living with fewer resources: Interactive systems can assist people in urban and other environments with understanding and practicing how to live with fewer resources, either as a matter of sustainable practices or as a matter of adapting to climate change or both. (v) Saving life: Social mechanisms—especially those that rely on interactive technologies—can play a role in fostering relationships between people at various levels of organization in order to ensure that as many people as possible have access to safe environments, with food and drinking water and that people are actively engaged in helping others. THE PERSPECTIVE OF DESIGN: DESIGN CRITICISM & CRITICAL DESIGN

Some respected scientists—such as James Lovelock—have made claims that the IPCC predictions are too conservative [27 as reported in 4]. Lovelock believes that we have already reached the point at which limiting anthropogenic production of green house gasses is no longer a viable mechanism for preventing climate change; and further, that positive feedback mechanisms are already in place which make dangerous global warming a certainty. Lovelock presents a catastrophic picture that the earth’s population— now approaching 7 billion—will be reduced to under 1 billion, before the end of the century.

This paper has outlined critical concepts and examples that are germane to conceptualizing HCI design for disciplinary and user engagement in sustainable food cultures in urban environments. The paper has also outlined the three core domains of interest for user-engagement: of people, place, and technology.

A changing climate and its associated implications per the two paragraphs above implies a need for the design of digital networking and interactive technologies that can help people at various levels—as individuals, small groups, governments, and global bodies—plan for the orderly adaptation to these effects. Such networks will need to make available data, visualizations, tracking, and predictive simulations about changing locations of food production and threats to particular urban environments.

Figure 1. Core domains of interest for user engagement

115

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

REFERENCES

Perceptively, technologies provide useful ways to help cultivate sustainable food culture from the individual to the global level. However, we must at the same time remember that network technologies are part of the broader contemporary influences shaping the ecology. Environmental impact is one of the main such influences. Recently Alexander Wissner-Gross at the Harvard University Center for the Environment has made a statement that has provoked much discussion amongst general public as well as in academic and commercial sectors: that ‘performing two Google searches from a desktop computer can generate about the same amount of carbon dioxide as boiling a kettle" or about 7g of CO2 per search’ [26]. This ‘statistic’ may in fact be controversial at best – nonetheless, the point that not even Google searches are without environmental impact holds.

1. Aleahmad, T., Balakrishnan, A.D., Wong, J., Fussell, S.R. & Kiesler, S., Year. Fishing for Sustainability: The Effects of Indirect and Direct Persuasioned. CHI conference, Florence, Italy. 2. Bang, M., Torstensson, C. & Katzeff, C., 2006. The PowerHhouse: A Persuasive Computer Game Designed to Raise Awareness of Domestic Energy Consumption. Persuasive Technology. 123-132. 3. Blevis, E., 2007. Sustainable Interaction Design: Invention & Disposal, Renewal & Reuse. Proceedings of the SIGCHI conference on Human factors in computing systems. San Jose, California, USA: ACM. 4. Blevis, E. & Blevis, S., 2010. Hope for the Best and Prepare for the Worst: Interaction Design and the Tipping Point. Interactions, 17.

For the ‘sustainability of’ a HCI design for sustainable food culture, the design needs to ensure participation of users in perpetual recreation of the technology and must also incorporate anticipation for the technology’s socio-cultural, health, and environmental impact. To achieve this, a continual examination of the technology’s effective is an imperative part of the overall development of the technology. Design theory is a rich and varied art with an extensive literature too broad to list here. Within design theory, one idea in particular that has elsewhere been applied to the notion of how interaction design is implicated in sustainability is the notion of design criticism and critical design as complimentary activities that are necessary companions in design processes [3]:

5. Blevis, E. & Stolterman, E., Year. The Confluence of Interaction Design & Design: from Disciplinary to Transdisciplinary Perspectivesed. Design Research Society Biennial Conference, Sheffield, UK. 6. Bonanni, L., Hockenberry, M., Zwarg, D., Csikszentmihalyi, C. & Ishii, H., 2010. Small Business Applications of Sourcemap: A Web Tool for Sustainable Design and Supply Chain Transparency. Proceedings of the 28th international conference on Human factors in computing systems. Atlanta, Georgia, USA: ACM. 7. Brinol, P., Rucker, D.D., Tormala, Z.L. & Petty, R.E., 2004. Individual Differences in Resistance to Persuasion: The Role of Beliefs and Meta-Beliefs. In E.S. Knowles & J.A. Linn (eds.) Resistance and Persuasion. Mahwah, N.J.: Lawrence Erlbaum Associates, 83-105.

We may distinguish here between design criticism – what is needed to understand and interpret present ways of being, and critical design – what is needed to ensure that our actions lead to sustainable future ways of being.

8. Brock, T.C., 1967. Communication discrepancy and intent to persuade as determinants of counterargument production. Journal of Experimental Social Psychology, 3.

Time being what it is, critical design takes place in the absence of complete understandings of present ways of being. Thus, design criticism and critical design are mutually dependent, ongoing, and co-evolving acts. Design without design criticism is unlikely to create critical design and criticism without critical design is unlikely to create design criticism. Design criticism is strategic. Critical design is tactical.

9. Burgess, J., 2006. Hearing Ordinary Voices: Cultural Studies, Vernacular Creativity and Digital Storytelling. Continuum, 20, 201 - 214. 10. Consolvo, S., Everitt, K., Smith, I. & Landay, J.A., 2006. Design requirements for technologies that encourage physical activity. Proceedings of the SIGCHI conference on Human Factors in computing systems. Montreal, Qubec, Canada: ACM.

In applying our framework to our future work on our project (http://www.urbaninformatics.net/projects/food), we expect to take this bi-valent approach—that is we expect to equally (i) look for and apply design criticism to various forms of the design of ubiquitous technologies for sustainable food culture that we may uncover by observation or from secondary sources, and (ii) suggest the design of new forms of interactivity that inspire sustainable food practices—while still preserving the joy of urban experiences.

11. Cornelissen, G., Pandelaere, M. & Warlop, L., 2006. Cueing Common Ecological Behaviors to Increase Environmental Attitudes. Persuasive Technology. 39-44. 12. De Solier, I., 2005. TV Dinners: Culinary Television, Education and Distinction. Continuum: Journal of Media & Cultural Studies, 19, 465 - 481. 13. De Young, R., 2000. Expanding and Evaluating Motives for Environmentally Responsible Behavior. Journal of Social Issues, 56, 509-526.

116

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

14. Disalvo, C., Sengers, P. & Brynjarsdottir, H., 2010. Mapping the landscape of sustainable HCI. Proceedings of the 28th international conference on Human factors in computing systems. Atlanta, Georgia, USA: ACM.

26. Leake, J. & Woods, R., 2009. Revealed: the environmental impact of Google searches. Times Online. 27. Lovelock, J., 2009. The vanishing face of gaia : a final warning New York: Basic Books.

15. Feenstra, G., 2002. Creating space for sustainable food systems: Lessons from the field. Agriculture and Human Values, 19, 99-106.

28. Marc, A.S., 2000. Some social implications of ubiquitous wireless networks. SIGMOBILE Mob. Comput. Commun. Rev., 4, 25-36.

16. Fogg, B.J., 2003. Persuasive Technology: Using Computers to Change What We Think and Do Amsterdam: Morgan Kaufmann Publishers.

29. Nicolescu, B., 2002. Manifesto of Transdisciplinarity New York: State University of New York Press.

17. Food and Agriculture Organizataion of the United Nations, 2008. Climate Change, Water and Food Security [online]. Food and Agriculture Organization of the United Nations. Available from: ftp://ftp.fao.org/docrep/fao/010/i0142e/i0142e07.pdf.

30. Papadaki, A. & Scott, J.A., 2006. Process Evaluation of an Innovative Healthy Eating Website Promoting the Mediterranean Diet. Health Education Research, 21, 206-18. 31. Parmar, V., Keyson, D. & Debont, C., 2008. Persuasive Technology for Shaping Social Beliefs of Rural Women in India: An Approach Based on the Theory of Planned Behaviour. Persuasive Technology. 104-115.

18. Food and Agriculture Organizataion of the United Nations, 2010. FAQ: What is Meant By Food Security [online]. Food and Agriculture Organizataion of the United Nations. Available from: http://www.fao.org/spfs/about-spfs/frequently-askedquestions-spfs/en/.

32. Parry, M.L., Intergovernmental Panel on Climate Change. & Intergovernmental Panel on Climate Change. Working Group Ii., 2007. Climate change 2007 : impacts, adaptation and vulnerability / Contribution of Working Group II to the Fourth Assessment Report of the IPCC Intergovernmental Panel on Climate Change [online]. IPCC Secretariat. Available from: http://www.ipcc.ch/publications_and_data/publications_ ipcc_fourth_assessment_report_wg2_report_impacts_ad aptation_and_vulnerability.htm

19. Foth, M., 2009. Handbook of Research on Urban Informatics: The Practice and Promise of the Real-time City Hershey, PA: Information Science Reference. 20. Hartley, J., 1999. Uses of television London ; New York: Routledge. 21. Jenkins, H., 2006. Fans, bloggers, and gamers : exploring participatory culture New York: New York University Press.

33. Ramadier, T., 2004. Transdisciplinarity and its challenges: the case of urban studies. Futures, 36, 423439.

22. Jones, P., Shears, P., Hillier, D., Comfort, D. & Lowell, J., 2003. Return to traditional values? A case study of Slow Food. British Food Journal, 105, 297-304.

34. Shirky, C., 2008. Here comes everybody : the power of organizing without organizations New York: Penguin Press.

23. Kaplan, S., 2000. Human Nature and Environmentally Responsible Behavior. Journal of Social Issues, 56, 491508.

35. Steel, C., 2008. Hungry City: How Food Shapes Our Lives London: Vintage.

24. Lappalainen, R., Kearney, J. & Gibney, M., 1998. A pan EU survey of consumer attitudes to food, nutrition and health: an overview. Food Quality and Preference, 9, 467-478.

36. Tuan, Y.-F., 2007. Space and Place: The Perspective of Experience Minneapolis: University of Minneapolis Press.

25. Lawrence, R.J. & Després, C., 2004. Futures of Transdisciplinarity. Futures, 36, 397-405.

37. Unfpa, 2007. State of World Population 2007: Unleashing the Potential of Urban Growth. New York: U.N.P. Fund.

117

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Distributed Usability Evaluation: Enabling Large-scale Usability Evaluation with User-controlled Instrumentation Lars Christensen & Erik Frøkjær Department of Computing, University of Copenhagen Njalsgade 128, building 24, 5th floor, DK-2300 Copenhagen S, Denmark {larsc,erikf}@diku.dk ABSTRACT

evaluations, and the communication and prioritization necessary for transforming redesign proposals into reality are complicated [20,23].

We present DUE (Distributed Usability Evaluation), a technique for collecting and evaluating usability data. The DUE infrastructure involves a client-server network. A client-based tool resides on the workstation of each user, providing a screen video recording, microphone input of voice commentary, and a window for a severity rating. The idea is for the user to work naturalistically, clicking a button when a usability problem or point of uncertainty is encountered, to describe it verbally along with illustrating it on screen, and to rate its severity. These incidents are accumulated on a server, providing access to an evaluator (usability expert) and to product developers or managers who want to review the incidents and analyse them. DUE supports evaluation in the development stages from running prototypes and onwards. A case study of the use of DUE in a corporate environment is presented. The study indicates that the DUE technique is effective in terms of low bias, high efficiency, and clear communication of usability issues among users, evaluators and developers. Further, DUE is supporting long-term evaluations making possible empirical studies of learnability.

We have tried to ease some of these difficulties by designing a technique called Distributed Usability Evaluation (DUE) with the following objectives: • Support the detection of usability problems covering aspects of effectiveness, efficiency and satisfaction. • Minimize the evaluation workload for users, evaluators and developers. • Enable evaluation of short- and long-term usage. • Strengthen the quality and validity of identified usability problems, for example by enabling realistic user-context and large-scale evaluation with many users. • Strengthen the communication of usability problems between users, evaluators and developers. To realize the objectives it was clear that an instrumentation tool was needed. We wanted automated screen video capture of user interaction and recording of voice commentary, while the user was using the evaluated application. Like in beta testing, this enabled us to have the user in charge of problem detection, without quite losing the information-rich context known from think-aloud evaluations.

Author Keywords

Usability evaluation, instrumentation, automation, remote, distributed, screen video, voice commentary, case study, software industry, evaluator effect, think-aloud, beta test, learnability.

To investigate the DUE technique we carried out an extensive case study. Here DUE was used to evaluate the usability of production software under active development in a business environment with 16 users during one month.

ACM Classification Keywords

H.5.2 Information Interfaces and Presentation (e.g., HCI): User Interfaces–Evaluation/Methodology; D.2.2 Software Engineering: Design Tools and Techniques–User Interfaces.

The next section summarizes the key challenges that served as background for our design. Then we present DUE with its Instrumentation Framework and how it can be used by users, evaluators and developers. Next follows a section describing the case study and the strategy for data analysis. Finally, we summarize and discuss the main results.

INTRODUCTION

Usability evaluations in industry are limited by a number of challenges, for instance: evaluations are very labor demanding, it can be difficult to establish clear goals for

BACKGROUND

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

This section will summarize and briefly discuss some of the key challenges in usability evaluation, which we aim to address by DUE.

118

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Measuring Usability

Emerging services, like for instance www.usertesting.com [26], offer internet-based remote think-aloud testing of websites. These services make it fast and easy to set up think-aloud tests and find participants, but many of the limitations and labour demanding tasks related to thinkaloud testing still remains, e.g. to extract, analyse and consolidate information from the videos.

A definition of usability, as the one given by ISO [15], gives us the initial step towards analyzing usability in applications. The next step, identifying and implementing reliable measures of usability, remains a challenge. Hornbæk [10] points out that usability cannot be measured directly and as a result of this, we find aspects of usability to measure and draw conclusions upon. Frøkjær et al. [4] make it clear that measures used to identify usability problems often do not take into account all three aspects of usability, i.e. effectiveness, efficiency and satisfaction; and a strong correlation between these aspects are often wrongly assumed. Identifying reliable measures remain an open-ended and commonly discussed challenge. Really useful measures cannot be identified at a general level, but have to be based on an understanding of the use situations within the particular application domain [4]. Users often have this understanding.

Thus, in spite of the assistance from various tools, usability evaluation in general is still a tedious task for evaluators, and often also for the developers. It involves extensive work to identify measures, set up evaluations with or without instrumentation, collect large amounts of data, analyse data, detect problems among the often vague measures, and communicate the problems [20,23]. The Evaluator Effect and Insufficient Analysis

The evaluator effect denotes that usability evaluators identify substantially different sets of usability problems when applying the same evaluation technique on the same application [17]. Facing large amounts of usability data, some of them vaguely documented, can make evaluator assessments complicated and unclear, thus increasing the evaluator effect.

Instrumentation and Automation

Instrumentation and automation within software evaluation have received some attention as a means to collect measures and supplement software evaluation, but have according to Ivory & Hearst [16] been underexplored.

Poor or insufficient analysis of usability data in practice, as observed by Nørgaard & Hornbæk [22], is partly a result of the complexity of the evaluator’s work when handling large amounts of usability data.

Early work by Hartson et al. [2,8,9] indicated that many challenges in usability evaluation could be met by using semi-instrumented remote evaluation and critical incident identification by users. However, Hartson et al. did not demonstrate the effectiveness of their approach in field studies.

These difficulties might be eased by tool support and stronger involvement of the users.

Recently instrumentation has received renewed focus as a means of collecting measures. Kim et al. [19] shows us that instrumentation can be a powerful tool to help detection of usability problems, especially when users are more involved than typically seen. User initiated events are seen as useful measures. Unfortunately the type of instrumentation used by Kim et al. requires program intrusion of the evaluated application, which limits practical usage in many situations.

Communication

The large amount of complicated data raises a huge communication challenge between people with quite different backgrounds: users with domain knowledge, developers with their technical insight, and evaluators with usability insight. Actually, this challenge is so large that it is often avoided in practice by more or less leaving out some of the stakeholders. However, studies indicate that involving developers, managers as well as users in the evaluation process can improve results [5,6,11,12].

Evaluation Tools and Workload

One possibility of strengthening the communication among stakeholders is to support the identification and recording of critical usability data in order to better understand and establish relationships among usability data [14].

Bateman et al. [1] recognize that setting up application intrusive instrumentation is a time-consuming task requiring technical insight by the evaluators. They argue that benefits can be achieved by being less intrusive; and propose a more intuitive way of adding instrumentation to an application.

Validity

The large workload on primarily the evaluators is probably the main reason for carrying out usability evaluations with few users over short periods of time. This makes it difficult to ensure external validity. The few users can make it doubtful that results are representative for all of the user population [3,25]. Further, a short evaluation period will only give snapshot insight in usability problems leaving out the users’ learning curve [7].

Other tools for usability evaluation, which offer some degree of instrumentation and are fully non-intrusive to the application, such as TechSmith’s Morae software, also support the task of usability evaluation. Here the evaluator is still left with the responsibility for problem detection and, as Howarth et al. [14] point out, usability evaluation is a tedious task even with this kind of software.

119

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

To strengthen the ecological validity, the circumstances of the usability evaluations should be as realistic as possible, while keeping attention to the risk of introducing disturbing experimental factors [24].

which have to be analysed thoroughly to identify usability issues, the focus in DUE is shifted to the user, who is given the responsibility to detect the usability issues. Thereby the primary instrumentation lays in the recorded video of the user’s screen, the recording of the user’s explanation, sound from surroundings, plus the collection of timestamps, and severity ratings of the user reported issues.

DISTRIBUTED USABILITY EVALUATION (DUE) The Instrumentation Framework

We have built an Instrumentation Framework, consisting of an Instrumentation Client and an Instrumentation Server, to realize the instrumentation and process automation necessary for meeting the DUE design objectives.

The user is instructed to have focus on elements which can have correlation to effectiveness, efficiency and satisfaction, but most importantly to use common sense and domain knowledge to report when something seems troublesome in actual use.

When doing a DUE evaluation the Instrumentation Client is installed on the computer of each participating user. The only requirements are that the computer must have a certain cpu power, a microphone, and a reasonable fast Internet connection. The Instrumentation Client manages user evaluation sessions. When an evaluation session ends, the Instrumentation Client transfers all collected data to the Instrumentation Server via the Internet.

In practice, when experiencing something as troublesome, the user can press a button to report and assess the severity of the issue (red, yellow or green) in the small Instrumentation Client visible on-screen, Figure 2 and 3.

All collected data are processed and stored on the Instrumentation Server for future use. The Instrumentation Server offers a web interface for evaluators and developers where they can log in and fulfil their role in the DUE workflow. See Figure 1 for the workflow of the four roles involved in DUE. These roles will be described in the following sections.

Figure 2. The small Instrumentation Client to the right of a web browser. Transparency enables placement in front of the application, if wanted.

Figure 1. Workflow in DUE. (1) The user starts with reporting issues, (2) the evaluator assesses these, (3) the development manager prioritizes, and (4) the developers respond. Finally the user receives feedback (1).

Figure 3. Close-up of the Instrumentation Client. The user’s severity rating of a usability issue is recorded by pressing one of the three colored buttons to the right.

After pressing the button, the user has to think out loud and explain the experienced issue vocally to the microphone, and maybe demonstrate further on screen by using the application if the issue was not evident from what was recorded before the button press.

The User Role

The instrumentation in DUE is centered on video recorded from the user’s screen and voice recorded from a microphone placed at the screen, while the user is using the application. The recording is continuously running while the Instrumentation Client is active, making it unnecessary to reproduce issues after discovery.

A timestamp at the user’s button activation is collected in the Instrumentation Client and is later used to create a bookmark in the recorded video of the user’s screen activity and voice commentary.

Instead of focusing on pre-selected measures and collecting large amounts of instrumented data from the application,

120

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The Evaluator Role

on the user’s ability to report when they experience something as troublesome and primarily have to decide if the reported issue is a usability problem or not.

When data is received on the Instrumentation Server the evaluator can log into the Instrumentation Server web interface from anywhere through a web browser, and review new issues (see Figure 4).

The Development Manager and Developer Role

Development manager and developers run through a similar workflow as the evaluator, prioritizing and responding to the issues that are approved as new problems by the evaluator. Summary of Characteristics

the

Instrumentation

Framework

This type of user-controlled instrumentation has the advantage that developers do not have to build anything into their application, and companies need not to be concerned about how the instrumentation will affect their application. Figure 4. Two new issues ready for evaluation on the Instrumentation Server

The Instrumentation Client can be run automatically on the user’s computer at specified times, taking care of data collection, and letting the user work and use the application without disturbances. The user presses a single button when experiencing something as troublesome, shortly explains and demonstrates the experienced problem, and carries on work.

The Instrumentation Server’s web interface ensures that the evaluator can run through all new issues in an efficient workflow. For each issue the evaluator can jump to where the user experienced an issue—20 seconds before the user pressed the button—review what happened in the video of the user’s screen, hear the user’s explanation, and see the user’s severity rating (see Figure 5).

The Instrumentation Server automatically prepares data for evaluators and developers, making it possible for them to jump directly to the issues in the video, and concentrate on reviewing and assessing these. Further, the issues are conveniently stored for communication and future use in the development cycle. CASE STUDY

As this was the first practical use of DUE, an exploratory quasi-experimental case study was chosen as the research strategy [27]. Propositions

Based on the design objectives we identified a series of propositions to be investigated in the case study: A1: DUE supports catching usability issues (effectiveness, efficiency and satisfaction) and other types of problems. A2: DUE reduces the workload needed for usability evaluation of software for users, developers and evaluators.

Figure 5. Issue evaluation on the Instrumentation Server. User screen-video running to the left, evaluations written to the right. Full-screen viewing of video is possible.

A3: DUE enables both short-term and long-term evaluations of software, with novice users and expert users.

The evaluator then decides whether the issue is a usability problem or not. If it is judged to be a problem, it is described and rated for its severity, or classified as a sameas problem, that is a problem found before, which then is linked to the prior description of the same problem.

A4: DUE strengthens the validity of identified problems:

The evaluators do not have to use time on analysing large amounts of instrumented data and reviewing many hours of video to identify usability problems. Instead evaluators rely

121

-

By the natural context of usage.

-

By minimizing disturbing experimental factors.

-

By enabling many test users.

-

By the possibility of long-term evaluation.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

A5: DUE provides an information-rich communication of the usability problems among users, evaluators, and developers.

Because of confidentiality agreements with the developing company we cannot show detailed pictures of F-Man, describe it in detail, or mention company names.

A6: DUE contributes to minimize the evaluator effect, and support a more thorough problem analysis.

F-Man has been on the market for a couple of years. It is under constant development and has never before been evaluated for usability, but is generally by customers considered to be user-friendly.

Strategy for Data Analysis

Following the general case study recommendations by Yin [27], we centered the case study on research questions designed to generate evidence that could clarify the propositions.

The users of F-Man were 16 users at a transportation company, who had been using F-Man for up to 8 months before the evaluation took place. The case study involved only one physical test site with the 16 users distributed in three offices.

Building a chain of evidence (Figure 6) ensured traceability all the way from evidence to final conclusions.

The role as evaluator was carried out by an external, highly experienced usability expert from industry. This evaluator has a Ph.D. degree in computer science and human computer interaction. The development manager role was carried out by the IT manager at the transportation company, and the role as developer was carried out by two developers at the developing software company.

Evidence  fact  proposition  conclusion

Figure 6. Chain of evidence, making it traceable how conclusions are supported by evidence.

The case study ran over one month, in total 18 work days as it was a period with many holidays.

All qualitative and quantitative evidence from the available sources were collected, see Table 1.

Installation and test of the Instrumentation Client on each participating user’s computer was done by a researcher, as the Instrumentation Client was a prototype without automatic configuration implemented yet. After installation the user was instructed in approximately 5 minutes.

Evidence was triangulated to support identified facts in the way Table 2 shows. In total 58 facts were identified. Each fact was assessed as supporting or opposing each of the propositions, see Table 3.

Due to very busy periods and customer service being a main concern for the transportation company, management preferred that users started the Instrumentation Client manually instead of this being done automatically. Because of this users were instructed to start the Instrumentation Client every day at different hours, or when they experienced an issue which could be reproduced. The Instrumentation Client automatically shut down after one hour. To help users in remembering this, a small gift was promised to the user being best at remembering the daily starts.

Alternative explanations that could challenge the propositions (rival explanations [27]) were investigated in a similar way. On this basis our conclusions are drawn. These are described in the Findings and Discussion section. The Case Study: Production Software under Active Development

To get a thorough investigation of DUE we teamed up with a software company developing and selling a fleet management and navigation product, here called F-Man. Source

Evidence

Usability issues

37 issues reported by users and evaluated by the evaluator. 30 issues prioritized by the development manager and responded to by developers. 37 issues reviewed for usability relevance by researcher.

Logged usage data

83 DUE sessions run by users, 57 hours total.

Observation

77 observations registered. 80 hours of user observation, 1 hour of evaluator and 4 hours of developers.

Timing of usage

13 user reports timed. 1 evaluator session timed. 1 developer session timed.

Interview

3 users interviewed in a total of 1:07 hours with 72 topics discussed. Evaluator interviewed for 32 minutes with 21 topics discussed. Development manager interviewed for 21 minutes with 27 topics discussed. 2 developers interviewed for 1:14 hours with 59 topics discussed.

Survey

8 users have answered the survey for users who have reported issues. 5 users have answered the survey for users who have not reported issues. 3 users chose not to answer any survey.

Documentation

1 e-mail from development manager to users. Table 1. Collected evidence in the case study.

122

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Fact 43 (A2, A5, A6)

Issues are easy to understand for evaluator, development manager and developers.

Evidence

Source

ID

All users who have reported issues state that the evaluator has understood these correctly.

Survey

S16

Evaluator understands issue at first review of video in 11 out of 13 issues.

Observation

O35

Development manager states that it is easy to understand issues.

Interview

I100

Developers state that it is easy to understand issues.

Interview

Proposition A5

DUE gives an information-rich communication of the usability problems among users, evaluators, and developers

Support

Fact

Description

X

30

Structure and documentation of issues is beneficial.

X

37

Real problem can be hidden for the user, but developers easily recognize the problem.

X

43

Issues are easy to understand for evaluator, development manager and developers.

X

9

DUE is supplemented or overridden if serious problem prevents work at all.

X

15

Some users find reporting to the microphone unpleasant.

-

-

etc.

Oppose

I159

Further evidence: O40, O50, O73, I28, I77, I79, I107, I115. Table 2. Example illustrating how Fact 43 was triangulated by the collected evidence.

To protect user privacy only the screen where F-Man ran was recorded. Most users had two screens. Furthermore users were able to stop the Instrumentation Client at any point and start a new session later.

-

Table 3. Example of proposition with supporting and opposing facts.

Login information to the Instrumentation Server was emailed to the evaluator who also received a short instruction by phone. The development manager and developers also received login information by e-mail and were given a 15 minutes demonstration. Their reviews of issues were done from work or at home via a web browser.

were primarily of technical nature. A single problem could be attributed to lack of knowledge among users. The evaluator, the development manager and developers agreed that these issues were valid problems. 13 problems were considered as “same-as” by the evaluator. Thus in total 19 unique problems were caught. Of the technical problems one was a “same-as” problem, thus 3 unique technical problems were caught.

Automatic e-mail notification had not yet been implemented at the Instrumentation Server, and a researcher called the evaluator, development manager and developers when a suitable amount of new issues reported by users were ready for assessment. This happened four times during the case study.

Through the researcher’s observations and interviews 4 problems were identified, which were not reported by the users. Three of these are considered relevant usability issues. One was cosmetic and only observed once during 80 hours of observation. A second was severe, but very close to other reports (slow search facilities). The third problem was unclearly described and occurred for one user during very busy periods. The fourth was of internal technical nature, and did not show up in the F-Man user interface.

A researcher was present at the transportation company observing half of the days. This set-up enabled analysis of potential influence by the researcher. One session with the evaluator was carried out at an external usability laboratory under video-recorded observation, and one session with the developers was carried out at the company under video-recorded observation. At the end of the case study video-recorded interviews and online user surveys were carried out, see Table 1. The case study ran without any technical problems.

Our analyses indicate that the identified problems primarily are problems concerning efficiency and satisfaction, while questions concerning effectiveness and general workflow issues get less attention. To get more attention to effectiveness problems it will probably be necessary to adjust the DUE technique by having users that are less habituated with the evaluated application. Other factors influence the DUE technique’s ability to catch problems, for instance the periodic activation of the Instrumentation Client, and personal differences among the users such as willingness to speak to a microphone without personal contact with the receiver of the message.

Collected data were transcribed, analysed and documented by the researchers for use as evidence and for the identification of facts. FINDINGS AND DISCUSSION A1: DUE supports catching usability issues and other types of problems

Severe usability problems were uncovered with DUE, for example a seemingly simple problem with the pick-up and

Users reported 37 issues through the Instrumentation Client. Of these issues 32 were usability problems, 4 problems

123

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

A3: DUE enables both short-term and long-term evaluations, with novice users and expert users

delivery ordering not being visible in F-Man. This had a huge impact in busy hours, as it resulted in users having to call drivers constantly over the phone. The problem resulted in poor service for customers calling to hear when they could expect pick-up or delivery, and on a daily basis many work hours were wasted. Finding such a problem was heavily dependent on the evaluation taking place in the real use context.

This proposition has not been directly tested in this case study, because all the participating users were experienced daily users of F-Man. The experiment did show, however, that using DUE was easy; the experiment was running for 4 weeks without any suspension. Testing with novice users would make it possible to follow how usability problems change as users move from novice to expert. Such studies can be very useful in research of learnability of complex software, an important aspect that we know only little about [7].

An important quality gained by DUE was that the developers got a very direct and sometimes completely new insight into intricacies of the users’ work situation. In summary, the DUE technique has shown its ability to capture a significant share of the problems that the users of the application perceive as important, problems that the developers and the external usability expert agree to recognize as important.

A4: DUE strengthens the validity of identified problems

To investigate this proposition we are distinguishing between internal, external, and ecological validity. These types of validity can be clarified by discussing the following characteristics of the DUE technique: (1) natural context of usage—this influences ecological validity; (2) minimizing disturbing experimental factors—this influences internal and ecological validity; (3) enabling many test users—this influences external validity; (4) the possibility of long-term evaluation—this influences external validity.

A2: DUE reduces the workload for users, developers and evaluators

The total amount of time used by users, evaluator, development manager and developers for this DUE evaluation is for instruction 2 hours 20 minutes, and 8 hours 3 minutes to locate, understand, describe and rate the 37 reported issues.

NATURAL CONTEXT OF USAGE: All users were able to use DUE in the form of the Instrumentation Client at their workplace. A few users with older PC’s experienced a decreased performance, but only one found this to be relatively disturbing.

All the empirical data indicate a small workload induced on the participants in carrying out the evaluation. Test tasks are unnecessary, as a broad range of tasks follow by the user’s ordinary work processes. The reporting of an issue is activated by a mouse click and includes the user’s spoken explanation. The average reporting time of an issue in this experiment was 1 minute 15 seconds. The users found it easy to use the Instrumentation Client. The evaluator reported that he found the issues unusually clearly documented, and thus easy and fast to understand and assess with an average time of 2 minutes 45 seconds spent per issue. The development manager and developers had a similar experience and on average spent a similar amount of time to understand and assess the issues.

The users’ busyness did influence the reporting activity, but we found no evidence that this has reduced the validity of the reported issues. Reporting activity was influenced if users heard others report what they thought to be the same issue. Thus some under-reporting of problems in this study took place, which undermines the possibility of using the frequency of a problem report as indicator of seriousness. The fact that the case study took place at just one location in a partly open office landscape made this factor important. If an evaluation can be distributed over different physical locations this factor will play a less important role.

As mentioned, the transportation company who participated in the experiment wanted the Instrumentation Client to be turned off at busy periods. In these periods this caused a few seconds waiting time to start up the Instrumentation Client and report an issue. Also some of the periodical activations, which was part of the experimental setup, were forgotten by users. This indicates that automatic starting of the Instrumentation Client would have reduced the workload even more and would have been more convenient for the user.

The developers and the evaluator emphasized that the reported issues have given them an insight into the users’ natural work settings, including a number of especially challenging situations, which otherwise had been difficult to achieve. This confirms that DUE has contributed important contextual information which has strengthened the ecological validity of the usability problems. MINIMISING DISTURBING EXPERIMENTAL FACTORS:

In summary, the case study shows how DUE induces a small workload on the users, the evaluator and the developers.

Compared to conventional think-aloud testing the evaluator found DUE recordings to be much less influenced by irrelevant and disturbing factors. The most influencing disturbing factors seem to have been

124

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the discomfort for some users of (a) speaking to a microphone, and (b) being monitored by the Instrumentation Client. Some users say that they find it difficult to describe an issue; but in only one case a description was found to be unclear and not really understandable to the developers and the evaluator. In this case the user has afterwards explained that she—in the situation of reporting—was fiery and thus unable to explain herself.

Observed factors Evaluator, development manager, and developer’s issue review takes 2:45 minutes per issue on average. User report rate is 0.35 issues per hour on average. User issue reporting takes 1:15 minutes per issue on average. Assumptions common to example 1 and 2 Evaluator, development manager, and developers each have 30 hours available for evaluation. 30 hours / 2:45 minutes per issue ≈ 654 issues in total can be evaluated. The Instrumentation Client runs for 1 hour each workday. One month is 22 workdays. Example 1: Number of users possible (Evaluation can run for one month) 22 workdays x 1 hour x 0.35 rate = 7.7 issues per user on average. 654 issues / 7.7 issues per user ≈ 84 users. 7.7 issues x 1:15 minutes ≈ 9:37 minutes spent per user on reporting.

Concerning both being monitored and speaking to a microphone, it must be noted that the majority stated that they did not feel any discomfort. For some users, however, this discomfort can have influenced their reporting activity. In the case study, the Instrumentation Client was running for one hour a day; and only one of the user’s two screens was video recorded. One could go further in providing the integrity of the user’s privacy, for instance by letting the user go through all of his/her video recordings, and delete clips that are undesirable.

Example 2: Time span possible (10 users are representative of the end user population) 10 users x 1 hour x 0.35 rate = 3.5 issues per workday on average. 654 issues / 3.5 issues per workday ≈ 186 workdays ≈ 8.5 months. 186 workdays x 1 hour x 0.35 rate x 1:15 minutes ≈ 81:23 minutes spent per user on reporting.

Table 4. Two hypothetical examples to illustrate number of users and time span possible in large-scale DUE evaluations. The average figures in the assumptions are based on measures from the case study.

ENABLING MANY TEST USERS: Based on measures of

“time used” for users, developers, and the evaluator from the case study, we have estimated that a DUE evaluation with 84 users can be done during a period of one month with an effective work effort of one week (30 hours) by an evaluator (Table 4), a development manager, and the group of developers. The participating users will on average contribute 10 minutes each. The instruction of how to use the instrumentation client will take further 5 minutes per user. The calculation is just a rough indication as it is based upon measurements of time used from this single case study. Time spent for a DUE evaluation will be heavily dependent of the number of issues reported, which again will be dependent of the software being evaluated and the involvement of the participating users. Note, that a DUE evaluation can be scaled, even after it has been started, by connecting or disconnecting users.

A5: DUE provides an information-rich communication among users, evaluators, and developers

For better communication, it is important that the usability problems are described as accurately as possible along with information-rich contextual information. Some of the developers told how they, before DUE, often experienced that they—after having spoken with a user about a problem on the phone—had not really understood the problem. The development manager added that he and the developers also experienced difficulties in their communication about usability problems. By collecting usability issues at one easily accessible place, the Instrumentation Server, the participants had one common information repository to service everybody. The study showed that the problems were easily understood by developers and the evaluator. They emphasized the effective way of combining video clips, and the user’s spoken explanations along with their mouse pointing. Often these descriptions of the issues contained contextual information, that were valuable in understanding the importance of an issue, for instance situations with interruptions by telephone calls, and difficulties in taking up work after being interrupted.

In this case study the users could work productively during the evaluation. If the software is less mature and work only is carried out for evaluation purposes, time spent by the user must be fully accounted to the evaluation cost. In the described example in Table 4 this would be 1 hour per day, totaling 22 hours per user. POSSIBILITY OF LONG-TERM EVALUATION: The case study lasted 4 weeks demonstrating that a DUE evaluation can go on for a long time. Based on the measures of time spent by the different participants in the current case study we can estimate that a DUE evaluation with 10 users lasting 8.5 months can be carried out with an effective work effort of one week by the evaluator, the development manager, and the group of developers (Table 4). Each of the 10 users can be expected to use 1 hour 21 minutes for reporting usability issues during these 8.5 months. Again, these numbers are just rough indications.

As mentioned above, some participants were not comfortable talking to the microphone or being video recorded during work; this can limit the usage of DUE, see the paragraph Minimizing disturbing experimental factors.

125

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

A6: DUE contributes to minimize the evaluator effect, and support a more thorough problem analysis

instrumentation with capture of screen video and voice recordings. The tool can be used for evaluation of prototypes and running versions of application software.

The detection of problems is entirely left to the users, and therefore the contribution to the evaluator effect from the evaluator’s problem detection activity is avoided. The rest of the evaluator effect, that is: (1) the evaluator’s severity rating of usability problems, and (2) the matching of similar reported usability issues as "same-as", is still a challenge. Instead we are left with a "user effect", in the sense that users identify different sets of usability problems when exposed to the same test setting [21]. With DUE we expect to meet this by involving a higher number of users, in a larger time span, than would be possible in conventional usability testing based on think-aloud protocols.

In a case study in industry DUE was demonstrated to be effective for catching important usability problems in a complex software product. The usability issues found related mainly to efficiency and satisfaction aspects, while issues related to overall effectiveness were rarely discovered. This indicates that DUE evaluations should be combined with evaluation techniques directed at identifying effectiveness issues. The contribution to the evaluator effect from the problem detection activity was partly eliminated by letting the users detect the problems. The involved stakeholders: users, developers, a development manager, and a usability expert, all expressed a high degree of satisfaction with the DUE technique. Information-rich descriptions of usability findings were stored in a common repository, which was easily accessible by all stakeholders. This served to clarify the understanding of problems and made the communication about possible solutions more concrete, avoiding many misunderstandings. The developers emphasized the usefulness of the direct and sometimes completely new insight into intricacies of the users’ work situation.

In the current study we have observed a tendency of the evaluator to rate the problems as more serious than the developers and the users. The evaluator, who was invited as an external usability expert, did only have a very brief introduction to the application domain and the users workflow. It could have been interesting to involve evaluators with a solid domain insight [6]. The task of matching usability problems as "same as" is an important aspect of usability analysis and far from an easily solved matter [13]. The repository of usability problems can to some extent support such analyses, for instance by grouping problems according to where in the application they were reported; also annotations to the problems might be to some use. But such use of the repository has not been experimented in this study.

Overall, the stakeholders assessed the quality and the validity of the DUE evaluation to be high. The important factors were the natural use context, few disturbing experimental factors, the ability of engaging many test users, and running the evaluation for a long period of time. Some users, however, reported discomfort by speaking to a microphone and by being monitored while working.

The evaluator found it easy to use the Instrumentation Server and emphasized that the issues were easy to understand—with one exception, the user reporting while fiery as mentioned above. The evaluator found it especially handy that he usually did not have to forward or rewind in the video. Starting the video clip 20 seconds before the user actually clicked for reporting an issue, established a sufficient understanding of the user’s situation. The 20 seconds worked fine and matches the “specious present” described by William James [18]. The evaluator found that the DUE evaluation seemed to have even more practical value than think-aloud testing. The provided information was vivid, informative, and convenient to use, he said.

The small workload to run a DUE evaluation makes it practically possible to perform large-scale usability evaluations, potentially revealing usability problems related to learnability as the user goes from novice to expert. Further, investigating the usability of a broad range of functionality in large and complex software becomes possible. ACKNOWLEDGMENTS

We would like to thank Dina Friis, Guðrún Hulda Jónsdóttir Johannessen, Esben Warming Pedersen, Kasper Hornbæk, Claus Topsøe-Jensen, Mikkel R. Jakobsen and Jakob G. Simonsen for valuable comments to an earlier version of this paper. Also anonymous reviewers gave many insightful comments. Special thanks go to Tobias Uldall-Espersen, the transportation company, and the software developing company—without their insight and generous cooperation this study had been impossible.

CONCLUSION

Usability evaluation draws heavily on scarce resources such as expertise in interaction design, programming, insight into the application domain and the company’s needs. Further, usability evaluation is time consuming and complex in many ways. In an effort to show that these difficulties might be reduced without compromising the quality of the evaluation work, a technique called Distributed Usability Evaluation (DUE) was designed. DUE is based on a tool with user-controlled problem detection, automation and non-intrusive

126

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

REFERENCES 1. Bateman, S., Gutwin, C., Osgood, N., and McCalla, G. 2009. Interactive usability instrumentation. In Proceedings of the 1st ACM SIGCHI Symposium on Engineering interactive Computing Systems, 45-54.

techniques for matching of usability problem descriptions. Interact. Comput. 20, 6 (Dec. 2008), 505-514.

14. Howarth, J., Smith-Jackson, T., and Hartson, R. 2009. Supporting novice usability practitioners with usability engineering tools. Int. J. Hum.-Comput. Stud. 67, 6 (Jun. 2009), 533-549.

2. Castillo, J. C., Hartson, H. R., and Hix, D. 1998. Remote usability evaluation: can users report their own critical incidents?. In CHI 98 Conference Summary on Human Factors in Computing Systems, 253-254.

15. ISO 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability. International Organization for Standardization, 1998.

3. Coolican, H., Research Methods and Statistics in Psychology, 3rd ed. Hodder & Stoughton, 1999.

16. Ivory, M. Y. and Hearst, M. A. 2001. The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33, 4 (Dec. 2001), 470-516.

4. Frøkjær, E., Hertzum, M., and Hornbæk, K. (2000). Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI Letters, 2(1), 345-352.

17. Jacobsen, N. E., Hertzum, M., & John, B. E. (1998b). The Evaluator Effect in Usability Tests. Conference Summary of CHI’98 Conference on Human Factors in Computing Systems, 255-256.

5. Frøkjær, E. & Hornbæk, K. Cooperative Usability Testing: Complementing Usability Tests With User-Supported Interpretation Sessions, Proc. CHI '05 extended abstracts on Human factors in computing systems, ACM Press (2005), 1383-1386.

18. James, W. 1890. Principles of Psychology. Henry Holt & Co., (Chapter XV, The Perception of Time)

19. Kim, J. H., Gunn, D. V., Schuh, E., Phillips, B., Pagulayan, R. J., and Wixon, D. 2008. Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In Proceeding of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, 443452.

6. Følstad, A. Work-Domaine Experts As Evaluators: Usability Inspection of Domain-Specific Work-Support Systems, IJHCI, 22, 3 (2007), 217-245.

7. Grossman, T., Fitzmaurice, G., and Attar, R. 2009. A survey of software learnability: metrics, methodologies and guidelines. In Proceedings of the 27th international Conference on Human Factors in Computing Systems, 649658.

20. Kjeldskov, J., Skov, M. and Stage, J. Instant Data Analysis. In Proc. Nordichi 2004 (2004), 233-240.

21. Lewis, J. Sample size for usability studies: Additional considerations. Human Factors, 36 (1994), 368-378.

8. Hartson, H. R., Castillo, J. C., Kelso, J., and Neale, W. C.

22. Nørgaard, M. & Hornbæk, K. What Do Usability Evaluators

1996. Remote evaluation: the network as an extension of the usability laboratory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 228235.

Do in Practice?: an Explorative Study of Think-Aloud Testing, Proc. of the 6th conference on Designing Interactive systems, ACM (2006), 209-218.

23. Nørgaard, M. & Hornbæk, K. Working together to Improve

9. Hartson, H. R. and Castillo, J. C. 1998. Remote evaluation for

Usability: Exploring Challenges and Successful Practices. International Journal of Technology and Human Interaction, 6(1) (2010), 33-53.

post-deployment usability improvement. In Proceedings of the Working Conference on Advanced Visual interfaces (L'Aquila, Italy, May 24 - 27, 1998). T. Catarci, M. F. Costabile, G. Santucci, and L. Taranfino, Eds. AVI '98. ACM, New York, NY, 22-29.

24. Preece, J. et al. Human-Computer Interaction, AddisonWesley, 1999.

10. Hornbæk, K. 2006. Current practice in measuring usability:

25. Spool, J. and Shroeder, W. Testing web sites: five users is

Challenges to usability studies and research. Int. J. Hum.Comput. Stud. 64, 2 (Feb. 2006), 79-102.

nowhere near enough. In Proc. CHI 2001 (2001), 285-286.

26. www.usertesting.com. Retrieved 2010 August 8th.

11. Hornbæk, K. and Frøkjær, E. (2005). Comparing usability

27. Yin, R. K., Case Study Research: Design and Methods, 4th ed.

problems and redesign proposals as input to practical systems development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM, New York, NY, 391-400.

Sage Publications Inc., 2009.

12. Hornbæk, K. and Frøkjær, E. (2008). Making use of business goals in usability evaluation: an experiment with novice evaluators. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, 903912.

13. Hornbæk, K. and Frøkjær, E. (2008). Comparison of

127

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

A Study of Mobile Mood Awareness and Communication through MobiMood Karen Church

Telefonica Research Via Augusta 177, 08021 Barcelona, Spain

[email protected]

∗

Eve Hoggan

Dept. of Computer Science University of Tampere Tampere, Finland

[email protected]

ABSTRACT

Nuria Oliver

Telefonica Research Via Augusta 177, 08021 Barcelona, Spain

[email protected]

HCI domain. Our moods, however, are complex. They are affected by many dynamic factors and can change multiple times throughout each day. Our mood can have significant implications in terms of our experiences, our actions and most importantly on our interactions with other people. For example, it has been shown that when we are happy we are more likely to communicate with others, while when we are sad we tend to distance ourselves from friends and family [16]. Furthermore, it’s been shown previously that moods are often a common theme in conversations with friends, highlighting the extent to which mood plays an important role in our daily life [4]. There has been lots of research over the past decade which explores the detection, monitoring and communication of moods through physiological sensing [9, 17, 25], through automatic detection in online blogs1 ,2 ,[8, 13], status updates3 and online diaries[24]. Each approach offers a range of trade offs in terms of validity, accuracy and privacy[7]. More recently, a variety of mobile services and applications have surfaced designed to help users communicate presence information in the form of short textual messages which often convey their mood[14]. However, the majority of existing approaches to the presentation and detection of moods do not facilitate the sharing of mood information within one’s social circle or within mobile environments. Existing mobile devices allow us to sense a range of contextual factors such as location, time and activity of users. Mobile handsets provide a great opportunity to capture information in-situ and gather feedback from users while onthe-move. Given the increased popularity of social mobile awareness applications such as Loopt4 and BrightKite5 , mobile users may also be interested in sharing moods with each other and may actually be able to positively impact each other’s moods. Furthermore, given their role as personal communication devices, mobile phones are a natural choice for sharing and communicating moods with others. In an effort to address the gaps in existing literature and to answer a series of research questions related to explicit mood sharing, communication and mobile awareness, we present the results of a live field study of a mobile application called MobiMood. MobiMood is a social mobile awareness application which allows users to submit and share moods and their associated mobile contexts with friends and others

Recent research shows that there has been increased interest in investigating the role of mood and emotions in the HCI domain. Our moods, however, are complex. They are affected by many dynamic factors and can change multiple times throughout each day. Furthermore, our mood can have significant implications in terms of our experiences, our actions and most importantly on our interactions with other people. We have developed MobiMood, a proof-of-concept social mobile application that enables groups of friends to share their moods with each other. In this paper, we present the results of an exploratory field study of MobiMood, focusing on explicit mood sharing in-situ. Our results highlight that certain contextual factors had an effect on mood and the interpretation of moods. Furthermore, mood sharing and mood awareness appear to be good springboards for conversations and increased communication among users. These and other findings lead to a number of key implications in the design of mobile social awareness applications.

Keywords Moods, emotions, mobile computing,awareness, social context, location, mobile interaction, field study

Categories and Subject Descriptors H.5.2 [Information interfaces and presentation (e.g., HCI)]: User Interface - Evaluation/methodology

General Terms Design, Experimentation, Human Factors

INTRODUCTION Recent research shows that there has been increased interest in investigating the role of mood and emotions in the ∗This project was carried out while Eve Hoggan was an intern in Telefonica Research.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16-20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

1

See See 3 See 4 See 5 See 2

128

http://nyti.ms/baswaG, last retrieved Aug 2010 http://www.wefeelfine.org, last retrieved Aug 2010 http://bit.ly/c4h64j, last retrieved Aug 2010 http://loopt.com, last retrieved Aug 2010 http://brightkite.com/, last retrieved Aug 2010

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the market. Loopt7 or BrightKite8 enable social serendipitous encounters and keeping track of what your friends are doing. Google and Yahoo also provide generic location sharing services (Latitude9 or Fireeagle10 ), along with readily available APIs that support development of location-aware mobile applications.

while on-the-move. It supports mood sharing in a similar way to microblogging services such as twitter6 , which allows status updates to be submitted and shared easily and quickly in the form of short snippets of text. In this paper we describe the MobiMood application and present the results obtained from a 2-week field study involving 15 users, split across 5 social groups. The findings of our field study lead to a number of important implications in future mobile social awareness applications.

Mood awareness

RELATED WORK MobiMood supports explicit mood sharing and awareness among groups of friends while on-the-go. Aside from mood, MobiMood also allows users to share other forms of context including location, time and social context (i.e. who I’m with). As such we have identified a number of related threads of research that belong to the general category of mobile social presence [3].

Location awareness While there has been several research projects that aim to connect electronic information to a physical location, e.g. GeoNotes [18], ActiveCampus [10], and UrbanTapestries [12]. In more recent times, focus has switched from tagging of locations to location sharing. For example, researchers working at Intel’s PlaceLab project have explored explicit sharing of location information in social communication using a prototype called Reno[22]. The goal of the Reno prototype was to understand how people leverage social context in mobile environments to share location. The authors carried out a small-scale pilot study with 7 early adopters using Reno over a period of 5 days. Their preliminary results showed that Reno facilitated more effective communication using a simple approach to disclose place. In [1], Barkhuus et al. described Connecto, a locationaware mobile application that allows location tagging and sharing among social groups. In a live field study the authors found that Connecto was used to express moods, lifestyle and events as well as location information. The authors highlight how the application allowed participants to ‘tell a story’ and express feelings relating to each others perceptions of each other. The WatchMe prototype[15] is a watch-based mobile communications device designed to support awareness and initiate communication among users. WatchMe supports multiple communication modes (text messaging, voice, etc.) and uses the display of photographs as emotional references to show users that others are thinking about them. The prototype supports sharing of location and activity information. In [2], Belloni et al. focused on mobile social awareness in the context of commuting. Through a web-based mobile interface users could explicitly self-report their current location, receive a list of train times in real time and find out which of their friends are in the same train as them. Through an informal study involving a group of 4 people, the authors found that users exhibited very different behaviours depending on their level of acquaintance with the person in question. For example, in some cases, users openly encouraged more interaction with their friends, in other cases, they avoided any form of interaction. More recently, commercial applications have appeared on 6

See

http://www.twitter.com

129

Microblogging services such as twitter provide insights into how and what people are doing. Although such services were not designed with mood in mind, recent results show that a high percentage of entries are in fact mood related, (e.g. in Twitter see: [5]). There have also been some projects that support explicit mood sharing. For example, the MoodJam project11 from the Human Computer Interaction Institute at Carnegie Mellon University provides a visualization of moods and other peoples moods based on colored strips and words. Users choose colours that feel most appropriate to them, thus creating a personalized mood representation. The website allows users to keep a record of their moods, to learn about mood trends and to share moods with others. The goal of the project is to increase mood awareness among users and groups, however, it should be noted that the project is desktop and no consideration has been given to mobile users. Nokia developed the Context Watcher application [11] with the goal of making context tracking, storage and use easy for users. The application runs on series 60 phones and allows context sharing with family, friends, colleagues, etc. The application supports explicit mood input as one form of context (along with location, activities, body data, etc.). The eMoto [23] prototype developed by Sundstr¨ om et al. supports mobile emotional messaging and communication through a series of emotion related gestures which are then mapped to a series of shapes, colours and animations. The authors carried out a user study with 5 participants and 5 friends as spectators, in which users reported on their friendships, the social experience and reactions to the emotional messages they received. Their results showed that emotional communication was not simply transferring information plus emotion from one person to another, but that the system must provide support for the sometimes fragile communication pattern that occurs among friends. They also found that users wanted their own personal ways of expressing themselves to one-another. Shirazi et al.[21] focused on sharing emotions through selfcomposed ringtones on mobile handsets. These melodies are used to share information about emotion is an easy and nonobtrusive manner. The authors carried out a preliminary user study involving 12 participants, the results of which highlight that these personal melodies have a stronger impact than pre-composed or downloaded melodies in terms of conveying mood or emotion. However, at the receiver end, users did express concerns regarding (1) misinterpretation of the melodies and the message it was trying to convey, (2) social embarrassment and (3) a lack of control. There are also a number of commercial mobile applications that support manual mood input but the majority do not support sharing of moods among friends/groups, e.g. 7

See See 9 See 10 See 11 See 8

http://loopt.com, last retrieved Aug 2010 http://brightkite.com/, last retrieved Aug 2010 www.google.com/latitude, last retrieved Aug 2010 http://fireeagle.yahoo.net/, last retrieved Aug 2010 http://moodjam.org/, last retrieved Aug 2010

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Mood of the Nation12 , My-Mood13 , and MoodSense14 . Although there has been some work that explores mood sharing in mobile environments, work to date has been limited with little focus on the role that mood awareness plays on communication in dynamic mobile environments or the impact that explicit mood sharing in social groups has on increasing awareness among users. By creating an application solely for the purpose of sharing moods within social groups we can study: (1) the impact on awareness and (2) the role of explicit mood sharing has on communication and in particular whether communication is initiated based on knowing the mood of others. As such the key contributions of this paper are as follows:

some degree. Each mood category is also associated with a different colour based on those established by Wexner: bluesad, green-energetic, purple-tense, red-angry, yellow-happy [26]. We chose orange for the custom mood (see Figure 1b). To input a mood, the user presses one of the buttons at the bottom of the screen and a bubble will begin to grow in the rounded box in middle of the screen (Figure 1a). The longer the user holds their finger on the button, the bigger the bubble will grow. The size of the resulting bubble is mapped to the intensity of the mood (1 to 10, where 1 is represents the lowest intensity and 10 represents the highest intensity). The intensity is also displayed on a progress bar so that more absolute feedback is provided. For example, if the yellow button is pressed and held until the bubble grows to its largest size, the participant’s mood is considered to be ‘happy’ with an intensity of 10.

1. An examination of how explicit disclosure of mood increases awareness and communication among groups of mobile users through a live field study in-the-wild.

Contexts

2. A set of design implications for future social mobile awareness applications.

Once the user has selected a mood and intensity level, he/she clicks ‘next’ and is taken to the context-input screen. On the context-input screen users record their situational and social contexts. The situational context allows us to determine more about the location of the user and is given by selecting one of four options: at home, at work, commuting, other. The social context 17 allows us to determine more about who the user is in the presence of when submitting a new mood. The social context is introduced by selecting one of six pre-defined options: friends, family, alone, partner, colleague, other. In addition, we log the device ID, the date and time and the physical location (in latitude/longitude form) of each user. By logging the physical location of the user we can map meanings to the situational contexts.

MOBIMOOD MobiMood is a proof-of-concept research prototype that enables groups of friends to share their moods with each other while on-the-move. The prototype consists of two components: (1) an iPhone application that allows users to record and share moods as well as comment on the moods of others; (2) a server that synchronizes and stores all mood details in the MobiMood database15 . The server forwards an up-to-date list of all moods to the mobile application. The server also consists of an email and SMS notification facility that informs members of the appropriate social network about new moods and new mood comments from friends. In addition, the server logs all interactions between the user and the iPhone application for off-line analysis of behaviour.

Mood Lists After submitting a mood, the user is presented with a list of the last thirty moods of their friends. The mood lists shows the name of the user who submitted the mood, the mood (in both textual and colour format), the date and time and the situational context. By using the tabs at the bottom of the screen, the user can also view their own previous moods (My Moods tab) or the moods of everyone else (Everyone tab) (see Figure 1c). Users can view the details of any of the listed moods by double-tapping on the mood in question.

Mood Entry When users launch the MobiMood application, they are presented with a mood input screen. The mood input screen shows six different coloured buttons at the bottom of the screen, each representing a different mood. Users can choose from one of five standard moods (sad, energetic, tense, happy and angry) or they can input their own custom mood, e.g. ‘bored’, ‘very excited’, etc. The standard moods are derived from a subset of moods found in Russell’s Circumplex of Affect [19] and XMPP [20]. Russell’s Circumplex is widely used in psychology and XMPP is a common standard used in Web based social applications. Both contain a similar set of moods16 . The Circumplex Model of Affect proposes that all affective states arise from two fundamental neurophysiological systems: one relates to valence (a pleasure - displeasure continuum) and the other relates to arousal or alertness. Each mood can be understood as a linear combination of these two dimensions, i.e. varying degrees of both valence and arousal. We chose a set of standard moods from each of these dimensions. By allowing users to set numerical values for each mood we could still capture opposing moods to

Supported Interactions This mood detail screen (Figure 1d) provides more details about each mood. The detail screen lists the user’s name, the mood, the date and time submitted, the intensity of that mood as well as any comments submitted by the users friends about the mood. This screen also includes 3 buttons that allow a user’s friend to interact with the mood entry. These buttons include an ‘add comment’ button, a button which initiates a phone call to the user in question and a button which initiates an SMS. The inclusion of these buttons allowed us to assess if mood sharing increased awareness and communication via comments, SMS messages and phone calls within the application.

12

See http://www.moodofthenation.net/, last retrieved Aug 2010 See http://bit.ly/dCFRlQ, last retrieved Aug 2010 14 See http://bit.ly/9uKEVZ, last retrieved Aug 2010 15 We use Apache for the server requirements and all data is stored in a MySQL database. 16 Related moods by Paul Ekman: http://en.wikipedia.org/ wiki/Paul_Ekman, last retrieved Aug 2010 13

USER STUDY In this section, we describe the results of a live field study of the MobiMood prototype with a focus on designing social 17

130

These labels were chosen based on informal pilot studies.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 1: Screenshots of the MobiMood application. sionals working and socialising together in Swansea, UK18 .

mobile awareness applications. Before conducting the field study, several pilot tests tool place in-house to evaluate the application design and the experimental methodology. The results of which led to several iterations of the MobiMood prototype with the final version shown in Figure 1. Note that in this study we were more focused on understanding the interesting and dynamic behaviours that emerged from using the MobiMood prototype in-the-wild rather than examining the usability/interface design of the prototype.

Group # Users # Male # Female Country

1 3 2 1 Scotland

2 2 2 0 USA

3 3 2 1 Italy

4 2 1 1 Scotland

5 5 4 1 Wales

Table 1: Details on the MobiMood participants

Participants

Procedure

To take part in the user study, participants needed to own an iPhone and be part of a group willing to participate in a 2-week study where they input and share their moods with friends. We chose to study the use of MobiMood in five close-knit groups who lived, worked and studied in different countries around the world. In total, 15 participants took part in the study (11 male and 4 female), ranging in age between 23 and 43 years (avg=28.6). The participants had a diverse set of occupations, including a journalist, a teacher, solicitors, IT professionals and students.The participants were given a small incentive of £20 for taking part and a £200 raffle was held at the end of the study and given to one participant. Table 1 shows more details about each of the five groups: number of users per group, the ratio of males to females and the country of origin for each group. Group 1 consisted of three young professionals from Glasgow, UK, two of which knew each other after having been flatmates previously, and the other was the partner of one participant (and well acquainted with the other). Group 2 consisted of two computing science PhD students from Indiana, USA, who work together in the same lab and often socialise with each other. Group 3 consisted of three professionals from Rome, Italy. Two of these are married and the other is a family member. Group 4 was made up of two young professionals in a longterm relationship with each other from Glasgow, UK. Lastly, group 5 included five members all of whom are young profes-

Before the field study began, users completed a pre-study questionnaire and installed the MobiMood application. The pre-study questionnaire was used to gather basic demographic information, details about their use of online social network sites as well as information about their moods and the factors that contribute to their moods. The live field study took place over two weeks in August 2009. During the study, we collected a series of log data for post-task analysis which included: listings of the calls and SMS sent between participants19 , the time, location, type and intensity of each submitted mood (participants could choose to omit their location data), what moods were viewed, comments on moods as well as whom the participants were with at the time of mood entry. Finally, participants were asked to complete a post-study survey to gather subjective information on their 18

Although the groups in this study represent relatively small samples, we focused on groups of users with close friendships rather than larger sample groups with less intimate relationships. If such an application was integrated with larger online social networks such as Facebook, we might expect the interaction to increase, especially given that users of online social networks would be able to select only the peers they want to interact with within an application like MobiMood. See http://bit.ly/awjxuu, last retrieved Aug 2010 19 Note that we could only count calls and SMS message initiated from within the MobiMood application

131

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2: Distribution of moods and average number of moods per user across custom and standard mood entries.

Figure 3: Moods submitted per user per group.

experiences with the application20 . In an effort to maintain the motivation amongst participants, the ability to view moods was restricted. MobiMood only allowed users to view other moods and associated comments once they had submitted a mood. One of the issues with evaluating social applications is that often they need to reach some mass level of usage before the system can be useful to users. Our study was relatively short and we wanted to ensure that enough mood entries were collected and shared. Given the natural imbalance that exists between producers and consumers of content within social systems its unlikely that these behaviours could emerge so quickly/naturally in a short time-frame. Furthermore, participants had to submit at least 15 moods before being able to see the submitted moods and comments of everyone in the study. Prior to this, participants could only view and comment on their own moods and those of their friends. At the end of study, we provided two visualizations of submitted moods to all users: one visualization at the group level and a second visualization showing the moods of all 15 participants over the 2-week period. We used email and SMS notifications to keep users informed of the interactions of others within the study21 . Whenever a participant submitted a new mood, all of the participant’s friends received an email notification. When a comment was added to a particular mood, the participant who originally entered that mood received an SMS to inform her of the new comment.

RESULTS Basic Usage Results In total, participants submitted 311 moods over the 2week period. 112 were standard moods (36%), while 176 were custom mood entries (56.6%)22 . Figure 2 shows the

Figure 4: Custom moods classified according to Russell’s Circumplex Model of Affect. distribution of mood entries per type as well as the average number of moods per user. Of the custom moods, 103 (58.5%) were unique custom mood entries, indicating that diverse sets of custom moods were submitted by participants. The most common standard mood was “happy” and the least common was “sad”, suggesting that users tend to share their positive emotions more easily than their negative emotions. We found an average of 20.1 submitted moods per participant (stdev=9.4) and an average of 62.2 moods per group (stdev=35.4). Figure 3 shows the number of moods submitted per user per group. We can see that even groups with only 2 participants (e.g. group 4) generate a high number of mood entries (> 70 moods submitted).

Custom Moods We found that 14 of the 15 participants submitted at least one custom mood, resulting in an average of 12.6 custom moods per user. When we looked more closely into the custom moods, we found 3 types of custom mood entries: (1) basic moods, e.g. “tired”, “excited” and “rushed”. 77% of custom moods fell into this category; (2) status or activity related moods (16% of the custom moods), e.g. “hating busy

20

Both pre and post-study surveys were carefully designed to include questions that would allow us assess participate reactions to an application such as MobiMood. 21 Emails are a common notification mechanism used in many social applications e.g. Facebook informs its users when friends comment on status, tag photos, etc. Note the email did not include a link to the application so if users did access MobiMood after receiving an email it was to interact in other ways, e.g. to post comments. 22 The type of the remaining 23 mood entries (i.e. standard or

custom) is unknown. The type is unknown for mood entries where the user did not want the application to track their location. We will discuss this later in the paper.

132

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

hot slow bus in the rain” and “looking forward to yoga”; and (3) combination moods, made up of a combination of a basic mood followed by a description of why the participant was in a particular mood: e.g., “grumpy (about going back to work)”, “glad, Monday done”, and “bored (of big brother!)”. 7% of custom moods fell into this category. In an attempt to better understand the custom mood entries, we manually classified all custom entries according to Russell’s Circumplex Model of Affect. Figure 4 shows the percentage of custom moods that fall into these dimensions. We find a total of 62.5% of the moods are actually unpleasant, and 62.2% fall into the deactivation dimension. Note that although we earlier found a very low occurrence of the explicit sad mood, we do find that users expressed more negative moods in their custom mood entries. In future work, we plan to investigate why and how users are ready – or not – to communicate these more negative moods with the goal of understanding how users might be persuaded to share their mood even when they are sad. We also find 50 (almost 30%) of the custom mood entries (out of the 176 that could be classified according to Russell’s model) are related to tiredness, implying that there are some patterns in the custom moods submitted. This is an area we’d like to explore further as part of future work.

Figure 5: Percentage of mood types by location.

Context and Mood We explored two different types of context in the MobiMood study: (1) location and (2) social context23 . The most popular location context chosen by participants was “home”, with 41.6% of all moods submitted at this location by 14 of the 15 users (average 9.3 moods submitted per user at this location). According to the answers from the post study questionnaire, users accessed the MobiMood application at home because there were often “at a loose end” and had more time to interact with their device. It was difficult to assess fully the effect of location on the moods experience by users given the volume and diversity of “custom” moods, however, as shown in Figure 5, we did identify a number of trends and correlations. For example, when the majority of participants recorded their location as “at work”, energetic, sad and angry levels were considerably lower (1.6% of moods at work were energetic, sad and angry), while “tense” (9.8%) was chosen more often. Whereas when location was set to “home”, energetic levels were higher. When we asked users if the location labels (i.e. at home, at work, commuting, etc.) attached to moods helped them to understand more about their friends’ moods, 11 users answered yes while the remaining 4 said no. Of the users who answered yes, their reasons included: “I can see how these effect my friends moods”, “they gave clues to what might be causing peoples moods”, and “if it was just angry it would be hard to know why they were angry without asking why but by adding commuting you would know that it was either delays/too many people etc.”. The users who reported no felt that not enough context was provided with just these labels, e.g.“too much variety of possible scenarios”. In terms of the social context associated with mood entries, we found that users submitted most of their mood entries when they were alone (35.5%). We found that 13 of the 15 users submitted moods while alone, with an average 8.5 moods submitted per user in this social context. Fig-

Figure 6: Percentage of mood types by social context. ure 6 shows the social contexts associated with individual moods. Interestingly, there were absolutely no recorded sad moods when participants were with friends or their partners and no angry moods submitted with family or their partners. Furthermore, a considerably higher number of custom moods were submitted when participants were alone (49% of custom moods). When we asked participants if the social context helped them understand more about their friends’ moods, only 7 (46.7%) answered yes. Of the users who said no, most felt that not enough additional detail was provided with these labels, e.g. “They did slightly but if at work i would assume they were with colleagues or at home it would usually be alone” while other users reported not noticing these labels as often, e.g. “didn’t really notice”. Based on initial user responses, it appears that the social context had less of an effect than location in terms of understanding why someone was in a particular mood. However, as mentioned previously, given the large volume of custom moods this is more difficult to assess. As part of future work we’d like to explore alternative interfaces for conveying such context to the end user which would enable us to assess fully the impact and effect of location and social context on mood sharing and mood awareness.

Sharing and Viewing Moods

23

One of the core features of MobiMood is to allow users to share moods among friends. In the pre-study question-

Social context in this regard refers to who the person is with when they submit their mood.

133

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ing that I could see everyone’s moods”, the results of the study show that all participants frequently attempted to view the moods submitted by strangers. On average, each participant attempted to view the list of everyone’s moods 13 times (min=1, max=61, stdev=13.9). Of the users who accessed the list of everyone’s moods, we found that 7 participants clicked on individual mood entries to access further details. On average, these 7 participants viewed an average of 6 specific moods submitted by strangers (min=1, max=26, stdev=26). These results indicate that our participants were quite curious to know what other users where doing, despite not being in the same social group.

naire we asked participants how they felt about the prospect of sharing their moods with others, while in the post-study questionnaire we gathered qualitative feedback about their experiences in sharing their moods during the study. Interestingly, we found differences in responses before and after using the MobiMood application. In the pre-study questionnaire we found users more reluctant towards sharing mood information with others. For example, in the pre-study questionnaire we asked users if they would like their mobile phone to be able to track their mood. 7 of the participants responded yes for reasons such as curiosity and the ability to track changes over time; 1 participant said no, while the remaining 7 participants said that it would depend on factors such as privacy and accuracy of the mood captured. When asked if they would like to be able to see their friend moods via their mobile phone, 10 participants said yes, mentioning curiosity again as a reason for it. One participant said, “It would make me feel more connected to them”; 3 participants said they would not like to track their friends’ moods with mobile phones, while 2 participants said that it would depend on the specific mood of their friend. We also asked participants if they would like their friends to be able to see their moods. In this case, the responses were less positive: Only 5 participants said yes. One participant stated: “I like talking about moods to people. There’s usually a story behind them!”; 8 participants said it would depend on factors such as the particular mood they were in; while 2 participants said no indicating that they would not wish to reveal more negative moods to their friends. For example, one participant commented: “sometimes hiding a bad mood is a good idea”. In the post-study questionnaire, users reactions to mood sharing were more positive. For example, in the post-study questionnaire, all 15 participants rated “viewing their friends’ moods” very highly (median=524 ) and likewise participants rated “liking the fact that friends could view their moods” also highly (median=4). Furthermore, based on the usage statistics we found that each participant viewed an average of 10.6 specific mood entries submitted by their friends (min=1, max=31, stdev=7.8). These positive findings are strengthened further when we asked users to tell us about their experiences in using MobiMood. Overall, participants indicated that sharing their moods and viewing the moods of friends’ as a positive feature of the application. For example: “I like being able to see how my friends are feeling.” and “Seeing how other people were doing was interesting, and is often lacking from other social networking apps”. As mentioned previously, the MobiMood application also allows users to view the moods of everyone within the application, i.e. the moods of people who are not within their social circle. We found that the users’ reactions to this feature was quite negative both before and after the user study. Almost the same number of participants (13 vs. 12 in the pre and post-study questionnaires respectively) stated that they would not like strangers to be able to view their moods. In both cases, participants raised privacy as their main concern: “because it’s something that is open to misinterpretation and abuse”, “mood its private or for close friends” and “it’s not really any of their business!”. However, although participants did not wish strangers to see their own moods and indicated that they were indifferent in terms of “lik-

Mood and Communication As part of this research, we were interested in determining whether the expression and sharing of moods prompts communication between users. That is, if you know your friend is in a particular mood are you more inclined to contact them? Most of the participants indicated that they liked being able to associate comments with moods (median=4). However, despite very positive subjective reactions to the comments facility, at the end of the 2-week study, participants only entered a total of 28 comments on submitted moods – an average 1.9 comments per person. Some of the questionnaire responses indicated that an improved mechanism for commenting may have facilitated more comments: For example, “I like being able to see how my friends are feeling. I also like the ability to comment on other people’s moods”. We found a similar trend in terms of the SMS/Call communication facility provided by Mobimood. For example, one user commented the following as what they liked most: “The integrated SMS/commenting feature. It meant that when I saw someone in a sad or negative mood I could instantly send them a text to offer support. It made me feel closer to my friends”. However in terms of actual communication, there were only 17 calls and 7 SMS messages25 recorded between friends throughout the 2-weeks. This suggests that users did not use MobiMood to communicate with each other regarding their moods. Given that we used email and SMS notifications informing participants when their friends submitted new moods and comments to existing moods, we hypothesized that users may carry out communication about their moods outside of the MobiMood application. In order to determine if this was the case, we asked participants in the post-study questionnaire if they initiated or experienced any communication about MobiMood outside of the application. Overall, 12 (80%) participants indicated that they utilised communication regarding their moods outside of the MobiMood application: 3 participants said they spoke about their moods via email, 3 participants phoned their friends, 4 sent SMS messages, 2 communicated through instant messaging systems and 1 participant engaged face-to-face conversations. Therefore, it appears as though sharing moods was a successful facilitator for communication outside of the MobiMood application. One participant said, “I liked being able to see my friends’ moods, and due to the fact that moods are generally one word (‘excited’, ‘happy’, ‘sad’, etc), it made me intrigued as to why they were feeling that way and what they were doing. It was a good catalyst for a conversation 25

We did not log any communication (e.g. phone or SMS messages) that took place outside of the MobiMood application.

24

Ratings were on a 5-point likert scale, where 1 indicates strongly disagree and 5 represents strongly agree.

134

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

with a friend, to see how they were”. Another user commented: “one of my friends using mobimood was a colleague, so it kind of became a ‘back-channel’ form of office communication, with the moods (‘frustrated’, ‘accomplished’, ’‘annoyed’ etc) having a shared meaning”. Also, “It made me think about their context. From my knowledge of what they were doing I was trying to guess why they applied a particular mood. From some mood messages we started conversations and this was nice”. We also explored whether the users context impacted a users willingness to engage in communication. We found that location context does have some effect on the willingness to communicate with no calls or SMS messages sent when location was set to “work”. All communication occurred when users set their location to “home” or “other”. However, 11 participants (73%) indicated that the location of a user did not affect the type of communication they would use. If a user‘s location was set to “work”, only 4 participants stated they would not call or SMS the person but they would choose a more subtle form of communication through comments or SMS. Conversely, if the location was set to “home”, participants felt comfortable calling their friend to discuss their mood. In the words of one participant, “if at work I would know not to call, a text or comment would be best or if at home would be fine to call”. The effect of social context was less important: 13 users said that the social context would not alter their willingness to interact. In other words, they would still call or send an SMS to friends even if their friends were in the company of others. However, one participant mentioned that “if (friends were) alone I would prob(ably) call”. Overall, participants found sharing moods to be a good springboard for conversation.

Figure 7: Sample visualization provided to participants at the end of the study. ticipants were provided with two mood visualizations; one for their group and a second visualization for all users across the 2-week study period. Figure 7 shows the visualization for everyone, i.e. all participants averaged over the 2 weeks. When asked how participants felt about this visualization, 13 participants said it was useful and some of the explicit comments made by participants related to increased awareness. For example, “It was cool to compare it to the public one. Also interesting to see that we all used the custom mood the most”, “I liked the look of it, and it’s fun to see how you fit in.”, and “I seen that the moods most used was custom and that suggested me that everyone likes to tell precisely how they feel”. Thus the usage results combined with answers to the questionnaire suggest that not only did MobiMood help users to become more self aware of their moods; it also supported awareness among friends and family.

Mood and Awareness Over the past two years there has been explosive growth in status update and microblogging tools that provide users with the ability to share private information about their daily lives, their experiences, their interactions and their feelings. Recent work has look explicitly at whether we can discern “moods” or “feelings” from social awareness tools such as twitter. For example, researchers from the University of Vermont have looked at using twitter as a type of collective mood ring, by analyzing tweets and deciphering the general mood of the public26 . Instead of focusing on implicit detection of moods in this manner, the goal of the MobiMood user study was to see how explicit mood sharing while on-the-go could increase awareness among groups of mobile users. In the post-study questionnaire, we asked users to tell us in their own words about their experiences with other peoples’ moods. Interestingly, when asked this question, some users made explicit connections between mood sharing and increased awareness. For example, “It made me more aware of how my friends were feeling and more likely to drop them a text if need be.”, “It was in some cases insightful to know how someone was feeling” and “It added a bit of awareness. I think its nice to know when friends are happy or stressed”. These results indicate that explicit mood sharing in-situ appears to have impacted on awareness in a positive manner. As well as attempting to increase awareness with the MobiMood tool, we also used external visualizations as an means for increasing awareness. At the end of the user study, par26

See

http://nyti.ms/1I0Ern,

KEY IMPLICATIONS Designing social applications is a difficult task[6] and when social applications are deployed in a mobile setting the environment becomes even more challenging. The live field study of MobiMood has led to a number of key implications in the design of mobile social awareness applications.

Self-expression is important In order to increase awareness within mobile groups, users must first feel like they can express themselves fully within their group and within the application. It’s important to remember that in social applications, people are the content 27 , and users of social services like to control how they express themselves and how that expression is interpreted by others. As mentioned earlier, a large proportion of mood entries were “custom” moods (almost 60%). The participants’ tendency to express their moods through the “custom” option suggests a need to express more information. In terms of expressing moods, the post study questionnaire revealed that participants would have liked to have more control over mood categories. One approach in future version of MobiMood could be to allow users to submit a combination of moods or to include a description with each mood to enable users to explain their choice of mood. Although this find27

“A social interaction design primer”, See: last checked Aug 2010

last retrieved Aug 2010

135

http://bit.ly/YOyC,

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ing is relevant to many social systems, providing innovative approaches to self-expression is particularly important when sharing information such as mood due to the intimate and complex nature of the information behind shared. One of the key challenge in providing rich self-expression capabilities in mobile environments relates to interface and interaction limitations. Touch-based interaction is a very personal modality and as such may be ideal for communicating certain moods or emotions. Another form of enriching the self-expression experience would be to incorporate sounds, haptics and images so users really feel like they have control over what they to reveal and to who.

easily find the content of interest and easily act upon that content. And providing notifications in social applications of this nature is important because it helps to keep users informed of the interactions of other users. We used visualizations at the end of the user study to help users reflect upon the mood of others within the study. Ideally, the MobiMood application would have included support for provide end-users with real-time, interactive visualizations of their moods and the moods of their group. Although this feature is not implemented in the current prototype, such visualizations could potentially increase awareness and self-reflection.

The Need to Highlight Context

Enhancing communication Given that mobile phones are used primarily as a communications device, social mobile awareness applications provide a unique opportunity to provide end users with an arena to collaborate and communicate with friends and other users while on-the-move. Perhaps the most interesting results from our user study arose from the use of MobiMood as a communication facilitator. We found that MobiMood generated a lot of conversations between users, with 12 users communicating even outside the application through email, SMS, phone calls, faceto-face conversations and instant messages. Furthermore, by providing users with a range of communication options we’re bridging the gap between these virtual and real worlds, between indirect communications like commenting towards direct communication and increased interaction. An important point to consider when facilitating communications within social mobile awareness applications is that the choice of communication channel used by end users tends to reflect intent and the type of awareness desired. This is especially true when communicating more intimate and personal information such as ones mood. For example, email is considered a good form of communication that supports one-to-one and one-to-many communications where the recipient can respond at their own convenience. SMS is mostly one-to-one, quick and easy to use and tends to be less invasive than phone calls. SMS supports a type of “I want to let the other person know I’m here but I don’t want to bother them”. Although phone calls can be invasive to the receiver they support more social dynamics of face-to-face type conversations and tend to be used in situations were an immediate responses is desired or needed or in situations that are more personal in nature. As such when designing social mobile services, designers should clearly identify the communications channel they wish to support, taking into account the intent of end users, the intimacy of the information being shared, as well as the advantages and disadvantages of that choice of communications channel.

MobiMood supports two explicit forms of context to go alongside the mood entries; (1) location and (2) social context. The location context in particular provided users more information so they could interpret or understand more about their friends’ moods. However, it was clear from some user comments that even with these contextual labels, not enough detail/clarification was provided and in some cases the labels were not explicit enough. Once again, although this finding is relevant to many context-aware applications, providing rich contextual information is even important when sharing moods among users due to the complex nature of moods and emotions. Users expressed a need and desire to know more about the context of their friends moods so they could better interpret and empathize with the moods expressed by their peers. The goal behind awareness applications is to provide users with a deeper understanding of a remote persons actions, activities or in the case of MobiMood, feelings. One of the key challenges is how to provide novel, rich and fun interfaces/interaction modalities to enable users to convey contextual information to their friends. There is room for improving the interface and input mechanism for conveying such context in the MobiMood prototype which would in turn enable us to assess fully the impact and effect of location and social context on mood sharing and mood awareness. As mentioned earlier haptics, gestures and persuasive UIs could all be explored to highlight context and help users interpret the moods fully thus improving awareness.

Interactions & Social Visibility One of the key findings of the MobiMood user study is that curiosity about our peers’ whereabouts and activities seems to be part of human nature. Participants were not only interested in their friends mood; they were also interested in the moods of strangers. MobiMood relied on users updating or sharing their moods with one another. Some users commented on forgetting to update their moods while busy. These users explicitly indicated that it was not that they didn’t want to update, they just forgot until an email reminder appeared. For example, “I did have trouble remembering to update it while being busy. If I saw more updates from my friends in my email I would have been reminded to update more often I think. I think in the future if you have larger groups so that more updates are sent people will remember to update more often.” To overcome the typical cold-start problem associated with social applications, there must be some social purpose (in our case mood sharing), there must be interaction and there should be social visibility. Mobile social awareness applications must engage users. Users need to be able to easily interact with the application,

SUMMARY AND FUTURE WORK We have presented MobiMood, a proof-of-concept research prototype which supports sharing of moods among groups of friends while on-the-move. We carried out a 2-week live field study (involving 15 participants split across 5 groups) that focused on explicit mood sharing and it’s impact on increasing awareness and communication among users. Our results highlight that mobile users enjoyed the ability to share moods with friends and that this sharing process facilitated a number of interesting conversations outside of the application. We also found that certain contextual factors

136

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

had an effect on mood, willingness to interact and the interpretation of mood. Based on our results we highlighted a number of important implications in the design of future social mobile awareness applications. We are currently investigating a number of avenues of future work related to the MobiMood prototype. We believe there is more interesting work to be done in the area of mood awareness and it’s affect on our social interactions. We are developing an improved version which incorporates some of the lessons we have learned. We plan to carry out a longitudinal field study involving more participants and more social groups. We would like to explore the social context of moods in more detail, to investigate how to facilitate more fruitful communications and conversations among users and how improving self-expression can impact on awareness within social circles.

12.

13.

14.

15.

ACKNOWLEDGEMENTS We would like to thank all the participants of the study and thanks to the NordiCHI reviewers who provided interesting insights and improvements for the final version of the paper.

16.

REFERENCES

17.

1. L. Barkhuus, B. Brown, M. Bell, S. Sherwood, M. Hall, and M. Chalmers. From awareness to repartee: sharing location within social groups. In Proceedings of CHI’08, pages 497–506. ACM, 2008. 2. N. Belloni, L. E. Holmquist, and J. Tholander. See you on the subway: exploring mobile social software. In Proceedings of CHI’09 Extended Abstracts, pages 4543–4548. ACM, 2009. 3. F. Bentley and C. J. Metcalf. The use of mobile social presence. IEEE Pervasive Computing, 8(4):35–41, 2009. 4. K. Boehner, M. Chen, and Z. Liu. Reflector: An emergent impression of collective experience. In Proceedings of the CHI Workshop on Providing Elegant Peripheral Awareness. ACM Press, 2003. 5. J. Bollen and A. P. H. Mao. Modeling public mood and emotion: Twitter sentiment and socio-economic phenomena. In Proceedings of WWW ’10, pages 1–3. ACM, 2009. to appear. 6. C. Crumlish and E. Malone. Designing Social Interfaces: Principles, Patterns, and Practices for Improving the User Experience. O’Reilly, 2009. 7. A. Geven, M. Tscheligi, and L. Noldus. Measuring mobile emotions: Measuring the impossible? In Proceedings of MobileHCI’09, pages 1–3. ACM, 2009. 8. A. J. Gill, D. Gergle, R. M. French, and J. Oberlander. Emotion rating from short blog texts. In CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 1121–1124. ACM, 2008. 9. A. Gluhak, M. Presser, L. Zhu, and S. Esfandiyari. Towards mood based mobile services and applications. In Smart Sensing and Context, 4793/2007:159–174, 2007. 10. W. Griswold, P. Shanahan, S. Brown, and R. Boyer. Activecampus: Experiments in community-oriented ubiquitous computing. IEEE Computer, 37(10), 2003. 11. J. Koolwaaij, A. Tarlano, M. Luther, and P. Nurmi. Context watcher - sharing context information in

18.

19. 20.

21.

22.

23.

24.

25.

26.

137

everyday life. In Proceedings of Web Technologies, Applications and Services, 2006. G. Lane, S. Thelwall, A. Angus, V. Peckett, N., and West. Urban tapestries: Public authoring, place and mobility, 2005. Project final report, Proboscis, UK, London, UK, 2005. G. Leshed and J. J. Kaye. Understanding how bloggers feel: recognizing affect in blog posts. In CHI ’06: CHI ’06 extended abstracts on Human factors in computing systems, pages 1019–1024. ACM, 2006. M. Luther, S. Bohm, M. Wagner, and J. Koolwaaij. Enhanced presence tracking for mobile applications. In ISWC ’05: Proceedings of the International Semantic Web Conference, 2005. N. Marmasse, C. Schmandt, and D. Spectre. Watchme: Communication and awareness between members of a closely-knit group. In Ubicomp, volume 3205 of Lecture Notes in Computer Science, pages 214–231. Springer, 2004. C. McIntyre, L. Clark, and S. Cross. The effect of induced social interaction on positive and negatives affect. Bulletin of the Psychonomic Society, 29:67–70, 1991. M. Morris. Technologies for heart and mind: New directions in embedded assessment. Intel Technology Journal, 11(1), 2007. P. Persson and P. Fagerberg. Geonotes: a real-use study of a public location-aware community system, 2002. Technical Report SICS-T–2002/27-SE, SICS, University of G¨ oteborg, Sweden, 2002. J. Russell. Aa circumplex model of affect. Journal of Personality and Social Psychology, 39:1161, 1980. P. Saint-Andre. Extensible messaging and presence protocol (xmpp): Instant messaging and presence, 2004. IETF RFC 3291. A. S. Shirazi, F. Alt, A. Schmidt, A.-H. Sarjanoja, L. Hynninen, J. H¨ akkil¨ a, and P. Holleis. Emotion sharing via self-composed melodies on mobile phones. In Proceedings of MobileHCI’08, pages 1–4. ACM, 2009. I. Smith, S. Consolvo, A. Lamarca, J. Hightower, J. Scott, T. Sohn, J. Hughes, G. Iachello, and G. D. Abowd. Social disclosure of place: From location technology to communication practices. IEEE Pervasive Computing, 3468:134–151, 2005. P. Sundstr¨ om, A. St˚ ahl, and K. H¨ o¨ ok. In situ informants exploring an emotional mobile messaging system in their everyday practice. International Journal of Human-Computer Studies, 65(4):388–403, 2007. J. R. Vittengl and C. S. Holt. A time-series diary study of mood and social interaction. Motivation and Emotion, 22(3):255–275, 1998. H. Wang, H. Prendinger, and T. Igarashi. Communicating emotions in online chat using physiological sensors and animated text. In Proceedings of CHI ’04 extended abstracts, pages 1171–1174. ACM, 2004. L. Wexner. The degree to which color (hues) are associated with mood-tones. The Journal of Applied Psychology, 38(6):432–435, 1954.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Engaging Spect-actors with Multimodal Digital Puppetry Céline Coutrix1, Giulio Jacucci12, Anna Spagnolli3, Lingyi Ma1, Matti Helin1, Gabriela Richard4, Lorenza Parisi5, Stefano Roveda6, Prayag Narula1 1 Helsinki Institute for Information Technology HIIT, Aalto University [email protected] 2 Department of Computer Science, University of Helsinki, [email protected] 3 HTLab Dept. of General Psychology University of Padova, [email protected] 4 Educational Communication and Technology, New York University [email protected] 5 Facoltà di Scienze della Comunicazione, Università di Roma “La Sapienza” [email protected] 6 Studio Azzuro, [email protected] ABSTRACT

spectators are invited to act in this fictional space, a truly engaging experience co-created by the puppeteer and the spectators can arise. Thus, spectators are not just passive recipients of the storytelling, but become active characters in it – spec-actors in short.

We present Euclide, a multimodal system for live animation of a virtual puppet that is composed of a data glove, MIDI music board, keyboard, and mouse. The paper reports on a field study in which Euclide was used in a science museum to animate visitors as they passed by five different stations. Quantitative and qualitative analysis of several hours of videos served investigation of how the various features of the multimodal system were used by different puppeteers in the unfolding of the sessions. We found that the puppetry was truly multimodal, utilizing several input modalities simultaneously; the structure of sessions followed performative strategies; and the engagement of spectators was co-constructed. The puppeteer uses nonverbal resources (effects) and we examined how they are instrumental to talk as nonverbal turns, verbal accompaniment, and virtual gesturing. These findings allow describing digital puppetry as an emerging promising field of application for HCI that acts as a source of insights applicable in a range of multimodal performative interactive systems.

The animation of digital objects or digital puppets does incite imagination of spectators in a particular way. Spectators interacting with such digital “beings” enter also a fictional space. Digital objects can be animated in many different ways and can be transformed in real time to provide novel possibilities for engagement and co-creation. Computer-mediated puppetry has been used mostly for animation production purposes and not for live public performances. In contrast, in this paper we focus on live digital puppetry with live audience using real-time multimodal animation as a promising area of application for engaging spectators. We describe a concrete system, Euclide, installed in a science museum in Naples, Italy. Euclide utilizes multimodal inputs: a data glove, a standard keyboard, a MIDI keyboard, and a mouse (see Figure 1, left). The system has a control center from which the puppeteer operates five different stations at the museum (see Figure 1, right). These are audiovisual output installations for the digital puppets, monitored by a camera and microphone. The puppeteer is able to switch between stations swiftly and choose to interact with passers-by. We are interested in characterizing the interactive and performative aspects of this application area by describing its three following elements:

Author Keywords

Multimodality, performative interaction, engagement, cocreation, virtual puppetry, field study, museum. ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

Puppetry is a 30,000-years-old art [4]. Most puppetry involves storytelling, and its impact is determined by the ability to create a fictional space for the spectators that has aspects in common with magic and with play. If the

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

•

How the puppeteer makes use of the complex multimodal system,

•

The interaction sessions in their structure and lengths on this example of a non-work related interaction,

•

How engagement is co-constructed by the puppeteer and “spect-actors”.

Others elements that are of interest are left for future work, such as considering the social context and the meaning making process. Those elements compose are what makes

138

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 1. Puppeteer interacting with the multimodal system Euclide in the control room (left and center) and interactive station and young audience interacting with the system (right).

puppetry an interesting application field for HCI. In order to study the first three, we collected composite videos of puppeteer, spect-actors, and puppet over several days, with three professional puppeteers. The findings utilize video analysis employing quantitative and qualitative methods. The variables we considered arose form these research questions and the data, as we explain in the Study section.

Chinese shadow puppetry has been implemented in a system called I-Shadows [14]. The installation allowed children to create stories for an audience. In this theater, the user interacts with the system by controlling a physical puppet of either a hero or a villain, whose movements are interpreted by a vision system that sends the information to the autonomous character’s environment. With CoPuppet [6], a system for collaborative puppetry is presented. The CoPuppet project explores the possibilities offered by multimodal and cooperative interaction, in which performers, or even audience members, are called upon to affect different parts of a puppet through gestures and voice.

RELATED WORK Real-time Digital Puppetry

There are surprisingly few examples of digital puppetry, given the possibilities offered by current technologies. Computer puppetry generally is used to refer to mapping the movements of a human performer to an animated character in real time [32]. Dontcheva et al. [13] introduce a novel motion-editing technique that derives implicit relationships between the animator and character. This and other work are generally motivated by the development of tools for animation production. Not only human performers are used to this end. Mazalek and Nitsche [28] addresses production and performative challenges involved in creating machinima through the development of tangible interfaces for controlling 3D virtual actors and environments.

Some systems are not directly puppetry applications but include interactive aspects whereby digital objects are influenced by spectators. Affective Painting [33] supports self-expression by adapting in real time to the perceived emotional state of a viewer, which is recognized from his or her facial expressions. Cavazza et al. [8] introduce a prototype of multimodal acting in mixed-reality interactive storytelling, in which the position, attitude, and gestures of spectators are monitored and influence the development of the story. Camurri et al. [7] propose multi-sensory integrated expressive environments as a framework for performing arts and culture oriented to mixed-reality applications. They report an example in which an actress’s lips and face movements are tracked by the EyesWeb system and her voice is processed in real time to control music.

Other examples include development of such production tools for pupils and use in everyday play. Barnes et al. [2] present “Video Puppetry,” in which the puppeteer first creates a cast of physical puppets, using paper, markers, and scissors. An overhead camera tracks the motions of the puppets and renders them on a new background while removing the puppeteer’s hand.

Frameworks for Performing Media and Spectators

Recently, several researchers have applied different performative or theatrical metaphors to describe the emergence of novel interaction formats and experiences that are related to real-time animated puppetry.

Liikkanen et al. [26] present PuppetWall, a multi-user, multimodal system intended for digitally augmented puppeteering. This application allows natural interaction to control puppets and manipulate playgrounds comprising background, props, and puppets. PuppetWall utilizes hand movement tracking, a multi-touch display, and emotional speech recognition for interfacing. Here, however, the idea is that the visitors are themselves having fun with the puppets and the puppeteer is not hidden backstage.

Dalsgaard and Koefoed Hansen [10] observe how the user is simultaneously operator, performer, and spectator. A central facet of aesthetics of interaction is rooted in, as they put it, the user’s experience of herself “performing her perception.” They argue that this three-in-one situation is always shaping the user’s understanding and perception of

139

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the interaction, and they address the notion of the performative spectator and the spectating performer.

Figure 1 (left) shows a hidden animator controlling the movements and mimicry of a virtual character through a multimodal interface including a data glove. The animator’s hand movements “activate’’ the virtual character, controlling the mimicking, and digital effects alter the animator’s voice.

Reeves et al. [31] present a taxonomy with four broad design strategies for the performer’s manipulations of an interface and their resulting effects on spectators: the “secretive”, wherein manipulations and effects are largely hidden; the “expressive,” in which they tend to be revealed, enabling the spectator to fully appreciate the performer’s interaction; the “magical”, where effects are revealed but the manipulations that caused them are hidden; and, finally, the “suspenseful”, wherein manipulations are apparent but effects are revealed only as the spectator takes his or her turn. Benford et al. [3] extend the above framework for designing spectator interfaces with the concept of performance frames, enabling one to distinguish audience from bystanders. They conclude that ambiguity to blur the frame can be a powerful design tactic, empowering players to willingly suspend their disbelief.

The rendering of the character appears on a screen in a second space, the “stage” (see Figure 1). Five stages are scattered about the museum. The animator monitors the audience members via a microphone and a camera and reacts to them (see Figure 1). Therefore, the puppeteer can react and respond to people talking to the character.

Figure 2. Virtual character of the Euclide system.

Also central to the discussion is the framework of “Interaction as Performance” [17][18][19]. This framework is based on anthropological studies of performance that have roots in a pragmatic view of experience. In particular, the framework proposes a variety of principles aimed at describing performative interaction. One of these is that of accomplishment and intervention. The etymology of the term “performance” shows that it does not have the structuralist implication of manifesting form but, rather, a processual sense of bringing to completion or accomplishing. The concept of event and processual character is also key: performances are not generally amorphous or open-ended; they have diachronic structure, a beginning, a sequence of overlapping but isolable phases, and an end. Expression and experience is another element of import. According to pragmatist views, an experience is never completed until it is expressed. Also, in an experience there is a structural relationship between doing and undergoing.

The system offers 100 different features to the puppeteer for animating the character, among them jumping (see Figure 2) or dressing like Santa Claus (see Figure 4). To allow use of this great expressive power, with many elements sometimes utilized simultaneously, different modalities are proposed. The interface includes 11 screens, two computer keyboards, two mice, a data glove, a microphone, headphones, and a MIDI keyboard, all in the control room (see Figure 1, left and center). Among these devices, three screens, one computer keyboard, one mouse, the MIDI keyboard, the microphone, and the glove are dedicated to real-time puppetry. The other devices are dedicated to system launch, switching between interactive areas or setting the puppet to inactive in order for the puppeteer to take a break. Facial Expressions

Using a data glove, the puppeteer can horizontally open/close both eyes by bending the index finger of the glove at i2 (see Figure 3), vertically open/close both eyes by bending the index finger of the glove at i3, and move the eyebrows around the eye by bending the middle finger of the glove at m2.

These novel frameworks originate from concurrent trends in HCI, including the emergence of installations as a delivery platform for interactive experiences. Installations as also tangible interfaces have the property of providing a stage on which the user becomes at times a performer. Other trends include attention to the fact that performers in general have more and more technology to mediate their interaction with spectators. The system we present now is a novel multimodal puppetry application that aims at engaging spectators in performative sessions of interaction. THE SYSTEM: EUCLIDE

Euclide is a virtual puppet (see Figure 2) that has an engaging role in the visit of a science museum in Naples, Italy. The system offers a multimodal interface to the puppeteer in order to animate a virtual puppet and entice the audience.

Figure 3. Control points on the fingers of the glove.

To control the mouth of the puppet, the puppeteer can bend the thumb of the glove at t1 to open/close the mouth and

140

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

bend the thumb of the glove at t2 to raise/lower the corners of the lips. This enables, respectively, making the puppet talk and look happy or sad.

constantly. To overcome this difficulty pointed out by a puppeteer, the thumb now controls the opening of the mouth.

The color of the skin can be controlled by four of the MIDI keyboard sliders, on the basis of the three RGBA channels.

Also, expressive gestures of the puppeteer were mapped to the expression of the puppet. For instance, clenching the fist makes the puppet look angry, opening the hand makes the puppet look happy, and relaxing the hand makes the puppet seems neutral.

Some costume accessories can be added to the puppet’s face, such as hats, glasses, and mustache, by pressing keys in the left portion of the MIDI keyboard (see Figure 1).

THE STUDY

Body Movements and Expressions

We studied the system in the science museum in Naples, where it is used for engaging the audience in their visit.

The puppet has global freedom of movement in 3D space. For this, the puppeteer uses the wheel of the mouse to translate the puppet in the depth dimension. Two sliders on the MIDI keyboard enable translation of the puppet vertically and laterally. The puppeteer can also change the orientation of the puppet to, for example, have its head upside down, by pressing the “M” key on the computer keyboard (see Figure 1).

Data Collection

We video-recorded the use of the multimodal puppetry system installed in the museum in winter and spring 2009. We recorded several hours of interaction from four viewpoints synchronized in one video (see Figure 4): the audience from front left and front right, and the real-time image of both puppet and puppeteer from the front. The recording also included sound. In total, three puppeteers interacted with the system. In addition to the recording, there was an interview with the most experienced puppeteer, to examine his use of the system.

The body of the puppet can be moved locally by jumping, turning, and bending its spring. Jumping extent is controlled by the little finger of the glove (l3 in Figure 3). The lateral movement of the mouse controls rotation extent around Bit’s vertical axis. Bending forward/backward is controlled with the y dimension of mouse movement. Jumping, rotation, and bending maximums are each controlled by one of the keyboard sliders. As well as the face, some costume items can also be added to the puppet’s body, such as a Superman costume, by pressing keys in the left part of the MIDI keyboard. Background

The background behind the puppet can be modified, from black to live video of the puppeteer or to still images or prerecorded videos. The interface also allows fine-tuning the transparency of the background (via the wheel on the MIDI keyboard), the orientation of the background in the vertical plane (“N” on the computer keyboard), the zoom of the background image (controlled via a potentiometer on the MIDI keyboard), and the 3D position of the background (in the vertical plane and near to distant, each controlled via a potentiometer on the MIDI keyboard).

Figure 4: Composite video, in which the puppet is dressed as Santa Claus. Data Analysis: Procedure and Reliability Coding Scheme

We employed constant comparison analysis [15] to the video data collected. While initially conceptualized as an inductive process, whereby theory emerges from close reading and analysis of the data, contemporary practice asserts that deductive or abductive processes can also be used in constant comparison analysis [25]. For the purposes of this analysis, we used an abductive process, letting the theoretical constructs emerge from the data as well as from existing theory such as the PAD scale [30] for measuring emotion.

The design of this complex form of interaction has been driven by the requirements from the museum to use generic and flexible hardware, the puppeteers’ requirements for a large number of functionalities, and usability. For instance, the location of the controls we presented here has been studied, to allow the puppeteer to perform particular movements simultaneously. For instance, moving the mouth and the eyes of the puppet have to be enabled simultaneously for puppetry.

Constant comparison analysis as conceptualized by Glaser and Strauss [15] is suitable for the analysis of multiple data sources, including observation. Constant comparison analysis is a fundamental element of grounded theory [15] and is a means of systematically generating theory by working closely with the data and setting aside previous theoretical assumptions. The coding process is methodical

Early versions of the system were not as easy to use, and the interface has been improved with puppeteers’ assistance. For instance, opening the mouth was tiring before: it was controlled by the index finger and done

141

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and starts with open coding (creating initial categories), continues with axial coding (parsing out relationships), and finally uses selective coding (coding around core categories) [34]. The theory is developed once core categories are repeated enough in the data for a point of theoretical saturation to be reached. We discussed the coding between three researchers and our final consensual scheme included 35 tracks for annotation. The audio was annotated using Praat software [5] and the video was annotated using Anvil software [23].

seldom-used features, such as costume or background change, are handled with less direct devices (e.g., the MIDI keyboard).

Selection of Clips

We selected clips via both a deductive and an inductive process. We applied a deductive process because clip selection was grounded in our research questions and objectives [12]. However, we were aware of a clip's narrative power [12] for understanding the phenomena. In other words, each clip was analyzed to understand whether it fit the scope of our research as well as how illustrative it was. Accordingly, the selection of clips also took an abductive approach, particularly influenced by Goldman's technique of negotiated clip selection [16]. The video material was evenly divided between the three coders.

Figure 5: Percentage of the duration of use for each feature (Logarithmic scale).

Figure 5 also shows that third most used features are the animation of the body (bending and rotation). Then comes the use of special effects, such as costume or background change used in specific situations when relevant. Least important features are more complex “physical” actions such as moving around.

Agreement Betweens Coders

For reliability of our annotation, we also assessed the intercoder agreement rate. As the basis, we use Cohen’s kappa index, which is a descriptive statistics that summarizes agreement as percentage of the cases on which the coders agree across a number of objects against bare chance. Index at 0.80 to 1.00 is considered very good agreement; 0.60 to 0.80 considered good agreement; 0.40 to 0.60 considered moderate agreement; 0.20 to 0.40 considered fair agreement; and 0.20 or less considered poor agreement [24]. In practice, if our assessed agreement was less than 0.6, we discussed between coders and made revision in the data, then continued testing and revising until the agreement was satisfactory, i.e. greater than 0.6.

Figure 6: Simultaneous use: no, individual, or multiple features.

In addition, we found that 2 to 4 features are used simultaneously 39% of the time (see Figure 6), whereas no features are used 27% of the time and a single feature 34% of the time. This highlights the actual significant simultaneous use by the puppeteers of the multimodal resources [27].

FINDINGS

In the presentation of the findings, we focus on how the puppeteers make use of the multimodal system, how the sessions are structured, and how engagement is co-constructed by puppeteer and spect-actors. These findings draw from the statistics distilled from the quantitative and qualitative analysis of interaction sessions.

More precisely, the most used set of features is speech alone, followed by speech and lips together characterizing the talking of the puppet and then lip movement alone to characterize expressions such as smiling. Speech without lip movements is the most frequent combination, meaning that the speech and lips are not fully in synchronization. This points to possible improvements in automating the lips’ synchronization.

Multimodal Puppetry

Several features’ use is important: the speech, lips, and eye movements, as well as bending and rotation (see Figure 5). This is related to the fact that these are distributed by microphone, different fingers of the data glove, and the mouse. Indeed, the interface for these features has been provided in the data glove and via the mouse in order for these to be easy to use simultaneously. These devices, respectively, are attached constantly to the puppeteer’s right hand and in the resting location for the left hand most of the time (i.e., when it is not required for operating a keyboard). These devices are used all the time to make to animate the speech, facial expressions and body of the puppet. Other

Figure 7 shows that the distribution of speech and betweenspeech durations for the puppeteer follows a power law. Indeed, the puppeteers keep segments of speech short, trying to make the audience react rather than monopolizing the conversation (see Figure 7a). In a similar manner, they try to show that the puppet is paying attention to the audience and therefore rarely let it remain silent for long (see Figure 7b).

142

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Comparing speech and silent use of features, we found that the puppet is considerably active also when the puppeteer is silent. Consequently, the use of the features goes beyond a basic animated character as we further explain in section “Resourceful Co-constructing of Engagement”.

In all sessions we analyzed, the number of people in the audience tends to increase until it reaches a maximum near the middle of the session and then decreases towards the end. The increasing phase demonstrates that the system appeals to visitors. The audience also tends to use body movement such as waving in the second part of the session. Length of Sessions

(a)

The length of session seems to follow a power law: four clips last more than nine minutes, five are between two and nine minutes long, nine last 1–2 minutes, and seven are very short. However, we should note that some of the sessions are a continuation: the same group already interacted with the puppeteer. Moreover, in some cases the sessions are interrupted by teachers or parents who want to move on in the visit.

(b)

Figure 7: Distribution of the 35 speech (a) and 36 silence (b) durations for puppeteers, in seconds.

Phase

Audience members’ action

Puppet’s reaction

Approach

Enter (one or several people, or only a voice)

Stops activity

Testing

Present themselves

Presents itself

Laugh

Skips happily Asks what is funny

Say bad words or abuse a bit

Repeats in a mechanical way Cries, complains, and goes away

Say a keyword

Changes costume Tells a story Sings a song

Ask questions

Answers normally Answers as if crazy or slow Answers and asks the audience the same

Greet

Greets

Playing

Ending

Structure of Sessions

Drawing from previous work on interaction with entertainment-related interactive installations [20], we analyzed the structure of the sessions. Different phases were annotated: Approach, Testing, Playing, and Ending. Approach is the phase in which participants enter the interaction area, observing. Testing is that in which they start trying to interact with the installation, by taking a particular action such as touching the screen in order to find out which actions have an effect on the installation. Playing is the phase in which participants interact with the installation in an aware, active, and involved way. This phase includes the climax or main action of the interaction session. Ending is the phase in which participants have their attention diverted from the installation before they leave. Figure 8 shows that these phases are balanced in the sessions, with the exception of Approach, which could not always be recorded. In this regard, the structure of the sessions as explained by the puppeteer during the interview confirms this distribution: more actions are proposed during the playing phase (Table 1).

Figure 8: Average relative length of phases in analyzed sessions (100% is the average duration of sessions).

Table 1. Summary of the structure of sessions as reported by the most experienced puppeteer.

Considering these four phases, we studied the evolution of the perceived emotion of the audience through pleasure and arousal (see Figure 9). For this we considered the facial expressions within the group, as well as their oral expressions and gestures. We found out that pleasure is high during the approach and testing phases while arousal is low. Pleasure is less important (p<0.01) during playing phase while arousal is more important (p<0.01). During the ending phase, the visitors show more pleasure (p<0.01) with less (p<0.01), but still positive, arousal. The evolution

Emergence of Performative Structures

We grounded the following analysis on the performative framework presented in the related work. Engaging the Audience Throughout the Sessions

People in the audience don’t talk to each other, 99% of the time, and they don’t pay attention to the area outside the interactive space, 98% of the time. They prefer to talk to the puppet (65% of the time). In addition, their pleasure and arousal was never annotated as negative.

143

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

of pleasure and arousal demonstrates that the audience starts interacting happily but calm, then gets excited and finishes the interaction happily and calm again.

visitors who would hardly be attracted by speech, such as very young babies or a far-away, busy audience. Nonverbal turns: In this case, nonverbal resources are used to make an individual contribution within a longer interaction that also features verbal exchanges. Examples include wearing some accessories on the special request of a visitor, and subsequently commenting on it, and explaining a detail by displaying a picture of it. Verbal accompaniment: This is the case in which nonverbal actions, such as acrobatic movements, are accompanied by speech. The communication relies on the nonverbal resource, which dictates its structure and meaning. An example is provided below: the puppet makes a jump commented upon by an accompanying onomatopoeic sound.

Figure 9: Evolution of pleasure and arousal in the sessions’ phases (annotated between -1 and 1).

We also noticed that in all sessions the audience’s members carefully take turns for saying goodbye to the puppet, regardless of the way they were interacting during the session, either through a leader or simultaneously. For instance, Figure 10 shows the audience interaction throughout time in a one-minute-long session: They mainly talk all at the same time in a chaotic way. At three occasions a leader emerge in the group. Before they leave, they speak one after the other in order to say goodbye. This ending interaction pattern is recurrent in every session we analyzed.

Example 1 (Disc 2, video 1, f 1:52) (Nonverbal events are in double parentheses. Extending over multiple columns indicates that the preceding sound is stretched.) Puppet:

Ho una molla I’ve got a spring Mi serve per saltare I need it to jump ⎡

Dong:: dong:: ⎤ Dong dong ⎣((Starts jumping))⎦

As the puppet starts jumping, the main resource to communicate with the visitor switches from verbal to visual: the onomatopoeic sound “dong dong” refers directly to the jump both in its duration and in its meaning. By producing it, the puppet continues the verbal communication with the audience even during the jump; exclusive room is not given to the visual resources, but the puppeteer signals the change in priority from verbal to visual.

Figure 10: Group verbal interaction with puppet in a oneminute-long session: The audience’s members mainly speak simultaneously, interacts through a leader three times, and take turns to talk to the puppet at the end.

Virtual gesturing: In the final case, priority is given to speech; visual features are used in synch with speech, following its structure and duration. The head movement in example 2 provides an instance of this.

Another interesting finding is that the structure of session can also be related to the use of the multimodal interface. Indeed, features like special effects (costume or background change) tend to be used during the Playing phase.

Example 2 (Disc 2, video 1, f 7:57) (The notation conventions are the same as for the previous example; in addition, events are in brackets when they overlap in time.)

Resourceful Co-constructing of Engagement The Puppet’s Multimodal Resources

The puppeteer acts by using visual effects (eyeglasses, masks, clothes, etc.), and performing virtual movements (jumping, rotating, etc.); the different ways in which these nonverbal resources are used can be distinguished on the basis of their relation to talk, as is done with natural gestures [22]. The following four cases have been observed.

Visitor:

Dove sei? Where are you?

Puppet:

Sono⎡::((pause))⎤ I’m ⎣((Looking down))⎦ ⎡qui dentro,⎤ inside here ⎣((Looking around))⎦

Pure special effects: This is the case in which nonverbal resources represent the predominant modality for interacting with the visitors; they are deployed to address

non mi vedi? can’t you see me

144

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Circular movement of the head is produced as the puppeteer starts replying to the question “Where are you?” and continues while the sentence is interrupted, filling the gap left by the search for an answer. The movement then stops and restarts with a different function when the answer “inside here” is finally produced: the puppet’s head, moving in a circle inside the monitor, shows what “here” refers to and performs a pointing gesture. In natural communication, verbal and nonverbal resources contribute to the creation of one, joint meaning and vary in their mutual dependence [22][29]. The four cases considered here suggest that this holds true for mediated communication as well, adapted to the specific resources of the communication medium – as is the case with the pointing gesture performed with the puppet’s head.

what it could do, in an explorative attempt that has already been observed with other performative technologies [20]. Since the visitor did not see any reaction except for a nonmotivated friendly look, she judged the character to be “dumb.” As the puppeteer started talking (and reacting to the specific, situated abuse), the abuse came rapidly to an end. The appearance of the puppet remained cartoon-like, yet the talk redefined the capacities of the puppet, reconfiguring its social presence. As is shown by De Angeli & Brahnam [11], visitors probe the cognitive abilities of a virtual character. The conversational agents studied by De Angeli & Brahnam, in this situation, mocked a human speaker and failed to show human conversational competence; in fact, the abuse often focused on the poor quality of the speech. A puppet instead can participate more properly in a verbal exchange, and this is probably responsible for the way in which abuse episodes are concluded. Let’s consider the case of irony. In extract 4, below, for instance, the puppeteer recognizes irony, which relies on the ability to understand implicit meaning [1], and is able to defeat the abuse.

Interactional Consequences

The multimodal resources do not just feed communication; they also define who the puppet is. While the repertoire of special effects per se is limited and cannot keep the visitor’s interest for a long time, talk can sustain the interaction longer. What the puppet does is entertain itself in conversations with the visitors, wherein talk is the main resource and the visual effects are instrumental to it. The visitors in our video recordings inquire about the puppet’s personal information (i.e., name, age, family, and sex), about what it can do, and about its general knowledge of the “real” world (TV shows, songs, and popular people). The puppet is so successful in this that most conversations are interrupted forcedly by an external intervention, from a parent or a teacher soliciting the young visitors to leave.

Example 4 (Disc 1, video 2, f 06:13) Child:

Ma::: chi e è tuo padre? But who’s your father

Puppet:

Ma io ne ho un sacco di papà Well, I have a lot of fathers

Child:

E allora tua mamma::: ((turns to a friend)) Then your mom:::

The puppet’s resources as a conversation partner distinguish it from other conversational agents, as is apparent from the cases of verbal abuse. In example 3, the puppeteer connects with a puppet located in a new room, with some young visitors trying to interact with it. Immediately after the puppeteer switches on, the conversation in example 3 unfolds.

Puppet:

↑Allora mia volume)) Then my mom

Child:

((pause)) ⎡che lavoro fa?⎤ What does she do for a living

Puppet:

⎣↑Allora mia ((increased volume)) Then my mum

Example 3 (Disc 1, video 3, f 00:20)

Child:

Che lavoro fa? What does she do for a living

Puppet:

AAAAAAH

Puppet:

mia mamma è la scheda madre del computer My mom is the mother board of the computer

(Same notations as in the previous examples; inaudible sound is between rounded brackets; non-verbal events are in double rounded brackets.) Child:

Vabbeh adesso basta. All right; let’s stop this

Child:

((turns back, surprised))

Puppet:

mamma?⎦

In this exchange, the visitor replies to the puppet’s announcement (“I have a lot of fathers”) with a question that implies offense to the puppet’s mother (“Then your mom?”). The puppeteer recognizes the implicit meaning in the visitor’s words before the sentence is even completed: he repeats the visitor’s ironic words twice at a higher volume and in an angry tone, responding to the indirect offense with an indirect threat. The visitor then completes the sentence (“What does she do for a living?”), the puppet reacts with a (funny) scream (“AAAAH!”), all visitors

Ogni volta che scemo, ogni volta che scemo. All the times dumb, all the times dumb. E tu invece come sei? And what about you instead, what are you?

Child:

((increased

Children: hahahah

(Scemo) Dumb

Puppet:

mamma?

((goes away))

The abuse takes place while the puppet is not animated. The visitor tried to interact with the puppet and to understand

145

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

laugh, and the conversation continues normally (“My mom is the motherboard of the computer”).

dedicated to the four inactive stations in the museum like presented in [9].

In synthesis, the different resources available to the puppeteer are combined together in multimodal communication with several specificities. The resulting social presence attracts the visitors effectively and avoids the low status that is attributed to other virtual characters.

Spect-actors and the Co-constructing of Engagement

The puppeteer uses nonverbal resources like visual effects and performing virtual movements. We examined the relation of these nonverbal resources to talk identifying types of use as nonverbal turns, verbal accompaniment, virtual gesturing. However we noticed that in the multimodal resources is talk that can sustain interaction longer. Talk is the main resource and the visual effects are instrumental to it. For example verbal abuses usually addressed to autonomic virtual characters (that generally are attributed a low status) are here resolved through irony by the puppeteer in an effective way. We also observed how the narrative of the sessions emerges from the interaction and contribution of both the puppeteer and the spect-actors.

DISCUSSION AND CONCLUSIONS

We described a system of digital puppetry and reported on a field study with the aim of characterizing a promising application field for HCI. The system we described featured digital puppetry using real-time multimodal animation. The attractiveness for HCI in this area lies in that it provides the possibility to introduce advanced interface techniques. Other examples of digital puppetry, such as those described by Liikkanen et al. [26] utilize expressive (emotional) feature-tracking from voice. This installation utilized a data glove and a variety of other input devices used in a multimodal way. In addition, puppetry provides a case for recent frameworks of interaction that focus on performative situations or installations. In particular, it illustrates a form of interaction where engagement is co-constructed by spectators and puppeteer through the interface.

This area of application is particularly engaging because the spectators are called upon to interact with the puppets in improvised sessions. They express themselves and therefore feel that they have the role of protagonist. Current analysis and frameworks for installations or performing media anticipate some of these themes. Dalsgaard and Koefoed Hansen [10] point to the multiple roles of the user operator, performer, and spectator (see [17], [18]). Jacucci et al. [19] point to a variety of elements characterizing interaction as performance, including the structural relationship between expression and experience. These frameworks ascribe to the user an important role in the construction of the resulting performance. We believe these frameworks can be useful in further analysis of this emergent field.

Multimodal Use of the System by the Puppeteer

We described the use of the system as truly multimodal since several features were used simultaneously. The use of features in ranked by duration served to (1) animate the speech, (2) give the puppet facial expressions, (3) animating the body, (4) use special effects, and (5) perform more complex “physical” actions such as moving around. The puppeteers used mostly the mouse and glove.

Beyond the role of the different features of the multimodal system, we showed how puppeteer and spect-actors accomplish engagement in the sessions. In particular, spect-actors have a key part in creating the narrative – the gags are often inspired by what the spectators say. While it is situated and emergent, we described how engagement is the product of particular performative strategies and skills and how it relies on collective accomplishments of the mediated puppet (puppeteer) and the spect-actors.

We inferred as implications for design the opportunity to automatically animate the lips and synchronize them with the speech of the puppeteer. This might free the puppeteer to concentrate more on expressive or symbolic acts. In addition, the expressive dimension of the interface, e.g. clenching the fist making the puppet look angry, can be further investigated. Performative Structures for Brief Interactions

In this case, the spectators were mostly pupils and teenagers and sessions lasted more than two minutes, on average. It must be noted, however, that some sessions are interrupted by teachers and others are a continuation of a previous interaction. While improvised, sessions conform to a general structure, which is also reported in interviews with the puppeteers. These structures have been observed to emerge in the use of installations [20, 21]. The groups of spectators generally are attentive to the installation (they did not talk to each other), actively interact with it, and show positive and growing interest as they interact. The puppeteer, therefore, is working with different resources, including a repertoire of gags, to be able to keep spectators engaged for several minutes.

REFERENCES

1. Attardo, Linguistic Theories of Humor. Berlin: Mouton de Gruyter, 1994. 2. Barnes et al., Video Puppetry: A Performative Interface for Cutout Animation, SIGGRAPH Asia 2008, 124:1124:9. 3. Benford et al., The Frame of the Game: Blurring the Boundary between Fiction and Reality in Mobile Experiences, CHI 2006 4. Blumenthal, Puppetry and puppets, Thames & Hudson, 2005. 5. Boersma, Weenink, Praat http://www.fon.hum.uva.nl/praat/

Implications of the found structure of the interaction lie in the design of an extended computer animated puppet

146

software,

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

6. Bottoni, et al., CoPuppet: Collaborative Interaction, Randy Adams, Steve Gibson and Stefan Müller Arisona eds Virtual Puppetry, In Transdisciplinary Digital Art. Sound, Vision and the New Screen, pp. 326-341.

information: designing for engagement at a public multitouch display . In: ACM CHI '10: Proceedings of the 28th international conference on Human factors in computing systems, Pp: 2267-2276, 2010.

7. Camurri et al. Communicating Expressiveness and Affect in Multimodal Interactive Systems, IEEE MultiMedia, 12(1), 2005, pp. 43-53.

22. Kendon, Gesture: Visible Action as Utterance. Cambridge, UK: Cambridge University Press, 2004.

8. Cavazza et al., Multimodal Acting in Mixed Reality Interactive Storytelling, IEEE MultiMedia, 11(3), 2004, pp. 30-39.

24. Landis, Koch, The measurement of observer agreement for categorical data in Biometrics. Vol. 33, pp. 159-174, 1977.

9. Coutrix et al., Interactivity of an Affective Puppet, Adj. Proc. of Ubicomp 2009. 10. Dalsgaard, Koefoed Hansen, Performing Perception— Staging Aesthetics of Interaction, ACM TOCHI, 15(3), 2008.

25. Leech, Onwuegbuzie, Qualitative Data Analysis: A Compendium of Techniques and a Framework for Selection for School Psychology Research and Beyond. School Psychology Quarterly, 23(4), pp. 587 -604, 2008.

11. De Angeli, Brahnam, I hate you! Disinhibition with virtual partners. Interacting with computers 20, 2008, pp. 302-310.

26. Liikkanen et al., Exploring emotions and multimodality in digitally augmented puppeteering, Proc. AVI 2008, pp. 339-342.

12. Derry et al., http://visa.inrp.fr/visa/presentation/Seminaires/Journees _inaugurales/Video_Gdlines_JLS5_09Derry.pdf, 2009.

27. Liikkanen, L., Jacucci, G. & Helin, M. (2009) ElectroEmotion: A Tool for Producing Emotional Corpora Collaboratively. In: the Proc. of Affective Computing and Intelligent Interfaces ACII 2009. Amsterdam, The Netherlands IEEE.

23. Kipp, ANVIL software, http://www.anvil-software.de/

13. Dontcheva, Yngve, Popović, Layered acting for character animation, ACM TOG, 22(3), 2003.

28. Mazalek, Nitsche, Tangible interfaces for real-time 3D virtual environments, Proceedings of the international conference on Advances in computer entertainment technology, 2007.

14. Figueiredo et al., Emergent stories facilitated, In Spierling, U., and Szilas, N., editors, ICIDS, Springer LNCS vol. 5334, 2008, pp. 218–229. 15. Glaser, Strauss, The discovery of grounded theory. New York: Aldine, 1967.

29. McNeill, Gesture and thought. Chicago: University of Chicago Press, 2005.

16. Goldman-Segall, Points of viewing children’s thinking: A digital ethnographer’s journey. Mahwah, NJ: Erlbaum, 1998.

30. Mehrabian, 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology: Developmental, Learning, Personality, Social, 14, pp. 261-292.

17. Jacucci, Interaction as Performance. Cases of configuring physical interfaces in mixed media. Doctoral Thesis, University of Oulu, Acta Universitatis Ouluensis, 2004.

31. Reeves et al., Designing the spectator experience, Proc. of CHI 2005.

18. Jacucci, G. and Wagner, I., Performative Uses of Space in Mixed Media Environments, In Davenport, E., Turner P., Spaces, Spatiality and Technologies, Springer, London, 2005.

32. Shin et al., Computer Puppetry: An Importance-Based Approach, ACM Transactions on Graphics, 20(2), 2001, pp. 67-94.

19. Jacucci et al., A Manifesto for the Performative Development of Ubiquitous Media, Proc. 4th Decennial Conference on Critical Computing, pp. 19-28, 2005.

33. Shugrina et al., Empathic painting: interactive stylization through observed emotional state, Proc. NPAR 2006, ACM Press, 2006.

20. Jacucci et al., Bodily Explorations in Space: Social Experience of a Multimodal Art Installation, Proc. INTERACT 2009, Springer, 2009, pp. 62–75.

34. Urquhart, An Encounter with Grounded Theory: Tackling the Practical and Philosophical Issues. In Trauth, E. (Ed.) Qualitative Research in Information Systems: Issues and Trends. Idea Group Publishing, London, 2001.

21. Jacucci, G., Morrison, A., Richard, G.T., Kleimola, J., Peltonen, P., Parisi, L., Laitinen, T., Worlds of

147

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Comparing User Interaction with Low and High Fidelity Prototypes of Tabletop Surfaces Jan Derboven†, Dries De Roeck†, Mathijs Verstraete†, David Geerts†, Jan Schneider-Barnes‡, Kris Luyten‡ † ‡ Centre for User Experience Research (CUO) Hasselt University – tUL – IBBT IBBT/K.U.Leuven Expertise Centre for Digital Media Parkstraat 45, 3000 Leuven, Belgium Wetenschapspark 2, 3590 Diepenbeek, Belgium {firstname.lastname}@soc.kuleuven.be {firstname.lastname}@uhasselt.be materials” that are different from the final product, such as paper, cardboard, etc. [20, 21]. High-fidelity, functional interactive prototypes are built at a later stage of the usercentred design process, and typically do use materials that you would expect to be in the final product, producing a prototype that looks and behaves much more like the final product [21]. In multi-touch technology and interaction, however, directness and visual interactivity is at the core of the experience and the interaction [16]. Because of that, it becomes hard to apply Wizard of Oz-like techniques to multi-touch directly, as problems with delay or obscured parts of the interface make it difficult for the „wizard‟ to quickly and accurately respond to user interactions. Some existing multi-touch developments deal with this by means of a „design by doing‟ approach, where ideas are implemented straight away and tested afterwards, without having a profound notion of the usefulness or chance to succeed at the start of the development [8]. From a developer‟s point of view, this is an understandable approach since the technical complexity of multi-touch hardware is relatively low. The downside of this is that implementing systems straight away is error-prone, and thus costs more time and money than creating low-fidelity prototypes. The required effort to update a system after it has been developed is significantly higher compared to creating several, rapid iterations of low-fidelity prototypes [23]. Furthermore, when implementation is started immediately, HCI practitioners coming from a nontechnical background are experiencing difficulties to fully participate in the process of creating prototypes [20], and exploring the potential of the new interaction paradigms.

ABSTRACT

This paper describes a comparative study between the usage of low-fidelity and a high-fidelity prototyping for the creation of multi-user multi-touch interfaces. The multitouch interface presented in this paper allows users to collaboratively search for existing multimedia content, create new compositions with this content, and finally integrate it in a layout for presenting it. The study we conducted consists of a series of parallel user tests using both low-fidelity and high-fidelity prototypes to inform the design of the multi-touch interface. Based on a comparison of the two test sessions, we found that one should be cautious in generalising high-level user interactions from a low towards a high-fidelity prototype. However, the lowfidelity prototype approach presented proved to be very valuable to generate design ideas concerning both high and low-level user interactions on a multi-touch tabletop. Author Keywords

Tabletop, multi-touch, creation, design.

prototyping,

fidelity,

content

ACM Classification Keywords

H5.2 User Interfaces: Input devices and strategies INTRODUCTION

In the field of HCI, methods for prototyping computer interfaces have quite a history [3, 7, 10, 20, 23]. For testing most graphical user interfaces, simulating interactivity can be done using low-fidelity prototyping methods: these are simple visualisation techniques of design ideas at very early stages of the design process, whose development does not need much time, such as paper prototyping. Low-fidelity prototyping methods typically involve “low-fidelity

To overcome these limitations, and stimulate the use of low-fidelity prototypes in the design of multi-touch tabletop interaction, we compare a low-fidelity and a high-fidelity prototype for the design of a multi-touch user interface.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

The paper is structured as follows. First, we will survey research in the area of prototyping, mostly focusing on multi-touch surfaces. We will then present the MuTable platform, which is used as a case for comparing low-fidelity and high-fidelity prototyping. The two test set-ups will be described in detail, before presenting the results which focus on the differences and similarities on different aspects

148

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

of the prototypes. We will conclude with findings about the value of multi-touch prototypes in various phases of development.

in a presentation. The scenario integrates searching and browsing for content with the creation of new content. A lot of work has already been done in the field of collaborative multi-touch applications, both on the side of detailed interaction design (for instance, gesture research as described in [12, 17, 25, 26]), as on a higher-level plane of application design and user collaboration [18]. Applications range from (collaborative) picture browsing and sharing [2, 5, 13], to gaming [1, 8, 15], storytelling [9, 22] and even specialized applications for hotels, restaurants and healthcare (e.g. the Sheraton Hotel, Resto‟Touch, and Patient Consultation showcased in [11]). However, significantly less attention has been paid to productivity tools and specifically content creation on multi-touch surfaces. The specific focus of the user tests described in this paper is on researching how people behave and interact with each other and the multi-touch work surface at a high level, while they are working on a creative assignment.

RELATED WORK

Low-fidelity prototypes can be used for different purposes. In a HCI context, paper prototypes are mostly used as tools for the evaluation of design/usability failures or successes [6]. The method is used to validate a prototype of which the findings are applied in a final product or application [1, 15]. Another approach to paper prototyping is proposed by Lim, Stolterman and Tenenberg [7], i.e. the use of a low-fidelity prototype as an explorative way to look at design ideas and as a way to think about design alternatives. Scott, Carpendale and Inkpen [19] used paper prototyping to categorize the type of interactions people have when they work collaboratively at a physical table. Findings of their study concern high-level interactions starting from a paper prototype, and do not focus on the relation with an actual multi-touch application. Research by Al Mahmud et al. [1] shows that interactivity and visualization is the core of a multi-touch experience, which is why the physical design of a low-fidelity prototype for multi-touch is something to be approached with caution. Therefore perhaps, few efforts have been made to experiment elaborately with „lowfidelity‟, physical interaction in a multi-touch development context. Examples are [14, 19, 24]. Both Piper [14] and Terrenghi [24] use a low-fidelity multi-touch oriented prototype in a focussed context, where the functionality of the tested application is relatively narrow. Furthermore, it is not always clear whether low-fidelity prototype studies allow for generalization to high-fidelity multi-touch products. Terrenghi [24] presents a study investigating this. In her study, a comparative analysis was made between a high and low-fidelity version of single-user puzzles and photo sorting tasks, focusing mainly on interface metaphors and lower-level interaction (such as one-handed vs. bimanual interaction).

The MuTable was conceived to be a multi-touch surface that is positioned horizontally, or at a small angle, convenient for people to work at while standing up. The surface was specifically intended for co-located collaboration when standing at one particular side of it, so all users face the screen in the same direction, unlike some other multi-touch hard- and software designed for access from all four sides of the surface (e.g. Microsoft Surface). This consideration also played its part in UI design, as the software was designed with users in mind standing in one specific orientation at the table. This setup makes the UI design of the MuTable software very suitable for touch tables that are placed against a wall, or that are wallmounted. When the table is positioned at a small angle in one direction, the table position will also be ergonomically better to work at, compared to a completely horizontal surface. Moreover, the MuTable design incorporates a fullscreen presentation mode in which slides created on the table can be presented. This functionality is aimed at wall projection, or wall-mounted tables that can be tilted from a small angle (in „working mode‟) to a vertical position, to serve as a presentation surface („presentation mode‟).

The approach taken in this paper focuses on higher-level interactions, such as interpersonal interaction between users. Specifically, the similarities and dissimilarities between low-fidelity prototypes and their high-fidelity counterparts in collaborative content-creation tasks are investigated.

METHOD

In our study, we performed two series of user tests: one focusing on low-fidelity prototyping and another on highfidelity prototyping. The two observation series, which are described in more detail below, were very similar in set-up. Different groups, equal in size and of the same age, worked on the same assignment. The main difference between the two tests was the material the participants worked with: the „low-fidelity‟ groups created a presentation with physical everyday objects (paper, scissors, etc.), while the „highfidelity‟ groups created a similar presentation using only digital content available on a multi-touch table.

THE MUTABLE PLATFORM

Our tabletop surface setup, conveniently called MuTable in the remainder of this paper, was used in a variety of public spaces, including museums, public events, libraries, and schools. While a lot of multi-touch applications typically focus on viewing video, browsing photographs and playing music, MuTable focuses on productivity tools for collaborative content creation. The usage scenario central in this paper is one where adolescents are asked to collaboratively use a tabletop interface for school-related work. Students are able to search for content (text documents, pictures, movies…), and integrate this content

When testing low-fidelity prototypes, and comparing them to their high-fidelity counterparts, it is important to choose the right prototyping focus. Arnowitz et al. [3] identify

149

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

several „prototype fidelity levels‟: a prototype does not have „just one characteristic of fidelity‟, but it has several fidelity levels. Arnowitz et al. distinguish between levels such as information design, interaction design, navigation model, visual design, etc. However, the focus aimed for in the MuTable low-fidelity prototype doesn‟t fit neatly into one of the levels described by Arnowitz et al. The aim is to look at a general level of interpersonal interaction and collaboration, and general interaction with the work surface and the material on it. In the test setup described below, the similarity in participants, group size, assignment, and tools should allow for a high-level comparison between the two series of user tests. The test setup does not, however, allow for detailed comparisons of specific interaction techniques, as the low-level interactions of handling scissors and picking up or folding paper in the low-fidelity prototype are not comparable to handling digital information in the highfidelity prototype. As also found in related work, such lowlevel comparisons would require different prototype setups, in which it can be more difficult to draw conclusions about high-level interaction.

mimics a user gradually exploring a system during first time use, opening and closing functionality.

Low-Fidelity Tests

The participants were asked to work at the prototype of the interactive table, standing up (see figure 2). Their task was to create a presentation in 45 minutes on the life and works of Leonardo da Vinci. This topic allowed the participants to create a presentation about the aspects of Leonardo they thought most interesting (arts or science), and it resembles a school assignment, as stipulated in the MuTable use scenario.

Figure 1: Low-fidelity test setup Test Design

For the test with the low-fidelity MuTable prototype, 16 adolescents aged 16-17 were recruited (7 female, 9 male participants). They participated in small groups (2 groups of 3, 5 groups of 2), in which all members of the groups already knew each other before the test: they were friends, classmates, or family.

Prototype Design

The low-fidelity prototype of the interactive table involved a table at which test users were asked to collaborate using only real-world, physical materials. The long side of the table (120x80x92 cm - LxWxH) was placed against the wall, in order to match the requirements of the envisioned touch table. At the start of the observations, only 3 sheets of A3 paper were on the table: these sheets were to be used to create a presentation on a given subject. In order to allow for a comparison between the low-fidelity and the high-fidelity prototypes, the former prototype‟s fidelity was increased by imposing a small number of restrictions. The materials to be used for the creation of the presentation were available in a separate material repository, which was hidden behind a curtain. The participants received a list of materials available for use in the presentation – they had to ask the test facilitators to bring them the material they needed. The repository included documentation materials, a typewriter to type text, a „recycle bin‟, and materials to paste (glue, tape, etc.), to cut (cutter knife, scissors, etc.), and to draw. In order to simulate screen estate usage, every material group (e.g. a set of scissors and knives) was in a separate container that had to be placed on the table. Participants were not allowed to stack the containers, or put them on the floor: all materials had to be put on the table, or given back to the observers.

Figure 2: Low-fidelity table usage

For analysis purposes, the test sessions were recorded using two video cameras. One of the test observers gave the assignment to the participants, in which the participants were asked to create a presentation for a travelling exhibition on Leonardo da Vinci. When finished, the participants presented their work and talked about the contents of their presentation. Before starting the test, participants were given a pre-test questionnaire asking about their background; after the test, they were given a post-test questionnaire asking how the assignment went. At the end of the test session, the participants received an incentive.

Since the materials were not immediately available, participants could not really anticipate what to expect when asking for specific work material (Figure 1). Test participants could ask the test observers at any time to bring them work materials from the repository, or to return these materials to the repository. In this way, the prototype

150

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

High-Fidelity Tests

members of the groups already knew each other before the test: they were friends, classmates, or family.

Prototype Design

The high-fidelity prototype offers similar functionality for creating presentations as the low-fidelity prototype discussed in the previous section. In general, the highfidelity MuTable platform offers a number of separate, small applications such as a typewriter, a presentation creation tool, a file browser, and a drawing application (see figure 3). The functionality is available on a surface of 110 cm x 55 cm, with a screen resolution of 1280x800 pixels. This allows two to four persons to collaborate at the MuTable in a comfortable way, depending on the specific tasks to be completed.

One of the test observers gave the assignment to the participants, which was the same assignment used in the low-fidelity tests: creating a presentation for a travelling exhibition on Leonardo da Vinci in 45 minutes. The test itself consisted of some ten minutes in which the participants were allowed to explore the table themselves. After some ten minutes, most participants arrived at a point where they needed some extra information and guidance on specific gestures and the functionality of the platform. To provide the test participants with extra information after their initial exploration, the complete MuTable functionality was explained to the participants in a short demonstration. Afterwards, they were allowed to continue working on their presentation for approx. 30 minutes. For analysis purposes, the test sessions were recorded using three video cameras. After completing the test assignment, participants presented the work they had created. Before starting the test, participants were given a pre-test questionnaire asking about their background. After the test, they were given a post-test questionnaire asking how the assignment went. At the end of the test session, the participants received an incentive.

All applications can be accessed through a central, circular menu, which can be personal (one menu per user) or common to several users: no restrictions have been imposed regarding „ownership‟ of content. The applications can be used together to create and search for content, and (re-)use that content in a presentation. The applications especially facilitate research on a specific subject, and creating new content (text, a presentation) on that subject. The highfidelity MuTable applications are the digital counterparts of the material used in the low-fidelity setup. Test participants in the high-fidelity tests, like the low-fidelity participants, could also do research and compile a presentation by reading documents, typing texts, using pictures, dropping unwanted material in a recycle bin, etc. In this way, both prototypes were very similar in their high-level functionality, regardless of lower-level, specific differences (such as the use of specific interaction methods, like gestures). 3.

6.

Comparing Low-Fidelity and High-Fidelity Tests

The observations and user feedback from both the lowfidelity and the high-fidelity test sessions were processed separately. In a first round, the test sessions were discussed and the common, recurrent themes/observations were logged. Based on this limited set of themes, the test sessions were reviewed a second time, fitting all relevant observations into the predefined themes. Afterwards, the observations were compared to each other, in order to find parallels in user behaviour between the two test settings.

1.

RESULTS

2.

5.

In this section, the results of the test sessions are divided into a number of key areas regarding high-level user interaction. The test results of both test sessions will first be discussed separately. After the separate discussion, an overall evaluation and comparison of the high and lowfidelity test results will be made.

4.

Table Layout

Figure 3: Main MuTable functionality, including 1. a central circular menu, 2. a Create submenu, 3. a typewriter, 4. a presentation creation tool (with a slide opened), 5. a file browser, 6. a piece of content (picture) opened.

Low-Fidelity Table Layout

Low-fidelity test participants tended to divide the table into several zones, similar to those described by Scott et al. [19] – for convenience sake, we will use the same terminology. One type of zone observed during the low-fidelity tests was the personal zone. This zone, typically the zone closest to a specific participant, was used only by that individual participant, and was primarily used for presentation creation, and to store ready-to-use documentation materials. The second type of zone was the group zone: this zone was used for common storage of the tools and unused documentation materials. Although some test groups clearly

Test Design

For the tests with the high-fidelity MuTable prototype, a test design similar to the low-fidelity test design was used. 16 adolescents aged 16-18 were recruited (9 female, 7 male participants – all were different participants from the ones that used the low-fidelity prototype). They participated in small groups (2 groups of 3, 5 groups of 2), in which all

151

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

were messier in their work than others, this division provided a structure for the test participants in which they knew where to look for a specific tool or content piece, even if one of the other participants used it last. E.g. the pair of scissors was always returned to the same storage/group zone of the table, regardless of which participant used it.

themselves, similar to the low-fidelity tests, in which participants also were free to divide the table according to their wishes. All test groups more or less naturally took a position between the two extreme approaches to table layout: a more fixed, clean layout emerged naturally from a task division between the test participants. Without a clear task division, the table layout tended to be messier.

The personal work zone consisted of the presentation being created, and numerous pages and scraps of content lying around, ready to be used in the presentation. The fact that the work zone consisted of current work materials and other content ready to be used, is consistent with the observations of Scott et al. [19] about storage zones, which states that „the location of a storage territory appeared to influence who utilized the resources contained within it‟. Shared items were indeed kept in the group storage zone, while „stored materials‟ ready to be used by one participant were kept close to that participant (see figure 4).

-

-

However, while Scott et al. only mention the use of piles to store content [19], the MuTable low-fidelity observations adds content layering to this framework. Instead of creating neat piles of content, participants tended to work on top of one or several layers of ready-to-use materials, randomly scattered on the table. In this way, most test groups effectively had more than one „layer‟ in their personal zones. The upper layer consisted of the presentation pages, and some tools they were actively using; the lower layer (often invisible, covered with the upper layer) consisted of content scraps lying around, waiting to be used. The participants lifted the upper layer in order to browse the underlying layer of content scraps.

Messy table layout. A messy table layout was in most cases the result of unclear task divisions between the test participants. Test groups that tended towards this side of the continuum mostly got annoyed with this arrangement after a while, commenting on it: „Shouldn’t we close some stuff first?‟, and „Maybe we should give the Recycle Bin a permanent place somewhere‟. Fixed table layout. This table layout mostly emerged from a more or less clear task division, e.g. one test participant typing text and creating slides, and another test participant searching for content such as images and documents. Four out of seven test groups evolved towards a more or less stable table arrangement in this way. For instance, one test group placed the keyboard in the lower left corner of the table, the presentation creation tool in the lower right corner, and the file browser in the upper right corner. The division leftright often demarcated the individual users‟ work spaces, while users kept the functionality they were working on most actively closest to them, and other functionality further away (division lower-upper half of the table – see also the section on interaction among test participants). One test group, however, also used a more or less stable table arrangement: they placed the typewriter centrally in the lower half, and the presentation creation tool centrally in the upper half. In this way, the table layout didn‟t really support simultaneous work: instead, they collaborated on the same tasks, the two test participants alternately working and observing.

Table Layout: Comparison

The table arrangements made by test participants during the low-fidelity and the high-fidelity tests were often very similar, in terms of global arrangement. Nearly all test groups divided the table in smaller vertical zones right in front of them, which were the users‟ personal zones. Apart from that, users also divided the space into horizontal zones: one personal work zone with material they were actively working on, and one common storage zone with material not used as actively, but still ready to be used.

Figure 4: Table layout emerging in most low-fidelity tests (three participants). The high-fidelity layout was often less clear-cut. High-Fidelity Table Layout

The MuTable high-fidelity prototype didn‟t impose any fixed table „layout‟. As in the low-fidelity prototype, applications as well as content could be placed anywhere on the table, and they could be resized (and, for content, also rotated) to any size. In the MuTable software, there are no defaults for window or content positioning, such as in graphical user interface (GUI) operating systems (e.g. Minimize and Maximize buttons in MS Windows, Split and Arrange All options in individual applications). Space and screen estate was to be divided by the test participants

Clear table layouts were somewhat more obvious in the low-fidelity tests, while layout in the high-fidelity prototype tended to be somewhat more „fluent‟. It is striking that in the high-fidelity prototype, a clear task division eventually lead to a more fixed layout, and an unclear task division lead to a messier table layout. Most test groups started out exploring with a messier layout – some groups later evolved into a more structured one. In the low-fidelity

152

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

prototype, this effect of task division was not that clear: the same table layout emerged, whether the participants had a clear task division or not. This leads to the conclusion that working with tangible materials that can be picked up and „layered‟, a suitable table layout emerges more fluently. It seems that the 2D, flat surface of the multi-touch table is somewhat limiting in this respect, an issue also explored by Terrenghi et al. [24]. Test groups that tried to work in an organic way, but weren‟t able to layer the material in front of them, often ended up with an unstructured mess – the more successful groups were „forced‟ into a more structured layout due to the two dimensionality of the content on the table. To make up for this, more explicit guidance on how to effectively divide the table into zones would be a good user support addition.

-

Turn-taking. Turn-taking collaboration was observed most often, interwoven by episodes of collaboration. Users didn‟t work individually, but took turns in actually interacting with the table, e.g. one user was just watching the other one interact with the table, or one user was reading text from a content PDF, while the other user was actually typing the text. These shifts between participants occurred mostly guided by negotiating, and comments such as „Let me try!‟ or „Can I type some text now?‟

The prevalent turn-taking behaviour obviously has an important effect on the results shown in graph 1. For instance, there was a 3-user test group in which the users didn‟t really take turns, but were actively working most of the time, dividing the table into 3 work zones (one zone per test user). With this in mind, it is understandable that two users of this group rate the table as „a bit too small‟, and only one user as „big enough, no room to spare‟.

Interaction among Test Participants Low-Fidelity Participant Interaction

When observing participant interaction, two general kinds of interaction could be distinguished, similar to the observations described in Gutwin et al. [4]. Test participants switched fluently between working individually on separate tasks and working together (e.g. working together creating content, in which one participant reads a piece of text out loud and the other one types out relevant items on the typewriter). Most of the time, participants worked individually. Even when they were all working on similar tasks, such as placing content on the presentation pages, participants tended to work individually (e.g. by each working on a different slide), merging their results afterwards. This phenomenon is described by Gutwin et al. [4] as „coupling‟: working individually in a group before „requir[ing] discussion, instruction, action, information, or consultation with another person‟.

The opposite observation is also true. Three groups out of seven mostly took turns in actively using the table; all users in these groups but one reported that the table was „big enough, with some space left‟, even though one of these groups had the maximum test group composition of three users.

High-Fidelity Participant Interaction

The interaction among the high-fidelity test participants generally went smoothly, apart from some minor incidents, such as users getting in each other‟s way while working on separate tasks (these incidents will be discussed in the section on territoriality).

Graph 1: Table size ratings in the post-test questionnaire. Participant Interaction: Comparison

Although overall, the participants in both sessions showed similar behaviour of alternating individual work with working together, an important distinction in the individual work was observed. While in the low-fidelity setting, people most of the time worked individually, in parallel, the high-fidelity setting showed people taking turns in working. In most groups, there was an active role („worker‟) and a passive role („onlooker‟), with participants constantly switching between the two roles. Since participants constantly had to work on the same surface, they tended to let one participant work at a task alone before doing work themselves. Only after some moments of on-looking, other participants started working themselves. „Tangible‟ work with 3D objects, as in the low-fidelity prototype, allowed more for individual work, without anyone looking. 2D interaction, on the other hand, often created situations in

In general, the same two kinds of interaction as described in the low-fidelity setting were observed: alternately working individually on separate tasks and working together. However, a distinction was seen between parallel, individual work, and turn-taking. -

Parallel, individual work. A minority of test users worked individually during the entire assignment, with participants working constantly on parallel, separate tasks. This parallel collaboration implies that all test users had a personal zone on the multi-touch table, in which they worked separately. When one participant found some content that could be used by another participant, it was exchanged in a fluent, natural way. Like in the low-fidelity tests, individual work was interrupted by episodes of merging results.

153

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

using too much space, use your own half!‟, etc.1 (see figure 5).

which participants tended to let another participant work, without „interfering‟ by starting to interact themselves. In this view, reducing interference would only be possible by increasing table size quite drastically. This behaviour impacted the way the table size was judged. With participants taking turns in working, the high-fidelity table was rated as „big enough, with some space left‟ by most participants. In contrast, most low-fidelity participants – working in parallel most of the time – rated their table as „big enough, no room to spare‟ (see graph 1). Territoriality Low-Fidelity Territoriality

In the low-fidelity test, users collaborated quite naturally. Participants exchanged material, passed material on to other participants, and often reached across the table to find the right tools to work with. Participants didn‟t really get into each other‟s way while doing this. These findings are similar to the results described in [19], where participants partitioned their interactions „with little to no verbal negotiation‟.

Figure 5: A high-fidelity test participant grabbing his head in frustration, when the other participant covers his work space.

This „territoriality‟ behaviour was, however, not related to the use of one common central, circular menu (see figure 3), or the use of individual personal menus per user. Both groups using only one common menu, and groups using more (individual) menus made remarks related to their „own‟ side of the multi-touch table.

Furthermore, users frequently exchanged tools and content material among each other. Participants passed on tools and showed photos and documents to each other. Larger tools, such as a typewriter, were not exchanged (which we thought was most likely due to physical properties such as weight). Instead, users changed places behind the table to type text on the typewriter. In general, users collaborated quite fluently, reached over to pick up the material they needed, and switched positions, e.g. to use the typewriter.

The territoriality issue is further complicated by users physically switching positions at the table. Often, when one user took over a task (such as typing) from another user, they did not move the applications on the table. Instead, they switched positions at the table, passing behind each other to reach the other part of the table in which, for instance, the typewriter was located. On the other hand, content such as pictures and books were resized and passed on, moving the digital content around instead of physically switching places.

High-Fidelity Territoriality

As already hinted at in the previous section, test participants often took turns working. Consequently, when only one user is working at the table, the user encounters no issues regarding the demarcation of his or her work territory. However, most groups worked simultaneously on more or less separate things at some point in the presentation creation (e.g. typing and browsing). This also means that the individual participants in these groups had their „own‟ functionality or content they were working on at any given moment. However, since there was no strict table division imposed, individual users‟ work sometimes got in the way of other participants‟ work. This was especially the case when resizing content and functionality on the table – not so much when moving functionality or content around the screen. In the last case, most participants appeared to „respect‟ each other‟s territory. When resizing content or functionality to get in the way of other users, remarks about it were made immediately: „Hey, this is MY part of the table!‟, „Keep your things in your own part, man!‟, „You‟re

In terms of effort, it is easier to move the functionality (which takes only one gesture), rather than to walk to the other side of the table. The position-switching behaviour suggests that territoriality doesn‟t mean „the side of the table with my content/functionality on it‟, as much as „the side of the table I am currently working on‟. While working on the collaborative assignment, users switched roles, and switched between working on different kinds of content – after switching, the „new‟ content or functionality users were working on quickly became their content or functionality. Furthermore, the „static‟ functionality and the „moveable‟ content suggest that table layout (see the section on table layout) has an important influence on territoriality, as well. Many participants seemed to consider the functionality on the table as part of the more static table layout, while content such as pictures etc. were not 1

Similar behavior has also been reported in [13]. In that experiment, however, users often didn‟t know other users interacting at the same multi-touch surface. Therefore, their reactions and comments often were more polite or covert.

154

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

considered to be part of that static layout. Therefore, functionality was not moved and passed on to other users.

were used only rarely were not returned, but stayed on the table, ready for use. Participants preferred to have all material on a rather crowded table, rather than use a cleaner, more organised table and go through the trouble of returning the materials and asking them back later.

Territoriality: Comparison

Although it was first thought that in the low-fidelity prototype, the typewriter was not moved around due to its weight, a similar observation was made in the high-fidelity tests. Although test users did exchange content (photos, articles,…), tools such as the typewriter tool or the content browser most often were left in its position, with the test users exchanging places.

Unused High-Fidelity Functionality

Like in the low-fidelity tests, test participants didn‟t use the recycle bin very often in the high-fidelity tests either. Instead, they bended the „rules‟ implicitly imposed by the platform, and used other, creative methods to dispose of unused materials. Participants resized the content until it was very small (and nearly invisible), or simply moved the unwanted content off-screen.

In addition, low-fidelity and high-fidelity prototypes results were quite different regarding territoriality conflicts: while in the low-fidelity setting, participants collaborated without any problems, the high-fidelity setting created a lot more territoriality issues. When comparing these territoriality issues to table size ratings (see graph 1), it becomes apparent that the type of prototype has an important effect on territoriality. While one would expect more problems to occur when participants judge the work surface as too small (i.e., in the low-fidelity prototype), the table size ratings given by participants show the opposite. In the low-fidelity setting, with a table that was rated as quite limited in work space, no territoriality issues ensued. In the high-fidelity setting, with a table that appeared large enough to most participants, several territoriality issues occurred. As in the participant interaction and table layout sections, it seems that the 2D high-fidelity vs. the 3D low-fidelity work space is an important factor here. Participants get in each other‟s way more easily on a 2D surface than in a 3D space.

As already mentioned in the table layout section, some groups tended to leave all material „lying around‟ on the table, like the low-fidelity groups. However, most of them soon got frustrated with the mess of content displayed on the table. Unused functionality: Comparison

Low-fidelity and high-fidelity tests were very similar in terms of functionality that was left unused by the participants: few test groups used the recycle bin. This suggests that low-fidelity prototypes potentially are adequate predictors of the perceived usefulness of functionality. The prototypes can predict which functionality users think of as useful, and which parts they think are redundant. However, it can be inferred that this predictive power will only hold when the interaction fidelity of the low-fidelity prototype is quite high on a more detailed level, as well. In the case of the recycle bin, lower interaction fidelity probably would have made the bin more useful. It can be assumed that the low-fidelity bin would have been used more often if users were allowed to place it underneath the table, or next to it, like a normal, real-world bin. Increasing prototype fidelity by only allowing the bin to be placed on the table made the bin less useful: it took up too much place on the table, compared to the actual benefit users had from having it on the table. Similarly, in the highfidelity prototype, the bin took up too much screen estate, given the limited amount of functionality it added.

To summarize, whereas the low-fidelity prototype allowed people to divide the available space in a natural way, as described also in the low-fidelity setting of [19], the highfidelity prototype tests in our study showed different results. Participants ran into territoriality conflicts and, depending on the nature of the items displayed (content vs. applications), they switched between passing on the digital content and physically switching places themselves to leave the table layout intact. This leads to the conclusion that specific issues, such as the 2D vs. 3D collaboration mentioned in the table layout comparison section, and a perceived difference between „static‟ functionality and „moveable‟ content (based on table layout), can have a profound influence on the way people experience territoriality in the real-world vs. in digital collaboration.

In sum, our findings suggest that, given the fact that interaction fidelity is high at a general level and at a lower, more detailed interaction level, low-fidelity multi-touch prototypes can be accurate predictors of the user‟s perceived usefulness of parts of the prototype.

Unused Functionality Unused Low-Fidelity Functionality

DISCUSSION

Although the low-fidelity tabletop prototype had a waste bin in which test participants could leave unused article scraps and other content material they did not use, it was not used as it took up too much table space. In the majority of user groups, unused scraps eventually made up the bottom layer in the work zone.

The results discussed above show that there are aspects to low-fidelity multi-touch prototyping that allow for prototyping, and aspects that resist prototyping. The way participants passed on content, but left functionality in place in the high-fidelity setting was similar to the lowfidelity setting, not because of the intrinsic properties of the functionality itself (as in the low-fidelity setting: the weight of the typewriter, etc), but because of the participant‟s

Participants tended to leave all material (both tools and content material) lying around on the table. Even tools that

155

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

implicit agreement on table layout. Similarly, the recycle bin was not used in both test series due to a comparable perception concerning its usefulness and the valuable screen estate it demanded.

interactions due to its physical presence on a table. This three-dimensional interaction allows users to easily „layer‟ paper documents, and retrieve them by picking up the paper. On a 2D surface, on the other hand, one is limited to overlapping content. Intrinsically linked to the above, the low-fidelity prototyping method presented in this paper is an excellent method for idea generation. In this way, the prototype has the function of a „filter‟, as proposed by Lim et al. [7]: it makes it possible „to examine an idea‟s qualities without building a copy of the final design‟. In the case presented in this paper, the low-fidelity prototype presented ideas about how to shape the interaction. The low-fidelity prototype inspired new concepts such as content layering and having all material immediately available, without „closing‟ functionality or using a recycle bin. This type of design ideas did not emerge during high-fidelity testing. LIMITATIONS AND FUTURE WORK

The study described above highlights some aspects of high and low-fidelity prototyping that are quite universal in multi-touch applications (e.g. the 2D vs. 3D distinction) and have been described in existing literature (e.g. the territoriality issues in [19]). However, future studies will have to confirm if the specific observations described in this paper can be generalised to multi-touch applications in general.

Figure 6: Similarities and dissimilarities between low-fidelity and high-fidelity prototypes.

However, the possibility to interact in a 3D space vs. on a 2D surface influenced the participant‟s interactions profoundly. While it was expected that the 3D tangible objects vs. 2D touch surface would influence more lowlevel interactions, the results show that higher-level interactions with the work surface and between participants are also influenced. This influence on higher-level interactions can be seen as a side effect of more obvious, low-level interactions, also affecting the usage of the prototype as a whole [7]. An example of this is the highlevel table layout being influenced by the (low-level interaction) possibility to stack items, and create „layers‟ of content. Although the influence of 2D vs. 3D on higherlevel interactions is often a more indirect influence, its effects cannot be ignored.

In addition, low-fidelity prototyping has some often-used, adequate methods such as paper prototyping for GUIs, using paper and post-its to create drop-down menus and other controls. For multi-touch tabletops, however, such often-used methods have not yet been established. It would be interesting to see in future work which innovative lowfidelity methods can be developed to make low-fidelity observations more predictive of high-fidelity results. CONCLUSION

We have reported on a study in which we compare highlevel interactions on a physical and digital prototype of a tabletop application. Although the low-fidelity prototype was designed to have a focus on global, high-level interaction, and not on detailed manipulation of objects, test results showed that differences in more detailed interaction and manipulation of objects also have a profound influence on higher-level interactions. In order to create testable lowfidelity multi-touch prototypes, we found that only very high-level user interactions can be generalised towards a high-fidelity prototype. In addition, the low-fidelity prototype approach presented proved to be very valuable to generate design ideas concerning both high and low-level user interactions.

We can conclude that when using low-fidelity, tangible prototyping as an evaluation method of high-level interactions, one should be cautious in interpreting the results. Although several low-fidelity observations were similar in the high-fidelity situation, the difference between 2D vs. 3D interaction also influences interaction with the surface and other users at a high level. Due to this influence, similarities between low-fidelity and highfidelity prototypes stop, and observations start to diverge. It is, however, difficult and unpredictable to pinpoint the exact point at which the observations diverge. As shown in figure 6, similarities between the low and high fidelity tests always lie on a high level of interaction. The dissimilarities found can mostly be related to the prototyping medium used and the specific constraints linked to that medium. For example, paper is a medium that enables three-dimensional

ACKNOWLEDGEMENTS

MuTable is a Belgian project funded by IBBT. Project partners in the project consortium are CUO (KULeuven), EDM (UHasselt), SMIT (VUB), Kunstencentrum Vooruit, Artec, STAM, Ravi, and University Library of Ghent.

156

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Mine, Don‟t Touch!: interactions at a large multi-touch display in a city centre. In Proc. CHI 2008, ACM Press (2008), 1285-1294.

REFERENCES

1. Al Mahmud, A., Mubin, O., Shahid, S., and Martens, J. Designing and evaluating the tabletop game experience for senior citizens. In Proc. NordiCHI 2008, ACM Press (2008), 403-406. 2. Apted, T., Kay, J., and Quigley, A. Tabletop sharing of digital photographs for the elderly. In Proc. CHI 2006, ACM Press (2006), 781-790.

14. Piper A.M., Hollan J.D., Tabletop Displays for Small Group Study: Affordances of Paper and Digital Materials. In Proc. CHI 2009, ACM Press (2009), 12271236.

3. Arnowitz, J., Arent M., and Berger, N. Effective Prototyping for Software Makers. Morgan Kaufmann, San Francisco, CA, 2007.

15. Piper, A. M., O'Brien, E., Morris, M. R., and Winograd, T. SIDES: a cooperative tabletop computer game for social skills development. In Proc. CSCW '06, ACM Press (2006), 1-10.

4. Gutwin, C., and Greenberg, S. A Descriptive Framework of Workspace Awareness for Real-Time Groupware. Comput. Supported Coop. Work 11, 3 (Nov. 2002), 411-446.

16. Quigley, A., Subramanian, S, and Izadi, S. Special issue on interaction with coupled and public displays. Personal and Ubiquitous Computing, 13, 8, SpringerVerlag, November 2009.

5. Kristensson, P. O., Arnell, O., Björk, A., Dahlbäck, N., Pennerup, J., Prytz, E., Wikman, J., and Åström, N. InfoTouch: an explorative multi-touch visualization interface for tagged photo collections. In Proc. NordiCHI 2008. ACM Press (2008), 491-494.

17. Saffer, D. Designing Gestural Interfaces. O‟Reilly, Sebastopol, CA, 2009.

6. Lim, Y., Pangam, A., Periyasami, S., and Aneja, S. Comparative analysis of high- and low-fidelity prototypes for more valid usability evaluations of mobile devices. In Proc. NordiCHI 2006. ACM Press (2006), 291–300. 7. Lim, Y., Stolterman, E., Tenenberg, J. The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer-Human Interaction. 15, 2 (2008), 1-27. 8. Marco, J., Cerezo, E., Baldassarri, S., Mazzone, E., and Read, J. C. Bringing tabletop technologies to kindergarten children. In Proc. British Computer Society conference on HCI 2009. British Computer Society (2009), 103-111. 9. Mazalek, A., Winegarden, C., Al-Haddad, T., Robinson, S. J., and Wu, C. Architales: physical/digital co-design of an interactive story table. In Proc.TEI 2009, ACM Press (2009), 241-248. 10. McCurdy, M., Connors, C., Pyrzak, G., Kanefsky, B., and Vera, A. Breaking the fidelity barrier: an examination of our current characterization of prototypes and an example of a mixed-fidelity success. In Proc. CHI 2006, ACM Press (2006), 1233-1242. 11. Microsoft Surface. www.surface.com. 12. North, C., Dwyer, T., Lee, B., Fisher, D., Isenberg, P., Robertson, G., and Inkpen, K. Understanding Multitouch Manipulation for Surface Computing. In Proc. IFIP TC 13 Conf. on Human-Computer interaction: Part II. Lecture Notes In Computer Science, vol. 5727. Springer-Verlag, Berlin, Heidelberg, 236-249. 13. Peltonen, P., E. Kurvinen, A. Solovaara, G. Jacucci, T. Ilmonen, J. Evans, A. Oulasvirta, and P. Saarikko. It‟s

18. Scott, S.D., Grant, K.D., Mandryk, R.L. System Guidelines for Co-located, Collaborative Work on a Tabletop Display. In Proc. ECSCW 2003, Kluwer Academic Publishers (2003), 159-178. 19. Scott, S.D., Carpendale, M.S.T. & Inkpen, K.M. Territoriality in Collaborative Tabletop Workspaces. In Proc. CSCW’04, ACM Press (2004), 294-303. 20. Sefelin, R., Tscheligi, M., and Giller, V. Paper prototyping - what is it good for? A comparison of paper- and computer-based low-fidelity prototyping. In Proc. CHI 2003,Extended Abstracts on Human Factors in Computing Systems, ACM Press (2003), 778-779. 21. Sharp, H., Rogers, Y., Preece, J. Interaction Design. Beyond Human-Computer Interaction. (2nd. edition) Wiley, West Sussex, England, 2007. 22. Shen, C., Lesh, N. B., Vernier, F., Forlines, C., and Frost, J. Sharing and building digital group histories. In Proc. CSCW 2002, ACM Press (2002), 324-333. 23. Snyder, C. Paper Prototyping. The Fast and Easy Way to Design and Refine User Interfaces. Morgan Kaufmann, San Francisco, CA, 2003. 24. Terrenghi, L., Kirk, D., Sellen, A., and Izadi, S. Affordances for manipulation of physical versus digital media on interactive surfaces. In Proc. CHI 2007. ACM Press (2007), 1157-1166. 25. Wobbrock, J. O., Morris, M. R., and Wilson, A. D. User-defined gestures for surface computing. In Proc. CHI 2009, ACM Press (2009), 1083-1092. 26. Yee, W. Potential Limitations of Multi-touch Gesture Vocabulary: Differentiation, Adoption, Fatigue. In Proc. HCI International 2009. J. A. Jacko, (Ed.) Lecture Notes In Computer Science, vol. 5611. Springer-Verlag, Berlin, Heidelberg, 291-300.

157

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Drop-and-Drag: Easier Drag&Drop on Large Touchscreen Displays Sebastian Doeweling SAP Research Center Darmstadt Bleichstr. 8, 64283 Darmstadt, Germany [email protected]

Urs Glaubitt Michaelisstr. 16A, 64293 Darmstadt, Germany [email protected] [15]). In office settings, larger displays were found to generally improve performance and were preferred to averagesized displays by the majority of users [8, 6]. Furthermore, tasks that require spatial orientation were found to benefit from the use of large displays (e.g. [22]). This effect was found to be stronger when resolution is also increased [1].

ABSTRACT

Large displays have been found to offer a number of benefits over average-sized desktop displays: They increase productivity in office settings, improve performance on spatial tasks and offer increased user satisfaction in several contexts. However, their physical dimensions can complicate drag&drop interactions for users, especially when touch or pen input is used. Existing approaches (e.g. push-and-pop) have addressed this problem for simple drag&drop operations, but fall short when it comes to more complex ones (e.g. dropping a target onto a currently hidden node of a file tree or a specific location on a digital map).

Additionally, Hawkey et al. [12] report that the interaction setting defined by large touch-operated displays, i.e. users near to each other as well as near to the display, is most effective for collaborative tasks. This claim is backed by a number of field reports, e.g. [20] (office and research settings) or [13] (digital design studios).

To address this issue, we propose drop-and-drag, an interaction technique which introduces fully interactive proxy targets and which allows the interruption and resumption of drag&drop operations. The results of a controlled experiment show that drop-and-drag is significantly faster than traditional drag&drop for sufficiently distant targets. Additionally, the results report improved user satisfaction when drop-and-drag is used, especially for complex drop targets.

However, as reported by e.g. Robertson et al. [19], there are also challenges that accompany especially the use of large displays: Discontinuities in the displayed image caused by bezels for installations formed by multiple screens/projectors; configuration problems when screens are added to or removed from multiple monitor installations; cursor tracking problems; problems with window and task management; and problems related to accessing distal information. As display size increases, distal access gains particular importance: large touch- or pen-operated wall displays may well confront the user with the unattainable challenge of interacting with elements beyond his/her physical limits; in multi-user settings spatial interference, both physical[16] and virtual[23], may be an additional challenge.

Author Keywords

drop-and-drag, wall-size display, drag&drop, touch screen, interaction technique ACM Classification Keywords

H.5.2 Information Interfaces and Presentation: User Interfaces—Graphical user interfaces

Drag&drop is a very common interaction with desktop applications that suffers from these limitations. Especially for touch-operated devices, which require the user to maintain physical contact2 during the whole operation, drag&drop becomes more challenging: if contact is lost, an accidental drop is performed, canceling the operation or even leading to unintended system actions such as moving files to an incorrect folder. Consequently, drag&drop was found to be more error-prone with increasing distance on touch-screen displays [18].

INTRODUCTION

Over the last few years, large interactive display walls as well as larger tabletop systems have gained attention in both research communities and the general public (especially the presentation of Jeff Han [10] at the TED1 conference in 2006 or commercially available systems like Microsoft Surface 1

Technology, Entertainment, Design

Permission to to make make digital digital or or hard hardcopies copiesofofall allororpart partofofthis this work Permission work forfor personal or or classroom classroom use use is is granted granted without without fee fee provided provided that that copies copies are are personal not made made or or distributed distributed for not for profit profit or or commercial commercialadvantage advantageand andthat thatcopies copies bear this or bear this notice notice and and the thefull fullcitation citationon onthe thefirst firstpage. page.ToTocopy copyotherwise, otherwise, republish, to post on servers or toorredistribute to lists, requires prior prior specific or republish, to post on servers to redistribute to lists, requires permission and/or a and/or fee. a fee. specific permission CHI 2009, 2010, April 4October - 9, 2009, Boston, Massachusetts, USA. NordiCHI 16–20, 2010, Reykjavik, Iceland. Copyright 2009 2010 ACM 978-1-60558-246-7/09/04... ISBN: 978-1-60558-934-3...$5.00. $5.00.

158

While state-of-the art approaches such as drag-and-pop [3] or push-and-pop [7] already offer support for drag&drop interactions with simple targets, they fall short when taking more complex ones into account, i.e. drop targets that may require state modifications prior to the actual drop operation: 2 strictly speaking, for some camera-based systems very close proximity is sufficient

Full Papers

• Simplify drag&drop interactions with distant targets.

minimal hardware reqs.

minimal overhead

minimal interference

To address this problem, we designed drop-and-drag with three major goals in mind:

direct input techniques

Missile Mouse Tablecloth ScaleView Portals Push-and-throw Drag-and-pop Push-and-pop The Vacuum

complex targets

A common scenario involving complex drop targets is copying a file from one file tree to another using drag&drop. In order to complete the operation, one must frequently modify the state of the target file tree, i.e. collapse or expand branches, to access the desired drop location. This has to be done either prior to or during the actual drag&drop operation. While the former is time consuming for remote targets on large screens, the latter is often burdensome, requiring the user to hover above a tree node for a given time in order to expand it. Furthermore, there may even be components such as digital maps which do not offer the possibility to modify their state during a drag&drop operation, at all.

easier interaction

Proceedings: NordiCHI 2010, October 16–20, 2010

+ + + + + +

o + + o +

o + + + + +

n/r + + + + + -

+ + o + o

+ o o o o -

Table 1. Review of state of the art techniques for drag&drop on large displays, according to the six categories introduced at the beginning of the related work section (abbreviated for this table). A rating of ’+’ has been given where the technique fits the requirement very well; ’o’ indicates an average fit, e.g. there is no active support, but also no obstacle to the requirement; ’-’ is used, when the requirement is not met. ’n/r’ (not rated) has been used for Missile Mouse and its requirements on hardware because support for direct input is missing there.

• Support drop targets that require complex interactions prior to the actual drop operation. • Design for minimal interference in multi-user settings. The remainder of this paper is structured as follows: First, we present related approaches and state-of-the-art techniques for drag&drop on large displays. Then, we elaborate on the design of our approach, following a discussion of the setup and results of a controlled experiment, which we conducted to analyze the performance and user satisfaction of drop-and-drag. Finally, we argue about the shortcomings and limitations of our approach, present our conclusions and propose further enhancements to our interaction technique.

Techniques facilitating general access to remote objects

Missile Mouse, devised by Robertson et al. [19], is a technique which is targeted at very large displays - with, however, mouse input. Because its primary focus is accelerating the cursor with relative input devices, it does not lend itself very well to touch-operated devices. Tablecloth [19] is another approach by the same authors, which allows the user to scroll the entire screen content for access to distant areas. However, there are two major issues when this technique is used for drag&drop:

RELATED WORK

A number of techniques have been proposed to ease the problems, which accompany drag&drop on large displays. We separate these into techniques that are not geared towards drag&drop interactions specifically, but aim at allowing more efficient access to remote objects on large screens in general and those that focus on drag&drop. All techniques were reviewed according to the following criteria. Although some requirements may be more relevant to some scenarios than to others, an ideal interaction technique will meet all of them, allowing optimal flexibility:

• First, it is geared towards single-user multi-monitor setups and facilitates access to remote areas by scrolling the relevant part of the desktop to the central screen - this may cause conflicts in multi-user settings where different users may want to access different content. • Second, in its current form, the user may either drag the desktop to access remote content or execute a standard drag&drop operation, but not both a the same time. Thus the user is effectively required to use standard drag&drop.

• support for easier interaction with remote drop targets. • support for complex drop targets.

ScaleView portals is an approach presented by Bezerianos [5]. These user-created portals represent alternative views of any area of the screen. Not only the visuals of the original area are recreated, but all actions performed within the portal are also delegated to the remote area and vice versa. Objects can be passed to the remote area by dragging them into the portals. Thus, the portals reduce the distance in visual and motor space alike. However, as ScaleView portals are geared towards more persistent interaction with remote locations, the creation of these portals imposes significant overhead for ephemeral interactions like drag&drop. Thus, drop-and-drag takes up the idea of fully interactive portals to

• support for direct input techniques, i.e. pen- and touchinput. • minimal requirements on available hardware (ideally the technique should work on single-touch or single-pen systems with pressure or proximity detection). • minimal overhead to invoke the drag&drop operation. • minimal interference in multi-user settings. The results of the review can be found in Table 1.

159

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

remote locations, but it modifies this idea for the context of drag&drop interactions.

covers towards the origin of the arc, i.e. the current pen position - to fit into the influence area, targets are reduced in size accordingly. The Vacuum can be used for both selection and drag&drop, depending on whether the user invoked it on empty screen space or on a specific object. While it does allow for complex interactions with remote targets, drag&drop operations with such targets will require the user to trigger two interactions: The first on empty screen space to manipulate the target, the second to perform the actual drop operation - an overhead that may be especially challenging for users unfamiliar with pen-input systems. Furthermore, The Vacuum is designed to use proximity information, i.e. a hover state, which is not available for a larger number of multitouch systems (e.g. those based on frustrated total internal reflection). Also, its clearly highlighted arc of influence may cause interference in multi-user settings.

Techniques which specifically address the challenges of drag&drop interaction on large displays

Push-and-throw, an approach proposed by Hasco¨et and Collomb [11], enables users to throw the object instead of dragging it. To counter the imprecision of human motor skills, a technique called trajectory snapping is integrated into pushand-throw. Trajectory snapping allows for minor inaccuracies by snapping the throwing trajectory to the drop target, if the drop target is not located on the trajectory but rather in close proximity. This technique can be expected to work well for simple, free-standing drop targets. However, it is questionable whether it is still applicable when multiple drop targets are clustered in a small remote area or when preceding interactions like expanding branches within a file explorer are required.

THE DESIGN OF DROP-AND-DRAG

Based on the evaluation criteria introduced in the related work section, we derived the following design decisions for drop-and-drag (respective evaluation criterion in brackets):

Baudisch et al.[3] describe a drag&drop extension called drag-and-pop which takes the opposite approach to pushand-throw. Upon the invocation of a drag-and-drop operation, so-called tip icons, which represent the original drop target, are shown around the drag source. The connection to the original target is made via a virtual “rubber band”. In order to avoid confusion caused by displaying too many tip icons, the set of targets is reduced to those which accept the data of the drag source and which are located in the direction of the initial drag movement. However, as the authors note, “the most common problem with drag-and-pop was getting the right group of targets to pop up”. Furthermore, the iconified versions do not allow complex interactions or precise positioning on the respective drop target.

• Easier interaction with remote targets will be facilitated via proxy targets, i.e. visually and functionally identical copies, of all compatible drop targets projected into the area the user can immediately reach3 . To support quick orientation, an algorithm that preserves the relative spatial layout of the original drop targets will be used to place the proxy targets in the area of immediate reach. (support for easier interaction with remote drop targets). • Proxy targets will be fully interactive and equal in size to the original. To allow for a maximum number of simultaneously displayed proxy targets at the same time, these targets consist only of the user interface components on which a drop can actually take place (e.g. only the file tree from an explorer window). Furthermore, the technique introduces a suspend mode for drag&drop operations to facilitate interaction with complex drop targets without burdening the user to maintain contact with the screen or additional hardware requirements4 (support for complex drop targets).

Based on push-and-throw and drag-and-pop, Collomb et al. propose a drag-and-drop technique push-and-pop [7]. When starting a drag motion, a take-off area is created, surrounding the current cursor position. This take-off area is still a miniature version of the screen as it is in push-and-throw. However, it contains tip-icons of all targets accepting the dragged object, thus eliminating the risk of invoking the wrong set of targets. Additionally, the tip-icons are positioned within the take-off area relative to their locations on the screen. As a result the layout of the take-off area is always the same for a given drag source, independent of its location or the direction of the initial drag motion. This enables the user to perform drag-and-drop actions based on muscle memory, i.e. the user memorizes the particular movement routine due to frequent repetition. Similar to push-and-throw and dragand-pop, push-and-pop does not allow for complex interactions with the available drop targets. However, the idea of projecting the relevant drop targets into the vicinity of the drag source while preserving the proportions of the original screen layout, is taken up in our approach.

• No assumptions will be made on whether there is pen or touch input (support for direct input techniques, i.e. penand touch-input). • Only single taps (clicks) will be necessary to operate the technique (minimal requirements on available hardware). • The technique will be invoked automatically once the user drags an object (minimal overhead to invoke the drag&drop operation). 3 To identify possible drop targets, hard-wired solutions (i.e. mapping drop targets and drag sources at compile time) are possible (and in fact this approach has been used for the implemented prototype). However, there are also more flexible solutions based on the semantic annotation of user interface components [17] 4 (contrary to Kobayashi and Igarashi, who use a throwing metaphor [14], the pause is triggered via a simple drop on empty screen space in drop-and-drag)

Another approach, based on drag-and-pop, is The Vacuum, developed by Bezerianos and Balakrishnan [4] and geared towards pen-input. This technique addresses the problem of getting the right group of targets to pop up by introducing a flexible arc which draws all the objects in the area it

160

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• No visual connections to the original targets are used (minimal interference in multi-user settings).

and the actual target, allowing the user to look for it in the original direction.

Thus, drop-and-drag enables the user to conveniently interact with distal targets, reducing the source-target distance in visual and motor space alike, while preserving the full set of possible interactions with these targets. Proxy Features

Created by cloning the visuals of the drop target, a proxy represents an alternative view of the drop target, similar to a ScaleView portal[5] with the drop target as its remote area. Additionally, all actions performed within the proxy are delegated to the actual drop target. If the visuals of the drop target change due to such a delegated action, the visuals of the proxy are updated as well. Thus, a proxy enables a user to access all functionality provided by the actual drop target. This includes common actions as clicking buttons, expanding and collapsing of trees or invoking menus, as well as more rare actions such as panning a map component to the desired location. Once the user starts to drag an object, his/her options to complete the drag&drop operation using the appearing proxies are twofold: • He/she can drop the dragged object on one of the proxies directly (given the target does not require any additional interaction), • or he/she can pause his/her interaction by temporarily dropping the dragged object next to the proxies. All proxies will remain visible and a resume button on each of them will then be enabled. Then, the user can perform the necessary changes to the state of the target object (without the burden of maintaining the drag&drop operation) and continue his/her drag&drop interaction afterwards. This is done via the resume button which causes the next click to be interpreted as a drop. As the user might want to nullify the paused operation instead of completing it, another button is added to allow the user to cancel the whole interaction, causing all proxies to disappear. An exemplary proxy target, using a file tree is shown in the steps 2 to 6 of the walkthrough in Figure 2. Proxy Placement

After a dragging motion is started and all drop targets accepting the currently dragged object are determined, the locations of the corresponding proxies need to be computed. The layout algorithm that we employ for this purpose was designed with three requirements in mind:

Figure 1. a) The initial version of the proxy positioning algorithm proxy targets are projected nearer to the user, regardless of the original target’s position. b) The refined version of the algorithm places proxy targets directly over the original targets if they are located within the reach of the user’s flexed arm. All other targets are projected into the area between the user’s flexed and stretched arm’s radius. This compensates for user’s moving the dragged objects too fast and overshooting close-by proxy targets.

We implemented these requirements through an algorithm that projects the location of a drop target into the screen space the user can currently reach, i.e. the screen space defined by a circle with the user’s arm length as its radius and the origin of the drag&drop operation as its center. Our first version of this algorithm handled all targets the same way: A proxy is positioned such that d/dmax = d /dmax holds true as illustrated in Figure 1(a). (rstretched denotes the distance defined by the user’s stretched arm; dmax is the maximum distance from the drag source location to the edge of the screen; dmax denotes the maximum projected length and is therefore equal to rstretched . d is the distance between the drag source and the original (potential) drop target P . d is the distance of the new proxy target position P  , calculated by projecting the ratio d/dmax onto dmax .) However, preliminary tests revealed issues with this implementation. Users often expected proxies to appear with a reasonable distance to the origin of the drag&drop operation. When dragging very fast, they overshot the proxy before it became visible and then failed to notice it immediately, as they focused on the original drop target. Furthermore, users frequently dragged over a short distance before dropping on empty space. This allowed them to take full advantage of drop-and-drag, i.e. they could interact with the proxy without being forced to keep on dragging the object. Nearby

1. A proxy should always be within the user’s immediate reach. 2. Spatial arrangement, alignment and proximity between the actual drop targets should be preserved. 3. In order to reduce the need for reorientation, the proxy should be placed on the line of sight between the source

161

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

proxies often interfered with this strategy by covering the relevant screen space. To avoid these issues, we modified the algorithm by distinguishing between two kind of drop targets: those within the reach of the flexed arm and those beyond this distance. If a target is located within the reach of the user’s flexed arm, its proxy is positioned right above it. All other drop targets are projected into the ring between the length of the user’s stretched arm (rstretched ) and the length of his/her flexed arm (rf lexed ), as illustrated in Figure 1(b). (dmax is the maximum distance from the drag source location to the edge of the screen minus rstretched , d is the distance between the drag source and the original drop target P minus rstretched , P  is the location of the projected drop target proxy. d is the distance of the new proxy target position P  , calculated by projecting the ratio d/dmax onto dmax , which denotes the difference between the user’s stretched and flexed arm’s radius.) This minimizes the amount of proxies located within very close proximity. As a result it is very unlikely that the user overshoots a proxy or that close-by proxies interfere with the intention to pause the drag&drop operation. At the same time it is still assured that every proxy is within immediate reach. Our current algorithm can be applied to an arbitrary number of drop targets simultaneously. For a high number of large potential targets, this may lead to occlusion problems. This and other possible limitations of drop-and-drag are discussed later in the next section. Exemplary drop-and-drag interaction

To illustrate the features offered by drop-and-drag, Figure 2 shows an exemplary sequence of actions, using drop-anddrag to move a folder from a newly attached portable flash drive to a folder currently not visible in the file explorer view of the user’s computer: 1. The file explorer showing the user’s computer is located in the upper left of the screen - the flash driver’s file explorer has just opened in the lower right corner. 2. The user initiates a dragging gesture on the folder to copy the file - a drop target proxy, which shows the file tree part of the “computer-file explorer”, is displayed close to the starting point of the dragging gesture. 3. As the the node/folder on which the user wants to drop the dragged folder, he/she pauses the drag&drop operation by dropping the dragged object on empty screen space. 4. Now he/she can conveniently interact with the file tree displayed by the proxy and expand the corresponding node. 5. By pressing the resume button at the lower left edge of the proxy, the user indicates that he/she wants to actually drop the dragged object. 6. Subsequently, he/she selects the now visible drop location. As a result, the dragged folder is copied to the folder

162

Figure 2. An exemplary drop-and-drag operation where pictures are copied from a flash drive to a sub-folder of “My Pictures”.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

within the file tree and the desktop returns to its initial state. 7. However, the state of the original file tree branch is now expanded as the state changes were forwarded by the drop target proxy. EVALUATING DROP-AND-DRAG

To assess the general performance of drop-and-drag and to investigate possibilities for future improvements, we conducted a controlled experiment comparing drop-and-drag to traditional drag&drop (later, in the discussion section, we elaborate on the advantages and limitations of this choice). Participants of this study had to perform drag&drop operations using both techniques in two different scenarios involving complex and simple drop targets. We had five hypotheses:

Figure 3. If the finger is not approaching the display surface perpendicularly, an unintended cursor position may occur.

issues. They were advised to touch the display perpendicularly if possible. Participants

Twenty-eight researchers and students from our lab volunteered to participate in our experiment. Five of them were female, twenty-three were male. Their ages were between twenty-three and forty. All but three participants were right handed. However, two of the three left-handed participants stated that they use the mouse with their right hand and were therefore treated as right-handed during the course of the experiment. None of the participants had any prior experience using drop-and-drag. They rated their experience in using touchscreen devices from 2 to 4 on a 5 point scale (1 no experience at all, 2 below average, 3 average, 4 above average, 5 extensive experience) with a mean value of 2.6.

H1 With increasing distances, drop-and-drag becomes significantly faster than traditional drag&drop. H2 The effect of H1 is more distinct in case of complex drop targets. H3 The average error rate of drop-and-drag does not exceed the error rate of traditional drag&drop. An operation is considered erroneous, if it is aborted or if the dragged object is dropped on a target, that was not in the correct state. H4 For more distant targets drop-and-drag will be less errorprone than traditional drag&drop. H5 drop-and-drag will exhibit a better user acceptance on large touchscreen devices.

Design

A within subject design with repeated measures and three independent variables was used. These variables were technique (traditional drag&drop, drop-and-drag), target distance (very close, close, medium, far) and target complexity (simple, complex). The target distance was not defined by static values, but rather in relation to the participants’ arm length. ’Very close distance’ targets were positioned at a distance of approximately 0.5 times the participant’s arm length, ’close distance’ targets at 0.9 times the arm length, ’medium distance’ targets at 1.6 the arm length and ’far’ targets at 2.5 the arm length. In case of ’simple’ drop targets, the dragged object could be dropped without any prerequisites. ’Complex’ targets, however, consisted of a panel containing two tabs. In order to perform a successful drop, the user was required to switch to the second tab (cf. Figure 4). In case of the traditional drag&drop technique, this forces a participant to reach for the target and select the second tab prior to the start of the actual drag interaction. By contrast, drop-anddrag enables the user to start the drag, perform a temporary drop anywhere on the screen, switch the tab on the displayed proxy and complete the operation via the resume button and a subsequent click on the actual target panel. Simple drop targets were 225 pixels wide and 210 pixels high. Complex drop targets were 24 pixels higher to accommodate for the additional tabs necessary to provide the tabbed pane. Thus, the area relevant to the drag&drop operation, i.e. the actual drop target, retained its size. Consequently, the size relevant to the index of difficulty according to Fitts’ law remains constant throughout the experiment as well. In summary the ex-

Apparatus

The user study was performed on a Barco[2] display wall with a screen width of 300 cm and a height of 150 cm. The screen was placed 87 cm above the ground and was operated at a resolution of 4200 x 2100 pixels. Single touch input was provided by an array of infrared sensors, fabricated by SMART technologies [21]. Inherent to this technique for recognizing touch input two side effects affected our experiment: First, a “touch” can be registered without physical contact to the screen. This sometimes confuses users who are unfamiliar with the device, as unintentional clicks are performed. Second, the accuracy of the touch input depends on the angle between the screen surface and the user’s finger. The flatter the angle, the more likely other body parts, e.g. other fingers, the wrist or the elbow, are registered by the infrared sensors as well, resulting in a cursor position that does not correspond with the one intended by the user (cf. Figure 3). This problem most often occurs when acquiring distant targets, as the user tries to maximize his/her reach by aligning his/her finger with his/her arm. As a consequence, this behaviour frequently leads to very imprecise and sometimes unintentional touch input. In order to avoid errors that are a mere result of this touchrecognition technique, participants were made aware of these

163

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

periment consists of 1792 samples (28 participants x 2 techniques x 2 target complexities x 4 distances x 4 repetitions).

Figure 5. A schematic screen layout for a right handed person. Each set of targets is depicted by a different symbol: target 1-4 by a square, target 5-8 by a pentagon, target 9-12 by a star, and target 13-16 by a triangle. The reach of the participant’s flexed and stretched arm is depicted by the two dashed circles.

Figure 4. The two states of a complex drop target: In the left part of the figure, the target does not accept any drops. The right side shows the target, where the correct tab is selected and drops are accepted.

Test Procedure

To avoid the impact of exhaustion or learning effects on the results, the order of the test runs was permuted over the experiment. Additionally, the single operations during each of the test runs were performed in random order.

Before performing any drag-and-drop operations, the participants were required to configure some user specific parameters before proceeding to the actual testing phase. Those parameters were: • The height at which the participant could comfortably touch the screen while standing in front of it, • the preferred hand, and • the length of both the flexed as well as the stretched arm.

Results

To test our hypotheses, we analyzed task completion times, error rates and user ratings for the two experiment scenarios. Additionally we collected qualitative feedback during and after the test runs that gave hints to possible improvements of drop-and-drag. Task Completion Time

After completing the configuration phase, the screen layout was adapted accordingly. If the participant was right (left) handed, the source of all drag-and-drop operations was placed at the left (right) edge of the screen. This was done to generate sufficiently distant targets, but should not affect validity - for more realistic scenarios, the placement algorithm will adapt to the possibly range in the other direction. On the basis of the length of the participant’s flexed and stretched arm, four sets of four targets each were generated (cf. Figure 5), one set of targets for each distance (’very close distance’, ’close distance’, ’medium distance’ and ’far distance’). Consequently, only eight targets were within the user’s immediate reach. The remaining targets required the user to walk towards them, as they had a distance of 1.6 to 2.5 times the user’s stretched arm length to the drag source.

Two-way repeated measures ANOVA tests revealed that distance significantly affected task completion times independently of technique and in both scenarios (simple targets: F (3, 216) = 34.00, p  0.01; complex targets: F (3, 216) = 34, 01, p  0.01). While the technique alone did not significantly affect the task completion times (simple targets: F (3, 216) = 0.1, p = 0.76; complex targets: F (3, 216) = 3.02, p = 0.08), a strong technique × distance interaction (simple targets: F (3, 216) = 6.67, p < 0.01; complex targets: F (3, 216) = 17.25, p  0.01) indicates that dropand-drag and traditional drag&drop are affected differently by changes in distance. Distance very close

Users were asked to accustom themselves with the screen layout and the different set-ups for about five minutes in order to avoid learning effects during the experiment affecting the results. Subsequently, four test runs, i.e. every combination of both drag-and-drop techniques and complex or simple drop targets, were conducted for each participant. Every test run consisted of at least sixteen drag-and-drop operations, one for each target. In case of an error, i.e. the dragged object was not dropped on the drop target, the operation had to be repeated. For each step in each test run there was only a single valid drop target. Furthermore, the participants were asked to look for the target first, i.e. before they initiated the logged drag&drop operation by activating the source with a button click.

close medium far

Simple Targets F (1, 54) = 15.88 p < 0.01 F (1, 54) = 4.80 p = 0.03 F (1, 54) = 0.01 p = 0.91 F (1, 54) = 5.73 p = 0.02

Complex Targets F (1, 54) = 29.58 p  0.01 F (1, 54) = 15.98 p = 0.01 F (1, 54) = 0.07 p = 0.79 F (1, 54) = 13.94 p  0.01

Table 2. Results of one-way repeated measures ANOVA tests. The nominal factor was technique (traditional drop&drag, drop-and-drag). The results show significant differences between techniques for ’very close’, ’close’ and ’far’ distances.

The results of additional one-way repeated measures ANOVA tests presented in Table 2 support this: there was a significant

164

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

difference between techniques for targets at ’very close distance’, ’close distance’ and ’far distance’. Only for targets at ’medium distance’ no significant difference was found (this can be attributed to the fact that drop-and-drag starts to outperform traditional drag&drop around this distance). These findings are reinforced by an analysis of the task completion times of all participants. As illustrated in Figures 6 and 7, traditional drag&drop remains faster than drop-anddrag for targets in the user’s immediate reach; this is the case for targets at ’very close distance’ and ’close distance’.

Figure 8. Relative time savings of drop-and-drag versus target distances in relation to arm length.

to be observed more clearly when drop targets require additional interaction to successfully complete the operation. Although drop-and-drag begins to be faster at approximately the same distance, regardless of target complexity, Figure 8 shows that the time savings when using drop-and-drag with complex targets exceed those for simple targets. We therefore consider our hypothesis H2 confirmed. Figure 6. Task completion times of test runs with simple targets versus target distances in relation to arm length.

Error Rates

Although error rates were expected to rise with increasing distance, no such correlation was observed. Overall error rates were relatively low. For simple targets, drop-and-drag performed better than traditional drag&drop; for complex targets, traditional drag&drop was slightly less error-prone (cf. Table 3). However, paired t-tests showed that both differences are not significant, due to a very high variance within our data coupled with a relative low overall error rate. The most frequent error when using drop-and-drag with complex targets was related to resuming a paused drag&drop operation. It seemed that despite the time the participants were given to familiarize themselves with drop-and-drag, some of them did not acquaint themselves with the concept of pausing and resuming an operation before completing some of the timed tasks. Summing up, H4 is rejected by our results, H3 is confirmed for the overall error rate, but the results for complex targets indicate room for improvement.

Figure 7. Task completion times of test runs with complex targets versus target distances in relation to arm length.

At a distance of approximately 1.6 times the user’s arm length, i.e. the distance of the ’medium distance’ targets, both techniques perform equally fast. When exceeding a distance of about 2 times the arm’s length, which was the case for the ’far distance’ targets, drop-and-drag clearly outperforms traditional drag&drop. For these targets drop-and-drag was on average 32% faster. Overall, these findings support the hypothesis that drop-and-drag becomes significantly faster than traditional drag&drop with increasing distances (H1). Furthermore, we hypothesized that this effect is supposed

165

Targets Simple Complex Overall

Drag&drop 8.57% 10.62% 9.60%

drop-and-drag 3.58% 12.71% 8.15%

Table 3. Mean error rates

Subjective Satisfaction

After completing all four test runs, the participants answered a short questionnaire in order to capture their subjective satisfaction concerning traditional drag&drop and drop-and-drag. Participants were asked whether the task was easy to fulfill using the different techniques and whether the respective

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

techniques provided reasonable support - their ratings were captured on a five point Likert scale (“1 being “full disagreement” and 5 “full agreement”).

drop targets in push-and-pop[7], we plan to re-implement this technique for future evaluations as, besides this, it meets most of our other requirements for ideal drag&drop techniques for large touchscreens very well.

While both techniques were perceived equally difficult in case of simple drop targets (the average rating was 3.4 for traditional drag&drop and 3.8 for drop-and-drag), drop-anddrag outperformed traditional drag&drop when operating on complex drop targets (average rating of 3.5 versus 2.4). Furthermore, only four of the 28 participants would still prefer traditional drag&drop when interacting with a very large touch-screen. This supports our expectation that drop-anddrag exhibits a better user acceptance due to the benefits it offers (H5). Further evaluation will however be necessary to test whether this hypothesis also holds against other supporting techniques for drag&drop (see next section).

CONCLUSION & OUTLOOK

When designing drop-and-drag, we aimed for a technique which makes drag&drop operations on large touchscreen displays both faster and easier, especially when more complex state manipulations are required. The findings of the controlled experiment show that this goal was met. Although traditional drag&drop exhibits faster performance when for targets within immediate reach, the ability of dropand-drag to efficiently bridge large distances, even when additional interaction complicates the drag&drop operation, is apparent (time-savings of up to 32% compared to traditional drag&drop). Furthermore, user satisfaction was improved notably: During their sessions multiple participants stated they felt more comfortable with drag&drop operations that use drop-and-drag rather than traditional drag&drop.

DISCUSSION

While the the results of our experiment show that drop-anddrag performs well, especially for targets in medium (1.5 × user’s arm length) and far (2.5 × user’s arm length) distance, there is also a number of limitations in both the evaluation and our current design itself.

Yet, there is room for improvement: Qualitative feedback from some study participants indicated a number of refinements of our technique. Two participants suggested that employing animated transitions when creating the drop target proxies could improve the usefulness of drop-and-drag. Adding a rubber band as in push-and-pop would also be an option, but was not proposed by participants. Further on, additional research will be necessary to determine whether these visual cues cause interference in multi-user settings.

Proxy size and placement

As noted when elaborating on our design in design section, our current design will work for an arbitrary number of potential drop targets, where each target results in a proxy in equal size. For a reasonably large number of targets this may lead to occlusion problems: Important information could be hidden from the user and he/she might be unable to access essential interaction elements such as buttons or menus. However, an approach that scales down the proxy targets size (like The Vacuum [4]) may also be challenging, especially when touch input is used and thus precision is limited by the human finger. More refined techniques may include fisheye view [9], but additional investigation should clarify the typical number of targets involved in drag&drop operations, first. Also, an analysis of the context users need to identify the proxy target (e.g. the surrounding explorer window for a file tree) could help to determine the necessary size.

More controversial was the way the drag operation is resumed. While one participant explicitly stated that he liked the possibility to complete the operation with two clicks, two others stated that they would prefer to resume the operation by another drag interaction, which would originate from a scaled-down version of the original source, placed at the current position of the resume button. The latter suggestions might also offer an improvement to the unexpectedly long time it took for some participants to get accustomed to pausing and resuming drag&drop operations.

Visual connections and interference

We deliberately decided not to include any visual connections between proxy targets and original targets, as we expected them to cause visual interference in multi-user settings. However, we noticed during our evaluation that some participants would reorient during the drag&drop operation, even though proxy targets were placed on the line of sight between drag source and original target. Thus further investigation on the interference caused by visual connections is necessary for multi-user settings. It may be reasonable to compromise and introduce lightweight connection visuals like the rubber-bands of push-and-pop[7].

Furthermore, as pointed out during the discussion of the limitations of drop-and-drag, future work will need to include refinements to the the current positioning algorithm which may have occlusion problems for a larger number of targets. Finally, while drop-and-drag did outperform the traditional technique, a more extensive comparison will also need to include state-of-the-art approaches like push-and-pop[7], at least for the case of simple drop targets. ACKNOWLEDGMENTS

We would like to thank all those participating in our experiments for their time and effort. Also we would like to thank both our colleagues and the reviewers for giving valuable advice on how to improve this paper and the presented interaction technique.

Evaluation against standard drag&drop

While the evaluation of our technique against standard drag&drop allowed a basic assessment of its performance, it allows comparison to state of the art techniques only to a very limited degree. Even though there is no support for complex

166

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The work presented in this paper has been partly funded by the German Federal Ministry of Education and Research.

12. K. Hawkey, M. Kellar, D. F. Reilly, T. Whalen, and K. M. Inkpen. The proximity factor: impact of distance on co-located collaboration. In GROUP ’05: Proceedings of the SIGGROUP conference on Supporting group work, pages 31–40. ACM, 2005.

REFERENCES

1. R. Ball and C. North. Effects of tiled high-resolution display on basic visualization and navigation tasks. In CHI ’05: Extended Abstracts of the SIGCHI conference on Human factors in computing systems, pages 1196–1199. ACM, 2005.

13. A. Kahn, J. Matejka, G. Fitzmaurice, G. Kurtenbach, N. Burtnyk, and B. Buxton. Toward the digital design studio: Large display explorations. Human-Computer Interaction, Volume 24(1 & 2):9–47, 2009.

2. BARCO. Barco control rooms. http://www.barco.com/controlrooms/. last accessed: 2010-05-11.

14. M. Kobayashi and T. Igarashi. Boomerang: suspendable drag-and-drop interactions based on a throw-and-catch metaphor. In UIST ’07: Proceedings of the ACM symposium on User interface software and technology, pages 187–190. ACM, 2007.

3. P. Baudisch, E. Cutrell, D. Robbins, M. Czerwinski, P. Tandler, P. T, B. Bederson, and A. Zierlinger. Drag-and-pop and drag-and-pick: techniques for accessing remote screen content on touch- and pen-operated systems. In INTERACT ’03: Proceedings of the IFIP TC13 Conference on Human-Computer Interaction, pages 57–64. Springer, 2003.

15. Microsoft. Microsoft surface product information. http://www.microsoft.com/surface/. last accessed: 2010-05-10. 16. C. M¨uller-Tomfelde and C. Schremmer. Touchers and mousers: commonalities and differences in co-located collaboration with multiple input devices. In CHI ’08: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 1149–1152. ACM, 2008.

4. A. Bezerianos and R. Balakrishnan. The vacuum: facilitating the manipulation of distant objects. In CHI ’05: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 361–370. ACM, 2005. 5. A. Bezerianos and R. Balakrishnan. View and space management on large displays. IEEE Computer Graphics and Applications, 25(4):34–43, 2005.

17. H. Paulheim. Ontology-based Modularization of User Interfaces. In EICS ’09: Proceedings of the SIGCHI Symposium on Engineering interactive computing systems, pages 23–28. ACM, 2009.

6. X. Bi and R. Balakrishnan. Comparing usage of a large high-resolution display to single or dual desktop displays for daily work. In CHI ’09: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 1005–1014. ACM, 2009.

18. J. Rekimoto. Pick-and-drop: a direct manipulation technique for multiple computer environments. In UIST ’97: Proceedings of the ACM symposium on User interface software and technology, pages 31–39. ACM, 1997.

7. M. Collomb, M. Hasco¨et, P. Baudisch, and B. Lee. Improving drag-and-drop on wall-size displays. In GI ’05: Proceedings of the Graphics Interface conference, pages 25–32. Canadian Human-Computer Communications Society, 2005.

19. G. Robertson, M. Czerwinski, P. Baudisch, B. Meyers, D. Robbins, G. Smith, and D. Tan. The large-display user experience. IEEE Computer Graphics and Applications, 25(4):44–51, 2005. 20. D. M. Russell, J. P. Trimble, and A. Dieberger. The use patterns of large, interactive display surfaces: Case studies of media design and use for blueboard and merboard. In HICSS ’04: Proceedings of the Hawaii International Conference on System Sciences, 2004.

8. M. Czerwinski, G. Smith, T. Regan, B. Meyers, G. G. Robertson, and G. Starkweather. Toward characterizing the productivity benefits of very large displays. In INTERACT ’03: Proceedings of the IFIP TC13 Conference on Human-Computer Interaction, pages 9–16. Springer, 2003.

21. SMART Technologies. Smart technologies. http://www.smarttech.com/. last accessed: 2010-05-10.

9. G. W. Furnas. Generalized fisheye views. In CHI ’86: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 16–23. ACM, 1986.

22. D. S. Tan, D. Gergle, P. Scupelli, and R. Pausch. With similar visual angles, larger displays improve spatial performance. In CHI ’03: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 217–224. ACM, 2003.

10. J. Y. Han. Low-cost multi-touch sensing through frustrated total internal reflection. In UIST ’05: Proceedings of the ACM symposium on User interface software and technology, pages 115–118. ACM, 2005.

23. T. Tsandilas and R. Balakrishnan. An evaluation of techniques for reducing spatial interference in single display groupware. In ECSCW’05: Proceedings of the European Conference on Computer Supported Cooperative Work, pages 225–245. Springer, 2005.

11. M. Hasco¨et. Throwing models for large displays. In HCI ’03: Proceedings of the British HCI Group Annual Conference, pages 73–77, 2003.

167

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Extending Boundaries with Meta-Design and Cultures of Participation Gerhard Fischer Center for LifeLong Learning and Design (L3D) University of Colorado Boulder, CO 80309-0430 USA [email protected] ABSTRACT

INTRODUCTION

Human-computer interaction (HCI) has refocused many research efforts within computer science from a technology-centered view to a human-centered view. The developments so far, however, have seen humans mostly as users and consumers rather than as active contributors and designers.

In the last 20 years, research and development in humancomputer interaction (HCI) have made major contributions to support the productive and creative autonomy of individuals. Historically, however, these possibilities often have been of interest and accessible only to a small number of “high-tech scribes.” Meta-design [21] is focused on the challenge of allowing users of software systems who are not primarily interested in software per se to modify, extend, evolve, and create systems that fit their needs.

This paper provides a conceptual framework based on meta-design and cultures of participation that democratize design and allows all stakeholders to evolve systems to fit their needs. It establishes a new extended discourse for HCI research in which information technologies are interwoven with human lives in all aspects of our existence (at home, working, teaching, learning, and being a citizen). Specific socio-technical environments instantiating the framework in different application domains are described, including: (1) environments for people with cognitive disabilities, (2) table-top computing systems for framing and solving complex urban planning problems, (3) modeling the buildings of the world in 3D, and (4) using Smart Grids to support energy sustainability. These examples show how metadesign and cultures of participation are design approaches that allow researchers and practitioners to extend boundaries by allowing all stakeholders to have more control over their artifacts and by providing opportunities to encourage and to support contributions by many people in personally meaningful activities.

What the personal computer has done for the individual, the Internet has done for groups and communities. The first decade of Internet use was dominated by broadcast models and contributed little to change the existing strong separation of designers and users imposed by existing media. Cultures of participation [19], supported by meta-design and the participatory web [31], represent an evolving framework to exploit computational media in support of collaboration and communication. Providing all citizens with the means to become co-creators of new ideas, knowledge, and products in personally meaningful activities presents one of the most exciting innovations and transformations of digital media, with profound implications to extend the boundaries of HCI research. This paper identifies boundaries, defines conceptual frameworks (centered on meta-design and cultures of participation), and describes socio-technical environments [27] (in four different application domains) grounded in these frameworks. It discusses the implications of these research activities to extend the boundaries of HCI.

Keywords

meta-design, cultures of participation, systemic problems, boundaries, distances, motivation, control, socio-technical environments

BOUNDARIES

ACM Classification Keywords

A basic challenge insufficiently addressed by prior HCI research is that almost all of the significant problems of tomorrow will be systemic problems, which cannot be addressed by any one specialty. These problems require multi-disciplinary, multi-sector, and international collaborations, providing opportunities for knowledge workers to work in teams, communities, and organizations that encompass multiple ways of knowing and collaborating. Our research has focused specifically on complex, systemic design problems requiring cultural and epistemological pluralism to make all voices heard. As stakeholders who

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

Permission to make digital or hard hard copies copies of of all all or or part part of ofthis thiswork workfor for personal or classroom classroom use use is is granted granted without withoutfee feeprovided providedthat thatcopies copiesare are not made or commercial advantage and and that copies madeorordistributed distributedforforprofit profit or commercial advantage that bear thisbear notice the full the first To copy copies thisand notice and citation the full on citation on page. the first page.otherwise, To copy or republish,ortorepublish, post on servers or on to redistribute requirestoprior otherwise, to post servers or to lists, redistribute lists, specific and/or a fee. and/or a fee. requires permission prior specific permission NordiCHI 2010, October 16–20, 16–20, 2010, 2010,Reykjavik, Reykjavik,Iceland. Iceland. Copyright 2010 ACM ISBN: ISBN: 978-1-60558 978-1-60558-934-3...$5.00.

168

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

are separated by spatial, temporal, conceptual, and technological distances [18] are brought together, boundaries can be overcome based on these distances. These distances will be briefly characterized as representing boundaries that need to be understood, respected, and extended.

These structures and skills often do not exist at the beginning but evolve incrementally and collaboratively. CoIs provide an example of the importance of combining voices from different communities.

Boundaries Based on Spatial Distance

The preceding subsections emphasized computer-mediated collaboration among humans to reduce the gaps created by spatial, temporal, and conceptual distances. In HCI systems, substantial information is embedded in computational artifacts.

Boundaries Based on Technological Distances

Bringing spatially distributed people together with the support of computer-mediated communication allows the prominent defining feature of a group of people interacting with each other to become shared concerns rather than a shared location. By allowing more people to be included, local knowledge can be exploited. These opportunities have been successfully employed by the open-source communities [34] as well as by social networks of people who have a shared concern (such as a family member with a disability [9]). Transcending the barrier of spatial distribution is of particular importance in locally sparse populations (e.g., as they exist among people with disabilities).

Unlike passive design materials, such as pen and paper, computational artifacts are able to interpret the work of designers and actively “talk back” to them [36]. However, barriers occur when the back talk is presented in a form that users cannot comprehend or when the back talk created by the design situation itself is insufficient, and additional mechanisms and information are needed. To increase the back talk of the situation, we have developed a series of critiquing systems [22] that monitor the actions of users as they work to achieve a common goal and inform the users of potential problems during this process.

Boundaries Based on Temporal Distance

Design processes often take place over many years, with initial design followed by extended periods of evolution and redesign. In this sense, design artifacts are not designed once and for all, but instead gradually change over long periods of time. For example, within most computer networks, when a new device or technology emerges, the infrastructure is incrementally enhanced and updated rather than redesigned completely from scratch. To be able to do this requires that new designers “collaborate” with the original designers. Long-term collaboration is crucial for the success of evolutionary development. This is hampered when individual designers are not informed about how the decisions they make interact with decisions that were made in the past from designers whose voices have been lost.

EXTENDING BOUNDARIES Supporting Users as Active Contributors with Meta-Design

In a world that is not predictable, improvisation, evolution, and innovation are more than a luxury—they are a necessity. The challenge of design is not a matter of avoiding the emergent, but rather of including it and making it an opportunity for more creative and more adequate solutions to problems. Meta-design [21] explores objectives, techniques, and processes to enable users to act as designers and active contributors, allowing them to create new knowledge rather than restricting them to the consumption of existing knowledge. The need for meta-design is founded on the observation that design requires open systems that users can modify and evolve [24]. Because problems cannot be completely anticipated at design time when the system is developed, users at use time will encounter mismatches between their problems and the support that a system provides. These mismatches will lead to breakdowns [22] that serve as potential sources for new insights, new knowledge, and new understanding. Meta-design advocates a shift in focus from finished products or complete solutions to conditions for users to resolve mismatches and repair breakdowns when they are discovered during use.

Boundaries Based on Conceptual Distances

To analyze the contribution of voices from different communities, two types of communities can be differentiated: communities of practice (CoPs) and communities of interest (CoIs). CoPs [46] consist of practitioners who work as a community in a certain domain undertaking similar work. Examples of CoPs are architects, urban planners, research groups, software developers, and end-users. CoPs gain their strength from shared knowledge and experience. However, they face the boundaries of domain-specific ontologies and tools, empowering the insiders but often creating barriers for outsiders and newcomers. CoIs [15,17,29] can be thought of as “communities-ofcommunities” brought together to solve a problem of common concern. Examples of CoIs are (1) a team of software designers, marketing specialists, psychologists, and programmers interested in software development; or (2) a group of citizens and experts interested in urban planning. Fundamental challenges facing CoIs are found in building a shared understanding, learning to communicate with and learning from others, and establishing a common ground.

Meta-design engages diverse contributors (individuals and communities) in designing and building their own tools and systems by democratizing design. Our future world will be substantially shaped by stakeholders who will design, build, and evolve their own devices—our goal is to inspire, shape, support, foster, and analyze these communities. Metadesign allows creative and unplanned opportunism by addressing one of the fundamental challenges of a knowledge

169

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

society: to create socio-technical environments in which all participants in collaborative design processes can express themselves and engage in personally meaningful activities.

Web 2.0 architectures) have emerged to support social computing. These developments are the foundations for a fundamental shift from consumer cultures (specialized in producing finished goods) to cultures of participation (in which all people can participate actively in personally meaningful activities). Cultures of participation provide foundations for this fundamental transformation by exploring and supporting new approaches for the design, adoption, appropriation, adaptation, evolution, and sharing of artifacts by all participating stakeholders. These cultures are not dictated by technology alone; they are the result of incremental shifts in human behavior and social organizations [6].

By allowing users to become co-designers, HCI needs to develop architectures, seeds, and features that provide all stakeholders with opportunities, tools, and social reward structures to extend and evolve systems to fit their needs. In current design practices, teams of developers function as experts observing use and designing solutions to fit existing work practices, typically with input and involvement from users at one or more iterative stages. There is one “official” future toward which design efforts are oriented and around which design decisions are made. There is an intentional focus on the immediate needs of the current users; the end result must be a complete artifact, leading to rigid design decisions made at design time. These techniques yield closed systems based on decisions made at design time, and stakeholders are incapable of modifying and evolving the system when new and unanticipated requirements arise during use.

A major objective of cultures of participation is to attract large number of contributors. A number of notable success models exist, including open source software, Wikipedia, Second Life, YouTube, and 3D Warehouse (our research is less focused on social networks that primarily serve communication purposes such as Twitter and Facebook). By breaking down the boundaries between producers and consumers, cultures of participation have created new opportunities and challenges.

Design practices based on meta-design are focused on systems facing unpredictable future developments and changes. Participatory design includes users in the initial design process, but these participants can represent only the immediate users and immediate needs. Meta-design applies design techniques that enhance the further development of systems at use time. Design power is shifted towards the users, allowing them to ultimately act as both designers and consumers of the system and allowing the system to be shaped through real-time use. Table 1 summarizes the different role distributions.

Breaking Down Strict Role Separations

New concepts and models establishing middle ground between consumers and producers are emerging, including the following: • Prosumers [44] are techno-sophisticated and comfortable with the technologies with which they grew up. They have little fear of modifying and evolving artifacts to their own requirements. They do not wait for someone else to anticipate their needs, and they can decide what is important for them. They participate in learning and discovery and engage in experimenting, exploring, building, tinkering, framing, solving, and reflecting.

Transcending Consumer Cultures with Cultures of Participation

The first decade of the World Wide Web predominantly enforced a clear separation between designers and consumers. New technological developments (such as participatory

Design Approach

Design Time

Use Time

professionally dominated design

users have no voice

users have to live with artifacts designed by others

participatory design

users are active participants; systems are designed as complete artifacts at design time

systems are designed with users’ input, but they cannot be evolved to serve unforeseen needs

users are active participants; systems are designed as seeds; design is focused on use and participation

users can act as designers and evolve the artifact to fit new needs

meta-design

• Professional amateurs [26] are innovative, committed, and networked amateurs working up to professional standards. They are new social hybrids, and their activities are not adequately captured by the traditional dichotomous definitions of work and leisure, professional and amateur, and consumption and production. These new classes of contributors form the foundation for social production and mass collaboration [6] by relying on the following facts: • A tiny percentage of a very large base is still a substantial number of people; • Beyond the large quantitative numbers of contributors exists a great diversity of interests and passions among users; • Although human beings often act for material rewards, they can also be motivated by social capital, reputation, connectedness, and the enjoyment derived from giving away things of value.

Table 1. The role of stakeholders in different design approaches

170

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Intrinsically motivated contributors can be found in the Long Tail [3], where people from around the world engage in topics and activities about which they feel passionate.

participation provide conceptual frameworks for the design of socio-technical environments because they give users design power to modify and evolve the systems according to their needs.

Open Design Spaces and User-Generated Content

This section briefly describes four specific examples that extend boundaries with meta-design and cultures of participation:

Cultures of participation emphasize the “unfinished” and take into account that design problems have no stopping rule, need to remain open and fluid to accommodate ongoing change, and can be characterized as having “continuous beta” as a desirable attribute rather than a to-be-avoided detriment. Cultures of participation can move from guidelines, rules, and procedures to exceptions, negotiations, and work-arounds to complement and integrate existing accredited and expert knowledge with informal, practice-based, and situated knowledge [42,47].

• the Memory Aiding Prompting System (MAPS), which fosters communities and creates new support tools for people with cognitive abilities and their caregivers; • the Envisionment and Discovery Collaboratory (EDC), which is a table-top computing environment supporting stakeholders from diverse backgrounds in face-to-face meetings;

User-generated content includes: (a) creating artifacts with existing tools (e.g., writing a document with a word processor) or (b) changing the tools (e.g., writing macros to extend the word processor as a tool). In specific environments, such as open-source software, the content is subject to the additional requirement of being computationally interpretable. Different activities for content generation (e.g., adaptation, generalization, improvement requests, specialization, and tailoring [2]) need to be supported.

• the three-dimensional (3D) modeling environment, which allows people from around the world to create and share 3D models; and • Smart-Grid developments, which involve consumers actively as decision makers to support new approaches toward energy sustainability. Cognitive Lever (CLever): Helping People Help Themselves

The CLever project has been a large-scale research project at the University of Colorado [9]. The project extended boundaries by complementing human minds through media and technologies. As an important development within CLever, the Memory Aiding Prompting System [8] is a meta-design environment in which caregivers can design task-support tools for people with limited memory and executive functions. Individuals with cognitive disabilities are often unable to live independently due to their inability to perform activities of daily living, such as cooking, housework, or shopping. By being provided with socio-technical environments to extend their abilities and thereby their independence, these individuals can lead lives less dependent on others. MAPS has explored meta-design by supporting mobile device customization, personalization, and configuration by caregivers and effective use by clients [10].

Rich Ecologies of Participation

Complex socio-technical environments cannot be understood as simple aggregations of the behavior of some nonexistent average user [37]. Stakeholders need to take on different tasks and responsibilities as they progress toward the demanding levels of participation. Most stakeholders will start as consumers, but only a small percentage will eventually become active contributors, curators, and metadesigners [33]. Creating innovative support mechanisms (e.g., learning environments, scaffolding, boundary objects) and social reward structures (e.g., intrinsic motivation, reputation economies, gift cultures) presents challenges for HCI research to reduce the funnel effect causing a limited migration toward more demanding roles [32]. SOCIO-TECHNICAL ENVIRONMENTS EXTENDING BOUNDARIES IN SPECIFIC APPLICATION DOMAINS

Meta-design is of critical importance for people with cognitive disabilities because they represent a “universe of one”: a solution for one person will rarely work for another. The success of MAPS is based on the empirical findings that (a) unexpected islands of abilities exist (clients can have unexpected skills and abilities that can be leveraged to ensure a better possibility of task accomplishment); and (b) unexpected deficits of abilities exist. Accessing and addressing these variations in skills and needs, particularly with respect to creating task support, requires an intimate knowledge of the client that only caregivers can provide. Currently, a substantial portion of all assistive technology is abandoned, resulting in the very population that could most benefit from technology paying for expensive devices that end up in the back of closets after a short time.

In the last decade, we have developed socio-technical environments to extend boundaries. In all of these application domains, socio-technical environments are needed because the deep and enduring changes of our ages are not technological in their core substance, but social and cultural. Changes in complex environments are not primarily determined by technology, but are the result of incremental shifts in human behavior and social organization. Sociotechnical environments are composed both of computers, networks, and software, and of people, processes, policies, laws, and institutions, thereby creating a complex web of socio-cultural concerns and requiring the co-design of social and technical systems. Meta-design and cultures of

171

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

framing and resolving complex urban planning by bringing together participants from various backgrounds in face-toface meetings. In this platform, the knowledge to understand, frame, and solve such problems does not already exist [16], but is constructed and evolves during the solution process—an ideal environment to study meta-design and cultures of participation. The EDC represents a socio-technical environment that incorporates a number of innovative HCI technologies and approaches, including (a) table-top computing, which fosters and invites participation by maximizing the richness of communication among stakeholders in face-to-face interaction; (b) the integration of physical and computational components, which supports new interaction techniques [14]; and (c) an open architecture, which supports metadesign activities.

Figure 1 Meta-design in MAPS: Empowering caregivers to act as designers

A unique challenge of meta-design in the domain of cognitive disabilities is that the clients themselves cannot act as designers, but the caregivers must accept this role (see Figure 1). Caregivers, who have the most intimate knowledge of the client, need to be empowered to become the end-user designers. The scripts needed to effectively support users are specific for particular tasks, creating the requirement that the people who know about the clients and the tasks (i.e., the local caregivers rather than a technologist far removed from the action) must be able to develop scripts.

Figure 2 shows three different but integrated components of the EDC: • the action space (bottom pane), in which stakeholders engage in participatory problem solving and decision making related to urban planning issues that are of concern to all participants; • the reflection space (top left pane), in which task-relevant information is displayed for the design created in the action space;

Caregivers generally have no specific professional technology training, nor are they interested in becoming computer programmers. Based on extensive end-user support, MAPS allows caregivers to design complex multimodal prompting sequences (including sound, pictures, and video to be assembled by using a film-strip-based scripting metaphor).

• embedded visualizations using Google Earth (top right pane), in which the impact of new buildings are shown. The vision of the EDC is to provide contextualized support for reflection-in-action [36] within collaborative design activities. In our research with the EDC during the last decade, we have extended the following boundaries:

The design of MAPS involves three different groups of participants: assistive technology professionals and special education teachers, parents of clients, and professional caregivers. MAPS was tested with representatives of several different groups, resulting in the identification of the following requirements for meta-design: (a) discover and learn about the client’s and caregiver’s world and their interactions; (b) observe and analyze how tasks and learning of tasks were currently conducted; (c) understand and explicate the process of creating and updating scripts; (d) comprehend and analyze the process of using the scripts with a real task; and (e) gain an understanding of the role of meta-design in the dynamics of MAPS adoption and use.

• Transcending limited engagement with cultures of participation. Participants are more readily engaged if they perceive the design activities as personally meaningful by associating a purpose with their involvement [7,35].

By designing the MAPS environment to enable script redesign and reuse, caregivers were able to create an environment that matched the unique needs of individuals with cognitive disabilities. MAPS represents an example for extending boundaries by supporting meta-design, embedding new technologies into socio-technical environments, and helping people with cognitive disabilities and their caregivers have more interesting and more rewarding lives. Supporting Collaborative Design with the Envisionment and Discovery Collaboratory

• Figure 2. Different components of the EDC: action space, reflection space, and visualization support

The Envisionment and Discovery Collaboratory (EDC) [4] is a long-term research platform that explores conceptual frameworks for extending boundaries in the context of

172

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Participants must be able to naturally express what they want to say [28], and the interaction mechanisms offered must have a “low threshold” for easy participation and a “high ceiling” for expressing sophisticated ideas [38].

models. In addition, the environment supports tagging, ratings, and reviews by the participating community. Interested users can utilize 3D Warehouse for creative collaborations by sharing, downloading, modifying, extending, and reusing existing models. Google Earth has the capability to show objects contained in 3D Warehouse. shows Downtown Denver, Colorado, in 3D as an example.

• Transcending group think by involving stakeholders with different interests and different knowledge. More creative solutions to problems can emerge from the collective interactions with the environment by heterogeneous communities (such as communities of interest [17], which are more diverse than communities of practice [25,46]). Boundary objects are needed [41] to establish common ground and establish shared understanding for communities of interest.

In an ongoing collaboration with our partners from the Google Boulder office in Colorado, we are extending boundaries by pursuing the following research issues. • Allowing users to act as active contributors to 3D Warehouse requires extensive learning support to achieve sufficient mastery of the tools and environments provided.

Obstacles to further investigate the above objectives rest with the difficulties of democratizing the design of the EDC [45] by providing more control to the participants. Currently, EDC participants have to customize the system at the source-code level to reflect the specific characteristics of the city and its urban planning problem. Support for end-user developments is important for the EDC, because each urban planning problem is unique: it has to take into consideration the geography, culture, and population of specific cities. In most cases, EDC developers (the metadesigners) do not have sufficient knowledge of the problem and the social context; they do not know which issues are of greatest concern to the city planners and citizens and which conflicts need to be resolved through the EDC system.

• Assessing the effectiveness of different reward structures (recognition by the community and featuring the best models in 3D Warehouse and Google Earth) for motivating users to participate in the collaborative effort to model the whole world. • Supporting richer ecologies of participation, including roles such as creators, raters, curators, power users, and local developers, while attending to the diversity and independence of participants. Smart Grids and Energy Sustainability

Emerging changes in renewable and sustainable energy include rethinking how electricity is produced, transmitted, distributed, and consumed. The Smart-Grid vision [12] combines electrical and intelligence infrastructures. In March 2008, Boulder, Colorado, was selected to serve as the first Smart-Grid City in the United States. The vision and objective behind Smart-Grid Cities concept is focused on a number of infrastructure upgrades and customer offerings, including:

Modeling the World in 3D: SketchUp, Building Maker, 3D Warehouse, and Google Earth

Having the whole world modeled in 3D and allowing users to explore this virtual world on their computers is the objective behind Google’s effort to integrate SketchUp, Building Maker, 3D Warehouse, and Google Earth. The amount of work and local knowledge needed to model billions of buildings is beyond the scope and capability of any locally operating development team. It requires the contributions of a large user base, and as such represents a unique, large-scale example for assessing the conceptual framework underlying meta-design and cultures of participation.

• transformation of the existing metering infrastructure to a robust, dynamic electric system and a communication network for two-way communication throughout the distribution grid;

SketchUp and Building Maker are 3D-modeling environments (http://sketchup.google.com/). SketchUp allows users to develop sophisticated and highly creative models, but it requires a substantial learning effort. Building Maker is a tool for creating simpler models from aerial images with less effort. Powerful learning mechanisms for these systems are critical to allow everyone who wishes to contribute to learn how to do so. The 3D Warehouse is an information repository for the collection of models created by all users who are willing to share their models (see http://sketchup.google.com/3dwarehouse/). It contains tens of thousands of models from different domains, including buildings, houses, bridges, and so forth, and it uses collections to organize

Figure 3. Downtown Denver, Colorado, in 3D

173

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• conversion of substations to “smart” substations capable of remote monitoring, near real-time data, and optimized performance; and

complemented by designing socio-technical environments focused on reflective communities. There is an urgent need for understanding, stimulating, fostering, and supporting creative co-production with meta-design and cultures of participation beyond support for solitary activity.

• installation of programmable in-home control devices (at the customer’s invitation) and the necessary systems for more intelligent home energy use.

As cultures evolve, specialized knowledge becomes favored over generalized knowledge [39]. Taking advantage of the distances (spatial, temporal, conceptual, and technological) discussed earlier in this paper represents promising developments for making as many voices as possible heard, exploiting the symmetry of ignorance between different communities of practice with communities of interests [20], and creating shared understanding among stakeholders from different disciplines [40]. Specifically, the EDC is an attempt to extend boundaries by addressing these challenges.

The Smart-Grid vision represents a challenging application domain for meta-design and cultures of participation. Our research [13] is grounded in the basic assumption that to harvest the benefits of Smart Grids, the underlying technologies are necessary but they are not sufficient. To take advantage of Smart Grids, humans must be engaged as active decision makers, and not only as passive consumers. The socio-technical environments must identify and offer the right balance between “automating” and “informating” [30,48]. Taking advantage of the potential of the new emerging technologies will require new mindsets that lead to changes in the way users (as individuals, communities, organizations, and governments) think and learn about energy sustainability.

Understanding the Impact of Personally Meaningful Activities

Actively contributing in cultures of participation, even when supported by powerful meta-design environments, requires a substantial amount of learning and engagement. All humans have a limited amount of time and attention, so they have to carefully choose the activities in which they decide to become active contributors. Socio-technical environments can fail in two directions:

DISCUSSION AND REFLECTIONS

A promising methodology to extend boundaries in HCI research is to exploit the synergistic efforts by applying and testing the conceptual frameworks (meta-design and cultures of participation) in the socio-technical environments discussed in the previous section and in return modifying and expanding the conceptual frameworks by the demands for specific applications. provides a summary how the four different application domains extend boundaries. The following subsections describe a number of themes that originated from these synergistic efforts.

• By forcing participants to act as designers when they prefer to be consumers; this is the case in personally irrelevant activities (depending on the individual, examples of such environments might be having to check one’s own baggage at airports or check out one’s own groceries at supermarkets); • By limiting participants to consumer roles when they

Transcending the Unaided Individual Human Mind

want to be active contributors and designers; this is the case in personally relevant and meaningful activities (examples of such activities for different individuals and communities are provided by the four application domains discussed in the previous section—providing more independence for a family member with a cognitive disability by using the MAPS system represents a personally meaningful problem for the caregiver).

The primary HCI objective of the past to support and educate individual reflective practitioners [36] needs to be Application CLever EDC

3D Modeling

Smart Grids

Extending Boundaries universal design; caregivers are empowered to be active developers support for small groups in face-to-face meetings with table-top computing, boundary objects, visualizations, and simulations; individual artifacts (urban planning environments) are complex and wicked problems decentralized mass collaboration, rich ecology of participation; the complexity is primarily in the aggregate (hundred thousands of models exist in the 3D Warehouse) environments to support, inform, and motivate users to act as active decision makers in the use of energy

Motivation, Control, Ownership, Creativity, and Quality of Artifacts

Meta-design and cultures of participation create a new understanding of motivation, creativity, control, ownership, and quality [6]—topics that have not been considered of great importance for HCI in the past, but which enter center stage in extending the boundaries for the next generation of HCI systems. Motivation

Human beings are diversely motivated beings. We act not only for material gain, but also for psychological wellbeing, for social integration and connectedness, for social capital, for recognition, and for improving our standing in a reputation economy. The motivation for going the extra

Table 2: Summary of how the different applications extend boundaries

174

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

step to engage in active participation was articulated thus [35]: “The experience of having participated in a problem makes a difference to those who are affected by the solution. People are more likely to like a solution if they have been involved in its generation; even though it might not make sense otherwise.” There is widespread evidence that people attach more value to things they create themselves (the so-called “Ikea effect” [5]). Meta-design relies on intrinsic motivation for participation; it offers a new platform for human connection by bringing together otherwise unconnected individuals and replacing common background or geographic proximity with a sense of well-defined and shared purpose. Motivation is critical in the framework and systems discussed in this paper because participation cannot be enforced, only encouraged, fostered, and supported (as indicated by the proverb: “one can lead a horse to water but one cannot make it drink”). Research in motivation transcends usability concerns (what people can and cannot do) by focusing on what people want to do [11].

of those around them) [43], (c) decentralization (participants are able to specialize and draw on local knowledge) [3], and (d) aggregation (mechanisms exist for turning individual contributions into collections). By focusing on open systems and on the unfinished, meta-design and cultures of participation create architectures and seeds that can be evolved on an ongoing basis.

Control

CONCLUSIONS

The importance of the distribution of control has been emphasized as important for architecture [1]: “I believe passionately in the idea that people should design buildings for themselves. In other words, not only that they should be involved in the buildings that are for them but that they should actually help design them.” Meta-design and cultures of participation distribute control among all stakeholders in the design process and are based on the observation that shared control will lead to more innovation [45]: “Users that innovate can develop exactly what they want, rather than relying on manufacturers to act as their (often very imperfect) agents.”

Democratization of the development and evolution of personally meaningful socio-technical environments is more than letting users customize their applications; it represents a fundamental new challenge allowing and supporting the co-creation of situational applications. Empowering and motivating users to become active contributors and decision makers is a grand challenge problem for HCI research. In doing so, we should not ignore that boundaries are in many contexts important means for distinguishing communities and strengthening their identity. The challenge is not ignoring or eliminating boundaries, but respecting them and making them traversable as needed to explore new approaches and new frontiers.

Quality of Artifacts

Quality assurance in systems created by cultures of participation is often grounded in the following concern: “How are we to know that the content produced by widely dispersed and qualified individuals is not of substandard quality?” An interesting case study to reflect upon this concern was done in the context of Wikipedia. The journal Nature has compared the quality of articles found in the Encyclopedia Britannica with those in Wikipedia [23] and has come to the conclusion that Wikipedia comes close to Britannica in terms of the accuracy of its science entries.

Ownership ACKNOWLEDGMENTS

Our experiences gathered in the context of the design, development, and assessment of our systems indicate that meta-design methodologies are less successful when users are brought into the process late (thereby denying them ownership) and when they are “misused” to fix problems and to address weaknesses of systems that the developers did not fix themselves. Meta-design works when users are part of the participatory design effort in establishing a meta-design framework, including support for intrinsic and extrinsic motivation, user toolkits for reducing the effort to make contributions, and the seeding of use communities in which individuals can share their contributions.

I thank the members of the Center for LifeLong Learning & Design (L3D) at the University of Colorado, who have made major contributions to the ideas described in this paper. I specifically would like to thank: Elisa Giaccardi, who is the co-developer of our meta-design framework; Stefan Carmien, who created and evaluated the MAPS environment; Hal Eden, who is the major architect and developer of the EDC; John Bacus from Google Boulder, who provided unique insights into the design of the 3D modeling effort; and Holger Dick and Hal Eden, who are developing the Smart Grid environment. The research was supported in part by (1) grants from the National Science Foundation, including: (a) IIS-0613638 “A Meta-Design Framework for Participative Software Systems,” (b) IIS-0709304 “A New Generation Wiki for Supporting a Research Community in ‘Creativity and IT’,” and (c) IIS-0843720 “Increasing Participation and Sustaining a Research Community in ‘Creativity and IT’”; (2) a Google research award “Motivating and Empowering Users to Become Active Contributors: Supporting the Learning of High-Functionality Environments”; and (3) a

Creativity

Where do new ideas come from in meta-design environments and cultures of participation? Their creativity potential is grounded in user-driven innovations, taking advantage of breakdowns as sources for creativity, and exploiting the symmetry of ignorance [20]. To increase social creativity requires: (a) diversity (each participant should have some unique information or perspective), (b) independence (participants’ opinions are not determined by the opinions

175

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

major grant “CLever: Cognitive Levers: Helping People Help Themselves” from the Coleman Institute for Cognitive Disabilities at the University of Colorado.

14. Eden, H. Getting in on the (inter)action: Exploring affordances for collaborative learning in a context of informed participation. In G. Stahl (Ed.), Proc. CSCL '2002 Conference, Boulder, Colorado (2002), 399-407.

REFERENCES

15. Ehn, P. Manifesto for a digital bauhaus. Digital Creativity 9, 4 (1998), 207-216.

1.

2.

Alexander, C. The state of the art in design methods. In N. Cross (Ed.), Developments in Design Methodology, John Wiley & Sons, New York, 1984, 309-316.

16. Engeström, Y. Expansive learning at work: Toward an activity theoretical reconceptualization. Journal of Education and Work 14, 1 (2001), 133-156.

Andersen, R. and Mørch, A. Mutual development: A case study in customer-initiated software product development. In V. Pipek, M.B. Rossen, B. deRuyter, and V. Wulf (Eds.), End-User Development, Springer, Heidelberg, 2009, 31-49.

3.

Anderson, C. The Long Tail: Why the Future of Business Is Selling Less of More. Hyperion, New York, 2006.

4.

Arias, E.G., Eden, H., Fischer, G., Gorman, A., and Scharff, E. Transcending the individual human mind— Creating shared understanding through collaborative design. ACM Transactions on Computer HumanInteraction, 7, 1 (2000), 84-113.

5.

Ariely, D. The Upside of Irrationality – the Unexpected Benefits of Defying Logic at Work and at Home..HarperCollins, New York, 2010.

6.

Benkler, Y. The Wealth of Networks: How Social Production Transforms Markets and Freedom. Yale University Press, New Haven, CT, USA, 2006.

7.

Brown, J.S., Duguid, P., and Haviland, S. Toward informed participation: six scenarios in search of democracy in the Information Age. The Aspen Institute Quarterly 6, 4 (1994), 49-73.

8.

Carmien, S. Socio-Technical Environments Supporting Distributed Cognition for Persons with Cognitive Disabilities. Ph.D. dissertation, University of Colorado at Boulder. http://l3d.cs.colorado.edu/~carmien/.

9.

Carmien, S., Dawe, M., Fischer, G., Gorman, A., Kintsch, A., and Sullivan, J.F. Socio-technical environments supporting people with cognitive disabilities using public transportation. Transactions on HumanComputer Interaction (ToCHI) 12, 2 (2005), 233-262.

17. Fischer, G. Communities of interest: Learning through the interaction of multiple knowledge systems, In Proc. IRIS'24, Ulvik, Norway (2001), 1-14. 18. Fischer, G. Social creativity: Turning barriers into opportunities for collaborative design. In F. deCindio and D. Schuler (Eds.), Proc. PDC’04, CPSR, University of Toronto, Canada (July 2004), 152-161. 19. Fischer, G. End-user development and meta-design: Foundations for cultures of participation. Journal of Organizational and End User Computing 22, 1 (2010), 52-82. 20. Fischer, G., Ehn, P., Engeström, Y., and Virkkunen, J. Symmetry of ignorance and informed participation. In Proc. PDC’02, Malmö University, Sweden (2002), 426-428. 21. Fischer, G., and Giaccardi, E. Meta-design: A framework for the future of end user development. In H. Lieberman, F. Paternò, and V. Wulf (Eds.), End User Development. Kluwer Academic Publishers, Dordrecht, The Netherlands, 2006, 427-457. 22. Fischer, G., Nakakoji, K., Ostwald, J., Stahl, G., and Sumner, T. Embedding critics in design environments. In M.T. Maybury and W. Wahlster (Eds.), Readings in Intelligent User Interfaces. Morgan Kaufmann, San Francisco, 1998, 537-559. 23. Giles, J. Internet encyclopaedias go head to head. Nature 438 (2005), 900-901. 24. Henderson, A., and Kyng, M. There's no place like home: Continuing design in use. In J. Greenbaum and M. Kyng (Eds.), Design at Work: Cooperative Design of Computer Systems, Lawrence Erlbaum Associates, Inc., Hillsdale, NJ, USA, 1991, 219-240.

10. Carmien, S.P., and Fischer, G. Design, adoption, and assessment of a socio-technical environment supporting independence for persons with cognitive disabilities. In Proc. CHI 2008, ACM Press (2008), 597-607.

25. Janis, I. Victims of Groupthink, Houghton Mifflin, Boston, MA, USA, 1972. 26. Leadbeater, C., and Miller, P. The Pro-Am Revolution – How Enthusiasts Are Changing Our Economy and Society, 2008. http://www.demos.co.uk/files/proamrevolutionfinal.pdf

11. Csikszentmihalyi, M. Flow: The Psychology of Optimal Experience, HarperCollins Publishers, New York, 1990. 12. Department of Energy, Smart Grid. http://www.oe.energy.gov/smartgrid.htm.

27. Mumford, E. A socio-technical approach to systems design. Requirements Engineering, 2000, 59-77.

13. Dick, H., Eden, H., and Fischer, G. Energy sustainability and Smart Grids: Fostering and supporting cultures of participation in the energy landscape of the future. (submitted for publication, 2010).

28. Myers, B.A., Ko, A.J., and Burnett, M.M. Invited research overview: End-user programming. In Proc. Human Factors in Computing Systems, CHI'2006, Montreal (2006), 75-80.

176

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

29. National Research Council. Beyond Productivity: Information Technology, Innovation, and Creativity, National Academy Press, Washington, DC, 2003.

39. Simon, H. A. The Sciences of the Artificial, 3rd ed. The MIT Press, Cambridge, MA, USA, 1996.. 40. Snow, C.P. The Two Cultures, Cambridge University Press, Cambridge, UK, 1993.

30. Norman, D.A. Things That Make Us Smart AddisonWesley, Reading, MA, USA, 1993. 31. O'Reilly, T. What Is Web 2.0 – Design Patterns and Business Models for the Next Generation of Software. http://www.oreillynet.com/pub/a/oreilly/tim/news/ 2005/09/30/what-is-web-20.html.

41. Star, S.L. The structure of ill-structured solutions: Boundary objects and heterogeneous distributed problem solving. In L. Gasser and M.N. Huhns (Eds.), Distributed Artificial Intelligence, Volume II, Morgan Kaufmann, San Mateo, CA, USA, 1989, 37-54.

32. Porter, J. Designing for the Social Web. New Riders, Berkeley, CA, USA, 2008.

42. Suchman, L.A. Plans and Situated Actions. Cambridge University Press, Cambridge, UK, 1987.

33. Preece, J., and Shneiderman, B. (2009) The reader-toleader framework: Motivating technology-mediated social participation. AIS Transactions on HumanComputer Interaction 1, 1 (2008), 13-32.

43. Surowiecki, J. The Wisdom of Crowds. Anchor Books, New York, 2005. 44. Tapscott, D., and Williams, A.D. Wikinomics: How Mass Collaboration Changes Everything, Portofolio, Penguin Group, New York, 2006.

34. Raymond, E.S., and Young, B. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. O'Reilly & Associates, Sebastopol, CA, USA, 2001.

45. von Hippel, E. Democratizing Innovation, MIT Press, Cambridge, MA, 2005. 46. Wenger, E. Communities of Practice – Learning, Meaning, and Identity. Cambridge University Press, Cambridge, UK, 1998.

35. Rittel, H. Second-generation design methods. In N. Cross (Ed.), Developments in Design Methodology, John Wiley & Sons, New York, 1984, 317-327.

47. Winograd, T., and Flores, F. Understanding Computers and Cognition: A New Foundation for Design Ablex Publishing Corporation, Norwood, NJ, USA, 1986.

36. Schön, D.A. The Reflective Practitioner: How Professionals Think in Action, Basic Books, New York, 1983.

48. Zuboff, S. In the Age of the Smart Machine, Basic Books, Inc., New York, 1988.

37. Shirky, C. Here Comes Everybody: The Power of Organizing without Organization. Penguin Books, New York, 2008. 38. Shneiderman, B. Creativity support tools: Accelerating discovery and innovation. Communications of the ACM 50, 12 (2007), 20-32.

177

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Wattsup?: Motivating reductions in domestic energy consumption using social networks 1

Derek Foster1, Shaun Lawson1, Mark Blythe2 and Paul Cairns2 2 Lincoln Social Computing Research Centre Department of Computer Science University of Lincoln University of York Brayford Pool, Lincoln Heslington, York {defoster, slawson}@lincoln.ac.uk {mblythe, pcairns}@cs.york.ac.uk

ABSTRACT

energy at home and fly too often. And yet they do next to nothing to change this. Indeed a recent article in the Ecologist on unnecessary travel was written on board a transatlantic Boeing 747 flight [30]. At the moment we want to cut carbon emissions without changing our lifestyles; we do not want sustainable technologies but perhaps we want to want them. How then can technologies designed to change behaviour become compelling, desirable and enjoyable?

This paper reports on the design, deployment and evaluation of “Wattsup”, an innovative application which displays live autonomously logged data from the Wattson energy monitor, allowing users to compare domestic energy consumption on Facebook. Discussions and sketches from a workshop with Facebook users were used to develop a final design implemented using the Facebook API. Wattson energy monitors and the Wattsup app were deployed and trialled in eight homes over an eighteen day period in two conditions. In the first condition participants could only access their personal energy data, whilst in the second they could access each others‟ data to make comparisons. A significant reduction in energy was observed in the socially enabled condition. Comments on discussion boards and semi-structured interviews with the participants indicated that the element of competition helped motivate energy savings. The paper argues that socially-mediated banter and competition made for a more enjoyable user experience.

It is increasingly recognised that interaction design must address issues of sustainability, e.g. [4, 24, 33]. There has been much work in HCI in the past decade on persuasive technology, e.g. [9, 18, 23]. However Fogg recently pointed out that persuasive technologies very often fail and urged practitioners to think small [19]. Monitoring technologies alone (e.g. pedometers) are often not enough to make meaningful changes in behaviour. This paper draws on previous work on persuasive technologies, e.g. [18, 19] in order to address environmental concerns over domestic energy consumption. Households are responsible for 30% of the UK‟s total energy use [12]. Since 1970, household energy demands have grown by 32% [29] and still continue to grow. Energy demands have grown because of increases in home temperature and the proliferation of consumer electronics [14]. Ulrick Beck identifies increasingly individualized forms of living such as: “living alone, single parenthood, non-marital cohabitation, childless marriage, serial marriage” and “living apart together”, where partners live in separate dwellings” [2]. The impacts of these trends are social but also environmental in that one person living alone uses more energy than two in the same household [3]. Rising energy consumption currently means increased CO2 emissions so domestic energy consumption is very much a world problem e.g. [20, 46].

Author Keywords

Sustainability, Persuasive Technology, Social Networking, Competitive Energy Saving, User Experience ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

It is generally acknowledged amongst scientists and, increasingly, politicians and corporations that current levels of energy consumption are not sustainable [25]. Many people already know very well that they consume too much Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

This paper reports on the design, deployment and evaluation of a Facebook application which aimed to encourage energy saving by using live and historical energy feedback in a social-normative context.

178

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

BACKGROUND

numerical display that can show units like watts and pounds sterling.

Numerous studies have demonstrated that energy usage falls when people know it is being monitored [46]. Studies in Environmental Psychology have shown that feedback on energy consumption can achieve behavioural change though it is not necessarily sustained without timely reminders [11, 16]. The goal of this project therefore, was not just to effect behavioural change but to demonstrate larger reductions in energy consumption through the addition of a social aspect in monitoring energy usage. The desire to belong and willingness to adapt behaviour to follow what others are doing has been seen as a fundamental motivator [1]. Social norms such as peer pressure have also been seen as a means of changing behaviour to align with the ideas or beliefs of groups [41]. It has been argued that computers now operate as social actors designed to influence our behaviour, filling the roles of teachers, sales people and health agents and [17, 18]. Computers have, for example, been shown to help people overcome their fear of public speaking [43]. Recent work on persuasive technology has argued that reciprocal interaction through instant messaging and “pokes” can be effective strategies for persuasion [47]. Whilst there is insufficient space to give a full account of the psychological theories of social motivation, effecting behavioural change through computer mediated social networks seems promising.

Figure 1: The Wattson energy monitor from DIY Kyoto

Smart monitors such as the Wattson are increasingly common and The UK government has committed to replacing all current electricity metres with smart metres by 2020. This is no guarantee that it will happen of course, but it indicates that the availability of such devices is likely to increase. Facebook

The social networking site Facebook now has over 250 million active users [15]. If the site were a country its population would be greater than that of Russia. Studies of Facebook have demonstrated that users read other people‟s postings, play games, upload comments on photographs and add to their own „profile‟ many times daily [27]. These sites provide a powerful means of delivering small, asynchronous applications to peer groups of likeminded real-world friends in a manageable and pleasant way. There may then be potential in leveraging the engaging power of small applications, offering rich social interactive features to change behaviour.

Energy Monitors

In many homes, electricity meters are often difficult to access (located in cupboards or corners) and usually difficult to read. Energy measurements in kilowatts are difficult to make sense of either in terms of finance or ecological impact [8]. Dynamically updating kilowatt readings are however much more understandable [48]. Studies of paper electricity bills have also shown that awareness and understanding can be increased through graphical representations of financial and normative comparisons [10]. However bills do not provide frequent feedback and a recent study found that continuous feedback over a long period is the best means of changing patterns of energy consumption [16]. The importance of raising awareness about patterns of consumption can be illustrated with reference to still widely held but mistaken beliefs about domestic appliances. One study of mistaken folk theories of energy consumption found that there was still credence given to such myths as - turning a thermostat up higher than necessary heats a room up quicker, and leaving computers and lights on consumes less energy than turning them on and off frequently [28].

Feedback Studies

In recent years there have been many studies of innovative ways of displaying energy consumption. The “power aware cord” offers per-appliance feedback in the form of a glowing power cord but does not indicate energy used or cost [21]. The use of portable and stationary, minimalist direct displays for the home was investigated in another recent study [48]. This found that occupants would move around the house turning appliances on and off to monitor the change of feedback on the portable display they carried and enjoyed this playful approach (ibid). The Phillips iCat study used a robot with happy or sad facial expressions corresponding to low or high energy use scenarios in a lab [37]. Although the study was limited to the lab and involved the use of a very expensive robot it indicated that participants responded more positively to the playful expression of the cat rather than a bar chart [22].

The Wattson home energy monitor is a standalone device which is designed to raise awareness of domestic energy consumption. It is an off the shelf technology which takes readings from electricity metres and displays the information as real-time energy usage data. It has a light system that provides ambient feedback as well as a

The two previously discussed studies did not employ the use of any normative feedback, social or otherwise. But some work has been carried out in using social platforms to motivate people to reduce their ecological impact. Mankoff

179

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

[34] proposed the use of web widgets like “badges” showing carbon footprints which could be incorporated on platforms like MySpace but the proposed implementation relied on users self-reporting their energy information. A recent study used a large situated display to compare energy consumption between dormitories of a university campus showing expenditure by both halls and individuals. The display was only updated bi-weekly yet resulted in massive reductions in electricity and water [36]. A Facebook group was created for the study but did not have a high uptake. A Facebook group network can only be fed updates in the form of self-reported textual or static image posts with little guided or sophisticated interaction. Facebook applications provide a more personalised experience through access to the users profile, profile actions and news feed. A Facebook application displaying personal student energy usage tied into each student‟s unique Facebook account profile may have been more effective.

Figure 2: WattsOn and Nabaztag YouTube Video

Participants commented on the comic potential of the device and the possibility of fun, playful interactions. This generated general discussions around the concept design of a socially enabled home energy monitor and what that might look like.

This study aimed to address a gap in current work on leveraging social platforms by embedding live, continuous energy data into a fully interactive socially-enabled energy application. Using the Facebook Developers Kit (FDK) API, Wattson devices were linked to Facebook allowing us to investigate whether sharing such information between friends might make for further reductions in energy consumption.

Pencils, coloured pens, scissors and other craft materials were also provided allowing participants to create their own interface elements.

DESIGN PROCESS

Focus groups were conducted with a convenience sample [40] of four Facebook users aged between twenty three and thirty eight. There were three males and one female and all were responsible for paying the energy bills in their homes. Discussions took place in a home lab on campus at a university and helped the participants focus on the home as a design space. It should be noted that as the participants were students on a Masters course in HCI they were more sensitive to issues of interface design than other groups might be.

Figure 3: Initial Sketches developed in focus group

A large number of ideas were generated and discussed. Various graphical metaphors were suggested such as a mouse running in a cage turning faster or slower depending on energy consumption, balloons with user‟s faces on them were pictured being inflated larger or smaller and floating higher or lower to indicate ranked consumption rates (see figure 3. There were a number of interesting suggestions. For instance: “a digital photo frame, if you aren’t using much energy then you see pictures of beautiful scenery or if you’re using a lot some stark scenery, almost like a piece of art that changes over time”. Smiley and sad faces were also suggested as simple but very easily understood graphical elements (see Figure 5).

Participants were shown the Wattson monitor and a selection of YouTube videos related to energy consumption. One video featured a Wattson monitor over a period of a few minutes displaying live home energy readings, from low to high when a cooker is switched on; simultaneously, a “Nabaztag” outputs a vocal message that a lot of energy is being used at that moment and flashes its lights.

The participants were then given a paper materials pack containing various paper interface elements such as user avatar icons, energy icons and CO2 icons to help create prototype interfaces. Plain interface template cards were added to the pack enabling the participants to place the interface elements on the blank interface templates.

180

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 4: Interface elements given to participants

Much of the discussion revolved around the difficulty of relating to the kilowatt as a unit of energy measurement. “Kilowatts, watts, I don’t want to see any of that, money yes.” There were interesting debates about whether financial representations were more powerful than ecological ones “I’m thinking a kind of erm some form of visual presentation....you’ll see more trees over time in an image if you use less energy, see trees knocked down if you use more” There were also interesting general discussions of issues such as privacy when sharing data raising concerns such as: “The risk of failure in front of your friends.” However it was generally agreed that introducing a competitive element between friends who were free to opt in or out of the group might help drive a reduction in consumption.

Figure 5: My Energy: Workshop sketches and final design

Sketches developed in the workshop (see the left hand side of figure 5) were developed into final designs (see the right hand side of figure 5). Workshop comments also informed the design e.g.: “What the dials can do is give you a comparison, although they are very abstract, I can still tell if I have used loads of energy today.” The final design then related quite directly to initial discussions. The Friends screen would display personal energy consumption against selected friends.

IMPLEMENTATION

Moving on from the conceptual design stage was made relatively straight forward due to the high quality of the user generated designs. Following discussions in the focus group it was decided that the main interface attributes for displaying energy would be expressed in Watts and pounds sterling (£) as well as CO2 emissions measured by weight. In addition to numerical representations, a graphical representation was selected to display alongside both numerical values for energy and Co2 emissions in the form of the happy/sad face theme as previously discussed. Three core interfaces were developed to provide an engaging user experience: My Energy, Friends and Rankings. The My Energy screen would show energy consumption with a dial visualisation and a seven day history bar chart.

Figure 6: “Friends” screen final design

Again this final design built directly on ideas and comments from the workshop “I can see it working amongst a group of friends, but finding a way to notify users of the group of whats happening’”.

181

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The Rankings screen would show a table of highest and lowest energy users of the application.

Figure 7; Rankings Screen, Initial workshops sketches and final design.

Figure 8: Technical Design Overview

Again there was a very direct link between the final design and workshop suggestions such as “„I thought about a leagues table based on points, slightly competitive but not with pressure and a bit where people can discuss it’” and rough sketches such as those on the left hand side of Figure 7. The ranked pyramid of the heaviest consumers suggested in the workshop (see the bottom left hand side of Figure7) was simplified to a ranking table similar to that imagined in another workshop sketch (top left hand side of Figure 7).

The diagram illustrates the Wattson sensor at the householders fuse box sending the current energy reading by radio signal to the Wattson energy monitor which stores the data in its 30 day dedicated memory. The Wattson is physically attached to a PC via a USB cable which transports the energy data from the Wattson to the PC via the desktop “Powometer” application where it is stored in a local SQL mobile database. At configured intervals Powometer sends the stored energy data to the myenergyusage.org service via the internet where it is stored in multiple MySQL databases for redundancy. Once the energy data is stored on the myenergyusage.org databases it is then presentable to authorised applications to make use of it. The Wattsup technical design and implementation tasks were approached using Evolutionary Prototyping [38].

Another important interface element was the integration of a comments board. The rankings table would allow users to visualise what their standing was against others but it would not facilitate friends commenting on personal or others‟ energy consumption. This feature was added following discussions of the importance of interacting with friends through such an application. A cloud tag feature was also added where users could note the devices they thought were high energy users.

EXPERIMENTAL METHOD Aim

As with any prototype design, choices were limited by time and material constraints. A number of interesting concepts were generated in the workshop which were not feasible within the scope of the project. As is well documented in the participatory design literature, the sketching process was invaluable not only in generating the final designs but also in recognizing a wider design space e.g. [6].

The aim of the study was to see if energy savings could be increased by the addition of a social element to monitoring. As it is already known that monitoring can reduce energy consumption the focus was on the social element and we did not consider households without any energy monitors of any kind. To this end, we made a socially enabled version of the Wattson energy monitor via Facebook. The hypothesis was that less energy would be used whilst the Wattson was socially enabled than when it was not socially enabled.

DEPLOYMENT

The application was developed using ASP.NET, Microsoft C#, HTML/CSS, XML, MySQL, Facebook Markup Language, and the FDK API. When completed it was deployed to the Facebook application platform.

Participants

Eight households were recruited to trial the prototype with each participant being the person responsible for paying the electricity bill. The number of participating households was limited by the number of devices available to the researchers. Selection of the participants followed a purposive sampling method [40]. The criteria for

To afford the functionality displaying live and recent energy data within a Facebook application, an elaborate application framework was designed and implemented as seen in Figure 8.

182

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

recruitment were that the lead participant in each household must be responsible for paying their household electricity bill and be a daily user of the Facebook website. None of the households had owned any type of energy monitor prior to the study. Participant

Profession

Age

No. Household Members

Diane Alice

Nurse Nurse Office Admin Programmer School Teacher Student Student Writer

26 23

2 2

49

2

32

2

41

2

31 20 40

2 4 4

Rachael Christopher Robert David Richard Shirley

with the required desktop software one week before the experiment officially began. This was done to assist in reducing any effects on participants‟ energy usage by initially using the Wattson on its own, as it was likely to receive attention as a new gadget in the house (and, as previously noted, decrease energy consumption). Each participant gave their informed consent and carried out the experiment in their own home. The experiment took place over a period of 18 days with 9 days in each condition. Half of the participants started in condition A with the other half starting in condition B, after 9 days the participants were sent an email through Facebook informing them of the changeover of conditions. The applications were then reprogrammed to perform in the alternative conditions with the relevant participants. Data was collected in the Wattson device itself, a MS SQL mobile database using the installed desktop software and the myenergyusage.org web service via a MySQL database. The Google analytics service was also used to record the number of Facebook application page views for each interface.

Table 1: Wattsup participant demographics

The participants were also recruited in 4 pairs who resembled one another in circumstances as much as possible. For example if a participant belonged to a family of four then another participant belonging to a family of four was recruited. In total the participants belonged to households with 6 couples and 2 families of four, so twenty people in all were involved in this study. The lead participants had all been regular users of Facebook for at least one year and were all friends who were on one another‟s Facebook friends list. Additional demographical information on the participants is detailed in Table 1 with pseudonyms used in place of the participant‟s real names.

RESULTS

The energy usage, in kWH, in both conditions for each household is summarised in Figure 9.

Design

The experiment followed a within subjects design [7] with each participant taking part in two conditions or social modes. In condition A the Wattsup application was socially enabled, i.e. users could see their friends‟ data as well as their own, in condition B the Wattsup application was manipulated so that there were no social features i.e. users could only see their own energy usage. The households were divided into matched groups and the conditions were counterbalanced between the groups to avoid ordering effects [7]. Group 1 started in condition A, group 2 in condition B and the groups switched conditions halfway through.

Figure 9: Wattsup participant energy usage in each condition

A Wilcoxon test, for comparing repeated measures of nonparametric data, showed that energy consumption was significantly lower when using the socially enabled application (Z= -2.1, N=8, p=0.036).

The independent variable was therefore the Facebook application‟s social mode, either enabled or disabled. The dependent variable was the energy used in Kilowatt Hour units with a total measurement being taken in each condition for each household.

A total of 130Kw units of energy saved by the participants in condition A as opposed to condition B. This amount of energy would be expended by leaving a 60W bulb on for 9 days and result in Co2 emissions similar to those produced by driving an average-sized UK petrol car for 399Km.

Procedure

The experiment required hardware in the form of a Wattson energy monitor and a Windows based PC running the Powometer desktop software available from www.myenergyusage.org to collect energy data from the Wattson monitor. The Wattson monitor was installed along

Additional data collected from Google Analytics highlighted the differences in user interaction activity between both conditions. In terms of the number of times

183

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Semi-Structured Interviews

the participants visited Wattsup, condition A was significantly more popular than condition B with a total of 263 versus 51 page views respectively. The number of visits to the Facebook application then showed a fivefold increase in the social condition. Participants spent most time on the rankings interface viewing and commenting on the rankings table. It can be assumed that participants enjoyed this feature the most due to the collective amount of page views for rankings as well as the average time spent viewing it

Participants took part in semi structured interviews about their experience with the set uy when the Wattson device was collected from their homes at the end of the study. They were asked: which condition they preferred and why, whether they had any problems using Wattsup and whether they would use it or something similar over a longer period of time. All of the participants preferred the socially enabled condition “I preferred the second one (socially) because I am quite competitive, it gave me further incentive. I think putting a bit of fun in it is quite important.”. All of the participants said they would be very interested in using the system over a longer period of time. With regard to problems using the device some suggested a more visual representation of Co2 emissions.

However with such a low number of participants it is worth exploring the result further by analysing the qualitative data collected. QUALITATIVE DATA

In addition to the descriptive and inferential statistics qualitative data analysis was undertaken on the comments board and on semi structured interviews following the trial.

As with other energy reduction systems participants found monitoring itself quite enjoyable: “Well, this morning we unplugged EVERYTHING one by one, room by room. The reading on the wattson went down by around 450 to 970 watts. Ha, we actually enjoyed investigating this though ;)”

Comments board

The comments board proved to be a popular feature with participants and comments were analysed using a small scale grounded theory approach [44]. Three main categories of comment emerged: banter, engagement and competition. Banter included teasing such as “how come you are at the top, cut down drastically on your cups of tea?” And gloating “good to see I'm higher in the table then you rob ha ha” as well as provocations such as ““energy vampire...you clearly are!!””. This kind of banter extended to joking exchanges such as:

Although financial and ecological concerns may also be important factors this and other studies clearly indicate that fun should be taken seriously, eg [5]. DISCUSSION

One UK study claimed that sustained behavioural change with domestic energy consumption was unlikely to alter until more than 3 months had elapsed [11]. However, the energy feedback in that study was not delivered within a socially enabled context; therefore it is possible that the claim of 3 months minimum for energy usage behaviour change may not hold when a social platform is used to deliver the feedback. Due to time constraints and resources available this project could not address the experiment duration issue for sustained behaviour change. These findings then may be viewed as a pilot study potentially leading onto a larger and longer term study.

Diane: “I’ve turned my washing machine settings down as this uses loads...no pun intended” Alice: “I once turned down a washing machine. Wasn't pretty. It went into a cycle of depression and self-loathing before finally giving me my socks back” Engagement included disclosure of information such as “Left my main PC on the last two nights. Made a massive difference to my scores.” And disbelief about energy usage “gone down in the rankings? I´m in Spain :S” Engagement could indicate enjoyment “Woohooo, looking good today!” but sometimes also mixed feelings of guilt and disinclination to change “I NEED TO STOP PLAYING PUTER! (idontwanttoidontwanttoidontwantto)”.

But the approach may be criticised more generally. It could be argued, for instance, that computers are part of the problem and not the solution. Part of the reason for the massive increases in domestic energy consumption is the proliferation of computers and other energy intensive devices. In 1995 29% of the UK population owned a PC, in 2006 this had more than doubled to 65% [35]. The power supply of today‟s computers is around 300w and growing as technology progresses, significantly more than computers of the 1990s [39]. Increasingly sophisticated components such as graphics cards and processors are using more energy with each subsequent generation.

Other comments indicated pleasure in the competitive aspect of the rankings “Take’s the top spot: D” this seemed enjoyable even to those who were not necessarily winning “I've been usurped”. This competitive element occasionally led to questions “hey [name omitted] what’s your secret? your energy rating is pretty good...”. Comments were made when participants moved both up and down the ranking table indicating engagement in the process throughout.

It is difficult to find reliable figures estimating the carbon footprint of an entity like Facebook. Some estimates place

184

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

CONCLUSION

the number of servers necessary as 10,000 with an additional 300,000 user PCs connected at any one time [32, 42]. While all carbon footprint estimates are disputable the footprint of Facebook and its users is clearly large, perhaps, as some claim, the equivalent of a major city. The number of Facebook users is often compared to the populations of nations. If Facebook is a country you need a computer to live in it. Using Facebook to reduce energy consumption then may be rather like eating more pies in order to lose weight.

The paper has described the design, deployment and evaluation of a Facebook application designed to allow friends to compare their domestic energy consumption. The quantitative and qualitative analysis of the data collected from participants in this study suggests that social networking sites may be able to play a role in reducing energy consumption in the home by making monitoring more enjoyable. This was a small scale study and only a larger investigation could conclusively determine how effective such applications may be. However, these results are encouraging. Competitive carbon counting appears to be both more enjoyable and more effective than individual monitoring.

However there may be a value to endeavours such as this beyond an immediate and measurable net reduction of personal energy consumption. Such studies cannot help but raise awareness of energy consumption if only because of the Hawthorne effect. For ecological change to take place there must also be ideological change. This study generated a considerable amount of press interest [e.g. 45] which indicates that sustainability is now a very real concern of users. That said, the focus on individuals and personal energy consumption must not detract from larger political interventions. As Paul Dourish has argued we must raise awareness not just of the ecological consequences of leaving the lights on but the consequences of our decisions at the ballot box [13].

The limits of how much any individual can achieve by changing their own lifestyles are often pointed out. But collective behaviour change even on a small scale is increasingly recognized as a key to tackling global warming. The International Energy Agency, for instance, estimate that devices on standby cause a full one percent of world greenhouse emissions, this is nearly equal to that of the entire aviation industry[46].

It has been argued that in an age of potential ecological catastrophe user centred design is no longer appropriate [e.g. 4]. The user‟s needs and preferences should not take precedent over the impact of their technologies on the environment. However this is to configure the user solely as a consumer. Users may also be considered as citizens. Although some scientists continue to doubt that global warming can be attributed to human activity it is clear that a great many users / citizens are now entirely persuaded and want to do something about it. The impact of our technologies is a real concern at the level of user experience. User experience is already broadly conceived in terms of social and psychological perspectives, it is becoming clear that it must be conceived in still wider terms to address the ecological challenges of the coming years.

Social networking sites like Facebook and Twitter are increasingly being appropriated by users for political and social ends. Facebook is of course primarily for fun but it may be that the enjoyable aspects of the service make for effective platforms for persuasive technologies.

Slavoj Zizek has argued that enjoyment is a political factor in any social structure, even, or especially if it is extremely repressive. The jokes and private satires against the party in communist states were no threat to the regime, indeed they were entirely necessary for its smooth running [49]. Lovelock has argued that if we were to take the ecological challenges that we face seriously then we would adopt systems of rationing much harsher than those of the second world war [31]. Even if the cataclysmic predictions of climate change are wrong it is clear that the way we consume energy must change. If harsh measures are to be endured then perhaps we must find ways to enjoy them.

REFERENCES

ACKNOWLEDGEMENTS

The authors wish to express their sincere appreciation to the project participants and their families for their invaluable assistance in the research and development of this project‟s. Particular thanks also go to Crispin Proctor for giving permission and assistance in accessing energy data online from the myenergyusage.org web service. Thanks also goes to the staff of the Lincoln Social Computing Research centre for offering their valuable opinions on various aspect of the project‟s work. 1. Baumeister, R.F., and Leary, M.R.: The need to belong: Desire for interpersonal attachments as a fundamental human motivation. Psychological Bulletin 117 (1995), 497–529. 2. Beck U. Individualisation: institutionalized individualism and its social and political consequences. Sage London UK, 2002 3. BERR, Energy consumption in the United Kingdom – BERR. http://www.berr.gov.uk/files/file11250.pdf 4. Blevis E. Sustainable Interaction Design: Invention and Disposal, Renewal and Reuse. In Proc. CHI 2007, ACM Press (2007), 502-512

185

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

5. Blythe, M.A., Overbeeke, K., Monk, A.F. and Wright, P.C. Funology: from usability to enjoyment. Kluwer Academic Publishers, Dordrecht, Germany, 2003

greenhouse gases. Office for National Statistics, London, UK 2004 21. Gustafsson, A. and Gyllenswärd, M. The power-aware cord: energy awareness through ambient information display. Ext. Abstracts CHI 2005. ACM Press (2005), 1423-1426.

6. Buxton B. Sketching the User Experience: Getting the Design Right and the Right Design. Morgan Kaufman. Amsterdam, Holland, 2007 7. Cairns, P., and Cox, A.L Research Methods for HumanComputer Interaction. Cambridge University Press, Cambridge, UK, 2008.

22. Ham, J. and Midden, C. A robot that says bad!: using negative and positive social feedback from a robotic agent to save energy. In Proc. HRI 2009. ACM Press (2009), 265-266. 23. Ham, J., Midden, C., and Beute, F. Can ambient persuasive technology persuade unconsciously?: using subliminal feedback to influence energy consumption ratings of household appliances. In Proc. Persuasive 2009, vol. 350. ACM Press (2009)

8. Chetty, M., Tran, D., and Grinter, R. E. Getting to green: understanding resource consumption in the home. In Proc. UbiComp 2008, vol. 344. ACM Press (2008), 242-251. 9. Consolvo, S., Everitt, K., Smith, I., and Landay, J. A. Design requirements for technologies that encourage physical activity. In Proc CHI 2006, ACM Press (2006), 457-466.

24. Huang E., & Truong K. Breaking the Disposable Technology Paradigm: Opportunities for Sustainable Interaction Design for Mobile Phones. In Proc. CHI 2008, ACM Press (2008), 323-332

10. CSE, Consumer Preferences for Improving Energy Consumption Feedback. http://www.cse.org.uk/pdf/pub1033.pdf

25. IPCC, IPCC Fourth Assessment Report: Working Group I Report “The Physical Science Basis” http://www.ipcc.ch/ipccreports/ar4-syr.htm

11. Darby, S. The effectiveness of feedback on energy consumption: A review for DEFRA of the literature on metering, billing and direct displays, Environmental Change Institute, University of Oxford, Oxford, 2006.

26. James, N., & Desai, P. A Study into the Development of Sustainability Rating for Homes. Advisory Committee on Consumer Products and the Environment BioRegional Development Group. http://www.defra.gov.uk/environment/consumerprod/ac cpe/research/pdf/accpe_final030711.pdf

12. Defra, Department for Environment, Food and Rural Affairs: Autumn Performance Report 2006. http://www.defra.gov.uk/corporate/apr/apr2006.pdf 13. Dourish P., Print This Paper, Kill A Tree: Environmental Sustainability as a Research Topic for Human – Computer Interaction. Working paper from the Laboratory for Ubiquitous Computing and Interaction at UC Irvine, n.d.

27. Joinson, A. N. Looking at, looking up or keeping up with people?: motives and use of Facebook. In Proc. CHI 2008. ACM Press (2008), 1027-1036. 28. Kempton, W. Two Theories of Home Heat Control, Cognitive Science 10 (1986), 75-90

14. EST, Energy Saving Trust‟s Response to the DTI Energy Review Consultation. http://www.berr.gov.uk/files/file31091.pdf

29. Kinver, M. Saving Energy on the Home Front (BBC News – 23/10/06) http://news.bbc.co.uk/1/hi/sci/tech/6056398.stm

15. Facebook, Facebook | Statistics http://www.Facebook.com/press/info.php?statistics

30. Lee M. How Do You Define Unnecessary Travel. The Ecologist. June 2009, 71-72

16. Fischer, C. Feedback on household electricity consumption: a tool for saving energy? Energy Efficiency 1, 1 (2008), 79-104

31. Lovelock, J. The Revenge of Gaia. Basic Books, 2006 32. Malik O. Facebook‟s Insatiatible Hunger for Hardware. Gigacom. http://gigaom.com/2008/04/25/Facebooks-insatiablehunger-for-hardware/

17. Fogg, B. J. and Nass, C. Silicon sycophants: the effects of computers that flatter. Int. J. Hum.-Comput. Stud. 46 (1997), 551-561.

33. Mankoff, J., Blevis, E., Borning, A., Friedman, B., Fussell, S., Hasbrouck, J., Woodruff , A., and Sengers, P. Environmental Sustainability and Interaction. In Proc. CHI 2007, ACM Press (2007), 2121-2124.

18. Fogg, B.J. Persuasive Technology – Using Computers to Change What We Think and Do. Morgan Kaufmann, San Francisco, 2003. 19. Fogg, B. Three possible futures for persuasive technology. In Proc. Persuasive 2009, ACM Press (2009)

34. Mankoff, J., Matthews, D., Fussell, S. R., and Johnson, M. Leveraging Social Networks To Motivate Individuals to Reduce their Ecological Footprints. In Proc. HICSS, 2007, IEEE, London, 87a

20. Francis, P. The impact of UK households on the environment through direct and indirect generation of

35. National Statistics, National Statistics Online http://www.statistics.gov.uk/cci/nugget.asp?id=1710

186

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

36. Odom, W., Pierce, J., and Roedl, D. Social Incentive & Eco-Visualization Displays: Toward Persuading Greater Change in Dormitory Communities. In Proc. of Public and Situated Displays to Support Communities. Workshop at OZCHI 2008.

43. Slater, M., Pertaub, D-P., and Steed, A. An Experiment on Public Speaking Anxiety in Response to Three Different Types of Virtual Audience. Presence 11, 1 (2002), 68-78

37. Philips, iCat http://www.hitech-projects.com/icat/

44. Strauss A. and Corbin J. Basics of Qualitative Research: techniques and procedures for developing grounded theory. Sage Publications, London, 1998.

38. Pressman, R.S Software Engineering: A Practitioner’s Approach, 6th edn. McGraw Hill, New York, 2005.

45. Thompson, R (2009) Facebook App Measures Your Carbon Footprint. Computer Weekly, August, 2009.

39. Przybyla, D. and Pegah, M. Dealing with the veiled devil: eco-responsible computing strategy. In Prco. SIGUCCS, ACM Press (2007), 296-301.

46. Walker G., and King D. The Hot Topic: How to Tackle Global Warming and Still Keep the Lights on. Bloomsbury, London, 2008.

40. Robson, C. Real World Research, 2nd edn. Blackwell Publishing, Malden MA, 1993.

47. Weiksner, M. and Fogg, B.J. Liu: Six Patterns for Persuasion in Online Social Networks. In Proc. Persuasive 2008, pp151-163

41. Schultz, P.W., Nolan, J.M., Cialdini, R.B., Goldsteing, N.J., and Griskevicius, V. The Constructive, Destructive and Reconstructive Power of Social Norms. Psychological Science, 18, 5 (2007), 429-434.

48. Yun, T. Investigating the impact of a minimalist inhome energy consumption display. In Ext. Abstracts CHI 2009. ACM Press (2009), 4417-4422.

42. Sen A. 10 Billion Facebook Photos – Is That Ecologically Sustainable? Trendspotting Blog. http://www.trendsspotting.com/blog/?p=487

49. Zizek S. (2008) For They Know Not What They Do: Enjoyment as a Political Factor. Verso, London, 2008

187

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Empirical Investigation of Web Design Attributes Affecting Brand Perception Franca Garzotto, Fabio Sorce Politecnico di Milano Departments of Electronics and Information Via Ponzio 34/A 20133 Milano, Italy {Garzotto, sorce}@elet.polimi.it

Davide Bolchini, Tao Yang Indiana University School of Informatics - Indianapolis 535 W Michigan St. 46202 Indianapolis IN U.S.A. {dbolchin,taoyang}@iupui.edu the corporate branding strategy but also to identify the “value” resulting from a web user experience in forming or sustaining a positive “brand image”. The work presented in this paper contributes to the understanding of this issue, focusing on the role of web design in e-branding.

ABSTRACT

The web has become a mainstream communication tool used by companies, institutions, celebrities, and politicians to establish, reinforce, or repurpose their brand, almost bypassing more conventional branding media. In this scenario, understanding the effects of the different design qualities of a website on users’ attitude towards the brand is of major importance. This paper contributes to the exploration of this issue by presenting a wide empirical study that investigates the degree to which users’ perception of a brand is affected by different design aspects of a website, namely usability, aesthetics, and communicability. The results of this multidimensional analysis have implications on HCI research and practice, as they provide empirically founded guidelines to prioritize design choices in relationship to branding goals.

The practice of marketing and industrial design generally acknowledges that the user experience (UX) with any well design reification of a brand – being it a product, a service, a communication or advertisement artefact - translates to a positive attitude towards the brand itself [47]. An implication of this general principle is that the quality of the UX with a website affects the gamut of feelings, beliefs, or judgments that users associate to the brand. Recent HCI studies pinpoint that people’s opinion on a web user experience is modulated by a number of factors of different nature: subjective (related to the individual predisposition of the user [26]), objective (related to the design characteristics of the website), and contextual (how the experience is situated in place and unfolds in time). The objective (i.e., design-oriented) features that are usually considered to affect the perceived quality of the UX include both pragmatic attributes, which focus on the fulfillment of user’s “operational” goals, and hedonic qualities, which affect the psychological well-being of the individuals [28]. The former are typically associated to the general concept of usability. The latter are related to a broad range of design attributes, such as aesthetics, attractiveness, engagement, joy, and fun [4, 18, 25, 28, 37].

Author Keywords

Brand, Web, User Experience, Design, Value, Usability, Aesthetics, Communicability, User Attitude ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

The promotion of a brand through the web — a process known as e-branding — is reaching in importance, if not overcoming, traditional paradigms of branding that support more passive, “one-way” forms of communication (e.g., conventional advertising) and are less engaging for users [9, 23, 30, 47, 48]. In this arena, it becomes important for many stakeholders - managers, designers, and marketers – not only to align the design of a web communication artefact to Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

188

The purpose of our research is to investigate the aggregated and individual effects of different objective, design oriented characteristics of a web UX on brand perception. We focus on a pragmatic design attribute – usability, and a hedonic attribute – aesthetics. In addition, we consider a third attribute of the design of websites – communicability – which, to our knowledge, has never been taken into account in existing studies in the design/e-branding domain. Specializing a concept of traditional semiotic theories [16, 39], we define communicability as the capability of the sole information content of a website, abstracting from the “channel” that wraps up it (interface and the interaction capabilities), to convey meaning. The “meaning” we are interested in is the set of messages that the brand underlying the web application intends to convey to its target.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The study involved overall 140 participants and was designed as a set of seven experiments. Each experiment considered a different website for which we evaluated the above design attributes and measured the effects of web exposure on users’ brand perception, which was operationalized using value-based metrics. We then analyzed the resulting data in order to identify the existence of significant correlations between each design attribute and the modification in brand perception, comparing the different findings.

brand value can be a moral, ethical, social, or cultural belief, benefit, or lifestyle which an entity is committed to thorough its overall behavior, its products and services, and is worthwhile for a given target group, either at individual or collective level. A brand value represents therefore those traits of an entity’s personality that the entity holds as salient to leverage on and to communicate via its products, interactive artifacts, services, or marketing actions. Syntactically, a brand value takes the form of a single statement or attribute, defining a trait of the brand, as in the following example of brand values for one of the applications considered in our study (Table 1).

To our knowledge, this is the first work that systematically investigates the effects of web design for e-branding purposes from such a multidimensional perspective, and provides some empirical evidence of the individual role played by different design qualities in the complex trajectory from a web user experience to the creation or modification of the brand image. At the same time, our findings raise a number of questions that highlight the complexity of this domain and the need for future research.

BV1

BV2

Magic/Fairytale Spectacular

BV3

BV4

BV5

Exciting

Amusing

Adrenalinic

Table 1. A subset of Brand Values (BVs) of Gardaland (the largest entertainment park in Italy - www.gardaland.com)

Consistently with this line of reasoning, we define Brand Perception in terms of Brand Values Perception, i.e., the users’ subjective opinion that an entity, its products and services, actually fulfill the promise of the values declared by their brand. Given a brand, we measure Brand Perception as the strength of the association, in the users’ mind, between the brand values and the entity, products, or services associated to the brand.

THE EMPIRICAL STUDY Research Goal

The research goal of our empirical study is to discover and isolate the individual effects of three web design attributes – usability, aesthetics, and communicability - on brand perception, identifying whether and at which degree these attributes are good indicators of the effectiveness of a website to convey the brand image, and discovering if there is any dominant factor in this process. We operazionalize and measure these three independent variables in seven different websites with the purpose of discovering their correlation with the change of brand perception (before vs. after using the site) as dependent variable.

Performance-based and Heuristics-based Usability

In our study we are interested in the most “classical”, functional view of web usability, which considers the operational ease of use and the degree of effective and efficient support to users’ information and operational goals. Among all the possible metrics that can be used to assess usability of a website, we adopt two complementary measures: standard performance-based usability — the degree to which users can successfully perform a set of tasks — and heuristics-based usability. The latter denotes the results of heuristics inspection of usability as performed by a team of expert evaluators examining the interface and judging its compliance with recognized usability principles. Rather than adopting Nielsen’s heuristics, we needed to consider finer grained and more structured set of usability principles that could specifically address the pragmatic qualities of the web design suggested by many existing web and hypermedia design models [41]. To this end, we adopted the set of “domain independent” heuristics1 provided by MILE+ method for web usability inspection [7]. MILE+ distinguishes between:

Study Variables Brand Perception

The modern concept of brand is extremely wide and deserves multiple interpretations that go beyond the simple, original notion of brand as a “sign” — word expression or symbol — which identifies the goods or services of a vendor and differentiates them from those of the competitors. A more recent approach to brand definition is more abstract, and encompasses the perceived qualities of both the good (i.e., the products or the service) and the “entity” that stand behind the good: a company, an institution, an organization, a person or, at a wider level, a community or a country. [17] proposes that the brand represents a promise of recognizable values that are unique to an entity and its goods, and they can keep to all their stakeholders - customers, trades, stockholders, employees, fans, or supporters. These are not necessarily functional, utilitarian qualities, but can be anything that gives rise to positive, emotional, or affective effects.

•

1

Content heuristics (8): they address pragmatic qualities of content like accuracy, consistency, linguistic

MILE+ also provides a set of heuristics for assessing usability features in specific web application domains, which are not relevant for the purpose of this study.

Embracing this approach, and in line with value-based design thinking in HCI [22] [10] [11], we model the concept of brand in terms of Brand Values [5, 6, 8]. A

189

Full Papers

correctness, readibility of the communicated to the users by a website;

Proceedings: NordiCHI 2010, October 16–20, 2010

information

•

Navigation heuristics (36): they consider the consistency and effectiveness of information architecture and link structures;

•

Presentation heuristics (31): they address the consistency and effectiveness of the presentation of the different interaction elements and of the overall page lay-out;

•

process that involves multiple subjective and contextual factors and elaborates both explicit and implicit (indirect, evocative, symbolic, or allusive) references to such “objects”. As a consequence, it is difficult to identify objective measures of communicability. It is more feasible to consider the perceived communicability of a website, as the results from users’ subjective rating of the evidence of brand values messages in contents elements only. Scoping Our Research

Technology heuristics (7): they focus on technologydriven features of the application, e.g., system messages or page loading time, or effects caused by implementation defects.

To restrict the boundaries of our investigation and better scope our research, we focused our study on a “category” of e-branding artifacts: brand-oriented content-intensive websites2. These are characterized by the goals that inform their design: reinforcing, establishing or repurposing a brand, and satisfying substantial information and knowledge needs of users. In other words, brand-oriented content-intensive websites are designed both to trigger positive feelings and attitudes towards a given brand and to support the effective use of a large quantity of articulated content.

Usability assessment carried on using MILE+ allows us to measure web usability (although subjectively as any inspection method) along multiple perspectives, which in turn enables us to explore the effects of pragmatic design qualities on brand perception of web users in a much finer grained way. MILE+ offers all the known advantages of heuristic evaluation and has been empirically proved [7] to be easy to learn and more effective (in terms of quantity of discovered usability defects vs. time employed in the inspection) than conventional Nielsen’s method, since it provides a systematic guidance to perform the inspection process.

It is important to characterize the investigated websites in terms of both the above design goals (brand and information purposes) because these criteria are appropriate for the design qualities we set out to investigate in relationship to brand perception. We consider, in fact, hedonic factors like aesthetics, attributes that are more pragmatic (usability), and features related to content (communicability and content accuracy, consistency, linguistic correctness, readibility). “Brand-oriented only” websites tend to privilege the hedonic aspects of the UX and the creation of emotional engagement with a brand. As such, they would offer less significant case studies than websites that are also content-intensive, because we aim at comparing how design attributes of diverse nature affect brand perception and also at understanding the branding role of web UX also along its functional, utilitarian, and more rational dimension.

Aesthetics

The model of perceived aesthetics used in our study follows the one proposed by Lavie and Tracktinsky [33], which differentiates between classical aesthetics and expressive aesthetics. Classical aesthetics refers to “traditional” aesthetic notions emphasizing orderly, balanced, and clear design. It includes attributes such as pleasant, clear, clean, symmetrical, and aesthetic. Expressive aesthetics is modeled by qualities that capture the design’s creativity and originality. Relevant items in this dimension are creative, fascinating, original, sophisticated design, and use of special effects.

Instruments

The study was organized in two sub-studies, the first one addressing the evaluation of usability, aesthetics, and brand values perception, and the second one addressing communicability. Each sub-study involved different subjects and, in part, required different instruments, as discussed below.

Communicability

As mentioned in the Introduction section, communicability denotes the “communication power” of the sole information content of a web system. More precisely, it expresses the degree to which text, image, video, sound, or animation units in a website manifest the brand values of the entity underlying the application and are capable to convey them to the users.

“Original” Websites

We selected seven brands – all related to consolidated companies or institutions in different business sectors and the corresponding websites (see Table 2), meeting the criteria of being brand-oriented and content-intensive, as discussed above.

Our concept of communicability does not imply that we consider explicit brand messages only, as determined, for examples, by the entity’s mission statements, brand mottos, tag line, or logo. As Eco teaches us [16], the meaning of content signs is not necessarily determined by whether they explicitly refer to actual “objects” – brand values in our case. Indeed, the meaning of a message that is formed in a person’s mind is a result of a complex interpretation

2

The term “content-oriented” website is commonly used in web engineering, in most cases as synonymous of “data intensive”, to indicate websites that provide large amount of information items.

190

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Website #1 Apple #2 Bang & Olufsen #3 Fender #4 Gardaland #5 Mirabilandia #6 Egyptian Museum Turin #7 Nintendo

url www.apple.com/it www.bang-olufsen.it www.fender.it www.gardaland.it www.mirabilandia.it www.museoegizio.it

Business Sector High-Tech High-Tech Musical Instruments Entertainment Park Entertainment Park Cultural Heritage

www.nintendo.it

Digital Entertainment

to re-implement each entire application in this form, we used the following approach. For each activity scenario of a given website, we defined, an appropriate “path” on the website, i.e., a sequence of pages that a “typical” user would traverse to complete the tasks in the scenario. The path was identified on the basis of the results of the first study of usability and aesthetics, identifying the most frequent steps followed by the users to execute the assigned scenarios. For each sequence of original pages we created a sequence of content-only html or flash pages that we then submitted to the participants involved in the communicability assessment study. A content-only page provides all text, sound, video, animation, and image units extracted from its corresponding page in the real website but it does not include any interaction and navigation capability, and have minimal lay-out (e.g., no background or special fonts - see Figure 1).

Table 2. The websites considered in our study. Brand Values and Activity Scenarios

For each website, we performed a detailed analysis to determine how the entity behind the application wants to be perceived by its target audience and to elicit the entity’s brand values. We followed the process guidelines provided by the value-driven requirements method illustrated in [5, 8], which structures brand values elicitation into a combination of business analysis, techno-organizational analysis, and user analysis. In short, we carried out brand value extraction from explicitly declared brand sources (institutional mission statements, press releases, and other public domain information such as investor relations material), interviews (via phone or face-to-face) to both institutional stakeholders (e.g., marketing managers or CEO’s) and brand “customers”, and direct, contextual experience of the product or services (e.g., visits to the entertainment parks or brand shops). The qualitative and quantitative data collected with these instruments were distilled in a list of (ten, on average) brand values for each website, which were further validated with institutional stakeholders.

Participants and Settings

A total of 140 users were involved in the overall study, which was carried on from February to September 2009. For each website, two subject groups were recruited and randomly assigned to either the study on usability, aesthetics, and brand perception, or to the study on communicability. For each website, we assigned ten participants to the first study and ten different participants to the second study. One hundred and twenty students of two master classes in HCI at Politecnico di Milano carried out the recruitment of participants (involving relatives and friends) as part of the course homework. All subjects were screened from a larger set, using a pre-test questionnaire that elicited demographic data and consistency of their profile with the brand target of each website. It is important to notice that we considered only persons who never used the website under study, to prevent pre-conceptions and biases derived from previous uses of the application. The sample was composed of males (56%) and females (44%) of different ages (65% aged 1835, 20% aged 35-50), balancing the age distribution according to the profile of the targets of the different brands. Reward for participation was the gift of an interactive CD (either a multimedia guide of a local museum or an interactive game, both developed by the research lab of one of the authors).

During this preliminary phase, we also identified an articulated profile of the main target users for each website, including their information and operational needs and the motivations for using the application. We distilled this knowledge into the definition of a set of activity scenarios that helped us screen the participants involved in the study and define the tasks to be assigned to the users during the exposure to each website. A set of tasks, specific for each application, operationalized high-priority goals for the intended users (e.g., “planning a family visit” on a museum website). The nature and sequence of the tasks were designed to frame a plausible and realistic experience with the website as it would arise in a spontaneous situation of use.

Observers and Usability Inspectors (13 overall) were members of our research lab in Milan, all with a significant experience on UX evaluation both in academic and industrial contexts. To preserve ecological validity, the first study (assessment usability, aesthetics, and brand perception) was performed in the natural usage environment of the participants (at home, work office, or university). The second, more controlled study, given the technical skills required to prepare and administer the communicability test, was carried out at the laboratory of one of the authors.

Content-Only Web “Sites”

Measuring communicability requires distinguishing the effects of users’ exposure to implicit and explicit messages contained in the pure content units from the semiotic effects of the interactive experience of contents as mediated by the interface. For each website, we therefore created a “content-only version”, where we removed all lay-out and interaction or navigation features. Since it would have been too expensive

191

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 1. Left: Home Page of an original website (www.gardaland.it); Right: content-only version (non-interactive experience).

Procedure

scenarios, the questionnaire used prior to the website explosure was administrated to the users, to collect their actual perception of the brand values after the experience with the website.

Measurements of Brand Perception and Performance

To assess Brand Values Perception, we used a questionnaire. Some questions aimed at identifying whether and at which degree users would associate an “entity”, or its products or services, to a set of brand values, inviting users to indicate, for each brand value, the strength of this association on a 5-points Likert scale (0 = Not at all; 4= Strongly). Other questions asked participants to express their level of agreement with a brand’s fulfillment of brand values statements, measured using a 5-points Likert scale (0 = Strongly Disagree; 4= Strongly Agree).

Inspecting Heuristic Usability

Each website was assigned to a team of 3 inspectors, who individually performed a systematic heuristics evaluation using the MILE+ method and its reporting instruments (structured data entry forms). Each inspector rated each applicable MILE+ heuristic with a 5 points Likert scale denoting compliance level (0=Very Low; 4 = Very High) and reported design problems according to shared guidelines. Then, as in conventional inspection methods [40, 46] the 3 inspectors came together to discuss the problems found and their severity, motivate their judgments, consolidate the results and finally converge their scores to an agreed level.

One important aspect of our procedure is the need to isolate the effects on brand perception resulting from the exposure to the website during the experiment from those resulting from previous experience with other brand reifications, e.g., traditional marketing actions, word of mouth, use of brand products or services or the website itself. To this end, we measured users’ brand perception before and after the use of the website, and considered the actual increment or decrement of these measures only. As mentioned in the previous section, study participants had never used the website before. This removed the potentially confounding factor that a previous exposure to e-branding actions could have introduced in our study.

Evaluating Aesthetics

Each website was evaluated by the users on each of the 10 adjectives (defined by the model in [33]) that denote classic and expressive aesthetics using a 5-points scale (0 = Strongly Disagree; 4= Strongly Agree). We administered a questionnaire after the exposure to a website and after compiling the brand perception post-usage questionnaire.

The brand perception questionnaire was administrated to the users before using the website. Then each subject was asked to use the website by following 3 assigned activity scenarios. The average overall duration of a session of use was approximately 40 minutes. During users’ execution of the assigned scenarios, observers collected data on performance, in terms of level of task success (task completion rate) and time-on-task. At the end of all

Evaluating Communicability

Each subject in the “content-only” group was exposed to a scenario-based experience of the content-only version of the assigned website. We removed any form of interactivity from the user experience by giving the facilitator the responsibility of opening, for each scenario, the corresponding “content-only” pages, one after another, in the proper sequence, at a pace defined by each user.

192

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Correlations

Dynamic elements (sound, video, animation) were also activated by the facilitator on user’s request.

To check whether the four predictors are measuring different characteristics of the considered applications, we looked for the presence of high correlations among the predictors. A high correlation may increase or mitigate the effect of a single predictor, which would make the situation much more complex to interpret. The VIF (Variance Inflation Factor), which detects strong linear relationships between predictors, shows that there are no strong correlations (1.196). The highest correlation is between classic aesthetics and expressive aesthetics, but it is not high in absolute terms (r=.478). Therefore, we can confidently conclude that the four predictors actually measured meaningfully different constructs of the user experience with the website.

A questionnaire was submitted to the participants after the overall content-only experience to assess communicability. Questions were articulated around brand messages, asking users whether and to what degree, in their opinion, each brand value “emerged” from the experienced content, on a 5 points Likert scale (0=Not communicated at all; 4=Strongly Communicated). RESULTS

We analyzed the data with standard effect and correlation analysis methods to investigate the relationships between the independent variables (performance-based usability, heuristics-based usability, aesthetics, and communicability) and the dependent variable (brand values perception). The main results are discussed in the rest of this section.

Overall, these preliminary results indicate that the selected websites are a “good” sample to represent content-intensive and brand-intensive web applications. They are normally distributed in terms of communication impact (Mean=.27, SD=.39), and they significantly differ in each of the four predictors.

Checking for Variability and Sample Quality Usability Performance

The level of usability performance on the seven websites is significantly different (one-way ANOVA, F(7,69) = 9.95, p < .01). On average, Turin Egyptian Museum has the highest score (mean = 88.65, SD = 5.36), while Gardaland the lowest (mean = 68.57, SD = 7.62). These results indicate that the sample of websites selected account for substantial variability in terms of usability performance, and this contributes to strengthen the overall reliability of the data.

Multi-Dimensional Effect on Brand Perception Factors Affecting Brand Perception Change

We run a multiple regression to discover relationships between performance-based usability, aesthetics, and content communicability as predictors, and brand perception change as the outcome variable (Figure 2). Based on the results of the multiple regression analysis (see Table 3), we used the standardized regression coefficient (β) as an indicator of the effect of a predictor on the communication impact. We were then able to quantitatively model how brand values perception is affected by usability performance, aesthetics, and content communicability. The model shows how much a given predictor affects the outcome variable (brand values perception), if the effects of the remaining predictors are held constant.

Classic and Expressive Aesthetics

The scales used to measure classic and expressive aesthetics are both reliable (Cronbach’s Alpha > .7). The websites are significantly different among each other both on classic (F(7, 69) = 5.29, p < .01) and expressive aesthetics (F(7, 69) = 5.19, p < .01). Apple has the highest score on class aesthetics (mean = 4.06, SD = .31), while Gardaland the lowest (mean = 2.88, SD = .78). For expressive aesthetics, Mirabilandia has the highest score (mean = 3.62, SD = .65), while Fender the lowest (mean = 2.38, SD = .86).

Constant Performance-based Usability Classic Aesthetics Expressive Aesthetics Content Communicability

Content Communicability

Like the other two factors, the values of content communicability of these websites are also significantly different (F(7, 69) = 6.87, p < .01). The content of Nintendo website has the highest communicability (mean = .44, SD = .34), however, the Fender website has the lowest and even a negative communicability (mean = -.31, SD = .41).

B -.740

SE B .388

β

.004

.004

.097

.206 -.024 .423

.069 .057 .106

.342* -.047 .416*

Table 3. Multiple Regression for Brand Values Perception: R2 = .364 (p < .001). * p < .01

Brand Values Perception

The communicability of the site is the predictor that most significantly accounts for the variability of the brand values perception (β =.416). In other words, the brand values perceived by the users when exposed to the multimedia content of the site only (in isolation from the interactive and graphic elements of the design) account for most of the overall brand perception in the fully interactive experience.

Overall, users’ perceptions of the brand values for each website are significantly different (F(7, 69) = 4.68, p < .01). The brand values for Nintendo website are best perceived by the users (mean = .64, SD = .46), whereas the brand values for Gardaland website are the worst perceived (mean = -.03, SD = .30).

193

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Classic aesthetics (β =.342) is the second most important factor that significantly affects brand values perception. Expressive aesthetics shows a negative, but not significant effect (β =-.47).

The fact that the considered design attributes can only explain 36.4% of the variation in brand values perception (R2=.364) is an indicator that there are a number of variables — of subjective and objective nature — we could not fully control for in our study, but have an impact on brand perception. Hedonic and pragmatic design factors other than usability, aesthetics, and communicability may play a strong role in forming or modifying the image of a brand in the user mind and are worth being investigated. Furthermore, the heterogeneity of the business sectors considered may have introduced a confounding variable. The selection criterion for the websites in our study was only to be brand-oriented and content-intensive, as this class of applications has design goals that are particularly relevant for the design attributes considered. As we wanted to study both hedonic and pragmatic factors in relationship to brand perception we needed to focus on websites whose design is informed by both branding goals and functional goals (such as information goals). Still, abstracting from the business sector brought in a broad variability of branding strategies and approaches. This in turn may have introduced a strong variability in the initial brand perception of the study participants who had familiarity with the brand; it may have acted as a strong mediating effect on the user’s judgment during the experiment [12] in spite of our attempt to control for this aspect (by isolating what impacted on user’s brand perception at the beginning through pre-use brand perception evaluation).

We also checked for a potential interaction effect between content communicability and classic aesthetics. In fact, we have seen that better communicability yields better brand values perception, but the effects of communicability could have been influenced by classic aesthetics (present in the stimuli of the communicability study). However, we verified that this effect is not significant (p > .05). Interestingly, usability performance does not significantly affect brand values perception (β =0.97). This suggests that, in our considered case studies, users could still strongly and positively perceive the brand values even when task performance is low but other design dimensions are well designed. Factors Correlating with Brand Perception Change

The assessed values of heuristics-based usability as a whole significantly correlate with the level of brand values perception (r=.817). This suggests that the level of usability of the design characteristics as inspected by the evaluators follows closely the variation in users’ brand perception. Still, when looking at the specific design dimensions considered during inspection, presentation design has the only significant correlation with brand perception (r=.775). The measures of the other pragmatic factors – usability of content, navigation, and technology aspects - do not have significant correlations.

On the other hand, focusing on brands in a specific business sector could reduce this variability factor but might introduce further complexity. Using the business sector as the only selection criterion for our case studies would have meant not to discriminate between brand-oriented only websites and websites that are also content-intensive, thus removing a confounding variable but adding another one.

In summary, we point out three key findings: 1.

Communicability is the most important predictor of brand perception impact, followed by classic aesthetics.

2.

As to the intrinsic characteristics of the application evaluated by inspectors’ ratings, presentation design has the strongest positive correlation with brand perception, while the other pragmatic factors seem to have a marginal effect.

3.

Overall, communicability, classic and expressive aesthetics, and usability account for little more than one third (36.4%) of the variation in brand values perception (R2=.364).

In spite of these limitations, we think that the results of our study have some interesting implications that are worth discussing and can be influential on the practice and research in user experience design and evaluation. Implications Content as Key Driver for the Brand Experience

Our findings clearly indicate that the brand values communicated by the content are the ones that mostly determine the overall brand perception. This goes much beyond the simplistic slogan that “content is king” and points to the fact that properly designed content is the most critical vehicle to effectively convey brand values and create a positive brand image in users’ mind. Note that content here does not indicate just “information”, but the organic, value-driven shaping of the messages (conveyed by various media) that the website intends to get across to its target users. We can summarize the key implication of our research as follows: Value-driven content is the primary concern when designing a website for branding purposes.

DISCUSSION Limitations of our study

Our research has its shortcomings which may have affected its validity, most notably the relatively limited sample size used for each website. Still, the fact that we have replicated the study in seven web applications using the same methodological framework partially compensates this drawback and strengthens the external validity of our results.

194

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Designing value-charged content plays an often underestimated role in UX design. A deep exploration of design concerns pertaining content requirements, conception, articulation, deployment and evaluation, and a seamless integration of this understanding into existing HCI methods and tools opens important perspectives for the future development of the field.

significantly affect brand value perception. This means, for example, that even strong (positive or negative) variations in user performance do not change significantly the user’s opinion on a brand. On the other hand, the overall usability of content, navigation and presentation design (as measured through usability heuristics expert rating) has a significant correlation with brand values perception. This suggests that the brand values communication effect originated by these pragmatic qualities is generally stronger than threats to the brand coming from breakdowns in user performance.

The Role of Aesthetics

The significant effect of classic aesthetics on brand perception demonstrates that implicit brand messages are effectively conveyed through design attributes that are very much related to the most conventional concept of “beauty”, such as clean, clear, pleasant, and symmetrical. This finding is somehow consistent with the high correlation found between presentation heuristics score and brand perception. In fact, the heuristics used by experts during the inspection of the presentation-related aspects of the website are conceptually akin to characteristics of classic aesthetics, since they mainly focus on the consistency of the visual characteristics of page elements and on their lay-out balance and regularity. In contrast, the attributes of expressive aesthetics, which concern more the emotional sphere, do not seem to significantly modify brand perception. This is quite surprising, since it seems to contradict, at least in part, the principles of emotional design [37] which suggests that emotions have a crucial role in the human ability of sense making. A possible explanation of this result is that users are more familiar with classic aesthetics attributes, which are more related to our conventional concept of beauty, and therefore can identify more immediately, transparently, and rationally the associations with intended brand values [46] Rating expressive characteristics, instead, require users to reach a deeper level of cognitive reflection, interpretation, and judgment, which in turn makes the association to brand values more indirect and distant.

RELATED WORK AND CONCLUSIONS

Recent studies pinpoint the complex role that different intertwined dimensions of the web user experience play in users’ judgment of a website and their attitudes towards its. Still, few of them specifically investigate these aspects in relationship to e-branding and brand perception. [12] illustrates that a subjective factor like brand attitude does influence the users’ evaluation of aesthetics, pleasure, and usability of websites, while [29] highlights the influence of brand awareness on the user evaluation of search engine results. [45] highlights the correlation between users’ perception of interface aesthetics and their judgment on system’s usability. [27] introduces a framework for users’ design quality judgments based on Active Decision Making Theory. [3] manipulates usability and aesthetics of an interactive system to assess the relative impact of these factors in users’ perceived “value” of the application. [32] considers a number of key visual design factors and identifies their quantitative relationships with 13 classes of emotions that people usually feel when viewing diverse homepages. In the field of persuasive computing [19], a number of studies have examined the design factors that affect consumer’s trust in e-commerce websites [21, 24, 44], influence people’s perception of the website credibility [20] and ultimately have an impact on consumers' product preference. [14] considers in particular the relationship between beauty and usability and the general impact of beauty on product choice and use.

It is also important to notice that the fact that classical aesthetic qualities of the design are related to the overall brand values perception across the sites analyzed, does not yet reveal which specific brand values are more influenced by this aesthetics dimension. Thus we cannot exclude that our results is influenced by the nature of the brand values of the websites considered in our study. A finer grained analysis of the relationship of each aesthetic dimension with specific values may reveal that the perception of values that have a more emotional, affective nature is indeed affected by expressive aesthetics. This analysis was outside the scope of our study, but the exploration of this and similar issues points to interesting future research directions. Based on the results of our study, we can suggest that after quality of value-charged content, designing for classic aesthetics is the most effective way to positively affect the brand image.

A number of authors claim that brand image - intended as the set of “brand associations held in consumer memory” [31] – is influenced by hedonic aspects of an interactive product [37] as well its pragmatic characteristics (e.g., usability [43]). [6] provides some empirical evidence to the correlation between usability and brand perception, and also offers a reusable, methodological support to replicate the reported study and to investigate additional design characteristics of a website in relationship to brand. [6] provides a useful operationalization of the notion of brand perception, which we have also used in the study reported in this paper, adopting a value-oriented perspective to model brand perception, and is centered around the concept of “value”. This notion is widely investigated in (online) marketing, also in relationship to measures of the perception of brand personality and the economic benefits derived from an improved brand image [1, 2, 13, 34, 35, 36, 38, 40, 42]. We instead embrace a notion of value that is

The Role of Usability

It is important to note the tension between two emerging results. On the one hand, usability performance does not

195

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

content – communicability, followed by classic aesthetics, are the most important factors to consider to effectively generating brand values perception. Still, we have assessed that the considered design factors can only explain little more than one third of the variation in brand values perception, at least in our case studies, that focused on a specific class of applications. If confirmed by further studies, involving a larger size sample of testers and a wider number of websites, these results mean that, beyond usability, classic aesthetics, and communicability, there are other, subjective and objective factors in the web User Experience that may account for the remaining variation of brand values perception. Further research is needed to discover them and indentify their role for e-branding. Finally, other genres of branded websites are challenging domains for investigating the applicability of similar user studies. We think for example of brand-oriented applications that are mainly intended to provide services, and therefore stress the functional qualities of the interface and the interaction, or those that focus almost exclusively on branding goals and thus heavily rely on special advertising-like multimedia effects. These classes of branded web systems may require similar approaches as ours, but different constructs and investigation methods, more relevant to the nature of the brand communication strategy at stake.

closer to what has been investigated in HCI design frameworks, such as Values-Sensitive Design [22] and Value-Centered Design [10]. More specifically, we take inspiration from Worth Centered Design (WCD) [11] This paper substantially extends the results of [6] for a number of reasons. [6] only considered how web usability as a whole affects brand image, without discriminating the role played by the different design dimensions that modulate usability, as we do in this paper. The work reported here also expands the analysis to a spectrum of brand affecting design dimensions beyond usability that include [classic and expressive] aesthetics and communicability. This is the first work that empirically investigates the effects of web design for e-branding purposes from such a multidimensional perspective, and provides evidence of the role played by the different design characteristics in the creation or modification of the “brand image”. Among the considered design factors, communicability is novel. A branch of HCI named Semiotic Engineering uses this term with a different meaning, as “the degree to which an interactive system communicates efficiently and effectively to users the underlying design intent and interactive principles” [15]. Instead, we embrace the concept of communicability from a different perspective, more related to traditional semiotic theories existing before the advent of interactive multimedia [16, 39]. Based on these approaches, we consider communicability as the fundamental semiotic property of the sole content elements of a website to more or less transparently communicate meaning. In particular, we evaluate the degree to which “pure” multimedia content, isolated from all the interactive elements through which it is delivered, conveys the intended brand image, and define an original procedure to measure it.

ACKNOWLEDGMENTS

A special thank to all master students of the HCI classes at Politecnico di Milano who contributed to user recruitment in their families and work places, and to all the participants who volunteered for the study. REFERENCES

1. Aaker D.A. The financial information content of perceived quality. Journal of Marketing Research, 31, 1996, 191-201

Our work provides contributions both from a research and practical perspective. In terms of contributions to the body of knowledge on user experience research and e-branding, we have: (a) analytically modeled and isolated some hedonic and pragmatic factors of web user experience design influencing brand perception, in a class of web application that are both brand-oriented and contentintensive; (b) for this category of websites, empirically demonstrated and discussed the relative importance and role of communicability, aesthetics, performance-based usability and heuristics-based usability to influence the user’s brand values perception of a website.

2. Aaker D.A. Dimensions of Web Personality. Journal of Marketing Research, 34, 1997, 347-356 3. Ben-Bassat, T., Meyer, J., Tractinsky, N., Economic and subjective measures of the perceived value of aesthetics and usability. ACM TOCHI, 13, 2, ACM Press (2006) 210-234, 4. Blythe, M.A., Overbeeke, K., Monk, A.F., Wright, P.C. (eds.): Funology: From Usability to Enjoyment. Kluwer (2005) 5. Bolchini D. Garzotto F., Value-Driven Design for “Infosuasive” Web Applications. Proc. 17th Int. Conf. on the World Wide Web (WWW’08). ACM Press (2008), 745-754 6. Bolchini D. Garzotto F. & Sorce F. Does Branding Need Web Usability? A Value-Oriented Empirical Study. Proc. INTERACT 2009, LNCS 5727, Springer (2009), 652-665

Besides, our work can have implications for the practice of UX design and e-branding. A question that seems to be crucial in the industrial world is “How do you prioritize interaction design when boundaries are set by time, money and resources?” (as raised at the Industrial Day UX session at Interact 2009) Our work sheds a preliminary light on this issue, as it provides implications for the consideration and prioritization of the design dimensions to inform effective user experience designs. Specifically, we have provided some empirical evidence that a semantic property of media

7. Bolchini, D., Garzotto, F. Quality and Potential for Adoption of Web Usability Evaluation Methods: An Empirical Study on MILE+. J. Web Engineering, 7 (4) Rinton Press (2008)299-317, 8. Bolchini, D., Garzotto, F., Paolini, P., Branding and Communication Goals for Content-Intensive Interactive Applications. Proc. 15th IEEE International Conference on Requirements Engineering, IEEE Press (2007) 173 - 182 173182.

196

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

9. Chaffey, D., Ellis-Chadwick, F., Johnston, K., Mayer, R. Internet Marketing - Strategy, Implementation and Practice. Prentice Hall (2006)

30. Kapferer, J. N. The New Strategic Brand Management: Creating and Sustaining Brand Equity Long Term. Kogan Page, London, 2004.

10. Cockton G. A Development Framework for Value-Centered Design. Proc. CHI 2005. ACM Press (2005) 1292-1295

31. Keller, K. L. Strategic Brand Management; Building, Measuring, and Managing Brand Equity. Prentice-Hall, 1998

11. Cockton, G. Designing Worth is Worth Design. Proc. NordiCHI 2006. ACM Press (2006) 165-174.

32. Kim J., Lee J., Choi D. Designing emotionally evocative homepages: an empirical study on the quantitative relationships between design factors and emotional dimensions. International Journal of Human-Computer Studies, Elsevier (2003) 899-940

12. De Angeli A., Hartmann J, Sutcliffe A. The Effect of Brand on the Evaluation of Websites. Proc. INTERACT 2009, LNCS 5727, Springer (2009) 638–651 13. Delgado, E., Munuera, J. L., Yagüe, M. J. Development and Validation of a Brand Trust Scale. International Journal of Market Research. 45. 1, 35-54. 2003 14. Diefenbach S., Haasenzahl M. The Beauty Dilemma: Beauty is Valued but Discounted in Product Choice. Proc. CHI 2009, ACM Press (2009), 1419-1426 15. De Souza C.S, Leitão C. F. Semiotic Engineering Methods For Scientific Research in HCI. Morgan and Claypull (2009) 16. Eco U. A Theory of Semiotic, Macmillan (1977) 17. Fisher-Buttinger C., Vallaster C. Connective Branding. Wiley (2008) 18. Fishwick, P. (ed.) Aesthetic Computing. MIT Press (2005) 405422 19. Fogg, B.J. Persuasive technology. Morgan Kaufmann (2003) 20. Fogg BJ, et al., , What Makes Websites Credible? A Report on a Large Quantitative Study. Proc. CHI’01, ACM Press (2001), 61-68 21. Friedman, B., Kahn, P. H., Howe, D. C. Trust Online. Comm. of the ACM. 43.12, ACM Press (2000) 34-40. 22. Friedman, B., Kahn, P. H. Human Values, Ethics, and Design. In J. A. Jacko & A. Sears (Eds.). The HCI Handbook. Lawrence Erlbaum Associates (2003). 1177-1201. 23. Gilbert A. C., Iacobucci D. Marketing Research: Methodological Foundations. Thomson South-Western Publishers (2005) 24. Ha H., Perks, H. Effects of Consumer Perceptions of Brand Experience on the Web: Brand Familiarity, Satisfaction and Brand Trust. Journal of Consumer Behavior. 4. 6, 2005. 438452. 25. Hartmann, J., Sutcliffe, A., Angeli, A. D. Investigating Attractiveness in Web User Interfaces. Proc. CHI 2007 ACM Press (2007). 387-396. 26. Hartmann, J., De Angeli, A., Sutcliffe, A.: Framing the User Experience: Information Biases on Website Quality Judgement. Proc. CHI 2008. ACM Press (2008), 855-864 27. Hartmann, J., Sutcliffe, A., De Angeli, A. Towards a theory of user judgment of aesthetics and user interface quality. ACM TOCHI 15, 4. ACM Press (2008), 1-30 28. Hassenzahl, M.: The Interplay of Beauty, Goodness, and Usability in Interactive Products. Human-Computer Interaction 19 (2004) 319-349 29. Jansen, B., Zhang, M., Zhang, Y. The Effect of Brand Awareness on the Evaluation of Search Engine Results. Proc. CHI 2007. ACM Press (2007) 2471-2476.

197

33. Lavie, T., Tractinsky, N. Assessing dimensions of perceived visual aesthetics of websites. International Journal of HumanComputer Studies 60, 2004, 269-298 34. Low G.S., Lamb C.W. The measurement and dimensionality of brand associations. Journal of Product and Brand Management, 9 (6), 2000 35. Macias, W. A Preliminary Structural Equation Model of Comprehension and Persuasion of Interactive Advertising Brand Websites. Journal of Interactive Advertising. 3. 2, 2003. 36. Müller, B., Chandon, J. The Impact of Visiting a Brand Website on Brand Personality. Electronic Markets. 13. 3, 2003. 210 – 221 37. Norman, D. A. Emotional Design. Basic Books (2004) 38. Page, C., Lepkowska-White, E. Web Equity: A Framework for Building Consumer Value in Online Companies. Journal of Consumer Marketing. 19. 3, 2002, 231-248 39. Pierce, C.S. Collected Writings. Indiana University Press (1982) 40. Romaniuk, J., Nicholls, E. Evaluating Advertising Effects on Brand Perceptions: Incorporating Prior Knowledge. International Journal of Market Research. 48. 2, 2006.179-192. 41. Rossi, G.; Pastor, O.; Schwabe, D.; Olsina, L. (Eds.), Web Engineering: Modelling and Implementing Web Applications, Springer (2008) 42. Slotegraaf, R. J., Pauwels, K.. The Impact of Brand Equity and Innovation on the Long-term Effectiveness of Promotions. International Journal of Marketing Research. 45. 3, 2008. 293306. 43. Spool, J.M. Branding and http://www.uie.com/articles/branding_usability/

usability.

44. Tsygankov, V. A. Evaluation of Website Trustworthiness from Customer Perspective, a Framework. Proc. 6th International Conference on Electronic Commerce. ACM Press (2004). 265271 45. Tractinsky N., Katz A.S., Ikar D., What is beautiful is usable. Interacting with Computers 13 (2000), 127-145 46. Tractinsky N. and Zmiri D. Evaluating the consistency of immediate aesthetics perception of web-pages. Int. J. of HumanComputer Studies 64 (2006) 10071-1083 47. Yoo B. Donthu N., Lee S. An Examination of Selected Marketing Mix Elements and Brand Equity. Journal of the Academy of Marketing Science, 28, 2, 2000. 1995-211 48. Ward, M. R., Lee, M. J. Internet shopping, consumer search and product branding. Journal of Product & Brand Management 9, 1 MC UP (2000) 6-29

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

“Luckily, I don’t need it”: Elderly and the Use of Artifacts for Time Management Leonardo Giusti FBK-irst Trento, Italy [email protected]

Eleonora Mencarini University of Siena Siena, Italy

ABSTRACT

Massimo Zancanaro FBK-irst Trento, Italy [email protected]

In this paper we adopt the ecological perspective proposed by Forlizzi and colleagues [2] as a way to make sense of the experience of aging. This framework takes into consideration the complex interactions between, and the experiences that result. New technology should be conceived of as a part of an interdependent system encompassing people, products, and activities to support elderly values and practices. In particular, in this paper we address the specific ecology of practices and values that mediate elderly people use of time-scheduling artifacts.

In this paper, we describe the artifacts and the practices that a group of elderly people use to plan future events and remembering information about future event at the proper time. The role of such artifacts in the construction of a narrative and emotional account of elderly life is discussed. A particular attention is given to the description of the values that mediate the use of time-scheduling artifacts. By taking into consideration the whole complexity of time management ecology, we finally suggest a number of design opportunities.

We present the results of several semi-structured interviews and some contextual interviews in the homes of a group of 19 elderly people with no specific health problems and with a rich social life.

Author Keywords

Elderly, Time-management,. ACM Classification Keywords

In the followings sections, we will describe in details the artifacts and the practices that elderly people use to plan future events and remembering information about future events at the proper time. Furthermore, the role of such artifacts in the construction of a narrative and emotional account of elderly life will be discussed. A particular attention will be given to the description of the values that mediate the use of time-scheduling artifacts.

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

In this paper, we describe a research work aimed at understanding how older adults in Northern Italy use lowtech artifact for managing their time. The ultimate goal being to inform the design of time-scheduling technologies to better fit this class of users.

By taking into consideration the whole complexity of time management ecology, we finally suggest a number of design opportunities. In particular, we feel that any new technology aimed at supporting time-management should be considered not only from a functional point of view: the role that such artifacts have in supporting elderly in making sense of time should be considered as well.

The adoption or rejection of a new technology by elderly people is known to be strictly dependent how they find it intrusive, complex and revealing their limitations or disrupting their home environment [1]. It is therefore extremely important to take into consideration existing practices and values when designing new technologies for elderly people.

RELATED WORK

In recent years there have been many studies on the design of time management tools. They mostly concentrate within two main areas: the environment in which time management takes place, either working place [3] or the domestic environment [4, 5] and the use of calendars and notes [6, 7] as management tools.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

Notes have three main functions: immediate use, temporary storage and perspective memory aid [6]. They focus on

198

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

present information and its future use as passive reminders; they have to be seen at the proper time by the user [6]. On the contrary, a calendar is a way to navigate the future (both personally and socially), it is an information management tool, a communication device (it provides awareness thanks to its visibility), it has a mnemonic function (reminder) and it provides a rationalized representation of time [4]. The calendar can be used to plan future events and to make decisions on how to organize the daily life.

retirement they find activities to fill their days, to socialize and to feel useful for the community. Activities and commitments make their life still meaningful while technology is perceived as threatening it [10, 2]. Literature on elderly and technology usually deals with home assistive technologies that aim to counterbalance physical impairments due to the age. To our knowledge, the only study involving elderly in the use of time management artifacts is Plaisant et al. [11]. This study focuses on the coordinating aspect of time managing, aimed at the whole family members and older adults were treated as special users. A noteworthy aspect that they reported is that maintenance of family calendars information in the majority of cases is up to more than one person, but when just a single person takes care of it, usually it is a woman.

In general, people still prefer paper to technological devices because of some of its availability (it is almost everywhere); portability (it can be brought everywhere) [8]; reliability (it does not run out of power); accessibility (anyone can use it) [8]; user-friendliness (it makes it easy to take notes when speaking on the phone) [5]; lack of structure (anyone can sketch on it in a way that suits his needs) [5]. Yet, paper can fail us by becoming misplaced, unintelligible, or ignored [6]. Indeed, the biggest contribution that digital technology has brought to time management until now is notification functionality. This function transforms a passive note into an active reminder [6]. On the other hand, digital calendars some disadvantages: in particular they must be powered up to display the content of the reminder; they are bulky and require too much time to enter information [6].

Forlizzi et al. [2] give important design guidelines based on elderly scale of values. Their research shows that elderly are constantly on the lookout for sense of self, independence and dignity; this affects how they use products. In the same way, Hirsch et al. [12] found that elders’ first aim is the maintenance of a good quality of life. Two main factors affect quality of life: independence and engagement. Between these two factors there is a special trade off to keep a stable equilibrium: as functional ability (and relative independency) decreases, connections to other people and engagement become exceedingly important in determining the individual wellness. A special matter in this area is users’ perception of their own abilities. Often perception is out of step with their actual capabilities. This can include an elderly operating at a level below his capacities, causing him to be fearful of attempting relatively safe tasks. Conversely, the elderly may overestimate her own ability, causing him to undertake risky tasks. This highlights that old people’s attitudes must be taken into consideration in design, in order to maximize acceptability.

Time management is often a social task. Organizing life in the immediate future may require information exchange with other people. Shared organizational systems turned out to be good for coordinating with other people in both domestic and work context because they share responsibility among people who use them [9, 5]. Indeed, the functionality of shared use relies on members’ compliance with coordination protocols [4]. Taylor and Swan [5] have shown how paper lists can help a mother to organize household activities and tasks as well as her children to grow in responsibility and to test the family order. On the contrary, Brzozowski et al. [9] have found that by using GCS (Group Calendaring Systems) in the working field the visibility of group members’ availability exerts a form of social pressure to encourage compromise and honesty but it can also raise problems relative to privacy and peer judgment about time allocation [3].

USER STUDY

The involvement of elderly people in a user research study greatly challenges the applicability of traditional methods such as in-situ interviews, shadowing or ethnography because of the peculiar psychological and ethical issues that enter into play [13, 14, 15]. Research in these contexts is often regarded as not merely difficult but often inappropriate and intrusive [4]: several researchers have noted the difficulty of obtaining the consent for in-home observations [16].

Other studies highlighted the emotional, reflective and narrative dimensions of time management tools [5, 8]. Sometimes people annotate events after they have happened in order to keep memory of them and to see them in a past perspective, often addressing to the emotional sphere [8].

For these reasons, we have adopted a comprehensive approach to motivate the participation of older people in the design team and to establish a long-term relationship with the other elders of the group. This approach is based on the work carried out by [13] and encompasses: (i) the endorsement by, and the direct involvement in the project of, local authorities and associations of older people with the role of mediators and guarantors; (ii) the establishment and nurturing of the sense of belonging to a pioneering

In digital calendars, personal signs and contextual marks for emphasizing certain entries reflect the emotional relation between entries and users. Yet, the sense of emotion embodied in material artifacts, such as paper, which is often missed by the technological tools [5]. Literature on the management of time by elderly people is too huge to be summarized here. An important aspect to mention is that elderly like having a busy life, after 2

199

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

group, with emphasis on the value of both individual and group contributions.

daily routines and were easily integrated in the everyday activities of the center.

Following these principles, we have involved in the study a day center for senior citizen located in the city of Trento in Northern Italy. The staff of the center has established during the years a well-consolidated relationship with the attendants and has greatly contributed to the creation of a trusted atmosphere between participants and researchers.

While a structured interview has a pre-defined, limited set of questions, a semi-structured interview is flexible, allowing new questions to be brought up during the interview as a result of what the interviewee says [17]. The interviewer has a framework of themes to explore. This framework is generally a grouping of topics and questions that the interviewer can ask in different ways to different participants.

The study was articulated in four phases: (i) Preliminary group meeting, (ii) Individual Semi-structured Interviews, (iii) Contextual Interviews and (iv) Final group meeting.

An important aspect of the semi-structured interview is that questions are asked when the interviewer feels it is appropriate to ask them. Elderly people could consider some questions as inappropriate or intrusive; therefore, it is important to adapt the interview structure and the questions to the different subjects’ personality and sensibility in order to do not create conflicts. Indeed, interview sessions were also an occasion to establish and consolidate a trusted relationship with participants.

Participants

The participants were 19 older people (16 women and 3 men; age range 60-87; mean 75) living in the urban area. All of them are fairly healthy and autonomous people; 14 live alone (at the national level: 77.9% of older adults live alone), one live with his son and the 4 are married. The gender asymmetry reflects the gender distribution in the center.

In our case, the framework was focused on the artifacts and the practices that people adopt to make use of time, with a particular attention to the emerging cultural and emotional values. In particular, it was articulated in four relevant dimensions of use of artifacts: planning, remembering, coordinating and reflecting.

People attending the day center are involved both in day center activities (for example, the creation of hand-made toys for charity sales, volunteer work, support to the staff in the organization of events and so on) and personal activities related to their family (such as taking care of grandchildren). In general, these subjects look for a social and active life and strive for having lot of commitments during the day.

The dimensions of planning referred to how people use artifacts to plan future events and the strategies to scheduling them. The dimension of remembering had to do with what artifacts and practices people adopt to remember things to do (i.e. call a friend, payments), appointments and other kinds of up-coming events (i.e. travels). The dimension of coordination investigates the social dimensions of the artifacts that elderly people use to manage their life. Finally, the dimension of reflection investigates the use of artifacts and the practices by which people look retrospectively to their life.

Preliminary Group Meeting

In the first encounter with the researchers both the staff of the day center and several of the older adults that regularly attend the center were present: by creating a safe and protected social context, reducing the individual involvement and mitigating the fear to be exposed to a novel situation, the participation of people was favored. The staff of the center presented the main objectives of the study and stressed the importance of belonging to a group of people that could play a critical role in the development of new technology aimed to improve the elderly people’s quality of life. Then, the researchers explained the main phases of the study, detailing what kind of participation was required in each different phase. At the end of the meeting, the staff and the researcher had a very short conversation with the people interested in continuing the study and the appointments for the second phase of the study were scheduled.

Beyond the understanding of the functional role of artifacts in mediating the activity of planning, remembering, coordinating and reflecting we also wanted to investigate the emotional and the cultural values tied to the practices of use. Contextual Interviews

A contextual interview [18] is an interview conducted in the same environment which is the focus on the interested for the research. In our case, the main goal of the contextual interviews was to observe the role of the artifacts, in the natural context of use. Contextual interviews may help revealing reveal latent and tacit cultural values that are difficult to verbalize and report during an standard interview because artifacts can be shown rather than simply described and practiced can be enacted rather than told.

Semi-Structured Interviews

The second phase of the study consisted in individual semistructured interviews conducted in the daily center. The reason to conduct the interviews in the center rather than in the homes is because that the participants know the center very well and perceive it as familiar and comfortable place; furthermore, the interview sessions became part of their

The contextual interviews were mainly used to refine the results of the data collected during in the previous phase.

200

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The results of the semi-structured interviews have been used as a lens to conduct the contextual interviews. We asked people to show off the artifacts they use to organize their life and to describe related practices. By directly observing people using artifacts, we had the opportunity to observe the context where artifacts were located, how people mark the different events in a calendar or the different post-it (colors, size, materials) they use to jotting down a reminder. In this way, it was possible to elicit the cultural, emotional and personal values related to the practices of use of the artifacts.

daily life of elderly people. It is not common that a routine activity is forgotten. Furthermore, elderly people routines are very flexible: failing in performing a routine activity does not have any major consequence. Time management becomes critical when they have to remember future extraordinary events, such as an appointment with the hairdresser or esthetician, physical examinations, meals or trips with other people. These events are deviations from habits and could be easy forgotten. Moreover, failing in attending such events could result in serious consequences: in general, these events are appointments with professionals or public officers and have a very rigid social and temporal structure.

Final meeting

In the final meeting, the results of the study was presented to the older people that participated in the study and the staff of the center.

Finally, deadlines, activities that have to be completed at a given time, are the most peremptory events. However, they are not felt as a problem because our subjects’ deadline are usually bills payments almost all of them have their bills automatically paid. A special case of deadlines are medication prescriptions that are usually dealt with ad-hoc strategies.

Data Collection and Analysis

We conducted 19 semi-structured interviews and 3 contextual interviews in the homes; each one lasted in average 18 minutes for the interviews in the daily center and 30 for the interviews at home. Each interview was audio-recorded and transcribed. During the contextual interviews, we took several pictures to document the use of artifact..

As we will discuss in the next session, routines do not require specific time management activities but what elderly find really difficult to deal with is a sudden change in their habits. For this reason, extraordinary events like appointments and deadlines require a particular attention. In the next session, we will describe several practices that elderly people have spontaneously elaborated to deal with the limits of their perspective memory.

The semi-structured interviews were analyzed by a grounded theory approach. Three researchers independently codified the transcriptions and identified the main emerging categories. The three sets of findings were then reconciled and consolidated as a list of strategies and practices use by the older adults in our group to manage their time, and in particular to plan future events, to remember upcoming events, to reflect on the past experiences and to coordinate with others. Furthermore, a specific attention has been paid to the role and the proprieties of the artifacts involved in these practices and to cultural and emotional values relate to their use. Some less clear aspects were further analyzed with the contextual interviews by focusing the interviews on those specific aspects.

Practices on planning

Planning is mainly related to the scheduling of extraordinary events, such as physical examination, personal care (i.e. haircuts, pedicure) or visiting friends and relatives. As many interviewees pointed out, it is not a critical issue in time management: “I don’t need to plan. I’m not busy. I mean…I really know when I’m available. I’m available every afternoon but Thursday.”

DISCUSSION OF FINDINGS

Planning is not perceived as a critical issue for several reasons. Extraordinary events are not so frequent and daily routines ,which occupy most of the time in elderly life, are very well-consolidated in memory, elderly perfectly know when they are available and will easily avoid any future conflict with extraordinary events. Also elderly routines are very flexible events that that can be changed or modified without any major complication.

How Elderly use time

We have identified three main kinds of events that need to be accommodated in the life of our subjects: routines, extraordinary events and deadlines. Our subjects daily life is mostly characterized by the presence of well-consolidated and regular routines. In general, the first hours of the morning are dedicated to personal care and to house care, in some case also to worshipping. The rest of the morning and the early afternoon are usually dedicated to more social and public activities as shopping, meeting friends or volunteering or recreational activities at the day center (i.e. ceramics, painting courses, etc) and so on Usually routines do not present particular problems in terms of time management. Once they have been defined, they create a rhythm of in the

Even when the elderly people life is full of arrangements, the emergence of future conflicts is unlikely: “Yes, I have many arrangements but I’m not obliged to attend them. I’m old and I like to be free and relaxed.” In general, our subjects do not use external support for planning (only two ladies with a very busy life mentioned the use of calendars and agendas). 4

201

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Practices on remembering

activity. Furthermore, the information recorded in the notes is continuously presented to individuals and their memory is implicitly reinforced day by day.

Differently from planning, remembering up-coming extraordinary events constitutes a critical time management task for older adults: if a considerable amount of time passes from when the appointment is fixed to when it should be attended, it could be very difficult to retain the plan in the prospective memory.

Embedding reminders into daily routines: another practice for maintaining a general awareness of approaching future events is to arrange notes in the physical environment so that they directly interfere with daily routines. For example: “I have a list of what I have to do in the next days in the bathroom…right there, in front of the WC. So, you know, every morning, when I wake up…I can check what I have to do”. Or the following: “I have a wall calendar near the TV. I watch TV every afternoon, after lunch. I mark important things with a red circle. So I can see them from distance. When there are TV ads, I give a look at the calendar…”

This “remembering task” can be articulated in three different sub-tasks: recording, maintaining and recalling. Recording refers to the moment when the date, the time and the place of a specific future event are jotted down information (usually on on a piece of paper). This information need to be stored for a certain period (for example, in the form of a note attached on the fridge) and then recalled at the appropriate time to prompt the intended action. In order to deal with these tasks, elderly people have elaborated different strategies none of them based on the use of digital artifact.

Creating a new routine: several subjects do not to store notes in visible place or to embed them in daily routine, rather they invent new routines to daily check what they have to do in the next days.

Most of our subjects record the information about future events using small sheets of papers, such as Post-It, notebooks pages, etc. In general, they take a note of the future events, by writing down the date, the time and the place. In several cases, a referring person (such as the doctor or the hairdresser) writes the note and hand it to them.

For example, a lady said that she collects all the notes in a drawer bedside the table and every night, before sleeping she controls if she has to do something the next day. Another lady adopt the same strategy collecting the notes in a pocket of her wallet. Moving information from the background to the foreground: a more complex strategy is to move the notes in different location to mark the approaching of an event. For example, “I have a note for every appointment. I stick my notes on a painting near the entrance door […]. Everything is there: appointments; programs; medical examinations and so on. […] I regularly look at them and when I know I have an up-coming appointment, I move the note from the painting to the entrance door so that the morning when I open the door I see it.” (see figure 2).

If the event is far away in the future, a common practice is to transcribe the note content in the calendar or to stick the note near the appropriate date. In few cases people use a sort of dashboard: they create lists of future events and delete them once they have been attended. Another strategy consists in having a dedicated place in the home where to collect such notes. Transcribing or collecting notes in more structured and permanent places reduce the possibility of losing them and allow referring when it is needed.

This excerpt points out an important issue: remembering the time and date of future events is not sufficient to trig an appropriate course of action at the right moment. Routines are really well consolidated scripts in elderly life and they could interfere with the execution of an unusual action schema. This phenomenon is called “strong habit intrusion” [20] and the resulting error is called “capture error” [21] that happens when a frequently used schemas (like “going to work”) take over from a less frequent, less activated schemas (like “visiting a friend”). In order to deal with these kinds of failures of the prospective memory, participants have elaborated different strategies to move the peripheral notes from the background to the foreground at the right moment. The subject mentioned before uses the strategy of moving the note from the showcase to the door so that she cannot avoid looking at them before going out. Another participant is used to transcribe a note from the calendar (used as a long-term storage) and to leave this near the alarm clock.

The effectiveness of a note to remember an event depends on its being noticed at the right time, not too early and not too late. Several strategies emerged from our analysis.

Figure 1. Notes on the fridge and on a dashboard in a living room.

Peripheral visibility: Wall calendars, dashboards and place dedicated to note collection in general, are usually located in the kitchen or in the entrance of elderly people houses (see figure 1). These locations guarantee high visibility: elderly people spend a lot of time in the kitchen and the entrance is a natural home passage. The notes remain in the background and do not interfere with domestic daily

Keeping the note until the end: as noticed before, in most of the cases the note is not written by the older adults but by

202

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

some other person and handed them. In many cases, we fund that elderly people keep the original note even when they transcribe them on the calendar.

emotional and narrative connotation. Another lady uses the calendar more as a sort of micro-diary than as a way of remembering future events: every night, before sleeping, she records with a list of keywords the most important facts of the day, personal thoughts, feelings and emotions. During the contextual interviews, the lady told us that she never throws away a calendar; she has the calendars of the last 12 years and she is used to look at them to remember what she did in the past. “When I’m alone at home, I really like to look at old calendar and say: oh, I did it, oh I did that, oh it was really snowing…”. During the contextual interview, we noted that there were a couple of weeks without any notes. She told us that it was a very sad period of her life and she did not do anything interesting. By discussing with the lady, we elaborated the idea that the calendar works as a sort of dynamic representation of personal moods and daily rhythms; it is a sort of wall display that stimulates self-reflection and contributes to create a self-understanding. By looking at the calendar, which is in a very prominent position in the dining room, this lady has an immediate representation of how she is in the current month.

Elderly people experiences some difficulties with working memory and having the note with all the information written down even if they transcribed them make them feel safer and confident.

Another subject has four calendars, a showcase full of notes, an agenda close to the phone and an agenda in her bag. All these artifacts have not a functional specification: information about future events is distributed across them in a redundant way without following any specific pattern. This practice is a clear example of a psychological need more than an organizational strategy: the calendar, the dashboard and showcases with notes are not only peripheral display that contributes to maintain a contextual awareness of future events, they also create a pervasive context that communicate a feeling of “a life full of arrangements” (see figure 3). As already discussed in literature, being busy and have the perception that time is not wasted is one of the most important value that characterize elderly life.

Figure 2. Notes stored on a dashboard (on the left) and then moved on the door. Practices on Coordination

Most of the interviewed people live alone and are autonomous, coordination with other family member is not a crucial issue. Even when daily routines are shared with other members of the elderly community, the coordination is facilitated by informal social context and it has never been mentioned as problematic in the interviews. However, we have identified an interesting coordination practice related to the moment of taking note of the relevant future event information. As already said, during the contextual interviews we have observed that elderly people often does not personally write the notes: it is the doctor, the day center staff or other people involved in the event that does so. Older adults have often hearing problems and some time also some difficulties in writing because of fine motor limitations, this practice therefore reduces the probability of transcription mistakes and misunderstandings. Practices on Reflection

Figure 3. Calendars with annotation and post-its.

Consistently to the literature of calendar usage, our subjects use the artifacts for managing future events, such as calendar, notes, dashboards also to support reflection on past experiences. For example, a lady mentioned the fact that she decides to plan an appointment with the hairdresser only after a month from the last time she had a haircut.

VALUES

Values are an important dimension that must be taken into consideration when designing for elderly [2]: values shape the practice of use, by directly influencing people attitudes towards products, both in terms of behaviors and emotions. In this paper we have specifically investigated the values embedded in time management practices and we have

However, the most interesting aspect of this topic is that the reconsideration of the past events could have also an 6

203

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

described how these values affect the way people use time management artifacts.

some cases they could also refuse to use them, even if they could positively support their independence.

In our analysis, three values emerged as crucial in relation to time management: the need of proving to themselves and other that they are not wasting their time; the fear of memory loss as an evidence of cognitive decline and need of assessing themselves.

It is important to note that the practices previously described about the physical arrangements in the environments of notes are not in contradiction with this value. The people who adopt such practice declared to have a very busy life and they really stressed this point. In this case, having a busy life not only justify the use of such artifacts but also make their pervasive use an added value. In both cases the ultimate purpose is to maintain and elaborate a representation of the self that is gratifying and acceptable from a cultural and social point of view.

Making a good use of time: social engagement, outdoor physical activities (such as mountain excursion, bicycling) and volunteer work are perceived as useful and rewarding activities that is important to carry out. Spending too much time watching TV, staying alone at home or eating preprepared food are associated with bad moods or negative feelings: “After lunch, I’m used to watch TV. However, after a while I really feel something that is pushing my back…it says me: you have to go out!” Another subject said “Before sleeping I’m used to look back at my day and I’m really happy when I see that I did many things, when I did a wonderful walk in the mountains, I’m really happy”

Assessing myself: a recurrent theme with our subjects is their need to continuously assess their memory skill, for example “I still have a good memory. I remember everything, even the birthday of my grandchildren. Sometime I say: tomorrow it should be the birthday of Giovanni. Then I go to see the calendar - where I wrote down the birthday dates of all my grandchildren -…and I’m always right!” or a similar “I keep my notes in my wallet. I don’t need them. I remember things by myself. I use these notes only to see if I’m right”

From our interviews, it clearly emerges that elderly people like to be busy, to have a day full of arrangements and an intense social life. This confirms many studies that point out the importance for elderly people of being engaged with the social and the cultural world.

This value is deeply related to what we have described in the previous section and to the importance that elderly people attribute to memory skills in the construction of the self. As it possible to observe in the excerpts, this subtle affects the way elderly make use of time management artifacts: they are used not only as tools to manage the upcoming future but they also support elderly people in their activity of constantly proving their memory skills.

This value influences how elderly people use artifacts for organizing their life. It is closely related to the practice of reflection. As already discussed, artifacts used for time management not only have a functional role: they are also used to support reflection about personal lifestyle. By looking to a showcase full of notes, most of our subjects elaborate the impression that they are actively living their life and still engaged with the “real” world.

DESIGN OPPORTUNITIES

By elaborating on the analysis previously presented, we have defined a set of design dimensions that are aimed to drive the development of time-management technology in the context of elderly life.

Luckily, I don’t need it: during one of the first semistructured interview, a subject answers to the question about what artifacts she uses to remember things: “luckily, I don’t need them. Not yet.”. This lady perceives artifacts for time management as evidences of memory problems; by saying she does not need artifacts for remembering things, she is asserting her cognitive well being.

Beyond functionality

When designing digital artifacts for time management in the context of elderly life, it may be useful to widen the design space and not considering only functional aspects but also the personal, emotional, narrative and leisure dimensions.

This theme emerged quite often in the analysis in a positive or in a negative form: some subjects were proud of their memory skills while some were feared the possibility of losing memory in the next years. Several contradicted themselves by neglecting the use of artifacts for remembering things to do and at the same time “confessing” to have a calendar for the appointments far away in the future.

In general, we have observed that time management artifacts have a double nature: on one side they are used to support functional task such as remembering upcoming events or planning future appointments; on the other side they enable elders in making sense of time, by stimulating personal reflections. In respect to this latter aspect, we can identify two different uses: artifacts used to support the elaboration of a narrative account of past and present experiences and artifacts used as a sort of “gaming platform” to test memory skills.

Since artifacts for time management are often associated with memory aids, they have also a symbolic value. Some people see them as the evidence of their diminished memory skills. This has an impact on practices of use: elderly people could try to minimize the use of such artifacts, to make them as less visible as possible and in

Narrative account: task management tools, such as notes, calendars or dashboards can also used as object to think with, in order to make sense of time and elaborate a

204

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

background to the background” we have describe how it is necessary to manipulate notes visibility in order to appropriately support the recalling process.

narrative and emotional account of personal life. From a design point of view, it could be interesting to investigate this dimension and to explore how to enrich the experience of use by working on the aesthetics of interaction [23] or by allowing people to integrate more personal contents.

In the section “luckily, I don’t need it” we have described the symbolic value of artifacts used for time management: they could be perceived as evidences of memory impairments. As a consequence elderly may tend to minimize their use or to make them invisible.

Gaming platform: time management artifacts could be used as a gaming platform to test personal memory skills. This constitutes an interesting design opportunity to change people attitude toward time management artifacts. Rather than being perceived as passive memory aids which could have a negative value, time management artifacts could be interpreted as tools that actively support elderly in improving or maintaining memory skills.

Time management artifacts should be considered within these complex set of values and practices; the way they are concretely used and adopted critically depends on personal attitudes. From a design perspective it is important to develop flexible technology that allow elderly to configure them according to their individual preferences. In particular, in respect to this class of artifacts, it is important to consider the visibility of information related to future events as a resource. Elderly should be enabled to modify and configure such a resource by moving information from the background to foreground or even making them invisible.

Beyond the domestic space

The design space may also be enlarged beyond the domestic space by taking into consideration a variety of different context of use. The whole lifecycle of event management might be considered: since when the information related to a future event is defined and recorded to when it is stored on a more persistent support, to when the information related to the event are referred to prompt the appropriate course of actions at the right time to when such information are used to support the older adult in arriving at destination and attending the event.

Proactive devices

As we have discussed in section “Making a good use of time”, elderly people are continuously looking on how to occupy their time: indeed, programs (such as theater seasons) are often near to their showcases and calendars.

The interaction context varies along with the evolution of the time management task: during the recording of the information related to a future event, elderly could be in an office (i.e. in front of an bank employee or a member of the day center staff); the temporary storage and the recalling of a note generally occurs in the domestic environment; then elderly tend to keep the note with themselves until the end of the attended event (this is a completely mobile context where the note is used, for example, to support elderly in remembering the appointment address). In particular, in the recording phase it is important to support the coordination between the older adult and other person involved: as we have discussed in the “coordinating” session, in order to minimize the probability of misunderstanding the note is written down by the other person involved in the event and not directly by the older adult.

This practice constitutes a design opportunity. We can consider time management artifact not only as passive tools that elderly people use to organize their life but also as proactive tools that suggest opportunities and possibilities. Calendars, dashboards and even showcases can be considered as interfaces to a wide set of services: by using such a familiar interface, elderly people could have the opportunity to navigate their future event reminders together with the proposals of public or private institutions such as cinemas, theaters, local events and so on. Public institution could also use these artifacts to communicate with elderly customers and present relevant information such as periodic physical examinations in the context of their daily life.

The design of task management technology should therefore take into consideration how information related future events are recorded, manipulated and propagated across different media, physical and social contexts, and it should guarantee a fluid user experience across different spaces and situations.

CONCLUSION

In this paper, we presented the results of a qualitative research aimed at understanding the time management practices of older adults. In particular, it emerged that although a busy life is often considered an important value that characterize elderly life, time management tools can carry a stigma of a prosthesis.

Visibility / Invisibility

In the section “peripheral visibility” we have described how elderly make arrangements of notes in the physical environment to support a peripheral awareness of future events they have to attend. In the “reflection” section we have discussed the role of such arrangements beyond its pure functional role (see also “narrative account”). Furthermore, in the section “moving information from the

Although the user group is limited in size, gender and geographical distribution, we feel that the issues emerged might have a broader validity and may contribute to a greater understanding on how to design for older adults.

8

205

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ACKNOWLEDGMENTS

11.Plaisant, C., Bederson, B. B., Clamage, A., Hutchinson, H., Druin, A., Shared Family. Calendars: Promoting Symmetry and Accessibility, ACM Transactions on Computer-Human. Interaction, 13, 3 (2006) 313-346

The work of this paper has been funded by the Internet of Services Joint Research Project at FBK. We would like to thank the “Centro Belenzani” day center in Trento and the people that participated in the study.

12.Hirsch, T., Forlizzi, J., Hyder, E., Goetz, J., Kurtz, C., and Stroback, J. 2000. The ELDer project: social, emotional, and environmental factors in the design of eldercare technologies. In Proceedings on the 2000 Conference on Universal Usability. Arlington, Virginia, 2000.

REFERENCES

1. McCreadie C and Tinker A. (2005) The acceptability of assistive technology to older people. Ageing and Society, 25, 1, pp 91-110. 2. Forlizzi, J., DiSalvo, C., and Gemperle, F. (2004). Assistive Robotics and an Ecology of Elders Living Independently in Their Homes. Journal of HCI Special Issue on Human-Robot Interaction, V19 N1/2, January, 2004.

13. Leonardi C., Mennecozzi C., Not E., Pianesi F., Gennai F., Cristoforetti A. Knocking on Elders’ Door Investigating the Functional and Emotional Geography of their Domestic Space. In Proceedings of ACM CHI 2009. April 2009, Boston, Mass.

3. Palen L., (1999) Social, individual and technological issues for groupware calendar systems, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.17-24, May 15-20, 1999, Pittsburgh, Pennsylvania, United States

14.Eisma, R., Dickinson, A., Goodman, J., Syme, A., Tiwari, L. and Newell, A.F. Early user involvement in the development of information technology related products for older people. International Journal Universal Access in the Information Society, 3(2), 2004, Lin1-140 Academia Press.

4. Crabtree, A., Hemmings, T., Rodden, T., and Mariani, J. 2003. Informing the development of calendar systems for domestic use. In Proceedings of the Eighth Conference on European Conference on Computer Supported Cooperative Work (Helsinki, Finland, September 14 - 18, 2003). K. Kuutti, E. H. Karsten, G. Fitzpatrick, P. Dourish, and K. Schmidt, Eds. ECSCW. Kluwer Academic Publishers, Norwell, MA, 119-138.

15.Marti, P., Giusti, L., Crafting Sensorial Stimulation and Social Exchange in Dementia Care, Proceedings of the 6th Conference of the International Society for Gerontechnology - ISG08, Pisa, Italy, May 20 - 23, 2008. 16.Dickinson, A., Goodman, J., Syme, A., Eisma ,R., Tiwari, L., Mival, O. and Newell A. Domesticating Technology: In-home requirements gathering with frail older people. Proc. of 10th International Conference on Human - Computer Interaction HCI (2003), 827-831 the Home. Housing Studies, Vol. 19, No. 5 (2005) 745– 763.

5. Taylor, A. S. and Swan, L. (2004) List making in the home. Computer Supported Collaborative Work, CSCW '04. Chicago, IL, 542-545 6. Lin, M., Lutters, W. G., and Kim, T. S. 2004. Understanding the Micronote Lifecycle: Improving Mobile Support for Informal Note Taking. In Proc. CHI '04. ACM Press, 687-694

17.Lindlof T. R., Taylor B. C., (1995) Qualitative communication research methods. Thousand Oaks, CA: Sage Publications, p. 195.

7. Payne, S. J. Understanding calendar use. HumanComputer Interaction 8, 2 (1993), 83--100.

18.Hugh Beyer, Karen Holtzblatt, (1998) Contextual design: defining customer-centered systems, Morgan Kaufmann Publishers Inc., San Francisco, CA.

8. Tomitsch, M., Grechenig, T. & Wascher, P. (2006). Personal and Private Calendar Interfaces support Private Patterns: e.g. Diaries, Relations, Emotional Expressions. In Proceedings of the Nordic conference on Humancomputer interaction

19.Glaser, B. G. and Strauss, A. L. (1967), The Discovery of Grounded Theory: Strategies for Qualitative Research. New York, NY: Aldine.

9. M. Brzozowski, K. Carattini, S. R. Klemmer, P. Mihelich,. J. Hu, and A. Y. Ng. groupTime: preferencebased group scheduling. In Proc. of CHI'06, pages 1047–1056, 2006

20.Reason, J. (1990). Human error. Cambridge University Press 21.Norman, D. A. (1981). Categorization of action slips. Psychological Review, 88, 1-15.

10.Doimo, L. A. ; Derntl, A. M. , (2006) Use of time in elderly people´s daily lives: an indicator method for lifestyle in old age.. BRAZILIAN JOURNAL OF GERIATRICS AND GERONTOLOGY, v. 9, p. 37-53.

22. Turkle, S. (2007) Evocative Objects --- Things we think with, MIT Press. 23. Rullo, A. 2008. The soft qualities of interaction. ACM Trans. Comput.-Hum. Interact. 15, 4 (Nov. 2008), 1-25.

206

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Studying Mobile Context-aware Social Services in the Wild Paul Holleis, Matthias Wagner

Sebastian Böhm

Johan Koolwaaij

DOCOMO Euro-Labs Landsberger Str. 312 80687 Munich, Germany {holleis, wagner}@docomolab-euro.com

net mobile AG Zollhof 17 40221 Düsseldorf, Germany [email protected]

Novay Brouwerijstraat 1 7523 XC Enschede, NL [email protected]

ABSTRACT

involves sensible context data has not been well exploited yet. In addition to technical aspects in the implementation of context-aware social services like IYOUIT, our research focuses on the usability, acceptance and market potential of such systems in practice. In this paper, we report on the results of a longitudinal user trial with 19 users from 6 countries over one month. We extend and discuss our longterm experiences made by releasing IYOUIT to the public since more than one year with the findings from this study.

We have implemented and evaluated IYOUIT, a contextaware application for the mobile phone that promotes a digital lifestyle, sharing, and life-logging approach for people on the go. The service incorporates context management technology to abstract data about and around the user into meaningful interpretations of the user‟s digital trace in the real world. Complementary to the public release of our service, we have conducted a longitudinal field study with 19 users for a period of one month. In this paper, we present findings from this coordinated user trial and provide researchers with advice on the design and implementation of similar systems.

After some related work, we present a system overview, followed by the setup and methodology of the study. After an in-depth analysis of the conducted user trial we summarize the results and derive guidelines for the design of context-aware mobile applications.

Author Keywords

Mobile Services, Context Awareness, Social Networking.

RELATED WORK

ACM Classification Keywords

IYOUIT comprises a broad range of concepts, ideas, and services into one system with various interfaces. To the best of our knowledge, no other system provides a comparable diversity of context-aware social services, also including an extensive mobile client application. A good overview on context-aware mobile computing research until the year 2000 is given by Chen and Kotz [5]. Interestingly, of the 19 projects they analyzed, only two used context information other than location (and the current time). These two added some notion of activity mostly recognized by specifically deployed sensors. In contrast, IYOUIT uses available phone sensors to sense and deduce a great variety of context information. Hong et al. [13] further report on general context-aware systems published until 2007. They categorize them according to five layers: concept and research, network, middleware, user interface, and application layer with various sub-categories. The diversity and rich feature set of IYOUIT is exemplified by the fact that it covers a total of 8 of the 19 categories of the first four layers and has elements from half of the application areas (information systems, communication systems, and web service). The authors also show that very few research projects provide a fully functional and publically available application.

H.5. Information interfaces and presentation (e.g., HCI): Miscellaneous. H.1.2 User/Machine Systems INTRODUCTION

IYOUIT is a mobile application that allows users to automatically collect context information centered on visited places and interactions with other people. It aims at making it easy to collect such data on a standard phone and facilitates instant and light-hearted sharing of personal experiences within communities together with rich contextual tagging for everyday life. In combining and aggregating context sources, IYOUIT derives a rich portfolio of personal information to be shared with others on the Web and on the mobile phone. By hooking up to Web 2.0 services like Flickr, Facebook, and Twitter, IYOUIT extends sharing experiences to social online communities commonly regarded as being widely adopted for the exchange of personal information. However, adding a mobile dimension to social communities that

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

In the last years, many academic and commercial social platform projects have been created. Many of them are specific to a particular type of information. For example, Picasa, Snapfish, and Panoramio concentrate on sharing images, Google Latitude, Shizzow, and Plazes help people share their current location, and EveryTrail, Dopplr, and

207

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

and mash-ups with other services and applications. The framework consists of two main parts. Management components, e.g., ensure secure authentication, implement finegrained access control, and allow the definition and exploitation of context ontologies. On the other hand, context provider components wrap basic context data sources and can implement aggregations and abstractions. The framework has been designed for an open integration with 3rd party services and is leveraging Semantic Web technology in various places. The platform can be accessed through a web platform and a mobile client application.

TripIt are used mainly for sharing travel plans. Obviously, several platforms exist that allow people to share messages, updates, and stories. Examples include Twitter, blogging, and other community platforms (see below) for short messages and status updates. As IYOUIT incorporates all these features, it can be used by a single person as a notebook and „digital life recorder‟ for his or her history of location and activity, encounters with others, etc. However, its true potential is unfolded with a community of users. On the same grounds, several social community platforms and applications have emerged. Most of them are specialized to some domain (e.g. LinkedIn for business; StudiVZ for education), restricted to some type of content (e.g. Flixster to movies; Last.fm to music), or are only available or popular within a certain region (e.g. Mixi in Japan; Qzone in China). Similar to IYOUIT, many of those have a web frontend and a mobile client application. However, especially the most popular platforms worldwide such as Facebook, Myspace, Orkut, and QZone do not exploit the rich feature-set of current smartphones and merely provide a simplified interface of their web portals. IYOUIT directly uses phone capabilities (see, e.g. [25] for a similar approach) and the users‟ mobility to create profiles and offer multiple services described below. In order to evaluate our system under real conditions and provide results with respect to its use, we chose a longitudinal field trial. Controlled lab trials or artificial test environments such as the one used for the evaluation of the health-monitoring application by Morla and Davies [15] cannot fully satisfy our needs as we wanted to focus less on the mere usability of the system and more on how it is perceived and used in the real world. There have in fact been very few studies examining the use of such powerful context-aware platforms with web-based and mobile interfaces. Most existing studies e.g. by Tungare et al. [23] concentrate on specific aspects or ways of applying the system or, like O‟Grady et al. [17], focus very much on the usability of the specific mobile application and its interface. In contrast, we were more interested in how users accept and use our system. A similar goal has been followed by Jacucci et al. [26]. However, their system is targeted at a quite specific usage scenario (communication at large events) and was trialed in very specific settings over a short period of time.

Figure 1: The IYOUIT component framework and connected 3rd party services. The Mobile Client

IYOUIT has been designed to run 24/7 in the background on modern phones. Our current implementation supports all phones of Nokia‟s S60 platform. Figure 2 shows some screenshots of the mobile client‟s interface. Each tab either displays a certain type of context information or accumulates various pieces of information in a contextual overview that helps assessing the situation of a user at a given point in time: Me: the home tab of IYOUIT and the standard entry point. It summarizes recently spotted information in the user‟s proximity, highlights the latest presence status, and provides pointers to more detailed information in other tabs.

THE IYOUIT PLATFORM AND SYSTEM

In order to cope with the plurality of context, IYOUIT is built upon a Context Management Framework (CMF) which hosts and combines various services and data sources [3]. As a prototype from our labs, IYOUIT is available free of charge. In addition to a mobile client available for Nokia S60 phones, we have implemented our own IYOUIT Web portal1 and open APIs to support 3rd party developments 1

Buddy: shows the real-time context of all social contacts, including their location, status updates, latest activities, shared photos, and more. It offers different views on context, various sorting orders and context-dependent groups. Places: summarizes the user‟s current whereabouts. The client uses GPS coordinates when available. Otherwise, the location is fixed using cell IDs or exploits existing location

http://www.iyouit.eu

208

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2: Various tabs of IYOUIT’s mobile client application.

information stored by the IYOUIT user community (e.g. at WiFi hotspots or fixed Bluetooth devices). Using statistical learning and clustering methods based on [16], places (abstractions from exact location values) are generated. These represent frequently visited places and map to basic semantic concepts such as the user‟s „home‟ or „office‟.

Initially, the level of accessible information is defined by the manually specified relationship to a contact (e.g. „wife‟, „colleague‟). In addition, it can be controlled at various levels (e.g. for all location updates or a single photo). Further tabs exist for displaying local information on a map, for showing GPS details and statistical overviews of the personal context history. Automatic context tagging is also available for books and products (via bar code scans) as well as sound and video recordings.

Experience: allows setting IM-compliant presence attributes like activity, mood or availability. Small text notes are automatically complemented with personal context information to be posted, e.g., on Twitter. Using built-in accelerometers measuring movement and gestures, we can automatically set states such as „sleeping‟ or „walking‟. Further (semi-)automatic methods are active research [11].

The Web Portal

While the primary personal use of IYOUIT is on the mobile phone, the application also materializes on the Web in its own portal. Daily hits on the IYOUIT Web portal currently average 100.000+ requests.

Photo: added pictures are proactively tagged with available context data and can be uploaded to 3rd party services such as Flickr. Beyond widely used tags such as geographic information, IYOUIT adds semantic descriptions like nearby people with the corresponding social relationships, the current place, qualitative weather annotations, etc. As a consequence, managing and sharing photos becomes more intuitive. For instance one can ‟find all family pictures taken at the holiday cottage on a sunny day„ or ‟subscribe to photos captured by friends on the university campus‟.

The Web portal offers similar interfaces as the mobile client but can be more suitable for the display and configuration of complex data. It shows latest public observations and selected bits of collected context together with hints on how the community is evolving. Users can conveniently define social relationships and configure personal privacy policies. Furthermore, the portal offers a hub to various other online services. Users can thus decide to share their personal context with, for instance, Flickr, Twitter, Dopplr or YouTube. The IYOUIT portal also serves as test bed for selected features from our research lab. In an experimental engine, for instance, users can configure automatic context blogging or profiling.

Events: displays a timeline of recently observed significant changes in the user context. Here, entries to places of interest are listed together with social encounters, network information as well as activities observed, e.g., by the accelerometer sensor of the phone. Actions such as switching the phone profile from „outdoor‟ to „meeting‟, can be attached to places. Similarly, activities such as „working‟ may be bound to nearby people.

Given the particular focus on real-time sharing aspects of personal data in the user study, the web portal and its features to display context in an aggregated view over a longer period of time has been of minor interest in this trial.

Scan: collects technical observations in terms of scanned Bluetooth devices and spotted WLAN access points. Observed devices can be associated with buddies while scanned WLAN access points are geo-tagged. Both can be associated with rules to trigger proactive actions such as setting the user‟s experience or automatically subscribing to WLAN network access in a certain place.

Public IYOUIT Community

IYOUIT makes it relatively easy to gather and share personal context data. During the design and implementation of the system, our assumption has always been that people will appreciate such sharing features given their privacy is preserved and under their control. We have started to test this assumption by releasing the system for public and free use about two years ago. For this reason, IYOUIT has also been promoted and eventually awarded as part of the

The owner of information (the user whose context is being sensed) has full control about what is accessible by whom.

209

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Mobile Experience at MobileHCI‟082. Since IYOUIT‟s public release mid 2008, more than 1.500 users from 50 countries registered with the service. More than 1.5 million location measurements were recorded, 10.000+ pictures were tagged and uploaded, and more than 55.000 local information items, such as weather forecasts, were requested. Albeit already of great value through direct feedback and observation, the global IYOUIT user community provides only limited contribution to a comprehensive and complete understanding of our system. To learn more about how mobile users perceive contextual tracking and sharing as well as other application features implemented in IYOUIT, a profound user study with a clear focus and a relatively controlled setting was indispensible.

collected feedback about the general concept, user experience and behavior, and application specific features and interfaces.

EVALUATION Figure 3: Demographics of the 19 study participants.

The study on which we report in this paper took place in 2009 and lasted four weeks. One main target of the study was to find details about people‟s acceptance of general context-aware sharing concepts and various special features within the IYOUIT system.

Final interviews: we also managed to have individual, faceto-face interviews with 14 participants to clarify some of their answers and to find reasons behind their judgment. Study Participants

Study Setup

We tried to recruit a variety of different user types from the existing user base and also encouraged additional persons who had not used IYOUIT before. In the end, we covered a range of different nationalities and age groups (26-65 with an average of 37 years, see Figure 3), 15 male, 4 female.

In the following, we briefly give an overview on the structure, methods, and tools we used. Initial tutorial: we met with all study participants to introduce the IYOUIT system, distribute compatible handsets if necessary and configure the application if not done already.

Table 1: Previous experiences and participation of the users.

Demographics: demographic information about participants was collected in this session with a short questionnaire.

Participants Characteristics Use of Facebook social platform Use of Flickr photo sharing Use of Twitter micro blogging Use of a location sharing platform

Triggering participation: as stated in Froehlich et al. [9], "participant compliance tends to decrease over time" and triggers can help to counter such trends. We provided weekly challenges and announced small prizes for those who would provide most constructive feedback. In fact, however, the consequences were very limited: only three persons attempted to address some of the tasks, and those only within the first two weeks. Thus, we can practically rule out any effect on the results of the study.

One half of the participants (47%) already knew IYOUIT (but only 21% for longer than 6 months before the study). We also aimed at a mixture of previous experiences with such platforms to study a potential correlation between the actual use of our system during the trial and previous experiences with similar services. Table 1 shows some of the characteristics.

Data Elicitation

During the study, we used three ways of gathering quantitative and qualitative data:

With respect to picture sharing, our sample mostly had experiences with Flickr or similar services. Regarding social community platforms, in total 17 systems have been in use by at least one participant. Among these, Facebook and LinkedIn were most prominent (mentioned >5 times).

Logging: the mobile IYOUIT client was fitted with additional routines to log and transmit information about actions and context experienced throughout the study in order to reinforce subjective opinions with actual behavior.

ANALYSIS

For this paper, we distilled results about four broad areas of interest and describe the instances from which we elicited the data: general use (usage data and personal feedback), context acquisition and sharing (photo sharing), context aggregation and abstraction (location and a concept called places), as well as social networking and privacy (connections to 3rd party sharing platforms and detailed treatment in the questionnaires).

Questionnaires: most subjective data has been gathered through questionnaires distributed by email as well as the IYOUIT Web portal. An initial questionnaire gathered demographic data and clarified previous experiences. After two weeks, a questionnaire checked people‟s participation and tried to spot trends. A final, elaborate questionnaire 2

Rate 58 % 53 % 32 % 21 %

http://mobilehci2008.telin.nl/?sub1=89

210

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

General Acceptance and Use

ly. Surprisingly however, and in contrast to what people indicated, the Experience Tab initiated more than twice as many interactions as the Location Tab. This indicates two aspects: the implementation and presentation of location information has been done in a way that needs little user interaction; and people are in fact interested in collecting and sharing more information than their position.

The application usage throughout the study was very high but varied largely between participants. As Figure 4 shows, the average uptime of the mobile client during the trial period was 38%. Given that these numbers assume a 24/7 usage, the measured average is actually surprisingly high.

Figure 4: Uptime of the mobile client application (avg. 38%). Figure 6: Types of context most important to users.

Although increased power consumption was mentioned as a major issue of the system by 47% of the participants, Figure 5 indicates hardly any correlation (r2 = 0.65) between problems due to power consumption and the actual runtime. The average number of times that people had the phone powered off because of an empty battery was only 4 times. In the end, battery problems seem to explain only 12% of the cases in which people had IYOUIT not running for some time. This is supported, e.g., by Froehlich et al. [9] who show that people most often recharge their phones considerably before the battery reaches a low level anyway. In addition, IYOUIT also offers several opportunities to conserve energy, e.g. cell triangulation instead of GPS for location estimation and an offline mode.

Still, location-based services are of high importance as the appearance of the Location, GPS, and Map tabs within the chart of Figure 7 proves. Less interesting seem to have been the possibilities to generate statistics (Statistics and Graphs) about past history of context of oneself or others. This correlates to the fact that no one described context history to be of top interest (Figure 6). However, this could potentially be improved with a better and more interesting user interface or more features such as comparing statistics across users. In fact, in the interviews, 43% pointed out that they would be interested in using IYOUIT as a life-logging application and a dedicated diary service was even requested as additional feature. This indicates a strong need for automatically generating and presenting context to the user in an appropriate way (cf. concept of places explained below). One interesting result from our final questionnaire was that people stated they learned habits of their buddies (2.9 on a Likert-scale from 0 to 4, σ = 1.0): “I just found out that a colleague’s hobby is airplane spotting! This buddy-based sharing is definitely a worthwhile feature for the application.” Japanese, male, 34 years.

Figure 5: Number of empty battery occurrences per users.

Context Acquisition and Sharing – Photos

To see which context categories are of most interest to users, we had the participants specify the top three most important types of information (cf. Figure 6). Location and pictures head the list followed by weather information. Curiously, people‟s activities and mood have not been mentioned very often. One advantage of running this study is that we were able to compare these statements with the actual behavior within the system.

The IYOUIT system provides an excellent framework for photo sharing. Pictures taken with the mobile client are automatically augmented with context information such as location, activity, or mood gathered by the application. Additionally, the user can manually associate tags and descriptions with photos. In conjunction with querying and grouping mechanisms, all this information helps in later retrieving specific pictures (using, e.g., information about the weather conditions or nearby people). As IYOUIT can be directly and seamlessly connected to 3rd party services such as Flickr and Facebook, features from these systems like Flickr‟s photo sets capability can directly be exploited.

Figure 7 provides more details about the most frequent use of the IYOUIT client. As expected, people were attentively following the context of their buddies. Also the Photo Tab for taking, tagging, and sharing pictures was used frequent-

211

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 7: Relative amount of use of the client’s different tabs.

Figure 9: Photos with manual tags and transferred to Flickr.

During the trial month, 540 photos were uploaded allowing us to derive interesting insights into the way people share and describe pictures. First, the number of uploads is not related to the uptime of the client. Second, as expected, our current system could not fully challenge dedicated picture viewing applications such as Flickr as it was not designed for this purpose. However, its feature set was rich enough to be attractive and often used. Automated tagging and the transfer to Flickr were very much appreciated. This can be seen in Figure 9 that shows the percentage of photos with manual tags and descriptions. The right bars show only those users that have manually tagged or described at least one picture during the month. The numbers are unexpectedly high as adding descriptions requires some effort: use of the menu and entering text on the phone. This indicates that adding information to pictures is important and IYOUIT can help greatly by automatically adding tags and a title 3.

One main issue that arose within the feedback through the questionnaires and interviews was that many people prefer their digital camera to the mobile phone camera, e.g. because of image quality, flash, and zoom capabilities. Therefore, it has been very well received that IYOUIT also allows for post-processing Flickr images: pictures uploaded from an external camera can be automatically augmented with related context tags at a later point in time.

Figure 10: Categories of manual tags and descriptions. Context Abstraction and Presentation – Places

IYOUIT collects much data while in use. This raw data is often only of limited value to the user and becomes meaningful only after it has been combined and abstracted. IYOUIT uses context ontologies and Semantic Web technologies to implement key abstractions and reasoning features. We refer to Böhm et al. [4] for the details. We exemplify context abstraction through the concept of places: an IYOUIT place is a semantic concept that abstracts from a geographic position and describes it by type and name. Places may expand to cover whole areas. The concept of places is needed for three reasons: first, location measurement on the phone has to rely on imprecise data (Fagerberg et al. [8], note that “place labels are useful no matter the granularity of the positioning system”); second, several real world locations consist of the same place – e.g. within an office building; third, abstractions are necessary to give a more appropriate meaning to locations, e.g. “office” instead of a street (see, e.g. Ashbrook and Starner [1]). As has been stated before, location is an import type of context and people often rely on and share such knowledge. Grandhi et al. [10], e.g, found that up to 77% of people observed in New York over a 3 weeks period were willing to share their location data (17% even with total strangers).

Figure 8: Number of photos shared using IYOUIT (avg. 30).

Out of all tags that have been manually assigned by the users, five categories of tags have been identified. A large number of tags detail the location where the photo has been taken (Figure 10). The mobile application actually could have derived 53% of those (such as city, proximity to a recreational area, or the position close to some landmark). Two other main categories describe some detail depicted on the photo (such as a person or event) or some activity. Descriptions are longer but used for the same purposes, but favor detail over location (e.g. if covered by a tag). These findings strengthen the need for a system providing smart algorithms for automatic tagging that, at the same time, allows for manually adjusting or adding information. 3

See http://bit.ly/IYOUITpic for an example picture

212

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The IYOUIT system holds an ontology of places shared by everyone to coherently share places between users (e.g. „home‟ vs. „office‟). Participants used 39 different place types for 235 places indicating that this mechanism is both appreciated and used. Similar to setting experiences manually, we suspected that the manual specification of places would be too much work for most users. Therefore, complementary to manual places, IYOUIT runs specialized data mining algorithms (cf. Nurmi and Koolwaaij [16]) over aggregated location traces and proactively generates place suggestions. Similarly to the reasoning by Ashbrook and Starner [1] as well as Hightower et al. [12], this is mostly based on the amount of time the user stays within a certain region. We also abstract from concrete positioning methods taking inherent (in)accuracy into account. The user is provided with such place suggestions and can easily decide to name and typify, to ignore, or delete them. This was very well received (about two-thirds of all suggestions adopted) and acknowledged in the questionnaires and interviews.

Although places are obviously very personal, we introduced a feature that enables people to inspect and subscribe to places of others (only for places explicitly marked as visible by buddies or the public). Within the trial group, already about 6% of the places have been used in such a way. This is one of the features we think has a high value, especially as soon as the IYOUIT community reaches a certain size and more places are being created.

Figure 12: Time users spent within a specific place (avg. 69%).

As a side result, it is interesting to see that many people seem to move between places quite often. The data gives an average of 8.9 times (σ = 7.6) that a user enters another place per day. Still, this seems to be quite specific to individual users (also indicated by the high standard deviation). Judging from the available data, one could categorize users into the three clusters „rather stationary‟, „average mobile‟, and „very mobile‟ with the distribution within the trial group being 8-5-3. This also suggests that specifying rules based on places, e.g. automatically setting one‟s status or activity, can provide a constant stream of sensible data.

Figure 11: Manually named and subsequently visited places. 4

Figure 11 shows the distribution of named places of each user with an average of about 15 places per user. The same chart also displays the number of places that have actually been visited during the trial period (this includes unnamed places, see below). Our usage data over a longer period of time shows that, for the majority of people, an average of 12 places seem to cover most regularly visited areas of interest. This is in accordance with previous research, e.g. by Hightower et al. [12] who found that an average of 7 to 8 places are visited regularly within a year and Zhou et al. [24] who discovered on average 15 significant places.

Privacy and Social Networking

From the user‟s point of view, IYOUIT is primarily a sharing platform for personal information. Here, the social aspects of sharing data and communicating with people, indirectly with context updates and directly by sending messages, blogging, or commenting on other‟s context are vital to the application. Privacy Policies and Directives

A Privacy Manager is an integral part of the IYOUIT system (for details see Böhm et al. [3]). In contrast to most platforms such as Facebook that allow full access to personal information for one category of „friends‟, IYOUIT extends to more defined categories of social contacts and makes sharing fully bound to the personal social network through a fine-grained access control. Previous studies, e.g. Palen and Dourish [19], suggest giving users full control over privacy directives. In IYOUIT, this has been fully implemented by controlling access to personal information with a general system of rules defined by the user and resource specific access directives as an exception to the general rules. These fine-grained access control mechanisms allow for the definition of rules for groups of people or specific buddies, for distinct context categories as well as the level of detail that is being disclosed (a requirement also

Furthermore, people overall spent over two-thirds (69%) of the time they used IYOUIT in the defined area of a place (Figure 12). In fact, some people have spent about 75% of their time within one specific place (provided one can meaningfully extrapolate beyond the uptime of the client). This statistic is actually very similar when considering the whole IYOUIT community during the trial period with an average value of 67% to be within a defined place, thus further confirming the validity of the place concept. 4

Note that in this and the following chart, only the data of 16 participants are visualized as the short usage times of the 3 users who had IYOUIT running for less than 2% of the time could lead to inappropriate values.

213

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

stressed by Consolvo et al. [6]). This means that, for example, access to the user‟s current location can be prohibited completely or allowed at a country, city or concrete address level. Users may also separately choose to grant access to historic information. Resource specific directives, on the other hand, provide means to explicitly choose the level of disclosure for specific items (such as a picture) case-bycase, independent of the general directive for this context category.

seem to fit the way people interpret these and want to have them: the user interface of the privacy manager was actually only accessed by 9 of the 19 users and only 6 of them modified the default directives. On the other hand, privacy directives could be adapted individually from the mobile client as well. The users‟ explicit opinions varied: 3 out of 17 people said to be fully comfortable in sharing their context with their buddies. However, in the trial, only one user actually granted full access for his buddies. 4 persons felt uncomfortable with sharing but again only one completely restricted access to others. This is similar to the findings reported by Consolvo et al. [6] who state that the Privacy Segmentation Model [18] into unconcerned, pragmatists, and fundamentalists through interviews does not seem to be a good predictor for actual behavior.

As Sadeh et al. [22] state, giving the user such a flexible and powerful control over privacy and sharing settings can increase the adoption of context-aware services. They also show ways of incorporating more complex rules for granting or denying access to information and provide guidelines in this direction. However, as they and Palen and Dourish [19] correctly point out, this can always be only partly a solution since (location) privacy is a "dynamic process of managing boundaries". Interestingly, the willingness to choose the level of disclosure for specific items rather than just relying on the previously defined rules or default settings varies greatly for different context categories. Our study participants often wanted to specify the sharing level of photos individually, whereas the sensitivity of location has been seen as rather static. For example, the majority of 62% of pictures were uploaded to be publicly visible, while 30% were for buddies only, and the remainder was only visible for the authors themselves. This indicates that there is a definite need for privacy preserving mechanisms and that they should be adapted according to the type of data.

Figure 13: Relationships used to categorize buddies. Social Networking Platform

In contrast to some systems for which a direct connection between usefulness and number of users is well known as the critical mass (see Rogers [21]), IYOUIT is already valuable without any or with only a small number of buddies. Yet, after the end of our study, everyone had a relatively large network of 2 to 25 buddies with an average of 10 and a total of 184 explicitly defined relationships. shows how study participants categorized their buddy relationships with 63% being tagged as business contacts, 7% as family, and 30% as friends. The high number of business relationships could indicate a potential use in the corporate setting, an application area that has in fact been suggested by several interviewees (Bardram and Hansen [2] show an application for hospitals using context to trigger social awareness between colleagues).

One finding derived from personal feedback and interviews is that it seems to become easier to define meaningful privacy directives when changing from low-level data to a higher level of abstraction. For example, most users easily understood how to judge the sensitivity level of places, whereas only more „expert‟ users were able to express the corresponding rules for spatial data in general. As a positive result, we regard that 79% were comfortable with sharing context with their buddies through IYOUIT. Furthermore, 63% of all trial participants expressed no need for a more sophisticated privacy management procedure. Others mainly argued for adding control about which picture is transferred to which 3rd party platform with which privacy setting or have not found the opportunities that the current system offers.

In order to make the most out of the IYOUIT platform and to increase added value, IYOUIT can be configured to connect to existing 3rd party services such as photo sharing platforms (Flickr), blogging platforms (Twitter), and social community platforms (Facebook). The automatic distribution of information across services was valued highly by many users. However, the actual representation or configuration of information within, e.g. Facebook needs to be revised as the following comment made clear:

We can support the discovery by various works [6, 20, 22] that people‟s sharing behavior depends on types or groups of people such as co-workers or close friends. To this end, IYOUIT allows to define personal social networks of buddies by defined and typed relationships such as „friend‟, „brother‟, „colleague‟, etc. which are more comprehensive than in most other systems. Relations are pre-configured in privacy settings that can be adjusted individually but mostly

“My friends really wondered why I suddenly started to comment on me waking up!” Italian, male, 26 years.

214

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

RESULTS AND OBSERVATIONS

The vast majority of uploaded pictures (90%) were also automatically transferred to Flickr. As Flickr employs a categorized privacy mechanism similar to IYOUIT, the access labels „public‟, „private‟, and „buddy only‟ are directly mapped (the buddy groups differ, though).

In the following, we distill and abstract from the study findings reported in the last sections to provide a set of outcomes that have been proven successful in the design of IYOUIT and can therefore also be relevant for those active in the area of context-awareness and social networks.

For Facebook, e.g., not having this mechanism, only pictures uploaded as public are transferred. In the questionnaire, 55% were very positive towards the automatic addition of status updates in their Facebook accounts and several of them requested an even tighter connection with the IYOUIT application.

General Usage

Combining data and meta-data from various sources and with various services is an important feature, live and also after data collection (e.g. post-tagging pictures on Flickr). Most important types of context as expressed by the users are location, pictures, weather and presence information.

Trends throughout the Study

We also tried to spot trends and changes in attitudes during the course of the study. However, besides a slight general decrease in use, there were no significant changes to be observed. We cannot clearly say whether this is due to the duration of the study or the mere absence of changes.

Power consumption is an issue although, in our study, this had only minor impact. Still, users see which features impact battery life and should be able to turn those off. Proactive versus Manual Actions

The concept of places has been widely understood and used. Users highly valued automatic place suggestions. In general, automatic abstraction mechanisms from raw (sensor) data proved to be an effective tool for leveraging personalized information exchange.

As one example of analysis of the quantitative data we collected, we let people judge whether IYOUIT increased their communication between buddies before and after the study. The plus of 0.4 points on the Likert-scale (to 2.1, σ = 1.3) was not significant (F = 0.71, p > .2).

Automatic context tagging can relieve users from manual input. However, what the system can derive automatically must be shown, with the possibility to manually adjust it.

Some behavior also seems to be an individual preference, independent from the amount of use. For instance, there was no correlation between the number of photo uploads and photo tagging per user (r2 = 0.07).

Automatic activity recognition needs more research before the process can (if ever) be fully automated. However, recommendations from which a user can choose can be a meaningful shortcut to express assumed presence states.

Some interesting results appeared with regard to thoughts about privacy. Our questionnaires before and after the study revealed that, on average, the use of IYOUIT made people more confident to share data with close family members and friends (see Figure 14). However, people also became aware that the information being disclosed can be rather delicate and detailed and the readiness to share dropped slightly for more distant relationships.

Privacy and Sharing

Explicit sharing is more subject to privacy concerns than automatic sharing: people tend to be reluctant to share manually entered information (such as mood) in detail. Automatically gathered, yet personal data (e.g. spatial information), however, has been regarded as less sensitive.

On the other side, two people acknowledged that they learned to restrict the amount of information made available and thus also became more comfortable to provide some data even to lose contacts.

3rd party services offer high potential, especially a connection to social platforms and automated information updates provide a strong added value for users. However, the user must remain in control of what is transferred and generated content should be explicitly marked or separated. A dedicated privacy mechanism that is accessible, understandable, and modifiable by the users is essential. Although people like to be in control and use an item-by-item strategy for some data such as images, privacy settings per type of buddy relation are accepted. Thus, we suggest changing the opinion of Consolvo et al. [6] and Lederer et al. [14] that "who is inquiring about one's location" is a strong factor to "the relation to the one who is inquiring". This implies a greater number of relationship types than available in most current systems is beneficial.

Figure 14: Willingness to share context with specific groups. People generally shared more information with family members or friends and became more careful in sharing with more distant persons.

215

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

SUMMARY

11. Györbíró, N., Fábián, Á. and Hományi, G. An activity recognition system for mobile phones. Mobile Networks and Applications 14(1). 82-91. 2009

We have evaluated IYOUIT, a freely available platform and mobile application for social, context-aware computing. A trial has been performed with 19 users over four weeks. Results have been presented in the categories of general acceptance and use, context sharing, context abstraction, as well as privacy and social networking. We collected and presented results in four categories about use, context sharing, context abstraction, and privacy.

12. Hightower, J., Consolvo, S., LaMarca, A., Smith, I. and Hughes, J. Learning and recognizing the places we go. In Proc. UbiComp’05. 159-176. 2005 13. Hong, J-Y., Suh, E-H. and Kim, S-J. Context-aware systems: A literature review and classification. Expert Systems with Applications 36(4). 8509-8522. 2009

We feel that this study has greatly enhanced our own understanding of context-aware social systems. In the future, we plan to further investigate and study the potential and acceptance of enhanced profiling algorithms to derive patterns about users, which promise more targeted personal services. As a starting point from a commercial point of view, 76% of the participants would accept targeted advertisements within the application. Complex derivations of suggestions and profile data will then, however, need further research to enhance their understanding and acceptance on the user side (cf. Dey and Newberger [7]).

14. Lederer, S., Mankoff, J. and Dey, A. K. Who wants to know what when? Privacy preference determinants in ubiquitous computing. In CHI’03 Extended Abstracts. 724-725. 2003 15. Morla, R. and Davies, N. Evaluating a location-based application: A hybrid test and simulation environment. IEEE Pervasive Computing 3(3). 48-56. 2004 16. Nurmi, P. and Koolwaaij, J. Identifying meaningful locations. In Proc. IEEE MobiQuitous'06. 2006 17. O'Grady, M., O'Hare, G. M. P. and Sas, C. Mobile agents for mobile tourists: A user evaluation of Gulliver's Genie. Interacting with Computers 17(4). 342-366. 2005

REFERENCES

1. Ashbrook, D. and Starner, T. Learning significant locations and predicting user movement with GPS. In Proc. ISWC’02. 101-107. 2002

18. P&AB. Consumer privacy attitudes: A major shift since 2000 and why. Privacy & American Business Newsletter 10(6). 2003

2. Bardram, J. E. and Hansen, T. R. The AWARE architecture: supporting context-mediated social awareness in mobile cooperation. In Proc. CSCW’04. 192-201. 2004

19. Palen, L. and Dourish, P. Unpacking “privacy” for a networked world. In Proc. CHI’03. 129-136. 2003

3. Böhm, S., Koolwaaij, J., and Luther, M. Share whatever you like. In Proc. Workshop CAMPUS’08. 2008

20. Patil, S. and Lai, J. Who gets to know what when: Configuring privacy permissions in an awareness application. In Proc. CHI’05. 101-110. 2005

4. Böhm, S., Koolwaaij, J., Luther, M., Souville, B., Wagner, M. and Wibbels, M. Introducing IYOUIT. In Proc. ISWC’08, 804-817. 2008

21. Rogers, E. M. Communication technology: The new media in society. 1 st edition. The Free Press. 1986

5. Chen, G. and Kotz, D. A survey of context-aware mobile computing research. Technical Report UMI TR2000-381. Dartmouth College. 2000

22. Sadeh, N., Hong, J., Cranor, L., Fette, I., Kelley, P., Prabaker, M. and Rao, J. Understanding and capturing people's privacy policies in a mobile social networking application. Personal and Ubiquitous Computing 13(6). 401-412. 2009

6. Consolvo, S., Smith, I. E., Matthews, T., LaMarca, A., Tabert, J. and Powledge, P. Location disclosure to social relations: why, when, & what people want to share. In Proc. CHI’05. 81-90. 2005

23. Tungare, M., Burbey, I. and Pérez-Quiñones, M. A. Evaluation of a location-linked notes system. In Proc. ACM-SE 44. 494-499. 2006

7. Dey, A. K. and Newberger, A. Support for contextaware intelligibility and control. In Proc. CHI’09. 859868. 2009 8. Fagerberg, P., Espinoza, F. and Persson, P. What is a place? Allowing users to name and define places. In CHI’03 Extended Abstracts. 828-829. 2003

24. Zhou, C., Frankowski, D., Ludford, P., Shekhar, S. and Terveen, L. Discovering personally meaningful places: An interactive clustering approach. ACM Trans. Inf. Syst. 25 (3). 12-42. 2007

9. Froehlich, J., Chen, M. Y., Consolvo, S., Harrison, B., and Landay, J. A. MyExperience: A system for in situ tracing and capturing of user feedback on mobile phones. In Proc. MobiSys’07. 57-70. 2007

25. Raento, M., Oulasvirta, A., Petit, R., & Toivonen, H. ContextPhone: A prototyping platform for contextaware mobile applications. IEEE Pervasive Computing, 4(2). 2005

10. Grandhi, S. A., Jones, Q. and Karam, S. Sharing the big apple: A survey study of people, place and locatability. In CHI’05 Extended Abstracts. 1407-1410. 2005

26. Jacucci, G., Oulasvirta, A., Ilmonen, T., Evans, J., & Salovaara, A. Comedia: Mobile group media for active spectatorship. In CHI‟07. 1273-1282. 2007

216

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Negotiating Privacy Boundaries in Social Applications for Accessibility Mapping Harald Holone Østfold University College, Norway [email protected]

Jo Herstad University of Oslo, Norway [email protected]

ABSTRACT

Typically, the approach to sharing of such information is largely based on personal communication, and the trust implied through personal connections. Making this process more public allows for greater reach of the information, and the collective gathering of accessibility information that can form the basis for route planning and other applications. On the other hand, exposing ones whereabouts, preferences, needs and opinions in a more public sphere raises issues not only related to trust, but also about aspects of privacy.

Privacy is often used as an abstract concept, and negotiating what information to disclose to whom, where, at what times, and in what situations is a challenging one. In this paper we apply a previously proposed framework from Palen and Dourish for understanding and discussing privacy to a setting of sharing and acting on information about physical accessibility. We do this by describing existing practice for sharing of such information among wheelchair users, and compare that with new practices emerging from the use of ICT, the mobile, collaborative route planning concept OurWay. Through highlighting these changes, we discuss concrete privacy issues, and hope to provide a contribution to users, designers and analysts for creating and using mobile, networked technologies for accessible navigation of urban and built environments.

We draw on research from Computer Supported Collaborative Work (CSCW) and Human Computer Interaction (HCI) to discuss the role of privacy and negotiation in collaborative systems. Further, we take a look at relevant accessibility literature to explore the need for user involvement in assistive technologies. To form a basis for our discussion of privacy, we build on the work by Palen and Dourish on privacy in a networked world [11], which in turn is adapted from Altman’s privacy theory from the 1970’s. The framework suggested by Palen and Dourish is useful both for illustrating the complex notion of privacy, as well as a vehicle for discussing the role of negotiation and privacy in a social application for accessibility mapping. We are using a concrete example firmly founded in an existing practice to discuss the boundaries and negotiation of information disclosure. In this context, the notion of privacy is not about law, it is about the experienced privacy.

ACM Classification Keywords

H.5.2 User Interfaces: User-centered design General Terms

Human Factors Author Keywords

CSCW, Social Navigation, Accessibility, Trust, Privacy, Negotiation INTRODUCTION

The contribution of the paper is twofold: First, we problematize the general use of social technologies for specific applications like accessibility mapping, and second we build on the work by Palen and Dourish to discuss the notion of privacy in social applications. The main research question we address in this paper is What privacy issues emerge when social software technologies are applied for specific situations like accessibility mapping?

The disability research area covers many aspects, including physical accessibility and e-accessibility. In previous work, we have suggested the use of ideas from social applications on the web to create and share information about physical access to built and urban environments. In this paper, we introduce the issue of privacy that arises when these technologies are applied in an attempt to solve the gargantuan task of collecting, disseminating and maintaining such information. We refer to this process Accessibility Mapping.

The paper is organized as follows: First we review relevant literature within the fields of accessibility, social applications, HCI and privacy. Then we re-cap the work with OurWay, a collaborative route planner, including our latest work relating to the re-framing of OurWay as a social application, and it’s potential for accessibility mapping. The discussion focuses on the notion of privacy in general, as well as specifically within the realm of accessibility mapping. Finally we conclude the paper with a summary of the role of existing practice and the

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16 - 20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

217

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

negotiation of privacy in mobile, networked systems for accessibility.

The role of negotiation in CSCW systems is put forward by Schmidt et al. as important for coordination and articulation work [14, 15]. A complementary form of negotiation of contexts created by introduction of new technologies is introduced by Satchell and Dourish on their study of non-use in HCI [13]. This perspective is useful for understanding mechanisms by which new technologies are introduced, but not necessarily accepted.

RELATED WORK

We now briefly review relevant literature in the fields of accessibility, social software and privacy. This literature provides a background for next section’s discussion of privacy in mobile, networked ICT for collaborative, accessible route planning.

Privacy

Accessibility

ICT-related privacy research often focuses on technology and implementation, and related to rules and regulations (see for example [2]). Palen and Dourish have proposed a framework for a more nuanced understanding and discussion of privacy in a networked world [11]. Building on the seminal work of Altman, they identify three boundaries central to the negotiation of information disclosure. The first and basic boundary is the disclosure boundary, that is, what information to reveal or share, and what to keep from others. Secondly, the identity boundary is defined by the role taken on by the user. For instance, if the user is representing herself personally, or if she represents a group such as a company or organization. Finally, within the temporal boundary the effects of persisted information is discussed. That is, the information left behind in a networked system can be interpreted by unintended recipients at a later time, and there is little or no way of controlling the interpretation of information, or the context in which it is interpreted.

Accessibility is a term with many connotations, and covers a wide range of topics, such as the accessibility and usability of ICT, as well as to accessibility in the urban and built environment. In the work with OurWay, the primary focus is on physical accessibility, and the communication of information relevant to moving around in the physical world. Universal design and accessibility in general is currently receiving a lot of attention, as can be seen for instance in programs within the European Union with regards to accessible technology and urban planning. Our current focus is on use of mobile ICT for sharing information about accessibility in the physical world. Also in the physical world, accessibility can have different meanings. For instance, in results from the AUNTSUE project [5], social aspects like fear of crime are mentioned as important factors when users assess the overall accessibility of an area. V¨ olkel et al. [17] describes requirements for accessibility annotation of geographic data, with the aim to collect detailed information for different needs and preferences.

Altman’s insight is that privacy is not a static set of rules, rather it is a dynamic process, a constant negotiation depending on the situation. Grudin puts this in the context of situated action, which is precisely what allows the constant negotiation just mentioned [6]:

Existing literature on accessibility focuses mostly on the interaction between a single user and a computer system, with some notable exceptions, e.g. [18] and [8]. Hedvall [8] argues that the accessibility field is lagging behind the HCI field, focusing mostly on regulations and predictability. His work bridges work in the HCI field and the accessibility field, and by comparing the evolution of the two fields, he suggests that the accessibility field has much to learn from the work in HCI and CSCW. In particular, the focus on the individual is key to Hedvall’s argument, in what he calls experienced accessibility.

Why then the uneasiness, the widespread attention to privacy? It may reflect an awareness at some level of something more fundamental than privacy that is being challenged: The steady erosion of clearly situated action. We are losing control and knowledge of the consequences of our actions, because if what we do is represented digitally, it can appear anywhere and at any time in the future. We no longer control access to anything we disclose. The role of negotiation, according to Schmidt, is a key component for a successful CSCW system. It seems obvious from the privacy work cited above that this is particularly true when it comes to issues of privacy. In fact, we would argue that the experienced accessibility discussed by Hedvall has a parallel in privacy, in what may appropriately be dubbed experienced privacy. This experience is difficult to assess in situ, since the control over interpretation is no longer in the hands of the user.

Social Software

Grudin, although not using the term social software, puts forward four characteristics he considers key in these kinds of technologies [7]: . . . they 1) can be extremely lightweight, 2) make information and activity highly visible, 3) provide individual and group benefits, and 4) are grassroots, self-organizing phenomena. Grudin refers to studies of using project blogs and wikis as well as the use of hash tags for coordinating and maintaining project-related content and activities. We have used Grudin’s characteristics for discussing OurWay as a social application.

218

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

RESEARCH SETTING AND METHOD

Dashboard [16] projects, where exposing the source and history of information is a key point. Perhaps the main drawback of the current OurWay prototype is that it only indirectly provides this information, through the resulting route suggested to the user. In other words, there are no clues provided as to who provided the annotations that led to the suggested route, or what context they were captured in.

We will now provide the background for our case, OurWay. First, the concept and early work is presented, before we summarize a discussion of OurWay as a social application. Finally, we present findings from interviews with wheelchair users to illustrate existing practices for sharing of accessibility information. The OurWay concept

Individual and group benefits

OurWay is a collaborative route planner, where users are providing feedback on accessibility through their interaction with the system. Users are equipped with mobile phones running a client application, which connects to a central route planning and feedback server over a mobile Internet connection. The client displays a map of the area, and allows the user to ask for a route between two locations, which is then displayed on the map. The user can at any time provide feedback to the central server about the part of the route being traversed. To keep the threshold for contribution low, we have used only three levels of feedback: good, uncomfortable, and inaccessible. The submitted ratings are attached to route segments, and are used as weights by the route planning server when calculating new routes.

Some individual benefits of a social software system for accessibility mapping are obvious, such as the opportunity to provide feedback on suggested routes and get immediate reward in the form of an alternative route from the system. The benefits for the group(s) are, as we have shown through the work with OurWay, more accessible routes over time. Another benefit for the group as a whole, is that this way of collecting and maintaining accessibility information is a potential resolution to the resource challenge mentioned in the introduction of this paper. Further, the social awareness of mobility challenges and special needs can be raised, also outside of the group of core users. Self-organizing grassroots phenomena

Social applications for accessibility mapping

There is obvious value in the networks, agendas and political power represented by established interest organizations. At the same time, parts of what make social applications work is the ability for users to rapidly form groups and processes, often to influence the organizations or establishments they are taking part in. There should be a mutual interest from institutions, individual users and user groups to make use of social software tools, however the suitable balance of initiative and power remains to be established. See for instance Borchorst et al. [3], where they look at the application of Web 2.0 technology in the interaction between citizens and municipalities in Denmark.

The OurWay project started as a technological prototype, without extensive work on designing it for collaboration. Throughout the project the focus has shifted towards the social aspects of use, and by using Grudin’s four characteristics we have re-framed the concept as a social application. A summary of this discussion is presented here, for more details see [9]. Lightweight

From the end user’s perspective, OurWay can be considered extremely lightweight. The feedback mechanism is a vast simplification of the typically form based inquiry taking place in traditional accessibility mapping. We do not propose that the OurWay concept should replace such detailed approaches, rather that the information generated by users can augment existing information. One obvious challenge is the use of mobile phones for people with special needs. Although mobile phones are ubiquitous among all user groups, the necessary HCI challenges related to use in the field must be considered carefully. One key argument for using an open, lightweight infrastructure for a social accessibility mapping application is that the opportunities for adaption to different needs and requirements are moved away from the core of the system and towards the end user.

Existing practice

We performed in-depth semi-structured interviews with three wheelchair users early in the OurWay project. The goal was to assess the interest in such technology, and the privacy issues related to use. Two of them have been active participants in organizations for the disabled over many years, and knew about the OurWay idea. The interviews were fully transcribed, and analyzed using an open coding process. This material is now used to describe existing practices for navigating the urban landscape with wheelchairs. Then we discuss the changes in practice introduced with new technology to provide a background for the discussion of negotiating privacy.

Information and activity made visible

For anyone, planning is an important part of a trip to an unknown location. The most common practice for planning a trip in a new or unknown urban landscape, according to our subjects, is direct communication with other trusted people who understand the needs of the person who asks:

A tool which provides route suggestions must adhere to high standards when it comes to dependable information and social transparency. There is little doubt that allowing the user to see the ratings and context which is used to calculate the routes is important. Some suggestions can be found in the WikiTrust [1] and Wiki-

219

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

“When you are in a situation like ours, as wheelchair and crouch users, we become very creative in finding solutions, and we always have to think ahead. We can’t just go.”

mous] says that the criteria for universal design has been followed, I know that it works for me. But I also know that if Mrs. Olsen comes along, and she has problems with her hip, I’m able to say that you shouldn’t trust that these toilets will suit you. Because they suit me.”

This communication often implies an intimate knowledge of each other, including previous common experiences and knowledge of each others respective needs: “If I know that he has been there, right, and I know that he and I share the same needs, if he says this worked for him, then I can go there, too.”

Some urban features, like shopping malls, are considered accessible “by default” by many users. Both because of easily accessible and designated parking spaces, and because of thoughtful design of the built environment:

“I would use a mobile phone, right. And talk to them and then ask [. . . ] are you sure it will work with my chair? Of course, if [friend] was the one sharing the information, I would have done it anyway, right. Because I know him so well.”

“When it comes to shopping malls, they have trolleys, shopping carts, baby strollers, and they build for them. Everyone must have access, because they need to shop. And if baby strollers and trolleys can get in, a wheelchair user can also get in.”

This is emphasized by the fact that many wheelchair users have different assistive technologies for different occasions, be it motorized chairs (light and heavy), manual chairs and in some cases crouches. Says one user of an electric chair:

“. . . and the fire regulations. These regulations demand quite a bit with regards to escape routes, the size of those, that there shouldn’t be stacks of goods placed everywhere, and so forth. This makes it secure enough for you to use it.”

“I sometimes use a manual chair, but then I need someone along to push me. [. . . ] I can make two trips to the center and back between charging the battery. Then I’m not using this chair, I’m using the large chair.”

Thus, we get the impression that shopping malls are preferred over shops in the streets because of the general accessibility: “You know that if you’re going shopping you go to the shopping malls. You don’t go looking for the smaller local shops, right?”

One frequently recurring theme from the interviews is that of individual needs and the issue of trusting information coming from others. This anecdote aptly illustrates what can happen if the shared understanding of needs is absent, or different reference points exist:

Several initiatives to map out the accessibility of urban areas has been undertaken, mostly (so far) based on paper maps and by the help of volunteers. Further, other less formal activities have been carried out, like this attempt to signal the accessibility of individual shops:

“I called a hotel to book rooms for the National Association of Disabled, and they told me that their hotel was accessible to disabled, and they knew that because a Paralympics team had staid there. [. . . ] There were no obstacles, right? These people even used the stairs to get to the dining hall. [. . . ] The receptionists experience was that she had seen many people in wheelchairs, and it all worked well in her hotel.”

“We had this campaign once where we sold accessibility stickers. We made the criteria, but we were obviously not sufficiently specific about the criteria, and then we sold these stickers [to the shops]. So when you came to a shop and saw the sticker, you knew that everything was OK. That was the intention. However, after a while, when different people did this [sold the stickers], we blurred the criteria, and we knew each other, and it became more important to sell stickers than to [indicate accessibility].”

Even interpreting standardized accessibility, for instance by signposting toilets as accessible, requires the similar knowledge. In this situation, intimate knowledge of the formal requirements are needed:

In summary, existing practices for sharing accessibility information is often based on individual needs and personal relations. The information is local, and is interpreted based on the individual and the situation. As the examples presented indicate, even formalized regulations and knowledgeable people can create uncertainty regarding the experienced accessibility. Having presented existing practice as a background, we now move on to the main discussion.

“The toilets, for instance, are a problem for many. Some want them high, others want them low. I want them low, as I’m 145 tall. There are no regulations on height for the toilet seats. A whole group in the Association of Disabled of course want to change the regulations so that all accessible toilet seats should be high, right. [. . . ] I know the regulations so well. It’s impossible to make a toilet seat both low and high, right. [. . . ] So if [anony-

220

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

DISCUSSION

conditions.

Introducing new assistive technologies will change the practice of getting around. Introducing aids for sharing of accessibility information will change the way people share this information. It will most likely not replace the existing practice, and it is key to our discussion that the technology is augmenting existing practice, not substituting it for something completely different. In the first part of the discussion we identify privacy issues that can result from use of a collaborative navigator, before we focus on negotiation of privacy in this context using Palen and Dourish’ framework in the second part.

Further, simply introducing a device like a mobile phone into the process of negotiating accessibility on the spot has practical implications, such as simply managing the extra device in an already demanding situation. Further, use of such a solution may be regarded as a special case of using a public terminal [4], which in itself has impact on the use situation. This is different from using navigation equipment in a car, where the car provides a private sphere in which interaction with the technology can be negotiated.

Privacy issues

Reach and persistent information

Staying focused on the individual, we now attempt to highlight some of the changes that will arise with the introduction of new assistive technology. We do this to answer our main research question.

Existing practices for sharing of accessibility information are largely local, personal and ephemeral. A mobile collaborative system can extend the reach of such information in more than one way. First, the potential number of people who can access and provide information to the system is increased. This means that information left behind can be used by people who would not normally have easy access to the information (they are not part of a local community, or lack personal relations in the area in question). It also means that users expose their activities to strangers, which raises privacy issues. Further, the exposure of activity can provide new and perhaps awkward situations, illustrated here by a quote from one of the interviews. The topic was whether annotating a goods lift as accessible could be justified if the benefits of getting to another floor were significant:

Change in practice

As we have shown, identifying accessible routes and locations for the individual is largely accomplished by intimate knowledge of respective accessibility needs and relations between people, and understanding of the criteria behind standardized accessibility. With the introduction of ICT for assistance, the communication forms changes, and the old practices cannot be directly applied. For instance, relying on persisted accessibility information provided by strangers will require support for sense-making, both through the technology itself, and by traditional means. However, to allow for this negotiation of meaning, the activity of other users in the system must be made visible, including information about the user(s) who provided the information.

“This is very dangerous. Because, as you say, this system could also be used by the municipality. If it [the use] in any way reflects that the group that I represent [when annotating the lift] accepts goods lifts as equally good alternatives to ordinary lifts, they might conclude that they won’t bother about [installing] an ordinary [accessible] lift.”

Providing details about other users in the system is a balancing act between full disclosure and privacy. Will full identities, activities and assistive technologies in use be required for this negotiation? Perhaps building trust networks and social relations into the application can be one way forward. In this way, some form of anonymity can be maintained, whilst still providing cues about the trustworthiness or relevance of information.

This statement highlights several privacy aspects, such as the representation of self and other, and the persistence of information over time and the interpretation of the persisted information. We will return to these issues in the next section.

In the existing practice, this information has an ephemeral and personal nature. Individual needs are communicated on a trusted need-to-know basis, and often it is not even an explicit part of the negotiation, as the knowledge is tacit and does not need to be addressed specifically. Another potential change in practice comes with the new opportunity for more spontaneous trips, since the information is readily available on the mobile device. There might be a shift from detailed pre-planning of routes to more ad-hoc route planning. The mixed reality of an annotated map with route planning capabilities provides the ability to adjust the route along the way, taking into consideration local and temporal changes in accessibility, such as construction work or weather

One of the premises for a collaborative navigator like OurWay is that users leave behind traces of their activity, in other words, that parts of their interaction with the system is recorded to facilitate calculation of accessible routes with updated information. Persistence of information is perhaps the most important source of privacy issues to be discussed in the next section. Persisting traces of user activity extends the reach, not only in terms of audience, but also over time and use contexts. Based on these observations about changes in practice and the introduction of technology to extend reach and persisting accessibility information, we switch gears and discuss these privacy issues with the help of the frame-

221

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

work proposed by Palen and Dourish.

the need for negotiation of what information goes out to whom, and what activity is triggered by interaction with the system. Some activity might be extra valuable to make public, for instance to make visible ones own contributions to the system and community of users in order to gain credibility or good reputation. Making activity visible can also be a means for limiting vandalism or improper use. However, as we shall argue later, there are instances where visibility of activity needs to be kept private.

Negotiating privacy

Our focus in this part of the discussion is on the need for negotiation of privacy in different situations, emerging from the introduction of ICT for sharing of accessibility information. Disclosure: Privacy and Publicity

The way we interpret the framework, the fundamental boundary is that of privacy and publicity, in other words what information users in a situation regard as private, and what information it is desirable or necessary to share with others. First, we need to identify the core types of information at stake. Obviously, positioning information is fundamental in a navigation system. This might be geographical coordinates, or approximate information such as venues used in the social application FourSquare. Time can also be crucial, for instance for taking into account the temporality of certain obstacles (like weather conditions), or time of day (related for instance to opening hours etc). The level of accessibility experienced by the user must also be recorded. In previous research, we have observed that it is largely inaccessibility which is recorded, and mostly when users have to provide feedback in order to get an alternative route from the route planner [10]. This might change, though, if such a collaborative navigator is used in a campaign setting.

In summary, the negotiation of what information to disclose must be left to the user, and whether or not activity should result in public traces must be negotiable by the user at any time. Identity: Self and other

The first question a potential user must answer is whether to join the service or not. Trivial as this might seem, it is a fundamental question, and one that is well illustrated by the social pressure to join and participate in successful on-line communities like Facebook or Twitter. Once a participant, it must be possible to quit the system, and potentially delete traces of ones own activity. Quitting or changing ones association with the service will be a public activity, which again touches on the social obligation to be a participant in the community. As an example of the publicity of private actions we present an anecdote about Facebook from one of our colleges: Gisle joined Facebook when it first opened for the international audience. He added many friends, including his wife who he identified as such. Realizing that this resulted in public information on his profile revealing his private family relations, Gisle decided to change the relationship from “spouse” to ordinary friend. The unintended effect of this change was that all his friends got a system generated status message from Gisle that said “Gisle is no longer listed as in a relationship”, which in turn resulted in a number of concerned friends contacting him to investigate what they interpreted as a terminated relationship.

Who you are, who you represent and the role you have in a situation are key to the negotiation of disclosure. For instance, some obstacles might be considered to apply in a specific context, and not be valuable to others. A user who considers an obstacle to be “private” must still be allowed to use the system, for instance for getting an alternative route, without this activity being persisted for later retrieval. Even when deciding to share the information with the community, the role has to be negotiated. Does the user represent “all” wheelchair users, or only users of motorized chairs or manual chairs? Does the user represent an interest organization with an agenda, or is he a casual user only using the system as a navigation tool? These are all questions that in the existing practice is negotiated from situation to situation, and similar ability for negotiation must be in place in the new practice. The role taken on by the individual is negotiated in the situation, and evolves over time as well. For instance, the amount of effort put into the “cause” can change: “[These days,] I don’t sit down and write to the management and tell them how it should be. I don’t, because I get so angry, I go home because I’m so upset. [. . . ] So I don’t go to that church, because I don’t like it. I think the ramp is ugly. Now, in my younger years [. . . ] I wrote an angry letter to the council down there, right. But I left it at that. I was very angry.” The experienced accessibility is personal, however it can trigger actions taken on behalf of a larger group: “[. . . ] I don’t think much about the fact that other people are going to that church later. In a way, I have told them what I think. [. . . ] Well, I react on behalf of all [the other users]” And if the context is explicitly coordinated by a group, the role taken on can be very different:

In this story, it was not Gisle who decided to make this a public announcement, it was the system. It illustrates

“Then, sometimes, the Association of Disabled has

222

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

decided to run a campaign against it [church accessibility]. And then I’ve gathered the church council. But I’m not taking on that fight regularly.”

it’s users. Again, Facebook is a prime example, where current analysis shows recent dramatic changes in privacy policy, where Facebook Inc. has made changes to their policy for providing access to user information for third parties1 .

Negotiation of identity is an involved and dynamic process. Capturing this process in an ICT system is challenging, and requires careful attention in the system design process. We suggest that the negotiation will take place mostly outside of the system itself, which means that the system must be open to the result of the negotiation, for instance by allowing anonymous or non-traceable activity, and the ability to revoke information at a later stage.

This also raises an important question about who owns the service and associated contributed data, and who can change the terms under which they are shared with others. By agreeing to conditions of use whereby the service provider can modify the rules without consent from the users, the user is effectively waiving the opportunity for negotiation of these issues. It is our recommendation that these issues must be publicly addressed and communicated to users prior to joining such a social service, and that any user should be allowed to easily opt out of the service and potentially remove traces of their activity and get their own data with them if they choose to do so.

To summarize, place, time and situation are key components in the process of negotiating ones identity - the answer to the question Who do I represent here, now?. An extra layer of concern is added when the reach and persistence changes are taken into consideration. The question then changes to Who do I represent here, now and later?. The time boundary is discussed next.

Another aspect of the time boundary is that of interpreting current activity in the light of previous activity. Again, an example from one of our subjects:

Temporality: Past, present and future

“. . . we’re very good at complaining when things are not OK, but we’re not very good at telling people that things are OK. [. . . ] I never go to the newspaper and tell them the new ramp is in place [. . . ] I like making a bit of fuzz, but I’m not very good at giving people a pat on the back.”

Arguably the most influential change to existing practice by introduction of a mobile, collaborative navigation system is that of persistence of information. Palen and Dourish address persistence by using the temporality boundary. In existing practice, it is largely taken for granted that communication about accessibility is of local and ephemeral nature. Sharing accessibility information through technology implies persistence of information, and thus the impact of sharing traces of activities becomes difficult to assess, as is the negotiation of what to share with others.

By constantly and publicly being complaining about accessibility, there’s a risk of later being regarded as a negative person. This is true in existing practice as far as actively public statements (like in the newspaper) is concerned. By the introduction of ICT, the aggregated statements made by single users can portray a negative image of a user, even though the user is only leaving traces by making use of technology as a tool to solve local, personal navigational problems. The converse is also true, of course, as in the example with the goods lift, where a positive attitude towards a undesirable solution can be interpreted as legitimizing sub-standard solutions - thus implicitly making the activity trace represent the user group collectively, and not merely as coming from an individual user.

The example about accessible goods lifts is an example of this, and the user who presented this continues: “. . . basically because we see that there are, like, economic interests at stake, and they would effectively create a barrier for many of us, in contradiction of rules and regulations. An we do definitively not want to be part of legitimizing that. You will find that attitude, guaranteed.” This user has been active in organized accessibility work for many years, and clearly comes across as reflected about the possible interpretations of information, made by others at a later time. There are numerous examples of lack of this kind of reflection, perhaps most notable in current use of on-line social applications like Facebook. For many, the idea that someone “outside the group” will have access to relatively private information seems elusive, and one likely cause is the apparent ephemeral nature of on-line conversations, and in particular the lack of technological understanding by users [12].

One could argue that aggregating and anonymizing information could limit the risk of standing out from the crowd as a person who provides critical feedback, however the use of aggregation also introduces new problems. These includes a higher risk of the information being interpreted as representing a consensus view withing a group, and it also makes it more difficult to assess the reliability of the information, which today is largely based on personal connections.

Further, changes in privacy policies for such on-line communities often introduces breaches of contract with

1

223

http://w2.eff.org/deeplinks/2010/04/facebook-timeline/

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

CONCLUSION

3. Borchorst, N.G. and Bødker, S. and Zander, P-O. The boundaries of participatory citizenship. In ECSCW 2009: Proceedings of the 11th European Conference on Computer Supported Cooperative Work, 7-11 September 2009, Vienna, Austria, page 1. Springer, 2009.

The first question that has been addressed in this paper is specifically about what privacy issues that emerge when social software technologies are applied for situations like accessibility mapping. Introducing new technologies is not only about what the specific technology “does”, but also about what existing practices are changed with the new technology in use. We have seen that there is a balancing act between the disclosure of information and managing privacy when using social systems for route planning. What was local and only disclosed to trusted others in an existing route navigation practice, can potentially be disclosed to all other users of the system now and in the future, and thereby changing the reach of the information.

4. I. Bratteberg and S. Kristoffersen. Irreversibility and forceback in public interfaces. In Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges, pages 63–72. ACM, 2008. 5. G. Evans. Accessibility, Urban Design and the Whole Journey Environment. Built Environment, 35(3):366–385, 2009. 6. J. Grudin. Desituating action: Digital representation of context. Human-Computer Interaction, 16(2):269–286, 2001.

Through this study, it is seen that the everyday practice of route navigation is fundamentally changed when introducing social applications. This view is of importance, since it is often lost when discussing the potential benefit of new, networked technologies. The second research question is about what forms of negotiation emerge in accessibility mapping. The framework presented by Palen and Dourish was applied in order to discuss the negotiation of boundaries related to disclosure, identity and temporality.

7. J. Grudin. Enterprise knowledge management and emerging technologies. In Hawaii International Conference On System Sciences, volume 39, page 57. IEEE Computer Society, 2006. 8. P.O. Hedvall. Towards the Era of Mixed Reality: Accessibility Meets Three Waves of HCI. In HCI and Usability for E-Inclusion: 5th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society, USAB 2009, Linz, Austria, November 9-10, 2009, Proceedings, pages 264–278. Springer-Verlag New York Inc, 2009.

The everyday practice of negotiating information disclosure is changed when introducing social software. The negotiation will both take place within the system, but also by using technologies and communication channels outside the system. Thus, the system must be open to the result of such negotiation.

9. H. Holone and J. Herstad. Social software for accessibility mapping: challenges and opportunities. In Unitech 2010, Oslo, Norway. Tapir Forlag, 2010.

Privacy issues are both about enforcement of laws and the practice of responding to specific circumstances, like when planning and making a journey. The impact that mobile, networked technologies have on the practice of handling privacy issues is important. Our understanding of privacy issues, and the way this changes with introducing new technology is of crucial importance for users, designers and analysts. By addressing specific use situations of social software, we hope to contribute to this important topic.

10. H. Holone, G. Misund, H. Tolsby, and S. Kristoffersen. Aspects of personal navigation with collaborative user feedback. In Proceedings of the 5th Nordic conference on Human-computer interaction: building bridges, pages 182–191. ACM New York, NY, USA, 2008. 11. L. Palen and P. Dourish. Unpacking” privacy” for a networked world. Proceedings of the conference on Human factors in computing systems, pages 129–136, 2003.

ACKNOWLEDGMENTS

We want to extend thanks to the ICCHP 2010 and Unitech 2010 anonymous reviewers, and to Sigurd Dahlseide and Torleif Gravelsæter for performing one of the interviews.

12. S. Patil and A. Kobsa. Uncovering privacy attitudes and practices in instant messaging. In Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work, page 112. ACM, 2005.

REFERENCES

1. B.T. Adler, J. Benterou, K. Chatterjee, L. De Alfaro, I. Pye, and V. Raman. Assigning trust to wikipedia content. In WikiSym 4 rd Intl Symposium on Wikis, 2008.

13. C. Satchell and P. Dourish. Beyond the user: use and non-use in HCI. In Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7, pages 9–16. ACM, 2009.

2. V. Bellotti and A. Sellen. Design for Privacy in Ubiquitous Computing Environments. Proc. of the European Conference on Computer-Supported Cooperative Work, 273, 1993.

224

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

14. K. Schmidt and L. Bannon. Taking CSCW seriously. Computer Supported Cooperative Work (CSCW), 1(1):7–40, 1992.

17. T. V¨olkel, R. K¨ uhn, and G. Weber. Mobility impaired pedestrians are not cars: Requirements for the annotation of geographical data. In Proceedings of the International Conference on Computers Helping People with Special Needs (ICCHP’08), pages 1085–1092. Springer, 2008.

15. K. Schmidt and C. Simonee. Coordination mechanisms: Towards a conceptual foundation of CSCW systems design. Computer Supported Cooperative Work (CSCW), 5(2):155–200, 1996.

18. F. Winberg. Supporting Cross-Modal Collaboration: Adding a Social Dimension to Accessibility. In Proceedings of Haptic and audio interaction design: first international workshop, HAID 2006, Glasgow, UK, August 31-September 1, 2006, pages 102–110. Springer-Verlag New York Inc, 2006.

16. B. Suh, E.H. Chi, A. Kittur, and B.A. Pendleton. Lifting the veil: improving accountability and social transparency in Wikipedia with wikidashboard. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 1037–1040. ACM, 2008.

225

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Transferring Qualities from Horseback Riding to Design Kristina Höök Mobile Life @ Stockholm University Forum 100, 164 40 Kista, Sweden [email protected] focus. The renewed HCI interest lies in designing for experiences – beyond efficiency and task completion [19]. In some respects the ergonomists were more sensitive of the body than we are.

ABSTRACT

We see more and more attempts to design for bodily experiences with digital technology, but it is a notably challenging design task. What are the possible bodily experiences we may aim to design for, and how can we characterise them? By analysing a horseback riding experience, we came to identify the following themes: (1) how certain kinds of bodily experiences are best understood through experiencing them yourself – the bodily ways of knowing, (2) how rhythm and balance create for particularly strong physical experiences of this kind, (3) how movement and emotion coincide in these experiences, (4) how the movement between seeing our own bodies as objects vs experiencing in and through our bodies is one of the ways we come to learn the language of expressing and understanding bodily action, and (5) how this in turn lets us describe the sensitive and delicate relationship of wordless signs and signals that represent, in the case described, two bodily agents – a human and a horse. When the human-horse relationship is really successful, it can be described as rare moments of becoming a centaur. We translate these themes into design considerations for bodily interactions.

As digital products have become an integral part of the fabric of everyday life, the pleasure they give, the contribution to our social identity, or their general aesthetics are core features of their design. We see more and more attempts to design for bodily experiences with digital technology, but it is a notably challenging design task. With the advent of new technologies, such as biosensors worn on your body, interactive clothes, or wearable computers such as mobiles equipped with accelerometers, a whole space of possibilities for body-based interaction is opened. But this begs for answers to three questions. First, what kinds of experiences are we aiming to design for? Glossing them over as all being about designing for flow [3] or inducing a game play experience is too vague [8]. We need to drill deeper and better understand exactly what experiences we are talking about. Are we designing for pleasurable or unpleasurable ones? Are we designing for those that are subjective and unique, or ones that are common and shared? Ones that deliver serendipitous experiences or ones that are evocative and emotional? These are not all the same, even if all may emphasise aspects of bodily experience.

Keywords

Embodiment, experiential qualities, bodily interaction, autoethnography ACM Classification Keywords

H.5.2 Information interfaces and presentation (e.g., HCI): Miscellaneous.

A particularly difficult issue lies in understanding how these experiences may unfold over time – both in the particular interaction with and manipulation of the artefact but also as parts of our everyday on-going lives. As Löwgren [17] puts it, a gestalt for interactive artefacts is defined as a dynamic gestalt which “we have to experience as a dynamic process” (p. 35 - 36).

INTRODUCTION

Originally, HCI spun off from ergonomics through its focus on the cognitive aspects of interaction. With the turn to third-wave HCI with its focus on experiences [2], we are now in a sense turning back to the body in interaction. But contrary to the main focus of ergonomics, instead of documenting and altering routines, finding error-free ways of involving operators in the complex beast of man and machine as a unity, or measuring how the body has its limitations, this turn back to bodily interaction entails a new Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

226

Second, once we know what kind of experience we are aiming to design for, we need to describe them in a form that makes sense and that we can share within a design team. Ways of knowing can arise from your bodily acts without any language translation in-between. The feel of the muscle tensions, the touch of the skin, the tonicities of the body, balance, posture, rhythm of movement, the symbiotic relationship to objects in our environment – these come together into a unique holistic experience. It is not the ability to fulfil a task, but the experience of the corporeality of doing so that matters here. Those descriptions also need to be shared with the users that we invite to test our designs, or even participate in the design process.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Thirdly, if we try to design interaction that builds on bodily movement, seeking certain experiential qualities, many different aspects of the interaction have to be fine-tuned to enable the experience, as, for example: •

•

•

the riding experience after each of the seven lessons. These notes were juxtapositioned with the written transcripts of the videos. I also discussed the material with two colleagues who helped me analyse the data. The accounts I provide below are sometimes quite detailed as I am trying to come to the essence of the riding experience.

the timing of interaction: movement has to render response in exactly the right moment for exactly the right kind of length of time in order to create for a particular experience [27] linking emotion and movement: certain movements and body postures are more likely to coincide with certain emotional experiences [4, 13, 25] harmony of modalities: the modalities of the interaction, such as graphics, haptics or gestures, all have to speak together – harmonize [26]

At the time, I was re-learning how to ride in English style as I had been riding Icelandic style for many years. The Icelandic style of riding is quite different from English: the Icelandic horses are smaller, have more gaits (including tölt and pace), the saddle is placed further back on the Icelandic horse, the stirrups are kept longer, and the horses are trained to interpret the signs and signals the rider emits (from legs, hands, body posture) differently. The way to ride English style is to try to be relaxed and balanced, not disturbing the horse through squeezing your legs too tightly or pushing with your rear muscles. The aim is not to put any tension into the horse – while on Icelandic horses, it is necessary to put tension into the horse in order to enable (unusual) gaits such as tölt. The account given here is therefore the story of an instructional situation. Teaching and learning are at core.

To shed some light on these issues, we decided to study a non-digital activity that is unambiguously about the body’s involvement – a mundane activity to some of us – horseback riding. Also, as it happens, horseback riding is an activity the author has a passion for. There is a plenitude of activities that we do for the pleasure of moving – dancing, sports, jogging, cycling. The pleasures of these activities are of course not only soft, nice movements, since some of the activities involve pain, applying yourself really carefully to make your body do them, adjusting your own body in various ways, even making your body build certain muscles that you normally do not use so much, embarrassment when you do not get it right, and so on. But they are still rewarding. The pleasure of moving a muscle that is aching, or the adrenalin rush when you ride your horse and almost loose control, makes some of us feel that we are living life to its fullest – away from the safe office environment were the excitements (and there are of course plenty of them) will not, to the same extent, move your muscles, bones, joints and stimulate the nervous system of the whole body.

THE FELT EXPERIENCE OF RIDING

There are many ways I could look at the data from this horseback-riding experience. I could look upon it mainly as a learning experience, analyzing each painful step of getting my body to behave in the English-riding style, and figuring out the ways by which my teacher Christian is trying to help me ‘see’ and correct my posture. This is similar to a study by Laurier and Brown, where they talk about Laurier’s father teaching him to see the various phenomena involved in fly fishing [14]. I could describe the ergonomics of the interaction between me and the horse in terms of errors, task completion time or bodily muscle acts. Or I could analyze how the two agents (me and the horse) are communicating. I will attempt to give some accounts of all those situations, but I want to approach the difference between describing various moments in the interaction (from what can be seen e.g. in a video of my riding), to instead describing aspects of the feeling of my experience. Rather than glossing over that experience as flow or other experiential qualities, we would like to describe the unique experience of horseback riding with Christian and his horses, named Blue, Henry and Liberty in terms that hopefully preserve the essential elements of that experience. Here I do not want to suggest that there is a unique subjective experience that I want to describe, so much as preserve what the American philosopher Quine called the inscrutability of reference: the actual thing that is empirically existent, for any and all persons [22]. I would like to describe how we sometimes become part of each others’ lived experience [5], how I experienced my own body, the horse’s body and the rare moments when we were in synch and I became totally absorbed by the situation.

STUDY SET-UP

The study is done by the author, who is not a professional horseback-rider. Instead, we portray a mundane activity of someone doing her favourite leisure activity, an amateur. From here on, I will therefore use the pronoun ‘I’ when I speak about my subjective, personal experiences of riding. The account is an autoethnography, that is a form of autobiographical personal narrative that explores the writer’s experience of life. It has been used in the sociology of new media, journalism and communication studies. Autoethnography (though controversial) has recently started to be used in HCI [15], most notably by Boehner and colleagues who also use it as a design method [1]. The study was done through video taping and taking notes of seven riding lessons. The videotaping was done both by the riding instructor, Christian, and by cameras placed next to the paddock where the riding happened. The instructor also watched one of the videos and made comments to it that I used in the analysis. All of what was said in the videos was transcribed, and snippets of video were cut out and analysed in detail. I wrote down detailed accounts of

The following themes are extracted from the analysis of my experience. They have to do with the various orientations to

227

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

my knees tightly to the horse’s sides and pushing with my sitting bones into the saddle, I could not understand what he meant. I tried making my heel the lowest point of my body rather than my toes, but I was still not getting it. It became a contrived position for my body, my heels were forced down and as soon as I forgot about pushing them down, they crept up again. Christian decided to help me understand where my weight should be through a special exercise. He asked me to stand up in the saddle, see Figure 1, at first while the horse was standing still, then while walking and finally while trotting. Standing in the saddle like this forced me to try and find my own balance without squeezing my legs to the sides of the horse whatsoever. Suddenly I realized what Christian had meant by putting weight into my heels. My whole weight had moved down to my heels and it was from there that I found my balance. I felt my own weight travelling down, gathering around my ankles, and the sole of my feet became like a floor. When I was allowed to sit down back into the saddle, I knew what the feeling should be – which muscles should be activated, what my ankles and feet should feel like and I experienced, inside my body, inside my balance, my own weight.

Figure 1 Standing in the trot to feel the weight in my heels, finally loosening my legs from Blue’s body.

experience that are constituted in the domain of horse riding: •

• • •

•

•

how certain kinds of bodily experiences are best understood through experiencing them yourself – the bodily ways of knowing, how rhythm and balance create for a strong physical, aesthetic experience how movement and emotion may coincide in this particular instructional setting how I move between seeing my own body as an object vs. experiencing in and through my body is one of the ways we come to learn the language of expressing and understanding bodily action how this in turn lets us describe the sensitive and delicate relationship of wordless signs and signals that represent, in the case described, two bodily agents – a human and a horse rare moments of becoming a centaur – that is feeling as ‘one’ with the horse

While I can rationalise what was needed afterwards and decompose exactly which body parts needs to be loose or tensed, it is not this decomposition that I learnt. I learnt a whole body schema, a wholeness that only made sense if I got all parts at the same time. This was a key element in the vocabulary of rider-horse relationships, how the sense of being a centaur is realisable. This insight did not come from being told a visual metaphor (I had already been told, repeatedly, in various ways to put weight into my heals), or from Christian showing me through riding himself what it should look like. In my case, I had to experience it myself in order for it to be meaningful to me. This is not to say that it is impossible to be told how to move your body or simply imitate and thereby constructing meaning. The more experienced you are in listening to bodily instructions, as a rider, dancer or in other bodily activities, the better you will be in following the instructions and thereby experiencing the movement. But the knowing itself does not come from the description of movement – it comes from the experience of it. Or in this case, the concept “putting weight into your heels” did not make sense to me until I had experienced it – the meaning of the words was intrinsically tied to the action. Without the action they conveyed insufficient knowledge to let a person ‘do’ the action in question

Bodily ways of knowing

The interaction with a horse is obviously not word-based. It happens through physical signs and signals: the riders use the muscles in their legs, the placement of their sitting bones, bodily balance, head movement, hand and arm connection to the horse’s mouth and sometimes tone of voice. The horse talks back through its movement, direction, pace, activations of muscles that can be felt throughout the horse’s body, its head movements, tail movements, flipping ears, bend of neck and noises. In order to be a good rider, you need to learn this wordless language. As in any language, understanding and communication arises in interaction over time. When you have experienced a particular bodily schema or concept yourself this understanding may arise. Let us provide one example of such a process.

Out of rhythm and embarrassment

As horse and rider move together, they create a rhythm. Depending on the gait, it can be a two-beat (trot, pace), three-beat (canter), or four-beat (walk, gallop, tölt), in different paces. To allow the horse to keep the beat in a balanced way, the rider needs to make herself invisible in the saddle, not disturbing the rhythm.

During the lessons, Christian kept saying that it was important that I put weight into my heels. This was where my balance was going to come from. As I had been squeezing

228

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

In one of the earlier lessons, as I was repositioning my body from Icelandic style to English, I totally lost everything. In particular, I lost my ability to follow the rhythm of the horse. I was ‘behind the movement’ as Christian phrased it when commenting on the video of me riding: “As you come into better balance the horse is more forwards. But then you lent forward and pulled on the reins so that he slowed down and fell in on the circle. There is still the tendency with the leg to go too far forwards so the seat of the rider is pushed to the back of the saddle and the shoulders come forwards so the rider is turned as what we would call ‘behind the movement’. You are not in time with the horse but you are always trying to catch up with the horse. You can actually see that quite clearly there. He is trotting in one rhythm and the rider is in a slightly different rhythm.”

Figure 2 Feeling very embarrassed by how badly things are going. I make a stupid remark, lean forward as if to release tension, sit down again, and pull my shoulders up, face upwards, giggling awkwardly.

horse compared to the small Icelandic horses. The movement was more of a huge big ship moving up and down, rather than the fast up and down rising on Icelandic horses. I had to make my body listen to and get into this bigger rhythm. As I was trying to adjust my way of sitting, experimenting with where my weight was, loosening the grips of my knees, avoiding to flex my back too much, pushing my heals down, and avoiding to push my sitting bones into the saddle, the rhythm did not come ‘naturally’. I did not have the ability nor prior experience of the feeling that would allow me to attune my body posture to how the horse responded, and make it respond the way I wanted.

The experience of being ‘behind the movement’ is very awkward. The trot is a 2-beat movement in which the horse’s legs move in diagonally opposite pairs separated by a moment of suspension. In a rising trot, the rider needs to rise from the saddle in every second stride. You need to let the movement come from the horse moving you upwards. When it does, you continue the movement forwards, rather than straight up. In the next stride of the horse, you then come back down, and if you are in rhythm, you will come back into the saddle, briefly, at exactly the right beat, immediately going up again. Your whole body needs to listen to the rhythm of the horse, because unless you are in perfect synch with the horse, the horse’s strides will not be perfectly even – your movement disturbs the horse.

Being inside a dynamically rapidly changing situation where you are simply not getting it right, it is very hard to make sense of what to prioritise, what to fix. When Christian stopped me, I was therefore totally devastated by the situation. I was out of synch with the horse and I had no hypotheses about what to do or what was actually wrong. I tried joking about it and made some embarrassed movements with my body, as you can see in Figure 2.

The problem was that I was sitting back into the saddle with a ‘splat’ slightly out of rhythm with the horse. Given how many years I had been riding before going to lessons with Christian, it was horribly embarrassing for me to be out of synch. Following the rhythm of the horse is one of the most important pleasures of riding. As discussed by others [21] rhythmic movement as in dance or riding, moves us in way which are immediately appealing. But just as it can be very awkward to watch someone dancing out of rhythm, it is very awkward to experience it. Being out of rhythm on the horse, is especially awkward as it inflicts on the horse to have the weight of an adult human rider bouncing down into its back at the wrong moment, just as the horse is about to rise.

In a sense, this is similar to Ingold’s description of plank sawing [9]. He nicely shows that no matter how experienced you are, sawing through yet another plank will never be exactly the same. There are number of different sensations that synchronise in a particular riding experience. Rapidly adjusting to each one of them, will in every new instance be a unique experience. Getting into the rhythm – being one with the horse

Towards the end of the third lesson, after having been embarrassed, out of rhythm and very confused, I had, in a way, given up all my good intentions of putting my body in the right place and working hard to make it work. Instead, I allowed myself to be distracted by Christian who was talking about something irrelevant, I relaxed and then I finally got the rhythm of the horse. Suddenly, without consciously trying to, I shifted my weight further back in the saddle and suddenly it worked. I rose at the right moment, I was relaxed in the way I was sitting and the whole thing felt fleeting and nice.

But my embarrassment did not arise only from knowing that Christian could see me and condemn my riding. Nor from my worry about how the horse had to deal with my weight bouncing into his back at the wrong moment, even if this also contributed to it. The embarrassment came from the actual physical experience of being out of rhythm. Our bodies are used to rhythms, our own bipedal swagger [25], our mother’s heart beat, waves beating the beach, music and dancing, and, for those who are fortunate enough to experience it: the horses’ different gaits.

At this point, as I was relaxing, just feeling the rhythm of the trot, I was not really thinking of where my different body parts are placed, nor was I thinking about the rhythm as such. I cannot say that any clearly formulated thoughts flew through my mind. Indeed, one could say that I am not

This particular incident happened when I was still trying to get to terms with how much bigger the stride was with a big

229

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 3 With my body in balance, more or less straight back, weight in my heals, in rhythm with the horse’s trot, rising at the right moment, and overall relaxed.

thinking at all. Thinking is precisely not what is required, I was simply entirely absorbed by the situation. My muscles were doing what they should be doing, my eyes were directed towards where we should be going, my hearing followed the rhythm of the horse’s hooves on the ground, and I felt as one with the horse. We were no longer separated but moved together with one intention, one will. My whole self was simple there, in synch with the movement.

trying to do then their body takes over their reflexes kick in and then people tend to ride in a more relaxed and better balanced way because they are not thinking too much about what they are doing, they are just doing it and that is better. So sometimes you just chat about going to the hairdresser or something.” When Alan Kay talked about how he worked through the ideas of the desktop metaphor and the use of the mouse for drag-and-drop, point-and-click, he picked up on the same insight [10]. He had been inspired by a TV-show of how a tennis teacher was able to distract his pupils by making them sing songs while learning to do backhands, forehands and serve. Through these distractions, the pupils were distracted and their bodies simply imitated the right physical movements without giving it much thought.

In Figure 3, we can see that my body is not leaning forwards, my legs are not too much forward (though still not perfectly in line with shoulder and hip), and I am looking in the direction of where we are heading. I asked Christian to comment on the video of me, and at this point, when I get into rhythm, he said: “Much better rhythm. And again, the rider is better balanced the horse goes forwards more fleeting. He is more even in his rhythm. A lot of this you can see straight, anybody watching this can see the difference. There is much better. You can argue that yes ok you are still a little bit forwards and the rider is still a bit behind the movement. But overall it is much better.”

Seeing my body as an object vs experiencing in and through it

When learning bodily practices such as tennis, golf or skateboarding [29], you occasionally will have to change your posture or aspects of your movements in order to correct some aspect of your performance. These changes can be very painful as they feel unnatural – the prior, faulty, behaviour keeps interfering. Those situations require that you consciously reflect upon different body parts, where they are, what they feel like in that position and how to move them into the new, required position or tension level.

He also confessed that he had used one of his strategies to get me into this state: “Very often, especially with adult riders or nervous riders, if you stop them thinking too intensely about what they are

Figure 4 My leg actively lifted away from the Henry’s side, without putting any weight into the heel, leaving me unbalanced (left). Leaning far too much forwards (middle) annoying Henry. Not putting any weight into my heel on Blue (right).

230

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Christian, talked me through one of these re-positionings of my posture:

tion, see Figure 6, he touched or pointed to parts of my body that needed to relax or be more tensed, he rode the horse himself so that I could watch his body and imitate, and he made me do some ‘extreme’ exercises to provide me with a bodily experience (as described above when standing in the stirrups), or as he phrased it, helping me to create “a muscle memory” of where he wanted certain parts of my body to be placed or what it should feel like.

“Sit in the centre of the saddle, balance squarely on both seat bones, your back should be straight, your legs should relax along the horse’s sides, and the idea is that there is a straight line from your shoulder, through your hip to your heal. Your arms should fall softly by your sides. There should be a straight line running from your shoulder, through your elbow, through your wrist and down the rein to the horse’s mouth. There should be a ‘spring’ between your heal knee and thigh that makes it possible to do a rising trot. You should put your weight in your heels and not lean your body forwards but be straight.”

Through these exercises, Christian occasionally helped me to get into a position where I was balanced, experiencing the kind of interaction between me and the horse that he wished to see. Being in synch with the horse, following its rhythm, and being perfectly balanced myself, happened rarely. But at those moments when I went from actively thinking about where a certain body part was placed, to actually feeling that “this is a good position”, I also typically would switch from seeing that body part as an object from the ‘outside’, to instead being ‘inside’ my body, feeling the interaction from inside my body. In Figure 5, we can see one such moment. Again, my body is not leaning forwards, my legs and knees are relaxed, and my head is turned in the direction of where we are going. The horse also has a nice forward movement. It gave a sense of relaxed concentration – I am totally inside the experience, but in a relaxed way.

You may note how each body part has to be in the right position and how each muscle needs to be activated or relaxed exactly in the right way. In this description, there is an emphasis on being soft, relaxed and balanced. Christian kept emphasising that I should keep my own balance, that I should not disturb the horse. Christian used some visual metaphors to describe what it should feel like, such as “the horse should be flowing like a river through the reins, your arms, and between your legs”. As I had been riding in the Icelandic style for most of my adult life, the re-learning of how to sit proved to be very hard. The problem for me was that as soon as I tried to do something with the horse like steering it or asking it to move somewhere, rather than just thinking hard about how I sat or where my different limbs where, I immediately fell back into squeezing my knees tight to the horses sides, pushing my legs to be straighter and more forwards, and pushing my butt hard down into the saddle. My whole body was screaming that Christian’s way of sitting was not the way I was used sitting and I comfortably moved into the position my body ‘knew’ best.

Learning a movement, a posture, a rhythm, as was required of me here, is to try learn a habit or as phrased by Game [5], “the movement of music, riding, writing, lives in us as we live in it.” Being In the World Together

When things go really well, I feel the presence of the horse’s way of being in the world as part of my own and we act together. This sense of another agent being present relies on a process of recognition. The horse and my self become one, as the centaur mentioned above. But sometimes – indeed in my case quite often as I struggled with learning to ride in a new way with horses unfamiliar to me – my orientation to other, the animal in this case, evokes the idea of the mind and how it is somehow separate from the body. To be able to merge with a horse, one has to get the feeling that the horse’s mind and its body are as one, and the mind is not being mischievous in a way that leads the body to convey misleading signals.

When re-learning how to position my body, my body become an ‘object’ that I controlled with my will. Or rather, what has hitherto become a matter of habit, a way of the body that one takes for granted, now becomes something that I focused on. My mind starts to think again, about how the body should be. During my third lesson riding with Christian, he had decided that he was going to change my way of sitting and thereby influencing the horse. He did so by lounging the horse around him so that I could focus entirely on how I was sitting without having to steer the horse or control it. Halfway into the lecture, I was still sitting all wrong. Since Christian had asked me to relax my legs and last time he had complained about my knees squeezing, I was now letting them hang down totally free from the horse’s sides – even actively lifting them away from the horse’s sides, see Figure 4. To overcome my difficulties Christian had to find a way to show me what the problem was. Describing it through the visual metaphors he used had not helped. In the seven lessons I took with Christian, he used a range of ways of showing me: he showed with his own body what my body was looking like when I was in the wrong posi-

This sense of another mind being present relies on what Jane Heal describes as an empirical exercise of the imagination [6]. I recognize the individual horse, how I imagine it is thinking, it’s personality and I ‘talk’ to it. Through this operation, I come to see that it is not alien to me at all, but an individual that I emphatically recognise through its horselike way of thinking and being in the world and how it is trying to listen to me. The problem with Christian’s horses is that I did not know them as I was a newcomer to his stables – my ability to construct in my mind’s eye, a sense of them, of their personality, of their mind, could not work. The ultimate test of this failure: I did not recognise what they were thinking.

231

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 6 Christian showing how my butt is sticking out and my legs are pointing forwards (two leftmost pictures). Then showing what my body position should be, and how I can be rising from the position (three rightmost pictures).

Their stride was much bigger than the Icelandic horse and it made me feel that maybe this is a different kind of animal. Not a horse as I know them but some other kind of intelligence. These horses do in fact have quite different behaviours compared to Icelandic horses as they are more nervous, higher strung (a quality that breeding has emphasised as these horses were used in war and therefore bred to be very fast and have quick reactions).

interested in me. I therefore tried to distract him, make him listen to me. I pulled at the reins to say “hello, I am here – listen to me”. I shifted in my seat, and I made my legs ‘present’ in the interaction. But it felt as the more I tried to get in contact, the more he closed off and did not want to listen. He just became more silly, more easily spooked. I did not feel that I was in control of the horse, something that makes any rider very nervous. As I did not feel that I could decide what to do on the riding track (Christian is in charge of what we do after all), I could not use the tricks that I would have liked to use – going into smaller volts, changing direction, distracting Liberty and getting his attention through saying “here I am, I want to do these things, let us have fun together”.

Christian’s way of teaching and my focus on the way I was sitting did not give me much room to establish this kind of relationship or explore who the horse was. His horses were also very well behaved, allowing me to concentrate on my way of sitting rather than controlling the horse. An exception was perhaps the horse named Liberty. Liberty was a huge, big horse that kept chewing on everything in a playful style, mirroring a teenager. Not dangerous, but a bit nervous and silly. The kind of horse who like to pretend to be spooked by some dangerous shadow in a corner, allowing it to get away with jumping here and there in a way that makes riders like me totally distraught and immensely nervous.

As I was doing all of this ‘give me attention’-business, I fell back into a bad habit of watching Liberty’s head. By doing so, you can get some more signs of where the horse is heading, what it is thinking or looking at, and whether it is listening to you. But the rider should not be watching the horse’s head. My head should be directed towards where we are going, since the balance of my head and the position of my shoulders tell the horse where we are heading. I also need to plan ahead where we should be going next, preparing the horse and myself for what is going to happen. If I look in the direction of where we are going, I will still see the horse’s head in the periphery. And by looking at where we are going and planning our route, I stay in control, rather than acknowledging and giving the horse the control. On and off, I tried to stop watching Liberty’s head, but because of this lack of contact or dialogue between us, I kept looking back at his head to be prepared for any action he would take. We were not ‘one’, but two that were not collaborating.

At the beginning of the lesson with Liberty I was immensely tensed. Liberty has already on our way towards the riding track decided that there were ghosts everywhere and was ‘spooked’. I could feel his tension and how it increased by my tensed movements. My butt-muscles and armmuscles were extremely tensed. I tried to make them relax, but failed. The tension in my arms made them straight and stiff rather than bent at the elbow and relaxed along the sides of my body. Christian could see how tensed I was and asked me to stop being so silly. During the lesson, I felt that Liberty was not listening. My feeling was that I needed to take more control, provide more signs to make him listen to me and start collaborating. I did not feel that Liberty was testing me, more like he was genuinely into his own thing, being spooked, and not that

Towards the end of the lesson we finally connected. I was asked to canter, and for some reason, this made a lot of my tensions go away. I took more control in the canter, was not

Figure 5 Finally, I am in the right position, balanced, and Henry is “flowing freely” with a forward gait (more or less).

232

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

worried that he would start being spooked. As the speed increased, some of the nervous energy was let loose, and the steering was easier. The speed and the rhythmic beat of the canter are exhilarating in themselves – it became physically enjoyable. In particular, I felt that the canter helped me to get into rhythm and after each canter both Liberty and I were more loose, relaxed, flowing in our joint stride. Liberty’s canter was big in its stride. It had a nice big, rolling rhythm. Together we went from canter to rising trot on and off. Both felt really good and relaxed. Liberty became more interested in the dialogue. The experience of it was fun, which in turn made the whole thing work. I only had to think ‘forwards’ or ‘let’s canter’ and it happened.

agents with quite different ways of being in the world – my own human body’s perceptual way of being in the world, vs. the horse body’s perceptual being in the world. I learn to express with the horse so that I, as a rider, can come to emphatically understand how the horse lives in its world – our world in common. When I do, and when I manage to capture the horse’s interest and attention, we sometimes get into an experience where there is only one being, one center of experience, a centaur, an animal of two in a world made the same for both: for the horse and for me. But when this is achieved, the two do not entirely merge. Important in horse-back riding is to leave space for the horse do its job – there is this space that I need to open for the horse to prefer to go there. I need to be out of the way, balanced in my own body, to not disturb the horse and its movement. As a rider, I found it very interesting to hear Christian’s account of how he could see when I was sitting wrongly or inflicting in a bad way on the horse:

To make this happen, true sympathy is required [5]. You have to recognise the otherness and difference in the horse and create a lived experience together. You have to forget about your own human self and instead turn yourself into a centaur self – consisting of two agents acting together. It is not without risks or wilderness, you might fall, the horse might take off in an unexpected direction. But when the connection and collaboration between the two minds happen, we both enjoy it.

“I watch the horse. The horse tells me what the rider is doing. The shape of the horse. The dynamic confirmation. The shape of how the horse is moving. We saw that earlier on this video clip earlier when you were out of balance and the horse was going very short in his stride and he was coming into the middle of the circle all the time. And then when you became better balanced he opened his stride and went into the bigger circle.”

TRANSFERRING LESSONS LEARNT TO DESIGN

In my view, there is no need to replace the horseback riding experience with a digital riding experience. This is not why I wanted to describe this experience. Instead, I want to transfer some of the qualities in the human-horse interaction into other applications we might want to create with digital technology – what Ljungblad and Holmquist name transfer scenarios [16]. Their idea is that we can learn from people’s specific practices and use the qualities of those practices and transferring them into innovative design that can attract a more general audience. They have tested this process on e.g. studying reptile owners and transferring qualities of their relationships with their animals to the design of home robots. The difficult step in their method is narrowing down on (1) what the qualities are, and (2) which are worth transferring and (3) how to implement those in an innovative design. Here, I want to suggest what some of the more important qualities from my horseback riding experience that we may want to transfer to design.

This could be a quality of design that we might find interesting to transfer to some kinds of human machine interactions – such as robot-interaction or interaction with various intelligent systems in our environment. Rather than making that relationship an explicit language, it could build on bodily signs and signals, allowing robot and machine to create a mutual understanding (cf. the work by Sengers [24] and discussions on machine intelligence by Taylor [28]). Bodily Learning

Many of the lessons learnt above where about bodily training. I found learning processes that to a large extent are already known from the literature on learning: •

I also want to point out that while others have also considered the importance of the body in design, e.g. [7, 12], their focus is most often on how to improve a function or serve a another purpose in the interaction, not on the aesthetics and enjoyment as an end-goal in itself (with some notable exceptions [21, 27, 30]).

•

•

A sensitive and delicate relationship of wordless signs and signals

The relationship between the horse and myself is based on non-verbal, very subtle signs and signals – a language of sorts even if not lexographical. There is a mutual space of signs and signals that we are involved in – to some extent translated by Christian, the instructor – but to a large extent happening between the two of us. This language worked to bring us together even though I assume that we are two

If you distract the conscious self, you can make people use their ‘muscle memory’ or imitate and thereby get it right [10] That you can get into an all-absorbing state as I did towards the end of the lesson – which is similar to flow, being in the zone in sports, and a range of similar descriptions That we move back and forth between states where we can think about body parts and where they are placed or what they are doing as objects or instruments (present at hand) and states where we are inside the experience doing the thing (ready to hand)

But when we design interactional systems involving our bodies, we rarely discuss how this bodily learning process is supposed to happen. Intuitively, many designers build these kinds of processes into their systems, but there is no systematic description of how to do it. In a sense, HCI-

233

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

research has been very concerned with zero-learning time. But as we could see from my account of riding, a great deal of the pleasure comes from applying yourself really hard, learning the movements that are needed in order to get to states of aesthetically appealing rhythmic interaction. Rhythm – an aesthetic experience

I have kept coming back to rhythm, as it is an important aspect of riding. At the same time, it is an interaction quality we rarely talk about in our design processes even if we know that any interaction, be it bodily or a more traditional desk-top system, reveals its dynamic gestalt over time and space [17]. Any such interaction will have its rhythm – its ebbs and flows, intense or more slow reflective parts. But in the horseback riding, rhythm is a more intense experience, strongly intertwined with all the different parts of the riding activity. Many of our everyday bodily experiences that we may want to mirror in interface design for sports, life style and heath applications will be rhythmically oriented. They may follow our circadian rhythm (biological processes following our 24-hour day), heartbeat, movement-oriented rhythm (picking up on walking or running behaviours), or everyday events (sleep, work, lunch, work, home). In my group, we have designed one such system named Affective Health [18, 23]. It picks up on bio signals from sensors placed on the body, transmits them in real-time via Bluetooth to your mobile phone were they are portrayed as an animated, pulsating torus. Similarly, the artist George Khut has created a range of art installations where visitors can see their bodily states mirrored in captivating, colourful animations [11]. These are just a couple of examples of interactions where our own bodily rhythm makes the interaction come alive – obviously we could imagine many more. Especially interesting are rhythmic interactions where the technology has some agency, moving us rather than us moving it. Describing Movement Experiences

Above, I tried to provide an account of exactly how I was sitting, which muscles were tensed, how I balanced on my sitting bones, the direction of my head and vision, the position of my legs and the tension in arms and hands. Still, the experiences I am trying to describe are wordless, and putting detailed descriptions of them still fails to cover the complexities and the uniqueness of my embodied experience. We rarely describe bodily, experience-oriented, system interactions in such a detailed way (in CSCW, there are system interaction descriptions based on conversation analysis, but their focus is mostly on the functionality, not on the subjective experience.) But in order to design for such interactions, we will have to do so. In particular, we need to start considering the aesthetics of movements. Why, for example, certain rhythmic movements feel awkward, and other states makes us leave our analytical selves behind, and instead allows us to be all-absorbed by the experience. Through my description of riding I want to show how those bodily postures corresponded to my experience of being in those lessons with Christian. That experience was a mixture

234

of my bodily posture, my expectations on my own expertise and embarrassment of having to relearn so much, my interaction with the horse, with Christian, and with the physical location – my embodied, lived experience. When MerleauPonty writes about the body he begins by stating that the body is not an object [20]. It is instead the condition and context through which I am in the world. Our bodily experiences are integral to how we come to interpret and thus make sense of the world. Our experience of the world depends on our human bodies, not only in a strict physical, biological way, through our experiential body, but also through our cultural bodies. Merleau-Ponty [20] attempts to get away from the perspective of the doctrine that treats: “perception as a simple result of the action of external things on our body as well as against those which insist on the autonomy of consciousness. These philosophies commonly forget – in favour of a pure exteriority or of a pure interiority – the insertion of the mind in corporeality, the ambiguous relation without body, and correlatively, with perceived things”. (p. 3-4) Through integrated descriptions of both our experiential and cultural bodies, living everyday, mundane, practices such as horseback riding, we may be able to reach a body of work that we can use in the design processes. Similar to any artists, as designers we have to practice our sensitivities in seeing, feeling, doing, hearing bodily experiences. SUMMARY

From a detailed account of my horseback riding, I have extracted a number of experience-oriented qualities. I have hinted at how those might be considered in design processes, such as: • • • •

Designing spaces for mutual wordless understanding between human and machine Letting bodily learning take time and be a pleasure in itself Putting more emphasis on the aesthetic pleasures of rhythm when designing for bodily interaction Finding ways of describing experiences of bodily interactions that can serve as inspiration to design

Hopefully these kinds of insights can take us beyond designing for the experiences of the body to the narrow domain of games and entertainment. In my view, we need to return to being more concerned about our bodily experiences, returning to the care that ergonomics shows the body, but with a stronger focus on experience, and not only function. Acknowledgments

This study was done while I was a visiting researcher at Microsoft Research in Cambridge, U.K. I would like to thank Richard Harper and Alex Taylor for comments on this study and their help in the analysis of the data. REFERENCES

1. Boehner, K., Sengers, P. and Warner, S. (2008). Interfaces with the ineffable: Meeting aesthetic experience

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

on its own terms. In ACM Transactions on ComputerHuman Interaction, 15 (3).

16. Ljungblad, S. Holmquist, L-E. (2007) Transfer scenarios: grounding innovation with marginal practices, Proc. of the SIGCHI conf. on Human factors in computing systems, San Jose, California, ACM Press.

2. Bødker, S. (2006) When second wave HCI meets third wave challenges. Keynote paper, Proc. of NordiCHI.

17. Löwgren J. (2001). From HCI to Interaction Design. In Chen, Qiyang (ed.), Human-Computer Interaction: Issues and Challenges, Hershey, USA: Idea Group Inc.

3. Csikszentmihalyi, M. (1990). Flow: The Psychology of Optimal Experience. Harper Perennial, London. 4. Darwin, C. (1872/1998) The expression of emotions in man and animals, Third ed. by Paul Ekman. Oxford University Press, 1872/1998.

18. Kosmack Vaara, E., Höök, K., Tholander, J. (2009) Mirroring bodily experiences over time, Proc. of the 27th international conf. extended abstracts on Human factors in computing systems, Boston, MA, ACM Press.

5. Game, A. (2001) Riding: Embodying the Centaur, 7 (4): 1, Body & Society, 2001; 7; 1.

19. McCarthy, A. Wright, P. (2004) Technology as Experience. Cambridge, MA: The MIT Press.

6. Heal J. (1989). Fact and Meaning: Quine and Wittgenstein on Philosophy of Language, Blackwell, Oxford.

20. Merleau-Ponty, M. (1962). The Phenomenology of Perception, C. Smith, trans., London: Routledge & Kegan.

7. Hillerup Fogtmann, M., Fritsch, J., Kortbek, K. J. (2008). Kinesthetic interaction – revealing the bodily potential in interaction design, Proc. of the 20th Australasian Conf. on Computer-Human Interaction: Designing for Habitus and Habitat, 2008, Australia.

21. Moen, J., (2006). KinAesthetic movement interaction: designing for the pleasure of motion. Doctoral Thesis, KTH, NADA, Stockholm, Sweden. 22. Quine, W.V. (1960). Word and Object, MIT Cambridge.

8. Isbister, K. Höök, K. (2009). On being supple: in search of rigor without rigidity in meeting new design and evaluation challenges for HCI practitioners, In 27th international conf. on Human factors in computing systems, Boston, MA, ACM Press.

23. Sanches, P., Höök, K., Kosmack Vaara, E., Weymann, C., Bylund, M., Sjölinder, M. (2010). Mind the Body! Designing a Mobile Stress Management Application Encouraging Personal Reflection, In Designing Interactive Systems, DIS 2010, ACM Press.

9. Ingolds, T. (2006). Walking the plank: meditations on a process of skill, in J. R. Dakers (ed), Defining technological literacy: towards an epistemological framework New York: Palgrave Macmillan, 2006), pp 65-80.

24. Sengers, P. (1998). Anti-Boxology: Agent Design in Cultural Context. Ph.D. Thesis, School of CS, Carnegie Mellon University. Technical Report CMU-CS-98-151.

10. Kay, A. (2003). Education in the digital age. (27:45) http://video.google.com/videoplay?docid=2950949730059754521&hl=en.

25. Sheets-Johnstone, M. (1999) Emotion and Movement: A beginning Empirical-Phenomenological Analysis of Their Relationship, Journal of Consciousness Studies, 6, No. 11-12, pp. 259-277.

11. Khut, G. (2006) Interactive Art as Embodied Enquiry: Working with audience experience. In Edmonds, E., Muller, L., & Turnbull. D. (Eds.) Engage: Interaction, Arts & Audience Experience. University of Technology, Sydney, Creativity and Cognition Studios Press.

26. Ståhl, A., (2006). Designing for emotional expressivity. Unpublished Licentiate Thesis in Design, Umeå Design Institute, Umeå University, Sweden.

12. Klemmer, S., Hartmann, B., Takayama, L. (2006). How bodies matter: five themes for interaction design, In Proc. of DIS 2006: ACM Conf. on the Design of Interactive Systems. State College, PA, 2006.

27. Sundström, P., Ståhl, A., and Höök, K. (2005). eMoto Affectively Involving both Body and Mind, In Extended abstract CHI2005, Portland, Oregon. 28. Taylor, A. S. (2009). Machine intelligence, Proc. of the 27th international conf. on Human factors in computing systems, Boston, MA, ACM Press.

13. Laban, R., Lawrence, F.C., (1974). Effort Economy of Human Effort, Second ed. Macdonald & Evans Ltd., London, UK.

29. Tholander, J., Johansson, C. (2010). Bodies, boards, clubs and bugs: A study of bodily engaging artefacts. In Extended Abstracts 28th ACM Conf. on Human Factors in Computing Systems, Atlanta, ACM Press.

14. Laurier, E. and Brown, B. (2004) Cultures of seeing: pedagogies of the riverbank, Manuscript, Institute of Geography, Edinburgh Univ., 2004 online: www.geos.ed.ac.uk/homes/elaurier/texts/Laurier_cultur es_of_seeing.pdf

30. Tweneboda Larssen, A., Robertson, T., Edwards, J. (2007). The feel dimension of technology interaction: exploring tangibles through movement and touch, Proc. of the 1st Intern. Conf. on Tangible and Embedded Interaction, ACM Press.

15. Ljungblad, Sara (2009) Passive photography from a creative perspective: “If I would just shoot the same thing for seven days, it's like...What's the point?”. Proc. of 27th International Conf. on Human Factors in Computing Systems, Boston, MA, USA, ACM Press.

235

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Controlling the Use of Collaboration Tools in Open Source Software Development Heli Ikonen, Netta Iivari & Henrik Hedberg Department of Information Processing Science, University of Oulu P.O. BOX 3000, 90014 Oulu, Finland [email protected], [email protected], [email protected] ABSTRACT

model relies on communities where large amount of people can improve the software, adapt it to their personal or organisational needs, and fix defects. [31] The most wellknown OSS products are Linux operating system, Apache web server, OpenOffice.org application suite, and Mozilla Firefox web browser.

This paper analyses control in the open source software (OSS) development context, focusing specifically on how the use of collaboration tools, such as bug trackers and mailing lists, are controlled in OSS projects. The tools are vital for the functioning of distributed OSS projects. One OSS case project was analysed in the paper. The findings show a surprising amount of control in the OSS project. In addition, when compared to traditional information systems (IS) development projects, different control modes and mechanism were found. Strong evidence for informal ‘clan’ and ‘self-control’ were found, but also a lot of formal control was in use. The results are discussed in connection to OSS literature and critical IS literature.

The OSS project is often depicted as a hierarchical or onion-like structure [5, 6, 13]. Usually, the decision-making power is centralized on a core team consisting of one to a few skilled and recognized developers. Other active participants positioned in the next layer are often called 'codevelopers' or simply 'developers' emphasising the importance of the programming. However, it must be noted that there are also other tasks in OSS projects, such as documenting, web page and other infrastructure maintaining, and testing. The outer layer of the OSS project structure consists of users. The furthermost individual is solely using the software, but some users may also communicate with the project by, for example, filing bug reports. The granularity of the structure may vary from project to project, but in most cases it forms the same hierarchical setting with very few developers in the centre making decisions and increasing amount of individuals around them with lower activity and communication levels.

Author Keywords

Control, open source software, collaboration tools ACM Classification Keywords

H.5.3 Group and Organization Interfaces INTRODUCTION

This paper examines control in the open source software (OSS) development context. More specifically, the research question addressed in this paper is: 'How is the use of collaboration tools controlled in OSS projects?' OSS means software of which source code is available for everyone to read and modify. Open Source Initiative defines OSS through its license, which has to comply with the given criteria (see http://www.opensource.org/docs/ definition.php). The key idea is to let software evolve freely without any restrictions to its usage or development. Traditional software development is carried out in closed settings, where only a few paid people can access the code, and end users have just the binary form of the software enough for using it. On the contrary, OSS development

The OSS development process has been introduced as a solution to software industry's problems of slow and expensive software development that does not fulfil quality expectations [10]. It has been argued that OSS can even challenge commercial software products in the quality of the product as well as with their development speed and cost [31]. OSS communities are in many cases based on voluntary work [9]. However, OSS development has been noticed to evolve into a more commercial direction. OSS developers are increasingly getting paid to do their jobs and the users are more often willing to pay for open source products or the supplemental of open source products. [11]

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

In OSS development it is usual that the members work at geographically dispersed locations. Carmel and Agarwal have argued that geographically dispersed organizations cannot work efficiently without control and collaboration. Collaboration to these researchers is "the act of integrating each task with each organizational unit, so the unit contributes to the overall objective" and control is "the process of adhering to goals, policies, standards or quality

1 236

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

levels." [4: pp. 23] A geographically dispersed organization can form social relations and create organizational tacit knowledge. It has been argued that this happens through a collaborative development environment. [3] Due to geographical dispersion, OSS developers need to have reliable and efficient collaborative development tools that offer them a channel to communicate and to distribute the work tasks between them [23]. This way of working also requires collaboration systems to work efficiently [9]. However, open source communities are organized quite freely and due to this, methods, processes and tools are heterogeneous [5]. Most commonly used development tools in OSS communities are OSS solutions themselves. Widely known and used collaborative development tools are bug trackers (e.g. Bugzilla), revision control systems (e.g. CVS Concurrent Versions System), automatic code builders (e.g. Tinderbox) and different way to communicate via Internet (e.g. mailing lists). [22]

developers. [25, 32, 34] Therefore, OSS development has made the long-standing boundary between design and use very fuzzy [1, 15]. Furthermore, OSS development has blurred the boundary between leisure and work, since it typically relies on voluntary developers with motivations such as gaining a reputation among peers, loving to code, altruism, fighting against proprietary software, learning and scratching own itch, but also companies and paid developers have entered the scene with economic motivations [2]. In this paper, we complicate the picture even further by examining the OSS developers as users of their collaborative development tools, and focus particularly on how these users and their tool usage are controlled in this voluntariness and openness emphasizing context. This control is partly enabled by the way these tools have been designed, which bears some implications for HCI research and practise. The paper is structured as follows. The next section discusses existing research on control in traditional IS and OSS development contexts, as well as outlines some studies critically addressing the issue of control in IS context. In the third section, the research setting and data collection and analysis methods are outlined. The fourth section presents the findings of our empirical examination, the fifth section discussing their implications. The final section summarizes the results and their limitations and identifies a number of paths to future work.

There is a lack of research on the adoption and use of these collaborative development tools in OSS projects. However, there is an ample literature base on the adoption and use of different kinds of information systems (IS) in organizational context. This literature has long ago indicated that the success of IS implementation is not selfevident. Orlikowski, for example, has shown in numerous studies (e.g. [26, 28]) that there might be huge differences between the goals of the management and users, and users might be very powerful in obstructing the implementation process. This may be the case even if the system was technically of good quality. Users may still choose to use it in different, surprising or even in harmful ways. [28]

RELATED RESEARCH Previous studies about control

Control covers all the methods and processes that aspire to predictable behaviour. In IS context, the target for control is the user and his/her way of using it so that it would be consistent. [16] Control is usually divided into two subcategories: formal and informal [29, 16, 4]. Formal control typically consists of budgets and explicit guidelines. Informal control could be for example peer pressure. [16] In this study, we have chosen to use Kirsch's portfolio of control to recognize and classify different forms of control. Kirsch was chosen for real examples of control in commercial information systems development (ISD) projects. Kirsch’s definitions are based on Ouchi's view of control, which is widely used also in IS studies. Kirsch has divided the two subcategories of control into four modes: behaviour, outcome, clan and self-control. The control mechanisms that are used to exercise the control mode overlap among the different modes. The implementations of these four modes are introduced next.

The traditional way to control how an IS is to be used is to provide user trainings in the implementation phase. In groupware systems’ case, this has been said to be particularly efficient. [27] In training sessions, users can compare their experiences and create shared work practices for using the system. Also guidelines, walk-throughs and recommendations by expert users may be used for establishing the desired usage patterns. [28] Furthermore, project plans and meetings can be used for this. [16] Supervisors are typically assumed to exercise control, but actually it is not always related to a relationship between a supervisor and a worker, but control can also exist between same level workers, even in hierarchical organization [17]. OSS development process is a complicated phenomenon that is influenced by technology, human behaviour, economics, culture and society [32], offering a new and fascinating research context for Human Computer Interaction (HCI) research. Traditionally, software has been developed by technically skilled engineers for people without these skills, but in OSS development the boundary between developers and users has been blurred. The developers tend to be the users of the software they develop, and all users need to be considered as potential

Behaviour control is a formal control mode that is implemented with rules and procedures. There are rewards based on following those rules. Behaviour should be visible and observable. Outcome control is also formal control mode that is implemented by the outcomes and goals. Also with this control mode rewards are based on producing

2 237

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

those goals. Outcomes should also be measured against those goals. [16]

regulating relations, and where the network consists of individuals rather than institutions." So he does not see control or regulations as important factors in OSS development model. [19]

Clan control mode is implemented in common values, beliefs and problem solving philosophies. Also the identification and reinforcement of acceptable behaviours is implemented. New members are selected with care and taught to follow the clan regulations. These regulations are commonly unregistered. It is also noticed that specific task goals evolve over life of the task. Finally, self-control mode is implemented with task goals and procedures that individual define. Individual also monitors, rewards and sanctions self. Rewards are based partly on individual's self-control skills. Self-control is visible in individual's efforts. [16]

In another study, the results are somewhat the same. Persson et al. concluded in their OSS project case study of Argo project that control is weaker in an OSS project than in a typical commercial systems development enterprise. They specify this as lack of predictable time scales. They also notify that the developers in Argo have wider access to the code base that they would have in a commercial project and this can be seen as a burden to developers as they cannot focus on their specific modules. [30] Thus, it seems that previous studies have come up with contradictory conclusions about control in OSS communities. The way of working openly has been seen as an advantage and a disadvantage. The aim of this study is to clarify OSS communities’ way of controlling by focusing on their collaborative development tool usage, as using them is essentially important to OSS communities as many researchers have concluded.

Also users of IS can be involved in controlling an ISD project [16] In addition, communication is nowadays a mediating factor in both control and collaboration [4]. In overall, it is important to understand that control is directed to the interaction between the user and the system. This is based on the view that qualifies technology as a neutral element and suggests that all the factors that determine and constrict the usage of the system are coming from the user's attributes or from the organization's attributes in which the user is a part of. [26, 28]

Critical studies

In this study we also approach controlling from a critical point of view. Studies concerning controlling have introduced many different critical views to it. In this chapter, we introduce some of them, which have also inspired us completing this study.

Previous studies about open source and control

OSS development process has been characterized as meritocratic. Decision power is normally concentrated on only a few people. This has also been seen as a problem: for governance and quality practices trusting only to a few people's know-how the quality of the software is depended on how good these few people are. [24]

It has been said that control is based on invisible social norms and relations. Control can therefore provide an efficient way of using a system consistently or as an opposite; control can also completely prevent the usage of a system. Doolin has carried out an empirical study of how an IS can be used in controlling a workplace. According to him, the thought that IS users are only items that the system monitors is far too simplistic. Users can always choose to work in another way. Interesting comment that Doolin represents is that the introduction of an IS can lead to an organization-wide debate about what the system can monitor. The users can question the entire system and the need for it when they question the monitoring and controlling aspects of the implemented system. Doolin has said that using too many resources for governance can add the possibility for the resistance of the system. Too much control can also make users questioning the purpose and usage of the whole system. [8]

Latteman and Stieglitz have proclaimed OSS communities to be highly depended from different support and communication systems. They mention revision control systems, news groups and mailing lists as examples. They particularly emphasize the importance of these systems during the community’s growth. [18] Gallivan has conducted a study where he examined the results of nine different OSS case studies and focused on the findings of control and trust. His discovery was that OSS projects have clearly used a variety of control mechanisms and policies. Gallivan also suggests that the OSS projects cannot operate effectively based only on trust. [14] Lundell et al. have studied communication, cooperation, and control in the OSS development context. They conclude that control in OSS projects is generated through maintaining a shared vision that guides developers to share a common understanding of the goal of the project and protects it from forking (branching). [20]

Zuboff has also argued that too strict control can lead to a deskilling of user and the freedom to use a system differently should be seen as a learning possibility and may lead to an increase of new skills. Zuboff has stressed the possibility for the negative consequences of technologyassisted control and surveillance. IS can store all tasks and operations and there may be no one supervising them. [35]

Ljungberg argues that OSS project "could be seen as a virtual organization, but where trust has replaced law in 3 238

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Earlier we noticed that OSS development process is a complicated phenomenon influenced by technology, human behaviour, economics, culture and society [32]. In a larger perspective it is worthwhile to notice how control is a part of the society and its trends. Many scholars (Zuboff, Lyon, Deleuze) have used a prison type designed by Jeremy Bentham, the Panopticon, as a metaphor for computermediated surveillance and control. The first who made Panopticon well-known was philosopher Michel Foucault who saw it as "both a sign of and metaphor for this new disciplinary society" [35]:

One OSS project was selected as a case for this study. Before selection the following criteria that the case should fulfil were defined: the project should have publicly available information about project management and it should have a public bug tracking system. The project should be medium sized so that there would be enough data for analyzing without being over-exhausting. Too large and too small projects were discarded. Other criteria were that the project should be active one and develop an open source licensed software product. The selected case project is based completely on voluntary work. The project has forked from different project in 2002 and it publishes software under LGP-license (Lesser General Public). Project has documented their project activities remarkably well in project's wiki. The project has also defined its purpose in vision document in the wiki. It is argued that this community should be driven by “innovation” and “experimentation” and that it should be “fun”, “stimulating” and “educational” (Project's Community Vision, wiki, 2009).

"Hence the major effect of the Panopticon: to induce in the inmate a state of conscious and permanent visibility that assures the automatic functioning of power. So to arrange things that the surveillance is permanent in its effects, even if it is discontinuous in its action; that the perfection of power should tend to render its actual exercise unnecessary; that this architectural apparatus should be a machine for creating and sustaining a power relation independent of the person who exercises it; in short, that the inmates should be caught up in a power situation of which they are themselves the bearers." [12: pp. 201]

The data was collected in a time period of May to October 2009. At the time there were about 20 developers in the project with two of them as lead developers. The project stored most of its development related data in the wiki. It also functioned as the project’s public web pages. In addition, the project used several other collaborative development tools: mailing list, bug tracker Mantis, revision control system Git and IRC for communication. The project had stored mailing list conversations to public archives since 2006. The mailing list and Mantis were examined totally. There were 1813 bug reports on Mantis and about 1500 e-mails in mailing list archive at the time of data gathering. Between the years 2006 and 2009 project had organized several IRC meetings. The minutes from these meetings were stored to a website which was publicly available. All these minutes were collected and examined for this study. From the wiki, 8 different web pages with their history data were collected and analysed. The collected pages represented guides and definitions of different work tasks. In addition to all material being publicly available, the permission to use data was asked from a lead developer of the project.

Deleuze argues that the disciplinary society that Foucault represented is already changing to a control society where people are confined by continuous control and instant communication. Control for him has especially taken place in the continuous monitoring of people. He also suggests that control in society cannot be explained just by the machines and computers that are used for control; it must be seen as collective arrangements where the machines are only one component. [7] Also Lyon has made a significant contribution in the area of surveillance and control. He writes "... to a virtual reality where the spaces and times to be controlled are manufactured ones. The environment can be perfectly controlled. Indeed, it no longer requires control; the environment is control". [21] RESEARCH DESIGN

This research was conducted as a qualitative case study. The case study method aims to understand complex social phenomena such as organizational processes. Case study method also aims to investigate a phenomenon within its real-life context and within multiple sources of evidence. Case study can benefit from previous theoretical propositions: they can guide data collection and analysis. [33] In this case study previous theoretical contributions were used as sensitizing device guiding and inspiring empirical analysis.

The data analysis was executed as follows. The email archive and bug tracking system data were examined in two phases. First phase was to make a swift look on the collected material and make a selection of seemingly interesting e-mails or bug reports. In a second phase, this selected material was inspected more carefully and Kirsch's definitions were used to identify different kinds of control methods and their implementations. For IRC meeting minutes and wiki pages there was no need for the first selection phase as their overall content was smaller so they were inspected with Kirsch's definitions as a classifier. A

OSS projects use many kinds of collaborative development tools efficiently and are actually highly reliant on them due to the distributed nature of the projects. It has been noticed that as a by-product the tools offer valuable research data [23]. All the data in this research was gathered from Internet sources that are public to all.

4 239

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

number of quotes were collected during this phase as examples of different control modes and related issues.

reporting system Mantis and in the mailing list to refer to the wiki documentation and to ask to read the instructions from there:

The data that was used as a source for the empirical results was collected from several sources. From the project's mailing list archive, totally 103 e-mail were analyzed. The archive consisted of emails written during the years 20062009. From Mantis, the project's bug tracking system, 51 bug reports were analyzed. In Mantis archives the oldest reports were written in year 2005, yet the newest were from the summer 2009. Of the project's IRC meeting minutes, 10 of them were analyzed. These meetings were conducted in years 2006-2008.

You should check out the wiki related to installation to get more information. (Lead Developer, Mantis).

The project participants did also look after that the Mantis users were following the documentation: See http://project.sourceforge.net/index.php/Reporting_bugs which is a reference that contains useful information related to what should go into bug reports. (Developer, Mantis).

It was frequent to refer to the wiki. The referred pages included also other pages that were not related to Mantis or a bug:

From the project's wiki the following pages were analyzed: • Changelog Format • IRC-Meeting • Release Checklist • Submitting Code • Development Processguide • Wikipolicy • Project's Vision • Reporting Bugs The wiki pages were written in years 2005-2009.

Next time you should at least read the style guidelines :) (Developer, Mantis).

The guideline refers to project's Coding Style page. It is noticeable that the advice to read the wiki for more information was given between developers and between developers and users. However, the strictest control was in use in the project IRC meetings:

EMPIRICAL RESULTS

I propose that we have a regular (e.g. every week or so) project meeting for contributors and other interested parties, on #channel @ Freenode.... (Developer, project's mailing list).

This section examines how the OSS project participants as users of their collaborative development tools are controlled. The empirical results are grouped based on the control mode that they are identified with.

The project had formed a very precise guide to communicating in IRC meetings. The guide defined that all should prepare a one-line status report, which they should present in the beginning of an IRC meeting:

Behaviour control

Behaviour control is a formal control mode that is based on rules and procedures and rewards based on following them. Mechanisms for this mode in Kirsch’s study were project plans, meetings, progress reports, walk-throughs, ISD technical documentations, organizational roles, conference calls, management travel, and systems demonstrations. This control mode was the most important one in Kirsch’s cases: all projects had project plans and different types of reports and meetings. However, only one project had ISD technical documentation. [16]

I have updated on the wiki the IRC Meeting page and added that several items should also appear in one line. Our goal is to keep things as succinct as possible, so think catefully your statements in advance and condense them into one line. (Developer, project's mailing list).

This line should contain information about what the developer has done (DONE), has there been any problems (BLOCKING), what the developer will do next (TODO) and whether he had anything he wanted to discuss about (DISCUSS). The line could look like this:

Based on the empirical analysis, we claim that behaviour control was the strongest control mode in the case project as well. The project had even more this type of controls than Kirsch’s cases. The most important finding was the wiki documentation. There was a considerable amount of documentation stored in the wiki (see the previous section). There were checklists for different work phases, like Release Checklist for publishing a new version of the software. There was also a guide to using the Git revision controlling system and policies for using and editing wiki and for committing a bug report.

DONE: pack libjava together with classfiles into the jar, much x4x development, TODO: have a look at the new full methods, probably useful, BLOCKING: none, DISCUSS: javabindings in next version. (Project's IRC meeting log)

However, as an IRC meeting went on, the discussion changed to be more informal. The project did also quit having these meetings: the last meeting minute was from year 2008. As mentioned, behaviour control is connected with observability: project members’ actions should be visible. This indeed happens; the tools the case project uses make possible for anyone to observe how the project is working.

Actually, the wiki can be seen as the most used tool in controlling the project. It was a common event in the bug

5 240

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Another observation related to behaviour control is that in Kirsch's definition there were rewards as a result of following the rules. However, there is no evidence that the case project used anything particular to reward the developers. Still, this does not ensure that there weren't any.

Kirsch cases there were only some findings of clan control: two cases had IT culture related findings but the two other cases had no clan control mechanisms identified. [16] This mode appears as a strong control mode in the case project. The strongest mechanism for this mode seemed to be peer pressure. Findings in the case project show that it had a particular culture of its own and also open source culture can be seen as an effective factor. Physical and IRC meetings were used as mechanisms of clan control. The project members had met face-to-face once a year in the years 2008 and 2009 at open source developers' meeting:

Outcome control

Formal outcome control is executed through measuring outcomes and rewards based on achieving the goals. The mechanisms for this mode in Kirsch’s study were project plans, requirement documents, standards, organizational roles, presentations to user community, system testing and 'walking around'. [16]

Still another idea that came up in the FOSDEM [Free and Open Source Developers' European Meeting] is that we ought to document the development process somewhat better and have a good discussion about it to ensure that everybody is one the same level. (Lead developer, project's mailing list).

This type was the least used mode of control in the case project. However, comparison between a traditional ISD project and an OSS project is difficult, since the case project didn't have budget or time related targets or requirement documents. The case project had the Vision document that described the aims of the project at general. They also had some organizational roles defined. One of the developers conducts himself as a person who is in charge of the wiki:

As the quote shows, there were discussions about the project’s ways of working in those meetings, so the meetings were identified as part of the control mechanisms of the project. For this mode it is characteristic that the rules are not articulated, but they are still followed [16]. There were some findings of this kind of action:

My approach up to now has been that I have allow people to edit and add anything they want, and then I may have reformatted the content and polished it a little, if required. It might be the case that I am the one who takes care of the wiki, but I do not like to think that I am alone responsible for all that is there. (Developer, project's mailing list)

Please don’t change reporter to me in case I haven’t got any idea how on earth it concerns me :) (Developer, Mantis).

The bug report that was directed to this developer was not in his area of responsibility. However there were no findings indicating that these areas were defined publicly.

Surprisingly he incites others to use the wiki without following the wiki policy.

There were also some indicators that suggested that the participants were not supposed to question the project’s way of working:

The project's Vision document, as mentioned, defines that the target goal for the case project is to be an innovative and experimental project. The document can also be seen as a high level agreement of the common goals for the project. Furthermore, it should be noticed that the extensive documentation found in this project could be seen also as an outcome control type. The documents aim for the outcomes to be something that has been defined beforehand. Thus, some of the wiki documents can also be seen as outcome control: for example filing a bug report was defined strictly in the Bug Reporting page and there was also very precise checklist for publishing a new version of the software in Release Checklist page.

Keep in mind that there aren’t guarantees in this business. It is very simple. In case project sucks, you just go somewhere else. :p (Developer, project's mailing list).

There also seemed to be implicit rules maintaining that using personal email is not desirable although it was not forbidden in any of the documents. A developer, replying to another person in the project’s mailing list, claims with an exclamation mark that the other person had exhibited “bad behavior” and commands the recipient to go “sit in the corner”. This case was about replying to a personal email address and not to a mailing list address, but as mentioned, there was no written policy guiding towards the proper behaviour in the project’s mailing list.

Clan control

Clan control is an informal control mode based on unregistered information. Control is based on common values, beliefs and problem solving philosophies. It is also influential how the new members are selected and taught to follow the clan regulations. One mechanism for clan control is also the clan culture guiding the clan’s actions. Other mechanisms identified are the culture and the norms of the organization’s IT department, peer pressure, photo exchange, team member travel and conference calls. In

Self-control

Self-control is visible only in individuals' efforts and is an informal control mode. An individual monitors, rewards and sanctions himself. Rewards are based partly on individual's self-control skills. Kirsch has defined this mode to be implemented through organizational roles, culture of the IS department, and project goals and procedures. In the

6 241

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

findings by Kirsch only two projects had some mechanisms used and the other two had none whatsoever. This was the least used control mode in the Kirsch's study. [16]

members to do certain tasks, meritocracy being widespread in OSS projects [24]. It was mentioned earlier in connection with the behaviour control mode that typically there are rewards as a result of following the rules, but in the case project no evidence related to this was found. On the other hand, there is a lot of research on OSS developers’ motivations, which may explain why they don't need any rewards for following the rules of the project or that they gain rewards but they differ radically from those in the traditional IS development. Based on this study, it appears that peer pressure and selfcontrol seem to be important factors contributing to the compliance with the rules.

On the contrary to Kirsch's findings, there were many findings of self-control in the case project. The findings can be connected with organizational roles and project culture, but self-control could also be seen to operate via the methods and collaborative development tools of the project encouraging the developer to control himself. The working methods and the tools used could be seen as a part of the project culture. Evidence of self-control can be found for example in a comment in the bug tracker in which a developer is asking for the lead developer to close an incorrect bug report filed by the developer himself. The problem in the report was that the problem did not actually exist; the developer just misread the log. The developers is requesting for closing the bug report and argues that the reporter (i.e. himself) is either “a brainless idiot” or “illiterate”.

Results compared to previous OSS studies

Some results of this study seem to be in line with the previous studies, but also some exceptions were found. Gallivan concludes in his study that OSS projects use a variety of control modes and suggests that the earlier assumption that the work in OSS projects is mainly based on trust is not durable. He argues that control in OSS projects is implicit and invisible and therefore it is hard to notice. He also states that OSS projects cannot use only trust to organize the work, on the contrary, the project members observe each others' actions. To sum up, he concludes that there is a surprisingly great amount of different control mechanisms used in OSS projects and many of them are social and self-control sort of control. [14] The results of this study are in line with Gallivan's view: lots of control was found in the case project. Some were characterized as social and self-control modes, but besides there was also strict behaviour control identified in the way that the case project used formal documentation.

Another, somewhat similar example of self-control can be found from another comment in the bug tracker, in which the developer requests to delete his “stupid patch” and asks someone else to do it. In this case the developer has made a software patch for a bug that was reported but he is not satisfied with it and asks publicly to remove his patch. DISCUSSION

All the control modes identified in the earlier IS literature were evidenced in this study. Some of the control modes even seemed to be more evident in this OSS project than in the traditional IS projects. This is somewhat surprising, since OSS development is praised as a voluntariness and openness-emphasizing context.

Latteman & Stieglitz [18] propose that social controls are especially important in the OSS context, even more important than in commercial projects. This view also equals with the results of this study. However, they promote the importance of trust among social control methods, which in the light of this study is questionable. In the case project there seemed to be little left for trust: most of the important tasks were described in the wiki with details. In addition, the developers monitored each other’s actions carefully. Also Lundberg [19] has said that OSS projects have replaced direct orders with trust. This statement cannot be supported with the results of this study.

On the other hand, there were some difficulties in applying the Kirsch’s [16] framework in this new context and in comparing the results. Next few observations related to this are outlined, after which the results of this study are discussed in connection to the existing OSS research as well as in connection to critical IS tradition. One problem in applying the Kirsch’s framework is that in it ‘organizational roles’ was associated with three different control modes: behaviour, outcome and self-control. In this study organizational role is discussed in connection to outcome and self-control. However, there was only one finding related to an organizational role in the case project, the developer who mainly was responsible for wiki. There surely are more informal roles in the project but only one explicit role was identified from the gathered data. Also, some code related tasks were concentrated only on lead developers to do (i.e. merge), but actually this could be seen more as an evidence of the clan control mode; the culture in this OSS project as well as in OSS projects in general seems to allow only the highly experienced and recognized

On the other hand, the results of this study seem to support Latteman's and Stieglitz’s view that social control exists in different kinds of social rituals, ceremonies, morals, and peer reviews. They linked the motivations of OSS developers with the use of these kinds of controls, as the normal monetary rewards are not used in OSS projects. [18] They also make an interesting claim: ”Even if direct and indirect monitoring may be possible, direct and indirect control is hardly feasible” [18: pp. 7]. In the case project,

7 242

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

there were many signs of indirect and direct monitoring and the project also used control successfully as they controlled whether the project members followed the instructions stored in the wiki. So, based on this study, one could claim that both direct monitoring and control can and do exist also in OSS projects.

It was interesting to notice that the metaphor also fitted to this case. One of the researchers that linked this metaphor to technology-enabled supervision and control was Zuboff. She argued that one of the threats of technology is the expansion of control and surveillance which technology can assist. Systems record every action people execute without any supervision. [35]

Lundell et al. emphasize the importance of a shared vision as a part of project control. They suggest that this is an easy way to maintain a common understanding of the goals of the project and in their opinion a shared vision also protects against project forking. [20] The case project had a shared vision: the project's vision document in the wiki. However, there was no evidence that this document supported control methods in any way. On the contrary, the vision document seemed to be conflicting with the way the project actually worked. In the project's vision document the lead developer emphasised fun, innovation and experimentation. However, the culture in the case project seemed to be based more on control and discipline than having just fun and experimenting.

In Deleuze's post-Foucauldian society, control society, people are confined by collectively arranged continuous monitoring and control. As he emphasizes it is not only the machines that monitor and control, it is the whole arrangements. [7] The case project participants did work efficiently as part of the monitoring and controlling processes which were computer-mediated, but which also needed the human actions for the real control to be implemented. Furthermore, Lyon's view that the environment is control [21] is also linkable to the collaborative development tools that the OSS projects use. These tools are designed and implemented by other OSS projects [34]. Maybe due to that background and the conventions of OSS projects, they are designed to be very open for monitoring as well. As the OSS projects do not work in co-located setting due their members’ dispersed locations, control might be built into their virtual environments. Gallivan concluded that control in OSS projects could be implicit and hard to notice. In the light of this study, it could be argued that the collaborative development tools that the OSS projects use are essentially the source for control and controlling the use of them is almost equal to controlling the whole project. Maybe the implicit decisions made in the design and implementation of these OSS tools is the key to understanding of the OSS projects ways of working and controlling, altogether.

Regarding the possibility of forking, in OSS development it is a typical way to resolve a conflict related to organizational or technical disagreements: if one founds a new project he can control it as well as continue from the existing foundations. However, it is not given that other developers will follow, and thus, established projects usually stay alive and successful branches are rare. This situation may be changing as companies are entering into OSS development. They have resources to ensure the progress of the development, but the project leaders may not be recognized and trusted, as they are just paid workers. Persson et al. conclude based on their OSS project case study that in OSS projects control is weaker than it is in a typical commercial systems development enterprise. They specify this weakness as lack of time related control and time related goals in comparison to a commercial ISD project. [30] In the case project, there was no evidence of time related goals or controls either. Another similar observation is related to the openness of OSS projects working. Also Persson et al. notice that OSS projects are more open to feedback and monitoring because the OSS projects way of working is so transparent. All developers can access and review each other’s code. [30] This was true also in the collaborative development tools of the case project: they were accessible even for outsiders.

On the whole, it seems that monitoring and control are essential parts of OSS project's working. It is surprising as the OSS projects are still in many cases based on voluntary work. Interestingly, there was no visible critique among the participants against the controlling and monitoring. This differs from the traditional view that strict control leads to criticism and the more resources are used to maintaining control the more critique against it appears. As Doolin has put it in IS context, the critique can lead users to start questioning the whole purpose of the system [8]. In the case project analyzed in this paper, instead, one might claim that the users of these OSS tools were controlled and they were willing to allow that – they voluntarily subjected themselves to that. However, maybe the OSS developers as users differ from the user population of many other IT tools. This is a question to be answered by future research.

Results connected to the critical studies

The way OSS projects operate is very open and exposed to anyone who is interested. The case project allowed a free access to documents, logs and conversations. This openness for anyone to observe can be linked to a view that technology can enable continuous observation that can be seen also as disciplinary power, much quoted Foucault's metaphor Panopticon as a sign of disciplinary community.

CONCLUSIONS

This paper inquired control in the OSS development context, focusing specifically on how the use of collaboration tools is controlled in OSS projects. The tools

8 243

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

are vital for the functioning of distributed OSS projects. One case was analysed in the paper. The findings show a surprising amount of control in the OSS project, even more than in traditional ISD reported by Kirsch. All identified control modes were found in this case, but emphasis was different: traditional ISD had mostly behaviour control whereas clan and self-control were strong in this case. Surprisingly much formal control was also found. Different control mechanisms were in use than in traditional ISD projects. In OSS projects, control focuses more on outcomes and behaviours but not on time related objectives. Maybe because of that, Persson et al. did not found much control in OSS projects, as opposite to this case. However, in more commercial based OSS project this finding could be very different. Regarding the existing research, this study contradicts the view that trust is an essential factor in OSS projects. Instead, control was very dominant; it seems that there wasn’t much space left for trust.

mentioned here. It has been proclaimed that innovation and learning in IS context is based on the user's possibility to "choose to do otherwise”. [28] It would be an interesting area of research to try to investigate how OSS projects could release their full potential in innovative and educational way of working. In addition, it would be interesting to study how OSS projects led by companies can control their communities: is it money that ensures resources, or should they invest on recognized individuals who lead the project following the OSS philosophy. REFERENCES

1. Barcellini, F., Detienne, F. & Burkhardt, J. (2009): Participation in online interaction spaces: Design-use mediation in an Open Source Software community. International Journal of Industrial Ergonomics 39. Pp. 533-540. 2. Bonaccorsi, A. & Rossi, C. (2006): Comparing Motivations of Individual Programmers and Firms to Take Part in the Open Source Movement: From Community to Business. Knowledge, Technology, & Policy 18(4). Pp. 40-64.

The publicly open environment OSS projects work in is an ample environment for controlling and monitoring and realizes Lyon's view of environments that are the control. As the critical opinions argued, controlling can be also seen as undesirable and harmful phenomenon. In this light it is surprising that the developers voluntarily put themselves under monitoring through the open collaboration environment and they did this without any noticeable resistance.

3. Booch G. & Brown A.W. (2003): Collaborative Development Environments. Advances in Computers, Vol. 59. 4. Carmel E. & Agarwal, R. (2001): Tactical Approaches for Alleviating Distance in Global Software Development. IEEE Software, 18(2). Pp. 22-29, March/April, 2001.

Regarding implications for design, this study highlights that design of tools always entails also the design for monitoring and control - in the workplace as well as nowadays in the everyday life and during leisure activities. The designers should always keep this in mind and consider its consequences in their specific design context. However, it is not only the designers responsible for this, but it also always depends on the users and on their tool use. Users may choose to use them in different, surprising or paradoxical ways (cf. [28]), or even choose not to use them at all. In our case, the users voluntarily subjected themselves to the continuous monitoring and control, but this surely is not the case in every design context.

5. Crowston K & Howison J. (2005): The social structure of free and open source software development. First Monday, 10(2). 6. Cox, A. 1998. Cathedrals, Bazaars and the Town Council. Available from: http://slashdot.org/features/98/10/13/1423253.shtml. Accessed 22 March 2004. 7. Deleuze, G. (1995): Negotiations 1972-1990. New York: Columbia University Press. 8. Doolin, B. (2005): Information Systems, Power, and Organizational Relations: A Case Study. Information Systems Journal, 14(4). Pp. 343-362.

Regarding the limitations of this study, only IRC meeting minutes were examined, but not actual conversations and other synchronous communication. Maybe critique towards strict control could be found there. Furthermore, OSS projects vary in every characteristic. As a single case study, this does not aim to be a comprehensive view of control in OSS projects. Therefore, different kinds of OSS projects should be examined and compared to this study. Finally, interviews made amongst the case project participants would have complemented this study tremendously. Interviews were planned and the intention was to interview the lead developers but this didn't come true.

9. Feller, J. & Fitzgerald, B. (2000): A framework analysis of the open source software development paradigm. In Proceedings of the 21st International Conference on information Systems (Brisbane, Queensland, Australia). AIS, Atlanta, GA. Pp. 58-69. 10.Fitzgerald, B & Ågerfalk, P. (2005): The Mysteries of Open Source Software: Black and White and Red All Over?, Hawaii International Conference on System Sciences, p. 196a, Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05) - Track 7, 2005.

Interesting paths for future work have been already identified in this paper. In addition, few new topics are 9 244

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

11.Fitzgerald, B. (2006): The Transformation of Open Source Software. MIS Quarterly, 30(3). Pp. 587-598.

24.Michlmayr, M. & Hill, B. M. (2003): Quality and the Reliance on Individuals in Free Software Projects. In Proceedings of the 3rd Workshop on pen Source Software Engineering. Pp. 105-109.

12.Foucault, M. (1977): Discipline and Punish: The Birth of the Prison. New York: Peregrine books.

25.Nichols, D. & Twidale, M. (2006): Usability Processes in Open Source Projects. Software Process Improvement and Practice 11. Pp. 149–162.

13.Gacek, C., Lawrie, T. & Arief, B. (2001): The many meanings of Open Source. Technical report. Centre for Software Reliability, Department of Computing Science, University of Newcastle, Newcastle upon Tyne, UK.

26.Orlikowski, W. J. (1991): The Duality of Technology: Rethinking the Concept of Technology in Organizations. Organization Science, 3(3).

14.Gallivan, M. (2001): Striking a balance between trust and control in a virtual organization: a content analysis of open source software case studies. Information Systems Journal. 11(4). Pp. 277-304.

27.Orlikowski, W. J. (1992): Learning from Notes: organizational issues in groupware implementation. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work (Toronto, Ontario, Canada, November 01 - 04, 1992). CSCW '92. ACM, New York, NY. Pp. 362-369.

15.Iivari, N., Karasti, H., Molin-Juustila, T., Salmela, S., Syrjänen, A.L. & Halkola, E. (2009): Mediation between design and use – revisiting five empirical studies. Human IT – Journal for Information Technology Studies as a Human Science, 10(2). Pp. 81126.

28.Orlikowski, W. J. (2000): Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations. Organization Science 11 (4). (Jul. 2000). Pp. 404-428.

16.Kirsch, L. J. (1997): Portfolios of Control Modes and IS Project Management. Information Systems Research. 1997. 8. Pp. 215-239.

29.Ouchi, W. G. (1979): A Conceptual Framework for the Design of Organizational Control Mechanisms. Management Science 25(9). Pp. 833-848.

17.Kirsch, L. J., Sambamurthy, V., Ko, D-G. & Purvis, R. L. (2002): Controlling Information Systems Development Projects: The View from the Client. Management Science. 2002. 48. Pp. 484-498.

30.Persson, A., Lings, B., Lundell, B. Mattsson, A & Ärlig, U. (2005): Communication, Coordination and Control in Distributed Development: an OSS Case Study. In Proceedings of the 1st International Conference on Open Source Systems (Scotto, M. and Succi, G, eds). Pp. 8892. Genoa, Italy.

18.Latteman, C. & Stieglitz, S. (2005): Framework for Governance in Open Source Communities. In Proceedings of the 35th Hawaii International Conference on System Sciences. 2005.

31.Raymond E. S. (1999): The Cathedral & the Bazaar. California: O'Reilly.

19.Ljungberg, J. (2000): Open source movements as a model for organizing. European Journal of Information Systems. 9(4). (Dec. 2000). Pp. 208-216.

32.Ye, Y. & Kishida, K. (2003): Toward an understanding of the motivation Open Source Software developers. In Proceedings of the 25th international Conference on Software Engineering (Portland, Oregon, May 03 - 10, 2003). International Conference on Software Engineering. IEEE Computer Society, Washington, DC. Pp. 419-429.

20.Lundell, B, Lings, B, Ågerfalk, P & Fitzgeard, B. (2008): The Distributed Open Source Software Development Model: Observations on Communication, Coordination and Control. In Proceedings of ECIS'2006, Gothenburg, Sweden. 21.Lyon, D. (2001): Surveillance Society. Monitoring everyday life. Buckingham: Open University Press.

33.Yin, R. K. (2003): Case Study Research Design and Methods. 3rd edition. California: Sage Publications.

22.Michlmayr M., Hunt F. & Probert D. (2005): Quality Practices and Problems in Free Software Projects, International Conference on Open Source Systems, Genova.

34.Zhao, L. & Deek, F. (2005): Improving Open Source Software Usability. In N. Romano (ed.), Proc. of 11th Americas Conference on Information Systems – A Conference on a Human Scale, 11.-14. August, 2005, Omaha, USA, pp. 923-928.

23.Michlmayr M. & Senyard A. (2006): A Statistical Analysis of Defects in Debian and Strategies for Improving Quality in Free Software Project. Bitzer, J. & Schröder, P. (ed): The Economics of Open Source Software Development, Published by Elsevier B. V.

35.Zuboff, S. (1988): In the Age of the Smart Machine: The Future of Work and Power. New York: Basic Books.

10 245

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Piles, Tabs and Overlaps in Navigation among Documents Mikkel Rønne Jakobsen and Kasper Hornbæk Department of Computer Science, University of Copenhagen Njalsgade 128, Building 24, 5th floor, DK-2300 Copenhagen, Denmark {mikkelrj, kash}@diku.dk ABSTRACT

Much research has aimed at supporting general window switching and switching between tasks or activities. General window switching techniques [26,29] help switching between applications, which are often visually distinct. However, switching between documents often takes place within one application and documents switched between may be visually similar. Also, while users’ frequent switching between tasks or activities needs support [7,18,27,28,30], often users must switch between multiple documents related to one task.

Navigation among documents is a frequent, but ill supported activity. Overlapping or tabbed documents are widespread, but they offer limited visibility of their content. We explore variations on navigation support: arranging documents with tabs, as overlapping windows, and in piles. In an experiment we compared 11 participants’ navigation with these variations and found strong task effects. Overall, overlapping windows were preferred and their structured layout worked well with some tasks. Surprisingly, tabbed documents were efficient in tasks requiring simply finding a document. Piled documents worked well for tasks that involved visual features of the documents, but the utility of recency or stable ordering of documents was task dependent. Based on the results, we discuss the effects of spatial arrangement, visibility, and task-dependency, and suggest areas for future research on document navigation and its support by piling.

Problems with existing switching mechanisms

One facility for supporting navigation among documents is overlapping windows, standard in most operating systems. Hutchings et al. [19] found that 78% of the time people had eight or more windows open, making locating a particular window within a group of overlapping windows difficult. Switching between windows using the taskbar is also hard because users may only see a short part of the windows’ titles and because the taskbar occasionally collapses documents into one tile [13].

Author Keywords

Document navigation, window switching, overlapping windows, piled windows.

Another facility, using tabs to represent multiple documents within a single window, is seen in many web browsers, spreadsheets, and programming environments. But tabbed interfaces can only show a limited number of documents at any time, necessitating further interaction with scrollbars or drop-down lists. When several documents are open, tabs provide minimal visual cues for finding a particular

ACM Classification Keywords

H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6): Graphical user interfaces (GUI). INTRODUCTION

Users frequently navigate between multiple documents, for instance to copy-paste text, compare web pages, or capture notes when reading. But navigation between documents is hard. Multiple windows or tabs are increasingly used in web browsing, for example, but users find it hard to manage multiple web pages [31]. In programming, source files are often viewed one at a time, requiring frequent navigation between files. Tabbed document interfaces that are common in widespread programming environments, such as Eclipse and Visual Studio, make navigating among many source files cumbersome [21]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

Figure 1: Interface with two piles of spatially arranged Figure 1: Interface with two piles of spatially arranged documents. documents.

246

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

document. Although recent studies show increased use of tabs in web browsing [15,31], attempts at improving or reinventing tabbed interfaces are needed [3]. BeaudouinLafon [6] suggested techniques for interacting with tabbed documents, such as leafing through tabs, but these techniques have to our knowledge not been widely adopted.

!"#"$"%"&'( !"#$%#&&'( #()&")%*+,+-.'( #))#*+,-(./)( !0#**%*+( ./01!2",)%"3,)'(

Still other facilities support navigation among multiple documents by a notion of piles. People pile and spatially arrange paper documents to see many of them at once, to structure their tasks, or to remind [22,25]. Although techniques for working with piles of electronic documents have been explored [4,5,6,23], questions about the design of interfaces that use piling remain unanswered. For example, how can a pile be spatially arranged so as to support navigation between its documents? Furthermore, empirical studies that investigate the usefulness of piling electronic documents are lacking.

9/014,*$!(

(

#))#*+,-(

4#*1#&&'( )/&-+)&"0)%%'(

$/(%*0),#!,(1!,)(0/*$)/&( $/(),-10,(1!,)(,../)$!(

5%,6,-(

(,.+)1,1&%'( $)#*!%,*$&'(

$/(")/5%-,(5%!1#&(01,!( $/(),-10,(1!,(/.(!"#0,(

.1&&7!%8,( ./)(0/*$,*$(%*$,)#0$%/*( ),"),!,*$,-( 6%$3(0/*$,*$( ./)(/5,)5%,6( #0)%,220)/"",-( ( #))#*+,-(%*(

3"4,2* /)-,)( )2)(&"!,*

Table 1: Taxonomy of document interface designs that use piling. The terms describing the interfaces used in the experiment are highlighted in bold.

This paper explores the design space of interfaces that support document piling and investigates possible benefits of spatially arranged piles (see Figure 1). In an experiment, we compare variations of interfaces for supporting navigation between documents, including two that arrange documents in a pile. The results contribute insights about the effect of visibility and spatial arrangement in different tasks involving navigation between documents. These findings may inform design of techniques for navigating between documents and indicate areas where further empirical research is needed.

hiding the title bar of windows underneath, as illustrated in Figure 2 (a). Spatial memory can be utilized to help people organize and find documents. Figure 2 (b) illustrates a spatial arrangement of documents in a pile that makes the spatial location of individual documents more distinct. Data Mountain, which allows users to arrange web page thumbnails on a perspective plane, showed faster and more accurate retrieval of pages compared with a text-based bookmark list [14].

SUPPORT FOR DOCUMENT NAVIGATION

Studies of office work have found that people benefit from piling as a way of spatially grouping documents [22,23,25]. Arranging and piling electronic documents may also help users navigate among documents. However, methods for spatially arranging documents and techniques for interacting effortlessly with them are needed.

Scanning through documents in a serial manner may be easier if the documents are aligned. Compared with the arrangement in Figure 2 (b), the alignment in Figure 2 (a) and (c) may provide for easier scanning of the documents. Agarawala et al. explore pile interaction techniques that use various aligned layouts for leafing through piles in a serial manner [4].

The use of electronic piles has been researched mainly as an alternative to filing documents by storing them as named files in hierarchical folders, and work on iconic document representations [4,5,23]. Instead, we focus on using piles for supporting navigation among documents that are in use. Based on a review of the literature, we have developed a taxonomy that incorporates important dimensions in the design of document piling interfaces (see Table 1). We relate each dimension to previous work and we explain the approach taken in the design of the piling interfaces used in the experiment presented later in the paper. Spatial arrangement

(a)

Documents can be spatially arranged to help users navigate among them. Four concerns of spatial arrangement in design of piling interfaces are visibility, spatial memory, scanning, and focus/periphery.

(b)

(c)

Figure 2: Spatial arrangements of four documents. (a) Pile of documents with titles and part of content visible, aligned diagonally. (b) Pile of documents with titles and part of content visible, arranged irregularly. (c) Document window with titles partly shown in tabs; tabs aligned horizontally.

Visibility of documents reminds users of the documents and allows them to interact with the documents. In overlapping window interfaces, windows open in a cascade to keep from

247

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Focal/peripheral areas seem to play a role in the spatial layout of documents [25] and how users manage space in large or multiple displays [8,16]. Robertson et al. divide the display into focal and peripheral regions in a window interface that supports task switching. Their Scalable Fabric uses animated transitions to shrink windows to thumbnails as they are moved into the peripheral region [27].

Transient views that can be called up temporarily can support navigation without permanently using display space. Widespread examples of transient views are the Exposé feature on Mac OS X [1] and the - window switching technique. Several techniques have been explored for browsing and leafing through piles that use transient views of the piled documents [4,5,23]. However, we are unaware of studies that compare how users navigate among piled documents with permanent and transient views.

In this paper, we arrange a pile of documents vertically so as to make the upper part of each document in the pile visible, including its title. To further improve visibility in piles with many documents, we reduce overlap by staggering the documents, which may also help users find documents by spatial location.

In this paper, we arrange documents in a pile to provide permanent visual cues for navigation so as to compare piles to permanent views techniques for navigation between documents that are common in widespread interfaces.

Manual vs. automatic layout

Document interfaces must provide the user with control of the layout to arrange the documents in ways that are appropriate for different tasks, with as little effort as possible.

Document representation

Full-size views of documents allow the user to interact directly with the documents’ content. In contrast, small document representations using thumbnail views, icons, or text make many documents visible in limited display space.

Manual layout in overlapping window interfaces allows users to pile windows manually—users have full control over moving, resizing, and overlapping windows. However, arranging windows and switching between overlapping windows require time-consuming window management [10]. Compared with overlapping windows, automatically tiled windows may increase performance, but users may prefer overlapping windows [9,20].

Windows can be reduced for instance by scaling the windows [11,27,28], or by scaling or cropping the content of windows [24]. Using a 3D document space, The Web Forager showed a document at a focus position in full size for direct interaction between user and content, whereas documents that are not the immediate focus are placed at greater z-distances (and thus smaller) [11]. Aiming to reduce screen clutter, Miah and Alty studied an adaptive window management system that reduced the size of windows [24]. They found that identification of a window became difficult when the window’s size was below 30% screen utilization. Their results also indicate marginally higher accuracy in identification tasks when a window’s content is cropped rather than scaled. Users may identify a document by a thumbnail view from among visually distinct documents (e.g., pages from different web sites). However, if documents are visually similar (e.g., source code files or pages from digital library), a thumbnail view contains no salient features for identifying the document.

Automatic layout of documents in piles may help users arrange documents and navigate among them with fewer efforts. Task Gallery provides an “ordered stack”, in which windows are automatically laid out with fixed distance between them, as well as a “loose stack”, which users lay out manually [28]. Clicking on a window in one of these stacks moves the window forward to a selected window position. The study focused on the system’s task management support and it is not clear how the stacks were used. Lightweight interactions have been explored for automatically gathering a selection of items into a pile on the desktop, for manipulating piles, and for arranging piles in different layouts with few user efforts [4]. However, it is not clear how these techniques would work for piles of document windows.

Semantic zooming or use of colors can make visually similar document thumbnails distinguishable. Other features of the document representation can help navigating among documents in piles. For instance, to help finding one among many equally-sized windows piled on top of each other, Beaudouin-Lafon suggests rotating windows [6].

In this paper, we explore piling interfaces that automatically maintain the layout so that all documents in a pile, and in particular the documents’ titles, are visible. When the user opens or closes a document, or drags a document from one pile to another, the piled documents rearrange to maintain a consistent layout of documents. We thus aim to reduce the user efforts required for rearranging and navigating among documents in piles compared with overlapping windows.

In this paper, we change the size of documents in a pile so that documents at the bottom of the pile are smaller. To reduce a document’s size, we scale the document’s content, but show its title in a fixed font size so that the title remains readable. The readable title is aimed at helping users in searching among visually similar documents.

Permanent views vs. transient views

Permanent views that support navigation among documents provide visual cues of the documents. For instance, in many desktop environments the task bar often shows windows as icons or text, including windows that are hidden.

Adaptive vs. fixed ordering

The order that documents are shown in the display may influence how users navigate between the documents.

248

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

In most desktop environments, windows overlap each other in the order they were last recently used—and windows are shown in this order when switching using -. Recent research adapts the order of windows used in window switching techniques based on semantic or temporal information about the windows [26,29]. These examples use adaptive ordering: the order of windows changes as the user switches between the windows. Alternatively, documents can have a fixed order. In this case, the user may order documents manually or documents may be ordered automatically by the system (e.g., by creation date). In this study we investigate different ways of ordering documents in a pile to see how that influences the way users navigate among the documents. We use a fixed ordering where documents are piled in the order they were opened and an adaptive ordering where documents are piled in the order they were most recently used. Summary

We have described the design space of interfaces for document piling based on five dimensions in the taxonomy shown in Table 1. Using this taxonomy, we have explained the approach taken in the interfaces used in the experiment. Earlier studies of document piling have mainly focused on piling document files in desktop environments, use of iconic representations, and use of transient view techniques to provide visibility of a pile’s documents. In contrast, we focus on piling documents in use, documents represented as zooming windows, and spatial arrangement to provide visibility of a pile’s documents. We are unaware of earlier studies of automatically arranging piles of documents that aim at providing permanent visual and spatial cues for navigating among the documents.

Figure 3: The Overlapping interface. The bar at the bottom of the display contains a tile for each document.

brings the document on top of other documents. Tiles appear from left to right in the bar in the order the documents were opened. All tiles have the same size and all fit in the bar, but only about 10 characters of a document’s title is shown. A tooltip with the entire title appears if the mouse cursor hovers over a tile. Rationale: This interface is included in the experiment because overlapping windows are standard in most systems and graphical user interfaces of many systems include a task bar. Studies have found that the task bar is often used for switching between windows [19,29]. Hutchings et al. found that 78% of the time people had eight or more windows open and consequently, users may experience problems with using the taskbar because only a short part of the windows’ titles are visible [19].

EXPERIMENT

To investigate the influence of spatial arrangement and visibility of documents on how users navigate among documents, we conducted a controlled experiment in which four interfaces for viewing multiple web pages were compared. We use web pages because the Web is a widely used source of information, for instance in sense-making of research literature using digital libraries [32], and because users often have many web pages open simultaneously [31].

The Tabbed interface

The Tabbed interface (shown in Figure 4) contains one window fixed in the center of the display. The window contains multiple documents represented by equally sized tabs under the title bar (see below). Tabs appear from left to right in the order the documents were opened. If there are more documents than tabs, which are minimum 120 pixels wide, a chevron appears to the right of the rightmost tab. Clicking the chevron opens a drop-down list of the titles of all the documents in the order they were added. Documents not visible in the tabs are at the bottom of the list on a gray background. This tab behavior resembles that of the Safari browser, and drop-down lists are common in tabbed document interfaces.

Interfaces

The interfaces consist of windows, where each window contains only the view of a web page, framed by a border and a title bar at the top. The active window is indicated by blue. Windows have a fixed size of 800 x 700 pixels. The Overlapping interface

In the Overlapping interface (shown in Figure 3), windows can be moved by dragging the title bar. Document windows open in a cascade to avoid hiding title bars of windows underneath. A horizontal bar (see below) at the bottom of the display contains a tile for each document (similar to the Taskbar in Windows XP or Vista [2]). Clicking on a tile

249

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The Piling Recent interface

The Piling Recent interface (shown in Figure 5) contains document windows piled in the order that they were last used, with the least recently used document at the bottom of the pile and the currently active document on top. As shown in Figure 6, documents are reduced in size and arranged vertically relative to their position in the pile so that the document at the bottom of the pile is smallest (at 30% size) and nearest the upper display border. Documents are reduced in size by zooming the view of the documents’ content. However, the document titles remain in a fixed, readable font size. When documents are opened, their windows are staggered so as to reduce overlap. Also, the staggering may help in remembering documents’ location. If the user clicks on a document in the pile, the document moves to the top of the pile and the other documents are rearranged, all with animated transitions. Rationale: Compared with Overlapping and Tabbed, the Piling Recent interface aims to provide better visibility of the documents, and titles of most documents are readable in full. The Piling Ordered interface Figure 4: The Tabbed interface. The document tabs below the window title bar contains the tabs for seven of the documents. Clicking the chevron near the right border opens a drop-down list showing all document titles.

The Piling Ordered interface (shown in Figure 7) is similar to Piling Recent, except document windows remain in the order they were opened. When clicked, a document is moved from its place to the top of the pile—the document returns to its place in the pile when another document is selected.

Rationale: This interface is included in the experiment because tabbed interfaces are common in widespread web browsers, spreadsheets, and programming environments. Common in many tabbed interfaces is that only a limited number of documents are visible as tabs and users either have to scroll in the tabs or otherwise get a view of the documents that are not shown.

Rationale: The spatial layout of document windows in Piling Ordered is more stable compared with Piling Recent, which can potentially help users in revisiting previously used documents.

Figure 6: The size and vertical location of a document at each position in a pile of five documents. At the top of the pile, the document is shown at full size.

Figure 5: The Piling Recent interface.

250

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

When document titles were not all shown in full, participants could not determine if the term occurred in a document title directly in the interface and titles were particularly truncated in the tiles in Overlapping and in tabs in Tabbed. In the Tabbed interface, however, participants could call up the list to read the titles in full. Navigate All tasks required participants to navigate to every document shown. An example task read: “Click the link to the first author of the document with the highest ‘Citation Count’” Each task used a different set of documents. We hypothesized that participants could traverse the documents quickly in the bar of Overlapping, and also in Tabbed in as far as the documents were visible in tabs. Compare References tasks asked participants to inspect the ‘References’ of three of the documents and determine the most referenced author. Each task of this type used a different set of ten documents. The three documents named in the task text had about 15 references each and at least one author referenced in all three documents.

Figure 7: The Piling Ordered interface. Participants

Eleven participants (eight male) between 25 and 51 years old (M = 34.6) were recruited by word of mouth.

We expected Piling Recent to be suitable for this task because participants could easily find the three documents close together at the top of the pile. We also were curious to see if participants arranged windows for comparison with the Overlapping interface.

Tasks

Ten web pages were opened in the interface before each task. Pages were selected from among 420 web pages from the ACM Digital Library and showed data on papers published 1994 to 2005. Documents in the experiment were thus all uniform in layout and appearance. Four types of task were used.

These four tasks were chosen to cover basic types of navigation among documents that occur in web browsing or programming. The tasks are similar to tasks used in earlier research [e.g., 20,29]. We are aware that not all user tasks that involve document navigation are among these four types of task. Also, tasks are limited in that they are taken out of context and involve already opened documents.

Navigate Title tasks required participants to do 25 tasks on finding a particular document, for instance “Click the link to the first author of the document: Pen computing for air traffic control”, where the document title varied between eight of the ten documents shown. We wanted to see how spatial layout of pages in the interfaces influenced performance in re-finding documents. Therefore, we used a Zipf-like distribution similar to the one used in Tak et al. [29] for selecting the target documents. The distribution was generated by randomly selecting eight targets. One target was then cued 10 times, one 5 times, then 3, 2, 2, 1, 1, and 1 for the others.

Materials

Participants used the 1440 x 900 pixel 15” display of a laptop computer. A mouse with scroll wheel was used for input. Tasks were presented in a view in the left side of the display. The interfaces used an area of 1040 x 878 pixels and document windows were 800 x 700 pixels. Design

The experiment employed a within-subjects design with the factors interface type (Overlapping, Tabbed, Piling Recent, Piling Ordered), and task type (Navigate Title, Navigate Term, Navigate All, Compare References). The order of interface type and task type was systematically varied and counter-balanced across participants. Participants completed 35 tasks with each interface: 25 Navigate Title tasks, four Navigate Term tasks, four Navigate All tasks, and two Compare References tasks.

We hypothesized that participants could find the cued document title quickly by scanning in the bar of Overlapping or the tabs in Tabbed. Navigate Term tasks required participants to find and compare documents with a given term in the title. An example task read: “Click the link to the first author of the document with ‘learning’ in the title that has the second most recent Year of Publication.” Each task of this type used a different set of documents, four of which contained the given term in the title.

Procedure

Participants were first given an introduction lasting about 15 minutes, which included instructions on how to use each interface and time for participants to practice tasks similar to the experimental tasks. Participants then completed tasks using each of the four interfaces. After completing the tasks

We hypothesized that participants could find the term in document titles faster and with less interaction using either Piling interface compared with Overlapping and Tabbed.

251

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

with each interface, participants received a questionnaire about satisfaction with the interface just used. The questionnaire contained nine questions from QUIS [12], two questions from NASA-TLX [17], and two additional questions asking how easy it was to find or re-find documents. Also, participants listed benefits and drawbacks of the interface. After completing all tasks, a final questionnaire asked participants to rank the four interfaces by order of preference. Last, participants were given opportunity to comment on the interfaces. The experiment lasted about an hour and 15 minutes for each participant. RESULTS Accuracy

We find no differences in the accuracy with which participants answer tasks, F(3, 8) = 0.25, ns. On average, participants answered 87% of the tasks correctly, ranging from 88% (with Overlapping) to 86% (with the two Piling variants). We had also expected no difference, as the tasks were relatively easy to solve with all interfaces.

Figure 8: Average task completion times (in seconds). Bars indicate the standard error of the mean.

reason, supported by our informal observations, is that participants sometimes lost track of the first document they viewed in sequentially traversing the documents. Hence, participants may have spent time looking at the same document twice.

Task completion time

We find an overall interaction between task and interface, F(9, 2) = 34.93, p < .05, and thus proceed to analyze the time differences on a per task basis (with statistics for significance omitted to save space, all differences are p < .05 with linear contrasts or post-hoc tests). Figure 8 shows the task completion times for each interface.

For the Compare-references task, we find that Piling Ordered (M = 131.9s) is significantly slower than the other three interfaces (Ms between 113.5s and 116.7s). One reason for this is that participants had trouble remembering the location of the three documents they had to compare references in. Compared to the other tasks involving revisiting documents, this task was cognitively more demanding. It might therefore have helped participants that the three documents could all be found in the order they were last used at the top of the pile in Piling Recent.

For the Navigate Title task, a post hoc comparison show no difference among interfaces, but to our surprise Tabbed was the fastest interface (M = 5.47s, SD = 2.84), followed by Overlapping (M = 5.66s, SD = 2.30), Piling Recent (M = 5.68s, SD = 2.71), and Piling Ordered (M = 6.21s, SD = 2.54). In particular, the Tabbed interface performed well even for documents that users had to access using the chevron and the drop down list.

Satisfaction and Preference!

A multivariate analysis of variance on the 13 satisfaction questions shows no overall differences in satisfaction between interfaces, Wilks’ != .274, F(3, 8) = 1.18, p > .3. Only for a question on whether tasks may be completed "with difficulty" or "with ease" do we find a significant difference, F(3, 40) = 3.20, p < .05, so that the tasks solved with the Overlapping interface are perceived to be easier than tasks with the other interfaces.

For the Navigate Term task, we find that the two Piling interfaces (Ordered: M = 53.3s, Recent: M = 53.1s) are significantly faster than the other interfaces (Overlapping: M = 60.6s; Tabbed: M = 63.8). This confirms our hypothesis that participants could find documents with the term faster using the Piling interfaces because titles of most documents are readable in full. Because the term is not always visible in truncated titles in Overlapping or Tabbed, participants may have spent more time switching between documents or calling up tool tips to look for the term.

In order of preference the interfaces are: Overlapping (M = 1.91, SD = 1.04), Piling Recent (M = 2.27, SD = 1.19), Tabbed (M = 2.82, SD = 1.08), and Piling Ordered (M = 3.00, SD = 1.00). However, an overall analysis of variance on the ranks suggest that these differences are only marginally significant, F(3, 40) = 2.361, p = .086.

For the Navigate All task, we find that Overlapping (M = 21.6s) is significantly faster than Tabbed (M = 35.1s). They are in turn significantly faster than Piling Ordered (M = 39.6s), which is significantly faster than Piling Recent (M = 47.3s). Looking at average task completion times, all participants were faster with Overlapping. This finding agrees with our expectation that participants could traverse the documents by clicking each tile in the bar. One reason participants were slower using Piling Recent compared with Piling Ordered is that the fixed document order in Piling Ordered helped participants revisit documents. Another

About Overlapping, six participants listed the taskbar as an advantage, three mentioning the arrangement of tiles in a row. Three participants liked that windows could be arranged for comparing their content. Two participants said it was hard to read or get overview of the titles. About Tabbed, participants described as advantages that it was “clear”, “simple”, and “familiar”. Also, three participants mentioned the alignment of tabs in a row. Six

252

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

participants expressed uncertainty about how the dropdown list in Tabbed worked, although two participants said it gave good overview of document titles. Overlapping Tabbed Piling Recent Piling Ordered

For both Piling interfaces, four participants mentioned overview of all documents as an advantage and three participants said the interfaces were good for finding a document. Five participants said about Piling Recent that it was confusing how documents were ordered, but three participants liked having recently used documents on top. Two participants said locations of documents were hard to remember in Piling Ordered, while two participants found it good for revisiting documents. Comments from three participants suggested problems in navigating through all documents using Piling Ordered.

Overlapping Tabbed Piling Recent Piling Ordered

10.6 8.3 7.2 5.7

11.4 12.3 13.9 11.9

5.7 5.0 5.2 5.9

75 109 111 105

32 53 192 121

123 84 99 94

For the Navigate All task, participants navigated in average to two more documents with Piling Recent than with Piling Ordered. This difference supports the explanation that participants sometimes lost track of the first document they viewed when using Piling Recent, resulting in slower performance. Looking at the mouse pointer travel distance, it seems Overlapping and Tabbed required less user efforts (32 and 53 in average distance) in navigating through all documents than the Piling interfaces (192 and 121 in average distance). This supports the explanation that the aligned tiles and tabs help scanning through the documents. For the Compare-references task, participants navigated to between 5 and 6 documents in average with all interfaces; which are surprisingly few switches between documents considering that participants had to compare about 15 references in each document. Piling Ordered was slower maybe because it was be difficult to remember the location of the three documents. Participants navigated only slightly more with Piling Ordered than with Piling Recent. It is unclear why the mouse pointer travel distance was larger with Overlapping compared with the other interfaces. It might be because some participants rearranged the documents to better compare them.

For the Navigate Term task, participants navigated 10.6 times in average using Overlapping and 8.3 times using Tabbed. In contrast, participants navigated 7.2 and 5.7 times in average using Piling Recent and Piling Ordered. This suggests that participants were faster with the Piling interfaces compared with Overlapping and Tabbed because they had to navigate to fewer documents to answer the task. Participants navigated to fewer documents with Tabbed than with Overlapping, which might be because some participants used the menu to find documents with the given term. It is not clear why participants navigated to more documents with Piling Recent compared with Piling Ordered. We wonder if the changing order of the documents made it difficult to revisit the correct document. The mouse pointer travelled relatively less between mouse events with Overlapping compared with the other Compare References

Compare References

interfaces, which indicates the benefit of traversing all the document tiles in the bar.

We analyzed data logged during the experiment to help uncover differences in how participants’ used the interfaces to complete the tasks. We summed the number of times that participants navigated to a document by clicking either in the document’s window, in a tile (only in Overlapping), or in a tab or pop-up menu (only in Tabbed). Table 2 shows the average number of such navigation actions for each interface. Also, we summed the distances that the mouse pointer traveled between mouse button events or tooltip call-ups (shown in Table 3). Distance was calculated as the diagonal between screen coordinates of the mouse pointer. Because the Navigate Title task involved only single-step navigation, we did not analyze the data for that task.

Navigate All

Navigate All

Table 3: Average distance that the mouse pointer traveled between mouse events for each interface (100=average distance for task across interfaces).

Interaction with the Interfaces

Navigate Term

Navigate Term

DISCUSSION

We emphasize three findings from the experiment, relate them to the literature, and elaborate on task effects. Visibility. The spatial arrangement in the Piling interfaces makes all documents visible and most titles readable in full. This might explain why participants completed Navigate Term tasks faster using the Piling interfaces. In contrast, document titles are truncated in Overlapping and Tabbed so participants had to inspect each tab or tile, either by using tooltips, by clicking the tab or tile to bring up the document, or by using the drop-down list in Tabbed. We note that the web pages used in the experiment were uniform in layout and appearance. However, it seems likely that the visibility in the Piling interfaces would improve navigation between visually distinct documents, for instance because users recognize content. Mander et al. studied piles of documents with varied content, but their piles were tightly arranged and used thumbnails [23].

Table 2: Average number of times that participants navigated to a document.

253

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Layout. The aligned layout of tiles and tabs in Overlapping and Tabbed allow for scanning document titles and for clicking through each document in sequence, with little mouse movement. Participants thus completed Navigate All tasks faster using Overlapping and Tabbed compared with the Piling interfaces, which seemed to require more efforts in navigating through all documents. In Piling Recent, participants could continually click the document nearest the upper edge of the display to bring it on top. The staggered placement required moving mouse horizontally, however, and to some participants it seemed unclear when all documents had been viewed. In Piling Ordered, participants had to move the mouse to each document in the pile to bring it on top.

In an experiment with 11 participants, we have compared four interfaces that support navigation among documents: one using overlapping windows, one using a tabbed window, and two that arranged documents in a pile. Strong tasks effects were found in task completion times. In tasks that used visual features of documents, participants performed significantly better using piles, likely because visibility of documents was better compared to overlapping and tabbed windows. In contrast, participants performed better using overlapping and tabbed windows in tasks that required participants to scan through all documents. The aligned layout of document titles in the task bar and the tabbed windows’ tabs provided for scanning titles in sequence without much effort. Overall, participants preferred overlapping windows, commenting the taskbar as an advantage. In conclusion, results suggest that automatic spatial arrangement of documents in piles provides visibility of documents useful in some tasks, but suggest that techniques for aligning documents could be useful in tasks that involve serial scanning.

We were surprised that the Tabbed interface performed well, despite the efforts required in navigating to documents not visible in the tabs. Still, only ten documents were used in tasks, and we wonder how tabbed interfaces scale to more documents. Also, participants were effective at navigating many windows in sequence using the taskbar. However, in real life the taskbar often contains different application windows and documents are sometimes collapsed into a single tile.

For future work, several questions about the use of piles in interfaces for supporting navigation remain to be addressed. First, to provide benefits similar to those enjoyed from arranging and piling documents on desks, more work is required that looks beyond navigation among documents in a single pile. We have begun exploring interaction techniques that allow users to effortlessly arrange documents side by side, in multiple piles (see Figure 1). Another issue concerns how users will arrange and pile documents in focal and peripheral regions, for instance by arranging and ordering documents in piles in the periphery to structure their tasks and piling documents in the focus that are frequently used. This requires empirical studies of people working on more complex tasks than the simple navigation tasks used in the present study.

Further, while techniques for leafing through many documents in tabbed windows [6] and in piles [4] promise to support looking sequentially through documents, we have yet to see empirically studies of their use. Our data suggest that this might be particularly useful for piling interfaces. Stability. Piling Ordered was slower in Compare References tasks, which suggests that participants had difficulties remembering where the documents were. In contrast, Piling Recent seems more useful in this task because the three recent documents all were near the top of the pile. Tak et al. reach a different conclusion in a study of window switching interfaces: they found that a spatially stable layout allows faster switching to windows often switched to than a recency layout [29]. The importance of stability and recency in the spatial layout clearly depends on the task performed.

Also, participants in this study only navigated among 10 documents, with limited screen size. We wonder how piling interfaces scale compared to widespread techniques in navigating among many documents and with large displays. Last, it remains unclear how transient techniques can use spatial arrangement and representation of documents to support navigation among documents. We will address this in future work and explore how transient techniques can be used for leafing through document piles.

Moreover, in this study participants navigated only between documents that were unknown to them. The usefulness of piles in real use situations, where users open and arrange documents themselves, remains to be further researched. In particular, we are curious if spatially stable piles of documents, which were inferior in this experiment, might be useful for structuring work with multiple documents.

1. 2.

CONCLUSION AND FUTURE WORK

3.

REFERENCES

Overlapping windows and tabbed document interfaces are widely used in desktop environments, but they provide limited support for navigating among documents. Aimed at providing such support, this paper has investigated the use of automatically arranging documents in piles. Based on five dimensions, we have explored the design space of interfaces for navigation among documents using piles.

4.

254

Exposé. Apple Computer, Inc.: http://www.apple.com. Windows Taskbar. Microsoft: http://www.microsoft.com. Reinventing Tabs In the Browser, Mozilla Labs Design Challenge 2009. http://designchallenge.mozillalabs.com/summer09/. Retrieved 2010 May 9. Agarawala, A. and Balakrishnan, R. (2006), Keepin' it real: pushing the desktop metaphor with physics, piles

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

and the pen, in Proceedings of CHI '06. ACM: New York, NY, USA. p. 1283-1292. Aliakseyeu, D., Subramanian, S., Lucero, A., and Gutwin, C. (2006), Interacting with piles of artifacts on digital tables, in Proceedings of AVI '06. ACM: New York, NY, USA. p. 159-162. Beaudouin-Lafon, M. (2001), Novel interaction techniques for overlapping windows, in Proceedings of UIST '01. ACM: New York, NY, USA. p. 153-154. Bernstein, M. S., Shrager, J., and Winograd, T. (2008), Taskposé: exploring fluid boundaries in an associative window visualization, in Proceedings of UIST '08. ACM: New York, NY, USA. p. 231-234. Bi, X. and Balakrishnan, R. (2009), Comparing usage of a large high-resolution display to single or dual desktop displays for daily work, in Proceedings of CHI '09. ACM: New York, NY, USA. p. 1005-1014. Bly, S. A. and Rosenberg, J. K. (1986), A comparison of tiled and overlapping windows. SIGCHI Bull., 1986. 17(4): p. 101-106. Bury, K. F. and Darnell, M. J. (1986), Window Management in Interactive Computer Systems. SIGCHI Bull., 1986. 18(2): p. 65-66. Card, S. K., Robertson, G. G., and York, W. (1996), The WebBook and the Web Forager: an information workspace for the World-Wide Web, in Proceedings of CHI '96. ACM: New York, NY, USA. p. 111-117. Chin, J. P., Virginia, A., and Norman, K. L. (1988), Development of an instrument measuring user satisfaction of the human-computer interface, in Proceedings of CHI '88. ACM Press. p. 213-218. Czerwinski, M., Smith, G., Regan, T., Meyers, B., Robertson, G., and Starkweather, G. (2003), Toward characterizing the productivity benefits of very large displays, in Proceedings of Interact 2003. p. 9-16. Czerwinski, M., van Dantzich, M., Robertson, G., and Hoffman, H. (1999), The Contribution of Thumbnail Image, Mouse-over Text and Spatial Location Memory to Web Page Retrieval in 3D, in Proceedings of Interact '99: Edinburgh, UK. Dubroy, P. and Balakrishnan, R. (2010), A study of tabbed browsing among mozilla firefox users, in Proceedings of CHI '10. ACM: New York, NY, USA. p. 673-682. Grudin, J. (2001), Partitioning digital worlds: focal and peripheral awareness in multiple monitor use, in Proceedings of CHI '01. ACM: New York, NY, USA. p. 458-465. Hart, S. G. and Staveland, L. E. (1988), Development of NASA-TLX: Results of empirical and theoretical research, in Human Mental Workload, P.A. Hancock and M. Meshkati, Editors. Elsevier: Amsterdam, The Netherlands. p. 139-183. Henderson, J., D. Austin and Card, S. (1986), Rooms: the use of multiple virtual workspaces to reduce space contention in a window-based graphical user interface. ACM Trans. Graph., 1986. 5(3): p. 211-243.

19. Hutchings, D. R., Smith, G., Meyers, B., Czerwinski, M., and Robertson, G. (2004), Display space usage and window management operation comparisons between single monitor and multiple monitor users, in Proceedings of AVI '04. ACM Press: New York, NY, USA. p. 32-39. 20. Kandogan, E. and Shneiderman, B. (1997), Elastic Windows: evaluation of multi-window operations, in Proceedings of CHI '97. ACM: New York, NY, USA. p. 250-257. 21. Ko, A. J., Aung, H., and Myers, B. A. (2005), Eliciting design requirements for maintenance-oriented IDEs: a detailed study of corrective and perfective maintenance tasks, in Proceedings of ICSE '05. p. 126-135. 22. Malone, T. W. (1983), How do people organize their desks?: Implications for the design of office information systems. ACM Trans. Inf. Syst., 1983. 1(1): p. 99-112. 23. Mander, R., Salomon, G., and Wong, Y. Y. (1992), A ``pile'' metaphor for supporting casual organization of information, in Proceedings of CHI '92. ACM: New York, NY, USA. p. 627-634. 24. Miah, T. and Alty, J. L. (2000), Vanishing Windows--a technique for adaptive window management. Interacting with Computers, 2000. 12(4): p. 337-355. 25. O'Hara, K. and Sellen, A. (1997), A comparison of reading paper and on-line documents, in Proceedings of CHI '97. ACM: New York, NY, USA. p. 335-342. 26. Oliver, N., Czerwinski, M., Smith, G., and Roomp, K. (2008), RelAltTab: assisting users in switching windows, in Proceedings of IUI '08. ACM: New York, NY, USA. p. 385-388. 27. Robertson, G., Horvitz, E., Czerwinski, M., Baudisch, P., Hutchings, D. R., Meyers, B., Robbins, D., and Smith, G. (2004), Scalable Fabric: flexible task management, in Proceedings of AVI '04. ACM: New York, NY, USA. p. 85-89. 28. Robertson, G., van Dantzich, M., Robbins, D., Czerwinski, M., Hinckley, K., Risden, K., Thiel, D., and Gorokhovsky, V. (2000), The Task Gallery: a 3D window manager, in Proceedings of CHI '00. ACM: New York, NY, USA. p. 494-501. 29. Tak, S., Cockburn, A., Humm, K., Ahlström, D., Gutwin, C., and Scarr, J. (2009), Improving Window Switching Interfaces, in Proceedings of INTERACT '09. p. 187-200. 30. Voida, S., Mynatt, E. D., and Edwards, W. K. (2008), Re-framing the desktop interface around the activities of knowledge work, in Proceedings of UIST '08. ACM: New York, NY, USA. p. 211-220. 31. Weinreich, H., Obendorf, H., Herder, E., and Mayer, M. (2008), Not quite the average: An empirical study of Web use. ACM Trans. Web, 2008. 2(1): p. 1-31. 32. Zhang, X., Qu, Y., Giles, C. L., and Song, P. (2008), CiteSense: supporting sensemaking of research literature, in Proceedings of CHI '08. ACM: New York, NY, USA. p. 677-680.

255

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Determining Usability Requirements into a Call-forTenders. A Case Study on the Development of a Healthcare System Timo Jokela Joticon Oy & University of Helsinki Finland timo.jokela@{joticon.fi, helsinki.fi} ABSTRACT

reported even in main newspapers. A room booking system of a city was not taken into use at all, due to its usability problems. Another public authority had a new website developed, where even the employees could not find basic information such as events or contact information.

Systems of public organizations, the development of which is acquired through public tendering, often suffer from poor usability. To resolve this issue, we explored how to determine usability requirements into a call-for-tenders. Our case is a usability-critical healthcare system to be developed for a city in Finland. We explored different options, and ended up with two measures: task completion success rate for defining effectiveness requirements, and a complementary measure that we call design solution success rate. We could not find appropriate ways for defining requirements that directly address efficiency and user satisfaction. Our case shows that the tendering context sets specific restrictions to the selection of usability measures, especially from the viewpoint of target setting and objective verification.

Why do these kinds of usability problems exist? It is reasonable to examine the specific development context: public tendering. A public authority needs issue a public call-for-tenders when acquiring the development of a new software system. The call-for-tenders includes the requirements for the system. The selected software development company is then committed to deliver what is required in the call-for-tenders.

Usability, user experience, measures, target levels, requirements, success rate, tendering, call-for-tenders

If usability is not among the requirements in the call-fortenders, it is understandable that software development companies do not consider usability when preparing their tenders. Vice versa, if usability is not required, inclusion of usability activities would lead to a more expensive tender and thereby make the tender less competitive.

ACM Classification Keywords

A Background Study

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

We studied public call-for-tenders for software systems in Finland over a four months period [23]. Many call-fortenders did not include any kinds of usability requirements. Those that did, typical requirements were kind of:

Author Keywords

INTRODUCTION

Systems in public organizations widely suffer from poor usability. A travel management system, used in many public organizations, is a revealing example of consequences the usability problems may lead to. Users report that making a single travel expense report may take up to three hours, and requires contacting user support every time. User failures and negative feedback were

• "Error situations should be avoided" • "The user's memory load should be minimal" • "The software should have a modern and user-friendly interface." The substance of these requirements represents usability. The problem is, however, that fulfillment of these kinds of requirements is not objectively verifiable. It should be agreeable between the customer and the software developer whether the requirements are fulfilled or not.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

For example, how to verify whether an implemented system meets the requirement "Error situations should be avoided", or the requirement "The user's memory load should be minimal?" These kinds of "requirements" are not true

256

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• We address a research problem – usability measures in the requirements context – that is not much studied. The focus of the research has been on measures in the evaluation context.

requirements; they can rather be regarded as wish lists that the development companies do not need to fulfill. An example of another type of requirement was: • "The software must be usability tested. A certificate of the test should be provided”.

• As we will show, the call-for-tenders context is a relevant aspect since it sets specific challenges to usability requirements. We found no empirical research on usability requirements in this context.

The fulfillment of this kind of requirement is verifiable (‘a certificate’). But in this case the validity is the problem. Validity means that the contents of the requirements should be right; i.e. the fulfillment of requirements truly means appropriate and good usability. A usability test – even multiple tests – does not guarantee good usability.

• The user base of systems of public organizations is remarkable; especially the growing number citizens who are becoming the users of e-governance applications. We find that the need for this research is most relevant from the pragmatic viewpoint, although in his remarkable study on usability measures, Hornbaek [11] does not include this topic among the research foci.

In one call-for-tenders, three user tasks were identified, with a requirement of the style: • "Log in. Success rate without any user guidance: 95%”. One problem here is the lack of coverage of the requirements. The requirements should cover widely enough the different areas of the system. It is not likely that there would only be three user tasks, and thereby the requirements are not comprehensive. (There also is a problem with the target setting; see discussion to follow).

LITERATURE REVIEW Usability Requirements Generally

Usability requirements at a general case are discussed rather widely in literature e.g. [8], [29], [26], [10], and [30]. The existing literature mainly focuses on describing and exploring the concepts and formats related to the definition of usability and the contents of usability requirements document, such as [15] and [13].

In all, our finding was that none of the call-for-tenders included usability requirements that had been verifiable, valid and/ or comprehensive.

Literature discussing target levels propose that one may include separate definitions for the target level (e.g. 90% of users can successfully complete a task) and the minimum acceptable level (e.g. 80% of users can successfully complete a task) [27]. Whiteside et al. [29] suggest that usability requirements be phrased at four levels: worst, planned, best and current.

Usability Measures vs. Usability Requirements

There exists substantial research on measuring usability. A large number of various usability and user experience measures can be found from the HCI literature [11]. Research has been carried out in understanding the correlations between the different usability measures [12] in an evaluation context. Lately, the research has expanded from usability to user experience measures [22].

There are not many empirical research reports on the use of quantitative usability requirements in practice. One of the few is one by Bevan et al. [5] who conducted case studies on quantitative usability evaluations following the Common Industry Format for usability testing, CIF [1] in a purchaser-supplier setting. Two of the case studies included the determination of quantitative usability requirements. In one case, five experts and four non-experts attempted to complete different sets of “typical tasks” in an existing system, and the goal of the new system was to “at least equal and if possible improve on these success rates”. Another case included a step “specification of the usability requirements”. It is reported that the cases were successful; however, the methodological aspects are not discussed in detail. - The author with his earlier colleagues report on measurable usability targets in product development [19].

Our research context represents another aspect of usability measures: rather than measuring usability in the evaluation context, our focus is how to determine measurable usability requirements. Defining measurable usability requirements induces a specific challenge that is not part of ‘only measuring usability’: one should be able to set target levels. For example, let’s say that a user satisfaction questionnaire SUS [6] is considered as a relevant usability measure. Measuring with SUS is quite straightforward. However, if one wants to use SUS in usability requirements, then one should determine the target level. Then we have a question: what should be the appropriate target level (e.g. 70, 80, or perhaps 90; SUS ratings are between 0 and 100)?

Usability Requirements in Software Contracting Research Problem and Relevance

Some authors have studied usability requirements specifically in the public software contracting context, e.g. [2], [24], [9], [3]. Their focus is on usability requirements after the selection of the software developer.

Our research problem is how to determine usability requirements for a call-for-tenders, so that they are verifiable, valid, and comprehensive. We find our research relevant for the following reasons:

There exists only limited research on the specific context of defining usability requirements in the call-for-tenders

257

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• The use of the system by the citizens will be voluntary.

context. Lauesen [21] is one of the few who specifically addresses the tendering context. He proposes an approach where critical user tasks and the related critical usability factors – such as efficiency, learnability and understandability - are identified, and metrics and target levels are chosen. He, however, does not go to details or report empirical experience.

• The citizens should not end up with erroneous results, whatever task they are to perform. The erroneous results would lead to waste of healthcare resources, or even to negative consequences in the health of citizens (e.g. recording measurement results erroneously). • The systems should be very easy to use by the healthcare personnel, too. They also have a lot of experience of systems of poor usability, and have a very critical attitude towards new systems.

A major example of measurable usability requirements is Voluntary Voting System Guidelines (VVSG) recommendations [16] where ‘overall performance metrics’ are defined. Five usability metrics are identified:

THE PROCESS FOR EXPLORING THE USABILITY REQUIREMENTS

• Total Completion Score: the proportion of users who successfully cast a ballot;

Overview

• Perfect Ballot Index: the ratio of the number of cast ballots containing no erroneous votes to the number of cast ballots containing one or more errors;

The original setting of the work was not a research study. The case study is a normal commercial development project, with the specific feature that usability was seen truly critical for the success of the system. The author’s task was to determine 'usability requirements’. Defining usability requirements in the context of public tendering, however, proved to be challenging and got a nature of research work.

• Voter Inclusion Index: a measure of both voting accuracy and consistency; • Average Voting Session Time: mean time taken per voter to complete the process of activating, filling out, and casting the ballot; and

The requirements team consisted of a requirements analyst, two healthcare professionals and a usability consultant (the author). Experts on tendering software systems reviewed the requirements at several points of the requirements process.

• Average Voter Confidence: mean confidence level expressed by the voters that the system successfully recorded their votes. VVSG, however, is not a call-for-tenders setting in the sense that a contract would be made with single development companies, prior it is to develop the system. Further, VVSG is setting where big resources are expected (e.g. tests with 100 users).

The main source of data arose from the reflective thinking of the of the author, the author’s discussions and feedback with the representatives of the project team, raising the issue in a discussion list of usability professionals, in personal discussions with other usability professionals, and also – by coincidence – in an ISO usability standards meeting where the topics very close to the research problem were discussed. Overall, the requirements were composed iteratively: there are approximately 10 iterative versions of the requirements.

Summary

In summary, there exist substantial earlier work related to usability requirements but only limited earlier work on usability requirements in the context of public tendering. As the basis of our work, we utilize the various measures described in the literature, and the approaches for determining target levels. The idea of Lauesen, as well as the example of VVSG, is based on defining usability measures, and setting target levels. This is also our approach that we took in our project.

Phases

The determination of usability requirements consisted of the four main phases as illustrated in Figure 1: definition of strategic usability targets; identification of user groups; definition of the context of use; and definition of operational usability targets.

RESEARCH CONTEXT: THE HEALTHCARE SYSTEM

Our case study is a requirements determination project for a health-care system where the customer is Oulu, a city with 140 000 habitants in Finland. Healthcare costs getting higher all the time. It is hoped that the new system would decrease costs, through providing a platform through which the citizens can do part of their healthcare activities in their homes through Internet.

Defining Strategic Usability Targets

Strategic usability targets are business-driven high-level goals that address the benefits of usability, without going to the details of user tasks, usability measures, etc. A number of key stakeholders, managers and initiators were interviewed, and the strategic requirements for the system were determined. The usability strategic targets were determined as:

The usability of the system should be good for several reasons: • The financial success of the system requires that the system will be widely used, by majority of the citizens.

• The users (citizens, healthcare personnel) should be able to use the system without training

258

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• All users (citizens of the target groups, healthcare personnel) should complete their tasks correctly, no matter which task they are to perform

• The user experience when carrying out routine tasks should be high as well, so that the users would use the system regularly (for example, citizens with chronic diseases have such routine tasks)

• The first trial of using the system should be such a positive experience, so that the user would continue using the system also in the future

Overall, usability was seen truly critical to the success of the system.

Figure 1. Usability requirements determination covers four steps (the reference: the JFunnel lifecycle model [17], [18])

identified dozens of user tasks, defined task goals to each of the tasks, and characterized the tasks based on whether they are ‘occasional’ or ‘routine’. An example of task definition is provided in Table 1.

Identification of User Groups

The basic categorization of users was obvious: the citizens and the healthcare personnel. • Citizens include the potential customers of the system (e.g. asthma or diabetes patients) who are accustomed to use basic Internet services, such as network banking and purchasing of tickets.

A specifically challenging job was to determine the task goals. The goals had to be defined precisely enough, so that they could be used as criteria for the correct completion of the tasks.

• The healthcare personnel consists of all those professionals (doctors, nurses, etc.) that need to use the system.

The determination work was based on: • The knowledge that the project team had on the applications and users

These two main groups, naturally, fall into sub categories. We, however, did not find necessary to define the sub categories at the requirements phase.

• Discussions and workshops with users, especially healthcare personnel • Interpretations, questions and comments imposed by the researcher

Definition of the Context of Use

Identification of user tasks, task goals and characteristics was a laborious task. The system contains several applications (appointment booking, weight control, recording of measures, etc.). For each application, we

Field studies – observations of users – were not carried out. First, we had resource limitations. Second, the healthcare personnel of the project team had a good knowledge on user

259

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• User satisfaction can be measured with a questionnaire, for example SUMI [20] , SUS [6] and QUIS [7]

tasks and task goals – although the knowledge was rather tacit. Most of the users tasks of the system are carried out even today, with exception that they are currently carried out by healthcare personnel only (when citizens visiting hospitals or health centers). Task name

Reserve a time in the laboratory

User

Citizen

Task goal

The following will be correctly defined: date and time; the correct laboratory unit. The user becomes informed: what is the latest time for canceling or changing the time; how he/she should be prepared

Task characteristics

We started to formulate the requirements based on the 9241-11 attributes of usability. The task proved not to be simple, and several iterations were required. THE RESULT: THE USABILITY MEASURES, MEASURING INSTRUMENTS AND TARGET LEVELS

A measurable usability requirement consists of three elements: • Definition of the usability measure. The attributes effectiveness, efficiency, and satisfaction are of too high level of abstraction. One needs to define the precise measure(s) for the attributes. In some sources, e.g. VVSG [16], the measure is part of the definition of the measuring instrument. We, however, find it clearer to define these separately.

To be carried out occasionally

• Description of the measuring instruments. One needs to define exactly how the measuring is to be carried out.

Table 1. An example of the definition of a user task, task goal and task characteristics

• Target level. The target level defines ‘how good the system is to be’; it is the ultimate requirement for the usability.

Determination of Operational Usability Targets

The outcome of this phase is the ultimate goal of the requirements work: the usability requirements. As said, the aim was that the requirements be verifiable, valid, and comprehensive.

In the following, we discuss the measures from each of the usability attributes – effectiveness, efficiency, and satisfaction – separately. We justify our selections, and also describe why some potential measures were abandoned.

When examining the strategic usability goals, we concluded that the definition of usability of ISO 9241-11 is applicable in our case: “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [13].

Measure for Effectiveness Definition of the Measure

One of our strategic usability objectives was that “all users should be able to carry out their tasks correctly”. A natural basic choice for the measure is Nielsen’s task success rate: “the percentage of tasks that users complete correctly” [25].

• Effectiveness: users should be able to correctly perform their tasks • Efficiency: it should be easy to do the routine tasks

We, however, modified the Nielsen’s measure in two steps. First, we modified the measure as to “the percentage of users that complete a task correctly”. We call this task completion success rate. The reason for this modification was that we wanted to identify exactly those tasks that users have problems with. Thereby we can report the software developer exactly those parts of the system where improvements are required.

• Satisfaction: the user experience should be of high level, because the use of the system is voluntary and one should want to use the system after their first experience We are aware that there exist many other potential attributes of ‘user experience’ [22]. We, however, found the ‘traditional’ usability attributes adequate, because they very well match with the strategic usability objectives. In addition, this system is not meant for competitive consumer market, and we reasoned that no specific ‘gimmicks’ are needed, just a highly usable system.

The other modification was about the ‘percentage’. It is not possible to verify a requirement that is formulated in a style “90% of users should complete a task correctly” (unless one tests with the whole population). When one tests with a sample population, one only gets a certain confidence on the minimum and maximum number of users that complete a task correctly.

The literature reports a bunch of various usability measures for each attribute: • An example of an effectiveness measure is task success rate: the percentage of tasks that a user can successfully complete [25].

We also considered the style “9 out of 10 test users should complete a task correctly”. The problem in this style is that the result depends on the set of test persons, and thereby the results do not necessarily represent the whole target population.

• Efficiency can be measured by the mean time needed to successfully complete a task.

260

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Thereby, our conclusion was to define the measure in a statistical form: 95% confidence that at least a predefined percentage of the target population completes a (specific) task correctly. We defined a task to be completed correctly when • The outcome of the task is correct

Number of test persons

Maximum number of unsuccessful attempts

10

0

20

1

26

2 Table 3. The maximum number of unsuccessful attempts 1

• The user believes that he/ she has accomplished the correct outcome

Measures for efficiency

In other cases, the task is not completed correctly.

Efficiency was found strategically relevant, especially because some users are expected to use the system routinely. We explored three alternatives for measuring efficiency

For comprehensive requirements, we had to define what is the ‘correct completion’ for each task. This led to a document of 70 pages that includes 20 – 50 definitions per application (an example is the definition: see Table 1).

• Task time • Adherence to design requirements

Measuring Instrument

• Task efficiency success rate

The measurement instrument should answer the question: How the task completion success rate is measured? The natural choice for the measuring instrument is usability testing.

Efficiency is typically measured through task time. We, however, did not find feasible to determine efficiency requirements using tasks time, because we did not have any measured benchmarks for task times. And we did not have resources to benchmark task times during the project.

A definition of the usability tests was produced, defining who are the test users, what are the tasks that the users are to do, what is the criterion for ‘task completion success’, etc. We defined, for example, that “the customer may select any set of user tasks, presuming that they represent natural order of work flow of the test person”. We provide the software developer the possibility to comment on selection and definition of the test tasks, prior the tests.

We also considered using design requirements, i.e. adherence to usability guidelines. We especially considered ISO 9241-110 [14] that includes appropriate guidelines, for example: “The steps required by the dialogue should be appropriate to the completion of the task, i.e. necessary steps should be included and unnecessary steps should be avoided.”

Target Level

We, however, decided to reject the measure because we could not understand how such requirements could be objectively verifiable. The verification would solely be based on expert reviews. We considered that this might lead to problems in agreeing on the results.

We set the target level as to: 95% confidence that at least 75% of target population completes the task correctly, as illustrated in Table 2. Users

Tasks

Target: the number of successful users

Citizens and healthcare personnel

Any task

95% confidence that at least a 75% of target population completes the task correctly

We further considered a subjective measure: the percentage of users that do not perceive the task unnecessary complex to do. Through this measure we hoped to addresses those design problems that lead to unnecessary user steps or actions. We, however, found difficult to measure this ‘perceived complexity’ in a valid way.

Table 2. The target level for task completion success rate

The percentage (75%) may seem quite modest - our strategic target was defined as to “all users should complete their tasks correctly”.

Our conclusion was we could not find an appropriate way for defining directly efficiency requirements.

We chose the target level because “at least 75% with the confidence of 95%” is statistically quite a high target anyway. For example, it we test with 10 users, all of them should be successful. See Table 3.

Satisfaction is relevant generally, and especially in this case because the strategic target was that users’ experience should be positive. The requirement was that after a first trial, a user should want to use the system again.

Measures for satisfaction

For measuring satisfaction, we considered using SUS, based on recent promising findings [28] [4]. We, however,

1

261

See http://www.measuringusability.com/comp_sample.php

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

abandoned SUS because we were not able to find any basis for defining the target level (i.e. should the target level be 70, 80, or perhaps 90?).

Overall, we found important to have a way capture other but the effectiveness aspect (task completion success rate); and design solution success rate is a complementary aspect.

Norms exist for the SUMI questionnaire [20]. Thinking afterwards, we could have considered more seriously the use of SUMI. Still setting the target level may have been a problem. In addition, SUMI is a ‘heavy weight’ commercial questionnaire, and its costs may have been a hinder.

We believe that the measure is objectively verifiable, although we see the risk that agreeing on a user problem may be more difficult that verifying the outcome of a task. Measuring Instrument

The measurement is usability testing. The tests can be the same ones as those for measuring the task success rate (see discussions above).

We also considered ISO 9241-110 guidelines. Our rationale was that designing the user interface following the guidelines (e.g. “If typical input values are required for a task, these values should be available to the user automatically as defaults.”) would probably have a positive impact on user satisfaction. We rejected the measure because of the same reasons as with efficiency: we considered that this might lead to problems in agreeing on the results.

Target Level

We set the target level: 95% confidence that at least a 50% of the target population do not have problems with a specific design solution, as illustrated in Table 2. We set the target level lower than the effectiveness target because we find that effectiveness is the key attribute.

Our conclusion was we could not find an appropriate way for defining directly satisfaction requirements. A mixed measure

We anyway found it important to have other usability requirements determined, in addition to effectiveness. We wanted to capture efficiency and satisfaction in some way. Our solution was a metric that we call design solution success rate.

Users

Tasks

Target: the number of successful users

Citizens and healthcare personnel

Any task

95% confidence that at least a 50% of the target population do not have problems with a design solution

Table 4. The target level for design solution success rate Definition of the Measure

We define design solution success rate as 95% confidence that at least a predefined percentage of target population do not have problems with a specific design solution.

Table 5 illustrates that if we test with 10 users, two users are allowed to have problems with a specific design solution.

A ‘design solution’ may represent different areas of design: terminology, presentation of information, error handling and recovery, dialogues, tables and forms, wizards, the steps required to accomplish a user task, etc.

Number of test persons

Maximum number of user problems

10

2

16

4

21

6

We defined that a user has a problem with a design solution when

Table 5. The maximum number of user problems

• When performing a task, a user gets into a trouble due to a specific design solution: makes a longer stop, is uncertain how to continue, makes errors, gets frustrated, etc. A user is not regarded to have a problem if an error was just a slip, or the problem is a minor one.

Summary

In summary, we ended up with two measures: task completion success rate and design solution success rate. We could not find a way to define direct measures for efficiency and user satisfaction (Figure 2).

• A post-task interview confirms that the user problem was due to the specific design solution. We are aware that user errors are typically considered as measures of effectiveness [13]. But our hypothesis is that design solution success rate contributes for the evaluation of efficiency and satisfaction: • Time spent recovering from problems would impact efficiency. • Frustration with problems with design solutions would lead to dissatisfaction. Frustration with unnecessary complex steps would also reveal efficiency problems.

262

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• Validity. We deduced the usability requirements from the business needs of the system. We justified the selection of the measures and target levels. We believe that we reached at least reasonable valid requirements; but we advice the reader to critically go through our justifications. • Coverage. The coverage is based on the systematic determination of user tasks (with outcomes). We tried to capture all the relevant tasks; the document is approximately 70 pages.

Figure 2. The usability measures of the case project CURRENT STATUS OF THE PROJECT

The usability requirements were defined as obligatory requirements in the call-for-tenders. In other words, the development company is obliged to develop a system that meets the usability requirements.

When we contrast our results with work of others, we can find that • Unlike what is proposed in literature (e.g. [27]. [29]), we found not applicable to determine ‘acceptable’ and ‘minimum acceptable’, or ‘worst, planned, best and current’ levels of requirement. At least in our case our conclusion was that one needs only one target, and that is the minimum acceptable level.

The city received several tenders. We found this as a good result: the tough usability requirements did not scare the software suppliers from making tenders. At the time of writing this paper, the supplier is selected, and the implementation of the system is about to start.

• In our context, we found that there was no reason to categorize tasks into more or less critical ones. We just could not identify any task where user failure would be acceptable. Also this made the determination of requirements easier; we did not need to identify less and more important tasks.

DISCUSSION

We determined usability requirements – measures, measuring instruments and target levels - into a call-fortenders for the development of a usability-critical healthcare system. We explored different options, and ended up with two measures:

• When contrasting our measures with ones of VVSG, some differences can be found. For example, in contrast with our Task Completion Success Rate, VVSG has a measure Total Completion Score (TCS) that is the proportion of users who successfully cast a ballot, whether or not the ballot contains erroneous votes. In other words, TSC includes incorrectly completed tasks, and another measure for measuring the proportion of successful votes. In our context, we found that it was adequate to use one measure only for effectiveness.

(1) Task completion success rate: 95% confidence that at least a predefined percentage of the target population completes a (specific) task correctly (2) Design solution success rate: 95% confidence that at least a predefined percentage of target population do not have problems with a specific design solution call We could not find appropriate ways for defining requirements directly for efficiency and user satisfaction. Our study shows how the tendering context set specific restrictions to the selection of usability measures. The measures should be driven by the strategic usability goals of the system. One should select measures where one has a clear base, on which to define the target levels. The measuring instrument should objective, fair to the developer, and practical to use without extensive resources. We have presented reasoning on choosing the measures and setting the target levels, as proposed in [11].

The starting point of our research was solely practical. Still, we can find that we addressed some of the challenges posed in Hornbaek’s study [11]: • We interpret that the task completion success rate is about ‘macro tasks’, and the design solution success rate is about ‘micro tasks’. Thereby, we make the distinction proposed in [11], and even focus on the ‘macro tasks’, as suggested. • The design solution success rate can be regarded as one approach to extend measures of satisfaction beyond postuse questionnaires, although one needs to consider that it is not a direct user satisfaction measure.

We aimed for usability requirements that are verifiable, valid, and comprehensive. To which extent we were successful then? • Verifiability. Task success rate should be rather verifiable; probably the biggest risk is in the definitions of outcomes of the user tasks (there are so many user tasks). A risk may be in agreeing the problems when testing the design solution success rate. Naturally, we know more about this after the tests have been carried out.

Limitations

While we believe that the effectiveness measure - task completion success rate - is valid (a slight modification of an existing measure), we understand the potential problems of the design solution success measure. We do not have any evidence about how it correlates to user satisfaction. The idea of measuring design problems is very close to one of measuring user errors, which is often used as a measure for

263

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

effectiveness. Further, although we expect the measure be objective, we admit that there are risks, too.

usability’ with questionnaires? Or should there be developed other kinds of subjective measures for satisfaction for defining requirements?

At this stage, we do not yet have empirical results on the impact that the usability requirements have on the development project and the usability of the final system.

There is also need for research for how to deduce the appropriate measures and target levels from the business context of the system. We explain how we came up with the specific measures and target levels in the case of our healthcare system. But we find that there exists only limited guidance on how to do this.

Implications to Practice

We believe that this work is useful for usability practitioners who are involved in preparing public call-fortenders for software systems. This article provides lessons learnt, and also examples for how to determine usability requirements.

CONCLUSION

We determined usability requirements into a call-fortenders for the development of a usability-critical healthcare system for a European city. We determined effectiveness requirements using a variation of an existing measure. Direct measures for efficiency and satisfaction requirements could not be determined; we used an indirect measure that we call design solution success rate. Our experience shows that the tendering context sets specific considerations to the applicable types of usability measures.

The paper also shows that determining appropriate usability measures and setting target levels is a challenging task. One should aim to usability requirements that are verifiable, valid and comprehensive. Defining usability requirement requires quite remarkable resources and usability expertise. We believe that this work is interesting also for a larger audience of usability practitioners. The author with earlier colleagues reported a positive experience of setting measurable usability requirements in product development [19]. We proposed in that “it is not necessary to know how to measure exactly at the time of determining the requirements”. The public tendering context, however, is more demanding. Now we find that our earlier conclusion is not true: for objective verification, one absolutely needs to define the measuring instruments in the requirement phase.

ACKNOWLEDGEMENTS

I wish to thank the team and management of the ‘Omahoito’ team for co-operation and an interesting case: Riikka Hirvasniemi, Keijo Koski, Jarkko Polvi, Maarit Salmi and Seppo Voutilainen. I also wish to thank Jeff Sauro for the hint for the statistical style of defining usability measures. Further I thank the CloudWS program of Tivit and Tekes, which supported especially the theoretical part and writing of this work. Finally, I thank the anonymous reviewers for the valuable comments.

Topics for Future Research

Only few case studies are published on measurable usability requirements. This article is one of them, in the specific context of public tendering. More empirical studies are welcome, also in the wider context of defining measurable usability requirements.

REFERENCES

1.

Natural follow-up issues are to examine is to see how the development project progresses, and what will be the usability of the final system. One specific interest is to analyze how the design solution success rate measure works in practice: how easy it is to identify and agree on the user problems at the level of individual design solutions.

2.

3. 4.

Another viewpoint of measurable usability requirements is the impact on software development companies. They should be able to deliver ‘measured usability’ – not only e.g. number of usability tests or user studies. This probably is a new challenge. But as we see, verifiable, valid and comprehensive usability requirements this is the only way to assure good usability from the purchaser’s point of view.

5.

6.

There is also need for more theoretical work. We totally agree with the need for research to “validate and standardize the host of subjective satisfaction questionnaires”, proposed by Hornbaek [11]. But in addition to this, we find that research is needed also in how to set target levels using the satisfaction questionnaires. Is it feasible at all to determine target level levels for ‘good

7.

8.

264

ANSI, Common Industry Format for Usability Test Reports. 2001: NCITS 354-2001. Artman, H. Procurer Usability Requirements: Negotiations in Contract Development. in NordiCHI 2002. 2002. Århus. Artman, H. and S. Zällh, Finding a way to usability: procurement of a taxi dispatch system. 2005. Bangor, A., P. Kortum, and J. Miller, Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies, 2009. 4(3): p. 114-123. Bevan, N., et al., Guide to specifying and evaluating usability as part of a contract, version1.0. PRUE project. 2002, Serco Usability Services: London. p. 47. Brooke, J., SUS - A "quick and dirty" usability scale. 1986, Digital Equipment Co. Ltd. Chin, J.P., V.A. Diehl, and K.L. Norman. Development of an instrument measuring user satisfaction of the human-computer interface. in Proceedings of SIGCHI '88. 1988. New York. Good, M., et al. User-derived impact analysis as a tool for usability engineering. in Conference Proceedings

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

9.

10.

11.

12.

13.

14.

15. 16.

17.

18. 19.

on Human Factors in Computing Systems. 1986. Boston. Gulliksen, J., et al., Key Principles of User-Centred Systems Design. Behaviour & Information Technology, 2003. 22(6): p. 397-409. Hix, D. and H.R. Hartson, Developing User Interfaces: Ensuring Usability Through Product & Process. 1993, New York: John Wiley & Sons. 416. Hornbaek, K., Current practice in measuring usability: Challenges to usability studies and research. Int. J. Human-Computer Studies, 2006. 64. Hornbaek, K. and E.L.-C. Law. Meta-Analysis of Correlations Among Usability Measures. in CHI 2007 Proceedings. 2007. San Jose. ISO/IEC, 9241-11 Ergonomic requirements for office work with visual display terminals (VDT)s - Part 11 Guidance on usability. 1998: ISO/IEC 9241-11: 1998 (E). ISO/IEC, 9241-110 Ergonomics of human-system interaction -- Part 110: Dialogue principles. 2006: ISO/IEC 9241-110: 2006 (E). ISO/IEC, 25062 Common Industry Format (CIF) for usability test reports. 2006: ISO/IEC 13407: 2006 (E). Jeffrey, W. and &al, Voluntary Voting System Guidelines Recommendations to the Election Assistance Commission. 2007, Technical Guidelines Development Committee, United States Election Assistance Commission. Jokela, T. A Two-Level Approach for Determining Measurable Usability Targets. in 5th COST294MAUSE Open Workshop on Valid Useful User Experience Measurement (VUUM). 2008. Reykjavik, Island. Jokela, T., Usability Guided Interaction Design (in Finnish). 2010: Väylä-Yhtiöt Oy. Jokela, T., et al., Quantitative Usability Requirements in the Development of the User Interface of a Mobile Phone. A Case Study. Personal and Ubiquitous Computing, 2006. 10(6): p. 345-355.

20. Kirakowski, J. and M. Corbett, SUMI: The software usability measurement inventory. British Journal of Educational Technology, 1993. 24(3): p. 210-212. 21. Lauesen, S. Usability requirements in a tender process. in OZCHI’98. 1998. Adelaide. 22. Law, E., et al. Towards a UX Manifesto. 2007. Lancaster, UK: http://www.cost294.org. 23. Lehtonen, T., et al., To Which Extent Usability Truly Matters? A Study on Usability Requirements in Callfor-Tenders of Software Systems, Issued by Public Authorities. Manuscript (in review for publication). 2010. 24. Markensten, E. and H. Artman. Procuring a Usable System Using Unemployed Personas in NordiCHI 2004. 2004. Tampere. 25. Nielsen, J., Success Rate: The Simplest Usability Metric. http://www.useit.com/alertbox/20010218.html. 2001. 26. Nielsen, J., Usability Engineering. 1993, San Diego: Academic Press, Inc. 358. 27. NIST, Proposed Industry Format for Usability Requirements. Draft version 0.62. 8-Aug-04. 2004. 28. Sauro, J. and J.S. Dumas. Comparison of Three OneQuestion, Post-Task Usability Questionnaires. in CHI 2009. 2009. Boston. 29. Whiteside, J., J. Bennett, and K. Holtzblatt, Usability Engineering: Our Experience and Evolution, in Handbook of human-computer interaction, M. Helander, Editor. 1988, North-Holland: Amsterdam. p. 791-817. 30. Wixon, D. and C. Wilson, The Usability Engineering Framework for Product Design and Evaluation, in Handbook of Human-Computer Interaction, M. Helander, T. Landauer, and P. Prabhu, Editors. 1997, Elsevier Science B.V: Amsterdam. p. 653-688.

265

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Descriptive Quality of Experience for Mobile 3D Video Satu Jumisko-Pyykkö(1), Dominik Strohmeier(2), (1)Tampere University of Technology P.O.Box 589, 33101 Tampere, Finland satu.jumisko-pyykko, timo.j.utriainen{@tut.fi}

Timo Utriainen(1), Kristina Kunze(2) (2)Ilmenau University of Technology P.O.Box 100565, 98684 Ilmenau, Germany dominik.strohmeier, kristina.kunze{@tuilmenau.de}

ABSTRACT

context of use. To understand this construct of quality components, subjective quality evaluation experiments with prospective users are conducted.

Perceptual quality evaluation experiments are used to assess the excellence of multimedia quality. However, these studies disregard qualitative experiential descriptions, interpretations, and impressions of quality. The goal of this paper is to identify general descriptive characteristics of experienced quality of 3D video on mobile devices. We conducted five studies in which descriptive data was collected after the psychoperceptual quality evaluation experiment. Qualitative semi-structured interviews and written attribute description tasks were conducted with over 90 naïve participants. The experiments contained an extensive and heterogeneous set of produced quality by varying content, level of depth, compression and transmission parameters, and audio and display factors for 3D. The results showed that quality of experience is constructed from four main components, 1) visual quality, 2) viewing experience, 3) content, and 4) quality of other modalities and their interactions.

The term ‘quality’ has two dimensions: ‘the degree of excellence of something’ and ‘a distinctive attribute or characteristic possessed by -- something’ [24]. The excellence of multimedia quality is conventionally explored using quantitative quality evaluation experiments according to standardized procedures (e.g. by ITU [12,13]). Within these approaches less attention has been paid to understand the descriptive experiential dimensions of multimedia quality. According to previous descriptive studies, multimedia quality of experience is a more complex phenomenon than just a collection of impairment characteristics [16]. The active nature of human perceptual processes, in which material-driven characteristics as well as high-level knowledge, emotions, and attitudes are represented, is a necessary part of it [6,8,16,21,25]. For example, experienced audiovisual quality of mobile video is characterized by the impressions of audio, video, audiovisual, content, and usage factors [16]. In overall, current descriptive studies are rare. They target on 2D systems, explore mainly one medium at the time, or use settings irrelevant for 3D on mobile devices, and finally conclude quality of experience under the constraints of the study-dependent characteristics [1,10,16,22,23,27,28,32].

Author Keywords

Quality perception, quality of experience, multimedia, 3D ACM CLASSIFICATION KEYWORDS

H.5.1 Multimedia Evaluation/methodology

Information

Systems:

INTRODUCTION

Goal of this paper is to identify general descriptive characteristics of experienced quality of 3D video on mobile device. We conduct five qualitative studies where data-collection is based on interviews or written attribute descriptions of naive participants, and collected after a psychoperceptual quality evaluation experiment. The studies contained a broad set of heterogeneous, produced quality factors by varying content, level of depth, compression and transmission parameters, audio and display factors for 3D. We summarize the results as a model of descriptive quality of experience for 3D mobile video. Our work benefits academia to build up deeper understanding on the nature of perceptual quality and practitioners to develop descriptive (vocabulary-based) evaluation tools and apply the gained knowledge in the system development.

3D is expected to emerge from cinema to personal consumer electronics. For 3D on mobile devices, interest has been expressed towards video, television, and gaming [19]. In these applications, 3D can provide a greater immersion experience for users if this added value is not restrained by erroneous image quality [32]. The overall excellence of 3D is influenced by content production, required high-level of compression, requirements set by transmission channel, and finally, the quality of the display. However, the final quality is determined by user’s perception, influenced by his/her characteristics and the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

266

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

QUALITY OF EXPERIENCE – 3D VIDEO ON MOBILE DEVICE AND ITS EVALUATION

pattern of low- and high-level factors can be identified in other related descriptive studies.

Quality of Experience (QoE) describes the relationship between perceived and produced quality and is also understood as a “multidimensional construct of user perceptions and behaviors [35]”. Produced quality represents provided quality of a certain system and is limited by its constraints. In mobile 3DTV systems, produced quality is characterized by various factors along the production chain on the levels of content, media and network [7]. Impairments like blocking artifacts or frame loss occur, propagate jointly over the chain and can influence the user’s experienced quality [2,7,32].

The majority of descriptions of quality are focused on lowlevel factors containing the descriptions of apparent features of uni- and multimodal material. Low-level factors in descriptive evaluation of audio and video often offer a description of either the excellence of stimuli or, with the use of antonymous expressions, their perceived impairments. In image and video quality research, low-level factors relate to the sharpness of the stimulus, its colors or brightness [1,10,27,28,32]. This is also valid for 3D videos (blurry, artifacts [33], brilliant [32]). In addition, low-level factors in 3D research often characterize an impairment of video quality through the display technology and crosstalk [32,33]. In audio research, assessors’ low-level descriptions lists timbral aspects (hissing, metallic [22]; “timbral balance between low-frequency and high-frequency” [22]) and distortions of the sound [22,23]. The combined audiovisual low-level factors emphasize motion, its errors and the synchronization between audio and video, and the balance between the qualities of the two media [16,26].

Perceived (or experienced) quality describes quality from the user’s point of view. It is characterized by the user’s active perceptual low- and high-level processing. Low-level processing extracts relevant features from the perceived sensory information. In vision, color, brightness, contrast, motion, and stereoscopic information are processed in the early stage of the perceptual process [8]. In audio, pitch, loudness, timbre or runtime differences for localization are relevant features [21]. High-level cognitive processing refers to the interpretation of stimuli through their personal meaning and relevance to human goal-oriented actions. This process is influenced by our knowledge, expectations, and emotions [6,25]. It affects the importance of each sensory attribute and more broadly enables human contextual behavior and active quality interpretation [14, 16,25-27]

Going beyond the apparent features of stimuli, the highlevel factors describe content, usage factors, and in 3DTV research, the added value of 3D. While only few were identified in descriptive audio quality research [22,23], high-level factors emphasized in visual quality evaluation, especially related to 3D videos. One of the key factors is object recognition [1,32]. In addition, the look of the stimuli is interpreted and assessors describe content as “professional” [27] or “amateurish” [27] or classify stimuli as “artistic” [27] or “life-like” [10]. A special aspect related to 3DTV is the added value which refers to an enhanced experience of 3D video in comparison to existing systems and an impression of increased reality or “emotional experience” [10] of the content [5,10,32].

To assess the impact of quality factors on perceived quality, different methods are applied in audiovisual and 3D-related research. Standardized psychoperceptual evaluation methods and their extensions study perceived quality quantitatively [9,12,13,18]. They are useful tools to examine a predefined set of technical quality parameters. However, their ability to discover new quality factors and quality structures is limited. To overcome this shortcoming, descriptive methods have been developed applying either interview-based approaches [16,27] or sensory profiling methods [29,32]. These descriptive methods extend quality evaluation with a hypothesis-free quality factor elicitation.

Taken together, currently existing descriptors construct audiovisual (3D) Quality of Experience from components of low-level and high-level factors as proposed by JumiskoPyykkö et al. [16]. Especially low-level factors show strong study-dependent results highlighting specific excellence of the system under evaluation. The high-level factors of 3D video were identified as content, usage factors, and the added value of 3D. Although the components of visual and audio descriptions can also be found in audiovisual descriptions, the results of multimodal descriptions show that audiovisual quality is more than just their combination. Following, we present our elicitation of experienced quality factors for audiovisual 3D quality perception.

DESCRIPTIVE QUALITY OF EXPERIENCE: FROM CHARACTERISTICS OF STIMULI TO BEYOND IT

In descriptive assessment methods for multimedia quality, assessors describe the factors that form their individual (multidimensional) QoE rationale. In contrast to quantitative approaches [12,13], quality is thereby expressed qualitatively in terms of open questions or sensory attributes and their definitions [16,27,32]. In general, Jumisko-Pyykkö et al. [16] showed that people derive two different kinds of quality descriptions. “Lowlevel factors” are directly derived from the characteristics of the presented stimuli, whereas “high-level factors” represent attributes that take into account users’ goals of using the system or their knowledge about the system. This

RESEARCH METHOD

The experienced quality descriptions were collected in five studies (following exp1 to exp5) into different research challenges for mobile 3D television and video. The experiment characteristics are summarized in Table 1.

267

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Goal – We investigated audiovisual experienced quality under monoscopic and stereoscopic video presentation (exp1, exp4, exp5), studied the influence of video coding methods, coding settings, and transmission parameters on experienced quality of mobile 3D television (exp2 to exp5). All experiments were conducted in controlled laboratory environments. In addition, exp4 was conducted in indoor quasi-experimental (home-like) settings and exp5 in two different real-life-like settings.

positive/negative?” The interviewer used only the terms introduced by the participant. In exp 4, the interview was complemented with a stimuli-assisted description task in which a set of stimuli was shown one by one in randomized order and the test participant was shortly interviewed after each presented video. Stimuli - Heterogeneous stimuli material that represented the characteristics of potential contents for mobile 3D TV [19] were used in the experiments. The stimuli contained synthetic and natural video scenes, varying depth levels, and differing amounts of details (spatial) and movement (temporal) [Table 1].

Participants – Participants were aged 18-45 years and both genders were equally represented. The majority of them represented naïve participants (not studying, working or otherwise engaged in fields relating to video processing or television). The participation to only one of the experiments was accepted.

The variables of the experiments were selected to have impact on spatial, temporal, and depth quality of the test items. To study a quality comparison between 2D and 3D video, the presentation mode of the stereoscopic display was varied. Different coding methods, quality levels, bit rates, frame rates, and transmission error rates were varied for mobile 3D television and video. The related audio varied presentation mode (mono, stereo) and bit rates. All test stimuli were encoded using the recommended codecs: mp4v, H.264/AVC Reference Software JM 14.2 and H.264/MVC reference Software JMVC 5.0.5 and H.264/AVC (x264 b1077) with baseline profile for video. Audio was encoded with WMA9 and AAC-HEv2.

Procedure - All studies consisted of two parts: psychoperceptual and descriptive quality evaluation. Prior to the psychoperceptual evaluation sensorial tests, demographic data collection, and quality anchoring and training took a place. During the test session, the stimuli were presented one by one and rated independently and retrospectively using a bidimensional research method of acceptance [13,18]. The quantitative evaluation task was repeated in different contexts of use (exp4). The post-test session collected qualitative quality descriptions using two different methods. Open Profiling of Quality (OPQ) was used in exp1 to exp3 [32]. During the attribute elicitation task of OPQ, test participants watch a subset of test stimuli. While watching, they write down their own quality attributes. Hereby, assessors are not limited in the derivation of quality attributes. Each individual quality attribute should be unique, it should describe exactly one aspect of perceived quality, and the assessor must be able to define it. These definitions are collected at the end of the elicitation task for each attribute to better understand their meaning. Again, the test participants write down these definitions using their own words (e.g. movement - movements cannot be identified). In previous OPQ studies, these descriptions have only been used for interpretation of the individual attributes in the final analysis [32]. However, these definitions represent another qualitative view on experienced quality. In this paper, we took the descriptions of attributes as an independent dataset of elicited experienced quality factors.

Presentation - Two different devices equipped with twoview autostereoscopic displays were used in the experiments. These displays provide two separate views, one for each eye of the observer by selectively blocking the light when viewed from different directions [4]. The brain fuses the two different images into a three-dimensional image freely floating above the display. Autostereoscopic displays do not require the use of specialized glasses to experience the 3D effect as the light filter is built into the display [4]. Audio was played back using headphones (exp1, 3: AKG K450; exp4-5: Sony MDR-E818LP). Method of analysis - The analysis followed the principles of Grounded Theory framework through systematical steps of open coding, concept development and categorizing [31]. The analysis of components of quality of experience was conducted in two phases: Firstly, all studies were analyzed independently. Secondly, over the studies analysis was based on the results of independent analysis. 1. Analysis of components of quality of experience in independent studies - The analysis contained three main steps: 1) Open coding towards concepts: In the interview studies (exp4-5) all recorded material was transcribed to text, meaningful sentences were extracted and open-coded from all data for creating concepts. This phase was conducted by one researcher and reviewed by another researcher. In the attribute data (exp1-3), as the descriptions were short (one sentence), well-defined, and exact compared to the interview data they were treated directly as the concepts in the analysis.

In exp4 and exp5, the post-test session contained a semistructured interview about experienced quality. During the interview the main question with slight variations was presented several times and the supporting questions were asked to clarify further the answers from the main question. The main question was: “What kind of factors you paid attention to while evaluating quality?” and the supporting questions were: “Please could you clarify was X among the positive/negative factors/pleasant/unpleasant?” and “Which of the factors you mentioned was the most

268

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 1. Characteristics of the experiments (VS=visual spatial details, VT=visual temporal motion, VD=amount of depth; L=low, M=medium, H=high) EXPERIMENT

DEVICE

EXPERIMENT VARIABLES

Exp1 [32] Sample size: 15

HDDP

Video: presentation mode (2D/3D); Audio: presentation mode (mono/stereo); Content: 6 contents

Exp2 [32] Sample size: 15

Exp3 [34] Sample size: 17

Exp4 [15,17] Sample size: 30

Exp5 [15,17] Sample size: 15

427x240px 3.5” headphones

HDDP 427x240px 3.5 “

HDDP 427x240px 3.5 “ headphones

3D LCD 400x480px 3.3 “ headphones

3D LCD 400x480px 3.3” headphones

STIMULI CHARACTERISTICS AND AUDIOVISUAL CODECS Length: ~ 18s Videos: Synthetic and Natural Presentation mode: 2D and 3D Quality level: Highly acceptable

Video: 4 coding schemes, 2 quality levels (low: 74-160kbps bit rate, high: 160452kbps); Content: 6 contents

Video: mp4v, 10-22Mbit/s, 25fps Audio: WMA 9, 48kHz 16bit Length: ~ 10s Videos: Synthetic and natural Presentation mode: 3D Quality level: Highly acceptable

Video: 3 coding schemes @ slice and nonslice mode, 2 MFER rates (10%, 20%); Audio: clean audio; Content: 4 contents

Video: H.264/AVC (JMVC 5.0.5) Audio: none Length: ~ 60s Videos: Synthetic and natural Presentation mode: 3D Quality level: Highly acceptable

Video: H.264/AVC (JM 14.2), MVC (JMVC 5.0.5) Audio: WMA 9 11/44.1 kHz Video: bit rate 160-1536kbps, frame rate 10- Length: ~ 30s 24fps, presentation mode (2D/3D); Videos: Synthetic and natural Content: 4 contents; Presentation mode: 2D and 3D Context of use: laboratory, home-like Quality level: Mainly unacceptable

Video: bit rate 320-768kbps, presentation mode (2D/3D); Audio: bit rate 18-48kbps; Content: 4 contents; Context of use: laboratory, bus-travel, station-wait

Video: H.264/AVC (x264 b1077) Audio: AAC-HEv2 (Nero 1.3.3.0), 44.1kHz 48kbps Length: ~30s Videos: Synthetic and natural Presentation mode: 2D and 3D Quality level: Mainly unacceptable

CONTENT CHARACTERISTICS VS VT VD Animation H H M Documentary H M H Videoconference M L M User-created H H H Music video M M M Documentary H M H Talking head Animation Feature film Horse Mountain Sports

M H H M H M

L M H L L H

L M M H H H

Documentary Animation Nature Roller

H H H H

M M M H

H H M H

Animation User-created Documentary Series

H M L L

H L M L

M M H M

Animation User-created Documentary Series

H M L L

H L M L

M M H M

Video: H.264/AVC (x264 b1077) Audio: AAC-HEv2 (Nero 1.3.3.0), 44.1kHz

2) All concepts were organized into subcategories and the subcategories were further organized under main categories. For interview data, the categorization was reviewed by another researcher. For the attribute data, the initial categorization was done independently by three independent researchers and final categories were constructed in the consensus between them. 3) Frequencies in each category were determined by counting the number of the participants who mentioned it. Several mentions of the same concept by the same participant were recorded only once. For 20% of randomly selected pieces of data (attribute descriptions or lettered interviews) inter-rater reliability is excellent (exp1-3: Cohen’s Kappa: 0.8, exp45: Cohen’s Kappa: 1.0).

the term ‘component’ to refer to any element of quality to combine the previously used terms of factors, components, dimensions, categories and aspects under the same umbrella. The identified concepts were categorized into initial subcategories and major categories by two researchers independently and reviewed by two other researchers. Final definitions for the categorizations were done in an agreement between two researchers. RESULTS

The results of components of quality of experience are presented in two parts. The main results of the independent studies are firstly described in two groups (based on datacollection method and display technique used). Secondly, components of quality of experience are summarized over the independent studies.

2. Analysis of components of quality of experience in all studies - The data was constructed from the sub-categories including their definitions from all experiments (1-5). These categories formed the concepts of analysis with equal importance between them. The equal importance was considered as the goal of the analysis to identify the general components for quality of experience. In this paper, we use

Quality of experience – study by study

Attribute based descriptions (exp1-3) - Experienced quality is constructed from components of visual quality (depth, spatial, temporal), viewing experience, content,

269

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

audio, and audiovisual quality (Table 2). The results showed similarities between studies, but also some studydependent characteristics. Visual quality, divided into depth, spatial, and temporal dimensions, was the most described component. In all experiments, visual depth quality is strongly characterized by the assessors’ ability to detect depth and its structure to separate the image clearly into foreground and background. Clarity of foreground and background, as well as a natural transition between them was underlined in the descriptions. Furthermore, the erroneous nature of depth was listed (exp1); artificial, strange or unnatural impression was created when the viewers felt that there was too much depth in the scenes, depth was somehow overdone, double images were visible, or when depth was constructed by multiple flat planes or cardboards. Assessor’s attribute illustrates well this failure in depth quality: ‘impression of an artificial world, one would not enter like this in reality’ (male, 26).

Visual spatial quality is expressed in the terms of clarity, color, brightness, impairments of different nature and the assessors’ ability to detect objects. In spatial visual quality, several negative categories were identified and they were all described with the same antonyms (clear, sharp, accurate, visible, error-free). In overall, spatial quality described the line from visibility of objects and overall image clarity to blurriness and finally to the structural imperfections such as blocking impairments and visible pixels. In cases of high video compression levels, the spatial impairments were highly announced in the results (exp2-3). Finally, visual temporal quality summarized the characteristics of motion from general mentions of motion and its fluency to impaired influent and blurry motion. The descriptions of temporal impairments (jerkiness, cut-offs) dominated when the transmission errors were introduced (exp3).

Table 2 Components of quality of experience, their definitions and percentage of participants’ attributes in this category in the experiments 1-3. COMPONENTS (major and sub) VISUAL DEPTH 3D effect in general Excellence of 3D effect Layered 3D Foreground Background VISUAL SPATIAL Clarity Color Brightness Blurry Visible pixels Detection of objects VISUAL TEMPORAL Motion in general Fluent motion Influent motion Blurry motion VIEWING EXPERIENCE Eye strain Ease of viewing Interest in content 3D Added value Overall quality CONTENT AUDIO AUDIOVISUAL

DEFINITION (examples) Descriptions of depth in video General descriptions of a perceived 3D effect and its delectability Artificial, strange, erroneous 3D descriptions(too much depth, flat planes) Depth is described having multiple layers or structure Foreground related descriptions Background related descriptions Descriptions of spatial video quality factors Good spatial quality (clarity, sharpness, accuracy, visibility, error-free) Colors in general, their intensity, hue, and contrast Brightness and contrast Blurry, inaccurate, not sharp Impairments with visible structure (e.g. blockiness, graininess, pixels). Ability to detect details, their edges, outlines Descriptions of temporal video quality factors General descriptions of motion in the content or camera movement Good temporal quality (fluency, dynamic, natural movements) Impairments in temporal quality (cut-offs, stops, jerky motion, judder) Experience of blurred motion under the fast motion User’s high level constructs of experienced quality Feeling of discomfort in the eyes Ease of concentration, focusing on viewing, free from interruptions Interests in viewing content Added value of the 3D effect (advantage over current system, fun, worth of seeing, touchable, involving) Experience of quality as a whole without emphasizing one certain factor Content and content dependent descriptions Mentions of audio and its excellence Audiovisual quality (synchronism and fitness between media). Total number attribute descriptions

270

Exp1 N=15 %

Exp2 N=15 %

Exp3 N=17 %

86.7 66.7 26.7 46.7 33.3

80.0 6.7 33.3 26.7 66.7

58.8 23.5 17.6 35.3

73.3 66.7 26.7 46.7 33.3 73.3

80.0 100.0 80.0 40.0 73.3 80.0

76.5 52.9 17.6 47.1 70.6 47.1

26.7 6.7 20.0

53.3 60.0 40.0 6.7

29.4 52.9 88.2 17.6

20.0 40.0 40.0 53.3

20.0 6.7 13.3 33.3

35.5 52.9 11.8 17.6

20.0 13.3 13.3

40.0 6.7

11.8 17.6 11.8 29.4

130

129

128

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 3 Components of quality of experience, their definitions and percentage of participants’ attributes in this category in the experiments 4-5. COMPONENTS (major and sub, (+) = positive; (-) =negative)

DEFINITION (examples)

VISUAL DEPTH Feeling of 3D (+) Artificial look (-) Fore-background (-) Seeing in two (-) Shadows (-) Color errors (-) See-through objects (-) VISUAL SPATIAL Spatial accuracy (+) Spatial impairments (-) Details VISUAL TEMPORAL Fluent (+) Influent (-) Blurriness of motion (-) Other movement (-) VIEWING EXPERIENCE Followability (+) Pleasant to watch (+) Unpleasant to watch (-) Ease of viewing (+) Difficulty of viewing (-) Need to focus (-) Viewing angle (-) Learning curve Long term viewing (-) Eyestrain (-) No symptoms (+) New experience Comparison to existing Immersion (+) Added interest (+) Enhanced authenticity (+) GENERAL QUALITY Constant quality As a whole Error free (+) AUDIO AUDIOVISUAL CONTENT

Descriptions related to three-dimensional presentation 3D feeling (feeling of space, depth, surroundings) Artificial look of 3D (e.g. scene is too deep) Bad relation between foreground and background Seeing objects in two Shadows around objects Color errors around object borders Ability to see through objects (solid objects become transparent) Spatial factors of video presentation Positive spatial factors (accuracy, clarity, sharpness, good details, good colors) Negative spatial factors (inaccuracy, unclarity, blurriness, bad details, bad colors, blocking, pixels) Details of spatial video in general (e.g. individual details undecipherable) Temporal factors of video presentation Fluent movement Influent movement (jerkiness, stops) Blurriness during motion Other errors of motion (e.g. pixels in motion) Viewer's task, relation to content, and eye-strain Ability to follow the video content, fit to purpose of use Pleasurable to watch, strongly positive experience Unpleasant to watch Ease of viewing Difficulty of viewing Viewing 3D requires extra effort or concentration Limited or narrow viewing angle The existence of a learning curve for watching 3D Not comfortable for long-term viewing Symptoms of eyestrain, headache, sickness Not experiencing any physical symptoms New and previously unrealized experiences General mentions of comparison to existing technology Impression that participants become a part of the events Increased interest in viewing the video Enhanced authenticity (naturality, life-likeness, realism) General quality related descriptions Constant quality during a clip Entire video affects the feeling of quality Quality is expressed to lack the errors (no annoying features) Audio factors (Bad audio) Audiovisual factors (how the audio supports video presentation) Quality is described in relation to any of the contents of experiment Total number of descriptions

Viewing experience described the users’ high-level interpretations of system (media, content) or their influences on the users. These subcomponents are not direct descriptions of representations of stimuli (e.g. colors, visible errors). However, they emphasize the interpretation of stimuli including users’ knowledge, emotions, or attitudes as a part of quality experience. The viewingrelated aspects highlighted the ease of viewing task and interests in viewing. Added value of the 3D was conveyed through the expressions of benefits over current similar system (2D on small screen), increased impressions of

Exp4 N=30 %

Exp5 N=15 %

86.7 50.0 56.7 53.3 70.0 60.0 46.7

93.3 20.0 33.3 33.3 33.3

93.3 96.7 33.3

80.0 46.7

20.0 53.3 63.3 36.7 20.0 36.7 33.3 20.0 20.0 53.3 46.7 36.7 70.0 20.0 40.0 16.7 6.7 46.7

6.7 20.0

6.7 13.3 6.7 33.3 26.7 33.3 20.0 53.3 53.3 33.3 40.0 26.7 40.0

10.0 10.0 33.3 6.7 6.7 100

93.3

778

194

20.0 6.7

involvement and tangible video. The assessors conveyed the impressions of this category in the terms of ‘how strong feelings and situations are transferred to me’ and ‘recognizing the beauty an ordinary image could not communicate’ (male, 23). Furthermore, experienced discomfort was described as a negative part of viewing experience. Finally, the construct of overall quality contained illustrates the total impression of quality. The final three classes highlighted content, audio and audiovisual quality.

271

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Interview based descriptions (exp4-5) - Quality of experience was formed from several visual quality components, viewing experience, quality factors in general, audio, audiovisual and content classes (Table 3). In overall, visual quality factors and viewing experience were the most commonly mentioned components containing several subcomponents and bipolar affective dimensions.

general quality component summarizes the overall impression of excellence of video and its error-freeness. Quality of Experience for 3D video on mobile device

The model of Descriptive Quality of Experience for 3D video on mobile device (DQoE- mobile 3D video), its components and subcomponents, their definitions and examples of subcomponent-dependent bipolar descriptive terms are described in Table 4.

Visual quality was defined by three subcomponents, depth, spatial and temporal quality similar to experiments 1-3. In the depth quality domain, feeling of 3D, artificiality and foreground-background relation were commonly announced. Such thoughts were described as: ‘[I liked] when it [image] was very clear, so that you didn’t have to put effort into seeing it properly, the whole system. For example, it really bothered me immediately when you couldn’t see the background properly’ (male, 39). In addition, several remarks about disparity errors between the two views were specified (seeing in two, shadows, colorerrors between views, see-through objects) underlining the common impairments of parallax barrier display technology [2]. Spatial quality was strongly defined by the factors of excellence (accuracy, inaccuracy). Finally, temporal quality especially referred to the fluency and blurriness of motion. Such thoughts were described as: ‘unpleasant was perhaps, especially scenes with fast motion were difficult to follow and they looked blurry at times’ (female, 43).

Quality of experience is composed of two main components, quality of visual modality, and viewing experience and two minor components, content, and audio and bimodal quality. Visual, audio and audiovisual quality underlines data-driven features of quality, i.e. directly detectable features and emphasizes characteristics of lowlevel perceptual processes as a part of quality perception. In contrast to the direct perceptual characteristics of media, viewing experience underlines the aspects of high-level perceptual processes where goal-oriented actions, knowledge, emotion and attitude are an essential part of quality interpretation. Visual quality is defined in detail with the aid of 16 subcomponents belonging to the groups of depth, spatial and motion quality. The excellence of depth is summarized as perceivable depth, its naturalness, composition of foreground and background layers, and balance between the qualities of these layers. Spatial quality is constructed from image clarity, block-freeness, colors, brightness, contrast and ability to detect objects and their edges. As a third component of quality factors of visual medium, motion is characterized by fluency, clarity and nature of motion.

Viewing experience described the factors of users’ viewing task and their relation to content as well as self-reported physical symptoms. Several commonly mentioned subcomponents defined the ease of viewing from several slightly different angles; the ability to reach the goals of viewing, a narrow viewing angle, and a need for extra concentration to learn to watch 3D were mentioned. Negative aspects of viewing 3D on a small display were described as eye-strain or by other self-reported uncomfortable feelings: ‘But when the picture quality is bad, then 3D doesn’t work. Then you can’t watch, because your eyes get tired. You really get a feeling that you have to focus and then you get these headaches and whatnot’ (female, 44). In contrast, positive experiences of watching videos in 3D presentation mode were announced as immersion (i.e. the feeling that participants become more mediated to the content) and enhanced authenticity. Finally, 3D on a portable device provided new experiences for the users, but its excellence was also evaluated against existing technology.

The second major component of quality of experience is formed from six subcomponents. Characteristics of viewing task are described by four components, ease and pleasantness of viewing, enhanced immersion and visual discomfort of viewing. Ease of viewing is defined as user’s ability to concentrate on the viewing task without extra effort or learning or interruptions by technology. Pleasantness of viewing represents the affective aspect of viewing in general. Enhanced immersion in viewing experience describes the viewers’ feelings of becoming a part of the events in the content or higher level of involvement, fun or tangibility. The enhanced immersion is also strongly related to positive impressions of 3D viewing. Visual discomfort summarizes the eye strain and other experienced discomfort that viewing possibly can cause on user. The comparisons between new and existing technology is based on user’s memory as defined by the subcomponent of impression of improved technology. Finally, the viewing experience is also characterized by an affective impression of quality as a whole (overall quality). The two minor classes summarize the quality of other related modality and interactions between modalities, and content-dependent quality.

In addition to visual quality and viewing experience, experienced quality was also constructed from quality of audio, audiovisual quality, content, and, finally, general quality components. Similar to the other experiments, audio and audiovisual quality was among the minor classes compared to visual quality. In the interviews, several quality descriptions were content-dependent resulting in a high number of mentions in this category. Furthermore,

272

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 4 Quality of experience for 3D video on mobile device – Components and their definitions COMPONENTS (major and sub) - Bipolar impressions VISUAL QUALITY DEPTH Perceivable depth Perceivable/Not perceivable Impression of depth Natural/Artificial Foreground-background layers Smoothly combined layers/Separate layers Balance of foreground-background quality Balanced/Unbalanced SPATIAL Clarity of image Clear/Blur Block-free image Block-free/Visible blocks Color, brightness and contrast Good/Poor Objects and edges Accurate/Inaccurate MOTION Fluency of motion Fluent/Influent Clarity of motion Clear/Blurry Nature of motion Static/Dynamic VIEWING EXPERIENCE Ease of viewing Easy/Difficult Pleasantness of viewing Pleasant/Unpleasant Enhanced immersion Enhanced/Not enhanced Visual discomfort Experienced/Not experienced Comparison to existing technology Improved/Not improved Overall quality Good/Bad CONTENT OTHER MODALITIES INTERACTIONS Audio Audiovisual

DEFINITION (examples) Descriptions of quality of visual modality, divided into depth, spatial and motion quality Descriptions of depth quality in video, characterized by perceivable depth, its natural impression, composition of foreground and background layers, and balance of their quality Ability to detect depth or variable amount of depth as a part of presentation 3D effect creates a natural, realistic and error-free impression instead of an artificial and erroneous impression (e.g. too much depth, double objects, shadows, seeing through objects) Depth is composed of foreground and background layers and the impression of the transitions between these layers can vary from smooth to distinguishable separate layers Balance between the excellence of foreground and background of image quality (e.g. sharp foreground, blurry background or vice versa, or they are otherwise not in balance) Descriptions of spatial image quality of video, characterized by clarity, block-freeness, colors, brightness, contrast and ability to detect objects and edges Clarity of image in overall -- Clear (synonyms: sharpness, accuracy, visibility) vs. unclear (synonyms: blur, inaccurate, not sharp) Existence of impairments with visible structure in image (e.g. blockiness, graininess, pixels) Excellence of colors, brightness and contrast Ability to detect necessary objects and details, their edges and outlines Descriptions of motion of video, characterized by fluency, clarity and nature of motion Excellence of natural fluency of motion -- Fluent (dynamic, natural) vs. influent (cut-offs, stops, jerky) Excellence of clarity of motion (e.g. accuracy under fast movement or movement out of screen) -- Clear, sharp vs. blurred, pixilated Nature of motion in the content or camera movements - Static (synonym: slow) vs. dynamic (synonym: fast) Descriptions of viewing experience, characterized by ease and pleasantness of viewing, enhanced immersion in it, visual discomfort and impression of improved technology and overall quality Easy to concentrate on viewing (e.g. free from extra effort and learning, viewing angle does not interrupt viewing) Pleasurable viewing experience, also for a longer period of time (e.g. 15min) Feeling of enhanced immersion into the viewing experience (impression of becoming a part of the events in the content, involvement, fun and improved impression of naturality, like-likeness, tangibility and realism) Feeling of visual discomfort (eye-strain) and descriptions of related discomfort symptoms (headache, general discomfort) Impression that provided quality of new technology (3D) is higher than quality of comparable existing technology (e.g. 2D video on a mobile device) Impression of excellence of quality as a whole without emphasizing a certain factor (e.g. excellence over the time, relation between erroneous/error-free) Descriptions of content, their content-dependency and interests in viewing content Descriptions of quality of audio modality and interaction between quality of audio and visual modalities Audio and its excellence Bimodal audiovisual quality (synchronism and fitness between media) and its excellence

quality goes beyond the apparent features of stimuli [6]. In addition, many of the subcomponents presented replicate some descriptions of previous work showing good reliability of our model [10,16,22,23,28,32-33]. In contrast, none of the previous work as such has constructed descriptive components for experienced quality of 3D mobile over-the-studies.

DISCUSSION AND CONCLUSIONS

The goal of this study was to identify the descriptive components of quality of experience for 3D video on mobile devices. We conducted five qualitative studies with heterogeneous stimuli material using written attribute descriptions and interviews in the data-collection. Our results showed that 3D video quality of experience on mobile device is constructed from four main components: 1) quality of visual modality in depth, spatial and motion domains, 2) viewing experience, 3) content, 4) audio and audiovisual quality. Our results confirmed that experienced

Experienced visual quality was described in three domains depth, spatial and motion. The depth was determined by the detectability of depth and its impressions, being in line with previous work [10,28]. The novelty of our work is towards

273

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the structured nature of depth - divided into foreground and background layers. In the case of comfortable 3D video, the composition of these layers is done successfully and the relation between the produced qualities of these layers is in balance. The need for balance between different medium qualities has been presented under multimedia studies [30], while our results suggest that similar phenomenon can also occur within the quality of one media. The requirements of structured depth have to be taken into account when designing 3D video contents and tools. The descriptions of spatial and temporal quality were similar to previous work on 2D mobile video [16]. Exceptionally, the blurriness during fast motion in vertical or depth direction was identified in all our studies. This may indicate a need for reduced speed in 3D scenes in order to provide viewers time to follow the motion as well as explore the content [10].

Fourthly, the current descriptive model may be expanded with the descriptions of other relevant work e.g. 2D and audiovisual quality. To conclude, this paper presents a general descriptive model of experienced quality for mobile 3D video. The main components of the model are visual quality, viewing experience, content and other modalities and their interactions. Novelty of the work was to construct the overall descriptive model, instead of study-dependent characteristics. Further work is needed to operationalize the model to act as an evaluation tool and to utilize it in the system development. ACKNOWLEDGMENTS

MOBILE3DTV project has received funding from the European Community’s ICT programme in the context of the Seventh Framework Programme (FP7/2007-2011) under grant agreement n° 216503. The work of the first author is supported by the UCIT graduate school.

Viewing experience, as a central component of quality of experience, is much more complex than the impression of pleasurable viewing. Firstly, the subcomponent of ease of viewing draws a connection between quality and usability. Secondly, positive affective experiences of 3D quality highlighted that 3D video on a small display, can offer impression of higher involvement or mediation. These terms have been conventionally associated with 3D presentations on large screens or virtual reality [5,10,11,28]. Thirdly, viewing experience was also influenced by visual discomfort [20], and the excellence of quality was weighted by the knowledge of existing quality. Finally, although the quality was divided into multiple factors, overall quality represented the user’s impression of quality as a whole showing that there is some kind of an evaluation composition structure (e.g. [3])

REFERENCES

1. Bech, S., Hamberg, R., Nijenhuis, M., Teunissen, C., de Jong, H., Houben, P., and Pramanik, S. The RaPID Perceptual Image Description method (RaPID). Proc. SPIE, Vol. 2657 (1996), 317-328. 2. Boev, A., Hollosi, D., Gotchev, A., and Egiazarian, K. Classification and simulation of stereoscopic artifacts in mobile 3DTV content. Proc. SPIE, Vol. 7237, No. 1, (2009). 3. Engeldrum, P.G. Psychometric Scaling. A Toolkit for Imaging systems development. Winchester: Imcotek Press, 2000. 4. Fehn, C. and Pastoor, R.S. Interactive 3-DTV concepts and key technologies. Proc. IEEE, vol. 94, no. 3, 2006, 524-538.

Although the quality of other media, their interactions and content were also listed as part of quality of experience, the accuracy of their presentations might be limited. The role of audio and the interaction between media might be smaller than in previous work [16] due to the dominance of visual variables in the experiments. Content-dependant quality descriptions were also strongly emphasized in the results of interviews (also in [16,27]). Accuracy of these components needs to be further clarified.

5. Freeman, J. and Avons, S.E. Focus group exploration of presence through advanced broadcast services. Proc. SPIE, Vol. 3959 (2000), 530-539. 6. Gibson, J.J. The ecological approach to visual perception. Houghton Mifflin, Boston. Lawrence Eribaum, 1979. 7. Gotchev, A., Smolic, A., Jumisko-Pyykkö, S., Strohmeier, D., Akar, G.B., Merkle, P., and Daskalov, N. Mobile 3D television: Development of core technological elements and user-centered evaluation methods toward an optimized system. Proc. SPIE, Vol. 7256, No. 1, (2009).

There are four suggestions for the further work on the descriptive quality of experience model. Firstly, the model needs to be further validated in both experimental and natural circumstances. Secondly, the model is necessary to be operationalized to act as a tool for a) vocabulary-based sensorial quality profiling (e.g. [1,22,23,29]), b) design/evaluation heuristics for professionals of 3D content creation and system development, c) comparisons of quality of 3D mobile multimedia systems for consumers. Thirdly, it would be valuable to understand the characteristics or profiles of different objective quality factors (e.g. artifact clusters [2]) on a descriptive scale. Ultimately, this could lead to the novel approach of development of quality metric going beyond the level-of apparent features of stimuli.

8. Grill-Spector, K., and Malach, R. The human visual cortex. Ann. Review Neuroscience, Vol. 27 (2004), 649677. 9. Gulliver, S.R., and Ghinea, G. Defining user perception of distributed multimedia quality. ACM Trans. Multimedia Computing, Communications and Applications, 2(4), 2006, 241-257.

274

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

10.Häkkinen, J., Kawai, T., Takatalo, J., Leisti, T., Radun, J., Hirsaho, A., and Nyman, G. Measuring stereoscopic image quality experience with Interpretation Based Quality methodology. Proc. SPIE, Vol. 6808 (2008).

22.Lorho, G. Perceptual evaluation of mobile multimedia loudspeakers. Proc. Audio Engineering Society 122th Convention, (2007). 23.Mattila, V.V. Descriptive analysis of speech quality in mobile communications: Descriptive language development and external preference mapping. Proc. 111th AES Convention, Paper Number 5455, 2001.

11.Ijsselsteijn, W., De Ridder, H., Freeman, J., and Avons, S.E. Presence: Concept, determinants and measurement. Int. Symp. on Electronic Imaging Science and Applications: Human Vision and Electronic Imaging V, vol. 3959, B. E. Rogowits and T. N. Pappas, Eds. SPIE, 2000, 520-529.

24.MOT Oxford Dictionary of English 1.0. The Oxford Dictionary of English, Oxford University Press 2005. Retrieved, 1.3.2010.

12.ITU Recommendation ITU-R BT.500-11. Methodology for the subjective assessment of the quality of television pictures, Recommendation ITU-R BT.500-11. Telecommunication Standardization Sector of ITU (2002).

25.Neisser, U. Cognition and reality, principles and implications of cognitive psychology. San Francisco: W.H. Freeman and Company, 1976. 26.Radun J., Virtanen T., Olives, JL., Vaahteranoksa M., Vuori T., and Nyman, G. Audiovisual quality estimation of mobile phone video cameras with InterpretationBased Quality approach. Proc. SPIE, Vol. 6494 (2007).

13.ITU Recommendation ITU-T P.910. Subjective video quality assessment methods for multimedia applications, Recommendation ITU-T P.910. Telecommunication Standardization Sector of ITU (1999).

27.Radun, J., Leisti, T., Häkkinen, J., Ojanen, H., Olives, J.-L., Vuori, T., and Nyman, G. Content and quality: Interpretation-based estimation of image quality. ACM Trans. Appl. Percept. 4, 4 (2008), 1-15.

14.Jumisko-Pyykkö S (2008) I would like to see the subtitles and the face or at least hear the voice: effects of picture ratio and audio-video bitrate ratio on perception of quality in mobile television. Multimed Tools Appl 36(1–2):167–184,2008

28.Shibata, T., Kurihara, S., Kawai, T., Takahashi, T., Shimizu, T., Kawada, R., Ito, A., Häkkinen, J., Takatalo, J., and Nyman, G. Evaluation of stereoscopic image quality for mobile devices using Interpretation Based Quality methodology, Proc. SPIE, Vol. 7237 (2009).

15.Jumisko-Pyykkö, S. and Utriainen, T. D4.4 v2.0 Results of the user-centred quality evaluation experiments. MOBILE3DTV technical report, 2009. http://sp.cs.tut.fi/mobile3dtv/results/tech/D4.4_Mobile3 DTV_v2.0.pdf

29.Stone, H., and Sidel, J.L. Sensory Evaluation Practices. 3rd ed. Academic Press, San Diego, 2004. 30.Storms, R. Auditory-visual Cross-modal perception phenomena. Doctoral dissertation, Naval Postgraduate School, Monterey California, 1998.

16.Jumisko-Pyykkö, S., Häkkinen, J., and Nyman, G. Experienced quality factors – qualitative evaluation approach to audiovisual quality. Proc. IS&T/SPIE 19th Annual Symposium of Electronic Imaging 2007, 650721.

31.Strauss, A. and Corbin, J. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 2nd ed., Sage, Thousand Oaks, CA, 1998.

17.Jumisko-Pyykkö, S., and Utriainen, T. A Hybrid Method for Quality Evaluation in the Context of Use for Mobile (3D) Television Multimed Tools Appl (in press). DOI 10.1007/s11042-010-0573-4

32.Strohmeier, D., Jumisko-Pyykkö, S., and Kunze, K. Open Profiling of Quality - A mixed method approach to understanding multimodal quality perception. Advances in Multimedia, vol. 2010 (in press).

18.Jumisko-Pyykkö, S., Kumar Malamal Vadakital, V., and Hannuksela, M.M. Acceptance threshold: Bidimensional research method for user-oriented quality evaluation studies. Int. J. Digital Multimedia Broadcasting, 2008.

33.Strohmeier, D., Jumisko-Pyykkö, S., and Reiter, U. Profiling experienced quality factors of audiovisual 3D perception, QoMEX (2010).

19.Jumisko-Pyykkö, S., Weitzel, M., and Strohmeier, D. Designing for User Experience – What to expect from mobile 3D television and video. Proc. uxTV 2008, 2008.

34. Strohmeier, D., Jumisko-Pyykkö, S., Kunze, K., Tech, G., Bugdayci, D., Bici, M.O. Results of quality attributes of coding, transmission and their combinations. Tech report (2010).http://sp.cs.tut.fi/mobile3dtv/results/tech/D4.3_M obile3DTV_v1.0.pdf

20.Lambooij, M., Ijsselsteijn, W., Fortuin, M. and Heynderickx, I. Visual discomfort and visual fatigue of stereoscopic displays: a review. J. Imaging Science and Technology, 53(3): 030201-030201-14, (2009).

35. Wu, W., Arefin, A., Rivas, R., Nahrstedt, K., Sheppard, R., and Yang, Z. Quality of experience in distributed interactive multimedia environments: toward a theoretical framework. Proc. ACM Multimedia MM '09. ACM,(2009),481-49.

21.Lewicki, M.S. Efficient coding of natural sounds. Nature Neuroscience, Vol. 5(4), (2002), 292-294.

275

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Unobtrusively Controlling and Linking Information and Services in Smart Environments Bastian Kriesten Christian Mertes René Tünnermann Thomas Hermann Cognitive Interaction Technology – Center of Excellence (CITEC) Bielefeld University, Germany {bkrieste, rtuenner, cmertes, thermann}@techfak.uni-bielefeld.de

ABSTRACT

Our living and work spaces are becoming ever more enriched with all kinds of electronic devices. Many of these are too small to provide the possibility to control or monitor them. Ambient intelligence is integrating many such devices in what are called smart environments to form a network of interweaved sensors, data displays and everyday devices. We present a method to intuitively issue control over smart objects in such an environment, to display data that smart objects provide and to manage the flow of information between objects in a smart environment. This is achieved by using touch-enabled mobile phones as readily available multi-purpose devices which are used to overlay real objects with virtual controls. We evaluated the system with a first qualitative user study. AUTHOR KEYWORDS

Figure 1: Illustration of the ad-hoc creation of a temperature-controlled fan.

Mixed reality, augmented reality, mobile interaction, mobile devices, home automation, ambient data streams ACM CLASSIFICATION KEYWORDS

H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Artificial, augmented, and virtual realities; J.9.a [Mobile Applications]: Location-dependent and sensitive

son touches an object – are already detectable by today’s sensor systems. Yet we are overwhelmed by digital information streams which are becoming ever denser with mail, instant messaging, RSS, Twitter and further examples from the continuously growing number of services.

INTRODUCTION

Our everyday environment becomes more and more filled with technical devices. These are ranging from simple lamps and digital thermometers to remote controls, mobile phones, television sets or recently even dedicated ambient displays. Modern sensors and algorithms enable us to sense and collect data about humans interacting in their environment. Many things – from the position of a person or object in a room, the ambient temperature or light intensity to what and where a per-

In our work, we present an architecture and system to create and control mappings from such information streams to the smart environment. This is done within the environment itself instead of using an inevitably indirect wimp-based interface. Our approach is to use modern touchscreen-based smartphones as the interface. The phone can be used to a) drag and drop information from one object to another, b) to connect a virtual wire from one object to another, so that the information (updates) an object provides will be send to the other object and c) to display an object’s information directly on the phone’s display, or d) to control a device or manipulate information associated with a smart object.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

1

276

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

An example is illustrated in Figure 1. At first the mobile phone displays the current temperature when pointing it at the thermometer (1). By touching the temperature reading, it can be picked up (2). Dropping it on the fan establishes the link (3). From now on, the fan will be switched on and off automatically depending on the temperature. The user might have to modify the threshold in a further step. The architecture enabling this kind of smart linkage is detailed in the infrastructure section. We used the mobile phone’s camera to detect proximity and the direction towards an object. On an augmented camera view on the touchscreen the controls and the offered information of the object in reach are displayed. Examples of the mixed reality view are shown in Figure 2. In the first screenshot (a) three colored buttons are drawn on the camera view of an ambient light. By a tap on a button the light’s color will be changed to the color of the tapped button. Secondly (b) a stop and a play/pause button as well as a progress bar are displayed on top of a loudspeaker. The button with the speaker symbol controls the volume: when pressed and held the phone can be turned upwards and downwards – like a volume knob – to adjust the volume accordingly. In the bottom half of the screen an explanatory text is displayed. On the right side (c) the picture frame application’s controls are shown. The dialog allows the user to change the picture and start/stop the display application.

(a)

(b)

(c)

Figure 2: Screenshots of various mixed reality views.

Touch Projector by Boring et al. [1] enables remote screen interaction including file dragging and dropping between different screens using touch phones. Their system does not deal with the control, augmentation and connection of real-world smart objects as ours does. Nonetheless, they describe a couple of improvements to the original system by Tani et al. which we will compare to the enhancements planned for our own system in the discussion section. Boring et al. [1] also provide a good survey of further work akin to mixed-reality-based interaction with mobile phones.

By using wireless networking it is possible to interact with the objects via the mobile phone and to manage and control their input and output data streams.

cristal is a more recent mixed reality approach to control smart environments. It was proposed by Seifried et al. [8]. They use an interactive surface on which a live video view from above the smart space is displayed. By touching objects on that surface, users are able to directly control or move them not only in the virtual but also in the real space. Like them, we also use mixed reality to manipulate smart objects, but where they use a single centralized interface we use a variety of decentralized mobile controls instead. Both methods have advantages and disadvantages. cristal offers a central, large and convenient interface, whereas our system runs on mobile devices that people have handy at most times and that can be used to interact with smart objects in the immediate vicinity. A fusion of both systems would be beneficial as both complementary interaction styles were supported, comforting varying needs and situations.

RELATED WORK

Relevant previous work has been done by Rohs who developed a visual marker recognition system for Symbianenabled mobile phones and used this amongst other things for gesture-based real world interactions with mobile phones [4]. Rukzios set an important cornerstone for smartphonebased interactions with the real world [6]. He was the first to analyze the potential of such kinds of interactions in different user studies and provided a taxonomy of physical mobile interactions. Our approach goes beyond these works with the interaction possibilities of a smartphone-based, touch-enabled mixed reality view. We also take the interaction with smart objects one step further to the direct control of a smart environment as a whole by shaping its data streams in an intuitive manner.

CONCEPTS

A mobile phone is usually within reach and therefore seems like the ideal choice for a universal device control tool. Modern smartphones are also capable mixed reality devices in that they have both good display possibilities with their built-in camera and a comparatively large display and the very direct input possibility of a touchscreen. This makes it possible to augment nearby real-

Using an augmented video stream to observe and control objects was introduced by Tani back in 1992 [9]. They used a live video view with augmented information through which the user was able to operate machines. 2 277

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 3: Digital picture frame application. The marker in the corner of the screen is recognized by the mobile phone which then displays controls on top of it to control the screen and to browse the picture collection.

Figure 4: Living room area in our ambient intelligence laboratory.

world objects with virtual controls and displays (Figure 3). The displayed controls can then be manipulated directly using the phone’s touchscreen. Smart objects can thereby offer input and output possibilities they do not provide through physical means such as buttons or displays. Even passive objects can receive a kind of virtual intelligence that is not inherent to them but given to them by the surrounding smart environment. We call the augmentation of an object with virtual interface elements which can be accessed when approaching the object with a mobile device, its ambient aura.

APPLICATIONS

Based on the described concepts we implemented a prototype system and integrated it into our ambient intelligence laboratory which is a smartroom that is in parts equipped similar to a living room (Figure 4). We implemented several base interaction paradigms: Controlling the Ambient Data Flow

Every mixed reality element in our system that is shown on the smartphone can be picked up by holding it on the touch screen and can subsequently be dropped on any other smart object. By this “drag and drop”-like gesture it is possible to move linked data from one object to another or permanently link one object’s output channel with another’s input channel. In our smartroom for example it is possible to grab a picture shown on a digital picture frame (Figure 3) and drop it directly to a projection surface. A linked music file can be dropped on a loudspeaker to have it played. In this way, handling digital data becomes closer to handling real world objects.

Data flows in the smart environment can connect any output pin of one node to the input pin of another. The mobile phone acts as an intuitive and direct means to establish and modify the emerging network of connections. It is itself also a node in the network and can directly display data output by other devices and exercise control over such devices, in addition to redirecting data flow from one device to another. The transition from direct input and output via the mobile phone to the redirection of input and output between devices should remain fluent. Both interaction types – direct input or output and the manipulation of these ambient data flows – should be very intuitive due to the physical nature of the interaction and its spatial coupling to the place where the effect is desired, instead of the place where the required devices are located. Data or data streams can simply be grasped, carried around and dropped where they should be displayed or be used to control a device.

Drag and drop gestures can also be used to establish device connections. Every output element that is dragged on another smart object can be linked with a compatible input element of the latter object like a connection through a virtual wire. It is possible to link the “new email” output symbolized by a vicarious email proxy object with a lamp’s on/off input element to automatically switch on the lamp when a new mail arrives. The thermometer’s output can be linked to a fan for example, to make it blow as soon as a certain threshold is exceeded (Figure 1). The thermometer’s output can also be used to continuously control the light color of ambient LED lights to visualize the temperature and adjust the

We expect that using the mobile phone itself to input or output data will be more useful in spontaneous and ad-hoc interactions while long-term display of data and monitoring should be more practical on persistent display devices.

3 278

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ambient lighting accordingly.1 This small collection of examples illustrates that our primary aims were to keep the interfaces generalizable and flexible and to minimize the setup time and effort for new connections, thereby leaving the concrete application design to the users. The system is meant to be tailored to each user’s individual and changing needs without requiring any programming or complex GUI handling.

be read by every visitor with an appropriate phone as a kind of virtual fridge magnet. Additionally, the door offers a button to globally toggle all devices in the room on or off. Furthermore, passive objects such as a printed album cover can be associated with the respective music, the artist’s MySpace site or other additional information. In our system these data can be accessed with the mixed reality view by touching the corresponding element on the smartphone screen. In case of the CD cover, the mobile phone would replay the linked audio files itself or it could initiate the transfer of the associated data to other devices.

As the input and output channels of different objects often have different data types we implemented automatic data transformations to link channels with different types of data. This enables the user to link nearly every output data to the Picture Frame as the data is converted to a text string. By this it is possible to display the current temperature value or the played music. A second transformation implements a threshold function. This function turns a device on or off if a certain threshold is reached. The last transformation maps for example temperature values to a color displayed by the ambient lights.

Interactive Objects

In accordance to the described interaction paradigms we implemented the smart objects that are listed below. These objects offer a good basis for further user studies and the demonstration of the interactive concepts. To see them in action, videos can be found on the web2 .

Thermometer A picture of a thermometer that gives access to the current weather measurements of a close-by weather station.

Physical and Virtual Device Control and Information Access

Several devices such as digital picture frames or loudspeakers are normally equipped with a very minimal set of – or even without any – input interfaces. Mostly this is due to technical, aesthetic or economic reasons. Through our mixed reality system we can control the digital picture frames (Figure 3) and the loudspeaker in our laboratory directly: when pointing the smartphone camera at a digital display or a loudspeaker, appropriate control elements are shown attached to the device. In case of the display the name of the current picture is shown and it is possible to browse through the integrated picture library or switch the screen on and off by pressing the virtual buttons. Pointing at a loudspeaker shows the title of the currently playing song, a progress bar to visualize the elapsed playing time and the usual stop and play/pause buttons. Additionally, the volume can be controlled by pressing a mixed reality button and turning the smartphone like a physical volume knob. In our laboratory, nearly every on/off device (for example the lamps and the fan) is integrated in a similar manner.

Picture Frame A screen that allows browsing through a digital photo library. Linked data streams can be rendered as image or as text. Loudspeaker It plays back dropped audio data. The mixed reality view shows the currently playing track, a play/pause and a stop button and the elapsed time as progress bar. Music Library Actual carriers or paper printouts of the covers that are linked with the corresponding music. Projection Surface Similar to the screen, a visual marker at the edge of a projection surface enables images or text to be dropped directly onto a surface where it is then displayed by a projector. Intelligent Door Offers a virtual button to globally switch off all devices in the room. The door additionally has the possibility to leave notes.

Augmenting Passive Objects

Several passive objects in our smartroom are endowed with an ambient aura and thus augmented with additional related information, output data or functionality. For instance, we used a wooden box as a proxy object for an email account. By pointing the camera at the box the users receive information about their email inbox, in particular the number of unread mails. In a similar manner the door is integrated as a smart object, allowing users to attach text messages to the door that can

Fan, Decoration Lights The fan and decorative chains of lights can be individually switched on and off. Ambient Lights There are LED lights embedded in the table whose color and intensity can be controlled. Card Game The box that holds a deck of cards is linked with a game rules website.

1 The relation is established instantly but could have to be parametrized and configured later on. For details on the underlying architecture see the infrastructure section.

2 http://www.techfak.uni-bielefeld.de/ags/ami/ publications/KTMH2010-UCL/

4 279

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Email Inbox This object provides information about the number of unread emails in the user’s email inbox. These are only a few sample objects. However, it is already possible to build combined functional units as diverse as e. g. the temperature-controlled fan described in the introduction, an email notifier using ambient lights or a “currently playing” display (connecting the loudspeakers to the picture frame). The number of reasonable applications should grow increasingly with the number of available components. INFRASTRUCTURE

The prototype system we developed consists of the Nokia 5800 XpressMusic 3 touchscreen smartphones used as mixed reality interaction tools and different smart objects tagged with a visual code. Each smart object’s functionality is provided by a software component that organizes the communication with the ambient intelligence network.

Figure 5: Schematic overview of communication framework.

Ambient Intelligence Network

In the context of this interaction tool, we developed a communication framework for ambient intelligence systems that is based on standard TCP/IP and the Extensible Messaging and Presence Protocol (XMPP)4 on the application layer, using the pubsub extension5 that implements the publish/subscribe paradigm. Figure 5 gives a schematic overview of the communication framework. Figure 6: The pubsub-based channel input/output data flow of the communication framework.

A smart object’s functionality is provided by an application that is connected to the infrastructure. These applications communicate with other infrastructure participants through broadcast channels based on pubsub nodes (see Figure 6). Data input and output to the object is realized by data channels of a certain data type that are represented as distinct nodes. Such an application publishes channel output to the output node and receives input by subscribing to the input node. An object’s metadata and a channel’s properties are published by the according application to an info node.

application as well as the smart object software. The library supports the connection of output channels to other objects’ input channels with automatic data transformation, regardless of whether the receiving node is a mobile phone or any of the smart objects. This communication framework provides an extendable and reactive loose-coupled system of distributed devices and services with minimal effort.

An object’s channels and its modalities are defined in the object’s meta information that is published to the object’s info node. A channel is described by 1) an ID, 2) a descriptive name, 3) the input/output possibilities, 4) a data type and 5) data constraints. The currently supported data types are: numbers, text, signals, on/off values, linked image data, linked sound data, linked files, URLs, vector graphics and smart objects.

Smartphone Application

The smartphone application creates the mixed reality interfaces based on live camera images, visual marker detection and the ambient intelligence network system. Figure 7 visualizes the application’s workflow cycle. At first the build-in camera is used to capture a video frame. These frames are analyzed to detect smart objects. The Visual Codes System (VCS) developed by Michael Rohs [5] is used to locate and identify objects with visual code tags. A visual code is a monochrome 10×10 dot matrix that can store 76 bits of data. This is used to store an ID that can then be used to find the

A library to use this communication infrastructure was developed in Python and is used by the smartphone 3 http://europe.nokia.com/find-products/devices/

nokia-5800-xpressmusic 4 http://xmpp.org/about/ 5 http://xmpp.org/extensions/xep-0060.html

5 280

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 7: Mobile applications workflow. Figure 8: Mobile applications user interface. associated application in the ambient intelligence network. After the marker recognition, the object specific information and the channel modalities are queried from the network system and are cached for faster subsequent requests. These data and the detected visual code positions in the picture are used in the following step to augment the picture and create a mixed reality view. This augmented view is displayed to the touchscreen. Any user input in reaction to this view is then processed and the input is sent to the smart object. This cycle is continuously repeated while the application is running.

• text and number data labels or input forms, • numbers in an interval that are shown as a progress bar or – in case of input modalities – as a slider widget (an alternative implemented input method for this is using physical rotation of the mobile phone similar to a tap), • signals and on/off Boolean values that are shown as either stateless or toggle buttons, • vector graphics that are drawn onto the object, • linked image files that are displayed on the object, • linked sound data, • linked files and websites whose file name or URL is shown.

In addition to the mixed reality view, the user interface (see Figure 8) has a second area in the bottom half of the screen. In this area, static – i. e. non-mixed-reality – content is shown. When the user points the camera at a smart object, the object’s name and description (made available through the communication framework) are displayed here.

Furthermore, we implemented the possibility to grab any shown data or control element in the mixed reality view and drop it on any other smart object, similar to the well-known “drag and drop” gesture from the desktop metaphor. This is used to establish persistent channel connections or just transfer output data of one object as input data to another object once.

The smartphone application is written in the Python programming language except for the image processing by the VCS which had been developed in C++ for Symbian.

There are also experimental input possibilities that mimic physical interactions. The volume button in the speaker UI can be pressed to then use the mobile phone like a water tap, turning it around the z-axis of the button to regulate the volume up or down. There is also a physical slider interface element that – when activated – measures the sideways movement of the mobile phone to adjust the slider position accordingly. These physical interaction styles turned out to be harder to handle than the interactions that used the more conventional GUI input explained above. A notable exception is the drag and drop interaction that combines a physical action for the data transport with the screen-only manual part of the interaction.

Generic Mixed Reality Interfaces

We designed various generic mixed reality user interfaces (UIs) for smart object channel interactions and included them in the smartphone application. The interface representation and interaction depend on the current channel output and the channel modalities, which contain the data type, data constraints and the mixed reality representation parameters. When focusing a smart object, all channels provided by the object application will be displayed at their relative position. By this, it is possible to create rich smart object interaction interfaces in a generic way. We designed configurable channel interactions for 6 281

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

was fun to use or showed that by spontaneous smiling or laughing or by exclamations such as “cool” or “awesome”. In one of the free response questions we asked whether the system went against the participants’ expectations at any point. Only one participant raised an actual objection against the system design here by stating “a self-explanatory menu especially for problems would be desirable”. One other participant criticized the flaky marker detection again and one participant responded “easier to use than expected” to this question.

EVALUATION Method

We evaluated our prototype system in a study with 9 participants, 6 male and 3 female, ranging from 23 to 35 years in age with the mean age being 28.8 years (median age 29). Every participant owns a mobile phone, one of them with a touchscreen and three more stated that they have had some experiences in the usage of a touchscreen in association with other devices. Three participants had used a system with a mixed reality view before6 . The user study was conducted in the living room part of our ambient intelligence laboratory depicted in Figure 4. In our scenario we chose a subset of the objects listed in the applications section to be made available for the interaction: Picture Frame, Thermometer, Ambient Lights, Fan, Decoration Lights, Loudspeaker, Music Covers. Participants were given a mobile phone with the software already running. Each participant was had to read written instructions consisting of five steps. Each step encompassed a conceptual explanation and a concrete task which had to be completed before moving on to the next step. See Table 1 for an overview of the explanations and tasks given and Appendix A for a translation of the exact instructions the participants received. The participants were observed and videotaped during the interaction. After performing the tasks, participants could continue to explore the possibilities of the system if they chose so and were finally asked to fill in a questionnaire. The questionnaire had five multiple choice questions with four options and seven free response questions. A translation of the questions are listed in Appendix B. Only the first and the last option were labeled. In the following we will denote the second and third option like the adjacent option without the comparative “very”. For example when the first option was labeled “very likely” and the fourth “very unlikely”, the second will be called “likely” and the third “unlikely”.

There were also some interesting anecdotal observations during the study. One participant took the physical interaction one step further by trying to spin the fan on with a fast circular gesture. Several people tried to link the outside temperature to the loudspeaker. Only one participant linked the temperature to the screen and one other used the text display capabilities of the screen to display the title of a piece of music. One participant did not remember that he had used the loudspeaker even after being reminded of the music task. The same participant also tried to link the picture frame with an item of the music collection to change the picture on a track change. This would have worked if the picture frame and the loudspeaker had been linked. Discussion

Our observations and the feedback from the participants indicate that the system has a strong appeal to new users. We furthermore noticed that most of the interactions we designed were usable without many explanations and despite the prototype character of the system. One of our main goals was intuitive and direct interaction which seems to have been achieved. The flexibility that arises from being able to freely connect various devices and channels also was received well by most of our test participants. On the other hand, the difficult handling of the mobile phone, namely the small, moving and sometimes flickering controls, seemed to be a major setback for the performance of the users. The users had difficulties in focusing the objects and keeping the smartphone still while touching the mixed reality elements on the screen.

Results

Figure 9 shows the results of the multiple choice questions. Each question had four options, 1 being most positive (options 1 and 4 bore question-specific labels). For the free responses, the issue raised the most was the jitter in the marker recognition. This point was brought up by 7 participants. The size of the display or the controls was criticized by 5 participants and it was mentioned by 4 participants that they would have liked the system to work at a greater distance to be more like a remote control. Connecting different devices was highlighted as a particularly exciting possibility by 5 of the participants. Six participants said that the system

Some usability defects for this kind of interaction metaphor were also discussed by Boring et al. [1] for their Touch Projector system. We think these image stability problems can also explain the discrepancy between the felt intuitiveness and the significantly reduced effectiveness (cf. Figure 9) for our system. This will therefore be the area in which improvements are most urgent. Many techniques are conceivable to cope with the sometimes poor marker detection and the involuntary movement of the device.

6 For example the Lego Digital Box, an augmented reality kiosk system that is showing a 3D Lego model virtually at the top of the box (http://www.metaio.com/fileadmin/homepage/ Presse/Dokumente/Pressemitteilungen/English/E_2009_01_15_ PR_LEGO.pdf).

Possible mitigation strategies for this include using a more stable object recognition that allows a greater distance and by using a larger display with a faster render7 282

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

How intuitive did you find the system?

7  

How quickly did you get used to the system?

7  

How likely is it for you to actually use such a system?

7  

How efficient to use did you find the system?

7  

How pleasant (“angenehm”) to use did you find the system?

7  

Phase 1

0  

Explanation general system overview

0  

2

direct object control

3

read out data

4

drag and drop

5

object connection

0  

0  

0   1  

2  

3  

4  

Figure 9: Questionnaire multiple-choice questions and results as histograms. The leftmost and rightmost responses had labels but for the sake of readability they were left out in this overview. Left indicates positive results and right negative.

Task familiarization and experimentation with the system switch on the picture frame and choose a picture read out the outside temperature choose one from the pieces of music on the table and drop it on a speaker connect the outside temperature to the LED light color

Table 1: Steps each participant had to perform in the given order.

ing7 and the design of an intelligent touch detection that is more flexible and adaptable. For example by evaluating more than the current frame when the user touches the screen, with consideration of the human response time. Or alternatively by freezing the view under certain circumstances such as the approaching of the user’s finger to the screen (or manually as done by Boring et al.). Some further strategies to cope with the handling problem will be discussed in the conclusion.

other hand, when the number of controls in a given context rises, either by having multiple controls attached to different parts of one object or by having multiple smart objects within reach, the spatial association between object and control should become a more important aspect of UI efficiency. This situation is of course more likely when the size of the objects’ ambient auras is increased to enable the remote control capability that many participants understandably wished for.

Other usability defects were revealed by the user study for which a fix is obvious. For example being forced to hold the finger down the whole time during the drag and drop interaction is cumbersome, but losing the object altogether when the marker flickers away in the wrong moment is just plain annoying. Locking and being able to recover the UI object in this case is important even if other interface improvements were to stop the controls from flickering away when the marker detection is interrupted.

CONCLUSION

We proposed a way to control, query and interconnect smart objects with a mixed reality smartphone application that uses visual markers to identify objects. Further we argued why we think this approach will significantly improve the usability of smart environments to the extent of opening up new use cases. Although the participants of the evaluation were excited about the possibilities and the intuitiveness of the system there is a strong need for improvements. We discussed some of the technical difficulties that users had and improvements that need to be accomplished to unlock the full potential of this approach.

On a more general note however, it has to be debated in which cases context-aware controls that are separate from the actual object (in our case, the bottom half of the screen lends itself, cf. Figures 2 and 8) might be more efficient to use, even if these were at the expense of immediacy and maybe pleasurability. We believe that for one control at a time, mixed reality interfaces might actually not reach their full potential. When the virtual controls are attached convincingly to the object, the movement of the control device constitutes an inherent disadvantage for the ease of handling. This can only be circumvented by sacrificing immediacy in one way or another. On the

The implemented system serves as a platform to control smart objects, ambient displays and their information flow directly in our familiar environment. The proposed method using visual markers is just one tool to identify those objects. In future work we will use RFID (Radio Frequency Identification) and NFC (Near Field Communication) as an additional method for the ambient aura use case to make object recognition more robust and flexible [6, 7, 2]. Bokode markers could be used to detect objects even from across the room [3].

7 Our system reaches approx. 10 frames/second at 300×400 pixels (on approx. 3×4 cm)

8 283

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

However, the required developments go far beyond mere technological improvements. The need to determine the circumstances under which mixed reality interfaces are beneficial and when they are not has already been mentioned. From the fact that we abandon the desktop metaphor altogether follows that new metaphors and according symbols need to be developed. How should for example connections between smart objects be represented when being drawn, to name just one such question that arises? Some participants expressed their wish for more feedback and we also believe that there are various possible visual, auditory and haptic cues that can provide a subtle and unobtrusive yet all the more helpful system response while performing tasks. The new workflows for tasks such as connecting devices and choosing options and parameters for these connections also needs constant revisioning accompanied by thorough user studies to optimize it.

[7] E. Rukzio, K. Leichtenstern, V. Callaghan, P. Holleis, A. Schmidt, and J. Chin. An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning. In UbiComp 2006: Ubiquitous Computing, volume 4206 of Lecture Notes in Computer Science, pages 87–104. Springer Berlin/Heidelberg, 2006. [8] T. Seifried, M. Haller, S. D. Scott, F. Perteneder, C. Rendl, D. Sakamoto, and M. Inami. Cristal: A collaborative home media and device controller based on a multi-touch display. In Proc. of the International Conference on Interactive Tabletops and Surfaces 2009, pages 37—-44, New York, NY, USA, 2009. ACM. [9] M. Tani, K. Yamaashi, K. Tanikoshi, M. Futakawa, and S. Tanifuji. Object-oriented video: interaction with real-world objects through live video. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, pages 593–598, New York, NY, USA, 1992. ACM.

These questions are not to be underestimated in their importance for the effective and easy to use control device of future smart environments, yet we believe the proposed system to be a powerful framework to address these questions.

APPENDIX A – INSTRUCTIONS FOR THE PARTICIPANTS

The following is a translation of the exact instructions and explanations given to the participants for the user study. The original instructions were written in German. The formatting has been altered in order not to interfere with the overall look of this work. The original also featured some explanatory figures and every step was printed on its own sheet of paper so participants could not inadvertently read ahead.

ACKNOWLEDGMENTS

This work has been supported by the Center of Excellence 277 Cognitive Interaction Technology funded in the framework of the German Excellence Initiative. BIBLIOGRAPHY

[1] S. Boring, D. Baur, A. Butz, S. Gustafson, and P. Baudisch. Touch projector: Mobile interactionthrough-video. In Proc. of the 2010 International Conference on Human Factors in Computing Systems, New York, USA, Apr. 2010. ACM.

Study on device control with mobile phones The AmILab is a laboratory for ambient intelligence. We investigate and develop new interaction possibilities for modern environments. This study takes place in our “living room area”. A mobile phone with a touchscreen is used as an interaction tool. On this phone the camera view is displayed together with virtual elements (mixed reality view).

[2] J. Bravo, R. Hervas, G. Chavira, S. Nava, and V. Villarreal. From implicit to touching interaction: Rfid and nfc approaches. In Human System Interactions, 2008 Conference on, pages 743–748, May 2008.

By means of the mobile phone you can interact with every object that bears a black and white marker.

[3] A. Mohan, G. Woo, S. Hiura, Q. Smithwick, and R. Raskar. Bokode: imperceptible visual tags for camera based interaction from a distance. In SIGGRAPH ’09: ACM SIGGRAPH 2009 papers, pages 1–8, New York, NY, USA, 2009. ACM.

During the experiment you will be given tasks that are to be solved with the system. Make yourself familiar with the functionality of the system and with its interaction possibilities. Then move on to the next page.

[4] M. Rohs. Linking Physical and Virtual Worlds with Visual Markers and Handheld Devices. PhD thesis, University of Technology, 2005.

Device control

[5] M. Rohs. Real-world interaction with camera phones. Lecture notes in computer science, 3598:74, 2005.

Explanation Devices can be controlled using the mobile phone. When you focus a suitable device, virtual elements will be displayed. These can be manipulated by using the touchscreen with your finger.

[6] E. Rukzio. Physical Mobile Interactions: Mobile Devices as Pervasive Mediators for Interactions with the Real World. PhD thesis, Ludwig-MaximiliansUniversität München, Dec. 2006.

Task Switch on the digital picture frame (Dell monitor) on the wall and choose a picture.

9 284

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Readout of linked data

Task Connect the thermometer output to the LED lights in the table in order to represent the outside temperature visually.

Explanation Using the mobile phone, digital data that are linked to objects can be queried. These can be static data such as digital documents or music as well as dynamic information such as sensor data or content on the Internet.

Finally, you are free to further explore the possibilities of the system if you wish. APPENDIX B – FREE RESPONSE QUESTIONS

The following are the translated free response questions on the questionnaire.

Task Read out the outside temperature. Drag and drop Explanation All of the digital data that are displayed on top the camera view can be picked up and dropped onto another device or object. To pick it up a virtual element has to be held with the finger. When now focusing another device or object, a frame will appear around the element. A green frame will mean that it is possible to drop the element there, a red one will mean that this is not possible. To drop, just let go of the element. You will then have to choose what to do with the data you dropped.

Question 1 Describe briefly your first impressions when using the system. Question 2 What did you especially like? Question 3 What did you especially dislike? Question 4 What would you change? Question 5 What did you find most disturbing while using the system? What restricted the usability the most?

Task Choose a piece of music from the collection on the table and let it play on the loudspeaker.

Question 6 At what points did the system not comply with your expectations?

Connecting devices

Question 7 Under which circumstances can you imagine to actually use such a system in your every day life?

Explanation You can also use the mobile phone to link devices together. To do this, pick any output element as described above and drop it on another device. By selecting the paper clip in the following step, the output will be permanently linked with the according input.

10 285

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Evaluating Multimodal Systems A Comparison of Established Questionnaires and Interaction Parameters Christine Kühnel

Deutsche Telekom Laboratories Technische Universität Berlin 10587 Berlin, Germany

Tilo Westermann

interactive media.virtual environments Universität Hamburg 22527 Hamburg, Germany

Benjamin Weiss

Deutsche Telekom Laboratories Technische Universität Berlin 10587 Berlin, Germany

[email protected]

Sebastian Möller

Deutsche Telekom Laboratories Technische Universität Berlin 10587 Berlin, Germany

ABSTRACT

Keywords

This paper describes the analysis of established and new questionnaires concerning their applicability for the assessment of quality aspects of multimodal systems. To this purpose, an experiment with 27 participants interacting with a a smart-home system via a voice interface, a smartphonebased interface and a multimodal interface, was conducted. Interaction parameters were assessed and related to constructs measured with these questionnaires. The results indicate that some of the questionnaires are suitable for evaluating multimodal interfaces. On the basis of correlations with interaction parameters subscales of these questionnaires can be mapped to quality aspects, such as effectiveness and efficiency. Recommendations are given how to meet two important evaluation requirements, namely which questionnaire to use for comparing two or more systems or system versions and how to identify factors or components in a system that have to be improved. This is another step forward to establish evaluation methods for multimodal systems.

evaluation, multimodal interaction, gesture, smart-home

INTRODUCTION Today, multimodal systems are well-established – owing to the ascribed advantages discussed in numerous publications (e. g. [27, 17]). Among those advantages are a higher reliability of the recognizer due to mutual disambiguation, or the flexibility offered to the users to select those modalities most adequate in their specific situation, and according to their preferences and capabilities. Along the way, guiding principles for the design of multimodal systems have been proposed [29] and the combination of modalities has been systematically described: at the fusion level with the CASE model [26] and at the user level with the CARE properties [5]. But once the system is – at least partly – built, methods for quantitative assessment are needed. With such an evaluation several goals are pursued, including the comparison of two or more systems or system versions (to verify assumed improvements) or the identification of factors or components in the system that can be improved [28]. Typically, the assessment is based on user ratings gathered with questionnaires. Additionally, in the domain of spoken dialog systems, parameters describing the interaction – such as number of turns and dialog duration – have been derived from log data and annotations [20]. Often, user ratings are described as ‘subjective’, while the interaction parameters are considered to be ‘objective’ [28]. This nomenclature implies a bias towards the ‘objective’ interaction parameters, while evaluations usually rely heavily on ‘subjective’ questionnaire ratings. In the PARADISE framework [34] questionnaire ratings – in terms of the users’ satisfaction with a spoken dialog system – are used as target for a prediction model based on a linear combination of parameters obtained from log and annotation data. Beringer et al. propose to transfer this approach by defining a methodology for the end-to-end evaluation of multimodal dialogue system (PROMISE [1]). Recently, a survey of principles, models and frameworks was published by Dumas et al. [6], identifying – among others –

Categories and Subject Descriptors H.5.2 [Information Interfaces and Presentation]: User Interfaces—Evaluation/methodology

General Terms Human Factors

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010 October 16-20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00

286

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the modeling of multimodal interaction as a future challenge. For these purposes questionnaires are necessary to assess the user ratings of multimodal systems. Furthermore, metrics from the domain of spoken dialog systems that are correlated with user ratings have to be transferred to multimodal interaction. For some aspects which are inherent to multimodal systems, such as modality fusion or fission, new metrics have to be defined as proposed by K¨ uhnel et al. [14]. This paper aims to contribute on two levels to the work done on the evaluation of multimodal systems: Established and new questionnaires are analyzed concerning their applicability for the assessment of quality aspects of multimodal systems. Interaction parameters are related to constructs measured with these questionnaires as further means of identifying issues and as a first step towards prediction of user ratings. Therefor, the research questions addressed are

To this end, we will first discuss work related to the evaluation of multimodal systems. Then, the methodology is described, including the system, the questionnaires and the test design. The result section is structured according to the two main questions addressed, namely the analysis of the questionnaires and the relation to interaction parameters. Finally, the results are discussed in detail and recommendations are given.

System Interfaces (SASSI) [11]. While the SASSI would have to be adapted to be used for multimodal systems, the authors do not recommend the use of the SUMI for the evaluation of multimodal systems. One aspect not covered by any of the questionnaires listed is the degree of intuitive use a system offers. Naumann et al. [23] attempt to bridge this gap with a questionnaire measuring intuitive use. Wechsung and Naumann [35] compared the different questionnaires focusing on usability and related constructs. They found that questionnaires designed for unimodal systems, such as the SASSI, are not applicable for usability evaluation of multimodal systems, since they seem to measure different constructs. In the paper by Naumann and Wechsung [22] subscales, such as efficiency, are compared to interaction parameters extracted from log-data. The authors conclude that questionnaire ratings and interaction parameters show only limited concordance. They accentuate the need for a reliable questionnaire for multimodal interfaces and state that the AttrakDiff [8] provides a suitable basis. Continuing the work by Wechsung and Naumann [35, 22] we analyzed a new and shorter version of the AttrakDiff, the SUS and the new questionnaire measuring intuitive use for their applicability to the domain of multimodal dialog systems. We also included the USE questionnaire proposed by Lund [18]. A brief description of the questionnaires is found in the next section. We did not apply the SUXES methodology as we wanted to focus on quality aspects and the comparison of questionnaires. Furthermore, multimodal interfaces and speech interfaces are still quite unfamiliar to the average user, and we feel that this would make the assessment of expectation rather difficult.

RELATED WORK

METHODOLOGY

(1) Do the questionnaires under consideration measure similar constructs? (2) Can these constructs be related to interaction parameters?

While for a long time focus has been on a systems’ usability, user experience seems to be the new catchword. This section will give a short overview on work done on evaluation of multimodal systems in general without discussing the relation of usability and user experience, which has been done by Nigel Bevan [2] and Effie Law et al. [15] among others. A methodology to assess user experience of multimodal dialog systems (SUXES) is described by Turunen et al. [33]. SUXES is a complete procedure, starting with an introduction to the evaluation and a background questionnaire. This is followed by an introduction to the application and the assessment of expectations of the users. Then, the user experiment is carried out, and the user experience is assessed. The questionnaires rely on a set of nine statements, related to speed, pleasantness, clearness, error free use, robustness, learning curve, naturalness, usefulness and future use. Interaction parameters are not analyzed. The method addresses the question to which degree the expectations are met by the actual experience with the system. The authors claim that the method is efficient and ‘particularly suitable for iterative development’. In [19] an overview on quality aspects of multimodal systems is given. Established questionnaires are discussed concerning their appropriateness for assessing these quality aspects. It is argued that the AttrakDiff [8] and the System Usability Scale (SUS) [3] cover most aspects (Learnability, Effectiveness, Efficiency, Aesthetics, System Personality and Appeal) at least partly. Further questionnaires examined are the Software Usability Measurement Inventory (SUMI) [13] and the questionnaire for Subjective Assessment of Speech

In this section the smart-home system and the different input modalities are briefly described, the test design is explained and the questionnaires are introduced.

I’m Home: Multimodal Interface to a SmartHome System Smart-Home System In the frame of the EU-funded IST-project 2001-32746 [21] a voice-controlled smart-home system was developed, which has been enhanced with an additional smartphone-based input. This system is set up inside a fully functional living room. A panoramic view of the room is depicted in Figure 1. Possible interactions include the control of lamps and blinds, the TV, an internet radio, an electronic program guide (EPG), video recorder, and hifi system. Furthermore, the system offers an archive for music and supports the generation of playlists. In the case of a smooth interaction – i.e. the users command is valid and understood by the system - system feedback is equivalent with the corresponding device feedback: e.g. lamps turning on or radio station changing. The TV program, available radio stations, lists of recorded movies, an overview of the users music (sorted by album, artist, etc.) or the playlist is displayed on the TV screen. Those lists are also displayed on the smartphone to allow touch input for the selection of list entries and the execution of corresponding actions, such as recording a movie or deleting a song from the playlist. In the case of communication problems – e.g. recognition

287

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 1: The living room where the smart-home system is installed. errors or incoherent commands – system feedback depends on the input modality. If voice input was used, a spokendialog recovery strategy applies. If a gesture was not recognized, an immediate recovery strategy on the smartphone is triggered: a list of possible options is displayed, of which one can be selected via touch.

Voice Interface Speech recognition was replaced by a transcribing wizard, but participants were told that there was a speech recognizer in place. A lapel microphone was used to further strengthen this impression. Additionally, the participants were asked to read a standardized text aloud – officially to train the speech recognizer. The replacement of the automated speech recognition with a typing wizard introduced an additional delay of 1.4 seconds on average for spoken input. Participants could communicated via command style language or continuous sentences. The natural understanding unit was also replaced by the wizard. Spoken output only occurred in case of out-of-domain commands, if the information provided by the user was insufficient or if the user asked for help. Here is a sample dialog for a simple task with statements of the participant (P) and system (S):

Figure 2: smartphone interface.

Task: ‘Turn on the ceiling light.’ P: Please turn on the lamp. S: I understood ’Turn on the lamp’. Which lamp would you like to turn on? P: Help. S: You can turn on the wall lamp, the ceiling lamp or all lamps at the same time. P: Turn on the ceiling lamp.

ing the radio button while moving the phone to the right would switch the radio station, while it would switch to the next TV channel if the TV button was pressed. Reusing the same concept for different system controls reduces the gesture set considerably. Before the gesture input could be used, the recognizer had to be trained with a set of gestures, which also have to be related to the functionalities of the system (such as ‘turn on the TV’). For the experiment, this was achieved by showing a video clip of each of the seven gestures to the participants, which they were then asked to repeat five times to train the recognizer. With every gesture a short description was recorded by the participant and stored for the ‘help’-functionality. Recognized gestures elicited an immediate vibration feedback. Furthermore, the participant could touch the GUI menu buttons to see which devices could be controlled, or which gestures had been recorded. The menu button with the question mark opened a detailed help screen.

Smartphone Interface A gesture recognition for simple and often used gestures (TV, radio, lamps, blinds) was combined with touch input via a GUI for more complex interactions (see Figure 2, lower buttons) and implemented on an Apple iPhone 3GS which communicated via wireless LAN with the smart-home system. As suggested by [36], we have followed a participatory design approach involving possible end users in the design process to develop the gesture set. This gesture set had then been tested concerning requirements proposed by [25]: the gestures should be easy to perform and remember, intuitive, metaphorically and iconically logical towards the functionality, and ergonomic. A set of seven three dimensional gestures was used in the experiment (see Table 1 and Figure 3). The gesture recognition based on accelerometer data was triggered by the selection of a device via the GUI (see Figure 2, upper buttons). The same gesture could thus be used to initiate the same effect for different devices. Press-

Multimodal Interface The system offers in principle three input modalities, namely spoken input, gestural input and touch input via a GUI presented on the smartphone. Spoken input can be used for every type of interaction. Additionally, for simple and often repeated interactions, such as turning on the TV, gestural

288

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Gesture

Command

Device

Swing up

Volume up Brighter Open

TV, Radio Lamps Blinds

Swing down

Volume down Dim Close

TV, Radio Lamps Blinds

Point forward

Turn on/off

TV, Radio

Swing to the right

Next channel Stop

TV, Radio Blinds

Swing to the left

Previous channel

TV, Radio

Move down and up

Turn on/off

Lamps

Tilt

Show EPG Show available channels

TV Radio

They were recruited on the university campus and were paid for their time. Participants were asked whether they owned an iPhone or a Nintendo Wii or had the opportunity to play with it at friends places. Only two owned a Wii and four owned an iPhone. But eleven participants (=40.47%) had at least some experience with either one. Every participant interacted three times with the smarthome system, each time using a different interface: the voice interface, smartphone-based interface or the multimodal interface, resulting in three conditions. The order of the conditions was randomized across participants. After an introduction to the experiment a short video clip was shown, giving an example of the possible interactions with the particular interface. Each input modality was trained – as described above – just before interacting with the interface. Afterwards, the participants were guided through a first interaction (lowering the blinds and stopping them midway). Then, they were left alone in the room and asked to follow the instructions displayed on a screen in front of them. After solving eight to eleven tasks (scenario A, B or C) the participant was asked to fill out a questionnaire. This procedure was repeated twice (see Figure 4), each time with a slightly different set of tasks. One example of the tasks to be performed with the system is given below, with the possible input modality indicated in brackets.

Table 1: Gesture-command mapping.

(a) Swing to the

(b) Swing to the

right

(c) Swing upwards

1. Turn on the ceiling light and try to dim it. (gesture, voice) 2. Turn on the TV. Navigate to the television-stations Sat1 and RTL. Turn down the volume and then turn off the TV. (gesture, voice) 3. Play the biathlon video. Mute the sound. (touch, voice) 4. Delete two tracks from your ‘favorites’ playlist. Add two new titles. (touch, voice) 5. Find out which movies are running tonight (gesture, voice) and record one of them. (touch, voice) 6. Zap through the radio stations. Turn down the volume. Switch to the next station.(gesture, voice) Mute the sound. (touch, voice) 7. Show the list of MP3s. Navigate to the albums of an interpreter. (touch, voice) 8. Now show the titles of an album. Play a track and then stop it. (touch, voice)

left

(d) Swing

(e) Tilt

downwards

On average it took the participants 9.46 minutes to complete one scenario (SD=3.40). (f ) Move down and up

(g) Point

for-

Questionnaires

ward

As motivated above we analyzed four different questionnaires, which are here describe in some more detail:

Figure 3: The gesture set.

• A new and shorter version of the AttrakDiff questionnaire [10]: ten antonym pairs rated on a 7-point scale ([-3,3]), yielding the subscales ‘Attractiveness’ (ATT), ‘Pragmatic Qualities’ (PQ), ‘Hedonic Quality - Stimulation’ (HQ-I) and ‘Hedonic Quality - Identity’ (HQ-I).

commands are possible. Likewise, more complex interaction tasks can be solved by touch input, for instance the generation of a playlist. In terms of the CARE properties [5], gestural and touch input can be described as assigned input, while the combination of both smartphone-based input options are equivalent to spoken input. The system does not accept complementary input, and redundant input is simply ignored. Thus, depending on the task the user could only select between two modalities: spoken input and gestural input, or spoken input and touch input.

• The System Usability Scale (SUS) [3]: ten items rated on a 5-point Likert scale with 1 - strongly disagree to 5 - strongly agree. • The scales assessing ‘usefulness’ (six items) and ‘ease of use’ (four items) of the USE questionnaire [18], rated on the same scale as the SUS.

Test design

• The questionnaire described in [23], assessing intuitiveness (QUESI): fourteen items, rated on the same scale

We asked 27 young adults (Mage =26.1, SDage =3.89) - of which 14 were female - to participate in our experiment.

289

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

interacting via voice (M = 7.91, SD = 1.23) and slowest when interacting via the smartphone interface (M = 11.95, SD = 4.19). When they had the option to choose freely, dialog duration was on average 8.73 minutes (SD = 2.72).

introduction pre-test questionnaire

age gender experience

Recognition Rate For the voice interface perfect recognition was simulated to balance the perfect recognition of the graphical user interface. In the smartphone-based interaction only about half of the tasks had to be solved with gestures, the other half relied on touch interaction, where no recognition errors occurred. For the gesture recognition the recognition rate (given in percent) shows a high variation between participants with a mean of 51.33 and a standard deviation of 17.38. The highest recognition rate achieved was 76.92%, while the lowest was only 15.79%. This is mostly due to false positives (M = 17.19, SD = 9.40). In the case of multimodal interaction users had the option to choose their preferred modality and even switch modalities whenever they felt like doing so. In this case again the recognition rate shows a high variability (min = 11.39, max = 94.60, M = 64.23, SD = 18.64) due to the different preferences of the participants.

video clip

repeated for each modality

(training) guided interaction

scenario A, B, C: 8 -11 tasks

questionnaire

AttrakDiff SUS USE QUESI

Task Success

Figure 4: Procedure.

Task success had to be annotated by hand based on video logs. As no time restrictions were applied, all participants managed to finish all given tasks. Task success would therefor be constant. Thus, we defined task success as the percentage of user input which led to a system response indicating the success of the action. Recognition errors and incoherent input, for example, reduced the task success. With the voice interface participants reach a comparatively high task success of 81.5% on average (SD = 9.5). The task success was lowest with the smartphone interface (M = 65.17%, SD = 8.5). With the multimodal interface participants were more successful (M = 72.63%, SD = 10.07).

as the SUS, yielding the subscales ‘Subjective mental workload’ (W), ‘Perceived achievement of goals’ (G), ‘Perceived effort of learning’ (L), ‘Familiarity’ (F), and ‘Perceived error rate’ (E).

Interaction parameters During the whole experimental session log-data, such as timestamps for user input, system output and response was recorded. From this log-data efficiency related parameters, e. g. dialog duration and number of turns, can be computed. After asking the participants for their acquiescence a video camera was set up in a corner of the room to record user and system behavior. The video data was, for example, used to annotate the performed gestures and thus – in comparison with the log data – to compute the gesture recognition rate.

Modality Choice During the multimodal scenario participants had the option to choose their preferred modality. This choice could be made for every single interaction. Even though gesture recognition rates were low, participants still interacted via the smartphone as shown by Figure 5. Of the smartphonebased input, 26% was gestural input. Participants always tried to finish the given task and no time limit was applied. Thus, the often erroneous gesture recognition led to a higher task duration (Spearman’s ρ = −.47, p < .01) in both, the smartphone and the multimodal context, as many interactions had to be repeated or even retraced in the case of false positives. Only in 27% of the cases did participants switch to spoken input when their gesture was not or falsely recognized. The majority tried again and again using the smartphone.

RESULTS To address the questions stated above, namely (1) do the questionnaires measure similar constructs and (2) can these constructs be related to interaction parameters, the section is structured as follows. First, the results obtained for the interaction parameters are displayed as these will be referred to in the following analysis of the questionnaires. To answer the first question, global scales and subscales are then compared concerning the results obtained for the three different interfaces. Where relations can be assumed, correlations between global scales and subscales and interaction parameters are calculated to answer the second question.

Questionnaire Ratings

Interaction Parameter

Scales and subscales were calculated according to the manuals [8, 3, 18]. For the AttrakDiff all negatively poled items were recoded, so that higher values indicate better ratings.

Dialog Duration Dialog duration could be extracted automatically from logdata and was measured in minutes, as the time from the beginning of an interaction to its end. There was a significant difference for the different interfaces (F (2, 72) = 13.06, p = .000, η 2 = .265). Participants solved the tasks fastest when

Comparison of usability measurements In M¨ oller et al. [19] usability is described as resulting from ‘joy of use’ and ‘ease of use’, being thus influenced by hedonic and pragmatic aspects. Following this concept of usabil-

290

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

100

Average number of user turns

recognition rates as well as dialog duration. Table 3 shows that this is indeed the case. The strongest correlation is found between the SUS and the recognition rate (r = .55).

mobile phone voice

Pragmatic Quality Pragmatic quality is a measure for a system’s perceived usefulness, according to Hassenzahl et al. [9]. Characteristics, such as ‘easy’ and ‘predictable’, assess the user’s need for a controllable system. This scale should measure efficiency and effectiveness, and should therefor be correlated with dialog duration [22] and task success. The later was declared ‘the simplest usability metric’ by Nielsen1 . The results obtained confirm the assumption phrased above, both for dialog duration (Pearson’s r = −.46) and task success (Pearson’s r = .67).

80

60

40

20

Intuitive Use 0 voice-only

mobile phone

To measure the intuitive use an interfaces offers, a new questionnaire (QUESI) was developed by Naumann et al. [23]. The five subscales (min=1, max = 5) of the QUESI are ‘subjective mental workload’ (W), ‘perceived achievement of goals’ (G), ‘perceived effort of learning’ (L), ‘familiarity’ (F), and ‘perceived error rate’ (E). They yield consistent results concerning the ranking of the three interfaces (see Table 4). As found on the scales SUS and USE the voice interface is rated best on all QUESI subscales and the smartphone-based interface is rated worst. According to M¨ oller et al. [19] intuitivity should have an influence on ‘ease of use’. Therefor, high correlations are expected and indeed found between the global QUESI scale ‘intuitiveness’ (mean of all items) and the ‘ease of use’ subscale of the USE (r = .85, p = .000). Furthermore, we would expect the subscales ‘subjective mental workload’ (W), ‘perceived achievement of goals’ (G), and ‘perceived error rate’ (E) to be influenced by interaction parameters. The Pearson’s’ correlations displayed in Table 5 corroborate this.

multimodal

User interface

Figure 5: Modality choice. M

SD

F(2,78)

p

η2

voice mm phone

1.63 1.65 0.87

1.08 0.97 1.65

4.19

.019

.098

SUS

voice mm phone

78.52 67.04 56.11

13.61 18.41 21.86

10.15

.000

.206

USE

voice mm phone

3.55 3.24 2.89

0.67 0.76 0.92

4.73

.012

.108

Scale

Interface

ATT

Table 2: Mean and standard deviation of questionnaire scales measuring usability for the three interfaces as well as ANOVA results.

Hedonic Quality

Seite 1

Apart from quality aspects related to – comparatively easily measured – constructs, such as effectiveness and efficiency, recent work emphasizes the importance of hedonic quality (e.g. novelty, beauty) [31, 8]. This aspect can be further divided into ‘stimulation’ and ‘identity’, both measured with the AttrakDiff questionnaire introduced above. On those scales, no differences of the three interfaces could be found (cf. Table 6). These scales are supposed to measure different aspects of perceived quality and should thus not correlate strongly with any of the other scales apart from

ity we presume that scales measuring usability are the SUS, with ratings ranging from 0 to 100, and the ‘Attractiveness’ scale of the AttrakDiff (Min = -3, Max = 3). We also consider the mean of the USE ratings (Min = 1, Max = 5) as a usability measure. The QUESI is supposed to measure intuitive use and subscales related to this concept. The analysis of the QUESI is therefore not included in this paragraph. As shown in Table 2 the three (sub-)scales ATT, SUS and USE yield differences between the interfaces, and also roughly the same ranking. On the scales SUS and USE the voice interface is rated best, followed by the multimodal interface and the smartphone-based interface. Post-hoc tests reveal that only the differences measured with the SUS are significant between all three interfaces. On the ‘attractiveness’ and the global USE scale only the differences between voice and smartphone interface are significant. Pearson’s correlations between the scales are given in Table 3. The strong correlations (r > .7) indicate that these scales indeed measure a similar construct. As argued by Naumann and Wechsung [22] correlations should be found between scales measuring usability, and

ATT SUS USE

SUS

USE

dd

rr

ts

.73**

.71** .82**

-.25* -.48** -.39**

.26* .55** .47**

.39** .54** .50**

Table 3: Correlations r between usability scales and between these scales and the interaction parameters dialog duration (dd), recognition rate (rr), and task success (ts). A double star indicates a significance level of .01 and a single star a level of .05. 1 http://www.useit.com/alertbox/20010218.html, last accessed 2010/04/19

291

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

M

SD

F(2,78)

p

η2

voice mm phone

3.56 3.10 2.64

0.82 0.98 0.74

7.73

.001

.165

G

voice mm phone

3.98 3.58 3.20

0.81 1.00 1.03

4.51

.014

.104

L

voice mm phone

3.90 3.52 2.98

0.91 0.98 0.87

6.84

.002

.149

F

voice mm phone

3.53 3.22 2.84

0.87 0.80 0.93

5.66

.005

.127

E

voice mm phone

3.54 3.02 2.54

0.99 0.98 1.07

6.57

.002

.144

Scale

Interface

W

Scale HQ - Stimulation HQ - Identity

– Hedonic quality seems to be an aspect relatively unrelated with other constructs like intuitiveness or pragmatic quality. It does have an influence on usability as measured by ‘attractiveness’. (2) Can these constructs be related to interaction parameters? – Correlations with the interaction parameters ‘task success’ and ‘dialog duration’ indicate that the perceived usability and ‘pragmatic quality’ are indeed affected by effectiveness and efficiency.

As multimodal systems are becoming more and more prevalent their evaluation is an important issue. The multitude of possible modalities combined and the different ways to implement the combination of modalities (as described by the CARE and CASE properties [5, 26]) lead to a high degree of freedom which makes evaluation all the more complicated. Evaluation can, among other, be based on ques-

W G E

– No correlation of hedonic quality with the interaction parameters is found. We could thus replicate the findings of [24] and [30] for the field of multimodal systems that preference measured via questionnaires and performance measures are correlated – a result that would merely support common sense had not the opposite been found by Hornbæk and Law [12] and Frøkjær et al. [7], among others. An overview of questionnaires and quality aspects they assess according to our results is given in Table 7. We can also analyze our results in regard to the possible goals pursued with system evaluation [28]. Concerning the comparison of systems or system versions we find that the System Usability Scale has the highest sensitivity for the

ts p

r

p

-.40 -.35 -.51

.000 .000 .000

.56 .47 .48

.000 .000 .000

.70 .64 .67

.000 .000 .000

.047 .025

– The QUESI questionnaire appears to measure a similar construct as the ‘ease of use’ subscale (EoU) of the USE questionnaire.

DISCUSSION

rr

.152 .372

– We found that the ‘attractiveness’ subscale of the AttrakDiff, the System Usability Scale and the global scale of the USE questionnaire measure the same construct, namely usability.

The influence of the ‘pragmatic quality’ on ‘attractiveness’ is stronger than the influence of the hedonic aspects. A reason for this might be the usage mode our participants were in when interacting with the system. It has been argued in [9] that in a task-guided interaction (goal mode) ‘pragmatic quality’ should be more important than ‘hedonic quality’. As expected, no correlations are found between the interaction parameter and hedonic quality.

r

1.93 1.00

(1) Do the questionnaires measure similar constructs?

AT T = 1 + .5 · P Q + .34 · HQI + .17 · HQS

p

part. η 2

tionnaire ratings or interaction parameters obtained in user tests. Questionnaires have been developed assessing a systems’ usability (e. g. SUS [3], CSUQ [16], SUMI [13]), user interface satisfaction (QUIS [4]) or perceived hedonic and pragmatic quality (AtrrakDiff [8]). Depending on the system under consideration these questionnaires have been adapted – for example to assess website usability [32] – or specialized questionnaires have been designed (e. g. SASSI [11] for speech system interfaces). It is questionable though whether these questionnaires can easily be transported to the field of multimodal systems. M¨ oller et al. [19] for example do not recommend the SUMI for the evaluation of multimodal systems. In the presented paper four questionnaires have been used to evaluate three different interfaces, amongst those a multimodal interface. Based on the data the questionnaires have been analyzed to answer the following questions:

‘attractiveness’ – this construct being understood as resulting from a combination of ‘pragmatic qualities’ and ‘hedonic qualities’. Correlations of the hedonic quality scales are indeed highest among themselves (r=.74, p =.000) and with ‘attractiveness’ (r=.72 for ‘identity’ and r=.70 for ‘stimulation’). A simple linear regression analysis results in the model (R2 =.74, F(3,76)=72.56, p=.000):

dd

p

Table 6: ANOVA results for differences between the interfaces on hedonic aspects.

Table 4: Mean and standard deviation of questionnaire scales measuring intuitiveness for the three interfaces as well as ANOVA results.

r

F(2,78)

Table 5: Correlations between the interaction parameters dialog duration (dd), recognition rate (rr), task success (ts) and QUESI subscales.

292

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ACKNOWLEDGMENTS

QUESTIONNAIRES Quality Aspects

AttrakDiff

Usability Efficiency (dd) Effectiveness (ts) Intuitivity

(ATT) (PQ) (PQ)

SUS   

USE

QUESI

   (EoU)

(W,G,E) (W,G,E) 

We would like to thank Matthias Siebke and Falk Schiffner for their help in setting up and conducting the experiment. The project was financially supported by the Deutsche Forschungsgemeinschaft DFG (German Research Community), grant MO 1038/6-1.

References [1] N. Beringer, U. Kartal, K. Louka, F. Schiel, and U. T¨ urk. PROMISE: – A procedure for multimodal interactive system evaluation. In Proc. of the LREC Workshop on Multimodal Resources and Multimodal Systems Evaluation, Las Palmas, pages 77–80, 2002. [2] N. Bevan. What is the difference between the purpose of usability and user experience evaluation methods? In Proc. of the UXEM Workshop at INTERACT, 2009. [3] J. Brooke. SUS: A quick and dirty usability scale. In P. W. Jordan, B. Weerdmeester, A. Thomas, and I. L. McClelland, editors, Usability evaluation in industry, pages 189–194. Taylor and Francis, London, 1996. [4] J. P. Chin, V. A. Diehl, and K. Norman. Development of an instrument measuring user satisfaction of the human-computer interface. In Proc. of ACM CHI, pages 213–218, 1988. [5] J. Coutaz, L. Nigay, D. Salber, A. Blandford, J. May, and R. Young. Four easy pieces for assessing the usability of multimodal interaction: The CARE properties. In Human-Computer Interaction, Interact, pages 115–120. Chapman & Hall, London, 1995. [6] B. Dumas, D. Lalanne, and S. Oviatt. Multimodal interfaces: A survey of principles, models and frameworks. pages 3–26, 2009. [7] E. Frøkjær, M. Hertzum, and K. Hornbæk. Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In Proc. of CHI, pages 345–352. ACM, 2000. [8] M. Hassenzahl, M. Burmester, and F. Koller. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualit¨ at [a questionnaire for measuring perceived hedonic and pracmatic quality]. Mensch & Computer 2003. Interaktion in Bewegung, pages 187–196, 2003. [9] M. Hassenzahl, R. Kekez, and M. Burmester. The importance of a software’s pragmatic quality depends on usage modes. In H. Luczak, A. E. Cakir, and G. Cakir, editors, Proc. of the 6th international conference on Work With Display Units WWDU 2002, pages 275–276, Berlin, 2002. ERGONOMIC Institut f¨ ur Arbeits- und Sozialforschung. [10] M. Hassenzahl and A. Monk. The inference of perceived usability from beauty. Human-Computer Interaction (accepted), 2010. [11] K. S. Hone and R. Graham. Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI). Nat. Lang. Eng., 6(3-4):287–303, 2000. [12] K. Hornbæk and E. L.-C. Law. Meta-analysis of correlations among usability measures. In Proc. of CHI, pages 617–626. ACM, 2007. [13] J. Kirakowski and M. Corbett. SUMI: The Software Usability Measurement Inventory. British Journal of

Table 7: Questionnaires assessing quality aspects, with corresponding interaction parameter or appropriate subscales given in brackets. differences of the interfaces, which is in contrast to findings published by Wechsung and Naumann [35] but supports the result reported by Tullis and Stetson [32]. Respective the identification of problematic factors, interaction parameters can be called upon. The analysis of the interaction based on the parameters ‘modality choice’ and ‘recognition rate’ shows that participants prefer the smartphone-based interaction although gesture recognition rate is low. As a consequence emphasis should be laid on improving the recognition rate when proceeding with the engineering cycle. To waive this interface due to unsatisfying user ratings would be the wrong decision.

CONCLUSION The analysis of questionnaires and interaction parameters presented in this paper should serve the purpose of better understanding quality aspects, such as ‘pragmatic quality’ and ‘hedonic quality’ in the field of multimodal systems. Furthermore, recommendations can be given to meet two requirements connected with evaluation: (1) the comparison of two or more systems or system versions (to verify assumed improvements) or (2) the identification of factors or components in the system that can be improved. We found that the AttrakDiff, the SUS, as well as the USE are suitable for the usability evaluation of systems with multimodal interfaces. But the selection of an appropriate questionnaire for the evaluation depends on the aim of the evaluation: • According to our results, the SUS would be suitable for a mere comparison of systems, interfaces or system versions as only the differences measured with the SUS are significant between all three interfaces. • If more information about the reasons for an unsatisfying usability result is required, the AttrakDiff would be a better choice, as it explicitly distinguishes between pragmatic and hedonic aspects. • The additional analysis of interaction parameters offers an even better identification and understanding of usability problems, and allows to specifically improve problematic modules. Further research is still needed to better understand the concept of intuitive use, how this is linked to usability and to define interaction parameters related to this concept. The same holds for hedonic quality aspects, which gain in importance in explorativ settings applicable especially for entertainment systems.

293

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Educational Technology, (3):210–212, 1993. [14] C. K¨ uhnel, B. Weiss, and S. M¨ oller. Parameters describing multimodal interaction — definitions and three usage scenarios. In accepted for Interspeech 2010, 2010. [15] E. L.-C. Law, V. Roto, M. Hassenzahl, A. P. Vermeeren, and J. Kort. Understanding, scoping and defining user experience: a survey approach. In Proc. of CHI, pages 719–728, New York, NY, USA, 2009. ACM. [16] J. Lewis. Ibm computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction, 7(1):57–78, 1995. [17] R. L´ opez-C´ ozar Delgado and M. Araki. Spoken, multilingual and multimodal dialogue systems: development and assessment. John Wiley & Sons, Chichester, 2005. [18] A. M. Lund. Measuring usability with the USE questionnaire. Usability Interface, 8(2), 2001. [19] S. M¨ oller, K.-P. Engelbrecht, C. K¨ uhnel, I. Wechsung, and B. Weiss. Evaluation of multimodal interfaces for ambient intelligence. In Human-Centric Interfaces for Ambient Intelligence, pages 347–370. Elsevier, 2010. [20] S. M¨ oller, K.-P. Engelbrecht, and R. Schleicher. Predicting the quality and usability of spoken dialogue services. Speech Commun., 50(8–9):730–744, 2008. [21] S. M¨ oller, J. Krebber, A. Raake, P. Smeele, M. Rajman, M. Melichar, V. Pallotta, G. Tsakou, B. Kladis, A. Vovos, J. Hoonhout, D. Schuchardt, N. Fakotakis, T. Ganchev, and I. Potamitis. INSPIRE: Evaluation of a Smart-Home System for Infotainment Management and Device Control. In Proc. 4th Int. Conf. on Language Resources and Evaluation (LREC), volume 5, pages 1603–1606, May 2004. [22] A. Naumann and I. Wechsung. Developing usability methods for multimodal systems: The use of subjective and objective measures. In E. L.-C. Law, N. Bevan, G. Christou, M. Springett, and M. Larusdottir, editors, Proc. of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM), pages 8–12. Institute of Research in Informatics of Toulouse (IRIT), 2008. [23] A. Naumann, I. Wechsung, and J. Hurtienne. Multimodality, inclusive design, and intuitive use. In Proc. of the 23rd British Computer Society Human Computer Interaction Workshop and Conference (HCI 2009), 2009. [24] J. Nielsen and J. Levy. Measuring usability: Preference vs. performance. Communications of the ACM, 37(4):66–75, 1994. [25] M. Nielsen, T. Moeslund, M. St¨ orring, and E. Granum. A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In Proc. of Gesture-Based Communication in Human-Computer Interaction, 5th International Gesture Workshop, pages 409–420. Springer, 2003. [26] L. Nigay and J. Coutaz. A design space for multimodal systems: concurrent processing and data fusion. In Proc. of the INTERACT and CHI, pages 172–178, 1993.

[27] S. Oviatt. Multimodal interfaces. pages 286–304, 2003. [28] T. Paek. Empirical methods for evaluating dialog systems. In Proc. of the Second SIGdial Workshop on Discourse and Dialogue, pages 1–9, Morristown, NJ, USA, 2001. Association for Computational Linguistics. [29] L. M. Reeves, J. Lai, J. A. Larson, S. Oviatt, T. S. Balaji, S. Buisine, P. Collings, P. Cohen, B. Kraal, J.-C. Martin, M. McTear, T. Raman, K. M. Stanney, H. Su, and Q. Y. Wang. Guidelines for multimodal user interface design. Communications of ACM, 47:57 – 59, 2004. [30] J. Sauro and E. Kindlund. A method to standardize usability metrics into a single score. In Proc. of CHI, pages 401–409. ACM Press, 2005. [31] N. Tractinsky. Aesthetics and apparent usability: empirically assessing cultural and methodological issues. In CHI ’97: Proc. of the SIGCHI conference on Human factors in computing systems, pages 115–122, New York, NY, USA, 1997. ACM. [32] T. Tullis and J. Stetson. A comparison of questionnaires for assessing website usability. In Proc. of Usability Professionals Association (UPA). [33] M. Turunen, J. Hakulinen, A. Melto, T. Heimonen, T. Laivo, and J. Hella. Suxes – user experience evaluation method for spoken and multimodal interaction. In Proc. of Interspeech, pages 2567–2570, 2009. [34] M. A. Walker, D. J. Litman, C. A. Kamm, and A. Abella. PARADISE: a framework for evaluating spoken dialogue agents. In Proc. of the ACL/EACL 35th Meeting of the Assoc. for Computational Linguistics, Madrid, pages 271–280, 1997. [35] I. Wechsung and A. Naumann. Established usability evaluation methods for multimodal systems: A comparison of standardized usability questionnaires. In Proc. of PIT. Heidelberg: Springer, 2008. [36] J. O. Wobbrock, M. R. Morris, and A. D. Wilson. User-Defined Gestures for Surface Computing. In Proc. of CHI, pages 1083–1092. ACM Press, 2009.

294

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Rise of the Expert Amateur: DIY Projects, Communities, and Cultures Stacey Kuznetsov & Eric Paulos Human-Computer Interaction Institute, Carnegie Mellon 5000 Forbes Avenue, Pittsburgh, PA, USA {stace, paulos}@cs.cmu.edu ABSTRACT

This paper presents a large-scale study of Do-It-Yourself (DIY) communities, cultures and projects. We focus on the adoption and appropriation of human-computer interaction and collaboration technologies and their role in motivating and sustaining communities of builders, crafters and makers. Our survey of over 2600 individuals across a range of DIY communities (Instructables, Dorkbot, Craftster, Ravelry, Etsy, and Adafruit) reveals a unique set of values, emphasizing open sharing, learning, and creativity over profit and social capital. We derive design implications to embed these values into other everyday practices, and hope that our work serves to engage CHI practitioners with DIY expert amateurs. Author Keywords

DIY, online communities, motivations of contributors ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

DIY (Do It Yourself) practice predates recorded history as human survival itself often relied on the ability to repair and repurpose tools and materials. For hundreds of years, people have been fixing water leaks, remodeling their homes and decorating their clothes without hiring professional plumbers, architects or designers. Modern societies oppose the principle of self-reliance with massproduction and consumer economy. Tangible things can be bought. Professionals can be hired to build and repair. Artists can be employed to decorate or customize. Nevertheless, people all over the world continue to create and modify objects with their own hands, ranging from knitting, to gadgets, music, and software. We define DIY as any creation, modification or repair of objects without the aid of paid professionals. We use the term “amateur” not as a reflection on a hobbyists’ skills, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

Figure 1. Amateur radio operator in 1920’s (left) and typical punk culture zines from 1970’s (right).

which are often quite advanced, but rather, to emphasize that most of DIY culture is not motivated by commercial purposes. Over the past few decades, the integration of social computing, online sharing tools, and other HCI collaboration technologies has facilitated a renewed interest and wider adoption of DIY cultures and practices through (1) easy access to and affordability of tools and (2) the emergence of new sharing mechanisms. We begin this paper with a brief historic overview of DIY cultures. We then present a study of six modern DIY communities, providing insight into DIY as cultural movement and the technologies that support it. We highlight opportunities for HCI researchers to engage with DIY practices, suggesting design implications in the domains of physical and digital identity management, expressive knowledge transfer and design studio culture. DIY CUTLURES: HISTORIC OVERVIEW

One of the earliest “modern era” DIY communities formed among amateur radio hobbyists in the 1920’s (Figure 1). These hobbyists relied on amateur handbooks, which stressed “imagination and an open mind” nearly as much as the technical aspects of radio communication [20]. Ham radio enthusiasts often met in person to discuss their work as well as unrelated social subjects. They continued to thrive rebelliously during World War II, when a ban was placed on amateur radio communication. Rebellious attitudes continued to pervade pirate radio stations of the 1960’s and handmade ‘zines’ expressing the punk aesthetic in the 1970s’ (Figure 1) [37]. Later in the 1980’s, low-cost MIDI equipment enabled people without formal training to record electronic music, evolving into the rave culture of the 1990’s [26]. During this time, computer hobbyists also formed communities to create, explore and exploit software systems, resulting in

295

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Open source software is an artifact created and maintained by decentralized contributors, usually as a hobby. According to the Free/Libre and Open Source Software (FLOSS) study of 2,784 open source developers, the majority of respondents contribute in order to “learn and develop new skills” [18], as well as to share “their knowledge and skills” and improve “FS/OS products of other developers”. Wikipedia showcases another collection of artifacts: millions of encyclopedia-style articles. A survey by Nov classifies motivations of contributors into six categories, suggesting that fun and ideology (“information should be free”) are the top two motivations of contributors [30]. Seti@Home harnesses volunteers’ computers to analyze data in search of extraterrestrial activity, with members creating an intangible artifact of largescale computation. Over 58% of SETI participants indicate that they contribute to “find ET for the good of humanity”, while over 17% emphasize “keeping my computer productive”1.

Figure 2. Images from DIY Communities: ‘electric discharge through air’, presented at Dorkpot Sydney (top left), chocolate dipped macaroons sold on Etsy (top right), Instructables tutorial for a servo robot (bottom left), “Chain of Command” scarf on Craftster (bottom right).

Lastly, we draw a parallel between DIY objects and more personal, digital artifacts created through blogging and image tagging. Ames’ and Naaman’s study of tagging patterns in Flickr and ZoneTag suggests a taxonomy of motivations [2], including personal organization, communication with other community members, and organization for the general public. Nardi, et al. emphasize five motivations of bloggers, among them: personal documentation, emotional outlet, muse (formulating ideas in writing), and supporting specific communities such as poets or musicians [29].

the Hacker culture. Today’s DIY cultures reflect the anticonsumerism, rebelliousness, and creativity of earlier DIY initiatives, supporting the ideology that people can create rather than buy the things they want. Modern DIY Communities

Recent breakthroughs in technology afford sharing such that anyone can quickly document and showcase their DIY projects to a large audience. An emerging body of tools allows enthusiasts to collaboratively critique, brainstorm and troubleshoot their work, often in real-time [e.g. 1,11,32]. This accessibility and decentralization has enabled large communities to form around the transfer of DIY information, attracting individuals who are curious, passionate and/or heavily involved in DIY work.

We hypothesize that DIY communities are also driven by scientific pursuit, personal organization, community values and intrinsic enjoyment of creating DIY objects. However, unlike communities that revolve around the creation of digital artifacts such software, encyclopedia articles, or blogs, DIY communities showcase meta information: personal experiences and knowledge from creating physical objects, projected into the public sphere.

Thousands of DIY communities exist today, varying in size, organization and project structure. Some allow members to contribute asynchronously on a variety of topics [22], while others focus on specific projects such as knitting, crocheting [32] or hip craft [11]. Some revolve around smaller in-person gatherings [12] and some enable hobbyists to trade or sell their projects [14].

Prior Work Exploring DIY Practices

Tailoring communities surrounding CAD environment, software and PC customization focus on personal knowledge sharing. However, unlike DIY communities, which embrace sharing by professionals and amateurs alike, knowledge in tailoring communities (e.g. customization files) is dissipated by a few experts, while the majority of users merely adapt this information to their needs [17,25,36]. A popular workshop held at CHI2009 initiated an early dialog between the HCI and DIY communities [8]. In addition, prior work provides insight into DIY practices: Torrey et al. explore information seeking among crafters [34], while O’Connor provides an ethnographic study of glassblowing [31].

RELATED WORK

An extensive body of work explores online communities in the domains of gaming (World of Warcraft), compiling objective knowledge (Wikipdia), open source software, and social networking (Facebook, MySpace), to name a few [3,13,24,28]. Although there has been no large-scale study of online DIY communities, we draw from several prior findings, focusing on communities that produce ‘artifacts’- tangible or digital objects such as documents, software, or images. Such artifacts can be compared to DIY ‘objects’- items that are created, modified or refurbished by hobbyists and non-experts.

1 Seti@Home. http://seticlassic.ssl.berkeley.edu/polls.html, accessed May 5, 2010.

296

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Buechley et al. integrate hardware electronics into textiles to make e-textile technology available to non-experts [7], and Rosner, et al. explore IKEA hacking practices [33]. Lastly, Torrey et al. focus on How-To documentation [35], identifying several motivations for sharing: personal documentation, online identity, and finding employment. Unlike prior work, which focuses on one DIY community or skillset, our research explores DIY as a broad phenomenon spanning a variety of domains- art, electronics, craft, music, etc.

doing strange things with electricity” is applicable to most speakers who include fire artists, electronics enthusiasts, industrial designers, hackers, and musicians. Adafruit

Founded by Limor Fried, Adafruit is an online distributor (seller) of electronic parts and kits. These items are also featured at ladyada.net [1], a sister site hosting images and descriptions of projects that are documented by Limor herself. Adafruit hosts a number of general tutorials, with an emphasis on low-cost materials and easy-to-follow methods. The site thus serves as a portal for nearly 8,000 registered users who are curious about or struggling with electronics projects. A collection of forums enables members to discuss and troubleshoot their work.

RESEARCH SCOPE

We present the motivations, practices and sharing mechanisms that support DIY as a cultural movement. Driven by non-experts, this movement embodies creation, sharing and discussion of DIY practices from crocheting and design to robotics and auto repair. We formally define community as a group of people who share common goals and interests- communicating through mediums online and in person. We focus on six communities as a sample of the diverse materials, practices and sharing mechanisms among DIY practitioners. In selecting these communities we hope to capture DIY as a multi-faceted movement which invites all practitioners- knitters, roboticists, fire artists, mechanics, designers, hackers, musicians, etc.- to share ideas through a variety of mediums, including forums, instructions, images, video, and face-to-face meetings. We detail the specific structure and focus of each selected community below.

Ravelry

Founded by Casey and Jessica Forbes in 2007, Ravelry is an online community of knitters, crocheters, and other artists who work with yarn and dyes. The site aims to help enthusiasts “keep track of their yarn, tools, and pattern information, and look to others for ideas and inspiration” [32]. Unlike other communities, Ravelry content is visible only to registered users. The site boasts over 300,000 registered users despite that fact that one must receive an invitation in order to join. Members can upload images and descriptions of their projects, as well as purchase and sell related supplies such as yarn and dyes. Numerous forums also host general discussion and questions. Craftster

Instructables

This online community also revolves around craft such as knitting and crocheting, with an emphasis on sharing “hip, off-beat, crafty DIY projects” [11]. Founded by Leah Kramer in 2007, Craftster allows everyone to view the contents of the site, and boasts over 700,000 unique readers every month, and more than 170,000 registered users. All communication occurs through forums, which are classified into thirty major categories, including pottery, cooking, crochet, jewelry and trinkets, and image reproduction techniques. Members can share their work by uploading photos, descriptions and tutorials, as well as commenting and asking questions about other projects, ideas and techniques. In addition, Craftster often organizes challenges with winning projects shown as “Featured Projects” on the front page.

Self-described as a “web-based documentation platform where passionate people share what they do and how they do it”, Instructables was launched in 2005 by Eric Wilhelm of MIT Media Lab [22]. DIY projects are shared in a step-by-step instruction format, often accompanied by images and video. With over half a million users, projects range from educational, to practical, to whimsical, including “Building a Medieval Gauntlet”, “Making Simple PVC Flute”, “Controlling an Arduino Microcontroller with a Cellphone” or “How to Kiss”. Users are able to rate, critique, question and comment on each project, leave forum posts, and submit general public questions and answers. Moreover, each member profile contains an “Orangeboard” allowing users to post comments to each other. The site hosts several specialinterest contests every month, awarding winners with online badges, T-shirts and sometimes equipment.

Etsy

Etsy is self-described as an “online marketplace for buying & selling all things handmade” [14], striving to financially empower DIY enthusiasts. Projects are thus posted as catalogue items with images, descriptions, and prices. Initiated by Robert Kalin in 2005, the site has more than 70,000 registered users. In November of 2007, 300,000 items were bought through Etsy, amounting to

Dorkbot

Dorkbot consists of groups of people who meet in person to discuss projects ranging from ‘electronic art’ to new gadgets [12]. The first Dorkbot was founded by Douglas Repetto of Columbia University Computer Music Center in 2000, and has grown to include sixty six active chapters located all over the world, at the time of writing. During meetings, several speakers (who are selected by meeting coordinators) present their work, followed by questions, critique and discussion. The motto of Dorkbot, “people

297

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

more than $4.3 billion dollars in sales1. Etsy also offers a variety of community resources, including forums, live chat, a news blog, and portals to bring DIY’ers together based on interests and location. Moreover, Etsy’s Virtual Labs offer weekly online classes on a variety of topics. STUDY METHODOLOGY

Our methods consist of an online survey distributed to the six selected DIY communities, followed by a detailed questionnaire sent to a random subset of participants. Our survey, designed to be completed within 10 minutes (all questions optional), was publicized through a variety of mediums, including message boards, mailing lists and forum posts in each of the six DIY communities. We now discuss our data collected from 2608 respondents over the course of 14 months (with the bulk of the responses arriving within the first 2 weeks of posting the survey). In addition, we leverage 5 qualitative responses to our follow-up questionnaire to explore some of the more fascinating aspects of our numeric survey data. We introduce our results by first discussing our participants. We then detail our findings across 3 areas: (1) participatory involvement in and motivations for contributing to DIY communities, (2) DIY projects and practices, and (3) sharing DIY projects with DIY communities.

Figure 3. Number of respondents involved in each community, by time involved.

insight into this overlap. One participant emphasizes the value of idea exchange amongst people of diverse backgrounds involved in different DIY communities: “One of the benefits of being involved in more than one DIY community is to be able to [ex]change ideas with so many different persons, with different technical, artistic and professional backgrounds.” Another respondent highlights that communities provide different ‘audiences’, which in turn allow develop both creative and business goals: “I use some communities to connect with other business owners. It makes exchanging ideas easier, promoting products since we can cross promote or pool our designs together to create a larger promotional event. Other communities I use to better connect with my end users or buyers. Site where I can post my creations and get feedback from the community.”

PARTICIPANTS

We collected 2608 responses, with participants’ ages from 18 to 95. The response rate is overwhelmingly female (2287 female, 186 male, 11 transgender), perhaps due to the large number of respondents from knitting and crocheting communities, and we address this phenomenon in the limitations section. About half (49%) of the respondents hold an undergraduate degree, 21% completed a Masters, 23% finished high school, and just over 6% earned a higher degree (PhD, JD, or M.D.). Educational backgrounds range from nursing, theatre, engineering, art and zoology to name a few.

Lastly, one participant notes the importance of size: “Each has its own 'personality,' for example, one is more a group of friends who happen to knit, so we do it together. From that group, I get not only knitting support and advice, but life advice and companionship… As for an online community (Ravelry), that’s just a window into the entire world… I get new, different, fancy, fun, perspective, enlightenment.”

Overlap across DIY Communities

Figure 3 shows the number of survey respondents by time involved in each of the six DIY communities. Despite the large response pool, less than 20 participants belong exclusively to only one of Instructables, Adafruit, Dorkbot or Etsy. Respondents from Ravelry and Craftster are somewhat separable (149 and 75 unique members respectively), but their data reflects trends of the group at large. Moreover, participants from all six of the studied communities indicate involvement in other DIY groups, including Flickr, LiveJournal, Yahoo Groups, ThreadBanger, Make Magazine, Knitter’s Review, deviantART, Cut Out + Keep, and Crochetville. Qualitative data from our follow-up questions offers

Since our results are not separable by community, we present survey data as a reflection of the DIY movement at large- a phenomenon that spans a variety of domains and sharing mechanisms (and we discuss the limitations

1

http://www.nytimes.com/2007/12/16/magazine/16Craftst.html?_r=3&oref=slogin&ref=magazine&pagewanted=all, accessed May 9, 2010.

Figure 4. Contributions to DIY communities, by frequency.

298

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 6. Participants’ DIY project categories. Figure 5. Motivations for contributing to DIY communities.

of this approach later).

online reputation”, with 68% and 60% disagreeing with each motivation, respectively.

CONTRIBUTIONS TO DIY COMMUNITIES

Question Answering as an Instrument of Learning

Over 90% of our respondents contribute to DIY communities through questions, comments and answers, with 45% of participants responding to others’ questions and 43% posting comments or questions on a daily or weekly basis (Figure 4). While nearly 87% of participants also post images of their projects at least once a year, much fewer respondents showcase personal work through step-by-step instructions and videos. In particular, videos are the rarest contribution (more rare than in-person interactions) with less than 8% of participants sharing videos at all, and only 2% sharing videos on a daily, weekly or monthly basis. Surprisingly, despite the fact that only 5% of respondents are members of Dorkbot, the only community that officially revolves around in-person meetings, a third of respondents (34%) attend in-person meetings and over a quarter (26%) present their work inperson at least several times a year. Qualitative responses suggest that these meetings range from “a group of friends” to informal “knit-along’s”, to larger “evening gathering[s] for the community”, often organized outside of the six communities from our study.

How is responding to others’ questions (most frequent contribution) related to learning and inspiration (most supported motivations)? Following up with our respondents, we simply asked: why do you answer questions in DIY communities? One participant explained: “By responding, I have also gotten feedback on what I posted, and in at least 2 cases, was able to correct technique that (it turns out) I was doing wrong.” In fact all respondents suggested that the act of answering questions helps learning. Another participant explains: “It’s like that saying that you learn more by teaching and sharing with others. Every time I pass on a little bit of information to someone else, it helps to ingrain that knowledge in my head, even spur on a desire to learn more.” In addition, participants also highlighted the value of teaching others (“the other reason I respond to questions/comments is that my approach may help someone else”), as well as connecting with people in the community (“to pass on a little bit of wisdom or knowledge”).

Motivations for Contributing to DIY Communities

Above all else, our participants contribute to DIY communities in order to get “inspiration and new ideas for future projects” (81% strongly agree, 16% agree) and to “learn new concepts” (68% strongly agree, 29% agree). Participants also highlight motivations revolving around information exchange: receiving feedback on personal projects and educating others and both supported by 77% of respondents. Surprisingly, while meeting people with similar interests is the third most supported motivation (79% agree or strongly agree), giving back to the community is the third least supported, with only 51% of all respondents agreeing or strongly agreeing. A large portion of free responses emphasizes fun as a motivation: “have fun!” or “it’s fun!” Other comments revolve around learning, for instance: “to learn new techniques”, and community bonds: to “socialize” or “to feel connected to other like-minded people”. The majority of participants are not driven by “finding employment” or “improving

DIY WORK AND PROJECTS

In the second phase of our study, we explore the practices and motivations behind DIY work. The vast majority of our respondents (90%, 2285 in total) contribute to DIY projects. The majority (94%) of participants who do DIY

Figure 7. Costs and earnings per typical DYI project.

299

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 8. Motivations for contributing to DIY projects.

Figure 10. Deterrents for sharing DIY work with DIY communities.

work contribute to craft projects such as knitting or sewing (Figure 6). Other popular categories include food/cooking (51%), art (44%), and home improvement (35%). Most respondents contribute to more than one category, and all categories significantly overlap with craft (by 70% or more) and cooking (58% or more). Electronics is an exception, overlapping with craft by only 43% and cooking by 40%. Free response project categories range from “gardening” to “photography” to “automotive” among others.

communities in order to meet people with similar interests. One participant offers insight into this discrepancy during our follow-up correspondence: “Working alone allows you full expression of your individual creativity. Sharing it with other people with similar interests can provide constructive criticism from people who understand what you want to do.” Another respondent suggests that the complexity of a project determines if she works alone or with others:

Nearly two thirds of respondents spend between $11 and $50 on a typical project, and the vast majority (84%) does not get paid for their projects (Figure 7). Project cost correlates with project completion time (more than 87% of participants who spend under $25 on a project finish it in under 30 hours, while more than half of projects that cost above $500 require over 100 hours to finish). Conversely, how often participants work on DIY projects does not vary with project cost, such that 40% of all participants contribute to DIY projects a few times a week and another 40% do DIY work at least once a day, regardless of project cost. For 66% of respondents, a typical project takes less than 30 hours to finish (with 21% of respondents spending 1-5 hours, 24% spending 610 hours, and 31% spending 11-30 hours). There is no direct correlation between the time spent and amount earned per project.

“Personally, I waffle between wanting to practice my craft alone and practicing in the company of others. It depends on the project I have going. Easy projects lend themselves to schmoozing; more complicated projects require ‘me time.’" SHARING PROJECTS WITH DIY COMMUNITIES

More than 90% of respondents who work on DIY projects share at least some of their projects (Figure 9). Our data suggests a correlation between the percentage of projects shared and the amount of time spent on a typical project: less than 70% of participants who spend under 5 hours on typical projects share any of their work, while over 95% of respondents who work on longer projects (more than 30 hours) tend to share, with more than 10% sharing all their work.

Motivations for Contributing to DIY Projects

Lack of time is the primary reason for not sharing DIY projects with DIY communities, as indicated by over half of our responded (Figure 10). Other common deterrents are respondents’ negative assessments of their projects (lack of creativity, novelty or complexity), each selected

An overwhelming majority (97%) of our participants work on DIY projects in order to “Express myself/be creative”, with over 68% strongly agreeing with this motivation (Figure 8). “Learn new skills” is the second most supported motivation for doing DIY work (52% agree, 39% strongly agree). The least popular reason is to “Gain internet fame or reputation” with more than 70% of respondents disagreeing or strongly disagreeing with this motivation. “Make money” is the second least popular motivation (25% disagree, 15% strongly disagree). Interestingly, only 40% of participants contribute to DIY projects to work/spend time with friends, despite the fact that nearly 80% of respondents contribute to DIY

Figure 9. DIY projects shared with DIY communities.

300

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

towards Ravelry (300,000 members) and Craftster (170,000 members). However, size does not explain the relatively small participation from Instructables, which, at 500,000 members, is larger than both Ravelry and Craftster. Perhaps our overwhelmingly female response rate is due to a female majority in Ravelry and Craftster (71% and 68% respectively). We note that our methodology is not immune the common limitations of survey methods: self-selection bias, underreporting of egotistic values, categorizing participants’ motivations into pre-defined lists, etc. While our data depicts the practices and values of DIY cultures at large, future work can pursue inter-community comparisons: for instance, do users belonging exclusively to Etsy express financial concerns above creativity and learning?

Figure 11. Aspects of DIY communities that participants find influential for their work.

by about 23% of respondents. Less than 10% cite poor editing or uploading skills as a reason for not sharing, and slightly more respondents (15%) indicate that they do not have the right equipment to document their work.

DISCUSSION

We now reflect on the broader processes and values that underlie DIY cultures, highlighting four themes that distinguish DIY communities from the other, more widely-studied communities. In doing so, we hope to entice HCI researchers to engage with DIY practitioners as an emerging group of builders, tinkerers, and ‘experts’ who create novel objects and interfaces that can be imported into HCI work. Furthermore, DIY communities represent early adopters of new cultural practices that reform and repurpose technologies, exemplifying a vibrant creative spirit often misunderstood and certainly understudied by HCI communities.

Lastly, when asked which aspects of DIY communities are most influential for their work, the majority of respondents emphasize images of other projects (over 60%), followed by step-by-step instructions (over 40%). Surprisingly, feedback on projects is least found “very influential” by less than 20% of participants, despite the fact that it is one of the most frequent contributions. Inperson interactions are the second least influential for DIY work, perhaps due to the fact that they are rare as well as the social (rather than purely work-oriented) focus of many in-person events- e.g., smaller knitting groups. Referring to one such gathering, a respondent explains:

Low Barrier to Entry

With the majority of DIY projects costing less than $50, a low financial threshold enables people to work with a range of materials across different project domains. Just as most participants belong to several communities, many also work on more than one type of project, ranging from craft to cooking to electronics. We consider the overlap between communities and projects to be an exciting finding: accessibility of information and resources blurs domain distinctions, inviting knitters to tinker with electronics, roboticists to practice crocheting and mechanics to explore music. DIY communities lower the barrier to entry into different project domains by enabling what one respondent describes as “exchange of ideas with so many different persons with different technical, artistic and professional backgrounds”. DIY communities thus invite individuals across all backgrounds and skill levels to contribute, resulting in: 1) rapid interdisciplinary skill building as people contribute and pollinate ideas across communities and 2) increased participation supported by informal (“anything goes”) contributions such a comments, questions and answers.

“My in-person meetings are with friends that share my interests, not strangers. It provides a chance to socialize while sharing an interest.” Another respondent notes that meetings are used to transform online connections into personal relationships: “When I get to meet an individual that I've talked to online in-person, it really helps me to put a face to the name!” Thus, participants seek in-person contact to fulfill social as well as purely DIY-related goals. LIMITATIONS

The six DIY communities selected for our study provide a representative cross-sample of the methods and materials employed by DIY practitioners: Instructables- a large site showcasing a projects in a step-by-step format; Adafruita small community of electronics enthusiasts; Ravelry- a large portal for knitters, crocheters and artists; Craftster- a community of “hip” and “crafty” practitioners; Etsy- an online marketplace for handmade work; and Dorkbotartists who present ideas in-person. To gather unbiased probability sampling, we posted identical recruitment text on each community’s discussion forums and our data somewhat reflects site statistics: fewer responses from smaller communities- Dorkbot and Adafruit, and a skew

Learning

The breadth of communities and projects that are often associated with a single person suggests widespread information exchange. DIY communities serve as instruments of learning through features such as discussion forums, the ‘instructable’ format, images and

301

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

video. Our data reveals question asking and answering as the core process behind the propagation of methods and ideas. As one respondent pointed out, participants tend to “learn more by teaching and sharing with others”. We contrast this mechanism with the more ‘traditional’ dissipation of knowledge in academic fields: while research papers certainly do increase the fundamental state of the art, they often alienate a large portion of the general population by ‘talking at’ rather than ‘talking with’ the audience. Conversely, DIY communities entice learning by initiating a give and take dialogue between individuals across all backgrounds and skill levels. With ‘learning new concepts and skills’ as the second mostsupported motivation for both engaging in DIY communities and working on projects, DIY is a culture that aspires to explore, experiment and understand.

author’s name “indicates the status of the discourse within a society and culture”) [16]. By this view, DIY contributors become ‘authors’, and sharing takes on the form of ‘story-telling’ through a creative rhetoric. Again, we contrast this more natural approach of telling (for instance, showing a new gadget or sewing technique) with the more prevalent of top-down sharing mechanisms in CHI. With storytelling as a core process behind creative DIY sharing, we argue for importing similar practices into CHI (for instance, more emphasis on demos, etc.). Open Sharing

Motivations for contributing to DIY communities highlight information exchange as a core value: receiving feedback on projects, educating others, and showcasing personal ideas and skills are the top factors. Sharing is accessible to individuals of nearly any background, since lack of equipment or skill is not a significant barrier. Participants are willing to share both their projects (especially through images, which are uploaded by more than 80% of respondents at least once a year), as well as raw knowledge by commenting answering questions. Sharing extends beyond DIY work and methods, as nearly 80% of participants seek to ‘meet people with similar interests’, or as on participant stated, “to feel connected to like-minded people”. With nearly a third of all participants attending in-person meetings and almost a quarter presenting personal work at meetings at least once a year, in-person interactions serve as a “chance to socialize while sharing an interest”. Community togetherness compliments the predominantly solitary practice of doing DIY work: ‘working and spending time with my friends’ is not a popular motivation for contributing to DIY projects, thus suggesting that DIY is a culture that strives to share together while working alone.

Creativity

DIY communities and projects are driven by creativity. The vast majority of our respondents contribute to DIY communities not to gain employment, money, or online fame, but to express themselves and be inspired by new ideas. Moreover, creativity serves as a prerequisite for sharing, with nearly a quarter of the community refusing to share their work because it is self-perceived as uninteresting, not novel, or too simple. The types of projects showcased through DIY communities, ranging from robotic gardening tools to three-dimensional art quilts to ladybug cupcakes, embody the members’ drive for the unique, the whimsical and the artistic, to enable what one participant described as “full expression of your individual creativity”. This individual creativity echoes Csikszentmihalyi’s view of p-creativity, referring to individuals who are interesting, stimulating, and “experience the world in original ways” [9]. Personal creativity is, of course, different from historic genius- individuals such as Picasso or Einstein who impact our culture on a larger scale. DIY communities facilitate p-creativity by enabling millions of witty, curious, and enthusiastic contributors to share and draw from DIY knowledge. This knowledge affords inspiration and learning, which in turn lead to discoveries and innovations in personal DIY projects. According to Csikszentmihalyi, creative outcomes are validated by a “field of experts” who “recognize and validate the innovation”. DIY communities provide this mechanism through open sharing and feedback. The “field of experts” in DIY communities consists of hobbyists and enthusiasts who critique and learn from others’ work, giving rise to the expert amateur. With most respondents sharing some portion of their projects with DIY communities and more than half of participants contributing to a community at least a few times a week, sharing is the fundamental process that drives and validates DIY creativity.

DESIGN IMPLICATIONS AND FUTURE WORK

The principles embodied by DIY communities- low barrier to entry, learning, open sharing and creativity- can benefit a variety of other corporate, academic and nonprofit collaborative environments. These values drive the exchange of ideas that lead to new discoveries and innovations. We now highlight three areas of design implications for DIY and similar systems that leverage personal knowledge exchange. Integrating Physical and Digital Domains

A large body of literature in the social sciences explores material possessions as ‘extensions of self’: personal objects serve as points of reflection on one’s identity as well as expressions of idealized future goals and desires [4,10,23]. Our participants, who create and repurpose personal objects, use online communities to broadcast self-constructed material things into the public sphere. In doing so, they symbolically project personal goals, values and practices into the digital domain. These contributions remain detached from the physical objects and states that produce them. Simultaneously, participants seek physical, in-person meetings, which provide social intersections

Alternatively, Foucault discusses creativity as ‘author function’: the author produces a work but does not necessarily participate in its interpretation (e.g., the

302

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ranging from putting “a face to the name” to receiving “life advice and companionship”.

To compliment expressive idea exchange, future systems can explore light-weight mechanisms for gauging learning outcomes. While numerous contests and rating systems exist to indicate the popularity of online content, very few tools allow us to evaluate tacit knowledge transfer or learning experiences. Systems aiming to support scaffolding for knowledge exchange can allow users to quickly indicate how much they learned or search for content that other people have effectively learned from.

Our findings suggest a range of design opportunities for managing personal identities within and across physical and digital domains. Broadly speaking, new tools can serve to ensure continuity across online communities to which our respondents contribute, as well as in the physical world where they create material objects and seek personal contact. For instance, new systems can inform users when they are interacting with the same individual across different websites, or showcase one’s projects in single place even if they are initially shared in separate communities. Moreover, digital interactions can be exposed in physical spaces [e.g. 21] to reconcile online relationships with in-person experiences. Future work can import ongoing research of more sophisticated tools for identity management [e.g. 19] to explore the implications of linking virtual and physical domains: what are the social consequences of combining ‘personas’ across online and in-person communities? To what extent are users willing to share personal facts and objects across these domains, and how can identity management tools infer and address privacy concerns?

Supporting Iterative Studio Culture

Creativity- the core value embodied by DIY communities, simultaneously entices and deters participation: on one hand, our respondents want to share their projects to receive feedback and inspiration from the community; at the same time however, creativity is a filter for sharing work that is self-perceived as un-creative (not novel or uninteresting, etc). This ‘creativity barrier’ stems, in part, from current DIY sharing as a practice of showcasing functional and completed work. Broader participation can be encouraged by reframing DIY communities as instruments of iteration, critique and feedback on work in progress. With design studio culture as an emerging field in HCI [5,6,15], we emphasize the importance new mechanisms that elicit “feedback early and often”[6] in creative design processes.

New Forms of Knowledge Transfer

Our results reveal commenting, question asking and answering as the most frequent contributions in online DIY communities. Since these communities function as instruments of learning and inspiration, we emphasize new media that enable richer experiences of knowledge transfer beyond text-based question and answering. Despite the fact that many communities already support expressive mediums, for instance images, step-by-step instructions and video, and participants find these to be most influential for their work, such contributions are considerably less frequent than text-based posts. Videos in particular highlight this discrepancy: despite being the least common (under 2% of respondents upload a video more than several times a year), they are deemed ‘very influential’ by almost 17% of respondents.

DIY communities already do embody iterative feedback mechanisms through comments, forum posts and question/answer exchange. Many contribution formats are reminiscent of a studio culture, whereby users learn by ‘observing’ the ‘actions’ of others- tutorials, images, videos, etc. In the future, personal project sharing can draw from the successful contribution models of systems such as Flickr, Twitter, or Facebook, allowing for status updates, photostreams and similar ‘live’ formats. This approach can depict the current state of the work, enabling an iterative dialogue between the DIY’er, the DIY process, and the DIY community. Moreover, new tools can expose the role that smaller projects play in the ultimate creation of larger, high-profile work. For instance, a final robotics project may draw from one person’s image of a circuit, another contributor’s tutorial on motors, and a third individual’s video of vacuumforming. Future tools can focus on revealing the iterative processes between projects people to highlight that creativity is not confined to large-scale, complex work that tends to receive the majority of the “creative” credit.

With lack of time as the main deterrent for sharing DIY work, there is a tradeoff between the time required to make a contribution and the extent to which it facilitates knowledge transfer. Videos, for instance, require a long time to edit, but can influence the viewer in at least three powerful ways: 1) by physically illustrating the steps required to create an artifact; 2) by showcasing a new idea in its functional form; and 3) by directly ‘speaking’ to and engaging with the audience. Future work can focus on media for expressive knowledge transfer. In the case of video, for example, new tools can offer fast and intuitive editing, non-linear traversal, descriptive overlays, tagging, and linkage to related projects. Images and tutorials can be made more powerful through voice annotation, interactive features, and graphical search, among others.

CONCLUSION

We discuss DIY as a vibrant culture with a long history of learning, creating and sharing. Our study of six DIY communities suggests that these values are embedded in everyday practices and supported by the technologies that bring DIY communities into being. Drawing from numeric and qualitative data, we present opportunities for identity management across digital and physical domains, expressive knowledge transfer tools, and systems to support iterative studio practices. Above all, we argue for

303

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

increased engagement between CHI practitioners and DIY expert amateurs, and hope that this work serves to import DIY culture into CHI and vice versa.

19. Gross, B. M. and Churchill, E. F. Addressing constraints: multiple usernames task spillage and notions of identity. In CHI '07 Extended Abstracts on Human Factors in Computing Systems, San Jose, CA, USA, 2393-2398 (2007).

REFERENCES 1. Adafruit. http://www.adafruit.com/, accessed March 5, 2009

20. Haring, K. Ham Radio's Technical Culture. Cloth (2008). 21. Hosio, S., Kukka, H., Riekki, J. Leveraging social networking services to encourage interaction in public spaces. In Proceedings of the 7th international Conference on Mobile and Ubiquitous Multimedia MUM '08, 2-7 (2007).

2. Ames, M., Naaman, M. Why we tag: motivations for annotation in mobile and online media. Conference on Human Factors in Computing Systems, 971-980, (2008). 3. Beschastnikh, I. K. (2008) Wikipedian self-governance in action: Motivating the policy lens. The International Conference on Weblogs and Social Media, Chicago.

22. Instructables. http://www.instructables.com, accessed May 5, 2010.

4. Belk, R. W. (1998) Possessions and the Extended Self. Journal of Consumer Research, 15, 2, 139-168.

23. Kleine, S., Baker, S. An Integrative Review of Material Possession Attachment. Academy of Marketing Science Review. 1-39, (2004).

5. Blevis, E., Lim, Y.K., Stolterman, E., Makice, K. (2008). The Iterative Design of a Virtual Design Studio. Techtrends: A Journal of the Association for Educational Communications and Technology. Springer US. Volume 52, Number 1. 74-83.

24. Lampe, C., et al. (2006) A Face(book) in the crowd: social Searching vs. social browsing . CSCW’06.

6. Blevis, E., Lim, Y., Stolterman, E., Wolf, T. V., Sato, K. (2007). Supporting design studio culture in HCI. In CHI '07 Extended Abstracts on Human Factors in Computing Systems (CHI '07), 2821-2824.

26. McKay, G. Party & Protest in Nineties Britain. Verso, (1998).

25. Mackay, W., Triggers and Barriers to Customizing Software, Proc. ACM CHI, 153-160, (1991).

27. Morch, A. Three Levels of End-User Tailoring: Customization, Integration, and Extension. In Computers and Design in Context. M. Kyng & L. Mathiassen (eds.). MIT Press, 51-76, (1995).

7. Buechley, L. and Eisenberg, M. Fabric PCBs, Electronic Sequins, and Socket Buttons: Techniques for E-textile Craft. Journal of Personal and Ubiquitous Computing, 13(2), pp 133-150. 2009.

28. Nardi, B. et al. Strangers and friends: Collaborative play in world of warcraft. CSCW 2006: Proceedings ACM conference on computer-supported cooperative work. ACM Press, (2006).

8. Buechley, L., Paulos, E., Rosner, D., Williams, A. DIY for CHI: Methods, Communities, and Values of Reuse and Customization, Workshop at ACM SIGCHI, 2009.

29. Nardi, B., Schiano, D. J., Gumbrecht, M., Swartz, L. Why We Blog. Communications of the ACM, 47(12), 41-46. 2004.

9. Csikszentmihalyi, M. Creativity: Flow and the Psychology of Discovery and Invention. HarperCollins Publishers, Inc., New York, NY, 1997.

30. Nov, O. What motivates Wikipedians? Communications of. ACM, 50(11), 60-64. 2007.

10. Csikszenthmihalyi, M., Rochberg-Halton, E. The Meaning of Things: Domestic Symbols and the Self. Cambridge University Press, Cambridge, (1981). 11. Craftster. http://www.craftster.org/, accessed May 1, 2010

31. O’Conner, E. (2005) Embodied knowledge: The experience of meaning and the struggle towards proficiency in glassblowing. Sage Publications, Vol. 6, 183-204.

12. Dorkbot, http://dorkbot.org/, accessed May 5, 2010

32. Ravelry. https://www.ravelry.com/, accessed May 5, 2010.

13. Dwyer, C. Digital Relationships in the "MySpace" Generation: Results From a Qualitative Study. 40th Annual Hawaii International Conference on System Sciences (HICSS'07), (2007).

33. Rosner, D.K., Bean, J. (2009) Learning from IKEA Hacking: "I'm Not One to Decoupage a Tabletop and Call It a Day". Proceedings of CHI'09. 34. Torrey, C., Churchill, E., McDonald, D. Learning How: The Search for Craft Knowledge on the Internet. Proceedings of CHI'09: ACM Conference on Human Factors in Computing Systems, 1371-138, (2009).

14. Etsy. http://www.etsy.com/, accessed March 5, 2009 15. Fallman, D. (2003). Designing Design: Design-oriented Human-Computer Interaction. CHI’03, Ft. Lauderdale, FL.

35. Torrey, C., McDonald, D. W., Schilit, B. N., Bly, S. How-To Pages: Informal Systems of Expertise Sharing. Proceedings of European Conference on Computer Supported Cooperative Work. Springer, 391-410, (2007).

16. Foucault, Michel (1969), "What is an Author?", in Harari, Josué V., Textual Strategies: Perspectives in PostStructuralist Criticism, Ithaca, NY: Cornell University Press, 1979

36. Trigg, R., Bodker, S., From Implementation to Design: Tailoring and the Emergence of Systematization. CSCW Proceedings. ACM CSCW, 45-54, (1994).

17. Gantt, M., Nardi, B. A. Gardeners and Gurus: Patterns of Cooperation among CAD Users. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 107117, (1992).

37. Wright, M. A. (1998) The Great British Ecstasy Revolution. In DIY Culture: Party and Protest in Nineties Britain, edited by McKay, George. London: Verso, 228-242.

18. Ghosh, R. A. (2002) Free/Libre and Open Source Software: Survey and Study, Part IV: Survey of Developers. International Institute of Infonomics University of Maastricht, Netherlands.

38. Yamauchi, Y., Yokozawa, M., Shinohara, T., Ishida, T. (2000). Collaboration with Lean Media: how open-source software succeeds. CSCW '00.

304

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Do people say what they think: Social conformity behavior in varying degrees of online social presence Lieve Laporte & Christof van Nimwegen K.U.Leuven – CUO – IBBT Parkstraat 45 / 3605 3000 Leuven, Belgium [email protected] [email protected]

Alex J. Uyttendaele Eindhoven University of Technology P.O. Box 513 5600MB Eindhoven, the Netherlands [email protected]

ABSTRACT

For example, a fast growing number of people use live video chat such as Skype, social network sites such as Facebook or share content on portals such as YouTube.

In recent social media internet applications, many activities consist of voting, ranking, commenting and sharing. People using these applications can experience the social presence and influences of others, just as in real life. The difference is, however, that fewer communication channels are available in these online communication mediums. In this pilot study, we investigated to which degree these altered communication mediums affect people’s social conformity behavior. Based on a classic normative social conformity paradigm by Asch [1], we developed an online quiz. Two versions were created: a version where users see other players represented with only a picture, and a version with a live video stream. We studied the social conformity in these two online situations regarding three information types: visual perception, factual information and personal opinions. Results showed that participants answering factual questions in the live video variant, offering more social cues than the photo variant, followed the group more in giving incorrect answers. Furthermore, participants in both variants agreed with group opinions.

Figure 1. A discussion about racism, adapted from YouTube.

Author Keywords

Many of these typical Web2.0 applications consist of activities such as voting, ranking, participating in polls, sharing opinions and other forms of personal utterances. People not only “meet” in the same virtual space, they also engage in interactions where they are confronted with each others opinions. They are therefore bound to influence each other. These computer-mediated interactions and influences, however, are potentially different from real-life interactions, because fewer communication channels are available to its users in comparison to face-to-face situations. It might well be possible that communicating via less rich channels and being more or less invisible or anonymous towards each other causes people to behave differently. The fragment in Figure 1 is an example of the fact that perhaps people dare to behave ruder or utter more extreme opinions in online situations than they would have done in real life.

Online social presence, online social conformity, video. ACM Classification Keywords

H5.2. Information interfaces and presentation, H5.3. Group and Organization interfaces. INTRODUCTION

In the last decades, computer and communication technology have evolved enormously: the PC has become omnipresent, mobile phones are even more polyvalent nowadays, and the wired and wireless internet has become an indispensable part of everyday life. The impact of this ongoing evolution on people’s lives (both private and professional), on the way they live together and communicate with each other is probably tremendous. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. . NordiCHI '10, 16-OCT-2010, Reykjavik, Iceland Copyright © 2010 ACM 978-1-60558-934-3/10/10…$10.00

In the present study, we were interested in the effects of different communication mediums on a particular form of social influence, namely social conformity: the process by which an individual complies his attitudes and behaviors with recognized social patterns or standards.

1 305

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

CONTEXT AND BACKGROUND

personalities and intentions of its users. It is a function of the medium’s capacity for immediate feedback, the number of cues and channels available, the language variety and the degree to which attention is personally focused on the recipient. According to this model, face-to-face communication is the richest medium. E-mail and chat, on the other hand, are rather impoverished mediums. The second characteristic, social presence refers to the degree to which a medium expresses the physical presence of the communicating participants, through verbal and non-verbal cues. Differences in richness and degree of social presence of a particular communication medium might alter people’s behavior when interacting with each other via this communication medium.

Social influence can be considered as the core of many social interactions. These influences, whether they are intentional and deliberate or unintentional and unwanted, arise whenever people engage in interactions with each other. As such, people’s thoughts and actions constantly affect, and are affected by, their communications partner(s). Deutsch and Gerard [7] describe two forms of social influence. Informational social influence refers to our acceptance of information from someone else as evidence about reality. Normative social influence refers to our conformity to the positive expectations of others. These two forms of social influence are triggered by two psychological needs respectively: the need to be right, and the need to be liked. Amongst many other forms of manifestation, social influence plays a key role in social conformity. Social conformity can be defined as the behavioral or attitudinal compliance with recognized social patterns or standards. Already in 1956, the psychologist Solomon Asch demonstrated in his now famous experiment, that social conformity is fundamental to group processes [1].

Substantial research effort has already been devoted to social influence, and to social conformity in particular, in computer mediated environments. Connell et al. [4], for example, studied the effects of the use of different types of communication mediums (face-to-face, email and chat, and voice telephony) on ‘ingratiation’, the fact that a person behaves in a manner to be liked by another person. They found that users of less rich media, offering fewer communication channels and less social presence, feel less pressure to ingratiate. It seems that less rich media allow people to express themselves with less inhibition.

In Asch’s experiment, the influence of a majority opinion on an individual’s own opinion was demonstrated. Participants were gathered in groups to take part in a so called visual discrimination experiment. They were asked to, each in turn, compare the lengths of lines. To manipulate social influence, all but one of the participants were actually Asch’s accomplices. They were instructed in advance to make erroneous comparisons. During the experiment, the majority of confederates, who were the first to announce their decisions out loud (the ‘real participant’ was seated last but one), thus periodically unanimously picked the wrong line. Asch found that individual participants agreed with the obviously incorrect judgments of the accomplices on 37% of the critical trials [1].

Similar results were found in a study on computer mediated group decision making [8]. Groups making decisions using electronic communication used more explicit verbal statements of their positions, in comparison with face-toface groups. Politeness and acknowledgment of other people’s views decreased. According to Matheson et al. [9], this can be explained by the fact that computer mediated communication tends to reduce public self-awareness (the overt aspects of self which are sensitive to attention and evaluation of others), which might result in less inhibited behavior. Since physical presence of others is excluded, individuals might be less inhibited and more task-oriented than in face-to-face situations. Group processes in a computer mediated environment result in participants identifying less with the group than they would in face-toface situations. As nonverbal behavior is absent, participants’ positions can be held more strongly.

Since Asch’s experiment, the study of social influence has been greatly refined, focusing on comparisons across cultures and across different periods in history, and on variables that had an influence on the presence and the amount of social conformity [2]. As will be explained hereafter, some of these variables might also be relevant in the study of social conformity in online environments. For example, studies using different types of paradigm have already shown that exposing participants to each other in face-to-face interactions versus isolating them in separate locations has an important effect on the size of the conformity effect.

Research on social conformity has also found its way to very concrete internet applications. As literature on conformity suggests that people are easily influenced by a majority group, recommender systems, using people’s opinions about items in an information domain to help people choose other items, probably affect users’ opinions. Cosley et al. [5] have shown that people can indeed be manipulated; they tend to rate toward the prediction a recommender system shows, whether this prediction is accurate or not.

As computer mediated communication becomes an increasingly important part of communication practices, in organizations as well as in people’s leisure time, there is an increasing need to understand how the characteristics of specific communication mediums might alter the way people interact. Communication media are often described in terms of their richness and the degree of social presence they allow [6, 10]. Media richness refers to the effectiveness by which a medium can convey the

Two studies were explicitly based on the methods of Asch’s original social conformity experiment [3, 11]. In these studies, a computer program substituted for the physical

2 306

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

presence of the experimenter and the confederates. The participants were told that they were connected in a network with other participants. The same materials Asch used in the original experiment were presented on a computer terminal. The rest of the experimental design resembled Asch’s original design, except that participants were now asked to input their responses through a computer terminal. This study has shown that a simulated majority had a smaller effect on the judgments of the participants than was originally reported by Asch: 69% of the participants responded error free, as compared to less than 25% in Asch’s experiment; 95% of the participants made fewer than three errors, as compared to 52% in Asch’s study. Both studies explain their results in terms of the “less social presence approach”. According to Smilowitz et al. [11], physical absence of others allows individuals to be more responsive to the immediate textual clues. According to Cinnirella et al. [3], the key difference between a face-toface situation and a CMC situation, is visual anonymity, reducing social conformity behavior.

impressions (relevant to, for example, recommender systems). Two studies of online social conformity are presented. The first study concerned an online quiz with visual perception questions (the Asch replication) and with factual questions. The quiz was played in a photo-only and in a live-video version, by participants who were –online- confronted with deviant majority decisions. The second study was analogous to the first one, but it solely consisted of opinion questions. These studies should be considered as pilot studies in the domain of online social conformity. They are aimed at exploring behavioral changes related to technological evolution, allowing us to orient future studies towards more specific research questions. MEASURING ONLINE SOCIAL CONFORMITY

In both experiments, a similar general set-up was followed. Six people played an online quiz. They were not physically together, but communicated with each other through a computer network. All but one of the quiz players were collaborators (these will be called ‘acting participants’ hereafter) of the experimenter. They were instructed to respond on certain questions with wrong and unlikely, but unanimous answers. Only one single participant – the real participant – was not aware of this prearrangement.

This paper uses the same perspective to study social conformity in computer mediated environments. In line with the just described studies, we used the materials and the set-up of Asch’s original conformity experiment as a starting point. In these studies, however, one computermediated condition is compared to a face-to-face condition. According to the social presence approach the degree of social conformity is (among other things) a function of the amount of social cues that a communication medium offers. We therefore compared two computer-mediated conditions, offering more or less social cues. Whereas Cinnirella et al. [3] and Smilowitz et al. [11] made use of mere text-based communication through a computer terminal, we included a condition in which participants communicated with each other through a computer network and were able to see each other’s names and pictures (static), and a condition in which they could see and hear each other live on a video and audio stream.

In the first experiment, the quiz consisted of visual perception questions (replication of Asch) and factual questions, measuring factual knowledge. In the second experiment, the quiz consisted of opinion questions, probing for participants’ personal subjective ideas and opinions. Our main focus was on the degree of social presence that a particular communication medium offers, and the effect hereof on social conformity behavior. Different communication mediums were therefore introduced into the design of both experiments. The participants communicated their responses to the quiz questions to each other through one of the following communication channels:

Our hypothesis according to the social presence approach, states that conformity of the individual to the majority’s opinion would be higher in the condition where people could see each other on a video stream, offering more social cues, then in the condition with the static picture.

• Photo-only Participants were able to see the name and a photograph of the other participants on their computer screen. They responded to each question by clicking the desired answer.

Asch, Cinnirella et al. [3], and Smilowitz et al. [11] made use of a visual discrimination task in which line-lengths had to be compared. As we wanted to create realistic online situations in our study, similar to many of the typical social media activities described previously, we added factual questions and opinion questions to the design. After all, the online situations in which people engage often consist of uttering opinions and statements, thoughts, discussions on facts, and these activities are of a different nature than the mere perception material as in Asch’s experiment. As to the opinion questions, we made a further distinction between moral or political opinions (relevant to, for example, voting systems and discussion forums), and more general personal

• Live-video Participants were able to see the name and a live video capture of the other participants. Through the use of headphones and microphone, they could also hear and talk to the other candidates. They responded to each question by clicking the desired answer, and by calling it out loud at the same time. Participants in this condition were also able to talk freely to each other.

3 307

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Study 1: Visual perception and facts

factual knowledge questions. The materials to compose the perception questions were directly drawn from the original Asch experiment (Figure 3). However, Asch’ materials back then were presented on a large paper board that all participants watched from a distance, whereas our materials were presented on a computer screen. Therefore, the lengths of the lines we used in our judgment task were physically smaller than the line lengths in Asch’ experiment, to make them fit on the screen. The viewing distance, however, was much less. The factual knowledge questions were typical multiple choice questions. Each of them had three answer alternatives, of which only one was correct. The questions were drawn from Belgian politics, Belgian celebrities, geography, and typical facts about Belgium (Table 1). All of these questions were assumed to test general knowledge of Belgian participants. Each quiz session was played by 6 participants (1 real participant and 5 acting participants). The real participant was seated in a separate room with a one-way window, allowing one of the acting participants to observe him and to coordinate the course of the quiz session according to the participant’s behavior. The acting participants were also seated in separate rooms.

Participants and design

Thirty-one people, aged between 17 and 35, participated to this experiment. They were drawn from a student and working population. Half of them were in the photo-only condition, the other half were in the live-video condition. Every participant received a 20 euro reward voucher. The acting participants in each condition were colleagues from our research department.

No. 5 in row was the real participant, the others were acting participants Figure 2. Quiz screen lay-out, showing a quiz question about a Belgian comedian (the picture above). The real participant (always in fifth position) and acting participants used headphones to play the quiz in the live-video condition. Materials

The online quiz was constructed as a web application and consisted of a number of successive web pages, each of them representing one multiple choice quiz question. In the centre of each web page, the question was shown, with its three answer alternatives immediately below. At the bottom of each web page, the pictures and names (in the photo-only condition), or the live streams and names (in the live-video condition) of all quiz candidates were shown. Figure 2 shows a screen shot of the quizzes screen lay-out.

Figure 3. Example of a visual perception question, adapted from Asch.

Question

Answer alternatives

Of which colours does the French flag consist?

red-white

white-blue

blue-whitered

Where does the sun rise?

east

west

south

Which country is not a member of the EU?

UK

Norway

Denmark

How many spheres does the Atomium in Brussels have?

9

10

15

Of whose territory does the Iberian Peninsula mainly consist?

Spain

Netherlands

Portugal

When did Belgium become independent?

1873

1830

1831

Who was the Belgian prime minister before Leterme?

Verhofstadt

Dewael

Martens

When is the national holiday?

July 11

July 21

August 15

How many strings does a violin have?

4

5

8

Where does king Albert live?

Courthouse

Chateau of Laken

Chateau Belvedère

Table 1. Examples of facts questions.

The quiz questions in this experiment were of two types: the first part of the quiz consisted of 18 visual perception questions (as in Figure 3); the second part consisted of 18

Procedure

Each participant was welcomed and told that he would be taking part in an online quiz. No further explanation as to 4 308

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

the goal of the experiment was provided, but the general course of events was briefly explained. Each real participant did not physically meet the other acting participants, but was taken directly to a separate room, where he was seated in front of a PC with a webcam mounted on it. In the photo-only condition, a picture of the participant was taken with the webcam. This picture was quickly integrated into the quiz web application. In the livevideo condition, all participants used headphones (Figure 2). The web application with the online quiz was started up by the observer (seated in a separate room with a one-way window). The participant then entered the quiz and could see the other participants’ picture (in the photo-only condition) or a live video and audio stream (in the livevideo condition) of the other participants. At the moment that every participant (the real one and the acting ones) was online, the first quiz question was presented on the screen. The order in which participants had to respond to each question was mandatory. As is shown in Figure 2, the real participant was always in fifth position, meaning that he saw the responses of four acting participants before he could give his own response. Answering a question was done by clicking on the letter of the chosen answer alternative. In the live-video condition, the participants were also instructed to say their answers out loud. When the first participant answered a question, all the participant’s web pages were updated so the answer became visible. After this, the second participant could choose his answer. This procedure was carried out until every participant had answered the question. At that moment, the web page was again updated, now showing a button labeled “Next question”. Clicking this button directed the participants to the next question. When all questions were answered, a page with the message “Thank you for participating to this quiz” was shown. Playing a quiz took about 40 minutes.

responses would be different, indicating that social conformity had been at work during the online quiz. For each question, participants also indicated whether they were sure of their answer or not. After the experiment, each participant was asked to fill out a post-experimental questionnaire, designed to assess participants’ feelings while playing the online quiz and their perception of their own compliance. Measures

We here report the measures of the real participants (the acting participants of course had predefined answers). • Degree of conformity. The degree to which the participant followed the acting participants’ answers on critical questions (i.e. questions that were purposely answered erroneously by the majority). Factual questions were chosen such, that an average Belgian citizen would most likely know the answer to them. The degree of conformity was measured overall and separately for responses of which participants were sure versus unsure. • Consistency during and after play. The discrepancy between the answer that was given during the online quiz (in the presence of a unanimous majority), and the answer on that same question given afterwards, in absence of any other quiz participants. • Questionnaire. The questionnaire was designed to assess the participants’ feelings while playing the online quiz, the participants’ perception of their own compliance while answering the quiz questions, and the participants’ insight into the research goal. Results

1 Degree of conformity Table 2 presents the percentages of answers equal to majority answers (by the acting participants) on critical questions.

The most important manipulation in our experiment of course was the disagreement between the real participant and the acting ones: the actors were instructed to give erroneous or unlikely answers on 66% of the questions. For the visual perception trials, we followed the exact procedure of Asch’s experiment [1]: in a series of 18 line comparisons, 33% of the trials were neutral (the acting participants gave a correct answer), and 66% were critical (the acting participants gave an erroneous answer). This answering scheme was extended to the factual knowledge questions: 66% of the questions were critical, and 33% were neutral. The participant was always put in the fifth answering position.

Question type Communication medium

Perception

Facts

Photo-only

0%

15%

Live-video

0%

28%

Table 2. Percentage of critical (visual perception and facts) questions answered according to the majority’s answer.

Neither in the photo-only condition nor in the live-video condition did any of the participants ever choose the majority’s answer on critical perception questions. All of them answered every question correctly, regardless of the majority’s answer. However, this was not the case for factual questions. Participants did choose the majority’s answers on questions that were deliberately answered wrong, and this effect was significantly stronger in the livevideo condition then in the photo-only condition (t(29) = 2.37, p < 0.05).

After finishing the online quiz, each participant was presented with a paper quiz that contained exactly the same questions as those of the online quiz. The participant was asked to answer the questions while seated in a separate room, without any interaction possibilities with other participants. By asking the participants to answer the questions again, but this time without the presence of any other quiz participants, we wanted to examine whether

5 309

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Study 2: Opinions

In a second, further detailed analysis, the degree of social conformity was measured in accordance with participants’ sureness of their responses. First of all, participants in the photo-only condition reported being sure of their answer on 53% of the critical questions, compared to 48% in the livevideo condition (this difference was not significant). Next, the results presented in table 3 show that a considerable part of the participants, in both conditions, didn’t follow the majority, regardless of their sureness. However, a significant difference between the photo-only and livevideo condition was found for conformist answers of which participants were sure (t(29) = -2.69, p < 0.05).

In our second experiment, the online quiz consisted of questions or propositions representing moral and political opinions, and personal opinions on general issues. Participants and design

Fourteen people, aged between 17 and 35, participated in our second experiment. They were drawn from a student and working population. This time, the design was withinsubject, meaning that every participant played the quiz in both experimental conditions. Each of them played the first half of the online quiz in the photo-only condition, and the second half in the live-video condition. Every participant received a 20 euro reward voucher. The acting participants in this experiment were colleagues of our research department.

Sureness

Communication medium

Participant was sure

Participant was not sure

Photo-only

1%

14%

Live-video

9%

19%

Materials

Table 3. Percentage of conformist answers on critical questions of which participants were sure versus not sure.

2 Consistency during and after play All of the visual perception questions were answered the same during the online quiz and during the paper quiz afterwards. Concerning factual questions, consistency was 95% in the photo-only condition, and 94% in the live-video condition. This difference was not significant (p > 0.05). 3 Questionnaire Asked about the tension they experienced while playing the online quiz, participants reported low scores (all beneath 3 on a 1 to 5 scale between ‘no tensions at all’ and ‘very tense’). Asked to which degree they themselves thought to be influenced by the majority, they all reported to be hardly influenced at all (all beneath 3 on a 1 to 5 scale between ‘not influenced at all’ and ‘very much influenced’).

Question

Answer Likert-scale

1. Flanders has to become independent.

[1 = strongly disagree – 4 = strongly agree]

2. An unemployed person should receive 60% of his former income.

[1 = strongly disagree – 4 = strongly agree]

3. Death penalty sometimes is a just punishment. In those cases, it should be executed.

[1 = strongly disagree – 4 = strongly agree]

4. Migrants should either adjust or leave.

[1 = strongly disagree – 4 = strongly agree]

5. Politicians are profiteers.

[1 = strongly disagree – 4 = strongly agree]

6. How safe do you find bungee jumping?

[1 = very dangerous – 4 = very safe]

7. How young/old does this person look?

[1 = very young – 4 = very old]

8. How slow does this car look?

[1 = very slow – 4 = very fast]

9. How obese does this person look?

[1 = very skinny – 4 = very obese]

10. How dirty does this street look?

[1 = very clean – 4 = very dirty]

Table 4. Examples of opinion questions.

The same online quiz format as in the previous experiment was used here (Figure 4). But the quiz now consisted of 24 opinion questions. Half of these questions probed for moral and political opinions (for example, questions 1 to 5 in Table 4). The answers to these questions were a rating on a 4-point Likert scale between ‘agree’ and ‘disagree’. The other half of the questions were on very general personal opinions or impressions (for example, questions 6 to 10 in Table 4), such as ‘How fast do you think this car is?’ (the question was accompanied by the picture of a car). The answers to these questions were also a rating on a 4-point scale (in the given example: between ‘very slow’ and ‘very fast’).

Figure 4. Quiz screen lay-out, showing a quiz question about the legalisation of soft drugs, and 1 real participant and 5 acting participants names and pictures in the photo-only condition.

6 310

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Procedure

The procedure followed in the opinions experiment was similar to the procedure of the factual knowledge experiment. But as the quiz questions in this experiment solely concerned personal opinions, it was consequently not possible for the actors to give ‘erroneous’ answers. Instead, we let the acting participants express very unlikely or extreme opinions.

Communication medium

After playing the online quiz, participants were asked to answer the quiz questions on paper, as in our first experiment. They were also asked to fill out the postexperimental questionnaire.

Moral/political opinions

Personal impressions

Photoonly

30%

20%

Livevideo

33%

24%

In total

32%

22%

Table 6. Percentage of moral/political opinions and personal impressions that were equal to majority opinions. Type of opinion Moral/political opinions

Personal impressions

In total

Photoonly

83%

81%

82%

Livevideo

39%

58%

49%

Measures

• Degree of conformity. The degree to which the real participant followed the acting participants’ answers. The degree of conformity was measured overall and separately for moral/political opinions versus personal impressions. As opposed to the measures in the facts experiment, the participants’ sureness was not measured here, as this experiment concerned personal opinions instead of factual information.

Communication medium

Table 7. Percentages of opinions during online quiz equal to opinions during paper quiz.

• Consistency during and after play (as in experiment 1).

2 Consistency during and after play Table 7 displays the consistency data of the opinions experiment. The overall difference between the photo-only condition (82% consistent) and the live-video condition (49% consistent) was significant here (t(13) = 4.55, p < 0.01), thus indicating that participants more frequently express different opinions in the paper quiz versus the online quiz in the live-video condition as compared to the photo-only condition. Furthermore, within the live-video condition, a significant difference was found concerning the consistency on moral/political opinions (39% consistent) versus personal impressions (58% consistent) (t(13) = 2.65, p < 0.05). This difference indicates that participants express more different opinions in the paper quiz versus the online quiz concerning moral/political opinions.

• Questionnaire (as in experiment 1). Results

1 Degree of conformity The first analysis examined whether participants shared the (extreme) opinions of the majority of acting participants. Table 5 displays the percentages of answers equal to majority answers. These results indicate that the differences between the photo-only and the live-video condition were not significant (p>0.05). In a next analysis, conformity was measured on moral/political opinions versus personal impressions. Table 6 presents the results of this analysis, overall and for photoonly and live-video condition. Although none of these differences were significant, a certain tendency was noticeable in both conditions: conformist answers were more frequent on moral/political questions than on personal impressions.

3 Questionnaire Asked about the tension they experienced while playing the online quiz, participants reported low scores (all beneath 3 on a 1 to 5 scale between ‘no tensions at all’ and ‘very tense’). When asked to which degree they themselves thought to be influenced by the majority, they all reported to be hardly influenced at all by other answers (all beneath 3 on a 1 to 5 scale between ‘not influenced at all’ and ‘very much influenced’). An interesting observation, however, was made by analyzing our video recordings of the quiz sessions. When a question appeared on the screen, participants sometimes put their mouse cursor over the answer of their (initial) choice (e.g. answer ‘1’). But when the first four answers became visible (the participants were always seated in fifth place), participants often changed their mind and chose an answer closer to, or matching the majority’s answer.

% online answers equal to majority answer Communication medium

Photoonly

26%

Live-video

29%

Table 5. Percentage of opinions equal to majority opinions in photo-only versus live-video condition.

7 311

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

DISCUSSION

at least is not sure of his answer – and is confronted with a group of others all choosing one particular answer. However, even in cases where participants were unsure, they most often did not follow the majority. This could be explained by the fact that most majority answers on critical questions were very unlikely. Just as in the visual perception condition, participants might experience a limit to their willingness to conform. It might well be that they only comply their behavior when the majority’s responses are not ‘too wrong’.

The described experiments provide some clear indications that some of the well-known and typical human tendencies known from real life situations – such as social conformity are also present in virtual online situations, under certain conditions. First of all, the results showed a total lack of conformist behavior regarding Asch’s visual discrimination tasks, in contrast to previous research on social conformity [1]. A comment already heard in replications of Asch’s original experiment was that it took place in the 50’s, and that participants (and people in general) back then were more obedient than contemporary participants. If this was the case in face-to-face conditions already, then we would expect, according to the social presence approach (less social cues allow for less conformity), that conformity would hardly be observed in online situations offering less social cues (compared to face-to-face situations). Although recent studies [3, 11] do report a conformity effect with contemporary participants, even in an online situation where the only cues that people received from each other were their text-based responses, we are inclined to conclude from our results that the social conformity effect as measured by Asch, concerning majority opinions that are so clearly wrong, might be a bit outpaced nowadays. It could be that participants experience a limit to their willingness to conform in cases where the majority’s decision is ‘too wrong’. This would be an interesting future research question.

Concerning answers of which participants were sure, higher levels of conformity were found in the live-video condition than in the photo-only condition, although conformity was considerably lower here as opposed to the answers of which participants were not sure. This result not only further supports our research hypothesis, but also suggests that a mere conformity effect (as measured in Asch’s experiments) might be at work here. Indeed, participants stated that they were sure of their (majority) answers, and yet, they were wrong. In our second experiment – focusing on opinions instead of factual information – different results were obtained. A considerable number of conformist responses was measured here, but there was no significant difference between photoonly and live-video conditions. Participants complied about just as much in both online situations, the different degree of social presence did not seem to make a difference. However, the amount of conformity was quite high in both conditions. Perhaps, having to show ”your political face” to others in an online social situation is awkward enough as it is, and seeing a moving image in addition to other participants’ answers did not make a difference anymore. It could also be that his finding is caused by the withinsubjects design of our experiment. The fact that the same participants played the photo-only and the live-video version of the online quiz might have accounted for some sort of habituation to the situation. Seeing a live video stream of the quiz players that they have just met in the (photo-only) first quiz part might not make an impression anymore.

On factual questions, however, a conformity effect was clearly observed. Although many of the answers given by the group of acting participants were very unlikely, participants in both photo-only and live-video conditions were, to some degree, influenced in their judgments. Furthermore, our findings showed significantly higher levels of conformity in live-video situations, as opposed to mere chat situations, concerning obvious factual information. These findings are in line with the social presence model, stating that communication environments providing less social cues (i.e. our photo-only condition) allow for less conformist behavior. The only difference between our photo-only condition and our live-video condition was the absence or presence of a moving live video stream (in the photo-only condition all that participants saw of each other was a static picture), and the fact that the answers to the quiz questions also had to be given out loud in the live-video condition (in the photo-only condition, answers could only be given through mouse clicks). Both of these factors then (‘real life’ versus static, ‘anonymous’ conversations and non-verbal versus verbal answers) could have influenced the amount of social conformist behavior.

Remarkable however, in this respect, are the results we obtained of the consistency between online answers and paper quiz answers in the opinions experiment. A significant difference between photo-only and live-video condition was found here, especially regarding moral or political opinions. While participants agreed with the majority about just as much in the live-video condition as in the photo-only condition, when they were asked to give their opinion again in the paper quiz, in the absence of the online group, they suddenly expressed different opinions in a considerable part of the opinion questions of the livevideo condition. This finding strongly suggests that participants did not express their ‘real’ opinion during the online quiz, at least not in the live-video condition. The discrepancy between the conformity results and the consistency results might indicate that an effect of the

The conformity effect was especially observed in cases where participants were not sure of their answer. At first sight, this might seem like a very natural reaction in cases where someone doesn’t know the answer to a question – or

8 312

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

opinion questions posed in this experiment came into play. Indeed, due to the within-subjects design, questions in both conditions were of course not the same. It would be worthwhile to make a detailed analysis of the questions and majority responses in a later study.

solving, or forum discussions as in the YouTube example in our introduction, it could be interesting to consider in how far opinions and utterances of others can be (positively or negatively) influenced, both at the sending and the receiving end. We might be facing a trade-off here between on the one hand offering a maximum degree of virtual social presence, and on the other hand safeguarding the chance on unbiased, honest opinions.

Besides the communication medium effect, we found a slight tendency towards a greater degree of conformist behavior regarding moral and political opinions, as opposed to more general personal impressions. Moreover, this finding was affirmed by a very low degree of consistency on moral and political opinions. People in general might have a higher inclination to conform in opinion matters as opposed to factual information matters, and even more so in opinion matters concerning moral or otherwise sensitive themes. After all, being extremely sure about a certain fact - although others disagree - might not feel extremely discomforting. Expressing an opinion about a trivial fact (“Is this a good or a bad movie?”) might be slightly more discomforting. But when uttering an opinion about an important moral issue (“Death penalty should be executed in some cases.”), however, one could feel to be frowned upon the group in a very personal manner.

Based on the results of online social conformity studies, some sort of ‘social conformity optimum’ could be created, providing the exact amount of social cues desired in situations as the ones just described. FUTURE RESEARCH

As was already mentioned, this paper was aimed to be a pilot study. We realize that our design and current analysis have their limitations. Another experiment, including a face-to-face situation, offering the maximum amount of social cues, should be conducted. Furthermore, in contrast to Asch’s visual perception material where it was clear that every participant could easily pick the correct answer, our facts and opinions questions were somewhat more confusing. We could of course never be sure of the participants’ ‘real’ knowledge or opinion. Although a clear effect was measured even on factual questions that participants were sure of, it would be interesting to add a control group to the experimental design.

As a summary, our findings indicate that important differences might exist in the process by which people accomplish their communicative tasks, depending on the type of communication medium that they are making use of. In situations where people have to take a decision about a fact, they have themselves influenced by a deviant majority opinion, even when they claim to be sure of their decision. The more social cues of the group are present, the stronger this effect is. Uttering personal opinions in groups offering more or less social cues of their presence is apparently an even more sensitive issue.

Finally, some of the analyses made here could be further elaborated on. It would be interesting, for example, to make a profound analysis of the questions that were posed during the quizzes, and of the exact answers of the majority. It could well be, for example, that the majority is only agreed with if answers are not too far-fetched, or that conformist behavior is more likely to occur on particular factual information. Also, diving into content analysis of the video recordings that were made during the experiment could reveal some valuable behavioral information. Also open for deeper investigation is the amount of (non-)conformity. The continuum of the degree of social presence offered here can be extended in both directions (no presence at all, or even more social presence experienced by advanced immersive technology). Lastly, these experiments made use of online quizzes, but there is a myriad of contexts thinkable in which this issue could be investigated. Indeed, in more critical situations such as the medical or military domain mentioned before, it is even more important to realize that people never entirely act on their own, but always as part of a group, even if that group is a virtual one.

These differences between the social conformity effects of different types of communication mediums lead us to the question whether computer mediated communication is in some cases a satisfactory (or maybe more satisfying?) alternative to face-to-face communication or not, for a variety of communicative activities. While a certain amount of ‘group thinking’ might be beneficiary in some domains (for example meetings where compromises have to be established), in others it might have undesirable consequences. Think of a judging-committee having to give a verdict on a crime case, where independent judgments of each of the committee-members are essential. One could also think of communication channels in technology enhanced learning environments (e.g. groupware for educative applications), where messages have to be conveyed in a proper and direct manner unhindered by human tendencies. Other examples are safety critical situations such as military or vigilance contexts or the medical domain. But also in the realm of ‘everyday’ social network applications, such as recommender systems, voting tools, online events involving electoral statistics showing intermediate polls, computer mediated group problem

ACKNOWLEDGMENTS

The research described in this paper has been made possible by the Citizen Media project (FP6-Research).

9 313

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

6. Daft, R.L. and Lengel, R.H. (1986). Organizational information requirements, media richness and structural design. Management Science, 32, 554-571.

REFERENCES

1. Asch, S.E. (1956). Studies of independence and conformity: I. A minority of one against a unanimous majority. Psychological Monographs: General and Applied, 70, 9, 1-70.

7. Deutsch, M. & Gerard, H. B. (1955). A study of normative and informational social influences upon individual judgment. Journal of Abnormal and Social Psychology, 51, 629-636.

2. Bond, R. and Smith, P.B. (1996). Culture and conformity: a meta-analysis of studies using Asch’s (1952b, 1956) line judgment task. Psychological Bulletin, 119, 1, 111-137.

8. Kiesler, S. and Sproull, L. (1992). Group decision making and communication technology. Organizational Behavior and Human Decision Processes, 52, 96-123.

3. Cinnirella, M. & Green, B. (2007). Does ‘cyberconformity’ vary cross-culturally? Exploring the effect of culture and communication medium on social conformity. Computers in Human Behavior, 23, 20112025.

9. Matheson, K. and Zanna, M.P. (1988). The impact of computer-mediated communication on self-awareness. Computers in Human Behavior, 4, 221-233. 10. Short, J., Williams, E., and Christie, B. (1976). The social psychology of telecommunications. John Wiley & Sons.

4. Connell, J.B., Mendelsohn, G.A., Robins, R.W. and Canny, J. (2001). Effects of communication medium on interpersonal perceptions: don’t hang up on the telephone yet! GROUP’01, Sept. 30 - Oct.3, 2001, Boulder, Colorado, USA.

11. Smilowitz, M., Compton, D.C. and Flint, L. (1988). The effects of computer mediated communication on an individual’s judgement: a study based on the methods of Asch’s social influence experiment. Computers in Human Behavior, 4, 311-321.

5. Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., and Riedl, J. (2003). Is seeing believing? How recommender interfaces affect users’ opinions. CHI 2003, April 5-10, Ft. Lauderdale, Florida, USA.

10 314

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Using the Hybrid Simulation for Early User Evaluations of Pervasive Interactions Matthias Rehm Aalborg University, Niels Jernes Vej 14, 9000 Aalborg {matthias}@imi.aau.dk

Karin Leichtenstern, Elisabeth Andr´e Augsburg University, Universit¨atsstr. 6a, 86159 Augsburg {leichtenstern, andre}@informatik.uni-augsburg.de ABSTRACT

first real interaction devices to physical objects are mobile phones. Almost everybody owns a mobile phone and takes it around constantly. Recent phones support novel hardware and network facilities that enable different interactions to pervasive environments, such as an interaction with a public display to download displayed information. Ballagas and colleagues [1] give a comprehensive overview about the different input channels available with todays smart phones (e.g. the phone’s built-in camera or accelerometer) that can be used for mobile interaction techniques. Despite the promising potential of using mobile phones as interaction devices to pervasive environments, some problems emerge that need to be solved.

To reach a good user-friendliness, knowledge about user requirements is crucial in the development process of a product. The sooner the knowledge is achieved via user evaluations, the more money and time can be saved. In this paper we investigate an approach called hybrid simulation for the early stages evaluation of mobile applications where real mobile phones are used as interaction devices to a virtualised simulation of a pervasive environment. On the first sight, the method is cheap, easy and quick to use as well as more realistic compared to a virtual simulation only approach. In order to receive a more detailed insight in potential benefits and problems of the method, we performed a user study and compared results of a traditional laboratory study with the results of a study performed with the hybrid simulation.

Compared to the development for desktop settings, the development of mobile applications in the context of Pervasive Computing adds new layers of complexity. Ensuring user-friendliness in this context is a challenge due to a number of reasons. For instance, the developer has to cope with limited input and output capabilities of the mobile devices [8]. Moreover, the contextual setting of the interaction is unpredictable due to the mobility of the user [11]. In terms of Dey [5], information about the user’s situation is often unknown which includes knowledge about the requirements of users as well as their typical behaviour and preferences in the corresponding situations. To tackle these challenges, the application of the user-centred development process [33, 25, 16] is a possibility to obtain a good design for mobile applications.

ACM Classification Keywords

H.5.2 Information Interfaces and Presentation: User Interfaces—Evaluation/Methodology, Prototyping, UserCentred Design, Input Devices and Strategies Author Keywords

User-centred Design, Early Stages Evaluation Method, Hybrid Simulation, Pervasive Interface, Mobile Phones INTRODUCTION

The idea of Ubiquitous Computing and its synonym Pervasive Computing [36, 8] is to make the computer invisible in our everyday life in order to enable interactions with everything, everywhere at anytime [7]. The users can either directly interact with physical objects in their environment (e.g. interactive surfaces) or make use of interaction devices as medium to the objects. By this mean the user can request services for the object, such as the current status of a home appliance. The

A characteristic feature of this process is an iterative prototyping that includes several iterations of designing and implementing prototypes along with continuous executions and analyses of user evaluations with the generated prototypes. In terms of interface design, a prototype represents a partial simulation of a product with respect to its final appearance and behaviour [13]. The execution of user evaluations with the prototypes can provide interface developers with the relevant knowledge about the users and their requirements as well as behaviour. Traditionally, these user evaluations are either performed in-situ (field study) or simulated in a laboratory. The field studies are often postponed to the end of the development process because they typically require fully implemented applications. Addition-

Permission to make work forfor make digital digital or or hard hardcopies copiesofofall allororpart partofofthis this work personal or classroom use is granted granted without without fee fee provided provided that that copies copies are are not made or distributed for for profit profit or or commercial commercialadvantage advantageand andthat thatcopies copies bear this notice notice and and the the full fullcitation citationon onthe thefirst firstpage. page.ToTocopy copyotherwise, otherwise, or or republish, to post on servers to redistribute to lists, requires republish, to post on servers or toorredistribute to lists, requires prior prior specific specific permission permission and/or a and/or fee. a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. NordiCHI’10, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM978-1-60558-934-3... ACM ISBN: 978-1-60558-934-3...$5.00. Copyright 2010 5.00.

315

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ally, they are often expensive and difficult to organise and conduct. Compared to field tests, laboratory studies are easier to organize and conduct but often less realistic because it is difficult to ensure a test setting that is as similar as possible to the field setting. Additionally, it is often difficult to recruit a large number of subjects for traditional laboratory studies since laboratory studies normally require the physical presence of the subjects.

Overall, the results showed that field tests cannot be completely substituted by laboratory studies and should be used at least at the end of the development process to investigate specific user behaviour in different contextual settings. Laboratory studies, however, can be used during minor iterations of the design process if appropriate evaluation methods and techniques are applied. But at that point a problem emerges. Duh and colleagues [6, 28] as well as Kjeldskov [19, 18] see a lack of appropriate evaluation methods and techniques in laboratory settings. In particular when investigating user behaviour in early stages of the development process, there is a need for appropriate methods [6]. S´ a and Carricio reflected different low-fidelity evaluation techniques for user studies and discuss their feasibility [4, 3]. They also see a lack of appropriate techniques in early stages of the design process that reduce the costs, required time and organisational effort.

In this paper, we describe and investigate the concept of using the hybrid simulation as an evaluation method in early stages of the user-centred development of mobile applications. After a literature review of current evaluation methods, the concept and implementation of the hybrid simulation is described more detailed. Finally, a comparative user study is presented in order to reveal potential benefits but also problems of the hybrid simulation as evaluation method at early stages of the user-centred development process.

VIRTUAL AND HYBRID SIMULATIONS

One idea is to apply virtual simulations at the beginning of the user-centred development process. Simulations via virtual worlds can improve the development process [20] because they can mediate ideas of new products and support first user evaluations. In the context of Pervasive Computing, a literature review showed a tendency to apply virtual simulations in order to investigate the pervasive environment itself and its performance [35, 32]. The use of virtual simulations for the investigation of pervasive interaction devices has not been focused so far. The projects that do address this aspect [14, 24, 35, 2], mostly used virtual simulations which directly included the use of the interaction device into the virtual world. Mobile devices are not longer physical available for interactions anymore. Instead, they are just virtually represented and have to be controlled via keyboard or mouse which leads to a disruption of the real usage of the device. For example, Manninen [24] used virtual representations of interaction devices in his setting. His main objective was to easily develop and test different virtual worlds and their input devices. Barton [2] is also interested in similar objectives. He developed UbiWise that is a simulation tool for Ubiquitous Computing applications. This simulation tool helps to investigate applications that use cameras or mobile phones as interaction devices. As the devices are only represented in a 3D virtual world, the user has to interact via the traditional input methods (e.g. mouse) which mean a marginal mapping to the real usage of the device and the application. Certainly, there is a need to consider the level of immersion that can often not be met by a simulation alone.

FIELD AND LABORATORY STUDIES

The most reasonable evaluation method is the execution of a field test in the real environment of the user. Different field studies with mobile applications have been executed and described [11, 29, 30]. For example, H¨akkila and colleagues [10] performed a user study in the city centre of Oulu. They used different applications, such as an event calendar or guidance service during their field test. This test was performed as a diary study combined with user interviews which gave valuable data of the user’s behaviour. There is evidence that field tests provide very realistic and valuable data because they are performed in the real world with real contextual constraints. The execution of field tests, however, also causes problems. Field studies might lead to uncontrolled contextual settings rendering the outcome useless. Moreover, field tests are often time-consuming to coordinate and conduct as well as quite expensive. Thus, the idea is to simulate parts of the real setting in a more controlled environment and conduct laboratory studies instead. A review [17] showed that 71% of the user studies for mobile phones are conducted as laboratory studies. But do laboratory studies really substitute field studies? Several comparisons of field and laboratory studies have aimed at answering this question [15, 6, 19, 18, 12]. For instance, Kjeldskov and Stage [19, 18] searched for appropriate techniques which enable evaluations of mobile applications in the field and in the laboratory. They found some differences between field and laboratory studies, such as the social comfort of use but pointed out most basic usability problems as similar. Other studies (e.g. Kaikkonen and colleagues [15]) validated these results. They revealed laboratory studies as sufficient in most cases due to the fact that field studies do not often provide an added value.

To solve the problem of the insufficient mapping, we searched for other approaches of simulations which involve at least parts of the real world. Morla and Davies [26] present such work. They used simulations to test a location-based application for health monitoring. Using their simulation environment, they can virtually evaluate the performance of sensors attached to a wearable

316

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

medical monitoring system. Although their main objective was not to investigate user interactions with pervasive environments but instead the performance of the pervasive environment, Morla and Davies’ work gives a first impression of a so-called hybrid simulation. Hybrid simulation means an integration and combination of the real and the virtual world. The medical monitoring system is not simulated and really reacts on the artificially generated contexts. A similar approach called dual reality is introduced by Lifton and colleagues [23]. They used Second Life as a virtual visualisation tool of streams which are generated from real world sensors (e.g. temperature sensors). Driving simulators1 also aim at the idea of the hybrid simulation. The users interact with a real steering wheel and dashboard while they are driving through a virtually presented test route. In our work, we do not use the virtual world as a visualisation platform for the performance of real devices as Davies and Lifton. We apply the virtual world as an evaluation platform for user studies similarly as used for driving simulators but we use real mobile phones as interaction devices and a virtual simulation of the pervasive environment. A similar idea is also described by Haesen et al. [9]. They used a virtual simulation of a museum to execute a user study where real mobile phones are applied to interact with the virtualised museum. Haesen et al. consider the concept of the hybrid simulation as a promising new evaluation technique in early stages of the user-centred design process. In contrast to them, we use a well-known platform for the virtual simulation of a pervasive environment (Second Life) as well as a tool to quickly and easily generate prototypes (MoPeDT). Additionally, we executed a user study to get insights to potential benefits but also problems when applying the hybrid simulation.

user interface developers in the user-centred prototyping of pervasive interfaces for mobile phones. Applying MoPeDT, applications for mobile phones can be designed, evaluated and analysed that support different pervasive interaction techniques [21] for the interaction with physical objects (e.g. products in a shopping store, objects of art in a museum or home appliances). For instance, the mobile phone and its built-in NFC2 reader can be applied to select a home appliance via an RFID tag that is attached to it. Further supported interaction techniques of MoPeDT utilize the user’s speech or location for interactions with physical objects. After having selected a physical object, different services and their contents are loaded from the database and displayed on the mobile phone, such as a detailed description of the selected object or information about the object’s origin. The idea to interact with physical objects and provide services to these objects follows the idea of Pervasive Computing. In order to support interface developers to user-centred develop and evaluate prototypes in the term of Pervasive Computing, MoPeDT employs a client-server architecture and software modules [21, 22]. For instance, a software module for mobile phones is used to generate prototypes of a mobile application. Additionally, whenever tool-supported user studies have to be conducted, the architecture’s component called evaluator is applied. Other components of the architecture are the main server, the database as well as the sensors and actuators. The database persistently stores all information about the physical objects (e.g. a detailed description) that can be requested by the mobile user via the main server. The sensor and actuator can be used to collect or display additional information about the pervasive environment. Sensors, such as a temperature or a humidity sensor can collect, interpret and broadcast contexts to the main server. The main server can forward these contexts to interested users or actuators. Actuators, such as a public display can receive and display contexts or other information (e.g. video content).

HYBRID SIMULATION

Applying the concept of the hybrid simulation to our domain, the user still interacts with the pervasive environment via a real mobile phone but the pervasive environment is now virtually represented in a simulation. Pervasive Interfaces Developement

Overall, using MoPeDT, different prototypes of pervasive interfaces for mobile phones can be implemented and user tests can be performed in real world settings of a pervasive environment as well as in a laboratory setting which has now been extended to also support user evaluations in hybrid simulations.

The pervasive interface development requires combined knowledge of software and usability engineering. This includes complex aspects, such as the implementation of the network communication and the interface programming of the mobile phone. As a consequence, the implementation of the different prototypes often takes too much time. There is a need to reduce the implementation time in order to be able to spend more time on the evaluation of the prototypes. To reduce the implementation time and to improve the interface’s friendliness, the literature approves the need for software tools, e.g [27].

Hybrid Simulation of Pervasive Interfaces

We developed a tool called MoPeDT (Pervasive Development Toolkit for Mobile Phones) [22] that supports

MoPeDT can also be used to generate prototypes for mobile phones which should be evaluated via the hybrid simulation. Therefore, some adaptations are required. Figure 1 illustrates the architecture of MoPeDT which is required to perform the hybrid simulation. The main difference to the general architecture of MoPeDT [21, 22] is the shift of the pervasive environment from the

1

2

http://www.carrsq.qut.edu.au/simulator/

317

http://www.nfc-forum.org/

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

ment. We propose to employ Second Life to simulate a real environment which has been augmented for context dependent interactions. Apart from setting up the simulation server, three steps are necessary for simulating a pervasive environment in a hybrid simulation. The environment itself has to be modelled, it has to be equipped with physical objects and sensors, and it has to allow for communicating with the outside world such as the real mobile device. Modelling the Environment

The most basic requirement is a virtual representation of the environment, in which the user evaluation should take place. To this end, standard modelling tools can be employed making it necessary to import the resulting models in Second Life or in-world modelling tools are used that supply basic functionalities. Figure 4 provides knowledge about the environment which was created for our study. Based on the requirements from our previous user evaluation which constitutes our benchmark (see Section The Reference User Study), a living room and a bathroom were modelled along with the furniture and devices like DVD player and TV.

Figure 1. MoPeDT’s extended architecture that illustrates the Shift of the Pervasive Environment in the Virtual World.

real world to the virtual world using a platform for a virtual world simulation, such as Second Life. Now, this simulation contains the virtual representation of all physical objects. All information about the physical objects is still stored in the database and can be accessed by the users with their real mobile phones. Thus, the user still makes use of a real mobile phone as an interface to the pervasive computing environment even though the physical objects are not longer physically present. Another difference to the former setting is the need for a representation of the user in the virtual world. With this avatar the user can interact via the keyboard within the virtualised pervasive environment, such as moving around to get closer to physical objects. These interactions can also create contextual information like a location event which is sent to the main server and then forwarded to interested mobile devices. The next section gives more information about these aspects and the simulated pervasive environment.

Making the Environment Pervasive

The challenge of a hybrid simulation is to realise the complex interplay between sensors, physical objects, and the mobile device, which can be seen as the inherent characteristic of a pervasive environment. The general idea is to use the real mobile for the interaction with the virtual world. This is not always possible. In our proof of concept study, objects are equipped with RFID tags to allow NFC with the mobile phone. Creating a virtual RFID tag is no challenge but of course this tag cannot be read out by the real mobile device. Thus, it is necessary to create a virtual representation of the mobile device for some of the contextual input. In the current version of our hybrid simulation, a virtual mobile device is used for registering the contextual input that is provided by the simulated environment. The real mobile device handles the output and the user interactions. Details are given in the next section.

Simulation of the Pervasive Environment

To simulate the pervasive environment, we make use of an open source version of Second Life, which is called Open Simulator3 . Open Simulator allows setting up one’s own virtual world that behaves exactly like Second Life and can be accessed with the same viewers. Thus, in the remainder of this paper we will use Second Life and Open Simulator as synonyms.

Apart from the virtual representation of the mobile device, we have realised physical objects which are triggered by user actions, sensors which are triggered by changes in the environment, and additional contextual input. To create physical objects, home appliances (e.g. TV and DVD player) were augmented with RFID tags allowing for near field communication (NFC), and with virtual IR sensors to register remote activation (see Section The Reference User Study for information on the interaction techniques). A virtual temperature sensor was necessary to register the effects of manipulating the heater. To this end, a model of the heater and its radiation was realised. To allow for indoor localisation of the user, WLAN access points have been installed in the environment. By measuring the signal strength, the user’s location can be approximated. Again, a model for the

Second Life represents one of the first massive 3D multiplayer platforms which is not primarily concerned with gaming but aims at establishing a general virtual meeting place. Thus, every conceivable type of interaction is in principle possible, be it buying or selling virtual or real goods, be it playing out as a real DJ in a virtual club, or be it pervasive interactions in an Ambient Intelligence landscape. Central feature of Second Life is the use of avatars which represent the real user in the virtual environment. We [31] have shown that Second Life can serve as an evaluation platform for multi agent systems involving the user in her natural environ3

http://opensimulator.org

318

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

THE HYBRID SIMULATION OF A SMART ENVIRONMENT

radiation was integrated. In both cases, a simple linear degression was chosen as a suitable model (see Figure 2).

Morla and Davies [26] describe several requirements of a hybrid simulation (e.g. the application of the device’s intrinsic code) which we considered as fulfilled when combining the features of MoPeDT with Second Life. Using MoPeDT we can upload the intrinsic code of the mobile phone application to physical devices which enables live user interactions on real mobile phones. At the same time, contexts can be generated via the user and his mobile phone as well as via the connected real or simulated sensors. All user evaluations can be logged and reproduced via MoPeDT’s architecture and the supported evaluation component. Even so, to get more insights in benefits and problems of using the hybrid simulation for early user studies, we decided to compare the results of a user study performed in a traditional laboratory setting with the results achieved by using the hybrid simulation. To this end, we implemented the scenario of the reference study as a hybrid simulation.

Communication

Two types of communications take place in our pervasive environment. Sensors directly communicate with the main server of the MoPeDT architecture. Physical objects on the other hand communicate with the virtual representation of the mobile phone, which in turn communicates with the main server (see Figure 1). So far, communication is based on the in-world scripting language. In order to allow for a stronger inclusion of the real mobile phone also for registering the virtual input that is provided by the simulated environment, it seems inevitable to integrate a context class in the Open Simulator server. The only sensor in our reference study which registers environmental changes is the temperature sensor. Thus, in each room, one sensor is placed that registers the local temperature according to the settings of the heater and its degression model (see Figure 2). Each sensor sends this information along with its ID as an HTTP request to the main server.

The Reference User Study

We conducted the reference study as a traditional real world simulation of a smart environment in a laboratory [34]. The main objective of this study was finding out whether users apply different mobile interaction techniques dependent on contextual conditions in a smart environment. In our setting, the smart environment contained several physical objects (e.g. a TV or a heater) which could be addressed and controlled via a mobile phone. For example, the mobile phone could be applied as a remote control to change the status of the heater by switching it on or off or by changing its temperature. In the context of the reference study, we investigated the use of the mobile interaction techniques: touching, pointing and scanning. When using the techniques touching or pointing the user has to physically touch or point at the intended physical object with the mobile phone in order to address it. Scanning is a technique to address a physical object by using the mobile phone’s graphical user interface and select the intended physical object out of the detected and graphically listed physical objects.

The home appliances have been augmented with RFID tags and IR sensors. Having identified a specific physical object activates its context on the mobile device, i.e. the services of this object become available to the user. To read out an RFID tag, the user moves her avatar towards the physical object. The avatar is holding the virtual representation of the mobile phone. The phone serves as a virtual RFID reader, simulating NFC. Thus, if the RFID tag is in a certain range (less than 30 cm) of the mobile, its ID is registered by the mobile device which sends a HTTP request to the main server containing its own ID and the object’s ID. The IR sensor allows remote activation of an object’s context. To achieve this goal, the user has to point the virtual mobile in the direction of the physical object (see Figure 4). Having been activated, the sensor sends the object’s ID via virtual Bluetooth to the mobile which in turn sends it to the main server via an HTTP request.

The reference study was performed with 20 people in a living room of a smart environment. All participants were sitting on a couch while they had to solve four different tasks in order to call a service of the intended physical object under different context conditions. (1) First, the user had line of sight to the physical object. The distance to the physical object was about three meters. (2) For the second task, the users were in front of the physical object. The distance to the physical object was about ten centimetres. (3) For the third task, the user did not have line of sight to the physical object. The physical object was located in another room and the distance was about 20 meters. (4) Finally, the user did not have a line of sight to the physical object. The physical object was located in the same room. The distance to the physical object was about four meters. To

The environment might also provide additional information that has to be actively transformed into contextual input. The indoor localisation in our scenario is of this type. WLAN access points have been installed in the environment (see Figure 2), which are utilized to calculate the rough location of the user. This location is the room the user is currently in. Thus, the room sets the interaction context by making only the services available on the real mobile phone which is defined by the appliances in this room. To estimate the user’s location, the virtual representation of the mobile phone registers the signal strength of the WLAN access points at the user’s current location and triangulates this location. The mobile’s ID is sent along with the location ID to the main server.

319

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2. Degression Model for Signal Strength of WLAN Access Points. Example for Access Point 2.

get a line of sight to the physical object, the user had to move about one meter. To cover most casual activities, users provided information afterwards about their behaviour and preferences when lying or standing. The results of the reference user study led to the following three findings. (1) Users tend to switch to a specific mobile interaction technique dependent on location, activity and motivation. (2) The current location of the user is the most important criterion for the selection of a mobile interaction technique. (3) The user’s motivation to make any physical effort is generally low.

completely realised on the real mobile phone (see Figure 3) and therefore no adaptation in the virtual world is required. The user navigates through different screens and finally selects the intended physical object in order to use a service for this object. Thus, scanning is quite similar to our reference study.

The Implementation of the Test Setting using the Hybrid Simulation

The reference study constitutes our benchmark for performing a similar test, this time making use of a hybrid simulation. Thus, we first modelled the living room and the bathroom and the required physical objects. Figure 4 shows the perspective of the avatar when sitting on the couch. In front of the avatar is the DVD player within line of sight. To the left of the avatar is the radio within touching distance. The idea is to select the different physical objects by using one of the three mobile interaction techniques. Once the user has selected one of the physical objects, the respective services are displayed and the user can select one of them by using the mobile phone’s graphical user interface. In the following we shortly describe the implementation of the three different mobile interaction techniques. Implementation of Scanning

We applied MoPeDT to generate the pervasive interface for the mobile phone that supports scanning. It is

320

Figure 3. Screens to perform scanning.

Implementation of Pointing

In contrast to scanning, pointing requires a direct user interaction with the physical objects. Figure 4 shows the implementation of pointing in a hybrid simulation. The user applies the virtual mobile phone to point at a physical object in order to perform the selection. The virtual phone can be positioned by pressing the four navigation keys. By hitting the ’PgUp’ key, an IR beam is emitted that is registered by the IR sensor of the virtual physical object. The information about the object’s ID is then transmitted to the server of MoPeDT’s architecture that forwards this context to the real phone and the application running on it. Now, the mobile phone loads the services of the selected object and displays them on the real phone.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Implementation of Touching

variables are location and activity with the different levels of our reference study. Thus, the participants of our user study had to perform exactly the same tasks as for the reference study. As dependent variables, we analysed the user’s preference for a mobile interaction technique in the different settings of independent variables. Thus, the experimental design is an exact replication of the reference study.

Touching is realised by getting very close to the physical object in the virtual world and touch it with the virtual mobile phone (see Figure 4). Once the user has touched the physical object, a script sends the identifier to MoPeDT’s server and the mobile phone as described for pointing.

Executing the Experiment

In this section we shortly describe how the user test was conducted. Pre-Experiment

Before we started the experiment, each participant of our user study was introduced to the correct usage of the mobile phone and the three interaction techniques. Moreover, we introduced the Second Life simulation. Prior to the test, the subjects could freely exercise each interaction technique and the use of the Second Life environment.

Figure 4. Pointing (left) and Touching (right)

USER STUDY WITH A HYBRID SIMULATION

After having implemented a hybrid simulation based on the idea of the reference study, we conducted a user study. First, we describe the experimental setting, then we report the user study and finally we illustrate our results.

Experiment

We conducted the experiment with 20 subjects aged 23 to 32 with an average age of 27.25. The Second Life environment ran on an ordinary computer that required no special hardware capabilities. The participants of our study could navigate through the virtual world using the avatar to trigger the different contexts of pointing and touching. The pervasive interface ran on a Nokia 6131 NFC that could be used to perform scanning and to retrieve the services for incoming contexts triggered via touching or pointing. After the explanation of the mobile interaction techniques and the virtual test setting, we sat the avatar on the couch in Second Life. This was always the starting position for each task. Now, the participants of our study had to complete the four tasks described earlier. After each task, we asked them about their attitude if the avatar would stay beside the couch or lie on the couch instead of sitting. Therefore, the subjects had to fill out a questionnaire that addressed the different test settings. This questionnaire was identical to the reference study.

Experimental Setting

The main objective of the experiment was finding out whether hybrid simulations of pervasive environments via Second Life can potentially be used as an evaluation method in the development process of a pervasive interface for mobile phones. In this context, benefits and problems should also be revealed by gaining practical experience. To address these aspects, we used our reference study and implemented a test setting. Based on this test bed, we deployed the experimental setting of the reference study, conducted a user experiment and compared the results. Hypotheses:

We used the findings from our reference user study and formulated them as the following hypotheses, falling into three categories. • H-1: Similar to the reference study, the users also tend to switch their mobile interaction technique based on their contextual situations when evaluating them with a hybrid simulation.

Results

For the hybrid simulation and the reference study, location could be identified as the crucial contextual factor for the decision of an interaction technique. An ANOVA test revealed these differences in location to be highly significant for the reference study (touching: F=19.225, p < 0.01, pointing: F=123.36, p < 0.01, scanning: F=10.769, p < 0.01). A similar result was obtained with the hybrid simulation. Again, location was the crucial contextual factor that dominated the choice of an interaction technique with significant differences depending on the location (touching: F=12.013, p < 0.01, pointing: F=39.2, p < 0.01, scanning: F=9.604, p < 0.01). No effect was found for the activity, i.e. it did not matter if the user was sitting, standing or lying. A post-hoc test revealed the dependencies between locations and

• H-2: Similar to the reference study, location is also the most important contextual criteria for selecting a mobile interaction technique when evaluating them with a hybrid simulation. • H-3: Similar to the reference study, the user’s motivation to make any physical effort is also generally low when evaluating them with a hybrid simulation. Independent and Dependent Variables

In order to investigate our hypotheses, we defined independent and dependent variables. The independent

321

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

interaction techniques. Touching was preferred in scenario 2, where the user is close to the desired object. Pointing is preferred in scenario 1, where the user is around 3 meters from the object but the object is in her line of sight. Scanning at last is clearly preferred if the object is in another room. There is a tendency to also prefer scanning if the object is in the same room but not in the line of sight (scenario 4), but this preference is not significant. Scenario 4 reveals that the chosen technique might be dependent on the activity but the results are not conclusive in either study.

to get a deeper insight whether the hybrid simulation is really meaningful. The now described advantages and disadvantages mainly base on our gained practical experience by the execution of a user study as a hybrid simulation. In some points the hybrid simulation benefits compared to a traditional laboratory setting and virtual simulation alone approach. (1) Compared to a traditionally laboratory setting there is no need to physically rebuild the user’s environment in a laboratory anymore. Thus, the designers can save money and time. (2) Relying on the hybrid simulation, even initial ideas of pervasive interfaces for mobile phones can easily and efficiently be mediated and investigated because the mobile application can be tried out and demonstrated in the corresponding simulated pervasive computing environment. (3) Another benefit is the ease of changing the environment. Different models of physical objects can rapidly be generated, modified and deleted. Thus, different settings of a pervasive environment can be arranged and tested in user studies. Using Second Life as virtual world adds further advantages. (4) Due to its widespread use, it is known to a great number of users who do not have to be introduced to the specifics of using the virtual environment. (5) A further advantage is the mobility of the test setting. Because the application realises a multi player platform over the internet, it can be accessed anywhere anytime. Thus, user tests can be run outside the laboratory in the user’s familiar surroundings. (6) This can also reduce the organisational effort of subject recruiting since the subjects do not need to be physical present anymore that is similar to the execution of online surveys. Consequently, user studies can quick and easily be conducted with a large number of participants. Second Life has attracted a large number of users. These are potential test users for our interaction concepts. Of course some restrictions apply like the necessity of compatible mobile devices. (7) Finally, in contrast to virtual simulation alone approach, the hybrid simulation also arise the benefits that the evaluation can be performed more similar to the real setting. The users can directly interact with the real mobile phone which can increase the level of immersion.

The findings from our hybrid simulation are comparable to our reference study. (H-1) They provided evidence for our first hypothesis: Similar to the reference study, the users also tend to switch their mobile interaction technique based on their contextual situations when evaluating them with a hybrid simulation. None of the participants used the same interaction technique in each tasks. Each of the participants assessed the situation and balanced reasons which interaction technique would fit best to which context. (H-2) We also could corroborate the second hypotheses: Similar to the reference study, location is also the most important contextual criteria for selecting a mobile interaction technique when evaluating them with a hybrid simulation. The execution of the user test with the hybrid simulation led to the result that location is the most important context factor which influences the decision for an interaction technique. In all four tasks the users tended to use the interaction technique dependent on the location of the avatar and the physical objects. If touching or pointing were possible they preferred these techniques. If there was no line of sight, the subjects tended to switch to scanning in order to prevent movements of the avatar. (H-3) The third hypotheses could also be partly proved: Similar to the reference study, the user’s motivation to make any physical effort is also generally low when evaluating them with a hybrid simulation. The subject’s motivation to spend physical effort was almost as low as in the reference study. But, in the hybrid simulation more subjects were willing to move the avatar in Second Life for performing touching or pointing, however, this difference is not statistically significant. A higher subject’s motivation to spend physical effort is not completely surprising when using the hybrid simulation because the subjects did not have to actually move themselves but just navigate their avatar through the environment which is not comparable in effort to the real setting.

Despite these promising benefits, there are also problems. Of course, there inevitably is an offset between a real world setting and a hybrid simulation. (1) The user moves virtually instead of physically which means a break because the user requires less motivation and less physical effort to move and explore the virtual setting. (2) A further problem of the hybrid simulation is the level of immersion for the mobile interaction. In our user study we applied scanning, pointing and touching as interaction techniques. Scanning is easy to evaluate with a hybrid simulation because it is completely realised on the mobile phone and therefore quite similar to the real usage but techniques, such as pointing and touching lead to a further break because they in-

DISCUSSION OF THE HYBRID SIMULATION

Our user study showed that we gained very similar knowledge about the user’s behaviour from results of the hybrid simulation compared to results from the reference study. Consequently, a first indicator points to the assumption that the hybrid simulation seems to be an appropriate evaluation method for early stages of the design process. Detailed benefits and problems of the method, however, must also be addressed in order

322

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

evitably require interactions with the pervasive environment. Our implementation of pointing is fully realised in the virtual world but instead should preferably be realised in the real world to reduce breaks to the real usage of the phone. We came up with the idea to replace our implementation of pointing and instead use the accelerometer of the mobile phone to point towards the screen of the virtual world simulation for selections of the objects, such as the DVD player. (3) Having too many interactions in the virtual simulation also leads to a problem of usability. Sometimes users had problems to perform pointing in our setting because it required the knowledge of key sequences. (4) A last problem is to generate the pervasive environment as realistic as the real setting. Developers require appropriate skills to virtually model the pervasive environment and set up the whole system to run a hybrid simulation.

3. Marco de S´a and Lu´ıs Carri¸co. Low-fi prototyping for mobile devices. In CHI ’06: CHI ’06 extended abstracts on Human factors in computing systems, pages 694–699. ACM, 2006. 4. Marco de S´a and Lu´ıs Carri¸co. Lessons from early stages design of mobile applications. In MobileHCI ’08: Proceedings of the 10th international conference on Human computer interaction with mobile devices and services, pages 127–136. ACM, 2008. 5. Anind K. Dey. Understanding and using context. Personal Ubiquitous Comput., 5(1):4–7, 2001. 6. Henry B. Duh, Gerald C. B. Tan, and Vivian H. Chen. Usability evaluation for mobile device: A comparison of laboratory and field tests. In MobileHCI ’06: Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services, pages 181–186. ACM Press, 2006.

CONCLUSION

In this paper, we investigated the hybrid simulation as an evaluation method in early stages development of pervasive interfaces for mobile phones. First, we described how to build this kind of simulation. Then as a first research step, we applied our prototypical setting of a smart environment and conducted a comparative user study in order to find potential problems and benefits of the method. Potentially, the hybrid simulation has many benefits, such as it can easily address a lot of subjects which can save time and money when organising and performing user evaluations. However, there are also problems which have to be considered, such as the level of immersion for the user. The experience we gained through the execution of a hybrid simulation points to a need to keep as many user interactions as possible in the real world and try to simulate as less user interactions in the virtual world as required. In that way, the offset between the two worlds can potentially be reduced and consequently the quality of the results improved. In future work we will address the mentioned problems more detailed to find appropriate solutions. But despite these problems, we already consider the hybrid simulation as a very promising method to improve the user-centred development of applications in the context of Pervasive Computing.

7. Gita Gopal, Tim Kindberg, Tim Kindberg, and et al. John Barton. People, places, things: web presence for the real world. In In proceedings WMCSA2000., pages 365–376, 2000. 8. Lada Gorlenko and Roland Merrick. No wires attached: Usability challenges in the connected mobile world. IBM Syst. J., 42(4):639–651, 2003. 9. Mieke Haesen, Joan De Boeck, Karin Coninx, and Chris Raymaekers. An interactive coal mine museum visit: prototyping the user experience. In HSI’09: Proceedings of the 2nd conference on Human System Interactions, pages 543–550, Piscataway, NJ, USA, 2009. IEEE Press. 10. Jonna H¨akkil¨a and Minna Isomursu. User experiences on location-aware mobile services. In OZCHI ’05: Proceedings of the 17th Australia conference on Computer-Human Interaction, pages 1–4. Computer-Human Interaction Special Interest Group (CHISIG) of Australia, 2005. 11. Jonna H¨akkil¨a and Jani M¨antyj¨arvi. Developing design guidelines for context-aware mobile applications. In Mobility ’06: Proceedings of the 3rd international conference on Mobile technology, applications & systems, page 24. ACM, 2006.

ACKNOWLEDGEMENTS

This research is partly sponsored by the research unit “OC-Trust” (FOR 1085) of the German Research Foundation (DFG).

12. Morten Hertzum. User testing in industry: A case study of laboratory, workshop and field tests. In In A. Kobsa and C. Stephanidis (Eds.), User interfaces for all. Proceedings. 5. ERCIM workshop, pages 59–72, 1999.

REFERENCES

1. Rafael Ballagas, Jan Borchers, Michael Rohs, and Jennifer G. Sheridan. The smart phone: A ubiquitous input device. IEEE Pervasive Computing, 5:70–77, 2006.

13. Stephanie Houde and Charles Hill. What do prototypes prototype? Handbook of Human-Computer Interaction.

2. John J. Barton and Vikram Vijayaraghavan. Ubiwise: A ubiquitous wireless infrastructure simulation environment, tech. report hpl2002-303. HP Labs, 2002, 2002.

14. Dongsik Jo, Ungyeon Yang, and Wookho Son. Design evaluation using virtual reality based

323

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

prototypes: towards realistic visualization and operations. In MobileHCI ’07: Proceedings of the 9th international conference on Human computer interaction with mobile devices and services, pages 246–258. ACM, 2007.

26. Ricardo Morla and Nigel Davies. Evaluating a location-based application: A hybrid test and simulation environment. IEEE Pervasive Computing, 3(3):48–56, 2004. 27. Brad Myers, Scott E. Hudson, and Randy Pausch. Past, present, and future of user interface software tools. ACM Trans. Comput.-Hum. Interact., 7(1):3–28, 2000.

15. Anne Kaikkonen, Aki Kek¨ al¨ ainen, Mihael Cankar, Titti Kallio, and Anu Kankainen. Usability testing of mobile applications. Journal of Usability Studies, 1(1):4–17, November 2005.

28. Jakob Nielsen and Thomas K. Landauer. A mathematical model of the finding of usability problems. In CHI ’93: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 206–213. ACM Press, 1993.

16. Eeva Kangas and Timo Kinnunen. Applying user-centered design to mobile application development. Commun. ACM, 48(7):55–59, 2005. 17. Jesper Kjeldskov and Connor Graham. A review of mobile hci research methods. In Mobile HCI, pages 317–335, 2003.

29. Antti Oulasvirta. Finding meaningful uses for context-aware technologies: the humanistic research strategy. In CHI ’04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 247–254. ACM Press, 2004.

18. Jesper Kjeldskov, Connor Graham, Sonja Pedell, Frank Vetere, Steve Howard, Rine Balbo, and Jessica Davies. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour and Information Technology, 24:51–65, 2005.

30. Antti Oulasvirta, Esko Kurvinen, and Tomi Kankainen. Understanding contexts by being there: case studies in bodystorming. Personal Ubiquitous Comput., 7(2):125–134, 2003.

19. Jesper Kjeldskov and Jan Stage. New techniques for usability evaluation of mobile systems. International Journal of Human-Computer Studies, 60(5-6):599–620, May 2004.

31. Christian Pallay, Matthias Rehm, and Ekaterina Kurdyukova. Getting acquainted in second life: human agent interactions in virtual environments. In ACE ’09: Proceedings of the International Conference on Advances in Computer Enterntainment Technology, pages 36–43. ACM, 2009.

20. Sangyoon Lee, Tian Chen, Jongseo Kim, Sungho Han, Zhi-geng Pan, and Gerard J. Kim. Affective property evaluation of virtual product designs. In VR ’04: Proceedings of the IEEE Virtual Reality 2004, page 207. IEEE Computer Society, 2004.

32. Vinny Reynolds, Vinny Cahill, and Aline Senart. Requirements for an ubiquitous computing simulation and emulation environment. In InterSense ’06: Proceedings of the first international conference on Integrated internet ad hoc and sensor networks, page 1. ACM, 2006.

21. Karin Leichtenstern and Elisabeth Andr´e. The assisted user-centred generation and evaluation of pervasive interfaces. In 3rd European Conference on Ambient Intelligence (AmI), 2009. 22. Karin Leichtenstern and Elisabeth Andr´e. Mopedt: features and evaluation of a user-centred prototyping tool. In EICS ’10: Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems, pages 93–102. ACM, 2010.

33. Stephanie Rosenbaum, Janice Anne Rohn, and Judee Humburg. A toolkit for strategic usability: results from workshops, panels, and surveys. In CHI ’00: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 337–344. ACM, 2000.

23. Joshua Lifton and Joseph A. Paradiso. Dual Reality: Merging the Real and Virtual. In Proceedings of the First International ICST Conference on Facets of Virtual Environments (FaVE), July 2009.

34. Enrico Rukzio, Karin Leichtenstern, Vic Callaghan, Paul Holleis, Albrecht Schmidt, and Jeannette Chin. An experimental comparison of physical mobile interaction techniques: Touching, pointing and scanning. pages 87–104. 2006.

24. Tony Manninen. Multimedia game engine as distributed conceptualisation and prototyping tool contextual virtual prototyping. In Proceedings IMSA2000 Conference, Las Vegas, NV, IASTED/ACTA Press 2000; 99104, pages 7–8. University Press.

35. Ichiro Satoh. Flying emulator: Rapid building and testing of networked applications for mobile computers. In in Proceedings of Conference on Mobile Agents (MA2001), LNCS, pages 103–118. Springer, 2001.

25. Ji-Ye Mao, Karel Vredenburg, Paul W. Smith, and Tom Carey. The state of user-centered design practice. Commun. ACM, 48(3):105–109, 2005.

36. Mark Weiser. The computer for the 21st century. Scientific American, February 1991.

324

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

XTag: Designing an Experience Capturing and Sharing Tool for Persons with Aphasia Abdullah Al Mahmud, Rikkert Gerits Dept. of Industrial Design Den Dolech 2, 5600 MB Eindhoven, The Netherlands [email protected]

Jean-Bernard Martens Dept. of Industrial Design Den Dolech 2, 5600 MB Eindhoven, The Netherlands [email protected] prominent and disabling characteristics of aphasics. The basic premise in our research is that enabling aphasics to better recount and express what has happened to them during the day can help them to regain some confidence in their ability to communicate with others.

ABSTRACT

In this paper we describe the design and exploratory field evaluation of an experience tagging and sharing application for people with expressive aphasia. We conducted a probe study with representatives from this target user group to gain a deeper understanding of the potential use of technology to capture and share everyday experiences. We used the obtained insights in the design of a new experience tagging tool (XTag). Our field study with the resulting prototype suggests that multimedia (picture, audio and GPS) indeed offer great potential for assisting aphasics while retelling their past experiences. Specifically, the tagging application improved support over a digital camera as it could be more easily operated single-handedly, which was much appreciated by aphasics. We also share some methodological lessons that we learned from our study.

Aphasia is an acquired communication disorder that cannot be cured, so that most aphasics have to learn how to cope with this disability. Specifically, individuals affected by expressive aphasia have difficulties expressing themselves through language. They can potentially benefit from technical support during the creation of a story and/or while expressing it. Aphasic individuals usually retain the ability to recognize image-based representations of objects [33]. This is extensively used in different assistive tools [29, 36], which are often referred to as Augmentative and Alternative Communication (AAC) devices. Both high-tech and low-tech AAC devices, are used extensively in aphasia therapy [35]. AACs mostly assist users in basic communication needs during the early stage of aphasia. However, AAC devices often fail to meet the needs beyond the initial stage. Therefore, the acceptance of AAC devices among aphasics decreases in the long run [21]. Though AAC devices have obvious merits, they also have drawbacks such as (a) AAC devices mainly support needbased communication, (b) AAC devices depend on the use of icons and symbols, which restricts the use since not all functions can be mapped to icons that are easily understood and remembered, (c) AAC devices stigmatize people, which implies that they prefer not to use them in a social setting.

Keywords

Aphasia, sharing experiences, photo tagging, technology probe, digital photo, storytelling, inclusive design ACM Classification Keywords

H5.2. [Information interfaces and presentation]: User Interfaces-Evaluation/methodology, user-centered design; K.4.2 [Computers and Society]: Social issues-Assistive technologies for persons with disabilities INTRODUCTION

An important part of our personal life is being able to share with others different kinds of stories. Storytelling is a key element of social interaction, as it allows a person to develop his or her bond with other people [30]. When a person, due to injury or disease, looses or becomes limited in his/her ability to use language (for example, because of aphasia) she or he also loses the ability to share stories, which can in turn lead to increased social isolation [16]. Language disturbances interfere with narrative communication at the single word or sentence level [26, 4], so that this ability to share stories is one of the most

People have various communication needs such as sharing information, social closeness and social etiquette [24]. AACs fall short to meet such needs of people with aphasia. There are very few examples of systems that aim at supporting such higher level communication needs. One example is the ESI planner [28]. ESI is a multi-modal daily planner designed to enable individuals with aphasia to independently manage their schedule. There are other isolated systems such as desktop-PDA [7], and Photo Talk [2] aimed at persons with aphasia. All these systems are intended to support some specific tasks, and are less suitable for complex communication needs such as storytelling. Social exclusion and social isolation are often uttered as prominent characteristics of persons with aphasia [9, 12]. We believe that technology, when properly

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

325

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

designed, has the potential to assist in reducing such social isolation. Specifically, we think that technological support to capture, build and share daily stories is likely to help aphasics to reduce social isolation and to improve their quality of lives.

We discuss different stages in the design of a new capturing and sharing device, called XTag, intended to enable aphasics to better capture everyday experiences. This device is intended to be used in combination with other life logging tools. XTag is different from existing life logging systems in at least two respects: (1) it assists in capturing experiences manually, and (2) it assists in categorizing the captured materials, which should make it easier to query them later with the help of a narrative interface. Our long-term goal is to use XTag as one of the components in an overall system that helps aphasics in communicating everyday experiences as a form of conversational storytelling. The contributions of this paper are therefore: (a) a better understanding of how technological aids can support communication in persons with aphasia, (b) the design of a specific photo tagging and sharing application (XTag) that is inspired by principles of storytelling, and (c) the results of a limited field evaluation of XTag.

RELATED WORK

Storytelling is not only essential for the transfer of information, but also for promoting social connection and maintenance or establishment of social roles. Therefore, storytelling exceeds the ability to express direct needs and wants and supports other communication goals. There are several distinct aspects that need to be addressed when designing a storytelling system for aphasics, i.e., (a) the ability to (re)collect materials in support of the story, (b) the ability to build a story from these materials, and (c) the ability to express the story to others or to let others explore the story by themselves. Each of these components creates a challenge in its own. Storytelling can potentially be done without (spoken) language by means of multimedia such as photos, as was demonstrated in some recent prototypes [5]. However, these systems were not tailored to persons with aphasia. Digital photos are beneficial for several purposes such as for cognitive assistance [11], supporting conversation [17], creating personal stories [20, 27], and self expression [23]. As the cognitive abilities of many people with aphasia are not seriously affected, we can assume that they can potentially utilize digital photos to express daily experiences if appropriate support is provided.

In the following sections we first describe the requirements gathering process that involved a probe study with two aphasics. In the conceptual design stage following this study three concepts were explored before settling on one specific concept for further development and testing. The resulting prototype was evaluated in a limited field test with the same two aphasics used in the probe study. We present the outcomes of this field test together with some methodological lessons learned. We finish by comparing and contrasting our own design to existing solutions in the extended discussion part.

Life logging tools such as SenseCam [6, 14] help their users to collect daily experiences continuously and passively. The downside of such systems is however that the volume of collected information grows very fast and that an increasingly large effort is required to select from these materials for story creation [8]. Life logging tools have however been used by patients with dementia and Alzheimer to assist their memories, rather than to build stories [22]. AACs such as TouchSpeak [36] and MindExpress [29] have been used by persons with aphasia to express simple stories, despite the fact that these AACs offer only very limited storytelling support. One early exploration of improved storytelling support for persons with aphasia was discussed in [10]. The resulting prototype has some obvious limitations: 1) it was not easy for aphasics to capture events with the web camera while the camera was attached to a tablet pc, 2) the prototype did not offer ample support to organize the captured materials, such as photos, while constructing and expressing experiences, and 3) there was no provision for aphasics to collect experiences in a mobile setting and later arrange those materials to create stories or re-experience the past. There is hence ample room for improving different aspects of the storytelling experience. In this paper we focus on how people with expressive aphasia can be assisted in gathering story materials such as photos, audio, etc.

REQUIREMENTS GATHERING

Technology probes have been used frequently in design environments where it is difficult to apply standard ethnographic methods [18]. We conducted a technology probe study with two aphasic participants. The objective was to determine the potential impact of media such as photos, sound and GPS on the ability of aphasics’ to (re)tell personal experiences. Data was collected through logging of probe usage and by conducting semi-structured interviews. Participants

Two participants (P1 & P2) were recruited through a local rehabilitation center. Earlier a certified speech therapist administrated the Aachen Aphasia Test (AAT) to participant P1. The Aachen aphasia test is a standardized test for aphasics to determine progression in different stages of the aphasia [19]. The test addresses six languagerelated aspects: spontaneous language, token recognition, repeating, written language, naming, and comprehension. The other participant P2 was unable to do the AAT and the speech therapist conducted an Implicit Association Test (IAT) that provides a measure of strength of automatic associations [13]. P1 is an adult male (age 56) who, as a result of a stroke about one year ago suffers from aphasia and lives with his

326

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

wife. P1 speaks full sentences at a fluent pace, but sometimes makes word-choice errors. His AAT test shows that his speech, writing and language comprehension impairments are mild. P1 was unable to return to his job as a mechanical engineer following his stroke. Despite P1’s communication and physical impairments, he is still comfortable performing many activities independently. P1 attends speech therapy once a week. P2 is an adult male (age 53) who, as a result of aortic surgery 2 years ago, suffers from severe aphasia and right-sided paralysis. P2 is unable to speak more than a few isolated words, and is able to write only partial single words. The semantic association test showed that he suffers from severe speech and writing impairments. His comprehension and visual semantics are moderately impaired. P2 was an assistant manager at a grocery store and he was unable to resume work due to his brain injury. However P2 still is able to operate appliances like desktop computers, TV sets, small PDA like devices. For communication he uses limited speech, gestures, or written notes. P2 attends speech therapy once a week. P2 lives with his wife, daughter and grandson.

Figure 1. The technology probes used in the study.

all times. The events P1 captured were: playing with the dog, internet banking, cycling, watching TV, waking up, breakfast, cab ride, speech therapy, relaxation, and fitness. P2 did not use the sound recorder and the GPS data logger. He captured images of meeting people at the rehabilitation center, of relaxing, a picture of his therapist, etc. The captured images were categorized as people (who), places (where) and activity (what) category. Most of the captured materials were related to the activity category. P1 used pictures and audio for assistance while sharing experiences. His memory and communication skills were sufficient to share experiences without the help of additional media. However, he did point out that pictures and audio supported him to explain and share his experiences in a more detailed manner. Some of the pictures that P1 had taken were very blurry up to the point where he had problems remembering their content. While discussing the usefulness of the photos for experience sharing, P1 mentioned that the chronology of events was of no great importance to him. Chronology of pictures and other media within a single event however did matter and such media should hence be properly sequenced in time. A few usability problems were uncovered during the field study. As explained, the autofocus feature of the digital camera created problems in that several pictures were out of focus. Both participants were also uncomfortable with operating the camera single handedly, as the camera was clearly not designed with such usage in mind. P1 wished to take pictures of himself when he wanted to share his feelings, which was simply too difficult with the existing camera. We analyzed the picture content to understand what kind of stories people conveyed with the photos. We observed that aphasics easily recognized the photos and the underlying stories behind the photos. Words expressing emotions were often used while expressing stories. Digital photos created shared content and context for the discussion partner. Since the communication partners were not present during the photo capturing, there was a need for detailed descriptions while showing the captured information, unless the materials were self explanatory or preprocessed. For P1 less prompts were required to extract the underlying stories from the photos. Spontaneous narrations were most frequent for P1, while for P2 induced narrations were dominating. This was in accordance with the fact that P1 only has mild aphasia, while P2 is a severe aphasic. The word/sentence length in case of P1 was significantly longer than in case of P2. P2 found it very difficult to express himself through the photos. He was

Procedure

The researcher met the participants at home and meetings were accompanied by family members for support. The first meeting lasted approximately one hour and background information, skills and test strategies were discussed. Probes for the test were: a digital camera, a voice-recorder, and a GPS data logger (Figure 1). Participants were asked to take pictures of events/experiences they would like to share, for a period of one week. Audio samples could be collected as well. Furthermore they received a GPS data logger which kept track of their locations over time. At the subsequent meeting (day after capturing), the researcher asked questions about how they had been using the probes for capturing experiences. The discussion involved several steps. Firstly, sharing the captured experiences without the help of pictures, audio, or GSP information as participants had to tell about the experiences they captured just from memory. Secondly, the participant used the captured pictures, audio and GPS data for retelling the experiences. Participants narrated the experiences by browsing through the pictures one by one with the help of MS PowerPoint. During this round the researchers asked questions to elicit detailed stories and provided cues that could be suitable for aphasics while retelling an experience. Thirdly, the participants were asked to cluster and categorize pictures into several events. Finally, the participants were asked to answer some post-test questions and rate some statements. The sessions were videotaped and observation notes were taken for further analyses. Results

Both P1 and P2 used the probe(s) outside of their home and in a social setting such as during a group therapy meeting. P1 took 19 photographs and P2 took 8 photographs. P1 recorded 1 audio sample and used the GPS data logger at

327

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2. Expressing emotions - Concept 1 (left), concept 2 (middle), & concept 3 (right).

mostly focusing on what was in the pictures rather than on the underlying experience. For P2, much support was required from the communication partner in the form of cueing to answer something properly, or questioning to gain details of the event being portrayed. For example, in one of the photograph, P2 recognized the person but was unable to name her. His wife and daughter were helping him by uttering the first few letters of the name.

for happy. By shaking the cylinder vertically the intensity of the color can be influenced, hence expressing the intensity of the mood. The third idea (Figure 2-right), which will be explored further, is to add information that is not continuous but discrete, i.e., by selecting an emotion and a category from a number of predefined options. In all concepts, the capturing action could be performed by a camera mounted on an ear headset (or on a pair of glasses).

Design Implications

Final Design: XTag

We have observed that digital photos, sound and GPS data have potential advantages for persons with aphasia. However, how to utilize those media for expressing day-today experiences is less obvious. From the probe study we summarize the following requirements for improved support:

The initial design ideas were discussed with the speech therapist. The design opportunities were reviewed and it was decided to develop a tool that integrates the capturing of pictures with sound recording. The action of taking a picture was divided into multiple actions, namely aiming, triggering and adding a mood or category. To separate the triggering and aiming actions a remote triggering device was proposed. As we have observed in the requirements gathering phase, operating and aiming a digital camera is cumbersome for aphasics. Therefore, similar capturing devices such as a mobile phone camera may also pose a challenge for them. Consequently, a sunglass camera would potentially be helpful for aphasics as a capturing device. A sunglass camera has several advantages for aphasics such as a) aphasics can wear the glasses and look at a particular object and can take pictures easily, b) aiming the sunglass camera is easy as it is in line with head movement and reduces the burden for aphasics as they do not need to grab the capturing device with both hands. We used an off-the-shelf sunglass camera that can capture 1.3 megapixel still images (at a resolution of 1280x1024) (Figure 3). The sunglass camera has a remote control (RF remote frequency: 2.4GHz) to capture photos and 1 GB of storage. We designed a new handheld controller (tagging tool) to add more functions to the existing remote controller. The resulting tagging tool has several buttons which reflect some storytelling principles, such as answering (some of) the 5 Ws (who, where, when, what and why). Three of the buttons are labeled as: who, where/what and emotion. Either auditory or visual feedback should be provided when one of these buttons is pressed. The four remaining buttons are used to add emotional expression to a photo, and have emoticons on them. The four emoticons are happy, sad, amazed, and

• Aphasics would benefit from a simpler capturing device that they can operate with their non-dominant hand only. • Categorization of the captured media would help aphasics to recall events later on. Support to tag and express emotions therefore seem interesting. • Support for categorizing and re-telling the underlying experiences could be useful. DESIGNING A TAGGING AND SHARING TOOL Conceptual Design

Several design ideas were generated through brainstorming. All ideas aim at coupling extra information to the picture being taken, such as the mood of the user at the time when a picture is captured. The large button on the top of the remote controller device in Concept 1 of Figure 2(left); triggers the digital photo camera. By sliding the slider to the left or right aphasics can express if they feel happy or sad when taking the picture. The second idea (Figure 2-middle) is an extension of the first idea. Instead of adding a one dimensional emotion to the picture, this idea proposes using two-dimensional information. Again, by pressing the button on top of the cylinder the digital photo camera is triggered. The top part of the cylinder can however also be turned, and feedback is provided though changing the color of the light within. The different colors represent different moods such as red for angry and yellow

328

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

angry. It has been demonstrated that such emotions can play a facilitating role in the communication with language-impaired people [25]. To re-experience the captured materials a narrative interface was designed which can playback the captured materials (pictures and audio) together with the annotations made with the remote control.

experience are printed out in the narrative template at the time the experience is shown in the narrative interface (Figure 5). The user can transfer the captured pictures, audio and annotations into the narrative interface by connecting the XTag device, the sunglass camera and the GPS logger through USB connectors. The scrollbar on the right in the interface allows to scroll through the captured experiences of a single day, while the calendar allows to select the day of interest. Two pictures, captured with the “who” and “what/where” button, respectively, are shown next to a textual field (currently containing day and time of the recording) and an emoticon field. The user can select the “who” or “what/where” picture by clicking on it, after which he can start and stop playing the corresponding audio recording by clicking on the play button underneath the pictures. Users can play the sound recordings by clicking the play button and scroll through each experience by using the scroll button situated at the right side of the narrative interface.

Figure 3. The sunglass camera and the remote controller [32] Implementation: XTag and Narrative Interface

The tagging tool (Figure 4) was prototyped using an Arduino controller and the remote controller of the sunglass camera. A sound recorder was integrated into the tagging tool. The casing was designed in SolidWorks[31] and the narrative interface was designed using Flash and Action script. The tagging tool has three large input buttons to select the categories. In addition there are four smaller input buttons to select the desired moods. Pictures are captured by pressing any of the trigger buttons (who, what/where). Sound fragments are captured by pressing a button placed on the left side of the controller. A sound recording can be stopped manually by pressing this button a second time, or the user can leave it up to the system to stop the recording automatically after 1 minute. A typical experience capture follows a sequence of actions. Firstly, pressing the ‘Who’ button trigger a picture capture and will turn the indicator next to the button green. The user can make a (first) sound recording while this green light is on. Secondly, pressing the next button which is ‘where/what’ button will trigger a second picture recording, as well as a recording of the GPS coordinates. The indicator light next to the ‘Who’ button turns red, indicating an off state, and the indicator light next to the ‘Where/What’ button turns, indicating an active state. The user can again make a sound recording if he wants to. Finally, the emotion button is pressed, which turns the green light next to the ‘Where/What’ button red, a desired emotion can be chosen using any of the four buttons with emoticons. The XTag device is now ready to capture a new experience.

Feedback Session with the Speech Therapist

The XTag prototype and the narrative interface were demonstrated to the speech therapist. The goal was to understand the suitability of the concept for aphasics and also to discover potential design flaws. The main concern was whether or not, in her opinion, people with aphasia would be able to understand and operate the tagging device and the narrative interface. The therapist advised us to reduce the number of options of the tagging tool as they might confuse persons with aphasia. She also mentioned that the narrative interface would be understandable by the aphasics given that some sort of support would be available from the communication partner. She appreciated the interface as it is based on explicit categories of information that are easy to understand. Based on her comments we reduced the number of emotion buttons from the original 8 to the 4 shown in the final design of the tagging tool in Figure 4.

Upon receiving the signal of one of the trigger buttons the Arduino performs a number of actions. First the Arduino activates the remote control of the sunglass camera. This is done by bypassing the button of the existing remote control of the sunglasses camera. Secondly, the Arduino stores information in its internal memory in the form of text strings. Some string contain information about the activated category (who, what/where, or emoticon), including button events related to the sound recording (start and stop time) or the selected emoticon. Another string contains information about the date, time and location (GPS coordinates) of the recording. All strings related to a single

Figure 4. The tagging tool (left and middle) and the sunglass camera on the top of a cap (right). FIELD EVALUATION

The objective of the test was to ascertain the impact of early categorization of images at the same time as the capturing event. The same two participants (P1 and P2) that assisted us earlier were introduced to the technology

329

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

probes which they were allowed to use for one week. The probes were: the sunglass camera, the tagging tool (see Figure 4), and a GPS data logger. Since both the participants already wore spectacles it was not feasible for them to wear the sunglass camera. Equipping the sunglass camera with prescription glasses was also not an option. Therefore, we attached the sunglass camera to a cap. After handing the probes there were several meetings over the week where the researcher asked questions to the participants about how they had been using the probes for capturing experiences. The discussion involved several steps. Firstly, participants had to tell about the captured experiences from memory. Secondly, the participant used the captured pictures, audio and GPS data for retelling their experiences. The captured experiences were presented on the narrative interface during this stage. Finally, the usability of the concept was discussed through a set of questions and (yes/no) statements. The partners of the aphasics were present during the interview sessions. The interview sessions were video recorded and field notes were taken.

both participants expressed that they found the concept of XTag useful and they understood its operating principles. We now provide more detailed discussion about the feedback on different aspects of the concept. Categorization and tagging

Both participants liked the idea of simultaneously tagging pictures when capturing. P1 mentioned: I think this is very convenient; categorizing pictures later on takes too much time and effort. The selections who and what/where are okay, but the freedom within one experience is very limited. Narrative interface

Both P1 and P2 liked the narrative interface as it gave a clear overview of the captured experience. However, participants mentioned that the narrative template was probably too restrictive in its current form. P1 mentioned: The interface is clear and simple, which is what I like about it. However I think I would like to see an overview of all the experiences that I took on a single day before focusing on a specific one. Both categories are always shown equally. I would like to see the category which isn’t applicable blend into the background. This way my attention can be focused only on the category that I want to tell something about, so if I tell something about a person the category what/where should fade out. Both pieces of advice within the previous comment would be worthwhile to consider in a future version of the interface.

Figure 5. The narrative interface showing the two pictures associated with a single experience, next to information on the day & time and the attached emotion. Audio recordings can be activated through the play button at the bottom. Results

Here we present general findings followed by the analysis of the two case studies. P1 only used the probes for 2 days, capturing 2 events, due to battery failure. No sound recordings were made but the GPS logger was used at all times. P1 did not have any problems operating the device. The amount of constructive feedback on the concept was satisfactory. Because of the battery failure the sunglass camera started to capture images automatically, with surprisingly positive results. P2 captured daily events such as gardening, having a chat with friends, but did not capture any GPS or audio data. P1 used the device for mundane tasks such as when he was riding a bike (see Figure 6). This was convenient for him since he could operate the camera easily while he was on the move. P1 was more independent while using the probes and expressing stories than P2. During the interview sessions

Figure 6. Some of the pictures captured by participants with XTag Advantage over traditional photo camera

Participants appreciated the design which was better than a normal digital camera. The main advantage was aiming the camera. This was justified from the comments of P1: With this tagging system you’re able to easily aim the camera just by looking at the desired object. However, the focus point of the camera is a bit problematic because the camera is now placed on a cap on my head. P2 liked a normal digital camera since it had a display device which gave direct feedback. However, he

330

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

appreciated that the tagging tool was much easier to carry and operate.

form it might cause difficulties because all the categories were present at all times. The category ‘emotions’ could be left out completely or at least should be extended with a neutral emotion. Sharing the experiences with the narrative template made it easier for aphasics to concentrate on the important aspects of the story.

Handheld controller for tagging

P1 and P2 found operating the tagging tool very easy. The buttons were large enough and the distance in between them was sufficient in order to avoid unwanted selections. The buttons for selecting the categories (who, what/where, emotions) were also clear though sometime P2 was confused by the emoticon buttons as they were smaller than the other buttons.

The placement of the photo camera on the head of the participant was considered easy, although the view of the camera in the current solution was slightly different than our initial solution. Operating the tagging tool was simple and understandable, but in its current form it might be daunting, because awareness of the system was constantly present. This might hinder daily activities. A solution would be to capture experiences semi-automatically. Additional media other than photos was useful for retelling experiences and audio was most suitable when telling about an event whereas GPS data for telling about trips. It would be advisable to convert the GPS data into an understandable picture format, such as a map. Both the audio and the GPS data should be more clearly linked to the correlating picture.

Use of multimedia to share experiences

P1 reflected that audio and GPS data would be useful additions to the narrative template for sharing experiences. GPS data was most suited for explaining the location of pictures, whereas audio would help aphasics to remember the events. He quoted the following: I like both, audio and pictures. Audio is suitable when I want to capture experiences. Audio helps me to associate images with words. Images in combination with audio can help me refresh my memory. I think audio files should be coupled to the correlating picture in a way that when wanted you can play the audio file when looking at a picture. GPS logging is very handy when I’m traveling; this way I can show where I’ve been in an easy way. The plotted map of the GPS data could be directly linked to the pictures I took. But there should be links to the pictures instead of pictures integrated into the map that would only distract me when sharing my experience.

The battery failure brought new insights to the research, namely semi-automatic capturing of events (experiences), a concept that requires the user to only initiate the capturing at regular time interval (such as 5 minutes). Such a sequence of events may still be tagged with a capturing mood. Although during the field study only few audio files were recorded, aphasics still saw the benefits of adding audio. We think that the use of GPS data would be too complex to understand for severe aphasics. Overall, the findings suggest that the concept of simultaneously tagging and capturing has merit for our aphasic participants.

However, for P2 there was no particular choice of arrangement of the collected media. He liked the pictures most and the audio. He did not capture GPS information as it was a bit complex for him to understand the photos with the GPS locations.

Another remark about the field test is the short testing period. A preliminary usability test could have indicated that the battery life of the tagging tool was limited to 14 hours. Although only two participants were recruited for the field study, still the field test provided us with valuable insights.

Manual vs. automatic capturing

The participants mentioned that there could be a semiautomatic mode for capturing photos. For example, capturing an experience just by pressing the start button and pictures will be taken every 10 minutes. This would help the person to concentrate on the activity without constantly having to think about taking pictures. Another recommendation was to record sound fragments every time a picture is taken. Therefore, while sharing experiences with photos audio can be played to help refresh the memory of the aphasic.

Comparison and Contrast with Capturing and Sharing Devices

other

Experience

Here we would like to discuss the comparison and contrast of our approach with other works. First of all, our approach is different from existing storytelling system [8] in a number of ways. Our work extends the option of capturing life experiences to a mobile setting, whereas existing systems [8] are more limited, as the capturing device is a web camera that needs to be connected to a computer, such as a tablet pc. As it is well known that most aphasics have right sided paralysis, operating such a camera-computer combination is quite tedious for them. Moreover, this would be mostly restricted to an indoor (home) context. In our approach it was feasible to collect experiences both at home and outside. The sunglass camera helps to focus on a particular thing and reduces the burden of operating it. The notion of a

DISCUSSION

The field test showed that XTag helped to view the captured experiences by creating a shared content and context for both the aphasic person and their communication partners. The narrative template with the three different categories was clear and understandable. However, enhancing the freedom to go back and forth between the different categories within capturing one experience would make the device more flexible. The narrative template provided guidance, but in its current

331

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

camera build into a pair of sunglasses, although popular in areas of security, seems to be quite novel for our application of supporting aphasics. The use of the tagging tool helps to capture the information in a more structured way, which also reduces the burden to organize the materials for story building afterwards. The prototype reported in [10] does not provide support for the organization of the captured materials. The narrative interface that we propose helps to view the captured materials in a more structured way. Our approach is also different from traditional tagging applications [3] where pictures are explicitly tagged as people, places, etc. In our approach the captured photos are categorized as who, where/what, emotion and date and time. These categories add an additional label to the captured information and are in agreement with categories used in existing Augmentative and Alternative Communication (AAC) devices and also in the therapy book for aphasics created by the Dutch aphasia union (afasie.nl). These categories are known to be effective for persons with aphasia. In our approach with XTag information is captured manually by the participants, whereas in traditional life logging systems the information is collected autonomously [6, 14]. One obvious difficulty is the huge number of materials collected by existing life logging systems which would probably requires partner assistance to help filtering irrelevant information. Another concern with a system such as SenseCam is that it uses a fish eye lens. The resulting pictures are therefore not rectangular and of insufficient quality for subsequent viewing. The pictures taken by the sunglass camera are rectangular in shape and have fairly high resolution. SenseCam needs to hang on the user’s neck and thereby it moves frequently when the participant is on the move, which adds to the risk of capturing images with motion blur. In the sunglass camera solution, the camera is usually more stable. As evident by our field study, autonomous or semi-automatic capture might be helpful such as when the participants are on the move. However, we believe that the ideal device should have both automatic and manual functionalities for capturing life experiences. As found in earlier research, assistive technologies tend to stigmatize their users and therefore, people don’t want to use them in a social setting [21]. Therefore, our concept would help to reduce this barrier as the solution is more unobtrusive, i.e. other people might not notice that someone is using the sunglass camera and looks like a disabled person.

with aphasia who have unique problems in reading, writing and or speaking in addition to physical disabilities. However, the post-capturing interface of Isaac is interesting and might be useful for people with aphasia. The application needs to be redesigned to make it suitable for people with aphasia, however. Improvements of the Tagging and Sharing Tool

Problems were discovered with the shape and design of the tagging tool during the field study. The shape and design of the tagging tool were judged as too big and ergonomically weak. This problem was admitted by the researchers, but due to time and manufacturing limitations improvements could not be carried out. The tagging tool needs redesigning, aesthetically as well as electronically, to create a smaller shape and more robust operation. Improvements can be done to the tagging procedure as well. At present tags are stored on the tagging tool, while synchronizing pictures with the correlating tags is done manually. Further improvements are required to the presentation of audio and GPS data, whereas current research has confirmed only the merit of those media. The presentation of the events for sharing is another aspect that should be investigated. Test results show that presentation of events using the designed template (who, what/where, emotion) has advantages over the use of pictures without the narrative interface. Presenting an overview of the pictures first and then highlighting the core categories step by step as they were captured and fading out other categories could be a solution. It might also be helpful to add additional labels to the captured pictures. This could be accomplished while eliciting detailed stories from the captured materials by the communication partner by asking questions. For example, the categorized pictures can have additional tags such as name of the person, place etc. Enriching pictures with additional labels would be helpful for people with severe expressive aphasia. Redesign of the capturing device should also be investigated, as the current sunglass camera solution causes problems if users already wear prescription glasses. In the next design iteration the camera device will be integrated into the aphasics’ own glasses. We believe that for P2 due to his severe aphasia it would be advantageous to do some preprocessing with the captured images such as annotation with text/icons to clarify the story to the story recipients. Notes on Methodology

Repetitive question asking worked well for the aphasics as we observed in both the pre and post test interviews. Since aphasics mostly remain passive and quiet, repetitive question asking is most suitable for them to extract opinions. Such an approach followed from requirements gathering with Alzheimer’s patients [15].

Isaac [20] is an early application which has been designed to capture and edit photos. It is based on a version of ‘Apple Newton’ with a CCD camera and phone handset integrated in a modified cover. The total weight of Isaac is 2 kg, including a 7.2 V/2.5 Ah NiMH power pack. Therefore it is not suitable for aphasics to handle the camera. With Isaac users can select a particular photo and annotate it with the help of a keyboard. The application contains complex menus and is not intended for persons

It has been observed that involving the help of domain experts [1] can assist in the design process, despite the fact that these experts are not aphasic themselves. We however observed that the feelings and enthusiasm of aphasics could

332

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

not be understood just by asking experts. Experts can serve as a complementary source of information but never as a substitute for working with aphasics. Especially during the design stage, concept validation through expert feedback is very helpful. However, to understand specific usability problems we must deploy the system with real aphasics.

are a couple of things we can improve. One is the size of the tagging tool and also the look and feel. The other one is improvements to the narrative interface. One of the prospective directions is to enhance the interface with additional functionalities for off-line tagging (adding more information related to the 5 W’s). Another prospective direction is to look at offline ways of distributing collected experiences, such as through email. Aphasics are slower in processing but can spend time to compose stories and share them with other people. Email is such an application that aphasics can benefit from. Therefore, we would like to build add-ons for the narrative interface. The envisioned tool will be able to handle diverse communication needs of aphasics (face-to-face, offline production of stories etc.). Another key direction is the improvements of the capturing device itself i.e. the sunglass camera. As we learned that many aphasics wear prescribed glasses, it would be advantageous to build a pair of glasses with a built-in camera that would allow to simply changing the glasses within.

Mild aphasics are capable of giving their opinions and they have empathy for other severe aphasics since they can better understand and express their experience having been affected by aphasia. Therefore, our selection of two aphasics with quite distinctive abilities turned out to be very fortunate. We also believe that deploying prototypes with mild aphasics could be helpful for early evaluation. Many aphasics need explicit goal setting which is also applied during the therapy period [34]. This aspect needs to be realized if we expect them to accomplish preset tasks. Aphasics either forget things or do not feel interested to accomplish the assigned tasks. We came to know this fact from P1 and used the goal setting technique for P2. We validated this remark of P1 by setting an explicit goal for P2. During the first few days of the probe study, P2 did not capture anything. Later, with the help of his speech therapist and family members we did set a goal for P2 by discussing this with him and by writing it on a piece of paper to remind him of his goals. This helped him to remain focused to use the probe. Therefore, it is very important to determine how new technology would fit into aphasics’ personal goals which might eventually help aphasics in changing their quality of lives.

ACKNOWLEDGMENTS

We thank all the participants and their family members for their cooperation. We also thank the speech therapist for her support. REFERENCES

1. Allen, M., Leung, R., et al. (2008). Involving domain experts in assistive technology research. Universal Access in the Information Society 7(3): 145-154. 2. Allen, M., McGrenere, J., and Purves, B. (2007). The design and field evaluation of PhotoTalk: a digital image communication application for people. In Proceedings of Assets ‘07, 187-194.

CONCLUSION & FUTURE WORK

It is evident that we need to design applications that provide support beyond needs and wants. We need to realize that providing support for sharing daily stories can potentially increase the independence and social affiliation for aphasics. Our study revealed that pictures, audio, and GPS logging can have a positive effect on the ability of aphasics’ to share their experiences, and factors that influenced the capabilities for sharing personal experiences were identified. An application that allowed aphasics to simultaneously tag pictures while capturing was developed and tested with the target group. The participant found the concept useful and understandable.

3. Ames, M. and M. Naaman. (2007). Why we tag: motivations for annotation in mobile and online media. In Proceedings of CHI ’07, ACM, 971-980. 4. Ash, S., et al. (2006). Trying to tell a tale: discourse impairments in progressive aphasia and frontotemporal dementia. Neurology, 66(9):1405-1413. 5. Balabanović, M., Chu, L. L., and Wolff, G. J. (2000). Storytelling with digital photographs. In Proceedings of CHI ‘00, ACM, 564-571.

Field studies were conducted for a short period of time and therefore long term research is needed to ascertain the impact and adoption of such devices such as the one that we propose. More extensive research is necessary with more aphasics to acquire more accurate data and generalize the findings. Conducting field studies with aphasic persons is challenging, but it is essential for evaluating such products. The evaluation of the tagging tool is a first step in a more accurate and intuitive manner of sharing personal experiences for aphasics. Below we describe our future plan with this project.

6. Berry, E., N. Kapur, et al. (2007). The use of a wearable camera, SenseCam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report. Neuropsychological Rehabilitation 17(4): 582-601. 7. Boyd-Graber, J. L., Nikolova, S. S., Moffatt, K. A., Kin, K. C., Lee, J. Y., Mackey, L. W., Tremaine, M. M., and Klawe, M. M. (2006). Participatory design with proxies: developing a desktop-PDA system to support people with aphasia. In Proceedings of CHI '06. ACM, 151-160.

First of all, we would like to build the second generation of the tagging tool and narrative interface. We see that there

333

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

8. Byrne, D., Jones, G.J. F. (2009). Exploring Narrative Presentation for Large Multimodal Lifelog Collections through Card Sorting. In Proc. ICIDS 2009, pp. 92-97

22. Lee, M. L. and Dey, A. K. (2008). Life logging memory appliance for people with episodic memory impairment. In Proceedings of the 10th international Conference on Ubiquitous Computing, UbiComp '08, vol. 344. 44-53.

9. Cruice, M., R. Hill, et al. (2009). Conceptualising quality of life for older people with aphasia. Aphasiology, iFirst (1): 1-21.

23. Levin, T., Scott, B., et al. (2007). Aphasia Talks: Photography as a Means of Communication, SelfExpression, and Empowerment in Persons with Aphasia. Topics in Stroke Rehabilitation 14(1): 72-84.

10. Daemen, E., P. Dadlani, et al. (2007). Designing a free style, indirect, and interactive storytelling application for people with aphasia. LECTURE NOTES IN COMPUTER SCIENCE 4662: 221-234.

24. Light, J. (1988). Interaction involving individuals using augmentative and alternative communication systems: State of the art and future directions. Augmentative and Alternative Communication 4(2): 66-82.

11. Danielsson, H., & Svensk, A. (2001). Digital pictures as cognitive assistance. In proceedings of AAATE '01, Assistive Technology-added Value to the Quality of Life: 148-152.

25. Lorch, M.P., Borod, J.C., et al. (1988). The role of emotion in the linguistic and pragmatic aspects of aphasic performance. Journal of Neurolinguistics 11,12, 103-118

12. Davidson, B., T. Howe, et al. (2008). Social Participation for Older People with Aphasia: The Impact of Communication Disability on Friendships. Topics in Stroke Rehabilitation 15(4): 325-340.

26. Mar, R. (2004). The neuropsychology of narrative: story comprehension, story production and their interrelation. Neuropsychologia. 42(10): p. 1414-1434.

13. Greenwald, A., Nosek, B., & Banaji, M. (2003). Understanding and using the Implicit Association Test: I. An improved scoring algorithm. Journal of Personality and Social Psychology, 85(2), 197-216.

27. Mckelvey, M., A. Dietz, et al. (2007). Performance of a person with chronic aphasia using personal and contextual pictures in a visual scene display prototype. Journal of Medical Speech Language Pathology 15(3): 305-317.

14. Gemmell, J., Williams, L., Wood, K., Lueder, R., and Bell, G. 2004. Passive capture and ensuing issues for a personal lifetime store. In CARPE'04, pp. 48-55.

28. Moffatt, K., J. McGrenere, et al. (2004). The participatory design of a sound and image enhanced daily planner for people with aphasia, In proceedings of CHI ’04, pp.407-414.

15. Hawkey, K., Inkpen, K. M., Rockwood, K., McAllister, M., and Slonim, J. (2005). Requirements gathering with alzheimer's patients and caregivers. In Proceedings of Assets '05, pp. 142-149.

29. MindExpress. www.jabbla.com

16. Hillis, A. (2007). Aphasia: progress in the last quarter of a century. Neurology 69(2): 200-213.

30. Shen, C., Lesh, N., Vernier, F., Frolines, C., Frost J. (2002) Sharing and Building Digital Group Histories. In Proc. of CSCW’02, 324-333

17. Hine, N., Arnott, J. (2002). Assistive social interaction for non-speaking people living in the community. In proc of Assets 2002, 162-169

31.SolidWorks. http://www.solidworks.com/ 32.Sunglass camera: http://www.thinkgeek.com/gadgets/electronic/a0f3/

18. Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., BeaudouinLafon, M., Conversy, S., Evans, H., Hansen, H., Roussel, N., and Eiderbäck, B. (2003). Technology probes: inspiring design for and with families. In Proceedings of CHI '03, 17-24.

33. Thorburn, L., Newhoff, M., and Rubin, S. (1995). Ability of subjects with aphasia to visually analyze written language, pantomime, and iconographic symbols. American Journal of Speech-Language Pathology 4(4) pp. 174-179.

19. Huber, W., Poeck, K., & Willmes, K. (1984). The Aachen Aphasia Test. Advances in neurology, 42, 291.

34. Van de Sandt-Koenderman, W., F. & van Harskamp, et al. (2008). MAAS (Multi-axial Aphasia System): realistic goal setting in aphasia rehabilitation. International Journal of Rehabilitation Research 31(4): 314-320.

20. Jönsson, B., Svensk, A. (1995). Isaac, a Personal Digital Assistant for the Differently Abled, Proceedings of the 2nd TIDE Congress, 356-361. 21. Lasker, J. & Bedrosian, J. (2001). Promoting acceptance of augmentative and alternative communication by adults with acquired communication disorders. Augmentative and alternative communication 17(3): 141–153.

35. Van de Sandt-Koenderman, M. (2004). High-tech AAC and aphasia. Widening horizons. Psychology Press 18(3): 245-263. 36. TouchSpeak. www.touchspeak.co.uk

334

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Pipet: a design concept supporting photo sharing Bernt Meerbeek, Peter Bingley Philips Research High Tech Campus 34 (51) 5656 AE Eindhoven, NL +31402747535 [email protected] [email protected]

Wil Rijnen User-System Interaction Eindhoven University of Technology 5600 MB Eindhoven, NL +32485616907 [email protected]

Elise van den Hoven Industrial Design Eindhoven University of Technology 5600 MB Eindhoven, NL +31402478360 [email protected]

they are not much fun to interact with. In this paper we focus on an enjoyable interaction style with a future remote control, which is specifically designed for cross-device interaction in the living room.

ABSTRACT To support reminiscing in the home, people collect an increasing amount of digital media on numerous devices. When sharing their media with other people, distribution of the media over different devices can be problematic. In this paper, we address this problem by designing an innovative interaction concept for cross-device interaction to support groups in sharing photos using multiple devices. We designed and implemented the Pipet concept. Results of a comparative study show that Pipet resulted in a pragmatic and hedonic user experience.

We investigated innovative ways of interacting, such as physical interaction, instead of taking the standard set of buttons used on most remote controls. For the application context we chose to focus on a situation in which people use multiple devices to create one shared item, namely a photo compilation of a shared group activity. Each member of the group can bring his or her photos on their personal media devices, such as mobile phones, digital cameras, external hard disks and laptops. However, in order to make this shared compilation all the photos will have to be collected on one device.

CATEGORIES AND SUBJECT DESCRIPTORS C.0 [Computer Systems Organization]: General – hardware /software interfaces. H.5.2 [Information Interfaces and Presentation (e.g. HCI)]: User Interfaces - Haptic I/O, Input devices and strategies (e.g., mouse, touchscreen), User-centered design.

In order to investigate cross-device interaction we will start this paper with an overview of related work in the areas of physical interaction, multi-device collaboration, and memory recollection and photo sharing. After that, we describe the design exploration followed by the conceptual design phase and the prototype Pipet. Subsequently, we report on the evaluation of Pipet and the results. We will end with the discussion and conclusions.

GENERAL TERMS Design, Human Factors KEYWORDS Cross-device interaction, Photo sharing, Interaction design, Tangible user interface, Collaboration

RELATED WORK Physical interaction Designers and researchers explore new interaction styles that enable users to communicate with technology in a natural way, for example via speech, gestures, and physical interaction. Physical interaction or the interaction with physical artefacts in the context of an interactive product is a research topic that is becoming increasingly popular due to several reasons. For example, due to miniaturization of technology as described in the Ambient Intelligence (AmI) vision, as well as the desire to design interaction particularly for the physical skills people already have and that are not used in everyday human-computer interaction. Several topics are covered by physical interaction, e.g. embodied interaction [3], reality-based interaction [10], kinetic user interfaces [1] and of course tangible user interfaces which, as defined by Ullmer & Ishii [19], “couple physical representations (...) with digital representations”. The field of tangible interaction shows a recent focus on design, interaction and physicality [11]. A well-known example of a physical interface that is related to our work is the I/O brush [17]. This augmented paintbrush can pick up textures, colours, and movements from the real world and allows a child to make drawings with them.

INTRODUCTION The living room of today hosts an increasingly large number of technical devices, from entertainment systems, such as TVs, audio installations and game consoles, to fully automated homes, with complete atmosphere creation systems, including lighting, temperature and curtains. To make all these features accessible anywhere in the room, most of these systems come with remote controls (RC), one for each device. Even though some of these devices have the same functionality or play the same types of media, e.g. send a picture from the game console to the RGB LEDs that are part of the atmosphere creation device, this is not yet possible through one remote control that can be used across different devices. In addition, most remote controls are designed to be efficient and effective with a button for every function, but Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

When we look at examples of physical interaction, we often see a focus on single user interaction [15,17,23]. There is some work on collaborative physical user interfaces, in particular on digital

335

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

tabletops [18]. Another example of a collaborative physical interface is the Tangible Video Editor, a multi-user, tangible interface for sequencing digital video [22]. Also Nintendo Wii is an example of multi-user physical interaction in gaming. We want to combine physical interaction with cross-device interaction that supports simultaneous co-located collaboration.

using multiple devices with sharing photos and reminiscing talk. And unlike most previous work, we focus on making this activity a fun and enjoyable experience that can take place in the living room. In the next section, we continue with our design exploration. DESIGN EXPLORATION

Multi-device collaboration As described in the introduction, there is a growing number of devices in our living rooms. Many of these devices are standalone and controlled with separate RCs. The exchange of content between devices is often still difficult. Furthermore, people increasingly own personal mobile devices (e.g. smart phones and media players) containing media that can be exchanged with other people and other devices.

Introduction Following the recent focus of physical interaction on design, interaction and physicality we decided to use the design research approach according to Hoven et al. [9]. This approach facilitates designing within context of use, to start designing from a physical activity, to have an iterative process and to focus more on user experiences as opposed to usability. The main activities of our interest are recollecting and sharing memories in the living room. Hence, we decided to start exploring the design space by conducting a contextual inquiry using diaries and in addition we organized co-creation sessions to generate ideas.

Some work has been done on supporting the exchange of information between devices, both in individual and in collaborative settings. Pick-and-Drop [15] allows a user to pick up an object on a display and drop it on another display as if he was manipulating a physical object with a pen based device. Slurp [23] is another example of a device that allows a single user to move digital content from one device to another by physical interaction. Slurp can extract and inject digital media by touching and squeezing the digital eyedropper and gives haptic and visual feedback.

Sensitizing diary Six people (three couples, age 27-36) participated in the contextual inquiry. Two weeks before the creative sessions, participants received a sensitizing package. This package contained a sensitizing diary with open-ended questions and small exercises, cards with creativity triggers, and some crafts materials. The goal of the sensitizing package is to prepare people for the creative session by asking them to think about the topic of reminiscing and express related opinions, needs, aspirations, etc. People filled out the diary for five consecutive days and needed about 20 minutes per day (see Figure 1). The diaries were completed individually in their own living room.

Research on multi-device interaction in collaborative settings mainly focuses on meeting rooms. Several techniques for moving digital content from one display (e.g. personal tablet pc) to another display (e.g. shared projection display) are suggested and compared with respect to efficiency [13]. Most important in these settings is to create an efficient solution, rather than a fun and enjoyable experience.

Creative session Each couple that completed the sensitizing diary was visited for a creative session. First, the researcher recapitulated the diaries with the participants to shift to the right mindset for idea generation on the topic of interest and ask clarifying questions if necessary. In the first creative exercise, participants planned an event to reminisce with friends or relatives with whom they had a pleasurable experience, for example a holiday trip. After this cognitive mapping exercise, people were asked to project how this event would change if it were held in 2020. In the second creative exercise participants used handicrafts materials - including clay, cocktail sticks, fabrics, etc. – to make a creature that would support them in reminiscing with their friends or relatives. By working with a fictive creature, participants forget about technical limitations and focus on expressing their real needs and wishes.

Memory recollection and photo sharing Another focus of our work lies in the application area of sharing memories and everyday reminiscing. When we talk about “memory for the events of one‟s life”, which underlie everyday remembering, we talk about Autobiographical Memory [8]. And the process of remembering is often initiated and supported by memory cues [9]. In everyday life people often use photos for cuing their memories and sharing their experiences. For example, people get together to show their most recent holiday photos to each other. We all know that with the introduction of the digital photo camera these traditions are undergoing some transformations. First of all people create an increasing number of digital photos [16], while people are reluctant to organize their collections [21]. On the other hand, people are still eager to share in particular face-to-face or co-present. Within co-present photo sharing one can identify three types, namely storytelling, reminiscing talk and a combination of both [5]. In the storytelling situation people share photos with others who were not present at the time the photo was taken, while for reminiscing talk both photo owners and visitors were present, so both parties can give input to the conversation. We want to support the organisation of digital photos in the reminiscing talk condition, since we see this is even more difficult than only managing and organizing your own set of photos. Without being complete, we presented a short overview of related work. Our work extends this body of research by designing a physical interface that supports multiple users simultaneously

Figure 1 Examples of completed sensitizing diaries

2

336

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Main findings Obviously, the diaries and creative sessions resulted in a substantial amount of qualitative data, which we clustered and analyzed using affinity diagrams. In this section, we summarize the main findings of our design exploration.

putting the cube back on the table, as well as edit the pictures or add comments. XXL The concept XXL (Figure 3) is based on reaching consensus. All group members need to agree on all decisions to get the job done. Each group member browses their own set of pictures and sends them to a shared device – a huge inflatable remote control – by blowing over the display of their camera or phone. As the inflatable remote control receives more pictures it becomes bigger. By collectively pointing and manipulating the remote control, pictures are added to the compilation on the TV screen.

Participants regard the living room as both an individual and a social place. They refer to it as the place to withdraw from the surrounding world and to be safe and at ease, as well as the place where your friends and relatives are welcome. It is a place to relax and enjoy. We indentified two types of „memory cues‟ in the living room: visible cues and stored cues. The visible cues are positioned in a visible place and help to remind you of a person or event (e.g. a family picture in a frame on the wall). The stored cues are precious and often private, hence safely stored in boxes or cupboards to prevent losing it (e.g. a picture of a first lover in a shoe box). With regard to recalling memories, participants distinguish between reminiscing in an individual and a social context. Individual reminiscing mainly focuses on mood intensification and mood change, whereas social reminiscing has to do with feeling connected and having fun. Most participants enjoy making photo collages or photo albums and believe that it facilitates reminiscing with others. However, they regard this as a complex and time-consuming activity.

Dice The Dice concept (Figure 4) consists of a set of large dices that can display pictures on each of its six faces. The displayed pictures are taken randomly from the picture collections of the group members. By tilting the dice, the picture on the bottom is deleted and replaced with another picture. By choosing the tilting direction, people can influence which pictures are deleted. The pips on the upper face indicate how often people have to tilt the dice. In this way, the best pictures are collectively selected. The dices can be stacked, resulting in a physical 3D photo collage.

The main findings from the diaries and creative sessions are translated into design requirements, which are described in the next section. Design requirements We aim to design an interactive system that supports groups in reminiscing previous pleasant experiences. People should be able to use this system in a living room setting. The system should allow groups to jointly create a photo compilation and stimulate people to recollect shared experiences. Furthermore, it should support using photos from different people and devices, but offer seamless cross-device interaction. The system should offer more value as more people participate and everybody should have equal opportunities to participate. Finally, the experience of creating the compilation should be fun. The process is more important than the end-result (i.e. the compilation).

Figure 2 MirrorCube concept

CONCEPTUAL DESIGN Idea generation Starting from the requirements we stated in the previous section, we generated ideas for systems during a brainstorm session with six interaction designers. Based on these ideas, we developed five interaction concepts, which we describe in the next section.

Figure 3 XXL concept

Five interaction concepts Mirror Cube With the Mirror Cube (Figure 2), the compilation process becomes an interactive gaming experience. People sit around the table and browse through their own set of pictures. They can share a picture from their own selection with the group by sliding the picture from their camera or mobile phone towards a digital display table. Everyone has a Mirror Cube that he can put on top of a picture, thereby copying the picture to his cube. The compilation process starts when everybody has filled his cube with six pictures. People can add a picture to the compilation by

Figure 4 Dice concept

337

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

participants were asked to rank the concepts. For each concept, detailed user feedback was collected. It is beyond the scope of this paper to discuss it in detail, but overall XXL was appreciated the least (4 of 7 participants ranked it as the least appealing concept). Pipet was most appreciated (4 of 7 participants ranked it as the best concept). Its interaction style was considered universal and easy to understand, yet playful and spontaneous. Everybody can use it immediately, but you can also improve skills with practice. Based on the results of the concept evaluation, we decided to design a prototype of the Pipet concept to gather more detailed insights on the user experience of this interaction concept. Figure 5 Byb concept PIPET PROTOTYPE The metaphor of a pipette as used in chemistry is extended in both the design and the interaction. Comparable with transporting fluids from one place to another, the digital pipette is used to transfer content across devices. Because the Pipet has a very specific goal, the looks and the interaction can be designed specifically as well. Sucking and shooting is done by a squeeze action of the Pipet. As in chemistry, where one needs to be in the fluid to draw it up, the digital one will only get the picture if the Pipet is pressed against the screen of the smart phone. Likewise the content will be released if you squeeze without pressing it against the smart phone. Similar to putting the tip of the laboratory instrument in another reservoir to indicate in which one to release the fluid, the tip of the digital one has to point towards the screen the picture has to be transferred to. The faster the Pipet is squeezed, the faster the content will be shot. This results in a larger picture on the TV screen.

Figure 6 Pipet concept Byb Byb (Figure 5) is the acronym of „behind your back‟ and turns creating a photo compilation into a physical activity. People select a picture from their individual camera or phone by rubbing over the display with their T-shirt. The picture is displayed on the back of the shirt. When all pictures are selected, they are presented in a slideshow on the TV screen. A green check mark and a red cross appear on the back of randomly selected shirts. People can vote whether they want the picture in the compilation by touching the shirt with the green or the red sign. After the voting, each shirt shows a different editing tool (e.g. resize, rotate, etc.) on the back side. People need to work together to create and edit the compilation.

At a glance the Pipet concept may seem somewhat similar to Slurp [23] that uses the metaphor of an eyedropper. Like Slurp, Pipet supports touching and squeezing a digital eyedropper while providing haptic and visual feedback, with Pipet also adding auditory feedback. Where Slurp focussed on the extraction of digital media from physical objects and the selection of an appropriate display device to access it from, Pipet focuses on multi-user interaction and group collaboration using multiple devices simultaneously.

Pipet Pipet (Figure 6) offers groups a way to create photo compilations in a quick and easy manner. Each person has a digital pipette, which can be used to suck up pictures from cameras and mobile phones and shoot them from a distance onto a TV screen or smart phone. In a similar way, text message created on a phone can be send to the TV. The harder people squeeze Pipet, the larger the picture on the screen, but the more difficult it is to aim. Once a picture is shot on the screen, it cannot be removed, as if it were paint. Concept evaluation The five concepts were presented on A3 concept cards and discussed with seven participants. Most of the participants had also participated in the design exploration phase, which was considered an advantage as they were already more aware of the problems and needs in recollecting and sharing memories. Goal of the evaluation was to assess to what extent the concepts appeal and are perceived as a useful and fun alternative for reminiscing. Participants were asked to highlight the aspects they liked with green and the aspects they disliked with red. Furthermore,

Figure 7 Four 3D printed Pipet add-ons with Nintendo WiiMotes

4

338

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The Pipet prototype consisted of four hardware Pipets, four smart phones, a Windows PC and an LCD TV. The Pipets were prototyped by 3D printing a specially designed casing for the Nintendo WiiMote (Figure 7). Foam balls were used to differentiate the Pipets by colour and to provide spring action for squeezing the Pipets. Four HTC Touch Cruise smart phones were running a dedicated Windows Mobile picture browsing application (Figure 8) that could be used with the Pipets to browse photos and to suck up a photo (Figure 9) which could subsequently be shot to the TV by squeezing a Pipet. The Windows PC served as a communication hub for the Pipets and smart phones, gathering input and distributing output to these devices. Also the PC was running an application for creating photo compilations created in Flash which was shown on the TV screen (Figure 10).

Figure 8 Four smart phones with photo browsing application

The Flash application on the TV was used to create a compilation of photos per day of the shared holiday of the users. Arrow buttons on the TV screen could be used to navigate to different days, causing the photos related to the selected day to appear on the smart phones. In this way the people were informed visually about the change, next to an auditory signal. Feedback about the Pipet status was provided with a combination of visual, auditory and haptic feedback, as people may be focusing on different devices while interacting. When the Pipet tip is pressed against the mobile phone display, the Pipet tool starts vibrating. When the Pipet is squeezed the vibration stops and LEDs indicate that the photo has been sucked up by the Pipet. When the Pipet is aimed at the TV, a coloured cursor is displayed corresponding to the colour of the Pipet. When aiming at the screen and squeezing the bulb, the picture is shot and appears on TV. At the same time, a splash sound is played and the LEDs on the Pipet are switched off. When squirting a picture outside the screen, the picture will not appear. Nevertheless, the splash sound and the LEDs will indicate that the picture is shot and that the Pipet is empty.

Figure 9 Pipet “sucking” a photo from a smart phone

All users could simultaneously suck up pictures from their smart phones and splash them on the TV screen (Figure 11). In this dynamic activity other group members can immediately see which pictures have already been chosen, and react upon them. Each individual has a big influence on the result, because nobody can undo what someone else has shot to the TV.

Figure 10 Screenshot of photo compilation application on TV

The prototype did not provide all functionality of the Pipet concept. The ability to shoot photos to other smart phones was not implemented, as was the possibility to scribble text messages and shoot these to the TV. Also photos were clustered by date only, and not by location. PROTOTYPE EVALUATION Introduction Pipet aims to support reminiscing previous pleasant experiences by enabling groups to jointly create a photo compilation in a fun way. The goal of the evaluation was to find out how participants experience Pipet and to investigate the effect of Pipet on the group process. To answer these questions, we compared the Pipet prototype (P) with the control condition (C). The control condition (Figure 12) was very similar to the Pipet prototype in terms of functionality and innovativeness of the user interaction device. It only differed on the two key aspects of the

Figure 11 Four persons creating a photo compilation on the TV screen using their Pipet and smart phone

339

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Method The evaluation was conducted in a realistic living room setting in the Philips ExperienceLab, a user-centred research facility at the High Tech Campus in Eindhoven [20], and followed a withinsubjects design. Three groups of four people were recruited to participate in the evaluation (12 participants, age 18-30, 9 female). To make the setting as realistic as possible, all three groups consisted of friends that had actually been on holiday together in the past year. They were asked to provide their own set of pictures of that holiday, which were used during the evaluation session. Procedure and measurements Each group received a short explanation about the procedure of the evaluation. After signing informed consent forms, participants could briefly try out the system. When participants had no more questions, the actual task started. Participants were instructed to jointly create a photo compilation of the holiday trip they recently made. The TV showed empty slides with a certain topic or a specific day of the trip on which participants could place their pictures. The session finished when five sheets were completed or after 20 minutes. Groups alternately started with condition P or C. For the second condition, the same procedure was followed. After each condition, people were asked to complete a questionnaire. At the end of the evaluation, a short group interview was conducted. Participants received a reward for their participation.

Figure 12 Control condition: gyromouse Pipet system: the multi-user involvement and the fun interaction style („suck-and-shoot‟). The control condition consisted of a gyromouse that one group member at a time could use for selecting pictures from a shared space and dragging these to the compilation. Other group members could give instructions to the person holding the gyromouse or take it over. There were several reasons why the gyromouse was considered the best control condition for our evaluation. In our design exploration, we learned how groups currently create photo compilations. Often, they just do not create compilations together, because it is too much hassle. But in case they do create a compilation together, they often sit around a pc or laptop, which is controlled by one person. So if we want to compare our solution to the status quo, we would have a pc and mouse as control condition. However, we believe a comparison of our solution with a condition in which four people sit behind a laptop would lead to an obvious preference of participants for newer and more innovative interaction solution. Therefore, we chose for the gyromouse, which still represents the situation where one person is in control, but is new and innovative for our participants as our Pipet is.

We used different measures to address our hypotheses, including questionnaires, objective task measures, and a semi-structured group interview. The questionnaire measured two aspects of user experience (pragmatic and hedonic) [14], perceived control [7], group result satisfaction and group process satisfaction [6]. As an objective measure for people‟s level of contribution, we counted the number of photos added to the compilation. In the Pipet condition, this number was automatically logged. In the control condition, we counted this offline by analyzing the audio and video recordings. During the semi-structured interview, we gathered more qualitative data on how people experienced the two systems and advantages and disadvantages of the systems.

Hypotheses The focus of the presented work is on designing a physical interface that supports groups in photo sharing and reminiscing talk, and on making this activity a fun and enjoyable experience. Therefore, we formulated hypotheses to test the effect of Pipet on the user experience and the group process. Next, we enumerate our hypotheses, where P stands for the Pipet condition and C for the control condition:

RESULTS Next, we will summarize the results concerning the user experience (H1) and the effect on the group process (H2). Thereafter, other interesting findings are reported. User Experience The User Experience questionnaire was tested for reliability (Cronbach‟s α = 0.88), which indicates a high internal consistency of the scale. The scores of condition P and C were compared with a Wilcoxon signed rank test. The results (see Table 1) indicate that Pipet scores significantly (α = 0.05) higher on pragmatic user experience, hedonic user experience, and overall experience. The qualitative results support these findings. Participants indicated that Pipet was fun to use and 9 out of 12 participants expressed that they preferred Pipet over the control condition (1 out of 12 preferred the control condition, 2 participants had no preference).

User Experience H1. Overall user experience of P higher than in C H1a. People experience P as more pragmatic („useful‟) than C H1b. People experience P as more hedonic („fun‟) than C Effect on group process H2. P supports group process better than C H2a. People contribute more equally in P than in C

In sum, the qualitative and quantitative results give strong support for hypotheses H1, H1a, and H1b. So indeed the user experience, both pragmatic and hedonic, of Pipet was better than in the control condition.

H2b. People perceive more control in P than in C H2c. People are more satisfied with end-result in P than in C H2d. People are more satisfied with group process in P than in C

6

340

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Control

DISCUSSION The Pipet prototype proved to be significantly more fun and useful than the control condition for collaborative photo compilation creation in the living room. Also the contributions of the participants were more equally distributed using Pipet and the perceived control for group members turned out to be higher.

Pipet > Control

Pipet

Mean

SD

Mean

SD

z

p

H1a: pragmatic UX

3.08

.42

3.60

.70

-2.04

.041

H1b: hedonic UX

3.37

.94

4.23

.58

-2.94

.003

H1: overall UX

3.23

.57

3.92

.58

-2.63

.009

The difference in satisfaction with the end-result and the group process turned out to be insignificant. For the end-result this could be due to the limited aesthetical and layout options available in both conditions, since the aim was to explore the merits of the interaction style. For the group process we observed a levelling effect in the groups of participants. For example, a dominant person preferred the control condition as she was in full control using the gyromouse, while a shy group member was left out. In the Pipet condition the dominant person was less satisfied with the group process since she had to share the control, while the shy person was more positive on the group process as she was more able to contribute to the compilation.

Table 1 User Experience results Control

Pipet

Mean

25%

25%

Min.

0%

12%

Max.

60%

36%

18.7%

7.1%

SD

Although the UX research filed is expanding, methodologically the field is not mature yet [11]. Although the notion of coexperience (“experience that users themselves create together in social interaction”) has been described [1], we are not aware of validated instruments to measure the user experience (UX) in settings where groups of friends or family interact with devices. Therefore, we used a measure for individual user experiences that has already been applied successfully in the domain of Information Systems [14]. However, we recommend future work on developing valid instruments to measure user experiences in groups for consumer products.

Table 2 Mean, minimum, maximum, and standard deviation in relative contribution to compilation (% of photos) Effect on group process We found no significant differences between the conditions in participants‟ satisfaction with the end-result (i.e. the photo compilation), nor in the satisfaction with the group process. Hence, we reject hypotheses H2c and H2d. We expected that Pipet would lead to more equal contributions of participants to the photo compilation. The percentage of pictures each participant contributed to the compilation was logged and this data indicates that indeed the contribution of each participant is more equal with Pipet. A fully equal contribution would imply an average contribution of 25% (100% / 4 group members). However, in C the contributions were significantly (α = .05, F = 7.05, p = .002) less equal than in P (see Table 2). Therefore, we accept hypothesis H2a.

The number of participants was small which might have affected the results, in particular when looking at group processes. However, we did find significant differences between the two conditions for most hypotheses. Another limitation of the evaluation is the choice for the gyromouse as the control condition. The gyromouse was selected because it is commercially available and it is a new interaction style for most participants. It resembles the current single-user interaction style for sharing digital photos in a living room. Results might have been different if we would have used another control condition.

Furthermore, we expected that participants would perceive more control in P than in C. The mean scores (on a 7-point scale) for perceived control were 4.45 for the control condition and 5.09 for Pipet. A Wilcoxon signed ranks test showed that the perceived control with Pipet was significantly higher (α = .05, z = -2.040, p = .041). Hence, the results of the questionnaire show that users indeed perceive more control with Pipet and support hypothesis H2b.

The Pipet prototype supported cross-device interaction between Windows Mobile smartphones and a Flash-based TV application. A commercial product will need to support a more diverse range of devices, including a variety of digital cameras, media players, laptops, tablet computers, PC‟s, and a plethora of smartphone variants (iPhone, Android phones, etc.). This brings additional technical complexity which is outside the scope of this paper. The complexity of setting up communication between various devices will hopefully be solved in the coming years through standardization efforts. However, the related user interaction complexity poses a worthy challenge for future investigation.

Additional findings From the interviews we learned that participants appreciated the idea of collaboratively making photo compilations in a living room setting. Not the photo compilation itself, but rather the social and fun process of reminiscing was the key reason why they liked it.

CONCLUSIONS In this paper, we described our efforts in designing a cross-device interaction system supporting groups sharing photos and supporting reminiscing talk, following a user-centred design approach. We explored the design space and specified user requirements for such a system. Based on the requirements, we developed five interaction concepts. After evaluation of the five concepts, we decided to proceed with the Pipet concept. We developed a prototype and evaluated this with groups of friends

Other applications of the Pipet concept were suggested by the participants, including video editing and file exchange between storage means. Another suggestion was to use Pipet for sucking up colours from objects in the environment and paint it as an artwork on the TV. Participants also mentioned using Pipet in public spaces, for example allowing people at a party to select songs from their portable music player and „shoot‟ it to the DJ.

341

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

that had been on holiday together in a realistic living room environment. The Pipet prototype proved to be significantly more fun and useful than the control condition (gyromouse) for collaborative photo compilation creation in the living room. Also the contributions of the participants were more equally distributed using Pipet and the perceived control for group members turned out to be higher. Hence, we conclude that we succeeded in designing a useful and fun interactive system that supports groups in sharing photos. The system allows seamless cross-device interaction and enables a physical interaction style („suck-andshoot‟). We also demonstrated that Pipet resulted in more equal contributions of group members to the photo compilation, which makes it a truly collaborative process.

interaction: a framework for post-WIMP interfaces. In Proceeding of CHI‟08 (Florence, Italy, April 2008), ACM Press, 201-210. 11. Law EL-C, Vermeeren APOS, Hassenzahl M & Blythe M. Towards a UX Manifesto. Proc BCS HCI Group Conference 2007, Lancaster, UK. British Computer Society 12. Mazalek, A., and Hoven, E. van den. Framing Tangible Interaction Frameworks. In "Tangible Interaction for Design" a special issue of AIEDAM 23 (Spring 2009), 225-235. 13. Nacenta, M.A., Aliakseyeu, D., Subramanian, S., Gutwin, C. A Comparison of Techniques for Multi-Display Reaching. In proceedings CHI‟05 (Portland, Oregon, USA, 2005), ACM Press, 371 - 380

ACKNOWLEDGEMENTS We thank TU/e Industrial Design‟s “/d.search labs”, for helping to realize the Pipet prototype and all participants in the creative sessions and prototype evaluation.

14. Ogertschnig, M., and Van der Heijden, H. A short-form measure of attitude towards using a mobile information service. In Proceedings 17th eCommerce conference (Bled, Slovenia, 2004).

REFERENCES 1. Battarbee K (2006) Co-Experience. Understanding user experiences in social interaction. Dissertation. 2nd ed. Univ of Art and Design, Helsinki.

15. Rekimoto, J. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments, In Proceedings UIST (Banff, Alberta, Canada, 1997), ACM Press, 31-39.

2. Bruegger, P., and Hirsbrunner, B. Kinetic User Interface: Interaction through Motion for Pervasive Computing Systems. In Proceedings of Universal Access in HCI, Part II: intelligent and Ubiquitous interaction Environments (San Diego CA, July 2009), 297-306.

16. Rodden, K., and Wood, K. How Do People Manage Their Digital Photographs? In Proceedings of CHI‟03 (Ft. Lauderdale FL, 2003), ACM Press, 409–416. 17. Ryokai, K., Marti, S., and Ishii, H.. I/O Brush: Drawing with Everyday Objects as Ink, In Proceedings of CHI‟04 (Vienna, Austria, April 2004), ACM Press, 303-310

3. Dourish, P. Where the Action Is: The Foundations of Embodied Interaction, Cambridge, Massachusetts: MIT Press, 2001.

18. Scott, S. D., Grant, K. D., and Mandryk, R. L. 2003. System guidelines for co-located, collaborative work on a tabletop display. In Proceedings of E-CSCW „03 (Helsinki, Finland, September 2003), Kluwer Academic Publishers, 159-178.

4. Eysenck, M. W., and Keane, M. T. Cognitive Psychology: A Student‟s Handbook, Hove, UK: Psychology Press, 2000.

19. Ullmer, B. and Ishii, H. Emerging frameworks for tangible user interfaces, IBM Systems Journal 39 (2000), 915-931.

5. Frohlich, D.M., Kuchinsky, A., Pering, C., Don, A., and Ariss, S. Requirements for photoware. In Proceedings of CSCW‟02 (New Orleans, LA, 2002), 166-175. 6. Green S.G., and Taber, T.D. The effects of three social decision schemes on decision group processes. Organizational behavior and human performance 25 (1980), 305-317,

20. Van Loenen, E., De Ruyter, B., and Teeven, V. ExperienceLab: Facilities. In Aarts, E. & Diederiks, E. (Eds.), Ambient Lifestyle: From Concept to Experience, BIS Publishers, 2006, 47-53.

7. Hinds, P.J. User control and its many facets: A study of perceived control in human-computer interaction, Technical Report Hewlett-Packard Laboratories, UK, 1998

21. Whittaker, S., Bergman, O., and Clough, P. Easy on that trigger dad: a study of long term family photo retrieval. Personal and Ubiquitous Computing 14, 1 (Jan. 2010), 31-43.

8. Hoven, E. van den, and Eggen, B. Informing Augmented Memory System design through Autobiographical Memory theory. Personal and Ubiquitous Computing 12, 6 (2008), 433443.

22. Zigelbaum J, Horn M, Shaer O, Jacob RJK (2007) The Tangible Video Editor: Collaborative video editing with active tokens. In Proceedings of TEI „07 (Baton Rouge, LA, USA, 2007) ACM Press, 43-46.

9. Hoven, E. van den, Frens, J., Aliakseyeu, D., Martens, J-B., Overbeeke, K., and Peters, P. Design Research & Tangible Interaction. In Proceedings of TEI‟07 (Baton Rouge LA, 2007), 109-116.

23. Zigelbaum, J., Vazquez, A., Kumpf, A., Ishii, H. Slurp: Tangibility, Spatiality, and an Eyedropper. In Proceedings of CHI‟08 (Florence, Italy, 2008), ACM Press, 2565-2574.

10. Jacob, R. J., Girouard, A., Hirshfield, L. M., Horn, M. S., Shaer, O., Solovey, E. T., and Zigelbaum, J. Reality-based

8

342

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

User Experience (UX) Patterns for Audio-Visual Networked Applications: Inspirations for Design Marianna Obrist1 , Daniela Wurhofer1 , Elke Beck1 , Amela Karahasanovic2,3 and Manfred Tscheligi1 Christian Doppler Laboratory for ”Contextual Interfaces” HCI & Usability Unit, ICT&S Center, University of Salzburg, Salzburg, Austria {firstname.lastname}@sbg.ac.at 2 SINTEF ICT, Oslo, Norway 3 University of Oslo, Oslo, Norway [email protected] 1

INTRODUCTION AND MOTIVATION

ABSTRACT

How to design and develop applications in order to lay the foundation for a good user experience (UX) has become a hot topic within the HCI community [22]. Designing for a good user experience has been investigated for different types of applications, such as social networking sites including audio-visual material (e.g. [20], [34]). Some of the most prominent examples for audio-visual networked applications are Youtube, Flickr, Facebook, and MySpace. They all build on user communities, provide networking opportunities for their members, and are strongly related to audiovisual user-generated content. Such social network and community websites have changed the way people use new media, by motivating their users to create personal profiles, share photos and videos, write blogs, etc. However, there is still a lack of knowledge on how the UX of such applications can be enhanced. Within this paper, we focus on how to design for a good user experience of audio-visual networked applications. More specifically, we report best practices for doing this by means of UX patterns, which represent possibilities and inspirations for designers. The developed patterns are a first step in giving designers an empirically grounded guidance on how to design for a better UX, and will be further extended with new insights gathered from users’ experiences.

This paper summarizes best practices for improving user experience (UX) of audio-visual networked applications such as YouTube, Flickr, or Facebook. Designing for a good UX is becoming increasingly important within the HCI community. However, there is still a lack of empirically based knowledge on how to design audio-visual networked applications for an optimal UX. Based on studies with more than 8000 users of ten different audio-visual networked applications, we have developed 30 user experience patterns (short UX patterns). Our UX patterns are build on the end users’ experiences investigated in lab and field studies in three different European countries. Most other pattern collections are based on the experience of designers or developers. In this paper we will present how we have developed the UX patterns and will describe the major UX problem areas found in detail. Our pattern collection can be useful to the designers of audio-visual networked applications and for the researchers working in the area of UX by providing empirical evidence on identified UX problems and suggestions for solutions referring to one or more of our UX patterns. Author Keywords

Patterns, User Experience, User Experience Patterns, AudioVisual Applications, Social Media, Social Networked Applications

This paper is based on our UX studies with users of audiovisual networked applications, which were conducted as part of a large three-year research project (2006-2009). Within this project, audio-visual networked applications were developed in three different European countries, on different platforms (web, mobile, IPTV – Internet Protocol Television), and addressed diverse target groups (online and offline community members, families, professional artists, children, etc.). More than 8000 potential and actual users were involved over the whole project in different design, development, and evaluation steps (e.g. from co-design sessions, lab studies, to field evaluation studies), with a huge amount of user feedback being collected. The knowledge we gained within the project’s evaluation phase resulted in a set of 30

ACM Classification Keywords

H.5.m Information Interfaces and Presentation (e.g., HCI): Miscellaneous; D.2.2 Software Engineering: Design Tools and Techniques

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, Reykjavik, LEAVE BLANK THE LAST 2.5 2010, cm (1”) OF THEIceland. LEFT COLUMN ON Copyright 2010 ACM 978-1-60558-934-3...$5.00. THE FIRST PAGE FORISBN: THE COPYRIGHT NOTICE.

1

343

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

UX patterns, summarizing the most important insights on how to improve UX in audio-visual networked applications. These patterns are intended to capture the essence of a successful solution to a recurring UX related problem or demand of audio-visual networked applications. This work is currently extended towards other contexts within a large national project on contextual interfaces.

count when constructing new buildings. Others made similar claims for explicitly considering human activity and a user’s experience (e.g. [11] or [25]). Van Welie et al. [41] also point out the need to consider experiences by describing them as the main user goals and tasks that need to be supported by the designers. Kohler et al. [25] state that there are no patterns describing solutions to motivational or other hedonic aspects. To overcome this, they suggest developing patterns that integrate broad and general principles of human behavior in the form of concrete solutions. In their article, they introduce one pattern dealing with a user’s motivation. However, existing attempts to focus on the user’s experience are fragmentary and lack a comprehensive pattern collection.

In this paper, we first provide an introduction to the patterns and their application in the field of HCI as well as their potential for UX. Afterwards, we describe how we developed and validated the UX patterns, how the resulting pattern collection looks like, and how the UX patterns can be applied according to the five major identified UX problem areas. Finally, we discuss the pattern approach as a tool for communicating knowledge on UX and inspiring designers of future audio-visual networked applications.

USER EXPERIENCE (UX) PATTERNS

Hassenzahl [21] states that a “good UX is the consequence of fulfilling the human needs for autonomy, competency, stimulation (self - oriented), relatedness, and popularity (others - oriented) through interacting with the product or service (i.e. hedonic quality)”. Pragmatic quality, such as the usability of a system, is also contributing to a positive experience, but only through facilitating the pursuit of meaningful hedonic needs. An important aim of “positive HCI” is to provide users with “outstanding quality experiences” [22]. Thus, developing applications that are explicitly intended to create a positive UX can be seen as an important step towards such a “positive HCI”. The most important characteristics of UX are its normative nature (differentiating between a positive, desired experience and a negative, undesired experience that a user can have when interacting with an application) [22] as well as its dynamic nature [28].

USAGE OF PATTERNS IN THE FIELD OF HCI

Since their emergence in architecture [1], patterns have been developed for a variety of application areas, ranging from patterns for successful software engineering concepts [19] to pedagogical patterns [18] for documenting good practices in teaching. In the last years, patterns and pattern languages have also gained popularity in HCI, as the broad application spectrum of patterns clearly shows. A detailed review on patterns and pattern languages in HCI as well as application areas of patterns in HCI can be found in [15]. In the field of HCI, patterns are often used in order to convey principles and best practices of good interface design (e.g. [3], [38]) and thus can be seen as a way of dealing with the increasing complexity and diversity of interaction design. A set of design patterns has been proposed in the area of product design [43] based on the product attachment theory. They describe how product interaction can help people to become who they desire to be. In the area of humanrobot interaction (HRI), design patterns were used for describing how to achieve sociality [23].

In order to preserve knowledge on UX and inspire designers on how to account for a good UX in audio-visual networked applications, we have developed 30 UX patterns. This extends the current state of the art on UX and patterns, as there is no structured guidance (i.e. patterns) on how to design for a better UX in such applications so far. These patterns were developed through an iterative process and clustered in five major problem areas. In the following sections we describe the whole data collection and clustering process in detail.

A recently published book targets the design of social interfaces [14]. Principles and patterns for social software are introduced by giving practical design solutions for improving the interfaces of social websites. Other research areas are dealing with the adoption of patterns for supporting innovative game design [32], for describing best practices in ubiquitous computing applications [10], and for teaching human-computer interaction principles [26]. Patterns have also been used for organizing and presenting ethnographic material (see [30], [13]).

Collection of User Experience Data

The UX patterns were extracted from the results of a threeyear European project, which aimed to develop novel forms of user communication and collaboration and to provide the users with an optimal user experience based on their own user-generated content. Apart from web-based applications, mobile and IPTV applications were developed within this project. Similar to applications such as YouTube, the developed applications enabled users to create their own usergenerated audio-visual content and share it over different platforms. In order to investigate UX, we collected data in the three testbeds located in different European countries. UX was defined by eight central factors that were identified as relevant for audio-visual networked applications. These factors were fun/enjoyment, emotions, motivation, user engagement, user involvement, co-experience, sociability, and

The adoption of the pattern approach for a variety of different fields of HCI shown above, illustrates the flexibility and broad applicability of patterns. Within the pattern community, there are demands for patterns explicitly dealing with the general human perspective and UX in particular. The first to emphasize a focus on social action and the human perspective were Alexander et al. [2]. According to them, it is essential to investigate how people experience architectural constructs and to take the user’s experience into ac2 344

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

usability (see detailed evaluation framework and description of the UX factors in [33]). This UX factor framework was applied for collecting user feedback from all testbeds and to detect common UX problems or demands and, in turn, to identify UX patterns. For the collection of the user data we used a combination of several qualitative and quantitative methods, such as interviews, focus groups, design workshops, questionnaires, lab and field studies, logging and semiotic content analysis, and the experience sampling method (ESM). A detailed description of all evaluation sessions, methods, and results goes beyond the scope of this paper. We focus here on the resulting patterns based on this process for improving the UX of audio-visual networked applications.

Table 1. Example for a UX Pattern (namely the Pattern 17 ‘Real-Life Integration”) illustrating the used Structure and Content

Name

Example: UX Pattern 17 “Real-Life Integration” (part of the third UX Problem Area - see Figure 2)

Problem

How can the users’ offline activities foster the use of an audio-visual networked application?

Forces

- The users want to support/strengthen their offline social relationships via an audio-visual networked application - The users want the possibility to promote offline activities that are closely related to activities on the application Use this pattern when your online community has close connections to an offline world/community.

Main idea of the pattern in a few words Problems related to the usage of the system, which are solved/improved by the pattern Further elaboration of the problem statement

Pattern Development and Validation

The UX patterns were identified by using a template that follows the pattern structure suggested by van Welie et al. [39] and Borchers [8]. Table 1 shows an example UX pattern called ”Real-Life Integration” taken from our UX pattern collection (see Figure 2). This pattern example reflects the main parts of a UX pattern, i.e. name, problem, forces, context, solutions, and examples. It also illustrates the content of a typical UX pattern. The list and details for all 30 UX patterns can be found online1 .

Context

Characteristics of the pattern usage context

Solution

Successful solutions for the described problem

Results of first evaluation phase Definition & Structure of UX Patterns

Examples

Examples of successful use of the pattern in a system

Initial UX Pattern collection Iteration

- Implement features that allow the mapping of offline social structures on the application - Possibility to announce important offline events on the application - Possibility to report offline events or activities on the application - Announcement of tour schedule of musicians on MySpace - Videos about offline community events on an IPTV platform - Regional groups on Flickr

Writer’s workshop with researchers

Our UX patterns were developed and refined iteratively, based on empirical data from user studies (see [35]) as well as on validation sessions with designers and developers (e.g. from co-design sessions, lab studies, to field evaluation studies). Figure 1 illustrates this iterative process. The UX patterns are based on the secondary analysis of the data collected during the evaluation of five applications and five use cases. During this evaluation we used in total 17 evaluation methods. The whole pattern development process was planned iteratively. First, eight researchers individually analyzed the empirically collected data to identify patterns in the three testbeds, using the same pattern template (see left column of Table 1). Then, the identified patterns from each researcher were compared and a common set of patterns was generated for each testbed. In the next step, the patterns from the different testbeds were merged, resulting in an initial UX pattern collection. Following the ideas presented by Borchers [7], we conducted a writer’s workshop with the researchers, in which the initial UX patterns were discussed and modified, and additional patterns were defined, resulting in an extended UX pattern collection. Another design and evaluation cycle within the project followed and revealed additional UX data, which were used to further extend and revise the UX pattern collection.

Extended UX Pattern collection Iteration Results of second evaluation phase

Feedback on UX Patterns by independent expert

Revised UX Pattern collection Iteration Interactive Pattern Workshop

Revised UX Pattern collection Iteration Pattern Checklist

Revised UX Pattern collection

Figure 1. Development and Validation of UX Patterns 1

http://hciunit.org/uxpatterns/

3 345

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

In order to validate the quality of the UX patterns, we developed a quality criteria framework which includes five main criteria, namely findability, understandability, helpfulness, empirical verification, and overall acceptance (see a detailed description in [42]). In order to validate the quality of the UX patterns, two validation sessions with potential users of the UX patterns (i.e. designers and developers) were conducted. The first validation session was arranged as an interactive pattern workshop, with six design students. The second validation session was conducted with six computer science students using a checklist we developed for estimating the quality of a pattern on a 5-point rating scale.

4. Raise and maintain the interest in the A/V application. 5. Support users in handling the A/V application. These problem areas are related to the overall goal to design for a better user experience in audio-visual networked applications and are provided as a starting point for inspiring designers in their design decisions. UX PATTERNS RELATED TO MAIN UX PROBLEM AREAS

In this section we present insights for each user experience problem area gained within our project. Thereby, we discuss the most important UX issues that account for a positive UX in audio-visual networked applications and give an overview about the related patterns. In favor of easier referencing, the patterns are consecutively numbered in Figure 2. In the very left column of that figure, UX pattern 6, “Content Co-Creation”, is further subdivided into additional patterns due to their high relevance for audio-visual networked applications. For details about single UX patterns and concrete solutions to the problems described below, we refer to our UX patterns website (see footnote 1).

The goal of these validation sessions was to evaluate the quality of selected UX patterns with regard to understandability, helpfulness, and overall acceptability, focusing on practical aspects in the usage of patterns by designers and developers. The results of the validations then provided the basis for two more revisions of the UX patterns. These revisions mainly concerned the grouping of the patterns, their naming, and the problem statement of each pattern. After the first validation session, the patterns were grouped according to more general problems they address. After each of the two validation sessions, the names of the patterns were iterated and verbs were removed from the pattern names. Furthermore, the problem part was described more concretely. The designers’ and developers’ rating of the content and the structure of the patterns implied that the general acceptance of the patterns was rather high concerning the individual ratings on the checklist. Interestingly, the stakeholders were not immediately convinced about the helpfulness of the patterns; however, after exploring some in detail, they reconsidered and decided that patterns would be helpful in the design process. A detailed description of the validation sessions and results including the applied quality criteria framework for validating patterns can be found in [42]. At the end of this iterative development process, our UX pattern collection comprised 30 UX patterns. The UX patterns have to be seen as suggestions for solving the identified problem area. Thus, the 30 UX patterns were further categorized by sorting them according to similar problems areas. These areas cover the main UX issues in audio-visual networked applications, which we identified based on the literature review and our empirical data. Our research team limited the categories to five problem areas so that designers and developers (intended users of UX patterns) could quickly find the solutions they need for a given UX issue. Further, an affinity diagramm was constructed which shows the relations between the patterns and their problem areas. Figure 2 provides an overview of the collection, with the patterns grouped according to the five main identified UX problem areas they address, namely:

Feeling of Being Part of the Community

This problem area contains patterns dealing with the question of how users of an audio-visual application can be given a communal feeling (established through the application). Figure 2 shows the patterns belonging to this category. With a total of eight patterns and six sub-patterns, this group is the biggest. This goes in line with our observations that a feeling of belonging is an important requirement for audio-visual networked applications. Moreover, a feeling of belonging is an important precondition for (re-)using an application. One issue which turned out to be central in this category was the possibility of giving an application a personal look, covered with UX pattern 1, “Self Presentation”. We found that the presentation of oneself and of others often leads to a more positive experience. For example, by personalizing the appearance of their profiles, users are presenting themselves to other application users. Conversely, by having the possibility to view profiles of other users, they can know with whom they are interacting. Audio-visual networked applications accounting for such a personal look should therefore provide the functionality for creating and viewing personal profiles. Strengthening the sense of community is another important issue supporting a positive experience in audio-visual networked applications. Having something in common with other application users or experiencing something together creates a communal feeling among the users. Such a sense of community can be for example enforced by establishing “social groups” through gaming competitions that require team play and cooperation. This and other solutions can be found in UX pattern 2, “Common Goals”. Letting users directly profit from the community aspect also accounts for a positive experience in audio-visual networked applications, which is the theme of UX pattern 3, “Content Sharing”. Such profit can be, for example, collective community content or the appreciation of other application members for the produced

1. Increase the feeling of being part of the community via an A/V application. 2. Increase the user’s trust in the A/V application. 3. Raise the personal and social benefit of using the A/V application. 4 346

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

How  to  design  for  a  be[er  U\  in  A/V  applica9ons]   Increase  the  feeling  of  being  part  of   the  community  via  the  A/V   applica9on  

Increase  the  user’s  trust  in   the  A/V  applica9on  

Raise  the  personal  and   social  benefit  of  using  the   A/V  applica9on  

(1)  Self   Presenta9on  

(15)  Privacy   Management  

(17)  Real-­‐Life   Integra9on  

(20)  Fun   Factor  

(25)  Ini9al   Support  

(2)  Common   Goals  

(16)  Gradual   Involvement  

(18)   Innova9on  U   Tradi9on  

(21)  Content   Variety  

(26)  Easy   Yrienta9on  

(19)  Explicit   Purpose  

(22)  Social   Rewards  

(27)   Informa9on   Management  

(23)  User   Responsibility  

(28)  Fast  Data   Input  

(24)  User   Challenges  

(29)  Dynamic   Informa9on   Grouping  

(3)  Content   Sharing  

(4)  Content   Crea9on  

(5)  Social   Interac9on  

(6)  Content   Co-­‐Crea9on  

(13)  Interest   Groups  

(14)  Visible   Interac9on  

(7)  User-­‐ Centric   Updates   (8)  Version   History   (9)   Consolida9on   View  

Raise  and  maintain  the   interest  in  the  A/V   applica9on  

Support  users  in  handling   the  A/V  applica9on  

(30)  Constant   Feedback  

(10)  Idea   Rooms  

(11)  Point  of   Reference   (12)  Quick-­‐ Start   Template  

Figure 2. Grouping of the 30 UX patterns under the 5 main identified User Experience Problem Areas for Inspiring Design of A/V Applications

content. For instance, features for uploading, viewing, and modifying collective content can support such an experience.

represents another way for improving their UX in an audiovisual networked application and supports the sense of community. UX pattern 6, “Content Co-Creation”, and the related UX patterns 7 - 12 further clarify this topic focusing on application features for collectively creating and modifying content (e.g. collaborative storytelling, video commenting).

UX pattern 4, “Content Creation”, covers another issue influencing UX in a positive way, namely the user’s feeling of being actively involved in the platform. The feeling of active involvement can be achieved when the users are given the possibility to act as “designers” and to actively contribute to the appearance and the content on the application. Thus, features that enable users to produce content and to place the created content on the platform are one possibility to account for this pattern. Providing space to the users for sharing their experiences and knowledge has also a positive effect on UX in audio-visual networked applications, as explained in UX pattern 5, “Social Interaction”. Letting the users interact with each other via the platform enabled them to partake or profit from their experiences, which turned out to be important for UX. Providing different means of communication is one way to support experience and knowledge sharing. Moreover, enabling users to collectively create content

UX pattern 8 “Version History” is in line with the Interaction History pattern proposed by van Welie [40]. They both focus on the importance of the use of history, but the pattern proposed by van Welie refers to the single-user context, whereas our UX pattern 8 identifies this need in the multiuser interaction context. Another way to improve UX is by supporting the users to find like-minded others on the audiovisual networked application. Meeting other users with similar interests thus accounts for a positive experience (see for example [37]). In order to achieve this, functions for forming and subscribing to special “Interest Groups” (see UX pattern 13) can be implemented in the application. In contrast, the findings of Ludford et al. [29] showed that dissimilarities among online community members led to increased

5 347

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

contribution and satisfaction within the community, because its members appreciated the feeling of uniqueness. Further research is needed to clarify which situations might benefit from different degrees of either similarity or uniqueness. Increasing social presence of the users is also important for a positive UX in audio-visual networked applications. Users experience the community as lively when other members are aware of them and their content, or when they are informed about other users’ activities and feelings. To support such a feeling of social presence among users, UX pattern 14, “Visible Interaction”, suggests displaying ongoing activities on the starting page or sending the users messages about their friends’ current status.

on- and off-line activities contributes to a favorable perception of the application. Another possibility to influence UX in a positive way is to offer a balanced amount of new and innovative versus traditional and well-known services and application elements. If a service is experienced as too familiar – e.g. because of other audio-visual networked applications with the same service – the users will not be motivated to try it, as such a service does not offer additional value. On the other hand, if the service is experienced as too different, users will have a hard time in identifying the purpose of the service and thus may be loathe to spend time investigating how it will benefit them. Consequently, a practical way of providing the users with a positive application experience is to blend innovative concepts with traditional audio-visual application elements. UX pattern 18, “Innovation & Tradition”, focuses on this issue.

Trustworthiness and Security

Another important problem area is related to the task of providing a trustworthy and secure audio-visual networked application [36]. There have been efforts to define a pattern language for trust in related areas, e.g. for online consumer trust in e-commerce [24] focusing on information policy, reputation and warranty policy. The UX patterns of this paper concentrate on user privacy and user involvement. As shown in UX pattern 15 “Privacy Management”, it is essential for a positive UX that the users feel in control over their self-generated content, in order to be able to shape their online identity and self-disclosure. The feeling of trust towards the application, its providers and the other users depends on a balanced degree of privacy and publicity of all personal information and uploaded content. This can be achieved by giving the users the opportunity to reveal different parts of their identity depending on the social context and to limit access to their content.

Explicitly stating the purpose of an audio-visual networked application to help users quickly identify the benefit of using it would seem to be an obvious requirement for a positive UX. Nevertheless, there are applications that do not manage to immediately attract users and make their purpose clear at a first glance. According to UX pattern 19, “Explicit Purpose”, this problem can be avoided by clearly communicating the defined aim, concept, and content via design and information architecture on the welcome page of the application. Potential users can then immediately know if the platform meets their expectations and requirements. Interest Maintenance

Next to raising the users’ interest in the usage of audiovisual networked applications, the challenge lies in maintaining that interest and in encouraging them continue contributing content. This UX problem area deals with the prevention of boredom and the elicitation of a joyful experience among the users of an application. UX pattern 20, “Fun Factor”, specifically refers to the fun-in-use factor of the provided services and functions of the audio-visual networked applications. In order to keep the users actively contributing to the application, it is beneficial to focus not just on the utility of the services but also on their entertaining character. Hence, one possibility to raise positive emotions during application usage is to provide the users with entertaining or surprising elements.

Building trust takes time and if users have to register for an application without being given the chance to explore the application and discover personal benefits beforehand, they may abandon the site. Bishop [4], for example, researched the question of how to persuade lurkers to become active contributors. They investigated a system that indicates the credibility of community members to lurkers in order to establish a trustful relationship between the members and motivate the lurkers to participate in the community. In our research, we also found that potential users should be gradually involved in the community to give them a positive, trustful UX of the application. Starting with the availability of basic functions of the application, users are continually provided with enhanced features after they passed through several levels of registration and user-generated content contribution. UX pattern 16, “Gradual Involvement”, provides further solutions for this UX problem.

Our findings from the user studies clearly showed that users strive for variety when using an audio-visual networked application. A loss of interest in the application, and thus a negative UX can be caused by outdated content and stagnating application development. To avoid this, UX pattern 21, “Content Variety”, suggests prominently placing and marking new content on the application, as well as keeping functionalities up-to-date or providing new ones.

Personal and Social Benefit

This collection of UX patterns comprises the challenge of how to raise the personal and social benefit of audio-visual networked application usage among its users. As described in UX pattern 17, “Real-Life Integration”, UX can be influenced in a positive way if the application includes aspects of the real life of its users. In other words, off-line activities of the users should be closely related to the activities within the application and vice versa. This tight relation of

One reason why people use audio-visual networked applications is related to the ‘social significance user value’ [9]. These users experience the application as beneficial if it provides social rewards for their contributions. As described in UX pattern 22, “Social Rewards”, social rewards can take the form of feedback or comments on contributions, or an 6 348

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

indication of which contributions received how much attention by other users. Similarly, Lampel and Bhalla [27] argue that status, as a kind of social reward, is a motive for contributing knowledge and content to the community and leads to a sustainable contribution behavior within communities. Apart from social rewards, the members of online communities experience their participation as valuable when they can take responsibility for the application. Giving users the opportunity to actively shape the application, increases their commitment to and involvement in it. UX pattern 23, “User Responsibility”, points out that this can be achieved by integrating tools for organizing contents, expressing opinions about the application, and getting in contact with application developers in the application.

worked application easier by offering suggestions or predictions based on previous user input. Another crucial point is the amount of information displayed to the user. Too much information can overwhelm the users, whereas too little can leave them bored and uninterested. One possible solution is to provide “Dynamic Information Grouping” (UX pattern 29), such as filtering information according to the user’s interests, friends, and groups. Finally, UX pattern 30 stresses the importance of “Constant Feedback”. Constant feedback from the application to its users contributes to a positive user experience, in the sense that the users need application and service status information about, for example, content upload progress and errors.

Providing “User Challenges” (UX pattern 24) also contributes to a positive UX, since the users of an audio-visual networked application can get bored when using the application for a longer time. By adding challenging elements like games, quizzes, and competitions to the application, users can enjoy competing with others and thus keep themselves from getting annoyed by the usual participation activities in the community.

DISCUSSION AND LESSONS LEARNED

Alexander et al. [2] already claimed that it is essential to observe and investigate how a user experiences architectural constructs (i.e. to take the user’s experience into account when constructing new buildings, rooms etc.). Existing pattern languages mostly do not deal with what users experience; rather, they deal with “technical” facts or are limited to usability issues. In this paper, we made a step towards focusing on user experience as a central topic for the HCI community. The most important aspect of the contribution is that we started with the users and used their input (collected in several studies - see e.g. [34]) to create our user experience (UX) patterns. As best we can tell, no other pattern collections describe best practices related to the design of user experience of audio-visual networked applications.

User Support

This problem area describes UX patterns that focus on helping users to handle the audio-visual networked application efficiently. It is therefore related to the usability of the application, which itself is an influencing factor on the UX. One possibility to immediately facilitate a positive UX for novice users is to provide them with “Initial Support” (UX pattern 25) - for example, by offering the users playful, interactive ways to get to know the application and the provided functionalities. Additionally, users will appreciate an “Easy Orientation” (UX pattern 26) within the audio-visual networked application, especially when it contains large amounts of user-generated content. Further, users are not interested in investing much time in searching for specific content. Thus, applying approved navigation structures and elements and providing interconnections between contents will help keep users from feeling lost in the application.

Other existing design patterns are still limited to mainly usability issues and do not take into account the huge amount of existing UX research, which has become an important area within the HCI community. We do not claim that all insights on UX, stored as UX patterns, are new. Some of our UX patterns, especially those dealing with usability issues (e.g. “Easy Orientation”, “Constant Feedback”, “Information Management”), may remind the reader of basic usability principles or existing interaction design patterns (see discussed similarities with references on pages 4 - 7). However, what distinguishes our collection from most other pattern collections is that our UX patterns represent empirically proofed insights on UX specifically in audio-visual networked applications. Also, instead of focusing on the user interface of an application, we focus on the user’s experience while interacting with the application (with usability issues of the interface as one of many factors influencing UX).

In general, a sophisticated information management will support the users in their consumption and contribution of usergenerated content on the audio-visual networked application. UX pattern 27, “Information Management”, states that the experience of the application can be improved by letting the users manage sets of objects like photos and videos, by using over-views and detail-views, by enabling user-defined grouping of content and information, and by integrating features for easily adding new content and deleting outdated information.

Similarities to our UX patterns could be detected in the book by Crumlish and Malone [14] about patterns for designing social interfaces to improve UX. Although their patterns address similar issues, there are two fundamental differences to our approach.

Redundant and needlessly complicated user actions (e.g. content upload and data input) within the audio-visual networked application leads to boredom among the users. One way to avoid this is to help users easily add data to the application by taking into account individual usage habits and typical choices of users. UX pattern 28, “Fast Data Input”, suggests making the input of text to the audio-visual net-

1. First, we build on the users’ experiences instead of building on the experience of designers or developers. The advantage of starting from the users’ point of view is that the patterns are directly based on their input and thus are not 7 349

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

biased by designers’ or developers’ interpretation of the users’ experience. However, it is necessary to validate the patterns with designers and developers, as they are the intended users of our UX patterns and should apply them in practice. Therefore, we iterated our UX patterns with designers and developers as part of the pattern development process.

the main UX problems. The developed patterns are providing designers an empirically grounded guidance on how to design for a better UX. A full list of our UX patterns can be accessed on the Web (see footnote 1). Experienced designers always have an advantage over inexperienced ones when designing a new system. Expert designers reuse solutions that have worked in the past. When they find a good solution, they use it again and again. Consequently, recurring patterns are formed. Such patterns help designers to reuse successful designs by basing new designs on prior experience. A designer who is familiar with patterns can also apply them immediately to design problems without having to rediscover them.

2. A second difference is that our UX patterns have a broader focus, because we looked beyond web applications – IPTV applications, for instance. Furthermore, the fact that we are validating our UX patterns against certain quality criteria [42] also differentiates our approach from that of most other pattern collections, in particular from the one of Crumlish and Malone [14]. When using our UX pattern collection in practice, the adoption of each UX pattern strongly depends on the context of use and thus must always be determined by the stakeholder designing or developing an application. Our pattern collection and the main identified UX problem areas provide a starting point for inspiring designers in their design decisions. The UX patterns should be seen as a collection of empirically revealed solutions, providing suggestions on how to solve particular UX problems rather than as fixed instruction for actions suitable for every context. We developed UX patterns in order to preserve knowledge on UX and to make it available for other researchers and practitioners in a comprehensive and easy to use format. During this process we experienced several challenges. For example, finding appropriate names was not always easy. Moreover, it was sometimes difficult to clearly distinguish between the forces and the solutions of a pattern. However, the close collaboration with expected pattern users for validating and improving the UX patterns turned out to be especially helpful for such issues. The examples (of the use of a pattern in a system) we found in the collected data were both positive and negative. In our patterns we describe only positive examples. In doing so, we risk missing some important messages from our users, but we preserve the “tradition” of patterns as describing best practices. Moreover, research in the field of teaching showed that negative teaching mechanisms like anti-patterns (i.e. patterns using negation) or negative guidelines confused the students and created pitfalls in the knowledge transfer [26]. As the process of developing patterns is a subjective activity, it might be that different researchers would identify different patterns based on the same data set. To overcome these limitations we followed an iterative development, merging, and evaluation process. During the analysis phase, several researchers were involved in individual as well as collaborative sessions.

Compared to existing pattern collections and common pattern development processes, we did not start from the designers’ experiences when we developed the UX patterns discussed in this paper. Our UX patterns are based on empirical data collected from users of an application. This is similar to the approach taken by Martin et al. [31] - they present patterns of cooperative interaction that they derived from ethnographic studies of cooperative work. By combining the pattern concept, an already proved and valuable approach, with an empirical data collection process, we gained a lot of insights on how users’ experience the interaction with a system, what kind of problems appear, and what further demands they have. Within the CHI community, several experts build their research on the pattern approach (most recently [5]). It was pointed out that patterns can facilitate the communication among all stakeholders, and are more than just a sort of poetic form of guidelines. Pattern languages are intended to be meta languages used to generate project-specific languages that are grounded in the social and cultural particularities of a given design domain. Moreover, it can be stated that patterns are important as they provide a common vocabulary [16], they are structured around the problems designers face, and they provide solution statements [17]. These pattern characteristics can reduce time and effort for designing new projects considerably [41] and support a better communication among different stakeholders. In summary, capturing useful design solutions and generalizing them to address similar problems is one of the big advantages of patterns, in part because the documentation and use of best practices improves the quality of design. Despite considerable research related to design patterns within HCI, there are few empirical studies evaluating them. Patterns can be validated in two ways. On the one hand, they can be validated by using expert/stakeholder-based evaluation methods, such as heuristics, checklists, workshops etc. On the other hand, patterns are validated through practical use by stakeholders, such as designers and developers. Borchers [6], for example, inspected whether a set of developed patterns met a set of defined criteria and evaluated their didactic usefulness. Cowley and Wesson [12] conducted an experimental evaluation of the usefulness of patterns, comparing a set of patterns with a set of similar guidelines. Kotze et al. [26] compared the use of patterns and anti-patterns in

CONCLUSIONS AND FUTURE WORK

Within this paper we aimed to fill the lack of knowledge on how to design for a good UX, in particular focusing on audio-visual networked applications. To do so we used the pattern approach and reported the development and validation process of our 30 UX patterns and provided insights on 8 350

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

education. As a first step for evaluating our UX patterns, we conducted two validations with designers and developers to enhance their quality (see section on “Patterns Development and Validation” and our “quality criteria framework” [42]).

8. J. O. Borchers. A pattern approach to interaction design. In DIS ’00: Proc. of the 3rd conference on Designing interactive systems, pages 369–378, New York, NY, USA, 2000. ACM.

The development of design patterns is an iterative process, and thus we plan to further validate the ones presented here. As audio-visual networked applications and users’ expectations related to their user experience will change over time, the UX pattern collection has to be constantly updated. To facilitate this process we use our established pattern website with its integrated comment/feedback channel. Overall, we will further investigate UX and try to preserve the collected knowledge based on the pattern approach in order to enable designers and developers to enhance UX in a positive way. Besides the context of audio-visual networked applications, we will further extend our patterns to other contexts in order to get a comprehensive pattern collection of so-called “contextual UX patterns”. Dependent on the specific context, the adequate sub-collection of patterns can be chosen. In particular, we will investigate UX patterns for the context car as well as for the context factory as part of a large national project on “Contextual Interfaces”.

9. S. Boztepe. User value: Competing theories and models. International Journal of Design, 1(2), 2007. 10. E. S. Chung, J. I. Hong, J. Lin, M. K. Prabaker, J. A. Landay, and A. L. Liu. Development and evaluation of emerging design patterns for ubiquitous computing. In DIS ’04: Proc. of the 5th conference on Designing interactive systems, pages 233–242, New York, NY, USA, 2004. ACM. 11. T. Coram and J. Lee. Experiences - a pattern language for user interface design. Website, 1998. Available online at http://www.maplefish.com/todd/ papers/experiences; retrieved at August 21st 2009. 12. N. L. O. Cowley and J. L. Wesson. An experiment to measure the usefulness of patterns in the interaction design process. In INTERACT, pages 1142–1145, 2005.

ACKNOWLEDGMENTS

This work was partly funded by the Federal Ministry of Economy, Family and Youth and the National Foundation for Research, Technology and Development (Christian Doppler Laboratory for “Contextual Interfaces”) as well as by the CITIZEN MEDIA research project (FP6-2005-IST-41). Thanks go to all people involved in the collection of the empirical data and the development of the UX patterns collection.

13. A. Crabtree, T. Hemmings, and T. Rodden. Pattern-based support for interactive design in domestic settings. In DIS ’02: Proc. of the 4th conference on Designing interactive systems, pages 265–276, New York, NY, USA, 2002. ACM. 14. C. Crumlish and E. Malone. Designing Social Interfaces. O’Reilly, 2009.

1. C. Alexander. The timeless way of building. Oxford University Press, New York, 1979.

15. A. Dearden and J. Finlay. Pattern languages in hci: A critical review. Human-Computer Interaction, 1:49–102, 2006.

2. C. Alexander, S. Ishikawa, and M. Silverstein. A Pattern Language: Towns, Buildings, Construction (Center for Environmental Structure Series). Oxford University Press, 1977.

16. T. Erickson. Lingua francas for design: sacred places and pattern languages. In DIS ’00: Proc. of the 3rd conference on Designing interactive systems, pages 357–368, New York, NY, USA, 2000. ACM.

3. E. Bayle, R. Bellamy, G. Casaday, T. Erickson, S. Fincher, B. Grinter, B. Gross, D. Lehder, H. Marmolin, B. Moore, C. Potts, G. Skousen, and J. Thomas. Putting it all together: towards a pattern language for interaction design: A chi 97 workshop. SIGCHI Bull., 30(1):17–23, 1998.

17. S. Fincher, J. Finlay, S. Greene, L. Jones, P. Matchen, J. Thomas, and P. J. Molina. Perspectives on hci patterns: concepts and tools. In CHI ’03: CHI ’03 extended abstracts on Human factors in computing systems, pages 1044–1045, New York, NY, USA, 2003. ACM.

4. J. Bishop. Increasing participation in online communities: A framework for human˜ncomputer interaction. Computers in Human Behavior, 23:1881–1893, 2007.

18. S. Fincher and I. Utting. Pedagogical patterns: their place in the genre. SIGCSE Bull., 34(3):199–202, 2002.

REFERENCES

19. E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design patterns: elements of reusable object-oriented software. Addison-Wesley Professional, 1995.

5. A. F. Blackwell and S. Fincher. Pux: patterns of user experience. interactions, 17(2):27–31, 2010.

20. J. Hart, C. Ridley, F. Taher, C. Sas, and A. Dix. Exploring the facebook experience: a new approach to usability. In NordiCHI ’08: Proc. of the 5th Nordic conference on Human-computer interaction, pages 471–474, New York, NY, USA, 2008. ACM.

6. J. Borchers. A Pattern Approach to Interaction Design. John Wiley & Sons, Chichester, England, 2001. 7. J. O. Borchers. Chi meets plop: an interaction patterns workshop. SIGCHI Bull., 32(1):9–12, 2000. 9 351

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

21. M. Hassenzahl. User experience (ux): towards an experiential perspective on product quality. In IHM ’08: Proc. of the 20th International Conference of the Association Francophone d’Interaction Homme-Machine, pages 11–15, New York, NY, USA, 2008. ACM.

Proc. of the SIGCHI conference on Human factors in computing systems, pages 235–242, New York, NY, USA, 2002. ACM. 32. K. McGee. Patterns and computer game design innovation. In IE ’07: Proc. of the 4th Australasian conference on Interactive entertainment, pages 1–8, Melbourne, Australia, Australia, 2007. RMIT University.

22. M. Hassenzahl and N. Tractinsky. User experience - a research agenda. Behavior and Information Technology, 25(2):91–97, 2006.

33. M. Obrist, A. Weiss, and M. Tscheligi. Evaluating user-generated content creation across contexts and cultures. In IAMCR2007: Proc. of the 50th Anniversery Conferneces of the International Association for Media and Communication Research, Paris, July 2007.

23. P. H. Kahn, N. G. Freier, T. Kanda, H. Ishiguro, J. H. Ruckert, R. L. Severson, and S. K. Kane. Design patterns for sociality in human-robot interaction. In HRI ’08: Proc. of the 3rd ACM/IEEE international conference on Human robot interaction, pages 97–104, New York, NY, USA, 2008. ACM.

34. M. Obrist, D. Wurhofer, and M. Tscheligi. User experience patterns: A useful way to support social media development. In Proc. of NordiCHI 2008 workshop, pages 80–85. TAPIR Akademisk Forlag, 2008.

24. E. Kaluscha and S. Grabner-Kr¨auter. Towards a pattern language for consumer trust in electronic commerce. In Proc. of the 8th European Conference on Pattern Languages of Programs (EuroPLop’03), pages 521–540. Universit¨atsverlag Konstanz UVK, 2003.

35. M. Obrist, D. Wurhofer, and M. Tscheligi. User experience patterns: A useful way to support social media development? In NordiChi2008, 2008. Workshop Position Paper.

25. K. Kohler, S. Niebuhr, and M. Hassenzahl. Stay on the ball! an interaction pattern approach to the engineering of motivation. In INTERACT (1), pages 519–522, 2007.

36. L. Palen and P. Dourish. Unpacking ”privacy” for a networked world. In CHI ’03: Proc. of the SIGCHI conference on Human factors in computing systems, pages 129–136, New York, NY, USA, 2003. ACM.

26. P. Kotz´e, K. Renaud, and J. v. Biljon. Don’t do this pitfalls in using anti-patterns in teaching human-computer interaction principles. Comput. Educ., 50(3):979–1008, 2008.

37. J. Preece. Empathic communities: reaching out across the web. interactions, 5(2):32–43, 1998.

27. J. Lampel and A. Bhalla. The role of status seeking in online communities: Giving the gift of experience. Journal of Computer-Mediated Communication, 12(2), 2007. Available online at http://jcmc.indiana. edu/vol12/issue2/lampel.html; retrieved at August 21st 2009.

38. J. Tidwell. Designing Interfaces : Patterns for Effective Interaction Design. O’Reilly Media, Inc., 2005. 39. M. van Welie. Patterns as tools for user interface design. Website, 2000. Available online at http://www.cs.vu.nl/˜martijn/gta/ docs/TWG2000.pdf; retrieved at August 21st 2009.

28. E. L.-C. Law, V. Roto, M. Hassenzahl, A. P. Vermeeren, and J. Kort. Understanding, scoping and defining user experience: a survey approach. In CHI ’09: Proc. of the 27th international conference on Human factors in computing systems, pages 719–728, New York, NY, USA, 2009. ACM.

40. M. van Welie. A Pattern Library for Interaction Design. Website, 2005. Available online at http://www.welie.com; retrieved at July 30th 2009. 41. M. van Welie and G. van der Veer. Pattern languages in interaction design. In Proc. of IFIP INTERACT03: Human-Computer Interaction, pages 527–534. IFIP Technical Committee No 13 on Human-Computer Interaction, 2003.

29. P. J. Ludford, D. Cosley, D. Frankowski, and L. Terveen. Think different: increasing online community participation using uniqueness and group dissimilarity. In CHI ’04: Proc. of the SIGCHI conference on Human factors in computing systems, pages 631–638, New York, NY, USA, 2004. ACM.

42. D. Wurhofer, M. Obrist, E. Beck, and M. Tscheligi. Introducing a comprehensive quality criteria framework for validating patterns. In 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns, pages 242–247. IEEE Computer Society, 2009.

30. D. Martin, T. Rodden, M. Rouncefield, I. Sommerville, and S. Viller. Finding patterns in the fieldwork. In ECSCW’01: Proc. of the seventh conference on European Conference on Computer Supported Cooperative Work, pages 39–58, Norwell, MA, USA, 2001. Kluwer Academic Publishers.

43. J. Zimmerman. Designing for the self: making products that help people become the person they desire to be. In CHI ’09: Proc. of the 27th international conference on Human factors in computing systems, pages 395–404, New York, NY, USA, 2009. ACM.

31. D. Martin, M. Rouncefield, and I. Sommerville. Applying patterns of cooperative interaction to work (re)design: e-government and planning. In CHI ’02: 10 352

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Collective Interaction by Design Collective controllers for social navigation on digital photos Marianne Graves Peter Gall Krogh Morten Boye Petersen Aarhus School of Mortensen Department of Architecture Department of Computer Science, Nørreport 20, DK- Computer Science, Aarhus University 8000 Aarhus, Aarhus University Aabogade 34, DKDenmark Aabogade 34, DK8200 Aarhus N, peter.krogh@aarch 8200 Aarhus N, Denmark .dk Denmark [email protected] mortensen.morten @gmail.com ABSTRACT

Collective interaction holds some qualities in terms of the strong social connection between close collaborators as well as the social engagement in a shared activity. We find that few interactive systems of today support collective interaction by design. Even though conventional systems may be adopted in this way, as exemplified above, we wish to point to the potential in designing to support shared control and negotiation of digital contents in everyday life. An effort in this direction is the development of the interaction model of collective interaction [13]. A limited number of interactive systems supporting collective interaction have been designed, and we expand upon this work using a research through design approach [11]. Thus we have developed a collective interaction prototype adopting “making as a method of inquiry” [ibid]. Previous prototypes, which can be characterized as collective interaction by design, have been developed for e.g. museums, a library and for collective exploration of digital photos in the home. In this paper we report on how we designed a prototype enabling people to navigate digital materials, in this case photos organized spatially by geographically positioned journeys, by two connected interaction instruments. Through their combined input, the interaction instruments control a shared cursor on the display. In addition, each controller provides tactile feedback on where the other person is moving. The prototype is developed for a home setting where people share a display, either remotely or co-located.

Author Keywords

Interaction,

Ditte Hvas Mortensen Department of Psychology, Aarhus University Jens Chr. Skous Vej 4, DK-8000 Aarhus, Denmark [email protected] k

object together, when children share a keyboard playing a PC-game together, e.g. one controlling the movement of a character, another controlling his weapons, in this way adopting the game in a way not anticipated by the designers.

This paper seeks to attract attention to the possibilities for designing for collaborative control and social negotiation in everyday interaction design. This work is starts out from the interaction model of collective interaction, which is a model depicting strong social connection between people interacting on a shared digital material with a shared goal. We adopt a research through design approach where we develop an exemplar collective interaction prototype for collective exploration of digital photos in the home and we evaluate the prototype in use. The exemplar prototype and experiences from trial use serves to refine the collective interaction model and identify qualities and shortcommings of collective interaction applications. In this way we wish to point to a design space, which can lead to new interaction techniques and -designs supporting shared social experiences around digital materials in everyday life.

Collective Interaction, Social device, tactile, shared display.

Thomas MøllerLassen Aarhus School of Architecture Nørreport 20, DK8000 Aarhus, Denmark [email protected]

Interaction

ACM Classification Keywords

H5.m. Information interfaces and presentation INTRODUCTION

Collective interaction happens often in everyday life, e.g. when two people coordinate their actions to carry a heavy Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

The prototype is developed as part of a project exploring new means for supporting connections between people, friends or family, who are apart as well as people who are

353

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

COLLECTIVE INTERACTION

co-located. We used the collective interaction model to guide the development of a collective interaction system for navigating digital materials in context.

In the following, the interaction model of collective interaction is briefly described. See [13] for a more elaborate description of the model. “An interaction model is a set of principles, rules and properties that guide the design of an interface. It defines the look and feel of the interaction from the user’s perspective” [2]. An interaction model defines a design space and is a way to move beyond “point design” [3], trying to capture more generally the characteristics of a group of interfaces and explore the qualities of interaction with interfaces inheriting from this model.

In the following, we position our contribution with respect to related work, before we describe the interaction model of collective interaction (CI). Next we describe early design concepts adhering to the CI model before we describe in more detail the prototype developed as well as the analysis of the results of a laboratory-based evaluation of the prototype with four couples representing different relationships. We conclude the paper discussing the experiences from the study refining the characteristics of collective interaction, and we discuss how this model points to future applications of collective interaction in the future.

[13] defines the CI model as “Collective Interaction is when multiple and collocated users share both one logical input channel and one logical output channel. The input channel may consist of a number of interaction instruments, which are logically coupled in the interaction. Collective Interaction requires more than one user for controlling and taking full advantage of the system. Through their interaction and negotiation, the users must actively coordinate their actions towards a shared goal”

RELATED WORK

Recently others have raised the need to focusing on shared experiences around interactive systems beyond what has previously been accomplished. Forlizzi and Battarbee [4] have proposed co-experience partly as a criticism of the highly individualistic approach to experience design. We sympathize with the criticism raised by Forlizzi and Battarbee [ibid], and where they focus mainly on how technologies per se can support co-experiences, our interest is to investigate how we can design technologies that invite co-experience by their design. Our thesis is that the CI model can contribute in this direction. Ludvigsen [14] also introduces a framework of interaction in social situations. He suggests this framework as providing a scale of engagement ranging from distributed attention to collective action. Ludvigsen [ibid] has been involved in the design of earlier systems adhering to the CI model and we apply this framework to analyze the experiences from trial use.

Figure 1 Two contrasting interaction models. Left: Single Display Groupware [16] Right: Collective Interaction

The Collective Interaction model can be described through comparison with the Single Display Groupware model (SDG) [16]. In both models, people share a display. In SDGs each person has an independent input channel, as illustrated in figure 1, whereas in the CI model people share one logical input channel. CI has the following characteristics:

A strand of research has investigated how sending messages in various forms can contribute to establish connections between couples living apart [e.g. 4, 10, 15, 19]. Much of this research have focused on designing for activities related solely to connecting, whereas we are interested in investigating how connecting can become a more integral part of otherwise mundane, purposeful everyday activities. [6] also explores means for drawing people into a shared experience, but they use other means than the tactile connection during the interaction itself. Thus elements of our systems can complement their approach.

1.

2.

Tactile interfaces has a long history in CHI, but most of these explore how individuals can improve their individual performance in various domains e.g. music composition [6] or feedback on basic manipulations like tilt [17].

3. 4.

The interaction model of collective interaction has been proposed already [13], however, few systems have been developed from this model yet. In this paper we wish to contribute to the continued development of this model adopting a research through design approach [20].

5. 6.

354

Collective interaction involves users actively collaborating and negotiating their concrete interaction Collaboration is instrumentalized, meaning that collaboration and negotiation is part of the concrete interaction. The interaction itself invites for human-human interaction beyond what is in the interface The spatial organisation of people induces expectations of use and contributes to the active collaboration A shared goal is established on the basis of sharing responsibility and negotiating of control Establishing shared goal through negotiation is essential both in order to achieve it and in order to challenge and thereby tease other participants.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

7.

The interaction may be asymmetrical, in the sense people take on different roles, but the efforts of all participating are accounted for and valued in the use of the system. As we illustrate later on, there are both advantages and disadvantages inherent in this model. It has been suggested that the CI model is particularly relevant in situations were social engagement and negotiation is important over efficiency. For instance in the areas of games, sport, teambuilding, multidisciplinary teamwork, training, and learning. Also in the domains of public libraries, museums, homes, and theme parks [13].

‘chair’ reacts on activity through displaying more pictures at a time. Moreover, pictures can be stretched and rotated through squeezing various ‘active zones’ distributed on the furniture.

Early examples

The collective interaction model has evolved through design iterations. An early example of a collective interaction design is iFloor [11] (see figure 2). The intention with iFloor is to pursue a richer social environment in a library facilitating visitors contacting one another.

Figure 3. Squeeze in use by a family

These early design concepts point to different ways of realizing the CI model. One of the experiences from the trial use of Squeeze was that people got frustrated because of lack of feedback on who was doing what making it hard to interpret the reaction of the furniture [13]. COLLECTIVE CONTROLLERS - THE DESIGN CONCEPT

The motivation behind the concept of collective controllers was to establish new means for sharing and exploring digital content on a shared screen both for co-located people and people who are apart in different locations or homes.

Figure 2 iFloor setup in a library

In the iFloor system, the Q/A’s are pushed to the floor using individual mobile phones and are displayed in the floor interface in an extending circular pattern (see figure 2). For interacting with the floor and navigating between the various messages to provide a response, visitors collectively move one cursor. In order to hit an area of interest on the floor, people need to coordinate their body movements around the floor. When the prototype is operated by a single user the cursor will be attracted to the rim of the display and thereby give little chance for exploring the whole interface, whereas if two or more persons are collaborating on moving the shared cursor the whole display can be explored.

Via each their handheld input device, users are offered the possibility to collectively explore digital contents. The devices are individual, but they share the control of one logical input channel controlling the spatial navigation on a map, and the navigation through slideshows with pictures. The devices are connected so that one user experiences tactile feedback about the other user’s spatial movement. The concept offers users the opportunity to negotiate through combinations of verbal, visual and tactile coordination. Furthermore this rich negotiation, especially the tactile feedback, was intended to introduce a sense of connection through use between the users. In this way the concept attempts to bridge the distance and create a sense of remote presence.

Another example is Squeeze [16] (see figure 3), which is a prototype designed based on the collective interaction model [13]. Squeeze consists of a house-camera and an oversized and interactive sack chair (figure 3), which is intended as an inviting setting for collective and playful exploration of the history of the home as captured through the pictures. Squeeze encourages humorous, active and collective ways of exploring the digital photos. The whole

The concrete prototype consists of screen displaying a simple world map organizing collections of digital photos (see figure 4), and two joystick-like input devices (see figure 5). One joystick for each user, two screens with shared contents for users who are apart and one shared

355

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

screen for co-located users. Approximately one third of the map is visible on the display, and tilting one of the joysticks left, right, forward or backwards causes the view to pan west, east, north or south accordingly. The thumbnails enlarge slightly if the shared pointer passes over them, and when holding the pointer on top of them a few seconds a slideshow of pictures from the location launches. To browse a slideshow the joystick is tilted to the right until all pictures have been viewed and the screen returns to map view. During the slideshow it is possible to go backwards by tilting the joystick left.

navigation speeds up letting the users reach their shared goal faster. One user can also decide to stay passive for a while or oppose a direction by navigating in the other direction. In this way a shared goal is negotiated. Since a user has the possibility to stay passive, only sensing the haptic movements from the co-user, the concept does not leave out single user interaction, and can in this way also work as a SDG system. Invitation to join in is supported in the way that if one user is exploring contents in one location, the unused joystick located elsewhere will lie wiggling and indicating the possibility to connect with a remote friend or family member. In designing the tactile feedback we encountered a challenge regarding what the tactile feedback should represent. We considered two alternatives, as illustrated in figures 6 and 7. The alternative differs as to whether the weight represents the resulting movement or alternatively such that one person feels the other person’s movement: • Resulting movement: Here the resulting vector is used as a common tactile feedback for both devices, giving both users the same feedback to let them sense how they are oriented in relation to the common result.

Figure 4 the map with photo collection

•

The partner’s movement: feedback to one device is only what the other is doing.

Figure 6 Feedback as resulting movement: the two movements even out each other and the feedback (weight) is neutral.

Figure 7 Eeedback on partner’s movement The feedback makes it harder for V1 to keep the position towards left. Similarly for V2, but in the opposite direction. In this way the movement is negotiated very tactically.

Figure 5 Collective controllers in action

The joysticks are held in mid air. As can be seen in figure 5, beneath the handle is mounted a weight, which mechanically shifts reflecting the co-users movements in real time. If people move in the same direction, the

In choosing between the two solutions we drew upon insights from psychology on proprioception [7].

356

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Proprioception is the sense a person has about the relative position of neighboring parts of her body. The spatial interaction is informed by a combination of the movements of each person, but the individual person already has proprioception of her own movements, so a combination of this with perception of the other’s movements through the tactile feedback is what informs the spatial interaction for the individual person. To include the individual’s own proprioception in the feedback would be redundant.

map and the slideshow to the users. Accelerometer readings from the two interaction devices are converted to vectors which drag the map. In practice this meant that if the two joysticks are tilted equally in opposite directions, the map does not pan at all, and if they are tilted in the same direction, the map pans at double speed. The accelerometer readings are also used to generate the tactile feedback in the other device and to neutralize the device’s own orientation so that the movements of the frame only reflect feedback from the other device.

Therefore we chose the second solution (illustrated in figure 7) expecting that it will make the tactile feedback more easier interpretable as negotiating with the other and thus be a good way to instrumentalize collaboration in this situation.

EXPERIMENT

We tested the collective controllers in a laboratory setting involving with four couples with different relations:

IMPLEMENTATION

The joystick-like prototypes consist of two main parts; a handle with an accelerometer attached, and a frame holding two servomotors and a counterweight keeping the frame in balance. The handle and the frame are connected by three cardan joints mounted in an L-shape, where the center joint is static and does not move, whereas the outer two are connected to rods that can be moved up and down by the servo motors, making the frame twist in relation to the handle. This causes the centre of gravity of the device to move and therefore changes the amount of force needed by the user in order to hold the handle in the desired position.

•

Malou and Mads: Couple living together

•

Camilla and Bo: Couple living together

•

Tine and Pernille: Friends living together

•

Maibritt and Dorthe: Two colleagues

Each session was divided in three parts. In the first part we asked the users not to talk to each other while navigating and browsing, but rather to rely on the feedback of the setup. We also did not explain the users the co-dependency of the joysticks and the unified input. In the second part we allowed them to talk to each other across the divider, and in this way imitate a real time audio connection between to homes or sites. In the third part the divider was removed and the two armchairs placed next to each other in front of only one screen. This last setup we also used for an interview about the experience and the concrete use. Each test lasted around fifty minutes in total and all four tests were recorded on video for later analysis.

Figure 9 Picture of the experimental setup in the first and second part

We acknowledge that the conditions for the experiment are artificial and quite different from the familiarity of the users’ home, just as the timeframe and the presence of the research team motivated the users’ commitment. However, keeping this in mind, our observations still enable us to identify general patterns of collaboration, which we describe below.

Figure 8 Prototype with counterweight removed for illustrative purposes.The liniear arrows indicate the movement of the rods and the circular indicate the resulting motions.

Handling accelerometer readings and controlling servomotors is done by an application which displays the

357

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

PATTERNS OF COLLABORATION AND NEGOTIATION

Maibritt: “But this isn’t Thailand.” Dorthe: “No.” Maibritt: “It really isn’t” Dorthe: “No. There are a bit to many stones.” Maibritt: “Yes.” [The Greece slideshow comes to an end and they return to the map.] Dorthe: “Should we go to Thailand?” Maibritt: “I think we should go to Thailand.” Dorthe: “…to see how nice it is.” Maibritt: “Actually it’s [the thumbnail] Indonesia.” [Simultaneously the tilt the handles to set the direction and navigate collectively towards the Indonesian thumbnail and quickly enter it.] […] Maibritt: “Dorthe, is this one of the islands you stayed on the last time you were there?”

In the following we describe the interaction, collaboration and negotiation observed during the experiment with the collective interaction prototype. We identified four patterns characterizing the interaction between the users and the prototype: Equally active, Taking turns, Main actor and active listener, and Close synchronization We use Ludvigsen’s framework of interaction in social situations [13] to support an analysis of the levels of social engagement during the lab trial uses. Ludvigsen’s framework provides a scale of engagement ranging from ‘distributed attention’ at the lowest level, where the only thing shared is the presence in the space, virtual or physical. ‘Shared focus’ is where the situation develops a single focus shared among its participants. ‘Dialogue’ in turn is where people invest themselves and their opinions in a dialogue visible to all participants. Finally, ‘collective action’ is the socially most engaging interaction. This is where participants are working collaboratively towards a shared goal. ##

Verbally referring to the desired content, through their conversation they form a shared goal, which is enhanced through the awareness of the others movements. In the form of collective action, they browse the contents they have agreed upon

Equally active: From Dialogue to Collective action Taking Turns – Shared focus

Three out of the four couples were very often equally active negotiating contents both verbally and through the tactic sensation of the other’s intention. Thus we saw a pattern of frequent shifts between dialogue and collective action. They discussed where and what to explore, both by verbally expressing their interests and by tilting their own handles to make the weight shift and indicate a direction to the other. Often both together:

From time to time for all four couples we observed a pattern of sharing through surrender of control, leaving mainly a shared focus. But one couple were special in that they exercised a very practical approach to taking turns, and did this quite often throughout the trial session: [… They circle the navigation cross around Iceland because they just got out of the Iceland slideshow too fast. Now they have trouble settling on the thumbnail together.] Camilla: “Can you hold it in place or should I?” Bo: “Well, I’ll do it. Just…” Camilla: “All right you do it.” [Camilla turns passive, watching on her screen how Bo slowly settles the cross and enters the slideshow.] Bo: “There.” Camilla: “Ok. Let me try to find it.” Bo: “Which part of it?” Camilla: “What I think might be a glacier. I just want to check.” [Bo turned the first couple of pictures together with Camilla, but now turns passive while Camilla browses through the pictures one by one.] Camilla: “There it is. I think it is. I’m not quite sure, ‘cause it’s so blackened from pollution.” [After considering the picture a bit Camilla takes the lead and holds her hand tilted to the right. Bo joins in and the browsing speeds up and out of the slideshow …]

Malou: “Try going downwards.” [She tilts the handle towards herself to pan the view south on the map, Mads follows suit.] Malou: “I’m kinda fascinated by those…”[Mads interrupts] Mads: “Hey! There’s also one [thumbnail] down here [on Antarctica].” [Mads adjusts his handle to navigate to the thumbnail.] Malou: “Waugh, it’s with penguins and stuff.” [With synchronous movements, they navigate to the thumbnail and try to settle on it, but have trouble keeping it still] Malou: “You move it.” [Mads manages to enter the slideshow, and they both take up tilting through the slideshow]

Observing them operating their joysticks, it’s evident that they are both equally active in moving to the slideshow, When browsing through the pictures of the slideshow they perform some micro-coordination where one of them sometimes take the lead in skipping to the next picture. During the slideshow Malou twice requests to go back and Mads obliges but he is also the one to move forward when they have talked a picture over followed by Malou's corresponding tilt of her handle.

Both Bo and Camilla are quite task orientated in reaching the glacier picture, and as they have just been through the slideshow once, there is no talk about the content until Camilla settles on the glacier picture. Since Bo does not comment on the picture, it seems in this example he indulges her interest but does not take part in it. In this way he dismisses a shared interest in the goal and therefore actually leaves little of a shared goal.

Less pronounced is the negotiation of Maibritt & Dorthe as they collectively tilt through the Greece slideshow appreciating the beaches and the sun, though both appear to have a thing for Thailand:

358

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Mads and Malou show a conversational approach to surrendering the control:

As Dorthe drives the storytelling forward, Maibritt listens attentively and exercises her possibility to stop or go back if parts of the story are particularly interesting or use the handle to speed up when her interest drops. With Tine and Pernille visiting Iceland, Tines story is the main focus

[Mads & Malou have seen all the slideshows more than once and then take to the map.] Malou: “Now you put it [the navigation cross] on a country you wanna go to.” Mads: “Aha.” Malou: “If you could decide yourself. Where you’d like to go the most.” Mads: “I know where.” [Mads navigates alone while Malou is passive.] Mads: “New Zealand. I’d like to go to New Zealand.” Malou: “Arh yeah.” [Malou takes over and navigates away past South America.] Malou: “Now look where I want to go. … Damn, where is it? Mads: “What do you want to find?” Malou: “I’d like to find….” [She struggles with adjusting the speed of the navigation.] Mads: “Hawaii?” Malou: “No, that’s here, right? I want to find something further down.” [Mads seems to have guessed the destination and joins in with the navigation] Malou: “There!!” Mads: “Arh the Galapagos.” Malou: “I’d also like to go… right about here…” Mads: “The Caribbean?”

Tine: “We can go to Iceland. I’ve been there also.” [Both tilt their handles navigation towards Iceland.] Pernille: “When?” Tine: “That was in high school. Sophomore year.” [Collectively they settle on the Iceland thumbnail and enter the slideshow and start browsing.] Pernille: “Which high school did you go to?” Tine: “Ikast High school. [They browse to a picture of a meadow creek.] Tine: “I remember being there.” […] [Tine tilts her handle to move forward and Pernille follows suit. The pictures flick by quickly] Pernille: “This picture …” [Both straighten their handles stopping on a picture of a tall waterfall.] Pernille: “Back…” [Pernille tilts her handle left and now Tine follows suit landing them on a picture of wide green fields.] Pernille: “This is reminiscent of that TV-show “Ørnen.” [Danish show aired on Danish national TV.] Tine: “Yes. Well it’s from up there.” [Collectively they tilt right, forward to the waterfall picture again.] Tine: “I’m not quite sure. But we saw a waterfall called Guldfos but it doesn’t look like this is it…” Pernille: “Guldfos….”

By agreeing on fully surrendering control and Taking Turns to navigate in the content Mads and Malou dismiss the shared goal, creating an asymmetrical attachment to the content. However they maintain a shared interest in the goal of seeing each other’s favorite places, which keeps the conversation alive. The two examples show how taking turns holds a possibility to connect and share as well as temporally abandoning the collective concept.

Tine has a lot to tell Pernille about her Iceland trip, but Pernille also asks her questions not directly relevant to the trip, just as she makes room for her own observations on the pictures by taking the lead in going backwards in the slideshow.

Main actor & active listener – between dialogue and collective action

Close synchronization – collective action

This pattern is somewhat in between the two former. Here one person takes the lead, and the other person soon adopts the goal as shared and contributes actively in the navigation and storytelling. Much like the example of Taking Turns by Mads and Malou the interest in the goal is shared, but though one is dominant here both users still collaborate in the interaction.

We also observed the users doing synchronised movements while browsing slideshows, where they would fall into silence and a rhythmic tilting and straightening of the joystick settling themselves on a pace that allowed taking in the picture while satisfying both their patience: [Tine and Pernille's hands are active tilting in synchronous browsing the Moscow slideshow pictures one by one, but they watch the pictures in silence.] Interviewer: ”What are you doing now?” Tine: ”I’m trying to browse controlled through the pictures.” Interviewer: “Yes.” Pernille: ”I think, I automatically have been following Tine’s movements. ’Cause it’s the pictures we’re watching. And we know that if we work together it will go fast, so here we tilt a bit and straighten up, so we have time to watch the pictures.”

[… Dorthe is commenting on slideshow pictures, which she pretends to be from her vacation in Greece. Both she and Maibritt coordinate the speed of navigation as they more or less synchronously skip from picture to picture.] Dorthe: ”This is from my trip to Athens….” Maibritt: ”Oh, what happened there? Can we go a back?” Dorthe: ”Sure we can.” [They simultaneously tilt their handles left and return to the picture.] Dorthe: “Ehh… I’m standing on some kind of fortress ruin. Actually what you can’t see is that there is water right down below. And that my husband just jumped in.” [Both laugh as they collectively tilt their hands to the right and skip to the next picture …]

The users browse through the pictures and take them in one by one, sensing in their hands an equal level of interest through the haptic feedback of the joystick.

359

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

The patterns did not occur in pure form throughout any of the four tests, and quite often the shift between patterns happened very quickly as a user suddenly changed focus. E.g. in the Taking Turns quote by Mads and Malou, Mads joins in with his joystick helping Malou to her favorite destination where they both move to in close synchronization.

And when knowing who the person is, it contributed to the experienced connection with the other and the rather detailed coordination on the speed of the slideshow: Tine: ”This is another kind of presence. Because if we where apart, I would know that the movements that I feel in my hand are not just by a machine but are actually made by Pernille at the other end”

EXPERIENCED QUALITY OF CONNECTION

Camilla: ”If I imagined being abroad I would find it nice, that there was some sort of interaction. Even if he takes over I would sense him through the machine.” Malou: “.. the sense of sharing is stronger in slideshows because I can tell when he’s seen enough of a picture and wants to move on.”

We did not explain the connection between the interaction instruments before the test began, as we were interested in exploring how people would interpret the connection or even if they were able to identify it. For all couples it took up to five minutes at most, to identify the connection. Once the concept of the prototype was clear none of the users expressed difficulty in mapping the shifting weight to the movements of their co-user

Dorthe: ”We can adjust the speed (of the browsing), because it’s easier when you have the contact. Then you know what’s said and then you can watch and then move on.”

Dorthe: ”You can feel the other is doing something – the joystick is moving in the direction the other is turning it. So you’re connected in some way.”

Common for all four couples, was the need to talk to each other in order to appreciate the functional and relational qualities of the design. Either in the form of a sound connection when not in the same place or through sitting together

Pernille: ”You can also see where the other one is going on this [the joystick]. I think if we didn’t have this it would take a longer time to figure out, that the other is counteracting.”

Malou: ”You can really sense there’s someone else. So in a way it can actually be more difficult to reach a goal if you don’t agree on what the goal is. Surfing around without talking about it, Mads’ presence is an obstacle to my exploration.

Dorthe: ”I think it’s a quite funny feature that you can go places together.”

Thus from this experiment it seems that the considerations around proprioception, i.e. the decision to only represent the other person’s movements in the tactile feedback made it easy to interpret the tactile feedback as the other persons actions, and in this way feel connected to the other person. Of course a comparative study between two different versions is needed to confirm this finding.

Maibritt: ”I thought it was very difficult with out communication [Skype]. Dorthe: ”Yes so did I.” Maibritt: ”I might as well have been sitting in my office alone. I didn’t have any connection to my sister, friend or whatever.”

Our experiment also illustrates the importance of actually knowing who is “at the other end”. As Bo points out, being physically influenced by another person requires a certain level of transparancy:

To Bo the design did not fulfill his expectation for an interface, and when asked about his experience he clearly expresses his reluctance towards the collaborative input into the navigation:

Bo: ”I think it’s ok that the counterpart is controlling if I know who the counterpart is. Otherwise it would just be annoying. Then you would just feel stupid.”

Bo: “It compromises the precision.”

To Bo the collaboration was an interference of use, and instead he suggested fitting buttons on top, allowing a user to either seize control or to surrender it completely. In this he did not dismiss the haptic feedback from the co-user, but through out the test he did not really approve of it either. Bo’s reluctance to negotiate the interaction throughout the trial suggests that there are people who due to their personal values and preferences will rarely find it appropriate to negotiate the interaction in the way promoted by the CI model. The remaining users on the other hand embraced the design and the possibility to explore the content together and almost exercise a relationship through the interaction:

Mads: ”About the presence and sensing that there is someone else, then it’s cool that it feels alive in some sense. It might also be because it’s something you know – that there is an intention behind the movements.” Malou: ”Yes”

In general the users responded positively to the sensation of haptic feedback from the co-user. Even before being aware of the co-dependence of the joysticks, one user expressed a sensation of something living within the mechanical movement: Malou: ”It really feels alive. -Like holding a budgie in your hand.” [Chuckling]

Pernille: “It’s about taking an initiative and about giving room at the same time.”

360

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Tine “I can’t help thinking that Pernille and I know each other so well, so we are so use to giving each other that space. It’s not just one taking control all the time.”

people were still sitting rather close in the laboratory when we simulated the situation where they are apart.

Pernille: [While trying to counteract Tine.] ”You can really work against each other. It could be kind of a relationship test” [Both laughing.]

With the experiences from this study we argue that the CI model is worth pursuing further in the future and that it can contribute to opportunities for interaction design providing a strong sensorial and social connection and relation between people in their everyday interaction with computers.

DISCUSSION AND CONCLUSION

As the design and trial use illustrates, the collective interaction model and exemplar design hold promise for supporting social experiences through instrumentalizing collaboration. In the experiment we saw how control was negotiated at times second by second in a way that can never be accomplished when the negotiation happens at device level. To contrast with other approaches Hornecker et al. [10] refer to previous studies suggesting that the number of input devices, in this case classic mice, enabled more active participation in children’s drawing activities, however, resulting in more parallel working on a drawing. CI depicts a solution contributing to the best of both worlds in this respect enabling both multiple input devices, but at the same time promoting close collaboration with these.

ACKNOWLEDGMENTS

We thank the couples who took part in the evaluation and we thank the Danish National High Technology Foundation for funding this research. REFERENCES

1. Bardram, J. E. (1998): Collaboratior, Coordination, and Computer Support: An Activity Theoretical Approach to the Design of Computer Supported Cooperative Work Ph.D. Thesis, Daimi PB-533, .Aarhus: 2. Beaudouin-Lafon, M. 2000. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In Proceedings of CHI '00. ACM, New York, NY, 446-453.

Compared to earlier research into the collective interaction model and exemplar designs, the experiences from this study add to and refine some of the earlier findings. The collective controllers exemplify how a system may bridge between the SDG model and CI model in different ways. Whereas iFloor is impossible to operate by one person and thus is a pure form of CI, the collective controllers may be adopted as a SDG system. This happens when people take turns with the controllers.

3. Beaudouin-Lafon, M. (2004) Designing Interaction, not Interfaces. In Proceedings of AVI’04. ACM Press, pp. 15-22. 4. Eichhorn, E., Wettach, R., and Hornecker, E. (2008) A Stroking Device for Spatially Seperated Couples. In Proceedings of Mobile HCI 2008, ACM Press, pp. 303306 5. Forlizzi, J., and Batterbee, K. (2004) Understanding Experience in Interactive Systems. In Proceedings of DIS 2004. ACM Press, pp. 261-268

The study also suggests that it is not only situations but also some people who will very rarely benefit from the opportunity to negotiate control and interaction.

6. Gabriel, R., Sandsjö, J., Shahrokni, A., and Fjeld, M. (2008) BounceSlider: Actuated Sliders for Music Performance and Composition. In Proceedings of TEI 2008, ACM Press, pp. 127-130

The collective controllers point to the potential in using tactile actuators to support the connection and negotiation in the interaction. At the iFloor prototype, feedback about other people’s influence are given in the form of lines on the display. In the Squeeze prototype, no feedback on the individual influence was given, which was experienced as frustrating. The tactile actuators establish a connection and a sense of the others’ presence in a way that none of the earlier systems did.

7. Gibson, J. J. (1966). The senses considered as perceptual systems. Boston, Houghton Mifflin. 8. Harboe, G, Metcalf, C. J., Bentley, F., Tullio, J., Massey, N, and Romano, G. (2008) Ambient Social TV: Drawing People into a Shared Experience. In Proceedings of CHI 2008, ACM Press

We have only connected two people in the setup. Technically, it is not a problem to extend the model where the tactile feedback represents the resulting movement of the others’, but it remains open how easy it will be to interpret this result, given the very clear indication that is was necessary to know who was causing the movement. Our study indicate that people felt connected when they could combine sound and tactile connection and we saw no difference between situations where people sat apart or beside each other. Our study is limited in the way that negotiating to grab the handles and engage in the collective activity would be very different in the two situations and

9. Hindus, D., Mainwaring, S. D., Leduc, N., Hagström, A. E., and Bayley, O. (2001): Casablanca: designing social communication devices for the home. Proceedings of the SIGCHI conference on Human factors in computing systems, Seattle, Washington, United States, Pp: 325 – 332 10. Hornecker, E., Marshall, P, and Rogers, Y. (2007) From Entry to Access – How Sharability Comes About. In Designing Pleasurable Products and Interfaces. ACM Press

361

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

11. Kaye. J. (2006) I just clicked to say I love you: Rich evaluations of Minimal Communication. In proceedings of CHI’07 ACM Press, pp. 363-368

Systems (San Jose, CA, USA, April 28 - May 03, 2007). CHI '07. ACM Press, New York, NY, 2609-2614. 17. Poupyrev, I., Maruyama, S. and Rekimoto, J. (2002) Ambient Touch: Designing Tactile Interfaces for Handheld Devices. In Proceedings of UIST’02, ACM Press, pp. 51-60

12. Krogh, P.G., Ludvigsen, M., Lykke-Olesen, A. (2004) Help me pull that cursor - A Collaborative Interactive Floor Enhancing Community Interaction. In proceedings of OZCHI 2004, ISBN:1 74128 079

18. Stewart, J., Bederson, B. B., and Druin, A. (1999) Single Display Groupware: A Model for Co-present Collaboration. In Procedings of CHI 1999. ACM Press, pp. 286-293.

13. Krogh, P. G., and Petersen, M. G. (2008) Designing Collective Interaction. In From CSCW to Web 2.0: European Developments in Collaborative Design, Springer Verlag

19. Vetere, F., Gibbs, M. R., Kjeldskov, J., Howard, S., Mueller, F. F., Pedell, S., Mecoles, K. and Bunyan, M. (2005) Mediating Intimacy: Designing Technologies to Support Strong-Tie Relationships. In Proceedings of CHI 2005, ACM Press, pp. 471-480.

14. Ludvigsen, M. (2005) Designing for Social Use in Public Places – a Conceptual Framework of Social Interaction. In Proceedings of Designing Pleasuable Products and Interfaces, DPPI 05, Pp 389-408. 15. Lottridge, D., Mason, N., and Mackay, W. (2009) Sharing Empty Moments: Design for Remote Couples. In Proceedings of CHI’09, ACM Press, pp. 2329-2338

20. Zimmerman, J., Forlizzi, J., and Evenson, S. 2007. Research through design as a method for interaction design research in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA, April 28 - May 03, 2007). CHI '07. ACM, New York, NY, 493-502

16. Petersen, M. G. 2007. Squeeze: designing for playful experiences among co-located people in homes. In CHI '07 Extended Abstracts on Human Factors in Computing

362

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Privacy-Awareness Information for Web Forums: Results from an Empirical Study Stefanie P¨otzsch Technische Universit¨at Dresden, Faculty of Computer Science 01062 Dresden, Germany [email protected]

Peter Wolkerstorfer Center for Usablitity Research and Engineering 1110 Vienna, Austria [email protected]

Cornelia Graf Center for Usablitity Research and Engineering 1110 Vienna, Austria [email protected]

A German newspaper reported about the case of a woman who runs a restaurant and was looking for help in terms of an online gambling issue. Therefore, she became member of a forum and posted a lot of personal details in this forum over time. With all this information publicly available, the woman attracted the attention of a cyberstalker who continued to spy on her and posted a lot of further allegations about her. Some of the stalker’s statements may be true, others are wrong, but in neither case the woman would have wanted this details published on the internet. The story went on for four years when the owners of the restaurant, which this woman was running, found some of the negative things about her online. For fear of a negative image of the restaurant, the owners finally canceled the contract with the woman, who some months later no longer even dares to have a nameplate on her front door [3]. This is only one of the extreme examples that appear in the media from time to time and all of these examples illustrate that the sharing of (too much) personal data with possibly millions of unknown people on the internet is a critical point from a privacy perspective and may result in negative consequences, like e. g., bullying, cyberstalking, harassment or identity theft.

ABSTRACT

While interacting with others on the internet, users share a lot of personal data with a potentially large but “invisible” audience. An important issue is maintaining control over personal data and therefore, in the first place, users need to be aware to whom they are disclosing which data. Based on the cues-filtered-out theory we introduce a new feature to support the privacy-awareness of forum users and tested it with 313 users. The results of our empirical study show that the presentation of privacy-related context cues indeed increases forum users’ privacy-awareness. This is an important precondition for users’ willingness to modify privacy settings or to use privacy-enhancing technologies. ACM Classification Keywords

H.1.2 Models and Principles: User/Machine Systems—Human factors; H.5.2 Information Interfaces and Presentation: User Interfaces General Terms

Privacy, Privacy Awareness, User-Centred Design, Empirical Study, Forum, Social Software, Social Web

Yet, social software also has positive sides: It enables its users to get in contact with like-minded people anytime, anywhere, no matter which kind of issues they like to discuss. The mutual exchange of information, both personal and nonpersonal, is the major feature of social software and the motivation why people use it. Further, from a social perspective, the exchange of implicit and explicit personal data allows people to get an impression of the potential interaction partners and their situations. In this sense, the disclosure of personal data contributes to the success of social interactions and the forming of communities [4]. This means, that users inevitably have to give up their anonymity to a certain extent to become part of the social web. However, they should not need to give up control over their personal data. Therefore, in the first place users need to be aware to whom they are disclosing which data and it is an interesting challenge for the HCI community to support users in making informed decisions whether and to which extent they disclose their personal data in social software.

INTRODUCTION

The success of the social web is based on the active participation of users and their willingness to contribute to the creation and improving of contents on the internet by sharing data and knowledge. By using social software, a lot of personal data is disclosed either directly, e.g., real name and date of birth on social networking sites, or indirectly, e.g., through editing specific topics in a wiki, commenting on blog entries or posting statements in a forum [9, 7]. The possibilities of the social web may enrich people’s life, however there are also privacy risks involed.

Permission to to make make digital digital or or hard hardcopies copiesofofall allororpart partofofthis this work Permission work forfor personal or or classroom classroom use use is is granted granted without without fee fee provided provided that that copies copies are are personal not made made or or distributed distributed for not for profit profit or or commercial commercialadvantage advantageand andthat thatcopies copies bear this this notice or bear notice and and the the full fullcitation citationon onthe thefirst firstpage. page.ToTocopy copyotherwise, otherwise, republish, to post on servers or toorredistribute to lists, requires prior prior specific or republish, to post on servers to redistribute to lists, requires permission and/or a and/or fee. a fee. specific permission NordiCHI 2010, - 20, 2010, 2010, Reykjavik, NordiCHI 2010, October October 16 16–20, Reykjavik, Island. Iceland. Copyright 2010 ACM 978-1-60558-934-3/10/08... ISBN: 978-1-60558-934-3...$5.00. $10.00.

It needs to be considered that there is an essential difference between actual privacy that is realised and secured by tech-

363

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

nical means on the one hand, and perceived privacy as an individual feeling of users on the other hand. Individuals decide and behave based on a subjective evaluation of their environment instead of collecting and analysing precisely all objective facts (cf. the theory of bounded rationality [15, 2]). Thus, our research about user-centred design and human behaviour in social software primarily focuses on the individual’s perception of privacy.

cues [6, 17]. Therefore, we propose to provide additional privacy-related cues in social software in order to raise users’ privacy awareness, help them to better assess their potential audience and eventually enable them to make better informed decisions whether to disclose personal data on the internet. Considering that privacy is only a secondary task for users, the privacy-awareness information should be easy and quick to understand and not hinder social interactions and communication as primary tasks in social software.

In the next section we point out the importance of perceived privacy in computer-mediated communication by referring to results from related research. Then we describe our approach for the integration of privacy-awareness information in web forums. This feature was empirically tested with 313 users. The hypotheses and the design of the study are explained in the following section. Descriptive statistics and findings of the user study are discussed subsequently. We conclude the paper with a brief summary of the results.

Recent research about privacy and the social web has especially focussed on social networking sites [1, 5, 18, 20] where users maintain explicit profiles. Other applications where personal data is mainly implicitly included in the usergenerated content have been widely neglected. However, protecting privacy is also an important issue in these applications and therefore we decided to use the example of a web forum to conceptualise and implement a feature for the presentation of privacy-awareness information. Obviously, manifold social and context cues are conceivable to enrich computer-mediated communication in general and web forums in particular. Our goal is to find and test practical privacyawareness information on real forum users’ perception of privacy. We do not aim to identify the most salient form of social cues. Thus, we distinguished between the following two different types of privacy-awareness information: First, the display of the potential audience replaces partly the missing context cues from communication partners and should remind users about the actual audience of their contributions. In face-to-face communication, individuals usually can see who hears them speaking, even if a large part of the audience does not actively take part in the conversation. In a forum (or other social software), it is impossible to ascertain the whole audience and users may especially forget about the mass of silent readers [5]. According to the 90-9-1 rule, 90 % of all users of an online community are lurkers who never contribute, 9 % are members who post from time to time and only 1 % are considered as regular active users [12]. This implies that the number of visitors of a post in a forum is on average ten-times higher than the number of the active forum members, which – at least in theory – could be known to the author of a post.

RELATED WORK

When asked whether they care about privacy and want their personal data to be protected, a lot of people indicate a privacy-aware attitude in general. Regarding their actions in concrete situations, the majority shows another behaviour [13, 1, 16]. The complex reasons for this privacy paradox are object of research from psychological, social and economic perspectives. A possible explanation is that, although caring for privacy, users “blind out” their privacy concerns after a while when they enjoy their computer-mediated communication and that people overestimate their level of privacy in online interactions [18]. In an early study, Sproull and Kiesler found egalitarian and deregulating effects of computer-mediated communication in comparison to face-to-face communication, e.g., the disclosure of personal data to a large audience within a company [17]. In a meta-analysis of 39 studies on self-disclosure in interviews, it was shown that subjects disclose more personal data in computer forms than they do in face-to-face interviews [22]. The finding was explained with a higher level of perceived privacy of the participants in the first case. Similar results are reported from a comparison about spontaneous self-disclosure in face-toface situations and computer-mediated communication scenarios [10]. The researchers confirmed a positive correlation between individuals’ self-disclosure and visual anonymity as one aspect that contributes to perceived privacy. Further studies about the privacy attitude and behaviour of Facebook users reveal that they do not underestimate privacy threats in general, however, they misjudge the extent of accessibility of their personal data [1, 19]. Altogether, these results suggest that the individual’s perceived level of privacy is a very important factor for the decision whether to disclose personal data on the internet. Therefore, users need to be (made) continuously aware which – actively or passively created – personal data is visible to whom. PRIVACY-AWARENESS INFORMATION

The cues-filtered-out approach implies that individuals are more lavish regarding the disclosure of personal data when they use computer-mediated communication technology than in face-to-face settings due to a lack of social and context

364

In our privacy-awareness information panel the display of the potential audience is realised, first, by a textual cue which says that a forum post is visible to all internet users and, second, by the indication of the concrete number of visitors of the forum within the last seven days. The second direction for privacy-awareness information aims at demonstrating users how not anonymous they actually are. Considering that individuals usually visit a forum by following a link on the internet, the IP address is a good context cue: it is highly individual, it reminds the user that she is less anonymous than often perceived, and it can be correctly shown to each user. For similar reasons we also included a textual cue about a user’s current location (city) in addition. The location can be derived from the IP address by using a geolocation database. It is less individual identifying then an IP address, however probably more understandable for those people who are not very familiar with technical terms.

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 1. Forum with privacy-awareness information (also available in German)

For the graphical presentation of the privacy-awareness information we followed well-accepted design guidelines [11] and used a box in a complementary colour (orange) placed on top of the forum (blue). The cues about the potential audience are presented on the left hand and are visually separated from the cues about the additional information that the provider receives. This is displayed on the right hand. Figure 1 shows an example of a forum with our privacy-awareness information panel.

Table 1. 2 × 2 design of the study

Numerical representation yes Textual representation no no CG ExG1 yes ExG2 ExG3

H2 – Numerical privacy-awareness information will show a stronger effect than textual privacy-awareness information.

EMPIRICAL STUDY

In order to test the validity of our assumption that the presentation of privacy-awareness information as additional context cues influences individuals’ perception of privacy, we conducted an online study with real forum users. In line with the arguments above, we expect:

The concrete objective of the privacy-awareness information is to continuously remind users about the potential audience if they contribute (personal) data to a forum. The privacyawareness information should also inform users about implicitly submitted data, which is known to the forum provider in addition to the actual contribution. Since users do not actively enter such data, e. g., their IP address or current location, in a web form and confirm its submission, they may especially not be aware of the fact that it is known by the a provider anyhow.

H1 – Participants who are provided with privacy-awareness information feel they have less privacy during their visit of the forum than participants from a control group. We further wanted to study whether there are differences in the effect of single privacy-awareness information according to their form of presentation. Considering that most information in the user interface of the forum is presented as text, we assume that the numerical cues in the privacy-awareness information panel stand out and will be perceived better. Here we also have to control for participants’ familiarity with technical terms, i. e., whether they actually know what an IP address is.

H3 – Participants who are provided with privacy-awareness information will be better informed about the potential audience of forum contributions and they will also be better informed about additional information that they implicitly transfer to the forum provider.

365

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

(a) Experimental group ExG1 (numerical cues)

(b) Experimental group ExG2 (textual cues)

(c) Experimental group ExG3 (textual and numerical cues)

(d) Control group CG (advertisment) Figure 2. User interface for experimental groups and control group (originally shown in German)

study group. In the following second part, subjects filled in a well-elaborated online questionnaire. In the questionnaire we asked about their perceived level of privacy during the visit of the forum, about their knowledge of technical- and privacy-related terms and their internet use in general. We also collected demographic data.

The study was a 2 by 2 design that distinguished two forms of presentation of comparable privacy-awareness information as independent variables: textual cues vs numerical cues (Table 1). As textual information the potential audience and the user’s current location were displayed. As numerical representation we showed the exact number of visitors of the forum within the last week and the subject’s IP address. Participants in the control group got displayed an advertisement instead of any textual nor numerical privacy-awareness information (Figure 2). Subjects of the study were randomly assigned either to one of the three experimental groups (ExGi ) or to the control group (CG).

DESCRIPTIVE ANALYSIS

327 participants completed the online study between mid of December 2009 and mid of January 2010. We excluded answers from those who ticked off that they have not seriously answered the questionnaire and also from those who needed less than four minutes to complete the whole study, because it was not possible to seriously answer all questions within such a short time frame. Since we want to test the effect of privacy-awareness information on forum users, we further did not consider answers from participants who stated that they have never even read in a forum. Altogether, 313 valid responses remain for further analysis.

We invited people to participate in the study by posting a link via mailing lists, forums and blogs on the internet. If they were interested they could follow this link and use their own computers in their familiar surroundings to do the study. To avoid bias towards privacy, we told them that we are studying usability aspects of a web forum. Further, we did not post the invitation on platforms with a special focus on privacy and data security. internet users were motivated to participate by the offer to win several vouchers for an online store after finishing the study.

Since the study was available in German language, our participants are mainly located in Austria (51 %) and Germany (44 %). The majority of 85 % has A-levels or a higher educational achievement. 45 % are employed persons and 42 % of the participants are students. An almost equal number of male (50.3 %) and female (49.7 %) subjects filled in the questionnaire.

The study, that was originally realised in German language, consisted of two parts. First, participants saw a realistic fullscreen screenshot of the forum including the orange box on top and two posts in a discussion thread about leisure-time physical activity (see Figure 1). We instructed the subjects to imagine that they are the author of the first contribution. The orange box either contained one of the three privacyawareness information panels or the advertisement (Figure 2). The rest of graphical user interface was identical for each

The participants of our study are experienced internet users. The majority of 88 % indicates that they use the internet for five years or longer. Regarding their daily use, we can identify three approximately equally large groups. A first group of 29 % of all participants can be labelled occasional users

366

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 4. Detailed reasons to use forums

Table 2. Knowledge of technical aspects of the internet

Inform myself about different topics / products Ask for advice / help Share my knowledge Pass the time Discuss own view on topics with others Report about own experiences Stay in touch with others Follow what happens in the life of others Get to know new people Use anonymity of the internet for communication Friends outside the internet have no time Showing off

Do you know... Yes / Correct No / Wrong ...what an internet browser 97.76 % 2.24 % is? ...what an IP address is? 86.90 % 13.10 % 85.30 % 14.70 % Give short explanation ...how to change your IP 52.40 % 47.60 % address? 48.88 % 51.12 % Give short explanation n=313 (100 %)

Table 3. Linkability by nickname

When you contribute to more than one forum, do you use the same nickname in more than one forum? Always different nicknames 24.28 % Same nickname in more than one forum 38.66 % No nickname (as anonymous / guest user) 4.47 % Not contributed to more than one forum 6.39 % Never contributed to any forum 26.20 %

multiple answers have been possible

93.29 % 56.87 % 37.38 % 32.91 % 31.63 % 25.56 % 21.73 % 13.74 % 9.90 % 8.63 % 5.11 % 3.51 %

n=313 (100 %)

Those 26.20 % (=82 participants) who claim to never have written a forum post were asked in a free text field about the reasons for not contributing. Besides a lack of time or interest, 20 % of the 82 indicated privacy concerns. This number underlines that forum providers should have an interest to develop privacy features for their users.

n=313 (100 %)

Selection of quotations from participants who indicate privacy reasons for not contributing to forums (originally posted in German):

who are online two hours a day at most. Second, we have 37 % normal users who surf the internet three to four hours a day. A third group of 33 % can be considered as frequent users since they spent at least five hours a day on the internet. We further asked participants about their technical knowledge. Results in Table 2 show that nearly all subject claim to know what an internet browser and an IP address is and that almost all of them were actually able to explain the second item with a few words. When it comes to the question how to change the own IP address, the ratio of participants who claim to know that and could also explain how to do it decreases to about 50 %, which is still quite good. These numbers indicate that the participants of our study are not that clueless when it comes to key technical terms related to internet usage as someone might think.

Q1 (from CG): “I do not feel secure, when I post something on public web sites.” Q2 (from ExG2 ): “I don’t want to reveal so much about me. Besides, I can get the information I am looking for somewhere else. If a need advise, I ask real people.” Q3 (from ExG3 ): “Not interesting for me, I don’t want to reveal information about me all over the internet where everybody may read it.” Table 4 lists different reasons why participants of our study use forums in general. We see that informing oneself about different topics and/or products, i. e., passively consuming information is the primary reason. However also the sharing of a user’s own knowledge, the discussion of own views and the reporting about own experiences are often stated motivations. All of the latter three reasons imply the disclosure of personal data.

As said previously, all participants whose answers are considered in the analysis read in web forums at least from time to time. Approximately three quarters of the subjects have also actively contributed to one or more forums and about one quarter claims to always use different nicknames for different forums which can be interpreted as very privacy-aware behaviour (Table 3). However, a huge part uses the same nickname in more than one forum which allows for linkability of the information that is provided across different applications and therefore can be considered as privacy-intrusive behaviour. When directly asked whether they care about privacy and the protection of their personal data on the internet, 89 % stated that they often or always do so. This means, among the participants of our study we see a discrepancy between the stated attitude and the actual behaviour with regard to privacy. This can be interpreted as evidence for the privacy paradox (cf. the section on related work).

RESULTS

To the best of our knowledge, at the time of the study there is no established scale available to measure people’s individual perceived privacy. However, in order to make participants’ perception of their own privacy in the forum – our dependent variable – comparable, we created the Perceived Privacy Index (PPX). Since privacy in general is a very ambiguous term, we decided to use four specific questions, namely participants were asked to specify how public and how private their contribution in the forum is and how anonymous and how identifiable they have felt during their visit of the fo-

367

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 5. Perceived Privacy Index for different groups according to the presented privacy-awareness information

Num, text or both others Num or both others Text or both others Both others

Min 0 0

PPX Mean 133.59 161.63

Max 280 292

0 0 0 0 0 0

128.27 151.76 133.96 147.58 123.83 145.85

266 292 280 292 248 292

ANOVA (F -test) F (1,311)=13.36; p=0.00*** F (1,311)=12.46; p=0.00*** F (1,311)= 4.08; p=0.04* F (1,311)= 7.89; p=0.01**

sign. levels: *** p < 0.001, ** p < 0.01, * p < 0.05

these groups they would intend to have access. Actually, the post that we showed to the participants was accessible for all people with access to the internet, i. e., all registered and unregistered users, the forum provider and the internet provider. The fact that the post from the example was completely public could be learnt from the privacy-awareness panel with textual information (if shown) and there was also a visual cue visible for participants of all study groups indicating that the post can be viewed without being logged in, which means it is visible for everybody with internet access. We found no statistical evidence that any of the presented privacy-awareness information leads to better informed users with regard to the question which groups really have access (see rows expected in Table 7). The comparison of the percentages of expected access vs intended access of different audience groups reveals that nearly all participants know about and agree with the access to all post for registered members. Also nearly all participants know that the forum provider has access and three-quarters stated that the forum provider should have access. Our results further show that a majority of participants knows that also unregistered visitors can see the post, however only about one-third would want unregistered people to view their posts. This means, there is a considerable difference between the percentage of participants who would let registered users read their posts and those who also would allow unregistered users access to their posts. This finding is very interesting considering the fact that in most forums on the internet anybody can easily become a registered member by providing a fake e-mail address and choosing a password. Thus, practically each internet user could have access in any case with no great effort.

Table 6. Regression models for Perceived Privacy Index

PPX (dependent v.)

Model 1, n= 313 Est StdEr p 159.16 5.73

Model 2, n= 313 Est StdEr p 161.63 6.62

Intercept Predictors Num or both −23.78 6.62 0.00*** −28.86 9.49 0.00** Text or both −14.12 6.61 0.03* −18.83 9.15 0.04* Both 9.88 13.25 0.46 sign. levels: *** p < 0.001, ** p < 0.01, * p < 0.05

rum. Each of the four items was measured on a 0 to 100% slider scale. Then, the PPX is calculated using Equation 1. The higher the PPX value, the more private a subject has felt.

P P X = (100 − public) + private + anonymous + (100 − identif iable)

(Equation 1)

The results, which are are listed in Table 5, clearly support hypothesis H1. The PPX is significantly lower, i. e., subjects feel they have less privacy, when textual, numerical or both types of privacy-awareness information are available. To further disentangle the effect of the numerical cues vs the effect of the textual cues, we used linear regression models. Table 6 shows that both kinds of cues significantly decrease participants’ perceived privacy and that hypothesis H2 can also be confirmed since the effect is indeed stronger – for both, estimate value and level of significance – for numerical cues. Thereby, from Table 2 we know that most of the participants really have a concrete idea of what an IP address is. We further see that there is no additional effect if both kinds of information, textual and numerical, are presented together as can be seen in model 2.

Furthermore, in an additional free text field, a dozen of the subjects said that they would like to authorise particular readers based on properties (e. g., others with more than ten posts) or based on their relationship to them (e. g., friends from other communities). An approach that addresses the idea of user-controlled, property-based access control for forums is discussed in [14]. The authors further argue that access control only based on relationships would not be suitable for forums in general since this requires that the author of a post and the users she wants to give access have to know each other before. This assumption does not hold for web

Regarding hypothesis H3 the picture is less clear. In order to test whether participants are aware of the potential audience, we asked them which of the four groups in Table 7 really have access to the forum post. We further asked which of

368

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 7. Expected and intended groups of people to access forum posts

All registered users expected intended Unregistered users expected intended Forum provider expected intended Internet provider expected intended

Advert. n=78

Num, text or both n=235

Num or both n=149

Text or both n=161

Both n=75

96.15 % 89.74 %

96.60 % 97.45 %

97.32 % 97.99 %

96.27 % 96.27 %

97.33 % 96.00 %

69.23 % 28.21 %

73.19 % 31.08 %

74.50 % 33.56 %

74.53 % 31.68 %

78.67 % 36.00 %

98.72 % 66.67 %

95.32 % 75.68 %

95.30 % 73.15 %

95.03 % 73.29 %

94.67 % 70.67 %

47.44 % 7.69 %

50.21 % 11.91 %

48.99 % 11.41 %

51.55 % 12.42 %

50.67 % 12.00 %

forums, where people with similar interests can meet and discuss without knowing each other in person.

Finally, we tested whether participants are aware of the information which they directly and indirectly disclose to the forum provider. Therefore we showed participants a list with specified information items. We asked them to decide for each of the items whether a provider knows this piece of data about them after they have visited the forum. An overview about the answers is given in Table 8. Actually all of the listed facts are known to the provider. Since the IP address is the item that most of the participants are aware of anyway, we found no significant difference between the experimental groups and the control group. The knowledge that the provider can infer the location of the user is lower in general and it increases significantly if privacy-awareness information are shown. The according regression model in Table 9 shows that really only the textual cue increases the knowledge, i. e., users who only were informed about the IP address (ExG1 ) do not conclude that this also conveys information about their location. Among all study groups, the majority of users is sure that the forum provider knows which posts in the forum they have visited, i. e., they are aware that their behaviour in the forum is monitored. Interestingly at least half of the participants of each group were sure that the provider does not know anything about them which can be learnt from the content of their contributions, which in our example was something about the author’s pollen allergy, i. e., health information. Yet, this percentages have to be regarded with caution since some participants may just have forgotten what was written in “their” post. On the other hand, users may also forget about real forum posts after a while and which personal data they have disclosed there.

We further asked subjects to name a concrete number how many people theoretically could access their post. Obviously there is no clearly correct answer to this question, however we were especially interested to see whether the display of privacy-awareness information about the audience leads to an increased estimated number compared to the number named by the control group. Considering a possible anchor effect [21] for the two experimental groups with the numerical privacy-awareness information, we further wanted to check whether the single information that posts in the forum are “visible for all internet users” (ExG2 ) leads to a higher variance in the answers in comparison to the cases when the information that the forum had 388 visitors last week is given (in addion) (ExG1 , ExG3 ). In the former case, the formulation leaves it completely up to the user to imagine a number of how much people “all internet users” might be, whereas in the latter case a number is already presented and may function as rough orientation. The answers, which are graphically shown in Figure 3 for each study group and on a logarithmic scale, support the assumption about an anchor effect. Though the medians (black lines through the boxes) are roughly on the same level for all groups, in the box plot for ExG2 (Text only), we see that the actual box, which depicts the interquartile range1 , is considerably longer and also the range between minimum and maximum values (without outliers) is much greater compared to all other groups. We cannot see this phenomenon if no privacy-awareness information is presented and also not if a concrete number is given. This means, if no concrete number is provided but a textual information about all internet users, at least some forum users do really imagine that not only hundreds but millions (> 220 ) of people theoretically can visit their contribution.

We also asked participants whether a forum provider knows which browser and operating system they have on their computers. This hint was not included in the presented privacyawareness information and the answers indicate that there is a considerable share of participants who believe that the provider does not know this kind of information or who are not sure about this question. In fact, forum providers - like all other web site operators - are able to read out the so-called

1 The interquartile range is the difference between the third and first quartiles, i. e., the “middle” fifty percent of all answers.

369

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Table 8. Users’ knowledge about data items known to forum providers

Advert. n=78

Num, text or both n=235

Num or both n=149

Text or both n=161

Both n=75

91.49 % 2.55 % 5.96 %

92.62 % 4.65 % 6.04 %

91.30 % 3.11 % 5.59 %

93.33 % 1.33 % 5.33 %

57.45 % 21.70 % 20.85 %

53.02 % 23.49 % 23.49 %

65.22 % 18.63 % 16.15 %

65.33 % 18.67 % 16.00 %

68.94 % 11.91 % 19.15 %

69.80 % 10.74 % 19.46 %

67.08 % 13.66 % 19.25 %

66.67 % 13.33 % 20.00 %

41.70 % 53.62 % 4.68 %

36.91 % 57.05 % 6.04 %

43.48 % 51.55 % 4.97 %

36.00 % 56.00 % 8.00 %

59.15 % 19.57 % 21.28 %

59.06 % 19.46 % 21.48 %

56.52 % 22.36 % 21.12 %

53.33 % 25.33 % 21.33 %

44.68 % 31.06 % 24.26 %

44.97 % 30.87 % 24.16 %

43.48 % 33.54 % 22.98 %

42.67 % 36.00 % 21.33 %

IP address known by forum provider 85.90 % not known 7.69 % unsure / no answer 6.41 % Contingency test χ2 (2)=4.26, p=0.12 Location known by forum provider 41.03 % not known 33.33 % unsure / no answer 25.64 % Contingency test χ2 (2)=6.79, p=0.03* Posts visited known by forum provider 70.51 % not known 15.38 % unsure / no answer 14.10 % Contingency test χ2 (2)=1.41, p=0.50 Content of own post (health information in the example) known by forum provider 41.03 % not known 50.00 % unsure / no answer 8.97 % Contingency test χ2 (2)=2.03, p=0.36 Browser known by forum provider 65.38 % not known 19.23 % unsure / no answer 15.38 % Contingency test χ2 (2)=1.41, p=0.50 Operating system known by forum provider 47.44 % not known 34.62 % unsure / no answer 17.95 % 2 Contingency test χ (2)=1.36, p=0.51 sign. levels: *** p < 0.001, ** p < 0.01, * p < 0.05

User-Agent string and therefore know which browser and operating system users run on their computers. Furthermore, considering details such as version numbers or installed plug-ins, the fingerprint of the web browser can be enough to re-identify a user [8]. There are settings and tools available to blur the information of the User-Agent string and to increase users’ anonymity. However, before users will apply this privacy-enhancing technologies, they need to be aware about the facts.

numerical cues. We also showed that privacy-awareness information enhances users’ knowledge about personal data that can be inferred by forum providers and that textual cues tend to stimulate users’ imagination of how many visitors potentially can see their contribution. Future research about privacy-awareness and user-centred design will include the transfer of the concept to other social software, such as wikis or (micro)-blogs. Then, further user studies will contribute to a more general understanding of the effect that single privacy-related context cues have on the perceived privacy of users and their self-disclosing behaviour across different types of social software. Having gained a better understanding, it will then be reasonable and very interesting to deploy a privacy-awareness tool in a real social software application and study the effect of the privacy-awareness cues on the long-term.

CONCLUSION

In this paper we have shown with empirical evidence that the presentation of privacy-related context cues promotes forum users’ privacy-awareness. This effect is found regardless of whether numerical, textual or both kinds of privacyawareness information are presented and is even stronger for

370

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

over accepting possible usability drawbacks and give up the convenience of personalisation when surfing on the internet in a privacy-friendly way, to extra money that needs to be paid for enhanced privacy and security features. It remains object of further research to precisely analyse and quantify the trade off between privacy and extra costs that privacyaware users are willing to accept in different use cases. ACKNOWLEDGEMENTS

Many thanks are due to Christina K¨offel for her excellent research assistance. We also thank all participants of our study. The research leading to these results has received funding from the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement No. 216483. REFERENCES

1. ACQUISTI , A., AND G ROSS , R. Imagined communities: Awareness, information sharing, and privacy on the facebook. In Proceedings of 6th Workshop on Privacy Enhancing Technologies (Cambridge, UK, 2006), pp. 36–58.

2. ACQUISTI , A., AND G ROSSKLAGS , J. Privacy and rationality in individual decision making. IEEE Security and Privacy 3, 1 (2005), 26–33. 3. B URGER , J. L¨ugnerin! Betr¨ugerin! http://www.zeit.de/2009/53/ Internetmobbing?page=all, December 2009.

Figure 3. Comparision of theoretically expected number of visitors by the four study groups

4. C UTLER , R. H. Distributed presence and community in cyberspace. Interpersonal Computer and Technology 3, 2 (1995), 12–32.

Table 9. Regression model for location Location known by forum provider = true (dependent var.)

Intercept Predictors Numbers or both Texts or both Both

n= 235 Est Std Er 0.41 0.06 −0.00 0.24 0.01

0.08 0.08 0.11

p

5. DANAH BOYD. Why youth (heart) social network sites: The role of networked publics in teenage social life. In MacArthur Foundation Series on Digital Learning Youth, Identity, and Digital Media Volume, D. Buckingham, Ed. MIT Press, Cambridge, MA, 2007.

0.95 0.00** 0.95

¨ , N. Reduced social cues / cues filtered out. In 6. D ORING Medienpsychologie. Schl¨usselbegriffe und Konzepte (Stuttgart, 2008), N. C. Kr¨amer, S. Schwan, D. Unz, and M. Suckf¨ull, Eds., Kohlhammer, pp. 290–297.

sign. levels: *** p < 0.001, ** p < 0.01, * p < 0.05

Even without privacy-awareness information users may know and accept that all information which they explicitly contribute to a forum or other social software application is public, potentially for a broad audience and for a long time period. However, the question about implicitly submitted data, which are at least known to the application providers, remains. Since users do not actively enter this data and confirm the transmission, they are not aware of the fact, that providers know, for instance, their current location or may reidentify them because of their web browser details. Privacyawareness is important since it is an essential precondition for the use of privacy-enhancing technologies. Using these technologies will help to protect users’ privacy, however it is also accompanied with extra costs. Only if users are (made) privacy-aware, they will see a clear need for these technologies and they finally might be willing to spend associated additional costs. These costs range from extra time and cognitive effort that is needed to pay attention to privacy hints,

7. E BERSBACH , A., G LASER , M., AND H EIGL , R. Social Web, vol. 3065 of UTB. UVK, Konstanz, 2008. 8. E CKERSLEY, P. A primer on information theory and privacy. https: //www.eff.org/deeplinks/2010/01/ primer-information-theory-and-privacy, January 2010. 9. G ROSS , R., ACQUISTI , A., AND H EINZ , III, H. J. Information revelation and privacy in online social networks. In WPES ’05: Proceedings of the 2005 ACM workshop on Privacy in the electronic society (2005), pp. 71–80. 10. J OINSON , A. N. Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity. European Journal of Social Psychology 31 (2001), 177–192.

371

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

17. S PROULL , L., AND K IESLER , S. Reducing social context cues: Electronic mail in organizational communications. Management Science 32, 11 (1986), 1492–1512.

11. KOYANI , S. J., BAILEY, R. W., AND NALL , J. R. Research-Based Web Design & Usability Guidelines. Computer Psychology, 2004. 12. N IELSEN , J. Participation inequality. http://www.useit.com/alertbox/ participation_inequality.html, October 2006.

18. S TRATER , K., AND L IPFORD , H. R. Strategies and struggles with privacy in an online social networking community. In BCS HCI (1) (2008), D. England, Ed., BCS, pp. 111–119.

13. N ORBERG , P. A., H ORNE , D. R., AND H ORNE , D. A. The Privacy Paradox: Personal Information Disclosure Intentions versus Behaviors. Journal of Consumer Affairs 41, 1 (2007), 100–126.

19. S TRATER , K., AND R ICHTER , H. Examining privacy and disclosure in a social networking community. In SOUPS ’07: Proceedings of the 3rd symposium on Usable privacy and security (New York, NY, USA, 2007), ACM, pp. 157–158.

¨ , S., AND B ORCEA -P FITZMANN , K. 14. P OTZSCH Privacy-respecting access control in collaborative workspaces. In Privacy and Identity, IFIP AICT 320 (Nice, France, 2010), M. B. et al., Ed., Springer, pp. 102–111.

20. S TUTZMAN , F., AND D UFFIELD , J. K. Friends only: examining a privacy-enhancing behavior in facebook. In CHI ’10: Proceedings of the 28th international conference on Human factors in computing systems (New York, NY, USA, April 2010), ACM, pp. 1553–1562.

15. S IMON , H. A. Empirically Grounded Economic Reason, vol. 3 of Models of Bounded Rationality. MIT Press, Cambridge, MA, 1997.

21. T VERSKY, A., AND K AHNEMAN , D. Judgment under uncertainty: heuristics and biases. Science 185 (1974), 1124–1131.

16. S PIEKERMANN , S., G ROSSKLAGS , J., AND B ERENDT, B. E-privacy in 2nd generation e-commerce: privacy preferences versus actual behavior. In EC ’01: Proceedings of the 3rd ACM conference on Electronic Commerce (New York, NY, USA, 2001), ACM, pp. 38–47.

22. W EISBAND , S., AND K IESLER , S. Self disclosure on computer forms: meta-analysis and implications. In In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Common Ground. (New York, 1996), ACM, pp. 3–10.

372

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Tactile Camera vs. Tangible Camera: taking advantage of small physical artefacts to navigate into large data collection Mathieu Raynal1 Guillaume Gauffre1 Cédric Bach1,2 Bénédicte Schmitt1,3 1

IRIT Université Paul Sabatier 31062 Toulouse cedex 9, France {firstname.name}@irit.fr

2

Metapages 12, Grand Rue Nazareth 31000 Toulouse, France [email protected]

ABSTRACT

Emmanuel Dubois1

3

Global Vision Systems 10 Avenue de l'Europe 31520 Ramonville Saint-Agne, France [email protected]

interactive systems. It led to the emergence of novel interaction techniques, especially for mass-market systems. Nowadays, as underlined by [12] and [17], recent advances adopt a new approach for reducing these gaps: it consists in focusing on users’ aptitudes and their environment. This approach therefore promotes a better integration with the physical environment of the user by merging the use of physical artefacts and digital capabilities.

This paper presents the design and evaluation of two interaction techniques used to navigate into large data collection displayed on a large output space while based on manipulations of a small physical artefact. The first technique exploits the spatial position of a digital camera and the second one uses its tactile screen. User experiments have been conducted to study and compare the both techniques, with regards to users’ performance and satisfaction. Results establish that Tactile technique is more efficient than Tangible technique for easy pointing tasks while Tangible technique is better for hardest pointing tasks. In addition, users’ feedback shows that they prefer to use the tangible camera, which requires fewer skills.

As a consequence, different paradigms and interaction forms emerged including mobile interactive systems, tangible user interfaces, mixed reality or tabletop interactions. Following their breakthrough in specific application domains such as, maintenance, surgery, learning or military applications [1], demonstrating their technical feasibility, these new forms of enriched or embedded interaction spread themselves in numerous public spaces, such as classrooms [23], sightseeing places, public transports and museums [5,11].

Keywords Interaction technique, mixed interactive systems, pointing task, usability study ACM Classification Keywords H5.2 [Information interfaces and presentation]: Input devices and strategies – Interaction styles General Terms Design, Experimentation, Human Factors INTRODUCTION

Through the use of metaphors, direct manipulation and exploration of different modalities combined or not, the reduction of the execution and evaluation gaps [30] has always played a major role in the development of Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

Figure 1: Luminous Table for Urban Design Course (http://web.mit.edu)

373

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Interaction in Public Spaces

The aim of this paper is thus to 1) present the two interaction techniques dedicated to the pointing in large space and, 2) to summarise results of the user-test comparing these two interaction techniques. First section synthesised existing interaction forms with large displays and physicality based interaction. Then, the implementation of the proposed interaction techniques is detailed. Finally, the two last sections describe the experiment and its main results.

In such public contexts, designers often provide large displays, tightly integrated into the environment, using video-projection or wall-screens. This is more and more prevalent when several users simultaneously interact with a large amount of information, such as a bus network, a site map, museum or artistic exhibits. Therefore, studies on the interaction with large data collections triggered new representation forms and interaction metaphors [7], or combination of representations [33]. However interacting with these display surfaces most often results in basic interaction techniques [24] because little attention has been paid to the study of their characteristics and impacts.

RELATED WORKS

With regards to the use of small devices for navigating through large data collections rendered via large displays, related works can be analysed through three different aspects: existing approaches for interacting with large displays, interests and forms of Mixed Interactive Systems (MIS) and finally optimizations of pointing tasks.

Meanwhile, the miniaturization of technologies raised the amount of small devices able to capture physical information and process digital data. They are even integrated into personal objects such as cellular phones, PDA or digital cameras for example. They support access, navigation and selection of data, but constrain the interaction spaces in terms of size. However, these objects are familiar to users: learning to manipulate them has to be done only once and the fear of using new technologies may be reduced.

Devices Used with Large Displays

Using a mouse to interact with large displays remains difficult since a display size larger than the input size deeply affects the pointing of distant targets [32]. Moreover, users are standing in front of large displays instead of sitting at a desk and therefore the use of traditional pointing devices is precluded. Considering these two facts, several works introduced some pointing devices to facilitate the interaction with large displays.

Therefore, our work explore a current challenge in public interactive spaces: successfully combining the use of small personal belonging, providing a spatially limited and constrained input interaction space, with the need of pointing in wide output interaction spaces such as large displays.

Among the proposed solutions, technologies enabling a direct pointing on displays constitute the main trend. The tracking of stylus [29] or pen [19] on whiteboards can use ultrasonic or infrared recognition. Direct tracking of users’ fingers is also implemented using visual recognition in infrared range [20] or visible range [27]. If pointing performances are satisfying [13], direct pointing constrains users to move around the display if targeted objects are not within reach. On very large displays, users can even not access to some areas especially with vertical displays. Therefore with direct pointing, interaction possibilities are dependent of the displays configuration. To avoid such a limitation, distant pointing is available with laser [10], eyetracking [34] or freehand technique [35]. However, distant pointing lacks precision, especially when targets are small.

Physicality-based Interaction Techniques for Remote Pointing

To contribute to this challenge, two interaction techniques were developed that exploit the physicality of a personal digital camera to navigate a large collection of pictures rendered via a large display. The exploitation of the digital camera provides two different user‘s input interactions with a data collection: • The first version takes advantage of the object itself, its position and its ability to be moved by the users; it is therefore a tangible interaction; • The second technique takes advantage of its tactile surface to capture the position of a finger or a stylus; it is therefore a tactile interaction.

In this context, 3D input devices can be used to support an indirect pointing [28]. It involves different input and output interaction spaces and can offer new perspectives. But it highlights three problems: 1) the use of a specific input device: it is not always relevant especially in public spaces, 2) the discontinuity of the interaction: input devices may require some users’ attention too, 3) the difference of scale between input and output spaces. Therefore, a promising way to consider should be the development of new interaction techniques, such as Mixed Interactive System (MIS), since they propose a wide range of devices, more customary to users and with larger interaction capabilities.

In both versions, the detected position, respectively of the camera and the stylus, is used to move a pointer in the pictures collection. This pointer is controlled through the so-called Softstick, a software enhancement allowing a modal behaviour of the interaction and a control of the pointer’s speed. To assess the adequacy of small physical artefacts to support the input interaction with a pointing task in a large output space, a user test protocol has been defined and applied: it aims at evaluating the usability of the systems, especially both performance and satisfaction.

374

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Mixed Interaction Interests

effective. Therefore interacting with pictures collections, which include few empty spaces, cannot take advantage of such kinds of optimizations.

In public spaces such as museums, getting used to an interactive element is rather rare, because users do not visit these places on a regular basis and interactive elements may be rapidly replaced with new ones. Rather than offering traditional desktop devices, an ecological design of the interaction techniques may lead to the use of the environment artefacts and/or the users’ own resources: this will contribute to the users’ appropriation and learning of the technique. The main objective of novel interaction forms such as MIS, tangible UI, ubiquitous and pervasive systems, is to populate the interactive space with physical elements: any artefact, the architecture, the ambient light or sounds are therefore potential interactive elements [22].

DEVICES FOR REMOTE POINTING

As enounced above, the main objective of our project consists in navigating into a large amount of data, actually a large collection of pictures (Figure 3) projected on a white wall. It is intended that users of our techniques have the possibility to navigate their own pictures: therefore the main design idea is to use their own digital camera as the primary interactive device. With this physical element, we propose two forms of input for pointing at pictures: 1) using a video-based localisation of the camera and 2) using the embedded tactile screen of the camera. Since such input techniques offer a quite limited range of input interaction values, they are combined with a digital tool, the Softstick further described in the following section.

Given the technologies involved, mixed interactions have a great potential for taking into account a wide range of physical dimensions: users or objects positions [21] or body gestures [2] are some examples. These possibilities now offer new ways of using the environment: by using water, bricks and clay [31] as interactive support instead of keys and buttons, by using image walls, see-through devices, and also light intensity, colours or gas pressure as feedback instead of only pixels. Such a potential opens up the design space of interactive systems that best fit the situation. Any physical artefact may be involved and related to a digital concept to define the best interaction according to the environment and the users’ goals and/or expectations.

Description of the Softstick

The Softstick is a software tool used to control the position of a selector on a display. The centre of the Softstick representation gives the position of the selector. The rest of the Softstick representation, a disk and a pointer, provides a feedback related to its behaviour. Indeed, one position of the interaction technique controlling the Softstick corresponds to one position of the pointer in the Softstick; in addition, the position of the pointer determines the motion (direction and speed) of the whole representation of the Softstick, i.e. of the selector displayed in the data collection. It thus supports a modal interaction to pilot the selector.

Involving MIS in public spaces is therefore a promising route. In addition, MIS can be more customary to users since their components may already be present and manipulated in the spaces; MIS can be more attractive because of their novelty and their use of common objects; finally MIS reduce the problems of damages or robbery since it can be based on artefacts owned by the users themselves. All these characteristics participate to a better acceptance by visitors that can increase the possibilities of knowledge transmission targeted in classrooms, museums, sightseeing places, and all public spaces.

disk

 v

However, when dealing with indirect pointing in large spaces, it is also necessary to put attention to the task itself and the way to optimize it.

 u

radius

Optimization of Pointing Tasks

Reducing the time needed to accurately point at distant targets is a problem already well studied and which produces many software solutions. Among them, three predominant forms of approaches emerge: they either propose to increase the speed of the pointer in empty spaces [6], or to always select the closest target [16,18] or to increase the number of on-screen pointers [7,25]. These techniques are effective when the number of targets displayed on screen is limited.

pointer Softstick motion u = log2(v)

Figure 2: the Softstick components

In concrete terms, the representation of the Softstick is composed of a disk with a specific radius and a circular pointer (Figure 2). The input technique drives the pointer inside the Softstick disk: the Sofstick disk therefore represents the available input interaction space. Above a predefined threshold of motion of the pointer (10% of the radius), the Softstick starts to move. That motion (u) is performed using as direction and speed (with a logarithmic

However, if the pointer flies multiple targets before reaching the desired target, these techniques are no longer

375

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

function) the vector (v) between the Softstick’s centre and the pointer. Therefore, the motion speed increases when the pointer moves away from the centre.

better distinguished its pointer. A progress bar was also added indicating to user how many acquisitions he had already performed and how many were left. These improvements led to reset a second set of interaction techniques, cleaned of major basic usability drawbacks and ready to be tested by users.

Having such behaviour and representation, the Softstick is able to manage the scale difference between a small input interaction space and a large visualisation space. Its modal behaviour has also consequences on the user’s physical actions and their number: clutching is avoided and thus reduces the appearance of musculoskeletal disorders.

In addition, a user testing has been carried out to identify problems that can only be revealed at runtime. During this pre-test we were both interested in usability problems and experimental protocol dysfunctions. We adopted an incremental strategy to iteratively improve the application and the protocol. Nine users participated to this preexperiment. All of them are daily computer users and have no view or audition impairments. They all completed the entire protocol including a training phase, targeting tasks, responses to standard questionnaires (e.g. SUS [9], some items of IBM CUSQ and SUMI were used) and in-house satisfaction questionnaires, interviews and debriefing. The average time to complete the entire protocol was 90 minutes per participant (SD = 10 minutes). This preexperiment revealed different bug related to the logging functionality and the Softstick velocity control. Additionally we computed a SUS score for both interaction techniques: each of them scored 72,5 which is marginally acceptable according to [4]. We also noted a learning effect due to a training phase on a single difficulty index. Obviously, we fixed all the drawbacks identified iteratively along these 9 testing and we ended with an “experiment ready” version of the interaction techniques, Softstick and experimental protocol, described in the next section.

Input Techniques

To control the behaviour of the pointer in the Softstick, two input techniques have been defined. In the first one, position of the Softstick’s pointer is associated with the location of the camera held by the user. The location of the camera is computed using marker-based video recognition; therefore a tag is placed on the camera. As a result, users interact with the pointer through wide arm motions: such non-fastidious motions limit the need of skill and focus for performing the actions. Conversely the second input interaction technique proposes smaller movements to interact with the Softstick. This technique is based on the use of camera’s tactile screen. This screen is actually simulated with an UMPC, providing a 7-inches tactile screen and unrestrictive software capacities. In this setting the user manipulates a stylus to control the position of the pointer in the Softstick. Although no data are displayed on the tactile screen, each position of the stylus on the tactile screen corresponds to a position of the pointer in the Softstick. These two input interaction techniques have been connected to the Softstick and used during user-tests with a large collection of data.

EXPERIMENT

As mentioned above, the goal of the experiment is to study two interaction techniques for large spaces navigation, based on the use of physical artefacts owned by the users. To study these interaction techniques we have adopted a composite approach, i.e. a multi-factors evaluation. Indeed we seek to measure the quality of the interaction techniques in terms of satisfaction and performance. Such an approach is in line with recent advances in HCI evaluation, which promote the place and importance of aspects, related to the user experience especially with advanced forms of interaction techniques.

Pre-Experiment

To finalize the development of these two interaction techniques and remove any remaining drawback that may jeopardize the experiment, a pre-experiment has been performed. This evaluation includes a document-based usability evaluation with ergonomic criteria [3] followed by a user testing. This evaluation is limited to the application displaying the pictures, the interaction techniques and the experimental constraints embedded in the application to support the part of the experiment related to the target acquisition tasks.

Comparison in terms of satisfaction is based on evaluation resources similar to those used in the pre-experiment. Performance analysis is based on the time required to perform the task and completed by the use of the Fitts law [15]. Slightly modified [26], the Fitts’ law is adopted in many HCI works and states that the time needed to point a target of width W, at a distance D from the started position of the pointer can be predicted with the following relation where a and b are constants whose values are obtained by linear regression:

The document-based inspection reveals a set of 12 problems mainly about legibility of target, prompting (e.g. progress in the tasks sequence), Grouping / Distinguishing by Format (e.g. inadequate use of colour to distinguish targets from other pictures), Feedback, Explicit User Actions, Consistency. This inspection led to improvements of the two prototypes and avoided the major drawbacks indentified with this Usability Evaluation Method (UEM). These new versions improved the distinction of the targets, added a transparency to the Softstick representation and

376

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

       

   

Communications between the UMPC and the experiment system were enabled thanks to the UPnP protocol. protocol

The setting and the hypothesis for these experiments are described in the following sections. Goals and hypotheses

Two wo interaction techniques exploiting physicality of small artefacts have been developed to navigate into a large collection of pictures (Figure 4). However the type of techniques considered is not traditional; the place and ro role of the physicality multiply the amount of interaction facets facets, i.e. channels constituting the interaction, and interaction targets,, i.e. entities involved in or supporting the interaction: this directly increases the amount of potential usability problems. As mentioned above, a multi dimensional evaluation protocol is required to compare them. In this context, a first goal is to study the applicability of a well known method in a specific context: the use of the Fitts’ law with mixed interaction techniques. The first hypothesis (H1) is that such a law is still applica applicable even for this new form of interaction technique and the second hypothesis (H2) is that Fitts’ law parameters will vary with the type of mixed interactive techniques considered. Furthermore, a second goal is to compare the two proposed techniques in terms of user performance (on pointing tasks) and in terms of user satisfaction. The third hypothesis (H3) is that it is quicker to point on a tactile surface, because it is quite similar to the use of a mouse; in addition, the meticulous pointing supported rted by this technique may contribute to maximize the efficiency. Finally a fourth hypothesis (H4) is that it is preferred to manipulate the camera itself, because motions require less skill and therefore require less attention on the actions performed; in addition, such rough gestures may imply less learning efforts and therefore be more appropriate to public spaces.

Figure 3: View of the overall setting (top), tangible (left) and tactile (right) t) techniques in use. Task and stimuli

The task consisted in the selection of a picture on a 24×12 matrix of pictures (Figure 4). The picture to be selected was p highlighted, and randomly computed so that it fits the constraints of a Fitts protocol (predefined set of variations in terms of size and distance). To select a target, the user had to move the Softstick onto the target and stop its motion. Participants were instructed to proceed as quickly and accurately as possible.

Participants

Twenty users, thirteen men and seven women women, have been involved in the study. The age range was 22 to 56 years ((M = 33, SD = 10,3). They were all regular users of mouse and keyboard. Ten of them frequently used tactile screen. Apparatus

The system was executed on a 2GHz Dual Core laptop computer with 2 Gb of RAM. A video projector displayed the task on a white wall. The projection on area was 2x1.5 meters and participants were removed from 2.5 meters. The experiment environment has been implemented with .NET using C# language. The video recognition ecognition used a 60 fps digital camcorder. Detection of the camera position was performed by the ARToolkit+ library [36]. An ARToolkit+ tag was therefore stuck on the camera. The tactile screen is the one of an UMPC (ASUS R2H 900 MHz with 1 Gb of RAM), providing a 7-inches inches tactile screen.

Figure 4:: experiment software

377

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Design

technique. This difference is significant (F(1, 1213) = 5.71, p = 0.017). However, this effect has only been identified for the ID equal to 4. In all other cases (ID ranging from 2 to 6), no significant differences have been observed.

Repeated measures within-subject design was used. The independent variables were Input interaction technique (Tangible technique and Tactile technique), Target Width (12, 30 and 50 pixels) and Target Distance (150, 450 and 750 pixels). The nine distance and width combinations gave height indexes of difficulty ranging from 2 to 5.99.

Similarly, we performed an analysis of the selection time according to the distance, and then according to the target’s size. As for the ID, there is no significant difference in the selection time between the two techniques. On the other hand, according to the target’s width differences between the techniques have been observed. For the target’s width equal to 50 pixels (i.e. the biggest one), participants were faster with Tactile technique (F(1, 1783) = 6.91, p = 0.009), while they are faster with Tangible technique for the small targets. The difference on small targets is marginal (p<0.1).

Participants were randomly assigned to 2 groups of 10. In the first group, participants began with the Tangible technique and ended with the Tactile technique; the order was counterbalanced in the other group. For each technique condition, participants completed 9 blocks: one block consisted of a ratio distance/width condition and was presented in random order. For each ratio distance/width condition, participants performed 15 targets selection. In summary, the experiment design was:

These differences confirm our third hypothesis: it is more efficient to select with the Tactile technique i.e. a technique which anyone is used to manipulate since it is a common form of interaction technique. However, this analysis also reveals some unpredicted limitations: the use of the Tactile technique is less accurate than the Tangible technique to point at small targets. This unexpected result plays in favour of the use and development of Mixed and Tangible forms of interaction: indeed we can infer that once the users will be familiarised with this new form of interaction, they will be able to perform task as quickly as with a Tactile technique but in a more accurate way.

• 20 participants × • 2 techniques × • 9 blocks of predefined ratio × • 15 target selections • = 5400 trials The dependant variable studied was the pointing time. Errors of pointing are not recorded: the target to reach remains the same until the participant successfully select the correct target.

Fitts’ law Analysis

A second part of the analysis consisted in computing the mean time required to perform a selection for a given ID. In Figure 5, X-Axis and Y-axis respectively represent the index of difficulty (ID, in bits), and the mean time computed over the participants for this ID (in seconds). Solid and dashed lines are the regression lines respectively obtained with Tangible techniques (         ) and the Tactile technique (      ²  ).

Variables

In order to verify that both tested devices follow the Fitts’ law, we have recorded for each pointing task the width of the target to be reached and the distance between the starting position of the Softstick and the selected target. In addition, we recorded all Softstick movements on the screen and the event that occurs during the selection of target. We associate with each collected event the time that has passed since the beginning of the experiment. RESULTS

Tangible Linear reg (Tangible)

Quantitative Measurements

This section presents the quantitative results computed by use of descriptive and inferential statistics (i.e. ANOVA).

Tactile Linear reg (Tactile)

9 8

Selection Time (in s)

Movement time

As a first analysis, we computed the mean time required to select a target is equal to 5434ms with the Tactile technique and to 5387ms with the Tangible technique. Regarding the time to select a target, there is no significant difference between the two techniques. An analysis of this selection time according to the index of difficulty (ID), defined by the ratio distance / width, shows that, for an ID equals to 4, the selection time depends on the technique used: participants pointed the target in 4984.6ms with the Tangible technique whereas they performed the same task in only 4617.7ms with the Tactile

7 6 5 4 3 2 2

3

4

5

Index of difficulty (in bits) Figure 5: linear regression MT vs ID

378

6

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

This first analysis shows two problems: 1) the correlation between Index of difficulty and Movement Time for both techniques is rather poor (r²~0.9). The scatter plot reveals that the mean movement time is above the regression line for IDs whose width of target equals 12 pixels (the smallest target); 2) the Fitts’ law intercept a is negative for Tactile technique. This negative value can be due to noise in data [37]: when the user misses the target, he must continue his pointing while he achieves it. These errors are more frequent for the smallest targets (and thus the highest IDs). This noise may explain points above the regression line and consequently a high slope and thus a negative intercept.

pointing tasks correspond to large target and/or reduced distance and hard pointing tasks are obtained with long distance and small target. Furthermore, this analysis demonstrates that the Fitts Law still applies with mixed interactive techniques, i.e. interactive techniques based on the manipulation of physical artefacts. Our first hypothesis is thus verified. Regarding the second hypothesis, we also established that the parameters of the Fitts law (a and b) also vary with the kind of physical actions a user has to perform with elements constituting the mixed interaction technique. For example, a lower value of the b parameter, such as with the Tangible technique, indicates a lower sensibility to the difficulty of the task to perform. In the present case, this may be linked to the resolution of the input interaction space: with the Tangible technique, required users’ motions are wider than those he has to perform with the stylus on the tactile screen.

In order to verify if the target with the width equal to 12 pixels is a problem for regression analysis, we calculated regression equation without the ID of the pointing task where the width of target equals 12 pixels (Figure 6). Regression equations are       ID with r    for the Tangible technique and        ℎ     for the Tactile technique. Without the small target, there is a high correlation between MT and ID for both techniques. So, 12 pixels are too small for the width of a target. Tangible

Tactile

Linear reg (Tangible)

Linear reg (Tactile)

Alternatively, having a higher value of the a parameter, such as the Tangible technique in comparison to the Tactile technique, means that even for very simple task, the time to perform the task will be longer. In the present case, this may be linked with the fact the user first has to perform a large motion with the camera to bring the pointer of the Softstick over the threshold triggering Softstick motions. This analysis may therefore suggest reducing the size of the ineffective physical area of manipulation of the camera until an appropriate minimum time is reached for simple task, or in comparison to another interaction technique.

9

Selection Time (in s)

8 7

However, further work will be required to enable a finer correlation between the Fitts parameters values and the mixed interaction characteristics.

6 5 4

Users’ preference and satisfaction

3

As previously mentioned different questionnaires and interviews have been used to gather users’ preference and satisfaction. This section reports the major results collected through these evaluation techniques.

2 2

2,5

3

3,5

4

4,5

Index of difficulty (in bits)

First we did not find major usability drawback about the Softstick and the application developed to support the targeting tasks. Interviews and specific questionnaires about the evaluation of the application and the Softstick only reveal few minor problems that could be solved easily. Therefore the results presented below will only focus on the interaction techniques.

Figure 6: linear regression MT vs ID without targets whose width equal 12 pixels

In both cases (with and without target with W=12), the profile of the computed regression lines is similar: they intersect and the one corresponding to the Tangible interaction is above the other for lower ID. In concrete terms, the regression lines intersect at an index of difficulty (ID=3,87 in Figure 5 and ID=5,11 in Figure 6). This intersection means that the Tactile technique is better suited for pointing task than the Tangible technique, if the pointing task is easy (ID<4). Inversely, the Tangible technique is more efficient than the Tactile technique for the harder pointing task (ID > 5). This analysis thus confirms the one made in the previous section: easy

Questionnaires analysis

We computed a SUS score to each IT: the Tactile technique scored 78.5 and the Tangible technique scored 77.5. The scores are better than the ones computed during the preexperiment and are good and acceptable according to [4]. This measurement of control firstly indicates that usability of IT cannot jeopardize the results of pointing tasks. Secondly it reveals a marginal difference in favour of the

379

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

CONCLUSION AND FUTURE WORK

Tactile technique. The answers to the items of the IBM and SUMI questionnaires are consistent with the SUS ones. But it is still difficult to conclude on preferences or satisfaction only on the basis of SUS scores; we therefore further discuss it in the following paragraphs, on the basis of the analysis of post-experiment interviews.

Our work aims at exploring new interaction techniques using a small input device to interact with a collection of pictures displayed on a large output space. Our approach consists in exploiting two complementary dimensions of the physicality of a personal digital camera: on one hand, the object itself and its position in the space is used as a support for the interaction; on the other hand, users can take advantage of the interaction with the tactile screen of the camera. Moreover, we carried out an iterative design, which involved a pre-experiment to help us identify the main usability problems and resolve them before the user experiment.

Post-experiment interview analysis

We asked participants to indentify the three best and worst points of each IT. These questions are usually a fair indication of improvements, indicating what could be developed to reach a better usability level. Regarding the Tactile technique, the most frequent answers mentioned by the participants are 

3 best points (a) very usable, (b) easy to learn how to use it, (c) intuitive.



3 worst points (a) not precise enough, (b) awkward to control velocity of the Softstick, (c) the screen shapes limit the freedom of movement.

Resulting interaction techniques have then been assessed in a users experiment. Different aspects of their use have been considered: user’s performance, preference and satisfaction. The user’s performance is based on standard measures (like mean time) and revealed that the Tactile technique is more efficient than the Tangible technique for easy pointing tasks, while Tangible technique is better for hardest pointing tasks. To complement the analysis in terms of users’ performance, this work also intended to study the applicability of the Fitts law in the context of advanced interaction techniques. Results of this users experiment allow concluding that the correlation with Fitts’ law is established, despite problems due to pointing at the smallest targets. Having shown in this work that Fitts’ law applies to the designed techniques, it opens up a new research avenue that aims at predicting performances of mixed interactive techniques. To do that, we envision in a near future to further explore the different characteristics of our techniques to establish links between the parameters of Fitts’ law and the characteristics of our technique (including the physicality involved in MIS).

And regarding the Tangible technique, the most frequent answers mentioned by the participants are: 

3 best points: (a) very usable, (b) easy to learn how to use it, (c) funny.



3 worst points: (a) less precise to point smaller pictures, (b) not easy to stay in the interaction area (i.e. in the camera field of view), (c) the size of the tangible camera is too big.

These qualitative results highlight one main difference about the best points: Tangible technique is mainly perceived as funny and the Tactile technique as intuitive. Regarding the worst points, it reveals on one hand that the Tangible technique may need to be more concrete and reliable: this seems to be technically easy to address through the use of a more accurate localisation system. On the other hand the Tactile technique shows bad points linked to the screen small size and definition. Consequently, these points cannot be solved without a radical change of IT including a bigger definition and size of the interaction surface.

The study was conducted in a laboratory and proposed an abstract context of use. Moreover, only 20 persons have been involved in the experience. This is sufficient for an exploratory study, but may represent a limit for the results. Finally, the study is based on prototypes and should be replicated with more practical and technically reliable applications.

About the users preferences between the two IT the answer show that 65% of participants prefer to use the Tangible technique. 55% of participants estimate that the Tangible technique is more usable than Tactile technique. On the same line, 60% of participants estimate that the Tactile technique is more constraining in terms of freedom of movement.

ACKNOWLEDGMENTS

Many thanks to Nathalie Sahuque, Céline Lemercier, Dong-Bach Vo, Theophanis Tsandilas and Philippe Truillet to their active participation in the study; and to the participants in the different phases of this study. Part of this work is funded by the Région Midi-Pyrénées - France, through the grant “Praxis Environment Project”.

To conclude we can say that our fourth hypothesis (H4) is confirmed. Overall the users prefer use the Tangible technique because it offers a stronger freedom of movement.

380

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

REFERENCES 1. Azuma, R. A survey of augmented reality. Presence:

12. Dourish, P. Where The Action Is: The Foundations Of

Embodied Interaction. MIT Press, 2004.

Teleoperators and Virtual Environments 6, 4 (1997), 355-385.

13. Faconti, G., Massink, M.: Analysis of pointing tasks on

a white board. In: Doherty, G., Blandford, A. (eds.) DSVIS 2006. LNCS, vol. 4323, pp. 185–198. Springer, Heidelberg (2007)

2. Aylward, R. and Paradiso, J. A. 2006. Sensemble: a

wireless, compact, multi-user sensor system for interactive dance. InProceedings of the 2006 Conference on New interfaces For Musical Expression (Paris, France, June 04 - 08, 2006). New Interfaces For Musical Expression. IRCAM — Centre Pompidou, Paris, France, 134-139.

14. Fish, R. S., Kraut, R. E., Root, R. W., and Rice, R. E.

1993. Video as a technology for informal communication. Commun. ACM 36, 1 (Jan. 1993), 4861. 15. Fitts P.M. The information capacity of the human motor

3. Bach, C. and Scapin, D. L. (2010). Comparing

system in controlling the amplitude of movement. Journal of Experimental Psychology, 47:381–391, 1954.

inspections and user testing for the evaluation of virtual environments. Intl Journal of Human-Computer Interaction, 26(8). p 786-824.

16. Grossman, T. and Balakrishnan, R. 2005. The bubble

cursor: enhancing target acquisition by dynamic resizing of the cursor's activation area. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 - 07, 2005). CHI '05. ACM, New York, NY, 281-290.

4. Bangor, A., T. Kortum, P., T. Miller, J. (2008) An

Empirical Evaluation of the System Usability Scale. Intl Journal of Human-Computer Interaction, 24(6). p 574594. 5. Bérigny Wall, B., Wang, X. InterANTARCTICA:

17. Grudin, J. The computer reaches out: the historical

Tangible User Interface for Museum Based Interaction, International Journal of Virtual Reality 8, 3 (2009) 1924.

continuity of interface design. Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, ACM (1990), 261-268.

6. Blanch, R., Guiard, Y., and Beaudouin-Lafon, M. 2004.

18. Guiard, Y., Blanch, R., and Beaudouin-Lafon, M. 2004.

Semantic pointing: improving target acquisition with control-display ratio adaptation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 519-526.

Object pointing: a complement to bitmap pointing in GUIs. In Proceedings of Graphics interface 2004 (London, Ontario, Canada, May 17 - 19, 2004). ACM International Conference Proceeding Series, vol. 62. Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, 9-16.

7. Blanch, R. and Ortega, M. 2009. Rake cursor:

improving pointing performance with concurrent input channels. In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09, 2009). CHI '09. ACM, New York, NY, 1415-1418.

19. Guimbretière, F., Stone, M., and Winograd, T. 2001.

Fluid interaction with high-resolution wall-size displays. In Proceedings of the 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, 21-30.

8. Bryan, D. and Gershman, A. The Aquarium: A Novel

User Interface Metaphor for Large, Online Stores. Database and Expert Systems Applications, International Workshop on, IEEE Computer Society (2000), 601.

20. Han, J. Y. Low-cost multi-touch sensing through

frustrated total internal reflection. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology (Seattle, WA, USA, October 23 - 26, 2005). UIST '05. ACM, New York, NY, 115118.

9. Brooke, J. (1996). Sus: A quick and dirty usability

scale. In Jordan, P. W., Weerdmeester, B., Thomas, A., and Mclelland, I. L., editors, Usability evaluation in industry. Taylor and Francis, London.

21. Hansen, A. S., Overholt, D., Burleson, W., Jensen, C.

10. Davis, J., Chen, X. (2002). Lumipoint: Multi-User

N., Lahey, B., and Muldner, K. 2009. Pendaphonics: an engaging tangible pendulum-based sonic interaction experience. In Proceedings of the 8th international Conference on interaction Design and Children (Como, Italy, June 03 - 05, 2009). IDC '09. ACM, New York, NY, 286-288.

Laser-Based Interaction on Large Tiled Displays. In Displays, Volume 23, Issue 5. 11. DiPaola, S. and Akai, C. Designing an adaptive

multimedia interactive to support shared learning experiences. ACM SIGGRAPH 2006 Educators program, ACM (2006), 14

22. Ishii, H., Ullmer, B. Tangible bits: towards seamless

interfaces between people, bits and atoms.

381

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, United States, March 22 - 27, 1997). S. Pemberton, Ed. CHI '97. ACM, New York, NY, 234-241.

29. Mimio interactive whiteboard, 2005.

http://www.mimio.com 30. Norman, D. A., & Draper, S. W. (1986). User Centered

System Design: New Perspectives on Human-computer Interaction (1er éd., p. 526). CRC.

23. Ishii, H., Ben-Joseph, E., Underkoffler, J., Yeung, L.,

Chak, D., Kanji, Z., and Piper, B. 2002. Augmented Urban Planning Workbench: Overlaying Drawings, Physical Models and Digital Simulation. In Proceedings of the 1st international Symposium on Mixed and Augmented Reality (September 30 - October 01, 2002). Symposium on Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 203.

31. Piper, B., Ratti, C., and Ishii, H. 2002. Illuminating

clay: a 3-D tangible interface for landscape analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Changing Our World, Changing Ourselves (Minneapolis, Minnesota, USA, April 20 - 25, 2002). CHI '02. ACM, New York, NY, 355-362.

24. Jota, R., Pereira, J.M., et Jorge, J.A. A comparative

32. Robertson, G., Czerwinski, M., Baudisch, P., Meyers,

study of interaction metaphors for large-scale displays. Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, ACM (2009), 4135-4140.

B., Robbins, D., Smith, G., and Tan, D. 2005. The Large-Display User Experience. IEEE Comput. Graph. Appl. 25, 4 (Jul. 2005), 44-51. 33. Santos, C.R.D. et Gros, P. Multiple Views in 3D

25. Kobayashi, M. and Igarashi, T. 2008. Ninja cursors:

Metaphoric Information Visualization., International Conference on, IEEE Information Visualisation (2002), 468

using multiple cursors to assist target acquisition on large screens. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 949-958

34. Skaburskis, A.W., Vertegaal, R., et Shell, J.S.

Auramirror: reflections on attention. Proceedings of the 2004 symposium on Eye tracking research \& applications, ACM (2004), 101-108.

26. Mackenzie I.S. Fitts’ law as a research and design tool

in humancomputer interaction. Human-Computer Interaction, 7:91–139, 1992.

35. Vogel, D. and Balakrishnan, R. 2005. Distant freehand

pointing and clicking on very large, high resolution displays. In Proceedings of the 18th Annual ACM Symposium on User interface Software and Technology (Seattle, WA, USA, October 23 - 26, 2005). UIST '05. ACM, New York, NY, 33-42.

27. Letessier, J. and Bérard, F. 2004. Visual tracking of

bare fingers for interactive surfaces. In Proceedings of the 17th Annual ACM Symposium on User interface Software and Technology (Santa Fe, NM, USA, October 24 - 27, 2004). UIST '04. ACM, New York, NY, 119-122.

36. Wagner, D. and Schmalstieg, D. Artoolkitplus for pose

tracking on mobile devices. Proceedings of 12th Computer Vision Winter Workshop, (2007), 139-146.

28. MacKenzie, I.S. et Jusoh, S. An Evaluation of Two

Input Devices for Remote Pointing. Proceedings of the 8th IFIP International Conference on Engineering for Human-Computer Interaction, Springer-Verlag (2001), 235-250.

37. Zhai, S. Characterizing computer input with Fitts’ law

parameters ― The information and non-information aspects of pointing.International Journal of HumanComputer Studies, 61(6), 791-809. December 2004

382

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Five design challenges for human computation Stuart Reeves and Scott Sherwood Department of Computing Science University of Glasgow, UK {stuartr, sherwood}@dcs.gla.ac.uk ABSTRACT

offering vast resources of computation. In particular, it is this potential for solving hard computational tasks that has drawn researchers to designing systems which involve human activity as an integral part of their operation. These attempts have been characterised as ‘human computation’ systems, employing humans as “processing nodes for problems that computers cannot yet solve” [29], while others have likened this opportunity to a dynamically available “remote server rackspace” of “distributed human brainpower” [34]. Motivating this trend is the knowledge that, in theory, interactive computation provides greater computational power than non-interactive algorithmic systems [33].

Human computation systems, which draw upon human competencies in order to solve hard computational problems, represent a growing interest within HCI. Despite the numerous technical demonstrations of human computation systems, however, there are few design guidelines or frameworks for researchers or practitioners to draw upon when constructing such a system. Based upon findings from our own human computation system, and drawing upon those published within HCI, and from other scientific and engineering literatures, as well as systems deployed commercially, we offer a framework of five challenging issues of relevance to designers of systems with human computation elements: designing the motivation of participants in the human computation system and sustaining their engagement; orienting participants, framing and orienting participants; using situatedness as a driver for content generation; considering the organisation of human and machine roles in human computation systems; and reconsidering the way in which computational analogies are applied to the design space of human computation.

Existing literature within HCI includes many impressive demonstrations of ways that human activity, coupled with computational processes, may be employed to solve varied problems, such as image recognition [28, 31] and audio tagging [17]. As yet, however, there is little research that offers generally applicable recommendations or design frameworks that might assist us in realising this potential power. Furthermore, there are numerous systems outside the domain of HCI that draw upon similar strategies to human computation. So, when we consider more general design frameworks for human computation, it appears sensible to examine the larger domain of systems that involve complementary human-machine relationships (‘human-based computation’), in which computer systems become partners in interactive processes [11]. For example, we have seen the spread of systems employing similar techniques to human computation in scientific domains outside computer science. ‘Citizen science’ projects have employed human computation methods, particularly for processing large data sets. There are also elements of human computation concepts within popular notions of ‘crowdsourcing’ [14] and the ‘wisdom of crowds’ [25], as well as increasing numbers of mainstream and commercial crowdsourcing ventures. Work done in HCI also links to other disciplines within computer science, and interest has been generated in information retrieval, natural language processing, genetic and evolutionary computation, and artificial intelligence communities.

Keywords

Human computation, design framework, games with a purpose, citizen science, crowdsourcing. ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

Recently within HCI, a strand of research has developed concerned with the ways in which we might harness human ‘computational power’ for the purposes of solving difficult computational problems. Most prominently this work has manifested itself within HCI via the so-called ‘games with a purpose’ (GWAP) genre (e.g., [28, 30, 31, 13, 9]), in which hard problems for computation (e.g., image labelling) are distributed to humans in the form of competitive games. This work has a wider significance. Human activity has been envisaged, particularly within networked and ubiquitous systems (e.g., the internet, ad-hoc networks, etc.), as Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

As a contribution to this growing literature, we develop five core challenges facing designers of human computation systems. Before presenting these distinct challenges, we review in more detail our own human computation game, ‘EyeSpy’, and the range of human computation systems within the literature. This literature, coupled with our own

383

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

human computation system design experiences, inform the design challenges.

‘rounds’ lasting 3 weeks in total. Participants’ interactions with the game were extensively logged. After the game’s end we interviewed all participants, the transcripts of which were qualitatively analysed for key themes of player experience.

HUMAN COMPUTATION SYSTEMS

In this section we review various human computation systems within HCI, as well as our own mobile human computation system [1]. We shall also examine other system examples with similarities to human computation, and begin to piece together a sense of the boundaries of the design space.

As a by-product of players’ activities, EyeSpy collects a high quality set of photos that are useful for navigation on foot around urban areas. In the game, players ‘tagged’ landmarks and other prominent urban features within their local environment. Other players then physically located and confirmed these ‘tags’ (see Figure 1) for which they earned points for themselves and the players who created the tags. Participants generated 257 georeferenced photo tags and 197 georeferenced text tags over the course of the trial. Through engaging in the game, players generated a set of photographic tags that consisted of varied pictures of buildings, streets, monuments, signs, and other objects in the environment. Crucially, in order to be successful within the game, players needed to ‘design’ tags of high quality for the purposes of play, i.e., to be very findable and recognisable images, that other players could rapidly go to the locations in question and align themselves so as to successfully confirm the tags. In this way, our study found that players demonstrated a concern for the navigational experiences of one another [1]. A follow-on experiment showed that the photo set’s navigational qualities, designed into them by players, significantly assisted their secondary use as navigational aids during a simple route finding test (subjects had to locate images from the photo sets, and corresponding geolocated images drawn from Flickr) [1].

Games with a purpose

Human computation ‘games with a purpose’ within HCI typically involve simple game mechanics in order to produce fun and enjoyable activities for players. As a byproduct of player activities, these systems generate useful data for other tasks. The ESP Game epitomises this [28]. Players are paired online via a website, and type relevant descriptions for a given image which, if matched with the other player’s keywords, score the players points. This activity results in the rapid collection of annotations for large numbers of images. In its design, the ESP Game addresses the hard computational problem of generating meta-data about large quantities of images in order to make them searchable. Various other games with a purpose have been devised in the style of the ESP Game, such as Verbosity [30] and Matchin [13] (like the ESP Game, both are presented as web-based games). Verbosity attempts to collect a database of ‘commonsense’ statements within the context of a game structure similar to that of the ESP Game. The purpose of Verbosity is the reuse of these commonsense statements within AI applications. Matchin on the other hand involves each user selecting an image they think the player they are paired with will prefer. If both players match in their predictions about what the other player will prefer, then the players both score points. Games produced within this genre also involve the categorisation of other media than images. TagATune [17], for example, involves categorisation of audio clips, where players work to commonly agree upon textual descriptions for a section of music that is played to both of them at the same time.

Thus, as a by-product of player activities, EyeSpy offers one solution to the difficult problem of selecting ‘good’ images to assist navigation. The interaction of players, and their activities with and around the game, can thus be seen as a way in which to ‘process’ the physical urban environment and extract suitable imagery that is navigationready. Whilst such a task may well be partially possible via existing machine-based algorithms for processing image databases (e.g., extracting ‘good’ navigational images from Google Streetview or Flickr’s geotagged image sets), the human work employed within EyeSpy permits the designer to leverage local knowledge and player orientation to ‘what anyone knows’ regarding the navigational features of the

von Ahn and Dabbish also characterise GWAP systems as follows: “output agreement”, where players must share the same input, and must match their outputs whilst being unable to see one another’s outputs; “input agreement”, where players receive different or identical inputs and, by sharing one another’s outputs, must determine between them whether those inputs are indeed different or the same; and finally, “inversion-problem” games, where one player receives an input, and the other player must determine this input, based upon the first player’s output [29]. It was within the context of human computation games that we designed our own system, EyeSpy [1]. EyeSpy is a simple mobile-based multiplayer game that generates photos and text labels for geographic locations. In order to test the game we recruited 18 participants who were each given a phone with the game software. The game itself ran for two

Figure 1: Browsing available tags (l); confirming a tag (r).

384

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

local area in which the game was played. This particular topic will be returned to in greater detail in our set of challenges for human computation systems designers.

collection techniques in urban environments in order to map out pollution data. Crucially for human computation, such systems support selection of data that is meaningful to urban inhabitants by through drawing upon human agency and local practices tied up in urban life. In contrast, using a fully automated system that drew information from dense arrays of static sensors in the environment might collect similar data, however may well entail great computational complexity in determining which readings were most meaningful to local inhabitants.

Citizen science

Games with a purpose are only one part of the range of systems that employ human computation. For example, there are increasingly many websites and applications that recruit members of the public in order to assist the processing of scientific data. This distributed ‘citizen science’ technique enables the analysis of large data sets that would be intractable via machine computation.

Crowdsourcing

One of the most interesting examples of citizen science, Galaxy Zoo, is a website that invites users to engage in the recognition of galaxy types drawn from image data generated by the Sloan Digital Sky Survey [8]. Users voluntarily classify images according to specific attributes such as the number of spirals, general shape (e.g., ‘cigar’ shaped), and any unusual features that might be visible for a given galaxy (see Figure 2, left). Recognition of particular types may involve recourse to the community of Galaxy Zoo users, where users negotiate an agreed classification of images and develop common orientations to the methods of classification (i.e., develop competence). In this and other ways, Galaxy Zoo users may engage in the community of others via forums on the website. Findings of users are highlighted to the community via “object of the day” postings and so on.

Systems popularly characterised as ‘crowdsourcing’ applications often rely on human computation methods in their configurations of humans and machines. One of the bestknown examples is Amazon’s Mechanical Turk website [19] in which tasks are distributed to users who subsequently receive payment based on their completion of the task. Many of these tasks involve hard computational problems, for instance, labelling images, the generation of summarisations of text documents, or transcribing audio and performing recognition on video streams.

Other groups of scientists with large data sets have taken a similar approach. Stardust@home [23], for instance, draws on visitors of its website to engage in the detection of interstellar dust particles embedded in the aerogel collector on NASA’s Stardust spacecraft (see Figure 2, right). For various reasons, these dust particles are very hard to detect automatically, and so analysis is conducted using humans.

A further example, perhaps better known to HCI audiences, is reCAPTCHA [32]. Web CAPTCHAs are used to secure websites against spammers, by challenging a potential user with a hard computational problem that is easy for humans to solve (in particular, the recognition of distorted text). The reCAPTCHA system modifies this by sourcing the distorted text from words selected from digitised books via an OCR process. Since OCR processes may fail to perform correct recognition, human activity (i.e., reading, recognising and rendering CAPTCHAs into plain text) may be employed to solve the problem. Thus, reCAPTCHAs use the ‘side effects’ or ‘by-products’ of human activity directed towards another purpose (e.g., accessing a portion of a website), as we saw with EyeSpy and other games with a purpose.

Other services provide mobile versions of the Mechanical Turk, such as TxtEagle, which recruits developing world populations for various tasks such as translation and transcription work (that often require highly localised knowledge). These tasks are delivered to participants by mobile phone in return for payment [27].

In contrast with the above two examples’ use of relatively raw data, citizen science systems also may involve humans manipulating abstract representations of data. FoldIt [7] in particular recruits participants to manipulate protein structures as represented in a downloadable application. Large numbers of participants ‘fold’ these protein structures into stable shapes—something that is a hard computational problem—thus leading to the mass collection of appropriate protein configurations.

Interactive computations

Our final systems to review differ in scale and form to citizen science, crowdsourcing and games with a purpose. Interactive evolutionary computation (IEC) and interactive machine learning systems [6] are employed for computational problems in which selecting optimal configurations that are best for a given process is computationally hard. So, in order to solve the problem of deciding upon an optimisation index, IEC systems rely upon human interaction, bringing a user’s activity to bear upon the task, and exploiting human judgement in order to guide the ongoing machine-computational process.

Finally, citizen science systems may also be mobile-based. Paulos et al. [20], for example, employ participatory data

Early work on ‘biomorphs’ [5], in which users select a particular evolutionary path from various machine-suggested paths, led to various other demonstrations of IECs. In an

Figure 2: Classifying galaxies with Galaxy Zoo (l); a slide from Stardust@home (r)

385

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

extensive review of the field, Takagi discusses how human evaluative abilities can be brought to bear on computational problems, such as image detection, in which current algorithms benefit from human guidance in order to finesse results [26]. In another example, human scientific expertise is brought to bear upon geophysics problems, such as determining the location of natural mines through modelling mantle convection within the earth [26]. (This is, interestingly, an inverse of the well-known expert systems found in artificial intelligence.)

of these systems. Some systems we have covered in our review push the boundaries. For example, systems like Matchin begin to involve more aesthetic judgements that are less clearly amenable to algorithmic approaches, i.e., determining human preferences for attractive imagery, although we note that services such as Flickr’s “interestingness” rating for photos [4] indicate algorithmic approaches. In a similar way, IEC systems may also combine more obviously subjective processes, such as guided music composition (Sonomorphs and GenJam [26]) or 3D lighting design, in which computer-generated arrangements of lighting in virtual scenes are selected by human intervention (i.e., to decide the appropriate configuration for the scene).

Interestingly, and unlike other systems discussed so far, IECs typically do not rely on mass participation, in contrast with ‘citizen science’, ‘games with a purpose’ and ‘crowdsourcing’ applications (although mass participation is not inconceivable). Instead they invert the model: multiple results are machine-generated and then presented to a single user for selection, rather than one machine distributing many tasks to multiple users.

Citizen science has also been used to describe systems that appear to sit further outside the boundaries of what we might consider human computation. For example, some projects involve large numbers of geographically distributed participants collecting biological data, e.g., Ebird or NestWatch. Such tasks are not obviously computational matters and have no clear algorithmic representation. Similarly, crowdsourcing systems also emphasise the blurry boundaries of human computation. The Mechanical Turk website, for instance, offers many tasks that fall well outside what we have considered so far as a relevant computational problem for human computation systems. These may be obviously non-computational tasks, such as writing reviews of music, or, instead, computational tasks that are relatively simple (i.e., not ‘hard’), such as scraping web content. Finally, some TxtEagle tasks, such as citizen journalism, also do not fit within the boundaries as they would stretch the notion of ‘computation’ in the term ‘human computation’.

What do we mean by human computation?

Based on our brief review, there are a number of systems that might plausibly be ‘human computation’ systems. As such, we can begin to piece together what human computation systems ‘look like’, albeit in an approximate way. Firstly we can say that the systems we have reviewed generally rely upon interaction between machine and human (e.g., via some kind of division of labour between the two [16]). Secondly, we can see that another important aspect shared by these systems is that work is done by a human that might otherwise be represented algorithmically for use in a computer system. Indeed, other authors ask similar questions of whether an algorithm could possibly produce similar results to their system in order to present it as being in the ‘human computation’ genre [3]. Bound up in this is the question of the level of output quality of an equivalent machine-computable version of the task, which itself often contrasts drastically with human computation system outputs. So, for example, whilst machine-based translation is possible, results remain poor outside of domain-bound tasks (e.g., weather reports) when compared with human-based translation systems.

DESIGN CHALLENGES FOR HUMAN COMPUTATION

In our short review above, the use of human computation methods within standalone, networked and mobile systems is very varied. With the increasing diversity of applications of human computation, there is a need for developing a design-oriented understanding for the ways in which humans and their interactions together may be employed as computational ‘components’ within machine processes.

We note that the classes of problems delegated to humans tend to be what are seen as computationally ‘hard’ tasks, such as computer vision problems, or natural language parsing, which are currently difficult for computers to perform, but easy for humans [29] (although there may be reasons to delegate computationally ‘easy’ or tractable tasks, which we shall discuss later). Generally we suggest that human computation tasks can often be characterised by knowledge-based procedures, expertise or skilled activity, or activity that is highly contextualised. Such tasks often draw upon commonsense and practical reasoning that everyday members of society engage in routinely.

Drawing from our study of EyeSpy and the systems reviewed in the previous section, here we develop five challenges facing designers of human computation systems. Some of these challenges also have relevance outside the domain of human computation we have loosely defined and in concluding we shall discuss this issue. Challenge one: designing for motivation and sustainability

Human computation tasks will require groups of participants to be recruited and then motivated and sustained in particular ways. As part of configuring player activity in EyeSpy, we had to consider how players could be motivated to engage in playing the game as well as reflecting upon the inherent ‘saturation point’ that was reached after a certain amount of play (i.e., when all the ‘obvious’ land-

However, as this approximate ‘definition’ might indicate, the boundaries of human computation remain fuzzy, particularly with regard to what constitutes a reasonable task. Thus there are some caveats in this rough characterisation

386

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

marks had been tagged). In this way a considerable issue for designers is the need to choose appropriate motivation and sustaining strategies for the task at hand and likely participant base. Here we will examine the motivation and sustaining strategies used by other systems we have reviewed as well as our own system.

the way in which user motivation is framed appears to be altruistic, in that participants are asked to contribute towards scientific endeavours. Of course, other motivational factors such as individual fun and enjoyment may still play a vital role in user engagement in such systems. There are also potential situations in which users may gain social status amongst other participants. For Galaxy Zoo, highlighting unusual images offers a ‘payoff’ for participants (e.g., an anomaly, “Hanny’s Voorwerp”, was named after the Galaxy Zoo user who detected it, Hanny van Arkel, bringing them to prominence and generating further scientific investigation [18]). Thus, for Galaxy Zoo, gaining participants relies on user self-motivation initially, but the configuration of the system maintains these users both through potential payoffs and social interaction in the form of community forums in which findings may be shared and discussed (indeed, this is how “Hanny’s Voorwerp” was first brought to attention).

Competition and gaming

In EyeSpy, we configured the motivations of players primarily by offering them a competitive game and scoring structure, which most players reported as a major feature of their own interest in taking part. This approach is common among games with a purpose. At the core of these gamebased and game-like systems is the notion of competition (often in the form of scoreboards or ranking schemes), and social interaction between players (via in-game chat, forums, etc.). So, in using the combination of competition and social interaction to form a key role in motivation to play, such human computation systems will as a by-product enable the processing of large amounts of data.

Compensatory payoffs are perhaps made clearer for Stardust@home users. Besides a more prominent user rankings system helping foster clearer competition between participants, any participants discovering a particle of interstellar dust are offered co-authorship on publications related to that particle, in addition to naming the particle themselves. This more explicit competition compares with strategies employed in EyeSpy and other ‘games with a purpose’. Stardust@home, whilst not being a ‘game’, presents an explicit motivational scoring structure with further motivation provided by the accolade of finding a particle track.

Characteristic of a competition or game-based motivational strategy is the increased separation between the role of the objects manipulated by players within the game, and the role of those same objects outside of the game. Thus, for instance, for ESP Game images, players interact with one another using the game objects (i.e., images) as the central focus of the work of playing the game. However, subsequently these images and descriptive tags are then used in a very different context, namely image search. This again is similar to EyeSpy, with players generating images for other players to confirm, whereas subsequently the same images were repurposed for navigational tasks of little relevance within the game for players.

Here we can return again to the notion of game structures and their merging with other motivational forces, such as altruism. At the same time we are also led to consider the separation between the payoffs for the designer, and the payoffs for the user. We find a range of choices with regard to how much human computation systems contain gamelike components.

Although a citizen science project, FoldIt is explicitly marketed to users as a “puzzle” game and includes game-like components such as leaderboards of most prolific ‘folders’. Further to this, FoldIt users may form teams in order to compete in a more collaborative way with other users. However, FoldIt is not framed purely as a game to potential users, and, like Galaxy Zoo and Stardust@home, highlights the scientific importance of a user’s activity (in this case helping find medical advances in treatments for HIV / AIDS, cancer and so on). In this way, FoldIt merges motivational strategies of gaming enjoyment, and at the same time implies that a participant’s altruism has some relevance for their engagement with the system. The ESP Game and others in the genre also highlight that the player is not only “having fun”, but helping “computers get smarter” [9], thus contributing to some scientific or engineering achievement. This leads us to examine the role of altruism—implied or otherwise—within human computation systems.

One side of this range is typified by citizen science projects that have hints of game qualities, and yet involve fairly explicit integration of the products of the task, i.e., payoffs for the designers, and what it is that users are achieving, i.e., the payoffs for the user. In Stardust@home, for example, we see a clear and explicit focus on the examination of technical data and the potential resulting scientific accolades for a discovering user. This payoff coincides with the payoff for the designers as well (e.g., a publication). At the same time, we also see on the Stardust@home site gamelike elements such as a leaderboard, which promotes competition and potential payoff mostly between users (since it is irrelevant for the scientific results how the details of any competition between users works out). The opposite side of this range are systems in which user payoffs and manipulation of task objects (e.g., enjoyment in the case of playing the ESP Game, and tagging its images) are very separate from the payoffs for the designer, and the subsequent repurposing of those objects (e.g., the ESP Game’s link to image search).

Altruism and status payoffs

A common motivation relied upon by the citizen science projects we have examined in previous sections is a sense of altruism on the part of the participants. In systems such as Galaxy Zoo and Stardust@home, a considerable part of

387

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Monetary motivations

Discussion

For EyeSpy, we also offered a nominal payment in order to compensate players for any inconvenience generated by playing the game (e.g., going out of their usual routines). This was also a motivational feature of the game, however was balanced with players’ desire to compete with one another and to win. Such monetary motivations are also present in crowdsourcing systems such as the Mechanical Turk and TxtEagle, except in these cases, providing a core component of participants’ interaction with the system (i.e., that, on the completion of a task, participants then receive payment). It is interesting see that identical human computation tasks found in the ESP Game and on the Mechanical Turk website may differ greatly in the strategy employed for motivation: the ESP Game is presented as a simple, fun game with the payoff being enjoyment (or perhaps a momentary distraction), with no monetary motivation whatsoever, whereas similar results are achieved through explicit monetary means as Mechanical Turk tasks.

Various obvious motivational strategies are offered in human computation systems: fun, altruism, social interaction, payment and competition, to name a few (as there are surely others). Strategies such as these also may be employed more generally in trials of interactive systems (e.g., paying compensation for participation is a common technique). We are also reminded that, for many of the human computation systems reviewed in this paper, social interaction is a core motivation for participation even when this appears peripheral to the actual task. Configuring motivation using these strategies forms a key design consideration for human computation systems, and will have significant knock-on effects for the quality or characteristics of the objects that come to be manipulated or analysed by users. Designers must take into account the various vectors of motivation that may come into play when engaging participants in their design, and be aware of motivations that are emergent and have not explicitly been designed into the system.

Sustaining human computation

Coupled with these issues, and raised in our analysis of EyeSpy, is one aspect of sustaining engagement, in particular the problem of how ‘complete’ work done by participants during some human computation task may be. As we found with EyeSpy, due to the players saturating a small geographic area with photo and text tags, play within that area (i.e., finding locations that had not been tagged already) became more difficult with time. Participants explicitly mentioned this as a demotivating factor in playing the game. Thus, sustaining play came at the expense of the motivation strategies we employed (fun, enjoyment and competition).

In these instances designers may attempt to minimise the possibility of deliberate disruption, motivating or assisting users with engaging in the task at hand. Designers are also constructing potential social interactions that may be folded into use with and around the system, and this is a further key component of motivation for users. For games this is most apparent when rules of competition provide explicit support for social interactions that motivate engagement in the human computation task. Finally, considering whether we might want a continually changing group of participants or a stable group is an important decision to address in system design. It may be a virtue for the human computation task at hand to have a rapid ‘churn’ of participants. This ‘lifetime’ of a user, combined with what the design requires for successful outputs of participant task work, may at one extreme maximise lifetime through maintaining user interest in the task, or at the other extreme ensure a continual flow of new users. Alternatively a significant component in the sustained success of a human computation task may be how to fit the system into existing everyday routines and interactions.

Sustaining motivation is also important in other systems. Projects such as FoldIt, for example, are enriched by participants becoming more skilled in their task, so in this case, fostering community and competition, and thus sustained engagement, becomes important. Contrastingly, systems like the ESP Game may benefit from a more rapidly churning user base in order to assist the reduction of players becoming very familiar with the system and potentially developing ways to ‘game the system’. However, within the literature, there are few accounts of how existing systems sustain participants over long periods of time, and the challenges that are faced by designers over this matter (or how the user base itself may change with time). Players may get bored by the simplicity of a game, or confused by over-complex rules (or perhaps confused by system ‘framing’, as in EyeSpy). Alternatively they may suffer from fatigue, drop out, or have problems weaving their play into everyday life. Other significant social factors will influence the success of human computation systems, such as how to ‘market’ them to potential users. Users may move on, and be replaced by entirely different people with markedly different practical engagements with the system. Many of these and other questions remain unexplored, but remain important factors when designing human computation systems.

Challenge two: balancing system design and user practices through orientation and framing

The second challenge involves understanding the nature of balance within human computation system design. This includes understanding participant use of that design, in practice. Once again, this challenge has relevance for other HCI systems. We found that EyeSpy’s game rules oriented players toward a strong concern for two aspects of navigation— recognisability and findability—which were in accord with our repurposing those images for navigation. Nevertheless, the game rules and context encouraged rather than enforced such an orientation. Through this we came to appreciate the importance of considering the context in which the game was presented. For example, even the language we used in

388

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

order to introduce and frame the game configured certain expectations about how to play (using the name ‘EyeSpy’ confused some players initially due to the similar name of the children’s game ‘’). Oriented by this, some players initially began creating ‘riddle’ tags that involved cryptic descriptions of locations, requiring ‘detective work’ on the part of other players (i.e., in keeping with the notion of ‘’). Rapidly, however, players ceased to create such tags, since the game encouraged an orientation to findable and recognisable tags. This ‘framing’ of the ways in which we presented the system to our participants thus challenged the system’s premises.

action users engage in [21]). The design of the system will shape, be appropriated, and be subverted by the practical and mundane activities of users. There are further complications for framing and orienting users human computation systems, however, particularly within systems that produce ‘by-products’. Once again, designer and user orientations may involve potentially conflicting or competing concerns, as well as more complementary or harmonious concerns. In systems with by-products, the framing of the objects manipulated by users within the system, the intentions in that manipulation, and the role and intentions of use of those same objects outside of the system (say, within some scientific practice), must be carefully balanced.

The way participants in human computation systems practically interpret system rules is influenced by the way in which the system itself is presented to them. By being aware of and even explicitly designing a process of framing in the deployment, we can attempt to orient participants to a particular way of conducting themselves in their interactions with and around the system, and with others. One practical way to do this is to take care with the language that is presented to users of the system, and perhaps employ scripted introductions and debriefings for users to carefully configure their expectations in ways designers intend.

There are of course many such dimensions and factors that play a role in successful configurations of these relationships. The challenge might be, for instance, balancing creativity and fun (as normally associated with gaming and more widely, systems that are enjoyable to use) with the requirement for quality system by-products. In order to meet such challenges, highly adaptive software design processes that involve significant levels of user feedback and offer the possibility of rapid updates in response, may form one way to quickly and iteratively fine-tune the relationship between system rules and human computation products. In this way a suitable balance between motivation strategies (e.g., enjoyment) and the quality of participant’s work may be reached more rapidly than for more traditional iterative design involving successive static software deployments.

Researchers in other domains have also described the relevance of system framing for users and how it influences their interpretation of the system and directs subsequent action [12]. This also has relevance for the design of other human computation systems we have reviewed; how the attitudes, norms and particular approaches of users in their interaction with one another are shaped by the system’s very construction and design. Matchin is of particular interest here because it throws into relief some of the mechanics present in other ‘games with a purpose’. In their study of Matchin’s results, Hacker and von Ahn suggest that cultural norms come to be reflected within the image selections, particularly the way in which considerations of gender came to feature in players’ interactions with one another (such as treating some images as ‘masculine’ or ‘feminine’, so influencing the other player’s preference) [13]. Thus, orientations to other participants as found in systems like EyeSpy and Matchin, forms a key part of the way that game objects are produced and manipulated.

Finally, we note that many systems are inherently ‘open’ in their construction; in particular the under-constrained nature of game rules fosters the construction and continual negotiation of shared understandings of ‘appropriate play’ in and around the game. Often within human computation systems design, this openness has often led to a preoccupation with issues such as accuracy, ‘cheating’ and ‘gaming the system’, and thus a concentration on the ways in which such activities may be curtailed in order to promote a particular quality of game objects (e.g., by-products). Whilst this is clearly an important feature, it should be coupled with a more general concern for how framings and orientations may shape user activity, in practice. Challenge three: using situatedness as a resource

The designer’s intent in constructing system rules, and how that intent is interpreted by participants, is also relevant. For instance, reCAPTCHAs present their purpose to users (e.g., to sign up to a service on a website), but hide their secondary purpose (e.g., to digitise scanned text). Instead of motivating participants with a game mechanic or relying on monetary or altruistic motivation, reCAPTCHAs exhibit an almost complete separation between the intentions of the participant in manipulating the object (i.e., rendering the image of text to plaintext) and the ways in which that activity is then used (i.e., to manually digitise books).

A key feature employed within the design of EyeSpy involved exploiting the situated, local understandings of players. In order to generate our useful data from EyeSpy, the design drew on ‘what anyone knows’ [10] about the local area in order to successfully select (and capture) navigationally relevant images. Players concerned themselves with other players’ potential routes, places locally considered to be central and so on. Social roles were also important to local understandings. This was demonstrated particularly by one of the participants in his orientation to ‘students’ as hypothesised recipients of his images (our trial involved mostly students, however non-students also played). However, although in EyeSpy the role of practical

The relationship between system rules and user practices is delicate and hard to predict (as are the specific sequences of

389

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

knowledge as deployed by players in the game featured as judgements regarding what people (‘anyone’) could find in a local area, we found that even this depended on the cultural positioning of players (e.g. as pedestrians in the city rather than drivers). More generally, then, in EyeSpy, exploiting the local knowledge of participants meant producing more culturally relevant images.

science’ systems may involve computer-based selection, organisation and distribution of the data to be processed, and alongside this, human innovation to conduct piecemeal analysis of the data. However, innovation work, i.e., reporting what can be ‘seen’ in, say, astronomical imagery in Galaxy Zoo, is, of course, the job for the human participants. IEC and interactive machine learning systems invert selection and innovation roles, with humans providing the selection work in order to guide ‘innovation’ on the computer’s part.

Other systems we have reviewed also purposefully exploit local knowledge. TxtEagle, for instance, relies upon the localised and practical understanding of its participants for a given area, in order to address highly specific linguistic problems such as providing local dialect terms to finesse translations.

In EyeSpy the machine role was selection of tags, using a strategy that distributed them randomly and without identifying information amongst players (who innovated by generating content). In our interviews, we found that players were often concerned with who else was playing the game. One player, for instance described it as “walking in the footsteps” of others whereas another “saw other people that [she] thought were doing the same thing”. In this way, organisation of the ‘thin channel’ between the selection interface and players helped motivate play, increasing the sense of intrigue reported by players and maintaining interest in the game.

This third challenge, like many of the others outlined in this paper, also offers an opportunity. EyeSpy demonstrated to us how human computation does not just involve producing ‘objective’ results, but can also be about using situated understandings to produce content that draws upon subjective, creative and practical knowledge. The centrality of situatedness in interaction has been a conceptual interest within HCI for some time (e.g., Suchman’s ethnomethodological analysis of situated action with technology [24]) and it is key for human computation system design that it both takes into account situatedness and even takes advantage of it.

For systems in which ‘innovation’ is the human role, we have also highlighted how human computation systems may be discussed in terms of content creation or content analysis (for humans in a ‘selection’ capacity, such as in an interactive machine learning system, the role will always be analysis). For EyeSpy, players created content which was then analysed by other players for validity, whereas for players of the ESP Game, the primary job is content analysis, and it is players’ various analyses of images which are then validated through gathering large numbers of results.

Thus as designers we must take note of everyday commonsense and localised understandings. These understandings may be a resource to exploit or a hindrance to work around. The situated nature of participant interactions with, via and around the system will offer the opportunity of taking advantage of local knowledge and practical understandings of the world that are often very difficult to access otherwise. EyeSpy and TxtEagle both demonstrate ways in which this may be exploited to the advantage of the system, and is particularly pertinent for any mobile human computation systems (of which we may see increasing numbers).

Challenge five: reconsider the utility of machine analogies in human computation

Our final challenge is conceptual. As mentioned in the introduction, much existing discussion on human computation systems has been based on information processing models of human activity. However, we suggest there are problems when using machine computation or abstract algorithmic processes as a design analogy for collected human activities. If we wish to use humans as ‘algorithms’ and “networked brainpower” [34] for hard computational problems, analogies between the two can potentially obscure the considerable design differences.

Although there is a clear opportunity here for design, so there is a corresponding potential danger in not accounting for such commonsense, culturally specific or highly localised understandings. This is especially true when deploying human computation systems on the web (e.g., GWAP, Mechanical Turk tasks). For instance, what might be the ways in which conflicts in ‘what anyone knows’ can come to bear when categorising the content of images found in the ESP Game? The geographic position of webbased participants is of relevance and we can imagine how one symbol may mean very different things to different groups of users (e.g., a swastika or manji, commonly used in Japanese maps to mark temples).

In what ways do the challenges for designing for machine algorithmic components and to designing for human ‘components’ differ? One immediate issue is that algorithms are generally deterministic and have known upper bounds calculation time (computational complexity). They are highly ‘accountable’ in that one can examine in detail precisely how an output was created. In comparison, within human computation systems, the time needed to obtain information is nondeterministic, and subject to the vagaries of human participation, motivation and conformity with regard to norms of interaction. For example, we were unable to predict how much content would be created during the 3 weeks

Challenge four: organising human-machine relations

A fourth challenge to designers is the organisation of the human and machine components in their system, particularly how to design their roles. One way, following Kosorukoff [16], is to analyse human computation systems as being based on variations of ‘selection’ and ‘innovation’ roles for humans and computers. So, for instance, ‘citizen

390

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

of trialling EyeSpy. In contrast to the fixed accountability of an algorithm, accountability in human computation systems is negotiated continuously between users themselves, which, again, was a key feature in the production of navigable images in EyeSpy—players’ activities were made accountable to one another in the game via tagging [1].

A final topic for discussion is the inherent imprecision of the boundaries and limits of what constitutes a human computation system. There are complexities in any potential definition, mostly due to the diffuse nature of what may or may not reasonably constitute a ‘machine-computable task’. We argue for the value of not providing too narrow or objectifying a definition, and that arriving at such a definition may not only be very difficult but perhaps inappropriate to the phenomenon under study.

Indeed, existing design techniques for human computation systems have been concerned with a very human problem – that of ways to preclude ‘gaming the system’ or feeding the system spurious data (e.g., [28]). This has highlighted the importance of moderation, quality control and ‘orchestration’ activities as vital components in keeping the system running successfully (see [15]), especially with large numbers of participants. However, this is only one part of a broader conceptual challenge in which human roles in computational systems are not seen as interchangeable with machine-based algorithmic components. So, as designers we should question the utility (and applicability) of simply repackaging design strategies that are useful for machine computation, when approaching human computation systems, their design, implementation, and evaluation.

In addition, our discussion retains some ambiguity partly because it is vital to understanding the relevance and scope of the challenges outlined in this paper. We have noted how some of the challenges have relevance both within and towards the boundaries of human computation tasks. For example, the importance of motivation strategies applies to citizen science projects such as NestWatch, which we have argued probably sits outside the limits of human computation. We note also that there are computational tasks which are very ‘easy’ for machines, however, the contexts that address such computational situations may well be ‘hard’ for other, non-computational reasons. The associated costs of developing machine-based algorithms might well be more costly than the more rapid development potential of humanbased solutions. Some tasks on the Mechanical Turk website, for example, rely on this fact (e.g., harvesting online data on businesses). Alternatively, logistics may mean that human computation methods work for the collection of data that could easily, but (in a monetary sense) expensively, be gathered. In these cases we might employ the non-monetary methods of human computation systems design, such as enjoyable game mechanics or altruism, in order to motivate and sustain participation.

DISCUSSION

This paper has offered five challenging dimensions along which to design human computation systems, providing a framework for designers that is derived both from our own study and wide ranging analysis of various systems in HCI and beyond. This framework serves multiple purposes: it provides both strategies and opportunities, but also sensitises designers to conceptual issues. The challenges we have covered in this paper vary in type from general interaction design issues, to particular opportunities that exist for human computation systems specifically, to questions that are hard to currently address fully. We can briefly reflect upon this framework for EyeSpy. (1) Designing for motivation and sustainability: we motivated our users with fun, competition and money, however we found the game design did not ensure it was sufficiently sustainable. This may have been solved with a more adaptive game design. (2) Balancing system design and user practices through orientation and framing: we experienced conflicts between our inadvertent framing and orientation of users (via the language we chose) and the way we intended users to engage with the system. Induction ‘rituals’ [2] and careful naming may have been of benefit. (3) Using situatedness as a resource: we created useful by-products (photos of landmarks, etc.) as a product of our system making a virtue of local knowledge as a game resource, although we note that the by-products were also a function of the particular group and mobility they employed—cyclists for instance would ‘see’ things differently (e.g., see [22]). (4) Organising human-machine relations: in EyeSpy, we maintained player interest by restricting the machine interface between players, making tags into ‘clues to be found’. (5) Reconsider the utility of machine analogies in human computation: we relied on the accountability of human activity, balanced via the restricted communication between players (see 4), in order to help drive successful tag creation.

CONCLUSION

This paper presented one way of understanding and structuring the emerging design space of human computation, and offers five key issues for consideration when designing such systems. We examined various strategies for designing and sustaining motivation, framing and orienting users, and organising human and machine components in terms of innovation and selection relationships. Opportunities such as exploiting everyday commonsense and localised understandings may also present themselves in human computation systems. We also explored conceptual issues such as the use of machine analogies. In this paper we have moved forward from demonstrations of particular designs and have begun to develop frameworks that offer more generalised directions for researchers and practitioners. This has not been without limitations, however; for instance, a restriction of our study of EyeSpy was the size and nature of the trial, and it is clear that mass participation is key in deepening our understanding of human computation system design. We are also particularly interested in further validating the framework by employing it in design formatively. Through understanding this space we hope to empower diverse groups of non-experts in de-

391

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

veloping their own human computation systems, as well as expanding human computation into areas it does not currently address.

18. Lintott C.J., et al. Galaxy Zoo: ‘Hanny’s Voorwerp’, a quasar light echo?, submitted to MNRAS. 19. Mechanical Turk website, http://www.mturk.com, verified 26/02/10.

ACKNOWLEDGMENTS

This research was funded UK EPSRC (EP/F035586/1, EP/E04848X/1, GR/N15986/01). Thanks also to the other members of the SUMGroup who helped directly or indirectly with this work.

20. Paulos, E., Honicky, R. and Hooker, B. Citizen science: Enabling participatory urbanism. In M. Foth, editor, Handbook of Research on Urban Informatics. IGI Global, 2008.

REFERENCES

21. Robinson, M. Design for unanticipated use. In Proc. ECSCW. Kluwer, pp. 187-202, 1993.

1. Bell, M., et al. EyeSpy: supporting navigation through play. In Proc. CHI, pp. 123-132, ACM Press, 2009.

22. Rowland, D. et al. Ubikequitous computing: designing interactive experiences for cyclists. In Proc. MobileHCI. ACM Press, pp. 1-11, 2009.

2. Benford, S. et al. The frame of the game: Blurring the boundary between fiction and reality in mobile experiences. In Proc. CHI, pp. 427-436, ACM Press, 2006.

23. Stardust@home website,

3. Bernstein, M. et al. Collabio: a game for annotating people within social networks. In Proc UIST, pp. 97100, ACM Press, 2009.

http://stardustathome.ssl.berkeley.edu,

verified

26/02/10. 24. Suchman, L. Plans and situated actions: The problem of human-machine communication. Cambridge University Press, 1987

4. Butterfield, D. S. et al. Interestingness ranking of media objects. U.S. Patent application 2006/0242139 A1, February 8, 2006.

25. Surowiecki, J. Wisdom of Crowds, Random House, 2004.

5. Dawkins, R. The Blind Watchmaker. Longman, 1986. 6. Fails, J. A. and Olsen, D. R. 2003. Interactive machine learning. In Proc. IUI ‘03. ACM, New York, NY, 3945.

26. Takagi, H. Interactive evolutionary computation: fusion of the capabilities of EC optimization and human evaluation, In Proc. of the IEEE , vol.89, no.9, pp.1275-1296, Sep 2001.

7. FoldIt website, http://fold.it, verified 17/09/09. 8. Galaxy Zoo website, http://www.galaxyzoo.org, verified 26/02/10.

27. TxtEagle, http://www.newscientist.com/article/mg20126956. 600-translations-by-text.html, verified 26/02/10.

9. Games With A Purpose website, http://www.gwap.com, verified 26/02/10.

28. von Ahn, L. and Dabbish, L. Labeling images with a computer game. In Proc ACM CHI, pp. 319-326, 2004.

10. Garfinkel, H. Some rules of correct decision making that jurors respect. In Studies in Ethnomethodology, pp. 104115, 1967, Prentice Hall.

29. von Ahn, L. and Dabbish, L. Designing games with a purpose. Communications of the ACM, 51(8): 58-67, 2008.

11. Goldin, D. et al. Interactive Computation: The New Paradigm. Springer Verlag: 2006.

30. von Ahn, L., Kedia, M., and Blum, M. Verbosity: a game for collecting common-sense facts. In Proc. CHI ‘06. ACM Press, pp. 75-78, 2006.

12. Grimes, A. et al. EatWell: sharing nutrition-related memories in a low-income community. In Proc. CSCW ‘08. ACM, New York, NY, 87-96, 2008.

31. von Ahn, L., Liu, R., and Blum, M. Peekaboom: a game for locating objects in images. In Proc. CHI, pp. 55-64, ACM Press, 2006.

13. Hacker, S. and von Ahn, L. Matchin: eliciting user preferences with an online game. In Proc. CHI ‘09. ACM, New York, NY, 1207-1216, 2009.

32. von Ahn, L., Maurer, B., McMillen, C., Abraham, D. and Blum, M. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science, 321 (5895): 1465-1468, 2008.

14. Howe, J. The Rise of Crowdsourcing, Wired, June 2006, http://www.wired.com/wired/archive/14.06/crowds .html, verified 26/02/10.

33. Wegner, P. Why interaction is more powerful than algorithms, Communications of the ACM, 40(5), May 1997, pp. 80-91.

15. Koleva, B. et al. Orchestrating a mixed reality performance. In Proc. ACM CHI, pp. 38-45, 2001. 16. Kosorukoff, A. Human-based genetic algorithm,

34. Zittrain, J. Ubiquitous human computing, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 366, No. 1881, pp. 3813-3821, 2008.

http://www.geocities.com/alex+kosorukoff/hbga/h bga.html, verified 26/02/10.

17. Law, E., von Ahn, L. Input-agreement: A New Mechanism for Data Collection using Human Computation Games. In Proc. CHI, pp. 1197-1206, 2009.

392

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Designing for crowds Stuart Reeves Dept. of Computing Science University of Glasgow, UK [email protected]

Scott Sherwood Dept. of Computing Science University of Glasgow, UK [email protected]

Barry Brown Department of Communication University of California, San Diego, USA. [email protected]

some exceptions, this work tends to address ‘audience-like’ formations (even where it deals with ‘crowds’, e.g., “crowd-and-DJ interactions” [10]) and is typically concerned with performance scenarios in which interaction between professional performer and audience is key. Issues such as the maintenance of awareness between collaborators, the ways in which they coordinate and design their conduct with an orientation to peripheral visibility and legibility, and conduct mutual monitoring of one another [16] have been applied to these new settings. For example, the availability and legibility of participants’ manipulations of an interface to surrounding audience has a clear importance for design within these public settings [23].

ABSTRACT

Designing for spectators and audiences presents new challenges to the design of technology. In this paper we focus our attention on understanding and designing for crowds as a distinct design topic. We present a study of one particular instance of crowd activity—football fans on match day. Close video analysis of interactions within the crowd reveals how crowds seeks to maintain membership through synchronisation of activity, but also how crowd support interaction between its members through co-ordination around shared objects and the ‘snowballing’ of songs and gestures. Drawing on this data we develop salient topics for HCI design for crowds, such as: reconceptualising interaction design to treat crowds as crowds rather than as groups of individual audience members; understanding intra-crowd interactions, via the use of shared objects and synchronising crowd interactions; and understanding the nature of peripheral participation in crowd activities, and interactions between distinct crowds. We also reflect on conceptual challenges that crowds pose for HCI as it increasingly develops its interests in public settings.

In this paper we develop this work further by focusing not on performance and performance-like situations, but instead convivial crowd-based settings. Crowds are distinct in that they offer a setting where large-scale participation is a key characteristic. Furthermore, this participation is not necessarily mediated by some singular ‘spectacle’ as with audience-performer scenarios, and any performance-like activities are more distributed, fluid and shared amongst members of the crowd. The relationships (and interaction) between members is much more varied than that between performer and audience, where (on the whole) there is a maintenance of shared attention. In crowds, individuals and the group maintain a shifting focus for participants.

Author Keywords

Crowds, design, spectatorship, sports fans. ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

This distinctness suggests the need to enhance existing models of audience-performer settings. Crowds also present distinct design challenges in how we can support interaction between a system and potentially thousands of users simultaneously. To explore these issues this paper we offers an empirical study of one crowd-based setting (a sports crowd). Using detailed video analysis of crowd interaction we develop observations concerning how individuals within crowds interact with each other, how they produce their status as members of a crowd, and how crowds manage their interactions with those on the periphery or outside the crowd.

INTRODUCTION

One key theme in HCI has been the exploration of new interactional situations that use technology. One recent area of interest has been settings in which users form part of an audience or crowd, spectating upon performer activity, and perhaps interacting with a system themselves [23]. Interest has also extended to situations involving the implication of passers-by in interactions with technology [4]. Studies of interaction in the performance arts [28], museums and galleries [18], city streets [9], funfairs [26] and clubs [10] have all expanded our understanding of these settings. With

RELATED WORK

Technical demonstrations of ‘crowd computer interaction’ [6] with systems to support spectators or audience (e.g., sports fans [31]) have been of interest to researchers for some time. One of the earliest audience-driven interaction systems was the ‘Cinematrix’, a large screen interactive [7], in which audience members (divided into two ‘teams’) held up coloured paddles in order to control interaction with a

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

393

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

simple game projected on a large screen. Thus, devices or controllers held by participants [31], or vision techniques to detect movement (e.g., [29]), coupled with large displays, have become the dominant paradigm for enabling multiscale crowd interactions. Interestingly, as noted by others, little of this work presents analyses of the material practices of users in these settings [21], and, although guidelines do exist [20], they are high-level [21].

experience as a group who have supported but also worked in football grounds. The authors are a mixture of non-fans and football fans and we found that this pairing of competent members of this particular culture, with those for whom the culture was alien provided useful alternate perspectives in triangulating the sense of what practical activities take place at and around football matches. We also draw upon our own experiences in and around matches, both as part of a more formal data collection mentioned above, and our own histories as fans, in order to make sense of our data, enabling us better understand, say, competence in starting songs in the crowd. Additionally, we note background use of informants and interviews with fans discussing the role of football in their everyday life.

So, in spite of emerging indications of interest in crowd interactions (e.g., [6, 24]), they can present something of a challenge for HCI, and there is little published work that analyses crowd interactions. Crowds can seem to be unlikely users of technology or alternatively present a difficult setting in which technology design may be deployed. We contradict this, however: crowds are already heavy users of technology, and making sense of the different social forms that crowds take is fundamental to understanding the diversity of modern technology use.

Thus, we employ video data presented here less as part of a systematic video corpus, but rather (using an interaction analytic, ethnomethodologically-informed approach) in order to exhibit certain phenomena found within our larger data set. From our experiences observational and otherwise, such activities as exhibited in our analysis are typical of crowds congregating around football events. In exhibiting these events we will take a practical orientation to this crowd, and pay close attention to the small details of collective football fan conduct and coordination. Just as when studying a language, one may only need individual examples of talk to elucidate our understanding as competent speakers (rather than statistical correspondences); thus our focus was on using this data to examine what we ‘already know’ from fieldwork and participation in these settings [25].

We might imagine that the considerable research focus outside of HCI upon the activities of ‘crowds’ would assist us in developing such understandings within HCI. However, often this research has been developed in terms of theoretical and abstract accounts (e.g., [22]) of the presumed hidden motivations of sports crowds (e.g., [27]), or theories of the sports crowd experience in terms of stadiums design (e.g., [1]), whilst ignoring what practical actions actually make up the recognisability of the crowd. With a few exceptions (e.g., [32, 11]), often endemic within crowd studies in general, and within many football studies in particular, is the focus upon problematic and exceptional crowds, such as crowds of unrest (e.g., hooliganism [19, 14] and outside of football). Thus, as noted by others [13], it often appears that the everyday sociality of fans, and their ‘mundane’ and routine formations have often escaped the majority of analytic attention. So, rather than develop abstract accounts and generic typifications of crowd behaviour or fan types (e.g., [12]), we wanted instead to understand more about the everyday, practical and mundane interactional ways in which crowds of fans constitute themselves at and around sporting events.

We noted that attending football matches often involves a great deal of time spent travelling to and from the stadium, as well as moments of extended ‘downtime’ like half-time, queuing, and drinking before and after a match. Although this time does not constitute the ‘main event’ it is nevertheless a fundamental component of the experience for many fans. So, in order to examine this significant aspect of ‘the football trip’ we were drawn to observing and capturing some of these moments. As such we attended and recorded part of the build-up to an international football match taking place in the UK.

UNDERSTANDING CROWDS OF FANS

Fundamental to designing for crowds is understanding what makes a crowd a crowd in the first place, how this is formed interactionally. We turn directly to a selection of video-based vignettes drawn from our corpus of data collected over the course of several football matches. These will be used to explore in detail the material practices of football fans here as they interact and participate with each other, and practically constitute what becomes see-able as a ‘crowd’.

There are numerous challenges one faces when attempting naturalistic audiovisual capture in public and semi-public spaces, however we found these challenges are often magnified for spectator sports environments, particularly so for football (e.g., achieving unobtrusive footage with good sound quality). Sequence 1: The crowd in the pub

It is match day for supporters of a national football team. We join these home fans as they converge in a local pub. There are many ‘gathering places’ in which fans will locate themselves on the day of a match, and this particular pub is no exception, in that fans will frequently meet here before home games.

Our approach has been broadly ethnographic; part of our corpus consists of observation work (and video recordings where possible) derived from attending various football matches. While not wishing to exoticise very ordinary competences, we also draw extensively upon our personal

394

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Many of the home fans in the pub are wearing blue shirts trimmed with white (the colours of the national team), along with other clothes associated with support of their team—hats, scarves, etc. It is notable that the extent of dress varies—whilst some supporters are wearing an extensive number of items relevant (i.e., seen by supporters as representative) of their support of the national football team, often involving quite extensive customisation of clothing (e.g., with badges), other supporters are only wearing a national team shirt with casual clothes, whereas others still are displaying no particular visible signs of their support other than appearing to be in visiting groups that contain fans wearing such items. Also present in the very same pub is a much smaller group of opposition supporters, made more visible by virtue of their red shirts displaying the colours of the opposing side’s national football team.

that are visible within the centre alone (due to their physical orientations). We can also see other members of the crowd performing similar actions around the same time, as well as people holding up cameras and beer glasses. Interaction within the crowd

Something that is greatly apparent initially is that we can see and hear different, and seemingly unlinked groups of supporters doing a number of things—timing their strikes of a plastic deer, coordinating their gestures at particular points in the flow of the song, organising their singing to be the right words and lines at the right time—in concert with each other. So, despite the distinct nature of these groups, they are all engaging in these activities of jumping, gesturing and so on, together. Similarly, many more fans across the crowd than just those displaying more observable bodily conduct are ‘joining in’ singing the song together. The paradox, however, is that most of these people that make up the crowd of fans will not know one another. How they are seen as being in a ‘crowd of fans’ and what are the observable and visible ways in which they do this? What are the competencies required to understand and take part in the activities constitutive of this observability?

In this excerpt from the data there appear to be various relatively distinct and separate groups of home fans, from a particularly physically and vocally active group towards the centre of the pub’s main area in front of the bar (see Figure 2, top), to more visibly subdued fans, such as those seen talking together at the edges of the crowd near a wall (see Figure 1). The environment of the pub itself is loud, with continuous talking and singing. Many present are at the moment we join them singing together a particular song that is associated with the home fans’ national team. The nature of the song is that it rises tonally through a scale for each line of the song, and then repeats. This provides natural climactic points as those singing reach the end of a verse.

In one sense the crowd here in our data consists of many different and distinct groups of individuals (e.g., friends, family) who have gathered in this particular place in order to attend the event together. In this understanding, the crowd here is potentially just clusters of these separate and unrelated groups. However, members of these separate groups orient themselves to the crowd at large beyond these immediate local groups. A common orientation to something (e.g., a match and a team), as well as an observable accountability to this, perhaps through wearing particular colours, forms one part of identifiability of the crowd of fans. That is to say, one key element of dress and behaviour is the production, for anyone observing, of being members of a larger crowd.

Many fans in the pub are singing this song together. As the song then approaches the end of a verse, the group towards the centre in particular, as well as others elsewhere in the crowd, begin jumping up and down, and raising arms. Their jumping up and down in concert, and in ‘time’ with the song is not strictly ‘in time’ but rather retains a ‘looseness’ that still is coordinated with each other around key moments. The song progresses and they, and others, cease jumping together but continue to gesture, raising arms and ‘pumping fists’, once again in approximate time to the rhythm of the song (see Figure 2, top).

Distinct groups embedded within the crowd of fans at large interact together in subtle and not-so-subtle ways. The observable and accountable crowd emerges via the practical conduct of these smaller groups. The various distinct groups and individuals within those groups conduct themselves with an orientation towards the possibility of collective participation across these groups.

Those performing these actions (particularly noticeable are those gesturing and jumping) appear to be located in observably different groups distributed spatially amongst the rest of the crowd. There are three apparently distinct groups

Thus, it is characteristic that the kinds of interactions fans engage in here are visible and audible not just within the immediate vicinity of known fellow supporters, but also at a

Figure 2: Supporters collectively ‘pumping fists’.

Figure 1: A segment of more ‘subdued’ supporters

395

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

distance. Returning to our data, and as we can see in Figure 2, arms are thrust high in the air, making the fans clearly visible within the sea of heads. Similarly, jumping inherently involves making oneself prominently visible to others via such a noticeable movement. Singing is, as we have described, ‘performed’ and designed by crowd members (perhaps mostly by those starting a song) to be clearly hearable for others within the crowd, as well as simultaneously forming an offer to others for the possibility of joining in. This orientation towards shareability is also a feature of jumping and other gestures, which furnish similar possibilities for others joining in.

This issue of appropriateness, and the sense that conduct is seemingly quite ‘performative,’ helps us relate to what has been suggested by some as the ‘carnival’ atmosphere of certain football crowds [11], in which the everyday definitions of the boundaries of appropriate behaviour are changed. Props, flamboyant attire and actions that would not be considered appropriate in the very same place with the very same people at a different point in time highlight the utility of this metaphor. Sequence 2: Deer and flags

Returning to the video recordings we can now begin to explore another important aspect of interaction we can see in the pub: the part that various objects play in collaborative action amongst fans.

We can also ask what other ‘work’ arm waving does here besides being visible for others (note that we use ‘work’ in the sense of an analytic device). We can interpret this in a number of different ways. It may be ‘about’ directing or punctuating speech or singing, perhaps for indicating some kind of defiance, or instead embodying and physically sharing emotional feelings (e.g., anger, excitement, joy) with others. There is also something inherently powerful and enjoyable about doing such actions with others at the same time. However, we must retain a perspective on what is reasonably observable here; and primarily this is the simple ways in which bodily and verbal conduct may be ‘augmented’ and thus made more public, shareable, and collaborative (in that it is possible to join in by virtue of the orientation towards hearability and visibility).

A short time later during another song, we recorded and observed an inflatable deer being bounced around between members of the crowd (see Figure 3, left). The relevance of this is related to the lyrics of the song being sung (which features a deer). The deer is bounced back and forth several times between various parts of the crowd before coming to rest. During this time various members of the crowd in different locations can also be seen jumping once again, and gesturing with arms. Later still, we observed a number of the crowd members holding up a large national flag for the opposing team (see Figure 3, right). Several opposition supporters are holding up this large flag above their heads, and moving its edges up and down. It appears that other fans, supporters of the home team, are also visible in taking part in this activity.

Actions that are produced synchronously with others, such as doing singing ‘in time’, doing jumping around the right moment, or gesturing at the same time, require a sensitivity in the production of fans’ own bodily and verbal conduct to those both immediately around them as well as those further away. Producing verbal and bodily actions that are part of collaborative conduct with others also involves ongoing mutual monitoring of one another’s actions, something that is perhaps most apparent in coordinating synchronous action. It is this basic interactional work of monitoring and producing ones actions with an orientation towards others’ own monitoring that forms a fundamental part of the ways in which the crowd constitutes its ‘crowd-ness’, and its apparent collectivity and becomes observable.

Coordination around shared objects

A crucial feature of our discussion of the bodily and verbal conduct of crowd members has been the availability of this conduct to other members of the crowd at a distance. The episodes described previously involving the inflatable deer and the flag also builds upon this point.

The importance of place

A final observation we can make about episodes like these is that the fans produce such actions in the context of the match day, and gain their relevance also from the location in which those actions are conducted. In understanding this, we can consider how the very same set of people could be present in this pub on a non-match day, however would have an entirely different character to what we see now. The sense of ‘place’ [15, 32] is constituted from a historical development of this pub as being one of many supporters’ ‘centres of gravity’ within the city that fans will congregate in before a prospective national match. It is thus appropriate and fitting for fan conduct to be played out here, just as it is the material actions of fans that also help construct the sense of place.

Figure 3: A plastic deer (circled, top) being bounced between members of the crowd (note: image has been sharpened); a flag being held by home and opposition fans (circled, bottom).

396

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Taking part in a strike of the deer involves participation between members of the crowd in distinctly different spatial places. In fact, the deer may be seen as a kind of ‘shared object’ for members of the crowd to coordinate with and around. The object itself (the deer) here does the work of enabling individuals in the crowd to interact with one another at a distance in some basic, lightweight fashion.

within earshot. As local known and unknown others join in, the volume of noise obviously increases, thus further increasing the song’s reach to the crowd at large. For successful song initiations, subsequently this ripple of others joining in begins to ‘snowball’, with larger and larger sections of the crowd joining in. Similar observations can be made about ‘Mexican waves’. The activities that make up a Mexican wave, such as holding up one’s arms, are designed to be visible in order for other members to participate, and at the same time forms the core activity of a Mexican wave (i.e., reflexivity). Participation in a Mexican wave also requires abilities of prospection for appropriate timing and synchronicity with others (i.e., seeing and projecting when the ‘right time’ is to be part of the moving body of the wave).

The flag also provides similar opportunities for those fans close to its location in amongst the crowd. It may also be thought of as a kind of ‘shared object’ that a number of fans may take part in holding together at the same time. We also have seen in the vignette how home fans located close by the flag could demonstrate their friendliness towards the opposition fans through joining in with this activity. This point regarding such ‘bridging’ interactions between groups of opposed supporters is relevant to the characteristics and expectations of being a ‘good fan’ supporting the home national team. This will be explored more fully in a later section where we will see a more fine-grained description of how interactions between such opposing groups of fans play out.

At this point we can also consider the observations of Clark and Pinch in their ethnography of the work practices of market traders [8]. In their study, the authors describe the ways in which groups of onlookers come to gather around market stalls through the strategies and techniques involved in traders’ performance of sales talk. As part of constructing an “edge” (i.e., a crowd of prospective buyers), traders rely on passers-by becoming increasingly likely to join the crowd when an existing group are already present. Thus, it becomes crucial for the trader to attempt to make at first one passer-by stop, as this then encourages others to stop, and so on.

We noticed other, less obviously shareable items also being passed around between fans; for instance, a pair of flashing glasses were exchanged between a number of fans within the same visiting group. This helps illustrate the diversity of forms of participation and interaction with and around objects—ranging from synchronous collaborative activities designed for a locally-situated group (waving a flag), to activities involving objects that are passed on between adjacent crowd members (the glasses), to objects that involve carefully timed responses between widely separated sections of the crowd (the deer).

Crowd uniformity

Having detailed the ways that members of the crowd attend to particular prominently visible and hearable activities, and join in with their own, we must also consider those not conducting themselves in such a way. In fact we may note that large portions of the crowd are not jumping, not raising arms and perhaps not singing. Alternatively, some fans are just singing, and others are talking. This leads to the rather obvious caveat in our analysis so far that crowds are not uniform or homogenous and should not analytically be treated as such.

Snowballing interactions

Fans’ orientation to collaborative observability is more clear when we begin to pick apart the practical work of singing a song together, not only with ones’ immediate group of friends but also with a crowd of fans in general. As part of our observations at and around matches we noted the ways that songs are sung in football crowds. When a fan wishes to begin a song, we observed that it is typically begun with deliberate loudness and slowness. So, at first it will often involve one fan in particular prominently and slowly singing the very first line of the song. Like the concern for visibility of gestures, in this situation involves a concern for hearability; i.e., that the first line becomes hearable as something that other crowd members can join in with. Thus, fans initiating a song craft it noticeably into ‘beginning a song’ such that it is an ‘offering’ to other members of the crowd. This offering also requires a sensitivity in terms of when it is produced; fans must be sensitive regarding the opportunities for when to start a song and when they should not.

As evidenced by the data, we can see that instead, the conduct of crowds of fans such as this have a sets of features directed both towards notional uniformity of presentation and non-uniformity of presentation. So, for example, we have seen how activities are performed and designed with an orientation towards and sensitivity to collective action that others can easily join in with. This may be, for instance, as simple as wearing specific colours (e.g., the wearing of national team strips), or items of relevance like team scarves and other significant clothes (e.g., national dress). It also may be bodily and verbal conduct, perhaps involving singing the same songs, gesturing in particular, commonly performed ways (e.g., arm raising). However, at the same time as we have seen there are many clearly distinct groups within the crowd, they may be entirely ‘disengaged’ in most of the other activities others are engaged in or in some cases be opposed to certain activities fellow

Mostly a fan’s song ‘offering’ is conducted in such a way so as to attract the attention of a local group of friends, however it also is be produced for any unassociated others

397

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

supporters engage in (see [11]). Further to this, configurations of apparel and particular customisations can be employed by fans, such as badges, hats and other props, in order to ‘stand out’ rather than as a presentation of uniformity. So, the supporters still retain an orientation towards uniformity of action and self-presentation, but the crowd’s members can choose their own level of observable adherence to this.

This sequence shows us the ways in which distinct sections of crowds of fans may interact. Although some actions may be designed to promote crowd homogeneity, e.g., gesturing together at the same time, as well as wearing, say, clothes of a particular colour, the form of ‘crowd’ we conceived of earlier in this paper, i.e., one that engages in varying amounts of collaborative action, may be internally divided in quite considerable ways.

Thus we can begin to see collaborative crowd interactions such as singing, gesturing and playing with ‘shared objects’ such as the deer or the flag, as being flexible activities. Members of the crowd can choose their own level of involvement in such collective activities; they can engage in, say, jumping in time to a song when they wish to and disengage from that when they wish to. Doing so does not make them ‘less of a fan’. This also is a feature of fan conduct at matches, as not all fans necessarily sing, jump, gesture, shout, stand up, and so on, and their level of engagement in such activities does not necessarily conflict with the notion of legitimately and actively ‘supporting’.

Here the opposition fans are faced with a ‘problem’ of sorts regarding the ways in which other fans act. The ‘problem’ is that because of their status (i.e., as supporters of the opposing team), the opposition fans are unable join in with the kind of collaborative crowd activities we have seen previously, i.e., jumping and ‘pumping fists’ in time with others’ songs associated with the home fans’ national team. The opposition are highly recognisable as opposing supporters by virtue of their different strips, their physical positions and so on. In the vignette several home supporters create a ‘bridge’ between the two crowds. In order to understand interactions such as these between groups within the crowd of fans here, we must take into account the particularities, context, history, ‘spirit’ and character of different groups of supporters. Interactions such as those seen between home and opposition fans are coloured most strongly by the home fans’ position as supporters of their national team and the normative character that support takes on: here, what is appropriate, expected, and normal behaviour is friendliness towards the opposing side and the breaking barriers (see [11] for more detail on this characteristic of this group of supporters). Committed home fans may demonstrate their own commitment through approaching the opposition in a manner of friendly rivalry. Thus, interestingly, such ‘bridging’ activities are part of what makes one a ‘good fan’ in this particular context.

However, we must temper this observation with our understandings of how the level of participation by fans is seen by others. For instance, in our observational data we found that not joining in may sometimes be seen as a technique of presenting oneself to others as having ‘been there and done it before’, i.e., that, coupled with age, reserved behaviour may be employed as a method for exhibiting experience as a fan. Crossing boundaries

Next we can look at how boundaries between very different groups within the crowds can be permeable. Here we examine a group of opposition supporters who are ‘embedded’ in amongst the larger crowd of home supporters. The group here are standing in a marked circle within this larger crowd (see Figure 4). Previously we very briefly saw how a flag featured as a shared object around which both home and opposition fans could both interact with one another around. Here we see some more detailed interaction between these opposing supporting groups.

Summary

In the sequences of interaction detailed in this paper, we have seen some of the various forms of collaborative action

As we join the action, a home supporter from the surrounding crowd approaches the opposition group, placing his right hand on the back of a member of the opposition supporters (see Figure 4), moving his arm further around the fan’s shoulder and turning to face him. Shortly after appearing to exchange a few words, the home supporter attempts to shake hands with the opposition fan. However, this particular opposition fan has a bottle in his right hand, is unable to reciprocate. Thus he raises his right hand to highlight the problem to the home fan. The home supporter, appearing to identify this problem then converts his gesture into a pat on the head of the opposition fan. Subsequent to this exchange, he then moves further into the circle of opposition fans. Following this, the opposition group is approached by two further home supporters who also begin talking with them (see Figure 4).

H O

Figure 4: A home supporter (H) approaching an opposition fan (O) (top left and right, marked); offering a hand (bottom, left); the head pat (bottom right). .

398

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

engaged in by large numbers of individuals and groups, enabling them to interact with one another in simple, elegant and often powerful ways. This was seen particularly through coordinated bodily and verbal conduct, such as the ways in which songs may be started, and bodily conduct may be timed with such songs, and thus with others. The interactions we have seen also hint at something of what it is to be a competent fan, particularly when considering how fans start songs (i.e., knowing when and how to conduct oneself).

in our analysis, but instead is produced by members’ conduct. For an audience this might be mutual orientation to a common subject (e.g., a performer), whereas for the crowds here there were a range of actual devices used—common dress or carrying or wearing particular objects (hats, flags). The synchronisation of behaviour is also important— jumping at the same time, walking in the same direction (marching) or common movements (waves, hand movements). Lastly there is a range of aural devices used to present membership in the crowd, such as singing, shouting, yelling, etc. The importance and centrality of synchronising activities—bodily or verbal—with one another cannot be underestimated. We have seen, for instance, disparate crowd members who are otherwise unassociated jumping and gesturing and singing in synchrony. This synchronicity and sensitivity towards timing conduct that is relevant for the moment is constitutive of the character of the crowd as it is presented to its own members as well as bystanders.

We have also seen the introduction of shared objects to crowd groupings, and that such objects may come to establish (fleeting and ‘lightweight’) connections between unrelated members of the crowd. In this way, shared objects offer the potential for fans to interact with one another in a very simple way. It was noted in our analysis that the crowd here is also not a homogenous, uniform entity, although it is often oriented to and designed as such by participants in their shared actions and attire. We also saw how engagement in these kinds of activities may also be highly flexible in terms of participation. The vignettes further illustrated the textured makeup of the crowd of fans, through seeing how opposing crowds may meet as in our example of interaction across crowd boundaries with fans’ ‘bridging’ interactions.

One key problem with designing technology to interact with crowds is how to not disturb this engagement. Most forms of technology are focused on interaction between a single user or a small group and a system. Even those systems that involve large displays usually rely on interaction with a small group (e.g. [21]). Yet this interaction scheme may tend to differentiate individuals from the crowd as a whole. Indeed, a crowd as a unit can come to be cast very quickly as audience or spectators [23]. This can create a ‘barrier’ within the crowd.

DISCUSSION

It is clear that crowded settings are a challenging environment in which to support interaction—the analysis here is beginning to uncover some ways in which technology could be threaded with interaction within crowds without disturbing or conflicting with crowds’ ordinary interactions and behaviour. We note that in many ways this paper is a continuation from existing HCI work on audiences (e.g., [23]), and bears a relation to CSCW topics such as the maintenance of mutual awareness and monitoring between people in workplaces (e.g., [16]).

What we find promising is ways of interacting with crowds that allow them to still be crowds, and produce that status simultaneously with any technological interactions. For these reasons technologies that focus on common behaviour are promising in contrast to systems where a small group or an individual is differentiated. One simple example is crowd cheering meters [2] which allow a crowd to synchronise and orient to a group activity together. Systems that involve a collective interacting together—perhaps on their own phones—and make this visible (say holding a phone up) are another simple example (see the [7] for an early example of this).

We will focus on four points here. Firstly, we discuss how technology can support ‘crowds being crowds’. That is, how technology can interact with users without causing them to no longer be seen as part of the crowd, or disturb the very behaviours which produce crowd membership in the first place. Secondly, we discuss ‘intra-crowd communication’—how technology can support interaction between members of crowd. Thirdly, we discuss how technologies can support interaction between crowds and those on the periphery or outside the crowd. This can be one important part of differentiating an ‘angry’ crowd from a friendly one. Lastly, we discuss broadly the potential for understanding crowds as the unit of design for interaction. This involves a break with seeing interaction in terms of a user, or an individual as part of a group, and instead interaction with the group itself as a meaningful unit.

This said, we do not wish to overplay the importance of supporting the homogeneity of crowd behaviour. Fans do wear similar colours and similar clothes, they sing the same songs together, and they make the same gestures together. However, as we have noted, crowds are not homogenous or uniform (since they are experienced as a multilayered complex of distinct groups that fans attend to), and participants may use complex configurations of dress and indeed their own participation as a method for ‘standing out’ from the potential uniformity of a crowd of fans. A fundamental aspect of this is the varying and flexible levels of engagement that fans as participants may involve themselves in. This raises the challenge of designing for flexible engagement for members of the crowd, who may have very different perspectives upon the amount they wish to ‘stand out’

Supporting crowds being crowds

One of the first key findings is that membership in a crowd is not necessarily something that we can taken for granted

399

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

or ‘blend in’. If we designed a system for members of crowds to interact with one another—i.e., acting as a shared object, we must consider the issues of crowds-withincrowds and the possibilities for ‘bridging’ between these groupings within crowds. This implies some support for multiple overlapping concerns, such as the ability to attend to one group at one time, and another group at a another time. Applications for crowd interactions should not rely on simplistic models of homogeneity or uniformity or aggregation (which is obviously an easier model to design infrastructure for, [7] being a well-known example). Instead we suggest systems designers give crowd members a choice or flexibility as to the scale of crowd they consider themselves part of, and offer them tools to manage the way they present themselves to the complex ecology of the crowd. We should consider the various sub- and super-crowds that people attend to at different points (e.g., at one point as a fan of the national team, at another attending to a group of friends).

weight and fleeting interactions. Another variety of digitally-shared data objects could rely instead on spatial proximity, as found in the flag example; digital objects in this example would rely on coordination between proximal fans. In particular peer-to-peer sharing could support the sharing of digital objects that serve a more individual form of passing on (one-to-one), as seen in our example of the flashing glasses. Another interactional form that exists in crowds is visible in our discussion of how songs or chants start. We witnessed the ‘snowball’ of different songs with one individual starting the interaction and it spreading to others. Many crowd songs are actually designed to support this—with a distinctive short first section which might be sung solo, followed by the participation of others. This sort of interaction in a crowd takes place with gradual accumulation; participation of each individual is optional, but routinely grows as a group synchronises themselves. Another example is how an audience might start to clap [17].

However, we must also potentially be wary of divides in crowds. As we have noted, there is a sense in which gestures, for example, are not produced as performative acts for an audience, but rather as both being about sharing with others in some synchronous activity as well as at the same time forming offers of participation for the crowd at large. Members of the crowd are typically more concerned with enabling participation than performing to one another, although we would not deny that elements of performancelike behaviour are indeed crucial to the conviviality of the crowd setting.

Supporting this sort of growth of action is difficult as it moves away from both a homogenous view of crowd state, but also of individual commands. Technically, voting interfaces go some way here, but without perhaps the simple dynamism of a group cheer or song. Interaction between crowds and peripheral participation

As we discussed above participation in a crowd has a certain fluidity and this is particularly so for those on the fringes of a crowd who either may act as spectators, partial participants or who may move between these roles. The relationship between non-crowd members and members can be key to the positioning of a crowd as ‘friendly’ or not— e.g., the particular efforts of the football fans in our paper to engage and if not incorporate, at least interact in a lighthearted manner with fringe crowd groups. These interactions might fail, of course, and interactions in even the most friendly crowds (carnivals, political marches, parades) can be fraught. Crowds are frequently policed when in contrast many crowded spaces (shopping streets) are usually not. One concern for technology then is in supporting interactions between the crowd and others, perhaps simply through explaining what a given crowd is and why it is together, or allowing communication across the boundary.

Intra-crowd interaction

Following from these observations one promising route is to seek to support intra-crowd interaction. As we saw from the fieldwork there are a range of different ways in which a crowd interacts with each other, and while the interactions between strangers may be on the whole very lightweight and almost trivial, they are key in helping the crowd gain a sense of common purpose, but also in preventing ‘trouble’ both interactional and more broadly. In our analysis shared objects like the plastic deer and the flag offered forms of interaction between members of the crowd—the deer enabled lightweight interactions between crowd members who were spatially distant as well as close by, whereas the flag enabled only those close by to take part in holding it. We also examined other objects that had to be passed around from individual to individual (such as the glasses).

It can be important to consider ways in which crowds might bring others on the periphery into the crowd itself. More broadly, these sort of transitions take place both within and across the crowd. A crowd may well consist of large numbers of participants engaging in collective activities designed for crowd participation, and yet at another moment engaging in very localised activities within their immediate group of fellow fans. Thus the address of any system [3] might at one moment be concentrated locally and at another to the crowd at large, and, perhaps also to distributed locations linked by digital communications technology. Further, the way in which interaction is done in these settings offers some purchase to ‘viral’ metaphors—and therefore suggests

Supporting such a variety of ‘shared objects’ seems an interesting research direction. It may be that these could be data objects inspired by this form of ‘moving’ interaction are not tied to any individually-held device, such as software components that ‘bounce’ between mobile phones belonging to spatially (and socially) distant members of the crowd. Such shared objects would not necessarily be ‘owned’ by anyone, but instead support extremely light-

400

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

that technologically mediated interaction may be supported by existing epidemic distribution techniques (e.g., [30]).

Mexican wave—participants carefully watch other crowd members’ actions, and use them to time their own participation, yet at the same time those selfsame actions that participants engage in also provide a resource for further others conducting their actions, much as with collective singing of songs.

A related issue concerns the possibility of supporting crowd interaction at a distance, in our observation and interview work, we have also noted the importance of considering the synchronicity and ‘quality’ of messages between groups of fans in distributed locations. For instance, in the common situation whereby some fans are unable to attend a match with their fellow supporters, updates about scores and other incidents on-pitch from those that are actually present (e.g., via SMS on a mobile device) obtain an important quality even though the information can be obtained from various forms of media (e.g., websites, newspapers, TV). With this in mind, we can also ask how this ‘quality’ of any synchronised but distributed group action will be affected by delay due to transport latencies (e.g., seconds, minutes, etc.)? Further, is the synchronisation method used, and is mitigation of the latency levels ‘good enough’? This obviously becomes a real challenge when considering network latencies and other issues associated with mobile systems, particularly when attempting to support shared expression across sites where this issue of quality of interaction may take on some relationship to the communications infrastructure it is conveyed across.

It is with this sort of real time behaviour in public settings that we can begin to see how the roles of participant and audience ‘snowball’ and diminish, fluidly and rapidly changing. Designing to exploit these sort of exchanges is a challenge in that any processing would have to be sensitive to both the ever changing crowd interactional space, but also that delays can radically change the meaning of an interaction. CONCLUSION

This paper has presented an empirical analysis of the material practices of crowds of fans attending a football event. Through exploring the observable ways in which they conduct themselves we have uncovered a number of potential sensitivities that designers could take into account when deploying technology in these new and exciting situations. Every crowd environment is different, however, and designers must carefully select the concepts they draw from this paper when applying it to their own design. Future work will seek to compare these sensitivities with other crowd types (e.g., music fans). Furthermore, although this paper has attempted to give balance to a discussion that is often concerned with extreme crowd behaviour, we emphasise the importance of considering the ethics of any design for crowds.

Conceptual contributions

In contrast with earlier work on participation frameworks, in this work we have attempted to more radically move away from the individual as the unit of analysis the user. This necessitates for HCI a serious challenge. Despite the focus on groups, or technology as a more broad aspect of our environment, often our interaction designs are still focused on a single user at a time, per interface, or at the very most a small set of users. Moving to considering how a system would interact with a crowd as a crowd is a challenging one that we have only started at in this paper.

Our main contribution is broadening recent interest in spectator experience, audience and performance-like situations to expand beyond audience-performer relationships to more radically participatory settings such as the football crowd, offering designers new insights into how practical action by fans may both direct how design can be done in these spaces as well as offer new possibilities.

Taking inspiration from our data, interaction between crowds and systems could perhaps not take the form of the deterministic single issue of commands. We take the model of the singing of songs as a surprisingly useful one. A single interactant collects others through their singing to produce synchronised group activity with individuals choosing freely to participate.

In concluding, we have identified and highlighted a number of key design issues: supporting the flexible participation status of crowd members that is sensitive to local and global groupings; supporting self-presentation and the subtleties of ‘standing out’, perhaps using customisation, and ‘blending in’ as part of a larger uniform whole; supporting synchronous, spatially distributed activity that provides offers of participation but not necessarily explicit ‘performative’ features; supporting interactions between subgroups within crowds but noting the sensitivity of address—i.e., some interactions may be appropriate at particular times and between particular groups; and finally, supporting shared objects and artefacts that may be offer the possibility for rapid and improvised collaboration between different spatial arrangements of fans (e.g., only those close by, or only those separated in space).

It may be that interfaces need to engage with user action more probabilistically than in terms of definite distinct commands. So, for example, a system might allow multiple choices to be made, with screen space dictated by the proportion of those who have chosen different options. Or we might consider majority rule as a guiding principle although supporting participation is a challenge. Going beyond the individual in this way is in some ways a more radical step than has been attempted in CSCW. A related point is the nature of time in crowd interaction and the importance of behaviour that is tightly coupled to what is happening at that point in time. For example, consider the detail of the feedback involved in taking part in a

ACKNOWLEDGMENTS

This research was funded by UK EPSRC (EP/E04848X/1).

401

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

REFERENCES

1. Bale, J. The spatial development of the modern stadium. International Review for the Sociology of Sport, 28(2-3), pp. 121-133, 1993.

17. Heritage, J. and Greatbatch, D. Generating applause: A study of rhetoric and response at party political conferences. The American Journal of Sociology, 92(1):110157, 1986.

2. Barkhuus, L. and Jørgensen, T. Engaging the crowd: studies of audience-performer interaction. In CHI ‘08 Extended Abstracts. ACM Press, 2925-2930, 2008.

18. Hindmarsh, J. et al. Creating assemblies: Aboard the ghost ship. In Proc. CSCW, pp. 156-165. ACM Press, 2002.

3. Bellotti, V. et al. Making sense of sensing systems: Five questions for designers and researchers. In Proc. CHI, pp. 415-422. ACM Press, 2002.

19. Hughson, J. Among the Thugs: The ‘New Ethnographies’ of Football Supporting Subcultures. International Review for the Sociology of Sport, 33(1):43-57, 1998.

4. Benford, S. et al. The frame of the game: Blurring the boundary between fiction and reality in mobile experiences. In Proc. CHI, pp. 427-436. ACM Press, 2006.

20. Maynes-Aminzade, D., Pausch, R. and Seitz, S. Techniques for interactive audience participation. In Proc. ICMI, 2002.

5. Bentley, R. et al. Ethnographically-informed systems design for air traffic control. In Proc. CSCW, pp. 123129. ACM Press, 1992.

21. O’Hara, K., Glancy, M., and Robertshaw, S. Understanding collective play in an urban screen game. In Proc. CSCW, pp. 67-76. ACM Press, 2008.

6. Brown, B. et al. Crowd computer interaction. In Proc. CHI (Extended Abstracts), pp. 4755-4758. ACM Press, 2009.

22. Redhead, S. Those Absent From the Stadium are Always Right: Accelerated Culture, Sport Media, and Theory at the Speed of Light. Journal of Sport and Social Issues, 31(3):226-241, 2007.

7. Carpenter, L. Cinematrix, video imaging method and apparatus for audience participation. US Patent, Nos. 5210604 (1993), 5365266 (1994).

23. Reeves, S. et al. Designing the spectator experience. In Proc. CHI, pp. 741-750. ACM Press, 2005.

8. Clark, C. and Pinch, T. The Hard Sell. HarperCollins, 1995.

24. Reeves, S., Sherwood, S., Juhlin, O. and O’Hara, K. Designing for Crowds. Workshop at Pervasive, Helsinki, Finland. 2010.

9. Crabtree, A. et al. Orchestrating a mixed reality game ‘on the ground’. In Proc. CHI, pp. 391-398. ACM Press, 2004.

25.Sacks, H. Lectures on Conversation. Blackwell, 1995.

10. Gates, C., Subramanian, S., and Gutwin, C. DJs’ perspectives on interaction and awareness in nightclubs. In Proc. DIS. ACM Press, 2006.

26. Schnädelbach, H. et al. Performing thrill: designing telemetry systems and spectator interfaces for amusement rides. In Proc. CHI, pp. 1167-1176. ACM Press, 2008.

11. Giulianotti, R. Football and the Politics of Carnival: An Ethnographic Study of Scottish Fans in Sweden. International Review for the Sociology of Sport, 30(2):191220, 1995.

27. Segrave, J. O. Sport as escape. Journal of Sport and Social Issues, 24(1):61-77, 2000. 28. Sheridan, J. et al. Understanding interaction in ubiquitous guerrilla performances in playful arenas. In Proc. British HCI, 2004.

12. Giulianotti, R. Supporters, Followers, Fans, and Flaneurs: A Taxonomy of Spectator Identities in Football. Journal of Sport and Social Issues, 26(1):25-46, 2002.

29. Snibbe, S. S. and Raffle, H. S. Social immersive media: pursuing best practices for multi-user interactive camera/projector exhibits. In Proc. CHI. ACM, pp. 14471456. ACM Press, 2009.

13. Giulianotti, R. The Sociability of Sport: Scotland Football Supporters as Interpreted through the Sociology of Georg Simmel. International Review for the Sociology of Sport, 40:289-306, 2005.

30. Tennent, P. Hall, M., Brown, B., Chalmers, M., and Sherwood, S. Three applications for mobile epidemic algorithms. In Proc MobileHCI ‘05, vol. 111. ACM Press, 223-226, 2005.

14. Giulianotti, R. and Armstrong, G. Avenues of contestation. Football hooligans running and ruling urban spaces. Social Anthropology, 10:211-238, 2002.

31. Tomitsch, M., Aigner, W., and Grechenig, T. A concept to support seamless spectator participation in sports events based on wearable motion sensors. Pervasive Computing and Applications, pp. 209-214, 2007.

15. Harrison, S. and Dourish, P. Re-place-ing space: the roles of place and space in collaborative systems. In Proc. CSCW, pp. 67-76. ACM Press, 1996.

32. Weed, M. The story of an ethnography: The experience of watching the 2002 world cup in the pub. Soccer & Society, 7(1):76-95, 2006.

16. Heath C. and Luff, P. K. Collaboration and control: Crisis management and multimedia technology in London Underground line control rooms. JCSCW, 1(1-2):69-94, 1992.

402

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Mobile Interaction with Real-Time Geospatial Data by Pointing Through Transparent Earth Erika Reponen Nokia Research Center P.O. Box 1000, 33721 Tampere, Finland [email protected]

Jaakko Keränen Nokia Research Center P.O. Box 1000, 33721 Tampere, Finland [email protected]

ABSTRACT

Traditional AR views are restricted to the visible surroundings around the user: an AR view augments the reality that is accessible via a camera with additional virtual information about the objects in the view [16]. The challenge is that if the user is interested in locations that are too far to be seen through the camera, there is no intuitive way to present such information as it would be confusingly overlaid on top of the objects seen in the viewfinder. Those distant targets may be just behind a building next to the user, but most interestingly even below beyond the horizon. In the latter case, the user should point towards the surface of the Earth to actually point directly at the target. This notion of pointing towards the surface of the Earth significantly differs from a traditional compass direction based mental model, where the Earth is essentially considered two-dimensional.

We present the user experience study results of a novel interaction concept that enables viewing and accessing geospatial data from all around the Earth, by pointing with a mobile device directly towards any physical location. We explain the relationship of this concept to traditional augmented reality and map based user interfaces, and we describe a prototype of the concept. In the evaluation we found that whole body interaction is a good way to browse geospatial content. The need for real-time information arises when using the concept. Reliable and detailed information is expected. Examining the Earth became interesting with the prototype. We also discuss the challenges faced in the prototype and suggest ways to tackle them. Author Keywords

First person view, geospatial data, location based data, augmented reality, embodied interaction, reality-based interaction

Global thinking is becoming more commonplace. For example, social media services such as Facebook [3] and Twitter [19] enable following events occurring all around the Earth, any time of the day. Google Earth [7] has already introduced a new, continually updated globe: a virtual alternative to the physical globes found on office desktops and schoolroom tables. Real-time video services, such as Qik [15], enable seeing videos from anywhere, any time. Nevertheless, faraway locations remain in people’s minds only as spots on 2D and 3D maps, and not as exact physical, real life locations. Our concept introduces a completely new mobile user interface that aims to correct this [18].

ACM Classification Keywords

H.5.2 [Information Interfaces and Presentation]: User Interfaces. INTRODUCTION

It has become common to tag data such as photos to physical locations. Photos and other data can be visualized on top of maps at the corresponding locations, based on the location metadata [4, 7]. Also locations of people, points of interest, or details of any pointed object may have location metadata and can be shown on a map. Another way to utilize location metadata is seen in the augmented reality (AR) views: the content is shown in the view so that it matches the visible physical location when seen through the view finder [6].

The paper is structured as follows. First, we present the related research in relevant areas, followed by an introduction to the MAA interaction concept. In the next section, we describe a prototype of the concept and the execution of a user study conducted with the prototype. The findings and their relevance is discussed in the following section. We conclude the paper with a summary of the findings and our future research directions, and at the end we list the references.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

RELATED WORK

Although there is a lot of existing work that is closely related to ours in the field of mixed reality, we have not

403

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

encountered research on this concept of seeing content through the Earth before. Research on reality-based interaction, augmented reality, spatial memory, whole body interaction, spatial interaction and live-video communication are related to our work and we will present some examples in those areas in this section to frame our research area. Jacob et al [9] have proposed the notion of reality-based interaction (RBI) as a unifying concept that ties together a large subset of emerging interaction styles. We consider our research to be part of RBI, positioned between AR and map based interaction. There is a lot of research on AR, Rekimoto et al. being one of the firsts to talk about an augmented interaction style which focuses on human-real world interaction and not just human-computer interaction [16]. Also philosophers like John Baudrillard and media artists such as Rebecca Allen and Myron W. Kryeger have been presenting various ideas related to mixing physical and virtual reality [1, 2, 11]. One specific related example concept is Nokia Point and Find [6, 13] which is a system that enables getting information about pointed objects through an AR view. Another example of a novel user interface, presenting the spatial relationships of the content is the Nokia Image Space in which by exploiting a sensor set of camera sensor, GPS, accelerometer and magnetometer the system automatically structures and creates a spatial presentation of the content [12, 20]. Jorge et al. use a peephole metaphor in their research about whole-body orientation in virtual reality interaction [10]. This mixed reality and AR research is interesting for us, because we think that our concept enables a kind of long distance AR view, combined to map data which is presented differently from traditional maps.

Figure 1. Concept illustration about standing on top of the spherical planet and seeing through the Earth by pointing with the device.

world, as seen through the Earth (examples in Figures 1, 2, 4). One analogy that helps in understanding the concept is that the device acts as a special lens allowing looking directly through the surface of the Earth all the way to the other side, as if the planet was a hollow sphere. Another way of thinking about the concept is having x-ray vision through the Earth. While traditional AR deals with things that are visible from the user’s current location, MAA is about the longer range of looking at things that are hundreds or thousands of kilometers away.

Real-time video connections between places out of viewing distance has been researched first by Kit & Sherrie who arranged a media art experiment “Hole in Space” [5] which is a live video connection between NY and LA. People's reactions towards seeing in real-time to the other side of the continent were surprised and excited. Newer research on the real-time video communication tells that also mobile phone videos are used for viewing any place on Earth in real-time [17]. Because real-timeness is a major aspect of our work we find the research in this area relevant to us.

The MAA user interface is based on a 3D model of the planet drawn on the display of a mobile device so that it matches with the physical reality around the user. The main idea in the concept is that by pointing with the device to any direction around him or her, the user is able to see which physical locations are in those directions, seen in firstperson view (Figure 2). In addition to geographical information, any geospatial content located in the shown places can be visible. The content can be for example photos, live videos, weather information, routes, Wikipedia articles, and people. This variety of content allows a MAA application running on a mobile device to act as a gateway to other content viewers, e.g., a web browser or a 2D navigation map.

CONCEPT

The concept we present here is a novel, embodied interaction method for browsing any geospatial content in an entertaining and captivating way. We call our concept “MAA”, which is a Finnish word meaning the Earth and as a longer term, in English, we use “Mobile Augmented Awareness”. MAA enables seeing information about a pointed location in a first-person view even if said location is not in the viewing distance of the device's camera. We mean not only showing what is behind any nearby obstacles but most importantly by turning the pointing direction downwards reaching locations on the other side of the

Geotagged content is commonly accessed via a 2D or 3D map interface for instance in a web browser. This allows the user to explore content positioned on a map. However, this type of an interface has a drawback: while the tagged locations exist in the real physical world, the user only gets an indirect sense of where the content is located in relation to her, as she has to think in terms of the map and not of the

404

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 2. Picture series of a transition form (1) seeing the surface of the Earth in the view finder image to (2) seeing the pointed locations through a transparent Earth to (3) seeing the opened geospatial content from the pointed location on the other side of the Earth.

physical space around her. In contrast, MAA allows the user to get a direct sense of the physical direction where the content and physical locations are in relation to her current surroundings, bridging the gap between the virtual geotagged content and the physical world.

represented by icons and labels but overlaid weather maps and different coloring. USER STUDY

Our research hypothesis is that MAA makes the user more aware of being part of the planet and that there are new interaction opportunities resulting from that. This hypothesis derives from the ability to interact with locations and content that is normally invisible to the user due to being too far away. Our concept allows the user to understand how this distant information relates to the physical reality around him or her, bringing new possibilities for mobile interaction with geospatial content by making it more entertaining and understandable.

We view MAA as an alternative for existing map based solutions for browsing and accessing geotagged content. The main difference is in the viewpoint: map based solutions work primarily in the third person, allowing the user to see a depiction of the world from space. MAA works in the first person, providing the user a more personal experience and a sense of what the planet is like in reality. The “Mobile Augmented Awareness” name is derived from the idea that this concept helps the user to understand that the Earth is physically below his or her feet, and it is not just a 2D map or a 3D globe one can rotate with a flick of a finger. We initially developed the concept because we felt that the interaction technique of pointing all the way through the Earth was underutilized in existing augmented reality user interfaces. As the concept of interaction with things directly through the Earth is foreign to most people, we emphasize entertaining and captivating aspects in the design. This makes it easier for the user to approach the concept. However, it is interesting to note that the idea of thinking about what is on the other side of the Earth is most commonly heard when talking with children: when they learn that we live on top of a spherical planet, it usually raises the question of where you would end up if you made a long enough tunnel directly downward into the ground. Older people appear to have given up thinking about such notions.

We conducted a user study to test the hypothesis and to collect feedback and first impressions of the user experience. We were also interested in studying the participants’ reactions to the mental model of a transparent Earth. This was an initial usability evaluation with an early prototype to determine the general feasibility of the concept of seeing through the earth to have access to geotagged content. The nature of the collected results was qualitative. Prototype

For a study in this early phase of research, we developed a prototype of the core aspects of the concept: a mobile device with a large screen showing a view through the Earth in the direction that the user is pointing the device at (Figure 4). The view gets updated in real time as the user changes the pointing direction. In the prototype the user’s location was fixed to his or her actual physical location on the planet. During the development of the prototype we consulted colleagues from other research teams in relevant areas such as augmented reality and photo sharing, and improved the design and implementation according to their expert opinions.

To make the user interface more entertaining and captivating, we utilize attractive visualizations combined with real-time aspects and animations. The goal is to give the user the experience of seeing the world as it exists in reality right at this moment. To avoid the overuse of icons to represent the shown content, we use a mix of different visualizations. For example, weather information and night/day variation (i.e., local time of day) are not

The prototype user interface was built out of three layers: Earth, content, and UI (Figure 5). The Earth layer showed a view of the back surface of the planet, of the areas that normally are below the horizon. The content layer was populated with photograph thumbnails and text labels

405

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Figure 3. The prototype user interface.

Figure 4. Prototype in use. Locations behind the horizon are accessible by using the device.

showing the locations of selected cities (e.g., New York, Sidney). On the UI layer there were touchscreen buttons for zooming the view in and out. A third button was reserved for toggling the overlaid content layer on and off.

degrees. The FOV angles of the maximum and minimum zoom levels were 20 and 100 degrees, respectively. All the content used in the prototype was stored locally in device memory; no real online services were used. The content comprised 22 city names and coordinates, and 144 photograph thumbnails placed in randomly chosen locations all around the planet. Half of the photographs had a daytime theme and the other half was taken in night time. This allowed us to show daytime photographs on the daytime side of the Earth and night time photographs on the other side, contributing to the real-time effect.

Oceans and continents as well as the polar regions were all shown in different colors. Water in general was transparent, so that the sky on the other side of the planet could be seen through them. The day and night sides of the planet were visualized using different color schemes. The approximate direction of sunlight was calculated at the time of day according to the clock of the device. Based on this information a set of colors was applied. For regions on the day side, the colors were light green and turquoise, and on the night side they were dark violet.

The content was presented as thumbnails and text labels as shown in Figure 3. The thumbnails were placed in an 8-by6 grid of square-shaped cells so that each thumbnail would appear in the cell that was currently behind the photo’s geotagged location. During normal use all the 48 cells would never be showing a thumbnail simultaneously so that a view of the Earth would be visible at all times. When the device’s orientation changed, the content layer would be cleared to allow an unobstructed view of the Earth. Once the device was again held stationary, the content would reappear gradually, one thumbnail at a time. In this implementation, the user was not able to interact with the content to, for instance, view photographs in full screen. The grid was used for visual clarity and preventing the showing of overwhelming amounts of information at the same time; instead, it was shown over a period of time.

The 3D model of the Earth used a single texture of 2048 x 1024 texels. Consequently, very deep zooming was disallowed to avoid overly blurry views. The default zoom level had a horizontal field of view (FOV) angle of 75

The prototype was implemented as a C++ application for the Nokia N900. The user interface was rendered with OpenGL ES 2.0 to achieve a high frame rate for the view of the 3D Earth. An external sensor module (SHAKE SK7 by SAMH Engineering Services) was attached to the back cover of the N900 to provide accurate compass heading values and device tilt angles in a way that did not hinder the user from holding the device (Figure 6). Figure 5. The three overlapping layers of the prototype user interface. . From bottom to top: A) 3D model of the Earth (parts of Europe and northern Africa visible here), B) geospatial content (photograph thumbnails), and C) controls (zoom out/in).

Procedure

There were two rounds in the study, and the same 12 persons participated in both. The participants were various professionals such as user experience designers, managers

406

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

measures practical quality, hedonic quality identity, attractiveness, and hedonic quality stimulation, and was translated into Finnish for this study (the native language of the participants). The free text questionnaire included questions about the user’s feelings after using the prototype, being partly same as the ones asked during the use. The questionnaire was used for supporting the live observations and getting the answers from the users also with their own words, not only as characteristics rated with AttrakDiff. We felt this would give us more inspiration for future design directions compared to using just AttrakDiff. The second round took place two weeks after the first round. For this time the participants explored the concept for 15 minutes. In this session we wanted to collect feedback how the participants experienced the concept when seeing it for the second time and what had changed compared to the first session. No new features or modifications were done to prototype. For the second session we brought a desktop globe for the users because we noticed in the first round that some users had difficulties believing that the prototype is actually showing the Earth as it is in real life. The physical globe was put on the table next to the participants, available to be examined if needed. During the sessions, some of the first round questions were asked again, with a couple of additional personalized questions based on the participant’s answers from the first round session. For example, if the participant still felt the same way about the possible uses for the application. At the end of the session the participants filled in the same two questionnaires as in the first session.

Figure 6. The compass was attached in a way that did not hinder the user from holding the device.

and engineers working in a large technology company. The gender of the participants was balanced and the persons were equally distributed to four age groups (26-30, 31-35, 36-40, and 41-45 years old). A two-round study was selected to give the users time to familiarize themselves with the concept between the rounds, as we were aware that the concept might be difficult for the users to understand immediately due to its novelty and the conflict with prior mental models which come from the use of twodimensional maps and globe-based user interfaces which can be rotated when interacting with the content not on the current physical view distance. We were interested in hearing how the participants felt about the concept on the second round when compared to the first round.

The data was analyzed after the two rounds of the study had been completed. Session observations and questionnaire results were combined to construct an affinity wall to find out the common themes between all participants. The responses to the AttrakDiff survey were analyzed separately and were used to determine how the perception of the concept had changed between these two sessions and to find out the general feeling about the concept and the user experience of it.

The study was conducted in a usability laboratory. During the first session the participants tried out the concept for approximately 30 minutes. We gave them very little information about the concept beforehand in order to collect their initial reactions to the concept (in the invitation we only said it is about using sensors in the navigation of a mobile application). As the session progressed we gave them more information and described the concept after they had first described their experiences.

RESULTS

In this section we will present findings from the user study, focusing on mobile interaction and the user experience. The findings are grouped in themes of whole-body interaction, the nature of the user interface as a window inside the world, the new way of exploring the Earth and other novel aspects, and finally the mental model utilized in the concept.

The session was organized as a semi-structured interview while the participants were interacting with the device. Questions such as: “describe what kind of thoughts this application raises”, “what can you do with this application”, “what kind of content you would like to have in this application” and “where is New York” were asked during the interaction. At the end of the session the participants filled in a questionnaire with free text answers for the 11 questions such as: “my primary feelings after using the application are ___”, “the application made me think that the Earth is ___” and “for my friends I would tell about the application that ___”. In addition to the free text questionnaire, the users answered to an AttrakDiff (www.attrakdiff.de) survey by Hassenzahl [8]. AttrakDiff

On the whole, the experiences of the participants were positive and the results did not contradict the research hypothesis of MAA providing a new interesting and entertaining way for browsing geospatial content, and the feeling of attachment to the Earth below one’s feet. The overall AttrakDiff score was clearly positive and together with specific attractiveness scale of the survey results the concept was seen as positive in AttrakDiff. Based on the

407

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

affinity wall findings, MAA was considered an inspiring, completely new way of thinking about the Earth and an intuitive way for accessing the geospatial content.

Interest towards seeing real-time events was mentioned. Observing what is happening right now in New York and seeing what kind of weather there is currently in a place of interest were also mentioned. Some participants said that it was good that they could see in real-time what time of day it is in any place. One of the participants for example commented:

The user quotes we present are translated to English from participants' native language. Whole Body Interaction

• “This would be good in social interactions with friends, so you can be sure not to call during the wrong time of the day.”

We found that whole body interaction, i.e., using one’s own body in relation to the device and environment is a good way to browse geospatial content. Several user comments tell about the easiness and intuitiveness of this interaction technique for browsing the Earth. For example, the users commented the following:

In addition to seeing the day and night regions, many participants wished that they could see the exact local time at the pointed location. Also the difference between daytime and night-time photos was mentioned by some users as being an important, self-evident thing: if it is nighttime at a location, the photos are night-time photos. Based on those comments as well as another user's comments about wishing to see real-time videos from a pointed location, this could be seen as a potential step forward from sharing live mobile video between places [17], and two steps forward from the first virtual window media art experiment “Hole in Space” by Kit and Sherrie [5].

• “This is easier than using a compass because there is a map and names of the places.” • “This is easy because you don't need to search for places by using UI buttons.” • “You don't need to select from a list.” • “Controlling the view is intuitive.” • “You don't need a user guide for this.” • “The idea of controlling the world by turning the device is nice.”

Reliable and detailed information is expected. The participants expected the prototype to present content coming exactly from the pointed location. It was selfevident to them that the content shown should be exactly from the pointed place. There were no exact comments about the photos which seemed to be in correct places but need for reliability is clearly seen in participants’ suspicious comments such as:

The prototype forced each of the participants to stand up and turn around at some point of the study session although they were given possibility to either sit or stand whichever they liked best. While part of the users jumped up form a chair immediately and enjoyed the intuitive browsing, some of the users considered it difficult to stand up and turn around if there is no real reason for it. Many users also hoped for traditional user interface controls for navigating the model of the Earth and the content, in addition to the ones that were provided. For example, a traditional map view from above was wished for. Also the possibility of changing one’s own location (traveling to another place on Earth virtually) and seeing what the world looks like from there was asked by many participants.

• “It is misleading when the photo of Lahti ski jump hills pops up in the sea.” • “Sauna photo in the South Pole?” • “I cannot see the connection with the locations and photos because there are cake photos and such in the sea.” Sometimes it was not clear to the users where a photo thumbnail was coming from. Either the pop-up effect used when a thumbnail appears was not strong enough, or the problem was that the photos were placed randomly around the world.

Window into the Earth

The real-time aspect of MAA was seen as inspiring. Due to the direct, physical, whole body interaction as well as seeing the difference between day and night areas, like they were at the present moment in reality, the participants expected all the presented content to be real-time. Seeing the difference between day and night regions and thus seeing in real-time what time it is in any place was found fun and interesting by several of the users. Also the color coding of the user interface for day and night parts was quite well understood and liked. One user put it as follows:

In general, things that were not corresponding to the users’ expectations were immediately mentioned. People are used to services such as Google Earth which provide street level views of the presented locations. That is why participants expected also this prototype to provide close-up views of the pointed locations. They expected the zooming in to take them to the street level of the view “just like in the Google Earth.” They also expected that selecting a content thumbnail by pressing it with finger would allow them to see the real photograph or some other details about the location. Most users felt that the zoom levels implemented in the prototype were insufficient.

• “Because there seems to be sunlight, this is somehow based on reality.“ Another use commented: • “[MAA is] like a direct window, it feels intuitive.”

408

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

New Way of Exploring the Earth

We found that examining the Earth became interesting using the prototype. This new embodied interaction style makes exploring the Earth and content around it more captivating, compared to the existing solutions. One user commented:

The study participants had no prior knowledge of the concept. The participants had not thought this way about the Earth before, of seeing directly through the Earth, and they found the application novel and unprecedented. Some felt it was “brain exercise” in a positive way. Other comments about the novelty were for example:

• “[This makes] it more fun to search for information, which is currently a boring task.”

• “On the other side of the world it is night-time at the moment. You don't usually think about things like that.”

Seeing through the Earth and becoming aware of what there is and what is happening in the pointed places was considered interesting. Comments from the participants indicate that this application encourages examining the Earth in general, to discover what kind of things exist out there. Participants commented that they could learn new things about the world by using the application. Examples about nature, people, culture, and geology were mentioned. Being able to discover information serendipitously without actively searching was liked, but in addition to this, participants also would have liked to search for specific things. Serendipity of the application was commented to be good:

• “As a small kid I started to wonder about where I would end up if I made a tunnel directly below my feet.” • “I have never thought that London is in that direction.” • “This is interesting because there is no other way to do this.” • “Seeing the whole world from one point.” • “This makes the Earth more intimate and more controllable.” • “This is difficult to perceive maybe because I never earlier thought about seeing through the Earth.” Many users commented that the map is mirrored, which is true when compared to the way we normally see the surface of the Earth, from above.

• “I can find information I was not even searching for.” • “Advancing liberal education.”

• “This is in some funny way. New York and San Francisco are in reverse order.”

• “The Earth is spherical and full of interesting countries and travel destinations.”

• “Kind of a mirror image. I understand it is so, but I need to think about this a little.”

Traveling in general was mentioned many times by the participants, but it is a common use case also with map and globe based solutions, so we do not concentrate much on it here. For example, sharing travel photos and routes as well as planning the travels were mentioned.

• “But where am I?” It was not intuitive to look downwards to explore the Earth, even though that is where the Earth is physically. Typically participants first pointed to some other directions than down, starting for example by pointing forward, then turning to both sides and only then after a while turning downwards or upwards.

The application raised interest in educational and human topics such as: • “Possibly an exploration application, for seeing the flora, elephant herds, even as real-time video.” • “Good for teaching: Look, there are sloths hanging in the tree, and information tagged to that.”

The Mental Model

Between the two rounds of the study, the users had two weeks to familiarize themselves with the mental model of the concept: the transparent Earth. The participants’ first impressions of the concept, when seeing it for the first time, were excited and interested. We collected lots of comments such as “this is fun,” ”exiting,” “this is a novel idea, I have never seen this before,” “using this is easy” and “this is interesting.” Similar responses were received also during the second round of the study, which means that it was not only their first impression as the participants interest had not died down during the two weeks. The biggest excitement about the novelty was replaced with more thoughtful interest during the second round, though. Lots of potential was seen in the concept especially when seen the second time, after having had time to think about it for some time.

• “Seeing the routes of migrating birds.” • “Photos from the South Pole. Where are the penguins going?” • “Seeing if the people are happy.” Also seeing the state of the Earth was commented being interesting using the prototype. Pollution, hurricanes and other such natural phenomena were mentioned. Also learning about the different cultures was mentioned. One user wanted to see the routes of wild animals. Novelty

The concept of seeing through the Earth was found novel and as such it required getting used to. However, it was thought to provide many new possibilities.

Practical and hedonic qualities of the prototype were evaluated using an AttrakDiff survey. There was no

409

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

• “I was thinking about this new way of perceiving the world.” • “Now it gives me great pleasure, after thinking about it for two weeks to be able to put my hands on it again.” • “When you have first understood it, it is clear; there is nothing weird about it.” DISCUSSION

The expectations of real-timeness and reliability of the presented information seem to rise from the concept of spatial and whole body interaction, which are part of reality-based interaction. Realistic and truthful information was expected even though the use of the application was considered entertaining. The combination of using one’s own body in the interaction, thinking about the relation of one’s own body to the Earth, the alive UI, and the showing of day and night areas seem to emphasize the feeling and importance of real-timeness.

Figure 7. Subscale median scores from AttrakDiff. All scores are on the positive side in both rounds.

statistical difference between the two rounds in the AttrakDiff survey results (Figure 7): a Wilcoxon Signed Rank Test revealed no statistically significant change in perception of Practical Quality, Hedonic quality Stimulation, Attractiveness or Hedonic Quality Stimulation of the concept. On the whole, AttrakDiff indicates that the concept was interesting, as the Attractiveness and Stimulation values were the highest ones and as all the values were on the positive side during both rounds. Using the prototype was experienced as confusing in the beginning, but it also stimulated a new way of thinking, of what the Earth looks like in a first-person view, and in which 3D direction places lie in reality. The real-timeness of the concept adds to the attractiveness and exploring the Earth itself is interesting.

We find the results about MAA being a new way of exploring the Earth very important and think that the comments tell about how this interaction method brings the Earth closer to the user and makes it feel more intimate to him or her. In the enthusiasm of the participants we can see that MAA brings new possibilities to their minds, things that they have not been able to do with available solutions. On the whole, many of the comments were about the application being a guide to the planet Earth. Several comments about the Earth being spherical are also interesting. While this is common knowledge, it is evidently not something that people think about on a daily basis. There were some concerns about the social acceptance of the interaction method. An interesting detail is that while a user who mentioned that social pressure is a problem said that he would look silly when turning in circles pointing downward with this, he did not consider it at all problematic to turn in circles with the device pointing upward to the sky to see the stars. We consider this a clear hint of how novel the concept is and how difficult it is to think about something you are not used to. Once a concept becomes familiar, such as observing the stars through a telescope, it becomes a natural and evident thing.

There was only one person in the twelve participants who did not understand the concept correctly during the first session. However, even that participant showed real interest in it all the time. Only two participants mentioned that they had not really been thinking about the concept, the prototype, or about seeing through the Earth in general between the study rounds. We purposefully did not ask the users to think about the concept since we wanted to know if they would be voluntarily using their time to get familiar with the concept. Ten of the twelve users said they had been thinking about it, a couple saying that they had been thinking about it a lot and were waiting eagerly to come back to see and try it again. This indicates to us that choosing a two-round user study method was suitable for this kind of a novel and unfamiliar concept. Even thought the participants did not have the prototype in their possession during the two weeks, they had the mental model in their mind:

As the world is tied together with global information networks and people have the ability to travel quickly to different sides of the world, it is more relevant to be aware of what the Earth is like in reality, and to be aware of what is happening on it. MAA is not only helping in awareness of where far-away places, people, and events physically exist in relation to the user, but it also has potential to make people aware of the state of the world. Since it encourages the user to think about himself or herself as part of the Earth, it has the potential to increase awareness of the user's own responsibility of the future of our planet.

• “I was thinking about why it was so difficult for me.” • “Now after thinking about it I do understand the idea of seeing through.”

Based on the interviews and comments from the user experience study, people seem to have a preconceived notion that there is nothing interesting below our feet —

410

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

that the Earth is essentially flat — but this application made people realize the actual state of affairs and forced them to stand up and turn around physically to see the whole world. This provides a potential link to spatial memory and cognitive maps, which have been researched a lot in cognitive psychology and neuroscience. People have an innate ability to build cognitive maps of three-dimensional spaces and objects [14]. People can use the cognitive map of the environment (spatial memory) by thinking about the landmarks and their spatial relationships to reach a target location. Also non-physical things can be remembered with help of spatial memory. For instance, method of loci is a technique which was originally used by ancient Greek and Roman orators to remember long narratives. They associated story fragments to different rooms in architectural space and used the architecture as a mental model to tell the story [21].

Another challenge is in visualizing the correct curvature of the planet’s inner surface. People are used to looking at the Earth from above the ground, and thus assume this to be the case also here. Especially if the horizon and the general color scheme give the impression that the user may be looking at a view of the Earth from above, the user may have considerable difficulties in understanding that it is not the outer surface they are seeing. While a color scheme that utilizes natural colors such as blue for water and green for continents aids in recognizing locations, it also gives the sense of seeing the outer surface. Visuals cues such as grid lines or other perspective cues may be helpful. In future iterations of the prototype we intend to experiment with different improvements to the visualization. The challenge with using a first person view like in the prototype is that regions near the user within a radius of a few hundred kilometers are almost flattened to the horizon, making it difficult to recognize locations and shapes. Also, content from a relatively large region of the planet is shown in a small section in the user interface. One solution is to provide an additional third person view that allows the user the see his or her nearby locations from above, as seen from space. However, the full impact of the first person view is then lost. Alternatively, one could show content from areas near the horizon in the sky, providing partial help in getting around the flattening.

We are interested in how people see themselves as a part of the physical world, using one’s own body to interact with the environment and content mapped to that. With MAA it becomes possible to form associations with the physical space and directions around you and the content, persons, or places you are interested in but who are out of your viewing distance. For example, when you are sitting in your office in Finland, you might learn to remember that everything happening in Italy can always be seen by pointing out through your office window.

Users were interested in viewing the world from a different location than they were physically in. However, if the location is changed, the user is no longer pointing in the actual physical direction of the content. This removes part of the sense and meaningfulness of the concept, although the benefits of the whole body interaction remain.

Evaluation

A two-round study method worked well in the user experience evaluation of a novel concept, as the users had time to familiarize themselves with it. When conducting a user study of a novel concept with the first version prototype, there is an additional challenge in analyzing which results are due to the immaturity of the prototype and which are innate to the concept. In our case, the prototype UI was so simplified that we are comfortable in assessing that the results are relevant for the concept itself.

A significant challenge in the sort of interaction utilized in the concept is that the user may not be able to very accurately keep pointing at a specific direction. Especially when pressing buttons on the device or on the touch screen, the direction may inadvertently change. Zooming in exacerbates the issue. One solution would be to allow the user to lock the orientation, or increase filtering in the orientation sensors.

Challenges

One of the first issues the users confronted was the fact that when compared to normal 2D maps or 3D globes, all the continents and other regions appeared to be mirrored in the prototype user interface. This poses a challenge as the user may not easily recognize what he or she is looking at. The severity of this issue depends on the user’s location on the planet; e.g., when standing near the equator in Africa, Scandinavia would appear flipped upside down, and the North and South Poles would be located opposite to each other about halfway down from the horizon. However, from northern Finland the South Pole is almost directly below the user, as expected, and the majority of the visible regions appear the right way up. While this may cause initial confusion, it is nevertheless a more accurate first person view of the planet than any 2D map projection or 3D globe could give. The challenge is in communicating this difference in the mental model to the user.

CONCLUSION

We have presented user experience study results of a novel interaction concept which provides world-wide access to geospatial data by pointing through the Earth. The concept enables seeing information about a pointed location in a first-person view even if the location is not in the viewing distance of the device's camera. Using a prototype of the concept, we conducted a user experience study with 12 participants. Findings from the study tell that the concept and prototype were seen as novel and interesting. There are advantages in using whole body interaction in browsing geospatial data. Participants mentioned using the prototype was easy and intuitive because there was no need to use buttons or lists to access

411

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

geospatial information. The interaction method of pointing through the Earth was well accepted after the mental model of the concept was understood. Participants thought that the prototype as a window to the other side of the Earth. The feeling of real-timeness was derived from the different visualizations of the day and night parts of the Earth, as well as from the embodied pointing based interaction happening in a first-person view. Content was expected to be realistic and truthful and to be shown in the correct locations on the 3D model of the Earth. Findings also indicate that the concept brings up new reasons for exploring the Earth. The novel mental model of seeing through the Earth is causing suspicious reactions because it makes the world look different to the maps that users are familiar with. The participants had not been thinking that most places and people of the Earth are actually below their feet. The concept got the participants inspired by the possibilities it provides for user interaction with the geospatial content. The concept made the world feel more intimate to the participants, although getting familiar with the concept takes time. In an AttrakDiff survey, the concept was found interesting as it was rated positive in attractive and hedonic qualities.

Aware Information to Mobile Users. In Supplemental Proceedings of the UbiComp 2007. 7. Google Earth. http://earth.google.com/ (accessed January 28, 2010) 8. Hassenzahl, M., Brumester, M., Koller, F. AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. in Mensch & Computer 2003. Interaktion in Bewegung, Ziegler, J., Szwillus, G. (Eds) B.G. Teubner (2003), 187-196. 9. Jacob, R.J.K., Girouard, A., Hirshfield, L.M., Horn, M.S., Shaer, O., Solovey, E.T., and Zigelbaum, J. Reality-Based Interaction: A Framework for PostWIMP Interfaces. In Proceedings of ACM CHI 2008 Human Factors in Computing Systems Conference, ACM Press (2008), 201-210. 10. Jorge, V.A.M, Ibiapina, J.M.T, Silva, L.F.M.S., Maciel, A., and Nedel, L.P. 2009. Using Whole-Body Orientation for Virtual Reality Interaction. In Proceedings of the XIth Symposium on Virtual and Augmented Reality, SVR'09. Brazilian Computer Society (2009), 268-272. 11. Krueger, M.W. Artificial Reality 2. Addison-Wesley Professional (1991).

In the future we will develop new, more full-featured versions of the MAA prototype that access real online content services and find solutions for the challenges faced in the first prototype. In upcoming evaluations we will concentrate on more detailed user experience questions but also continue studying the interaction technique of pointing beyond the physical viewing distance.

12. Nokia Image Space. http://imagespace.nokia.com/ (accessed January 28, 2010) 13. Nokia Point and Find. http://pointandfind.nokia.com/ (accessed January 28, 2010) 14. O’Keefe, J., and Nadel, L. The Hippocampus as a Cognitive Map. Oxford University Press (1978).

ACKNOWLEDGMENTS

15. Qik. http://qik.com/ (accessed January 29, 2010).

We would like to thank Hannu Korhonen for his assistance in conducting the user study.

16. Rekimoto, J., and Nagao, K. The World through the Computer: Computer Augmented Interaction with Real World Environments. In Proceedings of UIST'95, ACM Press, USA (1995).

REFERENCES

1. Allen, R. The Aspen Movie Map (1978). http://rebeccaallen.com/v2/work/work.php?isInteract=1 &wNR=15&ord=alph&wLimit=0 (accessed January 28, 2010)

17. Reponen, E., Huuskonen, P., and Mihalic, K. Primary and Secondary Context in Mobile Video Communication. Journal of Personal and Ubiquitous Computing, Volume 12, Number 4. Springer (2008).

2. Baudrillard, J. Simulacra and Simulation. University of Michigan Press (1994).

18. Reponen, E., Keränen, J., and Korhonen, H. WorldWide Access to Geospatial Data by Pointing Through The Earth. In Proc. CHI 2010 Extended Abstracts (2010). 3895-3900.

3. Facebook. http://ww.facebook.com/ (accessed January 28, 2010) 4. Flickr. http://www.flickr.com/ (accessed January 28, 2010)

19. Twitter. http://twitter.com/ (accessed January 29, 2010).

5. Galloway, K., and Rabinowitz, S. Hole in space (1980). http://www.ecafe.com/getty/HIS/index.html (accessed January 28, 2010)

20. Uusitalo, S., Eskolin, P., and Belimpasakis, P. A solution for navigating user-generated content. In Proceedings of ISMAR 2009, ACM Press (2009).

6. Gao, J., Spasojevic, M., Jacob, M., Setlur, V., Reponen, E., Pulkkinen, M., Schloter, P., and Pulli, K. Intelligent Visual Matching for Providing Context-

21. Yates, F.A. The Art of Memory. London, Routledge and K. Paul. Xv (1966). 400.

412

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Lightweight Personal Sensemaking Tools for the Web Brendan Ryder Dept. of Computing and Mathematics Dundalk Institute of Technology Dublin Road, Dundalk, Co. Louth, Ireland [email protected]

Terry Anderson School of Computing and Mathematics University of Ulster, Newtownabbey Co. Antrim, BT37 0QB, Northern Ireland. [email protected]

ABSTRACT

burgeoning store of information more accessible but also introduced the concept of personal associative trails [3], his envisioned sensemaking structure, where an individual could create, store, annotate and share personal trails through a body of knowledge. Web hyperlinks only go some way towards realising the richness of Bush‟s associative trails. In this paper we attempt to realise software support for these trails.

Sensemaking is an ill-defined, iterative and complex activity concerned with the way people approach the process of collecting, organizing and creating representations of information. The user needs to be supported in two cognitive tasks: „representation construction‟, which involves finding an appropriate structure to aid sensemaking and „encoding‟, which is populating that structure with meaningful information. Much work has been completed in the area of encoding, but the forms of representation construction and how they can be better supported in software require further investigation.

Sensemaking is central to the „knowledge worker‟. The term signifies someone who is using and sifting data (facts) and information, often with the aim not just of making sense of it (sensemaking) at some level, but of transforming and synthesising it into knowledge. In a given context, existing knowledge is used to understand the raw pieces of data and the available information is used to transform it into knowledge. Sensemaking is the activity that translates this information into knowledge. It is a continuous and iterative process, as new knowledge may enable more to be seen and understood in the available data and information. Marshall and Shipman [4] describe this as „information triage‟, the process of sorting through relevant materials and organising them to meet the needs of the task at hand, which is often timeconstrained and requires quick assessment based on insufficient knowledge.

This paper reports on the design, implementation and evaluation of a web-based personal sensemaking tool called Coalesce. It tightly integrates search facilities with the representation construction task through the SenseMap – an innovative interactive hierarchical mechanism for displaying, structuring and storing selected information. Results from controlled experiments indicate that Coalesce enhances users‟ searching, gathering and organizing tasks when compared to a standard browser and word processor combination, but without imposing an additional cognitive load. Author Keywords

H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6): Interaction styles, Prototyping, Evaluation/methodology.

This paper focuses on personal or individual sensemaking, as introduced above. While sensemaking at the organisational / corporate level is also a wellestablished area of interest in industrial knowledge management and practice [5], our interest lies in providing support to individuals.

INTRODUCTION

MOTIVATION AND APPROACH

Personal Information Management, Personal Sensemaking, Representation construction, Tagging. ACM Classification Keywords

Sensemaking is “the cycle of pursuing, discovering, and assimilating information during which we change our conceptualization of a problem and our search strategies” [1]. With the vast increase in readily available information on the web, and the rapid transition from paper to electronic documents, providing support for sensemaking is emerging as a significant sub-area in computing research [2]. Vannevar Bush in his seminal work on the Memex – arguably the conceptual pre-cursor to the web – recognised not only the need to make our

There is an abundance of software tools to help transcribe data, information and knowledge; witness the ubiquity of word processors. However, in the 21st century there are arguably no mainstream tools that assist to any great extent with making anything akin to Bush‟s associative links. The vital sensemaking transformation, with multiple aspects of organising, synthesising, labelling, annotating, recalling, re-working, and so on, that produces knowledge, has not been embedded into any established genre of computer-supported tool. Some research-driven tools are available, but they have yet to make any real-world impact. Where they do exist they tend to be designed for advanced or expert domain users, not for everyday sensemakers. In order to ensure that sensemaking activity is supported for a much wider audience, Russell, et al. [2] state that “sensemaking tools must fit into the flow and style of a user‟s work practices.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

413 1

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

They should build on patterns of representations that are well-understood, and both fast and simple to use”. They suggest that current sensemaking tool design is too “highend, complex and sophisticated” and predominantly caters for advanced or extreme users. Van Kleek, et al. [38] also found that a lightweight note-taking tool permits rapid capture and retrieval of short notes-to-self and effectively addresses people's needs.

developed a user model of sensemaking activity. His research examined how a group of Xerox employees made sense of laser printers and the output was used in the Xerox education division. Russell found that people make sense of information about a specific topic using cyclic processes of creating a well-organised structure (representation construction) and populating that structure (encoding the structure) with information relevant to the task at hand. This process is called the „learning loop complex‟ and can be seen in Figure 1. User studies found this recurring pattern emerged with a number of domainspecific sensemaking tasks.

This paper examines how mechanisms for information representation that allow easy organisation and reorganisation can be better provided to aid generalpurpose sensemaking tasks (specifically personal). It shows how simple, fluid, hierarchical and associative tagging, in the form of what we call a „SenseMap‟, and the exploitation of existing structured data improves sensemaking activities for individuals trying to comprehend a set of web documents. A review of the current models of sensemaking shaped a prototype called Coalesce, so named to emphasise its aim of tightly integrating support for web searching and organising / personalising selected information. By introducing a representation construction mechanism that avoids overcomplexity but also supports a high degree of representational richness, we hope to provide enhanced tool support for sensemaking. The remainder of the paper is organized as follows. After discussing current work in the literature relating to sensemaking the implementation of a proof-of-concept sensemaking prototype called Coalesce is described. The user studies of the prototype are then reported. Finally conclusions and future work are presented.

Figure 1. Learning Loop Complex of Sensemaking (Source: [10]).

Subsequent work carried out suggests important revisions to Russell‟s model and theory. Qu and Furnas [9] conducted an observational study to determine how people derive aspects of structure for their representations. While Russell‟s model posits a very clear separation between representation construction (i.e. finding a structure to aid sensemaking) and the encoding activity (i.e. populating the structure), Qu and Furnas argue that representation construction is much more tightly integrated with the encoding process. Sensemakers are not just getting „bags of facts‟ from their sources of information, but organised ideas, and they may incorporate some of this organisational structure in their own sensemaking. This recognises that both the content and structure of a resource are of value. That the structure a person uses to aid sensemaking evolves over time is clearly recognised by Jones, et al. [11]. They studied how individuals organise information in support of projects. Folder structures were used to summarise content as well as to organize it. The folders are information in their own right and represent the user‟s evolving understanding of a specific project and its components.

RELATED WORK

Software supported sensemaking requires mechanisms to find (through search and/or browse), collect, organise and categorise or label information. Increasingly support also needs to be provided for collaborative sensemaking as people share their sensemaking efforts [6], [7] and [8]. Work relating to the conceptual sensemaking model and the ever increasing use of documents available on the internet is also of significance. Finding is not explored any further due to space constraints, but is assumed to be provided by current web search engines. Also, collaborative aspects will not be discussed any further as the emphasis in this work is on personal sensemaking. Sensemaking

Sensemaking serves as an “organizing structure for personally meaningful information geographies: map-like workspaces which accumulate „trails‟ of our activity, which evolve over time but remain stable enough to provide the same fluency that we have with maps of physical places” [1]. It occurs when individuals face new problems or unfamiliar situations, which frequently happens when searching for information on a specific topic. The output of the activity is ill-defined, iterative and complex, “Information retrieval, organisation and task-definition all interact in subtle ways” [9].

A number of other research projects have examined sensemaking in specific contexts. The Scholarly Ontologies project [12] and the CiteSense project [13] proposed tools that support the interpretation of academic literature, specifically literature search, selection, organisation and comprehension. Other systems that have attempted to support sensemaking activities include:

Russell [10] pioneered the work on the concept of personal sensemaking at the individual level and

414 2

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

Universal labeller [14], gBIS [15] and Sensemaker [16]. Also, the PARC research group has examined methods related to automated sensemaking that involve techniques from natural language processing, machine learning and data mining.

[25]. Subject-based classification, as the name implies, is any form of content classification that groups objects by subject. However, there are a number of ways to describe subjects: controlled vocabularies, taxonomies, thesauri, faceted classification [26] and ontologies. All of these share the major disadvantage that a non-trivial amount of expertise and effort is required to correctly apply such a classification. By contrast, a folksonomy is an internetbased information retrieval methodology consisting of collaboratively generated, open-ended labels that categorise web resources. The labels are commonly known as tags and the labelling process is called tagging or social bookmarking. The comparative success of folksonomy-based systems is due in no small part to the use of unrestricted tags. While there is little mental effort involved in generating these tags, the downside is their uncontrolled nature, so that many users may use different tags to describe the same item.

Organisation

Organisation is central to the sensemaking process and can be examined from two perspectives: (1) interface and interaction design and (2) associative technologies, that is, the underlying data models for association. Much work has been carried out to examine how we interact with information resources. Typically concepts are managed in text form [17], using a hierarchy [14], or graphically using nodes and links [18] [19]. The information resources are normally associated in some form of visual container. The Nigara prototype [20] which aids navigation by providing a map-like view of hypertext documents supports informal organisation and classification. The Instant Bookplex system [21] contains a zoomable user interface (ZUI) for navigating through a spatial representation of a document collection. This ZUI supports extended reading in the collection using semantic zooming, graphical presentation of metadata, animated transitions and an integrated reading tool. The Hunter-Gatherer project, which examined information management on the Internet, produced interesting results in terms of information collection and management [17]. It found that users normally do not want to gather the contents of entire Internet documents, but instead desire to be able to select specific sections at a very fine level of granularity. They also want to be able to add and combine the gathered material in an arbitrary and flexible manner.

Despite the limitations of tags [27], commercial systems such as del.icio.us, Flickr and CiteULike have enjoyed immense success and have well established user bases. Research prototypes such as Dogear [28] have shown that social bookmarking can be useful in the enterprise. Tagging also played an important role in Piggybank [29] and Phlat [30] where it is used to create informal metadata thus increasing the availability of machineprocessible data, which aided search, discovery and management. In an effort to “provide in situ, low-cost tagging for web content”, the SparTag.us tagging system [31] utilises an intuitive Click2Tag technique. COALESCE PROTOTYPE

The Coalesce prototype is internet-based and aims to provide a unified interface that combines searching, organising and storing, with an emphasis on supporting representation construction in the sensemaking process. Figure 2 illustrates sensemaking with the topic 'Tea'.

A number of commercial and open source tools provide varying degrees of sensemaking capabilities. Clipmarks [39] provides clipping of web content together with collaborative tagging and StumbleUpon [40] provides a recommendation engine matched to personal preferences. Also, Microsoft OneNote [41] is designed for information collection, organisation and sharing.

The Coalesce prototype was implemented as a web application using AJAX and Software as a Service (SaaS), through open Web API‟s. The prototype‟s architecture adheres to a three-tier client-server AJAX architecture.

Producing software that models human association and conceptualisation involves combining research interests from diverse areas in computer science including AI, psychology and information retrieval. A number of associative techniques are used to represent knowledge and information including relational databases, the associative data model [22], semantic networks [23], semantic web technologies like RDF, topic maps [24] and web syndication and publishing frameworks like RSS and Atom.

The client executes in a standard web browser and does not require any specialised plug-ins or add-ons. For testing and evaluation purposes Microsoft Internet Explorer 6+ was utilised. The Google Web Toolkit [32] was used to develop the software, in conjunction with mygwt (now called Ext-GWT), a third party native GWT library. GWT provides interface components, intercommunication mechanisms and support for XML parsing. The server components were deployed on a J2EE compliant Apache Geronimo 2.0 for Windows application server. The application performs its communication using REST-based web services and requests are passed through an HTTP proxy server in order to overcome the same origin policy issue that arises when building AJAX applications.

Categorisation

During the sensemaking process individuals categorise and label information resources. The labels can be viewed as metadata - „data about data‟ - which can be used for both the management and retrieval of content. There are a number of mechanisms that can be used to categorise content. They range from formal subject-based classification to informal folksonomies or tagging systems to hybrid classification (coined „collabulary‟)

Application data is persisted in Google Base [33] and information is serialised in a specialised form of XML (see SenseMap below). This could support collaborative

415 3

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

capabilities that may be added in the future. Search functionality is provided by concurrently exploiting the multiple web search API‟s currently available and include Yahoo Web and Yahoo Image, Wikipedia, Flickr and Amazon Books. (Google was not used due to the limited number of API-allowed searches permitted per day.)

Figure 3 illustrates the proposed SenseMap. The hierarchical structure is very well known and is familiar to the user, thus providing an easily understood visualisation. The hierarchy provides a rich information representation without being overly complex and allows users to naturally and easily organise concepts. It also acts as a visual reminder where users can see the relationships between the concepts and the associated data they gathered at a particular point in time. With the SenseMap, the intention has been to avoid the tendency to design high-end systems for extreme users with complex visualisations but rather aim for simplicity because most sensemaking tasks are simple [2]. The interactively modifiable nature of a SenseMap is also of key importance. People need to be able to reorganise the information they have collected in a way that makes sense to them in the context of the task they are performing. Castello and Moreno [34] examined notetaking behaviour of students and found that “reorganising, reflecting, amplifying and synthesising the notes seem to help in information re-use and consolidation quite a bit”. The SenseMap allows the user to continuously modify and save any aspect of the hierarchy as they proceed, including the number, order and depth of the notes to any of its tags and links. The labels or tags that make up the hierarchical concepts are defined iteratively by the user as they “make sense” and are created manually as they search or can be recommended using a form of semi-automatic assistance from structured data sources that are available on the Internet. As the user selects specific tags that already exist in the SenseMap, suggested terms based on those existing tags are provided. This term recommendation is provided by the Yahoo Term Extraction service [35] and provides a form of content analysis which retrieves a list of significant words or phrases extracted from a body of text in either XML and/or JSON formats. Tags can be dynamically repositioned within the hierarchical structure as required (dashed line in Figure 3).

Figure 2. Coalesce Top-level Interface.

The output from each search tool appeared in standard HTML format in corresponding tabs. Coalesce aims to support very fluid working, allowing the user to move in any order among searching, reading and developing a SenseMap. The prototype also permits the simultaneous creation of multiple SenseMaps so the user can work on several independent sensemaking projects. The SenseMap

The representation construction aspect of the sensemaking process is encapsulated in a SenseMap. This is a hierarchical tagging structure that can be used to build the associative concepts that underpin the user‟s representation construction process. It is an attempt to provide a persistent and intuitive yet easily modified associative mechanism.

Figure 3. SenseMap and Information Snippets.

Fine-grained information resources, for example a paragraph of a web page, a sentence, or an image can be

416 4

Full Papers

Proceedings: NordiCHI 2010, October 16–20, 2010

tagged and associated with a tag group in the SenseMap. The gathered components are referred to as information snippets. Each label or tag can have multiple information snippets associated with it.

associated SenseMap concepts marked up in HTML. The content element is used as a wrapper for a non-validated XML representation of the associated concepts that is escaped into appropriate HTML.

The SenseMap and associated information snippets are stored independently using the structured format of Google Base (Figure 4). The user adds the tags in the SenseMap, and the tags are automatically added to the appropriate information snippets so that the user is never exposed to any of the low-level markup code.

The Information Snippet item type is used to represent information gathered from the Internet and could be a paragraph(s), a group of sentences or words or an image. (For scoping reasons the types of data that can be gathered is restricted to text and images). It can also include a user-defined annotation.

Google Base is an online database into which a user can add any type of content in a structured way and is based on the Google Data or GData protocol. GData is a protocol designed by Google that provides a simple standard protocol for reading and writing data on the Internet. It combines common XML-based syndication formats, Atom 1.0 and RSS 2.0, with a feed-publishing system based on the Atom Publishing Protocol (APP), plus some extensions for handling queries and authentication.

Figure 6 provides an example of an information snippet from the „Tea‟ SenseMap for information labelled „Types of Tea. The