Inhabited Information Spaces: Living with your Data
David N. Snowdon Elizabeth F. Churchill Emmanuel Frécon, Editors
Springer
Computer Supported Cooperative Work
011
011
011
011
11
Springer London Berlin Heidelberg New York Hong Kong Milan Paris Tokyo
i
Also in this series Gerold Riempp Wide Area Workflow Management 3-540-7643-4
Ralph Schroeder (Ed.) The Social Life of Avatars 1-85233-461-4
Celia T. Romm and Fay Sudweeks (Eds) Doing Business Electronically 3-540-76159-4
J.H. Erik Andriessen Working with Groupware 1-85233-603-X
Fay Sudweeks and Celia T. Romm (Eds) Doing Business on the Internet 1-85233-030-9
Paul Kirschner, Chad Carr and Simon Buckingham Shum (Eds) Visualising Argumentation 1-85233-664-1
Elizabeth F. Churchill, David N. Snowdon and Alan J. Munro (Eds) Collaborative Virtual Environments 1-85233-244-1 Christine Steeples and Chris Jones (Eds) Networked Learning 1-85233-471-1 Barry Brown, Nicola Green and Richard Harper (Eds) Wireless World 1-85233-477-0 Reza Hazemi and Stephen Hailes (Eds) The Digital University – Building a Learning Community 1-85233-478-9
Christopher Lueg and Danyel Fisher (Eds) From Usenet to CoWebs 1-85233-532-7 Kristina Höök, David Benyon and Alan J. Munro (Eds) Designing Information Spaces: The Social Navigation Approach 1-85233-661-7 Bjørn Erik Munkvold Implementing Collaboration Technologies in Industry 1-85233-418-5
Elayne Coakes, Dianne Willis and Steve Clark (Eds) Knowledge Management in the SocioTechnical World 1-85233-441-X
Related Title Richard Harper (Ed.) Inside the Smart Home 1-85233-688-9 A list of out of print titles is available at the end of the book
ii
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
David N. Snowdon, Elizabeth F. Churchill and Emmanuel Frécon (Eds)
Inhabited Information Spaces Living with your Data
With 94 Figures
011
011
011
11
iii
David N. Snowdon, BSc (hons) MSc, PhD Xerox Research Centre Europe, 6 Chemin de Maupertius, 38240 Meylan, France. Elizabeth F. Churchill, BSc, MSc, PhD FX Palo Alto Laboratory Inc., 3400 Hillview Avenue, Building. 4 Paol Alto, CA94110, USA. Emmanuel Frécon, MSc Swedish Insitute for Computer Science, Interactive Collaborative Environments Laboratory, Platforms for Collaborative Environments Group, Box 1263, 164 29 Kista, Sweden. Series Editors Dan Diaper, PhD, MBCS Professor of Systems Science & Engineering, School of Design, Engineering & Computing, Bournemouth University, Talbot Campus, Fern Barrow, Poole, Dorset BH12 5BB, UK Colston Sanger School of Management, University of Surrey, Guildford, Surrey GU2 7XH, UK
British Library Cataloguing in Publication Data Inhabited information spaces : living with your data. – (Computer supported cooperative work) 1. Human-computer interaction 2. Interactive computer systems I. Snowdon, David N., 1968– II. Churchill, Elizabeth F., 1962– III. Frécon, Emmanuel 004′.019 ISBN 1852337281 Library of Congress Cataloging-in-Publication Data A catalog record of this book is available from the Library of Congress Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. CSCW ISSN 1431-1496 ISBN 1-85233-728-1 Springer-Verlag London Berlin Heidelberg Springer-Verlag is a part of Springer Science+Business Media springeronline.com © Springer-Verlag London Limited 2004 The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Typeset by Florence Production, Stoodleigh, Devon, England Printed and bound in the United States of America 34/3830-543210 Printed on acid-free paper SPIN 10910238
iv
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Foreword The Human Touch: Reflections on i3
011
011
011
011
11
The Machine-centred Mind Set At the Chicago World Fair of 1933, the official motto was: “Science Finds – Industry Applies – Man Conforms”. To many of us today this seems quite shocking, yet it has been the driving force of much development in the last century. In particular, if you look at the rise of computing over the last 50 years, you will see that, on the whole, development has been extraordinary, but fairly straightforward: it can be characterised as trying to make “faster and faster machines fit into smaller and smaller boxes”. Starting from the time of the ENIAC, one of the colossal computers of the 1940s, most IT progress has been driven from the point of view of the machine. Since then things have changed – but perhaps not really that much. Even if computers can today calculate many times over what was possible a few years ago, and the machines have become somewhat less obtrusive, much of the “mind set” has stayed the same. It is the visions of huge calculating machines spanning massive rooms, trying to recreate an absolute artificial intelligence, that still haunt much of the thinking of today. Clearly, it is difficult to shake off old mind sets.
Alternatives Alternatives to the idea of fitting computing into ever smaller boxes can mainly be attributed to Mark Weiser. In his paper, “The Computer for the 21st Century”, he outlined notions of how computing could become integrated into the fabric of everyday life by becoming completely distributed into the environment. In this way computing would become “ubiquitous”. More recently, similarly inspired work on “tangible media”, by Hiroshi Ishii has emerged from the MIT Media Lab. Apart from this, the technological revolution of GSM and the mobile phone has also had its share of making information technology come out of its “traditional shell”. Alternatives to the machine-centred view to computing were also starting around the same time, such as the “anthropocentric” ideas proposed by Mike
v
Foreword
Dertouzos at the MIT Computer Science Lab; and in a similar vein, cognitive scientist, Don Norman has been pointing out the lack of well-designed information environments. Roughly at the same time, but from a different perspective, we started thinking about how to give technology more of a “human touch”. Now, in principle, this should not be that difficult, as technology is after all, made by humans. In practice, however, one has to go quite far to break down the machine-centred and box-centred ways of thinking. We decided that the only way to attack the problem with any significance was to try to invert the picture completely – that is, to start thinking from the human point of view and work outwards. Our idea of “human centredness” was that it should nurture technological innovation but within a broader context of human values and aspirations. This was not the same as “user” driven, or “defined by user needs”, all of which tend to become stuck in improving the status quo, but not growing beyond it. At the same time, we also wanted to make sure to break out of the box-centred ways of thinking as much as possible and avoid doing “traditional HCI”, which was mainly involved in improving computers as they were. Our ideas were designed to balance questions of technically “how”, with questions of “why?” and “what for?”. And the aim was to see if we could start restoring the balance between people’s inventiveness to make new machines, with the essence of being human. Our questions became rather: How can we reach a better and more fulfilling balance between technology and people? What could be new ways of thinking about the problems? What could be the new paradigms that could lay the paths for further research and development?
The i3 Research Programme It is along these lines that we launched our first call for proposals back in 1995. Our general aim was to look at the relationship between people and information technology in the future: how could people access and use information, and exchange things with others using information technology as a medium? A clear break was needed to get out of stale thinking. Therefore, we called for new paradigms of interaction and research on new interfaces between people and the world of information. We also asked how such work could intertwine human, societal and technological elements into one dynamic research activity. One of the main quotes from our call for proposals was: The goal of i3 is to research and develop new human-centred interfaces for interacting with information, aimed at the broad population (1996).
To help define a specific research agenda, we first had a competition for more specific visions of the future. “Connected Community” and “Inhabited Information Spaces” were selected as the two visionary themes on which we based a subsequent call for research projects. Even though it took some time
vi
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Foreword
● 1 ●●●●●●●●●●●●●● to have an extra layer of calls for proposals, in retrospect it was better to “reculer pour mieux sauter”. The two selected themes had similar yet contrasting underlying philosophies. The Connected Community theme, proposed by a team headed by Irene Mac William (Philips Design, Eindhoven) and Marco Susani (Domus Academy, Milan) asked: forget about virtual environments and trying to fit people into some artificial world – how can we help people in their everyday environment, and integrate technology into this? The idea is to understand how information and communication tools start making a difference when they are embedded in a real context, and start being more meaningful for actual people and communities. How can technology enhance these environments and activities, rather than replace them? The other schema, Inhabited Information, proposed by a team headed by Tom Rodden (University of Nottingham) took a slightly different perspective. It stated: the Internet and the Web already represent a suspended reality, and people want to participate more in these spaces. Given that this is a reality, how could it evolve in the future? How could we make it more accessible to the broadest possible public, and make it socially interactive for large groups of people, in meaningful ways? And in similar spirit to the first theme, how can such environments link to the physical everyday world rather than be removed from it? At a later stage, we decided to supplement the research with an emphasis on learning. We wanted to explore new relationships between learning and technology. The idea was that a lot could be learnt about designing new interfaces by looking at how children interact, play and learn. Similar ideas had been experimented in a Lego context by Seymore Papert of the MIT Media Lab. In 1997, we decided to have a call on experimental school environments (ese). This centred around learning for very young children, in fact, the 4–8-year-old age range. This age range struck us as being particularly challenging because at this stage children don’t have too many of the adult preconceptions of the world, and are still open to new things. Young children have a different kind of “language” – a form of communication and expression from which adults can learn a lot. From this we wanted to gain insights about how to design meaningful interaction tools for the population at large. The header of our call was: The aim of i3-ese, is to research new kinds of IT-based tools designed to enable new approaches to learning, focussing on the age range of 4 to 8 (1997).
From each of these programmes we selected a number of individual research projects. Together these spanned many universities, research centres and companies across Europe, and involved a mix of people from many walks of life – artists, designers, computer scientists, game companies, technology companies, experimental schools, teachers and children, people in communities, etc. At the same time all these different outlooks were united by the common vision: of exploring new relationships between people and technology.
vii
Foreword
Grains of the Future In this book you will find some examples of work in particular from the Inhabited Information Spaces Grouping. It is interesting to see how some of these ideas are still “futuristic” and others have started to become part of mainstream thinking and made their way into products. Some people say that you can find “grains of the future” in the present today – the only problem is, where do you start to look? One of the potential advantages of this book is that by looking at the research developments stretching out into recent past, one can identify how some grains developed into trends of the present, and other are still just emerging. For those still interested in seeking out “grains of the future”, this book will be a valuable source. Jakub Wejchert Information Society DG European Commission Jakub Wejchert grew up in Ireland, with a family background of artists and architects, of Polish origin. He studied natural science at Trinity College Dublin, specialising in physics, and holds a doctorate (modelling of non-linear networks) from the same institution. Later he worked in the USA with IBM research, working on computer graphics and interface design. He joined the European Commission in 1992. At the Future and Emerging Technologies unit, he set up and managed a number of research programmes such as i3 – intelligent information interfaces; i3 – experimental school environments; and the “disappearing computer”. He now works as an advisor on vision and strategy to one of the Directors in the Information Society Programme. Jakub lives in Waterloo, south of Brussels, with his wife and three sons. The opinions expressed here are those of the author and do not necessarily reflect the position of the European Commission.
viii
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Acknowledgements
011 The editors would like to acknowledge the European i3 initiative and all the authors of the chapters in this volume for their contributions. Much of the work described in this volume would not have taken place without funding from the European Commission. We would also like to thank SICS, XRCE and FX Palo Alto Laboratory for supporting our activities within this domain. Rosie Kemp and Melanie Jackson of Springer also deserve thanks for their help and support throughout the process of preparing this book for publication.
011
011
011
11
ix
This page intentionally left blank
Contents
011
011
011
List of Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Part 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1. Inhabited Information Spaces: An Introduction Elizabeth Churchill, David Snowdon and Emmanuel Frécon . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Chapters in this Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Pure Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Mixed Reality Environments . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.4 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.5 Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 5 5 6 6 7 8 8
Part 2. Pure Virtual Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2. WWW3D and the Web Planetarium Mårten Stenius and David Snowdon . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Producing a 3D Representation of a Web Page . . . . . . . . . . . . . . 2.3 Browsing the Web Using WWW3D . . . . . . . . . . . . . . . . . . . . . . . 2.4 Improving Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 The Web Planetarium: Creating a Richer Visualisation . . . . . . . . 2.5.1 Visual Differentiation of Nodes . . . . . . . . . . . . . . . . . . . . . . 2.5.2 The Web as a Road Network . . . . . . . . . . . . . . . . . . . . . . . . 2.5.3 Hybrid Browsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
11 11 12 13 16 20 20 22 22 24
3. PlaceWorld, and the Evolution of Electronic Landscapes Steve Pettifer, Jon Cook and James Marsh . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Background: The Physical and the Abstract . . . . . . . . . . . . . . . . . 3.2.1 Watching a Cityscape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 The Distributed Legible City . . . . . . . . . . . . . . . . . . . . . . . . .
25 25 27 28 29
011
11
xi
Contents
3.3 3.4
3.5
3.6
3.2.3 Finding “Something to Do” . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Abstract Influences: Nuzzle Afar . . . . . . . . . . . . . . . . . . . . . PlaceWorld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 The Design of PlaceWorld . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 The User Interface and Presentation System . . . . . . . . . . . . Technological Challenges for Electronic Landscapes . . . . . . . . . . 3.4.1 Synchronising the Behaviour of Entities . . . . . . . . . . . . . . . 3.4.2 Distribution and Communications . . . . . . . . . . . . . . . . . . . . 3.4.3 Defining the Behaviour of Entities . . . . . . . . . . . . . . . . . . . . 3.4.4 Methods and Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 The Distribution Architecture . . . . . . . . . . . . . . . . . . . . . . . System Support for PlaceWorld . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.1 Menus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 Access Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.3 Exploiting Subjectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.4 Becoming a Place Where Places Meet . . . . . . . . . . . . . . . . . Conclusions
4. Using a Pond Metaphor for Information Visualisation and Exploration Olov Ståhl and Anders Wallberg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 The Pond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 The Pond Ecosystem Metaphor . . . . . . . . . . . . . . . . . . . . . . 4.2.2 The Pond Example Application . . . . . . . . . . . . . . . . . . . . . . 4.2.3 The Hardware Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 The Software Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 The Pond Audio Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Observations from Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Summary and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
31 33 33 34 36 37 39 40 41 43 44 46 46 46 47 48
51 51 54 54 55 56 57 58 63 64 65 68
Part 3. Mixed Reality Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 5. City: A Mixture of Old and New Media Matthew Chalmers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Ongoing and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
xii
71 71 73 77 82 86 88
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Contents
● 1 ●●●●●●●●●●●●●● 6. Soundscapes Tony Brooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 The Soundscapes System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Therapeutic Uses of Soundscapes . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Artistic Performances Based on Soundscapes . . . . . . . . . . . . . . . . 6.4.1 Interactive Painting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 The Four Senses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
89 89 89 92 94 94 96 99
011
011
011
011
7. The Computational Interplay of Physical Space and Information Space Enric Plaza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 7.2 The Interplay of Physical and Information Spaces . . . . . . . . . . . 102 7.3 A Framework for Context-aware Agents . . . . . . . . . . . . . . . . . . . 104 7.3.1 Awareness and Delivery Services . . . . . . . . . . . . . . . . . . . . 105 7.3.2 Agents Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 7.4 The COMRIS Conference Centre . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.4.1 Delivery Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 7.4.2 Awareness Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 7.4.3 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Part 4. Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8. Communicating in an IIS: Virtual Conferencing Adrian Bullock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Virtual Conferencing – a Historical Perspective: Past, Present and Future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 What Do We Mean by Virtual Conferencing? . . . . . . . . . . 8.3 Approaches to Virtual Conferencing . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Early Videoconferencing . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.2 MUDs and MOOs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.3 The Arrival of Graphics . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.4 Video Comes of Age . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.5 Graphics Come of Age . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 Using Virtual Conferencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 Understanding Collaboration . . . . . . . . . . . . . . . . . . . . . . . 8.4.2 The Importance of First Impressions . . . . . . . . . . . . . . . . . 8.4.3 Sharing Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.4 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.5 Real Versus Abstract: The Role of Video? . . . . . . . . . . . . .
115 115 116 117 117 117 118 118 120 122 124 124 125 125 125 126
11
xiii
Contents
8.5 8.6
Virtual Conferencing Versus Telephony . . . . . . . . . . . . . . . . . . . Guidelines for Using Virtual Conferencing Effectively . . . . . . . . 8.6.1 What Is the Task at Hand? . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.2 Communication Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.3 Infrastructural Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
127 129 129 130 130 131
9. Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments Mike Fraser, Jon Hindmarsh, Steve Benford and Christian Heath . . . . 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 Awareness and Co-ordination . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Anticipation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Occlusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.1 Scaleability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.2 Reciprocity of Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5.3 Unrealism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
133 133 135 137 137 140 142 145 145 146 147 149 150
8.7
10. New Ideas on Navigation and View Control Inspired by Cultural Applications Kai-Mikael Jää-Aro and John Bowers . . . . . . . . . . . . . . . . . . . . . . . . . . 151 10.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 10.1.1 Challenges for Interaction Design . . . . . . . . . . . . . . . . . . . 152 10.2 Interactive Performances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 10.2.1 Lightwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 10.2.2 Blink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 10.3 Inhabited Television . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 10.3.1 Heaven and Hell – Live . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 10.3.2 Out of This World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 10.4 Production Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 10.4.1 Finding and Framing the Action . . . . . . . . . . . . . . . . . . . . 170 10.4.2 The Round Table: A Physical Interface . . . . . . . . . . . . . . . 172 10.4.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 10.5 Discussion: Navigation, Presence and Avatars . . . . . . . . . . . . . . 176 10.5.1 Avatar-centred Navigation . . . . . . . . . . . . . . . . . . . . . . . . 176 10.5.2 Object-centred Navigation . . . . . . . . . . . . . . . . . . . . . . . . 177 10.5.3 Activity-oriented Navigation . . . . . . . . . . . . . . . . . . . . . . . .178 10.5.4 Navigation as Montage, Dispersed Avatars . . . . . . . . . . . . 178 10.5.5 Accomplishing Presence and Intelligibility . . . . . . . . . . . . 179
xiv
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
Contents
● 1 ●●●●●●●●●●●●●● 11. Presenting Activity Information in an Inhabited Information Space Wolfgang Prinz, Uta Pankoke-Babatz, Wolfgang Gräther, Tom Gross, Sabine Kolvenbach and Leonie Schäfer . . . . . . . . . . . . . . . 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Related Work and Requirements . . . . . . . . . . . . . . . . . . . . . . . . 11.3 User Involvement and Studies . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3.1 Partner Settings and Evaluation Methods . . . . . . . . . . . . . 11.3.2 Do Users Meet at all in a Shared Workspace? . . . . . . . . . 11.4 The Tower Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Personalised Overview of Activities: The Tower Portal . . . . . . . 11.6 Awareness in a Working Context: Smartmaps . . . . . . . . . . . . . . 11.7 Symbolic Actions in a Context-based 3D Environment . . . . . . . 11.7.1 The Tower World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7.2 User Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.8 DocuDrama . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.9 Ambient Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.10 Lessons Learned About Awareness . . . . . . . . . . . . . . . . . . . . . . . 11.10.1 Awareness Is Something One Is Not Aware of . . . . . . . . 11.10.2 Synchronicity of Awareness . . . . . . . . . . . . . . . . . . . . . . 11.10.3 Walking and Talking Are Means to Achieve Awareness . 11.10.4 Peripheral Awareness in Electronic Settings . . . . . . . . . . 11.10.5 Awareness Is Double-situated: The Workspace’s and the Observer’s Situation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.11 Summary and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
181 181 182 184 185 186 188 189 191 194 194 196 198 201 203 203 204 205 205 206 207
Part 5. Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 12. DIVE: A Programming Architecture for the Prototyping of IIS Emmanuel Frécon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 The Virtual World as a Common Interaction Medium . . . . . . . 12.3 Partial, Active Database Replication . . . . . . . . . . . . . . . . . . . . . . 12.4 Programming the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.1 The DIVE Programming Model . . . . . . . . . . . . . . . . . . . . 12.4.2 Programming Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.3 Building your Application . . . . . . . . . . . . . . . . . . . . . . . . . 12.5 DIVE as a Component-based Architecture . . . . . . . . . . . . . . . . . 12.5.1 System Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.2 User-oriented Components . . . . . . . . . . . . . . . . . . . . . . . . 12.5.3 The DIVE Run-time Architecture . . . . . . . . . . . . . . . . . . . 12.6 The London Demonstrator: An Example Application in More Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
211 211 212 213 215 216 216 218 223 223 224 225 226
11
xv
Contents
12.6.1 Centre of London . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6.2 Collaboration Services for Use by Groups . . . . . . . . . . . . 12.6.3 Tourist Information Data Visualisation Service . . . . . . . . 12.6.4 Real-time Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . .
228 229 229 230 231
13. Communication Infrastructures for Inhabited Information Spaces David Roberts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.2 Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.3 Avatars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.4 Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.1.5 Communication Requirements . . . . . . . . . . . . . . . . . . . . . 13.1.6 Resources: Computers and Networks . . . . . . . . . . . . . . . . 13.2 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.1 Localisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.2 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.3 Persistence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2.4 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 The DIVE Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.2 PING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.1 Point-to-point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.2 Tunnelled Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4.3 Hybrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
233 233 234 235 236 237 238 240 240 241 246 251 252 256 256 260 263 264 265 266 266
Part 6. Community . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 14. Peer-to-peer Networks and Communities Mike Robinson . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Early Inhabited Information Spaces in CSCW . . . . . . . . . . . . . . 14.2.1 Rendering the Invisible Visible . . . . . . . . . . . . . . . . . . . . . 14.2.2 ClearBoard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2.3 Feather, Scent and Shaker: Supporting Simple Intimacy . 14.2.4 Gesture Cam: The Nodding Robot . . . . . . . . . . . . . . . . . . 14.3 P2P Themes and Overall Direction . . . . . . . . . . . . . . . . . . . . . . . 14.4 Design for Community: Inhabited Information Spaces . . . . . . . 14.4.1 Communities: An Aside on Definitions . . . . . . . . . . . . . . 14.4.2 Communities: An Aside on Use . . . . . . . . . . . . . . . . . . . . 14.4.3 Communities: An Aside on Philosophy . . . . . . . . . . . . . .
xvi
271 271 274 274 275 276 277 278 281 281 282 284
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Contents
● 1 ●●●●●●●●●●●●●● 14.5 P2P, Community and the Design of Inhabited Information Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 14.6 Concluding Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 15. Inhabitant’s Uses and Reactions to Usenet Social Accounting Data Byron Burkhalter and Marc Smith . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Netscan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.4.1 Social Accounting Data and Author-assessment Threads 15.4.2 Social Accounting Data and Newsgroup-assessment Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
291 291 293 294 295 295 301 305
References
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
Index
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
011
011
011
11
xvii
This page intentionally left blank
List of Contributors
011
011
011
011
11
Steve Benford Mixed Reality Laboratory, School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham, NG8 1BB, UK
[email protected] John Bowers Department of Numerical Analysis and Computer Science, Royal Institute of Technology, SE-100 44, Stockholm, Sweden
[email protected] Tony Brooks Aalborg University, Niels Bohrs Vej 8, DK 6700 Esbjerg, Denmark
[email protected]
[email protected] Matthew Chalmers Computing Science, University of Glasgow, 17 Lilybank Gardens, Glasgow, G12 BQQ, UK
[email protected] Elizabeth Churchill FX Palo Alto Laboratory, 3400 Hillview Avenue, Building 4, Palo Alto 94304, USA
[email protected] Jon Cook Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, UK
[email protected]
Adrian Bullock Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected]
Mike Fraser Mixed Reality Laboratory, School of Computer Science, University of Nottingham, Jubilee Campus, Nottingham, NG8 1BB, UK
[email protected]
Byron Burkhalter Department of Sociology, University of California, Los Angeles, 264 Haines Hall, 375 Portola Plaza, Los Angeles, CA 90095–1551, USA
Emmanuel Frécon Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected]
xix
List of Contributors
Wolfgang Gräther Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany wolfgang.graether@fit.fraunhofer.de
James Marsh Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, UK
[email protected]
Tom Gross Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany tom.gross@fit.fraunhofer.de
Uta Pankoke-Babatz Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany uta.pankoke@fit.fraunhofer.de
Christian Heath Work, Interaction and Technology Research Group, The Management Centre, King’s College London, Franklin-Wilkins Building, London, SE1 8WA, UK
[email protected] Jon Hindmarsh Work, Interaction and Technology Research Group, King’s College London, UK
[email protected] Kai-Mikael Jää-Aro Department of Numerical Analysis and Computer Science, Royal Institute of Technology, SE-100 44, Stockholm, Sweden
[email protected] Sabine Kolvenbach Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany sabine.kolvenbach@fit.fraunhofer.de
xx
Steve Pettifer Department of Computer Science, The University of Manchester, Oxford Road, Manchester, M13 9PL, UK
[email protected] Enric Plaza IIIA, Artificial Intelligence Research Institute, CSIC, Spanish Council for Scientific Research, Campus UAB, 08193 Bellaterra, Catalonia, Spain
[email protected] Wolfgang Prinz Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany wolfgang.prinz@fit.fraunhofer.de David Roberts Department of Computer Science, University of Reading, Whiteknights, Reading, RG6 6AY, UK
[email protected]
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
List of Contributors
● 1 ●●●●●●●●●●●●●● Mike Robinson Sageforce Ltd, 61 Kings Road, Kingston-on-Thames, Surrey, KT2 5JA, UK
[email protected] Leonie Schäfer Fraunhofer Institute for Applied Information Technology, Schloss Birlinghoven, 53754 Sankt Augustin, Germany leonie.schaefer@fit.fraunhofer.de Marc Smith Microsoft Research, Microsoft Corporation, One Microsoft Way, Redmond, WA 98052, USA
[email protected] David Snowdon Xerox Research Centre Europe, 6, chemin de Maupertuis, 38240 Meylan, France
[email protected]
Olov Ståhl Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected] Mårten Stenius Alkit Communications AB, Aurorum 2, SE-977 75, Luleå, Sweden
[email protected] Anders Wallberg Swedish Institute of Computer Science, Box 1263, SE-164 29, Kista, Sweden
[email protected]
011
011
11
xxi
This page intentionally left blank
011
011
011
011
11
Part 1 Introduction
This page intentionally left blank
011
011
011
011
Inhabited Information Spaces: An Introduction
1
Elizabeth Churchill, David Snowdon and Emmanuel Frécon
1.1 Introduction Studies of co-operative work have shown that, even when co-operation is not explicit, a surprisingly large amount of work relies on the knowledge of what other people are doing (or have done) so that work can be co-ordinated. Further, people collaborate over objects that are manipulated and exchanged: reports, diagrams, maps, books, models and drawings are all examples. People’s conversations and actions are part and parcel of producing the social frame within which work takes place (Giddens, 1984; Lave and Wenger, 1991). However, the technological circumstances within which that work takes place has a profound impact upon how it is achieved – technologies afford resources and constraints that affect practice. This volume offers readers an introduction to the field of Inhabited Information Spaces (IIS). We attempt to shed some light on the most important issues, including examples of representing information, how people interact in such systems, how IIS systems are constructed, and emerging notions of communities. There are already many books dedicated to “pure” information visualisation (techniques for visually representing information) and so in this volume we concentrate mainly on the value added by inhabited spaces rather than visualisation techniques per se. For a review of the field of “pure” information visualisation we recommend Card et al. (1999), who give a good overview of the field. The question addressed by IIS design is how best to design spaces and places where people and digital data can meet in fruitful exchange – that is, how to create effective social workspaces where digital information can be created, explored, manipulated and exchanged. IIS are the confluence of research into distributed, augmented and virtual reality (VR) spaces, information visualisation and computer-supported co-operative work (CSCW).
11
3
Inhabited Information Spaces
The term “Inhabited Information Spaces” derived from work being carried out within a number of European research initiatives (e.g. INSCAPE, COMIC) and laid out an agenda whereby virtual reality and information visualisation techniques were explicitly combined in support of collocated and remote collaborative work. Thus, in Inhabited Information Spaces (IIS) both information and people who are using that information (viewing it, manipulating it) are represented. This supports collaborative action on objects, provides awareness of others’ ongoing activities and offers a view of information in the context of its use. Thus, while information visualisation systems are useful tools in themselves, a representation of others who are also using the information and what they are doing with it could potentially add considerably to the value of such systems for co-ordination of collaborative work. Just knowing that many people are accessing a particular piece of information could be almost as useful as the piece of information itself. The specific representations can vary but in all cases they are manipulable: data can be interrogated, representations of people are mobile and interactive. Further, being shared, navigable, “live-in” spaces, information can be explicitly sought, can be “discovered by chance” (cf. Williamson, 1998), or “encountered” (cf. Erdelez, 1999) or tacitly consumed as part and parcel of navigating an “information neighbourhood” (cf. Savolainen, 1995). In Europe, the European Commission (EC) funds large programmes of research covering most fields of human endeavour (science, technology, medicine, culture etc.); one of the domains supported by the EC is Information Society Technologies (IST, http://www.cordis.lu/ist/). In 1996 the i3 (Intelligent Information Interfaces) network (http://www.i3net.org) was formed as a mechanism to create a community of people working on a number of research projects. One of the aims of this was to encourage researchers participating in European projects to exchange ideas and information and to allow collaboration on a larger scale. From 1997–2000 the i3 supported a programme called Inhabited Information Spaces and much of the work that appears in this book resulted from this programme. The field of IIS overlaps that of Collaborative Virtual Environments (CVEs) (see Churchill et al., 2001), as CVEs are one of the preferred implementation techniques for visualisation information in a collaborative way. However, IIS does not necessarily imply the use of online virtual environments technology – for example, it is possible to imagine a system that enables co-located groups to co-operatively work with information by using a display projected onto physical artefacts. Research into IIS also overlaps with work carried out in Social Navigation, which explicitly addresses social aspects of information seeking, searching and use (see Munro et al., 1999). The chapters in this volume cover all variants on IIS, the technology required to make it work and the social and psychological issues raised by such work. We also present recent innovations in “hybrid
4
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
An Introduction
● 11 ● ● ● ● ● ● ● ● ● ● ● ● ● ● environments” and augmented real-world environments. Our aim with this broad coverage is to offer readers the opportunity to reflect on the intersection of technology design, communication, representation, and collaborative work practices. Technological, psychological and sociological issues in the design and use of Inhabited Information Spaces are considered, including: design issues in the development of technology for human–human and human–system collaboration around information visualisations; applications demonstrating uses of the technology; and psychological/sociological analyses of the way such systems are used in practice. Our aim was to provide a broader perspective than solely graphical aspects of visualising information, systems aspects underlying the distribution and sharing of graphical and textual workscapes, or the communication and work practice aspects of the use of such systems.
1.2 Chapters in this Volume We have divided the chapters in this volume into a number of broad areas, although observations made by the authors often span these areas. The areas are pure virtual environments; mixed reality; communicationoriented systems and applications; construction; and community.
1.2.1 Pure Virtual Environments Stenius and Snowdon, in their chapter ‘WWW3D and the Web Planetarium’ (Chapter 2) describe the WWW3D 3D web browser, and how it turns browsing the web into exploration of a 3D space. The chapter describes the initial version of WWW3D and describes how it evolved into the Web Planetarium. Not only is the Web Planetarium more aesthetically pleasing and more scalable than WWW3D, but in this incarnation it has also been re-purposed to serve as a gateway between 3D environments. In Chapter 3, Pettifer, Cook and Marsh describe the Placeworld system, and its implementation in the Deva virtual environment. Placeworld was inspired by Jeffrey Shaw’s artwork “PLACE – A User’s Manual” and, like the Web Planetarium, aims to provide a connection between virtual spaces. The chapter describes the series of user trials that led to the final Placeworld design and how the Deva system is used to create a high performance virtual environment that implements the Placeworld design. This chapter also gives some insight into some of the issues that must be tackled to implement large scale IIS and CVE systems efficiently and serves as an appetiser for topics covered in more depth in Part 5 of this book. The final chapter in Part 2, Chapter 4, is by Wallberg and Ståhl, who use a pond metaphor for information visualisation and exploration. The Pond is a system that allows people to browse collections of multimedia data, such as music albums. The interface is presented in the form of a
5
Inhabited Information Spaces
large back-projected display on a table surface allowing several people to gather around it and use it collaboratively.
1.2.2 Mixed Reality Environments Chalmers, in his chapter entitled “City: A Mixture of Old and New Media” (Chapter 5), argues that one of the problems with “traditional” CVE systems is that they are disconnected from the physical environment and from other media (for a review of CVE systems see Churchill et al., 2001). He calls for an approach in which there is an explicit linking between different media, where people are considered to inhabit the “real” (physical) world, not the virtual, and have a number of media both “old” and “new” available to them. The chapter presents an experiment where visitors experience a gallery via different media – one by physically experiencing it, one via the web and one via an immersive 3D VR environment. All three are able to communicate via an audio link and are given awareness of the locations and actions of the others. The chapter presents the results of this experiment and the ways in which the participants used the features of the technology to interact and share their experiences. In Chapter 6, Brooks describes the Soundscapes system, which allows unencumbered interaction with visual and auditory systems projected into the physical world. This work illustrates a different form of information system, an auditory one, and offers an example of a mixed reality system, and example applications are covered including therapeutic use and public artistic performances. In the next chapter, Plaza illustrates the computational interplay of physical space and information space. Drives within mobile computing push computers further “into” the physical world (see also Mark Weiser’s vision for ubiquitous computing, Weiser, 1991). In order to design tools that are context sensitive and not inappropriately intrusive, it has been argued that such devices need to be “aware” of the activities of their users. This chapter describes the approach used in the COMRIS project in which wearable computers were linked with an information space composed of agents who attempt to find information useful to a person at a given moment.
1.2.3 Communication In Chapter 8, Bullock describes human–human communication via the medium of an IIS, considering design issues in the development of virtual conferencing. He makes the points that, for an IIS to function effectively, all technological elements need to work in concert. With video conferencing as a backdrop, Bullock explores opportunities and pitfalls of using IIS for mediated communication.
6
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
An Introduction
● 11 ● ● ● ● ● ● ● ● ● ● ● ● ● ● Fraser, Hindmarsh, Benford and Heath in their chapter “Getting the Picture: Enhancing Avatar Representations in Collaborative Virtual Environments” discuss how humanoid avatars are the most widely used type of avatar in VR systems. They note that although the avatars have humanoid forms they don’t have human-like perceptual abilities within the virtual worlds – the data that are relayed back to people from their avatar as prosthesis-in-the-virtual-world is in fact often misleading. This chapter examines some of the problems inherent in using different forms of avatar, and the problems this poses for collaboration in IIS. The authors describe some extensions the authors have made in an attempt to rectify some of the problems they have encountered. Jää-Aro and Bowers describe some new ideas on navigation and view control that have been inspired by cultural applications. This chapter describes the lessons learnt from a number of public VR and mixedreality performances. The authors discuss what they learnt in terms of the content of performances, the pacing and the means given to participants to navigate within the space. An important issue addressed is how such performances can be made accessible to non-interactive audiences who can only see a TV-like rendering of the event. In Chapter 11, Prinz et al. consider how to present awareness of the activities of others for better support of collaborative work. This chapter describes the TOWER (Theatre of Work Enabling Relationships) system, which provides a number of mechanisms to communicate awareness information to members of a work group both via 3D displays and also via Smartmaps integrated with the Basic Support for Co-operative Work (BSCW) document management system. Smartmaps are 2D displays based upon the tree-map visualisation technique. The TOWER world is an automatically constructed 3D environment that represents both users and documents and indicates the actions that users are taking with respect to the documents via symbolic actions and gestures performed by the avatars. DocuDrama allows 3D presentations of the past actions taken by members of a project team as a sort of 3D virtual theatre in which avatars look and turn towards one another to enhance the impression of an ongoing conversation, and camera navigation is carefully controlled in order to generate an interesting presentation. Finally, small simple robots were employed in order to give a tangible presentation of the activities of other users.
1.2.4 Construction Frecon’s chapter in Part 5 introduces readers to DIVE, a programming environment for prototyping IIS. The DIVE CVE system is described, with a focus on the mechanisms it provides to allow developers to rapidly develop VR applications, including IIS. DIVE provides several different APIs (application programmer’s interfaces) and mechanisms for
7
Inhabited Information Spaces
creating dynamic 3D content thereby allowing developers to choose the combination that works best for them. The chapter concludes with a number of examples that show how significant applications have been built using DIVE. DIVE is one of the oldest and most mature VR systems and the current version is the result of many years of experience; both the Web Planetarium and Pond systems are built in DIVE. Readers are urged to read these three chapters (2, 4 and 12) to fully understand what is possible with a mature VR system. In Chapter 13, Roberts considers communication architectures for IIS, describing the most important networking issues that need to be faced when trying to construct distributed IIS systems. He also details some of the techniques that can be used to create an illusion of a shared space in the face of delays caused by communications technology. A number of different CVE systems are described in order to give concrete examples of the techniques described in the earlier part of the chapter.
1.2.5 Community Robinson’s chapter deals with peer-to-peer networks and communities. Robinson argues that the concept of peer-to-peer and notions of community are heavily interdependent, and that the design of IIS would benefit by focusing more closely on community as the organising principle of peer-to-peer. The author first considers the early days of research into CSCW, analyses the metaphor of community and how it relates to electronically mediated communication, and draws parallels with the current state of peer-to-peer systems today. Burkhalter and Smith move away from what might be considered a true IIS in Chapter 15 to explore how the availability of social accounting data can help the auto-regulation of online communities. The authors consider Usenet news but the principles could just as well be applied to other systems. The intent of this chapter is to show how similar information might be used to help CVE-based communities such as those being created by there.com (http://www.there.com), if they ever reach the scale of communities such as Usenet.
1.3 Summary As noted above, the chapters in this volume cover a broad range of technical and social issues. All are focused on creating useful and “habitable” environments for information representations, seeking, searching and manipulation. There are many challenges to be faced at technical and at social levels, and far more research needs to be done on the long-term use of Inhabited Information Spaces, and how they co-evolve as a result of being regularly inhabited. We hope you enjoy reading about the work as much we have.
8
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Part 2 Pure Virtual Environments
011
011
011
11
9
This page intentionally left blank
011
011
011
011
11
WWW3D and the Web Planetarium
2
Mårten Stenius and David Snowdon
2.1 Introduction This chapter will describe an Inhabited Information Space based around a 3D visualisation of a portion of the WWW. The system was originally called WWW3D (Snowdon et al., 1996), and evolved from an experiment in immersive 3D web browsing into a richer inhabited information space called the Web Planetarium during the i3 eSCAPE project (Stenius et al., 1998). We include it here as an example of an inhabited space supporting a common modern activity – web browsing – and also as an example of how a basic visualisation was extended to correct deficiencies in the earlier system. WWW3D started life as an experiment in 3D web browsing. We were thinking about some of the original ideas behind VRML (Virtual Reality Modelling Language, www.vrml.org), namely that it would be a sort of 3D web. In fact many people working on VRML were inspired by the Cyberspace detailed in William Gibson ’s novels: Cyberspace. A consensual hallucination experienced daily by billions of legitimate operators, in every nation, by children being taught mathematical concepts . . . A graphic representation of data abstracted from the banks of every computer in the human system. Unthinkable complexity. Lines of light ranged in the nonspace of the mind, clusters and constellations of data. Like city lights, receding (Gibson, 1986).
However, VRML became more a means to represent 3D models with hyperlinks between them, than a richer way of experiencing the web. It so happened that as we were thinking about this during the summer of 1996 we were asked to provide a demonstration for SICS’s (the Swedish Institute of Computer Science’s) new large-screen immersive VR system. This system, while having impressive graphical performance, did not support interaction via a standard 2D GUI (Graphical User Interface); we therefore decided to create a web browser that was entirely 3D. While
11
Inhabited Information Spaces
the result was not a web browser as usable as Netscapes ’s or Microsoft’s for standard web browsing, the prototype had several interesting features and evolved into a more capable system during the i3 eSCAPE project. In this chapter we will describe the original system, WWW3D, and its evolution into the Web Planetarium. Just as a normal web browser, WWW3D allowed users to follow links and view the web pages associated with those links. However, rather than simply showing the current web page the user was exploring, WWW3D also visualised the structure of the portion of the web that the users had explored, historical information showing when pages had been last viewed and (thanks to the DIVE environment for which WWW3D was written) other users browsing the web at the same time. Note that WWW3D was never intended to visualise the web – the web is far too large for this to be meaningful, instead we simply visualise the portion of the web that users have explored and are exploring. We assume that existing search engines are sufficient to locate new pages of interest. The two key features of WWW3D were its method for representing web pages in 3D and the method for organising multiple web pages in 3D space. We will describe how this was done before continuing with a description of how the basic prototype was improved.
2.2 Producing a 3D Representation of a Web Page Many HTML tags provide semantic information about the marked-up text – that is, what the text is rather than how it should be displayed on the page. 2D web browsers will typically use this information to determine what style (i.e. font, font style, font size, etc.) to use when drawing a particular piece of text. WWW3D uses the information contained in HTML tags to produce a representation of the document in 3D space. A web document is represented as a sphere that is labelled with the document’s title. The contents of the document is placed around the inside surface of the sphere. Displaying large amounts of text in a satisfactory way is difficult in current VR systems, so textual information is currently represented by icons that can be unfolded to reveal the entire text. The first few words of the piece of text are displayed under the icon to give some indication of the contents. Images are displayed by texture mapping them onto polygons on the inside surface of the sphere. Finally, links to other documents are represented as icons labelled with their destination. To reduce the visual complexity of the virtual environment WWW3D makes extensive use of level of detail (LOD) operations. When viewed from outside, a document is represented as an opaque sphere and the actual document contents are not displayed. When a user enters a document to view it, the sphere is drawn in wire-frame so that the rest of the virtual environment is still visible.
12
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
011
011
011
Figure 2.1 A WWW3D representation of a web document seen from the inside. The spheres in the background represent other web pages.
Figure 2.1 shows the contents of a web document as displayed by WWW3D. Figure 2.2 shows the display generated when a user reads some of the text comprising a web document.
2.3 Browsing the Web Using WWW3D
011
11
When a user selects a link icon, WWW3D creates a new sphere representing the target document and places it near the document from which the user selected the link. In order to indicate the structure of the portion of the Web that the users have explored, WWW3D draws arrows between the spheres representing linked documents. If the documents are resident on the same Web server then the arrow is drawn in blue, otherwise
13
Inhabited Information Spaces
Figure 2.2 Reading part of the text of a web document.
it is drawn in green, thereby helping to provide additional information on the structure of the documents that the user has explored. In addition to this, the brightness of the arrow is dependent on the time since a user last followed that link thereby providing users with a visual representation of their browsing history. If WWW3D fails to fetch a document then a small red arrow is attached to the source document to represent the “broken” link . WWW3D is implemented using multiple lightweight threads so users do not have to wait for a document to be retrieved before selecting another link. This behaviour is essential if multiple users are to be able to browse independently. Users are also free to navigate through the space and browse other documents while waiting for a document to be retrieved.
14
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
As WWW3D parses a newly retrieved document, it checks for links to documents that the user has already explored and draws arrows to represent them. This means that at any given moment the complete set of links between documents is displayed without users having to follow every link. This is intended to aid the user by indicating links between documents that the user might have been unaware of. This also has the result that several users can be browsing different parts of the web and yet any links between the sets of documents they are exploring will be displayed. This might be useful since users will then have a visual representation of possible common interests. To produce an acceptable layout of the set of linked documents an incremental version of the Force Directed Placement (FDP) (Fruchterman and Reingold, 1991) algorithm is used. Links between documents act like spring forces that result in linked documents being moved closer together. Documents exert repulsive forces on one another which prevents documents being placed closer together than a userspecified minimum separation. At regular (user-specified) intervals WWW3D applies the FDP algorithm to refine the inter-document layout for a specified number of iterations. The more links a document has, the greater “inertia” it is considered to have when the FDP algorithm is applied. This has the result that heavily referenced documents are less likely to move and provides some stability to the visualisation. In addition damping is applied to try and prevent large changes to the visualisation for a given iteration of t he FDP algorithm – this helps to prevent the user from becoming disorientated. The result of this is that the inter-document layout gradually evolves over time to produce clusters of inter-linked documents. Instead of running the FDP algorithm until it converges (which could take a long time) only a specified number of iterations are executed. The rational for this is that the space will change anyway as users browse new web pages so it is not necessary to force users to wait until the layout stabilises before allowing them to continue browsing. If there is no change to the space (no new web pages) then the system will gradually converge to a stable state. However, the FDP algorithm suffers from the disadvantage that it can take a great many iterations to converge to a stable state. Figure 2.3 shows the display produced by WWW3D after the user has browsed a number of documents. The FDP algorithm has resulted in the formation of clusters of closely linked documents. The colours of the documents provide some indication of how long ago the user last visited them. Between invocations WWW3D stores information on the current set of documents, the links between them and the current 3D layout to a file. When a new instance of WWW3D is started, it reads this history file and displays the structure as it appeared in the last session. The contents of the documents found from the history file are not retrieved until the user
15
Inhabited Information Spaces
Figure 2.3 A collection of web documents, showing the links drawn between them.
enters a particular document. Since the document representations are opaque this process is invisible to the user except for an occasional delay in seeing a document’s contents.
2.4 Improving Scalability One problem with the WWW3D prototype is that it suffers from scalability problems both in terms of display and number of users. Since a single DIVE application is responsible for management of the visualisation, it is vulnerable to becoming overloaded if several users simultaneously place demands on it. In addition to this the complexity of the display increases as documents are added, lessening the advantage of
16
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
the displayed web structure information and making it harder to browse. Even though WWW3D makes extensive use of level-of-detail (LOD) operations for individual nodes, there will still be a point where the world becomes too complex to be rendered on even the most powerful hardware. For this reason we created a new version of WWW3D running on the MASSIVE-2 CVE (Benford et al., 1997a) that attempted to solve the display problem by making more extensive use of LOD effects combined with clustering of web pages. We decided to extend the use of LOD beyond the contents of a single web page to encompass groups of web pages (e.g. all pages on the same server, same domain name, owned by the same user etc.). Doing this meant that instead of showing all the pages in a website, or all the personal pages (home page and related pages) for a user, we could display an object that represented the whole group. The user would only see the contents of the group when a certain criteria was met – in this case getting close enough to the group. This meant that when a user was outside a cluster their client had no need to know anything about the web pages in the cluster or the users that were currently browsing web pages in the cluster. This would have the following advantages: ● ● ● ●
A reduction in the visual complexity of the world. A consistent metaphor since clusters looked and behaved similarly to individual web pages when users entered them. Increased interactivity since the computational load on the client machine was reduced. Reduced network bandwidth since MASSIVE-2 assigned separate multicast communication channels to each cluster. This meant that a user who could not see the interior of a cluster had no need to receive network updates concerning objects or users contained in that cluster.
We used a simple scheme based on URLs to perform our clustering. A more advanced alternative would be to explicitly consider the legibility of the CVE and cluster accordingly as is done by LEADS (Ingram and Benford, 1995) which is also capable of adding additional objects such as landmarks to aid users navigating through a CVE. Our new implementation was based on two fundamental building blocks, containers and links. Containers have the ability to enclose other objects in a spatial sub-region of the virtual space. Links connect the containers. A special form of container is the page, which represents a web page and its contents. A cluster is a container that is able to treat its contents as a sub-visualisation complete with layout, other containers and links. Figure 2.4 shows a cluster both from a distance and as the user gets sufficiently close to see the contents. In order to simplify the representation of links between clusters, we introduced compound links. A compound (or aggregate) link between two clusters is formed whenever there exists one or more links between the
17
Inhabited Information Spaces
a
b Figure 2.4 A cluster abstraction dissolves to display its contents when the user approaches: (a) the exterior of the cluster; and (b) the view seen by a user as they get closer to the cluster.
18
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
A
B
c
f d AB g e
011
011
Figure 2.5 A compound link (AB) between two clusters (A, B).
contained objects of the two clusters. Thus, a compound link both serves as a link structure between two clusters and as an abstraction for one or more lower-level links between the contents of these clusters, this is illustrated in Figure 2.5. Figure 2.6 shows a collection of linked clusters. When refined, the current application structure approach should lend itself to parallelisation: letting different processes handle different clusters. This in turn would enable a distribution of calculations on different hosts to allow more complex visualisations.
011
011
11
Figure 2.6 A set of interconnected cluster abstractions. Their sizes roughly reflect the number of objects (here pages) contained within.
19
Inhabited Information Spaces
2.5 The Web Planetarium: Creating a Richer Visualisation The Web Planetarium extends the WWW3D concept with the idea of the system becoming a gateway between different virtual spaces. The metaphor of a planetarium was used to create a new, more informative and more visually appealing look and feel, since the system was also going to be used in public settings using a large, dome-like display (Schwabe and Stenius, 2000). Nodes in the Web Planetarium display can represent either web pages, 3D models (in VRML or DIVE format) or 3D DIVE worlds that can be jumped to. The Web Planetarium can therefore also act as a space of portals to other spaces and provide a means of navigation between different virtual worlds. Compared to WWW3D the Web Planetarium provides a more visually interesting virtual space and different layout options.
2.5.1 Visual Differentiation of Nodes One problem with the WWW3D visualisation is that all nodes look alike. The only distinguishing feature is colour and that colour represents the time since the last access and not a feature of the node itself. When creating the Web Planetarium, it was felt more important for users to be able to quickly scan the space for potentially interesting pages than to show the usage information. The Web Planetarium replaces the use of colour by texture maps extracted from the web pages. Alternatively, the sphere representing a node can be replaced by a 3D model. Not only does this result in a more interesting environment but it allows the users to dinstinguish more easily one node from another and gain some idea of what the node represents. Figure 2.7 shows an example Web Planetarium view in which the texture maps clearly distinguish nodes from one another. A simple algorithm is used to select the image to texture onto a sphere – the first image found in a web page is used as the texture. In the case of a user’s home page this image is typically a photo of the user – this is easily visible in Figure 2.8 in which several of the spheres clearly show people’s faces. Obviously, a simple algorithm such as this will sometimes fail to produce an interesting image for one of several reasons. ● ● ●
20
There is no image on the web page. A banner advert is selected instead of an image representative of the content of the page. The image selected is a piece of web decoration (a line, a bullet point) instead of something really representative of the subject of the web page.
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
011
011 Figure 2.7 The 3D layout alternative emphasises the concept of a planetarium.
011
011
11
Figure 2.8 A screenshot of the Web Planetarium. Compared to the original WWW3D, one of the obvious differences is the use of texture mapping to give more information about the contents of web pages instead of a simple colour indicating the time since last access.
21
Inhabited Information Spaces
It would be hard, probably impossible, to produce an algorithm guaranteed to work, but there are a number of simple filtering operations that could be used to ensure that in many cases unsuitable images are rejected. ●
●
●
Reject images that are too small (according to predefined parameters) or are long and thin. This extremely simple approach was used successfully in the CWall electronic notice board (Snowdon and Grasso, 2002) and rejected most web decoration . Use a blacklist of URLs and URL components to reject banner advertisements or configure a proxy server that performs this function (Hood, 2000). There are websites that maintain such blacklists (@Man, 2000) so each user would not have to shoulder this burden. Reject images that do not come from the same server as the web page. This is a very simple approach that will work in many cases, but it may sometimes reject legitimate images.
An alternative approach could be to use a conventional web browser – embedded within the application or running in parallel – to generate a snapshot of the web page, including background colour, text layout, images and so on. The snapshot would then be used as an icon (or “thumbnail”) to be pasted on the outside of the 3D page representation. This method would have the advantage of creating a stronger connection to the original design of the web page as a whole.
2.5.2 The Web as a Road Network WWW3D and the Web Planetarium can generate very tangled and convoluted visualisations of nodes. Although users can manually reposition nodes this is not a solution that can be applied to large spaces. The Web Planetarium offers users the options of restricting the FDP layout algorithm to a horizontal plane, thereby giving the layout shown in Figure 2.9. A disadvantage of this approach is that in certain cases the FDP algorithm may take longer to converge.
2.5.3 Hybrid Browsing The original aim of WWW3D was to provide an acceptable and novel web-browsing experience using an entirely 3D interface. This made an interesting demo and made sense on a large screen or head-set based system but is less useful on the more common desktop-VR systems in which the VR system is just another window on the user’s 2D desktop. The Web Planetarium therefore allows users to combine the 3D view of the web structure with a traditional web browser view of the contents of a web page. This is illustrated in Figure 2.10 which shows a Web Planetarium view and a web browser (Netscape) view of the same web page side by side.
22
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
WWW3D and the Web Planetarium 2 ●●●●●●●●●●●●● ● 1 ●
011
011
Figure 2.9 The 2D layout alternative results in a road-like network of sites and links.
011
011
11
Figure 2.10 Using a conventional web browser as a side-viewer for HTML documents encountered in the Web Planetarium. Whereas the 3D view of the HTML document (to the right) gives an overview of the content and interaction points for links, Netscape provides a detailed close-up of the current page (to the left).
23
Inhabited Information Spaces
2.6 Conclusion This chapter has described WWW3D, a simple 3D web browser and its evolution into the Web Planetarium. Starting from a humble beginning WWW3D has been changed to address the issues of scalability and producing a more pleasing and informative visualisation. In addition it can now be seen as providing a means for navigating between disjoint 3D worlds as well as between web pages. However, the current version of the Web Planetarium is not without problems. One major issue is related to the viewing of 2D standard web content. Neither the current 3D view nor the separate browser and 3D views side by side really provide a convenient browsing experience. There is still work to be done to find the optimal way to provide information about web structure and a view on web page contents in an integrated way. The second problem with the Web Planetarum is that while it is perfectly possible for multiple users to share a single instance of the Web Planetarium there is no support given to the interrelation of different planetaria nor to different views on the same planetarium. The use of subjective visualisations (Jää-Aro and Snowdon, 2001) may provide a technique for merging planetaria since the same web page could be presented differently to different users, thus preserving individual layouts while allowing a degree of interaction between users.
24
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
PlaceWorld, and the Evolution of Electronic Landscapes
3
Steve Pettifer, Jon Cook and James Marsh
3.1 Introduction The eSCAPE project (ESPRIT Long Term Research Project 25377) set out to investigate the idea of an “electronic landscape” (e-scape): large-scale inhabited information spaces represented as a three-dimensional (3D) virtual environment (VE) in which users could interact with applications, find routes to other virtual worlds, access sources of data, and interact with one another at a social level. The project brought together computer scientists, artists, social scientists and graphics designers from Lancaster University, the University of Manchester, the Swedish Institute of Computer Science and the ZKM (Zentrum für Kunst und Medientechnologie – Centre for Art and Media Technology) in Germany. This chapter describes the evolution of PlaceWorld, an example e-scape. It is organised in four sections. We begin by chronicling the developments of the eSCAPE project that led to the implementation of PlaceWorld as its final demonstrator. We then consider in more detail the nature of PlaceWorld itself, and discuss how its development has influenced the underlying system technology. We conclude by tying together some of the technological issues with concrete examples of their use in the PlaceWorld landscape. The vision of an electronic landscape is easy to articulate, but difficult to make real. Distributed 3D virtual environments of any kind push the limits of today’s graphics, networking and processing technology; some of these issues will be considered in detail much later in the chapter. Add to these problems the social factors involved in making a virtual environment actually usable by the general public, and the issues become much more complex. The eSCAPE project’s approach to these difficulties is described in this chapter.
11
25
Inhabited Information Spaces
The project set out to address two problems. First there was the technological difficulty of integrating diverse environments and their applications. There is as yet no accepted standard technology for building a VE, and though the majority of implementers end up using OpenGL to render the visual part of the world at some point in their implementation, the sources of semantics and behaviour vary wildly from bespoke handcrafted pieces of code to those using higher level more generic systems such as MASSIVE (Greenhalgh, 1999), DIVE (Chapter 12) or DEVA (Pettifer et al., 2000). We will return to consider the way in which the DEVA VR system evolved to support electronic landscapes later in this chapter. The second, and much more open-ended, issue is that of how to make an environment really inhabitable and of practical utility to a user, and it is to this that we first turn. The methodology used in the eSCAPE project to develop effective electronic landscapes was iterative. Inspiration for the environments was drawn from two sources: artistic vision and ethnographic study of suitable real-world situations. The environments were then implemented and put in front of the general public for testing. Further ethnographic studies were then carried out on the installations, and the environments were refined and redeveloped in the light of the results from these studies. PlaceWorld combines lessons and technological perspectives discovered during the three-year period of this process.
Figure 3.1 Snapshots of PlaceWorld.
26
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Figure 3.1 shows snapshots from the final implementation of PlaceWorld, and will serve for now, with the brief description that follows, to give an overall flavour of the environment. Users entering PlaceWorld find themselves in an exotic landscape containing many artefacts. Situated here and there are cylindrical “buildings” covered with images from other worlds: these are portals that lead elsewhere. A network of pathways, hovering above the mist-covered landscape lead off into the distance in all directions. The scene is populated with towers, billboards, flying craft and other miscellaneous structures, and in the sky ethereal floating ribbons of light drift back and forth. Other users move around the landscape, represented by animated walking stick figure-like characters, and leave behind hazy coloured trails. Interaction with any of these entities brings up on screen a selection of possible actions. Touching a user’s trail, for example, even when that user is a long way away, reveals that it can be used as a shortcut to find its owner. Touching any other artefact allows it to be “picked up” and carried around, to be deposited later at another location. The remainder of this chapter explains why the esoteric nature of PlaceWorld is as it is, beginning with the project’s initial experiments with electronic landscape .
011
011
011
11
3.2 Background: The Physical and the Abstract eSCAPE examined two extreme styles of virtual environment, to extract from them appropriate features that would make an environment inhabitable. At one extreme, there was the physical or concrete “cityscape”: familiar, constrained, slow to change, easily understood. At the other extreme, the “abstract landscape”: unfamiliar, dynamic, unrestricted. Aspects from both these styles of environment were finally integrated into PlaceWorld. Virtual environments have the potential to include aspects of both these extremes: there is no technological reason why one should be confined to a groundplane in the virtual world just because gravity works in a particular way in the real world; there is no technological reason why one should be forced to walk from one part of a virtual world to another rather than simply “teleporting” to one’s destination, much like clicking on a link in a web page. On the other hand, it is easy to imagine getting completely disorientated in a space where there are no constraints on navigation, or of missing out on chance meetings and opportunistic discoveries if one’s only means of transport is by instantaneous hyperlink. Thus some balance needs to be achieved between these two extremes of physical and abstract. The evolution of the project is easier to tell from the perspective of the concrete cityscape metaphor rather than the abstract, simply because of familiarity with its nature and terminology. The following vignettes chronicle various attempts at building electronic landscapes, and lead towards the final design of PlaceWorld.
27
Inhabited Information Spaces
3.2.1 Watching a Cityscape The first environment built during the project was a straightforward cityscape (Figure 3.2), consisting of buildings, roadways, parks and street furniture generated by algorithms based on Hillier and Hanson’s (1984) studies of urban evolution. A set of ethnographic studies were completed to see how users behaved in this environment, setting the participants the task of finding a number of landmarks in the city such as fountains, statues or pavilions. Studies were carried out at a desktop computer, using a standard mouse as an interaction device. The results were in some ways surprising. In spite of the user being a lone inhabitant of the cityscape, and without prompting from the scientist, wayfinding in the city was almost invariably carried out on the pedestrian areas (pavements, parks etc.) in spite of there being not a single car in sight. Though there was no collision detection preventing passage through the buildings, users mostly followed the road system. Even though the mouse navigation included an ability to fly up in the air (a feature that the participants were told about, and which clearly would give them an advantage in terms of spotting land marks from on high), this was rarely used (indeed, one participant who did briefly levitate to a suitable
Figure 3.2 A virtual cityscape.
28
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
vantage point rapidly came back down to ground level having felt uneasy up in the air). Although this evidence was anecdotal rather than definitive (and there were a small number of exceptions in terms of behaviour), the studies suggested that even with little effort by way of interaction device to induce the sense of presence, that the general affordances of a real cityscape, transferred rather strongly to the virtual world. A more detailed description of the study can be found in Murray et al., 2001. On this basis, a more ambitious city-like environment, the Distributed Legible City, was built.
011
011
011
011
11
3.2.2 The Distributed Legible City The artist Jeffrey Shaw, a partner in eSCAPE, had once developed an artistic installation piece called the Legible City. In this work, a solitary cyclist, seated on a real bicycle and situated in a darkened room with a large projected wall, could cycle their way around interpretations of the cities of Amsterdam, Manhattan and Karlsruhe. Rather than ordinary buildings, the cities were populated with towering, solid letters, forming extracts from texts associated with the areas of the cities. For example, a virtual tourist in the Legible City could follow the ramblings of a taxi driver around Manhattan, or find sections of poetry in Amsterdam. Informal observation of the piece, which is still in situ in the ZKM in the real Karlsruhe, showed that, even with now outdated rendering technology and simple interaction style, the piece would capture people’s attention, and they would happily tour around the cities for some time, often trying to find out which piece of text was associated with their favourite part of the world. For entirely opportunistic reasons (much of the technology already existed and the piece was known to be “engaging”), the project decided to extend this installation to form a distributed environment, and so the Distributed Legible City (DLC) was constructed (Figure 3.3). Using consumer-level graphical accelerators, the original database of city structures was reused, and modified exercise cycles were pressed into action as interface devices. Rather than the large screen projection system used in the original work, stands for monitors were built in front of the cycles, and where the first Legible City used an LCD panel attached to the handlebars to display maps of the cities, in this new version the map was displayed virtually as an overlay on the monitor screens showing now the layout of the world as well as the positions of other inhabitants. For the purposes of communication, the users were provided with headphones and a microphone. Other users were represented by animated cycling avatars. Two versions of the DLC interface bike were situated around the ZKM, and another across a wide area network in Vienna, and the installation was made available to the public during a number of exhibitions.
29
Inhabited Information Spaces
Figure 3.3 Images from the Distributed Legible City.
For several days, the technology behaved perfectly; however, the overall result of the piece was extremely disappointing. No longer did users engage with the environment and there was no interest in exploring the world for its own sake. Quickly, users realised that the buildings offered no resistance to them cycling through them, and the overlaid map was discarded, with users making a beeline towards one another and completely ignoring their surroundings. Once users had found each other, another disappointing interaction took place. Unexpectedly, users were resolute that they should be facing one another in order to carry out any conversation (this was not a requirement on the audio layer, which had no directional bias). The positioning of the monitor in front of the cycle, and the inability to pedal backwards meant that a significant amount of effort was put into achieving “conversational orientation”, cycling round in ever decreasing circles until this was achieved. With this effort expended, the users exchanged little more than a few perfunctory greetings, and then left the installation. Our expectations of multi-user cycling tours of the cityscape were far from met. With hindsight, the problem was one of a mismatch between the affordances of the environment, and the expectations of the users. First, the main novelty of the environment to its user turned out to be the potential to find another inhabitant, and perhaps then to tour the world.
30
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Flashing arrows on the overlaid map made it clear that other users were around somewhere, and once users realised that the “buildings” did not block their attempt to cycle through them, the map became a meaningless diversion, which was not returned to as a useful navigation aid later (in spite of its apparent successful application in the original piece). Similarly, on a real bike one would expect to be able to cycle side by side with a friend, or at least to look over one’s shoulder to carry out a conversation with a cyclist behind. The fixed position of the screen in front of the cyclist, and the difficulty of arranging conversational orientation with these limitations, made casual touring and chatting unrealistic. To solve these problems, two simple modifications were made to the installation. First, the map was discarded (the alternative of forcing the utility of the map by making the buildings impenetrable seemed unproductive since the cities were large and users simply wanted to find one another in the first instance). As a replacement, a bird was positioned in the sky above each participant, which would fly in the direction of the nearest other user, meaning that now finding one another was straightforward: simply follow the bird. The most significant alteration was the introduction of a tracked head mounted display. The original fixed monitor was retained so that bystanders could see interactions within the environment; however, the cyclist was now able to look around in the environment, and to see the world in a different direction from that in which the cycle was facing. The improvement to social interaction with the new version of the DLC, this time tested out at ESPRIT IST98 in Vienna, was dramatic. With the ease of finding other users and of achieving a convenient orientation for comfortable communication, participants were much happier to spend extended periods of time exploring the cities together. The conclusion at this stage was that cityscapes provide easily understood metaphors for interaction, but these metaphors must be matched with the actual affordances of the environment and its input devices with great care.
3.2.3 Finding “Something to Do” For all the lessons learned from the Distributed Legible City, it was far from being an e-scape: it lacked interaction with other environments, and had no “information” as such beyond the possible interest in the cities’ texts. A more concrete cityscape was built to address these limitations. An ethnographic study of a real-world tourist information centre, in the northern England seaside town of Morecambe was carried out, concluding that it was an interesting enough location of social interaction and information browsing to warrant further attention. In particular the question of “What can we do here?” seemed as relevant to a tourist turning up in a new holiday location as it did to an inhabitant of a new electronic landscape. A virtual Tourist Information Centre (TIC)
31
Inhabited Information Spaces
was constructed to see in what way the behaviours observed in the real world TIC were transferred to the virtual landscape, and studies of this new environment were carried out in situ. The virtual TIC (Figure 3.4) was based on a map of Morecambe, and included 3D representations of the various tourist attractions. The map and its contents distorted dynamically to bring objects or areas of interest to a particular user into view, according to search criteria. Once more, users tended to follow city-like conventions in their exploration of the virtual world, and there was evidence that to some extent the environment supported social interaction and exploration. Its limitation, however, was clearly one of content. The hand crafted nature of the virtual Morecambe and the relatively small number of attractions represented within the world meant that its real potential as a source of holiday ideas was nowhere near reached. More importantly, it was also obvious that it was unrealistically costly for any individual to attempt to populate such a world. From this it was concluded that an important aspect of any worthwhile e-scape was the ability for its inhabitants and stakeholders to straightforwardly introduce their own content and to modify existing artefacts to reflect their own experience.
Figure 3.4 A virtual Tourist Information Centre, showing the deformable map and buildings.
32
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
3.2.4 Abstract Influences: Nuzzle Afar In parallel with the thread that has been described so far, informed very much by the “concrete” and “physical” end of the e-scape spectrum, a number of other studies and sub-projects were being explored under the eSCAPE banner. One of these in particular: a multimedia installation of an abstract nature called Nuzzle Afar had a significant influence of the nature of navigation and wayfinding in PlaceWorld. Nuzzle Afar, a multimedia installation by Masaki Fujihata, consisted of an enclosed room, with left and right side entrances. Within the room were two podia, in front of which were projection screens. Trackballs embedded on top of the podia allowed users to control movement through the computer-generated environment displayed on the projection screens, and microphones similarly located allowed distributed users to communicate. As users moved through the electronic environment, a string-like trace is left upon the virtual landscape, which may be locked onto and followed by another user. The enclosed space of the virtual world consisted of four walls, ground and sky plane. Upon the walls were images of “sense organs” (e.g. a hand, an eye, an ear, etc.). In addition to spherical “avatars” of unique colour, a sphere and a cylinder are placed within the virtual room. These latter objects were the means by which users could enter or depart a series of three rooms. Within each of the spaces in Nuzzle Afar navigation needs to be learnt anew: effort, space and travel have different relationships within each. When two or more users are in close approximation “within” any of the spaces, they are able to see a video image of one another mapped and wrapped, visor-like, around the middle of the sphere. This allows for recognition of the others’ identity. In meeting each other, any two users are able to enter a new virtual space which encompasses them, while locking out the previous environment and any other inhabitants. This new space is, however, visible to other users as a spherical object inside of which the colours of the two users “inside” merge. Once inside this new space, users are represented via their video images on a 2D square. When one or both of the users leave this space, a video still of the two users remains, along with details of the time and physical locations of the encounter. In spite of the very abstract nature of the Nuzzle Afar world, its means of wayfinding and of locating other users proved much more intuitive than our initial experiments with maps and other metaphors borrowed from the cityscape metaphor. The influence of this installation on the final implementation of PlaceWorld is described in the following sections.
3.3 PlaceWorld PlaceWorld became the final eSCAPE demonstrator and assimilated much of the work done in the earlier parts of the project. It aimed to
33
Inhabited Information Spaces
become an effective “place where places meet” building on the lessons learnt from studies of the smaller scale landscapes. Another multimedia installation piece by Jeffrey Shaw formed the basis for its artistic design, and a novel user interface device was commissioned for the purpose of public display of the new environment. The next sections describe the vision and rationale that underpinned the world.
3.3.1 The Design of PlaceWorld The design of PlaceWorld took as its point of departure many features of Jeffrey Shaw’s 1995 artistic work “PLACE – A User’s Manual” (Shaw, 1997). The paradigm that is embodied in both implementations is that of a navigable virtual environment that is populated by cylindrical artefacts that form portals into other virtual environments. The world is also characterised by pathways that provide identifiable routes between these places. As boundary architecture, cylindrical space is quite different from more typical rectangular spaces: the latter is perceived as an enclosure, whereas the former constitutes what is in effect a scaleable panoramic horizon. This aspect of the virtual cylinder is extended into the real world by the panoramic navigator interaction device described later. Cylindrical spaces are especially appropriate within PlaceWorld because they allow all the idiosyncratic virtual environments to define their own unique enclosures within a neutral and scaleable circumference. Despite the apparent exterior uniformity of these cylinders, attaching images to their surfaces offers an expressive way to signal their interior contents. In this manner, rather than by architectural variation, they constitute a characteristic and very expressive visual syntax for PlaceWorld. This cylindrical uniformity does not preclude the possibility to scale them variously in height and width, or to work with the relative transparency of their surface, and this allows further expression of spatial as well as temporal meanings as will be described later. From this initial vision, a graphic designer developed a design specification that took the earlier PLACE work, and extended it with new features that addressed the particular needs set by the objectives of an electronic landscape. One important aspect was the fact that an everyday urban landscape is characterised by the presence or absence of its population of citizens. PlaceWorld also socialises its unusual landscape with representations of its visitors and occupants, but in accord with the inherent properties of this information space, these avatars have unique design characteristics that enable them to effectively function as true citizens of PlaceWorld. A visitor in PlaceWorld would be expected to orient themselves to a large extent by the visual identities of the cylinders themselves – that is, by the individual images that would be textured on the outer surfaces of
34
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
these cylinders (a form of advertising their interior contents) and the general stability of the layout. Much as in the real world, it seemed useful to also establish some method of signposting that could offer and guide visitors to the various locations. Such signposts could also be set up by persons who had built a place in PlaceWorld as pointers to their location. The issue of the relative permanence of the overall geography of PlaceWorld and its cylindrically bounded environments was felt to be an important issue if regular visitors to PlaceWorld are to be expected to readily gain familiarity with it. On the other hand, the ability to introduce new content to an environment means that some controlled modification of the world must be possible, as was made clear by our experiments with the virtual TIC. A description of the access model – a set of malleable guidelines and policies that could develop over time under the influence of the inhabitants of the world – was developed to support coherent and manageable changes to the underlying structure. A brief description of the access model is given later in this chapter. The issue of balancing familiar structure with the desire for modifiable content formed a fundamental feature of the temporal nature of PlaceWorld’s landscape, and reflects once more the desire to combine appropriate aspects of the concrete and the abstract. Masaki Fujihata’s project Nuzzle Afar provided an exemplary demonstration of the value and effectiveness of attaching traces to users’ movements through a virtual environment. Colour coded for each visitor, these lines gave temporal evidence of people’s presence and their paths of interest. And the added feature that a visitor in such an environment could “hook on” to someone else’s trace and then get a roller coaster ride along that convoluted line that brought him face to face with that other person was also very effective and enjoyable. However, further discussion of this methodology revealed a basic weakness – as soon as there would be a large number of visitors in the virtual environment these lines would become an unintelligible tangle. Because we had decided to restrict movement to ground level in PlaceWorld, such traces could then only be drawn in one plane, thus adding to the confusion if there were many visitors. On the other hand, the restriction of these markings to one plane revealed the possibility of another very interesting design strategy. We realised that these traces could in fact be merged with the geographical and physical articulation of roadways in PlaceWorld. These roadways could be programmed to dynamically and temporally express the volume of their usage, being a virtual reflection of “desire lines” in the real world that are created, say, when pedestrians create their own shortcuts across grassy areas in parks. The hierarchy could be say trail, path, road, highway, which would be defined by the density of traffic experienced along these routes, and would change according to the varying density. In this way a new feature of informative comprehensibility could be added to PlaceWorld: the nature of the pathways would themselves express a history of visitor interests, so one could, for instance, choose
35
Inhabited Information Spaces
the well-trodden paths or venture along trails into relatively unfrequented regions, or forge a new trail into previously unexplored territory. Following this line of thought about the informative value of temporal signifiers in PlaceWorld, the notion was extended to the cylinders themselves, which expressed the temporal condition and/or the popularity of their contents. As with the tourist information centre, cylinders that are of interest or frequently visited were designed to increase in scale over time, with those of little interest shrinking into obscurity. A point of debate was the notion of hyperlinking the individual cylindrically bounded environments so that a visitor could “teleport” instantly from one related place to another without having to negotiate the PlaceWorld landscape. A more radical extension of this facility would be that a visitor could at will completely reorganise the distribution of cylinders in PlaceWorld to suit a specified personal need or desire (e.g. he only wants to visit the popular places, so these all become clustered around him and the others disappear). However, going back to the premise of PlaceWorld as a physical electronic landscape that should be strongly anchored in real world physics and experiences, it was felt that such hyperlinks and hyper-rearrangements should be kept to a minimum, since they were not entirely consistent with the “physical landscape” objectives set for this environment and shown to be effective in the previous e-scapes. As a compromise, the ability for a user to generate a hyperlink between places that they had already visited was made available, with the link being visible only to that user as a glowing trail in the sky, anchored at both ends on the groundplane.
3.3.2 The User Interface and Presentation System The Panoramic Navigator (PN) is a patented technology developed by Jeffrey Shaw and the ZKM whose initial function was to provide a panoramic method of allowing visitors to the ZKM to preview and explore the contents of the ZKM building. Using a 360-degree rotatable touch screen coupled to a video camera, this augmented reality approach enables visitors to the ZKM to look around the building (via the live camera image) and use the touch screen to access additional multimedia information that is attached to specific locations in that building. While these past implementations of the PN have used the real environment as its frame of reference for interactive information delivery, it was realised that this technology could also be an ideal interface to explore and interact with wholly virtual environments, and that this would be eminently suitable for PlaceWorld. What was needed was a method of generating a representation of that virtual environment around the PN, so that the touchscreen could then be used to interact with it. The design solution for this was to attach a video projector to the back of the PN (behind the touchscreen) and to put a circular projection screen
36
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
011
011
011
011
11
Figure 3.5 The Panoramic Navigator – Augmented Virtuality version, in use with PlaceWorld.
around the PN. In this way a new presentation and interaction method which embodies the innovative concept of augmented virtuality (PN-AV) was developed. This PN-AV technology is used in PlaceWorld to embody two distinct but interrelated functions: ●
●
to display the 360-degree immersive representation of virtual environment, wherein one also see the effects of user interaction with that environment. to give the user a touchable interface that allows him/her to explore and interact with that surrounding virtual environment by showing a parallel representation of that environment which is augmented by a dynamic set of user interaction tools.
The final version of the PN-AV, in use during a public demonstration, is shown in Figure 3.5. Having described the evolution of the PlaceWorld environment, we now turn our attention to the problem of its actual implementation, and describe the development of the DEVA VR system, which grew in parallel with the e-scape demonstrators. We will return to how aspects of PlaceWorld were supported by this technology at the end of this chapter.
3.4 Technological Challenges for Electronic Landscapes Until recently, technological limitations were the clear brake on progress for VEs (the graphics challenge is perhaps the clearest example). Beyond the technology the key task is that of writing software. The scale and complexity of this task is often underestimated, and it is here where we believe that the major problems lie. We do not at this time have adequate frameworks to simplify the task of implementing virtual worlds. Today, a person wishing to implement a challenging VE application has two broad options, as evidenced in current demonstrations. The most
37
Inhabited Information Spaces
sophisticated VEs are usually bespoke applications constructed from the graphics layer upwards; this is a substantial undertaking. The alternative is to use the VE system. These fall into different categories, (for example, VRML browsers at the lower end, up to VE systems such as DIVE and MASSIVE at the more ambitious extreme). The software support challenge to facilitate large-scale VE applications is twofold. First, to find techniques and algorithms to address specific needs in VEs. Collision detection, parallelism, distribution, synchronisation, navigation and so forth all require work of this kind. Secondly, and perhaps rather harder, it is to find frameworks that allow all the parts to be put together in “flexible yet powerful” ways. The rather trite nature of such a statement belies the difficulty of quantifying that task. Finding the “right” framework is particularly difficult in the case of virtual reality (VR), since it brings together a number of complex technical issues, and binds them with real-time constraints at the social/ perceptual interface. A desirable approach for the necessary flexibility to experiment is to build as little into the system as possible, so the system provides a set of mechanisms and policies and default behaviours that can be unplugged and tailored at each level. The issue of scale is an important one; simple small-scale VEs that do not challenge today’s hardware can be constructed by any number of means. Building large-scale complex applications raises a number of challenges. For shared VEs these relate, broadly speaking, to the following areas: number of entities; complexity of behaviour required of these entities; complexity of individual rendering techniques; number and geographical distribution of simultaneous users; and number of co-existing and interacting applications. Matters of scalability and synchronisation, and some architectural and network topologies for achieving this are discussed in Chapter 13. Here we present the approach implemented in the DEVA VR system, and arising from the observation that for a shared VE there is a natural distinction between the users’ perceptions of the world and what is actually “going on” within it. That is, between the distribution and simulation of application behaviours on the one hand, and the task of presenting a coherent perception of the VE to each user on the other. This is similar to the traditional philosophical distinction between subjective perception and the underlying objective reality. The reason that this is useful is that in VR it separates out the challenging graphics/interaction tasks from the semantics of the underlying world simulation. The distinction also legitimises efforts to make a perceptually smooth presentation of the world in the light of fundamental networking limitations. Architecturally we have addressed these two aspects of the VE (perception/reality) separately. First, rendering and spatial management seem to need special treatment that is different from current approaches to graphics toolkits. The MAVERIK system aims to address these issues. The unusual approach taken by the MAVERIK system is to avoid having any internal
38
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
“structure” for the representation of VEs and their contents. Instead, an object-orientated framework is provided that supports an application builder in implementing rendering, interaction and spatial management routines that are tailor made and appropriate to their particular purpose. This is as much a performance issue as programming elegance, for key graphics optimisations are highly application specific, and are generally unavailable when the application must export its representation into the VR system. The system is described in detail in Hubbold et al. (2001). The more difficult problem is that of defining the underlying behaviour of entities in a way that can be distributed to multiple users. It is to this problem that we now turn our attention.
3.4.1 Synchronising the Behaviour of Entities The distribution of semantics or behaviour in shared VEs is a particularly difficult issue. The ideal solution would be to describe the required behaviour in a single location and to have it instantaneously sent to all participants (a pure “client/server” approach). Current network technology is far too limited to make this solution feasible. An alternative is to replicate behaviour locally for each participant (a pure “peer-to-peer” approach); this introduces extreme synchronisation problems. The approach taken in DEVA is a hybrid solution, part way between client/server and peer-to-peer. The model is one of an objective and subjective reality; the former being located (logically) on the server, the latter being represented on each client. Each user interacts with the objective reality via their own subjective view. This introduces the idea that each entity in the virtual environment has two definitions, which may differ significantly in their semantics. Our usage of the term “subjective” is not intended to imply that each user experiences an entirely different VE (this would hardly count as a shared experience). Rather, we argue that an amount of the user’s experience may be decoupled from the “objective” behaviour of the world without disturbing consequences. Given that absolute synchronisation is in any case impossible, it is our contention that so long as the length of the delay is not too large, and causal events occur in the correct order, it is possible to accept a degree of subjectivity without it affecting the users’ understanding of the semantics of the application. Under these circumstances, given no other frame of reference the users are unlikely to even realise they are not receiving the same view of the world as each other (as will be described later, PlaceWorld demonstrated this phenomenon in extreme ways). In general it would therefore be useful to differentiate between what is “actually” happening in a VE and what the users perceive. In this way it is possible to optimise the manner in which information is transferred to minimise causal discrepancies. In the DEVA system this notion is implemented by describing an entity as comprising of a single “object” and a number of “subjects”, the former
39
Inhabited Information Spaces
part being the “objective non perceived” aspect of the entity and the latter being a collection of the “subjective perceivable” characteristics (e.g. its visible, audible, haptic parts). An alternative view of this distinction is that the “object” represents what an entity does, while its “subjects” represent what the entity looks/sounds/feels like when it is doing it. Communication between an object and its subjects can be implemented using whatever high-level “vocabulary” is appropriate. In this way the system minimises the need for strict synchronisation while maximising the accuracy of causally important events. One example of this is the use of twines (Marsh et al., 1999). Twines are lightweight parametric curves that are used to “smooth out” spatially and temporally the discrete updates to the visible aspects (e.g. position and rotation) of appropriate entities by interpolation thus increasing their “plausibility” by eliminating disconcerting jumps. We are also currently investigating an infrastructure for providing “quality of service” based on balancing such smoothing against frequency of updates depending on the perceived importance of various events. The flexible nature of the communications is also advantageous for subject-to-object communications; for example when a user “grabs” and moves an entity. Such manipulation with a classic client–server architecture would involve a round trip: the event is sent from the client to the server to be processed and then the effects of the event are then transmitted back to the client to be visualised. With the commonly available network technologies we are targeting for the client–server connections such a round trip would introduce an unacceptable lag. It is perceptually important for the cause and effect of the manipulation to be as tightly coupled as possible. With the separation of subject and object, changes caused by the manipulation can be immediately perceived by the subject, with a userdefinable policy function being used to determine when the object is updated. For example, an arbitrary fraction of the changes (say, one every fifth of a second) are transmitted back from the subject to the object and thus onto other visualisers connected to the server. Alternatively, a change in position is only updated to the object when the subject has moved a certain distance from its previously synchronised location. The goal is to ensure that the entity behaves “correctly/optimally” on the visualiser performing the manipulation and also that it behaves “plausibly/acceptably” on any other visualiser connected to the cluster (while accepting that latencies and bandwidth issues rule out strict and absolute synchronisation in such a distributed system).
3.4.2 Distribution and Communications The programming model employed by the DEVA system is one of communicating “entities” which can represent objects in the virtual
40
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
environment, the properties of the environment itself, or alternatively abstract programming concepts that have no direct representation to the inhabitant of the environment. These entities are coarse-grained programming objects, exporting a number of methods that can be called by other entities, and implementing these internally using optimised imperative code (currently written in C++). The DEVA programming model makes entity behaviour explicit, allowing entities to query one another’s features, those of the environment in which they exist, and to reason about these, rather than having behaviour emerge as an implicit side effect of a piece of code being executed. In this section we describe the makeup of an entity, and the mechanisms provided for enabling transparent and lightweight communication between entities distributed around the system.
3.4.3 Defining the Behaviour of Entities We can identify at least four conceptual sources of behaviour for each individual entity within a VE: 1. Behaviour unique to an entity. Entities each have their own particular role within a VE, for example the ability of a stopwatch to record elapsed time. 2. Behaviour common to a range of entities. Often many entities share some aspects of their behaviour through being of a similar type. For example, each pawn on a chessboard is subject to the same detailed rules governing movement; these rules do not apply in the same way to other pieces which have their own restrictions. However, all the pieces have in common the rules determining whether they may be “captured”; a notion that is specific to chess pieces but not necessarily to other entities in the same environment. Thus a hierarchy of common groups can be determined. 3. Behaviour common to all entities in a particular world. We generally consider gravity to be a phenomenon associated with everything found in the world around us rather than as a property of each individual entity. This also applies to social constructs such as “monetary value”. 4. Behaviour that is dynamically required at runtime. If an entity becomes inhabited by a user for example, it will behave differently – being controlled by a navigation device and so forth – to when uninhabited. Similarly an entity that is “set alight” will suddenly have properties previously unavailable to it (for example, being able to set fire to neighbouring entities or raising the temperature of an environment). Commonly, VR systems define the behaviour of an entity by attaching pieces of code, often written in Java, Tcl or some other scripting language. With little specific architectural support, however, it is often laborious to code and difficult to ensure consistency. The DEVA
41
Inhabited Information Spaces
programming model attempts to improve the situation by taking an object-orientated approach to the definition of behaviour and by providing features such as introspection (the ability to ask an object what methods it has, what parameters they take etc.) and run-time combination of environmental influences with entity-specific behaviour. This essentially allows the traditional inheritance graph to be modified while an entity is “live” in an environment, adding or removing properties and behavioural traits as necessary. Our solution to the problem of merging the various sources of behaviour that comprise an entity is to use “characteristics”. In DEVA a characteristic is a collection of methods and attributes relating to a single concept that can be attached or detached from an entity at runtime. DEVA supports three types of characteristic: 1. Innate: behaviours that define an entity and make it different to those around it, for example its physical shape. 2. Imbued: behaviours that are offered to an entity when it joins an environment, but can be over-ridden by that entity by a more specific innate behaviour (for example, an entity lacking mass in an environment that requires such an attribute, may inherit an imbued characteristic that approximates a default mass from its bounding volume; an entity that has a better idea of what its mass is can provide the functionality itself). 3. Enforced: behaviours that must be processed in order for an entity to conform to an environment’s requirements. The methods and attributes that comprise the characteristic are divided into a single “object” and multiple “subjects” as outlined in the previous section. While typically a characteristic will contain both parts, some characteristics are entirely abstract and have no directly perceivable representation in the virtual environment, that is, they have no “subject” part. The researcher trying out new low-level ideas is free to write characteristics directly in C or C++ that interface to the VR kernel at whatever level is appropriate. More general users are free to use the library of existing characteristics to construct entities without concerning themselves with their implementation. We speculate also that this characteristic system has the added benefit of being usable from an immersive “graphical user interface” allowing the user to “mix and match” characteristics at runtime. In order to facilitate efficiently moving entities between worlds with different behaviours, they contain two lists of characteristics, one inherited from the environment, and the other containing its own innate behaviour. Traditional object-oriented inheritance is supported through characteristics being able to load other characteristics when they are initialised. The order in which the characteristics are searched enforces the correct precedence.
42
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Income method call
Deva Entity
011
011
Characteristic 1
method 1 method 2 method 3
method 1 method 2 method 3
Characteristic 2
Characteristic 2
Characteristic 2
method 1 method 2 method 3
method 1 method 2 method 3
method 1 method 2 method 3
Characteristic 3
Characteristic 3
Characteristic 3
method 1 method 2 method 3
method 1 method 2 method 3
Imbued
Characteristic 1
method 1 method 2 method 3
Innate
Enforced
011
Characteristic 1
method 1 method 2 method 3
Optional return valve
Figure 3.6 Structure of a DEVA entity, showing the ordering of method processing.
When a method is called upon an entity, the two lists are searched in strict order. First of all the list of characteristics given to the entity by the environment is scanned for methods marked as being “enforced”. If one with the correct name is found, then this is called. Enforcing methods allows an environment to ensure all entities contain a particular method that cannot be overridden. Next, the entity’s innate characteristics are searched. If the method is still not found then the environment characteristics are searched again for methods marked as “imbued”. These are methods given to the entity by the environment but which can then be overridden by the entity itself. For example, an environment may enforce the notion of “solidity” upon all its entities (“you can’t pass through walls”). It may also “imbue” all entities with a concept of mass with a default value estimated based on its volume, but which the entity is free to override should it prefer. This process is illustrated in Figure 3.6.
011
11
3.4.4 Methods and Filters The strict order in which characteristics get searched for methods leads to another useful concept, that of the filter. As well as methods just returning results, they can be declared to be filters. These are allowed to return a new method call as their result, which then continues to
43
Inhabited Information Spaces
propagate along the characteristic lists. This is useful in a number of situations. Say, for example, it was necessary to constrain an entity to a plane. The entity inherits the imbued method setXYZ from its ThreeDSpace environment. In this case the entity would define a characteristic itself that defines setXYZ as a filter. Since this would be found before the imbued method, the entity’s filter is able to constrain the coordinates to a given plane before returning a new setXYZ message with the new co-ordinates that then propagates through to the environment imbued setXYZ method. Another example would be storing an entity’s state prior to migrating the entity to another server node process or prior to a complete shutdown of the system. Each characteristic defines a snapshot filter that adds any state variables it possesses to the input message and returns a new snapshot message. The final characteristic then returns a restore message containing all the variables that is sent to the new entity. The new entity contains a restore filter in each characteristic that originally stored variables, which then unpacks the message. Continuous behaviour is supported by permitting characteristics to define a special method called “activity” which is polled intermittently. Activities are not subject to the usual calling precedence, but always polled if defined.
3.4.5 The Distribution Architecture DEVA is logically a client–server architecture, which to a first approximation provides a single definitive locus of control for the VE using its server component, with “mirrors” of the entities being maintained in each client process. Behind the scenes, however, DEVA pragmatically manages the delegation of control dynamically to the most appropriate parts of the system, thereby achieving the highest fidelity of perceptual and causal coherency attainable for the application at hand. The “server” is in fact a cluster of processors running identical processes called “server nodes” that together form a single multithreaded parallel virtual machine capable of processing large numbers of entities. The intention is that the server provides a computing resource for multiple virtual environments, and maintains a far heavier processing load than any one user’s client could manage at any one time. A networking layer provides lightweight position independent messaging between entities. Entities are created in and managed by the server node processes, and client processes – such as a visualiser or user application – connect to the server to interact with and obtain state information about the entities. The server is persistent: it remains alive, processing any entities regardless of whether or not any clients are connected. Administrative tools exist to simplify the start-up, monitoring and shut down of the parallel server.
44
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
Creating and Addressing Entities Each server consists of a (large) fixed number, M, of virtual servers. The M virtual servers are trivially mapped using a lookup table to the N server node processes that comprise the server. This conversion takes place at a low level within the system and essentially hides the configuration of the server. An arbitrary virtual server is chosen to create and manage each entity (currently a random virtual server is chosen but this selection process could take into account loading factors). Each server node potentially manages multiple entities. Each entity is assigned a unique “pool ID” – an offset into the list of entities managed by a given server node. The location of an entity in the server is uniquely defined by the virtual server and pool IDs. This pool ID is not strictly necessary and is provided for efficiency only, since each entity has a unique name that can be searched for in the list of entities managed by the server node. When an entity is created a hash function is applied to the entity’s name to obtain a second virtual server. This virtual server manages the name of that entity, that is, it definitively knows the virtual server of where the entity is actually located and its pool ID. The name of the entity and the entity itself are managed by separate virtual servers. The same hash function is used throughout the system allowing any DEVA process to obtain the location of a named entity. This data is cached for future use. Entity name to location lookup, while a lightweight process, is a central and frequent task in a distributed system. A scheme where this load is spread equally across all server nodes is therefore advantageous. The main advantage of the addressing mechanism is that it allows entities to dynamically migrate across server nodes to help balance the processing load. When an entity moves it only needs to inform its name manager of its new location. DEVA processes – which now contain out of date cached data – will receive an error the next time they try to communicate with the entity at its old location. This error is trapped internally and the new location of the entity is obtained from the name manager; the originator of the communications with the entity is oblivious to the migration. The name manager can always be relied upon to know the correct information and its location is trivially obtained.
Server Reconfiguration It is also possible to migrate the names managed by a given virtual server onto a different server node by updating the virtual server-to-server node lookup tables in every client and server process. The migration of both entities and name management allows server nodes to be dynamically added and removed from a running server.
45
Inhabited Information Spaces
Networking Protocol Currently, standard TCP/IP point-to-point socket communication is employed although some work has been undertaken investigating multicast, since for local area networks at least it promises improvements in performance for our application. Although not a strict requirement, the communications strategy is based upon the assumption that inter-server node communication is fast compared to server–client communication. For example, the server nodes are connected via a dedicated network or protected to some extent from superfluous traffic by a bridge; while clients connect to the servers via a high-traffic shared LAN (local area network) or modem connection.
3.5 System Support for PlaceWorld In this final section, we describe how a number of the artefacts and features of PlaceWorld, informed by the previous multimedia installations and ethnographic studies, were supported by the technology of the DEVA system.
3.5.1 Menus Every entity in PlaceWorld is able to respond to the user in one way or another. When the user touches the entity on the navigator screen, a menu appears showing the available features. Most entities, for example, can be picked up and put in the user’s “pocket”, and in this way moved to another part of the same world, or indeed somewhere completely different. Even entities that were coded before PlaceWorld was conceived have this feature “enforced” upon them by PlaceWorld’s master environment giving a degree of coherence to the interface. Other entities hand-crafted for the world, such as the generators or noticeboards offer more sophisticated behaviour via the “innate” mechanism. Filters are used to cascade through the possible methods provided by each entity, combining the imbued, innate and enforced facilities available to the user and presenting these graphically at the interface.
3.5.2 Access Model PlaceWorld’s access model demonstrates a more sophisticated use of the characteristic architecture. In order to maintain a degree of order in the environment, users are able to specify rules that govern the use of entities they have placed in the world. The mechanism for describing these rules is beyond the scope of this chapter, however the method of
46
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
enforcing them can easily be explained. The PlaceWorld master environment enforces on its contents (which includes all its sub-environments and their contents) an “authenticate” filter that responds to all possible method calls. This effectively intercepts communication of any kind arriving at any entity in the world from any source. The authenticate filter examines the incoming method call, and matches the credentials of the caller against whatever rules are currently being requested by the individual entity via an innate method higher up the call chain. If the attempted invocation is valid according to these rules, the filter “releases” the original method call to cascade through the remainder of that entity’s methods, with the caller effectively unaware of the validation process that has occurred. Methods that are not authenticated are returned to the caller, which in turn has a method imbued by the environment to cope with such “return to sender” results. As before, entities that were coded without any knowledge of the access model automatically pick up a minimum amount of functionality to enable them to interact with the access model in a suitable manner; no extra effort on the part of the implementer is necessary, while those entities aware of their surroundings are able to easily extend the rule base in bespoke ways.
011
011
011
11
3.5.3 Exploiting Subjectivity PlaceWorld has comparatively relaxed synchronisation requirements. Interaction takes place mostly at a social level, and there is currently no detailed manipulation of shared entities in real-time such as might be necessary in a collaborative CAD package. The subject/object distinction embodied by DEVA allows us to take advantage of this by programming quite relaxed synchronisation routines for these entities. Though it is necessary to synchronise, say, two users attempting to pick up the same entity at the same time, or to reflect the change of text on a noticeboard to all users in a timely fashion, it is not necessary to synchronise the gentle floating effect of the PlaceWorld artefacts as they hover above the groundplane, nor, as another example, the swaying of the glowing hyperlinks in the sky. The public exhibition of PlaceWorld at the I3Net conference in Sweden and at the Sixth Doors of Perception conference in Amsterdam demonstrated that the relaxation of synchronisation and the use of subjectivity could be pushed further than we originally expected. Due to a technical problem with the graphics drivers at the time, it became necessary to use an extra rendering client to generate the display on the touch sensitive panel of the PN-AV device, rather than using the camera as was originally intended: the projected display on the external cylinder showed the user’s view of the world, and the display on the touch screen showed the same view, but augmented with the controls and menus etc. The idea of relaxed subjective behaviour had always been intended to overcome
47
Inhabited Information Spaces
network lag where users were geographically distant, and unable to see each other’s views into the environment: here, unusually, the same user was able to see one view on the touch screen in front of them, and “the same” view on the projected cylinder. The potential power of the subjective set-up was demonstrated when not a single user of the installation noticed that there were significant variations between the two displays that they could now see simultaneously: where their attention was focused (changes of text, avatars moving around the environment, entities coming and going), all appeared to behave normally. The aesthetics of the floating entities, which were in fact moving independently of one another with no attempt to synchronise the views, went unnoticed. More dramatically, one of the artefacts, a version of ‘QSPACE’ (Pettifer et al., 2001), is a graphical network much in the same vein as the Web Planetarium described in Chapter 2. To preserve bandwidth the distributed version of this artefact in DEVA was implemented with the force-placement routines as subjective behaviour, and the weights and connectivities that define the structure as objective and synchronised behaviour. What was overlooked during the implementation was that in the force placement routines, a small random factor was introduced to the positions of the nodes prior to application of the algorithm, as this seemed to produce more aesthetically pleasing results. The implication of this, however, was that although the logical and gross structure of the resulting 3D network was the same for each run of the algorithm, these small perturbations in the initial state of the network resulted in slightly different placement solutions. For the several weeks of implementation, and during the public demonstration of PlaceWorld, this went entirely unnoticed, even though, with hindsight, it was clear that the various visual representations of the QSPACE being manipulated by the collaborating users were very different. The fact that the logical structure of the artefact was coherent across views appeared to mask the visual differences in orientation and absolute positioning of the nodes.
3.5.4 Becoming a Place Where Places Meet PlaceWorld aimed to bring together many diverse virtual environments, not just trivially by allowing a user to “switch” from one application to another, but by making it possible for artefacts created in one place to be used elsewhere, analogous to cutting and pasting a spreadsheet into a word processor document. It was decided to integrate a number of existing virtual environments, including other multimedia art installations, into the PlaceWorld system. Artefacts and applications that are written with the DEVA framework in mind have this facility by their very nature; more challenging, however, was the problem of making legacy applications, such as the Legible City, or the Memory Theatre VR (an installation by Agnes Hegedüs) work within this
48
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
PlaceWorld, and the Evolution of Electronic Landscapes 3 ●●●●●●●●●●●● ● 1 ●●
framework (this last application, in particular, is interesting since it was written using Silicon Graphics Performer, rather than raw OpenGL, and therefore was already part of its own “system”). Figure 3.7 shows this and other environments embedded within PlaceWorld. MAVERIK’s (Hubbold et al., 2001) agnostic approach to data structures made this integration at the graphical level relatively straightforward, even to the extent of enabling the low-level graphics context to be shared between Performer and native MAVERIK objects, and DEVA’ s ability to combine behaviour at runtime based on enforced and imbued characteristics allowed legacy applications such as the Memory Theatre VR to pick up appropriate interaction features such as the menus and the ability to be carried around by a user.
011
011 a
b
c
d
011
11
Figure 3.7 Images from PlaceWorld and its embedded environments: (a) the QSPACE; (b) the Legible City; (c) Memory Theatre VR; and (d) the Advanced Interfaces Group’s laboratory.
49
Inhabited Information Spaces
3.6 Conclusions The vision of an electronic landscape remains an exciting one, and as graphics technology and higher performance networking become more affordable and available, there are increasing opportunities for the kind of research and development described in this chapter. The iterative approach of the eSCAPE project has been an effective way of understanding the social, aesthetic and technological problems associated with making these worlds meaningful and useful to the general public.
Acknowledgements PlaceWorld was influenced in many ways by all the partners and participants in the eSCAPE project: too numerous to mention, this chapter draws upon much of their work and our thanks go to them all. Special thanks are due to Paul Arnold, Annika Blunck, John Bowers, Andy Colebourne, Andy Crabtree, Timo Fleish, Simon Gibson, John Hughes, Andreas Kratky, John Mariani, Gideon May, Craig Murray, Andreas Schiffler, Jeffrey Shaw, Adrian West, and the project’s co-ordinator, Tom Rodden. The work was supported by a grant from the European Commission.
50
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration
4
Olov Ståhl and Anders Wallberg
4.1 Introduction The constantly increasing amount of information and media available in electronic form leads to a growing demand for new methods for searching and browsing. Traditional text-based database queries can be limiting, often requiring a user to know exactly what it is she is looking for and to express this interest using predicates in a query language such as SQL. Furthermore, to be successful in locating the right information, a user will often have to be familiar with the standard interface metaphor of desktop computers and know how to use a mouse and keyboard efficiently. Approaches to improve access to online information and to visualise it in an intuitive manner have been under development for a long time. Examples of systems that display information graphically in three dimensions include VR-VIBE (Benford et al., 1995a), QPIT (Colebourne et al., 1996) and BEAD (Chalmers, 1993). All of these systems require (at least to some extent) the users to use navigation to access particular information objects, because objects may obscure each other or may be out of view. However, navigation within 3D spaces is known to be difficult (Ingram and Benford, 1995), especially if the navigation needs to be precise. In this chapter we describe The Pond, a system used to search for and visualise data elements on an engaging tabletop display. The Pond uses methods of unencumbered interaction and audio feedback to allow users to investigate data elements, and supports shoulder-to-shoulder collaboration with the physical Pond artefact mediating the collaboration between those people gathered around it. The user interface is based on an ecosystem metaphor, presenting data elements in the form of shoals of aquatic creatures inside a virtual 3D pond. The design makes use of a
51
Inhabited Information Spaces
static view of the information space, making viewpoint navigation unnecessary. Instead the information creatures move and form groups, allowing the user to easily identify related information and to distinguish results from different queries. The work draws heavily on our experiences in developing two previous systems, and concepts and approaches can be traced back to these systems. The Web Planetarium, a 3D interpretation of HTML structures in the form of a graph (Chapter 2), and the Library Demonstrator, an information landscape constructed from the contents of an online library database (Mariani and Rodden, 1999), were both interfaces to active online information. Common features of these systems were the spatial arrangement of data elements, navigation around these data elements and the introduction of new data elements into the display. While these systems were successful in presenting information to end users, they were not necessarily easy to use. When using the Web Planetarium, new information was only introduced as a result of explicit interaction (clicking) by the user, and novice users could be too shy to discover this fact and not load new information. The positioning of this new data could also be problematic. In the Planetarium the user must be observant otherwise she may miss the introduction of the new data element. The Library Demonstrator addressed this problem using animation and self-organising models to show the emerging relationships between information, and it is this approach we build upon in The Pond with its shoal metaphor. Navigation around the data used a point and click method, automatically transporting the user to (or close to) the selected object. The core of this navigation technique was to adopt an object centric philosophy where users were explicitly freed from the overhead of having to manage their navigation and movement in three dimensions. This restriction of the overall freedom of movement meant that users were able to focus on the exploration of the information space. However, users still had problems navigating the structures. It could range from getting completely lost to not being able to look upon the data in the way they wanted. These systems were also subject to single user interactions and did not encourage a social atmosphere for the exchange of gathered information. When we started the work on The Pond, the objective was to design a multimedia user interface where users without any prior knowledge or acquired competence should be able to easily handle both single as well as groups of objects in an affordable and easy way. The objects in turn should be able to represent any type of information or media. They should be able to present themselves in a natural fashion at the user’s convenience. Also, they should disappear in non-obtrusive fashion when no longer needed. The user should be able to select, move, sort and explore objects of interest without having to confront the rigid hierarchy that is the hallmark of traditional file handling and database applications. Instead of using colour, form or position to indicate group or class
52
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
membership we wanted to use motion dynamics of objects to indicate these properties. If new information is requested, objects should just float to the surface and present itself to the user. Objects no longer relevant should slowly sink to the bottom and quietly disappear. The viewpoint should stay fixed above the virtual pond surface, thus making it unnecessary for the users to engage in navigation within the information space. An observation made early in the design process was the difference in the behaviour of users when confronted with a horizontally placed display vis-à-vis a vertically placed one. When a group of people gathers in front of a vertically placed display – be it a big screen TV or a wall projection – immediately a somewhat authorial situation tends to develop. The person in control of the display content is perceived as a teacher or lecturer. The rest of the group will play the role of an audience or school class. When, on the other hand, the display or projection area is positioned horizontally, people will gather on a more equal basis. With inspiration from as disparate sources as roulette tables, billiard tables and the military’s “classical” tactical war gaming board, it was decided that The Pond should use a horizontal display and that none of the sides of the physical artefact should be more important than any of the others. Tabletop displays have been in use for a number of years now, for example in visualising information (Kreuger and Froehlich, 1994) for command and control scenarios (Chase et al., 1998) and augmenting physical objects (Ishii et al., 1999). The table provides a natural centre for interaction to take place around and encourages collaboration between the users while they are interacting with the table (Benford et al., 2000). The developments in plasma display technology coupled with touch-sensitive surfaces now make it possible to dispense with potentially clumsy projected displays in favour of a neat, compact display. Interaction with The Pond is object based, using physical tags to load information into the application by placing them around the edge of the table (Ishii and Ullmer, 1997) (Ullmer et al., 1998). Unlike other approaches where the physical objects are placed directly on the table and manipulated (Underkoffler and Ishii, 1999; Fjeld et al., 1999; Schäfer et al., 1997), the tags used in The Pond are passive. They make it possible to load and store information, but do not manipulate this information any further. Interacting with the contents of The Pond is supported through direct manipulation of the virtual pond objects (Shneiderman, 1983) using the touch screen display. Typically, interactions with the tabletop systems discussed above make use of stereoscopic glasses, data gloves, and magnetic position trackers. These techniques are not used in The Pond as our aim was to create as easy and direct an interface as possible and not to encumber the user with devices needed to experience the material presented. In the remainder of this chapter we will describe The Pond as it stands today, details of its implementation, and observations from a study of system in use. We conclude with a discussion on some design choices and their implications on the potential utility of The Pond in different settings.
53
Inhabited Information Spaces
4.2 The Pond We now describe The Pond system in detail, examining its construction, both in terms of the physical artefact and the underlying software, and the way in which it is used.
4.2.1 The Pond Ecosystem Metaphor The Pond uses a user interface based on an ecosystem metaphor. The objective was to present an aesthetic that would hide the work chore aspect and act as a complementary backdrop for dialogue. When interacting with The Pond, the users see a 3D presentation of a virtual pool or pond (see Figure 4.1), an aquatic environment in which the information objects resulting from queries are presented as shoals of marine creatures. The visual presentation is complemented by a sound environment, consisting of a number of bubbling, splashing and whirling sounds, indicating various active processes inside the pond virtual environment. Each information creature has a simple navigational behaviour that governs its movements within the virtual pond. The behaviour is inspired by the boid flock model described in Reynolds (1987). The basic steering rules are: ● ● ●
Avoid collisions with nearby creatures. Match velocity with nearby shoal mates. Stay close to nearby shoal mates.
Figure 4.1 The Pond virtual environment.
54
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
These simple rules are then combined with additional influences like curiosity (a creature might temporarily break away from its shoal to “examine” its surroundings) and competitiveness (an urge to be in the lead), resulting in a very dynamic and fascinating pond environment, where shoals of information creatures move around in tight groups, avoiding each other as well as the pond walls, and reacting to interactions from the users. The movement of a shoal is entirely dependent on the combined navigational behaviour of its creatures, that is, there is no shoal “intelligence” that determines where the shoal is going or how its creatures should behave in certain situations. The argument for using a flocking algorithm for positioning the information creatures is not based on aesthetics alone. Our earlier experiences in information visualisation mixed with an ambition to explore a somewhat alternative user interface made us choose a self-animating system for the data elements. Not only animating during the creation/insertion stage but during the whole time the elements were available. In addition, the human visual system is quite good at separating objects with common velocity vectors making it easier to identify group belongings in a crowded environment. The Pond does automatic garbage collection, which means that one or several of the shoals that exists in the virtual pond may be removed. This may happen, for instance, if the environment is too crowded or because a particular shoal has not been interacted with for a long time. A shoal that is selected for removal by the system will sink down towards the bottom of the virtual pond, where it will disappear. However, should a user interact with a sinking shoal, for example, select one of its creatures, the shoal will return to the surface and another shoal might be removed instead.
4.2.2 The Pond Example Application
011
011
11
To investigate the concepts behind The Pond we developed an application that allows users to search for and browse music content on the Amazon.com web site. The music theme was chosen to maximise The Pond’s visual and sonic impact, being a theme common to most potential end-users. However, The Pond is not limited to this domain and is adaptable to other database resources. A search task is initiated by a user providing a keyword string, for instance the name of an artist, group, or a musical genre. The Pond communicates the keyword to Amazon.com and presents the resulting album hits as a shoal of creatures, each representing one specific album. The virtual creatures are represented by simple geometric shapes, which are texture mapped with the albums’ cover images. By interacting with a creature, users are able to access the album data (e.g., artist name, album and track titles, etc.) and play samples from some of the tracks. The virtual pond itself consists of a 3D model of a deep, narrow shaft that extends down from a watery surface.
55
Inhabited Information Spaces
The music-oriented Pond application was selected since we wanted an application that would allow us to explore and evaluate the ideas behind The Pond without having to implement a lot of advanced search functionality, and which would also be interesting and enjoyable to use (e.g., looking for and listening to your favourite music). Furthermore, the size of a “typical” query result was expected to be less than 50 items, which would allow us to present a number of simultaneous shoals in the virtual pond environment without making it appear too crowded.
4.2.3 The Hardware Platform The physical Pond artefact has the form of a desk, on top of which a large touch-sensitive plasma display is horizontally placed. On top of the display surface is a wide wooden frame with an irregular curved outline, representing the bank of the virtual water pond rendered on the display. The frame is covered by pieces of thick carpet so that users standing around it can comfortably lean over the display when interacting with the virtual pond environment (see Figure 4.2). Built into the frame are a number of speakers that are used to output various sounds and music samples. The use of audio is an important
Figure 4.2 The Pond desk.
56
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
feature in interacting with The Pond, and the sound system consists of several devices including a sampler, sub-woofer and amplifier. The frame also encloses several RFID tag readers. Each reader is entirely embedded into the frame carpet and uses three light-emitting diodes to indicate its position and state to the users. Users may initiate queries by placing a RFID tag on such a reader. The tag’s identifier, sensed by the reader, will identify a query keyword or phrase and the query will be initiated.
4.2.4 The Software Platform The Pond software platform consists of two different components (see Figure 4.3): ● ●
The visualiser application renders the view from the virtual pond environment on the plasma display. The pond manager application accepts query keywords from the users and communicates these to the Amazon web server. The resulting information is used to introduce and control shoals of creatures in the virtual pond.
The visualiser and the pond manager are built using the DIVE (Chapter 12) distributed VR system from the Swedish Institute of Computer Science (SICS). The 3D virtual pond is in fact a DIVE virtual world, shared by the visualiser and the pond manager. When the pond manager Tag readers
Speakers
011 Dive pond virtual world
Pond Manager
Query and result via web services API
011
11
Amazon Web server
Visualiser
Figure 4.3 Overview of The Pond software components.
57
Inhabited Information Spaces
application is started it loads a file containing a model of the pond graphical environment. Initially the virtual pond is empty, that is, it contains no information creatures since these are only created as the result of user queries. When the visualiser is started it joins the world created by the pond manager and will receive a state transfer containing the graphical environment. From this point on the two applications share the pond environment, exchanging DIVE messages to notify each other of any world changes (e.g., the introduction of information creatures or creature movements). The visualiser is responsible for detecting user interactions on the plasma display’s touch sensitive surface. If a user clicks on a creature, the visualiser distributes an interaction event that will be received by the pond manager. After examining the event, the manager determines the appropriate action to take (if any), which typically involves some change to the virtual creature (e.g., a change of appearance or position). The change will generate a new DIVE message, which will be received and handled by the visualiser, thus making the world change visible to the users. The visualiser uses a number of rendering plug-ins to enhance the visual appearance of the pond environment. A ripple plug-in creates ripples and waves that graphically deform the perceived environment when fingers are pressed on and moved over the touch screen of The Pond. A second plug-in is responsible for generating a caustic lighting effect on all objects in the water. These plug-ins use hooks in the DIVE renderer and operates on the image it generates. User input in the form of query keywords is handled by the pond manager. Queries are initiated either by direct request, that is, a user placing a tag on a reader, or more indirectly through inferred user interest. The manager continuously monitors the tag readers embedded in The Pond table and immediately senses any change in their status (i.e., RFID tags being added or removed). To get data requested by these queries the manager uses the Amazon.com web services (Amazon.com, 2002). With its web services program, Amazon.com offers third party web sites and applications the ability to search or browse Amazon’s product database. The manager retrieves information like album titles, artist names and URLs to cover images through the web services using XML over HTTP. This information is then used in the process of creating representations of individual albums and shoals representing more than one hit on a query subject. The pond manager also handles the audio output to The Pond table speakers, which is described further in Section 4.4.
4.3 Interaction Users standing around The Pond table are able to interact with it in several ways to perform various tasks. In the absence of keyboard or mouse devices, the users perform most interactions by tapping or
58
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
stroking the touch-sensitive display surface. Furthermore, prepared tags that are spread out on top of The Pond frame allow users to input information about queries without having to type on a keyboard. Queries are initiated by placing tags on the tag readers in table frame. Since each available query tag has a sticker attached to it showing the keyword text or an image of an artist or group, users are able to determine what keyword a certain tag represents (see Figure 4.4). When a tag is placed on a reader, the reader senses the tag’s unique identifier, which is pre-mapped to the keyword. As soon as a query is initiated, an empty shoal appears inside the virtual pond, representing the ongoing query. The shoal is indicated by a circle and a text string specifying the query keyword, as seen in Figure 4.5a. The circle will commence to float inside the pond environment, bouncing off the walls and avoiding other shoals. When the query results are initially delivered from the Amazon.com web server, creatures start to appear inside the empty query shoal. Each such creature represents an information element from the query result; in this case a CD album. As soon as a particular result creature has been created, it will begin to move around inside the virtual pond. However, since all creatures resulting from a particular query stay close together, different query shoals are easily identifiable, even with a vastly populated pond environment.
011
011
11
Figure 4.4 Query tags representing the keywords “Frank Sinatra”, “Billie Holiday”, “ABBA” and “Kraftwerk”.
59
Inhabited Information Spaces
a
b
c
d
e
f
Figure 4.5 Actions and events in The Pond: (a) To the right is an empty shoal, representing an ongoing query using the keyword “Nitin Sawhney”. To the left is a shoal of creatures representing a finished query. (b) Two shoals representing the results of queries using the keywords “Zeppelin” and “Queen”. (c) To the right a selected creature dragged a bit from its flock. (d) Playing sample one (out of five available) on the album “Siamese Dream” made by Smashing Pumpkins. (e) Three creatures dragged into a creel, limited by the pond walls and two buoys. (f) Getting related albums to an album called “Lamb” made by the group Lamb.
60
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
When all the results have been delivered and the corresponding virtual creatures created, the shoal circle label will change to only display the query keyword (see Figure 4.5b). Shortly thereafter the shoal circle and label will disappear, leaving behind only the creatures visible to the users. However, it is possible for users to make the circle and label visible again by tapping on any of the creatures belonging to the shoal. Each shoal member represents an information element that is part of the result of the corresponding query. The information includes name of group or artist, album title, and URL to the album cover image, as well as URLs to a number of short Real Audio® samples of some of the album tracks. Users are able to access this information by manipulating the creatures in different ways. A user selects a creature by tapping on it on the display, and is then able to see the information identifying the artist and title of the corresponding CD. This information is presented as a virtual text string, encircling the 3D creature and moving alongside it (see Figure 4.5c). The text will only be visible for a brief period (around five seconds) and will then disappear. The frame of the creature will become green to indicate the selection, and will remain so until changed back to white again when another creature is selected. Tapping once more on an already selected creature makes it float up to the surface and initiates playback of the corresponding album’s first Real Audio sample (see Figure 4.5d). By using the Real Audio player and the sample URL, the audio data is streamed over the network from the Amazon.com web site and output to the speakers embedded in table frame. By tapping repeatedly on the creature being played, users are able to step through the sample tracks available for that particular album. The text encircling the creature displays the number of the sample being played as well as the total number of available samples. By default, the viewpoint in the virtual pond environment is placed above The Pond surface, looking down, and at a distance from where the view is always guaranteed to include all the existing creatures. In this way the users are able to get a good overview of all the activity within the environment. However, this also constrains the creature representations to be rather small (as seen on the plasma display), which might present a problem when trying to identify the creature graphics of the CD albums they represent. In order to allow users to get a closer view of one of the shoals, a zooming mechanism allows for the translation of the virtual viewpoint to a position close to a shoal centre. The viewport is just large enough to encompass the whole shoal, with the benefit of making the creatures in that shoal together with their associated text strings and images appear larger. As a result, other shoals may end up out of view, not visible from the new viewpoint position. Another feature of the zooming mechanism is that, while zoomed, the viewpoint is attached to the shoal, which means that it will move as the creatures within the shoal moves. In this way the
61
Inhabited Information Spaces
viewpoint will always stay centred on the chosen shoal, even as this shoal changes its position within the virtual pond. While zoomed, the shoal creatures may be interacted with in the same way as before, for example, selected to initiate replay of the music samples. The zooming mechanism is triggered when a tag is placed on a reader while the corresponding shoal already exists in the pond environment. Removing the tag from the reader will reset the viewpoint to the default overview position. Thus, if a user places a tag associated with the keyword “Dylan” on a reader while a “Dylan” shoal exists, the viewpoint will change to a position close to this shoal, and stay there as long as the tag is on the reader. Whenever the tag is removed, the viewpoint is reset. Only one user at a time can use the zoom mechanism. If a zoom is active while a user initiates another zoom the viewpoint state will not be overridden and the second request is ignored. As users initiate more and more queries, older shoals may have to be removed in order to incorporate newer ones. To prevent a particular creature from being removed from the environment, it is possible for users to move individual creatures to safe areas, called creels. This is done by touching the creature with the finger, and then dragging the finger (and thus the creature) along the surface and releasing it over the creel area (see Figure 4.5e). Creels exist in several places in the virtual pond environment, close to the tag readers. Once inside a creel, a creature is constrained to move only within the creel boundaries. By moving several creatures, possibly from different shoals, into a creel, a selection shoal is formed consisting of creatures that a user finds interesting for some reason. Since this particular action shows an interest from the user in a specific data element, this instantiates an extra functionality. A more focused query is automatically launched pertaining to the corresponding album (usually resulting in fewer hits and consequently a smaller shoal), thus further populating the environment (see Figure 4.5f). In the case of Amazon.com the system issues a related albums query resulting in items that have a high chance of being interesting to the user as well. The creatures in the creels may be interacted with in the same way as other creatures (e.g., tapped on to play music samples), the only difference being that they won’t be removed from the environment as long as they stay inside the creel. Creatures that are dragged out of a creel will return to their native shoal, or form a new shoal if their native shoal no longer exists. By using a recordable tag, it is possible for a user to save the contents (i.e., creatures) of a creel selection shoal. A recordable tag is a RFID tag which is not pre-mapped to a search keyword. Placing such a tag on a tag reader next to a creel creates an association between the tag and the creatures within the creel area. When the tag is then removed from the reader, the creel’s selection shoal will disappear from the environment and may be regarded as being stored on the tag. By placing the same tag on a reader at a later stage, the creatures of the “saved” selection will
62
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
reappear, added to the corresponding creel shoal. Thus, creels and recordable tags allow users to save references to one or several albums of particular interest, and to gain quick access to these on a later occasion, as well as to share references with each other.
4.4 The Pond Audio Environment The sonic environment of The Pond consists of two different parts, the soundscape and the interface sounds. In accordance with the ecosystem metaphor, the nature of the soundscape is founded on a family of aquatic whirling sounds and a deep, obscure mumbling giving the impression of the data elements ascending from an abyss of ooze or mud. These ambient sounds fade out when samples from a selected creature starts to play, and fades smoothly in when the music stops. The interface sounds acts as a feedback mechanism to indicate user interactions like selecting/unselecting, clicking, dragging etc. This collection also originates from a number of concrete water sounds. Samples from different types of bubbles, a dripping tap, whirlpools etc. are heavily processed to suit their particular purposes. Examples include: ● ●
●
●
●
When a user initiates a query, the appearance of the query shoal is accompanied by the sound of sluggish bubbles rising from the bottom. When a sound file is retrieved over the network, the waiting time is masked with a bubble vortex that after a few seconds is smoothly merged with the music sample. When the user removes a tag from a reader, the action is accompanied by sounds of a cluster of bubbles being rapidly inhaled by The Pond itself. When the user draws his finger across the touch screen to drag an object, a glass organ sound reminiscent of drawing a finger along the damp edge of a crystal wineglass is heard. The visual zooming in of a shoal is illustrated by the familiar bubble sound gradually magnified through lowering of the pitch. When zooming out, the process is reversed.
Every time an interface sound needs to be played, the system will randomly choose a sound from a collection of sounds available for the particular interaction. There are ten query sounds, ten RFID sounds, ten dragging sounds, etc. The idea is to give the impression of the sonic interface being somewhat organic and unpredictable. The precise spatial placement of every sound is achieved through a builtin high quality four-channel sound system. A subwoofer in the table foot produces a deep and suggestive bass. The computer-controlled software mixer makes it possible to physically move sounds around and to create expressive musical gestures.
63
Inhabited Information Spaces
4.5 Observations from Use To test out The Pond a number of sessions were held with external participants in the expectation that some qualitative assessment of the effectiveness of system could be made. After being presented with an introduction to The Pond and the example application by one of the developers, the participants were told to explore the system, using the available tags on the table frame. The focus of the studies was on the technology-in-use (Button, 1992), that is, the sequences of interaction within which The Pond came to be used in real time. The developers stayed close the whole time to answer questions and to help out when problems occurred. Most sessions involved two or three participants and lasted for about an hour. Video was used to record each session and from this material a number of observations were drawn. The general impression is that the participants had very few problems in understanding the basic principles of The Pond design and its operation, for example, how to use the tags to initiate queries or how to interact with the creatures in order to play sample music. On very few occasions did the developers have to repeat the instructions given at the start of the session. The Pond is engineered to support hands-on experience and ease of use, and most participants did engage in various activities, together and alone, such as selecting CDs, listening to samples, transferring CDs to tags, and transferring CDs from tags after only a short time. During a discussion after one of the sessions, a participant stated: “The main good thing about The Pond is that the interface doesn’t require any special level of computer skills for using it.” Not only did the participants get the grasp of The Pond fairly quickly, but were also able to envisage its use for practical purposes in every day settings. For example, one of the participants suggested: “Place it in a Virgin Megastore, it can be used as a jukebox. Perhaps you should specialise it more so it fits into a category like film, books, articles, and radio. And you should be able to tell The Pond which CD you want and then be able to put it into your shopping basket.”
In some sequences of interaction it is noticeable that the sound worked as functional feedback. After the participants had recognised the connection between the function and the sound, the sound worked as a support for the user in interacting with The Pond. This can be seen in a sequence when a participant drags a creature out from a creel. In the sequence the sound that is heard as the dragged creature passes the creel boundary makes it clear to the participant that she has succeeded and the creature is released. The problems that participants had during sessions were mostly related to the hardware, and especially the touch screen. A serious drawback with the large area touch screen technology we currently use is a limitation when it comes to detecting multiple simultaneous touches.
64
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
This sometimes caused conflicts when more than one participant was trying to interact with creatures at the same time. For instance, it was impossible for two users to drag creatures at the same time. Also, if one user was dragging a creature while another user tried to select a different creature, the selection would most often fail. The reason for this is that when two positions on the touch screen are clicked simultaneously, a position in between the two will be returned by the driver software, most likely causing the second user to miss the intended creature. Future multi-user touch screens will alleviate this shortcoming. In the present set-up we try to work around the problem by making most interactions single click based, dragging being the exception. Another problem with the touch screen was that of sensitivity. Sometimes when the participants where dragging creatures they seemed to unintentionally lose contact with the touch surface, causing the creature to stop following the finger and return to its shoal. What happened was that even though their finger was actually in contact with the surface, they didn’t put enough pressure onto it, which made the driver report that the interaction had ended. A number of “phantom clicks” (i.e., the driver reported a click when there was none) were also experienced which had a similar effect on the interaction. We expect most of these hardware problems to disappear in the future as the technology matures. One of the software mechanisms that caused some problems was zooming. It was evident from the video material that zooming could cause confusion if being performed by one participant when others where trying to interact with creatures in shoals different from the one being zoomed (as these shoals would disappear from the view). Zooming probably requires synchronisation of the users’ activities in order to be useful. As it turned out, the navigational behaviour of the information creatures became a source of fascination for many of the participants when encountering The Pond, and it was not uncommon to see them standing silently at the side of the display for some time, simply watching the creatures swim back and forth.
4.6 Discussion One of the most frequently expressed comments from users was the desire to be able to search The Pond in a more dynamic fashion, that is, to issue custom search queries, which is not possible through the use of the tags at the moment. The seriousness of this limitation depends of course on the application and its intended use. The Pond is not primarily designed to be used for the task of quickly locating and browsing arbitrary known information items on the web, in which case the use of the pre-configured query tags would probably be unacceptable. Instead, the focus is more on applications where the size of the data set is fairly small (i.e., a product catalogue) so that the query tags more or
65
Inhabited Information Spaces
less cover the whole search space. In addition, if the database is hierarchical or allows items to somehow contain references to each other, the users can traverse the data not only by issuing direct queries using the tags, but also by exploring the creatures already existing within The Pond (following the inter-creature links to discover other “parts” of the data space). In such settings the tags can be seen as a way to introduce information objects that probably won’t be the end result of a user’s query, but merely a starting point for further exploration. Examples include using The Pond in a record store setting where the top 100 albums are to be displayed and played, or in a hotel lobby where menus from local restaurants, museum information, etc., can be examined in a collaborative fashion. In the Amazon.com data source set-up, a way of search refinement and database traversal was available through the use of related album queries. This introduces items that have a high chance of being interesting to the user. Combined with the fact that items considered uninteresting will sink and disappear, this has the effect of gradually refining the ecosystem to contain more and more interesting items. So even though very specific selections are being used as “seed” material, the contents of the environment might develop through time to contain a mix of several general tastes, specific favourites and complete wildcards. Our aim when designing the interface was towards a non-intrusive form of interaction, avoiding introducing keyboard type input. The RFID tags do serve well in cases where a hierarchical, finite and discrete database structure exists, as the music database in our case. Notice that the environment is in the most part populated by relations and not exact matches, so we force exploration on the users. It proved at times frustrating for users not to be able to directly summon the artist or track of their choice, and at times gratifying in discovering alternative music of their liking. A solution that would in fact allow users to input exact search keywords or query phrases and which wouldn’t entail the use of some sort of keyboard function would be to use voice as input. An issue that came up during the user studies was the possibility of modifying or even replacing the example application to add new functionality or to support different types of media. For instance, instead of saving references to albums onto recordable tags, one participant wanted to be able to save the actual sample data onto some kind of portable device, for example, a PDA, which could then be used for playback. This could be achieved by using some kind of point-to-point (e.g., infra-red or Bluetooth) connection to transfer the audio data from The Pond to the PDA. Also, since DIVE runs under PocketPC™ it would be possible to develop a PDA application which would join the pond virtual world to be able to access the information (e.g., music samples) represented by the virtual creatures. Modifying The Pond to support a different type of content media, or even combinations of different media, would require changes to the pond manager application (see Section 4.2.4 above) in two different ways. First, it would require the pond manager to be able to
66
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Using a Pond Metaphor for Information Visualisation and Exploration 4 ●●●●●●●●●●● ● 1 ●●●
communicate and issue queries to a database service different from the one provided by Amazon.com (assuming that Amazon.com does not support access to data elements of the “new” media type). This would be fairly straightforward since the code responsible for handling queries is isolated in one specific module, which can be easily replaced. It is also possible to add a second (or more) module to exist in parallel with the first if there is a need to connect to more than one database service during a session. Secondly, to support new types of content media would require new rendering mechanisms to present the creatures to the users when there is interaction with them. As is the case with the current Amazon.com web service version and the RealAudio player, special information rendering that goes beyond small pieces of text, images and texture-mapped videos will probably need external renderers. These can be in the form of DIVE plug-ins or external applications (e.g., Windows Media Player to render video material). The pond manager application defines a number of events (e.g., OnClick) that are fired when creatures are interacted with and needs to display details of their associated information (e.g., album title in the Amazon.com application). By modifying or replacing the current handling of these events in the pond manager code, it is possible to change the way the creatures present themselves to the users. One of the consequences of using a static view into the pond environment is that the display space is restricted, thus limiting the number of creatures that can be presented simultaneously. As described earlier, garbage collection can alleviate the problem to some extent by removing old shoals when new ones need to be introduced, but it does not help if the new query result is too large to fit into the virtual pond by itself. Making creatures smaller in size will make more of them fit before the pond becomes to cluttered, but also makes it harder to distinguish the album images, read the text strings, etc. One possible extension of the shoal concept that could help in presenting large data sets is hierarchical shoals. Assume that a shoal could be made up of sub-shoals as well as information creatures, and that a sub-shoal was initially represented by a single graphical object, just like an ordinary creature. When clicking on a sub-shoal object it would expand to display all of its elements, some of which might themselves be sub-shoals, and so on. An expanded sub-shoal might collapse into the single sub-shoal object after a certain amount of time, or possibly as the result of an explicit user interaction (e.g., clicking once more on the sub-shoal object). In this way it would be possible to make available a large number of information creatures to The Pond users without having to display them all at the same time. The rules governing the creation of sub-shoals are probably application specific, that is, the criteria used to determine when to create a sub-shoal and which creatures to add to it. Another possible solution to handle large query results would be to make each shoal present only a limited set of the resulting creatures at one time, and then allow the users to somehow (via some shoal interaction) make the shoal present
67
Inhabited Information Spaces
the next set of creatures, and so on. This is the method used by most Internet search engines where the user is presented with a list of about ten hits, and is then expected to use the “next” or “previous” links to move forward or backward in the result material.
4.7 Summary and Future Work We have presented The Pond, a multi-user system for browsing information (currently CD record data) on the Web using an engaging tabletop display. Users input musical search keywords (typically names of artists or groups) using RFID tags and are presented with matching Web information in the form of shoals of aquatic creatures inside a virtual 3D pond. The virtual pond environment is presented on a big touch-sensitive plasma display, which is placed horizontally to better support shoulder-to-shoulder collaboration between those people gathered around it. By touching the surface of the display users can interact with the creatures to access the information they represent, for example, play music samples from the corresponding CDs. A series of user studies have provided initial insights into the utility of The Pond. The results indicate that the device and metaphor are easy to understand and use, but also identify a number of problem areas. For instance, the touch sensitive display doesn’t currently support multiple simultaneous interactions, which sometimes caused the users’ actions to interfere with each other. The Pond has been demonstrated to members of the research community as well as to the public on numerous occasions. The feedback and observations from these sessions have been used to refine The Pond under an ongoing evolution process. Future ideas include support for multiple simultaneous interactions on the display surface, using a voice input system for entering search keywords, and using PDAs to extract and input information to and from The Pond environment.
Acknowledgements We would like to thank Lennart E. Fahlén, Jan Humble, Jenny Lundberg and Jonas Söderberg for their participation in the development of The Pond system. We would also like to thank Adrian Bullock for his contribution in documenting The Pond work as well as Andy Colebourne for his Pond-related efforts within the eSCAPE project. This work was undertaken as part of the Swedish Research Institute for Information Technology (SITI) funded Electronic Landscapes Project (ELP). We would like to thank all those who have experienced The Pond and provided us with invaluable feedback. The Pond work has also been described in (Ståhl et al., 2002).
68
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
Part 3 Mixed Reality Environments
011
011
011
11
69
This page intentionally left blank
011
011
011
011
11
City: A Mixture of Old and New Media
5
Matthew Chalmers
5.1 Introduction The majority of the chapters in this book describe what might nowadays be called “traditional” inhabited information spaces: collaborative virtual environments (CVEs) or virtual worlds. Although not all CVEs centre on computer-rendered 3D graphics, the paradigmatic CVE does so. Shared 3D virtual environments are emblematic of CVE research, but have only gained public acceptance in the form of computer games. The focused engagement in such games is designed to fit with the closed world of the virtual environment. A player can become immersed in a game – closed off from the “real” world – by attention as much as by apparatus. A personal computer (PC) at home can be as engaging as the head-mounted displays and immersive projection technologies in research labs. However, even a single-player non-networked game may be a resource for social interaction, for example played by one person while friends and family shout advice from the sofa, order pizza by phone and slip into the kitchen to get more drinks. A computer game is a resource for far more social interaction than the software’s architecture may suggest. In general, the wider context of use is hardly modelled or represented in the system. Games’ internal data structures are designed to be decoupled – that is, closed off – from the other media people use in everyday life, and this decontextualised design approach has worked well in this domain. Although many households, schools and workplaces have computers that could support 2D or 3D virtual environments, such CVEs are rarely used as a medium of family interaction, education or work. One reason for this may be the decoupling from the more traditional interactions of family members, the overall educational activities of the school and the business of the workplace. The information within the CVE would have to correspond with each user’s wider context, and this would require sensing and tracking of users’ activity in media beyond that of the computer. To paraphrase part of this book’s introduction, a large amount of
71
Inhabited Information Spaces
our activity relies on the knowledge of what other people do, and what people do in the home, street, school and workplace involves many noncomputational media. However, CVEs are designed on the basis of narrowly focused engagement, decoupled from their users’ wider context of family, friends, learning and work. Other than in games, CVEs’ decontextualised design approach has not led to popularity or widespread use. As the introduction to the book also points out, a number of researchers have begun to work on inhabited information spaces that are more “out in the world” than traditional CVEs. A rhetorical example the editors give is “a system that enables co-located groups to co-operatively work with information by using a display projected onto physical artifacts”. An IIS may include tangible artefacts in more traditional media, such as urban models and interaction devices made from wood, wire and plastic (Underkoffler and Ishii 1999), or tiles and pages made from toner and paper in augmented reality systems such as MagicBook (Billinghurst et al., 2001). The work discussed in this chapter is intended to go further in this direction. As part of the Equator interdisciplinary research collaboration (www.equator.ac.uk), the City project explores digital spaces that are peers with others, rather than a digital space that is the primary focus or locus of activity. For example, in our system, one person’s use of a 3D VR model of a museum exhibition is coupled with another person’s use of hypermedia describing the exhibition as well as a third person’s use of the traditional “bricks and mortar” exhibition. No one of the three is primary; each is part of the context of the other two. We combined the media of traditional exhibitions, mobile computers, hypermedia and virtual environments in one design, and support interaction between people using different subsets of this heterogeneous set of media. The project was initially theory led. A number of theoretical issues were outlined in a discussion document, and then exemplified by scenarios of technology use. As the project grew and developed, our theoretical issues, design scenarios, system development, evaluation of pilot trials, and observational studies all affected each other. Although I sometimes campaign for the rule of theory, not one of these areas is primary. Instead, each is part of the context of the others. The project has always aimed to get out into the streets of the city, but we decided to begin our work in a more controlled setting: the Mackintosh Interpretation Centre, a permanent exhibition devoted to the life and work of Charles Rennie Mackintosh (1868–1928). Mackintosh was a Glasgow architect, designer and artist, and several of his buildings and other institutions related to his work stand in the city. Often simply called the “Mack Room”, the Centre is comprised of textual and graphical displays with some original artefacts, as well as over 20 screens presenting video and interactive digital material. The Mack Room is in The Lighthouse, Scotland’s Centre for Design, Architecture and the City (www.thelighthouse.co.uk).
72
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
More generally, we are exploring the way that digital information is just another part of an interwoven set of media, the use of which constitutes inhabiting the city. Unlike traditional CVEs, we aim for systems coupled with and contextualised in everyday activity, and hence in accord with contemporary theory of the use of language and space. A basic theoretical premise is that we can only use digital media because of such interweaving and interdependence, and we are looking for ways to increase and take advantage of the interdependence of traditional and new media. This theoretical standpoint is set out in the next section of this chapter. The subsequent section outlines a system design based on this standpoint, followed by a discussion of user experience in trials of an implemented research prototype. Some of the details of our ongoing and future work are then outlined before a concluding section that offers more general reflections on the project.
5.2 Theory This section focuses on conceptions of space and the media associated with work, and how we often conceive of space as a medium that stands above or apart from others. It is this usually implicit assumption that lets us talk of information spaces as being “inhabited”. I would like to present an opposing view that treats information spaces as merely one medium among the many used in everyday life. My approach is based on experience with information visualisation and virtual environments, as well as some borrowing of structuralist semiotics (Saussure, 1983; Nöth, 1995) and philosophical hermeneutics (Grondin, 1994). Part of this latter work was set out in a recent paper in the Journal of Computer Supported Cooperative Work (Chalmers, 2003). When discussing work, and designing systems for remote collaboration, we all too often concentrate on emulating the spatial aspects of the workplace, for example modelling spatial forms and supporting remote communication that appears to be like face-to-face interaction. All design has to be biased in some way, and the bias towards space in CVE research may be due to it being technologically led more than sociologically, semiologically or philosophically led. The arrival of cheap graphics hardware and the eye-catching novelty of 3D images gave rise to a good deal of work that focused rather narrowly on the construction of rather decoupled and decontextualised information spaces. This is true in some of my own work over the past twelve years, ranging from Chalmers (1991) to Morrison et al. (2002), and there are strengths, weaknesses and alternatives to such a bias (Chalmers, 1999). Newer technologically driven research is weakening or revealing CVE research’s implicit assumptions of space’s primacy and independence. Many of the characteristic design principles and assumptions were established before the current fashion for mobile computers and ubiquitous
73
Inhabited Information Spaces
computing. Nowadays, it is possible to obtain tolerable frame rates for 3D graphics on a wirelessly net-connected handheld computer. A person can thus be walking down a city street with a friend, chatting about a museum they intend to visit, while simultaneously watching the avatar of another friend moving through a CVE – with that “remote” friend also taking part in the conversation. In this case, it would seem difficult to claim that the person “inhabits” the information space. One might ask whether the person is in digital or virtual space, or in real or physical space, but the question is based on two false dichotomies: digital media are no more or less real than older media, and computers are just as physical as buildings and books. The workplace has always been affected by communication with people in other locations. Many traditional, everyday and non-digital media support remote interaction, for example letters, books, maps and the landline telephone. There are already digital media in the contemporary workplace that support remote interaction, such as email and mobile telephones. Nowadays, why do we not speak of “entering cyberspace” when we use email, as people did a decade ago? Why do we not inhabit telephone space, or speech space, or MacDonald’s employee name badge space? I suggest that a principle from philosophical hermeneutics is useful here: we don’t talk about these technologies in such marked ways because we have appropriated them into our everyday life and language. We no longer “enter cyberspace” because email is so interwoven in our everyday life and familiar in our experience that we don’t need to mark it out in such a way. We don’t inhabit telephone space because we understand telephones, in particular how to present ourselves through them and how to present ourselves to “spectators” nearby who can perceive our use of them. We only “inhabit” virtual worlds because their designs are so new and decoupled from other media. Experience and understanding of such coupling lets us focus on the task of communication, not on the tool for communication, just as a carpenter engaged in his work focuses on hammering and not the hammer. We continually mix and couple media in our everyday communication: walking, gesturing and pointing while talking, and referring to places and what people did in them as one writes. Space is an essential part of this mix. It has its unique characteristics that differentiate it from other media, but it has no privileged position above or apart from them. It does not stand alone as a paragon for computational media to emulate. More generally, a medium cannot be fully used or understood in an isolated or “singular” way. People’s activity continually combines and cuts across different media, interweaving those media and building up the patterns of association and use that constitute experience and understanding. A person’s work or activity may be influenced by the configuration of space around them and the interactions that space affords, but also by books, telephones, hypermedia, 3D computer graphics and so forth. People act and work through the full range of media they have
74
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
ready to hand. A narrow emphasis on space as the paramount resource for activity underrates the influence of other media. Recent technological developments, such as mobile phones and email, heighten or highlight a phenomenon already familiar in the use of older media such as written text, maps and cinema, and well explored in older disciplines than computer science. For example, a city’s meaning is not just in its bricks and mortar, but also in our understanding and use of the information about it. At any time, one is likely to have symbols in a number of heterogeneous media available for interpretation and use. As I walk through a train station towards a city square, the map in my hand, the voice of a friend on my mobile phone, the signs informing me of exit routes and the posters advertising exciting shopping opportunities are all open for my interpretation and action. Temporally, symbols in an even broader range of media influence me, as my activity is influenced by my past experience and my expectations of the future. Past experience may include my previous visits to that city, my browsing of a web site with good maps to print out, and my experience of magazines, books and films about urban life, and so forth. My language and culture, spanning media old and new, affect me as much as the immediate perception of spatial form. Since Heidegger and Saussure, a fundamental tenet of philosophy and linguistics has been that language is constituted by all the symbols and all the media one uses, with each symbol interpreted through immediate individual perception as well as past social experience. Contemporary neurophysiology is in strong accord with this view (Churchland and Churchland, 1998; Edelman and Tononi, 2000), as is the field most obviously related to the design of space, architectural and urban design (Leach, 1997). The differences between media are usually very obvious. We can characterise media and treat each one as if it were an isolated individuated entity because of the senses we use in perceiving each one, and also because of our understanding of how to relate and to distinguish examples of each one. For example, it is easy to distinguish the spoken word “red” from the written word red because of the senses one uses in each case. Despite having the same letters, it is easy to distinguish tar from rat by looking at the order of letters within each written word. Simple rules about what one can immediately see, hear, etc. within a word begin to strain and then break when one considers, for example, how we distinguish homonyms such as rose. The written word rose can mean many things, including a flower and having risen. When spoken, the same syllables can also mean linear structures (rows), about or belonging to fish eggs (roe’s), moving in a boat (rows), small deer (roes) and multiple occurrences of the Greek letter (rhos). The word’s usage is understood through its context – one’s understanding of the other symbols co-occurring with its use – rather than perception of the word’s pattern of syllables or letters.
75
Inhabited Information Spaces
Context becomes progressively more important as we turn from thinking about the differences between media, and distinguishing symbols, to considering the similarities of media and the relatedness of symbols. For example, the spoken word “red” and the written word red are related because we can use either of them in the context of rose blooms, fresh blood, the former USSR and so forth. We understand, relate and differentiate symbols through experience of contexts of use within a culture. As shown in the early twentieth century by Saussure, this understanding is not solely dependent on the form or medium of each symbol, but also on how we use each symbol in the context of other symbols – and this context includes symbols in other media. Taking fuller account of the interdependence of media enriches our understanding of space and of work. Consideration of how to make systems that are consistent with this standpoint opens up new possibilities for technology design and for computer-mediated social interaction. More particularly, it opens up a wealth of approaches based on coupling and contextualisation. For example, a museum exhibition might be associated with a set of web pages so that walking into a room of a particular architect triggers the display of text describing the life and work of that architect. Similarly, reading the text might trigger display of a map or visualisation of the room, affording access to a structured collection of blueprints, design sketches and building models. The space of the room would be coupled with the text of the page, with each becoming part of the context of the other. In terms of social interaction, a person walking into the room might be made aware of a friend’s reading of the web page, and hence open to conversation about the exhibition despite the two people being geographically remote from each other. Our intention is to support social interaction, as is familiar in traditional museums where co-visitors use awareness of each other’s interaction with exhibits as a resource for their interaction with each other, and use interaction with each other in interpreting the exhibits (Falk and Dierking, 1992; Galani and Chalmers, 2002). The City project explores the coupling of new and traditional media, weaving them together to form resources for social interaction and interpretation. In particular, our 2002 experiment explored social interaction between people in different locations and contexts where, by definition, they have different resources at hand. As they discuss and refer to contextual information, heterogeneity of media is inevitable: one person can use the nondigital resources of his or her location while others have only digital representations of that location. A case that is more easily handled is audio: each person will hear his or her own voice and sounds from other nearby sources differently to others, because of the digitisation and transmission of audio, but we have become relatively accustomed to handling this. A much more challenging heterogeneity is that of people’s position, orientation and gesture within rooms and buildings. For example, the Mack Room presents much greater visual and tactile richness than
76
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
the room’s digital representations, e.g. maps and VR models. Unlike most earlier CSCW research, the City project addresses this inevitable heterogeneity by coupling media together, tracking activity in each medium and representing it in others, and so letting participants interweave these media in their social interaction.
5.3 System This section outlines our prototype system, beginning with its infrastructure: the EQUIP platform, the Auld Leaky contextual link server, the VR Juggler framework, and the Bristol ultrasonic positioning system. More detail of this system can be found in MacColl et al. (2002). The EQUIP platform is being developed within Equator to support information sharing between heterogeneous devices. It provides a runtime infrastructure to support interoperation between Java and C++, and supports extensibility, for example dynamic loading of C++ classes. The University of Nottingham is leading the development of EQUIP, with contributions from the various Equator projects. City uses it as a blackboard architecture through which VR Juggler, Auld Leaky and the Bristol ultrasonics interoperate. Data items representing user context, an underlying spatial model and context-dependent content are stored for manipulation by City clients and services. Additional EQUIP facilities support real-time 3D graphics and mathematical operations, and abstract, renderable scene graph nodes. In addition, interfaces between EQUIP and a number of other systems have been developed, including the University of Iowa’s VR Juggler. VR Juggler is used as the renderer for 3D graphics. It is described as a virtual platform for virtual reality (VR) application development. It is a high-level, cross-platform architecture supporting immersive and nonimmersive presentations. Both UCL and Nottingham have immersive projection facilities, and the UCL facility has been used for development and pilot trials in the City project. The 3D graphics rendering is used to provide an analogue to the traditional exhibition space visited by traditional visitors. For World Wide Web visitors, the space is represented as a 2D map. We also require a presentation of the information in the exhibition displays, and this is provided by Auld Leaky. Auld Leaky is a lightweight contextual link server being developed within Equator to store and serve hypermedia structures, using context to filter query results. The model used to define the structures is capable of representing a variety of hypermedia domains: link-based, spatial and taxonomic. Auld Leaky is being developed by the University of Southampton, and is written in Perl although it has a Java API. Information is encoded as an XML linkbase, loaded into Auld Leaky and queried using HTTP. The text and images of the hypermedia were taken from the Mack Room’s catalogue. Contextual queries are used to
77
Inhabited Information Spaces
generate location- and device-specific content to be delivered by the Apache World Wide Web server and servlet engine. For 2D or 3D renderings, a location can easily be derived from the position of a visitor’s avatar or map marker. For the mobile computer, an ultrasonic system is used to provide position information. The University of Bristol is developing a low cost indoor positioning system using a combination of radio frequency (RF) and ultrasonics (Randell and Muller, 2001) as part of its contribution to Equator. The system uses a single RF transmitter for synchronisation, with ultrasonic transmitters for positioning. The ultrasonics transmit at known intervals after the RF, and are received by a handheld or wearable receiver. Each second, the variations in flight time of the ultrasonic transmissions are used to calculate the spatial position of the receiver. The receiver incorporates a magnetic compass to provide orientation information. The City project installation involves eight ultrasonic transmitters covering the approximately 10 m by 20 m area of the Mack Room. The room is a challenging environment for ultrasonics, as it is split into two large areas by a partial “time line” wall and has some areas set up as cubicles within which ultrasonic reception is virtually impossible. For aesthetic and coverage reasons, the transmitters are set on top of walls, displays and cubicles so that ultrasonic transmissions are reflected off the ceiling. The system supports a shared visiting experience, with one visitor using a handheld or wearable computer in the Mack Room, a second visitor using the World Wide Web on a laptop or PC in another room, and a third using 3D graphics on a similar machine in a third room. These computers communicate through 802.11 wireless Ethernet. A separate audio subsystem, that we will not detail here, handles visitors’ speaking to and hearing each other. In discussion and development scenarios we name the visitors Vee, Dub and Ana respectively, and these will also be used in the remainder of this chapter. The names do have roots: Vee is for ‘visitor’, and Vee was the user in our first design scenario; Dub is from the first syllable of “double-U”, as in WWW; and Ana stems from ‘analogue’, playing with the way that the digital space of the VR is an analogue of the Mack Room. Some people have suggested that Vee should be the ‘virtual visitor’ but we decided to keep the name, so as to irritate those who think that Vee’s experience is not strongly influenced by digital media. Spatial awareness is supported by tracking activity in each of the 2D, 3D and handheld systems, sending position and orientation information for each one into EQUIP, and then rendering the information about all visitors to each individual visitor. The components of the prototype system operate similarly for each visitor, broadly as follows: 1. store spatial position and orientation in EQUIP; 2. retrieve and render positions of other visitors;
78
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
3. store named location in EQUIP in response to position change; 4. store content from Auld Leaky in EQUIP in response to location change; 5. format content for presentation and advise client program of availability in response to content change. For Ana, position and orientation information is automatically published in the EQUIP data space by the VR Juggler client, and the positions of all visitors stored in the data space are automatically rendered as 3D avatars. Figure 5.1 shows a non-immersive spatial awareness display for Ana, with avatars representing Vee and Dub (displaying only heads rather than complete avatars). Vee uses a Hewlett-Packard Jornada that polls position and orientation sensors, and sends the results via a proxy into EQUIP. The proxy is also responsible retrieving the information about the other visitors, and the positions and orientations of all visitors are presented to Vee on a 2D map. Figure 5.2 shows a visitor the Mack Room, with a handheld and an ultrasonics receiver. The figure also shows a close-up view of the handheld as an inset.
011
011
011
11
Figure 5.1 A non-immersive VR display of the Mack Room for Ana, with avatars representing Vee and Dub (displaying only heads rather than complete avatars).
79
Inhabited Information Spaces
Figure 5.2 A visitor in the Mack Room, “Vee”, with a handheld computer and ultrasonics receiver. The figure also shows a close-up view of the handheld as an inset.
Dub interacts with a Java applet in a World Wide Web browser frame. The applet communicates via a proxy that converts mouse clicks on a 2D map of the Mack Room to position and orientation information. The applet also displays representations of all visitors. An example of Dub’s map is shown in Figure 5.3, corresponding to Ana’s 3D display in Figure 5.1. The red boxes on Dub’s map are trigger zones, discussed in the next section. Vee’s map is similar to, but simpler than, Dub’s. Shared visiting requires a sense of shared context, and hence some comparability of the information available to each visitor, but we also
80
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
011
011
011
011
11
Figure 5.3 An example of Dub’s map. For paper publication, the image has been annotated with the names of the visitors.
wanted to maintain and explore a degree of the heterogeneity that is inevitable in remote collaboration. Vee has the rich environment of the Mack Room’s displays and artefacts, as shown in Figure 5.2. Providing hypermedia to Dub and Ana involves converting positions to named locations, querying Auld Leaky with the visitor’s device and location in order to generate informational content, and then formatting and presenting the content. In initial trials of our prototype system we did not deliver hypermedia content to Vee, encouraging her to use the rich content of the existing room when interacting with her friends. Dub and Ana have rich access to the web that Vee lacks, and they can move and jump between Mack Room locations in ways that Vee cannot. Ana’s
81
Inhabited Information Spaces
3D view of the Mack Room VRML model has greater visual richness than the 2D maps of Dub and Vee, but she also has visual occlusions to deal with. Positions are converted to locations by an EQUIP-connected service. These locations represent semantically significant volumes or extents within the spatial model (shown as red outlines on Dub’s map, as in Figure 5.3). Also, for each visitor a target is inserted into the EQUIP data space, currently equivalent to a 10 cm cube held in the hand. Detection of a collision between a target and a sensor invokes code that inserts a new (user, location) item into the data space. Adding or updating a userlocation item in the data space triggers a query to Auld Leaky and the results – a set of hypermedia fragments – are stored in the EQUIP data space. Adding or updating such a set of hypermedia fragments in the data space triggers formatting for delivery to the visitor. The content fragments are retrieved and combined into an HTML page. Dub’s applet displays the HTML page in a separate browser frame set aside for this purpose. Ana also runs a browser and applet that, when advised, displays HTML pages, i.e. Ana has no map display but does have a textual display. The overall effect, then, is that all three visitors have location information about their co-visitors’ shared awareness of location. Dub and Ana each have spatial and textual information about the exhibition, with the text updated as they use the 2D map and the 3D graphics. Vee has the traditional information of the exhibition room. Many of the artefacts and exhibits have corresponding representations for each of the three visitors. We sometimes refer to such artefacts and exhibits as “hybrid objects” because of the visitors’ tight interaction around corresponding heterogeneous representations. Space, text and audio afforded sufficient interaction, context and reference to support a shared visiting experience, as the next section discusses.
5.4 Use In the summer of 2002, the City project carried out a set of system trials in the Mackintosh Interpretation Centre. Rather than trying to make the best possible system for the Mack Room in particular, our focus was on general lessons we could learn for the design of systems involving heterogeneous representations and interactive media. We aimed to increase our experience and understanding of how these systems serve as constraints and resources for users’ interaction. We had already studied the use of a number of cultural institutions, including the Mack Room, without our technology (Galani and Chalmers, 2002), and were interested in comparing the Mack Room with and without our technological intervention. Fuller discussion of the trial can be found in (Brown et al., 2003). The trials involved 34 participants: ten groups of three and two groups of two. The groups of three consisted of a Dub (using the web), an Ana
82
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
(using VR) and a Vee (using a mobile computer). Dub was in the Interpretation Centre, while Ana and Dub were in separate rooms on a different floor of The Lighthouse. The pairs explored different combinations: Dub and Ana visiting without the physical visitor (one trial), and Vee and Ana (one trial). For the first half of the trial, participants were asked to explore the Centre together, to familiarise themselves with the technology and how they could co-visit. Since we were specifically interested in how the system supported social interaction, we introduced an artificial task for the second half of the trial. Each participant was given three questions, and the group was asked to answer these questions together. Some questions were designed to provoke open-ended discussion and interaction between the participants. For example, participants were asked “What is the group’s favourite Mackintosh painting?” and “What contribution has Mackintosh made to Glasgow?” as well as more factual questions such as “What was Mackintosh’s birthday?” This combination of open and task-centred behaviour allowed us to study activity that was typical of a museum visit, such as finding exhibits, and to observe how the system supported the shared aspects of visiting a museum. During the trial, use of the system was heavily logged and each Dub was video taped. After the trial, the participants were interviewed as a group in a recorded semi-structured debriefing. For analysis, we combined the map view used by participants, with the video and audio recordings. We analysed transcripts of the post-trial debriefings, and the logs of the visitors’ use of the system. We paid close attention to the details of how users interact with each other and with technology, especially through video analysis. With particular interest in the use of location and of exhibits, we looked for “critical moments” where the system was used in a way that would let us reveal design lessons, consistencies and inconsistencies with theory, and comparisons with earlier studies. The participants engaged in rich social interaction around the hybrid or coupled exhibits. When participants found that objects corresponded in this way they quickly were able to move on to using them in their shared tasks and activities, for example discussing the qualities of the object and comparing it to the other exhibits. In the following extract the participants discuss a set of Mackintosh pictures to decide which one they like the most. Square brackets (e.g. [pet]unias) show overlapping talk and italics shows a speaker’s emphasis. Vee: Petunias is errm better for me than Rosemaries Ana: Ok [pet]unias Dub: [hhh] Petunias it is Vee: Early work Ana: Hey guys see: this other one it’s really nice. It’s called Fort Mailly hhh Fort Mailly in nineteen twenty seven Vee: Nineteen twenty seven Ana: Yeah, it’s got the light
83
Inhabited Information Spaces
Vee: Yeah I know but I like [Roses] Ana: [Can you] see it? Dub: Fort Mailly? Ana: Hmmmm Dub: Yeah that’s quite nice Vee: I still prefer Roses
The visitors do not focus on the system or the media involved, such as the differences between the digital and the printed reproductions of the Fort Mailly painting. Instead they focus on its aesthetic qualities and the task of deciding which picture they like most. They don’t focus on the tool but on the task. However, interacting around these hybrid exhibits was not without its problems. In ordinary face-to-face interaction, we assume a degree of commonality in the objects that we can see, hear, touch and so forth. Each user of our system had to build an understanding of the perspectives the other users had of the Centre. They could often see similar things, or related things, but not the same things from the same perspective. Dub and Vee frequently guided Ana to specific exhibits verbally. We designed the system so that Ana did not have an overview map, and therefore might find that occlusion of objects in the exhibit was a problem. In turn, since Dub and Ana could shift attention between artefacts placed far apart in the room much more quickly than Vee, they frequently found information and then guided Vee to the corresponding location. Again this was often done verbally, but more spatial or graphical guiding is discussed later. Participants also made use of shared location and orientation, using icons on an outline map for Vee and Dub, and avatars in Ana’s 3D display. Shared awareness of location also allowed users to quickly move to their friends, and to quickly find or confirm the exhibits being discussed, that is, to quickly find what their friends were looking at and then move so as to look at the same or a closely related thing. They developed simple means to gesture. In one case, Ana moved her avatar back and forth in a “wiggle” so as to confirm which icon represented her and to show her location to Dub. This gesture was something like the wave of a hand used by someone to show he or she is in a crowd. Global location could be seen “at a glance” on the map, without the need for the visitors to use talk, but examples such as Ana’s wiggle show participants’ awareness of how they would be perceived by each other in different media. Participants learned about each other’s perspectives through questions and observations, building understanding of what they shared and what they did not, and thus how to more smoothly interact with each other through the resources at hand. Indeed, participants put considerable effort into designing their interactions to take into account the characteristics and limitations of their varying views of the Centre. For example, the above extract shows Vee emphasising and confirming the year in
84
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
which “Fort Mailly” was painted (“nineteen twenty-seven”). One of the largest exhibits in the Mack Room is a long wall of panels, with each of the chronologically ordered panels showing a number of images associated with a year of Mackintosh’s life. Mentioning a year to Vee would let her move quickly to a corresponding part of the exhibit. The information presented to Ana and Dub about this wall was broken up into a page for each year, so any one of the visitors could help guide the others to a particular image in that exhibit by specifying the year. The use of our system involved more talk, and louder talk, between the co-visitors than we observed in conventional museum settings. For example, during a post-visit discussion two participants were asked: Q: Is it different to a museum visit? A: Yeah, it’s really talkative. B: You kind of go “Mmmm, that’s nice” [. . .] If you find something interesting, you go “Look”, and “That’s over here”
Another commented on being able to talk without disturbing others: I quite enjoyed the social engagement . . . being able to talk about everything more and not feeling that you are disturbing. Not thinking about other users in the gallery, you know it’s kind of liberating . . .
I offer here a few of the many possible reasons for this, stemming from the use of multiple rooms, the audio hardware used and, perhaps more interestingly, the set of media available for interaction. Two of the participants were in rooms other than the traditional exhibition room, and so were less influenced by the normal hushed reverence adopted inside museums. All three participants were engaged in a trial of an unfamiliar technology, without established social norms for speech levels. Vee often talked more loudly and paid less attention to other members of the public in the Mack Room. Much like a telephone, the loudness of speech needed for audibility through the audio hardware may not be the same as in faceto-face interaction. Lastly, the participants often used shared audio because the coarse-grained representations of position and orientation did not afford familiar use of gesture and posture. However, participants were not solely using talk in order to circumvent the tools given; they were often talking while engaged in the tasks. Also, they developed gestures for the media at hand, for example Ana’s wiggle: a gesture made in a 3D VR to be seen on a 2D map. In everyday face-to-face situations, interaction between two people can be impeded for a number of reasons such as one or both of the participants choosing to interact with others (e.g. on the telephone) or to interact with objects, or by participants being forced to interact with other people or objects because of interruption, breakdown, occlusion and so forth. In the trial, interaction would often pause when participants found a difference between the visitors’ representations of the Centre. For
85
Inhabited Information Spaces
example, the room’s interactive video displays were only available to Vee. When a visitor started to use and talk about a display or exhibit that was not shared, the other participants would refrain from interacting and move on to other exhibits. Similarly, Dub and Ana generally used movement on the map or VR in order to access information, in preference to the more conventional hyperlink navigation. This seems to have been partly due to the fact that such spatial movement was more a part of the shared experience, in terms of conversational references to locations but also as a way of avoiding future confusion: following links to a web page about a new exhibit did not move the participant’s icon or avatar to the corresponding new position. (This capability was implemented during the trial period but, to maintain consistency, was not deployed.) This meant that “web movement” could leave a visitor’s icon in a potentially confusing place. While there were occasional interactional breakdowns, they were not fatal for the sense of a shared visit or for interaction. Overall, participants showed skill in finding ways to handle the differences between the different representations, and exploiting corresponding and coupled features. The system successfully supported a shared experience by enabling users to talk about and interact around the exhibition, offering a socially engaging experience beyond that available to a conventional web site visitor.
5.5 Ongoing and Future Work We continue to explore remote collaboration in cultural information and cultural institutions, in particular collaboration involving heterogeneous media. We support social context as a resource for the interpretation of information, and contextual information as a resource for social interaction. We are extending our system to be used in more of the city than The Lighthouse, adding GPS (Global Positioning System), dead reckoning, GPRS (General Packet Radio Service), and further 802.11 aerials. We are using the publicly available VRML model of central Glasgow from the University of Strathclyde, and 2D maps from services such as EDINA (www.edina.ac.uk). We have been undertaking field studies of visitors in a range of locations in Glasgow, seeing how their visits include far more than traditional cultural institutions, and how they use resources such as tourist information centres, maps, guidebooks and signage. We aim to run another field trial that involves a number of participants visiting the city, each of whom has a wearable computer on which tightly coupled 3D VR, 2D maps and hypermedia can be used. Each participant’s activity will be available to the others synchronously, much as in the Mack Room, but also asynchronously. One reason for this latter development is to move beyond the traditional objectifying or “scientific” systems of classification and retrieval
86
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
City: A Mixture of Old and New Media 5 ●●●●●●●●●● ● 1 ●●●●
that too often are the only means of access to digital information. Influenced by the theoretical standpoint outlined in Section 5.2 and the Recer recommender system (Chalmers et al., 1998), we have built a central information resource, connected to EQUIP and thus accessible via a variety of media and devices, and which stores a growing and evolving body of individuals’ paths or narratives through a range of symbols: our own images and fragments of hypertext, annotations made by users, locations in the city, and other locations and web pages worldwide. The system uses this resource to make contextually specific recommendations of people, places and things by comparing each person’s recent activity with similar sections of the past activity of selected others. Paths can also be more directly shown on maps, in VR models and woven into web pages. We will allow this body of information to grow as people use it, making new associations between symbols and adding in new ones. Information access based on evolving inter-subjective patterns of contextual association and use will complement access based on more static and objective interpretation. We are also experimenting with bridging between the two, using pre-written “official” explanations of the connections between symbols as a means to enrich the dynamically created recommendations. Remote collaboration brings abstraction and approximation as a system monitors and senses activity such as a person walking across a city. Issues such as sampling, resolution, delay, disconnection and uncertainty have to be faced as one decides how to represent the activity inside a system, even before one considers how to represent the activity to a remote collaborator. There is no getting away from the fact that activity is going to be interpreted through sensors and transducers such as cameras, GPS and ultrasonic systems, and any digital medium has characteristic losses and uncertainties. While we want to make new resources for interaction available to people, combining new media with old in a perfectly seamless way is not going to happen. Uncertainties and inaccuracies (“seams”) are an inherent part of any communicative medium, and people often learn to use these characteristics for their own ends. For example, mobile phones can be set to display the current cell, if the service provider permits, and some people choose to enable this facility. This is an elegant ambient or peripheral presentation of potentially useful information: users can choose what use to make of it, for example seeking a stronger signal by moving to a location that forces handover to another cell. Cell boundaries and signal strengths are interactional resources of the medium. Similarly, long-term use of videomediated communication was reported by Dourish et al. (1996) to lead to “complex patterns of behaviour built up around the interactional details of the video medium . . . When the medium changes, the mechanisms change too; but the communicative achievements remain.” Recalling a term used by Mark Weiser (1994), we see seamfulness as an important design goal for our future work. We plan to design in
87
Inhabited Information Spaces
explicit representations of the errors and uncertainties in our systems, letting people take account of the characteristic heterogeneity, errors and limitations of the systems we offer them. For example, we are starting to develop explicit presentations to accommodate uncertainty due to ultrasonic and GPS-based positioning, showing a person’s sensed position as a spatial extent, rather than as a point, and showing estimates of sensing accuracy and communications bandwidth on our city maps and models.
5.6 Conclusion The City project emphasises the interdependence of media, such as computer graphics and audio, with others. We have explored the combination of CVE technology with hypermedia and mobile computers, and also with the architecture and exhibits of the Mackintosh Interpretation Centre. Supporting broad social context in remote collaboration involves heterogeneity, and our project aimed to address this through coupling and correspondences between media. Users of a mixed collection of interactive media were able to enjoy a shared visit experience, engaging in collaboration through awareness of each other’s activity and through more focused talk and interaction around “hybrid” objects. By presenting the theoretical issues underlying much of this work, as well as system design and experiences of use, this chapter may serve to complement many other chapters of this book. Rather than considering that users inhabit our information space, we see people as inhabiting cities and towns, and using new technologies and older media to interact with friends, relatives and colleagues. Looking to the near future, CVE technologies will be widely available via phones and mobile computers. I suggest that taking fuller account of their use among a wider set of technologies and media, and designing for contextuality, heterogeneity and seamfulness, will greatly enrich our work.
Acknowledgements Special thinks go to all the City project members past and present, especially Barry Brown, Areti Galani, Chris Greenhalgh, Ian MacColl, Dave Millard, Cliff Randell and Anthony Steed. Also, we are all grateful for the generosity of our hosts at The Lighthouse, especially Lynn Bennett and Stuart MacDonald. Thanks also to Ziggy Stardust, Fugazi and Low. Equator is an Interdisciplinary Research Collaboration (IRC), supported by the UK Engineering and Physical Sciences Research Council, and the City project was supported by a donation from the HewlettPackard Art and Science programme.
88
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Soundscapes
6
Tony Brooks
011
011
011
011
11
6.1 Introduction Although purely virtual information spaces receive much attention and are indeed useful in certain situations, it can be difficult to integrate them with real-world activities and awkward to interact with them. As pointed out in Chapter 5, even notionally single-user activities often happen together with other people and a purely virtual environment tends to cut people off from their surroundings making it difficult for several colocated people to share an experience. Fraser et al. (Chapter 9) also show that collaborative interaction within a CVE is not quite as intuitive as it might first appear. For these reasons purely virtual environments may not be desirable in cases where the participants have a disability or where the environment is intended to be shared with other people, as in a public performance . The creative process is something that is inherent in most people, even the severely disabled, yet it is an often uncharted channel within which to explore their expressive human potential. “Productive creativity” is becoming recognised as a beneficial therapeutic treatment for people with disabilities. In this chapter the Soundscapes system, which is built to allow unencumbered creative expression for both able-bodied and handicapped people, will be discussed. The next section will describe the system itself and then further sections will show how it has been used therapeutically and also to stage public performances.
6.2 The Soundscapes System The origins of the Soundscapes system lie in a MIDI bass (a modified bass guitar that can generate MIDI – Musical Instrument Digital Interface – signals allowing the control of electronic synthesisers) and an expression pedal (a foot-controlled sound effects processor that can change the sound of an audio signal). From interactions with his severely
89
Inhabited Information Spaces
handicapped uncle in which the author played the bass and his uncle manipulated the expression pedal the author realised that even a simple means of expression could open up ways for people to communicate and could provide immense satisfaction. The author started searching for a richer means of expression and it soon became clear that the MIDI bass itself was too intimidating and required too much physical skill to be satisfying for people like his uncle. This led to a search for a rich expression medium that required no training to use. Although immersive reality hardware could track peoples’ movements in a natural way it was expensive and uncomfortable to use to the equipment since it had to be physically worn by the user. After much experimentation the author selected infrared movement sensors. Within the range of the sensor, movement from the flicker of an eye to a full body movement can be detected. The Soundscapes system uses three such sensors to allow triangulation of movement or triggering of three separate operations. Each sensor head is mounted on a flexible “gooseneck” support to allow for different configurations. The
Figure 6.1 Photograph of the first three-headed infrared movement sensor used by the Soundscapes system.
90
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
sensors output MIDI information that have been used to control movement and navigate through 3D space as well as controlling filters applied to computer-generated images. The Soundscapes system is formed from a library of tools for body function capture and a number of programs for creating the results that each user response determined. The library of input tools ranges from biofeedback sensors for brain activity, muscle tension, GSR (galvanic skin resistance), heart rate, to infrared (see Figure 6.1), ultrasound and video tracking (the EyesWeb http://www.eyesweb.org/ is used to perform the video capture) technologies. Most often the captured body movement data is transmitted using the MIDI protocol and the MAX program (http://www.cycling74.com/products/maxmsp.html) is used as a central hub directing and manipulating the data as needed. In conjunction with a physiotherapist the Soundscapes operator calibrates the system such that the range of velocities measured by the sensors matches the movement capability of the individual who will be participating in the session. The system is configured so that movement triggers audio and visual feedback with the exact nature of the feedback depending on the individual – we have found that some individuals respond better to audio stimulus and others to visual stimulus. System settings as maintained between sessions and movement data are logged so that the therapist can measure progress from session to session.
011
011
11
Figure 6.2 Version two of the three-headed infrared sensor system in use. The three sensor heads are visible at the centre of the image in between and a little below the performer’s hand and the image projected on the screen.
91
Inhabited Information Spaces
6.3 Therapeutic Uses of Soundscapes One of the uses of the Soundscapes system is to act as an “expression amplifier” for people with physical disabilities – it provides them with the possibility to generate sounds and images from whatever movement they are able to control and this can be a powerful therapeutic experience. The therapeutic use of Soundscapes is illustrated by the i3 funded future probe project called The World Is As You See It (TWI-AYSI, http:// www.bris.ac.uk/Twi-aysi/). TWI-AYSI was a successor to the CARESS project, which was also funded by i3 as part of the Experimental School Environments programme. The TWI-AYSI project members consisted of members of the Bristol University Electrical Engineering Department, and Stefan Hasselblad both of whom were involved in the EU CARESS project, and Tony Brooks. The CARESS project successfully motivated and empowered children to develop creativity, imagination and expression through interactive acoustic environments. The objective of TWI-AYSI was to answer the question “Can immersion in a visual environment hold similar potential for such children in terms of the aesthetic resonance they might derive from movement within such a visual space?” During the project we brought young children from a school for multi-handicapped people into the Centre for Advanced Visualisation Projection screen Video camera
Full scene camera
Infrared motion sensor
Video projector Disabled child Infrared motion sensor Video camera
Computer
Face shot camera
Camera feed quad splitter monitor & VCR
Figure 6.3 Typical set up as used in TWIAYSI (Sweden 2001) with Multiple Camera Analysis (MCA) utilising six video cameras and three infrared motion sensors.
92
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
and Interactivity (CAVI, http://www.cavi.dk) in Denmark. The children ranged in age from three-and-a-half to five-and-a-half. A common experience for all of the children involved was the manipulation of sounds, robotic lights and coloured images. However, in CAVI we also wanted to explore the possibilities for using the movement sensors to enable 3D navigation. MIDI signals from the movement sensors were received by a Linux workstation and translated to movement information that was used to control the rendering of the 3D world on a SGI Reality Monster (Silicon Graphics Inc.). In this experiment, the movement sensors allowed a person to control the movement of a rocket ship that was projected onto a large screen in front of them. It was possible to view the image in stereo using LCD shutter glasses . However, the children did not want to wear the necessary head sets and shutter glasses that were used to view the virtual objects and so we made the decision to project the ship in mono and the youngsters once placed with their head inside of the active sensor space, were able to control the various degrees of movement of the spaceship by small head gestures (a video of this is available online at http://media.nis.sdu.dk/video/ twi-aysi.html). A small gesture to the right and the rocket ship moved to the right, a movement left and the rocket ship moved to the left, a head movement down and the ship’s nose tipped down as if to dive, and a head movement backwards and the ship’s nose tipped up as if to climb. As an example of the results of this experiment we shall describe a session with a multi-handicapped five-year-old boy. At first we tried to place the 3D shutter glasses on him so he could view the scene in stereo but he shook his head to remove them. We removed the glasses and changed to a mono projection. This was a big success as the young boy was totally immersed for around six-and-a-half minutes. The young disabled boy was only able to express through a single verbal expression (a sort of “AYYE” phrase). While in the session and immersed in the interaction he could clearly be heard making his sound, which was translated by
Figure 6.4 Freja, a severely disabled girl with her helper Maggie. Freja is painting via her movement and can see herself in the window on the right. The colour painted depends on the velocity of the movement, with each individual having his/her own settings to match their ability. Freja’s face tells the story. Reproduced with permission from Stefan Hasselblad.
93
Inhabited Information Spaces
his helper as an expression of a joyful experience, and this was probably due to the fact that he quickly understood the functioning of the interactive space and was able to control the image on screen – an experience of controlling his environment that was probably completely new for him. As a result of the CARESS and TWI-AYSI projects we learned that when audio only or visual only feedback was present some people reacted better to audio and some better to video. When both audio and video were present in the environment a greater degree of immersion was observed. This work resulted in another European project to take these ideas further, called CARE HERE (http://www.bris.ac.uk/carehere), whose results are due to become available as this book goes to press.
6.4 Artistic Performances Based on Soundscapes 6.4.1 Interactive Painting The inspiration for “The Interactive Painter” came from a discussion at a party in which it was stated that some people believed that painting on canvas was “dead” since computers enabled us to do so much more with Video camera Projection screen Infrared motion sensors Back projected image
Mirro
Painter
Video camera Computer
Video camera
Video projector
Figure 6.5 Diagram of the equipment used in the interactive painting performances. The three sensors capture motion via infrared beams. In this diagram the video image is back projected but front projection is also an option to allow working by casting shadows and to overlap the video on a canvas.
94
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
images and colours. We decided to try to investigate a way for a traditional painter to utilise new technology while painting on canvas in the traditional way. The resulting performance involved the painter Manu Rich from Paris, France and within Tony Brook’s COIL (Circle of Interactive Light) interactive installation, it toured a number of museums of modern art in Scandinavia in 1998 and 1999 culminating at the Danish NeWave festival performed at the Gershwin Hotel, in Manhattan, New York 1999. More recently we have experimented with a therapeutic version of the installation for elderly and handicapped people. Interactive painting is designed to be a live experience with an audience and takes place in a darkened room, which provides a challenge to the painter since painting normally requires a well-lit room. The event makes use of projected computer-generated imagery and synthesised sounds that change according to the movement of the painter. In turn the painter may be influenced by the sound and images thereby creating a feedback loop. In order to give maximum freedom of movement, infrared and ultrasonic movement sensors were used to capture the painter’s movement, thereby allowing him to move freely without trailing wires. Owing to the size of the canvas and the limited range of the infrared sensors a jacket with reflective strips attached to it was worn by the painter, thereby allowing movement to be detected from a greater distance. The sensors were strategically set up around the canvas and translated light reflected from the jacket into movement information that a computer translated into sounds, colours and images. One of the primary aims of the performance was to see how the painter would be influenced by images projected onto the canvas. The light from the projected image would alter the appearance of the painted image and change how both the artist and audience perceive it. In order to do this a video projector is set up to project images onto the canvas. The movement sensors control filters applied to the image such that when there is no movement there is no image. As the painter moves then portions of the image are projected onto the canvas.
011
11
Figure 6.6 Photograph of an interactive painting session Gershwin Hotel, Manhattan, New York 1999.
95
Inhabited Information Spaces
In addition to the projected image, the painter’s movement also triggers sounds from a sound synthesiser. The author changed the synthesiser’s patch data in real-time to match the “mood” of the composition and also to try and influence the mood of the painter. The audience of an interactive painting performance therefore has many elements to focus on: ● ● ●
The painting that the painter is actually producing on the canvas. The movements of the painter, which the painter may change in order to change the sound and visuals as well as the painting. The sound and images caused by the movement of the painter.
The artist naturally tends to focus on the creation of the painting and not the multimedia experience. However, it was apparent that the sound and images had an effect on the painter and the painting. From experience we noted that smooth “pad” sounds tended to result in a corresponding smooth motion of the artist’s brush and passive choice of colour and similarly a hard-edged “Hendrix” guitar lick resulted in an aggressive motion and severe choice of colour.
6.4.2 The Four Senses In April 2002 the first collaboration between artist/researcher Tony Brooks and artist Raewyn Turner resulted in a series of multisensory performances called The Four Senses in Auckland, New Zealand, with the Aotea Youth Symphony, Touch Compass, (a mixed able-bodied/ handicapped dance company), and HANDSS (Hearing ANd Deaf Sign Singers, a deaf signing choir). The performances were an improvisation in light, sound and olfactory information. Brooks and Turner created a real-time translation of sound and the gestures of making that sound, into light and colour and multiple layers of smell. The concerts aimed to engage and reframe perception of music and to play with subjective experiences and simulated synesthesia. Each sensory element was constructed from information relating to the other elements. The associations and correspondences of the elements made by the audience were according to their own individual and personal experiences. Tony Brooks utilised movement sensors and video cameras to capture body part movement and translate it into painting with coloured light. In this way the orchestra conductor was able to “paint” the scene through his gestures. Similarly, orchestra members, dancers and a special signing choir for the deaf images were blended into the backdrop in real-time such that their velocity of movement affected the colour of image generation and collage composition.
96
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
Projection screens Percussion / Timpani
Primary camera capture areas
011
011
011
011
Lead cello
Lead violin
Deaf signing choir Basses
Singer Conductor Dancers
3 × roving video cameras
Figure 6.7 Diagram showing the locations of performers and projection screens for the Four Senses performance.
Raewyn Turner translated the music through drawing and proportional relationships into coloured light and olfactory stimulation. This was done using computer-controlled lighting rigs and by releasing aerosols into the ventilation system at key moments during the performance. The light collage thus created was a play of interaction between live video feeds and sensors, and coloured light pre-programmed to an interpretation of sound, each affecting the other in a dynamic visual loop. The fourth sense employed was touch. In order to give hearing impaired people a feel (pun intended!) for the concert experience vibrating cushions were used. These cushions were developed as an accessory for the Sony Playstation® and can transmit vibration at various levels of intensity. These cushions were coupled to the PA system so that they would vibrate according to the music played by the orchestra and a number of them were distributed throughout the auditorium. Hearing impaired people could sit on the cushions or hug them to their chests in order to get a “feel” for the music. In addition a deaf signing choir participated in the performance to counterbalance the conventional (but sight impaired) singer also performing on stage.
11
97
Inhabited Information Spaces
a
b Figure 6.8 Two photos of the Four Senses performance: (a) dancers in front of the orchestra; (b) an example of the projected video generated during the Four Senses performance. Reproduced with permission from Milan Radojevic annd Mirjana Devetakovic Radojevic.
98
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
Soundscapes 6 ●●●●●●●●● ● 1 ●●●●●
6.5 Conclusion We believe that there will one day be a programmable ion layer surrounding our body which will be able to stimulate our individual senses so that we are enabled towards a truly augmented reality without wearables and special hardware such as screens. This is not pure fantasy since a number of people in the Nanotechnology community such as Storrs Hall (http://discuss.foresight.org/˜josh/UFog.html) and Drexler (1992) are exploring methods that would allow the creation of a “Utility Fog” (Storrs Hall) that would, among other things, allow our senses to be stimulated and our movement sensed. Our human-centred work with people with special needs, who are disabled, elderly or in rehabilitation allows us, in certain circumstances and with certain individuals, to get closer to higher nuances of the senses and what the senses mean. By exploring that research and working in crossdisciplinary teams including neuropsychologists, computer scientists, human–computer interaction (HCI) researchers and others, we hope to work slowly towards that goal. We believe that the immersive “play” within an interactive environment has much potential to do good. Specific to the current work is the belief that everyone is so individual that a system is required that can be tailored to each desire, facility and requirement – this entails adaptability with a capital A. Libraries of input HCI devices, together with libraries of mapping devices and libraries of output software is the optimal way forward. Through non-fixation on interactive spaces through invisible system components – hardware (disappearing computers, sensors embedded in environments etc.) – we will obtain the mapping of body function subconsciousness that can help people. The subliminal pervasive aspects – or really more so terms such as “proactive computing” (that which focuses on improving performance and user experience through speculative or anticipatory actions) and “autonomic computing” (which focuses on improving user experience through the system’s self regulation) in relation to this work are preferred as both relate to the user experience rather than the artefacts often referred to in pervasive computing – are obvious. We also believe that responsive audio/visual/haptic feedback may have much more to offer in the future over and above what we now utilise, in fact we believe that correspondences between synchronised feedback especially sonic and visual (in the first instance) are only just scratching the surface and that there are many new discoveries waiting to be made.
Acknowledgements The CARESS and TWI-AYSI projects were funded by the European Union via the i3 network.
99
This page intentionally left blank
011
011
011
011
11
The Computational Interplay of Physical Space and Information Space
7
Enric Plaza
7.1 Introduction There is a current trend in computer science to develop new devices and applications that make the transition from desktop (and laptop) computing to computing devices that are embedded in the physical and social environment in which people live. Several approaches have been proposed in this direction and they have different names and focus on related but distinct issues. The first, known as ubiquitous computing, pervasive computing or the disappearing computer, focuses on embedding computing devices into the physical objects and surroundings where people work and live. A second related trend is that of wearable computers, that focuses on embedding personal computing services in devices that people can carry or wear while moving around in their everyday activities. Next, augmented reality focuses on enriching people’s perception of physical surroundings with computer-generated information. And finally, to be brief, there is a trend for developing autonomous agents that take on people’s goals and try to achieve them on their own. However different these approaches are, a common issue they all have to deal with is the awareness of the physical (and social) surroundings in which people interact with computing devices. Traditionally, computers (ranging from mainframes to personal computers) operate in a purely informational world – typically, screens and printers, plus a customised connection to manufacturing machines or task-specific sensors. The advent of the Internet and the World Wide Web links these computers into a common (or rather shared) information world. The relation of the Internet, as an information space, with physical space is a research issue that essentially deals with the issues of context awareness: who is where and when, with whom, doing what. This chapter focuses on the interplay of physical space where people act and live and
101
Inhabited Information Spaces
an information space where software programs reside, interact among them, perceive some properties of the physical world, and perform some tasks and actions on both the informational and physical spaces.
7.2 The Interplay of Physical and Information Spaces Since all computers, and the software they run, are potentially connected over the Internet, we can consider this as an “information space”. Then, the computing devices that populate the physical world, from personal digital assistant (PDAs) to the emerging “ubiquitous computing” devices, can be considered as the interface between the physical spaces people inhabit and the “information space” inhabited by software programs. The most critical issue to improve this interface is that nowadays software programs have little or no awareness of the physical space and of the activities in which people engage in that physical space. In this chapter we will first discuss the general issues that need to be addressed to improve the interfacing of physical spaces by the “information space” inhabitants, in particular awareness of the physical and social context of people. Later, we will present the COMRIS project and explain how these general issues are addressed and solved for a specific physical space (a conference centre) and a series of informational tasks useful in that space. Awareness of physical space involves more than merely spatial or geographic reasoning. People perform activities and interact with other people while moving in physical space, and the more aware they are of those activities the better the interface with the information space. Two levels of context awareness are required: 1. Physical sensors, determining the granularity of the perception of physical space activity. The physical sensors can be a Global Positioning System (GPS – giving a co-ordinate point); wireless tags that people wear to detect who is close by or in which room; or even more sophisticated speech capture and analysis systems (e.g. trying to determine topic, mood, etc., of ongoing activities). 2. Common sense knowledge, determining the inferences a system can make on the “world situation” given the physical sensor information. For instance, if a wireless tag indicates that a person is in a room of type Meeting Room, there is a series of inferences that can be made from a knowledge base that models business activities (e.g. the person is in a business meeting with other people and should not be interrupted unless there is something urgent). Perception using physical sensors establishes the baseline for awareness capabilities. For instance, GPS can be used together with personal devices like PDAs to yield information customised to the person (using the personal profile in the PDA) in that spatial situation (by a server that
102
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
has a database of services available near that location). However, this approach is very centralised, depending on a service provider holding geographical information. For some tasks it is better to add more sensors that allow the person’s behaviour to be determined: a microphone can be used to determine if the person is busy (talking with somebody) or idle. Currently, project Esprit “IT for Mobility” 26900 is developing a sensor board that could be integrated with mobile phones or PDAs. The sensors on this board include two microphones, a dual axis accelerometer, a digital temperature sensor and a touch sensor. With them, a computing device can locally infer different contexts where this device is situated – such as “sitting in a pocket”, “lying on the desk”, “in user’s hand” – allowing the device to adapt its behaviour to each context. Even discussing the physical sensors used we have had to include the idea of a computational model that transforms the raw input data into some interpretation of the state of affairs in the world. This is because context-awareness is essentially an interpretation of the world situation and, as such, what is needed is plenty of knowledge about what the world is like, in other words, common sense knowledge is needed. There has been a lot of research in artificial intelligence (AI) in the last 10–12 years, concerning the issue of common sense, with the Cyc project (http://www.cyc.com) being the most well-known endeavour. The current understanding of common sense can be summarised as: 1. an ontology, defining the objects existing in the world that we want to talk about, and 2. an inference engine, capable of using a model of the state of affairs in the world expressed in that ontology to conclude new facts or statements about that state; new facts that are “obvious” or “implicitly” known by people, by what we are calling “common sense”. Moreover, context aware applications need to have some properties that differ from current applications: they need to be persistent, responsive and autonomous. We will call this collection of properties continuative computing1 because they set apart context-aware applications from the usual applications oriented to input–output. First, a context-aware application needs to be persistent, that is, persistently in runtime state, non-terminating. Commonly, an application accepts input and produces output (in fact, the same definition of algorithm is based on the idea of transforming an input to an output) – an exception is programs that are operating system services (that are difficult to model in the algorithm paradigm based on termination). A common application is a file that, when a user needs it, becomes runtime, receives an input and after some processing time yields a result and then goes offline. 1
Continuative: tending or serving to continue.
103
Inhabited Information Spaces
A context-aware application needs to be non-terminating, awake and running persistently, much like an operating system or a PDA. Moreover, it needs to be persistent in order to be responsive: able to adapt and produce adequate responses when something changes in a context or the context changes to become a new context. Finally, context-aware applications need to be autonomous in the sense of having an identity persistent in time and a memory (or internal state) that is individual. Since changing context is one of the most important pieces of information a context aware application can deal with, it makes no sense that each particular physical space has a context-aware application that is independent from other locations. Since people move around it is better to think of a context-aware application as centred on users, like a PDA that a user carries around. In this way the context-aware application can know the past history of contexts of the user, and even learn to anticipate the most likely future contexts of the user and prepare for it. There is a current paradigm and associated technology that fits these requirements, which we have outlined as continuative computing: intelligent agents. Agents and multi-agent systems being developed in artificial intelligence are conceived of as autonomous, permanent entities capable of using ontologies to perform inferences for solving problems, and for co-operating and/or competing with other agents or people. Intelligent agents are classified as reactive or deliberative, depending on some design properties, but we will show later in the chapter that designing agents with a particular architecture like the one proposed, means they can also be responsive and thus exploit awareness of physical and social context to improve their performance on behalf of users.
7.3 A Framework for Context-aware Agents Our framework is composed of a collection of context-aware personal information agents (CAPIAs) working in an information space and a collection of human users interacting in the same physical space. A useful way to visualise this distinction is the dual space schema depicted in Figure 7.1. Human users, on the right-hand side of Figure 7.1, are in a location, interacting with other persons (who may or may not be users) in the context of social activities. Information agents, on the left-hand side of Figure 7.1, inhabit an information space where they interact with other agents and gather information in the interest of the users. Moreover, we have mediation services connecting the information space of agents and the physical space of human users. Specifically, we are currently using two mediation services, namely an awareness service and a delivery service (see Figure 7.1).
104
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
CAPIA
CAPIA
CAPIA
CAPIA CAPIA
Delivery services
Awareness services
Information space
Physical space Social activity
Figure 7.1 A schema of the dual space of information agents and human users with the mediation services between them.
7.3.1 Awareness and Delivery Services The awareness service takes charge of pushing information from the physical space to the information space. Specifically, the awareness service provides real-time information about the physical location and movements of users to CAPIAs. The specific data provided depends on the particular sensors available in the awareness service for a particular application. For instance, in the conference centre application the awareness service provides a real-time tracking of attendees’ location as well as the group of other attendees nearby a given attendee – see Section 7.4 for the features of the awareness service in the COMRIS Conference Centre application. The delivery service offers mediation and brokerage capabilities (subscribed by the human users) for delivering information from the information space to the physical space. Specifically, the delivery service provides the channels for delivering the information gathered by the CAPIAs to their corresponding users. For instance, in the conference centre application the delivery service allows information such as audio output to be sent by means of a wearable computer and HTML pages by means of screen terminals scattered through the conference building.
011
11
7.3.2 Agents Requirements The society of agents has to be able to communicate using a common ontology for a specific application, and they have to share a collection of interaction protocols appropriate for that application. Our approach is to use the notion of agent-mediated institution (Noriega and Sierra, 1999)
105
Inhabited Information Spaces
to specify the ontology and interaction protocols to be used by a society of agents for a particular application. In addition, to support the ontology and interaction protocols of an agent-mediated institution the agents should be able to handle contextawareness information. That is to say, a context-aware agent should be able to react dynamically when new physical context information is received from the awareness service. Moreover, since the future physical and social context of the user is not known, a desired feature of CAPIAs is the capability of gathering information that may become relevant in a future context. For instance, in the conference centre application, when an attendee is at a specific exhibition zone, the CAPIAs use the knowledge provided by the conference about the physical distribution of booths for trying to anticipate the next movement of the attendee. In our framework, CAPIAs are based on the distinction between two kinds of information valuation, namely interestingness and relevance. Information interestingness measures the intersection of a given piece of information with the user model a CAPIA has for the tasks with which it is charged. That is, interestingness: Info × UM |→ eI where Info is a given piece of information; UM is the user model; and eI is the estimation of the interest that the user has in Info. For instance, in the conference application a preliminary criterion for determining the interestingness of a given paper presentation is performed comparing the user interests (described as a collection of topics with different weights) and the keywords associated with the presentation (also described as a collection of topics with different weights). Then, other criteria such as the knowledge about the speaker or the user’s agenda increase or decrease the initial assessment. Depending on the physical and social context of the user and on the time, however, some information may be more or less relevant for the user at each particular point of time. Information relevance measures this intersection of a given information with the time and the context of the user. That is, relevance: Info × Time × UC |→ eR where Info is a given information; UC is the user context; and eR is the estimation of the relevance of Info in UC. For instance, in the conference application when an attendee is near an exhibition booth, the information related to the booth is estimated as more relevant. Another example of increase of relevance is when a conference event is close to start: a CAPIA has a time constraint for deciding if that event is useful for the user interests. We can say that, basically, the CAPIAs go on their tasks, interacting with other agents (and other accessible information resources), to gather information that is interesting for their users. Concurrently, each CAPIA uses an awareness service (see below) to keep track of the whereabouts of its user and decides which information is relevant for the user in a particular physical and social context. Clearly, interestingness and relevance are not completely independent, and the information gathering is correlated with the information the agent expects to deliver to the user, but for exposition purposes it is handy to talk about them separately.
106
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
011
11
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
At this point it is useful to put this framework into a concrete application to illustrate the dual space and the personal information agents’ exploitation of context awareness.
7.4 The COMRIS Conference Centre This section applies the framework of context-aware information agents to a particular application, that is to say in the physical location and the social activity context of a conference centre in the COMRIS project.2 We view the conference centre (CC) as an agent-mediated institution where a society of information agents work for, and are aware of, the attendees of a conference (Plaza et al., 1998). The ontology of the CC institution defines the conference activities that take place in the CC. Examples of conference activities are exhibition booths and demo events, plenary and panel sessions, and so on. The ontology also defines the roles that a person takes in different locations while performing different activities, such as speaker, session chair, attendee, organisation staff, etc. Other important elements defined by the CC ontology are the different locations of the conference such as the exhibition areas, the conference rooms, and the public areas – i.e. halls, cafeterias and restaurants. This information is used by the agents for reasoning about the movements of users in the conference. The schedule of conference events is also defined in the CC ontology. Finally, the CC ontology supports the definition by each user of the “instruction set” that their CAPIA should follow. The instruction set is entered by the conference attendee using a WWW browser while registering, and basically includes (1) an interest profile (specifying the topics, with weights, in which the attendee is interested); (2) those tasks the user commissions the PIA to do on their behalf (e.g. if they are interested in making appointments); and (3) the delivery modes that the CAPIA will use to communicate with the user. We implemented two types of CAPIAs in the CC application: CAPIAs representing interests of attendees and CAPIA advertisers. There is a CAPIA for each attendee, a CAPIA advertiser for each exhibition booth, and a CAPIA advertiser for each paper session. The goal of CAPIA advertisers is to convince people to attend the conference event they are representing.
7.4.1 Delivery Service The delivery service in COMRIS allows the users to receive information in two ways: by means of a wearable computer with text and audio output 2
COMRIS stands for Co-habited Mixed-Reality Information Spaces. More information is available at: http://arti.vub.ac.be/~comris/
107
Inhabited Information Spaces
and by screen terminals scattered throughout the Conference Centre. The wearable computer is used to convey short messages that are relevant for the user with respect to their current physical and social surroundings. The user can walk to a terminal if they wish to have more information about this message or other recent messages they have received. When the user approaches a screen the wearable computer detects this terminal’s identifier, and then sends this identifier to the user’s CAPIA. Once the CAPIA is aware of this situation, the agent sends to that screen the report of the performed tasks and the report of ongoing tasks. The delivery service comprises several components. The first component is the natural language generation (NLG) component. The NLG component receives the message sent by a CAPIA and generates an English sentence explaining the message content and taking into account the current attendee context and the sentences previously generated. Then, when the message has to be delivered as audio, the sentence structure is sent to a speech synthesis component that produces the actual audio heard by the user. Similarly, there are components that transform CAPIA’s messages into HTML or VRML in order to be delivered to the screen terminals.
7.4.2 Awareness Service The awareness service keeps track of the whereabouts of the attendees in the Conference Centre. In the COMRIS CC the detection devices are a network of infrared beacons (marking the different rooms, places and locations in the CC) and the wearable computers carried by the attendees. The COMRIS wearable computer (shown in Figure 7.2 and commonly called parrot) detects the infrared beacons and thus informs the
Figure 7.2 The wearable computer also known as “the parrot”. The CPU is on the front unit while the back one hosts sensors and batteries.
108
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
awareness service of the location of its user. Moreover, the wearable device possesses an infrared beacon, allowing the detection of other persons, wearing a parrot, located nearby. In order to have access to this information, each CAPIA in the information space “subscribes” its user to the awareness service. As a result, the CAPIA receives messages about the changes in location of that person and a list of other people close to that person. When the CAPIA interacts with other CAPIAs (representing other conference attendees), and decides that those CAPIAs are interesting persons, it subscribes those persons to the awareness service. Consequently, the CAPIA is aware of the location of the most interesting persons for its user and detects, for instance, when one of these persons is in the same location as the user – a most relevant situation to push to its user the information concerning that person who is interesting and nearby.
7.4.3 Tasks The tasks that the COMRIS Conference Centre supports are at the core of the activity in the CAPIAs. It is important to remark here that, in order to perform these tasks, the information agents use both the CC ontology and the awareness service to infer the situation of the user. That is to say, knowing that the user is in a particular place, the current time and the activity scheduled by the Conference for that place at that time, the information agent can infer the social activity in which the user is involved. The tasks performed by COMRIS CAPIAs and the scenes in which they are involved are summarised below. ●
011
011
11
●
Information gathering: this task is responsible for establishing initial conversations with other CAPIAs for estimating the interestingness of the attendees or conference events they represent. We say that the information gathering task constructs the interest landscape of a given attendee. The interest landscape holds all the information considered to be useful for the interest of the attendee and is used and refined in the other tasks. When the information-gathering task assesses a conference event with a high interestingness valuation, the information is directly delivered to the attendee. This delivery strategy was adopted for biasing the future decisions of the attendee. In CAPIA advertisers, this task has been specialised for attracting persons who might be interested into the conference events (exhibition booths or conference sessions) they represent. Appointment proposal: in this task, using the interest landscape, the CAPIAs try to arrange an appointment between two attendees. First, CAPIAs negotiate a set of common topics for discussion (the meeting content). When they reach an agreement, CAPIAs negotiate on the appropriate meeting schedule.
109
Inhabited Information Spaces
●
●
Proximity alert: in this task an attendee is informed that she is physically near to another person with similar interests – or near an exhibition booth or a thematic session with similar topics. Commitment reminder: this task is responsible for checking if attendees are aware of their commitments. The CAPIA uses context to determine that the user may be unaware of a commitment, e.g. if she is not near the location of an appointment (or a bus scheduled to leave) a few minutes before. Commitments of attendees are only noticed when the context information available to CAPIAs indicates that the attendee is not aware of the commitment (e.g. it is five minutes before the starting of a session chaired by the attendee and the attendee is physically in another place).
For each task several activities are launched in the CAPIA. For instance, when an agent in COMRIS is discussing appointments with several CAPIAs, each thread of interaction is managed by a distinct activity. The activities can start either by an internal decision of a CAPIA or because a CAPIA has received a request from another CAPIA.
7.5 Conclusions We have shown a specific context-aware application (the COMRIS Conference Centre) developed in the framework of the COMRIS project that illustrates the interplay of physical space and information space. We have seen that the physical infrastructure, consisting of an individual wearable computer and localisation beacons, was used as a “awareness service” by a society of agents inhabiting the information space. We have focused on the kind of software required to develop a context-aware application, showing that using an agent-based approach we can fulfil the properties we required in what we called continuative computing. We also discussed the kind of agent architecture that can exploit context awareness. The approach we presented was based on the idea of having a personal agent per person. This allows the continuative dimension of processing to be user-centred. As the user changed context, the personal agent received the corresponding perception data from the awareness service and followed the user to a new physical context. In addition to this, services that need to be provided to the users (services provided by the Conference organisation in our example) are agentified, that is, they are also provided by agents (and they also are aware of the contexts in which they are interested). Moreover, this approach is scalable: the agent could use another awareness service at a different conference centre. Moving from one application context to another (from one conference to another) requires a standardisation effort of the awareness services, but this effort is reasonable since it can improve the performance and lower the costs.
110
1 2 3 4 5 6 7 8 9 1011 1 2 3 4 5 6 7 8 9 2011 1 2 3 4 5 6 7 8 9 3011 1 2 3 4 5 6 7 8 9 4011 1 2 3 4 5 611
011
011
011
The Computational Interplay of Physical Space and Information Space 7 ●●●●●●●● ● 1 ●●●●●●
We see thus that context awareness can be integrated into an agentbased paradigm in a well-understood way. A better infrastructure on context perception, as we can expect to be developed in the next decade, can be integrated in the agent-based paradigm without major problems. The main reason for this is the AI approach to agents that employ ontologies describing the world. Clearly, with a better perception infrastructure the agents could perform better inferences about the state of the world. For instance, in the COMRIS conference centre the agents knew when two people were in the same room (using beacons) and when two people were in front of each other (using the wearable computer beacons), but since there was no microphone on the wearable computer there was no way to know if the user was already busy talking with someone or not. In addition to improved awareness services and perception infrastructures, a second issue that was considered in the COMRIS project but never tried was that of learning. Agent learning