Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences: Sharing Good Practices of Research, Experimentation and Innovation

TOWARDS SUSTAINABLE AND SCALABLE EDUCATIONAL INNOVATIONS INFORMED BY THE LEARNING SCIENCES

Frontiers in Artificial Intelligence and Applications Volume 133 Published in the subseries

Knowledge-Based Intelligent Engineering Systems Editors: L.C. Jain and R.J. Howlett Recently published in KBIES: Vol. 115. Vol. 104. Vol. 102. Vol. 87. Vol. 86. Vol. 83. Vol. 82. Vol. 79. Vol. 72. Vol. 69.

G.E. Philips-Wren and L.C. Jain (Eds.), Intelligent Decision Support Systems in Agent-Mediated Environments A. Abraham, M. Köppen and K. Franke (Eds.), Design and Application of Hybrid Intelligent Systems C. Turchetti, Stochastic Models of Neural Networks A. Abraham et al. (Eds.), Soft Computing Systems – Design, Management and Applications R.S.T. Lee and J.H.K. Liu, Invariant Object Recognition based on Elastic Graph Matching – Theory and Applications V. Loia (Ed.), Soft Computing Agents – A New Perspective for Dynamic Information Systems E. Damiani et al. (Eds.), Knowledge-Based Intelligent Information Engineering Systems and Allied Technologies – KES 2002 H. Motoda (Ed.), Active Mining – New Directions of Data Mining A. Namatame et al. (Eds.), Agent-Based Approaches in Economic and Social Complex Systems N. Baba et al. (Eds.), Knowledge-Based Intelligent Information Engineering Systems and Allied Technologies – KES’2001

Recently published in FAIA: Vol. 132. K. Nakamatsu and J.M. Abe (Eds.), Advances in Logic Based Intelligent Systems – Selected Papers of LAPTEC 2005 Vol. 131. B. López et al. (Eds.), Artificial Intelligence Research and Development Vol. 130. K. Zieliński and T. Szmuc (Eds.), Software Engineering: Evolution and Emerging Technologies Vol. 129. H. Fujita and M. Mejri (Eds.), New Trends in Software Methodologies, Tools and Techniques – Proceedings of the fourth SoMeT_W05 Vol. 128. J. Zhou et al. (Eds.), Applied Public Key Infrastructure – 4th International Workshop: IWAP 2005 Vol. 127. P. Ritrovato et al. (Eds.), Towards the Learning Grid – Advances in Human Learning Services Vol. 126. J. Cruz, Constraint Reasoning for Differential Models Vol. 125. C.-K. Looi et al. (Eds.), Artificial Intelligence in Education – Supporting Learning through Intelligent and Socially Informed Technology Vol. 124. T. Washio et al. (Eds.), Advances in Mining Graphs, Trees and Sequences ISSN 0922-6389

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences Sharing Good Practices of Research, Experimentation and Innovation

Edited by

Chee-Kit Looi National Institute of Education, Singapore

David Jonassen University of Missouri, USA

and

Mitsuru Ikeda Japan Advanced Institute of Science and Technology, Japan

Amsterdam • Berlin • Oxford • Tokyo • Washington, DC

© 2005 The authors. All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher. ISBN 1-58603-573-8 Library of Congress Control Number: 2005935898 Publisher IOS Press Nieuwe Hemweg 6B 1013 BG Amsterdam Netherlands fax: +31 20 687 0019 e-mail: [email protected] Distributor in the UK and Ireland IOS Press/Lavis Marketing 73 Lime Walk Headington Oxford OX3 7AD England fax: +44 1865 750079

Distributor in the USA and Canada IOS Press, Inc. 4502 Rachael Manor Drive Fairfax, VA 22032 USA fax: +1 703 323 3668 e-mail: [email protected]

LEGAL NOTICE The publisher is not responsible for the use which might be made of the following information. PRINTED IN THE NETHERLANDS

v

Programme Committee Conference Chair Yam-Sam Chee, National Institute of Education, Singapore Programme Co-chairs Chee-Kit Looi, National Institute of Education, Singapore David Jonassen, University of Missouri, USA Mitsuru Ikeda, JAIST, Japan Programme Committee Wing-Kee Au, University of South Australia, Australia Lyn Henderson, James Cook University, Australia Jon Mason, Education.Au Limited, Australia Ron Oliver, Edith Cowan University, Australia John Hedberg, Macquarie University, Australia Marlene Scardamalia, University of Toronto, Canada Carl Bereiter, University of Toronto, Canada Jacqueline Bourdeau, Tele-universite du Quebec, Canada Jim Greer, University of Saskatchewan, Canada Gordon McCalla, University of Saskatchewan, Canada Nelson Baloian, Chile University, Chile Qiong Wang, Peking University, China Zhiting Zhu, East China Normal University, China Ronghuai Huang, Beijing Normal University, China Ulrich Hoppe, University Duisburg, Germany Fong-Lok Lee, Chinese University of Hong Kong, Hong Kong Carmel McNaught, Chinese University of Hong Kong, Hong Kong Eugenia Ng, Hong Kong Institute of Education, Hong Kong Amy Soller, Institute for Defense Analyses, USA Kanji Akahori, Tokyo Institute of Technology, Japan Yoneo Yano, Tokushima University, Japan Tsukasa Hirashima, Hiroshima University, Japan Naomi Miyake, Chukyo University, Japan Riichiro Mizoguchi, Osaka University, Japan

vi

Kiyoshi Nakabayashi, NTT Resonant, Inc., Japan Yong-Se Kim, Sungkyunkwan University, Korea Duk-Hoon Kwak, Korea National Open University, Korea Dae-Joon Hwang, KERIS, Korea Insook Lee, Sejong University, Korea Kwok-Cheung Cheung, University of Macau, Macao Abtar Kaur, Open University Malaysia, Malaysia Lora A Roy O, Eindhoven University of Technology, Netherlands Piet Kommers, University of Twente, Netherlands Kinshuk, Massey University, New Zealand Ray Kemp, Massey University, New Zealand Raymund Sison, De La Salle University, Philippines Lung-Hsiang Wong, National University of Singapore, Singapore Gee-Kin Yeo, National University of Singapore, Singapore Seng-Chee Tan, Nanyang Technological University, Singapore Mun Kew Leong, Institute for Infocomm Research, Singapore Cher-Ping Lim, Nanyang Technological University, Singapore David Hung, Nanyang Technological University, Singapore Tak-Wai Chan, National Central University, Taiwan Shelley Shwu-Ching Young, National Tsing Hua University, Taiwan Fu-Yun Yu, National Cheng-Kung University, Taiwan Robert Lewis, University of Lancaster, UK Don Cunningham, Indiana University, USA Gerry Stahl, Drexel University, USA Dan Suthers, University of Hawai`i at Manoa, USA Chris Hoadley, Penn State University, USA Tom Murray, University of Massachusetts, USA Barry Fishman, University of Michigan, USA Michael J. Jacobson, Korea University, Korea Kohji Itoh, Tokyo University of Science, Japan Akira Takeuchi, Kyushu Institute of Technology, Japan

vii

Organizing Committee International Track Seng-Chee Tan

National Institute of Education, Singapore (Chair)

Huay-Lit Woo

National Institute of Education, Singapore (Secretary)

Yam-San Chee

National Institute of Education, Singapore (Doctoral Students Consortium Co-chair)

Kinshuk

Massey University, New Zealand (Doctoral Students Consortium Co-chair)

Mun-Kew Leong

Institute of Infocomm Research, Singapore (Interactive Events Co-chair)

Lung-Hsiang Wong

National Institute of Education, Singapore (Interactive Events Co-chair)

Gee-Kin Yeo

National University of Singapore, Singapore (Workshop/Tutorial Co-chair)

Tsukasa Hirashima

Hiroshima University, Japan (Workshop/Tutorial Co-chair)

Wenli Chen

National Institute of Education, Singapore

Ching-Puang Chin

Nanyang Technological University, Singapore

David Junsong Huang

National Institute of Education, Singapore

David Wei-Loong Hung

National Institute of Education, Singapore

Cher-Ping Lim

National Institute of Education, Singapore

Li-Ching Ng

National Institute of Education, Singapore

Gwendoline Choon-Lang Quek

National Institute of Education, Singapore

Shirley Seet

National Institute of Education, Singapore

Qiyun Wang

National Institute of Education, Singapore

Angela Foong-Lin Wong

National Institute of Education, Singapore

Philip Siew-Koon Wong

National Institute of Education, Singapore

Sheryl Eunice Wong

Nanyang Technological University, Singapore

viii

Local Track Sai-Choo Lee

Ministry of Education, Singapore (Chair)

Yew-Meng Kwan

Ministry of Education, Singapore

Bee-Kim Lim

Ministry of Education, Singapore

Elsie Mathews

Ministry of Education, Singapore

Shoo-Soon Wee

Ministry of Education, Singapore

ix

Additional Reviewers Rhodora Abadia Kanji Akahori Charoula Angeli Lora Aroyo Nelson Baloian Zhang Baohui Dencho Batanov Lars Bollen Doris U. Bolliger Jacqueline Bourdeau Christopher Brook Christopher Brooks Katy Campbell Janet Casperson Tak-Wai Chan Ben Chang Legend Chang Victor Chen Hercy N.H. Cheng Kwok-cheung Cheung C. H. Chiu Jenny Chiu Doug Clark Merlin Cruz Donald Cunningham Ian Douglas Ben Daniel Mansor Bin Fadzil Barry Fishman Isaac Pak-Wah Fung Janice Gobert Tiong Thye Goh Ian Green

Jim Greer Xiao qing Gu Andreas Harrer Yusuke Hayashi John Hedberg Eva Heinrich Lyn Henderson Tara Higgins Tsukasa Hirashima Christopher Hoadley Jeff Holmes Ulrich Hoppe Tomoya Horiguchi Ronghuai Huang Liu Huanglingzi David Hung Lynn Hunt Dae-Joon Hwang Kohji Itoh Michael Jacobson David Jonassen Youngcook Jun Tomoko Kojiri Siu Cheung Kong Djordje Kadijevich Kenji Kaijiri Toshinobu Kasai Akihiro Kashihara Abtar Kaur Elizabeth Kemp Ray Kemp Yong Se Kim K Kinshuk

Ingo Kollar Tatsuhiro Konishi Konishi Oskar Ku Duk Hoon Kwak Chih Hung Lai Fong-Lok Lee Mun-Kew Leong Robert Lewis Cher Ping Lim Eric Zhi-Feng Liu Christian Loh Joseph Luca Toshiki Matsuda Naomi Miyake Yusuke Morita Jon Mason Yukihiro Matsubara Kenji Matsuura Gord McCalla Kevin McElhaney Mark McMahon Sue McNamara Carmel McNaught Qi Meng Kazuhisa Miwa Yasuo Miyoshi Riichiro Mizoguchi Connie Moss Antoinette Muntjewerff Tom Murray Kiyoshi Nakabayashi Eugenia Ng Trang Nguyen

x

Roger Nkambou Michael Nott Ron Oliver Innwoo Park Niels Pinkwart David Raymond Ramesh Rayudu Rick Roder Hilary Rubin Priya Sharma Raymund Sison Jim Slotta Glenn Snelbecker Amy Soller

Gerry Stahl Sue Stoney Jan-Willem Strijbos Dan Suthers Katsuaki Suzuki Akira Takeuchi Seng Chee Tan Tiffany Tang Errol Thompson Roger Tipling Satoshi Togawa Maomi Ueno Kyoko Umeda Michael Verhaart

Eric Wang Toyohide Watanabe Rodney Williams Mike Winter Lung Hsiang Wong Chong-Woo Woo Hu Xiaoyong Lisa Yamagata-Lynch Yoneo Yano Gee Kin Yeo Shelley Shwu-ching Young Fu-Yun Yu Fei Yuan Yueliang Zhou

xi

Sponsors Platinum Sponsors

Diamond Sponsors

Gold Sponsors

xii

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Preface The 13th International Conference on Computers in Education (ICCE 2005) is being held from 28 November to 30 December 2005 in Singapore. It is the latest in a longstanding series of annual international conferences held in the Asia-Pacific region, highlighting top quality research on the application of computers in education. The theme of the 2005 conference is “Towards sustainable and scalable educational innovations informed by the learning sciences” with a subtitle of “Sharing good practices of research, experimentation and innovation.” This theme reminds us that we need to be cognizant of research that can inform and lead to sustainable and scalable models of innovation. In order to do so, we need to take an inter-disciplinary view of learning, such as that embraced by the learning sciences. One of the basic principles that underpin the learning sciences is to improve theories of learning through the design of powerful learning environments that can foster meaningful learning. Learning sciences researchers prefer to research learning in authentic contexts. They collect both qualitative and quantitative data from multiple perspectives and follow developmental micro-genetic or historical approaches to data observation. Learning sciences researchers conduct research with the intention of deriving design principles through which change and innovation can be enacted. Their goal is to conduct research that can sustain transformations in schools. ICCE 2005 will also demonstrate that learning sciences exist in the Asia-Pacific context. We have researchers and young academics within the Asia-Pacific Society for Computers in Education (APSCE) community who are concerned with issues of conducting research that can be translated into practice. Changes in practice are especially important to Asian countries because their educational systems are more centralized. Therefore, we believe that there is a need to reform pedagogy in a more constructivist and social direction in a scalable way. Overall, we received 271 submissions for full papers and posters, of which 75 (28%) were accepted as full papers and 87 (32%) were accepted as short papers. The conference also includes 9 interactive events, 2 panels, 9 papers in the Doctoral Student Consortium, as well as paper presentations in the Local Track in which teachers in Singapore share practices and experiences of using technology in the classroom. Also in the Proceedings are brief abstracts of the talks of the four invited speakers: Joseph Krajcik of the University of Michigan, Jeremy Roschelle of Stanford Research Institute International, Takashi Sakamoto of the Japan Association for Promotion of Educational Technology, and Stella Vosniadou of the University of Athens. The work to organize a conference of this size is immense. We would like to thank many people who have helped to make it possible. We thank Seng-Chee Tan, the chair of the Local Organizing Committee, and in particular the committee members: Wenli Chen, Ching-Puang Chin, David Junsong Huang, Li-Ching Ng, Choon-Lan Quek, Shirley Seet, Qiyun Wang, Angela Wong, Sheryl Eunice Wong, Philip Wong, and Huay-Lit Woo. The Program Committee is critical to having a strong program. Almost all full paper submissions had 3 reviews while every short paper had 2 reviews. Thanks to all the Program Committee members for completing their reviews. Thanks, too, to the reviewers who were recruited by Program Committee members to help out in this critical task.

xiii

The committees organizing the other events at the conference also have helped to make the conference richer and broader: Doctoral Student Consortium, chaired by Yam-San Chee and Kinsuk; Tutorials and Workshops, chaired by Gee-Kin Yeo and Tsukasa Hirashima; and Interactive Events, chaired by Mun-Kew Leong and Lung-Hsiang Wong. The local track programme, which is a platform for Singapore teachers to share best practices of using technology for learning, is chaired by Sai-Choo Lee assisted by Yew-Meng Kwan, BeeKim Lim, Elsie Mathews and Shoo-Soon Wee. Thanks to all these colleagues. Finally, we would like to thank Ben Chang and Hercy Cheng who provided the excellent conference submission and review system, as well as helped us to construct the proceedings. If you enjoy participating in ICCE 2005 or reading the Proceedings, we recommend that you consider joining the Asia-Pacific Society for Computers in Education (APSCE – http://www.apsce.net/), an active scientific community that helps to forge on-going interactions among researchers in the Asia-Pacific region. The Society organizes the annual ICCE conferences and will start its own journal, namely, Research and Practice in Technology Enhanced Learning. We certainly hope that you will enjoy the ICCE-2005 conference, and that you find it illuminating, stimulating, and enjoyable. Programme Co-chairs Chee-Kit Looi, National Institute of Education, Singapore David Jonassen, University of Missouri, USA Mitsuru Ikeda, Japan Advanced Institute of Science and Technology, Japan

xiv

Using Learning Technologies to Support Students in Developing Integrated Understanding Joseph S. Krajcik Professor of Science Education, University of Michigan, USA

Abstract. Using new technologies, learners have the ability to manipulate and create visualization and animations, create models of dynamic process, access unprecedented information over the Web, use probes to explore phenomena previously inaccessible, and collect and manipulated data. These applications allow students to engage in learning activities that they would not otherwise be able to do, transforming the classroom into a learning environment where students actively construct knowledge and build integrated understands. Researchers have now collected a growing body of evidence to support these claims. However, in themselves, learning technologies will not promote learning. To realize the their potential, certain conditions need to be met. These conditions include alignment of the technology use with the learning goals of the curriculum, teachers knowledgeable in using technology to promote learning, and technology designed for learners. In our work at the Center for Highly Interactive Classrooms, Curricula, Computing in Education (hi-ce) at the University of Michigan, we have explored for more than a decade how the to integrate technology into curriculum to meet learning goals and how to design technology for learners. In this session, I will discuss the potential of learning technologies and share how we have integrated technologies in curriculum materials so that they help learners build understanding that meet learning goals.

xv

Scaling Up Innovative Technology-Based Middle School Mathematics to a Wide Variety of Teachers Jeremy Roschelle Director, Center for Technology in Learning SRI International Abstract. Technology is often cited for its potential to create new opportunities for more students to develop deeper knowledge (National Research Council, 1999) and for “democratizing access to advanced mathematics.” (Kaput, 1994). The NCTM Technology Principle (ref) states not only that “Technology is essential in teaching and learning mathematics,” but also that technology “influences the mathematics that is taught and enhances student learning.” We report on a program of research that aims to evaluate this claim through a carefully designed randomized controlled field trial. In particular, we present evidence from the completed Phase I and discuss our upcoming Phase II. Prior research strongly suggests a relationship between students’ frequent use of representational tools and advanced levels of mathematics achievement. NAEP data, for instance, shows that frequent graphing calculator use correlates with students success at advanced levels of mathematics achievement. NAEP also highlights the challenge we face in enabling more middle schoolers to move from learning basic skills to mastering material at the more complex and conceptually difficult “proficient” level. Decades of technology R&D have resulted in the approach incorporated in tools such as The Geometer’s Sketchpad, Fathom, ThinkerPlots, and SimCalc, each of which provides new representational infrastructure by taking advantage of the new visual, dynamic, and manipulable properties of computer-based mathematics (Kaput, Noss, & Hoyles, 2002; Pea, 1987). Small-scale studies amply demonstrate that a representational infrastructure (RI) approach can produce gains in deep mathematical understanding for diverse children across a range of settings (Roschelle, Pea, Hoadley, Gordin, & Means, 2000). A metaanalysis that summarized findings from over 100 research studies involving 4,000+ experimental/control group comparisons found that both (a) representing knowledge graphically and (b) using manipulatives to explore new knowledge and practice applying it had an large effect size (Marzano, 1998). But concerns have been raised that only unusually gifted teachers with a high level of support can utilize such innovations. Our Phase I controlled experiment produced findings indicating that a wide variety of teachers with modest support could use SimCalc technology and curricula to improve student learning. Twenty-three 7th-grade math teachers from Texas, representing a wide variety of demographics, prior training, beliefs about students, and situation of teaching, participated in a field experiment in which one group utilized SimCalc technology and curricular materials to teach rate and proportionality while the other group used materials from a highly rated existing rate and proportionality workshop. Teachers were assigned to groups randomly, with some adjustment to ensure that minority and rural teachers were represented in the experimental group. In exit interviews, teachers in the experimental condition reported that they had been able to teach proportionality with new depth. Significant results from other data sets including the Texas Education Agency (TEA), (NAEP) and (TIMMS) corpi have influenced our data collection and choices. We also found that some teachers produced much higher student gains than others. In Phase II, we will collect a larger data set that will enable us to characterize and model the teacher backgrounds, beliefs, and practices best corrected with high student performance with these innovative materials.

xvi

Re-construction of Conceptual Framework on ICT in Education and eLearning in the Networked Learning Community Takashi Sakamoto President, Japan Association for Promotion of Educational Technology Professor Emeritus, Tokyo Institute of Technology, National Institute of Multimedia Education Abstract. The main activities of the AEN include interoperability of technology, international standards and certification, e-learning professional development, quality assurance, management, validation, portal site development and research trends analysis. A survey was conducted on the present state of e-learning in ASEAN+3 countries. In Japan, e-Japan Strategy Statement I & II, e-Japan Priority Policy Program 2003&2004, and IT Policy Package 2005 also promote IT use in education intensively. Based on these trends, changes have occurred in the use of ICT in education and e-learning: changes of human resources requested, need for expertise to deal with the solutions for the sustainable development for saving the environment and the human race, cultivating employment, emphasis on active self-learning in the networked environment, expansion of learning activities in the social context, advances of learning technologies, emphasis on cost-effectiveness and the wide introduction of quality assurance of all kinds of educational components with national and international standards. Following these recent changes in the learning environment, the conceptual framework for dealing with teaching and learning in the networked learning community should be re-constructed. The new paradigm is that teaching and learning in the wired environment represents the true nature of education, and traditional face-to-face lessons in universities and schools are conceptually considered as just one set of all kinds of educational components. Any educational provider can freely participate in the learning community. Even one person can offer courses to everyone in the world and provide any kind of certificate, diploma and degree. One-person private “juku” (cram schools) have appeared. These star leaders can organize consortia for star alliances which might be more influential in the industrial society than in traditional universities. In this context, the important issues seem to be quality assurance of educational systems, tools and contents, setting up national and international standards, copyrights, user protection, and blended learning. There is also a global need for collaboration within each sector as well as across different sectors such as academia, industry, government, culture, languages, countries, and ethnic groups. Advances of recent learning technologies could solve some of these problems. A conceptual framework for including and dealing with items shown above and an example of Job-IT literacy skill matrix standard will be also described.

xvii

Principles for the Design of Learning Environments to Promote Conceptual Change in Science Stella Vosniadou Professor of Cognitive Psychology National and Kapodistrian University of Athens, Greece Abstract. During the last years, a broad consensus has been achieved in educational psychology on certain principles that can guide the design of technology-supported learning environments. For example, most researchers seem to agree that the support of active learning and the guidance of students towards the acquisition of self-regulated processes are important characteristics of powerful learning environments. The importance of having educational tasks that students find relevant and meaningful is another, widely agreed upon principle. Finally, since learning is not an individual but a social affair, researchers usually agree that teachers should encourage collaborative learning in the classroom. In this talk, I will argue that while principles such as the above are necessary to be taken into consideration when designing learning environments, they are not sufficient by themselves. They need to be supplemented by principles emerging from research on the acquisition of subject-matter knowledge, particularly in the case of science teaching. They are at least two good reasons for this argument: First, the scientific ways of establishing reliable empirical knowledge, the methods of scientific argumentation and finally the goals of the scientific enterprise are different from those of other human activities (e.g., mathematics, history, etc.) and need to be taken into consideration when designing learning environments for teaching science. Second, scientific knowledge is the outgrowth of a long historical development that required significant scientific revolutions to take place. The body of knowledge that constitutes current science contradicts basic presuppositions of our intuitive knowledge about the physical world and requires radical conceptual change to be understood. In order to promote conceptual change in science, particular attention needs to be paid to the design of research-based, developmentally appropriate curricula, and to the enhancement of metaconceptual awareness and intentional (purposeful and goal-directed) learning. In this presentation I will focus particularly on the description of a computer-supported collaborative learning environment designed to encourage the development of critical discourse around students’ ideas and beliefs and also to give students more control over their learning.

This page intentionally left blank

xix

Contents Programme Committee

v

Organizing Committee

vii

Additional Reviewers

ix

Sponsors

xi

Preface Chee-Kit Looi, David Jonassen and Mitsuru Ikeda

xii

Keynote Speakers

xiv

Full Papers An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments Gerardo Ayala A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design Using UML Nilufar Baghaei, Antonija Mitrovic and Warwick Irwin A Case Study Describing the Creation and Implementation of a Capacity Building Program for Mixed Mode Delivery of Academic Programmes in Developing Countries Cleo Chadwick, Sheila Howell and Bakari Mwinyiwiwa

3

11

19

Fostering Learning Communities Among Teachers and Students: Potentials and Issues Ching Sing Chai and Seng Chee Tan

27

Group Metacognition in a Computer-Supported Collaborative Learning Environment Chris Chalmers and Rod Nason

35

Examining the Psychological Impact of Video Cases for Teacher Education Peter Y.K. Chan

42

Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring Ya-Wei Chang, Emily Ching, Chih-Ti Chen and Tak-Wai Chan

51

Public Computing, Computer Literacy and Educational Outcome: Children and Computers in Rural India Ritu Dangwal Summative Computer Programming Assessment Using Both Paper and Computer Pari Delir Haghighi and Judy Sheard

59 67

xx

Ontology-Driven Incremental Annotation of Educational Content for Instruction Planning Apple W.P. Fok and Horace H.S. Ip The Development of ClassSim: A Simulated Learning Environment to Support the Practicum Experience for Pre-Service Teachers Brian Ferry, Lisa Kervin, John Hedberg, David Jonassen, Brian Cambourne, Jan Turbill and Lisa Carrington Innovative Learning and Unlearning Di Fleming and Grace Lynch Implementation and Effectiveness of a Feedback Feature in Underlining to Promote Reading Comprehension of e-Learning Instructional Materials Yoshihiro Fukunaga, Tsukasa Hirashima and Akira Takeuchi

76

84

93

101

Interactivity of Exercises in ActiveMath Giorgi Goguadze, Alberto Gonzalez Palomo and Erica Melis

109

EpiList II: Explicit Instructions for Development of Generic Cognitive Skills Ghee Ming Goh and Chai Quek

116

An Evaluation Tool for Collaborative Learning Based on Communication Noriko Hanakawa and Nao Ikemiya

124

Adaptive Learning in a Mobile Environment Tai Wai Ho, Vincent To-Yee Ng, Stephen C.F. Chan and King Kang Tsoi

132

Conceptual Changes in Learning Mechanics by Error-Based Simulation Tomoya Horiguchi, Tsukasa Hirashima and Masahiko Okamoto

140

Exploring Fundamental Issues About Problem-Oriented Hypermedia Michael J. Jacobson

148

A Study of the Development of an Information Literacy Framework for Hong Kong Students Siu Cheung Kong, Fong Lok Lee, Siu Cheung Li and James Henri

156

An Experimental Study of a Cognitive Tool for Classroom Use: A Knowledge of Fraction Equivalence Siu Cheung Kong

164

The Implementation of an Information Literacy Framework: Key Issues in Professional Development Siu Cheung Kong, Fong Lok Lee and Shuk Fan Ip

172

Knowledge Work and Learning at the Workplace – Key Issues and Solutions Alexander Karapidis, Till Becker and Volker Zimmermann Scaffolding the Process of Collaboration: Exploration of Separate Control of Shared Space Lucinda Kerawalla, Darren Pearce, Jeanette O’Connor, Rosemary Luckin, Nicola Yuill and Amanda Harris

180

187

xxi

A System That Generates Word Problems Using Problem Generation Episodes Kazuaki Kojima and Kazuhisa Miwa Detection and Grammaticality Judgment of Forms in a Language Education System Oriented for Focus on Form Makoto Kondo, Takafumi Shiratori, Naoki Minai, Satoru Kogure, Tatsuhiro Konishi and Yukihiro Itoh Concerns and Preferences of Teachers Towards the Development of Virtual Classroom in Hong Kong Siu Keung Lam and Percy Lai Yin Kwok

195

203

211

Constraint-Based Error Diagnosis in Logic Programming Nguyen-Thinh Le and Wolfgang Menzel

220

Model Building as a Platform to Support Thinking in Problem Solving Chwee Beng Lee, Ioan Gelu Ionas and Matthew Schmidt

228

A Study of Curriculum Implementation and School Culture in the Information Age Shinn-Rong Lin, Hsiao-Ting Teng, Huei-Chuan Lin and Chuang-Chieh Chiu Sustaining Innovations in Singapore Schools: Implications for Research Work Chee-Kit Looi, Wei-Ying Lim and David Hung E-Learning System Acceptance: Implications to Institutional Implementation Strategies Will Wai-kit Ma and Allan Hoi-kau Yuen

236 244

252

Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation Naomi Miyake

260

Technology-Based Tools That Facilitate Data-Driven Decision Making Ellen B. Mandinach, Margaret Honey, Daniel Light, Juliette Heinze and Luz Rivas

267

CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD Yongwu Miao Development of an e-Learning Resource for Teacher Education Utilizing a VOD-Compatible Teaching Portfolio Having a Dual-Screen Synchronous Playback Function Hitoshi Miyata

275

283

Teachers as E-Designers: A Novel Approach to Professional Development in Elementary Science and Technology Gillian Mulholland, Lachlan Forsyth and Lynette Schaverien

291

On-Time Support with Supplementary Figures in Tutoring of Mathematical Exercises Yosuke Murase, Tomoko Kojiri and Toyohide Watanabe

299

xxii

The Capabilities Required By Online Tutors Mediated Through Learning Environment Factors C. Paul Newhouse and Doug Reid

307

Growth of Teacher Understanding of Mathematics Within a CSCL Environment Rod Nason and Mathew McDougall

315

Are Student Teachers Ready for E-Learning? Eugenia M.W. Ng

323

Learning Designs and Learning Objects: Where Pedagogy Meets Technology Ron Oliver, Ralph Wirski, Lisa Wait and Vivienne Blanksby

330

The Task Sharing Framework: A Generic Approach to Scaffolding Collaboration and Meta-Collaboration in Educational Software Darren Pearce, Lucinda Kerawalla, Rose Luckin, Nicola Yuill and Amanda Harris Structuring of Multimedia Contents for E-Learning Lessons in Multimedia Knowledge Base Rafisha Ramly, Eng-Thiam Yeoh and Kai-Li Ng Dialogue Games and e-Learning: The Interloc Approach A. Ravenscroft and S. McAlister From Response Systems to Distributed Systems for Enhanced Collaborative Learning Jeremy Roschelle, Patricia Schank, John Brecht, Deborah Tatar and S. Raj Chaudhury

338

346 355

363

Understanding Students’ Online Forum Usage Paulus Insap Santosa, Gee Kin Yeo and Julian Lin

371

Use of Dataloggers in Science Learning in Singapore Schools Whye Choo Seah, Daniel K.C. Tan, John G. Hedberg and Thiam Seng Koh

379

A Human Resource Model and Evidence Based Evaluation – Ontology for IT Skill Standards – Kazuhisa Seta, Mitsuru Ikeda, Kenji Hirata, Yusuke Hayashi and Ken Kuriyama Human Factor Modeling for Development of Learning Support System Facilitating Meta-Cognition Kazuhisa Seta, Kei Tachibana, Motohide Umano and Mitsuru Ikeda Digital Information Literacy: Explorations of History Tasks in Singapore Schools Sunita Shankar and John G. Hedberg The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns in a Computer-Mediated Learning Environment Hyo-Jeong So

388

396 404

413

xxiii

An Intelligent Agent that Learns How to Tutor Students: Design and Results Leen-Kiat Soh and Todd Blank

420

Computer-Supported Structured Cooperative Learning Leen-Kiat Soh, Nobel Khandaker, Xuli Liu and Hong Jiang

428

Sustaining Online Collaborative Problem Solving with Math Proposals Gerry Stahl

436

CHALLENGE FRAP – A Combination Guide and Reporting Tool for Problem-Based Exercises Terry M. Stewart, William R. MacIntyre and Victor J. Galea

444

Malaysian Female Pre-Service Teachers Online: Exploring Their Internet Use and Attitudes Su Luan Wong, Siew Fung Ng and Sai Hong Tang

452

Automatic Extraction of Learning Object Metadata (LOM) from HTML Web Pages Wai Yuen Tang and Lam For Kwok

460

Establishing a Probabilistic Model for Cognitive Learning Style Yoke-Jin Teh, Choo-Yee Ting and Somnuk Phon-Amnuaisuk DrilLs-M: Multimedia Testing System with Operation-Style Answering – Application to Kanji Letter Shape Learning – Hiroyuki Tominaga, Yu Kinugasa, Naoko Yamashita, Toshihiro Hayashi and Toshinori Yamasaki

468

476

The Development and Evaluation of a Japanese Language Paper-Grading Support System for the University Level Mio Tsubakimoto and Kanji Akahori

484

The Effectiveness of a Study Support System Based on Mobile Phones and Web-Based Information Sharing: Reporting Activities in a Class for the First Grade of an Elementary School Makiko Takenaka, Shigenori Inagaki, Hideko Kuroda, Masahiko Ohkubo and Akiko Deguchi

492

Probabilistic Learner Modeling in Scientific Inquiry Exploratory Learning Environment Choo-Yee Ting and M. Reza Beik Zadeh

500

A Research-Supported Approach to Using Weblogs as a Sustainable Educational Innovation in an Australian Post-Primary School John Turner and Adam Usher

508

Design Method for Learning Games Based on Key Structure and Behavior Takanobu Umetsu and Tsukasa Hirashima

516

xxiv

Automated Scoring for Creative Problem Solving Ability with Ideation-Explanation Modeling Hao-Chuan Wang, Chun-Yen Chang and Tsai-Yen Li

524

Teaching Strategies Ontology Using SWRL Rules Eric Wang, Leila Kashani and Yong Se Kim

532

Spatial Multiagent Student Diagnosis Carine Webber

540

Student Profiles, Reflection and Developing Autonomy in the Learning Process Ray Webster

548

Improving Chinese Language Competency of Ethnic Chinese Pupils with Story Building and Multi-User Virtual Environment – A Case Study Yin Mei Wong

556

Promoting Metacognitive Strategy Development Through Online Question-Generation Instructional Approach Fu-Yun Yu

564

Using Self-Response System and Online Learning Environment to Promote Science Learning in a Large College Class BaoHui Zhang, Ronald Patterson, Gail Richmond, Joyce Parker, John Merrill, Mark Urban-Lurain, Scott Rollins, Everett Weber and Tammy Long

572

A Three-Dimensional Context-Awareness Model: Support Learning Services Providing in e-Learning Environments Yanlin Zheng, Luyi Li, Hiroaki Ogata and Yoneo Yano

579

Effects of Resource Distribution and Feedback on Computer-Mediated Collaboration in Dyads Joerg Zumbach, Peter Reimann and Jochen Schönemann

587

Short Papers Online Feedback and the Real-Time Evaluation: Integrating Wireless Technology into Instructional Strategies Lyna, Lian Sai Chia, Ngoh Khang Goh and Widya Andyardja Weliamto

597

Comparative Study of Learning Effectiveness Between a Mobile Learning Tool and Paper Kanji Akahori, Yuuki Kato, Shogo Kato, Atsuhiko Sudo and Masayuki Ohmori

601

From Research and Development to Field Application: Collaboration Between Laboratory and Service Organization for Online Professional Education D. Michael Anderson and Penelope A. Lantz

605

A Resourceful Partnership: Work in Progress on the UK Teacher Training Resource Bank (TTRB) Michael Blamires and Simon Hughes

609

xxv

Communication Control and Support for Scheduled Collaborative E-Learning Freimut Bodendorf and Manfred Schertler Adaptive Learning Interface Customization Based on Learning Styles and Behaviors Hyun Jin Cha, Yong Se Kim, Sun Hee Park, Yun Jung Cho and Michael Pashkin Effect of Female Role Models in IT on Young Girls’ Attitudes Toward Computing—an Experimental Study Pi-Chi Chen and Pei-Chi Chan Learning OO with Peers Weiqin Chen, Jan Dolonen and Lars Eirik Rønning Gender Differences in Taiwan University Students’ Attitudes Toward the Web-Based Learning Ru-Si Chen and Chin-Chung Tsai Motivating Students by Technologies: A Case Study on Adopting TIPS Pedagogy in a General Physics Course Yu-Fen Chen, Sung-Bin Chang, Chen-Chung Liu, Yi-Wen Chu, Ben Chang and Tak-Wai Chan

613

617

621 625

629

633

Item Attributes Analysis of Computer-Based Test Based on IRT Deng-Jyi Chen, Ah-Fur Lai and Shu-Ling Chen

638

Enhancing the Classroom Learning Experience with Web Lectures Jason Day, Jim Foley, Remco Groeneweg and Charles van der Mast

642

Ontologies for Self-Regulated Learning Moma de la Torre and Raymund Sison

646

eThemes: An Internet Instructional Resource Laura Diggs and John Wedman

650

Pedagogical Agents in Multimedia Learning Environments: Do They Facilitate or Hinder Learning? Steffi Domagk and Helmut M. Niegemann

654

A Personalized Agents Platform Design and Implementation for Personalized Education Apple W.P. Fok, Xin Xiao, Yuanchun Shi and Horace H.S. Ip

658

AMPLIA Learning Environment Architecture Cecilia Flores, Louise Seixas, João Gluz, Rosa Vicari, Diego Patrício, Felipe Giacomel and Leandro Gonçalves

662

A Learner Model in a French Reading Tutor Didier Fragne, Rabiha Faqir and Nabila Sayouri

666

On Finding the Social Structure Among Learners in Group Learning Environments Isaac Pak-Wah Fung

670

xxvi

Enriching Specifications for Re-Usable Adaptive Learning Design Katerina Georgouli and Pedro Guerreiro

674

Teaching Thinking Skills in E-Learning – Application of the Bloom’s Taxonomy Tse Yew Lee and Tiffany Ho Peiwen

678

Higher Order Applications of Technology: Perception of WebQuests Among Pre-Service Teachers Hsiang-Ju Ho and Cynthia C. Choi

682

Educational Robot – A Case Study for Robot Learning Companion Sheng-Hui Hsu, Jen-Hang Wang, Ben Chang, Jwu-E Chen, Kuo-Chin Fan and Tak-Wai Chan

686

Late-Binding SCORM Contents in Dynamic Learning Environments Shueh-Cheng Hu

690

Pupils’ Attitudes Towards Using Cellular Phones for Learning Activities Tadashi Inagaki, Yuki Kobayashi and Hitoshi Nakagawa

694

Analysis of Collaborative Problem Solving Based on Thinking Styles Takafumi Ichihara, Kazuhisa Miwa and Norio Ishii

698

Project to Deliver Online Distance Learning for “Introduction to Database Systems” Between Two Campuses Nami Imanishi, Kuniharu Yokomura and Tsuyoshi Sano Design of a Guided Web Mail System for Senior Citizens Yurie Iribe, Kiichirou Sasaki, Mamoru Endo, Takami Yasuda and Shigeki Yokoi Stream Differences in Asynchronous Online Discussions: Analysing Student Cohesion and Roles in Technology-Mediated Project Work Classrooms Azilawati Jamaludin and Choon Lang Quek Learning Conflicting Information: Reading Versus Discussion Heisawn Jeong

702 706

710 715

Teacher Perception of the Use of Information Technology in Teaching: A School Case in Hong Kong Stephen Ping Wah Kong and Percy Lai Yin Kwok

719

The Effects of Textual Complexity Decrease on Activating Graphical Information in Second Language Reading Comprehension Yukari Kato

724

SMARTConsultant: Asking The Right Questions Elizabeth A. Kemp, Hiba Mustafa, David I. Gray and Raymond H. Kemp

728

ADEPT: A Tool for Developing Synthesis Skills Elizabeth A. Kemp, Timothy Pick and Raymond H. Kemp

732

xxvii

Extricating the Web of Learning: The Case for Learning Communities Elaine Khoo

736

Learning with Computer Modeling Software in a High School Chemistry Class: A Study of Conceptual Growth and Transfer HyungShin Kim

740

Development and Assessment of the “Science Net” e-Learning System for Science Education at the Elementary School Level Takeshi Kitazawa, Masahiro Nagai, Hiroshi Kato and Kanji Akahori

744

Automatic Generation of Courseware Reflecting Author’s Instructional Strategy Tomoko Kojiri, Yuji Kanematsu and Toyohide Watanabe

748

Teacher Development Through “Active Classroom” – A Learning Platform to Support School Development and Education Reform Bick Har Lam and Alison See Wai Yeung

752

An Intention-Oriented Model of Copyright Law for e-Learning: International Semantic Mapping of Copyright Laws Based on a Copyright Ontology Wenhuan Lu and Mitsuru Ikeda

757

Predicting High-Level Student Responses Using Conceptual Clustering Roberto Legaspi, Raymund Sison, Ken-ichi Fukui and Masayuki Numao Combinary Competence Reflections on Multidisciplinary Approaches to Digital Media and Learning Gunnar Liestøl The Development of a 3D Visualization Tool on the Topic of Cold Front Ming-Chao Lin and Chun-Yen Chang Action Research on Construction of Basic Courses Chief Teachers Pedagogical Model Based on School Network QiaoZhi Liu, LiMing Chen and ZhaoDe Zhou Multi-Tiered Peer Learning Support C.-K. Looi, E. Ong and L.-H. Wong An Initial Survey on Student Attitudes Toward Working with Marked Examples in the E-Learning Context Jia-Yi Lu, Eva Heinrich and Isaac Pak-Wah Fung

761

765 771

775 779

783

Promoting Critical Reflective Capabilities in a CSCL Environment Ada W.W. Ma

787

Password Input Method Using Icons for Primary School Children Takahiko Mendori, Natsuko Ikenoue and Akihiro Shimizu

791

Real Time Advice System for Sketch Learning Hirotaka Maeno, Tomohiro Iwaki, Masato Soga, Noriyuki Matsuda, Saeko Takagi, Hirokazu Taki and Fujiichi Yoshimoto

795

xxviii

Tangible Collaborative Learning Support System for Spatio-Temporal Contents Yuta Maruyama, Keigo Kitahara, Tomoo Inoue, Hiroshi Shigeno and Kenichi Okada Overcoming Hard-To-Reach Toolboxes with On-Demand Menu Options for Cascading Electronic Whiteboards in Classrooms Khaireel A. Mohamed and Thomas Ottmann

799

803

A Formal Representation for Interactivity in Asynchronous Communication Gyo Sik Moon

807

IT Enriched Science Education for Rural Schools B. Krishnarajulu Naidu and Jayanthi Sivaswamy

811

Inspection by Eye Tracking for Interactive e-Learning Materials with Synthesized Voice Navigations Kiyoshi Nosu, Ayako Kanda and Takeshi Koike

815

Multimodal and Textual Analysis of Students’ Digital Artifacts Uma Natarajan, Muthu Kumar and Libo Guo

819

Stylus or the Mouse? Using a Tablet PC with Young Children C. Paul Newhouse

824

What Made Student Teachers Sustain Their Interests in an Educational Game? Eugenia M.W. Ng and Lap Piu Lee

828

Students’ Metacognition, Strategy, and Affective Anxiety While Executing a Spreadsheet Activity Trang Thi Thuy Nguyen and Lyn Henderson Effectiveness of Collaborative Learning in the Network-Based Language Communities Yuri Nishihori, Nozomu Nishinaga, Keizo Nagaoka, Yoko Collier-Sanuki, Kenji Tanaka, Yuichi Yamamoto and Masahiro Harada Evolving Participation in Educational Technology Communities of Practice Kathy Nolan, Mhairi (Vi) Maeers, David Friesen and Alec Couros

832

836

840

Design and Implementation of Spoken Dialogue Based Multiplication Table Learning Support System Yasuhisa Okazaki, Hiroyuki Egashira, Kenzi Watanabe and Hiroki Kondo

844

Using a Blended Learning Approach to Support Problem-Based Learning with First Year Students in Large Undergraduate Classes Ron Oliver

848

Personal Stereo Videoconferencing System Supporting 3D Telepointers and Annotations Noritaka Osawa and Norio Takase

852

BeLife: A Simulation Tool to Support Learning About Photosynthesis Nuno Otero, Marcelo Milrad, André Vala and Ana Paiva

857

xxix

A Web-Based Basic Language Tutor Phanwoo Park and Tanja Mitrovic

861

Learning Sequences Construction Using Ontology and Rules Ruei-Yan Chen, Shian-Shyong Tseng, Chien-Liang Liu, Chun-Yen Chang and Chang-Sheng Chen

865

A Practice Example of Object-Oriented Programming Education Using WebCT Shigeru Sasaki, Hiroyoshi Watanabe, Kumiko Takai, Masayuki Arai and Shigeo Takei

871

Utilization of a Reusable File Format for Motional 3D Objects on an Educational System Kenji Saito, Keisuke Nakamura, Hajime Saito and Takashi Maeda

875

Design Strategies and Principles in VISOLE Junjie Shang, Fong-Lok Lee and Jimmy Ho-Man Lee

879

Multimodality-Based Model in the Class of Foreign Language Cecilia Silva

883

Paralinguistic Discussion in an Online Educational Setting: A Preliminary Study Karin Sixl-Daniell and Jeremy Williams

887

Three Stages for the Scenarisation of Learning Objects Sofiane Aouag

892

Analyzing Student Motivation and Self-Efficacy in Using an Intelligent Tutoring System Leen-Kiat Soh and Lee Dee Miller A Research Agenda for the Science and Design of Assessment for Deep Learning Michael J. Strait

896 900

Design and Development of a Web-Based Content Creation Application for Problem-Solving Exercises Based on Cognitive Apprenticeship Gek Hua Tan

904

Knowledge Building in Inter-School Learning Communities: Reflections from a Case on Project Learning in Hong Kong Christopher Tan and Percy Kwok

908

Verification of Prerequisite Relationship Among Learning Objects King Kang Tsoi and Vincent To-Yee Ng

914

The Development of the IASv1 Internet Anxiety Scale Kyoko Umeda, Mina Kurashiro, Tetsuro Ejima and Hironari Nozaki

918

Development and Evaluation of a Learning Support System for Learning by Following Akira Urao and Kazuhisa Miwa

922

xxx

E-Learning and Organisational Change: A Change Management Case Study Ray Webster

926

ShowMe the World: Learning with Global Peers John Wedman and Laura Diggs

930

An Instructional Model for Learning Theorem Proving with Dynamic Geometry Environment Wing-Kwong Wong, Bo-Yu Chen and Sheng-Kai Yin

934

An Eclipse Plug-in for SVG Animations in an Educational System for Programming Naoki Yanagisawa, Kenichi Takai, Koji Kagawa and Hiroyuki Tarumi

938

An Evaluation on Using Virtual Reality and 3D Visualisation in Science Education Yau-Yuen Yeung

942

First Observations from Reflective Learning on a Knowledge Repository Gee Kin Yeo, Paulus Insap Santosa, J.B. Foo and P.C. Low

946

An Open-Ended Framework for Learning Object Metadata Interchange Yuejun Zhang, K. Kinshuk and Taiyu Lin

950

A New Role of Lifelong Learning Support System Wei Zhou, Takami Yasuda and Shigeki Yokoi

954

DSC Papers Developing Multimedia Courseware Based on Multiple Intelligences Theory Bushro binti Ali

961

Classwride Computer Supported Reciprocal Peer Tutoring for Fraction Learning Yen-Hua Chen, Jen-Hang Wang, Jung-Feng Wu, Yang-Ming Ku and Tak-Wai Chan

963

Evaluation of a Constraint-Based Error Diagnosis System for Logic Programming Nguyen-Thinh Le

965

The Essential Elements in a Community of Enquiry – A Study of Computer-Supported Collaborative Learning in Higher Education Tengku Putri Norishah Tengku Shariman

967

Studying Scientific Epistemological Beliefs in a High-School Science Classroom Jarina Peer

970

Scaffolding Thinking for Novice Video Producers in a Design Critique Discussion Yiong Hwee Teo

972

Complementing a Web-Based Pedagogical Agent with an Enhanced-Search Framework Longkai Wu

974

xxxi

Orchestrating Talk for Meaning Making in a Science Learning Community Mediated by CSCL Jennifer Yeo

976

Collaborative Narratives: Collaborative Learning in the Blogosphere Jude Yew

978

Author Index

981

This page intentionally left blank

Full Papers

This page intentionally left blank

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

3

An Analytic Model based on ASP for Maintaining ZPDs in CSCL Environments Gerardo Ayala CENTIA, Universidad de las Américas, Puebla, México. [email protected] Abstract. This paper presents an analytical model of an agent about the capabilities of a learner for CSCL environments in DLV based on the ASP (Answer Sets Programming) formalism. The complete model is formally presented in the declarative language of DLV, a logic programming language for implementing ASP models. The model proposed provides a representation of the structural knowledge frontier and the social knowledge frontier of the learner, which are the components for the definition of the learner's zone of proximal development (zpd). Based on the zpd of its learner the agent can propose her a learning task and maintain the zpd for the learner in the group. The analytical model, as presented in the paper, runs using the DLV system that can be easily downloaded from the internet. Keywords: analytic model, agents, cscl, group configuration, learner model

Introduction Analytic models are considered a way to model group activities. According to Hoppe & Ploetzner [1] an analytic model, in the context of collaborative learning environments, is a "formally represented computational artifact that can be used to simulate, reconstruct, or analyze aspects of actions or inter-actions occurring in groups". Logical propositions have been useful to represent computational models, like the proposal for student modelling by Self [2] that integrates computational an theoretical aspects for modelling the domain, reasoning, monitoring and reflection levels of an agent. Self [3] also proposed some computational models to represent a viewpoint as a set of beliefs held by the agent. In this approach, the domain model is considered as a viewpoint, and the student model as an incremental viewpoint based on the domain. Logic programming was used in the GRACILE project [4] implementing intelligent agents for a CSCL environment in PROLOG. Declarative logic programming is widely accepted for the representation and manipulation of knowledge and belief systems. Beliefs systems for software agents deal naturally with incomplete information and non-monotonicity in the reasoning process. Nowadays, Answer Set Programming (ASP) is a well logic programming formalism known and accepted for non-monotonic reasoning and reasoning with incomplete information [5]. However, there are few real applications based on ASP an none concerning learning environments. In the context of agent based Computer Supported Collaborative Learning (CSCL) environments, logic programming allow us a more human-like heuristic approach for learner modelling and reasoning on learning opportunities and intelligent task proposals for the learners in a group. From the logic and formalization approaches, we have been working on the application of the answer sets programming framework for supporting collaboration in

4

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

agent-based CSCL environments [6] because its convenience for the representation and manipulation of the beliefs of the agent about the capabilities of the learners.

1. Using ASP for modelling CSCL environments Answer Set Programming (ASP) has been recognized as an important contribution in the areas of Logic Programming and Artificial Intelligence [5]. Nowadays, there is a significant amount of research work inspired on this formalism, well known and accepted for non-monotonic reasoning. ASP is more expressive than normal (disjunction free) logic programming and allow us to deal with uncertainty. Because beliefs can not be treated correctly in classical logic (PROLOG), ASP is useful as a logical framework for agent modelling. ASP allows us to have a direct and clear representation of the beliefs of an agent, specially in dynamic situations, incomplete information and uncertainty. ASP has two types of negation: weak negation (not x) and strong negation (~x). Using weak negation, not x in interpreted as "there is no evidence of x", while using strong negation, ~x means "there is evidence that x is false". DLV is a system for answer set programming [7]. The computational model of our agent is implemented in DLV, based on the ASP formalism. The result of a computation is a set of models, each one as a consistent explanation of the world, as far as the system can derive it. DLV tries to find all the models of the world which correctly and consistently explain the facts and rules of the program. A model assigns a truth value to each atom appearing in the program, and is represented as a set of atoms that are true in a certain model. If a program is inconsistent there will be no model. A model is considered an epistemic state of the agent, as a closed theory under the logic defined by a program [6]. Each model is interpreted as the implicit (derivable) and explicit beliefs of the agent. 1.1 Advantages of using DLV for agent modelling The advantages applying DLV as a logical framework for agent modelling in CSCL environments are: a) have a specification of the model as a set of rules and testing them with facts, in order to obtain correct and consistent models that ensure the coherence of the agent model. b) use double negation for derivation of beliefs under uncertainty and incomplete information. This is useful for representing the beliefs of the agent about the capabilities of its learner. c) use integrity constraints to represent conditions that can make the model inconsistent. d) have a formalization of the model.

2. Organization of the Domain Knowledge Our model is valid for structured knowledge domains, where we have knowledge elements (i.e. grammar rules for a second language collaborative learning environment) that can be defined and pedagogically or epistemologically organized. A knowledge element has an identification, which is represented by the predicate knowledgeElement(knowledgeElementId). For the pedagogical and epistemological organization of the knowledge elements, we adopt a genetic graph approach [8, 9]. The genetic graph is a very powerful structure for determining the learner's a knowledge frontier and learning opportunities. We have the following predicates and rules for its definition:

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

5

generalization(knowledgeElementId, generalKnowledgeElementId). specialization(KnowledgeElementId, SpecializedKnowledgeElementId):generalization(SpecializedKnowledgeElementId, KnowledgeElementId). refinement(knowledgeElementId, refinedKnowledgeElementId). simplification(KnowledgeElementId, SimpleKnowledgeElementId) :refinement(SimpleKnowledgeElementId, KnowledgeElementId). analogy(knowledgeElementId, analogKnowledgeElementId). analogy(KnowledgeElementId, analogKnowledgeElementId) :analogy(analogKnowledgeElementId, KnowledgeElementId).

2.1 Situations and Tasks A knowledge element is applicable in a given situation. For example, a grammar rule, as a knowledge element, is applied in a situation like a speech act, like “ask somebody to do something” or “apologize”. We have the following predicates to represent it: situation(situationId). applicable(knowledgeElementId, situationId).

We use integrity constraints in DVL in order to represent that if there is no information of a knowledge element applicable in a given situation, then the model is inconsistent: :- applicable(KnowledgeElementId,_), not knowledgeElement(KnowledgeElementId).

In the same way, if there is no information of a situation where a given knowledge element is applied, then the model is also inconsistent: :- applicable(_,SituationId), not situation(SituationId).

A task is a collaborative learning activity and implies one or more situations. This is represented by the predicates: task(taskId) implies(taskId, situationId).

In a similar way, if there is no information of a situation which corresponds to a task, then the model is inconsistent. Also, it will be inconsistent if there is no information of a task for a situation: :-implies(_, SituationId), not situation(SituationId). :-implies(TaskId, _), not task(TaskId).

3. Learner Model The agent identifies a learner with the predicate learner(learnerId). The set of beliefs concerning the learner capabilities corresponds to the beliefs of the agent about its learner's actual development level [10]. The belief that the learner is capable of apply a knowledge element of the domain is represented by capability(learnerId, knowledgeElementId). With the model we can derive new beliefs using derivation rules. In these rules the use of double negation allows the derivation with incomplete information. The agent believes that the learner is capable of applying a knowledge element if it believes that she is able to apply a knowledge element considered its generalization, and there is no evidence that the learner is not able to apply it:

6

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

capability(LearnerId, KnowledgeElementId) :capability(LearnerId, GeneralKnowledgeElementId), specialization(GeneralKnowledgeElementId, KnowledgeElementId), not ~capability(LearnerId, KnowledgeElementId).

Also, the agent will believe that the learner is capable of applying a knowledge element if she is able to apply a knowledge element considered its simplification, and there is no evidence that the learner is not able to apply it: capability(LearnerId, SimpleKnowledgeElementId) :capability(LearnerId, KnowledgeElementId), simplification(KnowledgeElementId,SimpleKnowledgeElementId), not ~capability(LearnerId, SimpleKnowledgeElementId).

When considered appropriate, the predicate ~capability(learnerId, knowledgeElementId) can be included when there is evidence that the learner is not able to apply the corresponding knowledge element. 4. Structural Knowledge Frontier The structural knowledge frontier is based on the concept of "knowledge frontier" of the genetic graph [8] and was used successfully in the GRACILE project [11] We call it structural knowledge frontier since it depends only in the genetic graph structure, and also in order to make the difference with the social knowledge frontier. A knowledge element is part of the learner’s structural knowledge frontier when there is no evidence that the learner is able to apply it, and the agent believes that she is able to apply another knowledge element, which is genetically related to the one in question. In the case of the generalization relation the rule is: structuralKnowledgeFrontier(LearnerId, KnowledgeElementId) :capability(LearnerId, RelatedKnowledgeElementId) , generalization(RelatedKnowledgeElementId, KnowledgeElementId), not capability(LearnerId, KnowledgeElementId).

The rest of the rules are similar, making reference to the refinement, analogy, simplification and specialization genetic relations. Also, a knowledge element will be part of the learner’s structural knowledge frontier if there is no evidence that the learner is able to apply it, and is considered an element of the basic knowledge set. We consider members of the basic knowledge set those knowledge elements which are not generalizations or refinements: structuralKnowledgeFrontier(LearnerId, KnowledgeElementId):basicKnowledge(KnowledgeElementId), not capability(LearnerId, KnowledgeElementId), knowledgeElement(KnowledgeElementId), learner(LearnerId).

5. Social Knowledge Frontier The concept of social knowledge frontier is based on the assumption that there are social issues that make the learners of a group to learn some knowledge elements first and more easily than others. Those issues may be the situations of their application, the relevance of it (be capable to apply the knowledge element in order to be a productive member in the group) or the nature and characteristics of it. For the agent, a knowledge element is part of the social

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

7

knowledge frontier of the learner if there is no evidence that the learner is able to apply it, but the agent believes that other learner in her group does. The rule is the following: socialKnowledgeFrontier(LearnerId, KnowledgeElementId):not capability(LearnerId, KnowledgeElementId), capability(OtherLearnerId, KnowledgeElementId), knowledgeElement(KnowledgeElementId), learner(LearnerId), learner(OtherLearnerId), LearnerId <> OtherLearnerId.

The social knowledge frontier implies the cooperation among agents, sharing their beliefs in their learner models. 6. Zone of Proximal Development In our model, the learning plan, and therefore the tasks proposals to the learner by her agent, are based on the representation of her zone of proximal development [10] or zpd. The actual development level of the learner is the set of beliefs concerning her capabilities in the domain. The potential development level corresponds to the structural knowledge frontier. A knowledge element is part of the zpd of the learner if is member of both, her structural knowledge frontier and her social knowledge frontier. The rule is: zpd(LearnerId, KnowledgeElementId) :structuralKnowledgeFrontier(LearnerId, KnowledgeElementId), socialKnowledgeFrontier(LearnerId, KnowledgeElementId).

7. Learning Plan and Task Proposal The learning plan of a learner is used in order to generate appropriate tasks proposals, and consists of the knowledge elements considered in her zpd: learningPlan(LearnerId, KnowledgeElementId):zpd(LearnerId, KnowledgeElementId).

However, in the case when the learner does not have a zpd, her learning plan will be her structural knowledge frontier (potential development level): learningPlan (LearnerId, KnowledgeElementId):not hasZpd(LearnerId), structuralKnowledgeFrontier(LearnerId, KnowledgeElementId).

The learning plan is individual. In order to keep a zpd for the members of the learning group, it is necessary to have an individual learning plan which also supports the maintenance of a zpd for the other learners in the group. This provides more learning possibilities to the group and therefore a maintenance of the motivation to participate. We defined the group supportive learning plan as the set of knowledge elements considered part of the learning plan of the learner and part of the structural knowledge frontier of another learner, who has no zpd. groupSupportiveLearningPlan(LearnerId, KnowledgeElementId):not hasZpd(OtherLearnerId), learningPlan (LearnerId, KnowledgeElementId), structuralKnowledgeFrontier(OtherLearnerId, KnowledgeElementId), LearnerId <> OtherLearnerId.

8

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

A learning task is defined as the cooperation of two learners, each one applying a knowledge element in a corresponding situation that occurs in the task. Once the agent has a representation of the learning plan of its learner, the next step is to propose her a task, based on that. First, the agent must determine if there is possible to assign a task based on the learner's group supportive learning plan. These tasks are called group supportive tasks. Working in a group supportive task, the learner cooperates with other learner trying to apply a knowledge element of her group supportive learning plan, in the corresponding situation, while the other learner cooperates by applying a knowledge element in his learning plan: groupSupportiveTask(LearnerId, KnowledgeElementId, SituationId, OtherLearnerId,OtherKnowledgeElementId, OtherSituationId, Task):groupSupportiveLearningProposal(LearnerId, KnowledgeElementId), learningPlan (OtherLearnerId, OtherKnowledgeElementId), KnowledgeElementId <> OtherKnowledgeElementId, LearnerId <> OtherLearnerId, applicable(KnowledgeElementId, SituationId), applicable(OtherKnowledgeElementId, OtherSituationId), implies(SituationId, Task), implies(OtherSituationId, Task).

In order to present an intelligent task proposal to the learner, the agent generates a set of task candidates. First, the agent will consider a group supportive task as a task candidate for its learner: taskCandidate(LearnerId, KnowledgeElementId, SituationId, OtherLearnerId, OtherKnowledgeElementId,OtherSituationId, Task):groupSupportiveTask(LearnerId, KnowledgeElementId, SituationId, OtherLearnerId, OtherKnowledgeElementId, OtherSituationId, Task).

However, if there is not a group supportive task for the learner, the task candidate will consist of the cooperation of the learner trying to apply a knowledge element of her learning plan, in the corresponding situation, while the other learner cooperates by applying a knowledge element in his learning plan. taskCandidate(LearnerId, KnowledgeElementId, SituationId, OtherLearnerId, OtherKnowledgeElementId, OtherSituationId, Task):not hasGroupSupportiveTask(LearnerId), learningPlan (LearnerId, KnowledgeElementId), learningPlan (OtherLearnerId, OtherKnowledgeElementId), KnowledgeElementId <> OtherKnowledgeElementId, LearnerId <> OtherLearnerId, applicable(KnowledgeElementId, SituationId), applicable(OtherKnowledgeElementId, OtherSituationId), implies(SituationId, Task), implies(OtherSituationId, Task).

The agent must propose its learner a set of learning tasks, and then the learner decides which one to commit to perform. The agent must present to the learner a list of tasks candidates. Among the set of task candidates, the agent determines some tasks as an intelligent proposal. A task proposal for a learner is a task candidate which implies that the knowledge element to be applied in the situation of the task is believed to be a capability of the other learner participating in the task.

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

9

TaskProposal(LearnerId, KnowledgeElementId, SituationId, OtherLearnerId, OtherKnowledgeElementId, OtherSituationId, Task):taskCandidate(LearnerId, KnowledgeElementId, SituationId, OtherLearnerId, OtherKnowledgeElementId, OtherSituationId, Task), capability(OtherLearnerId, KnowledgeElementId), KnowledgeElementId <> OtherKnowledgeElementId, LearnerId <> OtherLearnerId.

8. Results The computational model presented here is implemented in DLV. We performed several simulations, running the system in order to obtain a set of consistent and complete models for zones of proximal development and task proposals. At every time, we selected some changes in the capabilities of the virtual learners performing a task from those proposed by the agent. We ran the simulations with a domain of knowledge elements representing the grammar rules of Japanese Language, and we consider a small group of 4 learners. We ran all the simulations for the case of an homogeneous group as well as the case of an heterogeneous group. Homogeneous group: These simulations considered a group of four learners, all new in the domain (none was believed to be able to apply a knowledge element in the beginning). After running the model with its corresponding actualizations on the learners capabilities with respect to the tasks proposed, the average of zpd for a member of the group was of 1.31 knowledge elements. Due to the group supportive task, few times the zpd was an empty set, and there was always a recovery in the following task (see figure 1). Concerning the tasks proposed, the 68.57% of the task proposals implied the zpd of a learner. The 50% of them implied the zpd of both learners involved, while 33% only the zpd of one learner and 16.66% tasks did not imply the zpd of any of the learners. 4

3 learner 1 learner 2

2

learner 3 learner 4

1

0 1

2

3

4

5

6

7

8

9

10

11

12

figure 1. The agents maintain the zpd in a homogeneous group. (The x axis represents the number of session (time) and the y axis the number of knowledge elements in the learner’s ZPD).

Heterogeneous group: In this case we considered a heterogeneous group of four learners with the initial state as follows: one new learner (0 knowledge elements learned), one low level learner (2 elements learned), other middle level (4 elements learned) and a high level learner (6 elements learned). We ran the model with the corresponding actualizations on the learners capabilities with respect to the tasks proposed, The average of zpd for a member of the group was of 1.21 knowledge elements. For the new learner the average zpd was 2.125 knowledge elements; for the low level 1.25, for the middle level it was 0.625 and for the high level learner 0.875 (see figure 2). The 72.08% of the task proposals implied the

10

G. Ayala / An Analytic Model Based on ASP for Maintaining ZPDs in CSCL Environments

zpd of a learner. The 62.5% implied the zpd of both learners, while 25% only the zpd of one learner and there was a 12.5% of tasks which do not imply the zpd of the learners. 4

3 new low

2

middle high

1

0 1

2

3

4

5

6

7

8

figure 2. The agents maintain the zpd in a heterogeneous group.

9. Conclusions Using ASP is an interesting merge of AI&Ed approach to CSCL, applying the powerful concept of the genetic graph. The analytic model presented allows a formalization of notions of knowledge synergies in a learning group, which is a foundational issue for CSCL, from a computational point of view. With this model we have a more concrete and precise definition of zpd. DLV resulted in an appropriate system to elaborate a logically correct and consistent computational model of an agent for structured domain CSCL environments. Thanks to the concept of group supportive task the agents are able to maintain the zpd for the learners in the group. The analytical model, as it is presented in the paper, runs using the DLV system that can be easily downloaded [12]. References [1] Hoppe, U. H. & Ploetzner, R. (1998) Can Analytic models Support Learning in Groups. http://www.collide.info/Lehre/GestaltungInteraktiverLehrLernsysteme/downloads/esf98f.pdf [2] Self, J. (1993). Dormobile: a Vehicle for Metacognition, in Invited Talks, International Conference on Computers in Education ICCE'93, Taipei, Taiwan. [3] Self, J. (1992). Computational Viewpoints, Knowledge Negotiation, Moyse, R. & Elsom-Cook, M. (Eds.), Academic Press, 21-40 [4] Ayala, G. & Yano, Y. (1995). GRACILE: A Framework for Collaborative Intelligent Learning Environments, Journal of the Japanese Society of Artificial Intelligence, Vol.10. 6. 156-170 [5] Baral, C. (2003) Knowledge Representation, Reasoning and Declarative Problem Solving, Cambridge University Press. [6] Ortiz, M., Ayala, G. & Osorio, M. (2003) Formalizing the Learner Model for CSCL Environments, proceedings of the Fourth Mexican International Conference on Computer Science ENC 03, IEEE Computer Society and Mexican Society for Computer Science, 151-158. [7] Bihlmeyer, R., Faber,W., Ielpa, G. & Pfeifer, G. (2004) DLV- User Manual. http://www.dbai.tuwien.ac.at/proj/dlv/man/ [8] Goldstein, I. P. (1982) The Genetic Graph: a representation for the evolution of procedural knowledge, Intelligent Tutoring Systems, D. Sleeman and J. S. Brown (Eds.) Academic Press, 51-77 [9] Wasson. B. J. & Jones, M. (1988). Student Models: The Genetic Graph Approach. Inter. Journal of Man Machine Studies, vol. 28, 483-504. [10] Vygotsky, L. S. (1978) Mind in Society; the development of higher psychological processes, Harvard University Press, London. [11] Ayala, G. & Yano, Y. (1996) Learner Models for Supporting Awareness and Collaboration in a CSCL Environment. Intelligent Tutoring Systems, Lecture Notes in Computer Science 1086, Claude Frasson, Gilles Gauthier and Alan Lesgold (Eds.), Springer Verlag, 158-167. [12] Koch, C. (2004) The DLV Tutorial, http://chkoch.home.cern.ch/chkoch/dlv/dlv_tutorial.html

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

11

A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design using UML Nilufar BAGHAEI, Antonija MITROVIC and Warwick IRWIN Department of Computer Science and Software Engineering University of Canterbury, Private Bag 4800, New Zealand Phone number: ++64 (3) 3642987 Ext. 7756 {N.Baghaei, T.Mitrovic, W.Irwin}@cosc.canterbury.ac.nz Abstract. COLLECT-UML is an intelligent tutoring system that teaches Object-Oriented design using Unified Modelling Language (UML). UML is one of the most popular techniques used in the design and development of Object-Oriented systems nowadays. The Constraint-Based Modelling (CBM) has been used successfully in several systems and they have proved to be extremely effective in evaluations performed in real classrooms. In this paper, we present our experiences in implementing another constraint-based tutor, in the area of Object-Oriented design. We present the system’s architecture and functionality and describe the results of a preliminary study with postgraduate students who interacted with the system as part of a think-aloud study. Participants felt that using the system helped them improve their UML knowledge. A full evaluation study is planned for May 2005, which aims to evaluate the interface and the effect of using the system on students’ learning. Keywords: ITS, Constraint-Based Modelling, Object-Oriented design, UML

Introduction Previous work has shown that the Constraint-Based Modelling (CBM) [9] is extremely efficient, and it overcomes many problems that other student modeling approaches suffer from [6]. CBM has been successfully applied in a number of domains. These tutors, called constraint-based tutors [6] have been developed in domains such as SQL (the database query language) [5, 8], database modelling [11, 12], data normalization [7], and punctuation [4]. All three tutors in the database domain were developed as problem solving environments for tertiary students. Students solve problems presented to them with the assistance of feedback from the system. The punctuation tutor was developed with the goal of improving the capitalisation and punctuation skills of 10-11 year old school children. This paper presents our experiences in implementing a constraint-based tutor in the area of object-oriented design. UML modelling is one of the most popular techniques used in the design and development of object-oriented systems nowadays. UML was selected as an appropriate task for this research due mainly to its open-ended nature, and its complexity for novice designers. Although the traditional method of learning UML in a classroom environment may be sufficient as an introduction to the concepts of object-oriented design, students cannot gain expertise in the domain by attending lectures only. Therefore, the existence of a computerized tutor, which would support students in gaining such design skills, would be highly useful.

12

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

We start with a brief overview of related work in Section 1. Section 2 describes the overall architecture of the system, followed by the pilot study presented in Section 3. Conclusions and Future work are discussed in the last section. 1. Related Work Having found a lot of tutorials, textbooks and resources on UML, we are not aware of any attempt at developing an ITS for UML modelling. However, there has been an attempt [10] at developing a collaborative learning environment for object-oriented design problems using Object Modeling Technique (OMT). The system monitors group members’ communication patterns and problem solving actions in order to identify (using machine learning techniques) situations in which students effectively share new knowledge with their peers while solving object-oriented design problems. The system first logs data describing the students’ speech acts (e.g. Request Opinion, Suggest, and Apologise) and actions (e.g. Student 3 created a new class). It then collects examples of effective and ineffective knowledge sharing, and constructs two Hidden Markov Models which describe the students’ interaction in these two cases. A knowledge sharing example is considered effective if one or more students learn the newly shared knowledge (as shown by a difference in pre-post test performance), and ineffective otherwise. The system dynamically assesses a group’s interaction in the context of the constructed models, and determines when and why the students are having trouble learning the new concepts they share with each other. The system does not evaluate the OMT diagrams and an instructor or intelligent coach’s assistance is needed in mediating group knowledge sharing activities. In this regard, even though the system is effective as a collaboration tool, it would not be an effective teaching system for a group of novices with the same level of expertise, as it could be common for a group of students to agree on the same flawed argument. 2. COLLECT-UML: A Knowledge-Based UML Modelling Tutor COLLECT-UML is a web-based problem-solving environment, in which students are required to construct UML class diagrams that satisfy a given set of requirements. It assists students during problem solving and guides them towards a correct solution by providing feedback. The feedback is tailored towards each student depending on his/her knowledge. COLLECT-UML is designed as a complement to classroom teaching and when providing assistance, it assumes that the students are already familiar with the fundamentals of object-oriented software design.

2.1 Architecture The system has a distributed architecture [8], where the tutoring functionally is distributed between the client and the server (Figure 1). The application server consists of a student modeler, which creates and maintains student models for all users, a domain module and a pedagogical module. The user interface is Java-based (discussed in Section 3.3) and performs some teaching functions including immediate feedback for some problemsolving steps. The system is implemented in Allegro Common Lisp, which provides a development environment with an integrated Web Server (AllegroServe) [1].

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

13

Web browser Internet

Web Server (AllegroServe)

Logs

Student modeler

Student models

Session manager

Pedagogical module

Problems and Solutions

Domain model (Syntax and semantic constraints)

Figure 1: The architecture of the system

2.2 Domain constraints The system is able to diagnose students’ solutions by using its domain knowledge represented as a set of syntax and semantic constraints (88 semantic and 45 syntax constraints). Each constraint consists of a relevance condition, a satisfaction condition and a feedback message. The feedback messages are presented when the constraint is violated. It is common in any design problem to have two or more correct solutions for a problem, especially if the problem is complex. The system contains only one correct solution for each problem. However, the system is capable of recognizing alternative correct solutions, as there are constraints that check for equivalent constructs in the student’s and ideal solutions. Figure 2 gives examples of syntax and semantic constraints for the UML domain. 2.3 Interface The interface is an HTML page, containing a Java Applet, which was implemented using API specification for the Java 2 Platform, version 1.4.2 (Figure 4). The current version of the applet contains 4 packages, 75 Java classes and 6853 lines of code. In order to draw a UML diagram, the user selects the appropriate shape from the drawing toolbar and then positions the cursor on the desired place within the drawing area. Shapes can be resized by selecting them first, and then dragging the blue handles (shown as rectangles when the component is selected). The shape will remain red until the student selects a name for it from the problem text. A name can be selected either by double clicking on a word from the problem text, or by highlighting a phrase. The highlighted words are coloured depending on the type of the component. The feature is advantageous from a pedagogical point of view, as the student must follow the problem text closely. Many of the errors in students’ solutions occur because they have not comprehensively read and understood the problem. These mistakes would be minimised in COLLECT-UML,

14

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

; Semantic (117 "Make sure each subclass has at least one specific (local) attribute or method." (and (match SS SUBCLASSES (?* "@" ?subtag ?*)) (match IS SUBCLASSES (?* "@" ?subtag ?*)) (or-p (match IS ATTRIBUTES (?* "@" ?attr_tag ?attr_name ?subtag ?*)) (match IS METHODS (?* "@" ?method_tag ?method_name ?subtag ?*)))) (or-p (match SS ATTRIBUTES (?* "@" ?attr_tag1 ?attr_name1 ?subtag ?*)) (match SS METHODS (?* "@" ?method_tag1 ?method_name1 ?subtag ?*))) "specialisation/generalisation" (?subtag)) ; Syntax (100 "Check your inheritances. Two classes can not inherit from each other." (and (match SS SUBCLASSES (?* "@" ?subtag ?supertag ?*)) (not-p (test SS ("null" ?subtag))) (not-p (test SS ("null" ?supertag)))) (not-p (match SS SUBCLASSES (?* "@" ?subtag ?supertag ?* "@" ?supertag ?subtag ?*))) "specialisation/generalisation" (?subtag ?supertag))

Figure 2: Examples of syntax and semantic constraints

as students are required to focus their attention on the problem text every time they add a new component. Highlighting is also useful from the point of view of the student modeller for evaluating solutions [12]. There is no standard that is enforced in naming classes, methods, attributes or relationships. Since the names of the components in the student solution (SS) may not match the names of construct in the ideal solution (IS), the task of finding correspondence between the constructs of the SS and IS is difficult. This problem is avoided by forcing the student to highlight the word or phrase that is modelled by each component in the UML diagram. To create a new attribute or a method, the student needs to select the parent class first, and then either clicks on relevant toolbar icon or right-clicks on the class and chooses the “New ...” menu option. The properties of an existing component (class, attribute, method or relationship) can be changed by right-clicking on that component and choosing the relevant menu option. To connect two classes of the diagram, the student needs to select the appropriate type of relationship. A relationship will be shown in red if it is not properly attached to other classes. Clicking on “Student Model” button will display an overview of their knowledge. Currently, the system contains 14 problems, which cover different aspects of ObjectOriented modeling, and their ideal solutions. The ideal solutions are UML class diagrams that fulfil all the problem requirements. Figure 3 shows a sample problem and the internal representation of its ideal solution, which consists of 6 clauses (i.e. RELATIONSHIPS, ATTRIBUTES, METHODS, CLASSES, SUPERCLASSES and SUBCLASSES). The problem text is represented internally with embedded tags that specify the mapping to the components in the ideal solution. The tags are not visible to the student since they are extracted before the problem is displayed.

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

15

The applet saves the solutions submitted by students into XML files, which are converted to internal representation using an XSLT style-sheet. The constraints are applied to the internal representation of the solutions and feedback is given to students, using the messages attached to the violated constraints.

(5 ; problem number 5 ; difficulty "5. 5. An <E1> owner owns one or more <E2> vehicle s. Each <E2> vehicle has a <E2A1> gross weight . Each <E1> owner has a <E1A1> number (unique) and a <E1A2> name and <E1A3> register a number of <E1> vehicle s. Each <E1> vehicle can be either <E3> bike or <E4> car . For each <E3> bike , the software records the <E3A1> serial number and for each <E4> car , the <E4A1> license plate and the <E4A2> color are recorded." (("RELATIONSHIPS" "@ R1 association E2 E1 1..* 1 null null owns @ R99 inheritance E2 E4 null null null null null @ R99 inheritance E2 E3 null null null null null ") ("ATTRIBUTES" "@ E1A2 name E1 String private no @ E1A1 number E1 String private no @ E2A1 weight E2 float private no @ E3A1 serial_number E3 String private no @ E4A1 license_plate E4 String private no @ E4A2 color E4 Color private no ") ("METHODS" "@ E1A3 registers E1 void public no 1 vehicles List null null null null ") ("CLASSES" "@ E1 Owner concrete @ E2 Vehicle concrete @ E3 Bike concrete @ E4 Car concrete ") ("SUPERCLASSES" "@ E2 E3 @ E2 E4 ") ("SUBCLASSES" "@ E3 E2 @ E4 E2 ")) "5.jpg" "Vehicles")

Figure 3: A sample problem and its ideal solution

3. Pilot Study A pilot study was conducted as a think-aloud protocol in March 2005. The study aimed to discover users’ perceptions about various aspects of the system, mainly the quality of feedback messages and the interface. The participants were 12 postgraduate students enrolled in an Intelligent Tutoring Systems course at the University of Canterbury. Even though the target population for COLLECT-UML is undergraduates who are learning UML software design, it was not possible to gain access to this population at the time of the pilot study, because the UML class diagrams had not been covered in the lectures. The participants had completed 50% of the ITS course lectures, and were expected to have a good understanding of ITS. All participants except two were familiar with UML modelling. The study was carried out in the form of a think-aloud protocol [2]. This technique is increasingly being used for practical evaluations of computer systems. Although thinkaloud methods have traditionally been used mostly in psychological research, they are considered the single most valuable usability engineering method [3]. Each participant was asked to verbalise his/her thoughts while performing a UML modelling task using COLLECT-UML. Participants were able to skip the problems without completing them and to return to previous problems. Data was collected from video footages of think-aloud sessions, informal discussions after the session and researcher’s observations.

16

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

3.1 Students' impressions on interface The majority of the participants felt that the interface was nicely designed and the drawing area was big enough for them to work on the problems given. Three participants felt that some of the hints provided by the system were not helpful enough for them to correct their mistakes. For example, one participant had a class called Shape and an attribute (belonged to that class) called Origin. The attribute type had been specified as int, when the ideal solution expected them to have defined the attribute of type Point. The feedback in this

Figure 4: The current version of COLLECT-UML interface

case was: “Check your attributes. The types of some of your attributes have not been specified correctly”. Since the participant had defined several attributes, she was not sure which one the system was referring to. A modification was later made to the interface to highlight the errors in red. For creating new attributes and methods, participants tended to use the tool bar icons more than right click menu. A few participants felt that the help document provided by the system was too long and some of them were not keen to refer to it, even when they needed help with something. Two participants also expressed their desire to have access to glossary and a tutorial on how to use the system. These features will be added to the system in the future. In order to name a new component (class, attribute, method or relationship), the students were required to highlight (or double-click) the name from the problem text. Although some of the students found this somewhat restrictive, they became more comfortable with the interface once they had a chance to experiment with it. The students, who were more interested in typing in the names rather than highlighting the text, showed some interest after one of the researchers explained the reason behind restricting them.

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

17

The interface was modified to incorporate most of the suggestions mentioned above. The wording of one of the problems was changed after one participant commented that he found the problem text confusing. The initial version of the interface was restricting the users in a way that they needed to first click on the New attribute/method toolbar button and then highlight the attribute/method name from the problem text. Some participants suggested that it would be more convenient if the system would allow them to create new attribute/methods by first letting them highlight its name from the problem text. The interface was then modified to incorporate both functionalities; i.e. the users can first highlight (or double-click) the attribute/method name from the problem text and then click on the New Attribute/Method toolbar button or vice versa. It was observed during one of the sessions that one participant created a class, and chose the New Method menu option from the right-click pop-up menu. He was not sure what to do next, while the interface had disabled the toolbar and was expecting the user to highlight the name from the problem text. It was then decided to add some pop-up windows that would give information to novice users as to what they would need to do next. It displays the dialog window, each time the user wants to create a new class, relationship, attribute, or method (Figure 5). The system checks to see whether there are any attributes/methods specified already. If the user is about to create the first attribute/method for each class, the information window pops-up, otherwise, the system skips that level and prompts the user to highlight (or double-click) the name from the problem text, without showing the dialog information (as it is expected that the users would be familiar with what they would need to do then). 3.2 Students' impressions on feedback The majority of the participants felt that the feedback messages helped them to understand the domain concepts that they found difficult. For this study, we restricted the number of feedback messages up to 5 messages at a time. The constraints were implemented so that they would only check for necessary constructs that the students were supposed to have included in their UML diagrams (i.e. classes, attributes, methods and relationships). Therefore, the participants were allowed to define extra methods for example, if they thought there were needed. This was a feature several participants particularly liked about the system. 4. Conclusions and Future Work This paper presented COLLECT-UML, an ITS for UML modelling. An analysis carried out to investigate how participants interact with the single-user version of the system. Participants felt that using the system helped them improve their UML knowledge. Some of them experienced a number of difficulties interacting with the system. The video footage was useful in identifying the bugs in the system. All the bugs identified were fixed to make the system more robust. A full evaluation study is planned for May 2005. The study aims to evaluate the interface and the effect of using the system on students’ learning. It will involve secondyear University students enrolled in an undergraduate Software Engineering course. The data recorded in the student model will be analyzed to see how much students learn during their interaction with the system.

18

N. Baghaei et al. / A Constraint-Based Tutor for Learning Object-Oriented Analysis and Design

Figure 5: An information dialog popping up to guide the users as to what they would need to do next

The most important goal of future work is to extend the system to support collaborative learning addressing both collaborative issues and task-oriented issues. CBM has been used to effectively present knowledge in several tutors supporting individual learning. The comprehensive evaluation studies of the multi-user version of the system will provide a measure of the effectiveness of using CBM technique in intelligent computer supported collaborative learning environments.

References [1] [2]

AllegroServe - a Web Application Server, http://www.franz.com/ Ericsson, K. A. and Simon, H. A. (1984) Protocol Analysis: Verbal Reports as Data. The MIT Press, Cambridge, MA. [3] Nielsen, J. (1993) Usability Engineering. Academic Press Inc., San Diego, CA. [4] Mayo, M., and Mitrovic, A. (2001) Optimising ITS behaviour with Bayesian networks and decision theory, International Journal of Artificial Intelligence in Education, 12 (2). pp.124-153. [5] Mitrovic, A. (1998) Learning SQL with a Computerised Tutor. 29th ACM SIGCSE Technical Symposium, (Atlanta, 1998), pp.307-311. [6] Mitrovic, A., Mayo, M., Suraweera, P. and Martin, B. (2001) Constraint-based Tutors: a Success Story. 14th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems (IEA/AIE-2001), (Budapest, 2001), pp.931-940. [7] Mitrovic, A. (2002) NORMIT, a Web-enabled tutor for database normalization. ICCE 2002, (Auckland, New Zealand), pp.1276-1280. [8] Mitrovic, A. (2003) An intelligent SQl tutor on the Web. International Journal of Artificial Intelligence in Education, 13 (2-4). pp.173-197. [9] Ohlsson, S. (1994) Constraint-based Student Modelling. Student Modelling: the Key to Individualized Knowledge-based Instruction, (Berlin, 1994), Springer-Verlag, pp.167-189. [10] Soller, A. and Lesgold, A. (2000) Knowledge acquisition for adaptive collaborative learning

environments. AAAI Fall Symposium: Learning How to Do Things, (Cape Cod, MA). [11] [12]

Suraweera, P. and Mitrovic, A. (2004) An Intelligent Tutoring System for Entity Relationship Modelling. International Journal of Artificial Intelligent in Education, 14 (3-4). pp.375-417. Suraweera, P., and Mitrovic, A. (2002) KERMIT: a Constraint-based Tutor for Database Modeling. In: Cerri, S., Gouarderes, G. and Paraguacu, F. (eds.) 6th International Conference on Intelligent Tutoring Systems (ITS 2002), (Biarritz, France), pp.377-387.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

19

A Case Study Describing the Creation and Implementation of a Capacity Building Program for Mixed Mode Delivery of Academic Programmes in Developing Countries Cleo CHADWICK1, Sheila HOWELL2, Prof Bakari MWINYIWIWA3 1 Melbourne, Australia 2 RMIT University, Melbourne, Australia 3 University of Dar es Salaam, Dar es Salaam, Tanzania Abstract. In 2002 RMIT UNIVERSITY (RMIT) were awarded a World Bank (WB) four year contract (2003 – 2006) by the AFRICAN VIRTUAL UNIVERSITY (AVU) for x Delivery of a COMPUTER SCIENCE (CS) diploma and degree to 8 AVU Learning Centres (LC) in Africa using an INFORMATION COMMUNICATIONS TECHNOLOGY (ICT) mixed-mode delivery model, and x Building capacity in the LEAD PARTNER UNIVERSITY (LPU), in this case the UNIVERSITY OF DAR ES SALAAM (UDSM) in Tanzania, so that they could take over the management & operations of the program from RMIT as from the end of the contract - Dec 2006. The inclusion of effective CAPACITY BUILDING (CB) into the program was viewed as an essential and urgent requirement that would lead to an ongoing and sustainable program. This need is particularly important in the areas of ICT, where the technology has the potential to support education and contribute meaningfully to the development of economic and sustainable industries. To be effective, CB would need to be seen as a holistic process involving the whole organisation. This paper describes: x the complexities and key issues involved in developing a CB program with the primary objective being “To ensure that UDSM is “operationally ready” to take over the management and the delivery of CS Program from RMIT .by Dec 2006” x the ‘organisational mirroring’ planning approach created to develop the CB program and x the specific strategies and activities that formed the CB program. Finally the paper reflects on the strategies and activities adopted and discusses key issues arising during implementation and lessons learned to date. Keywords: Capacity Building, online education, sustainability, organisational mirroring, mixed mode delivery, developing countries

1. Introduction The tertiary education sector within Africa is characterised by x Extremely low access to higher education, x Over stretched government budgets for higher education; a dearth of skilled labour, and limited quality training facilities;

20

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

x x

Increasing isolation of Africa in emerging knowledge; and Poor and costly Information and Communication Technologies (ICT) infrastructure

The AVU was established by the World Bank (WB) in 1997, in order to harness modern ICTs to increase access to high quality, affordable tertiary learning opportunities, and to expand the pool of skilled labour for the new global economy. The AVU / RMIT - CS Program project, was the first accredited degree / diploma program that the AVU offered to its network. The CS program was to be offered to 8 Anglophone LC’s – in Kenya, Tanzania, Ethiopia, Rwanda, and Ghana[1]. It was also the first time in which the AVU wished to build capacity for ICT delivery of educational programs within an African educational institution – namely the UDSM. It was therefore critical that the CB activities that would be undertaken as part of this contract delivered a sustainable long term outcome, & ensured that by the end of the contract period – Dec 2006 - UDSM was successfully able to take over the operations of the CS program from RMIT. Capacity building with respect to developing countries is widely recognised as a vital & urgent need [2] [3] [4]. There has been a steadily increasing imperative to include major components of capacity building in aid projects, such that it is suggested that the World Bank puts the transfer of knowledge for development & the building of human & institutional capacity at par with the “transfer of financial resources & the building of physical capacity” [5] [6]. Successful CB is not restricted to the program at hand; rather it needs to be seen as a holistic combination of human resources, finances, management & social policy[7]. Traditionally, within the University context, capacity building has best been witnessed through infrastructure development & enhancing staff capabilities through Masters & PhD programs [8] [9] [10] [11]. Notwithstanding the many success stories that this approach has engendered, experimentation & acceptance of alternative models of capacity building is required [12].

1.1 Key Issues

x

x

x

The key issues facing the project team (AVU/ RMIT / UDSM) were In this project, - the development, delivery & management of online & mixed-mode programs, - a complex mix of academic, technical & management skills is required. The planned use of technology to enable a greater take up of tertiary education in Africa, led to the recognition that the successful transfer of skills in this project could have a lasting impact on educational opportunities across Africa. Potentially therefore Africa’s ICT infrastructure would prove a challenge. Reliable telecommunications infrastructure for satellite access was required in all of the countries in which the project was to operate in. The unreliability of such infrastructure was likely to influence the CS program & impact on long term sustainability, & so it was essential that the team develop innovative, long term sustainable ICT solutions At the commencement of the project there was a low level of awareness of the use of ICT in delivery of educational programs with both students & academics. This was likely to impact on the demand for such programs, as well as the acceptance of such a delivery mode by the academics. Due to a variety of factors, at the commencement of the project (Feb 2003) the development of ICT capacity (both infrastructure, & skill based) within UDSM was undergoing re-evaluation & expansion in a number of areas. Of key importance to the success of the capacity building process was the concept that UDSM would gain

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

x

x

21

ownership of the project & hence ensure its long term sustainability. There was a clear need to embed this ownership concept into the CB program, & ensure a high level of commitment.. Less than 10% of the total contract value was set aside for CB activities. Resource efficiency was therefore of critical importance to the long term success of the CS program & ongoing sustainability. This required innovative thinking & adoption of local area answers. The effect of the distance (RMIT in Australia & UDSM in Tanzania) on the successful implementation of a four-year CB program. Most of the team members had previously been involved in other CB programs, & from experience understood the effects that distance could have on effective communications, maintaining morale & motivation, & ensuring a common vision & team approach.

1.2 Political & organisational context Of significant importance to the long term CB outcomes was the political & organisational context of the various stakeholders. The key concerns were: x The initial Terms of Reference (TOR) provided for the RMIT tender were very vague about the CB outcomes & objectives, & open to interpretation. The only direction that the TOR provided was that CB would support the introduction of the LPU Model. x Once the contract was awarded there was enormous uncertainty regarding “the transitions or pipeline phase” which referred to the period of time post the 4 year contract time span (Dec 2006). The uncertainty concerned 2 key things (a) what support RMIT would continue to provide to UDSM & (b) who would accredit students who commenced the program during the 4 year contract period but completed post this. x Once the contract was awarded there was time pressure to commence the program & as a result there was limited opportunity for ALL the three parties (AVU / RMIT / UDSM) to consult & develop a joint understanding & vision of (a) what the CB outcomes would be (b) a joint understanding of how the LPU model would operate (c) how the transition / pipeline phase would operate & (d) the way forward. x During pre-implementation phase, the implementing agency, AVU, was still developing their own organisational structure & philosophy. Changes implemented at this time added further uncertainty to a common understanding & interpretation of the CB program proposed. To bring some certainty & clarity into the CB program, the team (RMIT/AVU/UDSM) decided to take an organisational mirroring approach. This was a unique approach created by the team which allowed CB to be viewed as two organisations mirroring each other within an integrated whole of organisation perspective. The key advantages included x It legitimised the concept of RMIT & UDSM mirroring each other systems, processes, & skill base for the purpose of capacity building, & allowed agreement for the RMIT systems & process to be the ‘starting point’ for all planning. It was acknowledged that the UDSM resulting processes might be very different from the RMIT approach, but that the RMIT processes represented the most practical “starting point” from which customisation could take place. This saved enormous time in the planning phase. x It provides an organisational planning framework for the development of CB outcomes, objectives & strategies. An organisational matrix was developed in which key functional area’s (ie Management, Human Resource, & Finances) were overlayed onto various organisational levels (ie Institutional, Operational, & Individual.). This

22

x

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

matrix was then used to assess UDSM’s organisational needs, & to design appropriate CB responses (training, study tours, development of procedure & manuals etc). Within a discrete period of time (6 months) it provided an opportunity for all the parties to discuss, agree, & to document a Capacity Building Plan (CBP). The CBP provided the road map for implementation of ALL the CB over the next four years, & could be used to monitor progress & assess the impact of any delays.

2. Description of the Capacity Building program The CB objective that was agreed & documented was - ‘To ensure that UDSM is operationally ready to take over the management & the delivery of the AVU Computer Science Program .by Dec 2006’. This section of the paper articulates the CB program, & describes the key issues, outcomes, objectives & strategies of each of the three organisational levels which were assessed in the planning phase – Institutional, Operational & Individual.

2.1 Institutional Level On the institutional level the objectives & strategies that were devised were aimed at ensuring awareness, acceptance, commitment &, finally, ownership by the highest levels of staff within UDSM & AVU. The institutional CB objectives were: x To develop commitment, ownership, & responsibility for CB process at the highest level within UDSM x To establish a clear governance & accountability structure for the implementation of the project x Promote a shared vision, & common understanding of the project between ALL stakeholders The organisational functional requirements & CB strategies are shown in the table below. Table 1 - Institutional Level Analysis Key Functional Areas Management – Ensure the integration & institutionalisation of the program Ensure high profile & senior management support HR Ensure organisation has skilled staff for program delivery Ensure trained staff are recognised & rewarded so that the organisation maintain their services Ensure ongoing internal skill base maintained Ensure that UDSM is not isolated

CB Strategies ¾ Establish Tripartite Legal Agreement – outlining the agreed CB objectives, resources & timelines ¾ Establish comprehensive CB governance structure – including a joint working party (UDSM / RMIT / AVU) to monitor the progress of the CB – see below for more information ¾ Documentation of Capacity Building activities into a Capacity Building Plan (CBP) – see below for more information ¾ Exposure of key senior UDSM & AVU personnel to RMIT operations ¾ Discussions on remunerations & conditions for key UDSM staff

Finances Ensure the financial aspects of the program are clear, understood by all parties, & documented Ensure that the program is profitable in the long term

Implementation is continuing, but to date the majority of the CB strategies for the institutional level have proved to be very effective. Of key importance has been:-

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

x

x

23

The Capacity Building Working Party (CBWP). This working party, which was established in June 2003, is chaired by the Vice Chancellor UDSM, & includes senior representatives from UDSM, AVU & RMIT. The main emphases of the CBWP are to (i) develop a high level profile within UDSM for the CB; (ii) ensure key decision makers within UDSM are involved & regularly updated; (iii) ensure a common vision & understanding; & (iv) establish a mechanism of reporting back on progress to date, any delays, & future activities. Overall the CBWP is an excellent mechanism of building high level commitment, & profile for the project. Documentation of Capacity Building Plan (CBP) - The CBP was a comprehensive document which outlined all details of the CB program. It included the CB framework & identified the key objectives, the activities planned over the period of the project, the associated schedule, & an agreed outcomes. This document provided the level of detail required to move the project through a very complex set of activities The document was developed by both RMIT & UDSM representatives, & signed off by the projects three most senior counterparts – The AVU Rector, the UDSM VC, & the RMIT Pro Vice Chancellor.

2.2 Operational Level The operational level of CB is strongly influenced by the field of endeavour; hence, the strategies developed reflect the realities of the program concerned, in this case the development, delivery, management & accreditation of online, mixed-mode CS program The operational objectives are x To identify, design, & implement all operational requirements for UDSM to be operationally ready by Dec 2006 x To effectively transfer from RMIT to UDSM - ALL necessary organisational systems, processes, & structures pertaining to the delivery of the CS program. This required an assessment of technical & management skills & expert knowledge of CS curriculum & student management issues. To manage this complexity & the size of the project, we divided the project into three different conceptual & practical groups (i) educational materials development, (ii) delivery & technology & (iii) governance. The organisational functional requirements & CB strategies are shown in the table below Table 2 - Operational Level Analysis Key Functional Issues Management – Build awareness of the new skills, systems & facilities needed to operate a mixed-mode program in 2007. HR – key UDSM counterparts identified & recognised by UDSM Finances – UDSM to be able to identify the resources required to take over delivery in 2007. IT – to ensure that UDSM had policies, processes, hardware & software in place that would enable the successful development & delivery of courseware using a mixed mode approach

Strategies x Undertake an operational analysis of UDSM including – review skills, systems, facilities – refer below for further information x Develop operational procedures manual for the management & delivery of a CS program x Review organisational structure – refer below for further information x Identification of equipment, facilities x Establish a plan for the gradual transfer of responsibilities from RMIT to UDSM

Implementation is ongoing, -key strategies which proved very effective have been

24

x

x

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

Operational analysis – review skills, systems, facilities – A detailed operational analysis was undertaken in which RMIT documented all the systems, processes, structures, skills, facilities, & equipment, that RMIT used. This document was then sent to all the UDSM counterparts, to identify the gaps between what was currently at UDSM & what would need to be in place by Dec 2006 for UDSM to be operationally ready for delivery. This proved a very effective means to identify what needed to be done, & formed the “WHAT” component of the CB Plan. Organisational structure – RMIT recorded the organisational structure which it had put into place for the AVU project , & for key positions identified roles, responsibilities, & accountability. This structure was forwarded to UDSM for their consideration. At the March 2005 CBWP meeting, UDSM presented their Faculty of Informatics & Virtual Education (FIVE) structure which identified a clear & unambiguous role for the delivery of online & distance education both for programs from external providers such as RMIT University & for UDSM –owned programs.

2.3 Individual Level The 3rd tier of capacity building effort is directed at individuals. To achieve the desired objectives of operating a large scale academic online program, a team of individuals each with specific skills & an overarching awareness of the complete project needs to be brought together & supported as they each bring their own skills to the jigsaw that represents the complete project. Strategies at this level include skills development, online mentoring, linking of counterparts etc. We would agree with Adderly [13] that in capacity building there is a need for training of individuals by means of both long term formal award programs such as masters programs & also short course participation. The individual level objectives were x To implement the most effective strategies for transfer of knowledge & skills from RMIT counterparts to UDSM counterparts x To ensure long term sustainability by supporting UDSM counterparts in developing new skills, acquiring new knowledge, & the building of confidence & ability. x To equip sufficient number of counterparts within UDSM with the necessary skills, confidence & motivation, so that UDSM would be ready to take over all responsibility in Dec 2006. The organisational functional requirements & CB strategies are shown in the table below Table 3 - Individual Level Analysis Key Functional Areas Management – release of staff as required for training programs. Development of systems & processes in readiness for delivery in 2007. HR – Training of individual UDSM staff through both long-term programs (eg Masters program) & short-term workshops. Recognition of staff workloads in the development, delivery & management of programs. Mentoring of UDSM counterparts to ensure the transfer of skills across the spectrum of tasks required to deliver an on-line mixed-mode program. Finances – Finance to support staff on training programs. scholarships for Masters students.

UDSM

IT – use of IT for online mentoring (chat, email, Website), - use of monitoring software to follow the mentoring process.

Strategies x In house training workshops x Masters scheme – please refer below x Teaming UDSM counterparts with RMIT counterparts – please refer below x Supporting UDSM counterparts through on line mentoring – throughout the handover process – please refer below

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

25

Implementation of these strategies is continuing, but what we are finding is that the transfer of technical skills can in part be delivered through short course training, but support in the day to day application of those skills is also essential for effective & long term use. Short courses can provide an excellent starting point, but the creation of expert users depends upon day by day support in the operation of those skills. Key strategies have been x Masters Scheme. 2 staff from UDSM completed a Masters in online education & have become qualified instructional designers. x Teaming UDSM counterparts with RMIT counterparts - The UDSM / RMIT counterpart relationship was fundamental to the success of the CB program. The key objective of the teaming of UDSM & RMIT counterparts was to develop trusting & supportive working relationships which allowed a seamless transfer of knowledge, skills, & management expertise. x Supporting UDSM counterparts through on line mentoring. - At the time of writing, all the necessary counterparts for semester 1. 2005 have been identified & the on-line mentoring will commence after the first workshop. Mentoring will cover the following activities at this stage:x Review of existing courses & preparation of one of the courses for semester 2, 2005. x Commencing the task of contextualising curriculum for delivery in 2007. x Support for the instructional design & development of new CS courses x Support for the delivery of an instructional design workshop for internal delivery by UDSM staff. In the online mentoring process, each 6 monthly Capacity Building Stakeholder Workshop operates as a “milestone” & each workshop will develop the timelines & activities needed to be completed before the next workshop. As items on the timeline become required, the RMIT mentor initiates discussion about the processes involved & provides support through email & online chat session as the UDSM staff member completes the task. Regular reporting takes place between RMIT & UDSM coordinators. Online software will enable all parties to review the progress of the mentoring process at any time, & a collaborative web site enables access to all CB materials.

4. Conclusions Capacity Building is an integral component of the AVU/RMIT CS Program. A comprehensive plan has been developed & detailed activities & timelines agreed between RMIT & UDSM. Technology will play a major role in the implementation of the CBP. The work in using IT tools as a part of the CB process is embryonic at this stage, & will be evaluated & assessed as the planned implementation takes place. To date, much has been learnt & a spiral method of implementation is gradually unfolding – tasks are undertaken in a small segment of the project, - we evaluate the results & the success of the chosen implementation method. From this, refinements are made before the next iteration of the plan which will involve a larger segment of the project involved in the same tasks. In this way, by 2007, many tasks will have been undertaken by UDSM on a number of occasions, &, importantly, all tasks will have been undertaken at least once. The successes of our approach to date can be summarised as follows:Capacity Building principles

26

C. Chadwick et al. / A Case Study Describing the Creation and Implementation of a CB Program

x

Building effective relationships between people & organizations is fundamental to the long term success of the CB program. x Documenting the plan helped to establish a common vision between all stakeholder & assisted in managing differing expectations x Supporting counterparts in them using new skills in their workplace, is essential in building knowledge & confidence in individuals Capacity Building strategies x The CBWP created an environment in which UDSM was able to operationalise its Faculty of Informatics & Virtual Education (FIVE) structures for online & distance education. x The skills-gap analysis undertaken enabled a comprehensive & effective CB plan to evolve. x Adoption of the organisational mirroring approach provided a good starting point for discussion within UDSM about their preferred approach to on-line & mixed-mode delivery. x The use of ICT for mentoring purposes is seen as natural & appropriate x Agreement of the HR issues surrounding the Masters students has enabled UDSM to ensure continuity of employment for these graduates. Items for re-evaluation in the next phase of implementation include:x Careful & thorough briefing of UDSM senior staff at the CBWP is essential for ongoing support. x Involvement of the AVU in all CB processes is required. x Continuous use of the on-line mentoring processes are required to provide support for the UDSM staff who will inevitably be tackling the project with inadequate resources & time.

References [1] Howell S. et al “Teaching Mixed-Mode: A case study in Remote Delivery of Computer Science” Educational Media International, pp291 – 300, 2004. [2] CIDA 1987, “Sharing Our Future” p.23. [3] Bolger J. “Capacity Development: Why, What & How”, Capacity Development Occasional Series, CIDA Policy Branch No.1, 2000 [4] Lavergne R. & Saxby J. “Capacity Development: Vision & Implications” Capacity Development Occasional Series, CIDA Policy Branch No.3, 2001 [5] Local Government Unit (LGU) Philippines LGU Assistance Portal, 2004 sighted 10th April 2005. http://www.lguportal.org/lgupor/Products/capacity.htm [6] World Bank 2002 Constructing Knowledge Societies – New Challenges for Tertiary Education, Washington DC. [7] Smith M.K. 2005 “”Community Development – an introduction to the idea” sighted, 8th April, 2005 from http://www.infed.org/community/b-comdv.htm [8] GTZ: Infrastructure & Human Resource Development Projects at the faculty of Engineering, University of Dar es Salaam. [9] NORAD: Infrastructure & Human Resource Development Projects at the University of Dar es Salaam. [10] NUFFIC: ICT Infrastructure & Human Resource Development at the University of Dar es Salaam. [11] Carnegie: Infrastructure & Human Resource Development at the University of Dar es Salaam. [12] Poverty Reduction Learning network for Eastern & Southern Africa (PRLN) 2002. http://www.africanprogress.net/capacity_building_brief.htm. Sighted 14 April 2005. [13] Adderley J. “Training – The Key to successful Systems in Developing Countries” 11th SIGCSE Technical Symposium, 1989

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

27

Fostering Learning Communities among Teachers and Students: Potentials and Issues a, b

CHAI Ching Singa, TAN Seng Cheeb National Institute of Education, Nanyang Technological University, Singapore [email protected] Abstract. This paper explores some potentials and issues of fostering learning communities through Computer-Supported Collaborative Learning (CSCL) technology in Singapore classrooms. The subjects were eight teachers who attended a professional development course conducted at the National Institute of Education. The course focused on preparing teachers in integrating information technology into teaching and learning. CSCL, in particular, Knowledge Forum®, was adopted as a platform for online discussion and knowledge building on these technology integration issues. Interviews with these teachers and analysis of their online discussion suggest that they appreciate the knowledge building approach as a means for the students and themselves to learn beyond classroom and in promoting in-depth learning. However, the teachers felt that the approach was time consuming and was more suitable for high-ability students. Keywords: Learning Community, Knowledge Building, Professional Development

1. Rationale for Adopting Knowledge Building Community This paper describes a study on the perceptions of two groups of Singapore teachers, all of whom experienced teaching and learning in a learning community mediated through Knowledge Forum (KF). Since teachers are the key agent for changes in effective classroom practices, the perceptions of these teachers towards the potentials and barriers of building learning communities are important to the development of more effective teacher training. Many of us are familiar with the role of a teacher as the sage-on-the-stage, transmitting knowledge that the teacher has conscientiously analysed and chunked for the students’ consumption. While one important task of the teacher is to provide information, this approach becomes problematic when it is the sole strategy used. Many educators have criticized this didactic approach for its epistemological assumption that knowledge can be transmitted as well as the assumption that learners are motivated to receive knowledge and they can re-create the same understanding intended by the teachers [1]. Dissatisfied with the didactic approach, educators have begun to look for alternatives. Among many recent education reform movements, constructivist learning, situated cognition, and communities of learners are rapidly gaining acceptance [2, 3]. These approaches have some commonalities: active engagement of learners, intentional learning, use of authentic problems and contexts, social collaboration, student-centred learning, and engagement of higher-order thinking. These reforms call for transformation of classrooms into learning environments for students to construct knowledge in authentic and collaborative settings. They also call for a change in teachers’ beliefs and practices about teaching and learning. In terms of beliefs, the change is from a realist/objectivist to that of relativist/constructivist view. Prawat and Windschitl had articulated the breadth and depth of the change involved in such conversion [4,5]. Teachers need to be dissatisfied with

28

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

current approach, understand the alternatives, assess them as plausible and useful, and integrate the new practices into their existing understanding and practices. In terms of practice, teachers need to be able to elicit students’ ideas, diagnose students’ understandings and scaffold students in constructing personally meaningful knowledge. Embracing this form of student-centred teaching requires teachers to be “practical intellectuals, curriculum developers, and generators of knowledge in practice” [6]. To facilitate teachers’ developments for the changes, it is necessary for teachers to test their ideas in their respective classrooms and observe the effects. In this study, the Knowledge Building Community (KBC) was adopted to engage teachers in solving authentic problems that they faced in integrating information technology into teaching [8]. The KBC is modelled after real-life knowledge workers like scientists who engage themselves in collaborative advancement of the understanding of a domain. Typical tasks of knowledge workers include identification of problems, formulating hypotheses, gathering relevant information, designing solutions, conducting empirical studies, and reflecting on the processes. These knowledge works are usually conducted in a community where members build on each other’s ideas. To engage learners in a KBC is equivalent to asking the learners to assume the role of knowledge creators in a particular field of practice or discipline. It allows the learners to deal directly with the problems that the practitioners are concerned with, thereby cultivating the necessary competencies and dispositions needed for knowledge work [9]. Case studies, conducted mostly in the Western context, have demonstrated that it is possible to engage schoolchildren in knowledge works [10]. Research has also shown that engaging students in KB promote depth of learning, inquiry and reflection [11]. The KBC model seems to be an appropriate model in facilitating the development of in-service teachers in constructivist-oriented teaching. Recent studies in teacher development provide further support that it may be beneficial to engage teachers as a community of practice supported by networked environments [12]. A community of practice affords valuable opportunities for the participants to appropriate ways of conceptualizing and solving problems [13]. Some teacher educators have recognized the potentials of engaging teachers in a KBC. However, the studies conducted to date have been confined to pre-service teachers [10,14]. In one study, Yuen interviewed teachers in Hong Kong who have implemented KF [15]. The teachers reported conceptions of learning and the knowledge construction processes seemed to be incongruent with the theories of the KBC. The study suggested further research in teacher professional development for the implementation of KBC as necessary.

2. Method In this study, the KBC model was adapted for two teacher professional development courses designed to develop appropriate epistemology and knowledge in in-service teachers for effective integration of a range of Information Technology (IT) for teaching and learning. Treating integrating IT into curriculum as an ill-defined problem, it was hoped that engaging teachers in a KBC would provide opportunities for teachers to coconstruct understanding about IT integration, observe different ways of integration, test out their ideas, reflect and discuss the emerging issues and evolving ideas with peers and experts [16]. The main research question guiding this study is “How do in-service teachers perceive teaching and learning in a KBC?” The findings will allow the researchers in assessing and refining the teacher KBC for future professional development. The participants for this study came from two groups of teachers who had attended

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

29

in-service program that led to the award of Advanced Diploma for IT. There were 33 teachers in total. All teachers experienced learning in a KBC for the first 8 weeks, with the use of Knowledge Forum (figure 1).

Figure 1: Screen Capture of Teachers’ Interaction in the Database. In the subsequent module, they were tasked to design lessons that employed computers as Mindtools. Mindtools are “computer applications that require students to think in a meaningful ways in order to use the application to represent what they know” [1]. The teachers were free to select the Mindtools they deemed appropriate for implementation. Some of them chose KF, which was classified as conversational Mindtool, while others selected spreadsheets and semantic networks. They shared their lesson plans, implementation records, and reflections in the database. They also performed peer critiques with the purpose of improving their lessons and discussed theoretical issues. This study adopted the purposeful sampling strategy for selecting research participants. The selection criterion was that the participants had experienced both teaching and learning in the KBC. Eight teachers who had implemented KBC in their classrooms were identified. The participants’ backgrounds were rather diverse. Their years of teaching ranged from 2 years to 22 years, and levels of teaching from Primary One (Grade One) to Secondary 4 (Grade 10). They granted the researchers permission for the interviews and the use of their online posts. The online posts were included as a source of data for the researcher to triangulate the teachers’ views. The participants were interviewed after they had completed the module. Each interview lasted for about 30 to 45 minutes. Semi-structured interviews were employed to allow probing or elaboration of pertinent issues. The interview questions asked were: 1. How do you feel about your learning experience in this module? 2. Do you think it is possible to create Knowledge-Building community in our local classrooms? Why or why not? 3. What are the barriers that you had experienced or you can foresee in implementing the CSCL technology? The study is qualitative in nature. The data from the interviews, participants’ reflections, and online discussions were analysed for recurring patterns and themes that

30

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

might indicate the teachers’ perceptions of the potentials and issues involved in the implementation of KBC. Such method is commonly recommended in qualitative research [17]. Themes identified by at least four teachers are reported in the following section.

3. Research Findings Analysis of the data revealed that the teachers appreciated the technical and the social affordances of the KBC and they perceived that they had learnt better compared to their usual learning experiences. However, they were concerned about not having enough time to facilitate students’ learning and about students’ ability to cope with the KBC model. These findings are elaborated below.

3.1 Knowledge Building Community fosters social constructivist learning through dialogic interactions and allows participants’ voices to be heard Teachers generally perceived that learning in a KBC allowed them to voice their ideas and improve their understandings through social interactions. For example, when asked about her learning experience in the module, Ann said, “I’ve just graduated so basically it’s just like different. There is a discussion here, for the rest there is no discussion there. Maybe it’s like here I teach, you listen, then you just fall out and it’s more exam-based. For this, there’s a panel to voice out your ideas.” Sue, Clare and GL believed that KBC helps them to learn better as the members challenged each other’s ideas. Sue said, “Especially when people question me about why is it that I make the decision against another decision… the line of questioning makes you think from another angle.” Clare reflected that “one strong point is that different people have different perspectives. You have a range of different ideas, which is pretty good. Because all along it’s like, okay you think in this way. You didn’t realized that the other person thinks in different ways so it’s like you open up your mind a bit more. That way you learn a bit more.” GL differentiated KBC with other online discussion experience: “I’ve used forum before but not like this one. This one is more active. In a way, you submit the assignment, they give you comments and then you give the comments back and so on. So it’s more like an educational forum.”

3.2 Knowledge Building Community leads to idea improvement The participants felt that the collaborative learning environment and inter-subjectivity afforded by KBC were important conditions for idea improvement. GL provided an account of how she experienced idea improvement: “Like for example when I post my lesson plan, they will try to like come to modify it, give me suggestions and so on. And then, by looking at other posts, I find that they are also doing the same thing for other people. So we’re actually helping one another, and the questions that they ask make you think maybe you should change your way of doing things a bit.” Other participants had also expressed their appreciation for the opportunities to be exposed to multiple perspectives which helped to improve their ideas. For example, Clare stated, “We can build upon our ideas, reflect upon them after reading peers' input, opinions and comments. By being aware of how others look at things from a different perspective, I feel that I can actually improve upon my own learning or ideas. Sue echoed similar opinion, “It is a great place for the advancement of ideas and practice as in a knowledge

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

31

building community, different people have differing opinions and perspectives of a certain problem or situation. Hence, with differing opinions, people are able to compare and make connections of what they know against new knowledge.” Carol related her experience, “When we were asked to peer critique, I was initially quite apprehensive. I was worried that I may offend my peers if I ‘over-critiqued’ … However, as we got used to the idea, I thought that the critiques were actually beneficial, especially when I was given concrete feedback on my draft. These feedbacks helped me to become more aware of my writing and highlighted points that I may have overlooked. We naturally began to show genuine interest in each other’s draft and were becoming more spontaneous in our response.” Carol’s reflection highlighted that teachers’ need to develop trust before they could provide substantial feedbacks for their peers. Thus, it is important to look into the social dimensions of teaching and learning in this environment.

3.3 Knowledge Building Community is feasible in school classrooms and is beneficial to the school students. The teachers perceived that the KBC model helped to promote research and collaboration among students. They reported that their students seemed to be more active, motivated, intentional and independent in their learning. “Pupils actually explore, research and they gather more information. It actually develops the knowledge because through selfdiscovery and some sort of inquiry learning,” Bill opined. Clare explained why her children became more active, “because in this KBC…they go about learning on their own… when they are asking questions, when they are trying to put in input, they have to read. If they don’t read, they don’t know what to put in…The more they do thing voluntarily on their own, it seems that they learn better. They learn faster.” Fiona felt that online discussion benefits her students because “they have more time for research…they get more info… Like it is also a knowledge base where you can actually give each other views.” Dorothy, in her reflection note, highlighted the progressive advancement of knowledge afforded by Knowledge Forum, “KF is great for collaborative learning and inquiry. As compared to traditional classroom discussions, knowledge is built progressively and captured in a database. As a result, participants can refer to and improve upon a rich network of information at any time.” The teachers’ opinions indicate their appreciation of the values of KBC supported by KF, both for the teachers and the school students. All of them recognized the potentials of the KBC approach in realising some of the important goals related to the local reform efforts [24] such as equipping students with learning, thinking and communication skills. The communal nature of learning in a KBC and the permanent records of interactions had stimulated some of the teachers to reflect and improve their practice. However, there were some perceived barriers that could prevent the teachers from fostering KBC.

3.4 The teachers believed that the KBC approach is more appropriate for high-ability students who have good command of language. The teachers mentioned that high ability students with good command of language are better suited for the KBC approach to learning. Most teachers implemented the KBC in the form of project-based learning. A substantial proportion of the students’ posts were reports of factual information as well as peripheral discussions like coordination of project work. In other words, since project-based learning usually had some pre-specified end products, the students’ attention was directed to task completion. Dorothy and her students

32

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

created the only database that did not have any product pre-specified. The focus was solely on creating explanations on “How cuts heal”. This resulted in a qualitatively different online discourse that was inquiry driven. When Bill, Carol and Fiona were asked to view Dorothy’s database, they were seemingly impressed but they attributed the success of Dorothy’s students to their ability. They also seemed to express a sense of disbelief that their students could conduct successful inquiry in a KBC. Bill said, “But East Top (fictitious name for the school), possible. You see, this is East Top standard. You get my school students, they can’t write like that.” Fiona opined that “if you are a good pupil, you want to learn more. But for the weaker students, they don’t even know what kind of question to ask.” Carol lamented that she didn’t have students will that high ability. When we asked the teachers about the possibility of implementing KBC in the form of student-initiated inquiry approach, their responses were similar. They believed that the approach is suitable for students with high motivation, as well as strong academic and language abilities. Clare questioned, “if they are on the average and the lower average, they are not motivated, so how to be student initiated?” Ann felt uncomfortable with her students’ language ability, “Their language usage, then may be their spelling everything…most likely.” GL felt that is it not suitable for lower primary students because “their language is not very good… if you ask them to type the question, they can’t really phrase the questions properly. They may not even know the correct terms to use.” In other words, the teachers seemed to believe that the students’ abilities to think and verbalize their thoughts are important pre-requisites for the KBC approach to work. Such perceptions may act as a barrier to the scalability of KBC. Another potential barrier perceived by the teachers is that of the time constraint imposed by other teaching tasks.

3.5 The teachers perceived time limitation as a major barrier to the implementation of KBC. Both for their own learning and facilitating students learning in a KBC approach, time constraint was perceived to be a major barrier by the teachers. In term of time constraints in implementing KBC in classroom learning, Dorothy was most explicit. She said, “I am alright with it (the KBC approach) but where is the time to do all this?” Bill echoed, “I would say that given the time, I would like to have more of this kind of activities. But I think in a normal classroom situation, it is quite difficult considering the syllabus that you have to complete.” The teachers also faced the problem of time constraints for their personal learning, as illustrated by Sue’s comment “I don’t have time to sit down and think through it really. The…the process of reflection, evaluation. It’s a very difficult process for me.” In summary, the teachers seemed to recognize that the KBC model enhanced teaching and learning by actively engaging students in the co-construction of knowledge. However, they were not confident about its applicability in a tightly scheduled curriculum, especially if the implementation were to include students with lower academic inclination. These concerns will be addressed in the discussion.

4. Discussion In this study, the teachers believed that participation in KBC could lead to construction of deeper understanding and idea refinement. This is an important goal of KBC [8, 11]. Similar benefits were reported by in a number of studies [10, 14]. Similarly, potential

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

33

barriers of time constraints and student ability were also reported [18]. At this stage, it is important to find ways to help teachers negotiate the barriers. Through the interview, we realised that teachers’ reflection in the interview has “nudging” effect on their own beliefs, which could possibly lead to a change in attitudes later. Ideas that flowed naturally out of the interviews had resulted in some possible ways to negotiate the barriers. One possible solution to the time constraint imposed by the tightscheduled curriculum is to shift KBC implementation to the school vacation. The teachers felt that school vacations might be a better time to implement KBC. Currently, it is a common practice for teachers to give holiday assignments, usually in the form of drilland-practice worksheets, to the students. Consequently, the teachers have to grade many assignments when school starts again. Students’ motivation in doing repetitive assignments is also low. The teachers believed that it is more efficient to replace assignments with meaningful inquiry that could help their students to apply and extend knowledge they acquired during the school terms. Teachers could monitor students’ progress and facilitate students’ learning during the vacations, thus eliminating the need to grade students’ holiday assignments. They could also evaluate students’ learning through their presentations when school term commences again. This seems more meaningful for both the teachers and the students. Another potential hurdle was teachers’ perception that KBC is primarily appropriate for high ability students. There seemed to be no apparent solution for this barrier. However, the interview process itself again served to “nudge” the teachers’ beliefs. The teachers seemed to be engaged in self-examination after they had voiced their views on this. Two teachers questioned the soundness of their view and proceeded to conduct further inquiry. They were interviewed for a second time. Fiona reflected on how she encouraged her students, “My point to the children is there is no such thing as stupid question…I told them I am not going to give answer. It is basically going to be interaction among all of you. I may go in to question you answer but I will not give you an answer.” Carol felt taking that extra step is important, “I just want to see if the students are able to pick up the important information and see its relationship, and then to think how this information can lead to other information which they feel are related.” Two factors seemed to motivate the teachers to embark on the second experiment. First, they were not sure about their views. Second, they wanted their students to be more inquisitive. One way to help teachers in negotiating the barriers is to provide opportunity for them to articulate their concerns and to confront their apprehensions. In this case, the teachers weighed their concerns with the potential instructional benefits of KBC. They decided that the KBC approach is worthy of further experimentation. This historical development of the events showed the conditions and pathways of conceptual change, echoing Prawat’s suggestion that teachers may change their pedagogy when they are dissatisfied with their current practice and be exposed to new alternatives [4].

5. Conclusion This paper documented preliminary findings on how some Singapore teachers view KBC as an innovative use of technology for teaching and learning. In particular, the teachers interviewed provided generally positive accounts of teaching and learning experiences in a learning community supported by CSCL. However, they felt that time constraint is an issue and that the approach is more suitable for students with higher academic ability. Implementing innovative approaches to teaching and learning inevitably involve complex changes. Among them, the change in teachers’ belief is essential for effective integration. However, changing a belief is a difficult process. It requires the teacher to first

34

C.S. Chai and S.C. Tan / Fostering Learning Communities Among Teachers and Students

encounter that situation that calls for the necessary changes. The KBC model seems to be a promising approach. It provides an appropriately challenging environment for teachers “to reflect on their own beliefs about teaching and technology, as well as to consider the real-world limits that exist in today classroom” [20]. The text-based nature of the discourse provides a means for the teachers to articulate the issues they encountered. Once the issues have been identified and articulated, teachers could examine the issues and embark on a professional problem solving discourse with colleagues and researchers. The community of practice afford multiple ways of conceptualizing and solving the problems when teachers are faced with difficulties. Building on current understanding of teachers’ perception, the next step is to engage more teachers in design experiments. It is hoped that future research efforts invested in this line of study would help the teachers and the researcher to co-construct adaptations of the KBC that help to better equip Singapore learners for the knowledge society. References [1] Jonassen, D. H. (2000). Computers as mindtools for schools: Engaging critical thinking. New Jersey: Prentice Hall. [2] Jones, B. F., Valdez, G., Nowakowshi, J., & Rasmussen, C. (1995). Plugging in: Choosing and using educational technology. Washington DC: Council for Educational Development and Research, North Central Regional Educational Laboratory. [3] Applefield, J.N., Huber, R. & Moallem, M. (2000). Constructivism in theory and practice: Toward a better understanding. The High School Journal, 84(2), 35-53. [4] Prawat, R.S. (1992). Teachers’ beliefs about teaching and learning: A constructivist perspective. American Journal of Education, 100 (3), 254- 305 [5] Windschitl, M. (2002). Framing constructivism in practice as the negotiation of dilemmas: An analysis of the conceptual, pedagogical, cultural, and political challenges facing teachers. Review of Educational Research, 72(2), 131-175. [6] Feiman-Nemser, S. (2001). From preparation to practice: Designing a continuum to strengthen and sustain teachers. Teachers College Record, 103(6), 1013-1055. [7] Scardamalia M. & Bereiter C. (1996). Computer support for knowledge-building communities. In T. Koschmann (Ed.). CSCL: Theory and Practice of an emerging paradigm (pp. 249-268). Mahwah, NJ: Lawrence Erlbaum [8] Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Lawrence Erlbaum. [9] Lamon, M., Reeve, R., Caswell, B. (1999). Finding Theory in Practice: Collaborative Networks for Professional Learning. Paper presented at the annual meeting of the American Educational Research Association, Montreal. Retrieved 14 Oct 2002 from http://ikit.org/abstract/finding_theory.htm [10]Scardamalia M. & Bereiter C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265-283. [11]Yamagata-Lynch, L.C. (2003). How a technology professional development program fits into teachers’ work life. Teaching and Teacher Education, 19, 591-607. [12] Lave, J., & Wenger, E. (1999). Legitimate peripheral participation in the communities of practice. In R. McCormick, & C. Paechter (Eds.). Learning and knowledge. (pp. 21-35 ). Thousand Oaks, CA: SAGE Publication. [13]Woodruff, E. & Brett, C. (1999). Collaborative Knowledge Building: Preservice Teachers and Elementary Students Talking to Learn. Language & Education 13 (4), 280-302. [14]Yuen, A. (2003). Fostering Learning Communities in classrooms: A case study of Hong Kong Schools. Education Media International, 40(1/2), 153-162. [15]Earle, R.S. (2002). The integration of instructional technology into public education: Promises and challenges. Educational Technology, 42(1), 5-13. [16] Miles, M.B., & Huberman, A.M. (1994). Qualitative data analysis: A sourcebook of new methods (2nd Ed.). Thousand Oaks, CA: Sage Publications. [17]IKIT, (2003). What is a barrier-breaking innovations? Retrieved Oct 3, 2003 from http://ikit.org/Barrier.htm [18] Zhao, Y., Pugh, K., Sheldon, S. & Byers, J.L. (2002). Conditions for classroom technology innovations. Teacher College Record, 104(3), 482-515.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

35

Group Metacognition in a Computer-Supported Collaborative Learning Environment Chris CHALMERS, Rod NASON Centre for Learning Innovation, Queensland University of Technology, Australia [email protected] Abstract. This study investigated within-group metacognition during group problem solving conducted within the context of a Knowledge Forum¤ CSCL learning environment. Three malfunctioning groups of students from a middle-grade primary school in an inner-city school in eastern Australia participated in this study. Prior to the study, the majority of the groups’ time at the computer was spent on non-productive conflict. During the study, the three groups were provided with group strategies and metacognitive scaffolds to facilitate group metacognition. The scaffolds and strategies used were derived from the literature on metacognition, co-operative learning, problem solving, and computer-supported collaborative learning. The study found that providing students with metacognitive scaffolds and group strategies resulted in positive changes in the students’ group work at the computer. The students developed an understanding of how to contribute effectively to their group which enhanced the groups’ problem solving and knowledge-building. Keywords: CSCL, Knowledge Forum, group metacognition, group problem solving.

Introduction While the majority of research into metacognition focuses on the individual student [1, 2], in a group context such as Knowledge Forum, students are encouraged to form collective knowledge [3]. Scardamalia and Bereiter [4] suggest that by focusing solely on the individual student, researchers have failed to address the dynamics required for progressive knowledge-building by groups of students. Metacognitive individuals plan, monitor and evaluate their learning processes [5]. These abilities to plan, monitor and evaluate learning processes are also essential for groups building a shared knowledge in a CSCL environment [3]. It has been stated that learning environments that merely provide group work do not promote true knowledge-building [6]. Studies of groups that have had no preparation for co-operative learning have found that students do not involve themselves fully in the group work situation and do not interact as a group [7]. The use of small groups during knowledge-building activities needs more than placing students into groups because students also need to learn how to think (or metacognate) about how to maximise their groups’ learning [8]. While there are several approaches to metacognitive instruction, the most effective involves providing learners with knowledge of metacognitive strategies, practice in using the metacognitive strategies, the use of appropriate tasks, and opportunities to evaluate the outcomes of the strategies [9, 10, 11]. Johnson and Johnson [12] suggest that to help students’ process how well their groups are functioning, members need to reflect on what

36

C. Chalmers and R. Nason / Group Metacognition in a CSCL Environment

worked well and what needs to improve. Students need to think on a metacognitive level about their group work [13]. A review of the research literature indicates that scaffolding of metacognition needs to consider three main aspects: (1) providing students with knowledge about the learning process [14], (2) introducing the self-regulatory practices of planning, monitoring and evaluating [15, 16], and (3) providing a supportive environment that fosters and values reflective learners [17, 11]. Group metacognitive strategies need to focus particularly on the self-regulatory practices of planning, monitoring, and evaluating co-operative group problem solving [15, 17]. Group planning strategies help groups identify what group skills need to be focused on. Monitoring strategies, such as commenting on how their group is going and how each member is helping the group, focuses students on improving their group work. Finally evaluation strategies help groups reflect on what group skills they use and what skills they need to incorporate in the future. Costa and O’Leary [18] refer this type of behaviour as co-cognition. They define co-cognition as the co-operative development of a group intellect which is used to plan, monitor, and evaluate group behaviour. They believe that groups of students need to develop co-cognition in order to collaboratively develop concepts and monitor their own group performance. Collins [19] feels that in order for knowledge-building to occur, students need to learn to apply co-cognition to their group situation, while communicating information to other members of their group. Cohen [20], however, pointed out that as well as metacognating on group skills or teamwork, groups also need to focus on the subject matter or task work.

1. Method According to Yin [21], the purpose of a case study is to ‘investigate a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident’. A case study methodology thus was deemed as being a most appropriate methodology for this study. A case study methodology also was utilised in order to bring out the details from the viewpoint of the co-operative groups from multiple sources of data [22]. Aspects of a teaching experiment approach were incorporated into the methodology in order to study the influence of group metacognition instruction on the co-operative groups [23]. The three groups of students participating in this study were purposefully selected from a cohort of two combined grade 4/5 classes who was involved in a larger research project in which the focus was on creating and maintaining a mathematical knowledge-building community of practice within a CSCL environment. Eleven groups of three students, including the three purposefully selected groups, within this CSCL environment were engaged in the task of creating an alternative mathematical model that could be used for ranking nations’ performance at Olympic Games which de-emphasised the mind-set of “gold or nothing.” Three students per group is considered the minimum size for a group and is a small enough number for a group to work comfortably around a single computer [24]. In order to create their model, each group of students was required to search for relevant information from a web based database and using this information (e.g., population, gross domestic product, literacy rates etc.) develop an Excel ® spreadsheet model which ranked 20 different countries’ performance at the Sydney Olympic games in a “fairer way” than by ranking in terms of gold medals won. Following the development of their initial model, each group of students was required to post their model into Knowledge Forum ® [25], a computer-supported collaborative learning environment that enables users

C. Chalmers and R. Nason / Group Metacognition in a CSCL Environment

37

to share their model with other groups in the community and also to view and make constructive comments about other groups’ models. After posting of their own model, each group of students was required to look at the models the other groups had posted on Knowledge Forum® and provide constructive feedback (such as comments, questions, propositions) to the other groups about their models. They also simultaneously engaged in the process of iteratively revising and improving their model based on the feedback they received from the other groups of students. It soon became apparent to all involved in the larger research project that the majority of students in the two grade 4/5 composite classes needed to improve their group workskills, as they were not working co-operatively. Three groups in particular were dysfunctional and were having a deleterious effect on the overall functioning of the learning community. Therefore, these three malfunctioning groups were purposely selected to participate in a satellite research study whose major aim was to investigate how the introduction of group metacognitive strategies could affect group problem solving work at computers. This satellite study also aimed to discover the nature of group metacognitive knowledge by describing how group behaviours changed due to the metacognitive experience.

1.1 Procedure In order to facilitate group metacognition within the groups, a fifteen minute lesson was presented at the start of three consecutive computer sessions. The goal of these lessons was to introduce specific skills needed for effective co-operative groups as well as to scaffold the group metacognition [26]. The skills were based on Dishon and O’Leary’s [26] group task and maintenance skills. The group diary was introduced during the third lesson to scaffold group metacognition and in order to document issues the group encountered in the process of working together [27, 28]. Lesson 1 entailed asking students to think about why they work in groups and what skills are needed when working in groups. During Lesson 2, students were shown a poster with the task and maintenance aspects of group work. The columns in the poster were later renamed by the students as the ‘What’ and ‘How’ of group work. Lesson 3 entailed students having increased responsibility for planning, monitoring, and evaluating their own group learning. After each lesson student groups recorded one ‘What’ and one ‘How’ behaviour in their group diary, which they used for self monitoring and evaluating of their group work. Each group of students eventually developed their own list of helpful behaviours for improving their group work [20, 29]. The student generated criteria (Figure 1) served as an indicator for group members to evaluate their own group performance [20, 29, 30].

Figure 1. Group generated list.

38

C. Chalmers and R. Nason / Group Metacognition in a CSCL Environment

The nine students from the classes were observed and videotaped to capture the discourse within their groups. Videotaping the group work ensured that the study encompassed aspects of group work that was not apparent until viewing of the tapes. The videotaping also assisted in analysing the interpersonal interactions of the group situations and was useful for portraying the context [31,28]. Video observations were based on group task and maintenance skills and salient portions from the videotapes, which showed evidence of students’ use of either task or maintenance aspects of co-operative group work, were noted and comments included in later focus group interviews [26, 8]. A focus group interview was conducted, and audiotaped, after each computer session where groups were interviewed together after each session on how they thought the group had functioned, focusing on their group metacognition of the task and maintenance aspects of group work [1]. A focus group interview was used to record students’ reactions towards group work, as well as to ascertain students’ perceptions of group processes [32]. During the focus group interview students were encouraged to evaluate and comment upon their own participation and their interaction with others in the group [33]. The goal of the focus group interview was to elicit students’ perceptions, feelings, attitudes, and ideas regarding group thinking, as the data from the focus groups are often richer than data from individual interviews [32]. Individual student interviews were also conducted at the conclusion of the study. The questions were similar to the questions used in a study by Whicker, Nunnery and Bol [34] in which students were interviewed on their perception of co-operative group work. An open-ended questionnaire was used, as well as a group cohesiveness scale questionnaire. Analysis of the interview data, and the group cohesiveness scale questionnaires, tended to corroborate the analysis of the observation data and group diaries.

2. Results The corpus of knowledge about group problem solving and learning, derived from research into collaborative and co-operative learning over the last forty years, indicates that students’ learning in successful groups can achieve higher cognitive levels than working alone [13]. However, prior to this study, it was apparent that the three groups selected were not functioning effectively. This was also the perception of the group members themselves and is exemplified by the following comment by a group member. At the beginning I didn’t really like it cause He wasn’t exactly uum…wouldn’t stop climbing over desks or hiding under the desks. Studies have shown that students do not engage in metacognitive thinking unless they are encouraged to do so [30]. The three main metacognitive skills of planning, monitoring and evaluating are the three main reflective strategies identified in metacognitive literature [9, 17]. These strategies were used by the groups involved in the study, where students noted any difficulties with working in their groups and solved the problem as a group [8]. A change was noted over the study due to the use of group metacognitive scaffolds. Students built an understanding that their co-operative process could be improved through the use of relevant metacognitive group strategies including planning, monitoring, and evaluating. By planning which group skills would be focused on, students were able to identify group skills that needed to be improved. Students were also able to monitor and evaluate their group’s success of the use of specific group skills, discarding inappropriate group skills.

C. Chalmers and R. Nason / Group Metacognition in a CSCL Environment

39

Following the introduction of the group skills and metacognitive strategies students indicated that they considered participation in the group useful for sharing ideas. These comments are typical of how students perceived the benefits of working co-operatively: Talking to other people in group, getting their ideas helped a lot. In the group people disagree sometimes, but you get knowledge in a group. At first I didn’t like it cause I didn’t really work well with them, but became better friends. It was sort of easier to learn the formulas. Better by having more than one person, more ideas. Analysis of videotapes of groups co-operatively working, as well as extracts from group diaries and interviews, suggest that a number of group behaviours and roles were adopted in order for the groups to work effectively together. The study showed that the groups working together around the computer developed an understanding of how to contribute effectively to the knowledge building progress of the group. The groups used group strategies and the scaffolded metacognitive elements of planning, monitoring, and evaluating, in order to reflect on how well they worked as a group and how they could improve their future group work. It was noted that students progressively incorporated effective group behaviours into their co-operative groups; group behaviours were seen to improve for all groups over the course of the study. Students considered a major factor of their improved group work was the use of the group diaries. This is consistent with the literature, which states that students who monitor their own learning process, through the use of diaries or learning journals, are more proficient group learners [17]. This is exemplified from the following student comment: The group diary helped enormously. I think we get on well (much better than we did) working in this program and with them was fun. This study showed that by using the group diaries and through instruction, students built a collaborative knowledge regarding effective groups. The results presented, while not a comprehensive analysis of group metacognition, show insights into co-cognitive behaviours evident in all three groups involved in this study. The final comment written by one of the students involved in the study shows the group reflection that developed within the study: There’s a lot to learn but not as much as there was. I’ve learned a lot about computer group work and how to agree and disagree nicely.

3. Discussion While numerous studies have been conducted that provide information on individual metacognition few studies have investigated social shared metacognition in collaborative problem solving activities [15, 35, 1]. A CSCL environment, such as Knowledge Forum¤, emphasises a group perspective rather than one of an individual student [4]. Working in-group situations within CSCL environments students need to learn how to think (or metacognate) about how to maximise their groups’ learning [8]. The current understanding of students’ group metacognition is based on a relatively small amount of research [2, 11]. Previous research has focussed mainly on the development of strategies for problem solving. This study sought to address the void by looking at students’ group metacognitive strategies incorporating group taskwork as well as group teamwork. While this study addresses this issue to a certain extent, group metacognition requires further study and has implications for Knowledge building for

40

C. Chalmers and R. Nason / Group Metacognition in a CSCL Environment

groups building collective knowledge within the Knowledge Forum© learning environment.

4. Conclusion Co-operatively working as a group entails students sharing understanding and encouraging each other to complete the group task. However, while several studies suggest that groups are more productive than individuals, not all groups are co-operative [13]. Children do not always know how to work co-operatively as a group [20]. In most classes students are assigned to work in a group as individuals, they can seek information from each other, but work in groups rather than as a group [13, 7]. In order to work as a group students need to build a shared understanding of what they are working on as well as how they are working together [36]. In order for groups to be successful depends on knowing how and when to use specific problem solving and group strategies, as well as planning, monitoring and reflecting on the use of the strategies. Group metacognition involves knowing how groups learn and includes strategies to plan, monitor and evaluate group learning. Planning to achieve the group task goal, as well as monitoring the group’s progress required to achieve the group goal are metacognitive processes. This study showed that introducing the group metacognitive scaffolds improved students group work. Future studies need to address how to transfer the face-to-face group metacognition to the online Knowledge Forum environment and how both the teamwork and the task work can be scaffolded in order to support online group metacognition.

References [1] Hoyles, C., & Healy, L. (1994). Learning mathematics in groups with computers: Reflections on a research study. British Educational Research Journal, 20, 465-484. [2] Hurme, T., & Järvelä, S. (2001). Metacognitive processes in problem solving with CSCL in mathematics. Retrieved May 6, 2005 from the University of Oulu, Department of Educational Sciences and Teacher Education Web site: http://www.ll.unimaas.nl/euro-cscl/Papers/70.doc. [3] Bereiter, C., & Scardamalia, M. (1989). Intentional learning as a goal of instruction. In L. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 361-392). Hillsdale, New Jersey: Lawrence Erlbaum. [4] Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3, 265-283. [5] Schraw, G. (2001). Promoting general metacognitive awareness. In H. Hartman (Ed.), Metacognition in learning and instruction: Theory, research and practice (pp. 33-68). Dordrecht, The Netherlands: Kluwer Academic Publishers. [6] Hewitt, J., Scardamalia, M., & Webb, J. (2002). Situative design issues for interactive learning environments: The problem of group coherence. Retrieved May 6, 2003, from University of Toronto, Ontario Institute for Studies in Education Web site: http://kf.oise.utoronto.ca/abstracts/situ_design/ [7] Ogden, L. (2000). Collaborative tasks, collaborative children: An analysis of reciprocity during peer interaction at Key Stage 1. British Educational Research Journal, 26, 211-226. [8] Johnson, D., Johnson, R., & Johnson-Holubec, E. (1993). Co-operation in the classroom (6th ed.). Edina, MN: Interaction Book Company. [9] Flavell, J. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence (pp. 231-235). Hillsdale, NJ: Lawrence Erlbaum [10] Wilson, J., & Johnson, P. (2000). Students thinking about their learning: Assessment to improve learning. Educational Research Quarterly, 24(2). [11] Xiaodong, L. (2001). Designing metacognitive activities. Educational Technology, Research and Development, 49(2), 23. [12] Johnson, D., & Johnson, R. (1993). Implementing co-operative learning. Education Digest, 58(8), 62-67.

C. Chalmers and R. Nason / Group Metacognition in a CSCL Environment

41

[13] Johnson, D., & Johnson, R. (1999). Making co-operative learning work. Theory into practice, 38(2), 67-73. [14] Hartman, H. (Ed.). (2001). Metacognition in learning and instruction: Theory, research and practice. Dordrecht, The Netherlands: Kluwer Academic Publications. [15] Blakey, E., & Spence, S. (1990). Developing metacognition. (Eric Reproduction Services No. ED327218.) [16] Schraw, G. (2001). Promoting general metacognitive awareness. In H. Hartman (Ed.), Metacognition in learning and instruction: Theory, research and practice (pp. 33-68). Dordrecht, The Netherlands: Kluwer Academic Publishers. [17] Gourgey, A. (2001). Metacognition in basic skills instruction. In H. Hartman (Ed.), Metacognition in learning and instruction: Theory, research and practice (pp. 17-32). Dordrecht, The Netherlands: Kluwer Academic Publishers. [18] Costa, A., & O'Leary, P. (1992). Co-Cognition: The co-operative development of the intellect. In N. Davidson & T. Worsham (Eds.), Enhancing thinking through co-operative learning (pp. 41-65). New York: Teachers College Press. [19] Collins, A. (1991). Cognitive apprenticeship and instructional technology. In L. Idol & B.F. Jones (Eds.), Educational values and cognitive instruction: Implications for reform (pp. 121-138). Hillsdale, NJ: Lawrence Erlbaum. [20] Cohen, E. (1994). Designing groupwork: Strategies for the heterogeneous classroom.(2nd ed.). New York: Teachers College Press. [21] Yin, R. (1994). Case study research: Design and methods ( 2nd ed.). Newbury Park, NJ: Sage Publications. [22] Tellis, W. (1997). Application of a case study methodology. The Qualitative Report, 3(3), 1-17. [23] Lesh, R., & Kelly, A. (2002). Multitiered teaching experiments. In A. Kelly & R. Lesh (Eds.), Research design in mathematics and science education. (pp. 197-230). Mahwah, NJ: Lawrence Erlbaum. [24] Samovar, L., Henman, L., & King, S. (1996). Small group process. In R. Cathcart, L. Samovar & L. Henman (Eds.), Small group communication: Theory and practice (pp. 7-11). Boston, MA: McGraw-Hill. [25] Scardamalia, M., & Bereiter, C. (1998). Web Knowledge Forum. User guide. Santa Cruz, CA: Learning in Motion. [26] Dishon, D., & O'Leary, P. (1984). A guidebook for co-operative learning: A technique for creating more effective schools. Holmes Beach, FL: Learning Publications [27] Hare, L., & O'Neill, K. (2000). Effectiveness and efficiency in small academic peer groups. Small Group Research, 31(1), 24-54. [28] Walker, R. (1985). Doing research: A handbook for teachers. London: Routledge. [29] Davidson, N., & Worsham, T. (1992). Enhancing thinking through cooperative learning. New York: Teachers College Press. [30] Gillies, R. (2000). The maintenance of co-operative and helping behaviours in co-operative groups. The British Journal of Educational Psychology, 70, (15), 97–111. [31] Kvale S. (1996). Interviews: an introduction to qualitative research interviewing. London: Sage. [32] Vaughn, S., Schumm, J., & Sinagub, J. (1996). Focus group interviews in education and psychology. California, CA: Sage. [33] Kormanski, C. (1999). The team: explorations in group process. Colorado: Love. [34] Whicker, K., Nunnery, J., & Bol, L. (1997). Cooperative learning in the secondary mathematics classroom. The Journal of Educational Research, 91(1), 42-48. [35] Hamilton, M. (1986). Knowledge and communication in children. In C. Antaki & A. Lewis (Eds.), Mental mirrors (pp. 78-96). London: Sage. [36] Mulder, I., Swaak, J., & Kessels, J. (2002). Assessing group learning and shared understanding in technology-mediated interaction. Educational Technology & Society, 5(1), 35-47. [37] Hyde, A., & Hyde, P. (1991). Mathwise: Teaching mathematical thinking and problem solving. Portsmouth: Heinmann.

42

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Examining the Psychological Impact of Video Cases for Teacher Education Peter Y.K. Chan Brigham Young University Hawaii 55-220 Kulanui Street, #1963, Laie, HI 96762 USA Phone: 1-808-293-3853; Fax: 1-808-293-3237 Email: [email protected]

Abstract. This study examines teachers’ cognitive development when interacting with video ethnography, a form of video cases. It used grounded theory to discover embedded meanings and relationships that emerge from descriptive data collected from six teachers. Findings revealed the categories of cognitive activities when using video ethnography and teachers’ progression in a cognitive development process through interaction with video ethnography. The study has implications in improving technology use in teacher development, production of video cases, and research on case-based pedagogy and other related areas. Keywords: Cases, Cognition, Ethnography, Teacher Education, Video

Introduction In recent years, video ethnography, and video cases in general, have attracted much attention in teacher education. As pedagogy for the professional development of teachers, these technology-based interventions are an enriched form of the case method. Traditional teacher education presents learners with abstract and generalized formal research findings. In contrast, cases are renditions of authentic, concrete teaching episodes (Bruner, 1986). As a form of video case, video ethnography has the potential to further expand the traditional text-based cases for use in teacher development. As a research method, video ethnography culminates a participant observer’s experiential study of teaching and learning as it unfolds in a live classroom. The resulting video case becomes the ethnographer’s qualitative account systematized according to outstanding themes and sub-themes. It is supported by multiple perspectives of the observed, concerned constituents, and related professional literature. Its purpose is to improve understanding by presenting practice juxtaposing with theory.

1. Statement of the Problem However, there is lack of understanding on how video ethnography may impact the thinking of teachers. Richardson and Kile (1999) pointed out that much of the literature on video cases is speculative and even promotional. Few systematic studies have demonstrated how they work in the minds of their beholders—teachers (Merseth, 1999). If teachers’ cognitive complexity is influenced through interaction with video ethnographies, then the use of these kinds of cases may hold important promise in promoting teacher change, because research shows that increasing teachers’ cognitive complexity leads to competence in handling

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

43

difficult tasks (McKibbin & Joyce, 1981). Therefore, prominent scholars in teacher education, for example, Richardson (1999) and Shulman (2001) have called for research that can illuminate on the relationship between video cases and teachers’ cognitive development.

2. Purpose of the Study In response to this noted deficiency in understanding how video ethnography relate to teacher thinking, the current study explores the cognitive activities of six teachers during and immediately after their interaction with video ethnography. As Carter (1990) indicates, “Learning-to-teach questions might well be unanswerable at a global level. What is needed, instead, are frameworks that focus more explicitly on what is learned and that specify more fully how that knowledge is acquired.” (p. 293) Therefore, the purpose of this study is to explore how video ethnography relates to the thinking of a small sample of practicing teachers. Through observations and interviews, theoretical propositions are developed to describe how teachers think about their own pedagogy as they respond to video ethnography.

3. Questions The overarching question for this study is: What is the nature of teachers’ cognitive activities when interacting with video ethnography? In exploring this question, four specific questions guide the study: 1. What cognitive activities are revealed in teachers’ thinking through their use of a video ethnography? 2. What patterns emerge from examining participants’ cognitive activities? 3. What theories of cognitive development in relation to video ethnography can be generated by cross-case analysis? 4. How might these theories guide future professional development research for teachers?

4. Defining Video Ethnography This study used a video ethnography—The Jean Turner Case (Harris, Turner, & Baker, 2001)—as the target object of investigation. It captures a master teacher, Jean Turner, working with fourth grade students in a morning of balanced literacy. A CD-ROM that typifies the video ethnography interface, it will be introduced in this section using much of Harris, Pinnegar, and Teemant’s (2005) work in introducing a similar title. Figure 1 shows the video ethnography’s basic interface. On the top left are the four study buttons representing four studies of the key components of this balanced literacy lesson: (1) morning of balanced literacy, (2) writing, (3) reading, and (4) management and motivation. On the top right are the nine probe buttons, which “represent examples of concepts that develop the theme of the study” (Harris et al., 2005, p. 150). In other words, each of the four studies brings forth a different set of nine probes. When teachers explore these studies, “they select a study button, and are immediately presented with the labeled probes, which contain video clips illustrating the probe and the larger focus of the study….Typically one of the probes in the study also allows the user to view all clips in

44

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

sequence” (Harris et al., 2005, p. 151).

Study buttons

Probe buttons

Video screen

Movie slider

Buttons for additional features

Audio slider for Commentaries Four sources of commentaries: literacy specialist, professional literature, case teacher, and another teacher

Figure 1: Video Ethnography Interface 5. Method 5.1 General Approach The study uses a grounded theory approach to discover explanatory premises that relate to learner cognition and its relationship with learning orientation. At its first stage, it selected a purposeful sampling (Patton, 2001) of six teachers stratified by their years of teaching and acquired basic understanding of their cognitive makeup including biographic information, learning orientation, and beliefs about effective teaching generally and literacy teaching specifically. The second stage comprised the major portion of the data-collection process. It included think-aloud and self-reports stimulated by video clips and commentaries. When each video clip or commentary ended, the researcher asked the participant, “What is going on here?” and “What are some of your insights inspired by this clip?” The purpose of these questions was to provoke open-ended responses to the video clip or audio commentary. The last stage was debriefing, which took place immediately after each participant finished reacting to the multimedia probe.

5.2 Participants This study used a purposeful sampling (Patton, 2001) of six teachers stratified by their years

45

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

of teaching to explore probable relationship between teachers’ seniority and their cognitive activities when using video ethnography. Two teachers came from each of the three strata, represented by the three teachers’ career cycles: survival (first year), adjustment (second to fourth year), and mature (fifth year up) (Burden, 1980). As the study used a video ethnography of a fourth grade balanced literacy class, participants of the study find it to be most relevant because they were also fourth grade teachers. Selected teachers had no training in balanced literacy, thus, avoided influence from prior knowledge of this area.

5.3 Data Collection and Analysis The study used questionnaires, interviews, think-aloud, and self-reports to collect participants’ verbal responses which were recorded on a tape-recorder and transcribed later. Its analysis process began by formulating a preliminary framework regarding teacher cognitions based on existing literature. As the coding process continued, new criteria emerged to offer specific depiction of novel accounts. Hence the resulting set of criteria became an indication of the participants’ cognitive activities, which in turns further develop the criteria. The frequency of each cognitive criterion is counted and correlated to participants’ learning orientation scores.

6. Findings 6.1 Cognitive Activities of the Video Ethnography Users Table 1. Final set of criteria for analyzing participants’ verbal responses Reflection Awareness

Comprehension

Acceptance

Rejection

Connection

Desire to act Requesting More

Applying

Sharing Experience

Comparing

Sharing Belief

Judging negatively

Not Liking

Disagreeing

Judging positively

Liking

Agreeing

Assuming

Expressing Uncertainty

Interpreting

Recalling

Advanced Noticing

Basic Noticing

Hence, the first part of findings includes the identification of different cognitive activities. The initial set of criteria based on literature included noticing, reflecting, and problem-solving, which soon became inadequate in labeling other verbal statements uttered by participants. Thus, new criteria emerged to depict novel incidents. Table 1 shows the final set of criteria. The two main cognitive levels are awareness and reflection: awareness is the fundamental and reflection is the sophisticated level of thinking. Awareness activities are categorized into three types: basic noticing, advanced noticing, and recalling. Reflective activities are categorized into five types: comprehension, acceptance, rejection, connection, and desire to act with fourteen subcategories (see Table 1). These categories are not predetermined coding criteria. Rather, they are the resulting labels from the researcher’s effort to depict the genres of cognitive activities that emerged from

46

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

transcriptions of participants’ protocol when they interacted with the video ethnography CD. 6.1.1 Awareness Awareness level activities are rudimentary to the higher reflective level activities. They were characterized as first becoming aware of or paying attention to what is seen and heard. Awareness activities include noticing and recalling. Noticing refers to what the participants observed during or after watching a video. It is further divided into basic and advanced noticing. Recalling refers to their recollection of a commentary. Examples of awareness include: • “They're listening to music” (Deborah, Basic Noticing). • “She's getting the class’s attention and the kids all seem really on task” (Jennifer, Advance Noticing). • “He was talking about how she was modeling during this clip, and how it was considered a shared writing” (Elizabeth, Recalling).

6.1.2 Comprehension Comprehension is the essential beginning level of reflection and the process of internalizing the encountered. It is the taking in (or not taking in) of an external stimulus to make meaning personal. It includes three activities: 1) interpreting—a participant’s description of what she has perceived rather than a direct account of an observed fact, 2) expressing uncertainty—an expression of incomprehension in the comprehending process, and 3) assuming—extending one’s comprehension of uncertain events by speculating based on what is known. Examples include: • “She's monitoring her class, but the kids are actually in charge of their own learning from what I'm seeing” (Elizabeth, Interpreting). • “The teacher's got some papers she's checking off as she watches them do their reading. I'm not sure what she's looking for” (Deborah, Expressing Uncertainty). • “I'm assuming that the balanced literacy and workshop are similar, or the same thing” (Elizabeth, Assuming).

6.1.3 Acceptance Acceptance refers to the favorable reception that a participant expresses. It contains three subcategories of activities: agreeing—a statement that explicitly uses the expression “I agree,” liking—a statement that explicitly uses the words “like” or “love” to express the participants’ preference of what they had seen or heard, and judging positively—an expression of a favorable opinion towards what is observed or heard and use words such as “good,” “great,” and “wonderful.” Examples include: • “Absolutely! I agree one hundred percent with what he said there. He said that it's important to have time in the classroom for the students to read, and I agree one hundred percent” (Carolyn, Agreeing). • “I like the fact that they're talking and discussing and making predictions and discussing it” (Elizabeth, Liking).

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

•

47

“She makes them accountable; very good management skills” (Deborah, Judging Positively).

6.1.4 Rejection Contrary to acceptance, rejection refers to participants’ negative remarks towards a certain aspect of the contents. It is a rare expression for most participants except for Carolyn, the most experienced teacher in the group, whose comments were significantly rampant with negative comments. There are three subcategories: disagreeing, not liking, and judging negatively. “Disagree” and “not liking” are expressions where participants explicitly said they disagreed or did not like what they had seen or heard. Statements that “judged negatively” expressed a negative sentiment but did not use either of these words. Findings in this area coincide somewhat with Fessler and Christensen’s (1992) idea that experienced teachers are more confident in voicing their contrary opinion to what authority has instructed. The two most experienced teachers, Carolyn and Elizabeth, had significantly higher numbers of rejection comments, while the other teachers did not show significant differences among one another. For example, Elizabeth says, “Reading aloud is a very key, essential point in my opinion. I don't think she's doing enough of that, at least from what I've seen” (Judging Negatively, Commentary 4).

6.1.5 Connection The connection category includes reflective statements that show explicit links between the presented materials and the participants’ own teaching and learning beliefs and experiences. It is further divided into three subcategories: sharing belief—an opinion a participant shared based her own value and experiences, comparing—a statement that shows a participant’s comparisons between the video ethnography contents and their own practices, and sharing experience—a statement about a participant’s own teaching and learning experience. Examples include: • “If we can teach kids to love to read, that would be the greatest gift that we could give them” (Deborah, Sharing Belief). • “I have kids who embellish and make things up as they go along and they sound like they know what's going on when they really have no clue” (Jennifer, Comparing). • “I try to do that with my students, to show them that I love to read and I love to get lost in books and to spend a lot of time doing that silent reading” (Heidi, Sharing Experience).

6.1.6 Desire to act Desire to act statements denote a desire to act. They consist of two subcategories: applying—an expression of the intention to apply what the participant had learned, and requesting more—an expression of the desire to know more about the particulars of the practice or to contact the case teacher for more information. • “This one let the students see the positive experience to be involved in creative writing, and even the teacher would like to ask questions and get some questions answered on what could make her story better…. I would like to do this in my own classroom. I enjoy writing and encourage creative writing a lot in my

48

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

•

classroom, and I would like to see something like this. I would like to try this and see how it goes with my own students” (Kimberly, Applying). “I want to know how long she takes to do that, and how long does she take to teach this process. Is it a matter of a few days, weeks? I want to sit down and pick this woman's brain, because I'm curious about how long she takes, because I like the fact that she's using this, and I think she sets up her classroom very well?” (Elizabeth, Requesting More).

6.2 Cognitive Development When Using Video Ethnography These six categories and 17 subcategories of cognitive activities stated above generated distinctive patterns in each participant. Cross-case analysis revealed a mental process that appears to be repetitive in all participants, revealing how one cognitive activity may relate to another. The resulting proposition is a cognitive development process that includes awareness and the five categories of reflective activities: comprehension, acceptance, rejection, connection, and application, as represented in Figure 2.

Acceptance

Connection

Rejection

Belief & Experience

Desire to Act

Comprehension

Awareness

Video Ethnography

Figure 2. Participants’ cognitive development process when using video ethnography Participants’ verbal reports showed that awareness is the fundamental, essential condition to reflection so it stays at the bottom of the graphic with an arrow pointing up. Teacher-learners must notice what is happening or has happened before reflecting on it. This noticing process goes from basic to advanced levels. That is, it goes from superficial descriptions of the encountered to contextual accounts using eloquent terms. The level beyond awareness is comprehension, which is the internalization of the encountered. The result of the comprehension process is increased understanding through interpreting and

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

49

assuming. Awareness and comprehension must be present in order for a teacher-learner to reflect further into the process. Rarely did any of the participants skip these two levels in their A think-aloud. Even when they did, we could assume they had gone through these steps without verbalizing them. Beyond comprehension are three parallel categories of activities: acceptance (liking, agreeing, and judging positively), connection (sharing belief, comparing, and sharing experience), and rejection (disliking, disagreeing, and judging negatively), placed just above comprehension in Figure 2. Acceptance, connection, and rejection are parallel to one other because any one of them could happen immediately after a participant had comprehended what was happening--they might even happen in any order. For example, a participant might feel positive (acceptance) about a particular comment and then compare it with her own teaching (connection), or she might share an experience first (connection) before stating her judgment of a clip. Such juxtaposition between a judgment (acceptance or rejection) and connection can be found in each participant. In most cases, this trio of cognitive activities concluded a participant’s reaction to a certain probe. At times, however, a participant might desire more information about the new technique, or she simply expressed a desire to apply what she had just learned. Such a desire to act, placed at the top level of the model, usually follows acceptance, but it might also occur after one has made a connection implying an implicit acceptance. Although we have not seen it in this study, it is probable that a teacher-learner may desire to apply a modified form of a rejected idea. These six categories of cognitive activities proceed from lower to higher levels, representing a cognitive development process that advances from initial cognition to action. A participant’s reaction to a certain idea might stop at any of these categories. That is, she might express her observation of a certain object or her interpretation of a practice, and then never mention it again. In Probe 8, for instance, Heidi simply restated the professional literature perspective without giving any of her own opinions. Within the scaffold of video ethnography, belief and experience are strong factors affecting a teacher’s advancement in the cognitive development process. As symbolized in a video ethnography case is used as the bottom-line stimulus tool for unlocking the cognitive development process. Next, the circular nodes represent the various components of the cognitive development process: starting from awareness at the bottom, to comprehension, to connection or judgment (acceptance or rejection), and eventually to a desire to act. The influences, belief and experience, are placed on the sides of the cognitive development process. Thus, this model ties together the four theoretical propositions and explains how teachers learn when interacting with video ethnography.

7. Conclusion This study responded to the need for research that examines teachers’ cognitive development when using video ethnography. It used a grounded theory approach in examining data collected from six teachers in three stages. Findings generated theoretical propositions that identified teachers’ cognitive activities when using video ethnography and explained how video ethnography scaffolds the development of these cognitive activities. It suggested a development model that illustrated how cognitive activities advances from one level to another. These findings have many implications in pedagogical, technological, and research areas.

50

P.Y.K. Chan / Examining the Psychological Impact of Video Cases for Teacher Education

8. References [1] Bliss, T., & Mazur, J. (1996). Creating a shared culture through cases and technology: The faceless landscape of reform. In J. Colbert, P. Desberg, & K. Trimble (Eds.), The case for education: Contemporary approaches for using case methods. Needham Heights, Massachusetts: Allyn & Bacon. [2] Carter, K. (1990). Teachers’ knowledge and learning to teach. In W. R. Houston (Ed.), Handbook of research on teacher education (pp. 291-310). New York: Macmillan. [3] Harris, R.C., Pinnegar, S., & Teemant, A. (2005). The case for hypermedia video ethnographies: Designing a new class of case studies that challenge teaching practice. Journal of Technology and Teacher Education, 13(1), 141-161. [4] Harris, R. C., Turner, J., & Baker, D. (2001). The Jean Turner Case: A Video Ethnography of Fourth Grade Balanced Literacy Classroom [CD-ROM]. Provo, UT: Harris Video Cases. [5] Hunt, P. (1951). The case method of instruction. Harvard Educational Review, 21 (3), 175-192. [6] Martinez, M (2000). Learning Orientation. Retrieved May 6, 2002, from the Training Place Web site: http://www.trainingplace.com/loq/loqinfo.htm (Used with permission by Margaret Martinez). [7] Merseth, K. (1981). The case method in training educators. Cambridge, MA: Harvard Graduate School of Education Doctoral Qualifying Paper. [8] Merseth, K. (1999). Forward: A rationale for case-based pedagogy in teacher education. In M. Lundeberg, B Levin, & H. Harrington (Eds.), Who learns what from cases and how? The research base for teaching and learning with cases (pp. ix-xv). New Jersey: Lawrence Erlbaum Associates. [9] Patton, M. (2001) Qualitative research and evaluation methods (Third edition). Thousand Oaks, CA: Sage Publications, Inc. [10] Richardson, V., & Kile, R. (1999). Learning from Videocases. In M. Lundeberg, B Levin, & H. Harrington (Eds.), Who learns what from cases and how? The research base for teaching and learning with cases (pp. 121-136). New Jersey: Lawrence Erlbaum Associates. [11] Shulman, L. (2002). Truth and consequences? Inquiry and policy in research on teacher education. Journal of Teacher Education, 53(3), 248-253. [12] Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (1988). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. In Tenth annual conference of the cognitive science society (pp.377-383). Hillsdale, NJ: Lawrence Erlbaum Associates. [13] Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks, CA: SAGE Publications.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

51

Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring Ya-Wei Chang1, Emily Ching1, Chih-Ti Chen2 , Tak-Wai Chan1 Graduate Institute of Network Learning Technology, 2Department of Computer Science & Information Engineering, National Central University, Taiwan [email protected]

1

Abstract. It has already been proven that peer tutoring is an effective way to engage students in learning. However, without technology support, it can be difficult to conduct peer tutoring program in a classroom. We propose a system called P3T that supports peer tutoring in one-to-one digital learning environment. P3T facilitates peer tutoring by scaffolding both tutors and tutees to prepare for the tutorial session and facilitating their elaborations during the face-to-face tutoring. An experiment was executed to test if the P3T system can help the activity go more smoothly than those without P3T system. This paper discusses the design of P3T models and system, and reports what is found in our experiment. Keywords: computer supported peer tutoring, computer supported learning by teaching, one-to-one digital learning, one-to-one computing environment, reciprocal peer tutoring

Introduction Bargh and Schul [1] have found that students in a teaching situation scored higher on an achievement test than those who did not teach. They suggested that preparing to teach someone could produce a more highly organized cognitive structure. It means that someone preparing to teach will reorganize the materials for clearer presentation and may clarify material while doing verbally teaching. Even though education and psychology researchers found that most peer tutoring classrooms benefit students positively in both cognitive and affective domains, such classroom peer tutoring programs without technology supports are usually inconvenient during the learning process. For example, the design that peer tutor in class read the questions and answers from studying sheets when interacting with the tutee may need many paper works and limit the learning task to be simple ones [2][3]. If a classroom can be well equipped with computing devices under wireless environment, students could benefit from a more convenient and interactive way to communicate with their peers and teacher. Those devices help to develop more ingenious instruction methods and aid in leading the activity proceed smoothly. Indeed, price of computers has been dropping so fast that we expect more and more students will gradually enjoy their studying in one-to-one (1:1) classrooms, where every student is equipped at least one wirelesscapable computing device. P3T stands for preparing and performing peer tutoring in a 1:1 classroom. It is a system designed to assist the student carry out three tutorial activities: learning about the materials, composing tutorial notes, and conducting face-to-face tutoring. We hypothesis and try to testify that computer-mediated face-to-face interaction plays a crucial element in our pedagogical design. This paper will pay more attention in the part of face-to-face tutoring. After the experiment we got feedbacks from the interviews, and questionnaires were done

52

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

by tutors and tutees. According to these data we can adjust P3T system for our future experiment. This paper intends to give a considerably detailed description of the system design and architecture, findings of our experiments, and future work. while the detailed survey and rationales of P3T activity are presented in [4] and [5]

1. Background The literature on peer tutoring programs has shown that having students teach their peers benefits both “tutor” and “tutee” [6][7]. In this section we summarize findings of some studies investigating the learning effects of peer tutoring. Learners can benefit from being taught by their peers. Schwenk and Whitman [8] pointed out that school faculty may have difficulty to make their students understand a procedure because they are “unconsciously competent,”. However, the students who just have learned the procedure may find it easier to teach because they are “consciously competent,” that is, they still have to think through each step of the procedure, one step at a time. On the other hand, learners also benefit from teaching their peers. In Annis[9]’s study which involved college students, those who prepared to teach and taught the contents of one thousand word articles tested higher than those who only prepare to teach. Both groups tested higher than those who were taught and those who taught themselves. Annis concluded that possible explanation for these results lies in a three-stage theory of learning that involves (1) paying attention to material to be learned, (2) coding it in a personally meaningful way, and (3) associating it with what is already known. Students who prepared to teach and taught may have learned the most because their responsibilities required them to perform all three of the steps essential to learning. Computer-supported peer tutoring were proposed by Chan and Chou [10][11]. They designed a Reciprocal Tutoring System (RTS) which had the role of tutor and tutee is alternative between the user and the computer [10]. In the Distributed Reciprocal Tutoring system (DRT) the proposed, two learners, working collaboratively but separately on two connected machines, play the role of tutor and tutee alternatively with a virtual peer as an intelligent “super-tutor” [11]. Both of RTS and DRT were used for college students to learn LISP by problem solving, and they didn’t involve students in face-to-face collaboration or elaboration.

2. Model Design P3T is designed to promote tutor’s innate motivation for comprehension, improve the learning effects during their preparation for teaching, and help them rearrange the material to explain in a clear way. And tutees should also be involved in active learning to increase their cognitive gains during the peer tutoring activity. Based on these issues, P3T is featured in three phases that we make a simple description in this section. The first phase proceeds before class while the second phase happens in class. The third phase involves the class teacher in leading a whole class discussion.

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

53

2.1 Preparation Phase This phase consists of four sub-phases: (1) Learning about the materials. Instructor provides two different materials which are similar in topic with the same unit, and all tutors are divided into two groups for reading different materials and then preparing to teach their own tutees. Tutees also have to read the assigned material. (2) Constructing individual tutorial notes. After studying the assigned materials, each tutor has to compose tutorial notes. (3) Assessing peers’ tutorial notes. Tutors who reading the same materials have anonymous evaluation on each other’s tutorial notes. (4) Collaborating for integrated tutorial note. Two tutors who learn the same unit with different materials have to integrate a common version of tutorial note for their posterior teaching.

2.2 Performing Peer Tutoring Phase This phase consists of three sub-phases: (1) Conducting expository tutoring. The integrated version of tutorial note is used by the two tutors to one-to-one teach their own tutee. As demonstrated in Figure 1, A teaches C and B teaches D. Tutor will also prepare questions in advance for tutee to answer during this phase(the fourth block in Figure 3), and so does the tutee(the third block in Figure 3). If problems occur during the teaching process, cooperative tutors could help each other. If the members in the same all confuse with one topic, they can send the question to instructor through the system, then the instructor will come to resolve their problem(the second block in Figure 3). (2) Tutoring assessment. Self-assessment and mutual-assessment were done between tutor and tutee. Tutor assesses her/his own teaching performance and also assesses her/his tutee learning attitudes, and vice versa. (3) Answering questions and group discussion. Every student individually answers questions distributed by the instructor and then discuss the solutions among the team members. During this step, students interact with each other face to face with the supports of P3T system that can show all members’ answer in one frame to facilitate discussion.

Figure 1 P3T ‘s interface and interactivity model

Figure 2 System architecture.

54

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

2.3 Instruction-led Discussion Phase The class instructor gives complementary instruction for the assigned material and explains answers of the questions in the performing peer tutoring phase. With the supports of P3T system, the instructor can view all reported answers and then adjust her/his explanations to give a complementary mental instruction. Instructor can also present some selected student answers to enrich the class discussion.

3. System Architecture The architecture of P3T System is shown in Figure 2. There are three kinds of participants in the whole P3T activity, which are the tutor, the tutee, and the instructor. Tutors and tutees are both students while the instructor means the class teacher. The following description will clarify the functions of the various modules, as well as their interactions: • Grouping Module enables the instructor groups all students into tutors and tutees. • In Assignment Module, tutors are assigned materials about the topic they are going to teach. • Composing Module assists tutor in the preparation phase, scaffolds tutors to structure their tutorial notes individually, allows tutor to download peer tutor’s tutorial notes so they can collaborate for an integrated version of tutorial note. The three modules mentioned above are designed for the preparation phase. • Inquiry module lets tutor and tutee to see the questions that they prepare in advance to ask each other during the teaching. Thus they can pay more attention in this subject which appears in the tutorial note. Even more, tutor may rearrange the teaching flow or decide when to resolve these questions. • Assessment Module enables self-assessment and anonymous peer assessment of tutorial notes according to the assessing rubrics and also provided self-assessment and mutual-assessment for each tutoring pair during tutoring assessment subphase. It also returns the assessment result to them. • In Presentation Module, the intergraded version of tutorial notes is retrieved and displayed for a tutor-tutee pair. The tutee notes down whatever she/he has puzzles in mind therefore tutor immediately grasps from the system notice and solve the problem in person. If none of them solve the problem, the tutor asks for help from the instructor. During peer tutoring, all of the students’ queries and answers are recorded into database which is connected to Assisting Module. • Assisting Module supplies an interface for the instructor to assist students to solve problems while difficulties occur in one-to-one tutoring; therefore, when the class instructor sees student’s signal of asking for help from the screen, she/he can either directly go to answer the student, or leave it to later instruction-led discussion in class. Then students won’t stagnate when they get confused with one topic. • In Quiz Module, every student will receive questions provided by instructor and submit answers individually. Answers will be recorded and scored (for those of multiple-choice questions) for Group Interaction Module and Monitoring Module. • In Group Interaction Module, the system allows each student to see team members’ answers in a frame and discuss each other’s answers. A special design is that associated with the same multiple-choice question, the differnt answers will be in different colors. This design helps team members see their knowledge conflicts and elicit deeper discussion. Students can always revise their answers

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

•

55

until the instructor stop this activity. Monitoring Module helps the instructor get the whole tutoring process in control. The instructor also can display some students’ answers to lead a further discussion. In Instruction notes DB and Assessment DB, instructor could check the each tutor’s tutorial notes to see if it meets the standard, and provide the best sample to other tutors for reference.

Figure 3 The top of this frame is the process guideline. The four blocks are for downloading tutorial notes, asking for help from instructor, displaying questions that tutee prepared, and my question to ask tutee in order.

Figure 4 The system allows each student to watch team members’ answers in a frame and discuss each other’s answers. The question type are multiple choice and open question. And the red color means different to other members.

4. Method Two experimental trials were conducted to verify the P3T System. In the first experimental trial, half of the participants perform P3T learning activity only by paper and pen, while the other participants using the P3T System. Then in the second trial, the participants who used paper and pen in the first trial performed the activity with P3T System, and vice versa. The purpose was to have all the participants experience both learning context, with and without the technology supports, so that they could provide more objective opinions in our survey.

4.1 Questionnaire Subjects were 26 students taking a gratuated-level course at National Central University, Taiwan. Students were administered an anonymous questionnaire following two experimental trials of P3T activity in 1:1 classroom. This questionnaire stress more about the advantage and disadvantage of P3T System.

4.2 Interview After the experimental trials, we interviewed 10 participants. They are interviewed about what they learnt and what had driven them to learn in each in-class sub-phase. All the interviews were voice-recorded. Some selected feedbacks were excerpted in Table 2.

56

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

5. Result 5.1 Questionnaire Participants were requested to fill in a questionnaire asking them to value if the system is good for the process of peer tutoring, from both perspectives of the tutor and the tutee. The results were presented in Table 1. Table 1. The satisfied degree of user for P3T system Questions Yes (%) 1. It is convenient to me to download the tutorial notes directly 20 (77%) from the P3T system during the tutoring session. 2. The function of prompting my peer tutee’s/tutor’s pre-class questions while I am tutoring/learning can reminds me where should 21 (81%) be emphasized more during the tutoring session. 3. When all the members in my team confront the same confusion, it’s very useful to me to ask for instructor’s help via the system, and 22 (85%) the instructor will come to resolve our questions. 4. It is comfortable to do self-assessment and mutual-assessment 13 (50%) privately through the system. 5. It helps me to look back my learning process because the system 22 (85%) reserves all learning profile. 6. It is good for my teaching/leaning reflection by the system showing 19 (73%) me the assessment results by my peer. 7. Teaching/learning through the system goes more smoothly than 19 (73%) using paper and pen. 8. Answering questions through the system helps my individual 15 (58%) thinking, as we can’t see peer’s answer. 9. It helps us to discuss the core question by see team members’ 21 (81%) answers in a frame and discuss the differences among our answers. 10. It facilitates the different answer distinguishability by different 25 (96%) colors. 11. In general , the user interface is friendly to me. 14 (54%) 12. The color manipulation in the interface is comfortable, not 14 (54%) pressured. 13. Learning to use the system is not difficult for me generally. 20 (77%) 14. The system runs smoothly for me generally. 16 (62%)

No Comment

No

6

0

5

0

2

2

11

3

4

0

6

1

6

1

9

2

5

0

1

0

10

2

12

0

5 9

1 1

According to the questionnaire results, the P3T system was the most useful for the participants during the sub-phases of answering questions and group discussion. 96 percent of the students thought that it fostered their awareness to distinguish different answers from other members’ by using different color, It also helped them discuss the core question by see all the teammates’ answers in a frame(Figure 4) and discuss each other’s answers (81%). The second most useful P3T function is offered during the sub-phases of conducting expository tutoring. They can download tutor’s tutorial notes easily (77%), ask for and get helps from the instructor via the system (85%), be prompted with questions that tutee prepared, and the question to ask tutee (81%) on the system directly. However, there are two students don’t like asking for help from instructor through the system. 85 percent of participants also agree that the system reserves all learning profile can help them to look back their own learning process.

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

57

5.2 Interview After the experimental trials, 10 participants were interviewed about how the system helped them preceed the peer tutoring activity during the face-to-face tutoring. All the interviews were tape-recorded. Some selected feedbacks were excerpted in Table 2.

Table 2. Some feedbacks from participants during the interviews 2-1 Conducting expository tutoring

“When I got into the system, I saw the question that my peer posted in advance, so I can rearrange the teaching flow when I tutored and paid more attention in the part about the question when I was tutored.” “When the members had the same confusion with one topic, we can send the question to the instrutor through the system, and then the instructor will come to solve our problem. It’s more convenient than raise my hand because it won’t break off our tutoring. And since the instrutor can see our questions in his/her screen, she/he can prepare or arrange more references to us, we can get more knowledge.” In the view of opposite: “It didn’t influence my teaching/learning attention by the system shows the questions my peer posted in advance. Because I prefer to learn or teach in my own way.” “I don’t like to ask the instructor’s help via the system. I still like to raise my hand and make sure that the instructor would aware that I have questions.”

2-2 Tutoring assessment

2-3Answering Questions and group discussion

3 Instructor-led discussion

“Doing self-assessment and anonymous peer assessment of tutorial notes according to the assessing rubrics through the system is more convenient than in paper and pan.” “The tutoring assessment results returned by the system could help me know my advantage or disadvantage during the teaching/learning process, and that the system reserves all learning profile enabled me to look back my learning process later.” “I felt good when my tutee answered the questions correctly, because that meant my teaching worked. “Answering the questions helped me re-think the knowledge I had just been told, knowing what belongs to whatΞ” “Seeing team members’ answers in a frame where the different answers were in different colors facilitated our discussion about each other’s answers step by step.” “We could always revise our answers during the disscussion, because the revising area was under the show-answer area.” “The instructor can view all reported answers and then adjust her explanations and instructions. Explanations from the instructor were very important, because there were some questions even my peer tutor was not sure about the answers. And I also would like to learn more from the perspectives of the instructor as well as the other small groups.” “Instructor can also present some selected student’s answers, sometimes I am afraid of this. I am willing to share my thought when I have confidence in my answer, while be afraid to talk when I am confused with the question that is discussed.”

6. Summary and Future Work Peers can be good teachers and effectively transmit their learning about the material to peer learners. They just have learned the target knowledge and still have to think through each step of it, so they can teach their peers in a clear step-by-step way. Active learning during the teaching session is another benefit of being taught by peers. The educational context of P3T activity involves tutors’ learning by composing tutorial notes individually and then

58

Y.-W. Chang et al. / Computer-Mediated Face-to-Face Interaction Supporting for Peer Tutoring

collaboratively, and tutees’ learning by pre-class study, and followed by face-to-face elaboration. The technology plays the roles of tutoring scaffolder, collaborative learning supporter, and elaboration facilitator. We design the P3T system to support preparing and performing peer tutoring in 1:1 classroom, and assist the students with three teaching activities: learning about the materials, composing teaching materials, and conducting faceto-face teaching. Our future work is to develope the P3T system completely, especially the pre-class stage. Computer-scaffolded tutorial notes composition is our original goal of P3T learning activity, so we will design more methods and references to help tutor design a great tutorial note in the preclass phase. And we will redesign the P3T learning activity to suit to elementary students. Scaffolding them to prepare the tutorial notes in an interesting way will make them willing to teaching peers, and the peer students would like to be taught by peer teachers.

References [1] [2] [3] [4]

[5] ˮˉ˰ʳ ˮˊ˰ʳ [8] [9] [10]

[11]

Bargh, J.A., Schul, Y. (1980). On the cognitive benefits of teaching, Journal of Educational Psychology, 72(5), 593–604 Greenwood, C.R., Delquardri, J.C., and Hall, V. (1989). Longitudinal effects of classwide peer tutoring, Journal of Educational Psychology, 81, 371–383 Fantuzzo, J. W., King, J. A., & Heller, L. R. (1992). Effects of reciprocal peer tutoring on mathematics and school adjustment: A component analysis. Journal of Educational Psychology, 84, 331-339. Ching, E., Chen, C. T. , Chou, C. Y., & Deng, Y. C. (2005). A pilot study of computer supported learning by constructing instruction notes and peer expository instruction. Proceedings of the 10th conference of Computer Supported Collaborative Learning (CSCL 2005). Ching, E., Chen, C. T., Chou, C. Y., and Deng, Y. C., P3T: A System to Support Preparing and Performing Peer Tutoring. Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005). Cohen, P.A.; Kulik, J.A.; Kulik, C.L.C. (1982). Educational outcomes of tutoring: a meta-analysis of findings. American Educational Research Journal, 19, 237-248ʳ Greenwood, C.R., Delquardri, J.C., and Hall, V. (1989). Longitudinal effects of classwide peer tutoring, Journal of Educational Psychology, 81, 371–383ʳ Schwenk, T.L., Whiman, N. (1984). Residents as teachers. Annis, L.F. (1983). The Processes and Effects of Peer Tutoring. Paper presented at The American Education Research Association Annual Meeting. Chan, T. W. and C.-Y. Chou (1995). Simulating a Learning Companion in Reciprocal Tutoring System. Proceedings of the first conference on Computer-Supported Collaborative Learning, pp. 4956. Chan, T.-W. and C.-Y. Chou (1997). Exploring the Design of Computer Supports for Reciprocal Tutoring. International Journal of Artificial Intelligence in Education, 8, 1-29.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

59

Public Computing, Computer Literacy and Educational Outcome: Children and Computers in Rural India Ritu Dangwal Research Head HiWEL/NIIT ltd, Synergy Building, IIT Campus, Haus Khas New Delhi –17, India Phone No: +91-11-26581017-20 extn:7022 Fax No: +91-11-26581022 [email protected] Abstract. This paper reports the research findings from a national research program conducted in rural India. In this research, children were provided unconditional access to public, outdoor computer. Evaluation was conducted on the children’s ability to learn to operate the computer and the effect of such playground computing on educational outcome. The results suggest that playground computers might have an important role to play in improving the outcomes of elementary education and in imparting critical life skills. Keywords: Evaluation, education, outcomes, computer, rural

Introduction: The Education for All movement took off in 1990 at the World Conference on Education for All. Since then, governments, non-governmental organizations, civil society, bilateral and multilateral donor agencies and the media have taken up the cause of providing basic education for all children, youth and adults. Around 83 countries are on track to achieve Education For All (EFA) by 2015. However, according to the 2002 Report, 28 countries, accounting for over 26 percent of the world’s population, may not achieve any of the three measurable Dakar goals i.e. universal primary education (UPE), gender equality and the halving of illiteracy rates. Two-thirds of these countries are in Sub-Saharan Africa and Asia, which includes India as well. India has the highest number and greatest diversity of grassroots Information and Communication Technology (ICT) initiatives in the developing world. There are a huge number of ICT projects and community centers in rural India for development purposes. And over half of the world’s ICT kiosk initiatives are located in India (Sood, 2003). Not only that but India is ahead of other developing regions when it comes to ICT initiatives in rural areas. One of the most pioneering works in this area is a research experiment on “Minimally Invasive Education” (MIE) or “Hole in the Wall project” initiated by Dr. Sugata Mitra, Chief Scientist, CRCS, NIIT Ltd. (Mitra 2000, 2003; Mitra and Rana, 2001).

60

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

The Hole in the Wall project: The first experiment was conducted in 1999, when one PC was embedded in a wall facing a slum in New Delhi to observe what children would do with it. Mitra (1999; 2000, 2001, 2003) hypothesized that “groups of children when provided appropriate resources will attain computer literacy with minimum intervention”. The three-year nation wide research program in India proved that groups of children can learn to operate computers with no adult intervention. Observations, anecdotes from community members indicated that a lot was happening as a result of the nature of the MIE learning stations. Teachers and Principals commented that children were performing better in schools and they attributed it to the learning stations. An MIE learning station has been designed such that computers are accessible from outside through holes in the wall. The present paper is about Zonal findings in India wherein MIE learning stations have been installed. It is also worth investigating findings at the National level (though not covered in the current paper but analysis is in progress). Research Design: The current design covers 4 Zones: South, North, East and West Zone: South Zone

Karnataka & Tamil Nadu

West Zone

Maharashtra & Rajasthan

North Zone

Jammu, Uttaranchal & Uttar Pradesh

East Zone

West Bengal

There are 17 sites where these MIE learning stations are operational. Out of the 17 sites, some sites have Internet connectivity while others have offline content. Offline content are freeware educational and fun games (science/ english/math/puzzles/quizzes etc). Sample: Sample size of experimental group = 250 and control group = 119 children. Experimental group consist of children in the age range 8-14 years (boys and girls), going to Government schools and are exposed to the MIE learning stations. While, control group consists of children falling in the same age range; belonging to the same socio-economic strata as the experimental group. The only difference being that they are not exposed to MIE learning stations. In this study, the dependent variable is the MIE Learning Station hence, any significant change in the performance of the experimental group can largely be attributed to the impact of MIE learning stations. Tools used a) Computer literacy: Icon Association Inventory (IAI) is used to assess the level of computing skills that children have achieved. b) Intellectual Maturity: The Draw a Man test has been used to assess the child’s intellectual maturity. c) School Academic performance: The school academic performance is measured by the aggregate percentage marks obtained by the experimental and control group children in their school examination. d) HiWEL English and Math test: These are two standardized tests developed in-house for English and Math.

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

61

Results: The results are based on nine-month research period at each zone. And is in the following sequence: Performance in icon test & the learning curve [measure of computer literacy] followed by results on intellectual maturity and academic results, respectively. Graphical representations is only shown for sites where the experimental & control group are at par with each other at the beginning i.e. baseline. Any difference in the post phase can then be attributed to the MIE learning stations. However, there are cases where the experimental & control group differ significantly from the beginning and hence, have not been reported in this paper. I have also not reported those cases wherein, no significant difference is observed in the beginning and on the 9th month (end of the research period) for experimental and control group. Zone-wise comparison between experimental & control group for each test has been studied. Again, only those cases have been touched upon where both experimental & control group were similar in the beginning. I will take this opportunity to state that in cases where they is significant difference between the two groups to start with (baseline), I could have discussed the relative changes, however this requires a different statistical technique and hence is not currently reported. The study is still in progress and would be reported in the forthcoming papers. - Performance in icon test Fig 1.0: Performance in icon test [North Zone]

Fig 2.0: Performance in icon test [East Zone]

62

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

In all the above figures, it is noted that there is a significant difference in the performance of experimental group on first day (i.e. inauguration day) and on 9th month (p= 0.000). Similarly, there is a significant difference between experimental group and control group on 9th month (p=0.000). - Learning Curves (icon test): A logistic model is used to study the behavior of performance over a period of time. They describe: a) carrying capacity of the curve, b) growth rate in that particular site, c) initial knowledge of the focus group children, d) knowledge achieved on the 9th month, e) any scope of further learning (by the children themselves), f) the “goodness of fit” of the model [R2 and MSE]. Fig 5.0: Learning curve -North Zone

Fig 6.0: Learning curve –East Zone Learning curve for East Zone

Learning curve for North Zone 60

50 45

50

Percentage

Percentage

40 35 30 25 20

30

20

15 10

10 5

0

0

0

0

10 0

Actual

200

Percentage D a y s

300

Predicted Percentage

Children started with an initial knowledge of 8.928% (i.e. 3.62% of the icons recognized); after nine months with a growth rate of 1.6% they achieved 98.73% of their potential. In other words, they recognized 43.05% of the icons. After the point of inflation i.e. 43.435% intervention will be required to enhance their icon recognition. R2 is 0.956, which shows a good fit with M.S.E=8.751

Learning curve for South Zone 45 40 35 30 25 20 15 10 5 0 0

10 0

Actual Percentage

Days

200

50

100

1 50 Days

Actual Percentage

200

300

Predicted Percentage

2 50

30 0

Predicted Percentage

Children started with an initial knowledge of 1.47% (i.e.5.75% of the icons recognized); and after nine months with a growth rate of 1.33% they achieved more than 87.22% of their potential. In other words, they recognized 49.87% of the icons. After the point of inflation i.e. 57.176% of icon recognition intervention will be required to enhance their recognition. R2 is 0.986, which shows a good fit with M.S.E = 3.658.

Fig 8.0: Learning curve West Zone

Fig 7.0: Learning curve South Zone

Percentage

40

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

Children started with an initial knowledge of 11.91% (i.e. they recognized 5.90% of icons) with a growth rate of 0.92%, they achieved 75.96% of their potential on the 9th month. In other words, they have recognized 39.86% of icons. Their point of inflation is found to be 50.63%, which means after this point, intervention will be required to enhance their erformance (icon recognition). The fit of the curve is good [R2=0.939] with an M.S.E= 7.1438

63

Children started with an initial knowledge of 24.57% (i.e. they recognized 10.75% of icons) and after nine months with a growth rate of 1.62% they achieved about 99.35% of their potential. In other words, they have recognized 44.67% of icons. After the point of inflation i.e. 44.96 % of icon recognition, intervention will be required to enhance their recognition. R2 is 0.9532, which shows a good fit with M.S.E = 7.119.

All graphical representations (figures) given below - Intellectual Maturity, Academic Performance, HiWEL Math and HiWEL English suggest that to begin with the experimental and control group were similar but after nine-months of exposure to the MIE Learning Stations, the experimental group has performed significantly better than the control group. Intellectual Maturity [IQ] Fig 9.0: IQ between the two groups [North Zone]

Fig 10.0: IQ between the two groups [South Zone]

- Academic performance Fig 11.0: Academic performance [North Zone]

Fig.12.0: Academic performance [South Zone]

64

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

- HiWEL Math Fig13.0:Performance in HiWEL Math [South Zone]

Fig.14.0 Performance in HiWEL Math [West Zone]

- HiWEL English Fig 15.0: Performance in HiWEL English [West Zone]

Interpretation: The above data suggests that there is a lot of zonal variation that is affecting the performance of experimental and control group. Zonal variation refers to the diversity within each site/location in terms of language, dialect, culture, geographic location, socio-economic factors, etc. Let us understand what I mean by the term “geographic” location. There is a site, in the North Zone named “Hawalbagh” where children come to the MIE learning station from as far as 25 km; whereas, there is yet another site in West Zone in Rajasthan, where covering a radius of more than 5 km becomes an alien territory for a child. Another type of geographic location is the proximity of the site to the city or main town, which also can be possibly causing a difference in the performance across zones. It is true that though the experimental and control group are matched but still differences have been observed in their performance prior to the installation of MIE learning stations. These could be attributed to many extraneous variables, which need to be investigated, separately. Interestingly, the findings are not consistent across zones for example on one parameter, there is no difference between experimental & control in the pre -phase, while for another zone, the same parameter shows difference between the two groups. As mentioned earlier, in the current paper, we have not discussed cases where the experimental and control group are not at par with each other when tested for baseline [pre-test]. In such cases, it is observed that the performance of experimental group is lower than the control group. But in the post-phase testing, the experimental group has performed better than the control group, though the difference is not significant, hence has not been reported in the present study. Let us examine each parameter individually: Before I share the findings on computer literacy, I will like to mention that while the experimental group is tested at

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

65

various time points, the control group is tested only on the 9th month. The reason why control group is not given similar treatment is to ensure firstly, no familiarity with the test. And secondly, that the difference in performance of experimental group is due to exposure to MIE learning stations. It can be observed from the graphs (fig 1.0, 2.0, 3.0 & 4.0) that the percentage of icons recognized by experimental group on 1st day and control group on 9th month is similar. In effect, any difference in the performance of experimental group is attributed to the learning stations. One can with confidence arrive at this statement because; over the nine-month research period the environment of these children has been relatively consistent (to begin and end with). Performance on Computer literacy: Children who have been exposed to MIE learning stations have gained computer literacy (as measured by IAI) over the nine months as against children who have had no exposure to MIE learning stations. In other words, MIE learning stations has led to computer literacy. However, computer literacy is not the same across zones for example, south zone recognized 39.86% of the icons, whereas, east zone recognized 49.87% of the icons on the 9th month. Learning curve clearly indicates that greater the exposure to MIE learning stations better is the computer literacy. It also suggests that in each zone there is still further scope of picking up computer literacy though this capacity varies across zones. For example, in east zone, children have acquired 49.87% of icons recognition and there is still further scope of recognizing 50.63% of icons. Whereas, in west zone, the children have already reached 98.35% of their potential (using the logistic model), leaving relatively little scope for further learning. Another interesting feature that can be observed in the learning curve is the Sshape. At all locations, the learning has been with multiple s-shapes. In other words, some learning takes place, followed by a plateau, after which again there is an increase in learning followed by a plateau. The plateau stage could be an indication that the child is crystallizing & assimilating his/her thoughts and then is ready to learn again. Intellectual Maturity: In the North & South zone, experimental & control group matched in their Intellectual maturity in the pre-phase. On the 9th month, the experimental group gained significantly more than the control group. This huge gain in the intellectual maturity of the child can be largely attributed to the MIE learning station, which acts as a catalyst in enhancing the cognitive growth of the child. Marked shifts in intelligence test scores suggest that environmental factors and personality characteristics may have significant effects on performance in the test {Paul H. Mussen, John J. Conger, Jerome, Kagan 1974). In our study, the learning stations provided an enriching environment for the child. Academic performance: It is observed that academic performance has increased significantly in two zones: North zone and South zone. The results are encouraging, as out of 17 sites, in 11 sites, children have benefited academically from the MIE learning stations. Everything else being constant between the two groups, the experimental group has gained from using the learning station. Interestingly, literature suggests that high intelligence may be expected to correlate significantly with educational achievement. The experimental group in the North & South zone has gained significantly in intellectual maturity and also in their academic performance. HiWEL Math & English: In the West zone, children have improved significantly in HiWEL Math & English over the nine-month period. While, in South zone, children

66

R. Dangwal / Public Computing, Computer Literacy and Educational Outcome

have gained significantly in HiWEL Math. One of the possible reasons could be the content that the children were exposed to. For example in West zone, in one of the sites, they had Internet connectivity as well as offline content. Similarly, in South Zone (2 sites) children were provided with Internet connectivity as well as offline content. Conclusion: MIE learning stations have impacted children in more than one ways. Despite the huge regional, linguistic, cultural variations among & across sites, there has been some common observations & findings. Most of the children have picked up computer literacy on their own, increased significantly in intellectual maturity, improved in school performance & in tests developed in-house. There is no doubt that family and school are critical institutions that influence the growing of a child, but it is also a preadolescent to early adolescent age for the child. During this time, family relationships are important, yet at the same time peer group relations take high priority compounded by the nature of the learning station. The design of these learning stations and the methodology of none or minimal intervention from adults creates an environment of flexibility and promotes maximum interaction among friends. Here, the child is an active "maker of meanings" in such a fluid and non-formal learning environment. It is in this context, that MIE learning stations have a huge potential and can be used as an alternative to formal schooling. Acknowledgments: I would like to thank Dr. Sugata Mitra, Manas Chakrabarti, Sujai Kumar and the research team for their support and guidance. References

1. Dangwal, R and Parimala Inamdar (2002): The GUI Icon Association Inventory and MIE – Sindhudurg Friday the 13th Conference, September 2002, page-31 – 42. 2. D’souza, M. and Mitra, S.(2004). A Validation of the GUI Icon Association Inventory. F-13 proceedings, volume XVIII, (NIIT internal document, to be printed) 3. http://www.dmu.ac.uk/~jamesa/learning/constructivism.htm. It talks about constructivist theory. Constructivism is the label given to a set of theories about learning which fall somewhere between cognitive and humanistic views. 4. Inamdar, P. (2004) Computer skills development by children using 'hole in the wall' facilities in rural India. Australasian Journal of Educational Technology, 20(3), pp 337-350. 5. MIE Users Manual (2003), Copyright Hole-in-the-wall Education Limited (HiWEL), New Delhi, India. 6. Mitra, S. (2000) Minimally Invasive Education for mass computer literacy. Proceedings of the CRIDALA 2000 conference, Hong Kong. 7. Mitra, S. and Rana. V. (2001) Children and the Internet: Experiments with minimally invasive education in India. British Journal of Educational Technology, 32(2), pp 221-232. 8. Mitra, S. (2003) Minimally Invasive Education: a progress report on the “hole-in-the-wall” experiments. British Journal of Educational Technology, 34(3), pp 367-371. 9. Paul H. Mussen, John J. Conger and Jerome Kagan (.1974). Child Development and Personality, Fourth Edition. 10. Pramila Phatak (2002) Draw-A-Man Test for Indian Children (Revision and Extension). M. S University, Baroda, India. 11. Sood, A.D. (2003) Information nodes in the rural landscape, i4d magazine, 1 (1), http://i4donline.net/issue/may03/pdfs/aditya.pdf. This article critically examines digital development in order to reveal the larger impact that ICTs could have on rural economics and societies, it goes further to identify information kiosks as the most effective vehicle for digital development. 12. Van Cappelle, F., Evers, V., Mitra, S. (2004) Investigating the effects of unsupervised computer use on educationally disadvantaged children’s knowledge and understanding of computers. Proceedings of CATaC’04, Karlstad, Sweden.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

67

Summative Computer Programming Assessment Using Both Paper and Computer Pari Delir Haghighi, Judy Sheard Faculty of Information Technology, Monash University, Australia Pari.DelirHaghighi, [email protected] Abstract. Selection of an appropriate assessment method has become a crucial issue in computer programming courses as many feel the traditional paper-based examinations are not delivering the desired results for either students or educators. The poor performance of students in written exams and their lack of real-world programming skills have encouraged educators to seek assessment methods other than pen and paper. This paper presents the findings of a study of an introductory computer programming final assessment which applied a combination of paper-based and laboratory exams. Pre-exam and post-exam surveys were conducted to find out students' attitudes towards the new assessment method and to investigate factors which have been shown in other studies to impact on test performance. Relationships between test anxiety and student characteristics of gender, previous computer experience and previous experience with computer-based assessment were investigated and reported. Keywords: introductory programming, laboratory exams, summative assessment

Introduction Assessment is an important component of the tertiary educational process. It provides a means to measure students’ learning of the course materials and determine whether they have the required knowledge and skills to enter the next academic level. Selecting an appropriate method of assessment can be critical to the achievement of educational goals and can affect students' attitudes and learning approaches towards a course of study. Furthermore, an assessment method can be a means to encourage and motivate students and result in a profound learning of a subject [6]. A traditional assessment method used in tertiary education for summative assessment in computer programming courses is a paper-based examination. However, there are indications that written examinations do not always deliver the desired results for either educators or students. Programming is a skill which can best be learned and demonstrated through practice. Concerns about programming skills of students have led educators to question whether written exams are effective in assessing students' programming abilities and in motivating students to practice programming on computer [8, 13]. To provide students with a more valid and effective assessment method, to encourage students to program and to increase students’ proficiency at programming and their learning outcomes, the final examination of an introductory programming course at Monash University trialed a combination of paper-based and laboratory exams for the first time in Semester 2 of 2003. Pre-exam and post-exam surveys were conducted to determine students' attitudes towards the new assessment method and find out any difficulties that they had experienced. A particular focus of the study was an investigation of test anxiety and its relationship to student characteristics of gender, previous computer experience and previous

68

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

experience with computer based assessment. This paper reports on the findings of this study and discusses issues with summative assessment in computer programming courses.

1. Summative Assessment in Computer Programming Courses A common method used in summative assessment in programming courses is the formal paper-based examination. Paper-based examinations are typically invigilated and therefore have the benefit of reducing opportunities for cheating. However, there are concerns that written examinations may not provide a fair testing environment for students to demonstrate their programming skills and their ability to deliver a working program [7]. In a paper-based examination, students may be required to write small programs or pieces of code but that does not give educators an accurate picture of students’ programming skills [3]. Other shortcomings of traditional assessment methods were revealed in a research study conducted by four UK universities. This study showed that formal examinations result in lower marks than other forms of assessment which included essays, lab reports, artworks and presentations [2]. Reasons identified for students achieving higher marks in these assessments included access to resources, less dependency on memory, direct feedback of assessment, and students responding positively to tasks that are more under their control [2]. In a desire to find an effective summative assessment method in computer programming courses, some educators have conducted examinations in computer laboratories. In a computer laboratory (lab) exam students typically use computers to perform small programming tasks. Several studies have shown that this form of assessment provides a more appropriate environment to test students' programming skills than the traditional exam [1, 3]. As English [7] proposes, the use of computers in programming course exams provides students with tools which enable them to demonstrate their practical skills. When students answer exam questions on computer, they are able to execute their code, test the results, correct their errors and finally see the result of the working program. This type of testing enables students to show a depth of knowledge of the subject that is not easy or possible to demonstrate with a paper-based assessment. Supporting this view, Califf and Goodwin [3] suggest that lab exams ensure that students can create a program from scratch, compile it, find any errors, and finally run the program thus giving educators a way of assessing hands-on programming skills that students have learned during the course. Another benefit of a lab exam is the motivation it can provide to students to practice their programming skills on computer during semester. Prior [13] found that if students face the prospect of an online assessment method they will be more inclined to do their studying on computer. It is reasonable to expect that this would lead to increased proficiency and improved learning outcomes. Providing support for this view, the results of studies by Jacobson [8] and Barros, Estevens, Dias, Pais and Soeiro [1] indicated that lab exams helped students to achieve better marks, with the numbers of students who obtained high marks increasing when paper-based exams were replaced with lab exams. These studies show that testing a practical skill on paper presents a danger of restricting students’ programming skills to the boundaries of pen and paper.

2. Test Anxiety An important issue to consider when introducing a new assessment method is test anxiety. Any innovative or new assessment approach can introduce test anxiety among students to some degree. Test performance may be affected by test anxiety and it may also affect test anxiety. Therefore, one of the important considerations in applying computer-based

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

69

assessment is to prepare students for the test and familiarize them with the new assessment method [12]. As Mutch [11] proposes: There might be a need to ensure that students do not meet a particular form of assessment at an advanced stage of their studies that they have not encountered earlier, if such assessment requires a degree of familiarity and practice for successful performance (p. 165).

Studies of computer-based tests have reported a relationship between test anxiety and test performance. Two factors which have been identified as influencing test anxiety are gender and prior experience with computers. Some studies revealed a negative relationship between test anxiety and previous computer experience [4, 5, 15]. However, the same studies showed inconsistent results regarding the relationship between test anxiety and gender. In this study we investigated the relationships between test anxiety and gender, previous computer programming experience and previous experience with the assessment method.

3. A Combination of Paper-Based and Laboratory Exams The reported benefits of laboratory exams provided the impetus to use this method in the final assessment of an introductory programming course at Monash University. The course lecturer considered that testing students in the same environment that they had studied and practiced computer programming would provide a fairer and more appropriate assessment method. However, there were some topics in the course which were not deemed suitable for a lab exam, such as design questions requiring students to draw diagrams, predicting output from a given code or designing a test strategy. Considering these issues, it was decided to employ a combination of paper-based and laboratory exams. The final exam consisted of two components. The first component was a paper-based exam and the second component was a laboratory exam. Questions for each topic to be examined were designed and tailored for either paper or computer. Therefore, students were given the benefit of the most appropriate testing mode for each type of question. They could demonstrate their knowledge of programming concepts, and code reading and understanding on paper, and their practical and problem solving skills on computer. The paper-based exam component consisted of the following types of questions: • Explaining concepts and theory of programming; • Predicting output from a given fragment of code; • Drawing a structure chart for a specified scenario; • Designing tests for a given program. The laboratory exam questions were more practical and appropriate to answer on computer. The questions on this component of exam were as follows: • Writing a small program; • Rewriting a given program by adding pieces of code and making some modification (based on previous assignment work); • Adding two functions to an existing program (based on previous assignment work). The paper-based component of the exam was worth 45% and was one hour in duration; the laboratory exam component was worth 55% and was two hours in duration. Longer time

70

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

was allocated for the computer exam as it was felt that the students would need more time to answer the questions on computer. The two components of the exam were held during one session. The paper-based component was held first. At the completion, students were given a 10 minute break and then proceeded with the laboratory exam. The order of the exams was a deliberate decision. It was felt that the students were likely to find the laboratory exam more stressful than the paper-based exam, and if the laboratory exam was held first, this could affect their performance on the paper-based exam. The literature reports several uses of laboratory exams in computer programming courses; however, there are not adequate cases available to make an evaluation of the merits of this form of assessment. Particularly, there is a scarcity of cases where a combination of paper-based and laboratory exams are used in the final exam. The evaluation study of an introductory programming final examination conducted using paper and computer assessment that is reported in this paper addresses this gap in the research.

4. Methodology The participants in this study were students of an introductory programming course, Semester 2 of 2003, at Monash University. Participants were mostly students in Masters and Graduate Diploma programs within the Faculty of Information Technology. Three data collection methods were used in this study, as follows: • A pre-exam survey consisting of two open-ended questions about students' opinions of the forthcoming exam; • A post-exam survey using a questionnaire consisting of three sections. The first section included demographic questions. The second section mostly contained questions to determine students' attitude towards the exam and their level of test anxiety. The third section contained three open-ended questions, two of which were the same questions used in the pre-exam survey; • Results for the introductory programming final examination. Of the 78 students enrolled in the introductory programming course, 65 responded to the pre-exam survey and 72 completed the post-exam survey. The results of all students were included in the analysis of results. The post-exam questionnaire used Likert scale questions in the second section. Based on the assumption that the Likert scales are not equal intervals, they were considered to be ordinal level measurements and non-parametric tests were used to analyze the data.

5. Results This section presents the key findings of the analysis of the survey data and the exam results. Of the 72 students who responded to the post-exam questionnaire, 75% were males and 25% were females. 61% of the students had previous programming experience. Most of the participants (77.8%) had no previous experience with a laboratory exam. Since both pre-exam and post-exam surveys used the same questions about students’ concerns and suggestions, the results of these two questions have been reported together to compare the students’ thoughts and feelings before and after the final exam.

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

71

5.1 Students' Attitudes towards the Assessment Method The results of the post-exam survey showed that 44.4% of the students preferred a combination of paper-based and laboratory exams, 33.3% preferred only a paper-based exam, and 15.3% preferred only a laboratory exam as the assessment method for the introductory programming final examination. The remaining 7% expressed no preference. More than half of the students (59.7%) agreed that the combination of paper-based and laboratory exam was a fair measure of programming concepts they had learned and only 9.8% of students disagreed. The remaining 30.5% were undecided. A Wilcoxon1 test (N=72) was conducted to examine students' opinions about the difficulty of the exam questions. This showed that students believed answering the laboratory exam questions on computer was harder than answering the written exam questions on paper and the difference was statistically significant (Wilcoxon, Z=-2.996, p<0.05). The mean of negative rank difference for the laboratory exam was 13.45 and the mean of positive rank difference for the paper-based exam was 11.58. The findings showed that the students felt that both the paper-based and laboratory exam questions were about what they understood rather than what they had memorized. Insights into the students' attitudes towards the combination of written and laboratory exams were gained from their qualitative responses in the open-ended questions on the surveys. In the pre-exam survey, there were a couple of typical comments that showed students’ disagreement with a laboratory exam: Innovations are welcomed but not at student cost. I don't see the point since all the computer skills we acquire should be covered in assignments and not exams.

However, in the post-exam the comments were generally favorable. A couple of typical responses were: I think this kind of exam is good and fair because it will show the student's capability in doing programming, not only in the paper test, but also implementing what they have understand in doing real programming in the exam. I think the objective of the course is not only understand the concept, but also doing the programming. Programming means practical work so it was appropriate to have a part of examination on computer.

In the post-exam survey, the most useful aspect of the laboratory exam reported by students was being able to use compiling and debugging facilities.

5.1.1 Concerns about the Assessment Method The time limitation and lack of previous experience with a laboratory exam were major concerns reported by the students before and after the exam. One of the reasons mentioned in regard to time by several students was slow typing. However, the issue of slow typing was reported less often in the post-exam survey. In the post-exam survey, another comment regarding time was that English was the second language for some students. One student 1

Wilcoxon is a non-parametric equivalent of a period t-test

72

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

suggested giving extra time for international students because it took them more time to read and understand the program specifications. The time limitation seemed to cause students anxiety before and during the exam. In the pre-exam survey, one of the students remarked: When we do programming during free hours we don't have to bother about the time. It is tension free because we don't have to submit it immediately.

In the post-exam survey, a similar comment was made: Please, allow more time for the computer-based assessment because during exam we are in stress so when the program doesn't work we get more tensed and the time limit adds to all these tensions. Having adequate pre-exam familiarization was the other important issue reported by students. As one student commented: A new concept or technique should not be test straightway in a final examination.

Some of the other responses for the post-exam survey showed that students were concerned about not having previous experience with the assessment method and about technology based problems. 5.2 Test Anxiety A focus of this study was an investigation of factors which could be related to anxiety. This section describes the results of examining test anxiety and its related factors. The questionnaire asked students how strongly they felt stressed and under pressure during the paper-based and laboratory exams. A Wilcoxon test (N=72) revealed that students experienced higher levels of anxiety in the laboratory exam than the written exam, and the difference was statistically significant (Wilcoxon, Z=-3.810, p<0.05). The mean of negative rank difference for the laboratory exam was 26.06 and the mean of positive rank difference for the paper-based exam was 16.25.

5.2.1 Gender There were no significant differences between the anxiety levels experienced by male and female students in either component of the exam. However, results of Wilcoxon tests to investigate anxiety level for each gender in the paper-based and laboratory exams revealed that male students felt higher levels of anxiety in the laboratory exam than in the written exam and the difference was statistically significant (Wilcoxon, Z=-3.363, p<0.05). The mean of negative rank difference for the laboratory exam was 21.05 and the mean of positive rank difference for the written exam was 13.05. The findings of the same test for females showed no significant difference.

5.2.2 Previous Programming Experience The relationship between test anxiety and previous programming experience was examined and the results of a Wilcoxon test (N=54) showed that students without previous programming experience felt higher level of anxiety in laboratory exam than in written

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

73

exam and the difference was statistically significant (Wilcoxon, Z=-3.970, p<0.05). The mean of negative rank difference for the laboratory exam was 18.25 and the mean of positive rank difference for the paper-based exam was 8.92.

5.2.3 Previous Experience with the Method To examine the relationship between test anxiety and previous experience with the assessment method, a Wilcoxon test (N=56) was undertaken. The results showed that those students without previous laboratory exam experience felt higher level of anxiety during the laboratory exam than the paper-based exam and the difference was statistically significant (Wilcoxon, Z=-3.990, p<0.05). The mean of negative rank difference for the laboratory exam was 20.16 and the mean of positive rank difference for the written exam was 11.64.

5.3 Exam Results The relationship between the students’ results for the paper-based and lab exams was examined. A Pearson’s correlation test showed that there was a significant positive relationship between the results of the paper-based and lab exams (r = 0.664, df = 76, p<0.05). A Mann-Whitney2 U test for the paper-based component of the exam revealed that males achieved higher results than females in the paper-based component of the exam and the difference was statistically significant (U=353.0, p<0.05). The mean rank for the male students marks was 41.41 and for the female students with the lower mean rank of 29.11. The similar Mann-Whitney U test for the lab exam showed that there was no difference between the male and female students’ marks.

6. Discussion Students found the laboratory exam questions harder than written exam questions and experienced higher level of test anxiety during the laboratory exam than the written exam. However, the preferred method of assessment for almost half the students was a combination of paper-based and laboratory exams. The majority of the students felt that this type of assessment was a fair measure of programming concepts they had learned. Furthermore, most of the students reported they would recommend other programming courses to use the combination method. However, the students expressed several concerns about the exam and almost all were about the lab exam component. One of the main concerns related to time constraints. As some of the pre-exam survey comments pointed out, students are used to practicing and completing their programming work at a time of their choosing and without any time limitations. During a laboratory exam, when students are required to write a program or parts of a program and make it work within a specified time limit, they could feel stressed and this stress may affect their test performance. It was observed that some students did not finish the lab exam, despite the fact that twice as much time was allocated for this component. It was therefore not surprising that in the post-exam survey many students suggested that extra time should be allocated for the laboratory exam component. This indicated to us that the time needed for this type of exam should not be underestimated. 2

Mann-Whitney U is non-parametric equivalent of a t-test

74

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

Another issue relating to time constraints reported by several students in the pre-exam survey was keyboarding skills. However, this was only mentioned by one student in the post-exam survey. A study of an electronic exam by Thomas, Price, Paine and Richards [14] investigated slow typing as a factor of this study in loss of time and the outcome of the study revealed that the typing skills could not necessarily be considered as a major hindering factor in an electronic exam for a course where students had a regular practice on computer to do their assignment work. The other main concern related to lack of familiarity with this type of assessment. It usually takes some time for students to become familiar with a new testing environment. In this study; the majority of the students had had no previous experience with a laboratory exam. Although, students reported they had received adequate guidance about the new assessment method, this preparation had only involved a pre-exam practice of logging in, downloading and uploading files from the account allocated on the system. In the post-exam survey, some suggested that the method could have been trialed in the unit test. Providing sufficient pre-exam familiarization is another important issue that needs to be considered when introducing a new assessment method. When two assessments are conducted sequentially, the experience of the first assessment could affect the second assessment. The scheduling of the lab exam after the written exam was a deliberate decision designed to minimize this problem, as it was anticipated that the lab exam would be the more stressful component. One student further suggested that the assessments could be conducted on a separate days to avoid any negative affect. Some of the results regarding test anxiety were consistent with other studies since they revealed a similar negative relationship between test anxiety and previous computer programming experience, and between test anxiety and previous experience with the assessment method. However, this was not the case with gender. The results showed that the male students experienced higher levels of anxiety in the lab exam than written exam but there was no difference in the anxiety level for female students. This finding was interesting because it contradicted other studies which have reported that males experienced lower levels of test anxiety than females when working with computers or other studies that found no significant difference between males and females [5, 11].

7. Conclusion A laboratory exam has the benefit of providing students with a more effective and appropriate assessment method to test programming skills. It assesses students in the same environment they have learned programming and offers a realistic environment where students can use the tools to compile and execute their program. Although our study indicated that the lab exam was the more stressful testing environment for students than the paper-based exam, the most preferred mode of assessment reported by the students was a combination of paper-based and lab exams and most students agreed this type of assessment was a fair measure of programming concepts. However, applying laboratory exams requires careful planning and consideration of exam timing, the selection and design of suitable questions for the exam mode, and the provision of sufficient preparation and familiarization with the method before the exam. It is also important to consider that some students may feel more familiar and confident with the environment of a written exam and the way they answer the questions on paper. For many students, a final exam means a written exam as this is the typical way of assessing in most programming courses. However, in a programming course final exam, students typically need to be assessed on a range of topics. Some of these may be topics

P. Delir Haghighi and J. Sheard / Summative Computer Programming Assessment

75

which are more suited to paper-based assessment. Therefore, replacing a written final exam with only a laboratory exam may not be the best solution. Applying a combination of paper-based and laboratory exams and designing questions according to each mode of the exam can provide students with the benefits of both methods.

References [1] Barros, P., Estevens, L., Dias, R., Pais, R. and Soeiro, E. (2003) Using Lab Exams to Ensure Programming Practice in an Introductory Programming Course. ACM SIGCSE Bulletin. Proc. of the 8th Annual Conference on Innovation and Technology in Computer Science Education, 35(3), 16-20. [2] Bridges, P., Cooper, A., Evanson, P., Haines, C., Jenkins, D., Scurry, D., Woolf, H. and Yorke, M. (2002). Coursework Marks High, Examination Marks Low: discuss. Assessment and Evaluation in Higher Education, 27(1), 35-48. [3] Califf, M. and Goodwin, M. (2002). Testing Skills and Knowledge: Introducing a Lab Exam in CS1. ACM SIGCSE Bulletin. Proceedings of 33rd SIGCSE Technical Symposium on Computer Science Education, 34(1), 217-221. [4] Chua, S.L., Chen, D., and Wong, A.F.L. (1999). Computer Anxiety and Its Correlates: a Meta-Analysis. Computers in Human Behavior, 15(1999), 609-623. [5] Coffin, R.J., and MacIntyre, P.D. (1999). Motivational Influences on Computer-related Affective States. Computers in Human Behavior, 15(1999), 549-569. [6] Daniels, M., Berglund, A., Pears, A. and Fincher, S. (2004). Five Myths of Assessment. Computing Education 2004: The proceedings of the 6th Australasian Computing Education Conference, 30, 57-61. [7] English, J. (2002) Experience with a Computer-Assisted Formal Programming Examination. ACM SIGCSE Bulletin. In Proceedings of the 7th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education. pp. 51-54. [8] Jacobson, N. (2000). Using On-Computer Exams to Ensure Beginning Students’ Programming Competency. ACM SIGCSE Bulletin, 32(4), 53-56. [9] Kuhne, T. (2001). A SmallTalk for Students-A Giant Leap for Studentkind, Part 1. Journal Of Object-Oriented Programming, 14(1), 19-25. [10] Malmi, L., Korhonen, A. and Saikkonen, R. (2002). Experience in Automatic Assessment on Mass Courses and Issues for Designing Virtual Courses. ACM SIGCSE Bulletin. In Proceedings of the 7th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education. pp. 55-59. [11] Mutch, A. (2002). Thinking Strategically about Assessment. Assessment and Evaluation in Higher Education, 27(2), 163-174. [12] Powers, D.E. (2001). Test Anxiety and Test Performance: Comparing Paper-based and Computer-adaptive Versions of the Graduate Record Examinations (GRE) General Test. Journal of Educational Computing Research, 24(3), 249-273. [13] Prior, J.C. (2003). Online Assessment of SQL Query Formulation Skill. Australian Computer Science Communications, 25(5), 247-253. [14] Thomas, P., Price, B., Paine, C. and Richards M. (2002). Remote Electronic Examinations: Students Experiences. British Journal of Educational Technology, 33(5), 537-549. [15] Woodrow, J.E.J. (1991). A Comparison of Four Computer Attitude Scales. Journal of Educational Computing Research, 7(2), 165-187.

76

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Ontology-driven Incremental Annotation of Educational Content for Instruction Planning Apple W P Fok and Horace H S Ip Centre for Innovative Applications of Internet and Multimedia Technologies (AIMtech) Image Computing Group, Department of Computer Science City University of Hong Kong [email protected] Abstract: Amidst the millions of pages available on the Web, it contains an invaluable educational resources pool for planning episodes of learning approach for instructions and guided learning. The un-structured nature and the lack of suitably annotated materials make retrieving the relevant learning contents that meet different pedagogical needs a difficult and time-consuming task. In this paper, we propose a framework as well as the design of a Personalized Instruction planner (PIP) that searches and annotates potentially useful learning content. Our approach is based on an ontology-driven and incremental approach to the annotation of educational content using the multi-agents infrastructure of a Personalized Education System (PES). The PIP which is a subsystem of the PES is composed of three software agents whose roles is to search, retrieve and classify potentially useful learning objects for personalized education guided by multiple ontologies that taxonomize subject domain, instructional design or pedagogy and content.

Keywords: Educational Personalization.

ontology,

educational

content

annotation

and

retrieval,

Introduction There exists a vast amount of potentially useful learning materials on the Web. However, due to the unstructured nature of such hypermedia content and the lack of suitably annotations specific for educational use, web-based learning systems are generally not able to take full advantage of such materials. To address the problem of searching for and then annotating useful hypermedia content from the Web that is deemed useful for educational or learning purpose, we need to deal with three inter-related issues. The first issue is: given the vast quantity of materials that are potential useful for education in the Web, it would be a mammoth task for any particular organization or group of educational professionals to systematically search for and then annotate these materials manually with common educational and instructional vocabularies. Instead, we might rely on a coordinate and distributed effort and allow users of these materials to annotate the materials as and when they found them or are using them, so that their annotations on the used materials can be re-used and benefit others looking for similar materials for their teaching and learning. This leads to the second issue: since users are annotating the materials as and when they are using it, we need to ensure that different users would annotate the materials based on a common understanding of the subject domain and a common vocabulary. In this paper, we propose a strategy and an implementation framework for addressing these issues and tapping the vast resources of potential educational content. Specifically, we propose an incremental and distributed strategy for identifying and annotating potential education content from the Web with the development of the appropriate ontologies for

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content

77

education and exploitation of existing meta-data Standard, such as LOM, for describing education content or object. We further propose an agent-based platform, a Personalized Instruction Planner or PIP that makes use of the above strategies to allow learners to search for and to annotate materials for educational purpose and makes use of the retrieved materials to construct an instruction plan based on the prevailing Standards for structuring and packaging educational content.

1. Personalization in Web-based Education Empirical studies have shown that given the same piece of information, individuals interpret it differently and learn it at different rates through different methods. Therefore, the challenge of effective instruction is delivering the desired instruction based on individual’s background, skills, and experiences that can take learning into learner’s personal world of knowledge and make it their own. As a means of dealing with the problem of organizing and utilizing relevant educational resources from the web, and offers better ways of assembling instructional resources, a conceptualization framework of Personalized Education, or PE, has been proposed [1][2]. The framework can be implemented using agent technologies and filtering techniques to deliver tailored-made education services. Our research aims to demonstrate a possible solution for Personalized Education – each individual learns in one’s own pace, own way, and own needs. In the following, we will first introduce the concept of ontology-driven annotation strategy for educational content, followed by the design and prototyping of a PIP system, a subsystem of the PES, which incorporates several instructional modules to retrieve relevant educational resources for personal development. The conceptual underpinning of the design is based on an ontology-driven incremental annotation of potentially useful education content on the Web.

2. An overview of Ontology-driven and Incremental annotation strategy To address the three inter-related issues alluded to above, we propose a strategy of ontology-driven incremental annotations which allows a piece of content to be annotated incrementally by different users so that the potential values of the initially unknown or un-structured materials can be discovered and packaged over time and over multiple uses. The problem of annotating a vast quantity of materials should not be done in an one-off effort, instead, we expect that the potential value or use of any single piece of content found on the Web could be identified and annotated incrementally, not by a single user, but by separate users who found and use the same piece of content for their (possibly different) learning needs over a period of time. The idea is that when a user found a piece of material useful for his or her learning need, he would annotate it appropriately so that future users with the same needs would be able to found it from the Web through a suitable search engine (e.g. a domain specific search engine). Over a period of time, the potential educational value of a piece of materials would be discovered and enriched as a result of the annotations that have been added incrementally by the different users. To provide a common vocabulary for such annotations, the solution lies in the development of a suitable ontology for learning in different subject domains. By exploiting the semantic web technology, different users would be able to supply their annotations based on a common underlying ontology or conceptual terms for the particular subject domain (i.e. instruction design models). This strategy treats a user or learner as both a contributor and a consumer of potentially useful learning materials available on the Web.

78

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content

They contribute by adding annotations to the materials that they have found useful for their learning needs, and they consume by being able to search for and retrieve suitable learning materials for their learning needs based on the annotations entered by previous learners. Finally, the issue of a common standard for educational content description can be addressed through using an existing standard for describing education content, the Learning Object Metadata (LOM).

3. Learning Object and the current state of development of ontology for education Although there is billions of information scattered across the web, there is no agreed standard for storing information on the web. Despite the varied definition of the concept of learning object, IEEE LOM [3], the IMS [4], AICC [5], and ARIADNE [6], that raised continuing debate, here we only concern about the potential use of those distributed valuable educational resources that can be referred to as educational objects, instructional objects, knowledge objects, intelligent objects or data objects. We briefly review some previous work that contributes towards making the best use of learning objects. The ideas of extending educational metadata schemas (LOM) to describe adaptive learning resources, Conlan et al. [7] added the “ref” attribute as a reference in adaptivity metadata element. The optional ‘ref’ attribute points to a reference document specifying the vocabulary with respect to the specified type of adaptivity. They also use ontology to construct a hierarchical structure of learning contents and several relational properties to guide activity sequence or activity content of learning. This highlights the importance of using the relevant subject domain vocabularies and those derived from some pedagogical theories. Recker and Wiley [8] proposes an instructional design theory for learning object design and sequencing called Learning Object Design and Sequencing Theory (LODAS). However, the lack of educational vocabularies for identifying learning objectives, instructional models or contexts limits the usability of the resources [9]. An ontological representation of learning objects is one way to deal with the interoperability and reusability of learning objects (including metadata) through providing a semantic infrastructure that will explicitly declare the semantics and forms of concepts used in labeling learning objects. On the other hand, to enable software agents to navigate the Web’s information space and accomplish tasks for humans, research in Semantic Web lead to the development of machine-understandable languages that link to ontologies which define the terms or vocabularies for content annotation. Three organizations, in particular, the DAML, OIL, and DAML+OIL have sped head some of the major developments in this area. Recent developments of domain specific ontology include a management infrastructure for business applications, KAON [10]; an ontology-based markup language developed for use on the Web, SHOE [11]; a controlled vocabulary of the Open Biomedical Ontologies, GENE [12]; and an ontology for education, EduOnto [13].

4. Personalized Instruction Planner (PIP): Framework Description To provide a flexible way to facilitate teachers to retrieve relevant learning objects to be deployed as personalized instructional resources and a context for our ontology-driven classification and annotation strategy, we propose here the framework of PIP which composes of THREE essential ingredients. First, an instructional design model which embraces varied instructional design processes. Second, a personalized search agent that is sensitive to the relations between individual user’s interests and the educational hypermedia resources. Third, an ontology-driven classification and annotation model which interacts

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content

79

with the retrieved search results and the users’ instructional design strategies to collect and to accumulate useful educational resources automatically. Personalized Instruction Planner

Searching Tools

Annotated Online Educational Learning Objects

A unit of well-structured instructional Learning Objects

Classification Agent Crawling Agent

Subject Domain Ontology

Instructional Design Ontology

XML RDBS

A deformable independent Learning Object

Personalized Instruction Planner/Tools

Reusable PIP Learning Objects

Personalized Searching Agent

PE Agents (PEAs)

Person and Content Ontology

PEOnto (Education Ontology) Databases

Figure 1: A Conceptual Framework of a Personalized Instruction Planner 4.1 Personalized Instruction Planner Planning is an essential skill that would ensure the quality of teaching and learning. PIP serves teachers to make their teaching more targeted to the needs and characteristics of individual learners by adjusting both the objectives and the methods of learning. To accomplish our goals, PIP records all instruction designs through an iteration of “Plan-Action-Plan” and identifies varied uses of different resources. 4.2 Ontology-Driven Content Search PE search encompasses not only the ability in recognizing, selecting, organizing, evaluating and associating teaching and learning resources, but also the ability in knowing, understanding, concerning and experiencing teaching and learning needs. Through diagnosing individual user patterns, search results can be dynamically filtered, matched and generated accordingly. We achieve this through the development of a Personalized Education Ontology (PEOnto) that guides the PE search through four inter-related educational ontologies, namely, The Curriculum Ontology, the Subject domain Ontology, Instruction Ontology, and the Content (LOM) Ontology. We build agents that commit to ontologies and we design ontologies to describe ontological commitments for the PEAs so that they can communicate well with the defined subject domain vocabulary items and share knowledge with and among each other. To attain the educational goal, it is essential to add educational goals that relate to the main goal of a unit, and to maintain the relation between the contents and situations addressed in the unit. For instance, the Instruction ontology shows classes of instructional-method variables and the major condition variables that influence each other. Instructional Conditions include Subject-Matter Characteristics, Goals, Constraints, Student Characteristics; Instructional Methods include Organizational strategies (Micro/Macro Strategies), Delivery strategies, Management strategies; and Instructional Outcomes such as Effectiveness, Efficiency, appeal of the instruction.

5. System Design of PIP Ontology-Driven Content Search and Annotations The personalized instruction planner is supported by three software agents, they are i) the web-crawling agent; ii) the classification agent; and iii) the search agent (Figure 2).

80

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content Web-crawling Agent

Internet

1 Databases

2

Education Ontology (PEOnto)

Classification Agent

3 Personalized Search Agent

Figure 2: An Ontology-driven agent-based PIP To enhance the distribution, flexibility, and reusability of learning components, the PIP framework is developed upon an agent-based platform, PES[1][2]. First, the Web-crawling agent used the subject domain ontology and the content description (LOM) ontology to retrieve relevant educational resources from the web. Then the Classification agent will based on the curriculum ontology and the instruction ontology to further classify the retrieved educational resources with respect to the education goals, learning objectives, and instruction design principles. Finally, the Personalized Search agent responds to user’s queries and collects the user’s feedbacks (i.e. usage results) with the information provided by the PEAs team. These agents together perform the following functions: − Retrieve relevant multimedia learning resources correspond to different instruction activities (e.g. for presentation, motivation or assessments); − Filter and classify retrieved information based on the PEOnto ontology that includes subject domain ontology, instructional design ontology, person and content ontology. − Construct a personalized search index table to facilitate personalization search; − Accumulate users’ feedback to improve the classification of materials. 5.1 Web-crawling Agent Triggered by a query (i.e. a keyword or a URL) obtained from the Ontology, the Web-crawling agent retrieves the relevant web pages, parses the associated content and extracts the relevant linked information recursively. While traversing web pages, textual information associated with learning material is extracted and used to construct a personalized/intelligent search index table to facilitate subsequent customization and personalization search. However, the web-crawler agent does not actually move around to different computers on the Internet, as worm or intelligent mobile agents do. The web-crawler agent resides in a machine and sends HTTP requests to existing search engines (e.g. Google, Yahoo) to retrieve relevant documents very much like a web browser does when the user clicks on links. The crawler uses the ontology that acts as the semantic search index to facilities searching across web pages. After traversing through chains of linked web pages, ideally, the web-crawling agent would accumulate sufficient relevant information from the Web for the Classification Agent to work on. 5.2 Classification Agent The Classification agent is responsible for auto-classification of retrieved content after the preprocessing stage by the Web-crawling Agent. The Classification agent acts under the guidance of the education ontology and perform classification based on the educational vocabulary items defined in the ontology. Currently many classifications are done based on concepts rather than goals. Learning activity and learning environment related to goals are incorrectly mixed [15]. Goal driven classification can be done through extracting extrinsic

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content

81

and intrinsic goals and systematizing them based on the relations explicitly described in the ontology. Through an application of mining techniques such as content mining, usage mining and event mining, indicative information can be gained. To further enhance the classification accuracy, the WebSifter II [16] methodology can be extended using a weighted semantic taxonomy-based classification technique. Based on a tree-structured educational goals and instruction events representation scheme which describe the relations between goals, sub-goals, and instruction events sequence, the content could be classified as best/good/medium/less in a precision scale. We can also elaborate users’ preference representation scheme to reflect a specific instruction decision-criterion. To rate the relevancy of the resources, a decision-analytic rating mechanism can also be incorporated. In addition to student learning, the Classification Agent incorporates multiple learning strategies to exploit different types of information either in the data instances or in the taxonomic structure of the ontologies. 5.3 Use of Instruction Design Ontology By addressing the fundamental concern of instructional design and incorporating educational psychologies to support the process of personalized learning through the PES, we attempt to incorporate a number of instructional design theory and practice (3-4 instruction design models for different learning approach) to form a dynamic Personalized Instructional (PI) model for an effective personalized instructional delivery. Therefore, we start by considering several instruction design models that teachers normally used in our (conventional or web-based/virtual) classroom such as the Blooms Nine Instruction model. The instruction ontology precisely describes the relations between learning activities, learning objects and the instruction steps/procedures. These ontologies are used to provide suggestions and guidance for teachers in designing and planning instructions and automatically annotate the used learning objects with expressive instructional goals or principles. This information is useful in the sense that the usability of each learning object can be easily identified and re-used. 5.4 Personalized Educational Search Agent The Personalized Educational Search Agent (PESA) is designed to act as a personal teaching/learning assistant for teachers and students. PESA grants users more expressive power in formulating their web searches and improve the relevancy of search results based on the user’s real needs. PESA helps to lessen user burdens in discovering, selecting and organizing relevant educational resources from the Web. PESA learns users’ interests through observing users’ browsing behaviors, and user activities while s/he is interacting with the system (i.e. the PES). The user profile is created and maintained from a content-based and event-based analysis of the used resources using Inductive Logic Programming techniques (e.g. CALVIN). It is able to expand upon the user submitted queries by exploiting the information represented in the individual user profile. Once the expanded query is submitted to and answered by the PESA, it performs a relevance ranking of the results based on the users’ interests. PIP employs semantic taxonomy-based PEAs to achieve THREE important and complementary goals in education: i) identifying different teaching/learning needs in different learning and personal development stage, comparing different teaching strategies with different learning styles, strategies, and preferences to clarify the relationship between them; ii) evaluating whether a specific material would be more suitable for a specific group of students; and iii) discovering and identifying the main characteristics in using varied teaching and learning materials among different teachers. Collaboration works between the

82

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content

users and the PIP can significantly improve the relevancy and the usefulness of the learning objects. The PEAs not only used the ontologies for communication between agents, but also automatically classify and annotate the retrieved and used learning objects with respect to the theory specified by the instruction design ontology and teachers’ professional design. Figure 3 illustrated The PIP Interface design.

Figure 3: The PIP Instruction Design Interface 6. An Implementation of PIP Personalized Searching Tools PE searching tools include domain-specific search and filters or advanced classifiers constitute inductive learning strategy with multi-dimensional resources packages. The educational resources collected and stored in the PE database are not only indexed through a general indexing scheme using keywords or document weight, we exploit the Semantic Web ontology indexing approach to perform descriptive instruction search for a particular resource that matches with a particular learning objective and learning preference. Customization and personalization search work asynchronously toward in-depth accurate information rather than breath/diverge information. A focus, controlled, expressive result set can be delivered rather than just a portal list of hyperlinks. To attain and to maintain the level of reusability and interoperability, both ontology are converted and modified based on the respective metadata standards including the IMS standard for describing a learner, LOM for the learning objects, MPEG7 for the multimedia resources and SCORM for the learning material sequencing. Figure 4 shows the PIP Personalized Education Search Tools and its results.

Figure 4: The PIP Personalized Education Search Tools The general assumption of our work is based on the beliefs that teachers are capable of designing, selecting, organizing and evaluating various educational resources independently for a particular teaching task. We believe the quality of each teaching material or resource is assured under the teachers’ professional judgments. Only because of teacher’s individual differences, resources variations and the problem of match existed. To eliminate this problem, collaboration between teachers is essential. Hence, teacher’s teaching styles and experiences can be shown and shared through the use of the PIP that supports incremental annotation of learning content and generates annotated resource as semantic-driven learning objects that can be reused more flexibly and accurately.

A.W.P. Fok and H.H.S. Ip / Ontology-Driven Incremental Annotation of Educational Content

83

7. Conclusion Using technologies to teach or learn nowadays is inescapable. Teachers’ professional development should not be blocked by their ability to exploit technologies. To insure teaching quality, teachers are facing four main challenges: i) producing digitalized educational resources; ii) incorporating digital learning resources with appropriate pedagogies; iii) modifying, reusing, or improving existing educational resources effectively; and iv) storing, retrieving and sharing educational resources as well as teaching experiences efficiently. PIP provides the opportunity for teachers to experience not only the potential values of using technology, but also the great invaluable experiences of integrating technology with the curriculum, the desire learning for students. In this paper, we introduce and demonstrate a feasible implementation of the Personalized Instruction Planner to facilitate teachers to collaborate in using varied teaching/instruction techniques to design lessons or thematic units. The beauty of using ontology-driven approach in PIP is sets of theoretical vocabulary items are expressively described with closely related educational goals and strategies. Static meta-data of learning objects can be semantically transmits to offer meaningful personalization services in education. PIP allows sharable, reusable, extensible education resources (the PE learning objects), instructional presentation methods, and activities and evaluation projects to be searched and indexed for personalized education. Collaboration can benefit both teachers and students. The proposed PIP can significantly achieving our education goal – personalized learning for whole-person development.

References [1] Fok, Apple W. P. & Ip, Horace H. S., (2004a), Personalized Education (PE) – Technology Integration for Individual Learning, Proceedings of Third IASTED International Conference on Web-Based Education, pp.48-53, Innsbruck, Austria, February 16-18, 2004. [2] Fok, Apple W. P. & Ip, Horace H. S., (2004b), Personalized Education (PE) – An Exploratory Study of Learning Pedagogies in Relation to Personalization Technologies, In W. Liu, Y. Shi and Q. Li (eds.) Advances in Web-Based Learning. Lecture Notes in Computer Science, Vol. 3143, Springer, Berlin (2004) pp.407-415 [3] LOM Learning Object Metadata (http://web.syr.edu/~jqin/LO/LOM_html/index.html) [4] IMS LRM Learning Resource Metadata, Version 1.2, (http://www.imsproject.com/metadata/. ) [5] AICC (http://www.readygo.com/aicc/ ) [6] ARIADNE Foundation of the European Knowledge Pool (http://www.ariadne-eu.org/ ) [7] Conlan, O., Hockemeyer, C., Lefrere, P., Wadde, V., Albert, A., 2001, Extending Educational Metadata Schemas to describe Adaptive Learning Resources, ACM ISBN 1-59113-420-7/01/0008, Available online at https://www.cs.tcd.ie/Owen.Conlan/publications/HT2001v1_Conlan.pdf [8] Recker, M.M., Wiley, D.A., 2000, A non-authoritative educational metadata ontology for filtering and recommending learning objects, Paper submitted for publication to the Journal of Interactive Learning Environments: Special issue on metadata, 25 September 2000. [9] Qin, J., & Hernandez, N., 2004, Ontological Representation of Learning Objects: Building Interoperable Vocabulary and Structures, WWW2004, New York, USA, ACM 1-58113-912-8/04/0005. [10] KAON http://kaon.semanticweb.org/ [11] SHOE http://www.cs.umd.edu/projects/plus/SHOE/ [12] GENE (http://www.geneontology.org) [13] EduOnto (http://web.syr.edu/~jqin/eduonto/) [14] Doan, A.H., Madhavan, J., Domingos, P., & Halevy, A., 2002, Learning to Map between Ontologies on the Semantic Web, WWW2002, May 7-11, Honolulu, Hawaii, USA, ACM 1-58113-449-5/02/0005 [15] Kasai, T., Yamaguchi H., Nagano, K., & Mizoguchi R., 2004, A Support System for Teachers of IT/Information Education based on Semantic Web Technology, In proceeding of ICCE’04, 1539-1547. [16] Kerschberg, L., Kim, W., and Scime, A., 2000, WebSifter II: A Personalizable Meta-Search Agent based on Semantic Weighted Taxonomy Tree, Available online at http://eceb.gmu.edu/pubs/WebSifterII(LasVegas).pdf

84

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

The Development of ClassSim: a Simulated Learning Environment to Support the Practicum Experience for Pre-Service Teachers Brian FERRY*, Lisa KERVIN*, John HEDBERG^, David JONASSEN#, Brian CAMBOURNE*, Jan TURBILL* and Lisa CARRINGTON* * University of Wollongong, Australia ^ Macquarie University, Australia # University of Missouri, USA [email protected]

Abstract. This paper reports on an online simulation designed to support pre-service teachers’ practicum experiences in the important area of literacy teaching. The purpose of the simulation was to develop pre-service teacher understanding of complex classroom situations associated with the teaching of literacy by giving them the opportunity to work within a virtual environment. Our trials of this software have revealed that it has provided pre-service teachers with time to think critically about complex teaching situations which relied on the teacher’s ability to respond to children’s experiences, engage with them in dialogue and negotiation as well as utilise a range of indirect instructions such as questioning, modelling and prompting. Pre-service teachers have acknowledged that their experience with the simulation helped them to make their practicum experience more focused by giving them the knowledge and experience to more fully appreciate the impact of subtle changes that experienced teachers made during lessons. Keywords: Simulation, pre-service teacher education, practicum

Introduction Education systems in Australia face a crucial period as many of the current cohorts of experienced teachers retire and their places are taken by the next generation of teachers. This places increased demands on teacher education institutions to graduate more teachers but at the same time stakeholders rightly demand that graduating teachers are knowledgeable, highly skilled and dedicated professionals [1]. It is crucial that teacher education programs support the entry of pre-service teachers to the profession by exposing them to the realities of the work of a teacher. It has been reported that the retention rate of beginning teachers is poor with statistics such as nearly 50% of teachers entering the profession leave within the first five years [2]. Such statistics presents a challenge to teacher education institutions, systems and schools to examine reasons for this apparent discontent and to find ways to better support the transition from university to schools.

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

85

The practicum is a critical part of initial teacher education but, as has been reported [1, 3], current school-based practical experiences are typically made up of a series of isolated, decontextualised lessons that can result in an unsupported and disillusioning experience. It has been recommended that pre-service teachers engage in an expanded practicum supervised by fully accredited teacher mentors but the cost of the practicum is a significant factor in a teacher education program [1]. In addition, a shortage of practicum places and suitable trained teacher mentors, creates a situation where it is difficult to make major changes to the current system without making major impositions on both schools and teachers. Our own anecdotal evidence has revealed that practicum experiences often do not allow the pre-service teacher to really engage in the problem-solving and decision-making work of a teacher. Teacher education needs to consider how pre-service teachers can experience these roles to better support their entry into the teaching profession. The pedagogical focus of the simulation we have developed is on the teaching of literacy skills in lower primary schools. These skills are considered one of the keys to success in schooling and an important focus area for pre-service teachers. Teachers of children in the early years of primary schooling need to provide appropriate sequences of learning experiences that develop reading and writing skills. Therefore, it is important that beginning teachers develop their own understanding of how children learn to read and write as well as consider the impact of classroom organization and discourse on student learning [4]. The simulation provides opportunities for pre-service teachers to engage with decisions concerning classroom management, organization of teaching and learning experiences and responses to individual students. The purpose of this study was to conduct research into the design and implementation of an online simulation designed to develop pre-service teachers’ pedagogical knowledge and practices about the teaching of literacy to children in the early years of schooling and to provide opportunity for pre-service teachers to assume the role of a virtual teacher. We considered a simulation to be an ideal vehicle to achieve the purpose of this study as it can engage pre-service teachers in an online environment that encourages sustained thinking about the appropriate use of instructional and classroom management strategies during literacy lessons. Simulations as learning environments have a long history of use in education and training [5]. It has been consistently argued that computer-based simulations can be powerful vehicles for learning by applying the critical characteristics of the traditional apprenticeship [6, 7]. As such they apply a situated approach to learning by focusing on the learning knowledge and skills in contexts that reflect the way that the knowledge is used in real life [8]. Jonassen’s research is supported by studies into the complex learning situations presented in computer games and other simulations [6]. Researchers have identified various overlapping learning principles that share four common features [3, 9, 10]. First, they involve socially shared intellectual work that is organised to achieve a task. Second, they contain elements of the traditional apprenticeship process [11] that encourage student observation and comment, make explicit much of the know-how acquired, and permit the participation of the relatively unskilled players. Third, they are organised around strategies necessary for acquiring a particular body of knowledge. Fourth, the process of playing a simulation or video game is focused on the individual, but makes use of a learning group to support decisions and provide reflection. Limited research has been conducted into the use of online simulations in teacher education as it has only been in recent years that ICT has evolved to the point where researchers can begin to take full advantage of its potential. A recent example is the work of Eckersley, Richards and Schofield [12] who report on the development of an online

86

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

simulation designed to introduce pre-service teachers to the broad work culture of primary and secondary schools. A simulation has the potential to provide a scenario that is relevant to a particular field or course of study. Quinn [13 p 103] reports that a scenario can be a meaningful exercise “…if there are goals established for the exploration that require successful application of some knowledge to achieve”. This is further enhanced with the inclusion of examples linked to the practice being explored and opportunities for reflection. The desire to further support the pre-service teacher practicum experience by providing them with opportunity to assume the role of the teacher coupled with the reported affordances of such environments provided a clear rationale for the development of the software reported herein. Once the user logs onto ClassSim, they are presented with a classroom-based scenario. The user is asked to assume the role of a virtual Kindergarten teacher and organize the literacy experiences for these children for a two-hour literacy block. Figure 1 illustrates how this scenario is presented to the user.

Figure 1: Introducing the scenario

Approach The study uses a development approach to the research [14] (also known as ‘design research’ ‘design experiment’ or ‘formative research’) and involved four phases that are characteristic of this approach as depicted Figure 2. Phase 1

Phase 2

Analysis of the practical problem by researchers and practitioners

Development of solutions within a theoretical framework

Phase 3

Evaluation and testing of solutions in practice

Phase 4

Documentation and reflection to produce ’design principles’

Figure 2: Research Design

This paper reports on each of these phases of the research. Phase 1: Analysis of the practical problem The following research questions were posed to gain an understanding of the practical problems that the simulation was to address. 1. What does the research literature report about how to best plan and organize literacy lessons in lower primary school classrooms?

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

87

2. How can pre-service teachers experience this knowledge in ways that encourage them to reflect on their pre-service teacher education experience and identify areas for future professional learning?

An extensive literature review was developed by the research team drawing upon current thinking about what constituted ‘best’ practice for classroom literacy experiences [15, 16, 17]. This provided the researchers with a platform of current research to frame the devised scenario. Further, the researchers drew upon their own classroom experience and data collected from observation of teachers to develop ‘teacher-created scripts’. The research team [18] has previously reported on the development of these scripts. These scripts were shared with a reference group of literacy experts and expert teachers who suggested modifications. Storyboards of these events were then created in PowerPoint and again reviewed by the reference group. These events were later used as an initial framework for the first version of the simulation design. Phase 2: Development of solutions within a theoretical framework The challenge for designers of simulations is to create these environments in ways that make them authentic learning environments. This challenge stimulated us to look for guidance from the literature and one the most promising articles was a review by [19]. Their review of the literature identified nine design elements of situated learning environments and the challenge for us was to operationalize as many of these as we could on an on-line simulation. The design of this simulation is based upon segmenting sets of authentic classroom teaching episodes into interconnected short-term events called cycles. Sets of cycles can link in a variety of ways to form a diversity of possible teaching episodes. The way that each teaching episode unfolds is based upon the decisions that users make about the management of the classroom, of students and of random events that occur during each cycle. The simulation allows the user to take on the role of the teacher of a virtual Kindergarten class (5 to 6 year olds) consisting of twenty-six children. The simulation cycles reflect the problem-solving nature of classroom life. The user is required to make a series of decisions about the management of the classroom, of students and of random events that typically occur during a lesson. At other times they are required to make decisions about the sequence of teaching, for example: do they begin a lesson with a sequencing episode, or a modelled reading episode, or a modelled writing episode, or a retelling of a familiar story episode? Each of these decisions has the potential to impact on subsequent decisions in each of these described areas. As the users make these decisions, the simulation allows access to a branching cycle, representative of a slice of time within the whole teaching period. The cycles available to shape the teaching and learning decisions within the simulation are represented in Figure 3. The user is presented with a number of minilessons, or ‘episodes’, that focus on the teaching of the concept of ‘days of the week’. We believe this to be a common focus within Kindergarten classrooms. These episodes have been divided into focusing on the language modes as the user selects from ‘Reading’, ‘Writing’ or ‘Language Activities’. The initial scenario presented to the users is continued as the time they have to complete this classroom literacy focus is indicated, as is the estimated duration of each of the episodes. Theory about the organization of classroom literacy blocks is available for the user to peruse in the summary material available on the right hand side of the screen. Such support materials have been devised to assist the users in making connections between the theory of their pre-service teacher education and what is happening within this virtual classroom.

88

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

Figure 3: The cycles within the simulation

Three targeted students have been represented in the simulation. These students are based on those students encountered in our own classroom teaching experiences in connection with research data gathered from classroom observations. These children are representative of the most challenging cases pre-service teachers may encounter in their practicums and as beginning teachers. One of these targeted children is Harley. Harley is medicated for Attention Deficit Hyperactivity Disorder (ADHD) and finds many classroom situations difficult. He is frequently not engaged during the running time of the simulation, pending the decisions made by the user. If he is not medicated he tends to distract and annoy other children. Another targeted student Gavin bullies Harley throughout the running time of the simulation. This situation needs to be monitored by the user in their role as virtual teacher. Figure 4 shows how information about Harley is presented in the form of teacher notes to the user. The notes are based on the type of records that teachers typically keep and are available to the user throughout the simulation. Such notes have been designed to add further depth and authenticity to the simulation and to also provide the user with information about these specific children which they may need as they make decisions and respond to individual needs. For example, information is available to the user about medication of students and bullying to support them in providing for Harley in the virtual classroom.

Figure 4: Teacher notes on students

Phases 3: Evaluation and testing of solutions Prototype versions of the simulated classroom have been implemented five different cohorts of pre-service teachers enrolled in their first year the Bachelor of Education program from 2004 to the present. The trial of the software with 187 first year Bachelor of Teaching students in 2005 will be reported on in this paper. The simulation software was

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

89

incorporated as a learning experience within a first year, first session core subject entitled ‘Curriculum and Pedagogy 1’. At the same time, these participants were also studying a core literacy education subject focusing on the teaching of reading. At the end of this session, participants were involved in a three-week practicum experience. These 187 pre-service teachers used the simulation in a laboratory over two 90minute sessions. They were arranged in adjoining pairs, so that they could converse and articulate their understanding and problem solving strategies when using the simulation. A research associate was employed to assist researchers to record observations on a protocol during each 90-minute period. The researchers used stimulated recall and semi-structured interview techniques at the end of each of these sessions to gather feedback from the preservice teachers. A purposive sample was selected from these participants to engage in further interviews with the research team before use of the simulation, after they had used the simulation and once they had completed their practicum experience. The data gathered were analysed using processes of data reduction, data display and conclusion drawing and verification [20]. Constant comparative methods [20] were used to determine issues and themes emerging from the data. This process was continuous and on going and informed future directions for the development of the simulation software. Research Findings Our data suggests that interaction with a classroom-based simulation is a feasible way to support and extend upon the existing classroom-based practicum experience. The analysis of participant reflections collected before using the simulation, after they had used the simulation and after the practicum experience supported this assertion. Participant reflections before using the simulation All of these participants had been exposed to demonstration lessons at local schools. These visits were scheduled 60-minute sessions where teachers demonstrated either a literacy or numeracy focus. Comments from participants revealed that these sessions were often difficult to deconstruct for two reasons. Firstly, due to their limited experience in the classroom, and the intricacies within a typical classroom environment, they often found it difficult to know what to ‘look’ at. Often the students found it hard to analyze these experiences, as they frequently didn’t know what it was they were experiencing in this environment. Secondly, they had little to no control over what was happening in the classroom. There was no opportunity to assist with planning for classroom organization or teaching and learning experiences or with responses to individual children. What they were experiencing, in fact, appeared to be a staged performance. Participant reflections while using the simulation The design of the simulation, which drew upon the conditions for authentic learning environments [19], appeared to support these participants. Many participants identified that the simulation environment was authentic with specific affordances that supported their movement through the scenario. One participant stated, “I feel that it was a realistic classroom setting with realistic students, teachers, parents and issues. It gave me an insight into what a real classroom setting will be like and what sort of decisions I will be making throughout the day”. Likewise, another participant acknowledged the authentic context of the simulated environment, which “…put me in teacher’s shoes to help me prepare and understand what is going to be expected of me as a teacher”.

90

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

Participants consistently identified the safety of the simulation environment as a positive affordance. One participant stated, “It allowed me to experiment with various situations without experiencing any 'real' consequences if decisions I made weren’t the best”. Another participant acknowledged “I would hate to think that we had to practice on real children with real feelings so this was very good!”. Throughout the running time of the simulation participants were encouraged to take risks and make decisions. The navigational design of the simulation was such that participants were ‘forced’ to make decisions at key points. One participant identified this design feature as a positive element of the simulation as it prepared her for classroom reality. She explained, “…because the navigational arrows would ghost until you answered the question made me realise that daily choices needed to be made for the classroom routine to continue”. The participants were given the responsibility to make decisions within the virtual classroom environment. The opportunity to not only make decisions but to also view the consequences of these upon the timing of teaching and learning decisions and with regard to the individual targeted children was a positive characteristic of the simulation many participants identified. One participant stated, “I realized I was in charge. I saw the impact each decision I made had on the class, the need to always be aware of the 'needy' children … and their affect on the class. The updates received helped keep me informed. I basically felt like I really was in charge of the class”. Having this responsibility appeared to exemplify the role of the teacher and the range of considerations a typical teacher needs to make continually – “…being placed in control of a class helped put things into perspective …the variety of lessons and circumstances incorporated into the simulation helped me to understand that different students have different needs … these needs come at various times and stages throughout the course of a lesson. It also opened my eyes to the amount of questions and decisions that a teacher has to make throughout the day”. The simulation design enabled the participants to go back through their decisions and change the decisions they had made to view alternate consequences. The majority of the participants identified this as a positive and supportive feature of the simulated environment. One participant stated, “In my opinion, the best feature of the software is being able to continually go back and try a different pathway with a decision so you can actually see how this affects each child. Then you can determine which decision is the most beneficial for the entire class, not just the mainstream students”. In addition, the simulation was seen by the participants as an additional practical classroom experience as “…we also got a first hand look at lessons and the everyday running of a classroom, the types of things that happen within the classroom and the different situations we are put in as teachers”. Participant reflections after practicum experience The majority of the participants clearly articulated that the simulation supported their practicum experience. One participant explained, ‘…being a first year student coming straight from high school and not having very much experience with children I thought it was really useful, providing insight before being thrown into the classroom’. When discussing their practicum experience, the participants made reference to key terminology and events within the simulation. One participant acknowledged frustration that ‘…not once did I get to experience modelled or guided reading and writing groups’ while on practicum. The inclusion of these teaching cycles within the classroom simulation appeared to prepare the students for their practicum experience and provide them with some indication of key elements within classrooms. As one participant stated, ‘…it gave me a bit of an insight into different things that I will have to deal with as a teacher’.

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

91

Each of these participants acknowledged that the classroom management elements incorporated within the simulation impacted upon their practicum experience. One participant identified similarities between the targeted students in the simulation and students within their practicum classroom. Another identified that they had made use of behaviour management techniques incorporated within the simulation in their practicum classroom. Input from the participants after the practicum also revealed some limitations of the current version of the simulation software. Firstly, the Kindergarten focus did not relate to all practicum experiences. While there appeared to be some transference of key classroom concepts, the need for the simulation to be expanded into other age groupings and curriculum areas became apparent. One participant stated, ‘for the stage that I was teaching in the sim did not help me as much as it would of if I was in a Stage 1 class. I would like to see the sim expanded to cover the different challenges that arise with the students of a Stage 2/3’. Concluding Comments Our data showed that the simulation has the potential to develop pre-service teacher understanding of complex classroom situations associated with the teaching of literacy by giving them the opportunity to slow down or accelerate classroom events, revisit and reflect on critical decision points and replay events in the light of new understandings. Interaction with the simulation software gave pre-service teachers time to think critically about complex teaching situations which relied on the teacher’s ability to identify and respond to the experiences of the targeted students, engage with them in dialogue and negotiation as well as utilise a range of indirect instructions such as questioning, modelling and prompting. Users reported that their experience with the simulation helped them to make their practicum experience more focused by giving them the knowledge and experience to more fully appreciate the impact of subtle changes that experienced teachers made during lessons. At this stage of our research we are confident that the design principles we operationalized, combined with our access to a large pool of authentic data, helped us design a simulation that contributed to the development of pre-service teacher understanding of the complex work of teachers in this virtual classroom but also in real classrooms. References [1] Ramsey, G. (2000). Quality Matters Revitalising teaching: Critical times, critical choices. Report of the Review of Teacher Education in NSW. Sydney: NSW Department of Education & Training. [2] McCormack, A. & Thomas, K. (2002). You’ss be OK!: Induction experiences and reflections of NSW beginning teachers in Physical Education. ACHPER Healthy Lifestyles Journal, 50 (1) [3] Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Education Policy Archives 8.1 Available at http://epaa.asu.edu/epaa/v8ni. Accessed 13 March 2003. [4] Gee, J. P. (2000). New people in new worlds: networks, the new capitalism and schools. In B. Cope & M. Kalantzis (eds.). Multiliteracies: Literacy Learning and the Design of Social Futures, Macmillan, Melbourne, Victoria. [5] Grabinger, R. S. (1996). Rich Environments for Active Learning. In D. H. Jonassen (Ed.) Handbook of Research for Communications and Educational Technology. NewYork: MacMilllan. 665-692. [6] Jonassen, D.H. (1997). Instructional design model for well-structured and ill-structured problem-solving learning outcomes. Educational Technology: Research and Development 45 (1), 65-95. [7] Jonassen, D.H. (2000). Toward a design theory of problem solving. Educational Technology: Research & Development, 48 (4), 63-85.

92

B. Ferry et al. / The Development of ClassSim: A Simulated Learning Environment

[8] Brown, J.S., Collins, A., & Dugduid P.(1989). Situated Cognition and the Culture of Learning. Educational Researcher, 18(10), 32-42. [9] Aldrich, C. (2004). Simulations and the future of learning. San Francisco, Pfeiffer. [10] Prensky, M. (2004). Digital Game-Based Learning. Paragon Press: New York [11] Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. [12] Eckersley, C., Richards, C. & Schofield, N. (2004). Evaluation of a Learning Object. World Conference on Educational Multimedia, Hypermedia and Telecommunications , 2004 (1)2004,718-3720 [13] Quinn, C. N. (2005). Engaging Learning: Designing e-Learning Simulation Games. San Francisco, USA: Pfeiffer [14] Reeves, T.C., Herrington, J., & Oliver, R. (2005). Design research: A socially responsible approach to instructional technology research in higher education. Journal of Computing and Higher Education, 16(2), 97-116. [15] Cambourne, B.L. (2001) What do I do with the rest of the class? The Nature of Teaching-Learning Activities, Language Arts, 79, 2, Nov 2001, pp124-135 [16] Comber, B., Badger, L., Barnett, J., Nixon, H., Prince, S., & Pitt, J. 2001 Socio-economically disadvantaged students and the development of literacies in school: A longitudinal study, Three Volume Report, South Australia Department of Education, Training and Employment and University of Adelaide. [17] Crevola, C. and Hill, P. (1998). ClaSS: Children’s Literacy Success Strategy: An Overview. Melbourne, Australia: Catholic Education Office. [18] Kervin, L., Cambourne, B., Turbill, J., Ferry, B., Hedberg, J., Jonassen, D. & Puglisi, S. (2005). From Classroom Reality to Virtual Classroom: The role of teacher-created scripts in the development of classroom simulation technology. In P. L. Jeffrey (ed). Proceedings of the 2004 Australian Association for Research in Education Conference. Melbourne, Australia: AARE (Australian Association for Research in Education) [19] Herrington, J., Oliver, R., & Reeves, T. (2003). ‘Cognitive realism’ in online authentic learning environments. In D. Lassner & C. McNaught (Eds.), EdMedia World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 2115-2121), Norfolk, VA: AACE [20] Denzin, N.K. & Lincoln, Y.S. (2000). Handbook of Qualitative Research (2nd Edtion). London: Sage publications.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

93

Innovative Learning and Unlearning Di FLEMING, Grace LYNCH lab.3000, RMIT University, Australia [email protected] Abstract. The biggest barrier to innovation and change is not our resistance to learning something new, but rather unlearning what we think is the truth. Learning contexts should be diverse to evoke and capture student imagination. The pedagogy that underpins this process needs to focus on the development of students’ ability to diagnose, simulate, problem solve, negotiate, construct, explore new ideas and exhibit their understanding. Keywords: innovative learning, digital design, design education, technology

1. Introduction The theme of innovation is everywhere these days – in magazines, books, articles, conferences, websites and government direction and policy. To innovate, being innovative, and innovation are all processes that have been integrated into industry thinking, education and government policy. The single most decisive factor in influencing a country’s living standard is economic growth. The engine of this growth, above all else, is innovation – almost regardless of other economic factors. This innovation, along with the knowledge development and management that drive it, are the building blocks of an information society and a knowledge economy. Yet the biggest barrier to innovation and change is not our resistance to learning something new but rather the process of unlearning what we think is the truth. 1.1. Australian Context The importance of innovation and creativity as an educational cornerstone is highlighted in the Curriculum Victoria: Foundations for the Future March 2004 [1] report as one of four key elements of a ‘best practice’ curriculum: x Equity and inclusiveness; x Encouragement of innovation and creativity; x Clarity and focus in content specification; and x Assessment for learning. The Victorian Curriculum Reform 2004 Consultation paper: A Framework of ‘Essential Learning’ [2] found that students need to leave schooling with a broad range of high-level skills which they can apply creatively to the real world. The report goes on to state that the curriculum must recognise that the structure and meaning of work has changed significantly since the 1970s: students are no longer preparing for one career in life, ICT is pervasive and the economy is increasingly based on service or ‘knowledge’ industries. Social and economic progress in Australia, not only in Victoria, will increasingly depend on active people with the capacity to solve problems, to create and to generate new

94

D. Fleming and G. Lynch / Innovative Learning and Unlearning

and better ways of doing things. At all levels, our society will require creative individuals able to communicate well, think originally and critically, adapt to change, work cooperatively, and remain motivated when faced with difficult situations [3]. 1.2. Definitions Design can be seen as a vital step in transforming ideas into creative, practical and commercial realities. Design optimises the value of products and systems and is therefore an important key to economic, social and cultural development. Other very narrow definitions of design from the Concise Oxford Dictionary include: a mental plan; a scheme of attack; an end in view; an adaptation of means to ends; a preliminary sketch for picture; and an invention. Design also involves planning and organising production, and evaluating products in a real context. Design does not necessarily require a product that can be seen or touched, design can be conceptual and/or practical and it can be real and virtual. “Design is both a noun and a verb, and has multiple senses. As a noun, it denotes a field as a whole, an end product or a thing – a designed object or entity – as well as a concept or proposal. As a verb, design denotes an action or process” [4]. Digital design is the intersection and integration of digital technology with the design process and across the many disciplines of design, including architecture, engineering, new media, industrial and product design. To be innovative is to do something different, to explore new territory or to take a risk [5]. The critical element of being innovative is newness and thus, creativity is frequently linked with innovation. Creativity has been defined as “the application of knowledge and skills in new ways to achieve a valued goal” [6]. Innovation can often be seen as an outcome of the broad exploration of ideas, materials, and technical processes that can occur during the design process. Another way of describing the linkage between creativity, design and innovation is that: x Creativity = ideas x Design = giving shape and form to ideas x Innovation = placing the design (product/process/system) into a context 2. Approach Traditional classrooms continue to provide the most economic and systematic way of educating young people. However, in order to encourage young people to generate new ideas, learning contexts should be diverse, so as to evoke and capture student imagination. Schools must create entrepreneurial attitudes to learning so that there are spaces created in which students can incubate their ideas and share their results beyond the walls of the classroom. These innovative ways are not necessarily in a physical location; at times, the space to innovate will be found within a changing mindset. The first element required in innovation is the desire to do something differently. There needs to be a reason to change the current way of working, or a challenge that needs to be addressed. Once it has been accepted that there is a challenge then a creative approach to the solution is the next element in practicing innovation – if a solution is not creative then there is no innovation [7]. The point of difference is getting educators to change their approach to teaching and learning, in essence unlearning from a solid foundation. This foundation has been built upon: how they were taught as a student; how they were taught to teach; and the way they

D. Fleming and G. Lynch / Innovative Learning and Unlearning

95

have been teaching. As Kaipa and Johnson explain when you join a new company you learn to do things the way they do, when you move to a new town, you pay attention to local people and unconsciously adopt their ways of saying, doing and being. We automatically get conditioned to the environment where we live and work. The same happens with teachers and teaching. We teach as we have been taught. “But just as we cannot plant new crops without first uprooting the old ones and giving the new seeds a chance, we need to unlearn before we can learn anew” [8]. Innovative learning is about unlearning old patterns and developing new patterns. It is about escaping the box. Unlearning then, is generating new ideas or concepts rather than just reformulating the old. Creativity and innovation occur during the process of unlearning. It is not the mere modification or restructuring of old material but rather the development of new possibilities and innovative approaches to situations that previously seemed stale or difficult [9]. In a society where the speed and capacity of change is so great it is now time to move away from a content-dominated curriculum. It is essential that students respond to big ideas and big questions. Through hypotheses, students are challenged in learning contexts that create a body of knowledge through which learning can be constructed and exhibited using contemporary learning tools so vital for access, acquisition and application of knowledge. Design based learning focuses on the learner’s discovery through self-awareness, motivation, relevance and action. Thinking about design is rarely implemented in schools and yet design literacy should underpin all ages and stages of learning, be it the study of poetry in grade five, mathematics in grade twelve, art studies at TAFE or engineering at university. Design literacy is about creativity and an ability to integrate ideas: for example, how learning relates to teaching, music relates to mathematics, art relates to science and nanotechnology relates to architecture. What learning is, how learning actually takes place, and the characteristics of learners should be major considerations in the design and development of teaching and learning environments [10]. We all hold views, however unconsciously they may be, of how learners learn, and this affects how we design and develop courses and programs. The techniques we use to increase learning are not as important as our attitudes and beliefs towards learning [11]. How the student learns is the focus of innovative teaching and learning. Innovative learning leads to students as learners, communicators, creators, designers, problem solvers and researchers. These roles are not mutually exclusive but are meant to overlap and interconnect. Developing a sense of a learning community is an integral component of ensuring that we meet the educational needs of students. A learning community values learning; enhances understanding of being an educator and a learner; focuses on action; and encourages collaboration, cooperation and collegiality. 2.1. Lab.3000 Lab.3000 – innovation in digital design is building, connecting and promoting the value of design in the State of Victoria in Australia through the integration and collaboration of design practitioners, industries, design professionals, ICT specialists, and tertiary institutions to foster innovation and enterprise. Funded by the State Government of Victoria and hosted at RMIT University in Melbourne, lab.3000 aims to position Victoria as a world leading centre for digital design and a hub for the intellectual and creative talents that underpin the growth of an innovation economy. Lab.3000’s core business is conducted via three entities: the Business Bureau, the Tertiary Bureau and the Education Incubator. These entities have enabled lab.3000 to

96

D. Fleming and G. Lynch / Innovative Learning and Unlearning

develop a more dynamic relationship between design professionals, manufacturing and new media industries and the strategic management of student innovation across the tertiary sector. The lab.3000 Business Bureau and Tertiary Bureau have been influenced by a number of international tertiary sector models that successfully promote and enable industry development and technology transfer: the Arabus Model Helsinki and NovaUCD Model (an industry centre at University College, Dublin). Lab.3000 has developed the Incubator in response to the Victorian Government’s commitment to its Innovation and Design agendas and to articulate the goals, vision, objectives and targets of the Blueprint for Schools report through innovative design thinking and practice. This paper focuses on the lab.3000 Incubator. Victoria’s first education incubator is a place where ideas can be tested, piloted, refined and brought to life. 2.2. Lab.3000 Incubator The Incubator is an open plan facility with different spaces for the students to use during their time. There are modern computer pods, quiet spaces, meeting areas, conversation areas for food and drink, small group work areas, and presentation spaces. There are 15 high end PCs all with zip drives, CD/DVD burners and the latest software. In addition 3 MACs, a scanner, and digital cameras are available. Students are encouraged to explore their creative potential and engage in the many dynamic possibilities that are offered by a career in digital design. There are 3.4 full time teachers including a teaching manager. A few examples of student projects are the design of new footwear, a new line and style of jeans, digital graffiti, and new computer games. Today, one of the most powerful aspects of working in a digital environment is the collaboration that comes naturally when students and staff are sitting side by side sharing something that has been written or drawn, programmed or accessed on-line. Today’s collaborator and team member could easily have been yesterday’s cheat. It is the process that the students’ have engaged in as much as the outcomes that are important. The relationship between learning and evaluation is reconsidered in a constructivist view. Constructivists prefer to focus on outcomes that naturally emerge from the learning experience. Therefore, it is more important to understand the journey rather than the destination [12]. The move towards digital work has only increased the differences required of assessment strategies for student work. This is applicable across all levels of education from elementary school through to post graduate work such as online doctorate theses. Institutions and organisations are continuing to struggle with this assessment issue. Learning contexts should be diverse to evoke and capture student imagination. The pedagogy that underpins this process needs to focus on the development of students’ ability to diagnose, simulate, problem solve, negotiate, construct, work in teams as well as assuming leadership roles, to explore new ideas and exhibit their understanding. A definition of a learning environment is “a space where the resources, time and reasons are available to a group of people to nurture, support, and value their learning of a limited set of information and ideas. Learning environments are social places even when only one person can be found there.”[13]. Learning is full of what Kaipa and Johnson [14] term “aha” moments. These are moments when a breakthrough in learning occurs, a transformation occurs and the new learning takes over the old. There are no real limits to learning, the limits we experience are generally there because we have placed barriers in the way. Without barriers, learning is deeper and easier.

D. Fleming and G. Lynch / Innovative Learning and Unlearning

97

In many schools, educators and government leaders are so intent on content and information transfer, they lose sight of the importance of the learning context. Innovation and creativity demands that we create time, space, and exploration of intersecting ideas and wide-ranging influences. The opportunities for learning that allow intersection and interaction are diminishing and “fact stuffing” wins out. For people to unlearn they have to discover that they should no longer rely on their current beliefs and methods, because current beliefs and methods shape perceptions and can blind people to potential interpretations [15]. The lab.3000 Incubator environment is a place for learning that defies the traditional instruction, content-based approach to teaching and learning. The use of the term lab is deliberate since a lab is a place where experimentation occurs. People who see themselves as experimenting are more willing to try something new and deviate from past practice [16]. When we deviate from the norm we create opportunities that may surprise ourselves. This is at the heart of innovative learning. Transformational learning is “the process of continually transforming perceptions through reflection…challenging existing frameworks, or at least suspending them, becomes a necessary part of the learning process if new meaning is to be made. Therefore it is necessary to unlearn in order to learn, that is, unlearning in the sense that you have to critically examine your constructs and be prepared to set them aside and look at things afresh” [17]. The following physical and attitudinal elements are essential to ensure students can experience and develop their creative and innovative capabilities: x x x x x x x x x x x x x x x x

Create space where construction can be real and virtual Employ teachers who are risk takers and facilitators Create possibilities that are free from internal and external coercion Use digital learning tools including design based software (Micro Worlds, Web construction, CAD, animation tools and peripherals) Access industry and higher education Visit diverse design/creative environments Explore unique combinations Encourage students to explore unfamiliar mediums and integrate these Permit experiments over and over Open to other perspectives and points of view Cultivate vitality and momentum Use multiple approaches Represent ideas graphically Plan strategically Build teams Design assessment pedagogy

As Reiber points out a “person’s interest in school learning rarely compares to the commitment that characterizes their learning outside of school” [18]. A learning environment can be created that enables people to devote effort and emotion to a task. The Incubator has created such a place. “Play is a suitable and respectable way to describe intense and meaningful adult learning” [19]. 2.3. Educational Programs When new programs and opportunities for learning become available, learning how, when and where to use them often becomes a process of trial and error. Digital design literacy is

98

D. Fleming and G. Lynch / Innovative Learning and Unlearning

critical for teachers as well as their students. There needs to be an avenue whereby teachers, industry professionals, and community groups can access resources and tools that can assist learners in developing design capabilities. Professional programs are developed in consultation with teachers, industry and community groups in response to the unique and changing curriculum needs of schools and the constantly evolving nature of design. Participants become learners, working with highly experienced and skilled educators and mentors, delivered through the lab.3000 Professional Development program. The Visiting Schools Program is designed to meet the needs of individual schools or school clusters across the State, and is conducted for small groups of students and teachers. Each program is unique whereby digital design techniques are integrated into the chosen theme or approach. For instance, one project has developed around water and conservation, since the State of Victoria has been experiencing a severe drought and all areas are on different levels of water restrictions. There are currently over 80 students from a cluster of schools participating in this program. The Once Upon a Time – Digital Storytelling program is a highly interactive educational program that excites the imagination and lends a strong personal voice to individuals. Once Upon a Time provides the tools and methodology through which students can assume the role of narrator to offer a glimpse or insight into their own personal world. The flexible and dynamic nature of digital storytelling, which encapsulates aural, visual and sensory elements, utilises the multitude of cognitive processes that underpin learning - from Verbal Linguistic to Spatial, Musical, Interpersonal, Intrapersonal, Naturalist and Bodily-Kinesthetic. Summer and Winter Schools are available to students of all ages who are eager to explore digital technology in an immersive and creative playing field for learning. Students use digital technologies to enhance their understanding of digital design through programs exercising analytical and simulation skills. Teachers in training and experienced educators can also benefit from the interactive group learning activities. 2.4. Outcomes At the heart of the Incubator programs is creative expression and appreciation. By inviting a range of educators, artists, architects, animators, and curators into the Incubator, many students who have become disaffected with traditional and predictable pathways have a chance to learn, and grow and in time mentor others. Visitors and guests to the Incubator over the past two years have included an animator from the movie series Lord of the Rings, an architect, and small business owners in web design, photography, graphic artist, games development, and multimedia. When these visitors come they work directly with the students on their projects and have made themselves available for ongoing electronic communication with the students. As Richard Florida states, “creativity is a broad social process and requires teamwork. It’s stimulated by human exchange and networks; it takes place in real communities and places” [20]. The Incubator is the place where students go to communicate, collaborate, be excited and create. It is a place that is safe, where divergent thinking and continuous try-outs are the way. By bringing students together in a context where subjects do not bind them, they are able to wrestle with ideas across cultures, music, art, mathematics, science, technology and so on where a vast array of concepts could and do emerge. The common link for the students in the Incubator is that they are learning and applying their learning on their own projects using a variety of digital tools and concepts.

D. Fleming and G. Lynch / Innovative Learning and Unlearning

99

Educational environments today have flexible tools with the potential for everyone – students and teachers to learn alongside each other. Multimedia or multiplemedia has become the learning currency. Students can draw upon a wide range of digital resources, and interpret and exhibit via their own website, a place where their current project is being constructed. Within a digital school environment, the only limit to the learning is the imagination, capacity, motivation and access to contemporary technologies. Unlocking the potential of learners as designers is premised on the shifting role of the teacher from instructor to facilitator. As a facilitator, the teacher can help unlock student design by using learning technologies and questioning strategies that explore different ways of looking at a problem, and by encouraging children to explore the unfamiliar. Such a curriculum is based on productive thinking where problems are explored by asking, “How many different ways can I look at this? How can I rethink the way I see this and what designs can I come up with?” Productive thinking can lead to various designs via multiple approaches. Teachers can encourage children to design by giving them the freedom to think, argue, make mistakes, go to different places and find ways of finding solutions rather than grasping the first seemingly correct answer. It must be acknowledged that this transformational learning process may cause discomfort for some students and teachers as it can lead to a confrontation of fundamental beliefs of teaching and learning. The challenge is to continue to deliver a high quality and creative learning experience for all [21]. Students who have participated in the educational programs at the lab.3000 Incubator have found their experiences to be rewarding, particularly for the freedom that it allowed them in bringing their creative talents to the program. Since its inception over 645 students, 56 teachers and 126 community members have participated in programs. The different learning environment in the Incubator and its subsequent impact on learning styles has made significant changes for the students. Some examples of students’ feedback are: x x x x x x x x x x

I am able to think more creatively I’ve had to make a few big changes to my way of thinking A program like this is so different from what I’m used to Since I started coming here, I have found that I haven’t been in as much trouble as I was in at school I’ve concentrated a lot more compared to when I was at conventional school Just more confidence It has made me look at the digital design industries totally different I have a better understanding of digital design The program has made me a different person I have benefited much more from this program and all my friends think so too

The State of Victoria in Australia has earned an excellent reputation for its initiatives in education both nationally and overseas. However, it is clear that the prevailing paradigm for learning is defined by pedagogies and regulations that do not challenge students. It is time to change the face of learning. It is not learning to learn, but rather learning to unlearn. Digital schooling is at the centre of the revolution of learning; however, it is not just about computing technology. Digital schooling requires a commitment to preparing our children for their futures and ensuring that schools are places where children love to go. The Incubator is such an environment.

100

D. Fleming and G. Lynch / Innovative Learning and Unlearning

3. Conclusion The lab.3000 Incubator was established to provide both a real and virtual space for young people to collaborate. It is a dynamic place where some students come and go and others stay. The Incubator is a space where individuals can express their creativity, explore new ways of thinking and find new ideas through the collision of the unfamiliar. If innovation means to bring into relationship, that which has not been previously related, then the Incubator provides an unique place where new approaches to teaching and learning are being experienced and shared. For individuals to explore new possibilities and opportunities, sometimes they need to be in a ‘space’ where there is permission to be different, to experiment by bringing together ideas that do not normally relate. References [1] Curriculum Victoria: Foundations for the Future. Victorian Curriculum Assessment Authority, Department of Education, State Government of Victoria, Melbourne Australia, March 2004. [2] Victorian Curriculum Reform Consultation Paper: Framework of Essential Learning. Victorian Curriculum Assessment Authority, Department of Education, State Government of Victoria, Melbourne Australia, March 2004. [3] Commonwealth of Australia, “Australia’s Teachers: Australia’s Future, Advancing Innovation, Science, Technology and Mathematics. An Agenda for Action”, Department of Education, Science and Training, Canberra, Australia, 2003. [4] Barnacle, R., “Mapping Design Research @ RMIT”, Prepared for Pro Vice Chancellor (Research & Innovation), October, RMIT University, Melbourne, Australia, 2003. [5] Lynch, G., “Towards Innovation”, Executive Excellence, 18 (8), August, Australia, 2001. [6] Commonwealth of Australia, “Australia’s Teachers: Australia’s Future, Advancing Innovation, Science, Technology and Mathematics. An Agenda for Action”, Department of Education, Science and Training, Canberra, Australia, 2003. [7] Lynch, G., “Towards Innovation”, Executive Excellence, 18 (8), August, Australia, 2001. [8] Kaipa, P. & Johnson, S. Igniting your Natural Genius. Mithya Institute for Learning, 1998. http://www.mithya.com/learning/igniting03.html accessed 12 April 2005. [9] Ibid. [10] Longstreet, W. & Shane, H., “Curriculum for a new millennium”, Allyn and Bacon, Needham Heights, MA, 1993. [11] Kaipa, P. & Johnson, S. Igniting your Natural Genius. Mithya Institute for Learning, 1998. [12] Rieber, L., “Using computer-based microworlds with children with pervasive development disorders: An informal case study”, Journal of Educational Multimedia and Hypermedia, 4(1), 1995, 75-94. [13] Rieber, L., “Designing learning environments that excite serious play”, Annual meeting of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE), Melbourne, Australia, 2001. [14] Kaipa, P. & Johnson, S. Igniting your Natural Genius. Mithya Institute for Learning, 1998. http://www.mithya.com/learning/igniting01.html accessed 12.04.2005. [15] Starbuck, W. Unlearning Ineffective or Obsolete Technologies. International Journal of Technology Management, 1996, 11, pp 725-737. http://pages.stern.nyu.edu/~wstarbuc/unlearn.html accessed 12.04.2005. [16] Ibid. [17] Martin, P. Unlearning and learning. In A. Herrmann and M Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum 7-9 February 2001. Perth [18] Rieber, L., “Designing learning environments that excite serious play”, Annual meeting of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE), Melbourne, 2001. [19] Ibid. [20] Florida, R. & Tinagli, I., “Europe in the creative age”, Alfred Sloan Foundation, DEMOS, 2004, p. 11. [21] Martin, P. Unlearning and learning. In A. Herrmann and M Kulski (Eds), Expanding Horizons in Teaching and Learning. Proceedings of the 10th Annual Teaching Learning Forum 7-9 February 2001. Perth

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

101

Implementation and Effectiveness of a Feedback Feature in Underlining to Promote Reading Comprehension of e-Learning Instructional Materials Yoshihiro FUKUNAGA1, Tsukasa HIRASHIMA2, Akira TAKEUCHI3 Graduate School of Computer Science and Systems Engineering, Kyushu Institute of Technology University 680-4 Ooazakawazu, Iizuka-shi, Fukuoka, 820-8502 JAPAN 2 Graduate School of Engineering, Hiroshima University, Japan 3 Dept. of Artificial Intelligence, Kyushu Institute of Technology University, Japan [email protected] or tel +81-928-51-2504

1

Abstract. In this research, a system was developed to enable underlining by learners to promote reading comprehension of e-learning instructional materials, and an attempt was made to further promote such underlining activity by evaluating and providing feedback on the learner's underlining. E-learning instructional materials implementing this feature were created and used as auxiliary instructional materials in actual classes, and a group of learners using instructional materials with underlining capability only were compared against a group of learners using instructional materials which also provided the feedback feature. The following results were obtained for the latter group: (1) The usage rate of the instructional materials was higher, (2) Underlining activities thought to be desirable for reading comprehension were more frequently seen, and (3) The rate of correct answers on a test for confirming reading comprehension was higher. These results suggest that the feedback feature for underlining which was implemented in this research is effective for promoting reading comprehension.

1. Introduction The rapid dissemination of the Internet sparked the current boom in Internet-based learning (e-learning) in a variety of areas (ALIC, 2003; 2004). There are various forms of e-learning, but at present, most e-learning for acquiring technical skills and qualifications (conducted mainly in companies and schools) is designed based primarily on a stage where the learner reads and comprehends instructional materials. This form of e-learning has high utility; it enables simplification of instructional material distribution, learning suited to the pace of each learner, and understanding of the overall progress situation. At the same time, however, there are problems because this form of learning does not greatly differ from conventional paper-based self-study in the core reading comprehension stage. The purpose of this research was to make better use of the interactive potential of e-learning in this reading comprehension stage. One activity which is often used to aid reading comprehension is "underlining" of key parts of the text. In previous research on underlining in paper-based reading comprehension, it has been assumed that simply underlining parts of instructional materials is effective for promoting reading comprehension (Uosaki et al., 2003). Therefore, if a system is designed so that learners can perform underlining to improve reading comprehension of e-learning instructional materials, it has the potential to be effective in promoting reading comprehension. However, if learners are simply asked to do underlining, we can't really say the system is making use of interactivity. So, in this research, we focused not only on enabling learners to perform underlining in reading comprehension materials, but also implemented a system for evaluating the finished underlining, and enabling the learners themselves to check the appropriateness of the underlining they have done by providing feedback on the evaluation results to the learners (Fukunaga et al., 2004). The kind of feedback motivates learners to do underlining more actively and appropriately, and we can expect this to yield a

102

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature

qualitative improvement in reading comprehension. The following sections describe this research. As background for this research, Section 2 discusses examples of previous research on the effects of instructional material underlining on reading comprehension. Section 3 explains the evaluation of underlining, and feedback based on that evaluation. Section 4 reports on comparative experiments regarding learning effectiveness, with and without feedback on underlining, conducted through actual classes. Finally, Section 5 provides a summary, and discusses the conclusions reached through this research. 2. Previous research on the effectiveness of underlining A variety of research results have previously been reported regarding the forms and effectiveness of instructional material underlining for reading comprehension. It has been reported that underlining, as a learning scheme, can be understood as a coding function or external memory function for promoting cognitive processing of the instructional content (Di vesta & Gray, 1973). When underlining is regarded as a coding function, it is thought to be a process of search/selection where the reader searches out information in the instructional materials which he or she believes to be important (Glynn, 1978). Marshall(1997) surveyed examples of the types of marks made in actual university textbooks, and pointed out that techniques like underlining, marker highlighting and circling were used for "marking", and indications like asterisks, stars and arrows were used as "landmarks" for sentences or words in the instructional materials, and that "memos" are also used. It is reported that memos are frequently used to record interpretations of the meaning of sentences or answers to problems, and markings and landmarks are used to indicate important concepts in the instructional materials. On the other hand, if underlining is regarded as an external memory device, the underlined text is effective only when the text is reviewed, and if the underlined part of the text is not reviewed, it is not effective as a memory device for understanding or reproducing the text content (Blanchard & Mikkelson, 1987). Seki (1997) clarified the point that emphasizing important concepts (which are the goals of instruction) beforehand in self-study instructional materials using prompts like arrows, underlining or bold characters promotes understanding of the indicated parts. Although prompts such as underlining provided at important parts of the instruction materials clearly improve understanding of content, it has been reported that underlining at non-target parts of the text hinders the understanding of content (Johnson & Wen, 1976). Also, when a learner actually reads material such as a textbook, they may do underlining in the material to improve their own understanding. This behavior is thought to be effective for improving the learner's own understanding of the textbook content. However, the learner himself or herself must pay attention to and select out important concepts in order to do underlining in the textbook in a self-motivated way. In other words, selective attention must be paid to the instructional materials in the recording activity of doing underlining while judging which parts are important concepts, and which are not, and thus this method is thought to be more effective for understanding content of instructional materials than the approach of emphasizing key concepts in the textbook beforehand as prompts and giving the information passively (Uosaki et al., 2003). 3. Feedback feature for underlining Making marks, such as underlining, at important parts of instructional materials, is a behavior which learners frequently use, and underlining by the teacher of parts thought to be important is not thought to involve a heavy burden as one aspect of instructional material creation. In this research, a system is implemented wherein underlining in instructional materials done by the teacher is taken to be the key (correct answer) data, and feedback is provided to learners by evaluating their underlining based on that key data. The following section describes evaluation of learner underlining using teacher underlining as key data, and feedback of that evaluation information to the learner. After that, the section reports on an investigation of the appropriateness of using teacher underlining as key data. 3.1 Evaluation of underlining The configuration of this system is shown in Figure 1. First, the teacher performs underlining of the

103

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature

presented instructional material using a computer, and the system saves that as key data. Then the system evaluates learner underlining based on that key data, and provides feedback to the learner regarding the evaluation results. The underline data from the teacher and the underline data from the student are acquired using the same mechanism.

Learner recorded information Instructional material

Learning trial Learning trial

Line Cancel

Learning time Learning time

Learning date Learning date

SAVE CANCEL

History End of learning

: Comprehension index : Learner underline data

Figure 1. Configuration of system for providing feedback on the comprehension index

Figure 2. Learner interface for reading comprehension

The learner interface for reading comprehension is as shown in Figure 2. As shown in the figure, underlining is done at parts of the instructional materials which are thought to be important, and when the "Save" button is pressed, underlining for those learning items (see Section 3.2.1) is compared with key data, and the results are fed back as a "comprehension index". Each time saving is done, it is counted as one learning trial, and saving and feedback are done in learning item units. If underlining which has been saved once is added to or deleted, and saving is done again after changing the underlined parts, underlining for that learning item unit is re-evaluated, and the learner can receive the new comprehension index. The comprehension index is calculated using the following formula. Comprehension index (C) [%]

Lc  Lw u 100 T

L c : Number of phrases where the learner underlined the same phrase as in teacher key data L w : Number of phrases where the learner underlined a phrase different from teacher key data T : Number of underlined phrases set by the teacher beforehand as key data for confirmation topics (important concepts acting as the instructional goal)

This comprehension index itself indicates the overall appropriateness of the learner's underlining, and is provided as an index to suppress behaviors such as not underlining or underlining too much. In this research, the purpose of feedback was to promote deeper learning comprehension by reviewing underlining, and the index was used not so much as an index suggesting a scheme for underlining (like the reproduction rate, or precision rate) but rather as an index suitable for indicating overall appropriateness. 3.2 Appropriateness of key data A critical issue for generating feedback on underlining is how to prepare key data on underlining. This system uses underlining provided by the creator of the instructional material him or herself. An investigation was done to determine whether this kind of key data is appropriate. In this investigation, we investigated the degree of agreement between underlining by multiple teachers with sufficient knowledge of the target instructional materials. However, underlining was done only once for a single learning item, and feedback on the comprehension index was not provided.

104

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature

3.2.1 Experiment materials and setup The instructional materials used in this research were materials for an "Information Processing Theory" course, treating content of the Information Processing Technician Examination (System Administrator Level I). The creators of the instructional materials were in charge of lectures for the course, and the materials were those actually used as auxiliary teaching aids for those lectures. These instructional materials were comprised of 31 learning items, with each learning item covering 1 or 2 A4 pages. The number of characters for each item was about 1600 on the average, and the number of phrases was about 250 on the average. The parts underlined beforehand by the creator of the instructional materials (the key data) were comprised, on the average, of about 70 phrases per learning item (i.e. about 28% of the whole). The instructional materials contained no diagrams or tables to facilitate understanding of the content. The subjects were two teachers of the information processing course, and although they were not directly in charge of lectures, they had sufficient specialized knowledge regarding "Information Processing Theory". 3.2.2 Experiment results and analysis The key data and underlining data provided by the subjects were compared for each learning item, and the results (indices) found by averaging (over the 31 learning items) are shown in Table 1. The reproduction rate and precision rate, averaged over the two subjects were, respectively, 59.5% and 88.2%. The high precision rate means that a little less than 90% of the parts thought to be important by the subjects were present in the key data, and the low variation suggests that the accuracy of the key data is high. The fact that the reproduction rate was not always high means that much of the underlining provided in the key data was not reproduced by the subjects, but we can allow this because underlining was done only once by the subjects, and although they had adequate knowledge of the material, they had never been in the position of directly teaching the course. Furthermore, when the indices of the subjects (teacher A and teacher B) were compared and a t-test was done, a significant difference was not found between the indices. For this reason, we determined that development of key data by having creators of instructional materials do underlining in those instructional materials has a certain degree of appropriateness. On that basis, we developed instructional materials based on this key data, and conducted experiments on learners. Methods of creating and evaluating more appropriate key data is an issue which we intend to pursue further in the future. Here, the reproduction rate and precision rate are calculated using the following formulas: Lc u 100 T

Reproduction rate (R) [%]

Precision rate (P) [%]

Lc u 100 Lc  Lw

L c : Number of phrases where the subject underlined the same phrase as in teacher key data L w : Number of phrases where the learner underlined a phrase different from teacher key data T : Number of phrases underlined by the teacher beforehand as key data for confirmation topics (important concepts acting as the instructional goal) Table 1. Results of comparing key data with subject underlining data Working time [Mins]

[Teacher A] n=31 learning items [Teacher B] n=31 learning items Teacher average

5.2 (2.8) 7.8 (4.6) 6.5

Reproduction rate [%]

Precision rate [%]

65.8 92.6 (10.8) (5.5) 53.6 83.8 (10.4) (5.3) 59.5 88.2 Figures in parentheses indicate standard deviation.

4. Evaluation experiment This section describes an experiment conducted to evaluate whether or not feedback of the comprehension index for underlining by the student is effective in promoting reading comprehension. Results on that experiment are also described.

105

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature

4.1 Experimental method In this experiment, the content was left the same as in Section 3.2.1, and two types of instructional materials were prepared: Instructional Materials I which only had the underlining feature, and Instruction Materials II which had the underlining feature as well as a feature for providing feedback on the comprehension index. The subjects were 240 freshman university students (who were studying in the humanities). These students were divided into 4 classes of 60 students each, and the experiment was conducted under 4 conditions, depending on whether Instructional Materials I or Instruction Materials II were used, and on whether the materials would be used for preparation or review (Table 2). The students were instructed to use these materials as auxiliary instructional materials for the class. The system was setup so that instructional materials for a certain learning item could be used prior to class for units of 5 or 6 learning items (in the case of preparation), and after class for units of 5 or 6 learning items (in the case of review). However, the system usage period was set to one week for those learning item units. Classes were held 12 times for each group (teachers generally did two classes on the content for each 5 or 6 learning item unit), and each subject was tested 6 times. These tests given once every two weeks, and were the fill-in-the-blank type where an appropriate word must be chosen. The content was almost the same as that used in previous classes. The number of problems in one test was about 10 problems for each learning item, for a total of 50 or 60 problems. Paper instructional materials with the same content as these instructional materials were provided, separately and beforehand, to the subjects to enable them to learn without using the system. Subjects were told beforehand that the test results would be reflected in their grades. Although use of the system was recommended, students were not told that such use would be reflected in their grades. Learning progressed at the same pace for all groups. With regard to homogeneity between groups, the classes corresponding to each group belonged to the same program, and classes were not formed based on ability. Therefore we feel there is no a priori difference between classes. Classes which might serve as a preparation for this subject were not held beforehand. In addition to the above, a subjective evaluation of the system was conducted using a questionnaire survey in the final class of the term. Table 2. System usage conditions for control groups and experimental groups Usage conditions

[Control group] n=60 persons Instructional Materials I, Preparation: [Experimental group] n=60 persons Instructional Materials II, Preparation: [Control group] n=60 persons Instructional Materials I, review: [Experimental group] n=60 persons Instructional Materials II, review:

Used Instructional Materials I (with underlining feature only) for preparation Used Instructional Materials II (with underlining feature AND comprehension index feedback feature) for preparation

Used Instructional Materials I (with underlining feature only) for review Used Instructional Materials II (with underlining feature AND comprehension index feedback feature) for review

4.2 Experiment results and analysis Table 3. Results on system usage under the preparation condition [Control group] n=44 persons Instructional Materials I, Preparation:

[Experimental group] n=59 persons Instructional Materials II, Preparation:

Number of uses* [Times]

Use time [Min]

Reproduction rate* [%]

Precision rate* [%]

Correct answer rate* [%]

3.1 (2.3) 6.0 (3.6)

12.4 (15.2) 10.8 (7.1)

24.8 (5.5) 61.2 (17.7)

67.7 (21.9) 81.5 (10.7)

52.3 (11.9) 73.5 (9.7)

Table 4. Results on system usage under the review condition Number of uses [Times]

Use time* [Min]

Reproduction rate* [%]

Precision rate* [%]

Correct answer rate* [%]

[Control group] n=46 persons 3.2 16.2 21.6 53.9 51.7 Instructional Materials I, Review: (1.1) (4.3) (12.9) (18.2) (8.9) [Experimental group] n=53 persons 3.1 7.2 45.2 77.0 63.6 Instructional Materials II, Review: (3.1) (4.9) (17.8) (13.4) (8.6) The asterisk (*) indicates that the control group and experimental group were compared, and the results of a t-test indicated significance at the 1% level (p<.01). Figures in parentheses indicate standard deviation.

106

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature

Even though use of this system was recommended, but not required, the number of users in each group who learned all the learning items using this system (i.e. the instructional material usage rate) was: 44 persons (73.3%) for Instructional Materials I and preparation, 59 persons (98.3%) for Instruction Materials II and preparation, 46 persons (76.7%) for Instructional Materials I and review, and 53 persons (88.3%) for Instructional Materials II and review. We feel that these learners made adequate use of the system, and we conducted analysis, taking these learners as the subject, with the aim of the investigating the effects of feedback (vs. no feedback). The results are shown in Table 3 and Table 4. Here, the values were obtained by averaging the values for each leaning item over all learning items. The number of uses is the number of times that underlining was saved for one learning item, and use time is the amount of time for each use. The reproduction rate and precision rate are the values for the underlining saved last for that learning item. The correct answer rate is the correct answer rate for problems in the test corresponding to one learning item. 4.2.1 Usage rate Although use of the system was recommended, it was not mandatory, and the number of subjects who did not learn using the system was not small. Conversely, we can say that the subjects who used the system used it on their own initiative. The instructional material usage rate for each group, indicated in Section 4.2, was clearly higher for Instructional Materials II than for Instructional Materials I. Since the only difference between Instructional Materials I and Instructional Materials II was the feedback feature, it appears that this feedback feature contributes to the high usage rate of Instructional Materials II. Therefore, this suggests that providing feedback on underlining evaluation results is effective for motivating underlining activity. 4.2.2 Number of uses and use time First, it is clear that underlining was reconsidered multiple times for 1 learning item, regardless of whether comprehension index feedback was provided, or whether materials were used for preparation or review. Here we can see the effect which underlining itself has on promoting reading comprehension activities. The biggest worry when providing feedback of the comprehension index is that "students will do underlining without reading comprehension, with the aim of simply improving their comprehension index". However, even in the case with feedback and review where usage was the lowest, the system was used for 7 minutes or more per time, so the above problem does not appear to be that notable. The differences between the preparation condition and review condition are considered further in Section 4.2.5. 4.2.3 Reproduction rate and precision rate The experimental groups had higher values than the control group for both reproduction rate and precision rate. Naturally, this is due to the fact that there was comprehension index feedback for the experimental groups, but investigating the correlation of the reproduction rate and precision rate yielded the results indicated in Figure 3 and Figure 4. Whereas a weak negative correlation was evident for control groups (with either the preparation condition or review condition), a strong positive correlation was evident for the experimental groups (with either the preparation condition or review condition). Results for control groups show that learners who provided many correct underlines also provided many incorrect underlines (high reproduction rate, low precision rate), and that learners who provided few incorrect underlines also provided few correct underlines (low reproduction rate, high precision rate). In contrast with this, results for experimental groups showed that many correct underlines were provided. This method of underlining is more refined than the method of underlining in the control groups, and comprehension index feedback can be regarded as promoting more desirable behavior in underlining intended to improve reading comprehension. 4.2.4 Correct answer rate The experimental group using Instructional Materials II with the feedback feature exhibited a high correct answer rate which was significant. These results, combined with the usage time described in Section 4.2.2 suggest that the underlining behavior of the experimental group described in Section 4.2.3 was done not simply to improve the comprehension index (which was provided as feedback). It contributed in a substantial way to reading comprehension.

107

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature 㪈㪇㪇

[Instructional Materials I, Preparation] Correlation Coefficient 䌛-䌝: -0.480 㪏㪇 a:-0.121 b:32.949

Average reproduction rate[%]

Average reproduction rate[%]

㪈㪇㪇

㪍㪇

㪋㪇

㪉㪇

㪇

[Instructional Materials II, Preparation] Correlation Coefficient 䌛-䌝: 0.625 a:1.036 b:-23.186

㪏㪇

㪍㪇

㪋㪇

㪉㪇

㪇 㪇

㪉㪇

㪋㪇

㪍㪇

Average precision rate[%]

㪏㪇

㪈㪇㪇

㪇

㪉㪇

㪋㪇

㪍㪇

Average precision rate[%]

㪏㪇

㪈㪇㪇

Figure 3. Correlation of reproduction rate and precision rate under preparation conditions 㪈㪇㪇

[Instructional Materials I, Review] Correlation Coefficient 䌛-䌝: -0.399 㪏㪇 a:-0.282 b:36.717

Average reproduction rate[%]

Average reproduction rate[%]

㪈㪇㪇

㪍㪇

㪋㪇

㪉㪇

[Instructional Materials II, Review] Correlation Coefficient 䌛-䌝: 0.683 a:0.905 b:-24.441

㪏㪇

㪍㪇

㪋㪇

㪉㪇

㪇

㪇 㪇

㪉㪇

㪋㪇

㪍㪇

Average precision rate[%]

㪏㪇

㪈㪇㪇

㪇

㪉㪇

㪋㪇

㪍㪇

Average precision rate[%]

㪏㪇

㪈㪇㪇

Figure 4. Correlation of reproduction rate and precision rate under review conditions

4.2.5 Preparation and review The difference between the preparation condition and the review condition was larger for Instructional Materials II (with comprehension index feedback) than for Instructional Materials I (with no comprehension index feedback). In particular, the learning time per item was approximately 65 minutes (6 times, 10.8 mins.) for the preparation condition and approximately 22 minutes (3.1 times, 7.2 mins.) for the review condition. This suggests that the reading comprehension support method where feedback is provided on the comprehension index exerts different effects with preparation and review. This does not simply mean that the system is used longer with the preparation conditions; it can also be interpreted to mean that, under the review condition, it is possible to obtain a higher score in a comparatively short amount of time. The current analysis does not cover the details of the underlining activity of each learner, so we cannot say that the difference between the preparation condition and review condition is certain, but in clarifying the range of application of this system, we feel it will be necessary to also investigate the effects of these sorts of use conditions. 4.3 Questionnaire survey A questionnaire survey was administered to learners in order to investigate how they subjectively evaluated this system and the feature for providing comprehension index feedback (i.e. how they accepted it, whether they thought it was effective). The questionnaire was comprised of four questions: "Initiative: Did you adequately prepare or review for class using the support system?", "Operability: Was the support system easy to operate?", "Effectiveness: Was the support system useful for preparation and review, and did it help you understand important concepts in the instructional material content?" and "Future potential: Would you like to use the support system again in the future?". Figure 5 shows the results obtained by dividing subjects into two sets (control groups and experimental groups) with 120 persons each and totaling the figures for each set. The questionnaire was administered using the web, and was given to all the subjects during the final class of the first term, so the response rate was 100%. Regarding the questions on "Initiative", "Operability" and "Future potential", there was no large difference between the control groups and experimental groups, and about 50% of the subjects responded in the affirmative. For the "Effectiveness" question, on the other hand, affirmative responses were about 33% for the control groups and about 53% for the experimental groups.

108

Y. Fukunaga et al. / Implementation and Effectiveness of a Feedback Feature

These results suggest that the presence of the feedback feature for the instructional materials had an impact on the effectiveness of system usage, from the subjective standpoint of the learners.

Figure 5. Tabulated results for each question on the questionnaire

5. Conclusions and future issues In this research, we implemented a feature for evaluating underlining and providing feedback on evaluation results as a method for exploiting the interactivity of e-learning to support underlining activities as one aid for improving reading comprehension of instructional materials. Instructional materials with this feature were developed, and evaluated in actual classes. The results suggest that: (1) Since there was an evident increase in the instructional material usage rate, the feature raised motivation toward underlining activities, (2) Underlining activities with a positive correlation between the reproduction rate and precision rate were seen, so the feature promotes more refined underlining activities, (3) Higher grades can be expected for reading comprehension confirmation topics (important concepts serving as the goal of instruction). For these reasons, we believe we have confirmed the effectiveness of this feature for supporting reading comprehension. This technique is a proposal for a method of evaluating the underlining used to promote reading comprehension, and we can expect these results to have a high degree of generality. Issues for the future include: (I) Further analysis of the data which has currently been obtained -- in particular, analysis of the activity of individual learners, (II) Refinement of the evaluation method and feedback method, and (III) Application to other instructional materials to show the generality of this technique. References [1] ALIC. (2003/2004). E-Learning White Paper. Advanced Learning Infrastructure Consortium, Ohmsha, Ltd. [2] Blanchard, J. and Mikkelson, V. (1987). Underlining performance outcomes in expository text. Journal of Educational Research, 80, 197-201 [3] Di Vesta, F. and Gray, G. (1973). Listening and note taking. Journal of Educational Psychology, 64, 278-287 [4] Fukunaga, Y., Takeuchi, A., Hirashima, T., Kunichika, H. (2004). A Scaffold Process to Facilitate Comprehension in Web-Based Training. IRMA2004, (pp.1187-1188). Proceedings of 15th Information Resources Management Association International Conference, New Orleans, America, IRMA Press. [5] Glynn, S. (1978). Capturing reader’s attention by means of typographical cueing strategies. Educational Technology, 18, 7-12 [6] Johnson, D. and Wen S., (1976). Effects of correct and extraneous markings under time limits on reading comprehension. Psychology in the Schools, 13, 454-456 [7] Marshall, C. C. (1997). Annotation. ACM1997, (pp.131-140), From Paper Book to Digital Library. Proceedings of the 2nd ACM international conference on Digital Libraries. [8] Seki, U. (1997). Effects of Itemization and Highlighting Key Words on Text Comprehension. Journal of Japan Society for Educational Technology, 21, 17-20 [9] Uosaki, Y., Itoh, H., and Nojima E. (2003). Effectiveness of Underlining the Texts on Readers’ Comprehension. Journal of Japan Society for Educational Technology, 26(4), 349-359.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

109

Interactivity of Exercises in ActiveMath a

b

c

Giorgi Goguadze , Alberto Gonzalez Palomo , Erica Melis a University of Saarland, Saarbrucken, Germany b German Research Institute for Artificial Intelligence (DFKI) Saarbrücken, Germany c German Research Institute for Artificial Intelligence (DFKI) Saarbrüucken, Germany [email protected] Abstract: Interactive exercising is one of the major ingredients of technology-enhanced learning. It reaches its full potential only, when appropriate feedback is given to the learner. This paper describes a principled approach for representing and processing interactive exercises in ActiveMath. This approach relies (1) on a modular architecture of the exercise player, (2) on a separation of different types of knowledge that have to be handled to generate feedback, and (3) on a generic representation of interactive exercises. Keywords: architecture, intelligent scaffolding, exercise representation, web-based mathematics application

1. Introduction Interactivity is one of the major advantages of technology-enhanced learning over traditional learning material. For an effective and interesting learning experience, a variety of types of interactive exercises is needed rather than multiple choice questions (MCQs) only. Moreover, it has been shown, that feedback is mandatory for fully exploiting the denefits of interactivity for learning [10]. Many e-learning systems do not provide feedback that goes beyond a correct/incorrect response referring to pre-defined results. Even in intelligent tutoring systems (ITS), this diagnosis is done for small domains only or feedback is authored. Such a program is more powerful, if it uses reasoners such as a computer algebra system (CAS) to evaluate the input of the learner. An author can encode the exercise using the language of a CAS and connect to the CAS to ”play” the exercise. This was our first approach in ActiveMath [1]. Previously, exercises in ActiveMath were programmed individually or as classes of exercises. Our experience is that such an approach is limited. On the one hand, it binds authors to the syntax of the computer algebra system. Moreover, CASs are not designed for representing human-like problem solving steps. Such exercises are not reusable with other CASs, and therefore, the learning environment is bound to the particular CAS. On the other hand, it is not possible to apply diěerent tutorial strategies to the same exercise and to adapt to the particular learner or learning situation. In order to extend the types of range of interactive exercises to respond to author’s wishes, to keep exercises modifiable and extensible, and to provide more elaborate feedback in ActiveMath, our goal was to allow for both, authored and generated feedback, for a variety of exercise types, and for a clear separation of functionalities necessary for interactive exercise. We report some main results on that way here. This paper is organized as follows. After short preliminaries about the ActiveMath

110

G. Goguadze et al. / Interactivity of Exercises in ActiveMath

system and the OMDoc knowledge representation we introduce the extensions to the knowledge representation of exercises and show how this knowledge representation works in the ActiveMath exercise subsystem. 2. Preliminaries ActiveMath is a web-based user-adaptive learning environment for mathematics [7]. It dynamically assembles courses of learning material for the individual learner according to her learning goals, preferences, and mastery of concepts. It makes suggestions on how to proceed with learning and navigation. The information about the learner and his learning context is represented in a Learner Model. The dynamic assembling of courses is performed by the Tutorial Component which selects elements to be presented to the learner, according to the information from the Learner Model. OpenMath and OMDoc Representations for Mathematics The content is annotated with metadata and stored in a knowledge base. The knowledge representation language in ActiveMath is OMDoc, a semantic markup language for mathematical documents [6]. OMDoc has evolved as an extension of OpenMath1 which is an XML-representation standard for mathematical formulas. The main difference between OpenMath and other representation formats for mathematical formulas, such as Presentation MathML and LaTeX, is that OpenMath deals with the semantics of mathematical expressions rather than with their presentation only. OpenMath defines so-called Content Dictionaries in which mathematical symbols are declared and their semantics is defined. OpenMath formulas are tree like XMLstructures of OpenMath symbols. The semantic representation of formulas allows for automatic translation of these formulas to (and from) the languages of different mathematical systems This provides the basis for interoperability of different computer algebra and other reasoning systems. OMDoc defines learning objects (LOs), such as exercises, definitions, and some relations between them. Such LOs can consist of text mixed with formulas in OpenMath format. We extended the OMDoc representation of the exercises to meet frequent usage requirements for interactive exercises. The new format is briefly described in the following section. A full account can be found in [4]. 3. Knowledge Representation of Interactive Exercises An exercise can be seen as a finite state machine of interactions, either predefined or generated. An interaction is a state of an exercise in which an interaction of the learner with the system takes place. The interaction contains feedback for the event that triggered it, an interactivity assignment that describes how to substitute certain parts of the feedback by interactive elements (e.g. replace a term by a blank), and one or more transition maps describing which interaction comes next depending on the learner’s input. We extended the knowledge representation of interactive exercises in [3] by introducing three layers of markup for each interaction of the exercise. The first layer consists of the 1

see http://www.openmath.org for the standard specification

G. Goguadze et al. / Interactivity of Exercises in ActiveMath

111

content, i.e. text possibly mixed with formulas (feedback). The second layer attaches interactive elements of different types to the places in the content marked for interactivity (interactivity assignment). The third layer defines conditions to be satisfied in order to proceed to further interactions (transition map). The element feedback contains textual content presented to the learner in the step.2 The optional attribute from allows to import the feedback elements from another interaction, and the attribute keep specifies whether the previous input plus feedback is shown after the step is performed or whether it is not. The interactivity assignment attaches interaction types to the content of the feedback element. It defines which parts of the content become interactive and what types of interactivity is to be used, e.g. selection (presented as choice questions or drop-down menus), fill-in-blank, mapping, or marking. The transition map contains the condition elements and a default for a default condition. Default condition is used if none of the other conditions are satisfied. Each condition has a pointer to an interaction to be executed, if this condition is satisfied. Currently, the condition element is a statement that may contain predicates syn_eq, num_eq and sem_eq. syn_eq means ’syntactic equality’ and calls a procedure that checks whether the input of the learner is syntactically equal to a given expression. num_eq means ’numeric equality’ and calls a procedure that checks whether the input of the learner is a number and is numerically equal to a given number. If the user input differs from prescribed result less then a given epsilon, then the input is considered numerically equal. sem_eq means ’semantic equality’ and calls a procedure that checks whether the input of the learner is the same mathematical object. For this evaluation a computer algebra system is used. Each interaction can be annotated with metadata describing competencies, relations to concepts trained in the step, as well as difficulty, mastery assessment to be sent to the Learner Model. Figure 1 shows a representation of an exercise step the rendering of which is shown in the Figure 2 (OpenMath formulas are abbreviated to linear notation for the lack of space). Try again: diff((x*sin(x+1))+2,x)=diff("blank1"),x)+diff("blank2",x) <sem_eq>x*sin(x+1)<sem_eq>x <default xref="int1_wrong"/> Figure 1. Exercise step representation 2

This is analogous to QTI’s material element

112

G. Goguadze et al. / Interactivity of Exercises in ActiveMath

In the Figure 1 interaction consists of two feedbacks, the first of which is imported from the previous interaction and kept in the next step. The second feedback is a hint statement and will disappear in the next step. The interactivity assignment defines 2 blanks for those two parts of the formula marked with IDs. In the transition map one condition is defined, containing the semantic comparisons with the correct results and one default condition. For all possible steps an exercise can contain fully authored interaction elements as well as an interaction_generator element that is used to generate interactions automatically. 4. Authored vs Generated Answers The architecture of the exercise system has the potential to process fully authored as well as (partially) generated exercise representations. Whether the interactions are generated depends on the availability of intelligent components that can help to diagnose errors of the learner in a particular learning domain and a component for generating feedback from the diagnosis. Part of the feedback generation is the application of tutorial strategies. In order to allow for diěerent tutorial strategies for the same exercise, these strategies have to be represented separately and applied to the exercise representation. The exercise subsystem renders interactive steps, evaluates the learner’s input, diagnoses and generates feedback, automatically generates interactions applies tutorial strategies for adapting to the learner and his learning situation, reports the achievements of the learner to the rest of the ActiveMath system. Consider the two versions of an exercise in Figure 2 and 3, whose interactions were completely generated by a reasoning component and whose feedback was inserted automatically by two diěerent tutorial strategies. Figure 2 shows the result of the application of the first strategy. In this strategy, when the learner asks for a hint, the system provides him with a simpler task by inserting the structure of the derivation rule to be applied here. In the second strategy, shown in the Figure 3, the needed derivation rule is formulated, and the learner is invited to try the same task again.

Figure 2. Partial insert -hint strategy

The hint in the first strategy is generated via the reasoner that computes concrete functions which are inserted into the rule application. The hint in the second strategy does not depend on the concrete function and does not have to be generated by the reasoner each time. The reasoner determines the appropriate rule and provides its link to the tutorial strategy.

G. Goguadze et al. / Interactivity of Exercises in ActiveMath

113

Figure 3. Conceptual hint strategy inserting the rule statement

5. Separation of Functionalities for Scaěolded Exercising It is a well-known fact that modifications and extensions are much easier to realize, if different types of functionalities and knowledge are separated by design and implementation. Therefore, the ActiveMath exercise player has a modular architecture. The diagram in Figure 5 displays a simplified architecture and information flow. The Exercise Manager is the main component of the exercise subsystem responsible for managing the collection of the exercise steps and for executing the exercise steps. In the initialization process, the Exercise Manager receives an exercise from the database and the learner data from the Learner Model. In each step it connects to an Interaction Manager in order to receive the next interaction to be played. In case of a fully authored interaction, the Interaction Manager is substituted by the default static Interaction Manager that is delivering the existing manually authored interaction from the exercise. After receiving a fully authored interaction, Exercise Manager sends it to User Interface Mediator which presents it to the learner. In case of a (partially) generated interaction, another generator is employed. This generator might need to connect to the Evaluator/Diagnoser which can use Domain Reasoner or CAS components to provide the diagnosis on the action of the learner. The Evaluator/Diagnoser can query external services such as Domain Reasoner or CAS for diagnosis on the input. Using this diagnosis the feedback and the next interaction are generated.

114

G. Goguadze et al. / Interactivity of Exercises in ActiveMath

Figure 4. Exercise Subsystem architecture After the learner has submitted his answers or requests3, the User Interface Mediator is translating this answer/requests into OpenMath and passes it to the Exercise Manager. The Exercise Manager connects to the Evaluator/Diagnoser component which evaluates the learner’s input by comparing it to each of the conditions in the transition map of the current interaction. The generator also applies a tutorial strategy from the Tutorial Strategy module to the step. As soon as an interaction has been assembled, it is returned to the Exercise Manager which is passing it to the User Interface Mediator. In each step, the Exercise Manager is also sending an event containing the learner’s input and information characterizing the step such as the competencies trained in the step, achievement of the learner w.r.t. the task, identifiers of concepts and misconceptions in erroneous steps. This event can be used for updating the Learner Model. The current implementation of the exercise subsystem is server-based and works with a tight integration of the components by means of procedure calls. The components of the exercise subsystem exchange fragments of XML documents (interactions or their parts) or only OpenMath formulas (e.g. for learner input). 6. Conclusion ActiveMath offers a general system-independent semantic markup language for representing interactive exercises of different kinds. The exercise subsystem of ActiveMath is playing such exercises in a browser. The exercise subsystem can connect to CASs and to other external 3

Requests allow asking for hints or other forms of help

G. Goguadze et al. / Interactivity of Exercises in ActiveMath

115

programs in order to diagnose the learner input and to generate feedback. There are no explicit calls to a particular external program in the exercise content. An exercise can be manually authored, generated completely, or partially generated/authored. In order to generate feedback and next interactions, the diagnosis of the learner’s input has to be made. For this, the connection to a reasoning component is established. Obviously, a bottleneck is to find and encode all relevant potential erroneous actions of the learner. For instance, in [5] 1000 buggy rules for the fractions domain were empirically found and narrowed down statistically to 300. User-adaptive presentation can be achieved via automatic enrichment of the manually authored exercise by the tutorial strategies which are called and applied to the exercise representation. Therefore, it is possible to reuse the same tutorial strategies for different exercises. The graphical user interface for presenting such an exercise to the learner is also interchangeable since the exercise representation does not determine the presentation/rendering. The presentation is handled separately by the ActiveMath presentation system [9]. Acknowledgment We are indebted to Claus Zinn for his clarifying interventions and help. This publication is partly a result of work in the context of the LeActive-Math project, funded under the 6th Framework Program of the European Community – (Contract IST-2003507826) and RTD project iClass, funded under the 6th Framework Program of the European Community – (Contract IST-507922). The authors are solely responsible for the content. References [1] J. Büdenbender, E. Andrès, A. Frischauf, G. Goguadze, P. Libbrecht, E. Melis, and C. Ullrich, Using Computer Algebra Systems as Cognitive Tools. 6th International Conference on Intelligent Tutor Systems (ITS-2002), Springer-Verlag, Series “Lecture Notes in Computer Science”, editors S.A. Cerri, G. Gouarderes, and F. Paraguacu, number 2363, pages 802–810, 2002. [2] Global Learning Consortium. IMS Question & Test Interoperability Specification: A Review . [3] G. Goguadze, E. Melis, C. Ullrich and P. Cairns, Problems and Solutions for Markup for Mathematical Examples and Exercises. In Proceedings of the Second International Conference on Mathematical Knowledge Management, MKM03, Andrea Asperti (ed.). . [4] Goguadze, G., González Palomo, A., “Knowledge Representation of Interactive Exercises in ActiveMath”, SEKI-Report, Nr. SR-04-10, University of Saarland, 2004. [5] M. Hennecke. Online Diagnose in intelligenten mathematischen Lehr-Lern-Systemen. PhD thesis, Fortschritt-Berichte VDI, Hildesheim, 1999. [6] M. Kohlhase. OMDoc: Towards an OpenMath representation of mathematical documents. Seki Report SR00-02, Fachbereich Informatik, Universität des Saarlandes, 2000. [7] E. Melis, E. Andrès, J.Büdenbender, A. Frischauf, G. Goguadze, P. Libbrecht, M. Pollet, and C. Ullrich. ActiveMath: A generic and adaptive web-based learning environment. International Journal of Artificial Intelligence in Education, 12(4):385– 407, 2001. [8] M. Mavrikis, A.Gonzaléz Palomo, Mathematical, Interactive Exercise Generation from Static Documents. . [9] P. Libbrecht, C. Ullrich, and S. Winterstein, An Effcient Presentation-Architecture for Personalized Content, in Proceedings of Berliner XML Tage 2003, editors R. Tolksdorf and R. Eckstein, Pages 379–388, 2003, ISBN 3885791161. [10] R. Schulmeister. Didaktisches Design aus hochschuldidaktischer Sicht -Ein Plädoyer für offene Lernsituationen. In U. Rinn and D.M Meister, editors, Didaktik und Neue Medien. Konzepte und Anwendungen in der Hochschule, number 21, pages 19–49. ., 2004.

116

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

EpiList II: Explicit Instructions for Development of Generic Cognitive Skills Ghee Ming GOH, Chai QUEK Centre for Computational Intelligence formerly known as the Intelligent Systems Laboratory, Nanyang Technological University, Singapore [email protected] Abstract. The importance of cognitive skills in human learning and reasoning has grown progressively throughout the years. This motivates the search for a knowledge representation that supports the development of a computational algorithm that performs as well as instructs such cognitive skills. Cognitive skills can be broadly divided into two types; specific and generic cognitive skills. EpiList I is an ITS that develops generic cognitive skills namely classification, generalization, and comparison skills. However, EpiList I develops the skills implicitly in the tutorials that explicitly tutors the student on the domain knowledge. This is a major weakness of EpiList I in developing cognitive skills; namely: classification, generalization and comparison. EpiList II incorporates explicit instructions to develop the cognitive skills in EpiList I. The explicit instructions in EpiList II monitor, tutor, and assess the student’s competency in the cognitive skills. The explicit instruction is equivalent to a closed-loop system. Keywords: ITS, explicit and implicit Instructions, generic cognitive skills, generalization, comparison.

Introduction In educational fields, schools and teachers are readily accepting the psychology of subject teaching and cognitive strategies developed by educational psychologists [7]. The importance of cognitive skills in human learning and reasoning has grown throughout the years. Several tests have shown that teaching cognitive skills to low-graded students will significantly increase the student’s performance, sometimes even higher than high-graded students [8]. This motivates the search for a knowledge representation that supports the development of a computational algorithm that performs such cognitive processes. Furthermore, it is desirable for this knowledge representation to be domain independent, subsequently yielding a domain independent computational algorithm. Cognitive skills can be broadly divided into two types; specific and generic cognitive skills. The former are skills that are applicable to a limited or a single domain while the latter are skills that are applicable to a wider or unlimited scope of domains. Several ITSs had been successfully developed to teach specific cognitive skills. The specific cognitive skills incorporated include math and science analytical skills [11], electronic circuit trouble-shooting skills [15], and programming skills [9]. The focuses of ITSs that teach cognitive skill are gradually shifting from specific to generic cognitive skills. Recent developments such as TAP [14] focusing on inquiry skills, SCI-WISE [12] and Inquiry Island [13] focusing on inquiry skills, and EpiComp focusing on generalization skills [2] highlight the research trend to be moving towards the teaching of generic cognitive skills.

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

117

EpiList I [5] is an ITS that focuses not only on the content knowledge but also on the cognitive skills involved in acquiring and organising the knowledge. The cognitive skills developed by EpiList I includes classification, generalization, and comparison. EpiList I is developed from the List-making and Compare-and-contrast Games in the set of Epistemic Games, a pedagogical theory of instruction, proposed by Prof. Collins and Prof. Ferguson [3]. It guides the student to perform classification in a bottom-up environment. Whenever the student makes mistakes in the classification, EpiList I would highlight the mistakes through a sequence of generalizations and comparisons. Almost all ITS that teaches specific cognitive skills employ the explicit instructions approach. However, few development of ITS, including EpiList I, incorporates implicit instructions to tutor the student’s generic cognitive skills. The issues of implicit instructions are the inefficient and unpredictable learning outcome of its instructions. These issues are present in EpiList I as it implicitly develops generic cognitive skills of the student. Hence, explicit instructions are highly desirable over implicit instructions. EpiList II employs a meta-cognitive teaching theory to explicitly develop the generic cognitive skills of EpiList I; namely, generalization and comparison skills. 1. Explicit Instructions of EpiList II “Implicit learning has been characterized as a subconscious process by which knowledge is acquired without deliberate attempt to learn. Its contrast, explicit learning, occurs when there is active engagement in conscious strategies to discover the rules underlying a task [10].” Thus, instructions that require the student to perform implicit or explicit learning are denoted as implicit or explicit instructions respectively. The instructions of EpiList I focus on correcting incorrect classification made by the student and migrating from student’s classification scheme to a significant and suitable classification scheme. The former is governed by the intra-scheme rules while the latter is governed by the inter-scheme rules. Both intra- and inter-scheme rules employ generalisation and comparison to highlight mistakes and migrate classification schemes. Hence, while the student acquires the domain knowledge through the tutorials, he also implicitly develops his generic cognitive skills. EpiList I does not ensure or monitor the development of these cognitive skills. Therefore, the instructions of EpiList I implicitly develop the student’s cognitive skills while explicitly instructing the student on the domain knowledge. From the field evaluation of EpiList I [6], it is observed that although the students show improvements in the cognitive skills taught by EpiList I; however, the improvements are not significantly great. This is due to the fact that implicit learning is time consuming [1]. The evaluation also reveals that very few students are able to precisely explain and describe the sequence of generalizations and comparisons in the tutorials. This demonstrates the limitations of implicit instructions on generic cognitive skills in practice. Hence, this motivates the research effort to develop explicit instructions of generic cognitive skills in EpiList I. The explicit instructions would ensure greater effectiveness as well as achieving desirable learning outcome in cognitive skills learning. EpiList II is a significant advancement over EpiList I by explicitly teaching generic cognitive skills to the student. In order to achieve explicit instruction for cognitive skills, an ITS has to: x Explicit monitor the student on his cognitive skills.G x Directly instruct and tutor the student on the cognitive skills.G x Assess the student on the knowledge and application of the cognitive skills.

118

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

1.1 Monitoring the student The monitoring of generic cognitive skills in EpiList II occurs while the student is guided by the intra- and inter-scheme rules. The intra- and inter-scheme rules of EpiList I are sequences of generalization and comparison. This provides the opportunity for EpiList II to capture execution of these cognitive skills and hence, determine the cognitive skills competency level of each student. When an inadequacy in a skill is detected, EpiList II attempts to explicitly tutor the student on that lacking skill. However, when such an opportunity is not available, other means should be provided to enable continue monitoring until the student has met the desired learning outcome of skills. Such non-opportunistic monitoring is provided by the non-opportunistic assessment that will be discussed in later section. Similarly, EpiList II would explicitly tutor the student on the skill that is captured as deficient from the non-opportunistic assessment. 1.2 A pedagogical model for tutoring the student The meta-cognitive pedagogical model employed by EpiList II is based on the cognitive apprenticeship model proposed by Prof Collins [4]. The instruction sequence of the cognitive apprenticeship is as follows: x Demonstrate (or make visible) the cognitive tasks x Demonstrate the application of the task in different context x Assess the studentG Using this model, the explicit instructions of EpiList II would first guide the student to perform the cognitive processes/tasks step by step. This would guide the student to recognize the sequence involved in the cognitive processes. Due to the limitation of the database in EpiList I, EpiList II could not demonstrate the cognitive processes under different context. Hence, after guiding the student to perform the cognitive processes, the instruction sequence would directly skip to assess the student. To access the students’ understanding of those cognitive skills, a test scenario is generated. This form of testing is termed as opportunistic assessment. 1.3 Assessing the student There are two forms of assessments, opportunistic and non-opportunistic assessments. The former assesses the student within a tutorial to ensure that the student has learnt the required cognitive processes. The latter assesses the student as and when necessary to ensure the student has achieved the desired cognitive skills competency level. Opportunistic Assessment. This assessment is undertaken, according to the pedagogical model described in Section 1.1, after the explicit tutoring of cognitive skills. Since the objective of the assessment is to assess the acquisition and understanding of the tutored skill, this assessment would target the tutored skill at the opportunistic instance within the tutorial interaction of EpiList I. Hence, opportunistic assessments are single page scenarios of the tutorials that test for one specific skill. Non-opportunistic Assessment. This assessment is presented when the learning objective of EpiList I is met but not that of EpiList II. When the desired learning outcome of EpiList I is met, the intra- and inter-scheme rules would not be fired. Hence, there is no opportunity for EpiList II to monitor the execution of the cognitive skills. Hence, the non-opportunistic assessment provides complete scenarios of EpiList I that would activate

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

119

the intra- and inter-scheme rules. This would create the opportunity for monitoring the student’s competency in cognitive skills until the desired learning outcome of EpiList II is met. 2. Architecture of EpiList II 2.1 Student’s Skills Competency Modeling The cognitive skills that EpiList I attempts to implicitly develops are chiefly the generalization and comparison skills. Comparison is the process of identifying the similarities and differences. The process of identifying the similarities is the comparing process while the process of identifying the differences is the contrasting process. Hence, the set of cognitive skills to be monitored by EpiList II are generalization, comparing and contrasting. Monitoring involves the counting of the number of times a student has correctly and incorrectly used the cognitive processes. A vector of integers, where each integer represents a cognitive skill, is used for the counting. A correct usage of a cognitive process increases the corresponding integer value of the vector, while the incorrect use decrements it. This counting vector is termed the UseCounter. Another vector, termed as WantedValue, indicates the desired values for the UseCounter. Hence, the WantedValue also represents the desirable learning outcome of EpiList II. EpiList II essentially overlays the UseCounter against the WantedValue to determine the cognitive competency levels of the student. The overlaying approach is able to identify the following competency conditions of the student: x Competency in the skill (UseCounter >= WantedValue) x Deficiency in the skills (UseCounter < 0) G x Insufficient competency in a skill (0 < UseCounter < WantedValue)G x Uncertain competency in a skill (UseCounter = 0) 2.2 Cognitive Rules The cognitive rules determine the appropriate actions for the student. The actions include providing tutorials for deficient skills and non-opportunistic assessments for insufficient or uncertain competency of skills. 2.2.1 Tutorials Table 1 Interaction between a tutorial and a student Semantic Dialogue Obtain properties of System: (Displays Tiger and a list of categories.) first item. Which of the following categories does this animal belong to? Student: (Selects “Warm-blooded”.) System: (Displays Crow and the same list of categories.) Obtain properties of second item. Which of the following categories does this animal belong to? Student: (Selects “Breathe through lungs”, “Warm-blooded” and “Lay eggs”.) Obtain properties of System: (Displays Lizard and the same list of categories.) third item. Which of the following categories does this animal belong to? Student: (Selects “Breathe through lungs”, “Cold-blooded”.) Generalize the three System: (Displays Tiger, Crow and Lizard as well as their categories that the items. student had selected previously.) Which of the following categories do all these animals belong to?

120

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

Explain the skill.

Assess the student.

Student: (Selects “Breathe through lungs”.) System: Well done. This is the process of generalization. Generalization is to obtain the categories that all the displayed animals belong to. System: Now I am going to test your generalization skill. (Display opportunistic assessment to assess the student) Student: (Answer the assessment correctly.) System: Well done. Now go back to the tutorial.

When a deficiency in a cognitive skill is detected, the respective tutorial will be activated to teach the student the corresponding skill. The Instructional sequences would guide the student to explicitly learn the cognitive skills. The dialogue in Table 1 is an example of the instructional sequence that would explicitly teach the student is to generalise ‘Tiger’, ‘Crow’ and ‘Lizard’. 2.2.2 Scenarios Table 2 Steps to generate complete scenario Steps Determine the skill to be monitored. Determine the specific rule of EpiList I to be activated. Randomly selects a few categories. Extract an appropriate set of items from the explicit domain knowledge in EpiList I based on the selected categories and rule. Determine the categorization of the items that would activate the correct rule.

Results Generalization skill Linear Negative Exemplar Rule “Breathe through lungs”, “Cold-blooded” and “Lay eggs” Tiger, Bird, Lizard, Shark

Categorize all animals into “Breathe through lungs”

Scenario is an important feature of the skill instructional loop. It essentially provides the conditions for both opportunistic and non-opportunistic assessments. Summarizing the details of the scenarios provided in Section 1.3, the single page scenarios are for opportunistic assessments while the complete scenarios are for the non-opportunistic scenarios. Although the two types of scenarios are different in structure and purpose, the generations or creations of both types of scenarios are very similar. Table 2 describes the procedure undertaken by the system to generate a complete scenario for non-opportunistic assessment. The complete scenario, of Table 2, should provide the opportunity to monitor generalization skill. Table 3 Steps to generate single page scenario Steps Identify the tutorial that activates the explicit cognitive instruction. Identify the cognitive skill to be assessed. Extract the categories from the tutorial. Extract an appropriate set of items from the explicit domain knowledge in EpiList I based on the selected categories and rule. Display the items in a page of the identified tutorial and display the page to assess the targeted cognitive skill.

Results First page of Linear Positive Exemplar Rule Generalization skill “Breathe through lungs”, “Cold-blooded”, “Warm-blooded” and “Lay eggs”. Horse, Cobra, Lion, and Dolphin

Display Horse, Cobra, Lion, and Dolphin in first page of Linear Positive Exemplar Rule with the categories “Breathe through lungs”, “Cold-blooded”, “Warm-blooded” and “Lay eggs”.

Table 3 presents the procedure to generate a single page scenario. The opportunistic assessment generated by the single page scenario of Table 3 is essentially an assessment for

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

121

the tutorial presented in Table 1. Hence, the resultant single page scenario is based on the information in Table 1. 3. Evaluation of EpiList II The evaluation of EpiList II was undertaken at Methodist Girl School, Singapore (MGS). The objective of EpiList II's evaluation is to demonstrate its improved efficiency in cognitive skill development. From the evaluations of EpiList I, it is found that primary 4 students are more receptive than primary 5 cohorts. Hence, the evaluation of EpiList II is conducted on primary 4 students only to enable a significant comparison with evaluation results of EpiList I to analyze the improvement in efficiency of explicit cognitive instructions. The primary 4 students that evaluate EpiList II are consolidated from four classes. In total, there are 140 ten years old students evaluating EpiList II. Similar to the evaluation of EpiList I, The evaluation consists of 3 phases. The first phase is the pre-evaluation test, which is a written test. The second stage is a hands-on session where the students go through the system twice. Lastly, the third phase is another written test, the post-evaluation test. However, in this evaluation of EpiList II, there is a slight change to the questionnaires and style to produce controlled evaluation. It has been argued that the improvement shown in EpiList I is due to the lower difficulty level in the post-evaluation. Therefore, the evaluation of EpiList II attempts to rebut such an argument. The method is to switch the questions that tests for the cognitive skills between the pre- and post-evaluation tests. Hence, if improvements are still registered in this swapped evaluation tests; it shows that the improvement is not due to the difference in the difficulty levels. The second change was made in response to the argument that the improvement in the cognitive skills was not due to EpiList I but the normal curriculum of the school. Therefore, for the evaluation this time round, not all the student will be involved in the hands-on session to use EpiList II. The student are split into half, two classes would go though all three phases of evaluation while the other two classes would only go through the first and last phases which are the answering of pre- and post-evaluation. In this way, the cognitive skills improvement of the student can be differentiated between EpiList and the school curriculum. 3.1 Result and analysis of the controlled group evaluation This section presents the results of the controlled group evaluation. This group of students answers the pre- and post-evaluation tests without being exposed to EpiList II. Figure 1 is the results of the evaluation tests of this group of students. From the results shown in Figure 1, the differences between the pre- and post-evaluation test is as small as 1% except for the difficult generalization skill. From these results, it can be deduced that although the school's curriculum contributes to the improvement of cognitive skills proficiency, but the contribution is relatively small and negligible. Using these results to compare with the results of evaluation in EpiList I [6], it can be seen that primary 4 students in the cohort of 2004, who evaluated EpiList II, is better than the primary 4 students in the cohort of 2003 who evaluated EpiList I in term of competency in cognitive skills employed in EpiList I. However, after the usage of EpiList I, the proficiency of last year's students exceeds this year's students. This shows that the improvement registered in EpiList I was due to the system and not from other factors.

122

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

Figure 1 Result of the Controlled Group Evaluation

3.2 Result and analysis of the EpiList II controlled evaluation This section focuses on the evaluation of EpiList II. The students in this evaluation follow the three phases of evaluation to evaluate EpiList II. The questions of this evaluation are swapped for the pre- and post-evaluation tests.

Figure 2 Result of the EpiList II Controlled Evaluation

The first and second bars of the chart in Figure 2 are the results of the pre- and post-evaluation tests of EpiList I. These results are there for the comparison against the results of the pre- and post-evaluation test of EpiList II, which is represented by the third and fourth bars of Figure 2. Focusing on the results of the EpiList II, increase or improvement in competency is demonstrated across the different cognitive skills. The improvement is, on average, 15% for generalization skills and 9% for comparison skills. This categorically demonstrated the fact that the questions of both pre- and post-evaluation tests are unbiased and they are able register the increment in competency of skills. When the results of EpiList II’s evaluation are compared with the evaluation results of EpiList I, it can be observed that the improvement in EpiList II is smaller than those of EpiList I. This could be due to the fact mentioned earlier that the cohort that evaluated EpiList II is better in terms of cognitive skills competency. As most of the student was already competent in these set of cognitive skills, therefore, to increase the competency level further requires greater effort and time. However, a significantly greater improvement in cognitive competency can be observed from the reasoning skills. As high as 51% of the student, as shown in Figure 2, have acquired or understood the positive and negative reasoning skills as compared to the average of 8% in EpiList I. This significant difference demonstrated the effectiveness of

G.M. Goh and C. Quek / EpiList II: Explicit Instructions for Development

123

explicit teaching employed in EpiList II. 4. Conclusion EpiList II is an ITS that develops generic cognitive skills; namely: classification, generalization and comparison by means of explicit instructions. Explicit instructions ensure the effectiveness of the tutorials and achieve the desired learning outcome for cognitive skills. It is necessary to monitor, tutor and assess the student’s acquisition of cognitive skills. This paper presents the methodology that EpiList II employs to accomplish such forms of explicit instructions. The examples of the tutorial and assessment provide a clear insight on EpiList II. One major disadvantage of EpiList II is that without a proper curriculum for the teaching of cognitive skills, the instruction becomes disorganised. The monitoring of cognitive skills is assessed and achieved mainly from the rules of EpiList I and the students are subsequently opportunistically tutored. Hence, current active research efforts within the group, the Centre for Computational Intelligence, investigate the use of a curriculum planner to systematically assess and teach cognitive skills. Other work in the group includes the incorporation of Multicausal Game in the form of food chain to develop cause-effect analytical skill. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

Anderson, Mike, (1998). Individual differences in Intelligence. In K. Kirsner, C. Speelman [et al.] (eds.), Implicit and Explicit Mental Processes. Lawrence Erlbaum Associates, Inc. Ang, Huey Min, (1999). EpiComp: An ITS for Multiple Concept Compare and Contrast Game. Final Year Report. Nanyang Technological University. Collins A., & Ferguson W., (1993). Epistemic Forms and Epistemic Games: Structures and Strategies to Guide Inquiry. Education Psychologist 28(1), 25-42. Collins, A., Brown, J.S., & Holum, A., (1991). Cognitive Apprenticeship: Making Thinking Visible. American Educator. Goh, G.M., Quek, C., (2004). EpiList: An ITS Shell to Develop Generic Cognitive Skills for Bottom-up Knowledge Construction. International Conference on Computers in Education, ICCE04. Goh, G.M., (2003). Student Evaluation of EpiList. Technical Report No. ISL-TR-01/03. Intelligent System Lab, School of Computer Engineering, Nanyang Technological University. Good, Thomas L., & Levin, Joel R., (2001). Educational Psychology Yesterday, Today and Tomorrow: Debate and Direction in an Evolving Field. Educational Psychologist, 36(2), 69-72. Mayer, Richard E., (2001). What Good is Educational Psychology? The Case of Cognition and Instruction. Educational Psychologist, 36(2), 83-88. Ng, F.C.N., & Butler, G., (1993). Specialized Theorem-proving in an Intelligent Tutoring System for the Dijkstra-Gries Programming Methodology. IEEE. O’Brien-Malone, Angela & Maybery, Murray, (1998). Implicit Learning. In K. Kirsner, C. Speelman [et al.] (eds.), Implicit and Explicit Mental Processes. Lawrence Erlbaum Associates, Inc. Cheng, Peter C.H., (1999). Unlocking conceptual learning in mathematics and science with effective representation systems. Computer and Education 33 (109-130). ESRC Centre for Research in Development, University of Nottingham. Shimoda, Todd A., White, Barbara Y., & Frederiksen, John R., (1999). Acquiring and Transferring Intellectual Skills with Modifiable Software Advisors in a Virtual inquiry Support Environment. IEEE Proceedings of the 32nd Hawaii International Conference on System Science. White, B., Frederiksen, J., Frederiksen, T., Eslinger, E., Loper, S., & Collins, A., (2002). Inquiry Island: Affordances of a Multi-Agent Environment for Scientific Inquiry and Reflective Learning. Proceedings of the Fifth International Conference of the Learning Sciences. Wong L.H., & Quek C., (1998). TAP: A Software Architecture for an Inquiry Dialog-Based Tutoring System. IEEE Transaction on System, Man and Cybernetics 28(3) 315-325. Yoshikawa, Atsushi, Shintani, Masafumi, & Ohba, Yujiro, (1992). Intelligent Tutoring System for Electric Circuit Exercising. IEEE Transactions on Education, 35(3).

124

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

An Evaluation Tool for Collaborative Learning Based on Communication a

Noriko Hanakawaa, Nao Ikemiyaa Faculty of Management Information, Hannan University, Japan [email protected]

Abstract. An evaluation too for collaborative learning has been developed. The tool is based on communication which occurs during the collaborative work. The tool consists of two sub-tools: a monitoring tool and a scoring tool. The monitoring tool can evaluate automatically students’ behaviors during collaborative work. The scoring tool can score automatically output products such as reports. The two tools are established by natural language techniques: TF/IDF method, vector space model method. As a result of experiments, we confirmed that the tool was able to provide rough evaluation of individual student in the collaborative learning. In addition, we confirmed that the tool has a weakness in the evaluation’s accuracies when students copy & paste sentences from web-sites to their reports. Keywords: Collaborative learning, communication, monitoring, scoring, natural language technique

1. Introduction In education of universities, it is well known that collaborative learning is effective technique for acquiring new knowledge[3]. In this paper, the collaborative learning means that students achieve a target of the group with collaborative work and interactive work. Of course, the students should perform individual duty of the collaborative work in the group. Because the collaborative learning is based on cognitive science theory, the effectiveness of the collaborative learning is assessed by many experiments. However, there is a significant problem of the collaborative learning in universities’ education. The problem is accuracy of subject results in collaborative learning. To make the results of subjects in universities’ education, teachers have to determine individual result of the subjects in collaborative learning. The easygoing way of the evaluation of the collaborative learning is to evaluate an output product of the collaborative work such as reports, programs. One product’s evaluation is equal to all members’ results of the subject. In this case, the accuracy of individual student’s result may be doubt because the rate of achieving individual duty in the collaborative learning is not considered. Only one student may be do all work in spite of collaborative work, the other students may be do nothing. In addition, even if teachers require the students to apply the rate of achieving individual duty, the answers depend on subjective views of the students. The best way of the evaluation in the collaborative learning is to monitor students’ behaviors by teachers. The teachers

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

125

should objectively judge the contribution of all students during collaborative working. However, it is difficult for a teacher to monitor the behaviors of a large number of students. Therefore, we propose a new tool for supporting the evaluation of the collaborative learning. The tool consists of two functions: measuring of group's ability as a whole during collaborative work, and evaluation of output products such as reports. Both functions can automatically execute to evaluate without teachers' large efforts. The evaluations generated by the tool are useful when teachers determine final results in collaborative learning. Section 2 presents related works, section 3 describes the detail of the tool, experiments and discussions are described in section 4. Summary and future researches are shown. 2. Related works Although many researches collaborative learning such as environments, distance learning, and computed supported learning have been proposed, there are few proposals about individual student’s evaluation in collaborative learning. Elis et al. have proposed valuable equipment for evaluation in collaborative learning[1]. The equipment and the way are depended on some specific cases such as specific subjects and specific subjects. Because our tool can be adaptable to various cases in collaborative learning, out tool may be more useful than the equipment which has been proposed by Elis. Hazeyama et al. have proposed a group organization system for group learning[2]. The system is valuable to acquire technical knowledge in collaborative work. However, the evaluation of the collaborative work is depends on the results of questionnaires to students. Because the questionnaires are based on the students’ subjective view points, the evaluation may be subjective results of subjects in collaborative learning. 3. The evaluation tool for collaborative learning 3.1 A basic concept of evaluation We think that the students in collaborative learning should be evaluated by not only output products such as reports and programs but also behaviors during collaborative work. If only output products are evaluated, all students’ results of the group will be equal to the results of the output products in spite that all students do not work equally. Teachers should evaluate the students’ results using both output products and behaviors during collaborative work. If teachers use two evaluation ways on different views in collaborative work, the evaluations will be able to supplement the demerits of the evaluation, each other. In addition, the evaluation using two ways should be executed automatically without teachers’ large efforts because teachers have to evaluate a lot of students, especially, in private universities in Japan. At least, data which was collected automatically by the tools should help the teachers make final results of many students. 3.2 An evaluation tool for collaborative learning

126

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

Group

Output Product (Reports)

Executing

Communication

Evaluation tool

Monitoring Tool

Group ability

Scoring Tool

Best answer

Evaluation of output products

Teacher

Figure1 An overview of the evaluation

Figure2 An image of the monitoring

According to the above concept, the evaluation tool consists of two functions. One is a monitoring tool, another is a scoring tool. An overview of the evaluation tool is shown Figure 1. When a group executes collaborate work, communication logs among students of the group are collected automatically to the monitoring tool. The group ability which presents the group state such as normal or confusion is calculated in real time in the monitoring tool. Therefore, the teachers can know the change of group state in time series. On the other hand, the scoring tool can calculate automatically results of output products such as reports using natural language techniques like TF/IDF method, and vector space models. The teachers can be supported to make individual student’s evaluation in the tools. 3.3 The monitoring tool At first, the concept of the monitoring tool is explained. The monitoring tool is to monitor students’ behaviors using communication during collaborative working. Communication logs are accumulated automatically in the tool. Group ability is calculated from the variance of the communication occurrence time about one topic. That is, when communication about a specific problem continues to a long period because a problem is not resolved, the variance of communication occurrence time about the problem is large. The detail theory is described in [4]. We think that communication is a most important linchpin of collaborative work. Good communication leads to good products, poor communication leads to confusion of work. Because the monitoring tool is embedded in the evaluation tool, teachers can grasp roughly what is happening during collaborative work. If the group ability calculated in the monitoring tool is low, the collaborative work may be executing under wrong situations. For example, only one student executes all work without the others’ collaboration, or the group is falling into confusion because of poor communication. Figure 2 shows the monitoring tool. Red circles in the monitoring tool means clusters of one communication topic, that is, one problem. A big red circle means continuous communication because the problem is not resolved smoothly. A small red circle means quick communication because the problem can be resolved quickly. In addition, because the monitoring tool can provide numerical values of group ability,

127

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

㪈㪉 㪇 㪏 㪍 㪇 㪋 㪇 㪉 㪇 㪉㪇 㪇

㪊㪇

㪋㪇

㪌㪇

㪍㪇

Figure4 Experiments’ results of Figure3 An image of the scoring the scorin tool teachers can compare easily the state of the collaborative work among groups. In addition, the monitoring tool has been evaluated using open source projects’ communication. We confirmed that the values of group ability can indicate project states such as confusion, and normal. The detail of the experiments is shown in [4].

3.4 The Scoring tool Next, the scoring tool is explained. The scoring tool’s purpose is an automatic scaring of reports of text files. If collaborative learning is executed in universities’ education, output products such as reports and programs occur in the collaborative work. Teachers should score the output products quickly. However, the scoring the reports of large volumes consumes the valuable time of the teachers. Therefore, the scoring tool supports the teachers to score reports of large volumes. Figure 3 shows the scoring tool’s image. The mechanism of the scoring is calculation of similarities between a best answer and student reports. If the report is similar to the best answer report, the report will be evaluated to high score. The calculation of the similarity between two reports is based on famous natural language techniques: TF/IDF method[6], and vector space model[5]. TF/IDF method provides index terms which are distinguished the best answer report from the other various documents in a document space. The multi-dimensions vector space model using the index terms is achieved. The inner product between two vectors: the best answer report and the other report in the multi-dimensions vector space model, means the similarity between the two reports. If the inner product is large, the report is not similar to the answer report, that is, the score of the report is low. In addition, the scoring tool also supports useful functions. The first useful function is that additional index terms can be set to the tool. According to the similarity’s calculation mentioned above, the index terms of the answer reports are generated automatically. However, sometimes teachers would like to add manually special keywords to the evaluation of the reports. The function of adding special keywords is supported. Next, the scoring tool supports a function of detecting injustice reports. Sometimes, students submit reports which are copied from friends’ reports. The title and some parts of the friends’ reports are changed to make the injustice reports. Teachers sometimes are troubled with such injustice reports. The trouble is not only difficulty of finding original reports which are not copied, but also wastage of time for detecting the injustice reports for teachers.

128

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

Figure6 Variation of the group Figure5 Communication logs in ability by the monitoring tool experiments Therefore, a function for detecting injustice reports is embedded to the scoring tool. The function is based on comparing the similarity among all student reports. If some reports are too similar, the tool regards the reports as injustice reports. In all cases, the final score of the reports should be determined by teachers. The scores generated by the scoring tool help the teachers judge the final scores of the reports. Figure 4 shows the results of experiments for accuracy of the scores that are generated by the scoring tool. The x-axis of Figure 4 means scores that are generated by the scoring tool. The y-axis means scores that are determined manually by a teacher without the scoring tool’s support. The correlation coefficient between the automatic scores and the manual scores is 0.79. Therefore, when the score of a report is extreme low, the teacher can eliminate the report from the evaluation task because possibility of irrelevant report is high.

4. Experiments 4.1 Experiments’ conditions and results To evaluate the usefulness of the evaluation tool including the monitoring tool and the scoring tool, we held experiments. The conditions of the experiments are as follows; (1) Subjects are four groups that consist of two persons, respectively . (2) Two subjects execute one collaborative work for making a report. (3) The theme of the report is “an information-oriented home”. (4) Some keywords for investigating “an information-oriented home” are provided to the subjects. The keywords are “multi-media home computing”, “appliance”, “AMIDEN architecture”, “human interface”, “middleware for electric equipments at home”, “home network”, “ubiquitous computing”. Table 1 Results of the experiments

Group1 Group2 Group3 Group4

Score by Scoring tool 36 46 34 42

Manual 75 60 60 65

Group Ability by Monitoring tool 10.9 9.7 9.0 15.0

Manual 8 10 11 12

Total result By tool 3.3 4.7 3.7 2.8

Manual A B+ B B

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

129

(5) Two subjects of a group communicate using only MSN messenger[7] without direct conversation. (6) Maximum time for making the report is 8 hours. (7) The way of investigating the theme is various: searching web sites, collecting books and article, newspapers. (8) We regard the special issue’s articles of Journal of IPSJ(Information Processing Society of Japan) as a best answer report about the theme. An example of the communication logs that were collected by MSN messenger in Japanese is shown Figure5. The communication logs include “sequential number”, “communication occurrence time”, and “communication content”. Using the “communication content”, each cluster which presents one topic was generated in the monitoring tool. The size of the cluster which presented a red circle in Figure 2 is calculated based on the communication occurrence time. The cluster’s size means the group ability. After that, the variation of the group ability was plotted as Figure 6. Figure 6 shows variations of the group abilities that have been calculated by the monitoring tool on time series. High values of the group ability mean more confusion state of the group, low values mean stable state of collaborative work in the group. The group ability of Group 4 kept high for long periods, that is, we can expect that Group 4 did not do the collaborative work smoothly. Group 2 was relatively stable during experiment’s period. The experiments’ results show to Table 1. There are three columns: “Score”, “Group ability”, and “Total result”. To compare the values calculated by tools and manual results by a teacher, the three columns include two sub-columns: “by tool” and “manual”, respectively. The comparing two values: “by tool” and “manual” is discussed in the following sub-section “4.2 Discussion” in this paper. Total result of Group2 is highest(4.7), Group4’s total result is lowest(2.8). In addition, the group abilities by the monitoring tool present states of the groups during the experiment periods, the scores by the scoring tool present evaluations of the output products of the reports about “an information-oriented home”. The high score by the scoring tool means a good report, the high group ability by the monitoring tool means no-good state of group such as confusion state. Please take care the different meanings between the two high values: the score and the group ability. The two values (Score and Group ability) can provide the different views in evaluating the collaborative work. For example, although the score of the report of Group 4 is high(42), the group ability is worse (15.0) than the other group abilities(10.9, 9.7, 9.0). We can expect that the collaborative work was not stable but the subjects of Gourp4 were able to make a good report. Therefore, the total result of Group4 is lowest. 4.2 Discussion In this section, we evaluate the accuracy of the values such as the scores and the group abilities, and total results in the experiments. To evaluate the accuracy of the experiments’

130

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

results, we analyzed all communication logs and the output reports. The sub-column named as “manual” presents scores and group abilities that have determined manually by a teacher. At first, we discuss the accuracy of the scores calculated by the monitoring tool. The most different point between “by monitoring tool” and “manual” is the score of Group1. Although the automatic score of Group1 is “36”, the score by the teacher is highest. That is, the teacher decided that the Group1’s report is best. However, the automatic score is the second lowest value. We investigated the reports in detail. The report of Group1 has a feature that is different from the other reports. The other reports were described in itemizations. We provided keywords shown in section 4.1 (experiment condition(4)). Therefore the subjects described the explanations according to each keyword in itemization. Especially, we were not able to understand the conclusion of the reports of Group2 and Group3 because the explanations according to keywords simply lined in the reports. In contrast, the report of Group1 had good conclusion using no itemization. The sentences as a whole were logical, and consistent. This is limitation of the TF/IDF method and vector space model. The logical sentences and consistency can not be checked by the scoring tool. However, the scoring tool should check whether the reports have conclusions. At least, if the reports consist of only itemizations, the scoring tool should calculate the low value as the report’s score. In addition, we found that executing “Copy & Paste” sentences from web-sites leads to high score. In the case of Group4, many sentences were copied and pasted directly from websites. Although the consistency among the sentences in the report was not good, the report’s score was calculated to high. Next, we analyze the group abilities. The group abilities of Group1, Group2, and Group3 by the monitoring tool are almost same, although the group ability of Group4 is very high. When we checked the communication logs of Group4, we found two significant problems in the communication logs. The first problem is that the subjects executed non-continuously the experiment. That is, the subjects of Group4 executed a part of the experiment on the morning at home. After that, the subjects went to their school. On the evening when the subject came back, the experiment was re-started at home. The time lag like this large interruption leads to miss-calculation of the group abilities. The other groups did not have such large interruption of the experiments. The second problem is that the communication included many meaning-less chats such as food topics, leisure topics. The meaning-less chat leads to a low value that is calculated by the monitoring tool. We think this calculation is correct because too many meaning-less chats occur in no-good state of collaborative work. In this case, the monitoring tool can calculate the correct value of the group ability Table 2 Comparing the expected results with results by the tools Condition of group and report Executing “Copy & Paste”

Expected results Low

Itemizing

Low

Having chat Not resolving problems

Low Low

Making good consistency as reports

High

Sharing work well

High

Results by Monitoring tool High

Results by Scoring tool High High

Low Low Not available High

N. Hanakawa and N. Ikemiya / An Evaluation Tool for Collaborative Learning

131

We summarize the accuracy of the results calculated by tool. Table 2 shows the summary of the accuracy. The column named “Expected results” means the values which we expect. The columns named “Result by Monitoring tool” and “Result by Scoring tool” present values that were confirmed in this experiments. If group’s condition is under “Having chat”, “not resolving problems”, the tool will calculate low values as no-good state of group and no-good product. If group’s condition is under “Sharing work well”, the tool will calculate high value as good state of group by the monitoring tool. However, when group’s condition is under “Executing Copy & Paste”, “Itemizing”, the tool was calculated high values although the expected results are low. In addition, the report’s condition of “good consistency as reports” was not able to be reflected to the values as mentioned above. At least, we can solve the miss-matches under “Executing Copy & Paste” and “Itemizing”. The problem according to “Executing Copy & Paste” will be solved by comparing automatically reports with websites’ sentences. Although the computational complexity of the similarities between reports and websites sentences is large, we will able to check the “Executing Copy & Paste”. The problem according to “Itemizing” will be solved by adding a function to the scoring tool. A function is to check reports’ structures such as “Introduction”, “Conclusion”. The function will be implemented easily to the scoring tool. 5. Summary We have developed an evaluation tool for collaborative learning. The tool consists of the monitoring tool that can calculate group state during the collaborative work, and the scoring too that can calculate scores of output products. As a result of the experiments, the tool’s features were clarified. In future, the tool will be renovated regarding the tool’s weaknesses that were discussed in section 4.2. References [1]Ellis, T.J., Hafner, W. “Peer Evaluations of Collaborative Learning Experiences Conveyed Through an Asynchronous Learning Network”, HICSS '05. Proceedings of the 38th Annual Hawaii International Conference on, 03-06, pp 5b- 5b, 2005. [2]HAZEYAMA,N., SAWABE,S., KOMIYA,S., “Group Organization System for Software Engineering Group Learning with Genetic Algorithm” IEICE Vol.E85-D No.4 pp.666-673 2002. [3] Slavin, E. Robert. Cooperative Learning: Theory, Research, and Practice. Needham Heights, MA: Ally and Bacon(1995). [4] Noriko Hanakawa, Kimiharu Okura, “A project management support tool using communication for agile software development”, Proc. of the11th Asia-Pacific Software Engineering Conference (APSEC2004), pp316-323, Dec. 2004. [5] Salton,G. and McGill, M. J.:"Introduction to Modern Information Retrieval",McGraw-Hill(1983) [6] http://mikilab.doshisha.ac.jp/dia/research/report/2004/0812/001/report20040812001.html [7] http://messenger.msn.co.jp/

132

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Adaptive Learning in a Mobile Environment Tat Wai Ho, Vincent To-Yee Ng1, Stephen CF Chan, King Kang Tsoi2 Department of Computing, The Hong Kong Polytechnic University, Hong Kong. [email protected], [email protected] 2

Abstract. Mobile devices are getting more and more popular. With the advanced wireless technology, students can assess Internet with their PDAs. This study aims to develop an adaptive e-learning system on PDA. The idea is to distribute learning materials to the students adaptively. The adaptive engine is based on two levels of Bayesian networks. The first level measures the student performance and the second level determines the redistribution of materials when a student needs retaking. We divide the student performance into the study performance and the quiz performance. The quiz performance is measured by a computerized adaptive testing (CAT) while the study performance relates to the learning behavior of a student. Preliminary experiments demonstrated that student performance has been improved via the adaptive system. Keywords: M-learning, intelligent tutoring systems, Bayesian network, adaptive testing

Introduction Nowadays, people use PDA (Personal Digital Assistants) to read and send e-mails, read e-books, take notes in class, play games, and store documents. The mobility of PDA makes it suitable for students to learn everywhere. Additionally, most new PDA models are integrated with wireless capabilities by which the owner can access teaching materials via the Internet. Most e-learners complain about a "one-size-fits-all" philosophy, a resulting cognitive overload and consequently the lack of personalization of existing applications [1]. It means that the system should be equipped with artificial intelligence that can monitor the study progress of students and provides adaptability. Learning through computer provides an edge over learning in class. In traditional classroom teaching, a teacher cannot tailor the contents and pace for each student. Instead, an Intelligent Tutoring System (ITS) can tailor to the needs of individual student. Adaptive learning is considered to be a critical success factor for e-learning systems. It enables students to select their modular components so as to customize their learner-centric environments. Most intelligent tutoring systems are characterized by a knowledge base, a tutoring strategy and, finally, a student model [2]. Adaptive learning can be achieved by 2 steps: 1. Learning diagnosis: Test a student using verified measuring theory, then estimate the student’s mastery degree of the teaching content. 2. Select teaching contents according to the student’s learning performance. For example, a long studying time of a page classifies that the content is too difficult for a particular student. Another page of simpler level is then selected.

1. System Design In order to enhance the learning efficiency and deal with the limitations in mobile devices,

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

133

teaching materials can be selectively distributed to students according to their learning performance. Here, we consider that the materials of a course are accessible on the web. A student can move forward and backward with buttons or select a particular page in the content page. (See Figure 1 and 2)

Fig. 1

A Learning Page

Fig. 2 Content Page which allows student to read a chapter actively

In our design, the learning performance of the students is assessed by Bayesian networks. Students can continue to study the next chapter if they have already mastered most of the important concepts in the current chapter. On the other hand, students need to retake a chapter if they have problems in understanding it. They are asked to re-visit (retake) some of the materials. Additional study materials may also be distributed to enhance their learning during chapter retaking.

1.1 Bayesian Network (Chapter Level) The chapter-level Bayesian network is shown in the Figure 3. This network is used to assess a student’s learning performance of a chapter comprehensively. The assessment is not based on one quiz result solely. A student may get a good result because of luck or the test is too easy. Also, a student may work very hard but still get a bad result when the test is too difficult. The classification of learning performance, therefore, is based on 6 various aspects. They are: 1. Review Score – revisiting time compared with the first visiting time, if any. 2. Content Score – number of times students clicked the “Difficult” button. 3. Reading Time – average reading time of students in that chapter. 4. Reading Strategy – whether the student learns actively or passively in a chapter. 5. Quiz Criterion Reference Result – to classify the student’s quiz result by his actual score. 6. Quiz Norm Reference Result – to classify the student’s quiz result by comparing it with his peers.

134

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

Figure 3 shows the casual relationships between the nodes in the network. For example, the classification of learning performance is conditionally dependent on the study performance and quiz performance. The study performance depends on the content difficulty, reading time, and reading strategy. The quiz performance depends on the results of content difficulty, quiz criterion reference result, and quiz norm reference result. The content difficulty is conditionally dependent on results of content score and review score.

Fig. 3

Bayesian Network (Chapter Level)

Due to the problem of having no previous data, proportional estimation techniques are used to estimate the probability of different states of the nodes in the network. When performance data exist, the classification of nodes should be calculated by the conditional probability of the preceding nodes. As shown in Figure 4, when a student finishes studying of a chapter and the quiz, the chapter-level Bayesian network determines the learning performance. If the learning performance is classified as “Adequate”, then he will move on to next chapter (Figure 5). Otherwise, he needs to re-visit that chapter again until his status is classified as “Adequate” (Figure 6).

Fig. 4 Students need to retake a chapter if the Learning Performance is classified as “Inadequate”

1.2 Bayesian Network (Page Level) When a student has to retake a chapter, the student needs not study the whole chapter again. It is because the student may have already mastered some concepts when he/she first reads the chapter. A page-level Bayesian network is used to determine which pages need to be redistributed to students (in Figure 7). Besides, additional references and examples may also be distributed to students in order to help them to understand the important concepts more clearly.

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

Fig. 5 Next Chapter is Shown

135

Fig. 6 Redistribution List

Fig. 7

Bayesian Network (Page Level)

Figure 8 shows the flow of redistribution. During chapter retaking, the system examines each page of the chapter by the network. Only the pages in the redistribution list are then shown to the student again.

Fig. 8 To add pages to Redistribute List that are classified as “Yes” in Redistribution node

136

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

1.3 Quiz Management System Apart from the two-level Bayesian network, a second major component in this adaptive learning system is the QMS (Quiz Management System). QMS is a CAT (Computerized Adaptive Testing) system which selects questions from the test based on the skill level of examinees. The prepared questions are stored in a question bank and can be reused. This system can provide adaptive tests to students in which different questions are selected according to their ability. IRT (Item Response Theory) is used to estimate their ability and then suitable questions would be selected from the question bank. IRT is a statistical framework, in which students are assumed to have different abilities on a unidirectional continue ranging from low to high ability. QMS deploys the three-parameter IRT logistic model to do the estimation. The three parameters are the discrimination parameter, difficulty parameter, and guessing parameter. The discrimination parameter indicates to what degree the question is able to distinguish student‘s ability. The difficulty parameter is the point on the ability scale where the probability of a correct response is 0.5. The guessing parameter incorporates the guessing factor on selecting answer into the model. [3] Figure 9 shows a screenshot of the tool to create question. At the end of a chapter, the student is prompted to start a quiz (Figure 10). A quiz description page is presented to student to brief about the information of the quiz (Figure 11). Figure 12 shows an example of a question. After completing the quiz, a quiz result page is shown to inform the student about the quiz result (Figure 13).

Fig. 9 To create Questions in QMS

The advantage of adaptive tests over the traditional test is that students do not need to spend time and effort attempting to answer those questions that are too easy or too difficult to them. Situations of students facing bored or fatigue can be minimized. The main function of QMS in this system is to test students using the testing items verified by IRT, then estimate students’ ability and the mastery degree of the knowledge according to their reaction.

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

Fig. 10 The End of a chapter

137

Fig. 11 Quiz Description Page

Fig. 12 The first question of the quiz

Fig. 13 Quiz Result Page

1.4 System Implementation All the teaching materials are stored as XML documents. The documents are parsed by a DOM parser when there is a HTTP request received at the application server (Tomcat). JAVA servlet located in the application server sends all the related attributes such as chapter number and page number to the DOM parser. The DOM parser returns all the required teaching materials to the JAVA servlet program such as page heading, page title and teaching contents. Finally, the JAVA servlet program generates a HTML page and sends to PDA pocket IE. (See Figure 14)

Fig. 14 System Architecture

138

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

2. Experiments and Results 15 students used this system to learn JAVA programming. 5 of them are novice in programming, 3 of them have learned other programming languages before. For the remaining 7 students, they have experience in writing JAVA programming language. Each student performed the experiment individually. When the student is classified as “Inadequate” for a chapter, he/she has to retake parts of the materials. Table 1 shows the number of retakes in each chapter. Chapter 1 2 3

Table 1 Number of Retake(s) per Chapter No. of students take the chapter No. of students retake that chapter 15 11 out of 15 15 10 out of 15 9 6 out of 9

Table 2 and 3 contain the statistics of the improvement of study performance and quiz performance after each retake. As shown in Figure 15, the study performance and quiz performance of most students can be improved during chapter revisiting. It proves that the system can accurately pinpoint the parts that students have not yet mastered or misunderstood and provide those teaching materials for revisiting. Besides, all students can pass through a chapter within 3 revisits. The evidence shows that the system can provide teaching materials to students adaptively and helps them to enhance their study efficiency. Chapter 1 2 3 Improvement

Table 2 Improvement of Study Performance 1st Retake 2nd Retake 7 / 11 9 / 11 8 / 10 8 / 10 3/6 4/6 18 / 27 (66.7%) 21 / 27 (77.8%)

3rd Retake 11 / 11 10 / 10 6/6 27 / 27 (100 %)

Chapter 1 2 3 Improvement

Table 3 Improvement of Quiz Performance 2nd Retake 1st Retake 8 / 11 10 / 11 6 / 10 7 / 10 6/6 6/6 20 / 27 (74%) 23 / 27 (85.2%)

3rd Retake 11 / 11 10 / 10 6/6 27 / 27 (100 %)

Fig. 15 Improvement of Student’s Study Performance and Quiz Performance

139

T.W. Ho et al. / Adaptive Learning in a Mobile Environment

A snapshot of two students’ performance data is provided in Table 4 to demonstrate how the system can track their learning progresses. Student A is a novice in programming and Student B has 1 year of Java programming experience. Table 4 Examples of Bayesian Network Data of 2 Students Student A Student B Review Score Content Score Content Difficulty

3/12 = 0.5 7/12

Moderate Little mastered Difficult

Reading Time (words/minute) Reading Strategy Study Performance

156

Insufficient reader 220 Passive Bad

Average reader Active Good

Quiz Criterion Reference Result Quiz Norm Reference Result Quiz Performance

2/8 = 25%

Fail D Fail

Above average A Excellent

Learning Performance of Chapter 1 Inadequate Remarks: There are 12 pages in Chapter 1 and 8 questions in Quiz 1.

1/12 = 0.083 3/12 = 0.25

6/8 = 75%

Rare Partial mastered Easy

Adequate

3. Conclusion A Web-based adaptive learning system is built for the use in a mobile environment. With the wireless Internet technology, students can access the teaching materials on a PDA anywhere. A two-level adaptability is designed to facilitate the learning process of the student. The chapter-level Bayesian network assesses the student’s learning performance. If the learning performance is classified as “Inadequate”, they will need to retake that chapter. The page level Bayesian network decides which teaching materials need to be redistributed to the student. Experiments show that the system is able to enhance the study performance and quiz performance of students. It indicates that the chapter-level Bayesian network can assess the learning performance accurately and the page-level Bayesian network can select teaching materials to students adaptively.

References [1] Blochl, M., Rumetshofer, H. and Wos, W. (2003). “Individualized E-Learning Systems Enabled by a Semantically Determined Adaptation of Learning Fragments”, 14th International Workshop on Database and Expert Systems Applications (DEXA'03), Prague, Czech Republic. [2] McCalla, G.I. (1992). “The search for adaptability, flexibility and individualization: approaches to curriculum in ITS”, in Adaptive Learning Environments: foundations and frontiers, NATO ASI Serie F, vol. 85, pp. 91-121, Berlin: Springer-Verlag. [3] Cooper, C. (2002). Individual Differences, 2nd Editions, London: Arnold.

140

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Conceptual Changes in Learning Mechanics by Error-based Simulation Tomoya HORIGUCHIa, Tsukasa HIRASHIMAb, Masahiko OKAMOTOc a Faculty of Maritime Sciences, Kobe University, Japan b Deptartment of Information Engineering, Hiroshima University, Japan c Department of Human Sciences, Osaka Prefecture University [email protected] Abstract. This paper reports on the experiment and its results which was conducted to examine what conceptual changes are caused by using Error-based Simulation (EBS) in learning mechanics. The EBS simulates the motion of mechanical systems based on students’ erroneous ideas to promote the awareness and correction of their errors. Ninety undergraduate students learned mechanics by solving problems and also worked on the pre-/post-tests. Two-thirds of them used EBSs and the rest didn’t. One-third of them were interviewed about their problem-solving. The results were as follows. First, for novice students, the EBS significantly increased their performance in problem-solving and its effect was much greater than the one of interviews for facilitating students’ reflection. Second, the effect of EBS was sufficient in simple problems, but insufficient in more complex problems especially for the correction of errors. Third, the effect of EBS on expert students was unclear in this experiment. Finally, though the effect of EBS on the transfer of learning wasn’t clarified sufficiently, it was suggested that the EBS promoted the deeper understanding toward the transfer which was beyond merely providing the explanation of correct solutions. From these results, we concluded the EBS was useful to cause some conceptual change in learning mechanics. Keywords: Conceptual change, Cognitive conflict, Mechanics, Error-based simulation

1. Introduction Students’ making errors is a great opportunity for learning [9]. The errors would clarify the fact that their knowledge is incorrect/insufficient, and suggest what to learn for making the correct solutions. It is, however, very difficult for students to utilize this opportunity because they can hardly recognize what the errors really mean. Insufficient recognition often makes them ignore the errors or patch their knowledge ad hoc. It is necessary to help students at this point. It is insufficient to simply tell them their solutions are wrong and give the correct solution. From such ‘instruction,’ students cannot understand why their solutions are wrong. They would swallow it without a deeper understanding and make the same errors again. In other words, this understanding is so superficial that it doesn’t conceptually change students’ misconceptions. In learning science, especially, students often have rigid preconceptions which block them from acquiring the scientific (correct) concepts. Therefore, a lot of methods for causing conceptual change have been proposed [10, 11] and some promising ones insist the following procedure is useful: (1) to make students be explicitly aware of the fact that their ideas (misconceptions or their results) are contradictory to some facts (2) to provide them with the explanation about the correct concepts or solutions The role of step (1) is to make students ready to accept the correct concepts/solutions provided in step (2). This step, in fact, tries to cause cognitive conflicts in students’ mind. Cognitive conflicts are the explicit representation of contradiction between students’ ideas and some facts, and have been proved to be useful to facilitate students’ conceptual change [2, 4]. It is, however, also well-known that cognitive conflicts are difficult to cause [3, 8]. Unless the facts which are contradictory to students’ ideas are meaningful and acceptable

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

141

for them, they often ingore or refuse (or only limitedly accept) the facts [1]. Insufficient cognitive conflicts in step (1) would hardly cause the conceptual change in step (2). Therefore, various forms of educational feedback have been proposed which provide the facts contradictory to students’ ideas in a meaningful and acceptable manner. The most popular way is to map the (abstract) concepts to the concrete world more familiar to students. For example, Takagaki [10] used the examples in which a mathematical concept ‘height’ was used in the level of daily experiences. Computer simulations of physical phenomena give another example in which physical concepts are embodied in concrete contexts [12]. These methods, however, have a latent limitation, we think. All of them provide students with ‘negative feedback’ in which the correct concept/solution must be given externally to cause the contradiction to the misconceptions. For example, even in the familiar world, the correct use of the concept ‘height’ is shown independently of students’ misconceptions. Even in concrete contexts, the correct physical phenomena are shown independently of students’ erroneous expectations. There is always the possibility that they don’t accept the (externally) given facts and cannot get intrinsic understandings. Error-based Simulation (EBS) [5] is a method for providing students with ‘positive feedback,’ which would be the alternative or complement to the methods above. It shows students what would happen if their (erroneous) solutions were correct. The results of erroneous ideas would often be unexpected ones which encourage students to reflect on why their solutions are wrong. For example, in scientific experiments, erroneous hypotheses produce surprising phenomena. Students wouldn’t be able to accept such results because the results contradict their ‘common knowledge’ (we assume they have the precedent knowledge required for this judgement). Then, they would intrinsically recognize their errors and be ready for conceptual change. The EBS is especially suitable for the learning domains in which there are rigid preconceptions to be conceptually changed and/or the correct phenomena are familiar to students (for the comparison with EBSs). Therefore, mechanics is the typical domain in which the EBS could be used effectively. However, there have never been any computer-simulation like EBS, that is, the simulation which shows unnatural and/or impossible phenomena. For example, if a student set erroneous equations of motion, a block might fall ‘upward’ or if she/he thought gravity was the only force acting on a book on the desk, it would sink into the desk! While the preliminary experiments proved the EBS to be educationally effective [5, 6, 7], the effectiveness of EBS hasn’t so far been quantitatively measured in the real learning processes. We, therefore, made such an experiment. In this paper, we report on the experiment and its result which was conducted to examine what conceptual changes are caused (or aren’t caused) by using EBSs in learning mechanics. It was planned to examine the effect of EBS on the microscopic change in expertise and on the transfer of learning. 2. Error-based Simulation [5] Before describing the experiment, we present some examples of EBS. Figure 1 shows the problems of basic mechanics. Suppose a student is asked to draw all the forces acting on the objects. If she/he has some misconceptions, she/he would draw incorrect forces (or wouldn’t draw correct forces). In this case, the set of drawn forces is her/his hypothesis and the EBS simulates the motion of the objects based on it. If the normal force acting on the block in normal-force-problem (Figure 1 (a)) wasn’t drawn (it is the very common error), the block would sink into the floor (Figure 2 (a)). If the reactive force acting on the dolly in tension-problem (Figure 1 (b)) wasn’t drawn, the dolly would go rightward faster than usual, so the string would stretch (Figure 2 (b)).If the gravity and its slope-parallel component acting on the block in component-force-problem (Figure 1 (c)) were simultaneously drawn, the block would sink into the slope (Figure 2 (c)). Such ‘impossible’ phenomena would strongly motivate students to reflect on their problem-solving1. 1

Not all of EBSs show the ‘impossible’ phenomena. There are errors based on which the erroneous motion is differ only a little from the correct one (suppose a student drew the slope-parallel force only in

142

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

(a) Normal-force-problem

(c) Component-force-problem

(a) Normal-force-problem



(b) Tension-problem

Figure 1. Mechanics Problems

 (b) Tension-problem (c) Component-force-problem Figure 2. Examples of EBS

3. Experiment 3.1 Purpose The experiment was made to examine whether the EBS cause some conceptual change in learning mechanics, and whether the effect of EBS interacts with students’ prior knowledge. We assumed that such conceptual change improves both the microscopic problem-solving performance of mechanics and the transfer of the learning, and that these improvements are better than the one by non-EBS methods. The improvement of the performance was measured by the change of students’ problem-solving scores in the learning session, and the improvement of the transfer was measured by the change of the scores in pre/post tests which were given before/after the learning session. In addition, the protocol of interviews in the learning session was analyzed to more qualitatively examine the conceptual change. Therefore, we had three conditions: EBS-with-interview-condition, EBS-condition , and control-condition. In order to measure the interaction between the effect of EBS and subjects’ prior knowledge, we had two subject groups; novices and experts, and compared their scores. 3.2 Subjects Ninety undergraduate students participated in the experiment. The half of them are majoring in science/engineering (expert group) and the other half of them aren’t (novice group). In each group, subjects were divided into three subgroups (15 students for each) which are component-force-problem). Though the mechanisms for managing EBSs based on the qualitative estimation of such differences have been proposed [5, 6, 7], we didn’t use them in the experiments described in this paper.

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

143

assigned to EBS-with-interview-condition, EBS-condition , and control group respectively. 3.3 Task and Equipment In the learning session, three problems of basic mechanics were used (Figure 1), which were shown and worked on by the subjects on the computer system. In each problem, a mechanical system was illustrated on a computer display and a subject was asked to draw all the forces (as arrows) acting on the objects by mouse operation. Then, the motion of the objects was simulated assuming that all the forces she/he had drawn were acting on them and the simulation was shown to the subject. Before/After the learning session, the subjects were asked to work on pre/post tests on paper. Pre- and post-tests were the same which consisted of 2 problems: slope-problem and slope-and-pulley-problem (Figure 3). In order to solve the former, it is necessary to use the concepts of normal force, tension and component force in an integrated way. In the latter, the change of direction of force by pulley was added to the former. (Problem-1) The dolly M1 and block M2 (which are linked each other by a massless string) are on a frictionless slope. They are being drawn upward by an external force. Draw all the forces which are acting on the dolly M1 and block M2.

(Problem-2) The dolly M1 and block M2 (which are linked each other by a massless string) are on a frictionless slope. The dolly M1 is linked to the block M3 by another (massless) string via a (frictionless) pulley. Draw all the forces which are acting on the dolly M1, block M2 and M3.

(a) Slope-problem (b) Slope-and-pulley-problem Figure 3. Mechanics Problems in Pre-/Post-Tests 3.4 Procedure The experiment was conducted in the order of pre-test, the learning session, and post-test. In the learning session, the following procedures were followed by each subject individually. The procedure of each condition consisted of three sequential subsessions, which were for normal-force-problem, tension-problem, and component-force-problem respectively. In each subsession of EBS-with-interview-condition, a subject made the first solution of the problem, saw the EBS based on it, interviewed by the experimenter, and then made the second solution (and saw its EBS). At the end of each subsession, the correct solution was explained to her/him. In EBS-condition, the interviews were omitted. In the control condition, a subject made the solutions on paper. She/he was provided with no EBSs and only with the judgements whether her/his solution was correct or not. The interviews were also omitted. The interviews in EBS-with-interview-condition were conducted to facilitate a subject’s reflection on her/his problem-solving. They consisted of the following questions and the subject’s answers to them (which were recorded on MiniDisk). - Did your drawing forces go well? - In what points did/didn’t it go well? - In the next time, in what point are you to take care? 4. Results and Discussion 4.1 Results

144

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

(1) Results of Problem-Solving Scores Table 1 shows the numbers of subjects who made the correct/incorrect solutions in each condition and problem in the learning session. It includes 18 small tables of 2(before/after EBS) x 2(correct/incorrect solution), in some of which the number of subjects who made the correct solution after EBS is greater than the one before EBS. This bias was proved to be significant by chi-square tests in the following 3 novice group cases: EBS-with-interview-condition of normal-force-problem (chi-square(1) = 3.39, p < .10), EBS-condition of normal-force-problem (chi-square(1) = 7.33, p < .01) and EBS-condition of component-force-problem (chi-square(1) = 6.14, p < .05). There appeared no significance in any expert group cases. Table 1. Numbers of Subjects’ Correct/Incorrect Solutions in Each Condition (unit: persons) Problem Condition

EBS-withinterview

Novice

4 4

2 9

13

0

15

6

3

12

4

11

3

12

Second

4

11

3

12

First

11

4

11

4

4 9

First Second First Control

Expert

11

11 6

First Second

EBS

EBS-withinterview

Normal-force Tension Component-force problem problem problem correct incorrect correct incorrect correct incorrect

11

14

1

12

3

First

10

5

9

6

Second

13

2

11

4

First

10

5

Second

10

5

9 9

6 6

Second

EBS

Control

4 5

11

1 7

14

3 2

12 13

14 15

1 0

11 13

4

10

5 5

10

10 8

2

(2) Results of Pre/Post Tests The examination papers of pre/post-tests were marked by the criterion that each drawn force which was correct got 1 mark and each drawn force which was incorrect lost 1 mark. By this criterion, the full marks of slope-problem are 9 and the full marks of slope-and-pulley-problem are 13. Table 2 shows the averages of subjects’ marks in each conditions and in each tests. For the novice group, three-factor ANOVA of 3(condition: EBS-with-interview/EBS/control) X 2(test: pre-/post-) X 2(problem: slope/slope-and-pulley) was conducted. As a result, the main effect of test was proved to be significant (F(2, 41) = 176.53, p < .01), that is, the marks of post-test were higher than the ones of pre-test in any conditions or problems. In addition, the main effect of problem was proved to be significant (F(1, 41) = 254.71, p < .01), that is, the marks of slope-problem were higher than the ones of slope-and-pulley-problem in any conditions or tests. For the expert group, the same three-factor ANOVA was conducted and the main effects of test and problem were proved to be significant (F(1, 38) = 10.23; 1554.33, p < .01). In addition, since the interaction between condition and problem was significant(F(2, 38) = 5.13, p < .05), sub-effect test (Tukey’s HSD) was conducted to reveal that there is no difference between 3 conditions in slope-problem while the mark in EBS-with-interviewand EBS-conditions were greater than the one in control-condition in slope-and-pulley-problem (p < .01).

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

145

Table 2. Average Marks of Pre-/Post-Tests in Each Condition (unit: marks) Test & Problem

Slope-andpulley-problem

Slope-problem

Slope-andpulley-problem

3.40

6.33

7.33

11.00

2.20

4.47

6.80

10.00

Control

2.57

5.71

6.50

9.00

EBS-with-interview

7.87

11.80

8.80

12.80

EBS

7.47

11.27

8.67

12.67

Control

7.92

10.92

7.92

11.50

Condition EBS-with-interview

Novice EBS

Expert

Post-test

Pre-test Slope-problem

*Slope-problem: full marks=9, Slope-problem: full marks =13

Table 3. Numbers of Subjects in Each Category in Interview (unit: persons)

Conditions

Right SE Wrong SE

No SE

Right SR Wrong SR No SR

15

0

0

1

9

1

12

1

2

0

8

3

Componentforce-problem

15

0

0

5

6

0

Normal-forceproblem

15

0

0

1

0

0

Tensionproblem

13

1

1

1

2

1

Componentforce-problem

14

0

0

3

1

0

Normal-forceproblem

Novice Tensionproblem

Expert

Self-regulation

Self-evaluation

Categories

(3) Results of Interviews From the viewpoints of self-evaluation (a subject’s awareness of whether her/his solution was correct or not) and self-regulation (a subject’s awareness of how to correct her/his erroneous solution), the following 6 categories were set for the subjects’ responses in EBS-with-interview-condition. (self-evaluation) - right self-evaluation: a subject evaluated herself/himself (i.e., her/his solution) adequately. - wrong self-evaluation: a subject evaluated herself/himself inadequately. - no self-evaluation: a subject couldn’t evaluate herself/himself. (self-regulation) - right self-regulation: a subject found the right way to modify her/his erroneous solution. - wrong self-regulation: a subject found the wrong way to modify her/his erroneous solution. - no self-regulation: a subject couldn’t find the way to modify her/his erroneous solution.

146

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

The subjects were categorized into them based on the protocol of the interviews. Table 3 shows the numbers of subjects in each problem and in each category. It shows that most subjects were aware of the correctness/incorrectness of their solutions by EBS, but that few subjects were aware of the way to correct them. 4.2 Discussion (1) Analysis of Microscopic Problem-Solving Performance According to the results of problem-solving scores, first, the number of subjects who made the correct solution significantly increased only in EBS-with-interview- or EBS-conditions of novice group. There appeared no significant change in any control-conditions (of novice group). This fact suggests that EBSs have the effect of causing some kind of conceptual change in learning mechanics. Second, the effect of EBS was seen in normal-force- and component-force-problems, but wasn’t seen in tension-problem. This fact suggests that EBSs are sufficient for the awareness and correction of errors in simple problems, but insufficient in more complex problems (in such cases, EBSs should be used with other methods of instruction). Finally, the interviews didn’t significantly influence the change in numbers of subjects who made the correct solution. They, therefore, seems to have no effect on any conceptual changes. (2) Analysis of Transfer of Learning According to the results of pre-/post-tests, the average marks in post-tests were significantly greater than the ones in pre-tests in both novice and expert groups, which suggests that the learning during the learning session was transferred to the post-test. However, since there was no significant difference between the three conditions, the explanation of correct solutions seems to be sufficient to cause the transfer even without EBSs. (In spite of this, the fact that the average marks in EBS-with-interview- and EBS-conditions were significantly greater than the ones in control-condition in slope-and-pulley-problem of expert group suggests the EBSs’ potential for facilitating a deeper and transferable understanding.) The results of interviews suggest that EBSs have the great effect on the right self-evaluation, while they don’t have sufficient effect on the right self-regulation. As for the transfer, this experiment didn’t clarify the difference between the effect of EBSs and the effect of explanation of correct solutions. However, the great effect of EBSs on the right self-evaluation suggests the fundamental conceptual change which would been hardly caused by merely giving the explanation of correct solutions. One of our future works is to analyze these effects separately. (3) Interaction between the Effect of EBS and Students’ Prior Knowledge As for the results of problem-solving scores, the EBS had significant effect on novice group in solving simple problems, while it didn’t on expert group because the problems used in this experiment were too easy to see some effect on experts. However, the result of sub-effect test in pre-/post-tests scores suggests that the EBS has some effect even on experts in solving complex problems. 5. Concluding Remarks The experiments reported in this paper proved that the EBS causes some kind of conceptual change in the learning of mechanics. The effects of EBS can be summarized as follows: - For novice students, the EBS has the effect of causing the conceptual change which is beyond merely providing the explanation of correct solutions.

T. Horiguchi et al. / Conceptual Changes in Learning Mechanics by Error-Based Simulation

147

- The EBS promotes the adequate awareness and correction of errors, and its effect is much greater than the one of interviews for facilitating students’ reflection. - The effect of EBS is sufficient in simple problems, but insufficient in more complex problems (especially for the correction of errors). In such cases, EBSs should be used with other methods of instruction. - As for the transfer, though these experiments didn’t clarify the effect of EBS sufficiently, we got some evidences which suggest that the EBS promotes the deeper understanding toward the transfer. - For expert students, the effect of EBS is unclear from these experiments. Our future works are: (1) to examine the effect of EBS on expert students, (2) to examine the possibilities of combining EBSs with other instruntional methods in complex problems and (3) to analyze the effects of EBS and explanation of correct solutions on the transfer separately.

Acknowledgment This work is supported in part by NISSAN SCIENCE FOUNDATION and Grant-in-Aid for Scientific Research from the Ministry of Education, Culture, Sports, Science and Technology of Japan. References [1] Chinn, C.A. and Brewer, W.F. (1993). Factors that Influence How People Respond to Anomalous Data, Proc. of 15th Ann. Conf. of the Cognitive Science Society, pp.318-323. [2] Fujii, T. (1997). Mathematical Learning and Cognitive Conflict, In Japan Society of Mathematical Education (Ed.), Rethinking Lesson Organization in School Mathematics, Sangyotosho, Tokyo (in Japanese). [3] Fukuoka, T. and Suzuki, K. (1994). A Case Study on Lower Secondary School Pupils' Conceptual Changes through Demonstration of Counterexamples: The Concept of Dynamics in the Case of Roller Coaster Models, Bulletin of Society of Japan Science Teaching, Vol.35, No.2, pp.21-32 (in Japanese). [4] Gagne, E.D. (1985). The Cognitive Psychology of School Learning, Little Brown and Company. [5] Hirashima, T., Horiguchi, T., Kashihara, A. and Toyoda, J. (1998). Error-Visualization by Error-Based Simulation and Its Management, Int. Jour. of Artificial Intelligence in Education, Vol.9, pp.17-31. [6] Horiguchi, T., Hirashima, T., Kashihara, A. and Toyoda, J. (1999). Error-Visualization by Error-Based Simulation Considering Its Effectiveness-Introducing Two Viewpoints-, Proc. of AI-ED99, pp.421-428. [7] Horiguchi, T. and Hirashima, T. (2000). A Method of Creating Counterexamples by Using Error-Based Simulation, Proc. of ICCE2000, pp.619-627. [8] Nakajima, N. (1997). The Role of Counterevidence in Rule Revision: The effects of instructing metaknowledge concerning non adhocness of theory, Japanese Jour. of Educational Psychology, 45, pp.263-273 (in Japanese). [9] Perkinson, H.J. (1984). Learning from Our Mistakes -A Reinterpretation of Twentieth-Century Educational Theory, Greenwood Publishing Group. [10] Takagaki, M. (2001). Teaching Strategies for Inducing Conceptual Change in Students’ Preconceptions About Height, Japanese Jour. of Educational Psychology, Vol.49, No.3, pp.274-284 (in Japanese). [11] Takagaki, M. (2004). How Undergraduate Students Change Their Preconceptions About Forces, Japanese Jour. of Developmental Psychology, Vol.15, No.2, pp.217-229 (in Japanese). [12] Towne, D.M., de Jong, T. and Spada, H. (Eds.) (1993). Simulation-Based Experiential Learning, Springer-Verlag, Berlin, Heidelberg.

148

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Exploring Fundamental Issues about Problem-Oriented Hypermedia Michael J. JACOBSON, Ph.D. Learning Sciences Lab, National Institute of Education, Nanyang Technological University, Singapore [email protected]

Abstract. This paper discusses theory and research issues associated with the use of hypermedia technologies to support learning in problem-oriented pedagogical environments involving cases. After considering research issues with problemoriented learning related to knowledge transfer and conceptual change, a design framework is discussed for a hypermedia system with scaffolding features designed to support and enhance problem-oriented learning. An ongoing project using a new version of this hypermedia approach with special ontological scaffolding to explore conceptual change and far knowledge transfer issues related to learning advanced scientific knowledge involving complex systems is also discussed. Keywords: Hypermedia, problem-based learning, transfer, conceptual change, complex systems

Introduction Although there is extensive use of hypermedia-based systems in education, the research on the efficacy of these systems is mixed. Many earlier systems have been criticized for focusing on the technology rather than cognitive and learning issues and for being atheoretical [1]. Of course, there has been research documenting factors associated with educationally effective hypermedia systems (including new adaptive hypermedia approaches), but there continues to be theoretical and methodological criticisms of much of the hypertext and hypermedia literature [2-4]. Given research on educational hypermedia appears to have reached an impressionistic vista, how might work in this area advance? 1. Research on Problems with Problem-Oriented Learning There is particular promise for the use of hypermedia technologies to support learner centered pedagogical approaches that involve the use of cases and problems. Research has documented how problem-oriented learning with cases (POLC)1 can help learners achieve important learning outcomes that include improved problem solving and knowledge transfer to new situations in university, legal, and business education [5-8]. There has also been an extensive amount of research on the efficacy of project-based learning in medical education over the past 20 years [9, 10]. In addition, POLC types of approaches have been used to help younger students in middle and high school to learn difficult science and mathematics concepts [11].

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia

1.1.

149

Inert Knowledge and Fostering Transfer in Problem-Oriented Learning with Cases

However, there has also been recent research that has identified cognitive difficulties associated with POLC. For example, Gentner, Loewenstein, and Thompson [8] at Northwestern University have revealed what might be called the “soft underbelly” of problem oriented learning. They have empirically demonstrated that the use of cases and problems can vary widely in terms of their educational effectiveness. In an elegant series of studies, Gentner’s group has been investigating issues related to the “inert knowledge problem” using the target domain of learning advanced negotiation strategies by undergraduate and graduate students [8, 12]. This research has demonstrated that there can be dramatically different learning outcomes in terms of knowledge transfer depending on how students studied two cases about negotiation strategies. These two cases varied considerably in terms of their surface features but shared a common negotiation principle. Students in what might be called the “advice groups”2 who were instructed to study the cases separately and to propose a negotiation solution for each case scenario. In a transfer task, the advice students were much more likely to use a common “naïve” negotiation strategy of compromise rather than more appropriate and sophisticated negotiation strategies such as trade-offs or contingency contracts that had been studied during the training period. Put another way, studying the cases in an isolated manner resulted in the students constructing “inert” knowledge that they had difficulty in applying to a new situation. In contrast, students in the experimental groups were instructed to contrast and compare the two cases and to think about similarities between the cases. These students were significantly more likely to use the appropriate negotiation strategy on the transfer tests than the students who studied the cases separately. Gentner and her associates [8] propose a theory of analogical encoding3 to explain these findings that inert knowledge was the result of studying cases in isolation while knowledge transfer of difficult concepts resulted from contrasting and comparing different cases. Overall, these findings by Gentner and associates are very important as they empirically document how a pervasive instructional approach in POLC (i.e., cases are studied in isolation and rarely are the students asked to contrast and compare the cases) 4 can result in inert or compartmentalized knowledge. However, the major positive finding from their program of research is that a relatively simple change in the nature of the learning activities students are involved with—such as contrasting and comparing different cases—can lead to dramatic learning gains that include knowledge transfer. 2. Designing Hypermedia to Support Problem-Oriented Learning with Cases Given the increasing use of problem- and case-oriented pedagogies in pre-college, university, and professional education, as mentioned above, the learning sciences issues of transfer and conceptual change have implications for enhancing the quality of learning that might be possible when using problem and case pedagogies. But how might teachers interested in using POLC pedagogies also deal with fundamental cognitive learning issues such as transfer? One approach is to use the representational flexibility of hypermedia technologies to develop tools for POLC that scaffold cross- or inter-case learning and problem solving activities.

150

2.1.

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia

Knowledge Mediator Hypermedia: Tools for Problem-Oriented Learning

Research involving the Knowledge Mediator Framework has been exploring ways that hypermedia systems might be designed to support learning with problems and cases [15, 16]. Knowledge Mediator (KM) hypermedia systems provide a set of hyperlinked resources that support learning activities in which a student receives special types of scaffolding in order to enhance the learning of conceptually challenging knowledge. The framework is also intended to help address the need of instructional designers who must confront the often difficult challenge of bridging from general theory and research principles to specific design features for useable systems [1]. The original framework, briefly described in this section, was articulated for non-adaptive hypermedia technologies, and significant conceptual change and knowledge transfer outcomes have been found in research involving the use of those KM systems [16-18]. Representational Affordances of Hypermedia Text & Symbols Visual Imagery Animations Digital Video odels & Simulations

Knowledge-in-Context Modular Cases Conceptual Lessons Contextualized Explanations of Concepts

Learning Scaffolds Conceptual Representational Problem Solving Ontological

Learning Tasks Cases Conceptual Lessons Problem Solving Guided Conceptual Criss-crossing

Figure 1. Schematic of main the functional elements for Knowledge Mediator hypermedia.

The primary functional elements in KM hypermedia systems are represented schematically in Figure 1. Three of these elements—Representational Affordances of Hypermedia, Knowledge-in-Context, and Learning Scaffolds—are design features for the link-node organization of KM hypermedia systems. The fourth element, Learning Tasks, refers to specific types of learning activities that are optimized for these design features. The arrows in the figure are intended to depict the interconnected ways in which the design features reinforce each other. These KM features are based on a series of studies over the past several years [15-19], however, given space limitations, a detailed consideration of each of the design elements in Figure 1 cannot be provided here (the interested reader may consult Jacobson [15] for a recent discussion).

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia

2.1.1. Educational Hypermedia Complex Systems

and

Learning

Conceptual

Perspectives

151

about

A Complex Systems Knowledge Mediator hypermedia system has recently been developed for use in a new study of knowledge transfer and conceptual change. What is distinctive about this hypermedia research is the challenging nature of the content the students are being asked to learn—core conceptual perspectives from emerging scientific knowledge in the field of complex systems—and the inclusion of ontological scaffolding (explained below) in addition to conceptual scaffolding and other features of Knowledge Mediator systems. The study of complex systems focuses on how the individual elements or agents interact with each other and their environment based on often relatively simple rules, which in turn may self-organize and show emergent and complex properties not exhibited by the individual elements. The core conceptual principles of complex systems, including new computational ways of doing science, have been found to apply in many different areas in the physical and social sciences [1-8]. Concepts related to complex systems thus may function as integrative cross-disciplinary principles that are essential to understanding new scientific perspectives, which suggests research exploring the transfer of these ideas to new problems and situations may be productive. However, prior work has demonstrated that many of the concepts related to the study of complex systems are counter-intuitive and thus pose particular conceptual challenges for students to learn. Consequently, complex systems also may function as an excellent domain in which to conduct conceptual change research given the sharp differences in how novices and experts think about how systems such as these function [20]. 2.1.2. Complex Systems and Problem-oriented Hypermedia Research A POLC hypermedia system about complex systems should provide an excellent opportunity to investigate cognitive and learning issues associated with conceptual change and knowledge transfer, which, according to the arguments presented earlier in the paper, are two special difficulties associated with learning with problems and cases. These two areas are considered in turn. 2.1.3. Fostering Conceptual Change about Complex Systems: The Potential of Hypermedia Enabled Ontological Scaffolding Many of the core ideas associated with new scientific conceptual perspectives about complex systems may be challenging for students to learn [20-23]. Research suggests that not only do individuals with expertise in complex systems have specialized conceptual understandings that novices do not (which is to be expected), but also that complex systems novices and experts use different ontologies and epistemologies when constructing solutions for “everyday” examples of complex systems such as ants foraging for food or traffic jams [20, 24]. Figure 2 shows a framework for the cognitive structure of knowledge about complex systems in terms of the relationship between experts’ and novices’ mental models and the corresponding underlying ontological beliefs. Complex systems experts were found to solve these problems using a set of ontological beliefs that involved de-centralized control and considered nonlinearities and randomness in the systems, while novices solved these complex systems problems using a set of “clockwork”

152

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia

ontological statements that described system control as being centralized and regarded system outcomes as being predictable. These findings are consistent with research on expert-novice differences [25-27] and with recent theoretical perspectives on conceptual change [28-32].

Figure 2. Diagram of hypothesized knowledge structure differences related to expert and novice complex systems problem solving. The “*” next to “multiple agents highlights that certain component beliefs might be used in two different mental models. This figure is adapted from Jacobson [20].

This research suggests that there are conceptual challenges students are likely to experience when learning certain complex systems concepts and that novices will likely need to enrich their ontologies in areas such as how complex systems function and the nature of control and causality in such systems in order to understand higher order concepts. Consequently, the Complex Systems Knowledge Mediator hypermedia system was authored to provide ontological scaffolding in addition to the conceptual and problem solving scaffolds that had been used in earlier research.5 Figure 3 shows with the box overlays where ontological scaffolding is provided in the Complex Systems Knowledge Mediator. The program refers to ontological beliefs as “Ideas about Complex Systems,” which for the current program focus on beliefs about control, actions, processes, and causality. In the Mini-Lessons (lower left portion of the screen), the learner has access to short conceptual change oriented lessons (e.g., use everyday examples of complex systems phenomena, discuss gaps in novice ways of thinking, introduce new ideas or concepts) about the higher order complex systems concepts, but also to general discussions that contrast and compare ontological ideas about how systems function such as decentralized versus centralized control or random versus predictable actions. In the cases, there are context specific texts in Conceptual Explanations about ontological beliefs relevant to each case, as well as Suggested Links in the Guided Conceptual Criss-crossing Panel on the right side of the screen that often includes links to Conceptual Explanations about ontological beliefs relevant to the problem being considered. In an ongoing study, students

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia

153

work collaboratively as dyads with the Complex Systems Knowledge Mediator (each student with his or her own computer) to investigate whether ontological scaffolding in conjunction with conceptual and cross-case problem solving scaffolds might help learners better understand the targeted complex systems concepts in the contrasting cases.

Figure 3. The ontological scaffolding in the Complex Systems Knowledge Mediator (shown in boxes) is provided in the Mini-Lessons on the left side of the figure (“Ideas about Complex Systems”), on the cases in Conceptual Explanations shown on the middle area of the screen, and in the Suggested Links section of the Guided Conceptual Criss-crossing Panel on the right side of the screen.

2.1.4. Complex Systems Conceptual Perspectives and the Possibility of Far Transfer As noted above, the issue of transfer related to problem-oriented learning is an interesting theoretical and research issue that is also of considerable practical importance in general educational settings that use POLC pedagogical approaches (e.g., medical and business education, and increasingly at the pre-college level). The general efficacy of studying contrasting cases to foster transfer has been discussed above [8, 12], as has the use of hypermedia enabled scaffolding with guided conceptual criss-crossing as in Knowledge Mediator hypermedia [16-18]. The ongoing study with the Complex Systems Knowledge Mediator includes within domain transfer items that have different surface features than the cases in the system but with deep structure conceptual relationships concerning the complex systems concepts studied in the hypermedia system. In addition, it has been suggested that given complex systems principles are being applied in a variety of areas in the physical and social sciences, such perspectives may function to cognitively interconnect subject areas that heretofore had been regarded as distinct [20, 22, 33]. Consequently, this study also explores the possibility that deeply learning core complex systems concepts might promote far transfer; that is, applying ideas learned in one domain to what is generally regarded as a different subject domain. To

154

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia

explore this thesis, the ongoing study also includes across domain or far transfer problem solving items. 3. Conclusion This paper has argued that there is an opportunity to conceptualize the use of hypermedia technologies as an enabling infrastructure for tools to support learner centered pedagogies such as problem-oriented learning with cases (POLC) that are of increasing interest in professional, university, and pre-college education. Specific problem areas that learning sciences research has identified with POLC are discussed, such as the development of inert knowledge by studying individual cases in isolation, as well as ways that hypermedia systems for POLC might be designed to overcome such difficulties as knowledge transfer and conceptual change. The hypermedia design features and earlier research involving the Knowledge Mediator framework are summarized and a discussion is provided of ongoing research investigating cognitive issues related to learning advanced scientific conceptual perspectives from the field of complex and dynamical systems. Finally, it is hoped that theory and research perspectives discussed in this paper will contribute towards work into how hypermedia (and other) learning technologies might be designed and used to enhance the knowledge and skills that may be developed using learner centered pedagogies involving cases and problems. 4. Endnotes 1

There are a variety of literatures from medical education, business and legal education, as well as cognitive and educational research that involve research on learning with cases, examples, and problems. Not surprisingly, there are variations in terminology, such as “case-based learning,” “case method,” “problembased learning,” “goal-based scenarios,” “anchored instruction,” and so on. The phrase “problem-oriented learning with cases” or POLC is preferred in this paper as these various approaches share strong family resemblances (despite their differences) that include the use of case or case-like materials as the context for problems or issues the learner considers. 2 This summary paragraph simplifies the discussion of four different but related studies in which there were two “generic” treatment groups that were varied in small ways in each of the studies. Detailed discussion of all the treatment groups in these studies is available in the Gentner and associates papers that are referenced. 3 Gentner and her colleagues [8] describe analogical encoding as a variation of structure-mapping theory that has been proposed as the mechanism for analogical reasoning [13, 14]. In analogical reasoning, a set of correspondences are highlighted between the shared relational structure of two analogs that might have different surface features, with a person’s knowledge of the source analog leading to understanding of the target analog. However, with analogical encoding, a learner may only partially understand two cases, but by comparing two cases that share an underlying principle or concept helps the learner focus on the structural commonalities rather than the idiosyncratic surface features. It is further postulated that the process of analogical encoding promotes schema abstraction and thus enhances recall and transfer. 4 This observation is based on conversations with university faculty colleagues who have been active in medical problem-based learning and the use of cases in university business schools. 5 Briefly conceptual scaffolding in Knowledge Mediator systems refers to the availability of context specific explanations in cases related to important abstract domain concepts, while problem solving scaffolding includes design features such as supports for cross-case learning activities based on conceptual dimensions of problems.

5. References [1]

Jacobson, M. J. (1994) Issues in hypertext and hypermedia research: Toward a framework for linking theory-todesign, Journal of Educational Multimedia and Hypermedia, vol. 3, pp. 141-154.

M.J. Jacobson / Exploring Fundamental Issues About Problem-Oriented Hypermedia [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21]

[22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33]

155

Dillon, A. and Gabbard, R. (1998) Hypermedia as an educational technology: A review of the quantitative research literature on learner comprehension, control, and style, Review of Educational Research, vol. 68, pp. 322-349. Shapiro, A. and Niederhauser, D. (2003) Learning from hypertext: Research issues and findings, in Handbook of Research for Education Communications and Technology, 2nd ed., D. H. Jonassen, Ed., 2nd ed. Mahwah, NJ: Lawrence Erlbaum Associates. Tergan, S. O. (1997) Conceptual and methodological shortcomings in hypertext/hypermedia design and research, Journal of Educational Computing Research, vol. 16, pp. 209-235. Hmelo, C. E. (1995) Problem-based learning: Effects on the early acquisition of cognitive skill in medicine, Journal of the Learning Sciences, vol. 7, pp. 173-208. Williams, S. M. (1992) Putting case-based instruction in context: Examples from legal and medical education, The Journal of the Learning Sciences, vol. 2, pp. 367-427. Barrows, H. S. (1996) Problem-based learning in medicine and beyond: A brief overview, in Bringing problem-based learning to higher education: Theory and practice, L. Wilkerson and W. H. Gijselaers, Eds. San Francisco: JosseyBass, pp. 3-12. Gentner, D., Loewenstein, J., and Thompson, L. (2003) Learning and transfer: A general role for analogical encoding, Journal of Educational Psychology, vol. 95, pp. 393-408. Albanese, M. A. and Mitchell, S. (1993) Problem-based learning: A review of literature on its outcomes and implementation issues, Academic Medicine, vol. 68, pp. 52-81. Vernon, D. T. A. and Blake, R. L. (1993) Does problem-based learning work? A meta-analysis of evaluation research, Academic Medicine, vol. 68, pp. 550-563. Cognition and Technology Group at Vanderbilt. (1997) The Jasper project: Lessons in curriculum, instruction, assessment, and professional development. Mahwah, NJ: Lawrence Erlbaum Associates. Thompson, L., Gentner, D., and Loewenstein, J. (2000) Avoiding missed opportunities in managerial life: Analogical training more powerful than individual case training, Organizational Behavior and Human Decision Processes, vol. 82, pp. 60-75. Gentner, D. (1983) Structure mapping: A theoretical framework for analogy, Cognitive Science, vol. 7, pp. 155-170. Gentner, D. (1989) The mechanisms of analogical learning, in Similarity and analogical reasoning, S. Vosniadou and A. Ortony, Eds. New York: Cambridge University Press, pp. 199-241. Jacobson, M. J. (in press) From non-adaptive to adaptive educational hypermedia: Theory, research, and design issues, in Advances in Web-based Education: Personalized Learning Environments, S. Chen and G. Magoulas, Eds. Hershey, PA: Idea Group. Jacobson, M. J. and Archodidou, A. (2000) The design of hypermedia tools for learning: Fostering conceptual change and transfer of complex scientific knowledge, The Journal of the Learning Sciences, vol. 9, pp. 149-199. Jacobson, M. J., Maouri, C., Mishra, P., and Kolar, C. (1996) Learning with hypertext learning environments: Theory, design, and research, Journal of Educational Multimedia and Hypermedia, vol. 5, pp. 239-281. Jacobson, M. J. and Spiro, R. J. (1995) Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation, Journal of Educational Computing Research, vol. 12, pp. 301-333. Hynd, C., Jacobson, M., Reinking, D., Heron, A., and Holschuh, J. (1999) Reading like a historian: Development of cross-text intertextuality and disciplinary knowledge in a hypertext environment. Presented at annual meeting of the American Educational Research Association, Montreal, Canada. Jacobson, M. J. (2001) Problem solving, cognition, and complex systems: Differences between experts and novices, Complexity, vol. 6, pp. 41-49. Charles, E. S. and d’Apollonia, S. (2004) Developing a conceptual framework to explain emergent causality: Overcoming ontological beliefs to achieve conceptual change, in Proceedings of the 26th Annual Cognitive Science Society, K. Forbus, D. Gentner, and T. Reiger, Eds. Mahwah, N.J.: Lawrence Erlbaum Associates. Retrieved from: http://www.cogsci.northwestern.edu/cogsci2004/sessions.html#emergent. Jacobson, M. J. and Wilensky, U. (in press) Complex systems in education: Scientific and educational importance and implications for the learning sciences, Journal of the Learning Sciences. Jacobson, M. J. and Angulo, A. J. (2000) Complex systems, cognition, and problem solving: A preliminary investigation of differences between novices and experts, in Proceedings of the Second International Conference on Complex Systems, Y. Bar-Yam, Ed. Nashua, NH: New England Complex Systems Institute. Jacobson, M. J. (2000) Butterflies, traffic jams, and cheetahs: Problem solving and complex systems. Presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Chi, M. T. H., Glaser, R., and Farr, M. J. (1988) The nature of expertise. Hillsdale, NJ: Lawrence Erlbaum Associates. Larkin, J. H., McDermott, J., Simon, D. P., and Simon, H. A. (1980) Expert and novice performance in solving physics problems, Science, vol. 208, pp. 1335-1342. Bransford, J. D., Brown, A. L., Cocking, R. R., and Donovan, S. (2000) How people learn: Brain, mind, experience, and school (expanded edition). Washington, DC: National Academy Press, pp. 357. diSessa, A. (1993) Towards an epistemology of physics, Cognition and Instruction, vol. 10, pp. 105-225. Chi, M. T. H. (2005) Commonsense conceptions of emergent processes: Why some misconceptions are robust, Journal of the Learning Sciences, vol. 14, pp. 161-199. Chi, M. T. H. (1992) Conceptual change within and across ontological categories: Implications for learning and discovery in science, in Minnesota studies in the philosophy of science: Cognitive models of science, vol. XV, R. Giere, Ed. Minneapolis: University of Minnesota Press, pp. 129-186. Vosniadou, S. and Brewer, W. F. (1992) Mental models of the earth: A study of conceptual change in childhood, Cognitive Psychology, vol. 24, pp. 535-585. Vosniadou, S. and Brewer, W. F. (1994) Mental models of the day/night cycle, Cognitive Science, vol. 18, pp. 123183. Goldstone, R. L. and Wilensky, U. (submitted) Promoting transfer through complex systems principles.

156

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

A Study of the Development of an Information Literacy Framework for Hong Kong Students Siu Cheung KONGa, Fong Lok LEEb, Siu Cheung LIc, James HENRId Department of Mathematics, Science, Social Sciences & Technology, The Hong Kong Institute of Education, Hong Kong b Department of Curriculum & Instruction, The Chinese University of Hong Kong, HK c Department of Education Studies, Hong Kong Baptist University, HK d Division of Information & Technology Studies, The University of Hong Kong, HK Email: [email protected] a

Abstract. Information literacy is the ability to master the processes of becoming informed. An information literacy framework should focus on the capacity of students to build both cognitive and affective knowledge. This entails the formulation of standards and indicators. Hence, we conducted a survey, for which a questionnaire was developed to allow us to understand the views of school principals and teachers on the implementation of an information literacy framework for Hong Kong students. The research findings showed that the standards and indicators we established were recognised. The practitioners agreed with us that the implementation of information literacy standards in schools was not just an issue of enforcing standards; the objective should be to build the capacity of learners and to cultivate school culture. Keywords: information literacy, capacity building, standards and indicators

Introduction In July 2004, the Education and Manpower Bureau (EMB) of the Hong Kong Special Administrative Region Government issued a document entitled “Empowering Learning and Teaching with Information Technology”. As the five-year strategy announced by the government in 1998 provided the essential infrastructure for advancing IT in education, one of the focuses in the new strategy was to guide practitioners in the integration of IT into learning and teaching, and to develop the appropriate skills, knowledge and attitudes in learners to ensure lifelong learning [1, 2, 3 & 4]. In achieving this, the EMB proposed that “a broad framework of ‘information literacy’ for students will be developed to help teachers and students have a clearer picture on the learning targets of using IT in education” [3, p.13]. This strategy was motivated by three developments in Hong Kong. First, the knowledge economy required our students to acquire the ability to process information to respond to the exponential growth of knowledge since the latter part of the twentieth century [5 & 6]. Second, the increasing popularity of digital culture meant that learners had to be equipped with information and communication technology skills in responding to the digitalisation of all industries from the middle of the twentieth century [7]. Third, economic

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

157

globalisation meant that our students had to develop global perspectives and be able to communicate and cooperate with people from different cultural backgrounds [8, 9 & 10]. Information literacy has spawned a proliferation of studies in the past two decades. The notion of information literacy, emerging with the advent of information and communication technologies, has been shaping the way in which people perceive, process, use and create information. Most of the contemporary interpretations of information literacy are inextricably intertwined with lifelong learning [8]. Information literacy is deemed pivotal to the pursuit of both personal empowerment and the economic development of a society. It is recognised as a kind of “new economy” [8] and includes lifelong learning skills that are essential for people to cope with the rapidly evolving changes in the era of information age. The World Summit on the Information Society in 2003 stated that “each person should have the opportunity to acquire the necessary skills and knowledge in order to understand, participate actively in, and benefit fully from, the Information Society and the knowledge economy” [10, para. 29]. Hence, information literacy for Hong Kong should be framed in such a way that students are empowered with the capacity for lifelong learning, and assume greater autonomy with and gain social responsibilities from their learning. Capacity building in this context refers to a process of developing in a learner the capacity to work both independently and socially, and to participate in, benefit from and contribute to the knowledge society and the wider global community. Thus, the emerging digital culture coupled with the trends in economic globalization and the need for developing a knowledge-based society, as depicted in Figure 1, have underpinned the infusion of information literacy in education and all spheres of political, economical and social life.

Figure 1: A conceptual model for Information Literacy

Background of the Study Information literacy is the ability to master the processes of becoming informed [11]. Hence, it is an essential capability that Hong Kong citizens need to adapt to digital culture, globalisation and the emerging knowledge-based society.

158

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

To develop a global perspective on and deepen our understanding of the current trend in the development of information literacy, a set of representative information literacy frameworks from different regions were selected for scrutiny. The eight selected models were SUNY1, ACRL2, AASL3, SCONUL4, AkASL5, WLMA6, ANZIIL7 and JULM8. Adopting the grounded theory approach [12], the goal of analysis was to arrive at a categorisation of information literacy frameworks and to identify key features within each dimension. While there were no a priori categories of information literacy standards, the analysis was initiated with open coding, based on the four overarching cognitive, meta-cognitive, affective and socio-cultural dimensions. After some initial exploration that was followed by iterative examination, a set of categories or a coding scheme of information literacy standards was derived. Despites the variations in scope and coverage exhibited by various information literacy models, the results of the analysis indicated that the standards of the eight selected frameworks could be categorised readily into the four key dimensions of learning: cognitive, meta-cognitive, affective and socio-cultural. The objectives of the information literacy framework development were therefore to enable our students to master the required information processing skills and understandings, to develop them as reflective learners, to enable them to appreciate the value of independent and community-based learning, and to empower them with greater autonomy and social responsibility over the use of information. To embrace all of the aforementioned dimensions of information literacy, eleven standards were formulated. Among those standards, four are in the cognitive dimension, three are in the meta-cognitive dimension, two are in the affective dimension and two are in the socio-cultural dimension. Table 1 summarizes the information literacy standards. Table 1: Information literacy standards in 4 dimensions: cognitive (C), meta-cognitive (M), affective (A) and socio-cultural (S). Code C1 C2 C3 C4

M1

M2 M3 A1 1

Information literacy standards An information literate person is able to determine the extent of and locate the information needed. An information literate person is able to apply information to problem solving and decision making. An information literate person is able to analyse the collected information and construct new concepts or understandings. An information literate person is able to critically evaluate information and integrate new concepts with prior knowledge. An information literate person is able to be aware that information processing is iterative, time-consuming and demands effort. An information literate person is able to plan and monitor the process of inquiry. An information literate person is able to reflect upon and regulate the process of inquiry. An information literate person is able to recognise that being an independent reader will contribute to personal enjoyment and lifelong learning.

State University of New York Association of College & Research Libraries 3 American Association of School Librarians and the Association for Educational Communications and Technology 4 Standing Conference of National and University Libraries from the United Kingdom 5 Alaska Association of School Librarians 6 Washington Library Media Association 7 Australian and New Zealand Institute of Information Literacy 8 Juarez University Libraries, Mexico 2

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

A2

S1 S2

159

An information literate person is able to recognise that information processing skills and freedom of information access are pivotal to sustaining the development of a knowledge society. An information literate person is able to contribute positively to the learning community in knowledge building. An information literate person is able to understand and respect the ethical, legal, political and cultural contexts in which information is being used.

Research Methodology A questionnaire was developed to allow us to understand the views of school principals and teachers on implementing an information literacy framework for Hong Kong students. The questionnaire was divided into four parts that covered goals, knowledge and attitude, implementation, and expected difficulties. A total of 3924 questionnaires were sent, along with a cover letter that explained the background of the study and the concept of information literacy, to all 1308 primary and secondary schools in Hong Kong in December 2004 to invite the participation of principals/curriculum coordinators, teachers responsible for coordinating IT across the curriculum and teacher librarians. The total number of questionnaires returned was 2608 for a response rate of 66.46%. Table 2 illustrates the demographic data of the survey. Table 2: Demographic data of the survey No. of No. of No of Response Category schools questionnaires questionnaires rate (%) invited sent returned Primary schools (AM) 153 459 309 67.32% Primary Primary schools (PM) 148 444 284 63.96% schools Primary schools (WD) 485 1455 996 68.45% Total primary schools 786 2358 1589 67.39% Secondary schools 14 42 4 9.52% (AM) Secondary schools 14 42 7 16.67% Secondary (PM) schools Secondary schools 494 1482 1008 68.02% (WD) Total secondary 522 1566 1019 65.07% schools Total 1308 3924 2608 66.46%

Results and Discussion The standards and indicators established were recognised by most of the respondents. Over 95% of respondents agreed that information literacy education is needed for Hong Kong students (see Figure 2).

160

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

No 4.97% Yes 95.03%

Figure 2: Results from all participants on whether information literacy education is needed for students

This shows that the practitioners considered that students being information literate was very important. The average ratings on the items regarding the four dimensions of the derived framework show that most practitioners were in agreement on each indicator. To further study the framework in terms of using IT for developing basic abilities in information processing, all participants were asked to rate the indicators of the four dimensions. Tables 3-6 show the average ratings of the cognitive, meta-cognitive, affective and socio-cultural dimensions, respectively. The average ratings of the indicators ranged from 3.30 to 3.54, which indicates that most participants either agreed or strongly agreed with each indicator. The average ratings of the indicators of cognitive dimension ranged from 3.30 to 3.48, of the meta-cognitive dimension ranged from 3.30 to 3.50, of the affective dimension ranged from 3.47 to 3.54 and of the socio-cultural dimension ranged from 3.44 to 3.51. This implies that the standards and indicators established for Hong Kong students were recognised by most of the school principals and teachers in Hong Kong. Table 3: Average ratings of the indicators of the cognitive dimension When students graduate, they are expected to be able to: g. Apply information in problem solving a. Identify a variety of potential sources of information b. Develop strategies for locating information e. Apply information to inform decisions c. Frame appropriate questions based on information needs f. Apply information in critical thinking i. Critically analyse information collected l. Determine the accuracy, relevance and comprehensiveness of information d. Determine the nature and scope of the information needed m. Assimilate new concepts into their knowledge bases and value systems h. Record, categorise and manage the information and its sources k. Make inferences, connections and draw conclusions j. Derive new concepts or understandings from the information collected

3.51 3.48 3.46 3.45 3.42 3.42 3.41

Standard Derivation (S.D.) 0.53 0.52 0.54 0.53 0.52 0.57 0.55

3.41

0.56

Average (1-4)

3.38

0.52

3.37

0.55

3.36 3.34

0.55 0.55

3.30

0.57

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

Table 4: Average ratings of the indicators of the meta-cognitive dimension Average When students graduate, they are expected to be able to: (1-4) b. Understand that information processing requires time, diligence and practice 3.50 a. Recognise that the information seeking process is evolutionary and changes 3.45 during the course of investigation e. Reflect upon the development process of the product/performance and 3.45 identify areas of improvement c. Define a manageable focus and timeline 3.42 g. Review the information seeking process and revise search strategies as 3.37 necessary f. Devise strategies for revising, improving and updating self-generated 3.31 knowledge d. Apply new and prior information to the planning and creation of a particular 3.30 product/performance

Table 5: Average ratings of the indicators of the affective dimension Average When students graduate, they are expected to be able to: (1-4) d. Recognise that being an independent learner will contribute to lifelong 3.54 learning b. Recognise and select materials appropriate to personal abilities and interests 3.51 c. Recognise that accurate and comprehensive information is the basis for 3.49 intelligent decision-making a. Read for information and pleasure 3.47 e. Recognise the importance of freedom of information access to a knowledge 3.47 society Table 6: Average ratings of the indicators of the socio-cultural dimension When students graduate, they are expected to be able to: Average (1-4) e. Understand and respect for the principle of intellectual freedom 3.51 b. Collaborate effectively in groups to pursue and construct knowledge 3.48 d. Understand and respect the principles of equitable access to information 3.48 a. Share knowledge and information with others 3.45 c. Recognise that information is underpinned by values and beliefs 3.44 f. Observe laws, institutional policies and social etiquette related to access 3.44 to and the use of information

161

S.D.

0.51

0.52

0.55

0.53 0.56 0.57 0.57

S.D. 0.53 0.51 0.53 0.54 0.53

S.D. 0.53 0.54 0.53 0.52 0.54 0.53

In the average ratings of the expected abilities possessed by primary students after graduation, four items in the affective dimension and three items in the socio-cultural dimension were rated in the 10 most important items that should be possessed by primary students when they were graduated. For secondary students, four items in the affective dimension and four items in the socio-cultural dimension were rated in the 10 most important items that they should possess. This illustrates that both the affective and socio-cultural dimensions were believed to be important elements for primary and secondary students. The difference between the abilities possessed by primary and secondary students can be seen from one item in the cognitive dimension. When compared to other items, primary practitioners did not expect too much from primary students in terms of being able to apply information in problem solving. In contrast, secondary practitioners considered this ability to be vital for secondary students. Primary school participants considered cultivating primary students to read for information and pleasure as an essential element of information literacy, whereas recognising that being an independent learner will contribute to lifelong

162

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

learning was regarded to be most significant for secondary students. Table 7 presents the 10 highest rated abilities expected of primary students, and Table 8 does so for secondary students. Table 7: The 10 highest ratings by primary school practitioners of abilities expected to be possessed by primary students Average Code When students graduate, they are expected to be able to: S.D. (1-4) A a. Read for information and pleasure 3.36 0.52 b. Understand that information processing requires time, diligence and M 3.33 0.50 practice b. Recognise and select materials appropriate to personal abilities and A 3.32 0.52 interests S a. Share knowledge and information with others 3.27 0.48 S e. Understand and respect the principle of intellectual freedom 3.27 0.55 S b. Collaborate effectively in groups to pursue and construct knowledge 3.25 0.54 S d. Understand and respect the principles of equitable access to information 3.24 0.54 a. Recognise that the information seeking process is evolutionary and M 3.22 0.50 changes during the course of investigation C g. Apply information in problem solving 3.22 0.54 d. Recognise that being an independent learner will contribute to lifelong A 3.19 0.64 learning

Table 8: The 10 highest ratings by secondary school practitioners of abilities expected to be possessed by secondary students Average S.D. Code When students graduate, they are expected to be able to: (1-4) d. Recognise that being an independent learner will contribute to lifelong A 3.49 0.53 learning b. Recognise and select materials appropriate to personal abilities and A 3.45 0.51 interests C g. Apply information in problem solving 3.45 0.53 S e. Understand and respect the principle of intellectual freedom 3.45 0.53 c. Recognise that accurate and comprehensive information is the basis A 3.44 0.54 for intelligent decision making b. Understand that information processing requires time, diligence and M 3.44 0.51 practice d. Understand and respect the principles of equitable access to 3.41 0.54 S information A a. Read for information and pleasure 3.40 0.54 f. Observe laws, institutional policies and social etiquette related to S 3.40 0.53 access to and the use of information S b. Collaborate effectively in groups to pursue and construct knowledge 3.40 0.55

Conclusion Information literacy is the mastery of the processes of becoming informed. It is considered to be crucial for people to cope with the rapid changes in the information age that are caused by the emerging digital culture, globalisation and the development of a knowledge-based society. These changes have necessitated the infusing of information literacy into the school curriculum and thus the development of an information literacy framework for Hong Kong students. The development of such a framework should enable students to master the skills that

S.C. Kong et al. / A Study of the Development of an Information Literacy Framework

163

are necessary for information processing, develop them as reflective learners, enable them to appreciate that being independent learners will contribute to personal growth, enjoyment and lifelong learning, and empower them with greater autonomy and social responsibility over the use of information in their individual and collaborative learning. The information literacy framework derived focused on the capacity of students to build both cognitive and affective knowledge. The standards and indicators of information literacy for students were identified in four key dimensions: cognitive, meta-cognitive, affective and socio-cultural. The research findings showed that the standards and indicators established were recognized. Over 95% of the respondents agreed that information literacy education is needed for Hong Kong students. The average ratings of the indicators (sets of expected characteristics that were developed based on information literacy standards) ranged from 3.30 to 3.54 on a 4-point scale, which shows that they were well-received. The implementation of information literacy standards in our schools is not merely an issue of enforcing standards; its objective should be to build the capacity of learners and to cultivate school culture.

References Curriculum Development Council. (2000). Information technology learning targets. Hong Kong: Curriculum Development Council. [2] Education and Manpower Bureau. (1998). Information Technology for Learning in a New Era Five-Year Strategy 1998/99-2002/03. Hong Kong: Education and Manpower Bureau. [3] Education and Manpower Bureau. (2004). Empowering learning and teaching with information technology. Hong Kong: Education and Manpower Bureau. [4] Education and Manpower Bureau. (2005). Overall Study on Reviewing the Progress and Evaluating the Information Technology in Education (ITEd) Projects 1998-2003. Hong Kong: Education and Manpower Bureau. [5] American Library Association Presidential Committee on Information Literacy. (1989). American Library Association Presidential Committee on Information Literacy, Final Report—1989. Washington, D.C.: American Library Association Presidential Committee. [6] Candy, P. (2002). Lifelong Learning and information literacy. White Paper prepared for UNESCO, the U.S. National Commission on Libraries and Information Science and the National Forum on Information Literacy, for use at the Information Literacy Meeting of Experts, Prague, the Czech Republic. Retrieved 20 January, 2005, from http://www.nclis.gov/libinter/infolitconf&meet/papers/candy-fullpaper.pdf. [7] Martin, A. (2003). Towards e-literacy. In A. Martin & H. Rader (Eds): Information and IT Literacy: Enabling Learning in the 21st Century (pp. 3-23). London: Facet Publishing. [8] O’Sullivan, C. (2002). Is information literacy relevant in the real world? Reference Services Review, 30(1), 7-15. [9] Rader, H. (2003). Information literacy – a global perspective. In A. Martin & H. Rader (Eds): Information and IT Literacy: Enabling Learning in the 21st Century (pp. 24-42). London: Facet Publishing. [10] World Summit on the Information Society. (2003). Building the information society: a global challenge in the new Millennium. Retrieved January 28, 2005, from http://www.itu.int/dms_pub/itu-s/md/03/wsis/doc/S03-WSIS-DOC-0004!!PDF-E.pdf. [11] Henri, J. (1995). The information literate school community: Exploring a fuzzy concept. Scan 14(3), 25-28. [12] Strauss, A.L. & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory. Thousand Oaks: Sage Publications. [1]

164

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

An Experimental Study of a Cognitive Tool for Classroom Use: A Knowledge of Fraction Equivalence Siu Cheung KONGa Department of Mathematics, Science, Social Sciences & Technology, The Hong Kong Institute of Education, Hong Kong [email protected]

a

Abstract. A cognitive tool (CT) that helps learners to generate a knowledge of fraction equivalence, which comprises conceptual and procedural knowledge, has been built. The aim of this research is to study the effectiveness of the CT in generating a knowledge of fraction equivalence for classroom use. Two groups of learners were randomly selected to learn the subject knowledge. One group, the CTWSGp, initially started to learn with the CT and then worked with drilling-orientated worksheets (WS), and the other group, the WSCTGp, learned in the reverse order. The results of pre-test-post-test experimental studies show that both groups had gained competency in the knowledge of fraction equivalence by the end of the learning period. An analysis of the results indicates that learners in the CTWSGp group learnt the subject matter better than the WSCTGp group, and that the CT is a more effective tool than drilling-orientated worksheets. Keywords: cognitive tools, experimental studies, fraction equivalence

Introduction This work was motivated by the need to overcome the shortcomings of the separation of knowledge from meaning in classroom instruction. For example, learners seldom understand the procedural knowledge that is associated with fractional operations [1]. A review of the literature indicates that learners have great difficulty in learning the concepts and procedural knowledge behind mathematical fractions [2, 3, 4]. Procedural knowledge is the knowledge that guides the performance of a task without the performer having access to the knowledge that underlies the performance [5]. Traditional classroom teaching adopts the algorithmic approach to teaching procedural knowledge, which has the shortcoming of separating knowledge from meaning. Figure 1 shows some examples to illustrate the traditional classroom approach to teaching algorithms for the computation of equivalent fractions. 5 5u ( ) ( ) 1 1u ( ) ( ) 2 2u5 ( ) a. c. b. 3 3u 5 ( ) 6 6u( ) 12 3 3u 2 ( ) Figure 1: Some examples that illustrate the traditional classroom approach to teaching algorithms for the computation of equivalent fractions.

We adopt the view that cognitive tools (CTs) are both mental and computational devices that can support, guide and mediate the cognitive processes of learners [6, 7]. A CT facilitates the cognitive functioning of learners by mediating their interpretation, reasoning and reflection on knowledge, which they are incapable of doing without support [8]. In this

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

165

study, a computational CT to help learners to generate a knowledge of fraction equivalence was built [9]. The aim of this research is to study the effectiveness of the CT in generating a knowledge of fraction equivalence. Experimental conditions were set up to study the effectiveness of the CT in different learning settings, which provided empirical findings that may lead to the formulation of advice on the classroom use of CTs. The knowledge of fraction equivalence has two parts: the concept of fraction equivalence and the procedural knowledge of computing equivalent fractions [10]. This CT is designed to develop a conceptual understanding of the subject matter amongst learners, as well as the associated procedural knowledge of the computation of equivalent fractions with graphical support. The CT aims to improve the communication of knowledge of fraction equivalence in three aspects to meet the diverse needs of learners in learning the subject matter [9]. The first design of the CT comprises an animation of the partitioning, or regrouping, process. The animation simulates the partitioning strategy through the additional partitioning of each part, and provides learners with an opportunity to grasp the idea of using the same unit to compare fraction equivalence and to see the inverse relationship between the number of parts and the size of a part of a unit that arises naturally in the animation, which are aspects of the problem that are commonly not realised by learners. Figure 2 shows the CT as a partitioning device to support the development of the concept of fraction equivalence. Learners are allowed to input any fractions to study their equivalent counterparts. q Learners can trigger an instant display by clicking the left button of the mouse on the upper half of the fraction bar. A pop-up menu for the selection of returning to the previous term or to the original fraction is displayed when the right button of the mouse is clicked on the fraction bar.

oLearners are required at random to adjust the size of the units so that they are equal before partitioning is allowed.

nThe graphical representation is displayed when learners input and confirm a fraction symbol.

p Learners can trigger an animation by clicking the left button of the mouse on the lower half of the fraction bar. Lines run down from each fractional part to simulate the further partitioning of the half into three sixths.

Figure 2: The CT as a partitioning device to support the development of the concept of fraction equivalence

The second design is a hypothesis-testing interface

a b

auc , which invites learners bud

to adjust parameters c and d to test possible fraction equivalent states. Learners can also trigger the comparison animation by dragging the hypothesised bar and dropping it onto the original fraction bar to compare equivalence. This option helps learners to understand the concept of fraction equivalence and simultaneously provides a way of finding equivalent

166

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

fractions. Figure 3 shows the CT as a hypothesis testing bed for reasoning on the method of finding equivalent fractions using the concept of equivalence.

Figure 3: The CT as a hypothesis testing bed for reasoning on the method of finding equivalent fractions with the concept of equivalence

The third design is a simplified feature of the second design for the comparison of the equivalence of fractions. The only difference here is that learners are allowed to compare two fractions that are either generated by the computer or inputted by the learners to compare their equivalence. Learners can trigger the comparison feature by dragging a fraction bar and dropping it onto another bar, which starts an animation that directly compares the two bars. This design allows learners who have shown no intention of representing fractions to compare equivalence to develop such knowledge. Research Questions The aim of this research is to study the effectiveness of the CT in generating a knowledge of fraction equivalence for classroom use. One of the principles of the evaluation of cognitive tools is the study of the knowledge that can be generalised by the learners after working with them [11]. To evaluate the CT, we investigate two specific research questions, as follows. 1. What are the learning outcomes of learners after working with the CT? 2. How can the CT be integrated with traditional classroom practices to achieve optimal learning outcomes? Methods This study uses an experimental design to study the knowledge that can be generalised by learners after working with the CT [12]. A primary school was invited to join the study. Two groups of learners were randomly selected to learn the subject knowledge. One group of students started learning with the CT in the first stage and then worked on drilling-orientated worksheets (WS), such as those depicted in Figure 1, in the second stage.

167

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

This group of learners was allowed to use the CT to validate the answers that they filled in on the drilling-orientated worksheets. The other group of learners started with the drilling-orientated worksheets in the first stage and then learned with the CT, and were not allowed to use the CT in the first stage. A quantitative pre-test and two post-tests were administered in both groups to measure the knowledge of fraction equivalence that the learners had gained before and after each stage of the experiment. Figure 4 shows the study flows of the two groups of learners. CTWSGp

Learning with the CT Pretest

Posttest I

Working on the drilling-orientated WS (CT allowed)

Posttest II

Working on the drilling-orientated Learning with the CT WS (CT not used) Figure 4: The study flows of the two groups of learners, the CTWSGp group and the WSCTGp group

WSCTGp

Subjects, Sampling and Profiles We sampled thirty-six learners in this study in the age group of eight to nine years. The learners were volunteer Primary Three learners in a primary school with one hundred and sixty-five third graders. Eighteen learners were randomly assigned to each of the experimental groups. The mathematics mean score is the mean results of learners in the mathematics tests and examinations of the school. Table 1 shows the profile of the learners in the two experimental groups of the study. Table 1: Profile of the learners in the two experimental groups of the study Profile CTWSGp WSCTGp Number of learners 18 18 Ratio of boys to girls 7:11 10:8 Mean age in years 8.82 (SD = 0.39) 8.99 (SD = 0.51) Mathematics mean score (max = 100) 70.37 (SD = 11.93) 70.76 (SD = 10.37)

Instruments A knowledge of fraction equivalence has two parts: the concept of fraction equivalence and procedural knowledge of the computation of equivalent fractions. The knowledge of the learners on fraction equivalence was measured by items that were extracted or modified from Niemi’s instruments for the assessment of the conceptual understanding of common fractions [3]. In the instrument, learners are required to draw diagrams to represent equivalent fractions, to judge the equivalence of fractions, and to enumerate, with examples, the number of equivalent fractions of a fraction to measure how well they have grasped the concept of fraction equivalence. Learners are required to answer eight questions on finding equivalent fractions to measure their capability to compute equivalent fractions. Table 2 shows the number of items in the instrument. The reliability coefficients reveal a substantial internal consistency. Table 2: Instrument to measure the knowledge of learners of fraction equivalence Num Cronbach’s Cronbach’s Cronbach’s Max. Instrument Measurement of Alpha Reliability Alpha Reliability Alpha Reliability Score Item (Pre-test) (Post-test I) (Post-test II) Concept of fraction FEConcept 5 12 0.549 0.531 0.566 equivalence Knowledge of computing FECompute 8 8 0.846 0.825 0.842 equivalent fractions

168

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

Learning and Teaching Activities The researcher was the teacher of the two experimental groups. The two groups of learners learned the subject domain in two computer rooms during four meetings with a total learning time of six hours in a period of two weeks. The course was conducted at the end of the school year. This topic would usually be taught at the beginning of the school year in grade four. Each learner was assigned a computer in the learning sessions, and the researcher and assistants coordinated the learning and teaching activities. The learners were organised into groups of three for discussion purposes. The typical learning events were to engage the learners individually, and occasionally in groups, to complete the tasks that were assigned on the guide sheets with CT support or by working with drilling-orientated worksheets; to encourage or guide the learners in using the CT to tackle incomplete tasks; and to organise peer tutoring or group discussion to consolidate the objectives of the learning activities. Tables 3 and 4 show the major learning and teaching activities of the learning sessions with the CT support and with the drilling-orientated worksheets, respectively. Table 3: Major learning and teaching activities of the learning sessions with the CT support Learning and Teaching Cognitive Tool Support Guide Sheet Questions Activities 1. Using the CT (the 3rd design) CT for developing the concept of a. 12 questions on representing equivalent fractions in geometrical fraction equivalence with an to answer questions on the form. guide sheet to understand the emphasis on the graphical b. 13 guiding questions on developing representation of equivalent higher terms of equivalent a concept of fraction equivalence fractions. fractions. by working with the CT. c. 6 discussion questions on fraction equivalence 2. Using the CT (the 1st and 2nd CT for developing conceptual and a. 13 questions on representing designs) to answer questions procedural knowledge on the equivalent fractions in geometrical form. on the guide sheet to computation of equivalent understand the higher terms fractions with an emphasis on the b. 17 guiding questions on developing a knowledge of fraction of equivalent fractions and graphical meaning of requiring a the method for finding equivalence by working with the common multiple of the nominator CT. equivalent fractions. and denominator of a fraction to find the higher terms of equivalent c. 4 discussion questions on fraction equivalence fractions.

Table 4: Major learning and teaching activities of the learning sessions with the drilling-orientated worksheets Learning and Teaching Static Graphical Drilling-orientated Worksheet Questions Activities Support 1. Using the worksheet with Static graphical a. 8 questions with static graphical support on the smaller numerical nominators diagrams are used meaning of finding the higher and lower terms of and denominators to teach the to illustrate the equivalent fractions. meaning and method of meaning of b. 11 questions on finding the higher terms and 11 on finding the higher and lower equivalent finding the lower terms of equivalent fractions. terms of equivalent fractions. fractions. c. 2 discussion questions on finding the higher and lower terms of equivalent fractions. 2. Using the worksheet with Static graphical a. 8 questions with static graphical support on the larger numerical nominators diagrams are used meaning of finding the higher and lower terms of and denominators to teach the to illustrate the equivalent fractions. meaning and method of meaning of b. 12 question on finding the higher terms and 12 on finding the higher and lower equivalent finding the lower terms of equivalent fractions. terms of equivalent fractions. fractions. c. 2 discussion questions on finding the higher and lower terms of equivalent fractions.

169

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

Results and Discussion In this section, we discuss the capability of the CT to assist learners to gain competence in the knowledge of fraction equivalence for classroom use. Figure 5 shows the effects of the treatments on the knowledge of the two experimental groups of fraction equivalence at the end of the first and second stages. ˣ̅̂˶˸˷̈̅˴˿ʳ˞́̂̊˿˸˷˺˸ʳ̂˹ʳ˶̂̀̃̈̇˼́˺ʳ˸̄̈˼̉˴˿˸́̇ʳ˹̅˴˶̇˼̂́̆

˖̂́˶˸̃̇ʳ̂˹ʳ˹̅˴˶̇˼̂́ʳ˸̄̈˼̉˴˿˸́˶˸ ˋˁ˃˃

ˋˁ˃˃

ˉˁ˃˃

ˉˁ˃˃ ˖˧˪˦˚̃

ˇˁ˃˃

˪˦˖˧˚ˣ

˖˧˪˦˚̃

ˇˁ˃˃

˪˦˖˧˚ˣ

˅ˁ˃˃

˅ˁ˃˃

˃ˁ˃˃

˃ˁ˃˃ ˣ̅˸ˀ̇˸̆̇

ˣ̂̆̇ˀ̇˸̆̇ʳ˜

ˣ̂̆̇ˀ̇˸̆̇ʳ˜˜

Figure 5: Effects of the experimental treatments on the two learning groups on their knowledge of the concept of fraction equivalence

ˣ̅˸ˀ̇˸̆̇

ˣ̂̆̇ˀ̇˸̆̇ʳ˜

ˣ̂̆̇ˀ̇˸̆̇ʳ˜˜

Figure 6: Effects of the experimental treatments on the two learning groups on their procedural knowledge of computing equivalent fractions

Table 5 show the means and standard deviations of the pre-test and post-test II measures of the learners who participated in the two experimental groups. The paired t-test shows that the mean difference between each pair of pre-test and post-test II measures of the concept of fraction equivalence and the procedural knowledge of computing equivalent fractions for both experimental groups is significantly different. In other words, both groups had gained a knowledge of fraction equivalence at the end of the second stage. Table 5: Means, standard deviations and paired t-tests of the pre-test and post-test II measures for learners who participated in the two experimental groups. Paired Paired CTWSGp WSCTGp Measure Measure t-test t-test Pre-test Post-test II Pre-test Post-test II M (SD) M (SD) t(17) M (SD) M (SD) t(17) FEConcept 5.72 (2.61) 7.33 (2.22) -3.107** FEConcept 4.22 (3.00) 6.78 (3.35) -2.997** FECompute 4.11 (2.49) 6.44 (1.76) -4.622*** FECompute 3.61 (2.93) 5.06 (2.41) -2.628** Note. n = 18 for both the CTWSGp group and the WSCTGp group for all measures. ** p < .01. *** p < .001.

Tables 6 and 7 show the means and standard deviations of the pre-test, post-test I and post-test II measures for learners who participated in the two experimental groups. In addition, we attempted to use the t-test results that are listed in Tables 6 and 7 to show the experimental effects of the CT in assisting learners to generate learning outcomes. Table 6: Means, standard deviations and t-tests of the pre-test measures for learners who participated in the two experimental groups. Pre-test t-test Measure CTWSGp WSCTGp M (SD) M (SD) t(34) FEConcept5 5.72 (2.61) 4.22 (3.00) 1.601 FECompute 4.11 (2.49) 3.61 (2.93) 0.551 Note. n = 18 for both the CTWSGp group and the WSCTGp group for all measures.

The two t-tests that compare the differences between the means of the pre-test measures show that none of the tests of equality of mean for the two experimental groups could be rejected. This finding substantiates our assumption that learners in both groups

170

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

had the same prior knowledge of fraction equivalence before the experimental studies. However, the differences between the two experimental groups in the FEConcept and FECompute measures in post-test I and the FECompute measure in post-test II are statistically significant in the four t-tests that compare the differences between the means of the post-test I and the post-test II measures. Table 7: Means, standard deviations and t-tests of the post-test I and post-test II measures for learners who participated in the two experimental groups. Post-test I Post-test II t-test t-test Measure Measure CTWSGp WSCTGp CTWSGp WSCTGp M (SD) M (SD) t(34) M (SD) M (SD) t(34) FEConcept 6.33 (2.66) 4.50 (3.05) 1.922* FEConcept 7.33 (2.22) 6.78 (3.35) 0.586 FECompute 5.83 (2.09) 4.44 (2.66) 1.740* FECompute 6.44 (1.76) 5.06 (2.41) 1.975* Note. n = 18 for both the CTWSGp group and the WSCTGp group for all measures. * p < .05.

Two results were obtained. Firstly, the mean differences of the FEConcept and FECompute measures between the CTWSGp and WSCTGp groups in post-test I as 5.83 4.44 0.58, converted to standardised progression scores are 6 .33  4 .50 0 .64 and 2.39 2 .86 respectively. Thus, at the end of the first stage the experimental effects of the CT on the average progression of a learner’s knowledge over the effect of the drilling-orientated worksheet treatment in the two areas are 23.91% and 21.91% respectively. Secondly, at the end of the second stage the mean difference of the FECompute measure between the CTWSGp and WSCTGp groups in the post-test II as converted to standardised progression 6.44  5.06 scores is 0.66 . In other words, at the end of the second stage the average 2.11 progression of the knowledge of learners in the CTWSGp group of procedural knowledge is 24.48% higher than the average progression of the knowledge of learners in the WSCTGp group. Two inferences can be drawn from this. The first result supports the assertion that the CT is a more effective tool in assisting learners in learning about fraction equivalence than the worksheet. The second result shows that starting to learn with the CT and then working with the drilling-orientated worksheets is a better design for the integration of the CT with classroom practice than the reverse order in terms of generating competency in the computation of equivalent fractions. However, Table 7 also indicates that the mean difference between the two experimental groups on the concept of fraction equivalence in the post-test II is not statistically significant. The different orders of using the CT and the drilling-orientated worksheets made no difference in building competence in the concept of fraction equivalence in the learners. The paired t-test results in Table 8 help to elaborate the role of the CT in assisting learners in the WSCTGp group to grasp the concept of fraction equivalence. Table 8: Means, standard deviations and paired t-tests of the pre-test and post-test I, and the post-test I and post-test II measures for learners who participated in the two experimental groups. Paired Paired WSCTGp WSCTGp t-test Measure Measure Pre-test Post-test I Post-test I Post-test II t-test M (SD) M (SD) t(17) M (SD) M (SD) t(17) FEConcept 4.22 (3.00) 4.50 (3.05) -0.377 FEConcept 4.50 (3.05) 6.78 (3.35) -3.473** FECompute 3.61 (2.93) 4.44 (2.66) -1.906* FECompute 4.44 (2.66) 5.06 (2.41) -1.571 Note. n = 18 for both the CTWSGp group and the WSCTGp group for all measures. * p < .05. ** p < .01.

Table 8 shows that the CT is an important tool for helping learners to grasp the concept of fraction equivalence. This is demonstrated by the fact that learners in the

S.C. Kong / An Experimental Study of a Cognitive Tool for Classroom Use

171

WSCTGp group grasped the concept of fraction equivalence significantly better (see also Figure 5) when the CT was introduced in the second stage of learning, and that the drilling-orientated worksheets did not help the learners to grasp the concept much when they was introduced in the first stage. This shows that the insignificant statistical difference between the two experimental groups in the FEConcept measure in the post-test II was due to the significant improvement of the WSCTGp group on the introduction of the CT tool. Conclusion

A CT to help learners to generate a knowledge of fraction equivalence is studied in two experimental settings. Three major findings are found from the statistical results. First, the CT assists learners to gain competency in fraction equivalence. Second, the CT is a more effective tool in assisting learners to gain a knowledge of fraction equivalence than using drilling-orientated worksheets alone. Third, starting to learn with the CT and then working with the drilling-orientated worksheets is a better design for the integration of the CT with classroom practice than the reverse order. Thus, teachers who integrate of the CT with traditional classroom teaching may achieve better learning outcomes for their learners in this subject domain. Future research work should study the way in which learners gain competency in the concept of fraction equivalence and the procedural knowledge of computing equivalent fractions through working with the CT during the learning process. This may help us to understand the underlying importance of conceptual understanding in the learning of procedural knowledge. Acknowledgement

The author would like to acknowledge the research grant support of the Hong Kong Institute of Education [RG6/2003-2004]. References [1]

Kong, S.C., & Kwok, L.F. (2002). Modeling a cognitive tool for teaching the addition/subtraction of common fractions. International Journal of Cognition and Technology, 1(2), 325-349. [2] Huinker, D. (1998). Letting fraction algorithms emerge through problem solving. In L.J. Morrow (Ed.). The teaching and learning of algorithms in school mathematics (pp. 170-182). V.A.: NCTM. [3] Niemi, D. (1996). Assessing conceptual understanding in mathematics: Representations, problem solutions, justifications, and explanations. The Journal of Educational Research, 89(6), 351-363. [4] Pitkethly, A., & Hunting, R. (1996). A review of recent research in the area of initial fraction concepts. Educational Studies in Mathematics, 30, 5-38. [5] Anderson, J. R. (1976). Language, memory, and thought. Hillsdale, NJ: Erlbaum. [6] Derry, S.J., & LaJoie, S.P. (1993). A middle camp for (un)intelligent instructional computing: An introduction. In S.P. LaJoie & S.J. Derry (Eds.), Computers as cognitive tools. NJ: Lawrence Erlbaum Associates. [7] Kommers, P., Jonassen, D.H. & Mayes T. (Eds). (1992). Cognitive tools for learning. Heidelberg FRG: Springer-Verlag. [8] Pea, R.D. (1985). Beyond amplification: Using the computer to reorganize mental functioning. Educational Psychologist, 20(4), 167-182. [9] Kong, S.C., & Kwok, L.F. (2003). A graphical partitioning model for learning common fractions: Designing affordances on a web-supported learning environment. Computers and Education, 40(2), 137-155. [10] Kong, S.C., & Kwok, L.F. (2005). A cognitive tool for teaching the addition/subtraction of common fractions: A model of affordances. Computers and Education, 45(2), 245-265. [11] Ohlsson, S. (1998). Representation and process in learning environments for mathematics: A commentary on three systems. Interactive Learning Environments, 5, 205-215. [12] Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.

172

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

The Implementation of an Information Literacy Framework: Key Issues in Professional Development Siu Cheung KONGa, Fong Lok LEEb, Shuk Fan IPa Department of Mathematics, Science, Social Sciences & Technology, The Hong Kong Institute of Education, Hong Kong b Department of Curriculum & Instruction, The Chinese University of Hong Kong, HK [email protected]

a

Abstract. Information literacy should be framed in such a way that students are empowered with the capacity for lifelong learning and assume greater autonomy over – and gain social responsibilities from – their learning. We conducted focus group discussions and in-depth interviews to collect ideas from practitioners on professional development in information literacy education. The research findings show that practitioners are concerned about the recognition of teachers and the training methods that will be used in the implementation of information literacy in schools. Professional development should focus on the content and the pedagogy of capacity building in school education. Three key issues in professional development are highlighted: realising capacity building, obtaining the consensus and support of school principals and teachers, and allowing practitioners the flexibility to organise their own development. Nine modules for the professional development of in-service and pre-service teachers are also proposed. Keywords: information literacy, capacity building, professional development

Introduction In July 2004, the Education and Manpower Bureau (EMB) of the Hong Kong Special Administrative Region Government issued a document entitled “Empowering Learning and Teaching with Information Technology”. One of the focuses of this new strategy is to direct school principals and teachers in the integration of information technology into teaching and learning, and to build up the appropriate skills, knowledge and attitudes in learners to ensure lifelong learning [1, 2, 3, & 4]. In realising this, the EMB suggests that “a broad framework of ‘information literacy’ for students will be developed to help teachers and students have a clearer picture on the learning targets of using IT in education” [3, p.13]. This strategy was inspired by three developments in Hong Kong society. First, the exponential growth of knowledge since the latter part of the twentieth century and the resulting knowledge economy necessitates that our students acquire the ability to process information [5, 6]. Second, with the increasing popularity of digital culture, learners must be equipped with information and communication technology skills to react to the digitalisation of all industries, which has been occurring since the middle of the twentieth century [7, 8]. Third, economic globalisation indicates that our students must develop global perspectives and be able to communicate and cooperate with people from different cultural backgrounds [9, 10 & 11].

S.C. Kong et al. / The Implementation of an Information Literacy Framework

173

Information literacy for Hong Kong students should be framed in such a way that students are empowered with the capacity for lifelong learning, and assume greater autonomy over – and gain social responsibilities from – their learning. The World Summit of the Information Society in 2003 highlighted that “each person should have the opportunity to acquire the necessary skills and knowledge in order to understand, participate actively in, and benefit fully from, the Information Society and the knowledge economy” [11, para. 29]. Capacity building, therefore, refers to the process whereby a learner develops the capacity to work independently and socially, and participates in, benefits from, and contributes to the knowledge society and the wider global community. In a project entitled “A Study on the Development of an Information Literacy Framework for Hong Kong Students”, an analysis of information literacy was undertaken using open coding, which was based on four overarching dimensions: the cognitive, meta-cognitive, affective and socio-cultural dimensions. To embrace all four dimensions of information literacy, eleven standards were subsequently formulated. Of the eleven standards, four are in the cognitive dimension, three are in the meta-cognitive dimension, two are in the affective dimension and two are in the socio-cultural dimension [12]. Table 1 summarises these information literacy standards. Table 1: Information literacy standards in four dimensions: cognitive (C), meta-cognitive (M), affective (A) and socio-cultural (S). Code Information literacy standards C1 An information literate person is able to determine the extent of and locate the information needed. C2 An information literate person is able to apply information to problem-solving and decision making. C3 An information literate person is able to analyse the collected information and construct new concepts or understandings C4 An information literate person is able to critically evaluate information and integrate new concepts with prior knowledge. M1 An information literate person is able to be aware that information processing is iterative, time-consuming and demands effort. M2 An information literate person is able to plan and monitor the process of inquiry. M3 An information literate person is able to reflect upon and regulate the process of inquiry. A1 An information literate person is able to recognise that being an independent reader will contribute to personal enjoyment and lifelong learning. A2 An information literate person is able to recognise that information processing skills and freedom of information access are pivotal to sustaining the development of a knowledge society. S1 An information literate person is able to contribute positively to the learning community in knowledge building. S2 An information literate person is able to understand and respect the ethical, legal, political and cultural contexts in which information is being used.

2. Research Methodology After proposing the information literacy framework, focus group discussions and in-depth interviews were conducted to gain an understanding of the opinions of practitioners and

174

S.C. Kong et al. / The Implementation of an Information Literacy Framework

related stakeholders on professional development in implementing the information literacy framework.

2.1 Focus Group Discussions Seventeen focus group discussion sessions, which included participants from education associations, education bodies, IT pilot primary and secondary schools in 1998, secondary, primary and international schools, and expert panels on information literacy, were conducted. Nine groups of primary and secondary schools were involved, which were stratified across five areas which incorporated eighteen districts in Hong Kong: New Territories East, New Territories North, New Territories West, Kowloon and Hong Kong and Islands. Ten primary and ten secondary schools were randomly selected in each district. The participants included teachers, teacher-librarians, school principals and representatives from other professional organisations. Table 2 illustrates the demographic data of the focus group participants. Table 2: Demographic data of the focus group participants No. of No. of responding Response No. of No. of invited Focus group focus participants organisations(y) organisations Rate(x/y) categories group (x) Education 1 8 11 7 63.64% associations Education bodies 1 7 15 7 46.67% Secondary schools 5 22 52 22 42.31% IT pilot secondary 1 7 10 5 50.00% schools in 1998 Primary schools 4 17 56 16 28.57% IT pilot primary 1 3 10 3 30.00% schools in 1998 International schools 2 8 16 6 37.50% Expert panels on 2 About 30 2 2 100.00% information literacy Total 17 About 102 172 68 39.53%

2.2 In-depth Interviews Each interview was semi-structured, with two interview guidelines for the two groups of interviewees, namely, educational and general. The interview guidelines stated only the major areas that were to be covered to ensure the smoothness and efficiency of the interviews but to allow flexibility. Each interview lasted for approximately one and a half hours. All of the interviews were digitally recorded with consent, and were summarised for further analysis. By applying the judgment sampling method, eleven groups of professionals, including education, information technology, commercial and industrial professionals, were recruited for in-depth interviews. Table 3 illustrates the demographic data of the interviewees.

175

S.C. Kong et al. / The Implementation of an Information Literacy Framework

Table 3: Demographic data of the interviewees In-depth interview

Group name

Advisory Committee on Teacher Education and Qualifications Hong Kong Teacher Librarians’ Association Parent Teacher Association Education Secondary school principals Officer in charge of education body Hong Kong Professional Teacher’s Union Primary school principal and teacher Legislative Member of the Legislative Council Council Printing and Publishing Industry Watch and Clock Hong Kong IT Federation Total 11

No. of participants

1

7 1 1 1 5 2

1 About 10 About 10 4 About 36

3. Results and Discussion

3.1 Concerns about Professional Development The participants in the focus group discussions and in-depth interviews were asked to provide their opinions on the expected difficulties of integrating information literacy into the school curriculum. The interviewees expressed their concern about professional development, and suggested some training methods for information literacy education. They also expressed their opinions on the possible increase in the workload of teachers and the pressure that they would be put under in the implementation of information literacy education. However, they did agree that practitioners should recognise information literacy. Table 4 shows the count of different opinions as taken from the written feedback to the focus group discussions on concerns about professional development. Table 4: Count of opinions from the written feedback from the focus group discussions on concerns about professional development Concerns about professional development Count % Training method suggestions 22 36.67% Information literacy should be recognised by teachers before the 13 21.67% start of any training Training will increase the workload and pressure of teachers 11 18.33% Professional development is a difficult task in the implementation 7 11.67% of information literacy in the education system Resources needed for training 2 3.33% Disagree on all staff training 1 1.67% Other difficulties with professional development 4 6.67%

176

S.C. Kong et al. / The Implementation of an Information Literacy Framework

Total

60

100.00%

Most of the participants in the focus group discussions and in-depth interviews recognised the need for professional development in the implementation of the information literacy framework, and some also suggested different methods by which this could be accomplished. For example, it was highlighted that both in-service teachers and pre-service teachers should be provided opportunities for training. The content of professional development should include the enforcement of information literacy for practitioners and the development of the ability to teach information literacy amongst practitioners. Further, it was stated that the content of professional development should be based on catering to the individual needs of different practitioners. Teachers could then identify their own needs when selecting the information literacy courses to attend, and training courses would therefore be a part of the Continuing Professional Development scheme, which is an existing mechanism for the professional development of teachers in Hong Kong.

3.2 Key Issues in Professional Development The role of practitioners is considered to be vital in ensuring the successful implementation of the information literacy framework for Hong Kong students. Professional development means the enforcement of information literacy and the development of the ability to teach information literacy amongst practitioners. A consideration of the concerns of practitioners about professional development as discussed in Section 3.1 generated three key issues as follows. In general, practitioners have no formal background in information literacy education, and rarely have experience in teaching it. There needs to be sufficient training on the implementation of information literacy to build capacity in learners. Therefore, one of the key issues in professional development is to realise capacity building to achieve the objectives of the information literacy framework through the engendering of knowledge about information literacy and the development of pedagogical principles for teaching it. In assisting school principals and teachers to develop the pedagogy of information literacy as a means to capacity building, an environment should be created that supports higher-order thinking and capacity building in the development of information-processing skills. The second issue is the importance of obtaining consensus and support from school principles and teachers on the implementation of information literacy before the initiation of professional development. To obtain the consensus of practitioners, professional development should help them to realise the importance of fostering in learners the capability to process information for learning, problem solving, and assuming social responsibilities as members of the knowledge society. To obtain support from school principles and teachers, a “pleasure space” should be created for teachers to fulfil the aims of professional development, in which sufficient resources, such as discussions of information literacy assessment in students and introductions to different information literacy models, are made available. Furthermore, professional development in information literacy should start as soon as possible to prepare practitioners for the implementation, and should be organised over an extended period to minimise the pressure on practitioners. The third issue is to allow flexibility for school principals and teachers to organise their own development programme. McKeown and Curran (2003) state that “flexibility in both time scheduling and training materials makes a significant difference to motivation

S.C. Kong et al. / The Implementation of an Information Literacy Framework

177

and the achievement of both personal learning objectives and organisational objectives” [13, p.142]. Flexibility in professional development is imperative for practitioners to acquire the necessary knowledge to realise capacity building in learners in school education. Web-based training materials should be considered to consolidate course content, and activities could be organised on the professional development days of individual schools so that practitioners can gain practical experience. Authentic experience should be provided for practitioners through visits to schools that have experience on information literacy implementation and experience sharing with practitioners who are experienced in information literacy education. To minimise the amount of time that practitioners have to spend attending initiatives on professional development, teachers should be encouraged to attend school-based courses or Web-based courses. Only a few teachers should be required to attend a full training programme, and they will then serve as trainers (seed teachers) in their schools.

3.3 The Content of Professional Development A seed-teacher training programme is suggested for in-service teacher training to implement the information literacy framework for Hong Kong students. Two seed teachers from each school, such as principals, vice principals, academic coordinators, curriculum coordinators, teacher librarians, or teachers who are responsible for coordinating IT across the curriculum, will receive central training through core modules. As the background of participants may differ, the training courses should also include elective modules. Seed teachers must attend the core modules and can then choose elective courses according to their needs. A similar programme is recommended for prospective teachers as a teaching training requirement. The aim of such a programme would be to develop the competence of pre-service teachers to implement the information literacy framework for Hong Kong students. Table 5 shows the proposed content of in-service teacher training in the implementation of the information literacy framework. The modules are aimed at introducing the basic information that is related to the implementation of information literacy in school education, such as assessment, school-based planning, pedagogy, and intellectual property rights. Another purpose of the modules is to allow participants to visit schools and share experiences with school principals and teachers who are familiar with the implementation of information literacy. Table 5: Content of in-service teacher training for the implementation of the information literacy framework Core/ Module Titles Elective Vision and mission of implementing information literacy in school education. School-based planning for information literacy implementation. Core Pedagogy of implementing information literacy as capacity building. Assessment in information literacy education. Information processing models and information literacy education. Intellectual property rights and information literacy education. Pedagogy of developing information skills in information literacy education. Elective Pedagogy of developing social responsibility in information literacy education. Experience sharing of information literacy implementation in school education (I) School visit. (II) Sharing with teachers.

178

S.C. Kong et al. / The Implementation of an Information Literacy Framework

4. Conclusion Information literacy is crucial for people to enable them to cope with the rapid changes in the information age that have been caused by the emerging digital culture, globalisation and the development of a knowledge-based society. These changes have forced the inclusion of information literacy in the school curriculum, and thus the development of an information literacy framework for Hong Kong students. The development of such a framework should enable students to master the skills that are necessary to process information; to develop them as reflective learners; to enable them to appreciate that being independent learners will contribute to personal growth, enjoyment and lifelong learning; and to empower them with greater autonomy and social responsibility in the use of information in their individual and collaborative learning. The information literacy framework should focus on the capacity of students to build both cognitive and affective knowledge. The findings of this research show that professional development is identified as one of the major concerns amongst practitioners. The practitioners who participated in the study agreed that they should be trained to ensure the success of the framework, but also reflected that practitioners nowadays are under heavy pressure, and that therefore teacher training should be in the form of flexible continuous professional development. One of the key issues in professional development is to undertake capacity building to achieve the objectives of the implementation. Another important issue is the obtaining of consensus and support from school principles and teachers to facilitate professional development that helps practitioners to realise the importance of fostering in learners the capability to process information. Furthermore, sufficient resources should be made available to minimise the pressure on principals and teachers. The third issue is the importance of allowing practitioners to flexibly acquire knowledge to realise capacity building in learners. Examples of measures that would increase the flexibility of professional development are Web-based training materials, activities through which participants gain practical experience, and exposure to authentic experience through school visits and experience sharing. These measures are recommended in response to the concerns of practitioners about flexibility in undertaking professional development and the stress from a possible increase in workload that might occur with the implementation of the information literacy framework. A seed-teacher training programme is proposed for in-service teacher training. As the background of the participants may differ, elective modules will be included alongside the core modules in the training course. Nine modules for the professional development of in-service and pre-service teachers are proposed, of which four are core modules and five are electives. Seed teachers must attend the core modules, but can choose elective courses according to their needs. The role of practitioners in guaranteeing the successful implementation of the information literacy framework is vital, and therefore carefully designed professional development in information literacy would make a significant difference to the implementation.

S.C. Kong et al. / The Implementation of an Information Literacy Framework

179

References [1] [2] [3] [4]

[5]

[6]

[7]

[8] [9] [10]

[11]

[12] [13]

Curriculum Development Council. (2000). Information Technology Learning Targets. Hong Kong: Curriculum Development Council. Education and Manpower Bureau. (1998). Information Technology for Learning in a New Era Five-Year Strategy 1998/99-2002/03. Hong Kong: Education and Manpower Bureau. Education and Manpower Bureau. (2004). Empowering Learning and Teaching with Information Technology. Hong Kong: Education and Manpower Bureau. Education and Manpower Bureau. (2005). Overall Study on Reviewing the Progress and Evaluating the Information Technology in Education (ITEd) Projects 1998-2003. Hong Kong: Education and Manpower Bureau. American Library Association Presidential Committee on Information Literacy. (1989). American Library Association Presidential Committee on Information Literacy, Final Report—1989. Washington, D.C.: American Library Association Presidential Committee. Candy, P. (2002). Lifelong Learning and Information Literacy. White Paper prepared for UNESCO, U.S. National Commission on Libraries and Information Science and the National Forum on Information Literacy, for use at the Information Literacy Meeting of Experts, Prague, Czech Republic. Retrieved 20 January 2005 from www.nclis.gov/libinter/infolitconf&meet/papers/candy-fullpaper.pdf. Johnson, H. (2003). The SCONUL task force on information skills. In A. Martin & H. Rader (Eds): Information and IT Literacy: Enabling Learning in the 21st Century (pp. 45-52). London: Facet Publishing. Martin, A. (2003). Towards e-literacy. In A. Martin & H. Rader (Eds): Information and IT Literacy: Enabling Learning in the 21st Century (pp. 3-23). London: Facet Publishing. O’Sullivan, C. (2002). Is information literacy relevant in the real world? Reference Services Review, 30(1), 7-15. Rader, H. (2003). Information literacy – a global perspective. In A. Martin & H. Rader (Eds): Information and IT Literacy: Enabling Learning in the 21st Century (pp. 24-42). London: Facet Publishing. World Summit on the Information Society. (2003). Building the Information Society: A Global Challenge in the New Millennium. Retrieved 28 January 2005 from www.itu.int/dms_pub/itu-s/md/03/wsis/doc/S03-WSIS-DOC-0004!!PDF-E.pdf. Education and Manpower Bureau. (In press). A Study on the Development of an Information Literacy Framework for Hong Kong Students. Hong Kong: Education and Manpower Bureau. McKeown, C. and Curran, C. (2003). ICT training: models of delivery. In A. Martin & H. Rader (Eds.): Information and IT Literacy: Enabling Learning in the 21st Century (pp. 132-143). London: Facet Publishing.

180

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Knowledge Work and Learning at the Workplace - Key Issues and Solutions Alexander KARAPIDIS1, Till BECKER2, Volker ZIMMERMANN3 1 Fraunhofer IAO,Nobelstrasse 12, 70569 Stuttgart, Germany 2 University of Stuttgart IAT, Nobelstrasse 12, 70569 Stuttgart, Germany 3 imc information multimedia communication AG, Altenkesselerstrasse 17/D3, 66115 Saarbrücken, Germany [email protected] Abstract. In an economy in which knowledge becomes more and more important as the main resource of economical success, workers, teams and companies need to develop the necessary competencies to be able to participate in a work life that is mainly based on knowledge productivity. In this paper we first outline knowledge work and its key issue areas. Second, we stress the necessity to combine work processes with training processes in the context of knowledge work. In the third part of this paper, we outline one approach developed by a consortium of research institutes and companies to overcome the gap between knowledge requested and content provided by a tool-based service to improve the skills of employees at the workplace. Keywords: personalised learning, individualised learning, work place learning, just in time, knowledge management

Knowledge Work New technologies, spreading in nearly all areas of economy and society, have been a major driving force behind significant changes in work organisation, production and society. These changes have led to the so-called “knowledge-based economy”. Over the last years, considerable attention has been payed to the analysis of "knowledge work" (Alvesson, 1993; Coulson-Thomas, 1991; Drucker, 1991). In the discussion about professional training, the term “knowledge work” refers to coping with work tasks which are complex or new at least to the person concerned in professional work. Knowledge work requires a large variety of information and sound knowledge as raw material. This generates new knowledge as a product. Davenport’s definition for knowledge work is as follows: “In our definition, knowledge work's primary activity is the acquisition, creation, packaging, or application of knowledge. Characterized by variety and exception rather than routine, it is performed by professional or technical workers with a high level of skill and expertise. Knowledge work processes include such activities as research and product development, advertising, education, and professional services like law, accounting, and consulting. We also include management processes such as strategy and planning." (Davenport et al., 1996, p. 54). By stressing knowledge work, one issue is that it is unclear what knowledge work could mean. Despres and Hiltrop, for example, stress, that “…the nature of knowledge-based work is fundamentally different from that known previously” and that “…new types of work are now being performed by new types of worker” (1995, p.12). This will be very obvious by interlinking knowledge work with professional training issues.

A. Karapidis et al. / Knowledge Work and Learning at the Workplace - Key Issues and Solutions

181

About knowledge work and professional training For enterprises it is necessary to be more effective and efficient in their work in order to be more competitive. At the same time the quality of both working life and the working conditions for the employee must not suffer, and these are just some of the aims. So, the main goal is to help to improve working conditions for knowledge and e-workers: x Comprehensive job content: The job tasks are planned, performed, checked and organised fairly independently by the employees. This makes the tasks attractive and motivating. x High individual demand on creativity and independent problem solving: Knowledge-intensive services are generated and provided in an interaction with the customer or co-operation partner, solutions must be developed spontaneously and on the spot. The job content, therefore, is challenging in a positive sense. x Situated, context-related and informal professional learning: Traditionally, research has focused on the transfer of knowledge from those who know to those who do not know. But most of the learning takes place as informal, situated and context-related learning, closely connected to the professional work. The learning arrangement will support this new view of learning. x New innovative types of learning: Some concepts are, e.g., just-in-time-learning, just-in-case-learning, Instant-Qualification. x Work in multi-disciplinary teams of experts: Actors with different competences, thinking patterns and interests co-operate for a limited period of time. Communication problems, interpersonal dynamics and unease, therefore, have a greater influence on the working processes. x High degree of fragmentation and limited possibilities of planning: Frequently, new job tasks appear suddenly and without warning. This forces the employees to reorganise their work sequences. Activities must often be interrupted and resumed later. x Job design for knowledge work: in the field of knowledge-intensive service work the issue is rather to counteract potential overstrain situations – for the protection of the employees as well as for the benefit of productivity. x Successful enterprises, especially SMEs, depend more on their intellectual capabilities than on their physical assets, often specialising in knowledge-intensive products. The strategic information and knowledge resources enable SMEs to recognise chances early and increase competitiveness. The ability to manage knowledge work via situated/informal, conceptualised learning arrangements and to convert it into new innovative products and services is a crucial factor for success. Expected long-term benefits from a concept of knowledge work include cost reduction due to greater efficiency, improved business processes and products due to better use of the staff’s knowledge and more efficient task performance. The bottleneck for the progress of a Knowledge Work Management concept is that research has traditionally focused on enhancing traditional training and transfer of knowledge from those who know to those who do not know. To solve this problem, the learner has to obtain his knowledge from an individualised learning support which is geared to his individual learning needs (personal and contextual). This way, informal and competence conductive professional learning must be supported. This means the direction of the knowledge transfer as learning is reversed to consider the real needs of the potential users in the companies. New technologies are enablers for knowledge work in supporting knowledge works more effective and efficient in the described way.

182

A. Karapidis et al. / Knowledge Work and Learning at the Workplace - Key Issues and Solutions

Key research issues that should be covered by knowledge work management concepts In this chapter the main key issue areas will be described and illustrated with short explanations and examples. New work organisation dealing with work processes: New forms of work division (in companies and between companies, e.g. knowledge work and decision making in inter-working learning environments). An example includes companies with knowledge-intensive work tasks that are often involved in networks, e.g. the media industry. There is no “full service provider”. Rather, there is one specialist for each specific work task. The task of the companies now is to buy these “product bits” and coordinate the specialists. Knowledge work is completed at the interfaces where the product is separated into small “product bits”, the organisation of the specialists and eventually the integration into a complete product. Knowledge management–the middleware between work- and learning process: Selection, design and implementation of knowledge management tools as key elements to support knowledge work. An example is the support of knowledge work by identifying the knowledge transfer tasks (training tasks), (e.g. conserve individual knowledge, knowledge exchange between teams). Identifying the best tools for these training tasks (e.g. knowledge product, instant-qualification, e-learning, etc.). Competence development by learning processes: Design of a new human resource management for knowledge workers including new career concepts, skill management and life-long learning. An actual example is the principle of the knowledge worker, “hire smart people and leave them alone“. Knowledge workers have to coordinate multiple tasks that they are involved in. They are working problem-oriented at the interface between different faculties. So, they cannot use the quality standards of their own faculty for their work. They must have multifaceted knowledge. They often work on special projects as an interface between company and departments. So, they cannot use the organisational structure of a company, e.g. departments, hierarchy, etc., for their orientation. To solve these problems there must be new concepts of HR management and training. Performance & Productivity: definition of standards to evaluate the concept: Definition of knowledge e-work including quality criteria and standardisation of knowledge e-work processes. An example is a consulting service project where the objectives are developed during the work process, not before. So, solutions have to be found “spontaneously” for new, unexpected problems. There are two conclusions to draw from this: It is not possible to standardise the processes and the performance has to be measured based on “soft skills” as the criteria could change during the project. Performance Management benchmark – the concept: Establishing an evaluation and benchmarking model for knowledge work companies. In this example, the problem is the following: In the same branches, different performance criteria could be defined, depending on the self-image. For example, consider the case of a school for the disabled that wants to teach the pupils as much as is possible for them - vs. the case of an elite school where they screen the pupils for the best and only want to train this group. For this example different benchmarking criteria are needed.

A. Karapidis et al. / Knowledge Work and Learning at the Workplace - Key Issues and Solutions

183

Knowledge Work Management – shrinking the work process with the learning process Again, research has traditionally focused on enhancing traditional training and transfer of knowledge from those who know to those who do not know. To solve this problem, the learner has to obtain his knowledge from individualised learning support, which is geared to his individual learning needs (personal context). This way, informal and competence conductive ways of professional learning are supported. That means the direction of knowledge transfer in learning is reversed to consider the real needs of the potential users in the companies. So, new technologies function as enablers for knowledge work management in so much as they support knowledge work more effectively and efficiently in the described way. The PROLIX1 consortium will develop the following solutions . The ideas are outlined and specified in a proposal for the 6 FP from the European Commission in the FP6-2004-IST-4. The overall guiding vision is the significantly improved performance of the employee at his workplace. Training the employee should enable him to learn directly at his workplace when the need for education is identified. The delay between identification of a learning need and the actual learning should not be long. Furthermore, the learning material must be targeted to the learner’s individual learning style and behaviour. In addition, the concept aims at seamlessly integrating learning at the workplace into the daily working life as much as possible. Organizations need to understand that only by providing the necessary means to learning at the workplace, they will be able to react quickly, cost-effectively and successfully in ever-changing markets. Therefore, the concept will fuse business processes with learning processes in corporate environments. The vision is a system that allows for business process-driven learning at the workplace, taking into account the single learners and their needs as well as the corporate requirements. In addition, the learner will be able to initialise a learning task himself, enabling self-guided learning in corporate environments. Seen from an organizational point of view, it will contribute significantly to the change of the management within companies, which needs to develop into a holistic learning organization, allowing for the integration of learning into daily working tasks. Corporate culture requires the provision of strategies, methods and concepts to satisfy heterogeneous learning needs. Mechanisms and concepts for the organizational introduction of TEL in corporations have to be co-ordinated with their philosophy and company vision. Scientific and technological objectives The overall goal is to research, analyse and develop a flexible and adaptive system which is capable of aligning training and knowledge production of people confronted with so-called “complex situations”, such as changes in work and business process, or other complex multivariable learning environments, which cannot be solved with traditional e-Learning or knowledge management approaches. Often, such situations also require a mix of individual and group learning, and of learning and knowledge acquisition/production. There must be a solution that strives to improve significantly the learning situation at the workplace by enabling the employee to address his learning needs directly without any further delay. Furthermore, companies will be able to increase their competitiveness with educational activities at the workplace, which are targeted towards market orientation, delivering the respective learning material tailored to the single user’s needs AND to the corporate 1

PROLIX consortium: imc AG, Germany; Synergetics, Belgium; IDS Scheer AG, Germany; GIUNTI interactive labs, Italy; QPR software, Finland; Immaginary, Italy; L3S, Germany; Fraunhofer IAO, Germany; Wirtschaftsuni Wien, Austria; University Vienna, Austria; NCSR Demokritos, Greece; VUB StarLab, Belgium; DFKI, Germany; Danube University Krems, Austria; Ernst Klett Verlag, Germany; Editis, France; The Office of the Government of the Czech Republic (Department of the Institute of State Administration), Czech Republic; EEA Communication solutions, Slovenia; British Telecom, UK; SCIE Social Care Institute for Excellence, UK

184

A. Karapidis et al. / Knowledge Work and Learning at the Workplace - Key Issues and Solutions

requirements. Therefore, a number of scientific and technological objectives, which address various issues involved in process-driven learning at the workplace, must be focused on. The overall objectives are the development of new learning scenarios in the workplace when coupling business with learning processes. Integrating knowledge management into this overall scenario will not only provide support for the learning employee, but also enhance the necessary organizational change, combining human resources development with corporate knowledge management and increasing significantly the company’s competitiveness. New learning scenarios for learning at the workplace The business process-driven learning management in corporations calls for new learning scenarios at the workplace (e.g. Senge, 1990; Prahalad & Hamel, 1990). The already existing scenarios do not provide sufficient support for the necessary combination of formal and informal learning at the workplace, where learning tasks might arise with or without any previously established plans, where knowledge is contained in formal and informal ways and where day-to-day requirements are constantly changing. Therefore, the new concept will extend existing, and establish new, learning scenarios that will support learning at the workplace more adequately. Aligning business processes with people and their learning processes at work provide orientation for, and drive, the learning processes in companies with criteria stemming from the related tasks; and the business processes at hand ensure a very tight and focused learning experience. The single learner is enabled to tackle his daily tasks more easily by being confronted with highly focused and strongly targeted learning processes as opposed to today, where learning processes usually mean some sort of training session at the workplace. Companies on the other hand can very effectively measure the performance of each of their processes, identifying gaps where they occur and being able to address and solve these without any delay. Especially the feedback of performance of working compared to the complex situation at hand provides very detailed and significant data on how to improve the skills of the employee and of the learning process Complement knowledge management with new e-learning strategies In most companies a concise knowledge management is in place, usually capturing quite nicely the explicit knowledge involved in each business process. Nevertheless, as soon as flexible and possibly complex situations occur, the internal company knowledge management struggles to identify and provide the necessary information to handle the situation. It is important to address this need to enable a more targeted and focused provision of learning resources and strategies and the exchange of information relevant for learning between learners. By providing flexible means of learning at the workplace, the employee as well as the company will be able to respond transparently and quickly to the changed situation. In turn, the knowledge management in companies will change to a flexible and effective information provision and capturing tool that significantly contributes to the company’s success. Support for organizational change and change of management The projected concept will provide the means to start the organizational change that is necessary in companies when e-learning strategies are implemented. Instead of forcing companies to additionally adapt to learning processes, as commonly done today, there will be a support for companies in introducing e-learning with a strong focus on their business processes. This support will enable companies to improve their profiles in order to either perform better in their already established markets or to acquire a share in newly developed markets. Additionally, the organizational change will also improve the employee’s situation

A. Karapidis et al. / Knowledge Work and Learning at the Workplace - Key Issues and Solutions

185

by certifying his continuous education activities. Thus, he will be able to prove his skills with widely accepted and certified documentation, e.g. based on jointly agreed competency profiles. A large number of stakeholders must be in target, ranging from companies, their business development and human resources management to providers of e-learning resources and the individual employee learning at the workplace. Therefore, various needs must be addressed in the approach, based on the following simple categorization: x Applications: Emphasis on the integration of formal and informal learning scenarios, taking into account the tacit and explicit knowledge involved in the current processes which is available to the company and the individual learner. x Learning scenarios: Accommodate various learning scenarios transparently, by adapting the content and its delivery, based on the learning context, the learning methods and personal learning styles, but also based on the resources available to the corporate setting. x Employees: Customize the content transparently according to the user’s professional situation (corporate learner, individual seeking professional enhancement for his career, adult learner seeking a degree, or HEI learner), x Organizations: Transparently integrate into the learning content access to related pieces of corporate information hosted by the corporate learning management system(s), interface with the Human Resources management system, with feedback loops and a focus on learning efficiency, curriculum integration, learner control and ROI, x Service infrastructure: Draw on seamless access to different, interchangeable and interoperable modules. Additionally, access to repositories and brokerage platforms, hosting content or information about the content (metadata) based on the companies’ own regulations, will be implemented based on standard interfaces. x Service providers will be allowed to build up new, innovative learning services by: o using standards-based building blocks, relying on public and market relevant interfaces, o reducing the cost of acquisition (purchase and integration) and operations, o lowering the entry threshold for a new supplier entering the market, o therefore increasing market transparency and corporate adoption, o ultimately, allowing an individual citizen to freely access the e-learning services he requires for a professional ambition, at any time, in any location and any situation. A system for process-oriented learning and information exchange The results at the end must be a closed-loop learning cycle x A support for the implementation of the internationally acknowledged P(lan)-D(o)C(heck)-A(ct) Philosophy within Organizations. Services that will have an important positive impact on both the Quality of Learning and the Quality of applying targeted competences to business processes for creating higher Performance, Productivity and Quality in an organization. 1. Plan - Competency driven analysis. During the re-engineering of business processes or offering of new services, a company has to evaluate their organizational structure to staff the new or modified processes. 2. Do- Content extended process design. Simulation processes will be used to train employees for their new job as well as to configure software systems and Business processes.

186

A. Karapidis et al. / Knowledge Work and Learning at the Workplace - Key Issues and Solutions

3. Check – Accessible process data during execution/learning. In order to be accepted by employees, it is important that people have access to the business processes in the training environment as well as in the execution environment. 4. Act - Get a feedback about the learning. From the learner’s and the business’s point of view it is important to know about the impact of the training on the business process and process performance, as it is the main purpose of a training to lead to improved processes. Because of the training it could be expect that there should be improvements in the performance, productivity or quality of business processes. If not, I might wish to adjust the training content / topics in order to optimise the effect. Summary In this paper we have argued that knowledge work in the work process and learning arrangements in learning processes are still discussed separately, seldom jointly. So, we have outlined what we have done in order to bring them together to close the gap between requested knowledge and learning activities at the workplace of (knowledge) workers. It is the hope of the authors to have shown in this article one way to overcome this gap. References [1] Alvesson, M., "Organization as rhetoric: knowledge-intensive firms and the struggle with ambiguity", Journal of Management Studies, Vol. 30 No 6, pp. 997-1015, 1993 [2] Back, A., Seufert, S, Kramhöller, S.: Technology enabled Management Education: die Lernumgebung MBE Genius im Bereich Executive Study an der Universität St.Gallen, in: io mamagement, 3, S. 36-42, 1998 [3] Becker, T; Gidion, G; Rickert, A.: Didaktische Modelle: Aufbereitung von e-Learning Content. In: Bullinger, H. J.; Weisbecker, A. (Ed.): Content Mangement - Digitale Bausteine einer vernetzten Welt, Stuttgart: IRB, 2002 [4] Coulson-Thomas, "IT and new forms of organization for knowledge workers: opportunity and implementation", Employee Relations, Vol. 13 No 4, pp. 22-32, 1991 [5] Davenport, T.H. et al: Improving Knowledge Work Processes, in: Sloan Management Review, Vol. 37, No. 4, 1996 [6] Davenport T.H, Beck J.C.: The Attention Economy, Harvard Business School Press, 2002 [7] Despres, C. and Hiltrop, J.M., "Human resource management in the knowledge age: current practice and perspectives on the future", Employee Relations, Vol. 17 No 1, pp. 9-23, 1995 [8] Drucker, Peter: The New Productivity Challenge. Harvard Business Review (1991) 6, S. 69-79 [9] Hermann, S.: Wissensarbeit erkennen und organisieren. In: Personalwirtschaft (2002) 6, 49-55. [10]Holzschuh, Gabriele, Karapidis, Alexander: Benchmarking organizational competencies with the dynamic e-assessment tool. In: Service Benchmarking, Walter Ganz (Ed.) / Josephine Hofmann (Ed.), IRB-Verlag 2002 [11]Karapidis, Alexander: Putting the competence card into practice - sample methods from two fit for service clubs. In: Fit for Service report 2002 - Service Benchmarking, Walter Ganz (Ed.)/Josephine Hofmann (Ed), IRB-Verlag 2002 [12]Kerres, Michael: Multimediale und telemediale Lernumgebungen, München Wien Oldenburg, 2001 [13]Klimsa Paul: Multimedia aus psychologischer und didaktischer Sicht, in: Issing, Ludwig J., Klimsa, Paul (Ed.): Information und Lernen mit Multimedia, Weinheim 1997 [14]Krapp, Andreas / Weidemann, Bernd (Ed.): Pädagogische Psychologie, Weinheim, 2001 [15]Nonaka, I., Takeuchi H.: the Knowledge Creating Company. How Japanese Companies Create Dynamics of Innovation, New York, Oxford 1995 [16]Pine B.J., Gilmore J.H.: The Experience Economy, Harvard Business School Press, 1999 [17]Prahalad, C. & Hamel, G. The Core Competence of the Corporation. Harvard Business Review, (MayJune), pp. 79-91, 1990 [18]Reinmann-Rothmeier, Gabi / Mandl, Heinz: Unterrichten und Lernumgebungen gestalten, in: Krapp, Andreas / Weidemann, Bernd (Ed.): Pädagogische Psychologie, Weinheim, 2001 [19]Schneider, Ursula (Ed.): Wissensmanagement: Die Aktivierung des intellektuellen Kapitals, Frankfurt a.M., 1996 [20]Senge, P. M. The Fifth Discipline: The Art and Practice of the Learning Organization. New York, Doubleday Dell Publication Group, 1990

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

187

             

                    !   " #    
$ %  &' ( )   *   + &  +    * ,  '*  *+ & ,  *   * ,  +, & *  & *  ,,, &- ( * * & * *  )   * , &  )-  )  (  & #,  & #* #, .###/  , '  * 0,   ,       )( -    *'   )    ' & 1)    )* ,  2 (   *'  (  , +  + '**  )     ) )  &  +3,  -    ) ++ * , & * ### & & ,(+  ) *  ,      '**  * )      & + *  )( ,-

   $  ) * &  ,    ,,-  

 

 & +   )   *  )( , ) & 0,   '+      4 (3' * '2-  * ,, ' ' , #,  & #* #,  (  &  0,   , & +  )(  &  3  ,,(-  '     *' * +  + )' 3,  )+ ,  ) (   *    ) * - 5* '2 ' & * ( , &  + ,(   ,,  )( +  5* 3  ,,* + * +   )  )  * '*  + 0 .-+- 678/-  * ' + 90 ) * 2: .6;8 ,;  3  + '**   )    - * ++ * ,  ) &&(    +  )( + )' , & *  .-+-6=8/- "'( *  ( * 2+ *   '2  ,     &  )(  .-+- 6?8/@   ) ,  *   ) .-+- 6;8/  & + ) &+  & 0,  * &  & ,  *  * 2  * *   )( 2  * ,, ' 2   )'  )(  3,( +-  )  ) )  9 3 * ( *  *   &

188

L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

  ,      * , &  ,) : .6A8 ,;=A/- 5*   & 3, '**  ) &  9* ( & ) + ,,B'* * ,  ,)  &  , & * ,) 3  (+: .6A8 ,;=A/- ,  0,   * & , +  '- * * 2   )( ,, 2 >  )  , & * 2- 3 ,( ( & * 2  ) '* ' , ,,  ( * 2  , * *  ,)  & ( - 5*  *  +*  & * , +*- 5* ( & )>(  2   ) *+*  *  )( +  )* ,,  2+  &&   * *   ++ .C 2' '*  2'  C 2' *  2' *  2'/- (  )>(  '  * 3,( +  * ,, ,  * ' , & * ' '* 0,   * * .C  2' '*  2'/- 5*  )  + & )>( + & )* ,,   +*  *  & * 3,( + ) * '  )  *  * & *  )+ ,- 5*    *&  & * * * * )   '* ,4+, '2+ 3,( - D 0,  3EF F   6G8 0,  *  & , , & ,,+  +, '2- *  *   * ' ,( , ,    , * ' )  ) * +,  ()   *  -  + * * & && 3,( '2+  * ,( , 2  , & * (  + ()   *  & * +,-      2  * *  & * +, C2' * H 2'  * * &   * ) & H ++- 5*   )    + * +  + ) ++     + & , ' + )  '    ) *  ,3&  ,- 5*   & *   * ' ) ) ' '* *  ' 3, * , ' ()   * )*  * ' &   2  * *( *+*   '  ) * *  )   ,( '   6-+- <8 * & * &' *      .-+- '3,/  + &  +  - "'(   1, *+ ,  & * )' ,   +, & *   * * * 2 )' * ) 32+ *  '*  *  C*2  * * , 6I8- * 32+ )2 ' *  * , &  *    * *   ,  * 2 '* &+ '* * ,- * *   (  *( ) 2 * 0,  *  )( , & +  ,  # #*2  2, 6J8 *( & * * ,( &  ,     ,( * 2 * &   )' * * -  + * *  ) * &' &   *   '    *   *(   * &     * ,+ 32+ * *  2 (- K& K  2   67L8 2 *    * *+   '** * * &  & C 30+ *  ' *   ) * & *      '  - 5* , * *  '*   * 3 , &&(   23*+ ) * ,  '   ,  *  '* ' 2  *  +  -  + * * &+   & & *  &  & * 2$ * *   +( * , & )+ 2   )' * '** *   * ,  ,   ,  - 5* , ,)       # &- D 0,  #* 6-+-778 0,  *' * + & ,    ,,  2' +3) + - " 2 7L37A      (  ,)  + (   * ' ( )    * '2 ,- 5*  '2 +* *  *       ,  +  , & * ,) -  + *  *+* * &  * ( ,  * &  .-+-67;8/

L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

189

' +    )(  *  * ,)  &   + * * )  + 4 (3+ * , ) '*  - 5*& * ,)  & (  (  +  * ,) 3 ,   ) 5  ' &    ( ,  & * '  * + , '**   ,)  & &&(  )  2 , -   ,   * & *+ &  & +  ,,  ) ' *( +  ( & 33 #,  & #* #,  ' , '  * *( 0,   - 5* &   ,   * 0,    ) , & *  +    2 '*    0  *  (   ,, )  & ++ ,3*   ) '* * * 0-    

      * *   '2+  )(    , * '*    ) ,    * ,@     > 32+ ) *   )(  * *    ' - #,  & #* #, .###/    & * 52 #*+ D'2 67=8  (  & *    + * ,  ++ '* * 2-  * ' *    ,  '**     & * > 3 23 ,  3,  ,- + * '  2+ >  ( ,, ' *( &&+ ,   *'  ,  * 2  *- 5* ### & ,( *  '* * ' , *  )        , * '  + & * 2- %( * *        '* * ' ,@    ,)     * , '2- C  *    &   ( * ' , (   * & * * *  *    &  ( * 2 , .*  * *  0  ,  */-   ' * (  )  ( , & * 2 , '**  '* +   & ( '*     )   -   &  +   &     2+     (  +  , & *    * -    * ( *   ,)  &     * * ) & 0,    1*+  & * - 5* ### &  * * * ,( * ,, & ,  1  )* ,  23, (  '** +( *  * , & )+ ++ '* *  (   & * 2@ * (  + *  + & +  *   + *  & 3,( 32+- D 0,      *(  ' & * ,  &* '* * + )& * * (  2  - * ,,   * ' ,  )*   0, '* * *2  *   & * 2- "'( * ### &    ,(  '*     & 1  * 2 ,@ *  0  ,  ( - ###  *'( )  &  & +  )( - 5* , , , *   & +  + )' *   *   *' * *  ( & + && '**  ) ,   + ,+  * 2  *- 5*   *  *    && )' * ,  *  ,       ) +   + + *  & *+*      )( '2 ) * *  )( .6J 7L8/  *  *  , *   & )( '*  * * ,  * 2    2   ) 1  ,  * *  +-  *(    C' + &  * ### &   & *

190

L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

 ) ,  && 2 . * ,++ +/  * +    ,3& + *  & * 3+ &  2 .-+- *'  '2  * '   /  , ,  & * 2   '*  .-+-  * & ' &  /-  &  '+ & * , & (+ +  +   ) & & 2+ * , ) '** +  *- 5* , &    & * , & + . )   , & * ( ,/  ) ++    )  ,&  & * - & & 0,  ' & *  *  ,  , * ,( 4 & * , '    * *   0*)+  *+* ( & +  * *   0 * , '* * -    

         5* ### & '    ,  2  '** , & *  ' 2   * ) & '  +*  .-+-   )0  + >/- 5* 2 ' * )      *(    2' + '**  *  *  '    * 2 , ) * ' )   * * ,-  * 2 1 & +  * *  *   *  , & * 2' +  * & & ,( - 5* '   +( ( ,3 )  - 5*   & *  '  , *   & * ### & '* +   &  +   &    &  +   &- 5* & '  + '** *     ###      *  )( ,+ *+* * 2-  5*30 I3  J33  & &     , *   #0   ,,  * - 5* ' ;; )  7? + - 5* *  '       , & *  +  *    * , ' * +   & * $ A    A   K  I   -     .&+ 7/ * *  *  + 3 &$ * *  +  2)  *  ' ,   +   - 5* '  C' + 2  ) ,   * * *  +  *   * + & *  ,- 5* &' )  *  &   * 2  * 2) '** ' )   , * &$  , '  ( *  )2 , *   . * E 2/  ' '  ( *  )2 ' *   . * M 2/   *,, & .' +/ . * 5 2/-  '  ,)     ,    * '  &' ( )   *   ,, * & & ,+ '  ( +  ' H .* *  )   67?8/ *        ) ,    - 5* '       & 2   ' ,)  &  *    * , )* + *  ,   ,+ * C' + 2-    ' 3  '  ( * ,( -

191

L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

 $ #+    K$ #+    $ #,  & & N +  , & N  , #* #, .### &/ 























































































































































































& *  +* *  D+ 7

& *  +* *  D+ 7)

& *  +* *  D+ 7

  K .&+ 7)/ * *    & * ,  ,  +   &' .  ) 6J8  )(/- 5* *  ' *  &  *      ) * *  * * '  & + 2- 5* *  +  * &  * O P 2 .+ , ' ,( /  * *   * +*  *  % 2 .+ , ' ,( /-  *   ' *& ,)  & * *    * ' 2  , * ' - "'(  * '      *  *     ' , '  (3 * ,( ,-  *,* *  '  ) *+*   K *   )   * )2  *   '   ) * *  *(+ * ' 2-  *  '   *(  '  * ,  1* * @ *   *   '  ) 12   2+  ,  -  K   +  2) 32+ * +*    -    .&+ 7/ * ,  * ### &- * *  * * '  & 2 .  K/ ) *   * * ' ,  '**  '2 -- * '    )2  + '* * ' + 2   '* * '  +  * & -  * ' * * ,  & * ' ,  2 ,- * , & *     ' (  !               M &+ '  +  * * & '**  ) ' +   & ( & 7/ >& & ,4' ;/ > + =/ > + ?/ '2+ '  * +  A/ 1 ,,  )  * 2  G/ 1)  ,, & ,-  , &+ & *  &  '* * *  '   * 2 * &  '2+ '* * *-   (  * * '  &&  * 1  & *  )  * * $ '* * 0, & ; , & *   , ' ,  )   * & +  ,( >& & *  .7A ,/ 4 & +  '2 '  * + .7 ,/ 4  *  + * ,+ .= ,/- "'(   +   * +* , & *  '

192

L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

  ,   & *  - & * + , ? ' +  *   A ' +    ' - 5*  *  *    ,  +   )  &  *,* * ###     * 1  & *  + - 5* & * ' *   * , '*  + * ### &   ( * *    ) +- % & * *   * ,   0*) *+ )*( '** ' ,)  '* *  * ### &- # *  *  )  ) * ,  , *   * '   ,+   ,(  - * *  '* ' 2+   4 2 & ' &     *  '* * *- 5* &+  * , * * +       (  *  1  &  *  ) ,  0 &   * * +  &"'( * & '+ ,  ( & * , & * ### &   > 32+ ) 2+ +  + (  0,  .     
* *  '      + > /$ 7 ; = ? A G < I

$   2  IL !  "  #"$%&' $ *  ! (" $ $   ))$  #'  *  *+* !  (      $ *$ ' $  * - 
5* *      *   + * * *( *+ *  .  ?/ )    +  & * ,)   ,  +  *   .  G/- 5*    ( * * )*  * *   + (   * & ' . 
L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

193

,   * , & &(  - 5*   * ,,     > + , *  ) .* ,, $ 9! *(  +- '  *(   ' *  2 ) *    + '** + + '** &:/- 5* ,  *   *  *      & > +     &+ =)-

D+ =$ ### , $+ D+ =)$ ### , +

)( & * (  * * *    * *   * 32+ ,- ,,  && * & * ,  * & , & * 2 + * * ### & ,, (  2+   & ,,- *    * ( , & +  + ,(  * & ) ,*  ,+  &&  *   & *  & * * - 5*     * & '+ 0,$

 $ ( +  +( * )+  * ( ) '  +  * * * $ **  'Q

 $  *2 * ) 0 )   *(* $  *2 *  '  $ 5* ,  1 )+ '* *  * +Q * $  * 0 + * ,   -$"   ./

 $ * )  ,, Q * $   G  + *  $  +    '- **   &&Q * $ 5* ,,     +$ ! *( + * (   *( + * ) ' *(  > ,2 - " )* * ,  * *   + * ### &  & '* * +  + .-+-  7/- 5* *  ) &&(   *  >  , '** * )* +- %( *   *  * *(  +  & * ,, * ++ *' *  * () *  

5*  *(  * , & * (  & 3 #,  & #* #, 3   * , &  )( -   &&(

194

L. Kerawalla et al. / Scaffolding the Process of Collaboration: Exploration of SCOSS

 + * + &$ ( +  ,  23, ( @ ,+ +  +@  + ( +- 5* & &  )     )    * (- 5*   '  & * & *  ,, '** ' +   *  & * -  & * ' '  2  0,  *  & & )*(    -  ,        '** )*(  ,, )  + 3   ### &    +   *+* + & *' * &  *  )( ,-  ,   (+ '** +  ,  )*  )( 2    & * ### & '  ) & )&  * 3*  ,- D* ' *(   * , & * ### &     ,(  ) ( )  * 2 , '**  )  )     '**    && &* && + & *  )( ,- +  678 * - 5 ( -  "+ D- - .7JJ=/ #  ,,*  +  - -# - %2  - -# .-/- 0    1 0& ( @ ' !26;8 * - .7JJ7/ 2    3$45   5   3$ $5  - "( ( @ )+ % - 6=8 2 - .7JJ?/ 0   $ 0 "  6  1-  +@ 6?8 # --  %* 5-E- .;LL;/ ++ 523    +  ;  = #* ( 2,- 7 $  503  0       0 "    2   6    ,026  8&-. K E =J3?G6A8 5  #-  *  #- .7JJ=/ +  > ,)  ,$ * ,    & *+ 2' +  #- >  #-  0      "  $  6G83EF F %- #* -      # - .;LL=/ *+ ')3)  )( + )  ,)   (+   &&  ,,- 9   )  5   9  6$    M -7= ;G=3;JJ- 6<8  )+ - K2 %- K  -  R   .7JJA/- 5* (  & *   )( +-  - #,  -   1$3 4: *$   $    -  ( 0& 7JJA- 6I8 #*+  - "'2 -  * - .7JJL/ C * *2  * ,$ 5*  & * +  *  & &  -  - K3K  - #  .-/ 0   $ 1  3   - 6J8 # #- #*2 E-  2, - .;LLL/ 5'   ,, &   )(   7 $ ; 9   
% @  %- ,,7L=377L67L8 K& #- K K- 2 - K K-  - " - " - + -   "- R%  - # - # - #)  !- N 50 E .;LLL/ ++ # + * +  +  ) )' + * -  7 $ 09-&&& 5* "+ *  , 73G6778 #* - .7JJJ/ , ,, &  )( 1- 7 $  <-$*  9   0          A3I %@ "'67;8 * K- # 2 K-  % - .;LLL/ (  & & +,  )  7 $ <<$*9   0          ?3< %@ "'67=8  - '  - 2 - !  -  " - .&*+/ 5* 2 *+ &'2 &  )  3 )- + & *  & & ,   #+, ;LLA 67?8  - .&*+/ #&%$ % ,   ,, & ( :    "07 =>> & ( & #067A8  - .;LL?/ :?$6   (@ )  450  A$ 5      7 A0$ 

$3  $    0$@$ $ 3    %#  ( & #067G8  -  '  - .;LLA/ 5*  &  ,,  &&  ,3*   ) '*  ,3) ( 7 $  - 9   0    5     9  6$   5 $-

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

195

A System that Generates Word Problems Using Problem Generation Episodes Kazuaki Kojima and Kazuhisa Miwa Graduate School of Information Science, Nagoya University Nagoya, 464-8601 Japan phone: +81-52-789-4748 e-mail: [email protected] Abstract. In mathematical learning, it is important to give learners problems that have various features in both surface problem situations and deep mathematical structures. In this study, we implement a system that can generate various problems by altering those surface and structural features for mathematical word problems using problem generation episodes. Each problem generation episode is knowledge comprised of a base example problem and a new analogical instance, which is regarded as a past case where an analogical instance was generated from an example problem. Our system generates problems applying problem generation episodes to initial problems stored in the system. In this paper, we describe our approach for generating mathematical word problems, and perform preliminary experiments to verify whether our system can appropriately expand the variety of problems.

Introduction In typical mathematical learning, a teacher first presents a solution method by example problems, and students then learn by applying the instructed example to the solutions of new analogical problems. Thus, a number of problems are needed in such mathematical learning. Not only the number, but also the variety of problems is essential in learning. It has been well recognized as crucial to give learners problems that have various features in both surface problem situations and deep mathematical structures (e.g., see [1]). Here, problem situations denote contextual settings expressed in problem texts, such as the purchase of goods or transfer by vehicles. However, problems similar/dissimilar in surface situations and solution structures to example problems are not always available in workbooks. In the research of educational systems, the focus is on the development of frameworks and mechanisms to support learning, but the production of learning data is also crucial [2]. Several systems that generate problems in mathematical learning have been implemented. For example, MathTeX [5] generates mathematical work sheets in a LaTeX format. But its work sheets don’t include word problems that have problem situations. Although, many programs pose word problems, most only provide the same type of problems by changing the parameters included in the problem texts. Takano’s WBT system [4] poses word problems; however, her system can generate only a limited variety of problems composed of the same solution and problem situations prepared in the system. In another domain, for example, Kanenishi’s WBT system [3] generates qualifying examination problems for the Information-Technology Promotion Agency Japan (IPAJ) using given domain knowledge. However, since problem situations are not crucial in this domain, the system doesn’t need to generate various problem texts. Therefore, it is an

196

K. Kojima and K. Miwa / A System that Generates Word Problems

important challenge to implement a system that can generate various problems controlled by both features of problem texts and solutions. In this study, we implement a system that can generate new analogical problems by altering the surface and structural features of example problems. Our system aims to expand the variety of problems using the initial problems stored in the system. To generate word problems, some technical issues such as common sense reasoning must be overcome. So we adopt the folloing two approaches: (1) our system generates and uses an episode, which is a case where a new problem is generated from an example problem, and (2) our system acquires common knowledge through interactions with a teacher as a user.

1. A System That Generates Word Problems Our system stores initial mathematical problems and generates new problems from them. First, it acquires problems and common knowledge through interactions with a user, which are then used to generate new analogical problems. Our study focuses on problem generation for a teacher as a user, not on support the learning of students. Of course, in the future our system will support learning by presenting problems to students after the acquisition of sufficient knowledge. In the current study, we choose a problem domain of word problems solved by simultaneous equations in Japanese middle school mathematical education.

1.1 System Construction Figure 1 shows the construction of our system, which is comprised of four main components: an input analysis interface, a dictionary database, a casebase, and a production engine. The input analysis interface generates problem data from problems input by a user. The dictionary database has a conceptual dictionary with a thesaurus, and an ontology database that has knowledge of words used in past word problems. The dictionary provides word knowledge needed in problem generation. The casebase consists of two different databases, a problem database and a problem generation episode database that stores knowledge used to generate new problems. The production engine generates new problems using the dictionary database and the casebase. Input Analysis Interface Production Engine

A user

action-generating rules

Dictionary conceptual dictionary ontology database

Casebase problem database problem generation episode database

Figure 1. System Construction

1.2 Summary of the System's Problem Generation Procedures Problem generation procedures by the system are as follows. 1. Generating problem data: To store problem data in our system, a user first inputs a problem text and its solution in plain text format. The input analysis interface then analyses them and generates problem data. After that, the system requires users to add new information or modify the problem data that the system cannot understand correctly.

K. Kojima and K. Miwa / A System that Generates Word Problems

197

2. Generating problem generation episodes: The system uses problem generation episodes to generate new problems. Each episode is comprised of a base example problem and another problem regarded as an analogical instance. An episode is automatically generated by the system using a base example problem and an analogical instance that the user selects from problems stored in the system. 3. Problem generation: The production engine generates new problems from problem data, using problem generation episodes.

2. System Implementation In this section, we explain in detail the representation of problem data and problem generation episodes and the generation process of problem data and new problems.

2.1 Representation 2.1.1 Problem representation An example problem is given as follows. I bought some 80 yen postage stamps and 50 yen postal cards for 3100 yen. The total number was 50 mai1. How many stamps and cards did I buy? Let x denote the number of stamps and y denote the number of cards. x + y = 50 80x + 50y = 3100 According to the equations above, x=20, y=30. Table 1. Example of Problem Representation Properties

a: postage stamps x 80

1: mai 2: yen 3: yen Object Category Verb Situation

Objects b: post cards y 50

c: (all) 50

3100 postal matters buy purchase

Our system represents problem indexes in the form of a table. Table 1 shows the problem representation of this example problem. In the upper table (called the indexing table) shown in Table 1, Objects indicate entity elements appearing in problem texts, and Properties indicate the attributes and values of Objects. In the indexing table, the vertical relationship between values corresponds to multiplication (1 x 2 = 3), and the horizontal relationship corresponds to addition (a + b = c). Other information of the problem, such as a verb indicating the problem situation and an operation needed to evaluate a value included in the equation of the solution but not in the problem text, is included in an additional table (at the bottom of Table 1.) Our system independently identifies the features of problem texts and solutions. The former features correspond to the parameters and the latter to the structure of an indexing table. Our system generates a new problem by altering an example problem. A problem that has a problem text different from the example problem is generated by changing parameters in the indexing table of the example problem, and another one with a different solution from the example problem is generated by altering the structure of the indexing table of the problem.

1

In Japanese, we must append a specific numerical classifier to the number of an entity. "Mai" is a classifier for the number of sheets, such as stamps or cards.

198

K. Kojima and K. Miwa / A System that Generates Word Problems

2.1.2 Problem generation episodes representation Each problem generation episode is knowledge comprised of a single base example problem (base) and a single new analogical instance (new instance), which is regarded as a past case where a new instance was generated from a base. A relationship between a base and a new instance is used to generate new problems. In problem generation, a new output problem (output) is generated from given input example problem (input) by mapping a relationship between a base and a new instance in an episode. Solutions Solution 1 Problem Texts

Problem Text 1

base

Solution 2 new instance

Problem Text 2 Problem Text 3

input

output

… Problem generation episode

…

Figure 2. Basic Problem Generation Idea Using an Episode

Table 2. Example of Representation of a Problem generation Episode Base example problem: I bought some 80 yen postage stamps and 50 yen postal cards for 3100 yen. The total number was 50 mai. How many stamps and cards did I buy? (The indexing table of the example problem is identical to Table. 1.) New Instance problem: I buy three hon pencils and two hon pens for 480 yen. My friend buys five hon pencils and two hon pens for 760 yen. How much are the pencils and the pens? (“Hon” is a numeral classifier for pencils and pens.) Objects a: pencils b: pens c: (all) 1: yen 50 x y 2: hon 3 2 Properties 3: yen 480 4: hon 5 3 5: yen 760 (Vertical relations in this table: 1x2=3, and 1x4=5) Altering Actions Altering Object a and b into different ones. Replacing Property 1 and 2 with each other. Adding Property 4 by copying Property 2. Adding Property 5 by copying Property 3.

Figure 2 shows the basic idea of problem generation using an episode. In Figure 2, the horizontal axis represents the features in solutions of problems and the vertical axis represents the features in problem texts. Suppose that an episode is formed, comprised of a problem that has Solution 1 and Problem Text 1 as a base and another problem that has Solution 2 and Problem Text 1 as a new instance. When a problem that has Solution 1 and Problem Text 3 is given as input, the episode enables the generation of a new problem that has Solution 2 and Problem Text 3 as output. Therefore, problem generation by our system is regarded as generating output from input through solving four-term analogy "base: new instance = input: output." With this basic idea, our system increases the number of problems and expands their variety using episodes. Table 2 shows an example of a problem generation episode. Each episode is generated from two problems that a user selects as the base and the new instance. The system provides the episode with procedures called Altering Actions that describe which indexes are altered and how in problem generation, based on comparisons between indexes of the base and the new instance. Our system generates new problems by altering indexes of problems using altering actions embedded in episodes.

2.1.3 Problem text templates In problem generation by our system, an indexing table of a new problem is first constructed, and then a problem text is created. Generally, text templates that have slots to

K. Kojima and K. Miwa / A System that Generates Word Problems

199

be filled by keywords or numeric parameters are used to generate texts. But an identical template of a problem text can't be adapted to problems that have different problem situations or solutions, because the structure of a problem text depends on features of the problem. In our system, templates of problem texts are automatically created based on problem information input by users. Each template is created from a problem text in the input analysis interface (described in next section) by replacing keywords and parameters extracted as indexes from the text with empty slots. When the system generates a new output problem, the problem text of the output is generated using the most suitable template created from other problems.

2.2 Generation 2.2.1 Generating problem data To store a problem in the system, a user inputs a problem text and equations of the solution in plain text format. In the first step to generate problem data, the input analysis interface constructs a structure of an indexing table based on analysis of the equations and fills cells in the table with numeric parameters included in the equations. Simultaneously, the problem text is divided into word blocks based on dependency between words after morphological analysis of the text2. Keywords of the problem text are extracted based on comparisons between the indexing table and word blocks. In keyword extraction, if a numeral that has a value equal to a parameter in the table is included in a word block, a numerical classifier on which the numeral depends is identified as a property, and the noun included in the block is identified as an object. A verb that represents a situation in the text is also extracted, and a superordinate concept of the verb is then identified as a problem situation by retrieval from the conceptual dictionary. In this step, any decision concerning keyword extraction that the system cannot make on its own is left to the user. Initial problem data are then generated. In the second step, the system gives users feedback to confirm the problem data after checking on unsolved information, including numerals in the problem text but not in the indexing table, and numeric parameters in the indexing table but not in the problem text. The system lists the unsolved information and requires users to add information or modify data. The system also requires users to give common knowledge (such as numeric classifiers for nouns) to new words not stored in the ontology database but extracted as keywords from the problem text. The words are stored in the ontology database. Finally, the system creates a template from the problem text by replacing keywords extracted as indexes from the text with empty slots. This template is used to generate texts of other problems generated by the system.

2.2.2 Generating output problems To generate output problems, input problems and problem generation episodes are selected. First, a user selects an input problem from problem data stored in the system, and then the system retrieves episodes that can be adapted to the input. Whether an episode can be adapted to a problem is judged based on the similarity between the problem and the base of 2

In Japanese, morphological analysis is needed prior to extraction of keywords from texts because words are not separated in the sentences.

200

K. Kojima and K. Miwa / A System that Generates Word Problems

the episode. Judgement criteria can be customized by the user. For example, if a criterion in which any episode can be adapted to problems similar to the base in both of the problem text and the solution is set, output problems generated by the system may be similar to the new instance of the episode. Otherwise, if a criterion is set in which any episode can be adapted regardless of similarity, innovative problems may be generated; however, the risk that inappropriate problems emerge increases. After selection of input and an episode, output is generated in two steps: generating an indexing table and a problem text. An indexing table of the output is generated by altering input using a problem generation episode. The basic idea of problem generation has been already shown in Figure 2. As described previously, each episode has altering actions to generate the new instance from the base. The production engine generates output by adapting altering actions to input. A problem text of output is generated from a text template by filling empty slots of the template with indexing table parameters. This template is basically the one created from the problem text of input. If the template cannot be adapted to the output (e.g., when output is different from input in problem situations), the system retrieves the most suitable template for the output from the problem database. If no templates are available for the output, one of the default templates prepared in the system beforehand is used. In this case, the system requires users to revise the problem text because the texts generated from a default template are often inadequate. A new template is created from the revised text and added to the problem database of the system. Generated output problems are presented to users. Only valid problems evaluated by the user are stored in the problem database. If needed, the system retrieves new objects from the conceptual dictionary and then asks the user to supply common knowledge to the objects. They are stored in the ontology database. The system acquires new knowledge through interactions with the user. Knowledge stored in the system is increases with a user’s utilization of it.

3. A Preliminary Experiment of our System We conducted preliminary experiments to verify whether our system can appropriately expand the variety of problems.

3.1 Procedures and Materials The experiments were performed in three steps: (1) storing problem data, (2) generating problem generation episodes, and (3) generating problems using the episodes. Seven word problems solved by simultaneous equations stored initially in the system were acquired from a general workbook. The following list shows some of them, and Table 3 shows their indexes. z z

z

Problem 2: I bought some 80 yen postage stamps and 50 yen postal cards for 3100 yen. The total number was 50 mai. How many stamps and cards did I buy? Problem 4: I buy three hon pencils and two hon pens for 480 yen. My friend buys five hon pencils and two hon pens for 760 yen. How much are the pencils and the pens? Problem 6: Mixing 6% saltwater and 9% saltwater will produce 900 g of 8% saltwater. How much of the two kinds of the saltwater are mixed?

K. Kojima and K. Miwa / A System that Generates Word Problems

201

The problem generation episodes used in the experiments were generated from pairs of problems selected from the seven problems, as shown in Table 4. Not every pair of problems was effective because duplicate identical episodes appear3. In the experiments, we stored 80 word knowledge in the ontology database and 25 rules to provide altering actions in the production engine. Additionally, the criterion to judge episode adaptation was set so that each episode can be adapted to any problem. Table 3. Indexes of Material Problems Problem Texts Situation

purchasing

population mixing

Object Category fruit postal matters books writing implements male and female water metal

x+y=c1 a2x+b2y=c3 1 2 3

Solutions a2x+b2y=c3 a4x+b4y=c5

4

5 6 7

Table 4. Generated Problem Generation Episodes E1 E2 E3 E4 E5

base example problems 1 1 4 1 1

new instance problems 2 4 1 5 6

Summary Altering objects Altering objects and solution Altering objects and solution Altering situation to population Altering situation to mixture

3.2 Results Table 5. Indexes of Problems Generated by the System Problem Texts

Solutions x+y=c1 a2x+b2y=c3 a2x+b2y=c3 a4x+b4y=c5 Object Category fruit 1 2 postal matters purchasing 3 books Good writing implements 4 Good (others) Good Good population 5 male and female Impossible (others) Fair Impossible mixing 6 8 (i) water Impossible (i) 7 metal Impossible Good (others) Good / Fair Impossible (ii) Good / Fair (ii) Good: problems generated appropriately Fair: problems whose problem texts partly needed to be modified Impossible: problems whose problem texts could not be generated appropriately (In cells divided into two, ones on the left/right sides are before/after adding Problem 8.) Situation

Table 5 shows the problem indexes generated by the system. The object category "others" corresponds to those not included in the material problems. The objects of others, such as sweets or soft drinks, were retrieved from the dictionary database. Because objects are randomly selected from the dictionary, each problem can have different objects in every generation. The examples of output problems are as follows. z z

3 4

Output 1 (Generated from Problem 6 by E1): Mixing [a2] %4 solution and [b2] % solution will produce [c1] g of [c3] % solution. How much of the two kinds of solution are mixed? Output 2 (Generated from Problem 2 by E2): I buy [a2] mai drawing paper and [b2] mai pasteboards for [c3] yen. My friend buys [a4] mai drawing paper and [b4] mai pasteboards for [c5] yen. How much are the drawing paper and the pasteboards?

E.g., E1 in Table 4 has the identical altering actions to an episode that has Problem 2 as a base and Problem 1 as a new instance.

Symbols such as [a2] and [b2] are parameter slots to be assigned numeric values when presenting the problems to students. And [a2] denote a value in cell (a, 2) in the indexing table (see Table 1 or 2.)

202

K. Kojima and K. Miwa / A System that Generates Word Problems

In Table 5, cells marked Fair include problems whose problem texts had to be modified partly based on common knowledge, such as "Solder A includes copper." (Solder is an alloy of tin and zinc.) Cells marked Impossible include problems whose problem texts could not be generated appropriately because no templates were available. We performed an additional experiment after adding Problem 8 (included in cell (i)) in the system. As a result, the system successfully generated Output 3 (included in the cell (ii)) by using a template generated from the newly added Problem 8. z z

Problem8: Mixing saltwater A by 100 g and saltwater B by 200 g will produce 300 g of 8% saltwater. Mixing saltwater A by 200 g and saltwater B by 100 g will produce 300 g of 7% saltwater. What are the concentrations of the mixed saltwater A and B? Output3 (Generated from Problem 7 with E2): Mixing paint A by [a2] g and paint B by [b2] g will produce [c2] g of [c6] % paint. Mixing paint A by [a4] g and paint B by [b4] g will produce [c4] g of [c7] % paint. What are the concentrations of the mixed paint A and B?

As indicated by the results of the experiment, problem generation by our system depends on the degree of dispersion in the problem indexes stored in the system. These results above confirm that our system can expand the variety of initial problems.

4. Conclusions In this study, we implemented a system that automatically generates mathematical word problems by altering the indexes of the texts and solutions of the initial problems. Our system generates new problems using problem generation episodes regarded as past cases of problem generation. In the generation, our system acquires knowledge through interactions with teachers as users. We performed preliminary experiments and verified that our system can successfully expand the variety of problems. One of the most important future works is to propose learning support by using our system. Our system can be used as an intelligent workbook. Furthermore, it may be used to support students' learning by problem generation. We believe that our system can support students’ idea generation by presenting various problems to them or evaluating their problem generation based on the similarity between problems generated by them and stored in the system.

References [1]

[2]

[3]

[4] [5]

English, L.D. (1997). Children's reasoning processes in classifying and solving computational word problems. Mathematical reasoning: Analogies, metaphors, and images (pp.191-220). Mahwah, NJ: Lawrence Erlbaum Associates. Tsukasa Hirashima, Taichi Umeda, Takahiro Shiki, Akira Takeuchi (2001). An Environment for Acquiring and Sharing Arithmetical Word problems – For Problems Solved by Specific Solution Methods –. Japanese Society for Information and Systems in Education, vol. 18, no. 3&4, pp. 284-296. (In Japanese) Kazuhide Kanenishi, Kentaro Hayashi, Hiroyuki Mistuhara, Yoneo Yano (2003). Drill Exercise Generation in WBT from Knowledge Based Components. Japanese Society for Information and Systems in Education, vol. 20, no. 2, pp. 71-82. (In Japanese) Atsuko Takano, Jun Hashimoto (2003). Drill Exercise Generation Based on the Knowledge Base. Information Processing Society of Japan, NL-160, pp. 23-28. (In Japanese) Hitoshi Hata (2002). MathTeX. http://www12.plala.or.jp/htown/mathtex/ (In Japanese)

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

203

Detection and Grammaticality Judgment of Forms in a Language Education System Oriented for Focus on Form Makoto KONDO, Takafumi SHIRATORI, Naoki MINAI, Satoru KOGURE, Tatsuhiro KONISHI, Yukihiro ITOH Faculty of Informatics, Shizuoka University, Japan [email protected] Abstract. This paper describes how to develop a language education system oriented for a pedagogical approach called focus on form (FonF). FonF aims at improving skills to produce grammatically correct sentences. The approach has attracted much attention in the field of second language education because it could overcome a potential problem of another pervasively adopted approach called communicative approach (CA). The CA puts the highest priority on cultivating communicative fluency and grammatical correctness is considered less important than conveying intention. As a result, there is a risk that learners would miss a chance to correct their misunderstanding of grammatical rules in target languages. If FonF instruction is effectively incorporated into a language education system oriented for the CA, it should overcome the problem by giving appropriate grammatical instruction while learners are engaging in conversation practice based on the CA. In this study, we selected 145 linguistic forms to teach based on the FonF approach and implemented functions necessary for FonF instruction: a function to detect the linguistic forms in input sentences, and a function to judge whether detected forms are used correctly from the viewpoints of morphosyntactic correctness and semantic correctness. Keywords: Focus on form, communicative approach, foreign language education, dialog system

Introduction In the field of second language education, a pedagogical approach called focus on form (henceforth, FonF) has acquired much attention as a teaching method to improve skills to produce grammatically correct sentences [1]. One of the biggest reasons for the attention is that FonF could solve a potential problem of another pervasively adopted approach called communicative approach (henceforth, CA). The CA aims at cultivating communicative fluency and puts a higher priority on conveying intention than on making grammatically correct utterances. As a result, there is a risk that learners would miss a chance to correct their misunderstanding of target grammar. If FonF is effectively incorporated into a language education system based on the CA, it should help overcome the problem. In our previous studies we developed a Japanese education system based on the CA [2,3,4]. The system engages in role play with a learner under a given situation. Since the previous system was developed based on the CA, it has the same problem as the CA has. This paper describes how to extend the previous system in order to incorporate FonF instruction into the system. Section 1 describes the outline of the previous system and some more details about FonF. Section 2 deals with how to decide forms to teach in FonF instruction. Section 3 and the subsequent section describe how to implement new

204

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

functions for FonF. Section 5 shows an example of actual operation of our extended system. The final section summarizes what has achieved in this study and points out some remaining issues.

1. Fundamental Considerations: Functions Necessary for FonF Instruction 1.1 Outline of the Previous System We are developing a Japanese education system by extending a Japanese dialog system developed by the JDT project [5]. The JDT system produces semantic representations from input and accumulates the representations as context information. The system performs problem solving by referring to the context information and its problem-solving knowledge. The JDT system is designed for Japanese native speakers, and some additional modules were necessary to develop a language education system based on the JDT system. The outline of our previous system is given in Figure 1, where the added modules are given in bold letters. We added three modules to the JDT system. Since the JDT system is not designed to accept ungrammatical input, modules to process ungrammatical input were necessary. We also added another module to judge whether input is relevant to a given situation, since role play requires judgment of situation relevance.

input system reply

Dialog Control

Input/Output Interface

Response Generator

Interpretation Part grammatical

Semantic Representation Generator for Grammatical Input

Grammaticality Judgment Component

process flow

relevant

accumulate semantic representations into Context Information

reference

Semantic Representation Generator for Ungrammatical Input

fail

succeed Situation Relevance Judgment Component

data flow

ungrammatical

succeed irrelevant

Situation Knowledge

control

fail direct reply

Problem Solving Part

produce semantic representations

Context Information

Problem Solving Knowledge

Figure 1. Outline of the Previous System

The grammaticality judgment component judges whether input is grammatical. The component was developed by adding grammatical rules to the original JDT parser [2]. We made the grammatical rules based on typical error patterns extracted from an error analysis of nonnative Japanese speakers [6]. The added rules make the parser fail in parsing ungrammatical input. As a result, success and failure in parsing reflect grammaticality and ungrammaticality, respectively. The semantic representation generator for ungrammatical input produces semantic representations by referring to the situation knowledge [2]. Since the parser does not

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

205

produce correct dependency structure from ungrammatical input, the generator must produce semantic representations without referring to incorrect dependency. The generator produces semantic representations by comparing content words (i.e. nouns, verbs, etc.) in input with the situation knowledge. The situation knowledge is a set of the JDT-style semantic representations that expresses what learners should convey in a given situation. The generator recovers learners’ intention from a set of content words in ungrammatical input by referring to the situation knowledge. The situation knowledge looks like Figure 2 when a given task is to search for a hotel with a hot spring spa in Hamamatsu.

Situation Knowledge

search@object = hotel hotel_existence@location = Hamamatsu hotel_inclusion@object = hot spring spa

Figure 2. Situation Knowledge

Semantic representations produced from grammatical input go to the situation relevance judgment component, which examines whether semantic representations are relevant to a given situation by referring to the situation knowledge [3]. If semantic representations are relevant, they are stored in the context information; otherwise, the input is processed by the semantic representation generator for ungrammatical input, which produces situation-relevant semantic representations.

1.2 Extension towards a FonF-Based System Before discussing extension aiming at FonF, let us consider what FonF is. According to FonF, grammatical instruction is performed when learners make some errors in using linguistic forms. Linguistic forms are word usages, grammatical rules, grammatical constructions, idiomatic expressions, etc. Notice that it is relatively easy to incorporate FonF into conversation practice based on the CA. The CA maximizes opportunities to use linguistic skills and FonF plays a role only when learners make some errors. FonF has two modes of instruction: explicit instruction and implicit instruction. In the explicit instruction, teachers give explicit instruction on how to use linguistic forms. The implicit instruction, on the other hand, only shows correct linguistic forms to learners. The explicit instruction is effective when learners have correct knowledge on linguistic forms to some extent, and the implicit instruction is effective when learners has less knowledge on how to use linguistic forms [1]. We need to consider the following five issues to implement functions for FonF. Since FonF instruction is made based on linguistic forms, (1) we must decide which linguistic forms should be taught. In order to perform FonF instruction, (2) the system should be capable of detecting relevant linguistic forms in input. (3) It should also be able to judge whether detected linguistic forms are used correctly. Since the choice between the explicit instruction and the implicit instruction depends on learners’ levels of understanding, (4) the system should be able to evaluate learners’ levels of understanding. In order to implement complete instruction, (5) we must determine how to instruct each linguistic form, especially in the explicit instruction. This paper deals with the first three issues and we leave the last two for the future work.

2. Selection of Linguistic Forms While FonF instruction is based on linguistic forms, CA instruction is designed based on

206

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

pragmatic functions like order, request, etc. Accordingly, linguistic forms should be categorized from the viewpoint of their semantic functions. We selected 145 linguistic forms from Japanese textbooks for beginners [7,8] and categorized them in accordance with their semantic functions. Since grammatical instruction, especially explicit FonF instruction, interferes with conversation practice, selection of linguistic forms should be made so as to make possible interference as little as possible. Accordingly, we selected forms that are difficult to acquire in the CA alone. We selected four types of forms: (1) multiple linguistic forms corresponding to a single semantic function, (2) linguistic forms rarely used in ordinary conversation, (3) linguistic forms baring less importance for conveying intention and (4) linguistic forms difficult for learners to use correctly. When multiple linguistic forms correspond to a single semantic function, learners may use only one of the forms to convey intention. It surely prevents them from learning the other forms. Rare linguistic forms hardly show up in role play and are difficult to learn in the CA alone. Forms like conjugation are often irrelevant to conveying intention and it is difficult to teach them in conversation practice alone. Forms difficult for learners to use correctly need extra instruction in addition to CA instruction. We categorized the selected 145 forms based on their semantic functions. For example, both “verb-yoo” and “verb-mashoo” are verbal forms expressing suggestion. We grouped them and other forms denoting suggestion into a function type suggestion. The selected 145 forms were categorized into 59 function types. Table 1 shows some of the function types and example forms belonging to each function type. Table 1. Function Type Function Form

Request V-tekure V-tekudasai V-tehosii

Desire V-tai

Report V-sooda V-toyuu

Source N-kara

Cause V-kara V-node V-te

Assumption V-tara V-nara

V: verbal elements N: nominal elements

3. Detection of Linguistic Forms Detection of the 145 linguistic forms is performed by referring to a function-form dictionary. The function-form dictionary stores information on correspondence between semantic functions and linguistic forms denoting them. In the dictionary, each entry consists of the name of a semantic function and subentries (i.e. linguistic forms corresponding to the function). Each subentry consists of the name of a linguistic form and morphosyntactic information on the form. Figure 3 shows how the information is stored in the function-form dictionary. If multiple linguistic forms belong to a single function type, information on every form is described under the same entry.

Function-Form Dictionary

function type 1 linguistic form 1 morphemes constituting the form morphosyntactic structure of the form

Figure 3. Function-Form Dictionary

Since every linguistic form has its own morphosyntactic structure, the selected linguistic forms can be detected from input by comparing the morphosyntactic structure of the input with morphosyntactic structures stored in the dictionary. The detection process

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

207

proceeds as follows; the system (1) picks up a morpheme in input, (2) search the function-form dictionary for linguistic forms containing the same morpheme (go back to (1) if no corresponding forms are found), and (3) checks whether the input contains one of the morphosyntactic structures associated with the obtained linguistic forms. Forms are detected if the answer to the process (3) is positive. The entire process is applied to every morpheme in linear order from the sentence-initial morpheme to the sentence-final morpheme. Linguistic forms in grammatical input are correctly detected by the process explained above; however, ungrammatical input may involve a sequence of morphemes that does not match with a linguistic form even if the ungrammatical sequence is intended to express that particular linguistic form. Take -kara for example. It is a nominal particle denoting a source (i.e. Japanese counterpart of English from). It is always attached to a noun denoting a source. Some Japanese learners, however, use this particle incorrectly and produce phrases like Burajiru kita kara roodoosha, which should be expressed as Burajiru kara kita roodoosha (laborers who came from Brazil) [6]. In this particular example, the process mentioned above does not detect the form “noun-kara” since -kara in the input does not immediately follow a noun. In order to deal with such ungrammatical input, strict matching of input with forms in the dictionary is insufficient. Accordingly, the comparison between input and forms in the dictionary is carried out taking the following cases into consideration: (1) morpheme sequences which would match with a form if order of the input morphemes were altered, and (2) morpheme sequences which would match with a form if intervening irrelevant morphemes were omitted. In addition to morpheme sequences that strictly match with a form in the dictionary, the system regards the two types of ungrammatical morpheme sequences as forms to be detected. Some linguistic forms cannot be identified based on their morphosyntactic structures alone. Take -te for example. It is a conjunctive particle denoting either cause or coordination. In the both usages, the particle is attached to a verbal element placed at the right-most position in a clause, and connects the clause to another clause. In such cases, the system lists up every candidate for the input and identification of the function type is performed in the semantic judgment process to be described in the next section. The detection process described in this subsection is carried out right after the syntactic parsing is completed. If a form is included in grammatical input, the name of the function type is attached to the relevant element in its dependency tree, and the name of the function type finally appears in its semantic representation attached to the relevant concept. If a form is contained in ungrammatical input, the name of the function type is attached to the relevant morpheme, and the semantic representation generator for ungrammatical input produces a semantic representation in which the name of the function type is attached to the relevant concept. Information on detected linguistic forms is stored with actual morpheme sequences in input.

4. Grammaticality Judgment of Linguistic Forms There are two judging criteria for grammaticality of linguistic forms: a morphosyntactic criterion and a semantic criterion. The morphosyntactic judgment checks whether a given form is morphosyntactically correct. The process is accomplished by strict matching of input with linguistic forms in the function-form dictionary. Since each linguistic form has its own morphosyntactic structure, grammatical forms in input necessarily have the same morphosyntactic structures as the corresponding forms in the dictionary. The process of morphosyntactic judgment is the same as the process of form detection described in the previous section except that the latter also involves detection of ungrammatical morpheme

208

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

sequences. Therefore, detected morpheme sequences are judged as morphosyntactically correct only if they strictly match with forms in the dictionary. The other detected morpheme strings are judged as morphosyntactically incorrect. The judging process is performed during the detection process. In order to perform the semantic judgment of a linguistic form, the system should be capable of identifying every possible situation in which the form can be used. It is almost impossible for a dialog system to perform the semantic judgment in real-world conversation because it requires almost infinite knowledge to accomplish the task. If a conversational situation is well delimited, however, the system can predefine possible linguistic forms used in the situation. Since our Japanese education system engages in conversation with a learner under a predetermined conversational situation, the situation is narrowly delimited and we can predefine a set of possible linguistic forms used in the situation. Our system has the situation knowledge storing information that learners should convey in a given situation (section 1.1). The semantic judgment is performed by comparing input with semantic representations in the situation knowledge. Semantic representations of learners’ input contain the names of the function types of linguistic forms detected in the input (section 3). We modified the situation knowledge so that semantic representations in the knowledge should involve the names of function types that can be used in a given situation. The names of function types are attached to the relevant concept in semantic representations in the situation knowledge. As a result, the system judges input linguistic forms as semantically correct if their semantic functions match with function types given in the situation knowledge. If form detection process detects more than one function type with respect to a given linguistic form (section 3), the semantic judgment identifies which form is used in the input. If one of the detected functions matches with the relevant function type in the situation knowledge, that function type should be the one used in the input. If none of them matches with the situation knowledge, the linguistic form is used semantically incorrect manner irrespective of its intended function type. The semantic judgment of linguistic forms is applied to semantic representations relevant to a given situation. This is because input irrelevant to a given situation is beyond the scope of the situation knowledge and the system cannot determine semantic correctness of linguistic forms detected in the input. Since situation-relevant semantic representations are always obtained as successful output from either the situation relevance judgment component or the semantic representation generator for ungrammatical input [3], the semantic judgment is carried out in the situation relevance judgment component and the semantic representation generator for ungrammatical input. The semantic judgment proceeds as follows. (A) Obtain a semantic representation matching with an input sentence from the situation knowledge. If the input matches with no representations in the knowledge, the semantic correctness is undetermined. (B) Pick up a linguistic form in the input. If no forms are detected, the judgment for the input is trivially completed. (C) Check whether the form picked up in (B) matches with a function type attached to the relevant concept in the semantic representation obtained in (A). If the function type in the input matches with the function type in the knowledge, the form is semantically correct; otherwise, the form is semantically incorrect. The process from (B) to (C) is recursively applied to every detected form in a given input. Suppose that a learner is given a situation in which the learner must find a hotel in

209

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

Hamamatsu and he/she is required to use a linguistic form denoting request. In this case, the situation knowledge contains a semantic representation given in Figure 4, where linguistic forms are placed in square brackets. The system understands that the hotel in question is located in Hamamatsu because the existence@location attribute of the hotel concept matches with the same attribute of the existence concept and the value of the latter is Hamamatsu [5]. Suppose further that the learner’s input is “Hamamatsu-no hoteru-o sagashi-tai (I want to find a hotel in Hamamatsu)”. The semantic representation derived from the input is shown in Figure 5. The semantic representation in Figure 5 matches with the semantic representation in Figure 4 since the object of the search is a hotel which is located at Hamamatsu in both representations [3,5]. On the contrary, each function type attached to the search concept does not match with the other. One is request and the other is desire. Accordingly the system correctly judges that the use of “-tai” in this situation is incorrect. search [request] ... search@object ...

nominative relative clause [relative clause]

hotel existence@location ...

[locative particle]

Hamamatsu ... ...

existence existence@location ...

Figure 4. Situation Knowledge with Linguistic Forms search [desire] ... search@object ...

hotel existence@location ...

Hamamatsu ... ...

Figure 5. Semantic Representation: I want to find a hotel in Hamamatsu

5. Example Operation We implemented the processes of form detection, morphosyntactic judgment and semantic judgment into our Japanese education system. Ideally, FonF instruction should be preformed based on learners’ levels of understanding, which should be worked out based on the result of the form detection and the grammaticality judgment of detected forms. At present, however, functions for calculating learners’ levels of understanding and performing actual instruction have yet to be implemented. Accordingly, this section provides output of debugging information on the relevant processes in order to show that the system works correctly with respect to the form detection and the grammaticality judgment of detected forms. input sentence: Hamamatsu-ni aru hoteru-o sagasu-daroo Given Situation: (I will probably search for a hotel in Hamamatsu) Find a hotel with a hot form: guess spring spa in Hamamatsu. attached place: search Requirement: morphosyntactic correctness: correct Use linguistic forms semantic correctness: incorrect denoting request form: locative particle attached place: existence [existence@location] Hamamatsu morphosyntactic correctness: correct semantic correctness: correct Settings in Situation Knowledge: input sentence: Onsen-tsuki-no hoteru-o sagashi-te [request] is attached to “search”. (Please search for a hotel with a hot spring spa) [locative particle is attached to a form: request link from “existence@location” to attached place: search “Hamamatsu morphosyntactic correctness: correct semantic correctness: correct

Figure 6. Example Operation

210

M. Kondo et al. / Detection and Grammaticality Judgment of Forms

Figure 6 shows how the system works. In this example, it is supposed that a learner is looking for a hotel with a hot spring spa in Hamamatsu, and that the learner is also required to use linguistic forms denoting request. The situation knowledge contains information on which linguistic forms can be used in this situation; that is, a form denoting request should be attached to the search concept and a locative particle should be attached to Hamamatsu. The first input contains a linguistic form denoting guess (-daroo) and a locative particle (-ni), and the second input involves a linguistic form denoting request (-te). The system correctly detects all those linguistic form in the input sentences. Although they are all morphosyntactically correct, the form denoting guess is used in a semantically incorrect manner. The given situation requires that forms denoting request should be used rather than forms denoting guess. The system correctly judges that all the detected forms are morphosyntactically correct and that only the locative particle and the request form are used in a semantically correct manner. The example thus shows that the system correctly performs form detection, morphosyntactic judgment and semantic judgment. In order to confirm the practical usability of the system, we need further experiments involving a wider range of data/subjects. We leave the issue for the future research.

6. Concluding Remarks We have extended our previous CA-based language education system in order to incorporate functions necessary for FonF instruction into the system. We have implemented three functions (form detection, morphosyntactic judgment and semantic judgment) and added them to the previous system. We have further confirmed that the added functions work correctly. Some other functions for FonF instruction (i.e. evaluation of learners’ levels of understanding and actual FonF instruction to learners) have yet to be developed and implementation of these functions remains as issues for the future research. Experiments involving a wider range of data/subjects also remain to be conducted.

References [1] C. Doughty and J. Williams. (eds.). (1998). Focus on Form in Classroom Second Language Acquisition. Cambridge: Cambridge University Press. [2] M. Suzuki, T. Itoh, T. Konishi, M. Kondo, and Y. Itoh. (2002). Interpretation of Ungrammatical Sentences in a Language Education System for Nonnative Speakers. Proceeding of ICCE 2002, Vol.1, pp.912-916. [3] T. Shiratori, T. Itoh, T. Konishi, M. Kondo, and Y. Itoh. (2003). Development of a Situation Relevance Judgment Component in a Language Education System for Nonnative Speakers. Proceeding of ICCE2003, pp.1215-1219. [4] T. Shiratori, N. Minai, T. Itoh, T. Konishi, M. Kondo, and Y. Itoh (2004). Error Detection and Response-Manner Selection in a Japanese Education System for Nonnative Speakers. Proceeding of ICCE2004, pp.1383-1389. [5] Y. Noguchi, Y. Ikegaya, A. Takagi, H. Nakashima, T. Konishi, T. Itoh, M. Kondo, Y. Itoh. (2002). A Framework for Semantic Representations for a Natural Language Dialog Systems. Proceeding of SNLP-Oriental COCOSDA, pp.231-236. [6] Y. Ichikawa. (1997). Nihongo Goyo Reibun Sho-Jiten [A Glossary of Errors by Nonnative Japanese Speakers]. Tokyo: Bonjinsha. [7] The Association for Overseas Technical Scholarship. (eds.) (1990) Shin Nihongo no Kiso I [New Introduction to Japanese I]. Tokyo: 3A Corporation. [8] The Association for Overseas Technical Scholarship. (eds.) (1993) Shin Nihongo no Kiso II [New Introduction to Japanese II]. Tokyo: 3A Corporation.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

211

Concerns and Preferences of Teachers towards the Development of Virtual Classroom in Hong Kong Siu Keung LAM, Percy Lai Yin KWOK Chinese University of Hong Kong (Hong Kong, China) Abstract. The study investigates the development of virtual classroom in secondary schools in Hong Kong from in-service teacher perspectives and focuses on teachers’ concerns towards the virtual classroom and their preferences on designing virtual classroom. Quantitative survey data came from 352 secondary school teachers and qualitative interview data was collected from 8 of the surveyed teachers. The results indicated that major concerns of most studied teachers were lower stages of typical non-user concerns profile, implying that virtual classroom was still at the infancy stage of development in some secondary schools of Hong Kong during the research period. A minority of teachers, who exhibited high competency in information literacy and were competent users of virtual classroom, had their concerns profiles shifted to higher stages of concern. An analysis of teachers’ paradigm for the use of virtual classroom revealed that most teachers still stuck to the traditional teacher-centered pedagogy and employed mainly receptive and directive architecture in teaching without using virtual classroom in guided discovery and exploratory approaches. Strategic plan and staff development program for the success of implementation of virtual classroom are finally discussed in this paper. Keywords: Virtual classroom, teacher concern, professional development

Introduction In principle, virtual classroom provide an innovative environment for teaching and learning, inferring that new student-centered pedagogical methods can be adopted for big successes in cultural transformation. In respect of this, teachers, who are the masters of virtual classrooms, are indispensable to the success of implementation of e-learning in schools. The first exemplary use of virtual classroom in Hong Kong could be traced back to 2003, during the school suspension period caused by the outbreak of SARS, via an e-Learning VITLE platform to help interested school teachers to interact with their students and to conduct lessons in virtual classrooms over the Internet (A Search for New Heroes, 2004), awakening local policy makers to realize the big potentials of online schooling. Noteworthy, only a few pioneer primary and secondary schools have even developed their own e-learning platforms to extend learning beyond traditional classroom in Hong Kong. There has been lack of research data, justifying policy decisions in developing virtual classroom locally. Facing all these local change ahead, it is worthwhile to investigate the development of virtual classroom in Hong Kong from the perspective of in-service teachers. Teachers’ concerns and preferences on

212

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

designing virtual classroom need to be scrutinized to reveal their paradigm on this new and interactive environment.

1. Literature Review 1.1 E- learning and virtual classroom Since e-learning is developing rapidly, its definition has to be adjusted regularly to accommodate its most recent development. E-learning is often defined as “wide-ranging uses of technology to enhance educational experiences” (Tan , 2003, p.190). Another definition is “a term covering a wide set of applications and processes, such as web-based learning, computer-based learning, virtual classrooms, and digital collaboration” (Kaplan-Leiseron, 2005). Some other definitions take a much simpler form such as “it is a methodology of transferring knowledge that is supported by multimedia content delivered synchronously and/or asynchronously over an electronic network” (HiSPEC Limited, 2003). One of the most popular strategies of implementing e-learning is by establishing an e-learning platform. “A learning platform is an application that enables students and staff to share information and use the school network to its full advantage. School work can be accessed from home as well as from school and many reporting and monitoring tasks can be integrated into the system” (Hertfordshire Grid for Learning, 2005). Indeed virtual classroom is one of the major components of an e-learning platform. To facilitate teacher’s use of virtual classroom, an e-learning platform is designed with various simple online teaching tools to facilitate teachers to construct virtual classrooms on their own with a need of just a few hours of training (Cheng, 2001). Kaplan-Leiseron (2005) defined virtual classroom as “the online learning space where students and instructors interact”, supporting all or most of the types of communication and learning activities available in the ‘traditional’ classroom … space like a classroom, where the ‘teacher’ or others may ‘lecture’ and where group discussions may take place” (Hiltz, 1994, p.6). Besides, a virtual classroom should be able to “incorporate a wide range of technologies and use a great number of software programs in its creation” (Lynch, 2004, p.22). Virtual classrooms, developed on an e-learning infrastructure, with distance learning and online learning modes, can provide an innovative environment for teaching and learning to take place in high schools (Zucker & Kozma, 2003). In view of this, new student-centered pedagogy should be adopted for the development of virtual classroom. 1.2

Teacher’s paradigm

‘Paradigm’ of teacher is a set of assumptions, concepts, values, and practices that constitutes a way of viewing the use of virtual classroom (Göktürk, 2005). Richard, Rhys & Levcho (1986) argued that with the development of information technology, we need a pedagogy increasingly directed towards an understanding of information and its manipulation, rather than a pedagogy oriented towards the acquisition and retention of factual knowledge. Unfortunately, most teachers “still believe that the essence of good teaching is the passing on of information from a teacher to a student who passively accepts, memorizes, and reproduces on worksheets and tests what the teacher deems to be important.” (Clark, 1997, pp.4-5). In a survey undertaken in one modern Scottish university, most academic staff members have

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

213

conceptions of the Web as a medium for information transmission and their uses of the Web are far removed from the concept of networked learning (Roberts, 2004). Among Hong Kong schools, despite the availability of the necessary infrastructure and support, innovative pedagogical practices have remained rare since the ‘Five-year Strategy’ was launched in 1998 (CITE, 2003). The Final Report of the Overall Study on Reviewing the Progress and Evaluating the Information Technology in Education (ReITEd) Projects during 1998-2003 reported that very little paradigm shift seemed to have occurred since the preliminary study conducted in 2000 – 2001 among the majority of Hong Kong school teachers in basic education. It also remarked that “actual classroom use of IT is still more related to teacher-centered rather than student-centered learning, involving predominantly didactic expository teaching such as explanation and demonstration. There was relatively less collaborative interaction focusing on facilitating learning and assessment or for tasks requiring higher-order thinking skills” (Education and Manpower Bureau, 2005, p.iv). Rakes and Casey (2002) gave a possible explanation to the lack of success in the use of technology in classroom as teaching training was viewed as just simple skill acquisition instead of as a change process that affected the behavior of individuals at a very profound level. Teacher, as a master of virtual classroom, totally controls how a virtual classroom works. If we could not see a paradigm shift among teachers, a virtual classroom would only be extended as another type of traditional classroom and no innovative pedagogical methods could be found. 1.3 Change as a process Hall & Hord (2001) stated that “change is a process, not an event” (p.4) and “an organization does not change until the individuals within it change” (p.7). Change should focus on individuals and first in terms of attitudes, then in terms of knowledge and skills (Casey, Harris & Rakes, 2004). “Most changes in education take three to five years to be implemented at a high level” (Hall & Hord, 2001, p.5). It implies that professional development programs should be dynamic, individualized, target-oriented and on-going. Teachers, as front-line workers, always play the crucial role for undertaking most changes in education. Fuller (1969) hypothesized that “beginning teachers had concerns in three major areas: self, task and impact” (as cited in Kirby & LeBude, 1998, p.2). Hall, Wallace & Dosset (1973), based on Fuller’s work, proposed the Concerns-Based Adopted Model (CBAM) to study the concerns of teachers facing the implementation of innovations. “To be concerned means to be in a mentally aroused state about an innovation” (Rakes & Casey, 2002, p.2). Teacher concerns, describing feelings and perceptions about the innovation and the change process, follow a developmental pattern named as the ‘Stages of Concern’ (Hall & Hord, 2001). The notion of ‘Stages of Concern’, consists of seven specific categories of concerns, with ‘awareness’ as the lowest level concern and ‘refocusing’ as the highest level of concern with full depiction in table 1. This model has become a change model widely used for staff development accompanying any educational innovation. “According to Hall, an individual’s concerns directly affect performance; and since concern levels correspond with levels of performance, lower concerns must be removed before higher level concerns can emerge” (Rakes & Casey, 2002, p.3). Concern is an ‘indicator’ of the progress of a change process.

214

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

Table 1. Stages of Concern: Typical expressions of concerns about innovation (Hall & Hord, 2001, p.61)

Area of concern

Impact

Task

Self

Stage of Concern 6 Refocusing

5 Collaboration

4 Consequence 3 Management

2 Personal 1 Informational 0 Awareness

Expressions of Concern I have some ideas about something that would work even better. I am concerned about relating what I am doing with what my co-workers are doing. How is my use affecting clients? I seem to be spending all of my time getting materials ready. How will using it affect me? I would like to know more about it. I am not concerned about it.

1.4 Instructional Architectures An e-learning platform should be supported with technology, including technology infrastructure, content development tools, learning management system (LMS) and learning content management system (LCMS) (Tan, 2003). To develop an e-learning platform in a school, most teachers should not be scared and discouraged by the technology part. Leafe (2001) discussed that a change for developing a learning community should not only involve “connecting the learning community”, but also involve developing a range of strategies for “learning within a connected community” (Leafe, 2001, p.181). Actually, teachers should spare most of their energy to develop contents or learning objects for e-learning platforms, instead of being burdened by the technology part. Teachers need to apply different instructional theories to design and develop their own virtual classroom. Clark (2002, p.10). proposed four main architectures for instructional design, respectively receptive, directive, guided discovery and exploratory. The receptive architecture features that teaching and learning are dispensing and absorbing information. The directive architecture requires frequent responses to questions from learners and gives appropriate feedback. Content is usually presented in short lessons, from simple to complex. Guided discovery architecture features constructive learning process mediated by problem-solving. Learners are given with authentic job problems to solve. Exploratory architecture incorporates high levels of learner control. Learners are provided with a rich network of relevant resources (Clark, 2002; Lim, 2002). Noteworthy, receptive and directive approaches in teaching and learning are traditional and still widely adopted by many teachers. Receptive and directive architectures could work well in both traditional and virtual classrooms. Nevertheless, virtual classroom, as discussed before, favors the instructional designs using guided discovery and exploratory architecture. Different instructional architectures (possible determinant for realizing full potential benefits of virtual classroom) are employed to design learning contents for the virtual classroom will lead to completely different learning outcomes.

2. Research Direction The purposes of this study were to categorize teachers into different stages of concern on virtual classroom, based on the Concerns-Based Adopted Model (Bybee, 2004; Hall & Hord, 2001); and to incorporate their virtual classrooms to reveal their paradigms on this new

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

215

environment, according to preferable types of learning activities. Four research questions are addressed as follows: i. What are the major concerns of teachers towards virtual classroom? ii. What are the factors affecting teacher concerns towards virtual classroom? iii. What are the preferences of teachers on designing virtual classroom? iv. What is the teachers’ paradigm for the use of virtual classroom? The collected data were analyzed to find out the factors affecting teacher concerns. The correlations between teachers’ stages of concern and preferences on designing virtual classroom were investigated. The implications of the findings for devising necessary strategic plan and staff professional development program for the success of implementation of virtual classroom were finally discussed. 3. Research Methodology The study consisted of two parts. Firstly, quantitative data was collected on the teachers’ concerns and preferences on designing virtual classroom, aiming at construction of a general view of teachers about the issue. Secondly qualitative data was collected through semistructured interviews so as to have in-depth understanding of teachers’ concerns and preferences on designing virtual classroom. Both types of data were used collectively to explore the teachers’ paradigm for the use of virtual classroom. 3.1 Stage One: A teacher Survey A teacher survey was conducted using both convenient sampling and snowball methods. The questionnaire was anonymous unless some teacher respondents were willing to offer their self-identify for follow-up interviews. Eleven schools were conveniently selected, resulting in 352 valid questionnaires. The questionnaire consisted of demographic information of the subject, measurement of teacher concerns about the development of virtual classroom, based on based on the Concerns-Based Adoption Model (Hall, George & Rutherford, 1979) and teachers’ preferences on designing virtual classroom. 3.2 Stage Two: Interviews In the second stage, qualitative research approach was employed. The design was phenomenological in nature. The purpose was to obtain a view into the participants’ lifeworlds and to understand their personal meanings constructed from their ‘lives experiences’ (Johnson & Christensen, 2000). Data was collected through in-depth interviews to investigate on teachers’ concerns and preferences on designing virtual classroom. Since the main purpose of the interviews was to elicit qualitative differences, the interview transcripts were examined in details by the researchers, who searched for significant statements relevant to the research questions. Those cited statements were used to understand the uniqueness of individuals, but at the same time, their commonality or essence (Johnson & Christensen, 2000).

4. Results 4.1 Major concern towards virtual classroom

216

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

Analysis of the openness of teachers to virtual classroom shows that teacher concern followed a developmental pattern. Non-users were intense with lower stages of concern while users had more intense higher stages of concern. The teacher survey revealed that major concerns of most teachers towards the virtual classroom were informational, personal and management ones, which were lower stages of concern, implying that the development of virtual classroom in those studied secondary schools of Hong Kong still maintained at an infancy stage. Upon schematic categorization of all interview transcripts, three independent areas of teacher concerns could be identified, namely, “time concern”, “student concern” and “technology concern”. The key ingredients of each concern areas of teacher concerns are given in Figure 1. Technology installation and maintenance adaptation user-friendliness

ƒ ƒ ƒ

ƒ ƒ ƒ Teacher

ƒ

Student their participation their initiative to learn their capability to learn independently inaccessibility to the virtual classroom

Time round-the-clock teaching preparation for teaching materials regular renewal of the contents other administrative work not related to virtual classroom

ƒ ƒ ƒ ƒ

Figure 1. Different areas of teacher concerns towards virtual classroom

4.2 Factors affecting teacher concerns towards virtual classrooms Teachers' concerns (by subject)

11.00

Teachers' concerns (by IT competency level) 12.00

10.00

11.00 10.00 9.00

Chi

8.00

7.00

Maths

CIT

6.00

Sci Hum

5.00 4.00

IIT

UIT

6.00

AIT

5.00 4.00 2.00

en ar

s es 1

In

rm fo

io at

n

al 2

P

so er

n

3

al m ge

e

nt ue eq

nc

e

b a s la an ol on M C C 5 4 Stage of concern

io at or

n

6

R

u oc ef

si

ng

Figure 2. Concerns profile and major subject taught.

Aw ar en 1 es In s fo rm at io na l 2 Pe rs on 3 M a l an ag em 4 Co en ns t eq ue 5 nc Co e lla bo ra tio 6 n Re fo cu si ng

Aw

BIT

7.00

3.00

3.00

0

Score

Eng

8.00

0

Score

9.00

Stage of concern

Figure 3. Concerns profile and IT competency level.

Results of t-test or ANOVA test with a significant level of 0.05 show that the stages of concern of teachers do not significantly depend on gender, teaching experience and major subject taught (except for Computer and Information Technology (CIT) teachers). However, it is found that the concern profiles of CIT teachers and teachers who had attained Advanced IT (AIT) competency level are significantly different from that of other teachers (Figures 2 and 3). They are more intense with higher stages of concern (consequence and collaboration). The

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

217

breakdown of users of virtual classroom indicates that the users actually came from a wide spectrum of major subjects taught and IT competency levels. It implies that teachers’ participation in using virtual classroom could effectively move the teachers, not only IT experts, from lower stages to higher stages of concern. 4.3 Teacher preferences on designing virtual classroom Quantitative survey data revealed that teachers’ preferred activities mainly involved low-order thinking skills and employed receptive and directive instructional architectures. Teachers perceived virtual classroom as a resource centre or a place of communication between students and teachers. Students continued to play a passive role and were reactive to teachers’ instructions. There was no evidence to support that this new environment was used for collaborative learning, activities involving high-order thinking skills, and activities using guided discovery and exploratory approaches. Analysis of transcripts of interviews with eight teachers A-H showed variation in teacher openness towards the development of virtual classroom. Four different categories of openness were identified. Teachers A and G belonged to the category of “not opening to change”; teachers C and D were “open to practice with reservation”. Teacher F was “open to exploration”, whilst teachers B, F and H were “regular users facing openness”. 4.4 Teachers’ paradigm for the use of virtual classroom Most of the interviewees (teachers A-H) claimed that the roles of teacher and traditional classroom could not be replaced by virtual classroom. Virtual classroom was to supplement traditional classroom or as an extension of traditional classroom. It could provide additional or other learning opportunities for students. Many interviewees preferred to use a virtual classroom as a resource centre. Teacher B uploaded exercises for students to work with. Teacher H uploaded both handouts and exercises to the virtual classroom. Teacher C was prepared to post additional teaching resources for students to read. Virtual classroom was also viewed as a place for students to communicate with their teachers and classmates. Evidences to support this idea could be found in the transcripts of interviews with Teacher D, Teacher E and Teacher F. Both Teacher D and Teacher E further pointed out that the virtual environment could encourage those passive or shy students to raise their problems in it. Teacher F commented that teacher could use this environment to counsel their students.

5. Implications and Discussion 5.1 Major obstacles Based on the above findings, the development of virtual classroom in some secondary schools in Hong Kong has merely attained an infancy stage. Virtual classroom in student-centered pedagogy has not widely adopted yet. Those teacher stakeholders were inherited with the traditional teaching paradigm to work in this new e-learning environment. This study did not provide evidences to support that virtual classroom have provoked new teaching pedagogy and provided an innovative environment for learning activities to take place. Virtual classroom is viewed as a supplement, instead of an alternative to traditional classroom. But it has given evidence that some pro-active teachers’ participation in using virtual classroom

218

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

could effectively move their concerns from lower to higher stages, whilst it also showed evidence that many teachers were unwilling to spend their time and energy into this new innovation. It followed that low teachers’ participation rate in virtual classroom was an obstacle to the development of virtual classroom in Hong Kong. Meantime, an innovative learner-centered model of virtual classroom should be re-conceptualized in Hong Kong, instead of marginalizing the dichotomy between teacher-centered and student-centered pedagogies (Moss, 2001). 5.2 Strategic plan and staff development program Some recommendations on how to remove the lower stages of concern of teachers can be entailed from some interview transcripts. Some teachers urged the needs of supports from the school, such as the employment of teacher assistants and I.T. technicians, to reduce teachers’ workload in the administrative work and preparation for the virtual classroom. The three areas of teacher concerns, respectively time concern, student concern and technology concern, identified in this study could provide information for individual school consideration in setting its strategic plan to remove the informational and personal concerns of teachers. Some surveyed schools also need to introduce some policies to encourage a higher teachers’ participation rate in using virtual classroom in future. Besides, some teachers suggested having sharing sessions for teachers, which encourage collaborative learning culture among teachers. Exposure to exemplary use of virtual classroom is also an effective way to help cultivate into teachers a new e-learning paradigm and readjust their teaching pedagogy with this innovation. As pointed out previously, change is a process, not an event, which takes years to complete. Year-round workshops should be accomplished by individual tutoring and regular help to suit the personal needs of individual teachers. A long-term action plan for staff development is essential. ‘Stages of Concern’ towards virtual classroom can provide data for reflecting the progresses of teachers along the line of change and give hints for arranging different development programs at different stages and phases. Special attention and efforts should be provided to help teachers of intense awareness concern within a virtual classroom. Concerns-oriented staff development program is a feasible solution to advance our education with information technology.

References [1] A Search for New Heroes. (2004). Online schooling to avoid SARS. Retrieved April 22, 2005, from https://secure.cwheroes.org/briefingroom_2004/pdf_frame/index.asp?id=4923 [2] Bybee, R. (2004). The Concerns-Based Adoption Model (CBAM): A model for change in individuals. Washington: National Academy of Sciences. Retrieved September 30, 2004, from http://www.nas.edu/rise/backg4a.htm [3] Casey, H. B., Harris, J. L., & Rakes, G. C. (2004). Why change? Addressing teacher concerns toward technology. New Orleans: NECC 2004, the International Society for Technology in Education. Retrieved November 7, 2004, from http://center.uoregon.edu/ISTE/NECC2004/handout_files_live/KEY_301813/WhyChangeCasey.pdf [4] Cheng, Y.Y. (2001). Impact of web-based development on current teaching and learning modes. eClass, 12 Retrieved Jan. 17, 2005, from http://www.eclass.com.hk/big5/about_concept.php [Chinese]

[5] Centre for Information Technology in School and Teacher Educations (CITE) (2003). A comparative case study of “Innovative Pedagogical Practices Using Technology” Key Findings. Hong Kong: The Hong Kong SITES Research Team, The University of Hong Kong.

S.K. Lam and P.L.Y. Kwok / Concerns and Preferences of Teachers

219

[6] Clark, E. T. (1997). Designing & Implementing an Integrated Curriculum – A Student-Centred Approach. Brandon: Holistic Education Press. [7] Clark, R. C. (2002). The New ISD - Applying cognitive strategies to instructional design. Retrieved November 9, 2004, from http://www.ispi.org/publications/pitocs/piAugust2002.htm#Applying [8] Education and Manpower Bureau. (2005). Overall study on reviewing the progress and evaluating the Information Technology in Education (ITEd) Projects 1998/2003 – Final report. Hong Kong Special Administrative Region, China: The HKSAR Government Printer. [9] Göktürk, E. (2005). What is paradigm? Norway: University of Oslo. Retrieved April 19, 2005, from http://66.102.7.104/search?q=cache:C7N7P6Y9q0YJ:heim.ifi.uio.no/~erek/essays/paradigm.pdf+%22meani ng+of+paradigm%22&hl=en [10]Hall, G. E., George, A. A., & Rutherford, W. A. (1979). Measuring stages of concern about the innovation: A manual for use of the SoC questionnaire. Austin: The University of Texas at Austin, Research and Development Center for Teacher Education. [11]Hall, G. E., & Hord S. M. (2001). Implementing change: Patterns, principles, and potholes. Boston: Allyn and Bacon. [12]Hall, G.E., Wallace R.D, Jr., & Dosset, W.A. (1973). A development conceptualization of the adoption process with educational institutions. Austin: The University of Texas at Austin, Research and Development Centre for Teacher Education. [13]Hertfordshire Grid for Learning. (2005). What is eLearning? Retrieved April 22, 2005, from http://www.thegrid.org.uk/learning/elearning/what/ [14]Hiltz, S. R. (1994). The virtual classroom – Learning without limits via computer networks. New Jersey: Ablex Publishing Corporation. [15]HiSPEC Limited. (2003). eKnowledgement – a quick guide to e-learning. Retrieved April 22, 2005, from http://www.hispec-consulting.com/doc/SSM-173.htm [16]Johnson, B., & Christensen, L. (2000). Educational Research – Quantitative and qualitative approaches. U.S.A.: Allyn & Bacon. [17]Kaplan-Leiseron, E. (2005). ASTD’s source for e-learning. U.S.A.: American Society for Training & Development. Retrieved April 22, 2005, from http://www.learningcircuits.org/ASTD/Templates/LC/LC_OneBox.aspx?NRMODE=Published&NRORIGI NALURL=%2fglossary&NRNODEGUID=%7bA1A2C751-7E81-4620-A0A352F3A90148EB%7d&NRCACHEHINT=NoModifyGuest#E [18]Kirby, B. M., & LeBude, A. V. (1998). Induction Strategies that work: Keeping agricultural, health and biotechnology career development beginning teachers in the classroom. Journal of Vocational and Technical Education, 15 (1). Retrieved November 7, 2004, from http://scholar.lib.vt.edu/ejournals/JVTE/v15n1/JVTE5.html [19]Leafe, D. (2001). Intranets – Developing a learning community. In M. Leask (Ed.), Issues in teaching using ICT (pp.181-189). London: RoutledgeFalmer. [20]Lim, K. C. (2002). Using the appropriate learning approaches in an e-learning environment. E-learning Competency Centre. Retrieved November 9, 2004, from http://www.ecc.org.sg/cocoon/ecc/website/services/article/elearn_approach.article [21]Lynch, M. M. (2004). Learning Online – A guide to success in the virtual classroom. New York and London: RoutledgeFalmer. [22]Moss, C. M. (2001). Finding balance: The vices of our “versus”. Retrieved July 9, 2005, from http://www.firstmonday.org/issues/issue7_1/moss/ [23]Rakes, G. C., & Casey, H. B. (2002). An analysis of teacher concerns toward instructional technology. International Journal of Educational Technology, 3(1). Retrieved September 8, 2004, from http://www.ao.uiuc.edu/iject/v3n1/rakes/index.html [24]Richard, E., Rhys, G., & Levcho, Z. (1986). Information Technology and Education – The Changing School. Chichester: Ellis HorWood Limited [25]Roberts, G. (2004). Teaching using the Web: Conceptions and approaches from a phenomenographic perspective. In Goodyear, P., Banks, S., Hodgson, V., & McConnell, D. (ed.), Advances in research on networked learning (pp.221-244). Massachusetts: Kluwer Academic Publishers. [26]Tan, C.Y.G. (2003). Educational Web-publishing: Design, creation and management. Hong Kong: Pearson Education Asia Pte Ltd. [27]Zucker, A. A., & Kozma, R. et al. (2003). The virtual high school: Teaching generation V. New York: Teachers’ College Press.

220

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.


           


     


             
  


!        

    

   
   
 
       
   

 
 
 

     
  
  
   
      

 

  !     
   
"    
#
 
 


     $  

  
 
  
        
  "  
   
 

 
 
 # 
  
 
   
  !" # % 


  " & 
 

$   '
     
     " (
  
     
     
       "  
    
   
    "
  

 
  



 

)


   
 

 
 *
   
    
   
  
 

 

 
  
  #    
 


       
 
    

  " 




 +  ,-.

        " (/ "(0 
(1&
 


  ,2.( 
 
 


 

  

     
   !  
 

  


   !

  




 
 

"  
  

     
 
  

 / )%+*0"          
                     
    
  3( 




 3# "
 


#  
 "

    
 
 

 
    " 
       "
 
     
 
  

   4 
 

 

 
 


5 


!
  (  

    
   
  

6 
 

N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming

221

 
  "    
 
   
 7
   
  

   

   4 

 8  
  
 

   9

 


 
 


 
 3
 
 #  

 
" 
    
    
  
S#
      
  8  
       
       
    
   
  S
  
 
 



4    ;;
    
 
 
       
"
 #
 
  
  

 
  !
  
 "  
  !   
 
 
  

 !

 




  

  #    

       
 3
  !
 


      
 
    "   
   )%+*4

 
   
 %& '  ! +    
    
      
   

     
  
   

  
        
  
  

   ,<=>. 
    
   ,?.  

     
  ,@AB-C.    
   ,--. 
  

 ,-2.D 
      
 "(
,-<. "
 


 8
 
    
#
 
 3 
  
  

 
  "       
  

 
  
 
   #
  S
 
    
  
 

 
 
    
  9 
E
F
   
 
 
D  
H ,--.
  

  
&  9 
  !
  
     ,-=.  

 !
  
 



   
   


  ,->."  !
  
(1& ,2.


 
 


,-?.D 
 #
 




  6 •  
 

 
 
   ! 
  
,->. • D

#   

# 
    



 
,-. • " 

 


  
    
 
 
      

222

•

N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming


               
          
     
      8                                     

,-2.   
 
  
     
  
 
 
 !  
      
   "   







 

 ,-.

(&  )        ))   D 
   
 

 
    
     4 #
  
      
"
 
    
    
 
        H
 

 
       
   

  6 
    
  
  

   ' 

 
   

  
   


  
 
 
   D

   
 





     


  9

  3

  
 


 
  ! 
          4  #
       
        
   

 E#  F,-@.   

                

"
 E#  F 
  5


  5 
   
 


   

  9  
   
     

 
 
 
6 pred(<>, [V2|V3]):-subgoal(1). pred(<>, [V5|V6]):-pred(<>, V8).   
  #  

   
 
  !#$% & 


  
"  
 

E#  F 


  ! 
  
 
 



   
  ! 4 
  
 
 

   
    
     
  
  "
 
 
E
 ! ! 
  '+
'6  
 

"  !
 

  
  E
  ! ! 
'8
':


,-A. 
    
  S
 

  
 ,-@. 
 
 3
  

     
 E #  F
 



   
" ''+ '6



 ' '+ '6
  !#$%  

   

 
 
 
"
  : pred(V1, [V2|V3]).

N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming

223

pred(V4, [V5|V6]):-pred(V7, V8).

 
   
    !

   ! 
     
      

  

 !   
   
    

 "

' !
' $7
   ,-B.    
       3  

      
 


D
   
 


 

    
#  
 !7

 
  

  
  3

 

  
  

   
  3
 
   
    
 " 
 
 


 
 
 7 ,2C.
 

 3


   
 
 !  S

 3
 


 
     

 
 
   % 




   
 
/

0

 
  !  
  


 (


 
3    3
  

 
  ,-B2-.



  ,2C.9
 
  
 8       " !

  
     

  

D 
 

    
 

   
 
 
 D ! 

  / 30
    
 !   
    
#  4
 ! 
 


 
 !
  
,-A222<.S
,22.
   !   
     
      !  7      !     I9 
 


I 9
 
  

 

D  
  
6




 
  
,-." 
     
  
 
 
"
#
    
     
 
4 
 
 
  
 
6 E           !" "# $   !
$  %    &          &  '
 '  %    &     ( "E F
 


E F

 
   
 
"
    


  
  


  5   7  
# 
 

 
 !   !     
   
   
 "


 
 
 
  
 

 
  
 

     

  !

  "      
 
  

! 
     

 
  


 
     6 • %# =% !''6
'
'
 • %# =% !'G$ % 6'

 

 • JK =% !' JK
 '
 JK 
 

 

224 • • •

N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming

$K =% !NK$K% $
NK!
$K% $ P$  6  

 D  




 





% =% !$ =% !$ =% !6 5    5    
   


 )%+* 

 J*&#  "  J*& 

 
  

 
  
 
 E#  F   


'
'
  
E
K
 F 

  
 



"  
 
 <"  4
$  
  

  *   # 
     
   
  
"
 -    
 Q$ %   Q$ % Q$   Q$ % W$! %# =% !X'XX'X Q$ % W$! Q$ % Q$   Q$ % $ Q$ % $ JK  % %#!

Q$ % $ JK Q$ % $ Y  X&$! Z$! %KK J Z[ %\ QZ]!X Q$ % $ Y  Q$ % $ Q$ % K% J^Q$ % K% J Q$ %  Table 1: Costs for constraints

% 2 < = >

% 
  5  
    
   
  
 

  


'
  

 

 
  #    
  *   
       #    

) *  " &   +  &    &      D


  

   
      
    L  9 

 
      S   
 
     


 
 " 8   

 

"      
" 
  
    
  
 






 
 

 
 
 
 
     

" 
 


   
 
 

" 

   
 



 

 
 "
  










   




 

 L  
   D

 
  
  
  
 
 
  L  %  


N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming

225

  

  
 


 


D 
 
   



 

 
 


     
        

  
" 

  



    
 
  
  
  D
     


  


   
 
   
 
  














 
  
 
4#










%# =% !'G$ %   

 

  

 
"
  
  

 

JK =% !' JK"    
   





#
     
 
  
 



    
  
 
 
  #4
$ 
7   
   
     #   *& !  )%+* 

 
     
/ -0"  
 
  
  #    


     
 




               
          
  
  
  


  
        L      "     
      
    

 
"
     



  "  
     
  
   

 
' 
  
     
  
  
 



    
    
 
    "  
 
  
    # 
    

 

Figure 1: Architecture of INCOM

" 
 

J*&   

 
"    
   
   
# 
   '
# 
    

 
 

 4

 

    
  
  
#    "
        
#4
  L   
3          


226

N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming

  
 
   

  


  
 


 
   
# L   
  
  
 
 

    + 
   
        
"
     
      
 
 


 


   
#     

    2 " 
  

4
$ 
    

   4

#

     " 

 


 
 

   #    '  

  
  


 


4  

 
 
  
    

   
 
# "4
$ 
 
    #

  

  !  ,&
   - ". 9  

    


  )%+*
 3H+(",-C.D3H+3+(2,@.
*'MM,2=. 

    
"


    )%+*
   

  
  

 )%+* # 

    
 
 
  5 D 


  
 
   2CC=NC>
   7
(
 

  
  

  
M 
  

  
  
 7      
)
    
  
  
    
 
 
 % 
2?-


BB  " 
 ?AAO

! 
 
  
,2>. +#  
 
 






  
  
 
 

9 

 
 
  

  
  # 

3"
  
 
 
   
  

   



       



    #  

     
 
7



  

#  
  
 
   
   



 

 #  S #  
  

 # 
  
 
 
    
 !."  " 

 '&
 %  7

 DCC<>

N.-T. Le and W. Menzel / Constraint-Based Error Diagnosis in Logic Programming

227

/ ,-.(+ % 

   P'$$ *%

 ,& - & + . /  &  "& 0& / &"&   -?@-ABS -BB= ,2.D*  
% 

6
 &*  
PQ

    1     &  2 *  * & 2# ,B<-B=CS
2CC- ,<.DD

P&
 &


 
* ->@>-22-BAC ,=.3Q
 * 3 2   *   4     * 3M R& %
R 5-BB= ,>.(J
T(%"

  

   

    222 .   ,   2  2B/=062?<-BB- ,A.&9 D

   6D
    
  * /=>0--<-@--BBC ,B.$9'  
      , /2C0-B>2B-@>-BA> ,-2.D*  
(+ '

  
 







  "  6   *       2& -C2?-BBB ,-<.RR PD  97
**
         6  *       2& A.D*  RR S*
 D

 

    
 

    7      - & <-<<222CC< ,-?.R9
 %"
 % 

  
 
       ,2-CA<-CA@-BB@ ,-@.3S
      * 8   2CC- ,-A.DS3S
 36D 
  
  ! 3S
 SMS
73
9    & &  &  #    +       &-?@-@@D#3  %(
%  -BBB ,-B."($7
  &
  


      , :-@<-B2-BB- ,2C.P7 $ 
 




  
   3   6 " "  ,&  ;>C>><=2CC= ,2-.*S  U3)U
&
 
   

*- , 9*4 4  << /206=-=@-BBA ,22.3S
DS "M*'  
%& 73
MH S( *(  3
  !   , 2C----<<-BB- ,2<.&( *R  
D  ! < =      9  -BB< ,2=.H%(  *)
*( 
* 
M 
M  ) 3
' -  9 @-AC2CCC ,2>.)"&'

  
% 

'M
  & 3
 3-<   %   % '
  2CC>

228

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Model Building as a Platform to Support Thinking in Problem Solving Chwee Beng LEE1, Ioan Gelu IONAS2, Matthew SCHMIDT2 1,2 University of Missouri-Columbia, USA [email protected] Abstract: Problem solving not only plays a central role in education [19] but is also the most important cognitive process in our everyday life. The authors of this paper argue that problem solving must be embedded in authentic learning contexts and supported by activities that seek to enable problem solvers to externalize their mental model in the most meaningful and engaging manner that will eventually lead them to the construction of a more sophisticated conceptual framework. One such way is through system dynamics modeling activities. To provide a sound argument for the use of system dynamics models in learning, we present three models that support thinking in problem solving. Keywords: Problem solving, system dynamics.

Introduction Problem solving not only plays a central role in education [19] but is also the most important cognitive process in our everyday life. In recent years, there is a growing interest in using problem solving strategies as a tool to improve students’ conceptual understanding. In traditionally taught courses teachers often assume that solving problems will definitely help develop students’ understanding of concepts and principles [10] and at the same time help transfer their problem solving skills to other learning contexts. Yet, research findings suggest that engaging in problem solving activities alone does not help to develop a deep understanding of concepts and principles [3] [12], let alone the transfer of knowledge [1]. There are several explanations for the failure of problem solving. However, it is not the intent of this paper to provide an extensive discussion on this issue. We argue that problem solving must be embedded in authentic learning contexts and supported by activities that seek to enable problem solvers to externalize their mental model in the most meaningful and engaging manner which will eventually lead them to the construction of more sophisticated conceptual frameworks. One such way is through system dynamics modeling activities. To provide a sound argument for the use of system dynamics models in learning, we present three examples where system models are being employed to support thinking in problem solving. All models are created using the system modeling software “STELLA.” 1. System Dynamics Modeling This paper argues that models offer students a platform to generate a higher level of cognition which is central to problem solving. According to Lesh and Doerr [9] models are “conceptual systems” consisting of elements, relations, operations, and rules governing interactions that are expressed using external notation systems. There are many different types of models, such as models that seek to represent the phenomena around us (physical

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

229

models) and models that are not visible but known to the human mind (conceptual models). Hong et al. [6] describe some of these, including abstract models (mathematical models), representational models (the central function of models used in science), and hypotheses and theoretical models (models constructed with theoretical principles). The models we are interested in are system dynamics models. A system model is a conceptual, conjectural representation of the dynamic relations among the factors of a system, resulting in a simulation that imitates its conditions and actions [7]. By using sets of building block icons, the learners externalize their understanding of the system and construct and test a model that is coherent with and explanatory for their conceptual understanding. Moreover, by creating a system model, learners are able to think about how a system changes over time, which is an implicit characteristic of any real world problem. In addition to building system dynamics models, learners can use models that have already been built in order to learn the nature of a specific system, test their hypotheses regarding the system and construct more robust mental models of such a system. By studying an existing but unfamiliar model, learners can gain insights into the inner workings of the system. Learners can then test their understanding of the model by “playing” with it. If the model does not agree with their preconceived idea(s), learners enter a state of cognitive conflict. To remedy the conflict, the learner can question the soundness of the model or of his or her conceptual understanding of the system. In the first case, the model builder would need to study the system more extensively and revise the model, thus conducting deeper learning, and in the second case the learner would need to modify his or her conceptual understanding, thus facilitating conceptual change and increasing domain-specific knowledge. 2. Cognitive Gains of System Dynamics Modeling We believe that during the problem solving process learners construct their initial mental model based on their available domain-specific knowledge, integrating new information into their model in the hope of solving the problems they encounter. Representing the problem by constructing models allows the learners to predict events and to generate explanations of phenomena. Perkins and Simmons [15] suggest that prediction and testing encourage “internal discourse” (perturbations) in learners thus creating cognitive conflict. System dynamics modeling offers such a platform for inducing and supporting cognitive conflict. When students discover the inconsistencies in their conceptual structures while modeling they are more likely to revise and refine their conceptual framework to resolve the perturbations or anomalies. However, it must be noted that models are more commonly used as pre-built engines in software applications [8] in which learners are only allowed to access and manipulate some of the variables that determine change. We firmly argue that to facilitate deeper thinking in learning and to induce conceptual change—a necessary catalyst in problem solving—students must build their own models. To be more specific, when the problem is presented, learners seek to represent it through modeling. By building a model, the learners have to articulate their reasoning through hypothesis testing, inferring, decision making and other cognitive skills. It is through this process of iteratively externalizing their mental representations that they seek ways to achieve a coherent conceptual model. System modeling is worth included in our learning context for several compelling reasons. First, real world systems are dynamic and the relations between variables are non linear. This assertion is in disagreement with some of the information and concepts presented by school textbooks which sometimes oversimplify the complexity of systems. Nonlinear models mean the rise of a more subtle and, consequently a more realistic vision of the world [5]. That is, non linear models make nonlinear behavior logical, reasonable,

230

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

and concrete [5]. Second, many science topics deal with systems, such as water cycle, ecology, and biology, and students’ understanding of such systems may be enhanced by creating, manipulating, and exploring computer models of those systems [15]. Moreover, by creating dynamic models, learners may be able to better understand the interrelationships between concepts in a larger context. System modeling is a highly engaging activity and it requires students to be committed to the modeling process as they will necessarily need to incorporate several strategies such as analyzing, synthesizing, evaluating, reasoning [16] and applying domain-specific knowledge to create a model that supports their conceptual understanding. Also, learners not only are able to create a model, they are also able to test and run it using various initial settings. Through this process of constantly revising and refining the model, the students are engaged in reflective thinking, and generate sound justifications that will eventually lead them to higher levels of cognition. 3. Gene Flow Model The initial argument in this paper is to have learners build their own system model. However, we recognize the fact that scaffolding students in such a modeling process is also crucial and beneficial to the development of a coherent conceptual framework, especially in highly complex problem, and when learners have no prior knowledge of model building. In such a modeling process, learners will necessarily need some form of support in mastering both software functions and the modeling process [21]. In this section, a pre-built gene flow model was presented to undergraduates enrolled in a course in Biological Anthropology at the University of Missouri - Columbia. The cognitive goal of presenting learners with the pre-built gene flow model was to provide them an interactive, online simulation to enhance their understanding of gene flow, and promote conceptual change. Through manipulating the variables of the pre-built model, learners were engaged in a hypothesis testing process, and this necessarily perturbed their existing conceptual framework and at the same time, learners got acquainted with the mechanism behind the modeling process. The ultimate goal was to gradually remove such scaffolding and have learners build their own models. To build the gene flow model, a team of developers met bi-weekly for one semester with a subject matter expert (SME), a professor of biological anthropology at the university. The SME was interviewed on multiple occasions and provided feedback for the duration of the design and development process. The team learned that the factor that causes gene flow is interbreeding between populations, which is influenced by the populations' migration, social rules and geographical distance. The final model (figure 1: capturing one section of the actual model) accurately demonstrates the effect of migration and social rules on gene flow based on learner-defined populations. This model not only enables learners to grasp a deeper understanding of the effects of possible variables have on gene flow; it also allows learners to predict the different outcomes of the interaction of variables and hypothesis testing. This process is crucial in problem solving as it increases learners ability make sound justification which is a predictor of solving ill-structured problems [6]. Migration and social rules are demonstrated in the gene flow model on the interface level with two sliders: "Level of similarity in social rules" and "Migration". Manipulation of these sliders affects the amount of interbreeding between populations. Additionally, there is a section labeled "Control panel for populations 1 and 2" (figure 2) which accepts learner-defined populations. Before the learner enters any variables into the interface, s/he first formulates a hypothesis of the outcome in terms of changes in the populations' gene pools over time. The learner then uses the gene flow simulation to test

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

231

these hypotheses. Based on feedback provided by the simulation the learner is able to recognize misconceptions by identifying inconsistencies between his or her mental model and the simulation's output. The learner can then reformulate and test hypotheses, iteratively enhancing his or her understanding of gene flow. Figure 3 shows the rates of homozygous recessive alleles (aa) for populations 1 and 2.

Figure 1: Part of the Gene Flow Model

Figure 2: control panel for populations 1 and 2

Figure 3: Rates of homozygous recessive alleles (aa) for populations 1 and 2.

4. Understanding a Social Phenomena Since uncertainty and change is more pervasive in social phenomena [22], dynamic system modeling offers a lens to better understand the complexity and to capture the dynamism of such phenomena. In this model, we seek to understand a social phenomenon - the increasing number of single graduate women in Singapore - through the system dynamics approach. In Singapore, the rate of single hood is fast increasing. In the year 2000, there were 26.7 % graduate women (age 40-44) who remained single as compared to 24% in the year 1990 [18]. According to various studies, if these ratios remain unchecked, the number of single hood will continue to surge and the aging population will increase exponentially. This will then become a challenging social problem. In this model (figure 4: capturing a

232

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

section of the actual model), we show the relationships between various variables that will lead to the increase in single hood for graduate women. To help us understand the increasing trend of single graduate women, and to predict the future trend of this phenomenon, we constructed a model that captures the interaction of possible variables that will presumably contribute to such increasing trend. The three variables included in this model are: men's choices (marrying foreigners and non-graduates), women's high expectation on marriage and women's priority (marriage is not the first priority on marriage). Note that since time is a crucial factor in dynamic models, we seek to model the phenomenon over a period of 12 years to capture the dynamic interaction between the variables on single hood.

Figure4: Model of a Social Phenomenon

This model leads us to conclude that when women are more educated, they tend to consider marriage as being of lower priority and at the same time, their expectation on marriage also increases. Therefore this group of educated women will spend more time pursuing other attainments (career, education, social work etc). As they place more priority on other Figure 5: Women Pursing Other Attainments, Women attainments, the likelihood of remaining single is higher. Figure 5 shows that the with Less Priority on Marriage number of women who have less priority on marriage and women pursuing other attainments increases over a period of 12 years. Such increases will in turn contribute to a rise in the number of single women. A real world phenomenon or problem is most of the time ill-defined and illstructured. Hence, to understand it, a systemic approach is needed, taking into the account

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

233

of the interaction between the contributing variables over time. Perhaps the most significant trend we presented in this model is the continuously increasing trend of single hood for well-educated women, depicted by the feedback loop (see figure 4). Such feedback loop represents a reality and that is as more well educated women remain single, they tend to devote their time and effort on further education and career. 5. Modeling Cognitive Conflict Last but not least a Stella model of Cognitive Conflict is presented to show that modeling can be a valid approach not only for physical or social phenomena, but can be used successfully to help the understanding of more elusive concepts such as learning theories. The model shown in figure 6 is aimed at understanding the impact of cognitive conflict in the learning process in science education where encountering information that contradicts the learner's current theories about the physical world is common occurrence [2], [11]. Gap needed to declare conflict

percent non conflicting

percent accepted

Existing concept not found

percent new

accepting

acknw concepts

interesting concepts

percent rejected

rejecting

Identifying existing concept

percent conflicting

missconceptions versus total concepts

Gap needed to declare conflict

conflicting concepts conflict found

percent new correct concepts

concept exposure

percent new conc incorrect

Individual acceptance level

Accepting as new concept

not rec concepts

percent accepting as new concept

Not reconciled

Rejecting conflictual concept

percent rejecting new concept

percent not reconciled

Willingness to reconcile reconciliation

miscon ceptions

deleting conflictual missconceptions

adding to misconceptions

percent reconciled incorrectly

percent reconciled missconceptions versus total concepts

deleting conflictual correct concepts

adding to correct concepts

correct concepts

percent reconciled correctly

Figure 6: Modeling Cognitive Conflict

Many researchers advocate the use of cognitive conflict to engage deeper learning. However, numerous studies show that in many situations, learners resist giving up their pre-instructional beliefs despite the conflicting evidence they face [2]. Vosniadou [20] argues that in science learning children assimilate information that might be inconsistent with their prior knowledge, forming synthetic meanings. Cognitive conflict alone is not sufficient to induce conceptual change; it has to be grounded in a more challenging context to create learning awareness. As suggested by Niaz [13], cognitive conflicts when used in teaching experiments must be based on problem-solving strategies that learners find relatively convincing. In addition, an existing body of research argues that conceptual change is a gradual process that evolves in time [20] as the learners accumulate knowledge. Based on existing research, the model in figure 7 is an attempt to represent how cognitive conflict impacts the learning process. The model concentrates on the processes that take place over time while the learner is exposed to and accumulates more

234

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

concepts, both correct and misconceptions. The intent of this model is to enable the understanding of the pivotal role cognitive conflict plays in learning. The model was developed starting from the hypothesis that the learner, as any individual, is continuously exposed to a variety of concepts (modeled as a random input variable). After the initial exposure, the concept is compared to the learner's existing knowledge (concepts) and either recognized as valid, or considered as conflicting or new. The newly acquired concepts can be correct conceptions or misconceptions, and will accumulate over time. A reconciliation process takes place for the conflicting concepts that will be either accepted or rejected. A control panel (figure 7) allows the user to set up a number of five initial variables to define learner’s characteristics. The “correct concepts” and “misconceptions” dials are used to set up the learner’s prior knowledge (or Figure 7: The Control Panel education level) while the three sliders define three relevant individual characteristics of the learner. Figure 8 shows a typical output of a simulation run. It presents in graphical format the behavior over time of a number of variables or ratios considered important for the understanding of the phenomenon. Besides building such a model, by running the simulation with 1: missconceptions É

1: 2: 3: 4: 5:

2: percent conflicting

3: percent reconciled

4: percent new

5: percent non confliÉ

0.45 0.11 0.84 0.575 0.5

3

2

3

4

1: 2: 3: 4: 5:

0.35 0.08 0.805 0.54 0.4

5

5

3

1

2

5

5

4

1

3

2

4

1

1: 2: 3: 4: 5:

0.25 0.05 0.77 0.505 0.3

1

0.00

Page 1

250.00

500.00 Time

750.00

2

4

1000.00

3:06 PM Thu, May 13, 2004

Model ratios showing the evolution in time

various initial settings, the learner is be able to make comparisons between various types of individuals at different levels of understanding and with different levels of education.

Figure 8: Behaviors of Variables over Time

Conclusions While existing research shows that modeling is a powerful instructional tool, we believe that its power is underestimated and therefore it is underused. In addition, while many people can easily understand modeling the physical world (see example 1: gene flow), they might have difficulties understanding that modeling can be used successfully to support teaching and understanding of more elusive concepts such as those embedded in social (see example 2) and cognitive (see example 3: cognitive conflict) phenomena. To support our assertion, we have shown three examples in which system dynamics models are used to support thinking in problem solving. The first case, a model of gene flow supports the argument that hypothesis testing, prediction, and providing sound justification(s) are crucial in problem solving, especially when dealing with highly complex contexts. In this model we also argued that pre-built models provide necessary scaffolding for such complex problems and can help prepare the learners to build their own conceptual models of complex systems. A second model shows how system dynamics modeling can be used to investigate and understand ill-structured and ill-defined social phenomena. The third and last model goes beyond real world phenomena into the realm of theories of cognition and learning as it presents a model of how conceptual change takes place. To conclude, it was our intent to support the argument that modeling can be a valuable tool in solving real world as well as conceptual problems using a practical,

C.B. Lee et al. / Model Building as a Platform to Support Thinking in Problem Solving

235

evidence based approach by presenting three models in three different realms of science and learning which in many situations require a non-linear approach to reach a thorough understanding. It is our belief and hope that the power of systems modeling will support its increased use in both education and research. References [1] Chipman, S. E., Segal, J. W., Glaser, R. (1985). Thinking and learning skills, Vol.2. hillsdale, NJ: Lawrence Erlbaum. [2] Chinn, C.A., & Brewer, W.F. (1993). The role of Anomalous data in knowledge acquisition: A theoretical framework and implications for science education. Review of Educational Research, 63, 149. [3] Clement, J. (1982). Students preconceptions in introductory mechanics. American Journal of Physics, 50, 66-71. [4] Giere, R. N. (1999). Using models to represent reality. In L. Magnani, N.J. Nersessian, &, P. Thagard (Eds.), Model-based reasoning in scientific discovery. New York: Kluwer Academic Press/Plenum Publishers. [5] Goerner, S. (1995). Chaos, evolution, and deep ecology. In Robertson, R., & Combs, A. (Eds.), Chaos theory in psychology and the life sciences. (pp. 17-38). Mahwah, NJ: Lawrence Erlbaum Associates. [6] Hong, N.S., Jonassen, D.H., & McGee, S. (2003). Predictors of well-structured and ill-structured problem solving in an astronomy simulation. Journal of Research in Science Teaching, 40, 6-33. [7] Jonassen, H. D. (2003). Using cognitive tools to represent problems. Journal of Research on Technology in Education, 356-381 [8] Jonassen, D. H., Strobel, J., & Gottdenker, J. (in press). Model building for conceptual change. Interactive Learning Environments. [9] Lesh, R., & Doerr, H. M. (2003). Foundations of a models and modeling perspective on mathematics teaching, learning, and problem solving. In Lesh, R., & Doerr, H. M. (Eds.). Beyond constructivism: Models and modeling perspectives on mathematics problem solving, learning, and teaching (pp. 3-34). Mahwah, NJ: Lawrence Erlbaum Associates. [10] Leonard, W. J., Dufresne, R. J., & Mestre, J. P. (1996). Using qualitative problem-solving strategies to highlight the role of conceptual knowledge in solving problems. American Journal of Physics, 64,14951503. [11] Limon, M., & Carretero, M. (1997). Conceptual change and anomalous data: A case study in the domain of natural sciences. European Journal of Psychology in Education, 12, 213-230. [12] Maloney, D. P. (1994). Research on problem solving: Physics. In: D. L. Gabel, (Eds.), Handbook of Research on Science Teaching and Learning, (pp.327-354), New York: McMillan. [13] Niaz, M. (1995). Cognitive conflict as a teaching strategy in solving chemistry problems: A dialecticconstructivist perspective. Journal of Research in Science Teaching, 32, 959-970. [14] Nickerson, R. S., Perkins, D. N., & Smith, E. E. (1985). The teaching of thinking. Hillsdale, NJ: Lawrence Erlbaum. [15] Perkins, D. N., & Simmons, R. (1988). Patterns of misunderstanding: an integrative model for science, math, and programming. Review of Educational Research, 58, 303-326. [16] Roberts, N., Anderson, D. F., Deal, R. M., Garet, M.S, & Shafer, W. A. (1983). Introduction to computer simulation: A system dynamic modeling approach. Productivity Press, Portland, OR. [17] Stratford, S. J., Krajcik, J., & Soloway, E. (1998). Secondary students’ dynamic modeling processes: Analyzing, reasoning about, synthesizing, and testing models of science ecosystems. Journal of Science Education and Teaching, 7, 215-234. [18] Singapore Statistics. Retrieved from http://www.singstat.gov.sg/papers/people.html#census2000 [19] Taconis, R., Ferguson-Hessler, M. G. M., & Broekkamp, H. (2001). Teaching science problem solving: An overview of experimental work. Journal of Research in Science Teaching, 38, 442-468. [20] Vosniadou, S. (2002). On the nature of naïve physics. In M. Limon & L. Mason (Eds.), Reconsidering conceptual change: Issues in theory and practice. (pp. 61-276). Amsterdam: Kluwer Academic Publishers. [21] Fretz, E. B., Wu, H. K., Zhang, B. H., Krajcik, J. S., & Soloway, E. (2002). A further investigation of scaffolding design and use in a dynamic modeling tool. Paper presented at the American Educational Research Association. New Orleans, LA. [22] Yilmaz, L, & Oren, T. I. (2004). Dynamic model updating in simulation with multimodels: A taxonomy and a generic agent-based architecture. Proceedings of Summer Computer Simulation Conference, July 25-29, 2004. San Jose, CA, pp. 15-19.

236

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

A Study of Curriculum Implementation and School Culture in the Information Age Lin, Shinn-Rong , Teng, Hsiao-Ting , Lin, Huei-Chuan , Chiu, Chuang-Chieh Graduate Institute of Learning and Instruction National Central University, Taiwan, ROC. [email protected]

Abstract. This study explores curriculum implementation and school culture in the information age, and explores the implied meaning of organizational culture that is proposed by Schein (1992,2004), namely, artifacts, espoused beliefs and values, and underlying assumptions. The study sample composed of two schools. One of the sampled schools is famous for its performance in integrating information technology into teaching and for having an abundance of hardware, software, and so on. Meanwhile, the other school lacked sufficient IT hardware. Participant observations, in-depth interviews, document analysis, and a focus group interview were used to obtain data. The artifacts included in preliminary results were computers and IT equipment, instructional websites, and computer arrangement in terms of teacher’s instructional perspective. The level of espoused beliefs and values includes collaboration and sharing experience, and the transformation of instructional paradigm (from teacher-center instruction to student-center instruction). Finally, the level of underlying assumptions includes space, time and enhancing the instructional effectiveness of IT. Keywords. School culture, curriculum implementation, information age, professional development

Introduction The rapid development of information technology has markedly impacted on education, society, and culture. The key changes between the industrial age and the information age in the area of education, for example, are a move from bureaucratic organization to team organization, from autocratic leadership to shared leadership, and from one-way communication to networking (Reigeluth,1994:6). The role of teachers has also changed fundamentally, for example, from being an instructor to a learning facilitator, and from “teacher-centered” to “student–centered’ (Cuban, 2001; Reigeluth, 1994; Roblyer, 2003 Sandholtz, Rinstaff, & Dwyer, 1997). In other words, information technology can improve student learning, especially achieving a shift from passive learning to active learning. Thus, school culture must be changed in response to the development of information technology, for example evolving from teacher isolation to teacher cooperation. Furthermore, ways of curriculum implementation in the information age would be different to those in the industrial age. The “Treasure” of the Educational Resources Information Center (ERIC) defines “school culture” as: “Patterns of meaning or activity (norms, values, beliefs, relationships, rituals,

S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age 237

traditions, myths, and so on) that are shared to various degrees by members of a school community.” Deal & Peterson (1999) classified the elements of school culture as follows: schools as tribes, vision and values, ritual and ceremony, history and stories, and architecture and artefacts. Schein (1992, p.12) defined group culture as : ” a pattern of shared basic assumptions that the group learned while solving problems of external adaptation and internal integration, that has worked sufficiently well to be considered valid and, therefore, to be taught to new members as the correct way to perceive, think, and feel in relation to those problems.” Schein further classified culture into three levels: artifacts, espoused values, and basic underlying assumptions. Artifacts include visible organizational structures and processes, and are difficult to decipher. Espoused values include strategies, goals, and philosophies, and thus include espoused justifications for shared values and beliefs. Finally, basic underlying assumptions include unconscious, taken-for-granted beliefs, perceptions, thoughts, and feelings, and are the ultimate source of values and action. The existing organizational culture can evolve in response to changes in the external environment. For example, rapid advances in information technology will lead to the emergence of new information related cultures. Schein (1992, p.282) noted that IT assumptions regarding people and learning include: (1) technology leads and people adapt; (2) all people can and should learn whatever is required to use technology; (3) people can and should learn the language of IT; (4) people can and should adapt to the IT “interface” including keyboards, command languages, windows, mice, and so on; (5) people already know how to communicate and manage; therefore, IT merely needs to enhance these processes; (6) people will use IT to enhance task performance and efficiency. In fact, Schein assumed that people use excessively rational assumptions when applying new technology due to their failure to recognize “how difficult, anxiety, provoking, painful, and time consuming new learning can be” (p.281). Besides the problems that generated by implementing curriculum using IT, another problem is the implications of assumptions regarding information technology for school cultures. Teachers perceived IT as a tool for improving teaching and learning. Through application and leading by teachers, the above assumptions, particularly assumption of (6), may become the expectations used by schools in constructing new school cultures in the information age. Therefore, this study examines curriculum implementation and school culture in the information age, and explores the implied meaning in relation to the three levels of organizational culture proposed by Schein (1992, 2004), namely, artifacts, espoused beliefs and values, and underlying assumptions.

1. Methodology This work describes a case of the exploration of the curriculum implementation and school culture in the information age. Observation, individual interviews, focus group interviews, and document analysis were used to gather the study data. Video camera, and digital recorder were used to record the observation and interviews, and all the contents were typed into transcripts for data analysis. This study examined two schools, one located in Taipei city (called “school A”), and the other located in Taoyuan County (called “school B”). Both of the schools mentioned above are located in the North of Taiwan. School A is an elementary school, and was well known as a leading school for the provision of information technology in Taipei city. This school had abundant information technology hardware and software, with every classroom having a

238 S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age

projector, computer, and Internet access. However, school B had only limited hardware and software provided in the classrooms (only three projectors in the whole school). The study was conducted from June 2004 to January 2005.

2. Results and discussion 2.1 Overview of curriculum implementation in school A. z Frequency of integrating information technology into teaching. Teachers who on the subject-matter apply IT in their curriculum implementation exceed teachers in charge of grade, called grade-teachers, for short. The main reason for more subject-matter teachers applying IT in teaching is because of time considerations. That is, since teachers can use the same digital instructional material in many classes, it is more worth while for teachers to invest their time in developing a single set of instructional material using IT and to apply that material in numerous classes. Meanwhile, many grade-teachers do not apply IT in their teaching in terms of heavy loads on teaching and classroom management. The main reason for few grade-teachers applying IT in their curriculum is because of the developed digital instructional material being used only every two years. Consequently, the grade-teachers in school A have little willingness to develop digital material for curriculum implementation. However, all the teachers in school A would apply IT in their teaching if there were some activities for outsiders to visit the school’s teaching practice on using IT. z Cooperative culture among teachers. Subject-matter teachers cooperated with teachers with high computer literacy to develop digital instructional materials. Some teachers teach in coordination with the arts and humanities. z Classroom web page. Every class has a web page with teacher and student portfolios. Moreover, some web pages have area where teachers and students can interact. z Ability to respond technological change. Owing to the study school being a key school in Taipei city, teachers are used to performing certain tasks required by the municipal government. The teachers also tried to adapt some various new products to improve teaching practice. For example, teachers must use a film studio to develop digital materials for teaching. z Climate for conducting innovation. The teachers are used to conducting innovations in their teaching practice. Additionally, the teachers can obtain some substantial support (such as equipment or space) from the principal if requested.

2.2 Overview of the curriculum implementation in school B. z Less than 10 percents of teachers use IT in their instruction. Because of school B lacking adequate equipment for using IT to implement the curriculum, less than 10 percent of teachers applies IT in their teaching. The school is quite academically oriented. Most of the teachers at school B focus on student academic achievement, and conduct teaching and learning using traditional methods. z Individual instruction web pages. Less than 10 percent of teachers have individual instruction web pages. Although the local government set a goal for every teacher to

S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age 239

construct their own instruction web pages, few teachers allocated time to their own instruction web page. z Classroom web page. Two thirds of classes have their own classroom web page; however, most of these the web pages are very basic, and few provide for interaction between teacher and students. z Climate for conducting innovation. The teachers are not used to implementing innovative methods in their teaching. It is difficult to get support from the principal. Teachers prefer to implement innovation privately, and only implement innovations publicly when they are certain of success.

2.3 Artifacts of curriculum implementation in the Information Age 2.3.1 Arrangement of computers in the classroom, and laboratory Schein (1992, 2004) pointed out that physical environment is one of the artifacts in an organizational culture. In the information age, the artifacts of curriculum implementation using IT generally include a computer, a projector and Internet access. Teachers arrange the computer differently depending on their teaching beliefs. Some teachers prefer to implement the curriculum in a “computer laboratory”, namely a room filled with computers. Meanwhile, some teachers prefer simply put some computers into a conventional classroom; an arrangement that enables teachers to remain sufficient space for lectures, group discussions, presentation and so on (see Fig. 1). The layout of the computers indicates teacher beliefs regarding curriculum implementation. That is the layout indicate whether they believe in a student-centered approach, a mixed approach, or a teacher-centered approach. For example, a teacher using a mixed approach in school B expressed her ideas about an ideal classroom for implementing the curriculum as follows: ….I would like have 35 computers, where every student can use one computer for every student, in a big room with a lecture area (large enough for 40 students), and then I would be able to be more flexible in my teaching. The most important thing is that the desks and chairs should not be fixed in place. If the desks and chairs are fixed, then it would be very inconvenient for me to conduct group interactive learning activities.

2.3.2 Web page Teachers in both schools used an instructional web page to extend student learning activities. Web pages are one of the artifacts of school culture for the curriculum implementation in the information age. However, web pages with rich content and interaction functions are related to teacher computer literacy and technical support provided by the school. Furthermore, it is becoming popular to display student portfolios on web pages. Thus, in the curriculum implementation in information age, web pages become an artifact of school culture.

240 S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age

White-Board & Screen

Computer Area Presentation & Activity Area

White-Board & Screen

)LJXUH&ODVVURRPDUUDQJHPHQWLQYROYLQJFRPSXWHUVSUHVHQWDWLRQDQGDFWLYLW\DUHD

2.4 Espoused belief and value of curriculum implementation in the information Age 2.4.1 Sharing and cooperation To increase the effectiveness of student learning when using IT, at both schools, teachers with teaching experiences cooperated with teachers with high IT proficiency. Cooperation and experience sharing among teachers is one of the espoused values for curriculum implementation in the information age. For example, a teacher in school A is familiar with Chinese calligraphy, but knows little about software and hardware, in the curriculum development process, this teacher cooperates with colleagues, friends, and learns some knowledge and skills by on-line. Furthermore, in school B, a teacher with high proficiency in computers invites a teacher specialized in Chinese. The motivation for cooperation of the teachers in school B is based on the belief that teaching with IT would make sense to students. The Chinese teacher said: I think our teaching that coping with IT would make sense to the students, especially to the learning process [with IT and independent learning]. Extended learning via curriculum implementation using IT making sense for students. Thus I coordinate with the computer teacher’s curriculum design and doing whatever she wants me to do.

2.4.2 Learner-centered approach and Teacher-centered approach The advancement of information technology, including the capacity of CPU and the delivery speed of Internet eliminate the obstacle of individualized instruction through computer and Internet, and increase the concern of some researchers regarding instructional approaches,

S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age 241

namely, leaner-centered (Cuban, 2001; Reigeluth, 1994; Roblyer, 2003; Sandholtz, Rinstaff, & Dwyer, 1997). That is, students are expected to be more active in learning activities, for example, by searching for information on individual, group discussion, or presentation basis. In this case, teachers in two schools applied Internet as a tool to increase teacher and student interaction, implying the espoused belief of teachers regarding teaching in the information age.

2.5 Basic underlying assumptions regarding curriculum implementation in the Information Age 2.5.1 Extension of Time and Space Traditionally, the work area of most teachers is limited in the boundary of school (especially within the classroom), and the daily work schedule of schools traditionally was based on the regulations of school administrators, generally lasting form 7:30 a.m. to 4:30 p.m. In fact, the work landscapes of teachers extended to elsewhere else, such as outside the school, home, library etc. In the information age, Internet access and the universality of laptops means that time for teachers to develop instructional material and to interaction with students is no longer limited to formal working hours. That is, curriculum implementation in the information age requires extending time and space from formal working hour to private hours, and from the school setting to non-school settings. Teachers invest considerable time on curriculum implementation with leading IT teachers feeling hesitant to implement their curriculum using IT again. Evidence of teachers’ overloading with integrating IT into the curriculum can be demonstrated through the replacement rate of personnel (generally not more than two years for the position of head of information technology section). Many teachers are not continuing to use IT on their curriculum implementation. Alternatively, some teachers may still be using IT in their teaching but rarely develop new instructional material. For example, over 5000 school teachers, educators, and administrators visited school A; however, most teachers are unwilling to the willingness to develop digital material for curriculum implementation unless visitors exist who can observe teacher teaching. That is teachers in school A can implement their curriculum using IT, but rarely use IT in the curriculum. In another example, teachers in school B expressed their concern about investing their time in using IT for curriculum implementation. Initially, they were motivated to join curriculum implementation with IT by financial aid and implementing what they learned in graduate school. The teachers invested considerable time to organize and develop the curriculum to cope with using IT, and experienced some difficulties, including lack of appropriate equipment, and lack of experience in conducting student-center learning activities. When the researcher asked them “were you jump into the project if you figure out everything?” They laughed and said “never”. To summarize, the basic underlying assumptions made in school cultures for implementing curriculum in the information age related to space and time can be described as follows: z Space for teacher teaching and student learning is not limited to the classroom and school. z Time for teacher to work and interact with students is not limited to the regular working schedule, and can include the private time of teachers.

242 S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age

2.5.2 Information technology can improve effectiveness of instruction. No consensus exists regarding the effectiveness of instruction. Cuban (1993) noted that researchers with different perspectives would have different preferences for certain forms of instruction over others. For example, cognitive psychologists prefer teaching with thoughtful, engaging students with active learning, exploring ideas and so on. Moreover, some scholars prefer teaching with high structure, helping students cultivate academic skills. Generally, the teachers in this study, who used IT to implement their curriculum, believe that using IT in their teaching is good for student learning. Furthermore the benefits to students are not limited to academic achievement, and may include student attitudes toward learning. For example, a teacher in school B expressed her perceptions of the benefit of applying IT in teaching as: “this form of teaching would make sense to students”. By the project, a study of outstanding women in Taiwan in the 20 century, student learned higher-order cognitive knowledge, and cultivated attitudes towards active learning. That is, teachers take it for granted that IT can improve the effectiveness of instruction. Based on the assumption it becomes an inert motivation for teachers to invest their time and energy in applying IT in their curriculum implementation.

3. Conclusions 3.1 Artifacts The artifacts of school culture used by teachers to implement curriculum in the information age include: related hardware, software, websites and arrangement of computers in the classroom. Related hardware and software are the basic artifacts used by teachers to develop and implement their curriculum. Web pages are another artifact used by teachers to share ideas and interact with students. Although web pages are not a physical object (like a computer, or a laptop), they offer a new means for teachers to construct a learning community for sharing their knowledge and experience in curriculum implementation.

3.2 Espoused values The espoused values of school culture used by teachers to implement curriculum in the information age include: cooperation and sharing, instructional approach shift (from teacher centered to student-centered). Teachers consider it extremely helpful to generate creative ideas by sharing experience with each other. Sharing teacher expertise can also help teachers to learn new knowledge and skills. As for the “instructional approach”, teachers tend to shift the approach from “teacher-centered” to “student-center”. That is, for teachers, there are need more professional dialogues and cooperation among them in the information age. Doing so, it provides more opportunities for teacher professional growth and achieving breakthroughs in curriculum implementation. Students are expected to play more active role in the learning process. Teachers act like a facilitator helping students construct their own knowledge. If administrators simply focus on equipping the infrastructure, and do not pay sufficient attention to changes in instruction approach and the systematic curriculum design, it is difficult for

S.-R. Lin et al. / A Study of Curriculum Implementation and School Culture in the Information Age 243

teachers to change their instruction approach fundamentally, from teacher-centered to student-centered. 3.3 Underlying assumptions The underlying assumptions of school culture for teachers to implement curriculum in the information age include: the extension of time and space, and technology being able to increase instructional effectiveness. Web pages are not only one of the artifacts of school culture used by teachers to implement curriculum in the information age, and also imply underlying assumptions about the “extension of time and space” in curriculum implementation. That is, time for teachers to implement curriculum do not limited in the time during which teachers can implement curriculum is not limited to working hours, and space for teachers to implement curriculum is not limited to schools or classrooms, and also extends to the Internet. Thus, the boundaries of school culture in the information age will be more ambiguous than in the industrial age. This phenomenon will make school culture more complicated and difficult to define in the information age.

References [1] Cuban, L.ΰ1993α.How teachers taught: constancy and change in American classroom 1880-1890. New York: Teacher College Press. [2] Cuban, L. (2001). Oversold and underused. Cambridge, MA: Harvard University Press. [3] Deal, T. E. & Peterson, K. D. (1999). Shaping school culture: the heart of leadership. San Francisco: Jossey-Bass. [4] Lesgold, A.(2003). Detecting technology’s effects in complex school environments. In Haertel, G. D. & Means, B. (eds). Evaluating educational technology: effective research designs for improving learning. New York: Teacher College, Columbia University. [5] Reigeluth, C. M. (1994). The imperative for system change. In Reigeluth, C. M. aGarfinkle, R. J. (Eds.) Systematic change in education. Englewood Cliffs, NJ: Educational Technology Publications. [6] Roblyer, M.D. (2003,). Integrating Educational Technology into Teaching (3rd edition) Columbus, OH: Merrill Prentice Hall. [7] Sandholtz, J.M, Rinstaff, C., & Dwyer, D.C. (1997). Teaching and Technology: creating student-centered classrooms. New York: Teacher College, Columbia University. [8] Schein, E. H. (1992). Organizational culture and leadership (2nd edition). San Francisco, CA. JOSSEY-BASS. [9] Schein, E. H. (2004). Organizational culture and leadership (3rd edition). San Francisco, CA. JOSSEY-BASS.

244

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Sustaining Innovations in Singapore Schools: Implications for Research Work Chee-Kit Looia, Wei-Ying Lima, David Hunga Learning Sciences Lab, National Institute of Education, Nanyang Technological University, Singapore [email protected]

a

Abstract. This paper explores the broad issues of why the meaningful use of ICT to engage in deep learning has not been widespread in K-12 schools. We argue a critical reason is the lack of sustainability in the implementation of ICT-enabled pedagogies in schools, arising from a failure to understand the conditions for systemic innovations. We describe the key issues related to sustainability from literature, and contextualized some of these issues using 2 examples of innovations that have been implemented in the Singapore schools. Implications are then drawn with respect to how research work should be conducted with a view of sustainability, how to build up capacity of researchers and teacher implementers to push innovations, and new methods of analysis for studying sustainability. Keywords: Innovations; Sustainability; Scalability; Professional Development; Research Work

Introduction Since the advent of information-communications technology (ICT) decades ago, research on how ICT can be used in education has been incessantly conducted. Despite “decades of funded study that have resulted in many exciting programs and advances have not resulted in pervasive, accepted, sustainable, large-scale improvements in actual classroom practice, in a critical mass of effective models for educational improvement, or in supportive interplay among researchers, schools, families, employers, and communities” (Sabelli and Dede, 2001). Appropriately, many are investigating as to why the adoption and appropriation of good ICT-enabled pedagogies have proven to be so difficult. How is it that businesses such as McDonalds is able to counter cultural and social differences and franchise its product, process and even its culture successfully across the globe; while the adoption of good teaching practices in education seems sluggish in comparison. Although the reasons could be multi-factorial and complex in nature, we believe that the key lies in innovations not being sustained in schools for a period long enough to reap the desired teaching and learning results. Schools either jump from one innovation to another, or that implementations failed to consider the complexity of the educational system, societal needs, policies, curriculum, pedagogy, practices, epistemic beliefs, skills, and others. At this stage, while we recognize that technology is only one tenet in the complex system of education, it is however the only one tenet that affords the catalytic effect to trigger change in the other components of the system. Thus it is our belief that the effective use of innovations for engaged forms of learning cannot be researched, planned and implemented in isolation but rather, concomitantly with the other components in the system in order for educational reform to be successful. We begin by understanding from the research literature issues about sustainability and scalability. We then move on to describe two local examples of innovations that have

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

245

been implemented, Knowledge Forum and Knowledge Community. Finally, implications are drawn to how research work should be conducted and how to build up capacity of researchers and teacher implementers with respect to their roles as change agents. 1. Issues and Frameworks about Sustainability It is our belief that meaning-making for learners should be facilitated with learning technologies or innovations. By innovations, we mean learning technologies that have the catalytic ability to trigger change such that “one innovation will reverberate through other aspects of pedagogic practice and will touch upon the wider realm of a teacher’s daily work so that it too will be changed” (Breen, 1994 p. 104). How then can the process of change be initiated in schools where innovation is both a catalyst and a facilitator for deep learning? Who should the change-agents be and what should their dispositions, epistemologies, beliefs be? Obviously, change is not easily implemented and sustained as a typical response from a rational human being to change is resistance. One of the issues faced lies in the tension between the desire to scale effective practice, on the one hand, and issues of adaptation and customisation on the other (Honey & McMillan-Culp, 2000). In other words, how hard do we rock the boat? To bring about change thus requires strong leadership and shrewd management. Such daunting task may not seem palatable at the onset to many school and policy leaders. According to Cohen and Ball (1999), this challenge can be analysed in terms of specifications versus development. Specification is the degree of details an innovation is described for school take-up such as specified curriculum and learning goals, while development refers to the provision of resources required to enact innovations such as curriculum materials, professional development and others. Specification without development requires teachers to figure out how to enact the innovation in their local setting, which can be a barrier to all but the master teachers. Development without specification provides resources for improvement, but not a clear picture as to what goals are to be attained. Therefore the skill of the adapter is critical is assessing what and how much to adapt an innovation depending on the contextual conditions of the educational environment. In fact, according to Dede and Wirth, intervention strategies are different for scaling up a single innovation into a school as opposed to a full-scale systemic reform of that educational organization. For the former, designers of innovations, together with their community of practice should design for attenuation as adapters from schools who are not involved in the development of the innovation will enact selected conditions favorable only in their context. In other words, designing for attrition requires identifying conditions for success likely to be attenuated in many contexts, then evolving the design to retain substantial effectiveness under those circumstances (p. 11). According Century & Levy (2002), changes must be contained within which the core intent and philosophy of the origin reform is still reflected. They have thus defined sustainability to be: “The ability of a program to maintain its core beliefs and values and use them to guide program adaptations to changes and pressures over time”. (p. 4) To do so, consideration has to be given to the influential factors (see Figure 1) at any levels of the school system in the progression of each innovation, from establishment, maturation to evolution. At each of these three phases, an innovation has particular goals and particular strategies for achieving those goals. To be sustained, goals must be realized at different levels of the school system, which require multiple strategies employed simultaneously. Thus at any point in the evolution of an innovation, attention must be given to the influential factors at any of the levels in the system such that the stakeholders’ orientation is accurate.

246

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

Factors that are related to the concrete element of a program/innovation

Accountability Instructional materials Leadership Money Partnerships Professional Development

Factors that relate to influences on a program/innovation and outcomes that result from the program/innovation Implementation & adaptation Philosophy Critical mass Perception Quality History – origins & longevity

Figure 1. Factors summarized from Century & Levy (2002). Sustaining change: A study of nine school districts with enduring programs The Letus community offers another perspective into the issue of sustainability. If an innovation is not usable, adoption is unlikely, not to mention sustainability or scaling up of technologies. Blumenfeld & team (Blumenfeld et. al. 2000; Fishman 2002; Fishman et al. in press) put forth that the three dimensions within school systems: school culture, capability of practitioners and policy/management are the key aspects to look into, thus the conceptualization of the Usability Framework. Together, the three dimensions when arranged in the form of three axes originating form a common point, they form the three-dimensional space known as the ‘usability cube’. An innovation can hence be placed within the space for gap analysis. Such analysis helps researchers attempting to do such work in understanding the opportunities and processes by which one needs to look into for a sustained success in schools. Yet another strategy proposed by Hargreaves and Fink (2000) is to use model successes to reculture as well as to restructure schools–adopt the philosophy and intentions into the existing culture of adventurous volunteer schools before enacting the structures for change. The goal of systemic innovations must first be established in the larger cultural context such that it becomes an “entire continent of change” (Hargreaves, Earl & Ryan 1996). In fact, they argued that educators should go beyond the policy makers by making their practice and improvements visible to the public such that a broad social movement for large-scale, deep and sustainable transformation can be created (Hargreaves, in press). Concomitant to this notion, Cuban, in an interview with John O’Neil (2000), highlight that the innovations that have the best chance of sticking are those that have constituencies grow around them. An example he cited is that of special education. When reform efforts reflect some deep-rooted socio concern such as preparing autistic children to lead fulfilling lives, it will be widely accepted over a sustained period of time. Community Science Surrounding

Values

Culture

Sustainability Equity Factors

Institution

Decision making power

Figure 2. Abstracted from Century & Levy (2002, p.18)

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

247

Figure 2 is an attempt by Century and team to map out the complexity of sustainability and scalability in a way that appropriately represents the layered nature and complexity of their interactions. In the center sits the innovation and its influential factors. These factors are embedded in the larger surrounding conditions such as the school culture, decision making power and equity issues. The next ring represents the content area and other correlated issues such as pedagogy, content knowledge, status of science learning, beliefs of teachers etc. Finally, in the broadest ring, there is the community and its corresponding socio-cultural issues exerting pressure on the innovations and the stakeholders’ decisions. 2. Innovations in Singapore Schools: 2 Case Studies 2.1 Knowledge Forum Knowledge Forum (KF) is computer-supported collaborative learning (CSCL) environment for users to engage in collaborative work and discussion, providing a record of the development of ideas and a way of tracking users’ contribution for assessment purposes. In addition to providing a computer-mediated communication (CMC) platform, it can model expert thinking and support the processes of scientific inquiry through the use of scaffolds (Scardamalia, 2004; Scardamalia & Bereiter, 1998). As an innovation that facilites engaged forms of learning, the affordances of KF allow identities to be developed in its users, of which in this context is learning to be scientists. As a research project, Knowledge Forum was implemented across 6 schools over a span of 30 months with an overarching objective of investigating the impact of CSCL technology on students engaging in scientific inquiry. Although the research project had a fairly specified direction in terms of its project goals, curriculum materials, teaching and learning philosophy, the innovation itself is fairly general, allowing for enaction across various school contexts. Viewed from a perspective of usability, the implementation of the innovation is seen to be geared towards success. Understanding the fact that each school has their own ‘personality’ due to the different socio-cultural forces acting on it, the research team took on a customized approach in planning the intervention program. In each intervention, the research team studied the classroom needs and constraints in each school by working closely with the teachers involved, whilst having the research goals and directions at the back of their minds. Under such guided-customized approach, not only were many technical glitches solved, enactions were able to be carried out across several diverse schools contexts - primary and secondary schools with students of varying academic competencies (Ibrahim & Tan 2004; Tan Hung & So 2005; Tan, Yeo & Lim 2005; Teo, Chiam & Ng 2004). It was found that students, even at the primary level, were able to engage in knowledge building activities and experienced a change in perception of learning, from one of personal pursuit to that of a social collaborative stance (Ow, Low & Tan 2004). KF in this case, has proven its potential to be scalable across students of varying ages with diverse academic capabilities. Another contributing factor to the successful implementation is the focus on the teachers’ perceptions and philosophy to learning. The research team, upon realizing the import of epistemic beliefs and outlook in learning, changed its strategy from working with teacher advisors in each school to focusing on teachers’ perceptions in constructivist, knowledge building type of pedagogy. Training on CSCL and knowledge building strategies were carried out extensively in the pre-service, in-service and Masters Courses, before partnerships for implementation were formed.

248

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

Although the research was found to have met its objectives, the sustainability of the innovation in these 6 schools is unknown. Considering the fact that the teacher implementers were trained in knowledge building concepts and strategies, they would supposedly dispose a changed epistemic belief and perception in their classroom practice. However, a translation from belief to practice has yet to be seen which we think is related to the following point. Another reason, we conjecture for innovation not being sustained in schools could be that the project dealt little with the larger scheme of things such as changes in student assessment, schools’ core curriculum, and providing on-going professional development for teachers using KF. The sustainability of the innovations is pretty much left to the availability of teachers to continue in their off-peak periods. Without an expansive holistic approach to implementation, it is currently dependent on the teachers’ enthusiasm to continue using the innovation, not to mention expanding the use to other teachers or subjects within each school. 2.2 Knowledge Community Knowledge Community (KC) (http://www.globalkc.net) is another CSCL tool used largely for project building and inter-school collaboration. Through the multimedia workspace, users collaborate with one another by participating in forum discussions. Thinking types such as My question or My analysis were designed to scaffold users’ discussions by prompting them various ways to present their findings and to collaborate with others in the knowledge building process (Kwok & Tan 2004). Similar to KF, KC is designed with constructivist, situated cognition type of epistemology for engaging users in deep learning. Since its introduction to Singapore in 2002, KC has seen over 50 schools using its platform for different subject domains, ranging from language learning (Ng, Tan & Kwok 2004) to scientific inquiry and project work (Tan in press; Quek, Tan & Chai 2003). Being web-based with a low cost start-up (there is no need to buy server space), coupled with an intuitive user-interface, KC bears good potential to be a catalyst for change in schools. Besides the occasional bandwidth slowdown due to extensive concurrent users, the innovation has good track record and enjoyed successful implementations in both primary and secondary schools. In fact, in one primary school, the success of KC in the learning of a mother-tongue language was extended from an intra-school implementation to an inter-school collaboration. Although KC is widely known as a platform for project work, there is no fixed curriculum attached to the innovation and as such, teacher-implementers need to craft the project scope, objectives, technology usage, curriculum and others. While this is a good strategy to keep enthusiastic teachers highly involved in the project because of the extent of planning and effort required, thereby increasing the probability of the innovation to be sustained; it is also a deterrent for ‘takeup’ in some others. The scalability of the innovation across schools was largely due to the dissemination of its easy use by word of mouth as the distributor of KC believes that true success will come when one convinced user influences another. Take for example, a particular Head of Department (IT) who is so convinced of this new pedagogy, besides sharing with her peers in her school, evangelizing the innovation; she has gone on further to disseminate this ICT-enabled pedagogy in educational conferences, addressing international audience. However it has been observed that KC, despite having some success in being scaled across 50 schools, has not been sustained longitudinally in schools. Anecdotal feedback has it that there is no concerted body or community formed, looking into the support and professional development of its members in the use of KC. As such, there is little knowledge sharing, even amongst the early adopters who have introduced KC to others. Reasons could be that teachers perceive themselves being transformed from expert practitioners to novices in

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

249

what they lack experience and knowledge in; or that they see the change as not requiring them to teach substantially different from what they have always been doing (Seow 1999). 3. Implications for Research Work Indeed for scalability and sustainability to occur, the effective implementation of innovations has to be planned and executed en masse with the other components in the system. The catalytic effect of such systemic innovations leads to 3 implications for research work. One is the approach in which ICT-research is being conducted and implemented in schools and the other, the capacity development of the researchers and/or practitioners in schools. The third issue concerns with the development of analytical methods for studying scalability. First, in a systemic approach to implementation, researchers have to consider how large-scale it should be. Such considerations hinges on how receptive or adverse the components of the system are to change. In our opinion, such balancing act of implementing innovations in schools for sustainability is best conducted in tight collaboration with the Education Ministry and related policy-makers who has good knowledge of the ‘ground’. By focusing on the contextual conditions of each school, such partnership ensures appropriate policy and decisions made, thereby garnering more support and commitment, either through increased involvement of the teacher-implementers or strong backing from the school leaders. This will also overcome issues of the disconnect between researchers and policy makers that has received considerable attention (Glover, undated; Swope, undated; Lomas 2000). Second, it is imperative to factor in an effective professional development model such that capacity building takes place both among the researchers and the practitioners. To do so, researchers, on one hand, will need to heighten their awareness of the actual classroom conditions that practitioners face, the epistemic beliefs, pedagogic skills, the level of mastery of the teachers and so on. On the other hand, the right beliefs and dispositions will need to be inculcated in our pre-service and in-service teachers, school leaders, and graduate students, to have expansive mindsets towards students’ learning. They would need to recognize that while they stick to what works in preparing their students to do well in examinations, they are also cognizant of the competencies and skills that students need to engender to live in a very different world, often characterized by complex, ambivalent and relativist. All in all, we hope to see a profusion effect, albeit slow in building, that will see a significant percentage of these enlightened practitioners pursuing their ideals and dreams about educational innovation, setting up good practices and inspiring others. Finally, developing new methods of analysis for studying adaptability and scalability, assessing the scalability of an educational intervention or innovation becomes an important research frontier (Dede, 2004). It is suggested that a generalizable metric be developed that would measure the degree to which the educational effectiveness of the design is robust despite attenuation of its conditions for success. This index could provide prospective adopters a better idea of what the likely effectiveness of an innovation would be in their own context. Such research project can be arduous, requiring strong statistics number crunching and while ensuring that the data analyses are accurate and strongly grounded. According to Bosco & Bakia (2004), making use of research work generally has had three consequences: 1) Provide better ways to make research accessible to practitioners and policy-makers such as a clearinghouse approach, 2) Develop structures that provide for communication between researchers and practitioners and policy-makers, and 3) Develop and implement a national research agenda. In Singapore, the first Masterplan for IT in

250

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

Education, initially implemented in 1997, had progressed into the second Masterplan (II) which spans 2003-2007, focusing on engaged learning through meaning use of ICT in practice. Thus, with a clear and focused national agenda, a systemic approach towards ICT-research partnering policy-makers, and an informed, purposeful focus on capacity development involving both practitioners and researchers to spread the meaningful use of technology; we see that together, they form an effective strategy in achieving the three consequences listed by Bosco and Bakia. 4. Conclusion In this paper, we described some of the issues related to sustainability and generalizabilty of ICT innovations in schools and we attempted to contextualize these issues in two examples implemented in schools. We further explored the implications of these issues to how research work should be conducted. Together with an alignment of efforts with various stakeholders at different levels of educational organization, unceasing efforts to build and develop our capacity in the educational workforce and development of analysis through design research for studying sustainability, we hope to see workable processes and technologies supported by research findings in schools. Acknowledgments This research work is funded by Learning Sciences Lab, NIE Singapore. References [1] Blumenfeld, P., Fishman, B., Krajcik, J. S., Marx, R. W., & Soloway, E. (2000). Creating usable innovations in systemic reform: Scaling-up technology-embedded project-based science in urban schools. Educational Psychologist, 35(3), 149-164. [2] Bosco, J and Bakia, M (2004). "National Research Agendas for ICT in Education: A Descriptive Analysis" Conference paper presented to the CoSN International Symposium. Washington DC. Retrieved April 14, 2005 from http://www.cosn.org/resources/international/2004_symposium_agenda_paper.pdf [3] Breen, M. (1994). The Teacher’s Voice in Curriculum Change. An unpublished paper presented at a cuuriculum symposium in Singapore, RELC. [4] Century, J. R., & Levy, A. J. (2002). Sustaining change: A study of nine school districts with enduring programs. A paper presented at the Annual Meeting of the American Educational Research Association, April 3, 2002. [5] Cohen, D. K., & Ball, D. L. (1999). Instruction, capacity, and improvement (CPRE Research Report Series No. RR-043). Philadelphia, PA: University of Pennsylvania Consortium for Policy Research in Education. Retrieved March 30, 2005, from http://www.cpre.org/Publications/rr43.pdf [6] Dede, C. (2004). Design for Defenestration: A strategy for Scaling Up Promising Research based Innovations. Chicago, IL: NORC. [7] Dede, C. & Wirth, T. E. Scaling Up: Evolving Innovations beyond Ideal Settings to Challenging Contexts of Practice. In Ed. Cambridge Handbook of Learning Sciences (to be published in 2006). [8] Fishman, B. (2002). Linking the learning sciences to systemic reform: Teacher learning, leadership & technology. Invited talk at the Annual Meeting of the American Educational Research Association, New Orleans, LA. [9] Fishman, B., Marx, R., Blumenfeld, P., Krajcik, J. S., & Soloway, E. (in press). Creating a framework for research on systemic technology innovations. Journal of the Learning Sciences. [10] Glover, D. (Undated) “Policy researchers and policy makers: Never the twain shall meet?”, Retrieved April 14, 2005 from http://www.eepsea.org/en/ev-8311-201-1-DO_TOPIC.html [11] Honey, M., & McMillan-Culp, K. (2000). Scale and localization: The challenge of implementing what works. In M. Honey & C. Shookhoff (Eds.), The Wingspread Conference onTechnology's Role in Urban School Reform: Achieving Equity and Quality (pp. 41-46). Racine, WI: The Joyce Foundation, The Johnson Foundation, and the EDC Center for Children and Technology.

C.-K. Looi et al. / Sustaining Innovations in Singapore Schools: Implications for Research Work

251

[12] Hargreaves, A. (in press) Beyond anxiety and nostalgia: Building a social movement for educational change. Phi Delta Kappan. [13] Hargreaves, A., Earl, L., & Ryan, J. (1996). Schooling for change. London: Flamer Press. [14] Hargreaves, A. & Fink, D. (2000). The three dimensions of educational reform. Educational Leadership, 57 (7), 30-34. [15] Ibrahim, A., & Tan, S.C. (2004). Computer-supported collaborative problem solving and anchored instruction in a mathematics classroom – An exploratory study. International Journal of Learning Technology, 1(1), 16-39. [16] Kwok, P & Tan, C. (2004). Scaffolding Supports in Project-based Learning through Knowledge Community (KC): Collaborative Learning Strategies and Pedagogical Facilitation. GCCCE 2004, Hong Kong. Retrieved Apr 2005. http://www.learningexpert.net/publication.htm [17] Lomas, J. (2000). “Connecting research and policy,” ISUMA, Spring 2000 Retrieved April 14, 2005 from http://www.isuma.net/v01n01/lomas/lomas_e.pdf [18] NG, K.E., Tan, C. & Kwok, P. (2004). A comparative study of Grade 6 students’ self-regulation on scaffolding narrative writing in Chinese through face-to-face communication and e-learning communities. ICCE 2004 Proceeding (Long Paper), Sidney. [19] O’ Neil, J. (2000). Fads and fireflies. The difficulties of sustaining change. Educational Leadership. April, 6-9. [20] Ow, E.G.J., Low, A., & Tan, S.C. (2004). Learning to be scientists through social collaborative discourse – A case study in a primary school. Paper presented at the International Association for the Study in Cooperative Education Conference 2004, 21 June to 25 June, Singapore. [21] Sabelli, N., & Dede, C. (2001). Integrating educational research and practice:Reconceptualizing the goals and process of research to improve educational practice.Retrieved Mar, 2005, from http://www.virtual.gmu.edu/SS_research/cdpapers/integrating.htm [22] Scardamalia, M. (2004) ‘CSILE/Knowledge Forum®’, in Educational technology: An encyclopedia, ABC CLIO, Santa Barbara. [23] Scardamalia, M., & Bereiter, C. (1998). Web Knowledge Forum. User guide. Santa Cruz: Learning in Motion. [24] Seow, A. (1999). Coping with School-based Curriculum Change. React. June, 30-34. [25] Swope, J. (Undated) “Closing the gap: New ways of strengthening the link between educational research and decision making on educational policies.” Retrieved April 14, 2005 from http://orion.forumone.com/gdnet/files.fcgi/111_swope.PDF [26] Tan, C. (In Press). 3-I Project Learning: Learning Ecology of Interdisciplinary, Inter-School and International Project Learning in Primary Schools through Knowledge Community, QEF, Hong Kong [27] Tan, S.C., Hung, D., & So, K.L. (2005, forthcoming). Fostering scientific inquiry in schools through science research course and Computer-Supported Collaborative Learning (CSCL). International Journal of Learning Technology. [28] Tan, S.C., So, K.L., Yeo, J., & Lim, W.Y. (2005, forthcoming) Fostering scientific inquiry skills with Computer-Supported Collaborative Learning- A case study. Journal of Computers in Mathematics and Science Teaching. [29] Teo, C.L., Chiam, C.L., & Ng, F.K. (2004). Fostering Knowledge Building among Low Achievers through Technologies: A Perspective from Singapore. Teaching and Learning, 25(1), 89-102. [30] Quek C. L. G., Tan, C., Chai C. S. (2003). Using a Conversation Tool in Interdisciplinary Project Crafting: An Exploratory Study. ICCE 2003, Hong Kong.

252

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

E-learning System Acceptance: Implications to Institutional Implementation Strategies a

Will Wai-kit MAa, Allan Hoi-kau YUEN b Hong Kong University of Science and Technology & Hong Kong Shue Yan College, HKSAR, China b Centre for Information Technology in Education, The University of Hong Kong, HKSAR, China [email protected]

Abstract. Today, most universities have enhanced their classroom courses with online learning systems. However, studies still find organizational systems with specific low usage, compared with the large population within the universities. In order to suggest effective institutional implementation strategies in e-learning system deployment, a validated model framework is applied to a local private tertiary institution. 507 respondents (students) completed an online survey during the second year of the system deployment. Significant correlations were found for performance expectancy, effort expectancy, social influence, and facilitating conditions, in predicting and explaining intention to use of the system. Age and voluntariness were found significant moderator to the determinants; while gender and computer experience were found non-significant in the model. Based on the results, implementation strategies, including institutional issues, management issues, technological issues, pedagogical issues, ethical issues, interface design issues, resource support issues, and evaluation issues were discussed. Limitations and further research agenda were also suggested. Keywords: Information technology acceptance/adoption, usage, e-learning system

1. Introduction Today, many universities employed online learning systems to support teaching and learning. However, the institutionalized scale of technology implementation involves a lot of parties. There are a lot of factors affecting the successful use of the system for effective teaching and learning support. Among the various critical factors affecting the success of this IT/IS implementation, acceptance of the technology becomes the very first critical factor. That is, whether users start to use the system. Who are the users? They include the institution administration, the instructors and the student learners. The institution needs to provide the system and install at campus. The instructors need to design and develop their course in the system. The student learners need to login the system to interact with the online course and to interact with the instructor or other peer learners. A survey of previous IT/IS studies found that systems implementation are not easy task and most organizational system usage are still low (e.g., [1], [5]). It is also found that IT/IS acceptance studies are always a core concern in information system research [4]. Here in this study, a local, private university implemented an e-learning system last year. At the second academic year of the usage of the e-learning system, it is prime time to learn more of the student users on how they perceive the system. The research questions are: 1.

What are the determinants of behavioral intention to use of the e-learning system?

253

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

2.

What relevant moderators will affect the level of these determinants towards the intention to use of the system?

3.

What organizational implementation strategies could be suggested based on the findings in (1) and (2)?

The paper follows with the methodology and the measurement instrument we use of this research. Then, the data analysis of the data collected from an online survey. Third, determinants and moderators are tested with regression analysis. Finally, discussions on the implementation strategies, limitations of the study and future suggested research are given.

2. Methodology To test our research model, we conducted a web-based survey in the institution’s e-Learning system (Interactive Learning Network, ILN) during the period Feb 15 to Feb 28, 2005. There was an announcement posted on the ILN. Student users were invited to complete the survey, through a hyperlink in the announcement. 507 students (out of a total of 2700 ILN student users or 19 percent of the total) responded and completed all questions. The table below showed the summary of the demographics of the respondents (see Table 1 and Table 2). Table 1. Demographics of Respondents (Valid N: 507) Gender Age

Program Department

Male Less than 19 19-20 21-22 Diploma Accounting Business Admin. Chinese Lang. & Lit. Counseling & Psy. Economics Eng. Lang. & Lit.

% (No.) 32.1 (163) 0.2 (1) 27.4 (139) 54.6 (277) 30.8 (156) 22.7 (115) 22.3 (113) 2.8 (14) 16.6 (84) 4.3 (22) 0.2 (1)

Female 23-24 Over 24

% (No.) 67.9 (344) 15.8 (80) 2.0 (10)

Bachelor History Journalism & Comm. Law Social Work Sociology

69.2 (351) 0.4 (2) 15.4 (78) 0.8 (4) 5.7 (29) 8.9 (45)

Table 2. Self-reported Computer Competence PC competence-Word Processing PC competence-Email PC competence-Internet Search PC competence-Multimedia Authoring PC competence-Installation of Software PC competence-Installation of Hardware

Mean 4.80 5.08 4.71 3.84 4.39 3.60

Std. Dev 1.324 1.343 1.462 1.488 1.636 1.677

2.1 Measurement Items In this study, we relied on a validated technology acceptance model framework, Unified Theory of Acceptance and Use of Technology (UTAUT) [8]. The framework contains four core determinants of intention and usage, and up to four moderators of key relationships. The four determinants include Performance Expectancy (PE; measured in 4 items); Effort Expectancy (EE; measured in 4 items); Social Influence (SI; measured in 4 items); and Facilitating Conditions (FC; measured in 4 items). The dependent variable is measured in 3 items of Behavioral Intention (BI). All items are listed as statements (see Appendix) and

254

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

respondents were asked to evaluate themselves to choose the best appropriate statement from a 7-point Likert’s Scale, where 1 represents Strongly Disagree and 7 represents Strongly Agree.

3. Data Analysis One-way ANOVA was employed to measure if there were any significant differences between groups, towards the composite scores of the four determinants, including performance expectancy (PE), effort expectancy (EE), social influence (SI) and facilitating conditions (FC) and the dependent variable behavioral intention (BI). Table 3. One-way ANOVA Gender Age Computer Exp. Voluntariness

F-value (*p<0.05; **p<0.01; ***p<0.001; ns – non-significant) PE EE SI FC BI 0.896 (ns) 3.519 (ns) 0.262 (ns) 3.262 (ns) 0.753 (ns) 1.217 (ns) 1.441 (ns) 4.763*** 3.144* 0.928 (ns) 3.646*** 5.523*** 2.257*** 3.959*** 2.701*** 13.965*** 15.477*** 12.984*** 11.785*** 15.632***

There were no significant differences between gender groups for all determinants scores. There were no significant differences among different age groups, except social influence (p<0.001) and facilitating conditions (p<0.05). There were very significant differences among groups with different levels of self-reported computer competence. There were also significant differences among groups who perceived the system as mandatory towards voluntary in use. These may have potential threats to the analysis results. These factors should be included into the model as control variables and their effects on the model should be noted. A summary of descriptive analysis of the construct items was listed (Table 4).

Table 4. Descriptive Analysis and Reliability Analysis PE1 PE2 PE3 PE4 EE1 EE2 EE3 EE4 BI1 BI2 BI3

Mean 4.87 4.79 4.52 3.95 5.01 4.82 5.18 5.24 5.41 5.72 5.41

Std. Dev 1.154 1.316 1.234 1.387 1.184 1.199 1.217 1.231 1.180 1.148 1.213

Cronbach’s Alpha 0.824

0.880

SI1 SI2 SI3 SI4 FC1 FC2 FC3

Mean 5.39 4.76 4.59 5.08 5.41 5.07 4.56

Std. Dev 1.195 1.240 1.226 1.355 1.211 1.117 1.093

Cronbach’s Alpha 0.725

0.697

0.889

We performed several tests on the measurement model to examine its validity and reliability. Reliability analysis with Cronbach’s alpha found that all constructs are either higher than or very close to the recommended value of 0.7 [6], demonstrating internal consistency. The three items of Facilitating Conditions has an alpha value of 0.697, which is below though very close to the recommended value. This may be a potential threat to the validity of the construct. Table 5. Pearson’s Correlation Coefficient Matrix PE1 PE2 PE3 PE4 EE1 EE2 EE3 EE4 SI1 SI2 SI3 SI4 FC1 FC2 FC3 BI1 BI2 BI3 PE1 1.00 PE2 0.57 1.00

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

255

PE3 0.63 0.68 1.00 PE4 0.44 0.49 0.46 1.00 EE1 0.49 0.45 0.48 0.33 1.00 EE2 0.49 0.53 0.61 0.41 0.60 1.00 EE3 0.41 0.36 0.42 0.30 0.66 0.55 1.00 EE4 0.42 0.41 0.44 0.34 0.67 0.57 0.82 1.00 SI1 0.36 0.28 0.32 0.14 0.33 0.28 0.21 0.24 1.00 SI2 0.46 0.46 0.49 0.30 0.45 0.53 0.33 0.37 0.42 1.00 SI3 0.48 0.40 0.47 0.27 0.39 0.44 0.36 0.34 0.48 0.34 1.00 SI4 0.45 0.37 0.47 0.21 0.38 0.36 0.32 0.33 0.37 0.35 0.43 1.00 FC1 0.40 0.49 0.42 0.29 0.42 0.43 0.42 0.46 0.36 0.43 0.26 0.33 1.00 FC2 0.43 0.43 0.50 0.30 0.54 0.50 0.52 0.60 0.29 0.40 0.33 0.36 0.56 1.00 FC3 0.48 0.38 0.45 0.34 0.37 0.42 0.41 0.38 0.26 0.37 0.43 0.33 0.32 0.41 1.00 BI1 0.52 0.47 0.48 0.31 0.60 0.45 0.53 0.55 0.39 0.54 0.31 0.40 0.49 0.50 0.37 1.00 BI2 0.44 0.38 0.35 0.26 0.52 0.40 0.41 0.44 0.43 0.47 0.30 0.37 0.47 0.40 0.29 0.70 1.00 BI3 0.54 0.47 0.44 0.40 0.54 0.41 0.48 0.48 0.38 0.47 0.33 0.36 0.50 0.49 0.37 0.72 0.76 1.00

All construct items are highly correlated, at p<0.01 level of significance. All correlations loaded on their respective construct are higher than the correlations among the constructs, demonstrating both convergent and discriminant validity.

3.1 Measurement Model Testing Linear regression analysis was employed to test the model fit. The four determinants were hypothesized to explain and predict dependent variable behavioral intention to use. Composite scores were used in the regression analysis. Moderating variables were added as control variables. The change in R-square was measured and compared. The coefficients and their level significance of constructs were measured and compared. Table 6. Model Testing Results Constructs PE EE SI FC

Model 1 (control variables) -

Model 2 (determinants) 0.128** 0.283*** 0.238*** 0.195***

Model 3 (All factors) 0.093* 0.244*** 0.209*** 0.200***

GENDER AGE EXPERIENCE VOLUNTARINESS

0.019 (ns) -0.060 (ns) 0.137*** 0.490***

-

-0.007 (ns) -0.086** -0.022 (ns) 0.166***

2

R 0.296 0.509 2 Adjusted R 0.290 0.505 2 0.213 ΔR Dependent variable: BI *p<0.05; **p<0.01; ***p<0.001; ns – non-significant

4. Discussions

0.534 0.527 0.025

256

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

The model testing results found that four determinants were significantly correlated with intention to use, including, performance expectancy (p<0.05; Ͳ=0.093); effort expectancy (p<0.001; Ͳ=0.244); social influence (p<0.001; Ͳ=0.209); and facilitating conditions (p<0.001; Ͳ=0.200). On the other hand, age (p<0.01; Ͳ=-0.086) and voluntariness (p<0.001; Ͳ=0.166) were also found significant in predicting intention to use. The model explained 53.4% change in variance of the dependent variable, behavioral intention to use of the system (R2=0.534). Here follows a discussion of the implications based on the data and model analysis results.

4.1 Institutional issues Before the institution launches the e-learning system, there should be a needs analysis. This is important to learn more about the instructors and student learners in order to provide the most appropriate service to them. It is important to view the system as service to users, instead of a machine. From the model testing results of the present student survey, performance expectancy, although is significant in predicting intention to use, it is however very weak to explain intention (p<0.05; Ͳ=0.093). That is, students do not think the system could improve much of their performance, or improving performance is not the main reason whether they use or not to use the system. A lot of previous studies found that perceived usefulness should be the most fundamental determinant to intention to use [4]. However, at the moment, the institution could not associate performance improvement with the use of the system. It is suggested that the institution should devise strategic plans to promote, to explain, and to demonstrate with best practices that the system usage will be associated with performance.

4.2 Technological issues Technological issues do not limit on hardware and software availability but also a strategic infrastructure planning. Rosenberg [7] states that e-learning infrastructure is based on an institution’s technological capabilities to deliver and manage e-learning. The institution should consider questions like: (1) What technological or technical capabilities are required to support e-learning? (2) What essential skills are needed by learners, instructors, and support staff to be successful in ever changing digital learning environment? [3] Model testing results above found that students have strong concern to the effort they need to pay in order to understand and interact with the system (effort expectance: p<0.001; Ͳ=0.244). Students associated much of the resources, knowledge and skills required to the intention to use (facilitating conditions: p<0.001; Ͳ=0.200). If the institution would like to uplift the level of intention to use of the system, the institution must devise concrete infrastructure planning to provide the necessary training to student users in order to interact with the system “free of effort.” Training should not be a one-off offer. It should be a continuous structured planning to provide the student users the resources, knowledge and computer skills in order to cope with their daily use of the system, to help improve the learning processes. On the other hand, hardware and necessary software applications should be provided in such a way that general access to the system is reliable and consistently available. In discussion with some of the students, it was found that they could not login the system most of the time at class. This becomes annoying. With this painful experience, it also explains why students concern much about the effort expectancy and facilitating conditions. Therefore, institution should have a thorough infrastructure planning in order to

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

257

uplift the level of intention to use.

4.3 Pedagogical issues Previous IT/IS studies suggested that social influence is more important under a mandatory condition [2]. That is, users will be more influenced by their important others, if they are required or expected to use the system. In the above analysis, it was found that social influence is significant and strong in predicting and explaining intention to use (p<0.001; Ͳ=0.209). However, voluntariness significantly moderated the levels of intention to use (p<0.001; Ͳ=0.166). On the one hand, instructors encouraged students to use the system. They provide all kinds of incentive, including bonus points added to frequent users. However, it is found that, under this circumstances, students are more prone to be affected by their important others, including their peers, classmates, instructors, and even the institution. What implications do these findings suggest as a result? The institution should cultivate an environment of group learning and experience sharing. The more support one gains from his or her social circle during the learning process, the more he or she intends to use the system. In this institution, it helps to organize teaching staff to join together to help each other. A “hub” is suggested so that each department would identify one key person to coordinate and to offer help. On the other hands, instructors are encouraged to give more group activities and group assessment methods so that students will form more close social circles among themselves.

5. Limitations of the Study While the study identifies a number of crucial factors influencing student adoption, the study is not without limitations. Owing to the pedagogical differences between instructors, each online course is designed and conducted in a very different manner. This will affect the perception of students, towards the learning system. That is, students’ opinion perceived about the course will project to the successfulness of the learning system. Therefore, there will be limitations in generalized the results from course to course, and from department to department. There will also be potential threat to extend the results to apply to other institutions. There is only the online survey. Therefore, there will be a potential bias that we only collect opinion from those who are using, and even who are keen to use the learning system. On the other hand, a significant shortcoming of the study comes from the student only subject. While the usage of the system depends also on the instructor, without considering the opinion from teaching staff will be a big hole for the complete picture. The alpha value for facilitating conditions was below the required threshold (α=0.697) though which was very close to it. This might pose a potential threat to the validity of the construct. However, as an exploratory study to the phenomenon, this study serves as a good start, even after considering its limitations at the present state.

6. Suggestions for Future Research The e-learning system involves the institution, the instructors and the student learners. We must consider also the resources and support given by the institution on the one hand, the instructors’ opinion and competence on the other hand, in addition to what we know about the student learners. Future study on how the instructors perceive the e-learning system is suggested. In addition to survey based study, there may be qualitative analysis, including

258

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

individual interview, focus group interview, and observations, in order to explain more of the barriers instructors have, resources and knowledge they require for the usage of the system. Beliefs may also change over time. It is also suggested future study on a longitudinal perspective, in order to identify and measure the changes over time.

7. Conclusion Most universities have enhanced their classroom courses with all sorts of online learning systems. However, without detail studies on the users about these systems, we are not certain about why these systems are still in low usage, compared with the large population within the universities. This study presents the results of an institutionalized online survey to collect students’ opinion about an e-learning system. It was found that there were four determinants, performance expectancy, effort expectancy, social influence, and facilitating conditions, significantly correlated and in predicting and explaining intention to use of the system. Relevant control variables were identified, where age and voluntariness were found significant moderator to the determinants. Based on the results, implementation strategies, including institutional issues, management issues, technological issues, pedagogical issues, ethical issues, interface design issues, resource support issues, and evaluation issues were discussed.

8. References [1] W.H. Delone, Determinants of Success for Computer Usage in Small Business, MIS Quarterly, 12(1), 1988, pp.51-61. [2] J. Hartwick, & H. Barki, Explaining the Role of User Participation in Information System Use, Management Science, 40(4), 1994, pp.440-465. [3] B. Khan, Managing E-Learning Strategies, InfoSci, 2005. [4] P. Legris, J. Ingham, P. Collerette, Why do People Use Information Technology? A Critical Review of the Technology Acceptance Model, Information & Management, 40, 2003, pp.191-204. [5] Mahmood, M.A., Hall, L., & Swanberg, D. L. (2001). Factors Affecting Information Technology Usage: A Meta-Analysis of the Empirical Literature, Journal of Organizational Computing & Electronic Commerce, 11(2), pp.107-130. [6] J.C. Nunnally & I.H. Bernstein, Psychometric Theory, McGraw-Hill, New York, 1994. [7] M.J. Rosenberg, E-Learning: Strategies for Delivering Knowledge in the Digital Age, McGraw-Hill, New York, 2001. [8] V. Venkatesh, M.G.. Morris, G.B. Davis, F.D. Davis, User Acceptance of Information Technology: Toward a Unified View, MIS Quarterly, 27(3), 2003, pp.425-478.

Appendix: Measurement Items (Adapted from [8]) Constructs / Items Performance Expectancy (PE): It is defined as the degree to which an individual believes that using the system will help him or her to attain gains in job performance (p.447). PE1: I would find ILN useful in my study. PE2: Using ILN enables me to accomplish tasks more quickly. PE3: Using ILN increases my productivity. PE4: If I use ILN, I will increase my chances of getting higher grades for the course. Effort Expectancy (EE): It is defined as the degree of ease associated with the use of the system (p.450). EE1: My interaction with ILN would be clear and understandable. EE2: It would be easy for me to become skillful at using ILN. EE3: I would find ILN easy to use. EE4: Learning to operate ILN is easy for me. Social Influence (SI): It is defined as the degree to which an individual perceives that important others believe he or she should use the new system (p.451).

W.W.-k. Ma and A.H.-k. Yuen / E-Learning System Acceptance

259

SI1: My instructor thinks that I should use ILN. SI2: My classmates think that I should use ILN. SI3: The instructor of this course has been helpful in the use of ILN. SI4: In general, the college has supported the use of ILN. Facilitating Conditions (FC): It refers to the degree to which an individual believes that an organizational and technical infrastructure exists to support use of the system (p.453). FC1: I have the resources necessary to use ILN. FC2: I have the knowledge necessary to use ILN. FC3: ILN is compatible with other systems I use.

260

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation Naomi Miyake School of Computer and Cognitive Sciences, Chukyo University, Japan [email protected] Abstract. Collaborative learning is a complex process with the potential to yield a variety of positive outcomes, particularly when supported by rich, well-designed uses of technology. The students are expected to learn not just the in-depth contents but also their integration through collaborative learning experiences. In addition, many collaborative learning practices expect the students to gain learning skills as well. These three different learning outcomes are compared in detail in this paper for three successful students observed in a highly collaborative, technology-supported college class. The comparison reveals that each acquired knowledge and skills in a unique way in addition to their basic learning. One student developed a solid, balanced understanding of all the learning materials, another integrated his choice of materials to attain insights into the topic of the class, and another student was very successful in strengthening a learning skill. This multifaceted pattern of outcomes reveals the richness of successful collaborative learning and compels us to explore divergent evaluation methods in learning-science research. Keywords: Divergent evaluation, Outcomes of collaborative learning, College education

Introduction Collaborative learning is a very complex process in which many factors interact to yield various positive outcomes. Recent studies indicate that this is particularly evident when support is available through rich, well-designed uses of technology. The students in this situation are expected to learn not just more in-depth contents than in regular learning situations but also to integrate what they have learned, rather than just remembering the information by rote. Many studies can be cited here; some reveal the effects of two sigma or even more when compared against regular classes [1]. Some studies also indicate that knowledge integration requires time, and the real effects through new methods of learning cannot be justly measured by simple tests like multiple-choice problems. [2] Many collaborative learning practices expect the students to gain learning skills as well as content learning [3, 4]. In this paper we compare three different learning outcomes from content understanding, knowledge integration, and learning skill acquisition, to verify whether they are simply different aspects of a united single learning process or are in fact different methods of learning that merit more attention as we evaluate our research. To this end, we present a detailed comparison of three successful students observed in a very collaborative, technologically-supported college class. The comparison revealed that each student developed his/her own unique abilities, which they brought to the basic required achievement of the class. One student developed a balanced and solid understanding of all the learning materials, another student was more concerned with unraveling the commonalities and differences among the learning materials and integrating the materials to reach an insightful understanding of the topic of the class. A third student successfully

N. Miyake / Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation

261

acquired the skill of asking questions to develop a skill that she wished to attain. This multifaceted pattern of outcomes is evidence of the richness of collaborative learning. It also compels us to seek a more robust theory for evaluating complex-learning outcomes and new developments of divergent evaluation methods for successful learning-science research.

1. Research Context We explored collaborative instruction to teach college sophomores introductory cognitive science to disseminate the basic cognitive skills for everyday problem solving to a wider audience. The research context we report here was a 13-class CSCL term involving repeated collaborative readings and short descriptions of basic research findings. This method, which we call the dynamic jigsaw, enables the students to form general or abstract concepts from the research findings to be applied to their everyday cognitive tasks. Their collaborative activities were supported by a concept-mapping tool called the Reflective Collaboration Note, which they could use through the Internet.

1.1 Teaching Objectives All material on each page should be centered on an A4 size (8.26 x 11.69 inch) page. The following margin settings in MS Word will produce the correct result, for A4 size paper: top 1 inch; bottom: 0.8 inch; left and right: 1.1 inch. It is important to check the margins even if you use this template, because they might have been overwritten by your local settings. Cognitive science covers many functions and mechanisms of cognition using multi-disciplinary methodology. It is thus important to form an integrated view of the findings to understand the implications of the field. This integration is difficult since findings come from research conducted with different participants performing different tasks, as well as on conclusions drawn from different methods of data processing. We can simplify it into the following three steps to facilitate the integration process. Step 1. Comprehend many research findings in terms of their themes, evidential data, and conclusions. Step 2. Form an initial or hypothetical theory to integrate the above research findings in terms of implications. Step 3. Find possible applications of the theory that makes it usable in the future. We implemented these steps in a 13-class term for sophomores majoring in cognitive science in a Japanese university. The course involves collaborative understanding of 24 learning materials, each representing classic research in three different domains of cognitive science. The students collaboratively read, explained, exchanged, and discussed the material to integrate it and ultimately form an abstracted view. We use a method we call the dynamic jigsaw.

1.2 Learning Materials The participating students studied 24 technical texts, or research reports. Each text was taken from a cognitive-science textbook to convey one research theme in 24 topic areas and was subsequently rewritten to fit onto two pages to facilitate reading. The 24 topic areas were categorized into three domains: (1) Language acquisition and developmental studies, (2) Interactive features of knowledge processing, and (3) Social and cultural biases in

262

N. Miyake / Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation

cognition. Each domain consisted of eight texts. The eight texts in each domain were further subdivided into four pairs. Pairs were formed with two research reports with apparent relationships. The participating students were expected to repeatedly exchange explanations of 24 selected texts of research findings with other students, and to gradually become able to give an integrated explanation of several findings during the course of thirteen classes in ten weeks.

1.3 The Reflective Collaboration Note: The Support Tool The collaborative learning in this class was supported by a CSCL tool called the Reflective Collaboration Note (ReCoNote). ReCoNote is a note-sharing tool that can adapt to group activities by flexibly utilizing a conceptual map. Links can be interconnected between notes laid out on a sheet and comments can be added to all notes. The sheets can be layered to represent a hierarchical structure for the concepts. Figure 1 depicts a typical sheet.

Fig. 1 Example of a concept map on ReCoNote.

1.4 The Dynamic Jigsaw The 24 texts were collaboratively read by the entire class through dynamic jigsaw activities. The students were given all the titles with short descriptions of the contents at the onset of the course and were encouraged to attempt to read several of them and to choose three or four that were most relevant to their interests and research concerns. The choices were then pooled and distributed so that each text was covered by two or three students. These first assignments were called core readings, and students who were assigned to the same texts were to work together to become our experimental experts on that subject. Aronson’s original style calls for a text to be divided into six parts and read by six members, each of whom is responsible for different parts. They then get together to answer questions covering all six parts, which requires them to participate equally to succeed. This method was invented to solicit equal participation in learning, but this can also be a model for the socially constructive interaction of researchers. They assume different responsibilities to study a common theme and exchange their findings so that each member can construct his/her own interpretation. They most often do this repeatedly, explaining their findings to different audiences so that they can examine them from different perspectives to explore various integration possibilities and reach a coherent comprehension. The dynamic jigsaw models this process and requires each participating student to first become an expert of his/her core topic, then repeatedly explain the core topic to different audiences, receive their explanations, and integrate them to form a new explanation to enhance understanding. The group of students assigned to each text studies it by answering questions posed on a Wiki to help them comprehend the structural elements of the text. The answers can be transferred onto a concept-mapping sheet on the ReCoNote described above.The students

N. Miyake / Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation

263

created these idea maps of their cores during the first two classes. The students were encouraged to ask questions during this phase, as well as to give a practice explanation to a TA or an instructor (there were four TAs and two instructors). They prepared the starting concept map of their cores, and in the process they were also encouraged to practice exchanging their explanations with members of the paired text and relate it to their core, to form an integrated understanding of the pairs. The maps, contributed by the entire class and covering all 24 research reports, were then kept on the web for sharing and modifying, both in and out of class periods. All the students were expected to know and be able to explain at least two research descriptions by the end of this phase. The students began to exchange their explanations with neighboring pairs when they felt ready to explain the core-based pair of texts. Each student was expected to know four texts at the end of this phase, two of his/her own plus two more explained by another student. A student was expected in the following phase to exchange explanations of the four texts he/she had integrated with another student’s integration of his/her four texts, to cover eight texts of the domain containing his/her core topic. By this time, each student was an expert of eight texts in one of the domains. These domain experts exchanged explanations of the domains toward the end of the term. The class was 90 minutes long and met once a week for the first nine weeks and twice a week in the last two weeks in the fall of 2004.

1.5 Data Collection and the Participants The data used in this study came from many sources, including the log of ReCoNote use, audio records, and the protocols of the students’ conversation during the classes, their term papers, and their answers in occasional interviews. There were 68 registered students in the class. We focus on three of these, ST, YO, and KM. They are all successful students who finished the course with good grades. We focus on them because we are interested in the best possible performance in the collaborative learning situation we created. They were also recruited because their records were complete.

2. Learning Outcome Measurements The outcome of the course was measured using the students’ final concept maps as well as their term papers, using three indices.

2.1 Basic Achievement Measure The students’ basic achievements were measured by how well each research finding was described in the term paper, with the necessary components, such as research evidence, in order. The paper was scored higher if the research findings were explicitly related. Additional points were given when it also described how the knowledge thus integrated could be used to solve daily problems or be applicable to other fields of science. The overall performance for this class was relatively high. All 24 learning materials were covered in 83% of the term papers, out of which 56% had the necessary components, including the theme, evidence, conclusions, and implication, in that order. This indicates that the students learned both the basic contents of the learning materials as well as how to compose concise summaries. The three participants we focus on in this study were within the top 10% of this class.

264

N. Miyake / Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation

2.2 Other Outcome Measures We focus here on three aspects of collaborative learning, which has been said to promote knowledge integration in addition to mere content learning. One aspect is the formal integrity, or how structured the concept maps were at the end of the term. Another is a more qualitative measure, focusing on the insight the students attained during the course. The third is the collaborative learning skills, such as question-asking and hypothesis solicitation, which are associated with collaborative learning experiences.

2.2.1 The Formal Integrity This index concerns the degree of integration, or the structural coherency, of the final concept maps the students created after the course, before the deadline of the term papers. The maps were converted into a schematic graph view, with the sheets and notes on them linked by arrows. Figure 2 depicts three graph views for the three target students. The student on the left, ST, created a far richer, better balanced map than the other two. It exhibits the most balanced structure, containing all the information presented in all 24 learning materials. N22

N9

N26 N5

N21

N23

N10

W21

N2 7

N18

W24

N24

N7

N23

RS71 RS65

N8

RS49

N25 N109

RS47

S3

N6

N5 4

N8 3

N4 6

N120

N7 6

N124

N127 N91

N48

RS92

N7 0

W20

S8

RS84

RS14

N113

N3

N89

N2

W13

N14

RS90

N117 N20

N13

S38

W22

S42

N1 9

N9

N8

RS31 S64 S63

S35

N1

N3 0

S2 N12

N12

RS15

S1

N14

N9 4

N110

N2 1

N26 N107

S24

N1 2

N17

N7

h203025

N103

N97

RS27

N112

S4 1

N9 6

N125

N105

N68

N102

N104

N32

N5

N9 8

S25

RS53

N38

N116

S4 4

N4

N10

S19

W7

S36

RS29

N108

S62 N7 4

N52

S57

S39 S4 0

W6

N28

S58

RS86

N15

N5

N3

S37 N9 3 N9 9

S23 N8 5

N106

N9

N27

N 101

S61 N11

h203063

S1

RS69

N9 5 N100

h203034

N1 8

N28

W19

N128

S6 0 S45

RS77

N10

N11

RS16

S34

N1 3

N1 6

S4

N26

S18

N6

N32

S4

N2

RS51

S43

S59

S20

N111

N2 8

S11

N25 N33

N119

N122

RS33

N87 N5 0

N30 N7 8

RS75

N36

N7 2

N29

W17

N8 0

N126

N29 N115

N114

RS66

N8 1 N121

RS88

N5 5 N5 6

N118

N31 N123

RS82

RS67

RS79

RS73

N34

N37

N16

N35

a. Final structure of ST b. Final structure of YO c. Final structure of KM Fig. 2. Schematic of the final concept maps created by three students

The interview also revealed ST’s motivation for creating the structure. When he was asked about the final goal of the course, he discussed these concept maps, referring to them as “an externalization of how I understood the materials. This reflects what is inside my head. If I were asked what I learned, I could come back to these notes and retrieve all the necessary details. I am confident about that.”

2.2.2 Insightful knowledge integration We are currently analyzing how the students’ utterances changed over the course, from the beginning of the term to the end, to clarify the process of how the knowledge was integrated by each student. We have audio records of each student’s conversations with other students during the class, as well as with TAs and the instructors. The transcripts are divided into utterance units and coded according to their contents; they were analyzed in terms of the development of the explanations they conveyed to other students, their questions and answers while discussing the material, and how each student attempted to unite the different reading materials covered in the jigsaw.

N. Miyake / Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation

265

These protocol analyses revealed the process for one of our target students, YO, who struggled to relate his core readings with others, and successfully achieved an insightful integration in the middle of the course. Insightful integration is a designation that indicates the conceiving of a non-conventional relation that would be agreed upon by professional researchers, even though it may not have occurred to them. YO’s core reading regarded the difference between procedural and declarative knowledge, which he felt was associated with the history of AI research. While he was attempting to explain to his partner, NT, how declarative knowledge was transferred to procedural knowledge through automatization, they came to mutually consider the question of why the transformation is one-way and not the reverse (that is, why the reading explained only the transformation from declarative to procedural knowledge, but not from procedural to declarative knowledge). This concern is expressed in their conversation on November 16, as follows. The line numbers indicate the utterance number of the day’s conversation. 16/Nov: While checking their notes, his partner voices a question about the text. 646 NT We have written that knowledge gets transformed from declarative to procedural. Is this final? 649-655 YO Yes, we finalized it…ah… maybe we can put it as a last comment. 666-675 NT We have talked… from declarative to procedural transformation rather extensively. But couldn’t it be reversed, like the other way around? Is this actually related to automatization? I don’t understand. 686-687 YO We don’t have good examples of automatization, yeah.

This conversation was resumed in two weeks and developed into YO’s insightful integration on utterance numbers 447 to 448 and again in 451 to 464. 30/Nov -1: After the pair exchanged their practice explanations, YO develops his own interpretation 441-445 YO I just thought… we might not use procedural knowledge intentionally, but to be conscious about it is, maybe, cognitive science… 446 TN Ah… 447-448 YO So, those human behaviors are difficult to explain, 449-450 KT …but we make them explainable in words. 451-464 YO Let me try again, cognitive science is a process of transforming procedural knowledge into declarative knowledge. How’s that?

Note that YO’s concept-mapping structure was not as thorough as that of ST. The goal they were aiming at could have been different, as revealed in YO’s comments during one of his informal interviews. He mentioned that it was important for him to understand the breadth of research in cognitive science so that he could determine the direction in which to expand his study. He was apparently trying to establish his own value for this science, which could have triggered the insight he gained during his class conversation.

2.2.3 Acquisition of learning skills The final index, also measured in the term papers, was the relationship between the implications they drew from what they had learned and their possible usage in everyday life. We call this measure “extendibility.” Our target student, KM, did not reach any insightful integration, nor did she exhibit a strong structured view of the concept mapping. Yet she scored among the highest on acquiring an important learning skill, that of asking questions. She wrote extensively in her term paper about the strategy of asking good questions. One strategy that she claims works for her is to start asking some apparent, easy question and have her partner answer that, to secure time to consider a more in-depth question. This works, she claims, because she knows the answer, so she does not have to concentrate on

266

N. Miyake / Multifaceted Outcome of Collaborative Learning: Call for Divergent Evaluation

listening to the response, and the partner’s reply often triggers a new question. She also mentioned in the interview how she had wanted to learn to ask questions, so she took some models from good question-askers among her classmates and practiced while she read the materials as part of her homework. Again, this is a clear achievement for her, which was possible because the class activities were heavily collaborative. These patterns, taken from detailed analyses of the logs of the computer tool, are the protocols of classroom conversations, careful readings of the term papers, and intensive interviews, none of which is regarded as an established method of evaluation. Nonetheless, some aspects of the learning outcome that are apparently important to the individual students become visible from this perspective. What I report here is only a fragment of the learning process, but one that plays a vital role in further development of the research field.

Acknowledgments This research is supported by SORST/JST and by Grants-in-Aid for Scientific Research, A, No. 15200020.

References [1] Hicky, D.T. et al., (2003) Integrating curriculum, instructions, assessment, and evaluation in a technology-supported genetics learning environment, American Educational Research Journal, 40, 495-538. [2] Clark, D., & Linn, M., (2003) Designing for knowledge integration, Journal of the Learning Sciences, 12, 451-493. [3] Scardamalia, M., & Bereiter, C., (1991) Higher levels of agency for children in knowledge building, Journal of the Learning Sciences, 1(1), 38-67. [4] Kolodner, J. (2002) Learning by design, Cognitive Studies, 9(3), 338-350.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

267

Technology-Based Tools That Facilitate Data-Driven Decision Making Ellen B Mandinach, Margaret Honey, Daniel Light, Juliette Heinze, Luz Rivas EDC Center for Children and Technology, USA [email protected]

Abstract. In this paper, we describe how technology-based tools support data-driven decision making in US schools. We examine how three different applications used in six different school districts either impede or facilitate decision making by educators across various levels of school districts. One of the major objectives of this work is to develop a theoretical model for data-driven decision making based on the outcomes of this research. Keywords: Data-driven decision making, technology applications, theoretical model.

Introduction Examining how technology-based tools can facilitate, support, and enable decision making, and how administrators and teachers use such tools and data to enhance instruction is essential if we are to understand how assessment data can be used effectively to inform educational decision making. This project brings together complimentary evaluation techniques, using systems thinking as the primary theoretical and methodological perspective, to examine the implementation and use of data-driven applications in school settings. The project has two goals: (a) to build a knowledge base about how schools use data and technology tools to make informed decisions about instruction and assessment; and (b) to develop an evaluation framework to examine the complexities of dynamic phenomena that will inform the field and serve as a knowledge building enterprise [1, 2]. Recently, the education community has become interested in data-driven instructional decision making, largely because growing numbers of school systems and states have the capacity to process and disseminate data in an efficient and timely manner [3, 4]. This trend has been further accelerated by the requirements of No Child Left Behind (NCLB) to use data to improve school performance [5]. Of the growing body of literature on the use of data systems, tools, and warehouses to support decision making processes in schools, research indicates that a host of complicated factors need to be addressed if these tools are to be used to support instructional improvement. There are a number of initiatives being implemented across the country for which research is in the formative stages [6 – 17]. Comprehensive reviews [18-20] of the tools are available, identifying some of the technical and usability issues districts face when selecting a data application to support instructional planning. Technical challenges include data storage, data entry, analysis, and presentation. Other challenges include the quality and interpretation of data, and the

268

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

relationship between data and instructional practices [21]. While debate about the merits of using state mandated testing data for diagnostic purposes continues, responding to accountability requirements remains a daily challenge that schools and districts must address now [22, 23]. Although high-stakes accountability mandates are not new, the NCLB legislation places public schools under intensified external scrutiny that has real consequences [24]. District and school administrators are struggling to respond to these heightened expectations, which by design call for different thinking about the potential of accountability data to inform improvements in teaching and learning. It is clear that NCLB is requiring schools to give new weight to accountability information and to develop intervention strategies that can target the children most in need. The growing interest in data-driven decision making tools is no doubt a direct response to these mounting pressures [18, 25]. 2. The Research The purpose of this work is to examine technology-based, data-driven decision making tools, their implementation, and impact on different levels of school systems. Examining different tools in diverse settings enables us to develop and validate an evaluation framework that will be sensitive to the dynamic and interacting factors that influence the structure and functioning of schools as complex systems [1, 2]. The research examines a methodological framework using systems thinking, and also presents a theoretical framework on how data-driven decision making occurs in school settings, and a structural framework that outlines the functionality of the tools that either facilitate or impede data-driven decision making. 2.1 The Technology-Based Tools The project focuses on three tools – a test reporting system, data warehouses, and diagnostic handheld assessments. The first application, the Grow Network uses a mix of print and web-based reporting systems. The print materials, called Grow Reports™, deliver well-designed, highly customized print reports to teachers, principals, and parents. Each report is grounded in local “standards of learning” (e.g., mathematical reasoning, number and numeration, operations, modeling/multiple representations, measurement, uncertainty, patterns and functions) that encourage teachers to act on the information they receive and to promote standards-based learning in their classrooms. When teachers view their Grow Reports on the web, these standards of learning link to “teaching tools” that not only help to explain the standards, but also are solidly grounded in cognitive and learning sciences research about effective math and literacy learning. Second, are two home grown data-warehouses, that enable administrators, teachers, and parents to gain access to a broad range of data. The systems store a diverse array of information on students enrolled in the districts public school systems including attendance information, the effectiveness of disciplinary measures, test and grade performance. This information is available to an increasingly larger set of stakeholders in a growing number of formats for use in various contexts. The data warehouse systems have accommodated new kinds of data, have created multiple mechanisms for making that data available in different formats, and are continuing to work with school-based users to further address their needs. The third application consists of handheld technologies to conduct ongoing diagnostic assessments of students’ mathematics learning and early literacy. In this system the teacher collects data on a handheld computer. Teachers upload their information from the handhelds to

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

269

a Web-based reporting system, where they can obtain richer details about each student. They can follow each student’s progress along a series of metrics, identify when extra support may be necessary, and compare each student’s performance to the entire class. The handhelds are: (a) built upon what we know from research about the key areas of mathematical knowledge and early literacy; (b) address real instructional challenges that teachers are facing and make the task of assessing student learning easy and practical to accomplish; and (c) tools to be applicable across multiple contexts and multiple curricula by addressing core learning challenges, not curriculum-specific skills and tasks. 2.2 The Research Sites and Data Collection Two sets of sites were used for each application. The sites for Year 1 were the original sites and the Year 2 sites were used for validating the initial findings. The New York City Public Schools and Chicago Public Schools served as the sites for the Grow Reports. The Broward County Public Schools in Florida and Tucson Unified School District in Arizona served as the sites for the data warehouses. Albuquerque, NM and Mamaroneck, NY served as the sites for the handheld diagnostics. Three of these sites represented the first, third, and fifth largest school districts in the United States. Research was conducted through interviews with administrators and through interviews and focus groups with teachers and students. Surveys also were given to teachers and administrators. Analyses are continuing as staff is using data to construct systems-based models of the interrelationships among important variables that influence the implementation of the tools and data-driven decision making in each of the sites. Data also are being analyzed in terms of the construction and validation of the theoretical framework for data-driven decision making and the structural functionality framework for the tools. 3. Results 3.1 The Development of Three Initial Frameworks The project is developing three frameworks: a methodological framework based on systems thinking; a conceptual framework for focused inquiry and exploration of data based on both theory and practice; and a structural functionality framework for the data-driven decision making technology-based applications. The methodological framework is founded on three principles. First, there is the need to recognize the dynamic nature of school systems in order to capture their complexity. Second, the methodology must account for the interconnections among the many variables that impact a school system. Third, the methodology also must account for the different levels of stakeholders within a school system. The goal, by the end of the project will be to have a systems model of each of the six sites, taking into account the dynamic nature of school, the interconnectedness among important factors, and the multiple levels at which schools must function. It is our hope that from these models, we will be able to draw parallels to other districts with similar characteristics and contexts, and providing a level of generalizability from the data. The conceptual framework approaches data-driven decision making as a continuum from data to information to knowledge. Figure 1 depicts the model that reflects our current theoretical thinking. Key variables include collecting, organizing, analyzing, summarizing, synthesizing, and prioritizing. These variables are manifested differently, based on who the

270

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

decision makers are and where in the school structure they are situated. The types of questions to be addressed are influenced not only by the location within the school hierarchy (i.e., class, school, district), but where along the data- information-knowledge continuum the focused inquiry falls. This conceptual framework further posits a continuum of cognitive complexity in data a decision making begins with data, transforms those data into information, and then ultimately into actionable knowledge. The data skills are collecting and organizing. The information skills are analyzing and summarizing, and the knowledge skills are synthesizing and prioritizing. Decision makers probably will not engage these skills in a linear, step-by-step manner. Instead, there will be iterations through the steps, depending on the context, the decision, the outcomes, and the interpretations of the outcomes. The structural functionality framework identifies six characteristics of tools that influence how they are used and by whom. The first is accessibility which is how accessible are the tools, and how do the tools support access to the data or information. The second is the length of the feedback loop. Feedback focuses on how much time passes between when the data are generated and when results are reported to the end-user. The concern is that the data are still relevant by the time they are reported. The third, comprehensibility, deals with: how understandable the functioning of the tool is; the clarity of presentation; and how easy it is to make reasonable inferences from the information presented. Flexibility is the fourth component focusing on whether there are multiple ways to use the tool and the extent to which the tool allows for manipulation of data. Alignment is the fifth functionality focusing on the extent to which the data align with the classroom, the standards, and to the curriculum. The final component is the link to instruction. It focuses on how the tool bridges information (either physically or conceptually) and practice. Figure 1. Theoretical Framework for Data-Driven Decision Making

3.2 Observations from the Sites We will summarize data that fall into two overarching topics. First, we will describe findings that relate to school issues. These include such factors as accountability and assessment,

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

271

professional development and training, leadership, and data use. The second focus is on the affordances of the technology. We examine how the six characteristics that form the structural functionality framework impact the use of data. School Issues. Accountability pressures by and large are one of the most important factors influencing the use of data and the tools. In the United States, there is increasing pressure at the local, state, and federal levels for schools to achieve performance mandates, as assessed by high-stakes tests. The more tests, the more pressures that are felt by the practitioners, and therefore the need to use data to make informed decisions about instructional practice that my lead to improving achievement, especially given the punitive consequences associated with failure. Because of this increase in testing, schools are faced with an explosion of data. The data need to be mined in different ways, and in particular, must be disaggregated. Simply put, there is so much data that educators are forced to use technological applications to deal with the wealth of data. As many educators say, they are data rich, but information poor. By this they mean that there is far too much information with which they must deal, but those data are not easily translatable into information and actionable knowledge. One goal of using the tools is to facilitate the mining of data from multiple perspectives that ultimately will provide the user with information from which they can make decisions. A byproduct of the increase in testing is what happens in the classroom in terms of time allocation. As more testing occurs, teachers are forced to devote less time to instruction. Teachers report that they must teach to the tests, and in doing so many important topics get omitted. Teachers feel that they are not teaching as much as they are doing test preparation. Teachers also feel that their typical classroom practices are being changed by these pressures. Teachers know their students and tend to use multiple assessment strategies, quantitative and qualitative, summative and formative, to measure student progress. These strategies translate into a wealth of classroom data that needs to be collected and analyzed. Thus the applications play a critical role in helping educators to manage and examine the plethora of data. Many teachers feel frustrated by the accountability pressures. Many see the use of data to make informed decisions a necessary survival strategy. Thus the applications by which data can be mined are key tools. Other teachers, however, are taking a more fatalistic approach. They feel that the pressures are just another passing fad and will fade in time, using a strategy to continue practice as usual. While yet another group of teachers are luddites who feel threatened by technology and balk at mining the data, entrusting that task to someone else in their school. Some teachers’ reluctance to use data and the tools is grounded in a lack of training or a mistrust of data. Two kinds of training are salient here. First, there is a need for training on the use and understanding of data. Second, there is the need for appropriate and timely training on the tools. Teachers rarely receive preservice or inservice training. There are relatively few courses offered in teacher training institutions on data, and only recently have such inservice workshops begun to emerge. While teachers need to understand data, they also need to know how to use the technology that makes data mining possible. Again, only recently have professional development opportunities become available. Leadership is the last major of the major school issues. Leadership makes a difference in terms of the message administrators communicate to their staff. In our experience, building leadership appears to be more important in facilitating or impeding the use of data and the tools. Although superintendents set the tone for a district’s philosophy, principles have more direct contact with the faculty and therefore more influence on what they do. A principal who is data-driven or technically savvy can exert substantial influence on the faculty, communicating

272

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

the importance and thereby stimulating use. In contrast, principals who are luddites communicate that technology and data are not important. They may not be impediments but they certainly do not urge their teachers to make use of the data and technology. 3.3 Affordances of Technology As mentioned above, we have identified six functions of the tools that contribute to their use. These characteristics play out differently across our three applications as well as other tools. It is clear that the more easily accessible, the more likely the tools will be used. The handhelds are easily accessible, even with minimal training. Teachers access data on the devices and almost as easily online with the downloaded data that allow for deeper data mining. In contrast, the interface of the data warehouses are much more difficult to negotiate and therefore far fewer teachers make effective use of the tool. Had the interfaces been more user-friendly, it is clear that many more practitioners would take advantage of the wealth of data that resides on the warehouses. The feedback loop is perhaps one of the biggest motivators for or impediments to use. The functionality involves both the form of assessment or data and the tool. The Grow Reports are seen as static data with less utility because of the five-month delay between testing to the delivery of the data. In contrast, the handhelds provide immediate data to teachers from which they can make informed instructional decisions. The warehouses are somewhere in between, depending on the type of data entered and mined, as well as who is accessing the data (the end user or the data inquiry specialist). The tighter the feedback loop, the more immediately useful that data appear to be. Comprehensibility deals with the understandability of the information. The more understandable, the more likely the tool will be used. Parts of the Grow Reports are highly comprehensible, while other parts are open to misinterpretation and ambiguity even by trained specialists. The handheld’s data are fairly easy to understand. Flexibility refers to the extent to which the tool can be used in multiple ways to examine data. The more flexibility, the more useful the tool will be. However, the more options a user has, the more opportunity for confusion. Looking at data in a variety of ways generally will help the user to understand more deeply the meaning of the information. This includes having a variety of visual displays such as tables, graphs, and charts, and presenting data at different levels of aggregation. Take for example the two data warehouses. One warehouse presents data at the individual student level. If an inquiry is made at the level of the class, a special data run must be made to aggregate the data at the class level. The other warehouse flexibly moves across different levels of aggregation and units of analysis – student, class, teacher, school, and district levels. Alignment refers to how well the data can be matched to standards, instructional goals, and classroom practices. The Grow Reports are customized to state standards and display student and class performance categorized into quartiles. Teachers can go to the Grow website to obtain instructional resources that may help to remediate particular performance deficits. In a similar manner, the sixth component, link to instruction, is manifested differently in the tools. The Grow Reports’ indication of how the class performed in relation to the standards and the online resources are intended to help teachers plan instruction. Perhaps the most aligned to instruction are the handhelds. The diagnostic assessments administered via the handhelds are intended to be translated immediately into instructional remediation, thereby blurring the distinction between assessment and instruction. The handhelds can truly be a powerful instructional tool, not just an assessment device. The data warehouses have less direct links to instruction, requiring the teachers to develop links to classroom practice based on the data they have accessed.

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

273

Summary Data collected from the first and second year of the project continue to be analyzed in relation to the three frameworks. Once the data have been fully analyzed, our goal is the produce systems models for each of the six participating districts, identifying the key factors that relate to the infusion of data-driven decision making and how the particular applications influence decision making across the various levels of the school system. These models, will inform the refinement of the three frameworks. The ultimate objectives from the development of these three frameworks is to provide a means by which to generalize our findings so that other districts’ forays into data-driven decision making can be examined and better understood. The methodological framework will enable us to examine more systemically the use of the technology-based applications as they are infused into the organizational complexities of school districts. The conceptual framework will help us to understand the decision making process and the cognitive processes by which individuals make educational and instructional decisions. The structural framework will enable us to examine the applications more specifically, focusing on particular characteristics that may facilitate or impede use and integration. These data only touch the surface of how three technology-based tools impact how educators use data to make informed decisions. It is clear that the characteristics of the tools, as well as a host of school variables affect use. The three frameworks posited here are works in progress based on nearly two years of intensive data collection in six school districts. These data will help us to further refine the theoretical model as well as the structural functionality framework. Our goal is to take our data and ultimately transform them to information and knowledge, just as we have posited in our theoretical framework.

Acknowledgments We thank to acknowledge the National Science Foundation for supporting this research.

References [1] Mandinach, E. B. (2005). The development of effective evaluation methods for e-learning: A concept paper and action plan. Teachers College Record, 107, 8, 1814-1835. [2] Mandinach, E. B., & Cline, H. F. (1994). Classroom dynamics: Implementing a technology-based learning environment. Hillsdale, NJ: Lawrence Erlbaum Associates. [3] Ackley, D. (2001). Data analysis demystified. Leadership, 31, 2, 28-29, 37-38. [4] Thorn, C. (2002). Data use in the classroom: The challenges of implementing data-based decision-making at the school level. Madison, WI: University of Wisconsin, Wisconsin Center for Education Research. [5] Hamilton, L. S., Stecher, B. M., & Klein, S. P. (2002). Making sense of test-based accountability in education. Santa Monica, CA: Rand. [6] Mitchell, D., Lee, J., & Herman, J. (October, 2000). Computer software systems and using data to support school reform. Paper prepared for Wingspread Meeting, Technology's Role in Urban School Reform: Achieving Equity and Quality. Sponsored by the Joyce and Johnston Foundations. New York: EDC Center for Children and Technology [7] Spielvogel, R., Brunner, C., Pasnik, S., Keane, J. T., Friedman, W., Jeffers, L., John, K., & Hermos, J. (2001). IBM’s reinventing education grant partnership initiative: Individual site reports. New York: EDC/Center for Children and Technology.

274

E.B. Mandinach et al. / Technology-Based Tools that Facilitate Data-Driven Decision Making

[8] Brunner, C., Fasca, C., Heinze, J., Honey, M., Light, D., Mandinach, E., & Wexler, D. (2005). Linking data and learning – The Grow Network study. Journal of Education for Students Placed at Risk, 10, 3, 241-267. [9] Honey, M. (2001). The consortium for technology in the preparation of teachers: Exploring the potential of handheld technology for preservice education. New York: EDC/Center for Children and Technology. [10] Honey, M., Brunner, C., Light, D., Kim, C., McDermott, M., Heinze, C., Breiter, A., & Mandinach, E. (2002). Linking data and learning: The Grow Network study. New York: EDC/Center for Children and Technology. [11] Heistad, D., & Spicuzza, R. (2003, April). Beyond zip code analyses: What good measurement has to offer and how it can enhance the instructional delivery to all students. Paper presented at the AERA Conference, Chicago. [12]Murnane, R. J., Sharkey, N. S., & Boudett, K. P. (2005). Using student-assessment results to improve instruction: Lessons from a workshop. Journal of Education for Students Placed at Risk, 10, 3, 269-280. [13] Mason, S. (2002). Turning data into knowledge: Lessons from six Milwaukee public schools (WP 2002-3). Madison: University of Wisconsin, Wisconsin Center for Education Research. [14]Chen, E., Hermitage, M., & Lee, J. (2005). Identifying and monitoring students’ learning needs with technology. Journal of Education for Students Placed at Risk, 10, 3, 309-332. [15] Lachat, M. A., & Smith, S. (2005). Practices that support data use in urban high schools. Journal of Education for Students Placed at Risk, 10, 3, 333-349. [16] Streifer, P. A., & Schumann, J. A. (2005). Using data mining to identify actionable information: Breaking new ground in data-driven decision making. Journal of Education for Students Placed at Risk, 10, 3, 281-293. [17]Wayman, J. C. (2005). Involving teachers in data-driven decision making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed at Risk, 10, 3, 295-308. [18] Stringfield, S., Wayman, J. C., & Yakimowski-Srebnick, M. E. (2005). Scaling up data use in classrooms, schools, and districts. In C. Dede, J. P. Honan, & l. C. Peters (Eds.), Scaling up success: Lessons learned from technology-based educational improvement (pp. 133-152). San Francisco: Jossey-Bass. [19] Wayman, J. C., Stringfield, S., & Yakimowski, M. (2004). Software enabling school improvement through analysis of student data (Report No. 67). Baltimore: Johns Hopkins University, Center for Research on the Education of Students Placed at Risk. Retrieved January 30, 2004, from www.csos.jhu.edu/crespar/techReports/Report67.pdf [20] Sarmiento, J. (n.d.). Technology tools for analysis of achievement data: An introductory guide for educational leaders. Retrieved March 7, 2004, from www.temple.edu/lss/. [21] Cromey, A. D. (2001, Spring). Data retreats: A conduit for change in schools. Using data for educational decision-making. Newsletter of the Comprehensive Center-Region VI, 6, 21-23. [22] Choppin, J. (2002, April). Data use in practice: Examples from the school level. Paper presented at the annual meeting of the American Education Research Association, New Orleans. [23]Pellegrino, J.W., Chudowsky, N., & Glaser, R. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press. [24] Fullan, M. (2000). The three stories of education reform. Phi Delta Kappan 81, 8: 581-584. [25] Hayes, J. (2004, October). Ed tech trends: A ‘first look’ at QED’s 10th annual technology purchasing forecast 2004-05. Paper presented at the NSBA’s T+L2 Conference, Denver, CO.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

275

CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD Yongwu Miao Institute of Computer Science and Interactive Systems, University of Duisburg-Essen, Germany [email protected] Abstract. IMS LD is a meta-language that can be used to specify a pedagogical method as a learning design. A learning design is a formal process model, encoded in XML, which can be played by any IMS LD-aware-player to learners online. In order to make it easy to edit learning designs, LD authoring tools are absolutely needed. Through a review of currently available LD authoring tools, we see that the development of LD authoring tools is still at an immature stage. This paper describes a LD authoring tool named CoSMoS, which can support editing learning designs at Levels A, B, and C. In this paper the main design ideas are presented and three design issues are discussed. Keywords: educational process modelling, IMS LD, learning design, authoring tool

Introduction In the development of learning technology standards, the release of IMS LD (stands for IMS Learning Design specification [7]) signals an exciting paradigm shift from defining, sharing, and reusing learning objects to designing, sharing, and reusing pedagogical strategies. IMS LD is based on the Educational Modelling language (EML) developed by the Dutch Open University OUNL [9]. IMS LD provides a framework for teachers to express the pedagogical meaning and reflect in a deeper and more creative way about how they design and structure activities. IMS LD can be used to fully describe a teaching-learning process in a unit of learning (UOL), including references to the digital and non-digital learning objects and services needed during the process. It enables to describe personalization aspects within a learning design so that not only content but also activities within a UOL can be adapted based on the personal characteristics and situational circumstances. The specification makes it possible that the learning designs been proved as effective are communicated and shared between teachers or archived for reuse on future occasions [10, 11]. IMS LD is a meta-language that is divided into three parts, known as Level A, Level B, and Level C [7]. Separate XML schemas are provided for each level, with Levels B and C each integrating and extending the previous level. By using IMS LD a teaching-learning process can be formalized as a computational model in XML format, which is a platform-independent web-standard notation for describing arbitrary structured data. This means that a learning design, encoded in XML, can be read and run by any IMS-LD-aware player. The problem is that authoring a learning design in XML is a time-consuming and error-prone task. There is an absolute need to research appropriate user interface models to allow easy creation of learning designs. Since 2003 many efforts have been made to empower learning designers and others to describe and share their teaching-learning designs. Recently several LD authoring tools have been developed. Compared to common XML editors, these LD authoring tools provide

276

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

user-friendly interfaces for learning designers to create, reuse, and customise UOLs without the need of editing any XML code. However, most existing authoring tools can only support editing UOLs at Level A. Because the elements added at Level B are something like variables, operators, and control flow statements, how to facilitate learning designers to handle such technical complexity is a challenge for the developers of authoring tools. This paper describes our work that can be regarded as the first attempt to take the challenge to facilitate editing learning designs at Level B.

1. Three Levels in IMS LD As mentioned above, IMS LD is divided into three levels: LD Level A contains the bulk of the IMS LD constructs, including roles, activities, environments, learning objects, services, plays, acts, etc. The main added value of Level A learning design to the domain of e-learning is that it defines activities and roles as reusable components that can be designed into a learning flow using the method element. IMS LD uses a metaphor of a theatrical play to model learning-teaching processes. A method can have one or more plays. Each play consists of a sequence of acts and each act contains a set of role-parts. Each role-part assigns an activity to a role. The specification also allows learning objects (e.g., web-content) and services (e.g., index-searching and conferencing) to be specified at design-time as placeholders within the learning design that will be instantiated by the run-time system. LD Level B adds properties and conditions to level A. Properties enable information about learners, roles and the state of the learning design itself to be maintained. The addition of properties is important for recording outcomes of participants and supporting personalization. Conditions enable learning designers to define rules that govern the behavior of the unit of learning as a whole what gets presented to individual participants. LD Level C adds notification to level B. Notifications provide a greater level of interactivity and control over a live learning design, as a form of event-driven messaging system within a LD player. A notification can be triggered by an activity completing or by a rule through the global elements and can also trigger a notification to be sent to a design element or human participants. A notification can be used to make a new activity available for a role to perform or to change the value of a property.

2. A Brief Review of Authoring Tools Related to IMS LD This section provides a brief review of currently available LD authoring tools according to the forms of representing learning designs in the tools. Existing XML editors and authoring tools provided by Learning Management Systems (LMSs) are not taken into account, because they are not LD-specific authoring tools. However, two non-LD authoring tools will be investigated, because these tools are e-learning-specific tools and built around a model of ‘activity’ and ‘learning flow’, rather than around ‘learning object’ and ‘content’. 2.1 Tree-based Authoring Tools A tree-based authoring tool presents the elements of LD as a tree structure. An interface is provided for users to navigate through the tree and to enter values for the elements. LD authoring tools adopting a tree-based approach are ALFANET, CopperAuthor, and RELOAD.

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

277

ALFANET (stands for Active Learning for Adaptive Internet), an EU project aims to develop new methods and services for active and adaptive learning [1]. In ALFANET, IMS LD is used to represent learning scenarios for supporting personalization. The menu buttons in the tool are organized as a hierarchical structure according to IMS LD element structure. It provides basic support for authors to fill the forms for defining elements at Level A. CopperAuthor is a tree-table-based authoring tool. It allows learning designers to construct and navigate the structure of the edited learning design. The generic information of each element can be entered in the cells of the table. This tool provides a primitive and direct manner to edit a learning design at Level A. RELOAD (stands for Reusable E-Learning Object Authoring & Delivery [18]) is a project, funded by JISC, UK. RELOAD LD editor consists of several edit pages and each page supports editing a type of element such as role, activity, environment, method, etc. An element tree on each page enables to navigate through all elements of a certain type. If selecting an element, the tool presents a user-friendly interface for authoring the element in a series of panels. The RELOAD LD editor is regarded as a reference implementation of LD authoring tool. In June the newest version of RELOAD LD editor was released, which can support learning design at Levels B and C as well. However, it supports editing conditions in a way that the representation structure of expressions and actions exactly reflects the element structure of IMS LD specification, rather than the conventions of viewing and writing expressions. In addition, at any point of time users can only view and edit sole operand. It is not convenient for users to perceive and define conditions. 2.2 Diagram-based Authoring Tools Diagram-based authoring tools represent IMS LD elements and their relations as graphical objects such as nodes and arrows. A tool in this category provides a graphical language for users to edit a learning design as diagrams. MOT+ system [17], developed by the CIRTA Research Centre, Tele-university, Montreal, is an interesting example of diagram-based authoring tools, which enables users to navigate in and construct hypertext documents that make up a learning design. A learning design represented as hypertext documents can be exported as IMS LD XML files. Now the developers of MOT+ are working on LD Level B and C editing functions. Another existing diagram-based authoring tool is ASK-LDT (Advanced e-Services for the Knowledge Society Research Unit [2, 8]), developed by the department of technology education and digital systems, University of Piraeus, Greece. The ASK-LDT editor provides a graphical language. The components of the graphical language include several types of nodes representing several predefined activity types (e.g., lesson, discussion, assessment and so on) and two types of arrows representing preceding and rollup relations between activity nodes. A learning design can be modeled as a diagram representing a learning flow by using the graphical notations. The resulting diagram can be translated into a XML representation automatically by using IMS LD Level B in a way to create a property with a “boolean” type for each activity and creating a simple condition clause for each arrow. Therefore, users can define a learning design with features of LD Level B without need to define properties and conditions. However, this tool offers a relatively easy path into LD authoring and can not really support learning designers to edit UOLs at Levels B and C. 2.3 Two IMS LD Relevant Tools Two non-LD authoring tools reviewed in this section are LAMS and COMPILE. Both are diagram-based authoring tools that are intended for normal teachers. Although these tools

278

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

do not implement the IMS-LD specification, they embody the core ideas behind the specification in terms of a focus on creating sequences of activities, rather than learning objects. An additional common feature of these two tools is to be integrated with run-time environments. LAMS (stands for Learning Activity Management System [12]) was developed by MacQuarie University in Sydney and WebMCQ Ltd. It provides a simple and highly intuitive user interface that allows the course designer to drag and drop typed activity nodes into the edit area and use connecting arrows to organize the activities into a sequential workflow [4]. LAMS provides a variety of software tools that are intended to support the creation and management of specific learning activities such as synchronous discussion (e.g., Chat), concept-mapping (e.g., Mind-Map) or role-play scenarios (e.g., Kartouche). The activity types are fixed, but these cover many basic activities carried out in the classroom. This use of familiar elements makes the application easy for teachers to comprehend, as this is the way that conventional lessons are planned. COMPILE (stands for Cooperative Open Multimedia-based Process-centered Integrated Learning Environment) was developed at Muenster University, Germany. It was developed by using a content-based and process-centered approach [13]. The aim of the system is supporting learning projects (formal collaborative learning processes [14]) and learning programs (informal collaborative learning processes [15]). The system provides two graphical languages for teachers to edit teaching-learning processes by using nodes such as courses, lessons, instruction activities, test activities, projects, project activities, and artifacts. Notations such as temporal-arrow, artifact-arrow, start, end, branch, fork, and join are used to connect elements as a learning flow and an artifact flow. The authoring tool enables to define properties and to define a transition condition for each branch. Because this tool is intended for teachers, the properties and conditions defined in COMPILE are much simpler than those in IMS LD.

3. Design and Implementation of CoSMoS This section briefly presents the design and implementation of a LD authoring tool and gives a quick overview of its particular features and functionality. CoSMoS (stands for Collaboration Script Modelling System) was originally designed and implemented for supporting formalization of collaborative learning processes [16]. The modified version is compatible with the IMS LD specification. This tool is developed for learning designers, who may be pedagogic experts, course planners, and instructional designers with knowledge about IMS LD. Therefore, the tool is designed in a way adhering rigidly to IMS LD. As shown in Fig. 1, the user interface of the tool consists of three components. At the top of the window, the menu bar and the toolbar of the tool list the main functions. The left pane shows learning designs in a hierarchy structure. It is allowed to edit multiple learning designs at the same time. Such a design can help to reuse definitions of elements across learning designs and makes it easy to specify the relations between multiple learning designs. The structure of a learning design is hierarchically ordered as the same as the structure of major elements of IMS LD. Such a structure can help learning designers to navigate in the element tree easily. Users can create new elements by adding leaves of the tree. The right pane is an edit area showing forms for editing elements. If a user selects an editable element node such as a role, a learning activity, an environment, an act, and so on, a corresponding edit form will be shown in the edit area. Users can enter values in editable input fields. If a value is a reference to an existing element, users can drag the tree node representing the element and drop it into the input field. Such a reference can be removed by

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

279

dragging it and dropping into the garbage icon in the tool bar. Using drag&drop to do constructive and destructive work intuitively is an important feature of the tool. Fig. 1: The User Interface of CoSMoS

Fig. 2 shows the form for editing a condition clause. The schema used to define Level B elements can be seen in the left pane. The form consists of three edit panels for editing conditional expressions, then actions, and else actions. Our approach is to provide a user-friendly representation of expression and to allow passing parameters by using drag&drop. When defining an expression, a learning designer can drag an operator from the schema tree (e.g., <, not, and etc.) and drop it in a blank field in the ‘If’ panel. The blank will be replaced by the dropped expression. For instance, if selecting an ‘and’ symbol, a dialog window will pop up to ask how many operands will be defined in this ‘and’ expression. If the answer is ‘3’, three blanks connected by two ‘and’ operators will appear. A blank field can be further specified by dragging an operator and dropping in the blank. For example, if a ‘complete’ symbol is dropped in the first blank. The blank will be replaced by ‘[an undefined process] is completed’. Dragging a process element (e.g., a learning activity titled “Enter your initial thoughts”) and dropping in this blank, the title of the learning activity will be shown in the blank (see Fig. 2). It is possible to define an expression by using a property. Fig. 2 also shows such an example property titled “What I think greatness is”. By using drag&drop, learning designers can define actions as well. The Fig. 2 shows examples of ‘then’ and ‘else’ actions: showing and hiding classes and sending a notification, an element at Level C. An ‘else’ has three options: no action, else actions, and else if condition. Note that the tool can support recursively defining ‘else if’ conditions. In order to aid learning designers to create UOLs effectively, the tool can validate the definition of any LD element at design-time and show warning messages and error messages of the definition. The value of the status of an element will be assigned by the tool

280

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

automatically according to the validation result. The tool can show the XML representation of any element in the edit area on demand. Experts can modify the XML representation directly in the edit area, which can be imported to the tool. The tool supports a deeper copy (rather than a reference copy) and paste in the same learning desgin and even in other learning designs. These functions help to reuse the definitions of elements easily. The tool is implemented by using pure JAVA. A defined UOL can be exported as an imsmanifest.xml file, which can be read and played by an IMS LD-aware player such as the CopperCore engine [3]. Fig. 2: Defining a Condition Clause

4. Discussions CoSMoS has been demonstrated and used in an UNFOLD (stands for Understanding New Frameworks of Learning Design [18]) workshop in April by participants including learning designers, experts, and the contributors of IMS LD. UNFOLD is an EU project aiming at promoting and encouraging the development and use of e-learning systems related to IMS LD. In the workshop, most comments were positive and some were valuable suggestions. Based on the comments and experiences, we can discuss some design issues, which should be taken into account when designing a LD authoring tool. Tree- and diagram-based tools: A tree-based authoring tool provides an overview of all major elements of a learning design. Such a tool makes it convenient to navigate in the hierarchical structure and operate on any element. The main disadvantage of the tree-based approach is that the relations between the elements can not be explicitly represented in a tree except for component relations. However, the diagram-based approach has strengths to represent relations between elements, and to construct and modify the relations within a diagram. The most attractive feature of diagram-based tools is that the visualized notations can help users to understand and represent learning designs intuitively. If a learning flow is drawn as an individual diagram by the tool (e.g., LAMS and ASK-LDT), the diagram

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

281

provides an overview as well. However, if the number of graphical objects increases, the limitation of the space and the increased complexity of the diagram will be a serious problem. A layer-diagram approach will be a natural choice. For examples, MOT+ has two layers and COMPILE can have arbitrary layers. Users have to navigate in a set of hypertext documents. It may lead to a classical problem ‘lost of way’ in hypertext documents. An additional difficulty is representing a relation between two nodes located in different diagrams. Finally, the diagram-based tool has a problem to load and present a learning design created by using other tools, because the graphical information is missing. Higher- and lower-level tools: According to [11], the judgment of higher- or lower-level tools is based on how much knowledge about the LD the users of the tools should possess. LD experts would like to use a tool that enables them to access the parts of a LD document without a need to enter repetitive text and to have their document checked for integrity. Learning designers need tools that enable them to represent pedagogic methods by defining roles, resources, and the flow of activities together with the various branching conditions. Normal teachers, who may have no knowledge about LD, need to be able to represent their pedagogic experience and instructional plans by using their conventional vocabularies such as course, lesson, presentation, discussion, voting, test, exercise, homework, and so on. It is a challenge to develop authoring tools for normal teachers. These tools should be based on a higher-level learning process modelling language and can translate the learning designs represented by the higher-level language into the IMS LD. For example, if a teacher creates a ‘voting’ activity in a learning design, it is not necessary to create a detail specification about roles, learning activities, environments, voting tools, properties, conditions, and so on. The teacher just need to select or assign a value or use a default value for each attribute such as the type of voting, voting issue, voting items, voting policy (e.g. majority agree or common agree) and so on. The authoring tool can automatically interpret such a higher-level description into a formal specification of a ‘voting’ activity in IMS LD. It is expected that a spectrum of authoring tools can be developed from lower-level to higher-level for different kinds of users. CoSMoS is a lowerand middle-level authoring tool. ASK-LDT, COMPILE, and LAMS can be regarded as, to some extent, higher-level tools. Recently, some attempts are tried to develop higher-level languages and tools [6, 7]. It is important to note that there is a widespread misunderstanding that diagram-based tools are somehow higher-level tools. For example, MOT+ has been categorized as a higher-level tool in [11]. Whether a tool is higher-level depends on whether the tool provides a set of higher-level notations. MOT+ is not a higher-level tool because the vocabularies used by the MOT+ LD editor are exactly those of IMS LD. Stand-alone and integrated tools: Most existing LD authoring tools are stand-alone and single-user tools. Perhaps it is because the development of the tools is still in the initial phase. Authoring a UOL is usually not an independent and individual work. An integrated and shared authoring environment is expected. Users with different roles in an organization or in a virtual community can cooperate to construct and share learning objects and to create and share learning designs. Furthermore, it should be encouraged to go one step more – build an integrated authoring, publication and delivery environment.

5. Conclusions and Future Work This paper briefly reviewed authoring tools related to IMS LD. The conclusion drawn from the review is that the development of LD authoring tools is still at an immature stage although several LD authoring tools have been developed. In particular, most LD authoring tools can only support editing learning designs at Level A. This paper described the

282

Y. Miao / CoSMoS: Facilitating Learning Designers to Author Units of Learning Using IMS LD

development of a LD authoring tool named CoSMoS, which can support editing learning designs at Level B and C. Recently RELOAD LD editor can support editing learning designs at Level B and C as well. Comparison with the RELOAD LD editor, CoSMoS supports editing learning designs at Level B in a way that is more close to the convention of viewing and writing condition clause. Our approach to allow easy creation of learning designs can be characterized by providing a user-friendly representation of condition clauses and by using drag&drop to do constructive and destructive editing work intuitively. The paper discussed three design issues. This discussion will be helpful for developers of IMS LD authoring tools to make decisions when designing their own authoring tools. The discussion has revealed that there are three more challenges related to LD authoring tools in the future: to develop a higher-level modelling language for describing teaching-learning processes, to develop a diagram-based authoring tool based on the higher-level modelling language, and to develop an integrated environment for authoring, publication, and delivery of learning designs.

References [1] [2] [3] [4]

ALFANET home page. Retrieved from http://alfanet.ia.uned.es/ ASK-LDT home page, Retrieved from http://www.iti.gr/db.php/en/pages/ASK.html. CopperCore home page, Retrieved from http://coppercore.org/ Dalziel, J.R. (2003). Implementation Learning Design: The Learning Activity Management System (LAMS). In: Proceedings of 20th Annual Conference of Australasian Society for Computers in Learning in Tertiary Education. Adelaide, 7-10 December 2003. [5] Goodyear, P. (2003) Working towards an e-learning design pattern language. Learning Technology, 5(4). [6] Hernandez, D., Asensio, J.I., Dimitriadis, Y. (2004). IMS Learning Design Support for the Formalisation of Collaborative Learning Flow Patterns. In: Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies, Aug.30 - Sep. 1, 2004, Joensuu, Finland, IEEE Press, pp.350-354. [7] IMS Learning Design Best Practice and Implementation Guide; IMS Learning Design Information Model; IMS Learning Design XML Binding. Retrieved from http://www.imsglobal.org/learningdesign/index.cfm [8] Karampiperis, P., and Sampson, D. (2004) A Flexible Authoring Tool Supporting Adaptive Learning Activities. In Proceedings of IADIS International Conference on Cognition and Exploratory Learning in Digital Age, Lisbon, Portugal. [9] Koper, E.J.R., “Modelling Units of Study from a Pedagogical Perspective: the Pedagogical Meta-model behind EML”, Educational Technology Expertise Centre Open University of the Netherlands. Retrieved from http://hdl.handle.net/1820/36 [10] Koper, E.J.R. & Olivier. B. (2004). Representing the Learning Design of Units of Learning. Educational Technology & Society. 7(3), pp.97-111. [11] Koper, E.J.R. & Tattersall, C. (Eds), (2005). Learning Design: A Handbook on Modelling and Delivering Networked Education and Training, Springer, Berlin. [12] LAMS home page. Retrieved from http://www.lamsfoundation.org/ [13] Miao, Y., Piontkowski, U., Keil, W., Becker, L., Laus, F. O. & Heckelmann, M. (2002). A Content-based and Process-centered Approach to the Design of an Integrated Cooperative Learning Environment. In: Proceedings of the World Congress Networked Learning in a Global Environment, Naiso. Academic Press. [14] Miao, Y., Piontkowski, U., Keil, W., Becker, L., Laus, F. O. & Heckelmann, M. (2002). Establishing an Integrated Web-based Collaborative Learning and Collaborative Work Environment for Student Working Groups. In: Proceedings of the 4th International Conference on New Educational Environments. Lugano. [15] Miao, Y., Piontkowski, U., Keil, K., Schmermund, A., Becker, L., Laus, F., and Breimann, C. (2003). Informal Cooperative Learning Program: Using Technologies to Bridge Theory and Practice. In: Proceedings of the 11th International Conference on Computers in Education, pp. 351-359, Hong Kong. [16] Miao, Y., Hoeksema, K., Hoppe, U. Harrer, A. (2005). CSCL Scripts: Modelling Features and Potential Use. In: Proceedings of Computer Supported Collaborative Learning Conference (CSCL2005), Taiwan, June 2005. [17] MOT+ home page. Retrieved from http://www.cogigraph.com:90/cogigraph/article.php3?id_article=55 [18] RELOAD home page. Retrieved from http://www.reload.ac.uk [19] UNFOLD home page. Retrieved from http://www.unfold-project.net:8085/UNFOLD/

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

283

Development of an e-Learning Resource for Teacher Education Utilizing a VOD-compatible Teaching Portfolio Having a Dual-screen Synchronous Playback Function Hitoshi MIYATA Faculty of Education, Shiga University, Japan [email protected] Abstract. We have developed a teaching improvement support system utilizing a web-based teaching portfolio provided with a VOD-compatible animated video clip function. This system has been used by active teachers since last year. This teaching portfolio has recently been given a dual-screen synchronous playback function for video clips of classes, and has been distributed as an e-learning resource to enable students to observe classes prior to practical teacher training. As a result, in comparison with video clips on a single screen, this dual-screen synchronous playback has been suggested as being effective as an instructional resource since there are differences in the manner in which students perceive the three points consisting of the sense of presence of classroom observation, understanding the reactions of students in response to questions from the instructor, and understanding the preferences and interests of students. Keywords: e-Learning , Teaching Portfolio, Video-on-demand, Teacher Education

Introduction Observation training is conducted as a part of preliminary guidance prior to teacher training. This observation training consists of going out to affiliated schools once or twice a year to observe classes conducted by teachers at those affiliated schools. Recently, classroom observation has been able to be realized using two-way teleconferencing systems by taking advantage of high-speed broadband networks between universities and affiliated schools. Although it is indispensable for students to actually go to affiliated schools to observe classes while meeting face-to-face with the instructors and students at those schools, we have developed an e-Learning resource for classroom observation that utilizes a web-based teaching portfolio as a means of supplementing preliminary guidance prior to teacher training so as to allow classes to be observed without placing any restrictions on place or time [1], [2]. This instructional resource realizes viewing of teaching plans as well as a dual-screen synchronous playback function that provides a screen depicting the teacher and a screen depicting the students. According to the literature that is “Inquiry, Modeling, and Metacognition” by White and Frederiksen (1998)[3], after analyzing videotapes of six local teachers teaching materials, they determined that it is not enough to simply provide teachers with Teacher’s Guides that attempt to outline goals, suggest activities, and describe, in a semi-procedural fashion, how the lesson might proceed. They argue that teachers need to develop a conceptual framework for characterizing good inquiry teaching and for reflecting on their teaching practices. To achieve the goal of enabling teachers to characterize and reflect on inquiry teaching, they are using a framework developed for the National Board for Professional Teaching Standards (Frederiksen et al., 1997[4]). This framework, which attempts to characterize

284

H. Miyata / Development of an e-Learning Resource for Teacher Education

expert teaching, includes five major criteria: worthwhile engagement, adept classroom management, effective pedagogy, good classroom climate, and thinking about the subject matter, to which they added engaging in inquiry. In this characterization of expert teaching, each of these criteria for good teaching is unpacked into a set of “aspects”. For example, the criterion of “good classroom climate” is defined as “the social environment of the class empowers learning.” Under this general criterion, there are five different aspects: engagement, encouragement, rapport, respect, and sensitivity to diversity. Each of these aspects is defined in terms of specific characteristics of classroom practice, such as “humor is used effectively” or “there is a strong connection between students and teacher.” Furthermore, each of these specific characteristics of classroom practice is indexed to a set of video clips, called “video snippets”, which illustrate it. This framework characterizes good inquiry teaching and provides teachers with video exemplars of teaching practice. Such video clips can be used to enable teachers to learn about inquiry teaching and its value as well as to reflect on their own and each others’ teaching practices. They tried the following approach with a class of 10 student teachers. The student teachers first learned to use the framework outlined in the preceding paragraph by scoring some of their materials videotapes. Then, they used the framework to facilitate discussions of videotapes of their own teaching. In this way, they participated in what they call “video clubs”, which enable them to reflect on their own teaching practices and to hopefully develop better approaches for inquiry teaching. Video clubs incorporate social structures designed to help teachers reflectively assess and talk about their teaching practices. The results so far have been very encouraging (Frederiksen & White, 1997[4]). 1. Objective of Research A study was conducted of the effectiveness of an e-Learning resource as an instructional resource for teacher training by distributing that e-Learning resource for classroom observation by students prior to teacher training by providing a dual-screen synchronous playback function for classroom video clips arranged in a web-based teaching portfolio. In this research, an analysis was conducted particularly on the effects of on-demand dual-screen synchronous playback on classroom observations by students using video clips. 2. Research Method 2.1 Web-Based e-Learning Resource The teaching portfolio was a lecture class type of classroom teaching portfolio entitled, "What would you do if you encountered a web page like this? - Questionnaire page and personal information" in a general learning class of fifth grade elementary school students. As shown in Fig. 1, the class teaching plan and a video clip of the class can be viewed on demand. The portfolio contains 13 video clips of three minutes duration or less that have been divided into chapters that provide a summary of the class. The following three types of scenes were captured to produce the video clips. (1) Scenes depicting the teacher: A camera is installed in the back of the classroom to film the teacher from the front (Video T). (2) Scenes depicting the children: A camera is installed in the front of the classroom to capture images of all of the children in the class from the front (Video B). (3) Scenes depicting classroom: A camera is installed to the side of the classroom to film explanations by the teacher and statements made by the children while changing the camera angle (Video C). (See Fig.2, Fig.3) 2.2 Survey Subjects

H. Miyata / Development of an e-Learning Resource for Teacher Education

(Survey 1) Teaching profession courses: The subjects of this survey consisted of 160 second year students enrolled in the Department of Education of Shiga University taking the "Practical Teaching Research" and "Utilization of Information Media" courses. The subjects used for analysis were divided into four groups as shown in Table 1, and 40 students were assigned to each group. (Survey 2) A total of 100 students consisting of 25 students from each group of the four groups shown in Table 1 were extracted at random and designated as subjects of Survey 2. 2.3 Survey Environment * Server: Windows 2000 MediaServer capable of streaming distribution * Client PC: Windows XP Professional * In-school LAN: Playback of gigabit L3 optic fiber video clips was performed on a PC monitor (size: 17 inches, resolution: 1024 x 768)

285

286

H. Miyata / Development of an e-Learning Resource for Teacher Education Table 1 Group Names, Numbers of Students and Presented Video Clips Type of Screen Group Name No. of Students Type and Size of Video Clip single screen 1C Group 40 Classroom scene (small) dual screen 2ts Group 40 Teacher (small) + children (small) dual screen 2Ts Group 40 Teacher (large) + children (small) dual screen 2St Group 40 Teacher (small) + children (large) (Small: 320 x 240 dots, Large: 640 x 480 dots)

2.4 Procedure (Survey 1) (1) Use of the e-Learning resource was performed from any PC installed in three computer training classrooms located at the university at a time convenient for the students within a designated one week period. The subjects viewed only one of the four types of video clips subject to the conditions shown in Table 1. (2) Survey of Understanding of Contents of Observed Class: A web-based test for confirming the students' understanding of the contents of the class was given using a predetermined fill-in-the-blank format relating to the contents of the observed class immediately after using the e-Learning resource. (3) Questionnaire Survey: The students were asked to complete a web-based questionnaire after using the e-Learning resource. The questionnaire consisted of a seven-level evaluation ranging from "1. Completely inapplicable" to "7. Extremely applicable". The questions were composed of a total of 30 questions and comments columns relating to questions pertaining to the degree of difficulty in using the e-Learning resource and the image quality of the video clips, questions pertaining to conveying the atmosphere and situation in the class, and the students' overall impression of the video images. (Survey 2) (1) The students were asked to read a web-based instructional resource containing an explanation of important aspects for classroom observation, select three important aspects of classroom observation from among them, and then reply over the web. (2) An environment was provided that allowed the students to compare four types of e-Learning video-based instructional resources that differed in type and size from the video clips shown in Table 1 that were used in Survey 1. The students then selected the single resource they thought was optimal for viewing the important aspect of classroom observation that they had selected, and then use that resource on any PC installed in computer training classrooms located at the university at a time convenient for the students within a designated one week period. 2.5 Survey Period: May to June, 2004 2.6 Analysis Method In order to analyze the effects of the number of video clips displayed, display size and on-demand dual-screen synchronous playback on classroom observation by students using video clips, a quantitative analysis was performed using statistical techniques on intergroup comparisons between each of the questions on the questionnaire. A qualitative analysis based on the Grounded Theory Approach of Glaser and Strauss (1967) was also made on anything noticed by the students from their classroom observations obtained from the comments they entered in the questionnaire.

287

H. Miyata / Development of an e-Learning Resource for Teacher Education

3. Results and Discussion 3.1 Survey 1 In order to investigate the students' understanding of the flow and contents of the class that they observed, a confirmation test was given to the students using a fill-in-the-blank format relating to the contents of the observed class. As a result of analyzing variance based on those scores, there were no significant differences observed among the four groups. In examining these results with respect to understanding the class contents and flow of the class, in the case of the lecture and explanation type of class observed in this survey, it was thought that the majority of the required information could be conveyed by audio alone provided the voice of the teacher is transmitted clearly, while the presented combinations of screen sizes were suggested to not have an effect on understanding of class contents. The results of replies to questions relating to the usefulness of this instructional resource for classroom observation are as shown in Fig. 4. The resource was evaluated highly by all groups. There were no significant differences observed in the evaluation scores among the groups. 㪭㫀㪻㪼㫆㩷㫀㫄㪸㪾㪼㩷㫈㫌㪸㫃㫀㫋㫐㩷㫎㪸㫊㩷㪾㫆㫆㪻 㪫㪿㪼㩷㫃㪼㫅㪾㫋㪿㩷㫆㪽㩷㪼㪸㪺㪿㩷㫍㫀㪻㪼㫆㩷㪺㫃㫀㫇㩷㫎㪸㫊㩷㪸㫇㫇㫉㫆㫇㫉㫀㪸㫋㪼 㪫㪿㫀㫊㩷㫎㪼㪹㪄㪹㪸㫊㪼㪻㩷㫉㪼㫊㫆㫌㫉㪺㪼㩷㫎㪸㫊㩷㪼㪸㫊㫐㩷㫋㫆㩷㫌㫊㪼 㪫㪿㫀㫊㩷㫎㪼㪹㪄㪹㪸㫊㪼㪻㩷㫉㪼㫊㫆㫌㫉㪺㪼㩷㫎㪸㫊㩷㫌㫊㪼㪽㫌㫃㩷㪽㫆㫉 㪺㫃㪸㫊㫊㫉㫆㫆㫄㩷㫆㪹㫊㪼㫉㫍㪸㫋㫀㫆㫅 㪇

㪈

㪉

㪊

㪋

㪌

㪍

㪎

㪝㫀㪾㪅㪋㩷㪜㫍㪸㫃㫌㪸㫋㫀㫆㫅㩷㫆㪽㩷㪼㪄㪣㪼㪸㫉㫅㫀㫅㪾㩷㪬㫊㫀㫅㪾㩷㪪㫐㫅㪺㪿㫉㫆㫅㫆㫌㫊㩷㪭㫀㪻㪼㫆㩷㪧㫃㪸㫐㪹㪸㪺㫂

In addition, the Kruskal-Wallis H test was used to statistically verify the presence of differences among the four groups for each of the questions of the questionnaire relating to the e-Learning resource. As a result, statistical differences (p<0.05) or trends indicating a statistical difference (p<.10) were observed for the 16 parameters shown in Table 2. According to Table 2, in the case of using dual-screen synchronous video playback as an e-Learning resource for classroom observation, it was clearly determined its use has an effect on (1) motivation and promotion of action, (2) conveyance of atmosphere and situation, and (3) overall impression of video images. With respect to (1), since there were many students who had viewed synchronous video playback for the first time, a sense of novelty was thought to have had an effect on the results, thus indicating the need for continued investigations. With respect to (2), whether or not children are present in the images contained in the dual-screen synchronous playback is considered to have had a considerable effect. In particular, having placed a microphone separate from the one used by the teacher in the center of the area where the children were seated is thought to have had an effect since it made it possible to listen to the children as they talked among themselves. In addition, there were differences in the favorable impression and sense of trust in the teacher based on the teacher's facial expressions. In the case of dual-screen synchronous playback, it was suggested that differences in the impressions on viewers differ depending on the which of the screens is made larger and the combination of sizes in which the screens are presented.

288

H. Miyata / Development of an e-Learning Resource for Teacher Education Table 2 Parameters for which Differences were Observed Among the Four Groups * Motivation and Promotion of Action 5. Want to observed classroom video clips of this type again. 6. Was able to concentrate on recording the class contents. 7. Was able to concentrate on views of the children. 9. Was able to view while concentrating on class references and instructional materials. * Conveyance of Atmosphere and Situation 10. The children appeared serious about studying. 11. Some of the children appeared to be disinterested. 14. Some of the children were not looking at the printout being used in the class. 15. The children were frequently looking at the teacher. 17. The children were closely listening to what the teacher was saying. 18. The children frequently took notes. 19. The teacher gave an impression of being able to be trusted. 20. I had a good impression of the teacher. 21. The video images of the class have a sense of presence. * Overall Impression of Video Images 25. The video images were of good quality. 26. The video images had a sense of presence. 27. It was difficult to determine how the different parts of the class were divided.

With respect to (3), for the parameter relating to the sense of presence of the video images, synchronous video screens of the same size for the children and teacher exhibited a significantly higher score than those in which the image of the teacher was small while that of the children was large, and those in which the image of the teacher was large and that of the children was small, thus suggesting that there is an optimum combination of image type and size in order to comfortably view two synchronous screens. In addition, the finding that there were more responses in the single screen video group than in the dual-screen synchronous video groups indicating that it was difficult to determine how the different parts of the class were divided is thought to have been caused by there being two times when the video happened to have been switched over to the children while the teacher was still talking in the video clip viewed by the students of the single screen video group in this survey. Namely, this can be interpreted as meaning that, in order to for students prior to teacher training to understand the different sections of a class and the transitions made between those sections, it was easier to determine changes in the subject matter being discussed since the video image of the teacher fulfilled the role of conveying non-verbal information of the teacher. 3.2 Survey 2 After having 100 survey subjects select three important aspects for classroom observation among 14 choices presented in the survey, the students then selected one e-Learning teaching portfolio for classroom observation from among four types of available video clips that differed in type and size, and then actually used that teaching portfolio. As a result, none of the students selected the single screen portfolio, while all 100 students selected a dual-screen synchronous video playback portfolio. Among the students that selected a dual-screen synchronous video playback portfolio, 41 selected that having a small image of the teacher and small image of the children, 37 selected that having a large image of the teacher and small image of the children, and 22 selected that having a small image of the teacher and large image of the children. Fig. 5 shows the frequencies of the important aspects of classroom observation selected by students in the three synchronous video playback groups. Each student was allowed to select three important aspects.

H. Miyata / Development of an e-Learning Resource for Teacher Education

289

Fig.5 Important Aspects of Classroom Observation and their Selectiom by Synchronous Video Groups

In order to examine which combination of screen sizes for synchronous video playback were selected for each important aspect of classroom observation, the data was tested by applying to a two-dimensional logarithmic linear model in a factorial design of important aspect versus group, and interaction was observed as a result. As a result of subordinate testing, there was clearly demonstrated to be a trend in which those students who selected the important aspects of classroom observation of "children's behavior" and "children's thoughts" selected synchronous video images of the same size for the teacher and children, while those students who selected the important aspects of "explanations given by teacher" and "instructional materials and teaching tools" selected synchronous video images in which the teacher was depicted larger than the children. Although it was predicted that students who selected the important aspects of "children's behavior" and "children's thoughts" would select synchronous video images in which the children were depicted larger than the teacher, there were many students who actually ended up selecting synchronous video images of the same size for the teacher and children. As a result of interviewing the students with regard to this point, it was mentioned

290

H. Miyata / Development of an e-Learning Resource for Teacher Education

that, "In the observing of a classroom, if the two screens of the synchronous video playback are of different sizes, viewing ends up being biased toward the larger screen. Although the reactions of the children are of the greatest interest to me when observing a classroom, since I have not yet begun my practical teacher training, I also want to carefully observe the actions and expressions of the teacher, and that is why I selected the portfolio having the same screen sizes. In the case of conventional single screen playback, since there was a time difference due to the effects of switching between cameras when an attempt is made to observe the reactions of the children in response to a question or statement made by the teacher, dual-screen synchronous playback makes this very easy to observe." Another response indicated that, "In the case of desiring to observe the actions and thoughts of the children as an important aspect of classroom observation, it is necessary to follow both the scenes of the children and teacher at the same time, and since your attention ends up shifting to the larger screen if the screen sizes are different, I selected a portfolio in which the same screen sizes were used for the teacher and the children." In addition, it was also mentioned that, "Viewing synchronous video clips of the same screen size creates a stronger sense of presence of being in the classroom than combinations of video clips of different size, and this makes it easier to observe the actions and thoughts of the children." In this manner, there is the possibility of not only arousing the attention of the students, but also various other factors, being intricately involved in video images of the children, video images of the teacher as well as multiple combinations of synchronous videos images, and because of this, it will be necessary in the future to add multiple comparisons to confirm differences between specific groups and other groups as well as conduct more in-depth studies by analyzing the relationships between parameters. In addition, although the class used in this survey was of the lecture type, an additional subject for future studies will be to determine which combinations of synchronous video images are effective in the case of exercise-oriented and practical training courses by comparing those findings with those obtained in this research. A portion of this research was conducted through assistance in the form of a research grant made in accordance with Subject No. 1450217 (Representative: Hitoshi Miyata) of Basic Research Receiving Assistance for Scientific Research Expenditures (c)(2) of the Ministry of Education, Culture, Sports, Science and Technology for FY 2003.

References [1] Hitoshi Miyata, Development of a Classroom Teaching Improvement Support System Using a Web-Based Teaching Portfolio with Video-On-Demand, the International Conference on Computers in Education, The Proceeding of ICCE 2003, Hong Kong, pp.365-372, Association for Advancement of Computing in Education(AACE), 2003. [2] Hitoshi Miyata, Evaluation of a Classroom Teaching Improvement Support System Using a Web-Based Teaching Portfolio with Video-On-Demand, International Journal on Advanced Technology for Learning, Vol.2, No.1, The International Association of Science and Technology for Development(IASTED), (Now printing), 2005. [3] White, B. & Frederiksen, J. , Inquiry, Modeling, and Metacognition: Making Science Accessible to All Students, Cognition and Instruction, 16(1), 1998, 3-118. [4] Frederiksen, J., & White, B., Cognitive facilitation: A method for promoting reflective collaboration, In Proceedings of the Second International Conference on Computer Support for Collaborative Learning, Mahwah, NJ: Lawrence Erlbaum Associates, Inc, 1997.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

291

Teachers as E-designers: A Novel Approach to Professional Development in Elementary Science and Technology Gillian MULHOLLAND, Lachlan FORSYTH and Lynette SCHAVERIEN NSW Department of Education and Training, Australia University of Technology, Sydney: Faculty of Education, Australia University of Technology, Sydney: Faculty of Education, Australia [email protected] Abstract. Providing effective, systemic, sustainable professional development for teachers of elementary Science and Technology is a key concern for large, geographically dispersed, centralised education systems. In this paper, we report on DESCANT-SciTech, an Australian Research Council project, being undertaken in collaboration with the New South Wales Department of Education and Training (DET). This project tests an innovative, e-learning strategy for seeding just such a future-oriented, scalable professional development system. The project’s strategy, informed by a generative theory of learning, casts teachers as designers of their own e-learning environment for professional development. We describe and analyse outcomes of the project’s initial phase, as the first cohort of teachers conceived this environment in community, supported by their consultant, DET officers and University researchers. We conclude speculatively, as two new cohorts of teachers to enter this environment.

Teacher professional development, elementary Science Technology, e-design, collective learning, generative learning theory.

Keywords:

and

Introduction Effective systemic professional development of elementary Science and Technology teachers is a key challenge for education providers. In particular: ƒ There is a history of low teacher confidence and competence in and valuing of the teaching of Science and Technology, in studies of teaching, teacher education and student learning internationally (see, for example, in Australia, [1], [2]). ƒ A growing realisation that rapid cultural change, particularly with the advent of elearning, requires a double-edged professional development response from education systems: Teachers need to be equipped to teach emergent disciplines in innovative and e-learning mediated ways [3], [4], [5]; and teacher professional development itself must at least keep pace with the educational possibilities new technologies afford [6]. ƒ A vast set of guiding principles, models and frameworks has been developed for teacher professional development in primary Science and Technology teaching, and elearning mediated teaching, from policy, commissioned inquiry, anecdote and scholarly practice (as recommended frameworks, such as [7], [8], [9] and outcomes of teacher professional learning research and development, such as [10], [11], [4], [12]). We have a track record of successful outcomes in small scale studies of teacher learning in elementary Science and Technology, addressing all three of these dimensions, most recently in e-learning mediated contexts. As precursors of the present investigation, these studies have adopted what are now being termed design-based research approaches.

292

G. Mulholland et al. / Teachers as E-Designers

Such research “blends empirical educational research with the theory-driven design of learning environments, [thereby fashioning] an important methodology for understanding how, when and why educational innovations work in practice” [13]. Central to the particular design-based research approach adopted here is a generative theory of learning. On that view, learners generate ideas, test them on their value and keep those that survive their tests (after [14], [15], [16], [17], and fully articulated in [18], [19]). In brief, a generative learning view recognizes this generate-testregenerate heuristic (and hence, knowledge-gaining or learning) at three nested levels: in the gene pool where genetic knowledge is gained, over reproductive time, through natural selection; in organ systems, over an individual’s lifespan, as an individual’s immune system or brain learns; and in communities and cultures where ideas are tested and refined over generations by groups of scholars. To design a successful professional development environment for teachers as generative learners, intimate, specific and explicit knowledge of how teachers generate and test ideas as learner-teachers, and of their insights into this process, must be gained and used. Hearing and responding to teachers’ own voices in the design of professional development, is justified on equity grounds; and hedges the chances that professional development approaches will succeed (see, for example, [20]). As well, such a research design acknowledges fundamental changes in the ways learning (and, in particular, teacher learning) is conceived. With the advent of new educational media and new ways of thinking about innovations, it is now clear that, in opportune circumstances, knowledge can be generated successfully within those organizations and systems in which it will be needed, rather than in research laboratories or academies for later export [21]. This paper reports the first phase of DESCANT-SciTech (Designing e-learning systems to celebrate and nurture teaching in Science and Technology), an Australian Research Council supported collaboration between the University of Technology, Sydney and the New South Wales Department of Education (DET). The project tests an innovative, future-oriented strategy for scaling up teacher professional development in elementary Science and Technology education in a large, operationally diverse and geographically diffuse education system. In Phase 1, a small group of teachers, supported by consultants, DET officers (in Science and Technology disciplines and IT domains) and University researchers, designed their own e-learning environment for professional development in science and technology education – an environment able to evolve through an iterative process of reflective immersion by subsequent cohorts. In Phase 2, two more cohorts will enter the environment and its worth for their professional renewal examined.

1. DESCANT’s Intervention Design: Phase One The DET selected two adjacent rural school districts to initiate the DESCANT project. The Science and Technology consultant (GM)1 called for expressions of interest. Local information meetings were held and eleven self-nominated teachers from six schools agreed to participate. The resultant network consisted of a broad cross-section of teachers (with respect to years of teaching experience and confidence in Science and Technology and their teaching). All had engaged in professional learning in Science and Technology in the preceding two years. In four of the six schools teachers were paired for support. During the first two and a half months, teachers engaged with each other, their consultant and university researchers in a rich variety of activities, including:

1

Here, initials denote university and DET project members, and first names denote teacher members.

G. Mulholland et al. / Teachers as E-Designers

293

ƒ

Immersion in a generatively theorised e-learning environment, The Generative Virtual Classroom or GVC [5]2, developed by the project leader (LS) to assist teachers to deepen their views about Science and Technology learning; ƒ Extensive webboard discussion (moderated by the consultant, the doctoral researcher (LF) and the project leader) of science and technology, and learning, teaching and professional development in these disciplines, supported by a range of resources including DET syllabus materials and academic research papers; ƒ Virtual excursions to Science and Technology professional development sites; and ƒ Deepening consideration of theories of learning and of their journeys in DESCANT. These experiences deliberately addressed the risk that without clarity about new paradigms or educational media, teachers might design an e-learning environment that “[filled] in the gray areas based on their existing understandings and practices” [20]. At the end of this period, a second workshop (the Design Workshop Day) was held. Over that day teachers participated in discussion about the (generative) theoretical bases of a range of e-learning environments and articulated their own purposes for the e-learning environment they were to design, as well as indicators by which they might gauge whether these purposes had been met. In the culminating afternoon session, the group conceived an e-learning environment for teacher professional development in Science and Technology. We now consider some findings of this first phase of the DESCANT project.

2. Some Findings of this First Phase of the DESCANT Project We present outcomes of Phase 1 of DESCANT in two parts. We describe, from evidence, the nature of the learning science that grew to inform this community’s aspirations for their e-learning environment. Then, we describe the community’s prototype design.

2.1 Growing a Learning Science to Underpin their E-learning Environment for Systemic Professional Development Immediately the Design Workshop Day began and teachers started to contribute in conversation, it became clear that many had adopted a strong and explicitly generative orientation towards learning. For example, Roger described his fascination with the thoughts of a child who was trying to make sense of a strobe in the GVC. Cynthia gave a detailed account of her dynamic conversation with her son concerning about breathing, a conversation she said she would previously have listened to passively but which she now used to explore how and why he thought the way he did. In the midmorning session, teachers agreed that the environment should address the following purpose: To understand better how we (students and teachers) learn, initially by consideration of a generative model of learning, specifically in the context of designing and making and investigating in order to improve student learning in K6 Science and Technology.

2

This e-learning environment is a nested pair of virtual classrooms: an elementary Science and Technology one and a tertiary one. Tertiary learners, as fly-on-the-wall observers, view and review exemplary videotaped excerpts from authentic Science and Technology classrooms at their leisure, contributing their ideas to a community archive, discussing them with geographically dispersed colleagues online and considering pre-recorded generative interpretations. In the present project, the GVC made generative learning principles more accessible to teachers for the purposes of critique and illustrated how e-learning environments can be designed explicitly on a particular (generative) theoretical base.

294

G. Mulholland et al. / Teachers as E-Designers

The group’s webboard conversations over preceding months had not prepared us for the group’s rapid decision to make generative learning theory central to their design. In fact, we had planned to dedicate part of a session to helping teachers assemble a learning model or theory on which to design their environment, using relevant DET syllabus documents and supporting materials. Prior discussions in the DESCANT steering group had stressed that though the GVC was built upon a generative learning theory, it was certainly not the intent of the project to proselytize. Rather, the GVC was simply an object-to-think-with (after [22]) in the project with respect to what a principled e-learning system, rather than a generatively principled one, might look like. Once the teachers had opted for a generative learning basis for their environment, we omitted the planned Design Workshop Day session on learning models and theories; and afterwards, in making sense of our findings thus far, we tried to understand, from the project data we have, how the group came to adopt such an apparently powerfully generative orientation to learning. At the start of the project, teachers immersed themselves in the GVC, and read selected papers posted in the webboard library about related studies of teachers’ and students’ learning. On first entering the GVC, teachers were struck by the power of the video excerpts they saw, in particular for the rare privilege it afforded to see inside classrooms other than their own. As well, teachers commented, as many others have, on the fact that the GVC shows only one or two students learning, rather than a whole class. Much early conversation occurred concerning the tension between provoking thinking about how students learn and providing models of classroom practice. Though we challenged them on the webboard to comment with precise reference to evidence of student learning in the videos they were watching, we noted that these teachers did not often do so. (Why such precision was so difficult is a matter on which we can speculate. Perhaps they were unused to talking about learning at this level of detail or they did not have the language to do so or they feared exposing themselves in a group they did not yet know well.) In any event, many members of the community were later to acknowledge the power of watching one or two students for their thinking about teaching. For example, Kirsty subsequently wrote about her DESCANT learning journey, the GVC … opened my eyes more to what was happening in my classroom. Even though I have always used small groups as a way of seeing the children working, their thought processes, the language they are using etc. I really did begin to reopen my eyes as to what they were saying and how they were saying it. I even found myself letting number groups run overtime as I got into some very deep discussions with kids about patterns and predictions in chance activities. (12 October) And most of these teachers eventually decided to focus on small groups of students in the e-learning environment they designed. In their earliest conversations online and face-to-face, there were signs that these teachers were already attending to aspects that, to a trained eye, could be considered generative, though the language of their descriptions did not always make this orientation explicit. Soon, however, some teachers were adopting the language of generative theory, notably first in making sense of what was happening in the DESCANT community, but also in what they recognised students as doing. For example, Vic commented, I'm intrigued by the community process here. As the eager learner I had a vague idea of a system of sharing and storing inputs......... other ideas were posted...........the gentle facilitators, [LF] and [GM], drew on their greater experience to give some shape to the suggestion, injecting vivid terms like 'atrophies' and so encouraged me to consider the next step. When teachers facilitate such a process we feel valued and stirred to advance. Think, toss

G. Mulholland et al. / Teachers as E-Designers

295

it in, re-think .......... or 'gen/test/re-gen' (generate-test-regenerate). Is that pretty close to what's happening? (30 August). Nevertheless, over much of the two and a half month period, such explicit reference to generative learning was rare. Webboard discussion was peppered with references to a range of ideas teachers had encountered and found useful. In particular, there was extended discussion and much favourable comment relating to 4Mat Training, but others including Multiple Intelligences, right brain/left brain, Herman’s Brain and Brain Gym were also discussed. It was only towards the end of this period of discussion, as the Design Workshop Day approached, teachers returned from their virtual excursions and embarked on writing their final syntheses that it started to become clear that many were adopting an explicitly generative view of learning. For example, picking up what she saw as a move towards a generative view of learning in the group, Cynthia remarked, I don't think it is the [ILF] e-learning environment that is the source of our queries. I think it might be our own hidden agenda, ie. children’s deep learning in a generative way as is reflected in good teaching and learning in science. (25 September). The form as well as the substance of the webboard conversations can be seen as building the wherewithal for such a movement. One of many examples makes this clear: Repurposing our spinning and weaving tasks (intended to generate summaries for the second workshop), Anna commented, “Wouldn’t it be great if you could set up team teaching situations [in the classroom] with weaving and threading functions?” (13 August). GM (14 November) wrote of her sense of this, “Participants are generating, importing, repurposing ideas left and right in a glorious free-wheeling think-fest, then continually testing them against their own and each other’s experience.” Certainly, many of DESCANT’s teachers have long been immersed in the language and ideas of the DET curriculum; and ideas about generative learning were only one part of the DESCANT project. As well, the project is remote, rural and online. However, it appears that these teachers have become every bit as immersed in generative thinking as others have been in face to face, sustained studies of teachers’ learning or in online coursework in which assessment requirements could arguably have played a part (for example, [4], [23]). Here, in DESCANT, clearly our intervention design created opportunities for teachers to develop new, strong, but also diverse generative positions on learning in ways that we are yet to understand fully.

2.2 Conceiving an Initial Design for DESCANT’s E-learning Environment Over the course of the afternoon session of the Design Workshop Day, in tune with their previously expressed purpose of enhancing understanding of learning, participants mapped an e-learning environment in the form of a journey, over which learner-teachers could chart progression. Participants suggested and discussed five journey components: ƒ A “self-test” with which learner-teachers would begin. This introductory activity would address technical set-up and learner-teachers’ expectations about the environment, both generating interest and making clear if learner-teachers’ own interests are compatible with the environment’s focus (Anna). Others suggested it might probe teacher-learners’ initial ideas about learning (JS, Cynthia) or guide navigation by offering suggested pathways (WH). ƒ A set of video excerpts of examples of student learning, captured in the designerteachers’ own classrooms and accompanied by textual material that would focus learner-teachers’ attention or prompt their interest.

296

G. Mulholland et al. / Teachers as E-Designers

ƒ

A private notepad within the environment in which learner-teachers could record their views as they progressed, as well as a forum where community discussion could occur and be facilitated. Conversation dwelt on whether such functionality would be public or private, editable or non-editable, and diverse views were expressed about this. Teachers justified their views with reference to their own experience in the DESCANT webboard and on the GVC. Despite sustained conversation about their diverse levels of comfort about posting in diverse environments, consensus seemed to be reached on the need for both public and private functionalities, but without the ability to edit in either. It was Roger’s experience of forever polishing and never posting, and hence losing the crucial record of his learning journey that clinched that decision: It’s like dragging a piece of brush behind you rubbing out your footprints – you don’t do it. … I concur simply because of the experience I’ve been through where I’ve held back from responding immediately and then felt that my responses were no longer appropriate ƒ A culminating task, in the form of a video excerpt, captured in learner-teachers’ own classrooms and accompanied by textual material. For GM, this task aligned with the reflective journals teachers had completed in the first phase of DESCANT. For Vic, it was important that some kind of baseline or entry story had been told for judgments to be made on progression. For Roger, such a task would spur the kinds of worthwhile workplace implementation that he was familiar with in his role as a Technology in Learning and Teaching facilitator. For LyF, such a task would encapsulate a learner-teacher’s progression and contribute to an expanding set of video clips for future cohorts of learner-teachers to consider. ƒ A rating process (proposed by Vic) whereby learner-teachers could judge the worth of videos or discussion comments, elevating over some threshold those contributions that are valued for some reason, so that other learner-teachers could easily find them. For Vic, such a process would address the challenge of keeping abreast of the rapid accumulation of large numbers of considered posts. For Kirsty, if 100 people valued a particular video for a specific reason as against five people that valued another, a learner-teacher might want to have a look at the former video to find out what was good about it. After this workshop day, teachers returned to their schools to record examples of students’ science and technology learning in their classrooms, selecting excerpts and writing accompanying texts for them. At the time of writing, it is 17 months later, and there has been a range of unanticipated difficulties that have delayed development. Two further workshops with the first cohort of teachers have been held. In the first, the community further elucidated their conception of the environment and in the second, teachers furnished it with their legacy to future cohorts. Now the DESCANT environment has been completed, the second and third cohort of teachers are just beginning their learning journeys – and the first cohort of teachers has committed to enrolling as mentors.

3. Discussion and Implications The first cohort’s commitment to DESCANT, despite its unusual demands and long delays, is of interest to us. We explain this commitment, at least partly, in terms of the trusting relationships that teachers developed within the cohort, the ways they came to understand their roleand a sense of mutual responsibility for the common project. These teachers’ brave decision to video learning events in their classrooms is evidence of both trust and commitment. It is not clear exactly when this decision was

G. Mulholland et al. / Teachers as E-Designers

297

made. However, their visits to the Inquiry Learning Forum made the possibility of capturing their own classroom video more real. They expressed a range of thoughts – trepidation, as they empathised with teachers whose lessons were receiving critique, alongside recognition of the value of that critique. Events in teachers’ own classrooms, including distressing ones, were brought to the webboard for discussion and received prompt, sensitive and often uplifting response. From the earliest stages, to our delight, teachers appropriated for themselves a role as researcher-designers, a role that constituted a significant shift from their usual one of learner-teachers in professional development contexts – or even of e-learning environment critics. It was as if they knew that they held the camera in their own hands. Post after post alluded in diverse ways to their new view. Four examples make this clear: ƒ Kate was conscious that being part of a research project directly affected what she did, And now you see a conundrum. I am happy to bare such thoughts on here but "reluctant" to let go in the GVC. Is it because I feel that in the webboard we are becoming a little community and can be friends? … I am probably only sharing these thoughts with you because I am part of a research team and these reactions could provide valuable insight into how other learners may feel about things, (the teacher part of me coming out) because I am a very "private" person usually. I haven't asked any questions here, but have I caused some?(15 August) ƒ Sarah distinguished evaluating an e-learning environment from what she was doing, Most of us have experienced frustration over the quality of the videos and the problems associated with their viewing. When we step back and think about why we're viewing them, though, their quality really is of little importance. When I first started delving into the ILF, I had it in my mind that I wanted to develop professionally via the ILF medium, but that's not what we're doing, is it? We're actually just looking at the ILF model and … it actually has more positives than negatives about it. (21 September). ƒ Kate wrote to LF, proposing many thoughtful explanations for what she saw, You said: “Assuming such anxiety is common, could this be related to the 'culture' surrounding the sharing (making public) of beliefs about learning (in the teaching community)? How solid do our ideas have to be before we are willing to make them public? Or perhaps more importantly, do we view the (public) ideas of other people as 'their final word' (cast in stone) instead of their 'evolving word' (stepping stone.)” I was chatting with [GM] about this topic the other day when she visited. Perhaps we do feel our ideas may not appear to be solid enough. My thought was that, having seen the expected change of outlook over time for students we perhaps fear the thought of having made changes ourselves (thereby illustrating "deficiency" in ourselves) OR that we won't have shown change. I suspect that experienced teachers (ones who've been around for a while) would be reluctant to reveal something that would be open to criticism. (22 August). ƒ Stepping further, Kate asked LS, “An ‘aha’ thought here! How much learning are you getting out of this whole thing?? (smile included) What discoveries about learning are you learning as you watch us go through this project?” (16 September). In view of these insights, it ought not to surprise us that teachers decided to turn the video camera on their classrooms. They had focused their lenses on the work of the DESCANT community and were already standing in front of the camera and behind it. In this initial phase, DESCANT teachers have been able to build a (generative) learning theory for themselves, and to conceive a rich and principled environment around learning episodes as sources of understanding for teaching. So, perhaps the thought of

298

G. Mulholland et al. / Teachers as E-Designers

capturing on video the ingenuity of their students (with their own subtle, significant support as teachers) might have begun to look not only urgent for their own professional development and that of others, but also much less threatening after all. Prior studies of learning in the GVC (for example, [24]) have described the mirroring that occurs once teacher-learners understand that students in the virtual classroom are learning in similar ways to themselves. An analogous finding is visible here, at the level of research-based designing (which is, of course, learning in its own right). These teachers appeared to realise, clearly and profoundly, that their work in DESCANT mirrors the work of many others in their vista: the research team as researcher-designers of the DESCANT project, LS as designer of the GVC, others as designers of ILF, and so on. Such a conclusion would help to explain the intense enjoyment and high attribution of value to the project that we have witnessed in this DESCANT group.

References [1] Department of Employment, Education and Training. (1989). The Discipline Review of Teacher Education in Mathematics and Science. Canberra: Australian Government Publishing Service. [2] Goodrum, D., Hackling, M. and Rennie, L. (2001). The Status and Quality of Teaching and Learning of Science in Australian Schools: A Research Report prepared for the Department of Education, Training and Youth Affairs. http://www.detya.gov.au/schools/publications/2001/science/index.htm [11 Sep 2005] [3] Schaverien, L. & Cosgrove, M. (1995). Technology learning 2: Towards reawakening the technologists within primary teachers. International Journal of Technology and Design Education 5(1): 51-68. [4] Schaverien, L. & Cosgrove, M. (1997). Learning to teach generatively: Mentor-supported professional development and research in technology-and-science. Journal of the Learning Sciences 6(3): 317-346. [5] Schaverien, L. (2000). Towards research-based designing for understanding fundamental concepts. Aust J. of Ed. Tech. 16(1):1-12. [6] Marx, R., Blumenfeld, P. and Krajcik, J. (1998). New technologies for teacher professional development. Teaching and Teacher Education 14(1): 33-52. [7] Loucks-Horsley, S., Hewson, P., Love, N. & Stiles, K. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, Ca: Corwin Press. [8] McIntyre, D. & Byrd, D. (eds.) (2000). Research on Effective Models for Teacher Education. Teacher Education Yearbook VIII. Thousand Oaks, Ca: Corwin Press Inc. [9] Sparks, D. & Loucks-Horsley, S. (1989). Five models of staff development. Journal of Staff Development, Fall, 10(4) http://www.nsdc.org/library/publications/jsd/sparks104.cfm [11 Sep 2005] [10] Bell, B. & Gilbert, J. (1996). Teacher Development: A model from science education. London: Falmer. [11] Clark, J. (2001). A generative model for teacher development in primary science. Unpub. PhD, UTS. [12] Smith, D. & Neale, D. (1991). The construction of subject matter knowledge in primary science teaching. Advances in Research on Teaching 2: 187-243. [13] The Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. [14] Edelman, G. (1992). Bright Air, Brilliant Fire. London: Penguin. [15] Plotkin, H. (1994). The Nature of Knowledge. London: Allen Lane and Penguin. [16] Plotkin, H. (1997). Evolution in mind: An introduction to evolutionary psychology. London: Penguin. [17] Plotkin, H. (2002). The imagined world made real. London: Allen Lane, Penguin Press. [18] Schaverien, L. & Cosgrove, M. (1999). A biological basis for generative learning in technology-andscience: Part I - A theory of learning. International Journal of Science Education 21(12): 1223-1235. [19] Schaverien, L. & Cosgrove, M. (2000). A biological basis for generative learning in technology-andscience: Part II - Implications for technology-and-science education. Int. J. of Sc. Educ. 22(1): 13-35. [20] Stein, M., Smith, M. & Silver, E. (1999). The development of professional developers: Learning to assist teachers in new ways. Harvard Educational Review 69(3): 937-969. [21] Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P. & Trow, M.. (1994).The new production of knowledge. London: Sage. [22] Papert, S. (1980). Mindstorms: Computers, Children and Powerful Ideas. NY: Basic Books. [23] Schaverien, L. (2002). Learning about e-Learning: Some insights into a postgraduate learning community. Proceedings of ICCE, Massey Uni, Palmerston Nth, NZ, 3-6 Dec. [24] Schaverien, L. (2003). Teacher education in the Generative Virtual Classroom. International Journal of Science Education 25(12): 1451-1469.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

299

On-time Support with Supplementary Figures in Tutoring of Mathematical Exercises Yosuke MURASE1, Tomoko KOJIRI2 and Toyohide WATANABE1 1 Department of Systems and Social Informatics, Graduate School of Information Science, Nagoya University 2 Information Technology Center, Nagoya University Furo-cho, Chikusa-ku, Nagoya, 464-8603, Japan Abstract. The objective of our research is to support for solving mathematical exercise in the collaborative learning environment. To promote the exchange of more varied opinions toward the correct answer, it is necessary for the support system to offer a communication means, recognize the learning situation, and provide appropriate advice according to the learning situation. In mathematical exercises, it is effective to propose a diagram to the learning group as form of advice. In this work, we construct a system that proposes supplementary figures in order to enable the group to derive the next step in a problem according to the current learning situation. In our system, a diagram which a learner draws is converted into an internal model. Then, supplementary figures according to the learning situation are generated by selecting and applying a rule for drawing diagrams that corresponds to the internal model. In this paper, we explain the overall framework for generating supplementary figures and show the results of an experiment to evaluate the system.

Introduction Collaborative learning through networks is an effective learning style that enables learners to enhance their creative skills [1][2]. In collaborative learning, the members of the group share learning materials and cope with a common learning task, and the collaborative learning proceeds through discussion. However, if the learners’ levels of understanding are different and communication among learners is ineffective, learners can never attain their learning goals. Therefore, it is necessary for a support system to offer a communication means, recognize the learning situation, and provide appropriate advice according to the situation. In this work, we focus on collaborative learning of high school-level mathematics. We have developed an agent that grasps the group’s levels of understanding and generates appropriate advice through conversations in a chat window [3][4]. However, in the real world the learners solve the mathematical exercises by not only writing equations but also by drawing diagrams so as to arrange the current learning situation and deduce a derivation method for the next step. Ito et al. analyzed the learner activities for drawing diagrams in geometry exercises [5][6]. Based on this analysis, learners can visually clarify features that could not be noticed from written descriptions. Furthermore, Larkin et al. demonstrated that diagram expression would be used to search for information needed for forming inferences

300

Y. Murase et al. / On-Time Support with Supplementary Figures

in the process of problem solving [7]. Therefore, it is effective to propose diagrams to learners as advice for estimating the derivation method for the next step when they fall into an impasse. The objective of our research is, therefore, to propose a mechanism that generates appropriate diagrams according to the current learning situation. To show the diagram to the group is not to give answers directly but to promote more active discussion.

Fig. 1. Learning support by sharing diagrams in the group

In a collaborative learning environment, the learners derive the correct answer while sharing diagrams among the group (Fig. 1). The group’s diagram is published for all members. All members can solve problems by updating the diagram while accessing it in turn. So, in order for the learners to achieve smooth discussion, we must develop an interface that can share the diagram among group members. Through this interface, a mechanism is constructed to generate supplementary figures that indicate the next derivation method as advice to the group. It is unsuitable to prepare the diagram statically as a piece of advice in an environment where the learner can draw diagrams freely. The diagram that each learner draws is not always identical to others, even if the learning situations are the same. If the diagram that one learner draws is different from the diagram illustrated by the system, it takes a long time for the learner to understand the system’s diagram. It is, however, helpful for the learner to receive advice based on the diagrams of the learning group. Thus, our system analyzes learners’ diagrams and generates a supplementary figure dynamically based on the learners’ diagrams. In addition, a supplementary figure is defined as an additional figure corresponding to the derivation method. The derivation method corresponds to a specific theorem, rule or formula. In our approach, stepwise generation of the diagram corresponding to each step is regarded as a forward reasoning in the production system. That is, an appropriate drawing rule that corresponds to a method for drawing a supplementary figure of a particular derivation method is selected and applied to the diagram of the current situation. First, the rules for drawing diagrams are prepared. Then, the diagram in each situation is generated dynamically by applying particular rules to the current diagram.

Y. Murase et al. / On-Time Support with Supplementary Figures

301

1. Overall Framework for Generating Supplementary Figures The mechanism that dynamically generates a supplementary figure as advice is composed of two layers (Fig. 2). In the model conversion layer, our system converts a diagram drawn by learners into an internal model represented inside the system, and the internal model expresses the diagram as a logical expression. In mathematical exercises, if learners understand specific shapes and positions, they can draw diagrams. Moreover, the position of figures can be expressed by positional relations with others. Therefore, the internal model is represented by figures that compose the diagram and relations between figures. Table 1 shows the internal model of a mathematical exercise for a two-dimensional function. Variables x and y indicate the names of figures. Currently, six kinds of figures and twelve kinds of relations are prepared in our system.

Fig. 2. System architecture Table 1. Internal model of a mathematical exercise for a two-dimensional function terms

objects

figure

point(x) segment(x) x-axis(x)

line(x) parabola(x) y-axis(y)

relation between two figures

cross(x, y) contact(x, y) parallel(x, y) top(x, y) edge(x, y) same_shape(x, y)

on(x, y) apart(x, y) vertical(x, y) pivot(x, y) same_long(x, y) opposite_shape(x, y)

On the other hand, in the diagram generation layer, the system holds rules for generating supplementary figures, which are called the rules for drawing diagrams, and selects the appropriate rule corresponding to the current internal model. Supplementary figures depend on the derivation method. The appropriate derivation method is applied to specific figures and relations between figures, and a specific figure is then generated. Therefore, the rules for drawing supplementary figures corresponding to each derivation method are prepared and the appropriate rule is selected using forward reasoning. The rule for drawing diagrams is expressed in the following form: (rule number IF THEN ) .

302

Y. Murase et al. / On-Time Support with Supplementary Figures

Because the condition that can apply to the derivation method has been decided, the conditional part of the rule for drawing diagrams is assumed to be an internal model, which is a precondition for applying the derivation method. It is also effective to show the result of applying the derivation method. Because the system adds a supplementary figure as an appropriate diagram, the operation part is assumed to be only an addition action, thus figures that are results of applying the derivation method are indicated as operational parts. The system can draw the supplementary figures by updating the internal model by applying appropriate rules for drawing diagrams. 2. Prototype System It is assumed that our system supports the entire group and generates supplementary figures by interpreting the diagram drawn on the group’s common screen. Since all members access the group’s diagram in turn, the system handles only one diagram. Therefore, when considering the mechanism that generates the supplementary figures, the function of our system does not change even if it is developed in a stand-alone environment. Currently, the prototype system that executes in the stand-alone environment is implemented. This system is mainly implemented using Java, with figures and relations managed as independent objects. Prolog is also used to select and apply the rules for drawing diagrams.

Fig. 3. Interface

3.1 Interface Figure 3 shows the interface of our system. The learner freely draws a diagram on this interface, and can also draw a figure by pushing buttons for drawing figures and clicking appropriate coordinates on the canvas for drawing diagrams. For example, if the learner wants to draw a segment, he clicks the origin point and the terminal point on the canvas. To date; six figures have been prepared, point, line, segment, parabola, x-axis, and y-axis. If special relations such as “top”, “parallel,” and “vertical” are needed, the learner pushes the

Y. Murase et al. / On-Time Support with Supplementary Figures

303

button for drawing relations. The system generates an internal model of figures and relations between figures and displays them in the list of figures and relations, respectively. A relation between figures is grasped by calculating the coordinates of each figure, and the learner can delete the figure easily by choosing the target figure from the figure list. When a new figure is added, it is drawn on the canvas in a different color in order to highlight it. Figure 4 shows an example of generating diagrams. The left side in Fig. 4 is a diagram drawn by a learner, while that on the right shows is a diagram generated by applying the rule to add for deriving the distance between a point and a line. Diagram input by learners

Diagram output from system

Fig. 4. Example of applying rule for drawing diagrams

If the learner requires supplementary figures, the “next” button is pushed, converting the diagram into an internal model. Then, appropriate rules for the current internal model are selected and a new diagram with supplementary figures is shown on the canvas as advice. If the learner pushes the “cancel” button, another rule is applied and another diagram is generated. After pushing “next,” if the learner wants to see the diagram of the previous state, the “previous” button is available for doing so. 3.2 Selection and application of rules for drawing diagrams Currently, in our system there are 31 different prepared rules for drawing diagrams in a two-dimensional function field. These rules are described by using Prolog as shown in Fig. 5. In that example, the rule corresponds to the rule for deriving distance between a point and a line, whereby if the point, the parabola, and the relation apart exist, the point and segment that connects two points are also added. In the ninth and eleventh lines, a Java method is recalled from the Prolog program, whereby the name of the generated figure is allocated by receiving the name of a new object from the Java program. When the rules for drawing diagrams are applied, the internal model and conditional parts of the rules for drawing diagrams are compared. The parameter of the figures and relations is substituted for the element parameter of the condition of the rule based on the concept of unification. If the rule’s condition satisfies the internal model, the figures and the relations of an additional list are added to the internal model. In the example given in Fig. 5, first the name parameters of additional figures are set as “A” and “B.” Then, the figures’ name parameters are received by the Java method call. In Fig. 5, parameter “C” is

304

Y. Murase et al. / On-Time Support with Supplementary Figures

set as the line name and parameter “D” is set as the point name. The matching process is performed on all prepared rules. If multiple rules match the current internal model, the rule which has the largest number of objects in the conditional part is selected. ( The rule to add for deriving the distance between a point and a line ) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

if(Obj) :point(X1), line(X2), apart(X1, X2), then(Obj, X1, X2), then(Obj, X1, X2) :/*** Additional Objects ***/ javaMethod(Obj, returnValue, A), add(point(A)), javaMethod(Obj, returnValue, B), add(segment(B)),

Conditional Part

Operational Part

/*** Additional Relations ***/ add(apart(X1, A)), add(edge(X1, B)), add(on(A, X2)), add(vertical(X2, B)), add(edge(A, B)), :

Fig. 5. Example of a rule for drawing diagrams

3. Evaluation We performed a learning experiment using our prototype system. The objective of this experiment is to show the effectiveness of the supplementary figures proposed by our system. In the learning experiment, nine subjects from our research laboratory solved three mathematical exercises using our system. The subjects were allowed to write equations on paper, but they were required to draw diagrams using our system. Then, if they fell into an impasse situation, they had to push the “next” button to acquire supplementary figures as advice. After the learning was finished, the supplementary figures generated by the system were evaluated. Supplementary figures were generated for 23 exercises. Table 2 shows the evaluation results for supplementary figures proposed by our system. About 65% of the subjects answered that supplementary figures were useful, though three of the subjects thought that the supplementary figures to be inappropriate. In those cases, we found that the system applied inappropriate rules and generated supplementary figures that were not effective for the exercises. Table 2. Evaluation of generated supplementary figures No.

Evaluation of supplementary figures

1

I could derive the answer using proposed supplementary figures I thought supplementary figures were useful, but I could not derive the answer at once. I thought some of the supplementary figures were not suitable. I could not get any ideas from the supplementary figures. I do not know whether supplementary figures are suitable or not. Total

2 3 4 5

Number of answers 9

6 2 1 5 23

305

Y. Murase et al. / On-Time Support with Supplementary Figures

In the two-dimensional function field, since the number of internal models was fewer than the prepared rules for drawing diagrams, there arose situations that the internal model described in the conditional part of the rule for drawing diagrams was equal to and included in the conditional part of other rules. Figure 6 shows the inclusive relation of rules for drawing diagrams. Based on the types of conditions, rules are classified into a tree structure. According to Fig. 6, conditions for the majority of rules are dependent on each other. That is, the condition of one rule is a part of the condition of another rule, so that the former rule is applied to various situations. At present, rules for drawing diagrams are applied in order in which according to the number of rule conditions, largest to smallest, because the rule with condition elements most suited to the derivation method was specified in the exercises. Thus, in this experiment, the rule that the number of elements is small is not applied, even if that rule is the most appropriate derivation method in the exercise. It is, therefore, important to apply an appropriate rule to adjust the order and to determine the range according to the exercise.

Fig. 6. Inclusive relation of rules for drawing diagrams according to their conditions

Finally, we examined the operationality of the interface. The result is shown in Table 3. Most subjects thought that the interface is valid for drawing diagrams. However, some inappropriate figures could be drawn sometimes because the figures were generated by pointing out specific coordinates. Therefore, it might be convenient to be able to draw figures by inputting a mathematical expression. Table 3. Evaluation of the interface’s operationality No.

1 2 3

Evaluation of the of interface’s operationality

It is easy to draw diagrams. It is not easy nor valid to draw diagrams. It is difficult to draw diagram. Total

Number of answers 2 6 1 9

306

Y. Murase et al. / On-Time Support with Supplementary Figures

4. Conclusion and Future Work We addressed a system that dynamically interprets a diagram drawn by a learner and generates supplementary figures that indicate a derivation method as advice. In our system, diagrams drawn by the learner was converted into the internal model. Then, appropriate rules for drawing diagrams were selected and applied to the internal models based on the approach of forward reasoning. We implemented the system working in a stand-alone environment and performed an experiment. The experiment results reveal the majority of subjects replied that the figures proposed by the system were useful and that the interface is powerful for drawing diagrams. In future work, first we must integrate the prototype system into a collaborative learning environment. Because it is necessary to manage a learning group using this system, the system needs to be extended as a server-client system. It is also necessary to carefully manage the members of the group, and to set access authority for drawing the figures. Next, we must determine whether our system promotes active discussions about various viewpoints. In collaborative learning, diagrams are not drawn in a stepwise fashion; therefore, the history of applying rules for drawing diagrams should be managed and rules need to be selected in tracing the history. Moreover, we are currently focusing on only a mathematical exercise involving a two-dimensional function. To generate advice that induces various viewpoints, we need to extend the target fields, such as to threedimensional functions, trigonometric functions and so on. Of course, the internal model and rules for drawing diagrams should be extended according to the target field. References [1] S. Williams and T. Roberts: Computer Supported Collaborative Learning: Strength and Weakness, Proc. of ICCE’02, pp. 328-331, 2002. [2] J. Cuena, A. Garcia-Serrono and M. Verdejo: The Role of Knowledge Based System for Automatic Coordination in Distance Learning, Collaborative Dialogue Technologies in Distance Learning, NATO ASI F-133, Springer Verlag, pp. 142-155, 1994. [3] T. Kojiri and T. Watanabe: Diagrammatic Advices in Mathematical Excercises, Proc. of ICCE’03, pp. 293-296, 2003. [4] T. Watanabe and T. Kojiri: Airchitecture of Collaborative Learning Environment as Infrastructure for Personal Activity, Proc. of CATE’03, pp. 420-425, 2003. [5] T. Ito, N. Ohnishi and N. Sugie: A Cognitive Model of Diagram Based Geometric Problem Solving, Transactions on IEICE (in Japanese), Vol. 24, No. 11, pp. 84-97, 1993. [6] T. Ito, N. Ohnishi and N. Sugie: Problem-Solving SCRIPT and Classification of Drawings for Expaining Human Drawing Processes, Transactions on IEICE (in Japanese), Vol. 26, No. 5, pp. 29-42, 1995. [7] J. Larkin and H. Simon: Why a diagram is (sometimes) worth ten thousand words, Cognitive Science, pp. 65-99, 1987.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

307

The Capabilities Required by Online Tutors Mediated Through Learning Environment Factors C. Paul NEWHOUSE & Doug. REID School of Education, Edith Cowan University & Faculty of Education, University of Wollongong [email protected]

Abstract. The Internet has provided distance education with a new delivery opportunity that tertiary institutions are rapidly deploying. However, little is known of the characteristics of effective human tutors to exploit this opportunity and what capabilities they require for particular online learning environments. This paper reports on an in-depth study examining the online learning milieu in six separate courses that incorporated the use of text-based online technologies at an Australian university. The study required a determination of the environmental factors affecting online tutor capabilities and the relationship between these capabilities and the factors. Thirteen environmental factors emerged from an analysis of the data and a set of critical online tutor capabilities based upon five categories distilled after an exhaustive review of the literature. From this a model of the mediated relationship between these capabilities and environmental factors was created and recommendations were made concerning the required capabilities of online tutors that will assist tertiary institutions in selection and training processes. Finally, the study identified a dissonance between the perceptions of tutors, students and course designers that needs careful consideration and further research. Keywords: Distance Learning, Open Flexible Learning, Tutor, Open Learning Systems, Higher Education

The past decade of research and development in educational computing has been dominated by the presence of the Internet, so much so in Universities that for some learning has become elearning. The drive towards online learning in Universities has been relentless whether based on an economic or educational rationale. There seems little doubt that increasingly courses in Universities will include substantial components online for all students [1, 2]. This requires University ‘teachers’ to adapt to a different type of learning environment and almost certainly places different demands and expectations upon them requiring some variation in the capabilities needed for traditional face-to-face instruction. For the purposes of this discussion the term ‘tutor’ will be used to refer to University ‘teachers’. There has been a lack of research into the capabilities required by online tutors [3, 4] and the environmental factors which affect tutor capabilities [5-7]. In the hiring and training of tutors it is important to know what capabilities are required, which are essential and which features of the learning environment may impact on those capabilities. It is currently unknown what these capabilities are, their relative importance and relationship to factors and components of the online learning environment. Much is known for face-to-face tutors but not for online tutors, and the two are likely to be different [4, 8]. That is, “teaching with technology to learners who are not physically

308

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

located in the same site where instruction is taking place requires a different set of skills and competencies than traditional education” [2]. Throughout the literature, there are authors who present a large number of capabilities in various states of organization [1, 9, 10]. Furst-Bowe [8] and Salmon [11] have detailed, comprehensive lists of competency areas and there are also discussions of levels of competence needed. Schoenfeld-Tacher & Persichette [6] examined the literature for factors that affect the capabilities required by online tutors to find: subject matter, context, delivery medium, learner variability, teacher preparation and experience in both content and pedagogy plus the synergy and interrelationships among the factors. These lists needed to be investigated to identify further factors and determine the relative importance of these factors and their relevance to tutor capabilities. Clearly further research was needed to identify whether the factors related to the online learning environment modify the essence of the capabilities as Kupritz [12] found who examined the way people communicated through different contextual mediums. Subject matter and delivery medium were two other factors that affect the capabilities required by online tutors [6, 12]. This complements the argument that competencies are dynamic and largely depend on social context [13]. There is evidence in the literature that online students want to be involved with units that have a tutor to help them [1, 7]. However, there is little detailed literature regarding the capabilities required to be an effective online tutor in relation to the roles they are required to play [3]. According to Cyrs [4], there is a need for the development of specific online skills for tutors. The study reported here limited itself to considering tutors in online text based tertiary education settings in Australia although the findings may be able to be generalised to other online educational contexts. The main research question was, “What are the relationships between text-based online learning environment factors in tertiary education and the required capabilities of tutors as perceived by the stakeholders?” The study aimed to consider the perceptions of the major groups of stakeholders, namely students, tutors and unit coordinators (administrators). Phipps and Merisotis [14] present a difference between student and tutor views and suggest that more research needs to be done on the tutors’ role in distance education. 1.

Theoretical Framework for Online Tutor Capabilities

The literature was exhaustively examined and lists of capabilities were organized into a number of categories. There are a number of authors who have written about online tutor capabilities [1, 9, 15-18]. A number of categories were identified after examining many of the categorization strategies and the individual capabilities presented throughout the literature. This process of organization involved creating a variety of ways to group the capabilities to determine what would be the most advantageous for the study. Throughout this process, definitions were refined to determine what an online tutor was in comparison to online instructors, online teachers, e-moderators, facilitators and online trainers. There were many methods of classification put forth in the examined literature and over 500 discrete capabilities presented. This led to a decision to focus on a limited number of foundational sources [e.g. 1, 4, 11, 13, 19, 20-24] with each additional source adding both individual capabilities and new organizational schema. Each individual capability was examined and sorted with other similar capabilities. This process of sorting the capabilities into groups resulted in twenty-four groups of capabilities. These twentyfour groups of capabilities were then defined and examined again resulting in the combination and sorting into five categories of capabilities. These five encompassed sets of sub-capabilities with all being labelled based on the terminology used throughout the

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

309

literature. There were five online tutor capabilities associated with the twenty-four related sub-capabilities as shown here. Content Expertise Knowledge and Skills, Enriching Interactions, Finding & Providing Resources, Question Analysis, and Relevant Tasks Course Management Institution Contact, Pedagogy, Management, and Administration Evaluation Assessment, Course Evolution, Feedback, and Monitoring Process Facilitation Communication, Values, Confidence, Disposition, Environment Creation & Maintenance, Facilitating, and Pedagogical Technical Knowledge Attitude, Choice of Resources, Technical Pedagogy, and Technical Support Figure 1 shows a graphical representation of the theoretical framework developed for the study which connects the online tutor capabilities categories as being mediated through the online learning environment composed of students, tutors and the online learning environment technologies, pedagogies and resources. The online tutor capabilities, Content Expertise, Process Facilitation, Evaluation, Course Management and Technical Knowledge act as the contact points through which the tutor interacts with the students and the online learning environment (OLE) technologies, pedagogies and resources. The tutor, through the five capabilities, comes first into contact with the OLE technologies. Through this contact, the tutor is interacts with the students, while the five capabilities are mediated by the OLE technologies. Students

Content Expertise

Process Facilitation

Tutor Evaluation

Technical Knowledge

Course Management

Figure 1: Graphical representation of the online learning environment.

2.

Method

This study was designed using an interpretative approach and an ethnographic model with an attempt to draw on a mix of qualitative and quantitative data collection methods for its data collection and analysis. The samples for this study drew from six selected online units offered at the University three main stakeholders in online education: tutors, students and unit coordinators. One tutor was selected from each of the six online units with each referred to by a pseudonym to maintain anonymity: Benny; Catherine; AC; Margaret; Lauchlin; and William.

310

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

Data were collected from tutors, students and unit coordinators who were contacted to provide input at the specific phases of the study (Pre-Unit, During Unit, Unit Wrap-Up, and Post-Unit). The four data collection phases took place throughout the entire length of a semester-long online unit and beyond. There was a research cycle of data analysis that ran throughout the data collection phases that allowed the analysis of data to inform later data collection. There were four distinct phases of data collection with the data sources utilized were: Pre-Unit online surveys of students and tutors; During-Unit interviews of all unit coordinators, electronic observation of students and tutors and face-to-face observation of tutors; Unit Wrap-Up online surveys of students and tutors; and Post-Unit interviews with all tutors and a sample of students. The online units in the study were selected mainly on the level of interactivity provided by the delivery technologies employed in each unit. The measure of the level of interactivity between students and tutor from the Technology Generations in Distance Education and Open Learning model presented by Oliver and Grant [25, p. 1]. Three units were selected that had a high level of interactivity and three units were selected that had a low level of interactivity. Instrumental in the selection of the online units were the unit coordinators. They had an intimate knowledge of the delivery technology, course content, pedagogical under structure of the unit and the tutors. There was an effort made to select units that had a number of online tutors, however there were no units available for study that had more than one tutor in addition to the unit coordinators who chose to take on many of the online tutor roles. 3.

Findings

As the study progressed, 13 factors affecting the capabilities of online tutors in the learning environment emerged. These factors were identified during the data analysis phase of the study with the final definitions given here. Community: The learning community (or lack thereof) created by the design of the unit, the actions of the tutors and the actions of the students. Content Milieu: Issues dealing with the educational material used in the unit; including how the materials were presented, access issues, and how the students interacted with the materials. Design / Pedagogy: How the pedagogy involved with the design and presentation of the unit affects the students and tutors. Facilitation of Learning: How the tutor helped the students interact the content without direct instruction which encompasses the tutors understanding of how learning takes place. Institutional Milieu: How the unit is affected by the policies, procedures and supposed beliefs of the institution that is offering it. Interaction student/tutor: The interaction between the tutor and the student in all situations, at a distance, in person and facilitated by technology. Management of Teaching Processes: The non-instructional teaching processes involved with tutoring, including marking, preparation time and time management. Student Expectations: What students believe as compared to what the tutor believes or what the situation really is. Student Responsibility: What students are responsible for according to the tutor, the unit designer and the University. Not necessarily what the students think they are responsible for. Subject Epistemology: The tutor showing an expertise in the content subject area. Technical Milieu: This was everything regarding technology including learning to use it, potential access problems, and how to use it in a proper pedagogic manner.

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

311

Tutor Experience: The experience (or lack thereof) the tutor has dealing with aspects of tutoring online and how that affects the unit being tutored. Tutor Personality: The tutor as a person dealing with emotions, behaviours and personality. The main stakeholders perceived the identified factors differently. The tutors had the most pragmatic view of the factors, the unit coordinators had the most overarching view and the students had the most student-centred view. During the analysis of the tutor interviews, it was found that five factors were associated with the main themes that emerged, namely, Interaction Student/Tutor, Technical Milieu, Tutor Personality, Design/Pedagogy, and Student Expectations, with the first three factors identified by all tutors as important, irrespective of the educational situation. The tutors viewed the identified factors in a practical manner as they had just fulfilled their role as a tutor. Efficiency was vital for the tutors especially in relation to helping the students. The effective interaction between students and tutor was clearly the focus of the tutors. This included the inclusion of their coping mechanisms such as Margaret’s decision not to access the Internet during the weekend or William’s discussion board postings continually requesting students to contact him so he could help them. The tutors did not emphasize their abilities a great deal and tended to understate their content expertise. The unit coordinators focused more on content expertise, technical milieu and institutional milieu than the other stakeholders. For example, Margaret’s unit coordinator stated that the tutor “should have a first degree in the content area and work experience with the content.” Benny’s unit coordinator commented that “To survive as an online tutor … relatively competent with computers usage, html, ftp, elementary ability of graphics, and a degree of technical skill on how to troubleshoot student problems.” For the students interaction between the students and tutor as well as student expectations were the focal points. One of AC’s students wanted to view feedback her classmates received so that she could learn how AC evaluated and “do better in the next assignment.” Catherine’s student wondered why they didn’t “go the full way and have digital conferencing with your picture up there.” One of Benny’s students argued that “Tutors need to be disciplined with email and troubleshooting by checking 3 times a day.” The distillation of the three key factors resulted from the effort to determine the criticality of the online tutor capabilities. The relationship between each of the 13 environmental factors and each of the 24 sub-capabilities was explored to determine the strength of each relationship. There was found to be different strengths of relationships based upon the mediation effect of the learning environment in which the relationship took place. This mediation affected each relationship as no factor and sub-capability was found to have a relationship outside of the learning environment. The exploration of the relationship between the factors and the capabilities resulted in the identification of the critical sub-capabilities of online tutors. Content Expertise – Knowledge and skills, and Enriching interactions Course Management – Management, and Administration Evaluation – Assessment, and Feedback Process Facilitation – Communication, Confidence, Disposition, and Values Technical Knowledge – Attitude, Choice of resources, Technical pedagogy, and Technical support Examples of the criticality of the sub-capabilities were evident throughout the study. For example, Content Expertise - Knowledge and skills was critical in Catherine’s unit as she tutored in a very detailed, data driven field in which the precise scaffolding of knowledge was critical. Content Expertise - Enriching interactions was critical in Benny’s unit as the strong student individuality led Benny to act complimentary to the unit

312

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

materials. Course Management – Management was critical in Margaret’s unit as she worked within the design of her unit and managed the unit as it was designed. Course Management - Administration was critical in William’s unit due to the amount of interaction the unit had with the organizations and structures of the institution. Evaluation - Assessment was critical in Margaret’s unit due to the institutional structure which required her to return some assignments before others had been received. Evaluation - Feedback was critical in AC’s unit with students submitting biweekly assignments and needs verbose feedback for the consumption of the entire class. Process Facilitation - Communication was critical in AC’s unit given the large amount of interaction. Process Facilitation Confidence was critical in Benny’s unit as he changed his preconceived notions about online education and accepted the role to students’ wanted him to fulfil. Process Facilitation - Disposition was critical in William’s unit as he was frustrated by the lack of interaction between students and the tutor but did not let this turn the tutoring experience into a disappointing one. Process Facilitation - Values was critical in AC’s unit as her unit had a wide cultural demographic of students with a range of beliefs on many topics. Technical Knowledge- Attitude was critical in Lauchlin’s unit as he had to content with a challenging technology situation and he wanted to keep the students positively engaged with the process. Technical Knowledge - Choice of resources was critical in Margaret’s unit as she avoided poor resources to concentrate on positive outcomes. Technical Knowledge - Technical pedagogy was critical in Catherine’s unit as she tutored large group online lecture sessions. Technical Knowledge - Technical support was critical in Lauchlin’s unit as access as well as student understanding of technology was challenging. Three factors were found to modify the essence of the online tutor capabilities: Student Expectations, Interaction student/tutor, Content Issues. An example of the effect of Student Expectations was that students who were apparently dissatisfied with their online unit and expressed this publicly, required tutors’ to have essentially different capabilities than students who solely required assistance with the content of the unit. Several students were quite aggressive and vocal in their expressed opinions and Lauchlin commented that, “Not much you can do about it if students give you pounding.” Other tutors, such as AC and Benny were never in a position to receive a pounding from the students because of the expectations of their students. An example of the effect of Interaction student/tutor was that there was a difference between units in how the sub-capability of ‘communication’ was achieved. William did not have access to his students’ email addresses and commented, “lack of student contact was frustrating, asking students to contact you and they don’t contact you.” On the other extreme, Catherine turned off her answering machine as a way to force students to use email as she “didn’t like the demands of the phone.” An example of the effect of Content Issues was the manner in which access to unit materials was a central theme in Margaret’s unit as there were materials which students had a great deal of difficulty accessing. AC’s unit was presented in three different formats, online, CDROM and paper-based, and it was not modified during the semester so her content was static throughout. AC did not mention anything about students having access issues with the content created before the semester but her unit was based on co-creation of knowledge. The views of what unit content consisted of modified the essence of the tutor capabilities. It is difficult to find a simple statement to encapsulate the relationships between the factors and the capabilities presented in this study. Each factor affected the capabilities to a greater or lesser extent, depending on the situation. Each situation was very complex involving many factors therefore the mediation involved was complex. This led to a revision of the theoretical framework upon which the study was grounded. The online tutors interacted with the students through online learning environment technologies. As this interaction took place, it became apparent that the capabilities of the tutor had a relationship with the factors that emerged from the environment. The learning

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

313

environment in which the relationships took place had a mediating effect on these relationships. There were different strengths of mediated relationships between each factor and capability. Examples of this include the strong mediated relationship between the capability Technical Knowledge and the factor Technical Milieu. There was a much weaker mediated relationship between the capability Evaluation and the factor Technical Milieu. Factors played different roles in different learning environments presented through the case studies where a factor may be prominent in one and negligible in others. While tutors may have needed to deal with similar factors, they needed vastly different capabilities because of the nature of their learning environments. For example, Benny wanted more community creation and public interaction between students and tutors while AC talked about being overwhelmed by the amount of public interaction her community entailed. These were two facets of the same factors. This uniqueness of performance caused the factors to have varying relationships with the capabilities as the nature of the factors was dependent upon the learning environment they acted within. There were also different perceptions of the mediated relationships between the factors and the capabilities as demonstrated by the groups of stakeholders. The tutors viewed process facilitation and interaction to have the strongest relationship, the unit coordinators had a view indicating the strength of the content expertise and the design / pedagogy of the unit, while the students emphasized the connection between evaluation and interaction between students and tutors. 4.

Conclusions

The purpose of this study was to examine the online learning milieu in an Australian University to identify the capabilities required of online tutors for particular online learning environments. This involved determining the environmental factors affecting online tutor capabilities and what the relationship was between the capabilities and these factors. This was explored through the perceptions of online tutors, students and unit coordinators and through observation of online and face-to-face interactions. There are a number of major outcomes of this study that have implications for educational practice. A primary outcome was the creation of a taxonomy of capabilities and sub-capabilities specifically for online tutors who do not have control of the design of their content. Secondly the study identified and organized the environmental factors which affect online tutor capabilities. This extended to the formation of a paradigm of the mediated relationships between online tutor capabilities and environmental factors which affect them resulting in the creation of a research-based model of the online learning milieu. The final major outcome was addressing the lack of depth in the literature regarding online tutors, their capabilities and the environmental factors which affect those capabilities. The lack of control tutors possess over the design and content make them fundamentally different from ‘face-to-face’ teachers and even online designers. It was this difference which defined the capabilities of the tutors as they were forced to adapt and ‘make do’ with what was available rather than change the design to better suit their skill sets. This gave rise to the importance of the identification and organization of the environmental factors affecting online tutor capabilities. In doing this, the study has provided a better basis for further research to be conducted. Further, this, in conjunction with the richness of the case studies, provides a structure for the professional learning requirements for online tutors. The study led to the formation of a paradigm of the mediated relationships between online tutor capabilities and environmental factors. Each mediated relationship between individual sub-capabilities and factors is complex and has not been addressed in any detail

314

C.P. Newhouse and D. Reid / The Capabilities Required by Online Tutors

in the literature. The mediation in these relationships forms the essence of the tutoring experience. This finding has provided a research-based framework of the online learning milieu. This framework begins to clarify how online education with human tutors functions and will assist unit designers as they produce online courses. The refining of this model through research in the future will further assist educational leaders, designers and tutors. This framework will assist educational leaders in the selection, training, and support of online tutors and provide tutors with a clearer understanding of what they are likely to encounter and the skills they need to develop to successfully support students within an online learning environment. 5. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25]

References Goodyear, P., et al., Competencies for online teaching: a special report. Educational Technology Research and Development, 2001. 1: p. 65-72. Simonson, M., et al., Teaching and learning at a distance: Foundations of distance education. 2 ed. 2003, Columbus, Ohio: Merrill Prentice Hall. Fletcher, M. Embedding expertise for online tutor development. in Australasian Society for Computers in Learning in Tertiary Education 2003. 2003. Adelaide. Cyrs, T., Competence in teaching at a distance. New Directions for Teaching and Learning, 1997. 71: p. 15-18. Levy, S., Six factors to consider when planning online distance learning programs in higher education. Online Journal of Distance Learning Administration, 2003. VI(1). Schoenfeld-Tacher, R. and K. Persichette, Differential skills and competencies required of faculty teaching distance education courses. 2000, International Journal of Educational Technology. Cashion, J. and P. Palmieri, The secret is the teacher: The learner's view of online learning. 2002, Australian National Training Authority: Leabrook, South Australia. p. 130. Furst-Bowe, J., An analysis of the competencies needed by trainers to use computer-based technologies and distance learning systems. Performance Improvement Quarterly, 1996. 9(4): p. 5778. Gustaffson, P. and D. Gibbs, Guiding or hiding? The role of the facilitator in online teaching and learning. Teaching Education, 2000. 11(2): p. 195-210. Berge, Z., The role of the online instructor/facilitator. Educational Technology, 1995. 35(1): p. 2230. Salmon, G., E-Moderating: The key to teaching and learning online. 2000, London: Kogan Page Limited. Kupritz, V., The medium is the message: Implications for teaching in cyberspace. Electronic Journal of Instructional Science and Technology, 1999. 3(2). Spector, J.M. and I. de la Teja, Competencies for online teaching. ERIC Digest, 2001. December: p. 4-5. Phipps, R. and J. Merisotis, What's the difference: A review of contemporary research on the effectiveness of distance learning in higher education. 1999, The Institute for Higher Education policy: Washington, D.C. p. 42. Volery, T., Online education: an exploratory study into success factors. Journal of Educational Computing Research, 2001. 24(1): p. 77-92. Bennett, S., A. Priest, and C. Macpherson, Learning about online learning: An approach to staff development for university teachers. Australian Journal of Educational Technology, 1999. 15(3): p. 207-221. Cooper, On-line courses: Tips for making them work. Electronic Journal of Instructional Science and Technology, 2001. 3(3). White, Students and faculty respond to online distance courses at grant macewan community college. Technological Horizons in Education Journal, 2000. 27(9). Berge, Z., Perceptions of e-moderators about their roles and functions in moderating electronic mailing lists. Distance Education: An International Journal, 2000. 21(1): p. 81-100. Duggleby, J., How to be an online tutor. 2000, Hampshire, England: Cambridge University Press. Kouki, R. and D. Wright, Telelearning via the Internet. 1999, London: Idea Group Publishing. Lamb, A. and W. Smith, Ten facts of life for distance learning courses. TechTrends, 2000. 44(1): p. 12-15. White and B. Weight, The online teaching guide: A handbook of attitudes, strategies, and techniques for the virtual classroom. 2000, Toronto: Allyn and Bacon. LaRue, B. and M. Sobol, The executive master class: Cyberspace and the new frontiers of executive education, in Handbook of online learning, J. Schoenholtz-Read, Editor. 2002, Sage Publications: London. Oliver, R. and M. Grant, Distance education technologies: A review of instructional technologies for distance education and open learning. 1994, Perth: Edith Cowan University.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

315

Growth of Teacher Understanding of Mathematics within a CSCL Environment Rod NASON, Mathew McDOUGALL Centre for Learning Innovation, Queensland University of Technology, Australia [email protected]

Abstract. This study focused on growth of understanding about ratios and fractions by a group of four pre-service teachers engaged in a computer-supported collaborative learning (CSCL) environment. Participation in the CSCL environment was found to facilitate the processes of “folding back” and “thickening” [1] of mathematical concepts and thus contributed to the pre-service teachers making considerable advances in their understanding of ratios and fractions. Keywords: cscl, mathematics education, pre-service teacher education, rational numbers

Introduction Most current pre-service teacher education programs attempt to address pre-service teachers’ limited understandings of mathematical concepts such as ratios and fractions by having them engage in workshops/seminars based on constructivist principles. However, most pre-service teachers find the experience of applying constructivist frameworks to themselves and of actively and reflectively constructing (or in most cases, reconstructing) mathematical knowledge results in much uncertainty, because they have construed their education around assumptions that their professors knew “the truth” and all that the preservice teachers need to do is write it down, study it, and pass a test on it to prove they know it [2]. Many pre-service teachers find that unresolved uncertainty such as that engendered whilst engaged in the process of constructing mathematical knowledge in these workshops/seminars to be a disabling experience [3]. CSCL knowledge-building communities have been proposed as a means to address these feelings of uncertainty that hamper many pre-service teachers’ construction of adequate repertoires of mathematical subject-matter and pedagogical content knowledge [4]. In knowledge-building communities, participants are engaged in the collaborative production of conceptual artefacts (e.g., mathematical models) that can be discussed, tested, compared, hypothetically modified and so forth and the participants see their main job as producing and improving such artefacts, not simply the completion of tasks [5]. In this paper, we trace the growth of understanding about ratios and fractions by a group of pre-service teachers participating over a period of three weeks in a computersupported collaborative learning (CSCL) knowledge-building community [5, 6].

1. Conceptual framework The educational effectiveness of CSCL environments is highly dependent on the quality of tasks [6, 7, 8] and the cognitive scaffolds [8, 9] provided to participants within the CSCL

316

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

environment. Therefore, the major foci of conceptual framework that informed the design of the CSCL environment in this study were on the design of tasks and design of cognitive scaffolds. Knowledge-building discourse is predominantly derived from investigations of real, authentic problems that learners really care about [6]. Scardamalia [6] also has suggested that idea diversity is essential to the development of knowledge advancement - idea diversity creates a rich environment for ideas to evolve into new and more refined forms. Therefore, the design of the task was influenced by Scardamalia’s [6] socio-cognitive principles of: (1) real-world authentic problems (i.e., it should be a real and authentic problem for the pre-service teachers that they would want to invest time and effort in), and (2) diversity of ideas (i.e., it should provide a context that would stimulate a diverse set of ideas from the pre-service teachers). According to Berenson [10] the task of lesson planning provides a ‘real-world’ authentic context for pre-service teachers to investigate and advance their subject-matter and pedagogical content knowledge of mathematics topics such as ratios and fractions. Also, previous research by proponents of Lesson Plan Study has suggested that lesson plan study provides contexts that encourage the presentation and discussion of diverse ideas by pre-service teachers [10]. Therefore, in this study, the task administered to the participants was to design an initial lesson plan for the teaching of fractions and ratios to Year 7 students. It was hypothesised that because this task was consistent with the principles of real-world authentic problems and encouragement of diverse ideas, it would provide a context to engender knowledge-building discourse about the subject-matter and pedagogical content knowledge of ratios and fractions. The design of the scaffolds used in this study was informed by Scardamalia’s sociocognitive principles for knowledge-building classrooms [6], by Nason’s [4] previous research on scaffolding knowledge-building discourse within CSCL environments for preservice teachers, and by Berenson’s [10] research on Lesson Plan Study. The following set of cognitive scaffolds was inserted into Knowledge Forum to facilitate knowledgebuilding discourse: 1) APPROPRIATE PRAISE, 2) HOW YOUR IDEAS HAVE INFLUENCED MY IDEAS, 3) HOW ARE YOUR IDEAS LIKE MY IDEAS, 4) HOW YOUR IDEAS ARE DIFFERENT FROM MY IDEAS, 5) QUESTION (I need to understand), and 6) PROPOSAL (How you can improve your lesson plan). Based on findings from previous research with pre-service teacher computer-mediated knowledge-building communities [e.g., 4], it was hypothesised that the critiquing and the posing of questions about other groups’ lesson plans facilitated by this set of cognitive scaffolds would induce knowledge-building discourse in the following ways: • Encouragement of the presentation and discussion of diverse ideas (consistent with Diversity of Ideas Principle [6]) • Focusing pre-service teachers’ attention on similarities and differences between the lesson plans (consistent with Knowledge Building Discourse Principle [6]) • Encouragement to re-examine key notions imbedded in lesson plans (consistent with Epistemic Agency Principle [6]) • Improvement of lesson plans (consistent with Improvable Ideas and Rise Above Principles [6])

2. Design and Method This study used a teaching experiment methodology [11]. Lesson plan study [10] was the vehicle chosen to frame the teaching experiment. This methodology was extended by

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

317

having the pre-service teachers collaboratively design lesson plans for the teaching of fractions and ratios within the context of a computer-mediated knowledge-building community. The ten participants (all Caucasian females) in the study were volunteers from a cohort of preservice teachers (N=350) enrolled in a four-year Bachelor of Education degree course at an urban Australian university. At an initial meeting at the beginning of the study, the participants were asked to form three small working groups. One group of four participants (Group 1) and two groups of three participants (Groups 2 and 3) were formed. These three self-selected groups were formed on the basis of friendships and the pragmatics of timetabling of the participants’ classes. This paper focuses on the growth in understanding by Group 1. The four members of this group (Karen, Tracey, Liz and Natasha) were in their mid-30s. Karen felt that she had a good understanding of ratios and fractions. However, an assessment of her knowledge about ratios and fractions conducted prior to this study found that her understanding was instrumental and rote in nature. Tracey, Liz and Natasha felt that they had very limited understanding of ratios and fractions. Their perceptions were confirmed in the assessment of their knowledge about ratios and fractions conducted prior to this study.

2.1 Procedure 2.1.1 Phase 1 - Generation of initial group lesson plans Each group was asked to meet twice face-to-face and engage in the development of an initial lesson plan on ratio and fractions for a class of Grade 7 students. These sessions were videotaped for later analysis. To facilitate this process, each group was provided access to a library of mathematics education source books and curriculum materials, teachers’ manuals and teaching journals and to a wide variety of continuous and discrete concrete teaching aids. At the conclusion of these meetings, each group was required to post its lesson plan onto Knowledge Forum ®, a computer-supportive collaborative learning environment. Once the lesson plan had been posted onto Knowledge Forum, it could be viewed by the participants in other groups and by the two expert mathematics educators who acted as mentors to the knowledge-building community.

2.1.2 Phase 2- Evaluation and revision of lesson plans During this phase that occurred over a period of two weeks, each group was engaged in two types of activities: 1) reading and constructive critiquing of other groups’ lesson plans, and 2) iteratively evaluating and revising their own lesson plans. During this phase, as well as reading the other groups’ lesson plans posted on Knowledge Forum, the participants also were encouraged to engage in constructive critiquing of and the generation of explanatory/elaborating questions about the lesson plans. The process of commenting and critiquing was facilitated by the set of cognitive scaffolds inserted in Knowledge Forum. Each group made three revisions to their lesson plan. These revisions were based on the feedback via Knowledge Forum from the participants in other groups. Revisions to the group lesson plans occurred in face-to-face group meetings. In these meetings, the groups reviewed and evaluated the comments they had received from the other groups, referred to source documents such as curriculum guides, text books and teaching guides, and then made revisions to their lesson plan. The revised lesson plan was then posted onto the

318

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

Knowledge Forum shared database to enable further feedback. These face-to-face meetings were videotaped to observe the process of developing the content and structure for the lesson plan.

2.1.3 Phase 3 - Presentation of final group lesson plan At the end of Week 3 of the study, each group was required to submit a final group lesson plan via Knowledge Forum. At the conclusion of this phase, each group of participants engaged in a focus interview during which they were required to reflect on what and they had learnt.

2.2 Analysis of data The primary focus of the analysis of data from videotaped observations, interviews, Knowledge Forum notes and lesson plans was on the growth of knowledge. The model developed by Pirie and Kieren [1] provided the framework for tracing the growth of understanding about ratios and fractions. Pirie and Kieren’s model is one of actions and interactions where growth in understanding occurs through a continual movement back and forth between and among eight layers of understanding activities as the individual reflects on and reconstructs current and previous knowledge [12]. In this study, we focused on the first four layers. The innermost layer is that of primitive knowing consisting of all of one’s previous knowledge and serves as the reservoir from which to build subsequent understanding. Moving outward within the model are the image making and image having layers. Image making is the activity of creating an internalised image, a representation that can be used in place of something that may once have been in the learner’s perceptual field. The result is a state of image having, where the constructed image is accessible and comparisons to it can be made. These two layers of activities play a prominent role in our analysis. The other layer of activities used here is property noticing. Identifying properties of the constructed image defines property noticing [1]. When faced with a problem at any layer that is not immediately solvable, an individual often returns to an inner layer of understanding. This shift to working at an inner layer of understanding actions is termed folding back. It enables the learner to make use of the current outer layer of knowing to inform inner understanding acts, which in turn facilitate further outer layer understanding. Consideration of the concept at a more basic level may result in the thickening of a concept. That is, the construction of a thicker, more robust understanding of the concept that can then be examined from the higher-level perspective. According to Pirie and Kieren [1], folding back and thickening are the primary mechanisms that give rise to a more sophisticated and deeper understanding of a concept. The secondary focus of the analysis of data from the videotapes of the planning sessions, Knowledge Forum database and from the interviews was on the factors that influenced the growth in mathematical knowledge. Therefore, quantitative data about participation in the on-line discourse was collected. As each lesson plan was posted, Knowledge Forum tracked the number of times it was read and by how many people. Knowledge Forum also was used to track the number of times each feedback comment was read and reacted to. The content of each of the comments was qualitatively analysed to note its contribution to the advancement of knowledge. The number of times each scaffold was used was also recorded. In addition to this, the notes produced each time a scaffold

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

319

was utilised were qualitatively analysed in order to ascertain how they contributed to growth in knowledge by the recipients of the comment.

3. Tracing Group 1’s Growth in Understanding In their initial lesson plan, Group 1 focused on the part-to-whole notion of a fraction (i.e., 3/8 means three parts out of eight equal parts) and the part-to-part notion of ratio (i.e., 3:5 means 3 parts to 5 parts). Group 1’s understanding of part-to-whole notion of fractions as reflected in their selection of instructional representations and the questions they utilised with the instructional representations was at Pirie and Kieren’s image having layer of understanding: their understanding was related to the image provided by the continuous pizza model representation of a fraction. Similarly, their understanding of the part-to-part notion of ratio was at the image having layer. However, their understanding of the part-topart notion of ratio was related to the image provided by a discrete set model (toy monkeys and zebras). The group failed to make conceptual linkages between the part-to-part notion of ratio and the part-to-whole notion of fractions. When their initial lesson plan was viewed by pre-service teachers from the other two groups, most of the constructive critique and questions focused on the instructional representations they had chosen. Based on an analysis of the Knowledge Forum on-line discourse, the revisions made to lesson plans, and data from post-study interviews with members of Group 1, it was found that the following thread of Knowledge Forum on-line discourse played a crucial role in advancing Group 1’s understanding of ratios and fractions. Carolyn: [PROPOSAL] To make it easier for children to learn, you should use your part/part example (i.e., toy monkeys and zebras) rather than the pizza model to link the part/part ratio and the part-whole fraction idea Lauren: [PROPOSAL] I agree with your comment. Why don’t you use the monkey and the zebra toys to model the part-part and part-whole ideas?

The on-line comments by Carolyn and Lauren proposing the use the discrete set model rather than the continuous pizza model provided Group 1 with the catalyst to fold back from the image having layer of understanding to the image making layer of understanding (see continuous line in Figure 1). At the image making layer, they were able to investigate the part-to-whole aspects of the discrete model by quantifying not just only how many monkey and zebra toys were in the set (i.e., the parts) but also how many toys were in the whole set. They thus were then able to remake the image of the part-to-whole notion of a fraction to include not only the continuous models (such as pizzas) but also discrete set models (such as toy monkeys and zebras). Thus, instead of their understanding of part-to-whole notion of a fraction being limited to just continuous model images (such as pizzas), the group now also had alternative discrete model images of fractions to complement their discrete model image of the part-to-part ratio notion. The generation of this extended (or what Pirie and Martin [12] refer to as “thickened”) part-to-whole fraction image led the group members’ understanding of both part-to-part ratio and part-to-whole fraction notions to progress from image having to the property noticing layer of understanding (see Figure 1). That is, the group members had extended their understanding to notice the commonalities and differences in properties between the mathematical notions and were now able to make conceptual links between these two fundamental mathematical notions.

320

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

Figure 1 Tracing Group 1’s growth of understanding of fractions and ratio

Their growth in understanding illustrated in Figure 2 was reflected in their final group lesson plan where: (1) discrete set models (dingo and echidna toys and unifix) were used as instructional representations for both part-to-part ratio and part-to-whole fraction notions, and (2) clear two-way linkages were made between part-to-part and part-to-whole fractions and ratio. Part-toWhole Notion of a Fraction

Image

Pizza Diagram

Type of representation

Continuous Model

Property noticing

Part-to-Part Notion of Ratios

Image

Image

Toy monkeys and zebras

Toy monkeys and zebras

Type of Representation

Type of representation

Discrete Model

Discrete Model

Figure 2 Group 1’s Final understanding of ratios and fractions

4. Factors influencing growth of knowledge The analysis of the data revealed that the lesson planning task and the cognitive scaffolding both played crucial roles in the growth of knowledge. Knowledge transforming discourse is central to knowledge building because it is the means through which knowledge is formed, criticised, and amended [6]. Therefore, it is

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

321

important to have most participants actively engage in the knowledge-building activity of the community. An analysis of the Knowledge Forum data clearly indicated that the lesson planning task provided the catalyst for high levels of participation and rich discourse in the Knowledge Forum environment by all participants. On three occasions, a single posting was read over 60 times. This occurred after each of the three groups had posted their initial lesson plans on the Knowledge Forum shared database. The number of different people reading these comments also showed that a majority of the community reviewed and critiqued these lesson plans. The community participants read each of the comments an average of 16.65 times. However, as well as simply being active in the reading and posting of comments, participants need to engage in deep-level critiquing of knowledge objects being constructed. To facilitate the process of building greater depth of understanding within the Knowledge Forum community, this study employed scaffolds to promote community discourse and cognitively prompt the participants in developing teacher knowledge through critiquing of the lesson plans. The analysis of the data indicates that the cognitive scaffolds facilitated discourse leading to in-depth analysis of important issues pertaining to the teaching and learning of ratios and fractions. The PROPOSAL scaffold was used fourteen times. It was found that comments that utilised this scaffold were crucial in helping each group of participants overcome impasses blocking the advancement of their lesson plans (and their growth of knowledge about ratios and fractions). The scaffold QUESTION was utilised ten time. It was found to help each group to organise and clarify their ideas about fractions and ratios. The scaffolds (HOW YOUR IDEAS HAVE INFLUENCED MY IDEAS; HOW YOUR IDEAS ARE LIKE MY IDEAS; and HOW YOUR IDEAS ARE DIFFERENT FROM MY IDEAS) were very effective in interconnecting ideas between the three groups that lead to the best ideas about the teaching of ratios and fractions rising up and coming to the fore as the study progressed.

5. Discussion By the end of this study, the group of four pre-service teachers had developed deeper understandings about ratios and fractions. This was evidenced by: 1) a thickening of their knowledge about the part-to-whole notion of a fraction at the image having layer from being one based on the continuous pizza model image to one that was also based on discrete model images too, and 2) the extension of their understandings about the part-towhole notion of a fraction and the part-to-part notion of ratios beyond the image having layer to the property noticing layer. Thus, by the end of the study, this group of pre-service teachers not only had richer, thickened conceptions about both the part-to-part notion of ratio and the part-to-whole notion of a fraction but also were able to conceptually link these two notions. The analysis of the data with respect to factors that underlay the growth in knowledge indicated that the Knowledge Forum discourse mediated by the lesson planning task and the set of cognitive scaffolds played a crucial role in facilitating the growth in their understanding. Previous CSCL research [6, 7] has highlighted the importance of providing participants in CSCL communities with rich authentic tasks that encourage discourse. The lesson planning task provided to the participants in this study situated the learning about ratios and fractions in a context that was authentic and meaningful for the pre-service teachers. This enabled the participants to relate ratios and fractions to their own previous experiences and to utilise their combined knowledge to engage in discourse about real-

322

R. Nason and M. McDougall / Growth of Teacher Understanding of Mathematics

world exemplars and instructional representations of ratios and fractions. Ironically, their limited but diverse repertoires of knowledge about ratios and fractions that they brought to this lesson planning task probably was one of the major factors that led to this particular task being so successful in acting as a catalyst for the knowledge-building discourse. This is consistent with Scardamalia’s [6] viewpoint that diversity of ideas is an important condition necessary (but not sufficient) for knowledge-building discourse. The implications of this for future studies involving lesson planning study within CSCL knowledge-building communities is that the topic(s) chosen for lesson planning should be complex, multifaceted and interrelated in nature (like ratios and fractions) and ones in which the participants have limited repertoires of formal knowledge but diverse repertoires of real-world exemplars. Previous CSCL research [8, 9] also has highlighted the importance of providing effective cognitive scaffolding within CSCL environments. The findings from this study confirm these previous findings. In this study, the scaffolds focused primarily structuring the preservice teachers’ endeavours to construct their lesson plans. However, Reiser [13] contends knowledge-building discourse maybe could also advanced by provoking issues with students where they are forced to confront key disciplinary ideas in their solutions to a problem. Therefore, in future studies involving preservice teachers engaged in lesson planning tasks within a CSCL environment, knowledge-building discourse possibly could be further enhanced by including what Reiser refers to as problematising scaffolds as well as structuring scaffolds.

References [1] Pirie, S., & Kieren, T. (1994). Growth of mathematical understanding: How can we characterise it and how can we represent it? Educational Studies in Mathematics, 26, 165-190. [2] Lampert, M. (1988). What can research on teacher education tell us about improving quality in mathematics education? Teaching and Teacher Education, 4(2), 157-170 [3] Floden, R. E., & Buchmann, M. (1993). Between routines and anarchy: Preparing teachers for uncertainty. Oxford Review of Education, 19(3), 373-382. [4] Brett, C., Nason, R.A., & Woodruff, E. (2002). Communities of inquiry among pre-service teachers investigating mathematics. THEMES in Education 3(1), 39-62. [5] Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah, NJ: Erlbaum [6] Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith (Ed.), Liberal education in a knowledge society (pp. 67-98). Chicago: Open Court. [7] Chaney-Cullen, T., & Duffy, T. M. (1999). Strategic teaching framework: Multimedia to support teacher change. Journal of the Learning Sciences, 8(1), 1-40. [8] Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences, 3(3), 265-283. [9] Weinberger, A., Fischer, F., & Mandl, H. (2002, January). Fostering computer supported collaborative learning with cooperation scripts and scaffolds. Paper presented at the annual conference on Computer Support for Collaborative Learning (CSCL) 2002, Boulder, USA. [10] Berenson, S. (2003, April). Lesson plan study: A viable pedagogy for teacher educators. Paper presented at the annual meeting of the American Education Research Association, Chicago, IL. [11] Steffe, L. P. (1991). The constructivist teaching experiment: Illustrations and implications. In E.v. Glaserfeld (Ed.), Radical constructivism in mathematics education (pp. 177-194). Dordrecht, The Netherlands: Kluwer. [12] Pirie, S., & Martin, L. (2000). The role of collecting in the growth of mathematical understanding. Mathematics Education Research Journal, 24(2), 127-146. [13] Reiser, B. J. (2002, January). Why scaffolding should sometimes make tasks more difficult for learners. Paper presented at the annual conference on Computer Support for Collaborative Learning (CSCL) 2002, Boulder, CO.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

323

Are Student Teachers Ready for E-Learning? Eugenia M. W. NG The Hong Kong Institute of Education, Hong Kong SAR, China [email protected]

Abstract. There are many articles about different strategies and techniques of electronic learning but there are not many findings about student teachers’ readiness in this direction. A questionnaire was developed by adapting Hiltz’s framework to find out student teachers’ perceptions on different areas which contribute to the success of electronic learning. There were a total of 64 responses and the respondents gave high ratings on most items asked. Their perceptions and preferences on E-learning clearly indicated their readiness to go for electronic learning partially or completely. The findings are useful not only to our institute but also inform others who are in the area of teacher education. Keywords: E-learning, student teachers, perceptions, preferences

1. Introduction Internet access is not restricted by distance nor time and its flexibility has begun to have a huge influence on education [1]. Most universities in developed countries have invested large sums of money in different electronic learning (E-learning) initiatives [2]. They realize that the traditional classroom setting, where the educators and learners meet at the same time and place is insufficient to meet the modern demand for education [3]. It was suggested that academics who used E-learning may become better teachers as they may reflect on their teaching and also initiate more dialogue with other teachers on the potential benefits of various teaching methods [4]. Myers, Bennett, Brown & Henderson [4] also found that “the largest effect on a faculty’s perception of online learning environments occurs when faculty’s reason for teaching the course is to develop new teaching skills” (p. 83). “Educational technologies may open new avenues for more students to access opportunities and information, increase forms of interactions among teachers and students and encourage collaboration across institutions” (p. 84). The Hong Kong Institute of Education (HKIEd) realized the potential of using information technology to assist learning and has supported many projects related to web-based learning in the past few years. It has adopted a popular learning platform, namely Blackboard, to facilitate this new mode of learning. Although there are many articles on how to measure the success of e-learning at different higher educational institutes [5, 6] there are very few articles on what kind of factors affecting E-learning for teacher education. It is understood that it might not be feasible or desirable to replace face-to-face lectures with e-learning mode completely as presentation and communication skills are considered essential attributes of teachers. However, given the positive feedback on using discussion forum to complement face-to-face learning at the HKIEd [7], easy access and high availabilities of computers at schools in Hong Kong [8], perhaps it is timely to explore what

324

E.M.W. Ng / Are Student Teachers Ready for E-Learning?

factors affect the success of E-learning implementation and also to examine if student teachers are ready to take some modules completely online. 2. A Framework of E-learning The success of E-Learning can be measured from both students’ and faculty members’ perspectives by using educational outcomes as indicators. To align with the educational reforms, learners should be prepared to develop “how to learn” skill and other generic skills and should focus more on problem solving skills rather than just on the content acquisitions [9]. Hiltz [10] identified four major approaches, namely, technological determinism, the social-psychological approach, human relations school and interactionist perspective, which affected the acceptance and diffusion of information technology and its impacts on society. Details of these approaches and their relationships with E-learning are delineated below: Technological determinism - The efficiency and effectiveness of the system design and implementation will directly influence user behaviour [11]. The resources such as hardware, software, and communication technologies could be important factors affecting user accessibility and reactions to particular aspects of E-Learning. Social-psychological approach - The attitudes and capabilities of educators and learners when using information technology can be grouped as: 1) the perceived value of computer-based information, and 2) computer literacy. Both factors are accepted as important factors influencing the success of E-Learning [10, 12-14] . Human relations school - The relationships and interactions among educators and students could be a significant facilitator or inhibitor to teaching and learning in E-Learning mode. Implementing E-Learning successfully requires task interdependence in which agreement and collaboration of members are necessary and essential. It is imperative that members are comfortable with open discussion of and decision making related to E-Learning [10, 15]. Interactionist perspective - The success of E-learning depends on the interactivity among elements mentioned above and they are influenced by information culture and national culture. Information culture refers to the characteristics of administrators, learners and educators in the use of collaborative information technology for knowledge sharing [14, 16]. National culture refers to “the collective programming of the mind that distinguishes the members of one group or category of people from another” [17]. In the learning environment, the interactivity involves the way people think, act and communicate. Draw from different literature, Hiltz [10] came up with a research framework with independent variables as technology, course and student characteristics, the intervening factors are the amount and type of use of virtual classroom and the dependent variable is better learning experiences. A questionnaire was developed based on Hiltz’s [10] research framework. However, various independent variables are reworded so that they are more appropriate to the learning environment under studied, they are: system resources, computer literacy, and perceived value of computer-based information, group dynamics and learning culture. Since it would be very difficult to measure human learning process and outcomes, the effectiveness of outcomes could be measured through the perception of students and their preferences [10, 18, 19]. Thus, the dependant variables for the success of E-Learning could be interpreted in terms of perception and preferences. Two sets of hypotheses are formulated as follows: 1(a): System resources are positively related to student teachers’ perceptions of E-Learning for teacher education.

E.M.W. Ng / Are Student Teachers Ready for E-Learning?

325

1(b): Computer literacy is positively related to student teachers’ perceptions of E-Learning for teacher education. 1(c): Perceived value of computer-based information is positively related to student teachers’ perceptions of E-Learning for teacher education. 1(d): Group dynamics are positively related to student teachers’ perceptions of E-Learning for teacher education. 1(e): Learning culture is positively related to student teachers’ perceptions of E-Learning for teacher education. 2(a): System resources are positively related to student teachers’ preferences of E-Learning for teacher education. 2(b): Computer literacy is positively related to student teachers’ preferences of E-Learning for teacher education. 2(c): Perceived value of computer-based information is positively related to student teachers’ preferences of E-Learning for teacher education. 2(d): Group dynamics are positively related to student teachers’ preferences of E-Learning for teacher education. 2(e): Learning culture is positively related to student teachers’ preferences of E-Learning for teacher education.

3. The Study There were seventy participants who were pre-service student teachers studying at the HKIEd. They took a module called Information Technology Supported Learning Environment (ITSLE) either as one of their Information Technology (IT) minor studies module or as an elective. The author perceived the concept of ITSLE would be best learnt by modelling the concept. The module material, discussion forum and announcements were put together on Blackboard to facilitate learner-cantered learning. To reflect the importance of active learning, 20% of the total participations were on student participations and half of them were on their online participations. Although we adopt face-to-face approach at HKIEd, one lesson was conducted completely online so that learners had an authentic E-learning experience. They were required to learn how to use software which had multi-media instructions and related resources on the web site. They were asked to do a task in group and which were to be submitted in two weeks time as evidence of their E-learning.

4. Findings and Discussion Student teachers were asked to fill in an online questionnaire during their break time towards the end of the semester. There were a total of 64 responses with a response rate of 91%. Table 1 shows that participants were mostly positive on the items asked as most items had more than half as “agree” or “strongly agree” answers. They gave the highest rating to the question 3.3 - usefulness of computer-based information (98.44%), followed by 6.1 - the course materials are better structured when made online (95.31%) and 5.3 - HKIEd provides a supportive environment such as rules, system or daily environments to learn on-line (93.73%). In fact, question 1- easy to get access to computer facilities outside campus had the highest strongly agree answer. This implies student teachers have good connected computer facilities at home.

326

E.M.W. Ng / Are Student Teachers Ready for E-Learning?

They gave the lowest rating to question 4.4 - if my views are different from others in a group, I am reluctant to express it out (26.56%), 4.1 - to express my views or ideas that contradict what instructors say is not comfortable (35.64%) and 5.1 - I prefer to take some modules completely delivered online (46.88%). These three items had low ratings but the results did not suggest our respondents were negative about E-learning. Indeed, it was quite the opposite. Their responses indicated that they were a group of respondents who were mature in thoughts and they were not afraid not to conform to the norms. However, they were ambivalent about replacing face-to-face learning (question 7.1, 46.88%) or taking some module completely online (question 7.2, 54.60%). The plausible reason was that they did not enjoy reading academic material on-line as much as listening to instructors in classrooms (question 5.1, 48.44%) even though they believed that the course materials are better structured when made online (question 6.1, 95.31%) and they are not completely confidence on their own learning as they still need some instructions from instructors rather than control my own learning process (question 4.3, 92.19%). Table 1. Results of the Questionnaire 1. System resources

1.1 Easy to get access to computer facilities outside campus

1.2 I have an advance and up-to-date software to facilitate my learning. 2. Computer literacy 2.1 Easy for me to use computer 2.2 My level of computer competency is good enough to access different materials

3. Perceived value of computer-based information 3.1 Stimulating 3.2 Efficient 3.2 Useful 4. Group Dynamics 4.1 To express my views or ideas that contradict what instructors say is not comfortable 4.2 I am more likely to accept lectures’ instruction rather than questioning them.

4.3 I still need some instructions from instructors rather than control my own learning process 4.4 If my views are different from others in a group, I am reluctant to express it out. 4.5 I am more likely to discuss assignments with my friends rather than tackling them alone. 5. Learning culture

S. Disagree

Disagree

Agree

S. Agree

0.00%

9.38%

53.13%

37.50%

3.13%

15.63%

73.44%

7.81%

0.00%

40.63%

54.69%

4.69%

1.56%

12.50%

75%

10.94%

0.00% 0.00% 0.00%

18.75% 7.81% 1.56%

75% 76.56% 79.69%

6.25% 15.63% 18.75%

1.56%

62.50%

34.38%

1.56%

0.00%

25%

70.31%

4.69%

0.00%

7.81%

82.81%

9.38%

6.25%

67.19%

20.31%

6.25%

0.00%

31.25%

56.25%

12.50%

327

E.M.W. Ng / Are Student Teachers Ready for E-Learning?

5.1 I enjoy reading academic material on-line rather than listening to instructors in classrooms. 5.2 I like to take part in discussion about my learning with instructors or other students by using e-mail or discussion board rather than doing the work alone. 5.3 HKIEd provides a supportive environment such as rules, system or daily environments to learn on-line. 6. Perceptions

3.13%

48.44%

45.31%

3.13%

1.56%

28.13%

64.06%

6.25%

0.00%

6.25%

89.06%

4.69%

6.1 The course materials are better structured when made online

1.56%

3.13%

89.06%

6.25%

6.2 The course content is easier to understand when made online

1.56%

17.19%

70.31%

10.94%

1.56%

10.94%

73.44%

14.06%

7.1 I prefer to have more E-Learning “lecture” to replace face-to-face learning

6.25%

39.06%

46.88%

7.81%

7.2 I prefer to take some modules completely delivered online

6.25%

46.88%

42.19%

4.69%

6.3 The course content is better illustrated using different digital formats 7. Preferences

Pearson Correlation was used to test the strength between two variables for the first set of hypotheses under investigation. It was found that only hypothesis 1(e): Learning culture is positively related to student teachers’ perceptions of E-Learning for teacher education is accepted in which the Pearson Correlation was significant at 0.05 level. This means that learning culture is significantly influence student teachers’ perceptions of E-Learning for teacher education. Learning culture has the highest Pearson Correlation (0.310) which means the participants’ perception of E-Learning is most dependent on the learning culture. The relationships between student teachers’ perceptions of E-Learning for teacher education and the five independent variables, actually, were not very strong since none of the five independent variables could meet the significant level of 0.01. The second set of variables were explored and we could see that the hypotheses 2(b): Computer literacy is positively related to student teachers’ preferences of E-Learning for teacher education and 2(e): Learning culture is positively related to student teachers’ preferences of E-Learning for teacher education were significant at 0.05 and 0.01 levels respectively. This means that the independent variables of computer literacy and learning culture are significantly influence student teachers’ preferences of E-Learning for teacher education. The variable of learning culture has the highest Pearson Correlation (0.558) which means student teachers’ preferences of E-Learning for teacher education depended on the learning culture the most whereas computer literacy was the second most important factor which affected their preferences of E-Learning for teacher education with the Pearson Correlation of 0.316.

328

E.M.W. Ng / Are Student Teachers Ready for E-Learning?

5. Conclusion and Implications This research provided a first step in exploring the different factors affecting the successful implementation of E-learning for teacher education. Student teachers were positive on most of the items asked which showed that they were ready to learn equally well if not better via E-learning under the current environment. They were also prepared to take some modules online which gave us the confidence to go in that direction. A number of factors were not found to have any significant impact on the perceptions nor the preferences of E-learning. It was encouraging to know that the independent variable of learning culture is the most significant factor and computer literacy was regarded as the second most important factor which will affect the successful of E-Learning for teacher education. There are two implications; the first implication is that the academics and HKIEd have provided a conducive environment for learners to go for E-learning. Another implication is that student teachers were well prepared to study online as they did not encounter any problems regarding systems resources, individual differences and group dynamics. However, we have to be mindful that this was an exploratory study with a group of student teachers who were either minor in information technology or having an interest in this area so that it is very likely the respondents had more positive attitude towards E-learning. The findings provided quantitative information from student teachers’ perspectives and it should be complemented by focus group meetings to search for a pattern of inter-relationship between different variables. It is necessary to have a holistic view for any forward planning. It is hoped that we can have focus group meetings with different categories of colleagues such as academics, administrators and information technology supporting staff. The focus of the meetings will be slightly different. Learners and academics will be asked on the pedagogical aspects, administrators will be asked for the future direction whilst the supporting colleagues will be asked on the technical aspects. The triangulation findings will enable us to gain an in-depth and better understanding of different perspectives of E-learning so that our student teachers are better prepared towards this new mode of learning (and teaching). The results would be more comprehensive if we could conduct some cross culture comparison with findings from other countries.

Acknowledgements The author would like to acknowledge Janice Burn and Nalinee Thongprasert for inspiring her to this research area and participants and colleagues who constructed knowledge with her when conducting this module. Special thanks must go to her research assistants, Lee Lap Piu, Lee Wing Kin and To Pak Ho who helped her in different areas of the research. The research funding from the Hong Kong Institute is also gratefully acknowledged (Project code: 1-4001-16-A377). References [1] [2] [3] [4]

Gaspar, R.F. and T.D. Thompson, Current trends in distance education. Journal of Interactive Instruction Development, 1995. 8(2): p. 21-27. Alexander, S., E-learning developments and experiences. Education & Training, 2001. 4/5(43): p. 240-248. Peraya, D., Distance Education and the WWW. 2005. Myers, C.B., et al., Emerging online learning environments and student learning: An analysis of faculty perceptions. Educational Technology & Society, 2004. 7(1): p. 78-86.

E.M.W. Ng / Are Student Teachers Ready for E-Learning?

[5]

[6] [7] [8]

[9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]

329

Hayashi, A., et al., The role of social presence and moderating role of computer self efficacy in predicting the continuance usage of E-learning systems. Journal of Information Systems Education, Summer 2004. 15(2): p. 139-154. Tham, C.M. and J.M. Werner, Designing and Evaluating E-learning in Higher Education: A Review and Recommendations. Journal of Leadership and Organizational Studies, 2005. 11(2): p. 15-25. Ng, E.M.W., Enhancing Flexible and Collaborative Learning for Preservice Teachers through a Web-based Learning System. Journal of Quality School Education, 2002. 2(53-63). Leung, C.K., T.Y.V. Ng, and S.F.A. Tam, Overall Study on Reviewing the Progress and Evaluating the Information Technology in Education (ITEd) Projects 1998/2003 - Final Report. 2005, Education and Manpower Bureau: Hong Kong. Seng, L.C. and S. Al-Hawamdeh, New mode of course delivery for Virtual Classroom. Aslib Proceedings, 2001. 53(6): p. 238-242. Hiltz, S.R., The virtual classroom: Learning without Limits via Computer Networks. 1994, Norwood, NJ.: Ablex. Mowshowitz, A., Virtual organisation. Communication of the ACM, 1997. 40(9): p. 30-37. Larson, M.R. and R. Bruning, Participant perceptions of a collaborative satellite-based mathematics course. American Journal of Distance Education, 1996. 10(1): p. 6-22. McCollum, K., A professor divides his class in two to test value of online instruction. Chronicle of Higher Education. 1997. 43(24): p. 23. Jarvenpaa, S.L. and D.S. Staples, The use of collaborative electronic media for information sharing: an exploratory study of determinants. Journal of Strategic Information System, 2000. 9: p. 129-154. Claver, E., et al., The performance of information systems through organizational culture. Information Technology and People, 2001. 14(3): p. 247-260. Davenport, T.H., Information Ecology. 1997, Oxford: Oxford university press. Hofstede, G., Cultures Consequences : Comparing Values, Behaviors, Institutions, and Organisations Across Nations. 2001, Thouson Oaks, London, New Delhi: Sage Publications. Hiltz, S.R., The virtual classroom: Learning without Limits via Computer Networks. 1994, Norwood, NJ.: Ablex. Alexander, S. and J. Mckenzie, An Evaluation of Information Technology Projects in University Learning. in Department of Employment, Education and Training and Youth Affairs,. 1998: Australian Government Publishing Services, Canberra.

330

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Learning Designs and Learning Objects: Where Pedagogy Meets Technology Ron OLIVERa, Ralph WIRSKIb, Lisa WAITb, Vivienne BLANKSBYb a Edith Cowan University, Australia b e-works, Australia [email protected].,au Abstract. This paper discusses an Australian project where online learning materials with problem-based learning designs are being developed in a form which facilitates their inclusion in digital repositories and learning management systems. To that end the products are being developed as properly formed SCORM compliant IMS content packages. This paper discusses issues associated with the design of online resources characterised by quality learning designs and their subsequent redevelopment as IMS content packages.

1. Learning Objects The term learning object is one of the current buzzwords within the learning technology community and its use proliferates in much of the current discussion. This is quite an intriguing action given that so few people actually agree on what precisely constitutes a learning object. Whilst few people agree on what a learning object looks like, most agree with the concept as a solution for the current problems with discovery, reusability and interoperability of learning resources. Among the many definitions of learning objects we read, are: • self-contained modules that can support some discrete forms of learning, • any entity, digital or non-digital, which can be used, re-used or referenced during technology supported learning [1]; • digital resources that can be used for any learning purpose [2]; and • a small learning module which serves a learning objective [3]. The major problem which seems to exist in all of this discussion is the reference to learning. It is clear that within themselves, the conception and description of these entities are related more to technical details and system architectures eg. size, form etc. than any learning attribute [4]. The current specification and standardisation associated with learning objects stem mainly from technical and development specifications with learning features noticeably absent. The apparent disconnection between most of the current work from learning issues appears to come from a number of factors in current work: • The notion of learning objects draws mainly from needs associated with reuse, interoperability and repackaging; • The desire to remove context from the content of objects improves reusability but limits inherent learning potential;

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

331

• Learning objects tend to be inadequately defined and relate to information and content presentation without any particular learning support; • There is uncertainty and doubt in issues concerning the scope and size of learning objects [5]; • The researchers developing technical standards often display basic misunderstandings of contemporary conceptions of meaningful learning [6]; and • The activity tends to be yet another example of technology rather than pedagogy leading the research agenda. A number of researchers have attempted to create learning objects that satisfy the reusability requirement and that are also pedagogically rich. In such instances, the developers set very firm guidelines concerning size, form, and assessment [7]. Learning objects in these instances take the form of discrete elements able to support very targeted learning outcomes eg. rules, facts, procedures. With such learning objects, researchers are anxious to remind users that the use of these objects in a meaningful learning environment is still very dependent on the learning design associated with their use. Even in this very precise form, the objects themselves are still dependent on other factors in order to be seen as components of a meaningful learning environment. 2. Pedagogy and Learning Objects Contemporary understandings of meaningful learning environments suggest that the forms of learning design that best support learning are those which are based on some type of problem-solving activity on the part of the learner. Contemporary conceptions of learning have been formed from the research into such learning areas as activity theory, situated cognition, authentic learning and social constructivism [8]. The emergence of instructional technologies coupled with these understandings of learning has led to the growing use of a variety of powerful learning environments across all sectors of education. In these uses there are many effective learning designs apparent including collaborative learning, problem-based learning, role-playing etc. [9]. These settings are ripe for the application of learning objects but often, it is these settings which have the least potential use for the objects that are derived from current activity. Much of the learning object activity is leading to the development of resources suited to learning designs that differ widely from those supported by contemporary understandings. There have been a number of attempts to marry contemporary work with learning designs to that occurring with learning objects. Contemporary learning environments tend to be very resource rich and rely on access to a variety of digital learning materials. The literature describes various attempts to establish repositories of learning resources eg. [10], [11]. Digital repositories tend to provide a means for teachers to discover and access learning resources but these resources often fall short of contemporary views of learning objects since they typically comprise materials for learning taken directly from their local contexts. True learning objects, as defined by SCORM conformance, contain many more features and properties than these resources as a consequence of their ability to track learning and support discovery, reuse and interoperability. This paper describes a large project in Australia which has been involved in the design and development of large scale online learning materials and which has used a number of strategies to support the reusability of the learning resources within. This project, the Flexible LearningToolbox project, has been characterised by the use of quality

332

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

learning designs coupled with deliberate strategies to facilitate the discovery and access of resources [12]. 3. The Flexible Learning Toolbox Project The Flexible Learning Toolbox Project commenced in 1997 and has continued as an ongoing series in which online resources for national curricula in the vocational education and training sector (VET) have been developed by teams throughout the country. Descriptions of the resources can be accessed at http://www.flexiblelearning.net.au/toolbox. Across the past 8 years over 90 separate Toolboxes have been developed, each representing around 400 hours of quality online learning [13]. A typical Toolbox contains complete online learning resources for a series of discrete modules that comprise a larger qualification, for example a Certificate or Advanced Diploma. The resources are created to specifications that require metadata be applied to the various resources and over time, have moved to IMS and SCORM conformance. Despite the fact that traditionally SCORM standards have led to the use of narrow learning designs, in the Toolbox project, the standards and specifications have been successfully applied to strong learning designs based on student-centred models of learning. 3.1 Early Stages The early stages of the Toolbox project were undertaken at a time when the notions of reusable learning objects had not yet been formalised. Our developers were encouraged to consider reuse through the application of metadata descriptions to all resources and the use of learning designs where instructional elements were separated from the underpinning content and course information. The instructional designers were encouraged to use activity and problem-based learning designs which required learners to access information as part of a problem-solving process usually en-route to developing a product or artefact. This approach was intended to create resources with strong potential for reuse across a broad set of contexts. The Toolbox resources that were developed were all described with metadata and the development processes enabled the large numbers of resources to be disaggregated. The various resources however were all marketed and delivered within their original contexts and there was little evidence of teachers and trainers using or resusing any of the resources. 3.2 . The Digital Repository In 2002 a project was undertaken to build a repository that would facilitate the organised storage and discovery of the vast number of discrete resources that had been produced to date. The development of a central store of the resources from the Flexible Learning Toolbox Project was an activity that sought to value-add to a very powerful set of learning resources [14]. The Digital Repository Project sought to develop a prototypical form of a repository that could be used to explore issues associated with the reuse of resources from the Toolbox project After initial explorations and inquiries, it was decided that an appropriate strategy would be to develop a system whereby the resources could be stored in one location and to compile a database from the metadata contained in each. While the use of metadata alone is not sufficient for resource reusability, metadata tags do allow for the location of resources and since the resources within the Flexible Learning Toolbox Project contained a

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

333

considerable degree of metadata information, this provided a strong support for any system looking to support resource discovery and access. The software architecture that was chosen to implement the digital repository is shown in Figure 1. The repository took nearly 2 years to build and was completed in 2003. The repository had a number of unique features, foremost of which was its ability to facilitate advanced searches on the resources’ metadata descriptors, a shopping cart type selection process of resources, and the download of a zip file to the user containing the selected resources (Figure 2). repository repository (hard drive)

toolbox 1 toolbox 2 toolbox 3

applications

robot

SQL DB (stores metadata) interface

ASP

browser (search, page preview)

Figure 1: Toolbox Digital Repository Architecture

Testing of the Repository revealed a number of issues that potentially limited the utility of the tool and its ability to facilitate the reuse of the digital resources it contained. a. Resources in the Repository In the first pass, the total number of files within the repository was over 130,000. A large portion of these files were ancillary and not be used to convey information to the end user. Of all the files found in the Toolbox repository, 14674 (11%) contained standard metadata. The overwhelming majority of files within the repository were images, with .gif and .jpg file types totalling 78,766 files (nearly 60%). The approach used of storing individual files had led to an inappropriate grain size in many instances.

Figure 2: The Toolbox Digital repository

334

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

b. Metadata integrity The functionality and utility of the Digital Repository relied heavily on the integrity of the metadata supplied by the developers for the individual resources. The quality of the metadata was seen to vary considerably across the resources. With many resources the content descriptors lacked the detail to be able to distinguish items from others. c. Metadata extent There were many files and resources in the various Toolboxes which contained insufficient metadata to be useful. While the products had undergone quite extensive quality assurance processes, it was not possible to view every file and resource and to check that it had appropriate metadata. d. Metadata for media files The metadata in the Toolbox resources had been applied only to the HTML pages. This meant that many graphics, movies, images etc. could not be discovered since the Repository could only discover those resources for which metadata was provided. e. Granularity The Digital Repository returns single pages from keyword searches with accompanying graphics and images. Unfortunately, in its developed form, the system did not have the capability to return collections of related pages as a single entity. f. Stylesheets When pages were downloaded from the Digital Repository and displayed, their appearances differ considerably and it was very clear that different pages have come from different Toolboxes [13]. These issues and others revealed quite quickly that the way the resources had been organised and collected in the Repository was going to limit the capacity of the system to support and promote resource reuse and it became evident that there needed to be a change to the development approach if the resources were really going to be able to serve the 2 planned masters, the original Toolbox context and future reuse in other settings. 3.3 SCORM and Content Packaging Toolboxes Given the unsuitability of the approach we had used with individual resources in the Toolbox Repository, an alternative solution promoting reusability was sought. The Toolboxes all tended to use contemporary learning designs based on knowledge construction [12] and the nature of the SCORM standard had until this point suggested that these resources might be difficult to build with SCORM [15]. As a trial activity to judge how SCORM might be applied in the design and development of Toolboxes, an activity was undertaken to produce a SCORM conformant Content Package using an existing Toolbox, The chosen Toolbox was built around an activity-based learning design and provided a sound model to test the processes by which Toolboxes might be designed and developed using SCORM A sample of the Grange Care Services Toolbox is available via the Internet at http://flexiblelearning.net.au/toolbox/series6/602.htm. The first noticeable aspect of the Grange Care Services Toolbox is the front page which is a Flash animation. The Flash animation allows the learner to enter either the home care or residential services components of the toolbox. Here, we immediately discovered the impact of the SCORM rule that disallows navigation between SCOs. To adhere to this SCORM rule and keep the Flash animation, one would have to treat the entire Toolbox as one SCO. This is not only inelegant but defeats the purpose of SCORM: creating reusable learning objects. To produce a SCORM conformant toolbox, we could remove the navigation contained within the Flash object and transfer the navigation to a manifest file that accesses separate SCOs.

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

335

There were a number of ways to create SCORM conformant content with this Toolbox. For example, it was possible to determine that the Grange Home Care area was one big learning object, and all other resources were assets. This way, the manifest file would contain the index HTML page of Grange Home Care as the only SCO. While this approach may have passed the SCORM conformance test, it did not provide a learning object of a suitable size for reusability. Another approach to building SCORM conformant content is to produce small content packages and combine those to produce a larger content package. Within the Grange Care example each task (nursing duty statements) within the unit could be considered a content package, containing individual tasks as SCOs. Each day of the week could also be a content package, containing sub-manifests that define duty statements. This idea could carry on recursively, incorporating duty statements as a content package, until eventually we encompass all learning content. From a theoretical point of view, this was an ideal solution. By creating content packages at a low level, we would have learning objects that were readily reusable, and because they were a content package they should be also be meaningful. Using the smaller content packages we created a larger content package that could be disaggregated by future users. Unfortunately, under the current definition of SCORM (1.2), the implementation for sub-manifest aggregation was awkward at best. The solution we chose for Grange Care was to create multiple SCOs and join them using the organisation section of the Manifest. Figure 3 shows the division of SCOs for our Content Package. At the lowest level we created a SCO for duty statements. This SCO contained assets that define how to complete a particular duty comprising sequential tasks. This represents the smallest reusable object. We used an individual duty statement, together with others, to form different sets of duty statements, each for a specific set of circumstances.

Figure 3: Multiple SCO layout

The activity to establish whether Toolboxes could easily be made SCORM comformant appeared to be very successful. The end product was a fully conformant product which still retained the student-centred learning design. The activity appeared to demonstrate many opportunities to be derived with little cost in terms of functionality and

336

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

limitations of learning design. The outcome added weight to the general thinking that subsequent Series of the Toolboxes all be developed to be SCORM conformant.

4. Summary and Conclusions All future instantiations of Toolboxes are now being built as SCORM conformant Learning Objects but not SCORM conformant Toolboxes. The 1.2 version of SCORM we were using had some limitations and shortcomings: for example, the technical implementation of the aggregation and disaggregation of packages is not ideal. However, the limitations on SCO sequencing and navigations are enhanced in SCORM 2004. SCORM conformance is not difficult to reach. The major components required are: standard and readily available API functions applied to SCOs; metadata used to describe the learning resources; and a well formed manifest file. As an extension to the Digital Repository project, we have taken a number of the earlier Toolbox projects and created SCORM conformant content packages from them. The revised Digital Repository is now populated with nearly 300 learning objects created from the Toolbox products. Whilst some of the learning objects are quite large in size, they demonstrate that it is possible to create SCORM conformant content packages from existing online resources based on a multitude of different learning designs. Our experiences with retrospectively creating SCORM conformant forms of the Toolboxes have demonstrated that despite our initial misgivings that this standard may have limited our capacity to develop online resources using problem and activity-based learning designs, in the end this was not the case. While the use of this form of learning design is able to be supported by current SCORM specifications, its use in this form does appear to generate larger learning objects than applications with more conventional learning designs. We are now addressing the problem of the grain size of the LOs by exploring instructional design strategies that enable a higher level of disaggregation of reusable resources. In some instances this involves separating instructional elements from informational elements and in others creating more modular forms for the learning resources. Designing the learning setting with reuse in mind can change aspects of the instructional design process. The intention in our further work is to explore strategies that will see us developing learning objects of varying levels of granularity. For example, exploring ways to locate some dependent and reusable resources external to the Learning Object to facilitate its use in other LOs without the need for multiple copies. This will enable the larger units of study which currently are delivered as single learning objects, to be formed from smaller objects (assets) which themselves can be used as discrete entities outside the Toolbox context. When this occurs, we expect the levels of reuse and repurposing of the Toolbox resources to grow rapidly. We are also exploring the packaging of Toolbox resources using SCORM 2004, and simple sequencing procedures.

R. Oliver et al. / Learning Designs and Learning Objects: Where Pedagogy Meets Technology

337

5. References [1] [2]

[3] [4] [5] [6] [7] [8] [9]

[10]

[11] [12]

[13]

[14]

[15]

IEEE LTSC, (2005). Learning Object Metadata Standard. Retrieved Aug 26 2005, from http://ieeeltsc.org/wg12LOM/lomDescription Wiley, D. (2000). Connecting learning objects to instructional design theory: A definition, a metaphor, and a taxonomy. In D. A. Wiley (Ed.), The Instructional Use of Learning Objects: Online Version. Retrieved 15 May, 2003, from http://reusability.org/read/chapters/wiley.doc Rehak, D. & Mason, R. (2003). In A. Littlejohn (Ed.) Reusing online resources: a sustainable approach to e-learning. London: Kogan Page. Laurillard, D. (2002). Can technical standards unlock the pedagogical innovations needed for elearning? Retrieved May 15 2003 from http://www.imsglobal.org/otf/dl02SeptSheffield.pdf Lasseter, M. & Rogers, M. (2004). Creating Flexible E-Learning Through the Use of Learning Objects. Educause Quarterly, 27(4) 72-74. Jonassen, D., & Churchill, D. (2004). Is There a Learning Orientation in Learning Objects?. International Journal on E-Learning 3(2), 32-41. Bradley, C., & Boyle, T. (2004). The Design, Development, and Use of Multimedia Learning Objects. Journal of Educational Multimedia and Hypermedia 13(4), 371-389. Jonassen, M. & Land, S. (2000). Theoretical Foundations of Learning Environments. Mahwah, NJ: Lawrence Erlbaum. Oliver, R. & Harper, B., Hedberg, J., Wills, S., & Agostinho, S. (2002). Exploring strategies to formalise the description of learning designs. In J. Herrington (Eds.) Proceedings of HERDSA. Joondalup: Edith Cowan University. Koppi, A. & Hodgson, L. & Bayly, J. (2000). The often missing but essential component for online learning: A learning resource catalogue. In (Bordeaux & Heller Eds.) Proceedings of Ed-Media International Conference, (pp502-506). Charlottesville: AACE. MERLOT (2000). Multimedia Education Resource for Learning and Online Teaching. Retrieved 14 April 2003 from http://www.merlot.org Oliver, R. & Blanksby, V. (2003). Online learning designs in the training sector. In G.Crisp, D. Thiele, I. Scholten, S. Barker & J. Baron (Eds.) Interact, Integrate, Impact: Proceedings of the 20th Annual Conference of ASCILITE (pp 364-374). Adelaide, ASCILITE. Oliver, R., Towers, S. & Pearl, N. (2000). The ANTA Flexible Toolbox Project: Developing Sustainable and Scalable On-Line Learning Materials for Vocational Education and Training. In J. Bordeau & S. Heller (Eds.) Proceedings of ED-MEDIA 2000. World Conference on Educational Multimedia, Hypermedia and Telecommunications. (pp 820-826). Virginia: Association for the Advancement of Computers in Education. Wirski, R., Brownfield, G. & Oliver, R. Exploring SCORM and the National Flexible Learning Toolboxes. In (R. Atkinson & R.Phillips Eds.) Proceedings of ASCILITE 2004. (pp938-947). Murdoch University, ASCILITE. Kraan, W, and Wilson, S.,(2002), “Dan Rehak: "SCORM is not for everyone"”, CETIS Discussion Article, CETIS, Retrieved January 15 2003, from: http://www.cetis.ac.uk/content/20021002000737

338

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

The Task Sharing Framework: A Generic Approach to Scaffolding Collaboration and Meta-Collaboration in Educational Software Darren PEARCE, Lucinda KERAWALLA, Rose LUCKIN, Nicola YUILL and Amanda HARRIS Ideas Lab, Informatics, University of Sussex, United Kingdom Abstract. This paper introduces a novel and general framework which operationalises the sharing of collaborative tasks between multiple users. In contrast to the majority of existing software used in a collaborative context, software developed under this framework provides each user with their own identical yet independent copy of the task which, by default, only they themselves can manipulate. This represents a departure from traditional turn-taking and dual control collaborative interaction styles and is intended to reduce the domination of one user over others as well as to provide the space for effective collaboration. The visual representation of agreement and disagreement is particularly emphasised since this has the potential to constructively mediate the resolution of collaborative disputes, especially if agreement is required at various points during the task. Under the framework, it is also possible to emulate traditional collaborative interaction styles and, more importantly, dynamically vary between them, thus manipulating the collaborative affordances of the user interface. This is likely to scaffold conversation between the users about how they are working together to complete the task; users would be collaborating about their own collaborative process. The framework therefore holds significant potential to scaffold not only collaboration but also meta-collaboration. Keywords. task sharing framework, collaboration, meta-collaboration, formalisation, operationalisation

1. Introduction When software is used collaboratively, the nature of the user interface is critical. Although, in general, dual control is the favoured style of interaction in CSCL [1], various studies have compared this to other collaborative interaction styles such as turn-taking (e.g. [2,3,4]). With any of these interaction styles, however, it is still possible (and common) for one user to dominate others and often this is detrimental to the quality of the collaborative process.

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

339

In order to address this problem, this paper introduces the Task Sharing Framework, an operationalisation of the sharing of collaborative tasks between multiple users. Using the example task described in Section 2 as a departure point, the core of the framework is presented in Sections 3–6. This is followed by a discussion of the rich collaborative potential afforded by the introduction of this generally-applicable framework into the research area.

2. An Example TSF Task Research in the Riddles Project1 has used tasks in which pairs of children collaborate to illustrate different interpretations of text. As an example, consider the task of illustrating a deliberately vague sentence such as the girl heard the cat by placing a girl and a cat on a background consisting of a tree.2 One child may think that the girl has climbed the tree in order to rescue the cat whereas the other may think that the girl is on the ground, calling to the cat in the tree. Using this example, Figure 1 demonstrates two of the ways in which the project has explored novel interfaces for collaborative tasks. The first, Figure 1a, consists of three panels. The two lower panels display each child’s elements in the child’s own ‘colour’: white for child 1 and diagonal stripes for child 2. The children are only able to move the elements that are within their own panel; they are not able to move their partner’s elements. The shared, third panel shows the elements belonging to both the children and also provides a representation of agreement and disagreement by colouring agreed elements in a distinct colour. Thus the cat is coloured black since both children agree it should be in the tree. On the other hand, the children disagree about the location of the girl so the panel shows her in each location using the relevant child’s colour. A second novel interface that the project has explored is shown in Figure 1b. In contrast to Figure 1a, it does not include a shared panel. Nonetheless, agreement and disagreement are still represented since each child panel itself highlights agreed elements. Thus in both child panels in Figure 1b, the cat is shown in black, the agreement colour. These interfaces have two novel, underlying characteristics in common. Firstly, each child manipulates their own identical yet independent copy of the task. This is designed to reduce the potential domination of one child over another and give them individual agency in the task process. Secondly, they both include the representation of agreement and disagreement. This is essentially automated spot-the-difference (and similarity) between the two pictures. The importance of this feature increases with the complexity of the task since the more complex the task, the more difficult it is to compare manually. 1 All the work described in this paper was funded as part of the Riddles Project (http: //www.riddles.sussex.ac.uk/), EPSRC grant number: GR/R96538/01. 2 This task is based on [5].

340

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

Shared Panel

Child 1’s Panel Child 2’s Panel (a)

Child 1’s Panel Child 2’s Panel (b)

Figure 1. Two novel collaborative interfaces.

3. The Task Sharing Framework Using this example as a departure point, a task under the Task Sharing Framework (TSF) consists of elements which users manipulate in order to reach task completion, however defined. Manipulating these elements will typically entail the manipulation of certain attributes of the elements. This abstraction of the nature of a task is consistent with existing examples of tasks in the literature: for example the alien pattern game discussed in Scott et al. [6], KidPad [3], the Klump [3], the mathematical game ‘Prime Climb’ [4] and the Belv´edere system [7]. Although these and other collaborative tasks can be conceptualised using this abstraction, the crucial difference is that under the framework, each user always has their own copy of the task that typically only they themselves will be able to manipulate. The core of the TSF follows the Model-View-Controller paradigm. A model (Section 4) is the underlying data; a view (Section 5) is a typically graphical representation of some or all of this data; and a controller (Section 6) provides a way in which the user can manipulate the model. Thus the view can be seen as how the model communicates with the user and the controller as how the user communicates with the model (via the view).

4. The Model: Encoding the State of the Task The model encodes the state of the task at any particular time. Of the three core components — model, view and controller — the model is pivotal since it is in the model that all information about the task is stored.

341

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

4.1. Task Representation In order to represent the example task introduced in Section 2, the elements are labelled: e1 for the girl and e2 for the cat (Figure 2a). The three possible locations for these shapes are also labelled: l1 holding the trunk, l2 on the branch and l3 on the ground (Figure 2b). The state of the task, S, can then be represented by a set of elements paired with their corresponding locations.3 For example, for child 1’s panel shown in Figure 1a, the girl (element e1 ) is holding the trunk (location l1 ). This can be represented by the pair e1 , l1 . The cat (element e2 ), however, is on the branch (location l2 ). This corresponds to the pair e2 , l2 . The task state, S, for this configuration then consists of these two pairs as shown in Figure 2c.

(b)

(a) e1

e2

l2

l1

 (c)

S=

e1 , l1  e2 , l2 



l3

Figure 2. (a) Element labels; (b) location labels; (c) the encoding of the task state corresponding to child 1’s panel in Figure 1.

4.2. Task Representation For Multiple Users Where multiple users are concerned, each user has their own copy of the task so each user ui would have their own task state Si . The underlying representation for the task as a whole is now of the form S1 , . . . , SU  where U is the number of users. In the example task, there are two users (U = 2). The task state, S1 , for user u1 (child 1) as in the previous section, currently consists of the two pairs e1 , l1  and e2 , l2 . User u2 (child 2) thinks that the cat should be on the branch (e2 , l2 ) and that the girl (e1 ) should be on the ground (l3 ). Task state S2 therefore corresponds to e1 , l3  and e2 , l2  (Figure 3b). The overall task state is then given by S = S1 , S2  as shown in Figure 3c.     e1 , l1  e1 , l3  (a) S1 = (b) S2 = e2 , l2  e2 , l2  (c)

    e1 , l1  e1 , l3  S = S1 , S2  = , e2 , l2  e2 , l2 

Figure 3. (a) the state S1 ; (b) the state S2 ; (c) the overall task state.

3 In general, however, task state would just consist of a set of elements; location would be an element attribute.

342

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

5. Views: How Users See the Task Graphically A view is responsible for displaying the overall task state graphically. The different example views of the task described in this section emphasise the representation of agreement and disagreement. This has the potential to constructively mediate the resolution of collaborative disputes, especially if agreement is required at various points during the task. In general, a view is conceptualised as a function that takes a set of users, A, that agree on the location of a particular element and returns a colour: either ci , the colour for user ui ; a distinct colour, ca , used for agreement; or  for ‘no colour’. This function is used for each of the element-location pairs in turn.4 A view therefore determines not only how elements should be displayed on the screen but whether they should be displayed at all. 5.1. Personal Views The two child panels at the bottom of Figure 1a are ‘personal views’. In a personal view, P Vi , only the elements belonging to user ui are displayed, all in the user’s colour ci . Thus child 1’s panel is P V1 and child 2’s is P V2 . Such views are defined by:  ci P Vi (A) = 

if ui ∈ A otherwise

5.2. Group Views In contrast to the child panels, the shared panel at the top of Figure 1a is independent of any particular user; it shows all elements of all users, colouring agreement as appropriate. It is defined here by:  ci GV (A) = ca

if A = {ui } otherwise

So if ui is the only user to have their element in a particular location, then the element is displayed in their colour (ci ). If this is not the case then the agreement colour is used. Thus the cat is coloured black (the agreement colour ca ), and the girl is coloured in c1 and c2 at the relevant locations. 5.3. Agreement Views The child panels in Figure 1b show agreement within them (so a shared panel is not needed). As with personal views, these ‘agreement views’, AVi , display only those elements that belong to user ui . However, the elements are coloured differently according to the following function: 4 Formally, A is a notational short-hand for A(e, l) where A(e, l) = {u : e, l ∈ S } and iS i the view function is called for each element-location pair, p ∈ P, where P = i=U i=1 Si .

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

⎧ ⎪ ⎨ca AVi (A) = ci ⎪ ⎩ 

343

if ui ∈ A and |A| = U if ui ∈ A otherwise

so that the agreement colour, ca , is used if all the users agree on the location of an element. This view is in fact a specific case of the more general notion of agreement between some but not necessarily all the users. This more general view can be defined by: ⎧ ⎪ ⎨ca AVim (A) = ci ⎪ ⎩ 

if ui ∈ A and |A| ≥ m if ui ∈ A otherwise

where m (1 < m ≤ U ) is the degree of agreement required between the users before the element is shown in the agreement colour. Using this notation, the child panels in Figure 1b correspond to the views AV12 and AV22 respectively. A further agreement view, AV m , can also be defined. In contrast to the other agreement views developed so far, it is independent of any particular user (which explains the absence of the subscript i). This view shows only those pieces on which m (or more) users agree:  m

AV (A) =

ca 

if |A| ≥ m otherwise

6. Controllers: How Users Manipulate the Task So far, there has been no explanation of how the user manipulates the task; it has been taken as implicit. A controller is conceptualised here in terms of a policy which, for a given a user, u, determines the set of pieces that they can control.5 It is this set of pieces that the user can manipulate in some way. For the purposes of explanation here, this manipulation is movement of that element. However, location can be seen as one of possibly several attributes of an element. In the same way that a personal view, P Vi , shows the elements for a particular user, a personal controller, P Ci , allows the manipulation of the elements for a particular user:  Si P Ci (u) = 

if u = ui otherwise

where  is the empty set. Therefore all the child panels in Figure 1 provide ‘personal control’. 5 The

input device for each user could be (parts of) the keyboard or multiple mice for example.

344

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

Similarly, just as a group view, GV , shows the elements for all users simultaneously on the same panel, a group controller, GC, allows each user to manipulate their own elements simultaneously: GC(ui ) = Si . Finally, as in the case of the shared panel in Figure 1a, it is also possible for a panel not to allow any control by any user. This is the null controller, N C(u) = .

7. Discussion The Task Sharing Framework as described in the previous sections provides a language with which it is possible to reason about and design collaborative tasks. Each of the panels in Figure 1 can now be annotated with one of the views from Section 5 and one of the controllers from Section 6. (It should be noted that the same model is used within each set of panels.) Shared Panel

GV

P V1

P C1

NC

P V2

P C2

User 1’s Panel User 2’s Panel

AV12

P C1 AV22

P C2

User 1’s Panel User 2’s Panel

Figure 4. Annotation of the panels in Figure 1 using the TSF.

Although this specific example has been used throughout the paper, the applicability of the framework remains general: the number of users is arbitrary; collaboration can be distributed or co-present; the notion of agreement between users is flexible; and the abstraction of the nature of a task means that the framework can be used in a multitude of contexts. For example, the TSF could be used in a collaborative brain-storming task similar to the Belv´edere system [7] where rather than constructing a single diagram together, each user constructs their own diagram and is able to view their level of agreement with other users, eventually co-constructing a final agreed solution. The TSF could also be used in collaborative programming. In such a situation, the notion of equality between elements could vary depending on user or system preferences. For example, functions may be required to be completely identical or, less strictly, equal subject to variable renaming. Real-world tasks that follow this framework have been explored empirically: Kerawalla et al. [8] discuss estimation tasks between child-child pairs and parent-

D. Pearce et al. / The Task Sharing Framework: A Generic Approach

345

child pairs. As a distinct alternative to turn-taking or dual control, these TSF tasks are demonstrated to afford individual agency in the task process and to be more effective at avoiding domination and mediating eventual agreement. TSF systems crucially have information about the task activity of each and every user. This carries significant implications for the scaffolding of collaboration since the system can prompt users, highlighting differences between their current task states. The potential for this has been explored empirically and shown to be effective [9]. It is also possible for a system to detect undesirable patterns of interaction such as one user consistently copying another. It could then intervene and prompt appropriately. The flexibility of the framework makes it possible to emulate traditional interaction styles such as turn-taking and dual control. Furthermore, this flexibility means that a TSF system can dynamically vary between interaction styles potentially by using collected information on user interaction patterns. Such variation fundamentally changes the collaborative affordances of the user interface and is likely to lead to conversation between the users about how they are working together to complete the task. In this way, users would be collaborating about their own collaborative process. The framework therefore holds significant potential to scaffold not only collaboration but also meta-collaboration. References [1] M. M¨ uhlenbrock, “Analyzing collaborative learning interactions in shared workspaces,” K¨ unstliche Intelligenz, vol. 1, no. 3, pp. 37–39, 2003. [2] K. Inkpen, J. McGrenere, K. S. Booth, and M. Klawe, “The effect of turn-taking protocols on children’s learning in mouse-driven collaborative environments,” in Proceedings of Graphic Interface ’97, (Kelowna, BC), pp. 138–145, May 1997. [3] S. Benford, B. Bederson, K.-P. ˚ Akesson, V. Bayon, A. Druin, P. Hansson, J. Hourcade, R. Ingram, H. Neale, C. O’Malley, K. Simsarian, D. Stanton, Y. Sundblad, and G. Tax´en, “Designing storytelling technologies to encourage collaboration between young children,” in Proceedings of the SIGCHI conference on human factors in computing systems, (The Hague, The Netherlands), pp. 556–563, 2000. [4] S. Scott, R. Mandryk, and K. Inkpen, “Understanding children’s interactions in synchronous shared environments,” in Proceedings of Computer Supported Collaborative Learning (CSCL)’02, (Bolder, CO, USA), pp. 333–341, January 2002. [5] N. Yuill and T. Joscelyne, “Effects of organisational cues and strategies on good and poor comprehenders’ story understanding,” Journal of Educational Psychology, vol. 80, pp. 152–158, 1988. [6] S. Scott, G. Shoemaker, and K. Inkpen, “Towards seamless support of natural collaborative interactions,” in Proceedings of Graphics Interface (GI)’00, (Montreal), pp. 103–110, May 2000. [7] D. Suthers, “Towards a systematic study of representational guidance for collaborative learning discourse,” Journal of Universal Computer Science, vol. 7, no. 3, 2001. [8] L. Kerawalla, J. O’Connor, D. Pearce, R. Luckin, N. Yuill, and A. Harris, “Scaffolding collaboration: exploration of Separate Control of Shared Space,” in ICCE 2005, (Singapore), 2005. Submitted. [9] J. O’Connor and L. Kerawalla, “Using discussion prompts to scaffold parent-child collaboration around a computer-based activity,” in AIED 2005, (Amsterdam), 2005.

346

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Structuring of Multimedia Contents for E-Learning Lessons in Multimedia Knowledge Base Rafisha RAMLY, Eng-Thiam YEOH, Kai-Li NG Faculty of Information Technology Multimedia University, 63100 Cyberjaya, Selangor D.E Malaysia E-mail: [email protected] Phone: +60-3-8312-5212, Fax: +60-3-8312-5264

Abstract. This paper describes the research on the structure of multimedia contents in the knowledge base of the Multimedia Knowledge Base E-Learning (MKBe-Learning) System [1]. The Object-Oriented Analysis and Design (OOAD) [2] of the MKBeLearning System identified the structure of the knowledge base to represent the lessons created by lecturers and the reusable contents of a subject. This is represented in the Instructional View and Knowledge View of the knowledge base. The initial contents developed for Basic Electronics [3] demonstrates these representations and the analysis of the results is given on the content structure and the categorization of contents in the knowledge base.

1. Introduction The importance of global flow of information has given impact in the education environment due to globalisation in delivering accurate educational resources at anytime and anywhere [4]. In recent years, there are a lot of research on e-learning systems and their contents. The educational contents are made attractive to the learners by applying and creating interactive multimedia contents. The combination of multimedia and educational materials results in the production of multimedia content applications that are interactive, multi-sensory, and visually challenging to the learners [5], [6], [7]. The Multimedia Knowledge Base E-learning (MKBe-Learning) System is a project funded by the Multimedia Research & Development Grant Scheme under the Multimedia Development Corporation (MDC), Malaysia [9]. The main objectives of the project is to develop an integrated environment where lecturers can create reusable multimedia contents in a knowledge base and improve the learning process through multimedia presentations of the educational contents of the knowledge base. As such, it is essential to properly create the representations of knowledge and multimedia contents in the knowledge base. These representations would be the basis for the development of the components in the MKBeLearning System. 2. MKBE-Learning System

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

347

The MKBe-Learning System [1] is an integrated e-learning system consisting of 6 components: ƒIntelligent Interface for student access to multimedia lessons. ƒInference Engine that personalizes the learning process according to the student needs. ƒPresentation Generator that creates the multimedia presentations based on personalization parameters. ƒAuthoring Tool for lecturers to create learning materials to be presented to students. ƒKnowledge Base that stores lessons and reusable concepts. ƒLearning Management System that maintains personal details of students and records of their learning process. The components and their relationships are shown in Figure 1. The lecturer will create the multimedia lessons using the Authoring Tool and store the lessons in the Knowledge Base. The student will access the lessons through the Intelligent Interface, where the Learning Management System will manage the student details and the learning records, Inference Engine will monitor the student activities and infer the personalization parameters and the Presentation Generator will retrieve the contents from the Knowledge Base and present to the student through the Intelligent Interface.

Figure 1: MKBe-Learning System

3. Knowledge Base Model In the MKBe-Learning System, the Knowledge Base stores lessons developed by the lecturer and the reusable concepts for the lessons. The lessons would be developed using the Authoring Tool and retrieved by the Presentation Generator to create the displays to the student. Object-oriented analysis of the use cases involving the Knowledge Base resulted in the class diagram in Figure 2. The classes can be separated into Instructional View [8] and

348

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

Knowledge View. Instructional View represents the multimedia lessons created by the lecturer whereas Knowledge View represents the reusable knowledge concepts of the subject concerned. The Instructional View consists of Subject, Course, Chapter, Lesson, Topic and WebPage classes. The Subject class represents the knowledge domain that the Course class would be a teaching instance of, for example Basic Electronics and Computer Programming. The Course syllabus consists of Chapters where each Chapter contains one or more Topics. The actual teaching of the Course is done through individual learning sessions represented by Lessons. The Lesson class contains Topics and each Topic can be viewed through Webpages. The Webpage class represents the actual presentation to the students, and may have different concept characteristics and media organized by the lecturer. This means that each lesson consists of many webpages prepared by the lecturer to present the topics given in the course syllabus. Some topics may be presented through a few webpages. Instructional View

Knowledge View Concept

Subject SubjectID 0...1 ConceptList CourseList Objectives SubjectName

1

contains 1

1 Consist of 0….*

0….*

Course Chapter

1 Consist of

CourseCode CourseID Pre_requisiteCode CreditHours Objectives URL_Links References Keywords Author SyllabusList LessonList

Consist of 1 0….*

Title Goals CourseID ChapterID Hours Keyword CoreTopics

InitialState Steps Explanation

0….*

UseCase

Related to

Usages CaseStudy Scenario Implementation

0….*

0….*

Explanation

1

Contains

Definition Description Content

Describe through 0….*

Information

Consist of

Fact

CharID CharName CharType

Example Label 1

0….*

Lesson LessonID LessonName CourseID LessonObj StoryboardList

Demo

0….*

Characteristics

1

0….*

Is prerequisite of

ConceptID Name PrerequisiteConceptID RelatedConceptID HasCore HasExample HasDemo HasUseCase HasGeneralInfo ParentID IsComposed SubConcepts

Topic Contains

Title contains Hours 1 ConceptID Author IsComposed SubTopics 0….* Objectives PrerequisiteTopicList RelatedTopicList Task 1

Graphics

Presented by

GraphicsNumber GraphicsType

0….*

Media

Animation

Name MediaType Description Location MediaID FileSize

AnimationNumber AnimationType Duration Audio

Consist of 0….*

WebPage WebID Title Hours Size

Video

Text

VideoNumber VideoType Duration ScreenSize

TextNumber TextType TextFont TextSize

AudioNumber Audiotype Duration

Figure 2: Class Diagram for the Knowledge Base

The structure of contents in the Instructional View was formulated through the study on the concept of Learning Objects [10] [11], the Cisco Reusable Learning Object Model

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

349

[12], the course description in Computing Curricula 2001 [13], and the Sharable Content Object (SCO) in SCORM [14]. The course in the Instructional View will be used to generate SCORM-compliant materials to support reusability of the knowledge base contents. With the high cost in the development of multimedia education materials, reusability is a major consideration for any developer of multimedia education. In the MKBe-Learning System, the content of the knowledge base would also be presented in different ways to students with different expectations. This differentiation will be based on personalization criteria inferred through the Inference Engine. The analysis of the knowledge base also identified representations of subject knowledge that are reusable to different lecturers. These are classes in the Knowledge View of the knowledge base in Figure 2. A Subject contains many concepts where a Concept is a representation of an idea that can be taught, explained or communicated by lecturer to the students. Concepts can be related to other Concepts via typed-relations such as pre-requisite and related-to. Each Concept is described through different Characteristics such as Information, Explanation, Example, Demonstration, and Use Case. Each Characteristic will be presented through the Media. The types of Media have been identified as Text, Graphics, Audio, Video and Animation. The reusable concepts in the Knowledge View are part of the Instructional View, without the presentation features such as layout and sequence. For new subjects, the concepts in the Knowledge View would be extracted from the Topics in the Instructional View, whereas the characteristics and media would be extracted from the webpages. These can then be used by other lecturers when creating other courses for the subject, and any additional concepts from the new courses would be added to the Knowledge View for reuse. 4. Basic Electronics Content Development In the MKBe-Learning System, the first subject to be created is Basic Electronics. The development of the Basic Electronics course follows a process illustrated in Figure 3. This development process is adapted from Dick and Carey’s instructional design methodology [15] and generic instructional design model known as ADDIE [16]. The requirements were identified through analyzing variant syllabi gathered from local and international higher learning institutions [3]. The syllabi are compared to define and categorize the topics based on their similarities. The three categorizations identified are Important topics, Relevant topics and Optional topics. The integration of all the syllabi is done to create generalized syllabus that describes the required contents of the Knowledge View, representing all the concepts for the subject. For actual development of a course, a ‘standard’ syllabus of the subject is created by a lecturer who is the subject matter expert. This syllabus would contain the topics that are likely to reuse based on the highest percentage of topics occurrences by lecturers. The standard syllabus will be reviewed by another subject matter expert to ensure that the learning objectives can be achieved. It is possible that the generalization process reduces the significance of some topics to be optional and may be omitted from the standard syllabus. Thus, the formalization process would be iterated to ensure that the standard syllabus can meet the learning objectives. The subject matter experts are responsible in ensuring that the contents developed in the knowledge base are highly reusable.

350

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

Analysis of Syllabus

Syllabus Integration

Syllabus

Generalized Syllabus

Course Formalization Standardized Syllabus Expert

Syllabus Review

Syllabus Verification

Lesson Creation

Lesson List

Lesson Description Form

Storyboarding Content Creation

Content Review & Verification

Knowledge Base

Figure 3: Process of Producing a Standard Course for Knowledge Base

The standard syllabus has to be verified by the subject matter expert reviewer to ensure the syllabus created will be used in content development. This process implies the reexamination of the syllabus information and topics that will be practiced in a particular educational environment. Thus, the verification process will be ensured that previously reported results are accurate. Once the syllabus is verified, lesson creation process will be followed. During the lesson creation process, three outputs would be produced; lesson list, lesson description form and storyboard. This process helps the subject matter expert to identify the appropriate content development within the knowledge base in order to assist them choosing the right teaching materials. The course content is then created based on the information gathered during the lesson creation process. Throughout the content creation process, the content developed will be stored in knowledge base. At the final stage, the content will be reviewed and verified by the subject matter expert to ensure that the course contents are developed as expected. 5. Content Structure The Basic Electronics course contents are structured according to the Instructional View in Figure 2. Based on the standard syllabus and the lesson creation process, the number of objects to be created in the course is shown in Table 1.

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

351

Table 1: Distribution of Objects in Basic Electronics Course Basic Electronics Number of Chapters 7 Number of Lessons 50 Number of Topics 60 Number of Webpages 5 – 10 per lesson

The development of the first 11 lessons in Chapter 1 has been completed and Table 2 shows the distribution of the different course content objects. In each lesson, there may be more than one topic to explain the concept that need to be delivered to the students. A lesson consists of a topic or a combination of a topic with the subtopics or combination of subtopics. A topic contains subtopics to explain a detailed concept. A webpage can contain display elements which will be termed as presentations. A presentation is used to describe a specific aspect of concept and categorized under characteristics in the Knowledge View of the knowledge base (Figure 2). Table 2: The Distribution of the Lesson Objects for Chapter 1: Direct Current Circuits Lessson

Topic

Webpage

L1

2

7

Presentation 9

L2

1

8

12 10

L3

2

5

L4

2

5

7

L5

1

8

18

L6

2

6

13

L7

1

5

8

L8

2

9

18

L9

1

6

11

L 10

1

8

16

L 11

1

6

15

TOTAL

16

73

137

Further analysis of the 11 lessons was done to identify the different types of presentations. The types of presentation are identified from the characteristics categorization in the Knowledge View in Figure 2. Table 3 shows the distribution of the different types of content presentations for each lesson in Chapter 1. The overall distribution for the types of presentations is shown in Figure 4. Based on Figure 4, Description is widely used to describe the concepts (42%). This is because the concepts for Basic Electronics subject are more likely to be presented as descriptive information. Example is the second highest which comprises of 25% since this subject is technical and need to provide a lot of examples to explain the concept and its application. Fact and Definition are the next percentage highest which comprises of 10% and 9%. Only 3% of concept is presented using the Content since this presentation is less likely to be presented as specific illustration of a concept.

352

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe Table 3: The Distribution of the Presentations Types for each Lesson

L 1

L 2

L 3

L 4

L 5

L 6

L 7

L 8

L 9

L 10

L 11

TOTAL

Lessons Types of Presentation

Definition

1

2

1

2

1

1

-

2

1

-

1

12

Description

4

5

3

3

4

4

3

4

3

7

5

43

Contents

-

-

-

1

-

2

-

1

-

-

-

4

Fact

1

-

-

1

3

1

1

1

1

2

3

14

Example

3

5

3

-

4

4

1

4

3

4

3

34

Demonstration

-

-

1

-

-

-

1

1

-

1

1

5

Usage

-

-

-

-

6

1

2

1

-

-

-

10

TOTAL

9

12

10

7

18

13

8

18

11

16

15

137

Types of Presentations Usage 7% Demonstration 4%

Definition 9%

Example 25% Fact 10%

Description 42%

Contents 3%

Figure 4: Types of Presentation Used in Chapter 1

For the 11 lessons in Chapter 1, the number of media items created in each lesson is shown in Table 4. The media items were created based on the storyboards created in the lesson creation process to realize the presentations in the lesson webpages. Based on Figure 5, Text and Animation are the most used media for the presentation of the subject which accounted for 36% and 24% of all the media items. Graphic (22%) and Audio (17%) are used more to illustrate and clarify the content and description of the concepts. Since this lesson is on concept of introductory to direct current circuits, more animations are used to explain the concept. Video is less likely to be used (7%) since video files require more bandwidth. Video is only applied for demonstration and use case.

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

353

Table 4: The Distribution of the Media Types based on the Lessons Lesson

Media Animation Audio

Video Graphic

Text

L1

4

4

2

-

4

L2

4

2

-

1

5

L3

3

2

-

3

3

L4

3

3

-

-

3

L5

3

3

-

4

6

L6

3

1

-

2

4

L7

1

1

-

2

3

L8

2

1

-

7

7

L9

1

1

-

4

4

L 10

3

1

-

5

6

L 11

3

1

-

3

4

34

24

2

31

53

Figure 5: Media Used in Chapter 1

6. Further Work The development of the Basic Electronics course will be completed and further analysis of the content distribution can provide a better representation in the lesson structure of the Instructional View of the knowledge base. Further refinement of the class diagram may be done based on the course contents, in order to produce an accurate representation of the course. The extraction of concepts, characteristics and media into the Knowledge View of the knowledge base is to be formalized, through the further analysis of the course contents. The extraction process may be translated into a generalized algorithm that may be applied to all subjects. The reusability of the contents will then be explored through the creation of other courses by other lecturers. 7. Conclusion

354

R. Ramly et al. / Structuring of Multimedia Contents for E-Learning Lessons in MKBe

Based on the object-oriented analysis of the MKBe-Learning System, the knowledge base structure has been created and implemented for the subject Basic Electronics. The initial multimedia contents created for the subject demonstrated the suitability of the content structure to store course contents created by lecturers. The contents can then be made reusable in the Knowledge View of the knowledge base. Further development of the Basic Electronics course contents and the reusable concepts in the knowledge base would be done and the process will be refined. Other courses will also be developed to explore the reusability of concepts in the knowledge base. Acknowledgement The authors acknowledge the financial support given by the Ministry of Science, Technology and Innovation through the MSC Research & Development Grant Scheme. The authors would like to thank Prof. Ryoichi Komiya and all members of the MKBe-Learning Project for their guidance and assistance throughout the project. References: [1]

Yeoh, E.T., Mohd. Yusof, M.F., “An Integrated Multimedia E-Learning System”, Proceedings for the International Conference on Computers in Education, Hong Kong, December 2003, pp. 794-801. [2] Booch, G., Rumbaugh, J. and Jacobson, I., “The Unified Modeling Language User Guide”, NJ: AddisonWesley, 1999. [3] Yeoh, E.T., Ramly, R., Ng, K.L., “Comparison of Electronics Courses For Development of A Multimedia Knowledge Base”, MMU International Symposium on Information and Communication Technologies, Malaysia, 2004, pp. 13-16. [4] Schott, M., Chernish, W., Dooley, K. E., Lindner, J. R., “Innovations In Distance Learning Program Development And Delivery”, Online Journal of Distance Learning Administration, Summer 2003, State University of West Georgia, Distance Education Center, Volume VI, Number II, (http://www.westga.edu/~distance/ojdla/summer62/schott62.html). [5] Ng. K.H., Komiya, R., “Multimedia Textbook for Engineering Courses”, Proc. IWNA 3rd International Workshop on Networked Appliances, Singapore, March 2001, pp. 122-128. [6] Bennett, C.L., Pilkington, R.M., “Using A Virtual Learning Environment In Higher Education To Support Independent And Collaborative Learning”, Proc. IEEE International Conference on Advanced Learning Technologies, 2001, pp. 285 -288. [7] Rami A., Piccoli, G., Ives, B., “Effectiveness of Virtual Learning Environments in Basic Skills Business Education: A Field Study In Progress”, Proc. of the International Conference on Information Systems, 2000, pp. 352-357. [8] Norliana Maleh, C.S. Lee, C.K. Ho, “Software Requirement Specification For Instructor Use Case”, Technical Report MKBE/AT/Norliana/001, MKBe R&D Laboratory, Multimedia University, Cyberjaya, Malaysia, 2004. [9] Multimedia Super Corridor, Multimedia Development Corporation, MSC R&D Grant Scheme, 2003. (http://www.msc.com.my). [10] Wiley, David A., “Connecting Learning Objects To Instructional Design Theory; A Definition, A Metaphor, And A Taxonomy”. In D.A. Wiley (Ed.). The Instructional Use of Learning Objects, 2000. (http://reusability.org/read/). [11] The Herridge Group, “Learning Objects and Instructional Design”, August 2002. [12] CISCO System, “Reusable Learning Object Authoring Guidelines: How to Build Modules, Lessons, and Topics”, 2003. (http://www.cisco.com). [13] ACM/IEEE-CS Joint Curriculum Task Force, “Computing Curricula 2001”, Final Report, December 15, 2001. [14] Advanced Distributed Learning Org., “SCORM Best Practices Guide For Content Developer”, 1st Edition, February 28, 2003. [15] Dick, W. and Carey, L., “The Systematic Design of Instruction”, NY: Harper Collins Publishing, 1996. [16] Steven J. McGriff, “ISD Knowledge Base / Instructional Design & Development / Instructional Systems Design Models”, Penn State University, USA, 2004, (http://www.personal.psu.edu/faculty/s/j/sjm256/portfolio/kbase/IDD/ISDModels.html). [17] Merrill, M. David. Chapter 7, “A Lesson Based On Component Display Theory”. In Charles M. Reigeluth (Ed.) Instructional Design Theories in Action, Hillsdale, NJ: Lawrence Erlbaum Associates, 1987.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

355

Dialogue Games and e-Learning: The Interloc Approach A. Ravenscroft & S. McAlister Learning Technology Research Institute (LTRI) London Metropolitan University, UK. [email protected] Abstract: Currently there is considerable enthusiasm for exploring how we can apply digital gaming paradigms to learning. But there are problems with the degree to which these approaches both fit with our learning landscape and define how they actually support the development of social or conceptual processes that are involved in learning. In contrast, this paper will describe a ‘dialogue game’ approach to e-learning that integrates with existing, or near-future, pedagogical and technological practices, and which seeks to explicitly link game-playing activity to the development of dialogical and reasoning skills that lead to improved conceptual understanding and collaborative knowledge refinement. This paper will present: the background and theoretical foundations for this approach; summarise computational and empirical studies that support it; and, describe a prototype socio-cognitive tool called InterLoc that mediates, structures and scaffolds educational dialogue games. This approach is demonstrated and the implications it holds for designing gaming and related types of educational interaction are then discussed. Keywords: dialogue games, learning dialogue, conceptual learning, socio-cognitive tool, e-learning landscape

1. Digital games and learning This paper proposes an educational articulation and application of ‘game’ concepts. These are related to, and yet a variation, from the currently popular approaches which are typically inspired by or build upon the influences of the digital (or video) gaming industry. Many e-learning researchers and developers are currently developing some interest and enthusiasm for this (video) ‘gaming paradigm’ in networked learning because they are inspired by the work of proponents such as Gee [1] & [2] and Prensky [13]. It is claimed that these approaches can raise the motivation, social and affective engagement, and cultural relevance of learning interactions (e.g., through using the ‘kids’ technologies or role playing in simulated situations). They also emphasise the role of pleasure, play and ‘learning by doing’ through collaboration and competition (e.g. see [7], or [4] for a review). But there is a problem with the degree to which these video game inspired approaches fit with our existing or near-future learning landscape. Gee [1] is quite explicit about this in his support for video game approaches, arguing that digital games, and the learning principles that underlie them, represent a somewhat revolutionary approach to learning and literacy development that cannot be easily accommodated or realised through traditional institutions such as schools. Indeed, he argues that video game approaches actually change the fundamental nature of learning and literacy. Prensky [13] offers a similar argument, which states that the prevalence of digital game-playing by children actually leads to the development of new types of cognitive skills, and these need to be understood and represented in learning situations in ways that move beyond our traditional ideas about education. Another problem with video game inspired approaches is that they are often weak in linking the game-playing activity to transferable social or conceptual processes and skills that constitute, or are related to, learning. So whilst these digital

356

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

gaming proponents may offer attractive arguments for ‘future’ learning, this paper will argue that we should also adopt a more problem-focussed, near-future and interaction design approach to digital games for learning. So we argue that a more formal and dialogical approach to games (e.g. [22], [16]) can be used to develop pedagogical innovations that address salient learning problems, such as the development of dialogical and reasoning skills in children and adults, within the evolving new media landscape. In brief, we argue that carefully designed digital games, such as computer-mediated dialogue games (e.g. see [17] for a review), can support engaging, relevant and important learning activities. The following sections: make the argument for a dialogical approach to learning; demonstrate how this approach can be realised through designing dialogue games; demonstrate a socio-cognitive tool called InterLoc that mediates educational dialogue games; and, then discusses the implications of this approach. 2. Dialogue and developing conceptual skills The social constructivism of Vygotsky [21] emphasises the role of dialogue and organised social activity in the development of higher mental processes and learning. Vygotsky considered language and dialogue to be the most interesting and powerful semiotic mediators and the primary tools for thinking. He claims that our higher mental functions, such as verbal thought, reasoning, selective attention and reflection, originate in the sphere of social activity. Development proceeds ‘from action to thought’ and therefore communication and social contact are essential to transform mental functioning from the ‘outside’. So if we accept this argument for the predominance of dialogue as a tool for learning and thinking from an interaction design perspective, it suggests that we could use games as paradigm for developing models and tools that realise desired dialogical and reasoning processes, leading to improved conceptual understanding and knowledge refinement in students. 3. Designing digital dialogue games for reasoning and knowledge refinement Previous research in e-learning dialogue has produced the methodology of Investigation by Design [16] to develop and investigate educational dialogue games. This approach combines discourse analysis and other dialogue game techniques to specify models that can be implemented as tools or modelling systems. These game designs and tools are developed through modelling social features of effective dialogical interaction, such as the roles of the interlocutors, the ground rules for commitment and turn-taking, and the type of speech-acts [19] that may be performed. The dialogue games that are developed, whilst sharing the same categories of features (e.g. defined numbers of players, roles, moves and rules) are distinctive in terms of the actual instances and numbers of these features. They are also different in terms of the particular learning problems they address and the learning processes they support whilst retaining certain ‘family resemblances’ [25]. The approach has been successfully used to design a number of educational dialogue game tools (e.g. DIALAB, CoLLeGE, CLARISSA, AcademicTalk), that are reviewed in Ravenscroft [18] and summarised below. 3.1. Dialogue games addressing particular learning problems and processes Previous research has shown this dialogue game approach to be effective in designing distinctive interaction scenarios addressing particular learning problems and processes. A facilitating dialogue game for collaborative argumentation led to conceptual change in science with adults and school children [14], [15]. An inquiry dialogue game supported improvements in the reflective and diagnostic skills of medical students [12]. And a critical discussion and reasoning game supported deeper and extended argumentation and reasoning in open and distance learning (ODL) students, who showed a more scholarly and dialogical approach to learning and reasoning when compared with a less structured dialogical approach (i.e. synchronous Chat) addressing the same task [8], [9]. Recent projects funded by the UK JISC (Joint Information Systems Committee) e-learning programme have built upon this previous work. These projects have refined the dialogue game design paradigm and developed an Open Source tool, called InterLoc, that supports the specification, integration and investigation of dialogue games for critical discussion and reasoned

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

357

dialogue in education. This tool supports a number of dialogue games. One game for critical discussion and reasoning has already been well tested and evaluated [9]. Other games for creative thinking and exploratory dialogue, that have been developed through adapting the critical discussion game whilst considering the work on creativity and exploratory talk proposed by Wegerif [23, 24], have been specified and are currently being implemented. These games are linked to a local pedagogical context and can link to other tools developed according to international standards for interoperability and reusability (see [10]). The following sections describe in more detail how the dialogue games are developed and specified in terms of the InterLoc approach. 4. Specifying educational dialogue games Through applying the Investigation by Design methodology described above the educational dialogue games are specified in terms of the goals of the interaction (e.g. critical discussion and reasoning, exploratory dialogue, creative thinking), the roles of the participants (e.g. discussant, facilitator), the Intentions/Moves that may be performed (e.g. Inform, Question, Challenge), the Locution Openers that actually express the surface-level realisations of the Intentions (e.g. “My evidence…”, “Is it the case that…”, “I disagree because…”) and the rules of interaction (e.g. about turn-taking and the legitimate sequencing of moves). These games can be designed ‘on paper’ using pre-defined templates and realised through instantiating the InterLoc tool (described below). 5. InterLoc: collaborative Interaction through scaffolding Locutions Figure 1. The InterLoc interface during a Critical Discussion and Reasoning game

The most important InterLoc interface for supporting the dialogue game interactions is shown in Figure 1 and described below. This tool is an improved version of AcademicTalk, that supported

358

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

critical discussion and argumentation specifically, and which has been demonstrated and evaluated by McAlister, Ravenscroft & Scanlon [8, 9]. Figure 1 shows how the interface manages, structures and scaffolds contributions during the dialogue game exercises. The interface content represents the dialogue game being performed, and is retrieved from pre-defined templates representing Critical Discussion and Reasoning, Creative Thinking or Exploratory Dialogue. Different sets of locution openers, rules and ‘preferred next openers’ configure the interaction to represent a particular dialogue game. The example in Figure 1 represents the Critical Discussion and Reasoning game. Key features of the tool are described below. 5.1. Managed synchronous dialogue InterLoc provides two viewing panes, one to browse the latest messages (top), and another to view the selected message and its antecedents (bottom), linked by the reply relationship. A message and its antecedents comprise an argument strand providing a coherent sequence of dialogue1. So unlike other synchronous approaches, such as Chat, where there is pressure to be 'first poster' to keep the reply near to the antecedent message it replies to [6], in InterLoc there is no such necessity, since every reply is placed next to its antecedent message when viewed as an argument strand. It is also possible to browse early messages in the discussion and reply to them, offering some of the reflective advantages of asynchronous discussion. 5.2. Locution openers to support critical discussion and reasoning InterLoc requires that the learners select a locution opener (e.g. “I think…”, “I disagree because…”, “My evidence…” etc.) from one of six pre-defined dialogue move categories (e.g. “Inform”, “Question” “Challenge”, “Reason”, “Agree”, “Maintain”) to perform their contribution and then complete the message in their own words. These locution openers are derived from previous work in Speech Act Theory [19], Collaborative working [20] and other empirical research performed by the authors and others (e.g. [11]; [15]; [9]). The openers have been developed and refined over several years and tested to ensure that they provide a relatively natural means of locution construction. This subtle design process has allowed us to avoid the problems of somewhat ‘unnatural constraining’ that has been observed with other approaches that have employed Speech Acts (e.g. Flores et al., [5]). 5.3. Dialogue rules and guidance on opener selection Certain openers are highlighted (i.e. bolded) when making a selection as suggestions to be considered in making a reply (see Figure 1). This 'preferred reply' set of openers is derived from formal dialogue game approaches (e.g. [22]) and corresponds to notions of well-formed and coherent dialogue (for example, “Can you elaborate?...” suggests a reply “Let me elaborate...”). However, learners are free to choose openers outside this preferred set, so a flexible, non-directive and yet constructive form of guidance is provided. The following section describes how dialogical and reasoning processes are supported by InterLoc. 6. Exemplar dialogue game interaction through InterLoc The interchange below demonstrates how a Critical Discussion and Reasoning Game was performed on the topic of “Who is the father of the Internet?”. This dialogue was performed during a study using (adult) Open and Distance learners at the UK Open University, based on a topic that was part of their course (Note: To respect conditions for anonymity the names of the actual students are changed in the presented exchange). The dialogue is chosen as an exemplar to demonstrate how the dialogue game instantiated in the InterLoc tool mediates the discussion. 1

The upper pane lists the tail (latest) message of each argument strand and when selected the strand is listed in the lower pane.

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

359

Future work will pilot the tool with varied types of learners, and in varied course related contexts, following the successful outcome of the first phase of user trials and previous research that evaluated an earlier version of this tool (see [8, 9]). The interchange demonstrates a number of interesting dialogical features and the educational value of these interactions. Firstly, although this is a short interchange (due to space restrictions specified for this paper), it is relatively well balanced with all participants making significant contributions to the development of the discussion. Secondly, there is a good variation in the type of contributions that are made, the extract demonstrates Informing moves such as “I think…”, Challenge moves such as “I disagree because…”, Agree moves such as “I agree because…”, and Question moves such as “Is it the case that?…”. Thirdly, there is a demonstration of dialogical reasoning and ‘inter-thinking’ leading to more sophisticated perspectives being developed during the collaborative knowledge building within the group, who challenge, question, and then elaborate or qualify propositions that are in play. This interchange is now described and commented upon in greater detail below. Table 1. A CDR game demonstrating challenging, justifying and elaborating ideas

1

Andrew

-- Father of the Internet - who are the main contenders and why?

2

Sandra

I think...Paul Baran for packet switching and distributed networks

3

Eric

I'm not so sure...Is Paul Baran father of the Internet or just father of the network?

4

Sandra

5

Eric

6

Dawn

I'm not so sure...whats the difference between father of network or network of networks? To summarise...There were many competing networks. The internet connected them all. We are looking for the father/mother of the internet. Also...however, Baran's ideas about the Internet's distributed form were very important in it's development I agree because...it was a very important break through

7

Sandra

8

Eric

9

Dawn

10

Andrew

11

Dawn

12

Eric

13

Dawn

14

Andrew

15

Eric

I disagree because...because it was only a building block towards the internet. It needed someone to draw it altogether. To summarise...then it took many parts to create the whole. Each part needs the others for the Internet to have came about An alternative view...someone needed to influence people's thinking and create a vision for the future I agree because...wasn't it Vannevar Bush who had the idea of creating a collection of knowledge which has links to others Is it the case that?...he could be called the father of the internet. Although his ideas are incorporated, so are many others. I agree because...other people had an indirect input to the net, such as Wiener, Licklider, who's theories were incorporated. Baran and Cerf worked on the theories Also...he has been very widely acknowledged by many who followed after him, along with Lick A counter-argument is...that these ideas only really lead to singular networks. It needed someone with the wider vision to suggest connecting them all.

The group are responding to a discussion thread introduced by the student facilitator – Andrew. The interchange shows the locution openers selected (in bold), and each message is a direct reply to the message above. Note that the set question is quite a contentious one, since, over the twenty year development of computer networking and eventually the Internet, there were many players and many ideas involved. This type of open question in our critical discussion game typically seeds the process of argumentation and reasoning, with players weighing up the merits of the contending arguments, examining the criteria used to make judgements and generally elaborating

360

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

the group’s knowledge and understanding. The participants are stimulated to look more deeply into the issues, as the provided openers stimulate reasoning leading to challenges (e.g. “I disagree because…”, “A counter-argument is…”), elaborations or qualifications (e.g. “I agree because…”) and some synthesis (e.g. “To summarise…”). Sandra makes the first contribution under this thread using the informing move ‘I think…’ – to suggest a possible ‘father of the Internet’, namely Paul Baran (2). Eric responds with a (mild) challenge move – ‘I’m not so sure…’ – which calls into question whether Baran’s influence is directly relevant (3). This stimulates Sandra to realise that she doesn’t clearly understand and accept this distinction he is making and so she issues a challenge using – ‘I’m not so sure…’ – in a way that also requests clarification (4) (Note: she could have used a more direct request for clarification with ‘Can you elaborate?’). To reply to Sandra, Eric makes a reasoning move about the scope of the question they are trying to settle using the opener ‘To summarise…’ to explain his position (5). Then Dawn makes a further informing move (6), using “Also…” to propose her position and Sandra supports this through using “I agree because…” (7) as they return to the argument for one of the early ideas behind the Internet, its distributed topology, as the key idea. Eric then challenges their position more strongly – using ‘I disagree because…’ – to dispute whether the idea of internetworking was the key factor and to elaborate his point further (8), arguing that networking was just part of the picture. Dawn apparently sees that both the previous positions have some validity and proposes a more sophisticated view of Internet development – using the reasoning move ‘To summarise…’ – which references the previous positions (9). The existence of this tension between two opposing positions was possibly the spur for Dawn (9) to initiate a form of synthesis. Andrew follows this through performing a challenge move that takes the discussion in an elaborated direction through suggesting that the group should also consider ‘visionaries’ as well as the implementers of the Internet (10), and Dawn (11) offers a support for this using – ‘I agree because…’ to introduce the notion that Bush, who had the idea of collecting knowledge, was also important. In reply Eric, however, does not apparently fall into complete agreement (12) and offers a leading question – using ‘Is it the case that?…’ – which reuses Dawn’s earlier argument about the building of several people’s ideas towards the eventual outcome, the Internet. This leads Dawn to rethink and consider others who may have had an equal input into the internet project, by agreeing with Eric (13). The extract finishes with Eric more confidently proposing the criteria for ‘father of the Internet’ as distinct from ‘father of the network’ (15) to Andrew using ‘A counter-argument is…’. This is a useful distinction to make between the superficially similar concepts of any computer network and the Internet. This interchange reflects a typical interplay between groups performing this dialogue game, where the approach leads participants into a relatively deep collaborative analysis of the issues through offering reasoned arguments and, in turn, critiquing the arguments before developing a more elaborate view. The use of the openers, particularly those that challenge, facilitate dialogical argument with clear, direct responses to the ideas of other players without causing offence or discouraging participation. Moreover, the participants from the twenty three sessions that have been studied thus far (see McAlister, Ravenscroft & Scanlon [9]) reported that they enjoyed the greater depth and intellectual challenge of these discussions, the constructive argument with their peers and they also found the experience enjoyable and absorbing. 7. Discussion: developing the digital dialogue game approach Recent work has performed a critical analysis of the design of this application and the dialogue game approach in general to consider how we should further develop this line of work. In particular, we have considered issues of motivation and pleasure, as it is typically articulated in relation to digital games, and considered these in the context of approaches that are more pedagogically grounded and related to existing learning practices and organisations. The following two points are being considered in ways that influence our ongoing design studies. Firstly, learners engaging in current educational practices, who may be quite mature and so typically not the focus of gaming research, will have more extrinsic as well as intrinsic motivation for performing learning interactions when compared to their younger digital gaming counterparts. Learners in existing educational contexts will also be aware of the broader cultural and educational relevance and value of certain skills, and this in itself is quite motivating. For

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

361

example, the need to develop critical thinking and reasoning skills to perform and participate in study, work, debates ‘in the pub’, or society in general may be more motivating, or as motivating, as any pleasure derived from playful interactions which practice these skills. Also, in these cases a significant amount of pleasure can be experienced through cognitive engagement and satisfaction with performance. We can get excited about ideas and debating them without employing overly sophisticated interfaces that may rely on 3D graphics, visual immersion and the like. In other words, motivation, pleasure and engagement don’t just emerge during (game) interactions, as these dimensions are heavily influenced by what the games ‘mean’ in the context of cultural expectations and practices in the real world. So we need to be careful when proposing a dimension such as ‘fun’ as an objective, in ways that that are divorced from reflective satisfaction and value. Secondly, considering the role of ‘meaning’, game interaction can be made more relevant (and hence motivating), and more valuable through considering the contextual and environmental factors that influence learning interaction and behaviour. For example, we could increase engagement through having an appropriate blend of players, with individual differences in their opinion or backgrounds to seed engaging arguments and discussions. The motivation for game playing will also be improved through having a good fit between the game-playing activity and the expectations for learning, e.g. through addressing an important role or niche for developing the practiced skills in terms of the curriculum or ‘life skills’ more generally. It will also be useful to allow players to develop and represent their actual identities online, as these are what will matter in their broader educational progression. Additionally, it is important to allow seamless integration with a learner’s personal learning environment or devices and other learning activities. In other words, an educational game that uses ubiquitous technologies and catalyses natural human learning processes will arguably be more ‘exploitable’ than one that relies on a specialised high memory box and high resolution screen that can only be used in a set number of ways, and typically, in one place (e.g. the bedroom). The work reported by Facer et al [4] showing how children, using mobile technologies, simulated lion behaviour in a wildlife game (‘Savannah’) is also an example of how engagement is assisted by the technological fit between the game interaction and the use of ubiquitous technologies familiar to learners. 8. Summary and conclusions This paper has presented an approach to designing and implementing educational dialogue games. Our reflection on the results of these projects has led us to address more fundamental questions about learning and the context in which it is performed. It has become clear that the value of, motivation for, and pleasure experienced with (digital) educational games will not just come from the interaction itself – but how the game interaction is integrated and interwoven with the broader education process and related technologies. So we should not just be asking ‘What makes a good educational game?’, but also ‘Why would we want to play it?’ and, ‘What long-term value do we get from it? Acknowledgements Recent work on the AcademicTalk and InterLoc projects has been carried out with the support of the UK Joint Information Systems Committee (JISC) in the framework of the ‘e-tools for teachers and learners’ programme. The content of this paper does not necessarily reflect the position of the JISC, nor does it involve any responsibility on the part of the JISC. We would also like to thank all other members of AcademicTalk project team who advised about the development of the InterLoc application. In particular we would like to thank Dr. Rupert Wegerif for his contributions to work specifying the particular dialogue games, Prof. Eileen Scanlon for advising on the general pedagogical approach and design and Prof. Tom Boyle and Prof. Oleg Liber for advising about Open Source and Standards issues.

362

A. Ravenscroft and S. McAlister / Dialogue Games and e-Learning: The Interloc Approach

References 1. 2. 3.

4. 5. 6. 7. 8.

9.

10. 11. 12. 13. 14. 15.

16.

17.

18.

19. 20.

21. 22. 23. 24. 25.

Gee, J.P (2003). What video games have to teach us about learning and literacy, Macmillan. Gee, J.P. (2005). Learning by Design: good video games as learning machines, E-learning, 2/1, 5-16 Facer, K. (2004). Computer Games and Learning: Why do we think it’s worth talking about computer games and learning in the same breath? Nestafuturelab discussion paper. Available http://www.nestafuturelab.org/research/discuss/02discuss01.htm Facer, K. , Joiner, R. , Stanton, D., Reid, J. , Hull, R. & Kirk, D. (2004). Savannah: mobile gaming and learning?. Journal of Computer Assisted Learning 20/6, 399-409 Flores, F., Graves, M., Hartfield, B. & Winograd T. (1988). Computer Systems and the Design of Organizational Interaction. ACM Trans. on Information Systems 6/2, 153-172 Herring, S. (1999) Interactional coherence in CMC. J. of CMC [online] 4, 4. Available: http://www.ascusc.org/jcmc/vol4/issue4/herring.html Kirriemur. J. & McFarlane, A. (2003). Literature Review in Games and Learning, NESTA FUTURELAB SERIES, Report 8, NESTA Futurelab, Bristol. McAlister, S., Ravenscroft, A & Scanlon, E. (2004a). Combining interaction and context design to support collaborative argumentation using a tool for synchronous CMC, Journal of Computer Assisted Learning: Special Issue: Developing dialogue for learning 20/3, 194-204 McAlister, S., Ravenscroft, A & Scanlon, E. (2004b). Designing to promote improved online educational argumentation: an evaluation study, In Networked Learning 2004, (eds.) Banks, Goodyear, Hodgson, Jones, Lally, McConnell & Steeples, Lancaster & Sheffield University, 541-548. Available: http://www.shef.ac.uk/nlc2004/Proceedings/Individual_Papers/McAlister_et_al.htm Olivier, B. (2004). Application & Tool Component Framework, JISC Technical Paper. Pilkington, R.M. (1999). Analysing Educational Discourse: The DISCOUNT Scheme. Version 3, January 1999. CBL Technical Report No. 99/2 Pilkington, R and Parker-Jones, C. (1996) Interacting with Computer-Based Simulation: The Role of Dialogue, in Computers & Education 27/1, 1-14 Prensky, (2001). Digital Game-Based Learning, McGraw-Hill Education. Ravenscroft, A. (2000). Designing Argumentation for Conceptual Development, Computers & Education, 34 , 241-255. Elsevier Science Ltd. ISSN 0360-1315. Ravenscroft, A. & Matheson, M.P. (2002). Developing and evaluating dialogue games for collaborative e-learning interaction. Journal of Computer Assisted Learning Special Issue: Context, collaboration, computers and learning 18/1, 93-102. Ravenscroft, A. & Pilkington, R.M. (2000). Investigation by Design: Developing Dialogue Models to Support Reasoning and Conceptual Change, International Journal of Artificial Intelligence in Education: Special Issue on Analysing Educational Dialogue Interaction: From Analysis to Models that Support Learning, 11/1, 273-298. Ravenscroft, A. (2004a). From conditioning to learning communities: Implications of 50 years of research in eLearning interaction design. Association for Learning Technology Journal (ALT-J), 11/3, 4-18. Ravenscroft, A. (2004b). Towards highly communicative eLearning communities: Developing a sociocultural framework for cognitive change. To appear in Land, R & Bayne, S. (Eds.) Cyberspace Education, Routledge, Chapter 9, pp 130-145 Searle, J. R. (1969). Speech Acts: An essay in the philosophy of language, Cambridge University Press. Soller, A. & Lesgold, A. (1999). Analyzing peer dialogue from an active learning perspective. Proceedings of the AI-ED 99 Workshop: Analysing Educational Dialogue Interaction: Towards Models that Support Learning. Available: http://lesgold42.lrdc.pitt.edu/wwwuments/Soller-Lesgold-AI-EDWorkshop.pdf Vygotsky, L. (1978). Mind and society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Walton, D. (1984). Logical Dialogue-Games and Fallacies, Lanham: University Press America. Wegerif, R. (2005). Reason and Creativity in Classroom Dialogues, Language and Education, 19, No. 3. Wegerif, R. (1996). Using computers to help coach exploratory talk across the curriculum, Computers and Education, 26, 1-3, 51-60. Wittgenstein, L. (1953). Philosophical Investigations, translated by G.E.M. Anscombe, Blackwell, Oxford, UK

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

363

From Response Systems to Distributed Systems for Enhanced Collaborative Learning Jeremy Roschellea, Patricia Schanka, John Brechta, Deborah Tatarb, S. Raj Chaudhuryc a Center for Technology in Learning, SRI International, USA b Center for HCI, Virginia Tech, USA c Department of Physics, Christopher Newport University, USA [email protected] Abstract. Innovators need a common, reusable framework for implementing patterns of collaborative learning on wireless handheld devices. Our project explores the application of an architecture from distributed system research, “tuple spaces,” to a wide variety of collaborative learning scenarios. We are testing this approach with documented scenarios from prior work published by our team and others. We are also designing new scenarios that leverage the architecture to engage learners optimally in cognitive, collaborative work. Our research explores the conceptual alignment, programming complexity, and classroom usability of this approach. Keywords. Collaborative and group learning; Learning systems platforms and architectures; wireless and mobile technologies

Introduction Collaborative learning has a well-established research base in education and social sciences [1, 2, 3, 4]. In classroom practice, coordination is fundamental to modern pedagogy. Groups of learners and their teachers routinely work in more complex configurations than in traditional lecture-based classes. They take roles, contribute ideas, critique each other’s work, and together solve aspects of larger problems, all to good effect [5, 6, 7, 8]. Managed flow of information and control is essential to the structure of many of these successful educational activities [9]. With the advent of mobile, wirelessly connected technologies in education [10, 11], innovators have begun to design applications to coordinate students’ work to produce better science, technology, engineering, and math (STEM) education outcomes. Yet there has been little work on formal computational foundations to support this core function of coordination [12]. Consider as an example, classroom response systems, which are an increasingly popular application in K-12 schools and universities across many subjects. These systems rapidly aggregate the input of all students anonymously and present a histogram that reveals the students’ understandings [13]. While research suggests that these systems can have powerful benefits in classrooms [14]—for example, by enabling student groups to focus on improving their conceptual understanding [15]—this technology is limited to a very narrow range of representational types and information-sharing topologies; generally speaking, the main data type is a response to a multiple-choice question and the only topology is many-students-toone-teacher. In contrast, researchers in the subfield of computer supported collaborative learning (CSCL) have documented how shared dynamic representations can support participation and discourse [16, 17, 18]. Researchers are calling for more powerful classroom

364

J. Roschelle et al. / From Response Systems to Distributed Systems

network technology that can expand to richer representations and interaction topologies [19, 20, 21] and have assembled prototypes of advanced uses, such as participatory simulations [22, 23], structured group reasoning tools [24, 25], comparison of scientific data sets [26], and ways to harvest students’ problem solving to highlight emergent mathematical and scientific structure [21]. Research has shown these prototypes to be educationally promising, but they have been hard to build with available technologies, and the resulting applications have been slow and unstable. Robust and responsive architectural support for coordinating multiple learners is thus an increasingly important topic [27, 28, 29]. Despite growing awareness of how the detailed structure of learners’ participation in activity affects learning [30, 31], there has been little progress directly linking the computational foundations of coordination (e.g., as found within distributed system research) with the educational fundamentals of participation.

1. Tuples Spaces for Collaborative Learning 1.1 Using Techniques from Distributed Computing Computer scientists working in the area of distributed computing have established a foundational abstraction called “Tuple Spaces” as a means for coordinating the work of multiple computers. Tuple Spaces provide a coordination language [32] that elegantly addresses the problem of organizing distributed scientific computations. Tuple spaces have grown beyond their original scientific niche. It has been claimed that they provide the best available technique for coordinating the participation of multiple mobile agents in collective activities [33]. They have been used to solve both technical and social coordination among these agents [34, 35]. A tuple space coordinates the computations of different processes by allowing the processes to read and write structured data to a common space [36]. Tuples provide, in essence, an associative shared memory. The word “tuple” refers to the use of vectors of primitive data types (text, numbers), which constitute the primary data type. In the classic “Linda” model [32], processes coordinate by writing data in the form of tuples to represent the state of computation and reading data by providing a template to be matched. The read operation blocks such that if no match is immediately found, the request will wait until one is available and then return immediately. Tuple spaces provide one critical additional operation, take, which allows exactly one process to read a match and take the tuple out of the space simultaneously. This last operation allows processes to coordinate such that one process uniquely works on each task. The combination of blocking reads and takes allows efficient, just-in-time execution of parallel processes without excessive central management.

1.2 Students as Distributed Processors: A Provocative Analogy Tuple spaces excel at optimizing the engagement of processors with different abilities (e.g. speed, memory capacity, etc.) working towards a group goal. If we consider students, not their computers, to be the distributed processor, we can pose the question of whether, by analogy, a tuple space can be utilized to engage and coordinate students with different abilities and preferences (e.g. to solve problems, communicate, critique, etc.) in working towards the goal of mastering classroom content.

J. Roschelle et al. / From Response Systems to Distributed Systems

365

Our project seeks to address this question. We conjecture that this architecture can provide new levels of cognitive and communicative engagement in classroom activities, yielding new levels of group performance. This socio-technical system can be conceived of as cybernetic in the tight coupling between “processors” giving and receiving information.

1.3 An Architectural Vision for Classrooms of Wireless Handhelds Wireless handheld devices, connected in a network, along with a projected display space, present an affordable and attractive hardware platform for classroom learning. We are proposing tuple spaces as an alternative to peer-to-peer, client-server, or web-based services for collaborative learning. We expect that a simple, general framework for coordinating the work of multiple learners will ease rapid prototyping, enable new types of activities, and make these systems easier to support in the field.

2. Using a Tuple Spaces Architecture in Collaboration Tools To test the feasibility of our vision, we plan to implement and test a variety of collaborative learning scenarios using Tuple Spaces. For all implementations, we are using the TSpaces server [36] and client programs written in Java. We have been first testing the tools on laptops and later will test them on handhelds, with applications that will inform us about the possibilities of both the technology and the pedagogy.

2.1 Scenarios We began the project by collecting collaborative learning scenarios from the literature. The group identified 41 scenarios. Of these, a set of 13 were selected as high priority because they (a) were deemed most important for use in STEM classrooms, (b) had been tried in classrooms and worked well in the past, and (c) clearly leveraged technology (e.g., would not be easy to do with paper and pencil). With this set, we also strove to span a wide variety of patterns (e.g., working in pairs, small groups, or an entire class; different workflows such as a pipeline or very distributed) and highlight a variety of issues (e.g., sharing, exchanging, publishing information) that would exercise the technology and pedagogy space and stimulate the imaginations of developers. So far we have implemented five activities representing these scenarios. Below we describe three of these activities and their Tuple Space implementations.

2.2 Question Posing The question-posing tool [37] is a twist on the conventional response system. In the conventional response system, the teacher asks a question and all students answer it. Using our tool, students determine what questions are important, either for the purposes of examining a new topic or for composing a set of review questions. The tool allows students to submit new questions to the question space, browse and rank other’s submitted questions, and cluster related questions. Submit is a simple write operation, browse is a read of a tuple that fits the “question” template. Rank involves taking a question tuple and writing a new version of the tuple with the applied rank as part of its data. Clustering here involves the writing of

366

J. Roschelle et al. / From Response Systems to Distributed Systems

“cluster” tuples that describe a relationship between one or more questions. Students can perform any of the tasks in any order they wish, and to whatever depth required. Thus, students more adept or interested in posing questions are free to do so as much as they wish, and students more interested in sorting and evaluating questions may do so, all without explicit central coordination. The result is a ranked set of important questions, with related items gathered in clusters.

2.3 A Novel Critical Reading Tool This text markup tool provides an application that can display a body of text and allow selections of that text at some specific grain size (e.g., lines, paragraphs, words). For example, students are presented with a poem, and asked to highlight all of the near-rhymes in the text. Students select relevant lines of text one-by-one, by checking a box beside ech line that illustrates a near-rhyme. When they are finished, they submit (write) their selections to the space. They can then browse and view (read) the selections made by others, which appear highlighted in gray on their text display. Their own selections appear highlighted in yellow. Bar graph displays to the right of each line indicate the number of students who highlighted the particular line, and the font size of more popular line is also increased to indicate frequency of selection. In this way, students’ markup is aggregated on each individual display, and discussion may then ensue about disagreement amongst the highlighted lines.

2.4. Draw My Molecule The molecule drawing tool provides a game-like environment to help students become familiar with a multiple scientific representations for various molecules and compounds. This tool provides an example of integrating exisiting software––in this case, ChemSense [38]––with the TSpaces architecture. Using this tool, students first submit (write) names of molecules to the space. Once several molecule names have been generated, students grab (take) a random molecule name from the space. The tools asks them to draw the molecule in one of three chemical representations: molecular formula, electron dot, or ball and stick. Students use the ChemSense drawing tool to draw the molecule, and then submit (write) their representation back to the space. If a student is unable to generate the representation, she can give up on that molecule-representation pair to release (write) it back to the space, and grab a different random molecule-representation assignment. Once a student submits a particular representation (e.g., electron dot) for a particular molecule (e.g., water), no other students will get this particular pair to draw. As a result, after some minutes of this distributed drawing activity, the matrix of moleculerepresentation types becomes mostly or even fully complete. In a final activity, students

J. Roschelle et al. / From Response Systems to Distributed Systems

367

use the tool to play a “slot machine” game [39] in which they get (read) three random representations from the space, and indicate, through checkboxes, which (if any) of the three representations are for the same molecule. If they indicate correctly, they “win” that slot round. Wins and losses are tallied for both individual and group scores.

3. Research and Evaluation Our overall objective is to develop methods, knowledge, and tools that better align social needs and distributed computing techniques with respect to the opportunity to improve collaborative learning. Below we describe three key research questions and our current progress towards answering them. 3.1 Is there a sound alignment of the tuple space formalism to the patterns and principles of collaborative learning? We are investigating this question by implementing a wide range of collaborative learning activities either previously described in the literature or of our own invention. So far, we have implemented five quite different activities and have found good alignment between the Tuple Space formalism and collaborative learning. We have found the general formalism to be fairly easy to use. The data management concepts are straightforward and primitive by design, and we haven’t needed to wrap the primitive operations. One area of difficulty is reconceptualizing pre-existing patterns of connectivity, such as peer-to-peer interaction, in the distributed Tuple Spaces perspective. We plan to further explore this question by implementing a high-level collaboration framework, such as that specified by the Collaboration Modeling Language, CML [40]. CML is a general framework for describing workflow and interaction in a networked classroom environment, such as group formation and the playing of specific roles,.

3.2 Can a wide variety of programmers adequately understand how to build designs for learning using these techniques? We are investigating this question by involving programmers from different settings and of different abilities in our work. So far, four different programmers have developed activities using the TSpaces environment. In most cases, implementing an initial prototype activity took about three days, including time spent on issues unrelated to TSpaces, such as user interface development (with Swing, a graphics library for Java) or content development. Database-like transactions were easily implemented using the transactions built into the TSpaces architecture. One advanced developer noted some difficulty giving up features builtin to more sophisticated databases, such as referential integrity and type protection. In the summer of 2005, we will host a 3-week International workshop in which students will code new collaborative learning tools using our framework. Further, in the fall of 2005, we will teach a course at Virginia Tech and study undergraduates learning and using the Tuple Spaces framework. These activities will further inform our understanding of the nature and development of expertise in programming using tuple spaces for social coordination.

368

J. Roschelle et al. / From Response Systems to Distributed Systems

3.3 Do tuple space servers work well in realistic classroom settings? Evaluation of Tuple Spaces in a classroom is scheduled to start in September 2005. Three challenges we anticipate are server overload, basic wireless infrastructure issues and setup/installation problems. For example, if a teacher instructs students to follow her in lockstep, and everyone in a class clicks on the same button at the same time, the server may overload. So far, our tests suggest that network traffic is fairly light, but we will need to track this. While our system does nothing to improve the wireless link layer, such improvements do not seem necessary. The system appears to tolerate network dropouts, partly because data packet transmissions are typically small and few with small retransmission cost. We are currently working on handling the set-up and installation problems that typically dog CSCW and CSCL systems, and which prove so difficult for teachers to handle.

4. Conclusion Innovators need a common, reusable framework for implementing patterns of collaborative learning on wireless handheld devices. Our project explores the application of an established architecture from distributed systems research, “tuple spaces,” to a wide variety of collaborative learning scenarios. Results to date indicate that this approach can both implement existing scenarios from the literature and enable new scenarios that may optimally engage learners in cognitive, collaborative, and cybernetic work. Our research is exploring the conceptual alignment, programming complexity, and classroom usability of this approach. We invite international partnerships in this research.

Acknowledgements This material is based in part upon work supported by the National Science Foundation under Grant Number #0427783. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References [1] [2] [3] [4] [5] [6] [7]

Koschman, T. (Ed.). (1996). CSCL: Theory and practice of an emerging paradigm. Mahwah, NJ: Lawrence Erlbaum Associates Koschman, T., Hall, R., & Miyake, N. (Eds.). (2002). CSCL 2: Carrying forward the conversation. Mahwah, NJ: Lawrence Erlbaum Associates. Wasson, B., Ludvigsen, S., & Hoppe, U. (Eds.). (2003). Designing for change in networked computing environments. Boston, MA: Kluwer. Webb, N. M., & Palincsar, A. S. (1996). Group processes in the classroom. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 841-873). New York: Macmillan. Angelo, A. A., & Cross, C. K. (1993). Classroom assessment techniques. San Francisco: JosseyBass. Hake, R. R. (1998). Interactive-engagement vs. traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66, 64-74. Hatano, G., & Inagaki, K. (1991). Sharing cognition through collective comprehension activity. In L. B. Resnick & J. M. Levine (Eds.), Perspectives on socially shared cognition (pp. 331-348). Washington, DC: American Psychological Association.

J. Roschelle et al. / From Response Systems to Distributed Systems

[8]

[9]

[10] [11] [12] [13]

[14] [15] [16]

[17] [18] [19]

[20]

[21]

[22] [23]

[24] [25]

[26] [27] [28]

[29] [30] [31]

369

Palincsar, A. S., & Herrenkohl, L. R. (1999). Designing collaborative contexts: Lessons from three research programs. In A. M. O'Donnell & A. King (Eds.), Cognitive perspectives on peer learning (pp. 151-177). Mahwah, NJ: Lawrence Erlbaum Associates. Guribye, F., Andreassen, E. F., & Wasson, B. (2003). The organisation of interaction in distributed collaborative learning. In B. Wasson, S. Ludvigsen & U. Hoppe (Eds.), Designing for change in networked learning environments (pp. 385-394). Boston: Kluwer. Norris, C., & Soloway, E. (2003, April). The viable alternative: handhelds. Retrieved May 12, 2003, from http://www.aasa.org/publications/sa/2003_04/soloway.htm Soloway, E., Norris, C., Blumenfeld, P., Fishman, B., J., K., & Marx, R. (2001). Devices are readyat-hand. Communications of the ACM, 44(6), 15-20. Malone, T. W., & Crowston, K. (1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87-119. Abrahamson, L. A. (1999). Teaching with a classroom communications system: What it involves and why it works. Retrieved February 20, 2004, from http://www.bedu.com/Publications/PueblaFinal2.html Roschelle, J., Penuel, W. R., & Abrahamson, L. A. (2004). The networked classroom. Educational Leadership, 61(5), 50-54. Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. The Physics Teacher, 69(9), 970-977. Cobb, P., Yackel, E., & McClain, K. (Eds.). (2002). Communicating and symbolizing in mathematics: Perspectives on discourse, tools, and instructional design. Mahwah, NJ: Lawrence Erlbaum Associates. Roschelle, J., Penuel, W. R., & Abrahamson, L. A. (2004). The networked classroom. Educational Leadership, 61(5), 50-54. Suthers, D. D. (2001). Towards a systematic study of representational guidance for collaborative learning discourse. Journal of Universal Computer Science, 7(3), 254-277. Abrahamson, L. A., Davidian, A., & Lippai, A. (2000). Wireless calculator networks: Where they came from, why they work, and where they’re going. Paper presented at the 13th Annual International Conference on Technology in Collegiate Mathematics, Atlanta, GA. Hegedus, S. J., & Kaput, J. (2001). New activity structures exploiting wirelessly connected graphing calculators. Paper presented at the 23rd Conference for the North American Chapter of the Psychology of Mathematics Education, Snowbird, UT. Stroup, W. M., Kaput, J., Ares, N., Wilensky, U., Hegedus, S. J., Roschelle, J., et al. (2002). The nature and future of classroom connectivity: The dialectics of mathematics in the social space. Paper presented at the Psychology and Mathematics Education North America conference, Athens, GA. Colella, V. (2000). Participatory simulations: Building collaborative understanding through immersive dynamic modeling. Journal of the Learning Sciences, 9(4), 471-500. Stroup, W. M. (1999). Participatory simulations: Network-based design for systems learning in classrooms. Paper presented at the Computer Supported Collaborative Learning Conference, Stanford, CA. Pinkwart, N., Hoppe, U., Bollen, L., & Fuhlrott, E. (2002). Group-oriented modeling tools with hetrogeneous semantics. Paper presented at the ITS 2002, San Sebastian, Spain & Biarritz, France. Suthers, D. D., & Hundhausen, C. D. (2003). An experimental study of the effects of representational guidance on collaborative learning processes. Journal of the Learning Sciences, 12(2), 183-219. Davis, S. (2003). Observations in classrooms using a network of handheld devices. Journal of Computer Assisted Learning, 19(3), 298-307. Chang, C.-Y., Sheu, J.-P., & Chan, T.-W. (2003). Concept and design of ad hoc and mobile classrooms. Journal of Computer Assisted Learning, 19(3), 336-346. Pinkwart, N., Hoppe, H. U., Milrad, M., & Perez, J. (2003). Architectures and educational scenarios for cooperative use of PDAs in wireless networks. Journal of Computer Assisted Learning, 19(3), 383-391. Roschelle, J. (2003). Unlocking the learning value of wireless mobile devices. Journal of Computer Assisted Learning, 19(3), 260-272. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press. Rogoff, B., Baker-Sennett, J., Lacasa, P., & Goldsmith, D. (1995). Development Through Participation in Sociocultural Activity. In J. Goodnow, P. Miller & F. Kessel (Eds.), Cultural practices as contexts for development (pp. 45-65). San Francisco: Jossey-Bass.

370

[32] [33] [34]

[35] [36] [37] [38]

[39]

[40]

J. Roschelle et al. / From Response Systems to Distributed Systems

Gelernter, D., & Carriero, N. (1992). Coordination languages and their significance. Communications of the ACM, 35(2), 97-107. Cabri, G., Leonardi, L., & Zambonelli, F. (2002). Mobile agent coordination models for Internet Applications. IEEE Computer, 33(2), 82-89. Murphy, A. L., Picco, G. P., & Roman, G. C. (2003). LIME: A coordination middleware supporting mobility of hosts and agents (No. WUCSE-03-21). St. Louis: Washington University, Department of Computer Science. Nichols, J. R., Yechezkal, S. G., Paschetto, J., Rush, K. S., & Martin, C. (1999). How the internet helps build collaborative multimedia applications. Communications of the ACM, 42(1), 79-85. Lehman, T. J. (2001). Hitting the distributed computing sweet spot with TSpaces. Computer Networks, 35(4), 457-472. Penuel, W. & Yarnall, L. (2005) Designing handheld software to support classroom assessment: Analysis of conditions for teacher adoption, Journal of Technology, Learning, and Assessment, 3(5). Schank, P., & Kozma, R. (2002). Learning chemistry through the use of a representation-based knowledge building environment. Journal of Computers in Mathematics and Science Teaching, 21(3), 253-279. Vahey, P., Tatar, D. & Roschelle, J. (2004) Leveraging Handhelds to Increase Student Learning: Engaging Middle School Students with the Mathematics of Change. Proceedings of the Sixth International Conference of the Learning Sciences (pp. 553-560). NJ: Lawrence Erlbaum Associates. Brecht, J., Pea, R., & Chung, M. (2002). CML – The ClassSync Modeling Language. Paper presented at the Computer Supported Collaborative Learning Conference, Boulder, CO.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

371

Understanding Students’ Online Forum Usage Paulus Insap SANTOSAa, Gee Kin YEOa, Julian LINa Department of Information Systems, National University of Singapore, Singapore [email protected]

a

Abstract. This paper reports the result of online survey to investigate online forum usage for different purposes. In general, there are two different purposes on why students engage in an online forum: to seek answer and to share ideas. Based on the Technology Acceptance Model and the Social Exchange Theory, we proposed three hypotheses. The hypotheses are to compare whether perceived ease of use and perceived usefulness have different effect on attitude toward seeking answers and on attitude toward sharing ideas, as well as whether intention to seek answers and to share ideas have the same effect on online forum usage. To test the hypotheses, we invited students who have some experiences on using online forum related to the course module. Of the three proposed hypotheses, two of them are confirmed by the data. Keywords: online forum, seeking, sharing, TAM, SET, PLS

1. Introduction Online forum is a very useful tool for education nowadays. It allows students and teachers to communicate asynchronously [29]. It allows users, both the students and the teachers, to post a message, as comments or questions, for others to read, and to which others can respond and answer questions compulsorily or voluntarily [34] and [23]. Due to the unstructured nature of the online forum, anybody could initiate the discussion. The asynchronous mode of communication allows all parties to carefully review the posted information before responding since there is no time limit to respond [28]. Further, parties can participate as many times as they want. Online forum is about sharing information. When individuals visit an online forum, they hope they can get valuable information including important and personally important information. This information may help individuals to cope with decision anxiety [13]. However, due to information overload [24] and cognitive overload [6], individuals may have a hard time to decide whether the information presented to them is valuable. To sustain information communication, Gatignon and Robertson [13] stated that information quality (positive vs. negative), consistency with other information, and credibility of source are very important factors. They also mentioned that poor information may cause interpersonal communication breakdown. As stated by [34] and [23], in general, there are two reasons why individuals may involve in an online forum. The first reason is to seek answers to help them in solving their problems, or just to browse information available in the forum to improve their knowledge. The second reason is to share idea by voluntarily answering the posted questions. As such, the purpose of this paper is to find an answer whether students who participate in online forum have the same intention to seek answers and intention to share ideas, which in turn affect their online forum usage. This paper is structured as follows. Right after this introduction, the theoretical background and hypotheses are presented. The subsequent section discusses the data

372

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

analysis, followed by the discussion of the result. This paper ends with the conclusion and direction for future study.

2. Theoretical Background and Hypothesis 2.1 The Technology Acceptance Model The Technology Acceptance Model (TAM) postulates that two particular beliefs, perceived usefulness and perceived ease of use, are of primary relevance for computer acceptance behaviors [9]. Davis [9] defined perceived ease of use as, “the degree to which a person believes that using a particular system would be free of effort” (p. 320). Perceived usefulness refers to “the degree to which a person believes that using a particular system would enhance his or her job performance” ([9], p. 320). In his framework perceived usefulness is a function of perceived ease of use. TAM, which is an extension of the Theory of Reasoned Action [12], has gain popularity since it was first proposed. Several replications have been made to test the robustness and validity of the questionnaire instrument used by [9] at the first place (e.g. [1] and [17]). In order to explain perceived usefulness and usage intentions in a social setting and cognitive instrumental processes, Venkatesh and Davis [33] have extended the original TAM. Another attempt to extend the original model of TAM was done by Dishaw and Strong [10] in which they combined technology fit construct that provide more explanatory power than either model alone. Several empirical studies have shown how this framework positively predicts intention to use technologies and its actual usage. Selim [30] uses perceived usefulness and perceived ease of use constructs to assess university students' acceptance of course websites as an effective learning tool. Course website usefulness and ease of use proved to be the key determinants of the acceptance and usage of course website as an effective and efficient learning technology. Hong et al. [18] used TAM on their investigation to determine factors that determine users' adoption of digital libraries. Their results strongly support the utilization of TAM in predicting users' intention to adopt digital libraries, and demonstrate the effects of critical external variables on behavior intention through perceived ease of use and perceived usefulness. Lin and Lu [20] used TAM to investigate factors why users accept or reject a Website. They showed that TAM fully mediated the Website usage behavior. Lu et al. [21] have also used TAM to predict wireless Internet usage based on several factors including individual differences and technology complexity.

2.2 The Social Exchange Theory Online forum is one kind of computer-mediated communication [22]. It allows individuals to communicate with others by posting written messages to exchange ideas. It uses asynchronous type of communication. The Social Exchange Theory (SET) [25] posits “individuals evaluate alternative courses of action so that they get best value at lowest cost from any transaction completed” ([15], p. 2). It assumes that the individuals’ willingness to participate in an exchange relationship is to satisfy their needs [2]. Exchange takes place on the basis of the symbolic value attached to things [11]. These valued items could be material, informational, symbolic, etc. [7].

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

373

SET comprises a set of analytical concepts and its corresponding assumptions that are common to any various forms of exchange [25]. These serve as the building blocks of social exchange: actors, resources, structures, and processes. The exchange activities in online forum can be viewed as a reciprocal transaction, in which one student initiates the discussion by posting a message without prior knowledge whether the other party will reciprocate [8]. According to Davis [9], perceived ease of use is a belief that using online forum will be effortless. In term of students’ physical activity, sharing ideas only requires students to do some typing of what they want to share, and seeking answers only requires them to browse. Perceived usefulness is a belief that using online forum would enhance students’ performance. Again, in term of students’ physical activity, seeking answers and sharing ideas would increase browsing experience that perfecting students’ mental model [27]. Based on the above assertions, the following hypotheses are proposed: H1. There is no difference in the strength of the effect of perceived ease of use on attitude toward seeking answer and on attitude toward sharing ideas. H2. There is no difference in the strength of the effect of perceived usefulness on attitude toward seeking answer and on attitude toward sharing ideas. According to the basic concept of SET (e.g. [25]), students will get benefit when somebody give answer to their doubts, and incur cost when students share their ideas. As stated by [15], students will try to get as much benefit as possible with as little cost as possible. As such, we hypothesized the following: H3. The effect of intention to seek answer on online forum usage is stronger than the effect of intention to share ideas on online forum usage. To answer the above hypotheses, a research model as depicted in Figure 1 is proposed.

Figure 1. The research model.

3. Research Methodology 3.1 Subjects and Survey Administration The survey subjects are undergraduate students in Singapore, and most of them are teenagers and young adults in their early or near twenties. They have experience on using

374

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

an electronic discussion forum, especially the one related to a particular course module. Surveys are administered to students using an online survey. Students were invited to voluntarily participating in the survey. All students complete all items within 10 to 15 minutes. Finally, a total of 103 responses are usable. Slightly more male students participate in the survey (52.4%)

3.2 Construct Operationalization Most instrument items are adapted from previous research. Several items are dropped or modified to fit the context of the research. x Following Davis [9], perceived ease of use (PEU) is defined as the degree to which students believe that using online forum to seek answers and to share ideas would be free of effort. Perceived usefulness (PU) is defined as the degree to which students believe that using online forum to seek answers and share ideas would enhance their performance. Perceived ease of use and perceived usefulness is measured using four, and three, 7-point Likert scale items, respectively. Items are adapted and modified from [18] and [33]. x Attitude to seek answers (AttSEEK) and attitude to share ideas (AttSHARE) is defined as a complex mental state involving beliefs, feelings, values, and dispositions to use online forum to seek answers and share ideas, respectively (http://www.cogsci.princeton.edu/cgi-bin/webwn). Each of these attitudinal constructs is measured using five 7-point Likert scale items adapted from [32] and [2]. x Intention to seek answers (IntSEEK) and intention to share ideas (IntSHARE) is defined as the degree to which the subject is inclined to participating in online forum to seek answers, and to share idea, respectively (modified from [16]). Each of these two constructs is measured using four 7-point Likert scale items adapted from [31]. x Online forum usage (OFU) is defined as the frequencies the users visit, contribute to, download from and rate the discussion forum. This is a selfreported usage similar to [5] and [27]. In this study, four-items are included for self-reported usage: 1) the frequency of visiting the forum; 2) the frequency of contributing to the forum; 3) the frequency of downloading; and 4) the frequency of rating the posting by others. The first item 1 is measured by a 7-point scale ranging from “almost never” to “more than once per day”. The other three are measured by a 7-point scale, ranging from “never” to “several times a day”. Due to the limited number of pages, questionnaire is not included in this report. However, they can be obtained from the authors upon request.

4. Data Analysis The described model is analyzed using both SPSS version 11 and Partial Least Squares (PLS graph version 3.0). PLS is a second generation multivariate technique which could assess the measurement model and the structural model (i.e. path coefficients, and R2) simultaneously in one operation. Additionally, PLS is suitable for a small set of sample size and is not sensitive to normal distribution requirement ([5], [14]). For testing path coefficients, t-value is assessed with a nonparametric test of significant known as bootstrapping [4].

375

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

4.1 Descriptive Statistics and Measurement Model Table 1 shows the descriptive statistics. All items have adequate reliabilities ranging from 0.88 to 0.96. Table 2 and Table 3 present discriminant validity of the constructs. Table 2 shows the results of factor analysis produced by SPSS. Table 2 shows that all loadings are higher than 0.60. Moreover, all loadings on their own constructs are higher than on the other constructs, i.e. comparing the loadings in columns shows that each indicator in the block is higher than other indicators from other blocks. Table 1. Descriptive Statistics Construct

PEU PU AttSEEK AttSHARE IntSEEK IntSHARE OFU

Number of Items 4 3 5 5 4 4 4

Composite Reliability (PLS) 0.921 0.885 0.939 0.958 0.947 0.964 0.894

N

Minimum

Maximum

Mean

103 103 103 103 103 103 103

1 1 1 1 1 1 1

7 7 7 7 7 7 4.25

4.37 4.87 4.54 3.62 3.93 4.28 1.99

Std. Deviation 1.29 1.04 0.99 1.06 1.27 1.28 0.70

Table 2. Factor Analysis produced by SPS Rotated Component Matrix (a) Component 4 3 2 1 -0.218 PEU1 -0.020 0.147 0.806 PEU2 0.089 -0.156 -0.236 0.801 -0.009 -0.290 0.152 PEU3 0.736 -0.221 -0.180 0.178 PEU4 0.806 -0.203 0.159 0.006 -0.222 PU1 -0.222 0.170 0.136 -0.228 PU2 PU3 -0.107 -0.318 0.294 0.251 -0.160 0.323 -0.269 AttSEEK1 0.621 -0.013 0.231 -0.341 AttSEEK2 0.632 -0.212 0.252 AttSEEK3 -0.141 0.785 AttSEEK4 -0.223 -0.227 0.160 0.798 0.144 AttSEEK5 -0.242 -0.198 0.761 -0.160 -0.106 0.060 AttSHARE1 0.863 0.075 -0.102 -0.075 AttSHARE2 0.885 0.062 -0.169 -0.136 AttSHARE3 0.901 0.266 -0.276 -0.155 AttSHARE4 0.813 AttSHARE5 -0.149 -0.316 0.175 0.829 IntSEEK1 -0.094 0.245 0.368 -0.212 IntSEEK2 -0.059 0.326 0.349 -0.260 IntSEEK3 -0.130 0.395 0.256 -0.209 -0.196 0.292 0.451 IntSEEK4 -0.198 0.203 -0.140 IntSHARE1 -0.176 0.857 IntSHARE2 -0.215 0.190 -0.157 0.861 IntSHARE3 -0.141 0.205 -0.146 0.823 IntSHARE4 0.184 -0.184 -0.180 0.767 OFU1 -0.050 0.173 -0.089 0.059 OFU2 0.040 -0.005 -0.010 0.106 -0.203 0.053 OFU3 -0.102 -0.080 0.211 -0.198 -0.035 -0.075 OFU4 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a Rotation converged in 7 iterations.

5 -0.114 -0.128 -0.167 -0.140 0.001 -0.006 0.322 0.126 0.205 0.183 0.094 0.068 -0.031 -0.092 -0.144 0.058 -0.039 0.106 0.085 0.071 0.108 -0.005 0.071 0.056 -0.021 0.835 0.867 0.810 0.726

6 -0.090 -0.237 -0.150 -0.132 0.366 0.201 -0.108 0.139 0.180 0.189 0.207 0.319 -0.065 -0.131 -0.065 -0.016 -0.086 0.680 0.721 0.712 0.635 0.124 0.163 0.204 0.360 -0.082 -0.038 0.224 0.176

7 -0.167 -0.171 0.001 -0.149 0.726 0.824 0.614 0.229 0.196 0.135 0.113 0.037 -0.075 -0.184 -0.142 -0.084 -0.111 0.235 0.137 0.168 0.109 0.038 0.107 0.118 0.045 0.196 0.086 0.016 -0.226

Table 3 shows the comparison of the square-rooted of the Average Variance extracted (AVE) produced by PLS, in which the square-rooted AVE for all constructs

376

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

(written as diagonal elements) are higher than their correlation to all other constructs (written as off-diagonal elements). Therefore, the results confirm that the constructs have adequate convergent and discriminant validity. Table 3. Comparison of square-rooted AVE and correlations among constructs PEU PU AttSEEK AttSHARE IntSEEK IntSHARE OFU

PEU 0.864 0.476 0.505 0.390 0.553 0.442 0.357

PU

AttSEEK

AttSHARE

IntSEEK

IntSHARE

OFU

0.849 0.574 0.499 0.528 0.384 0.248

0.869 0.552 0.679 0.552 0.333

0.906 0.367 0.392 0.186

0.904 0.614 0.267

0.932 0.135

0.823

4.2 Structural Model Figure 2 shows the results for the structural model. All hypotheses, except the path from intention to share to usage, are moderately to highly significant. Perceived ease of use has positive effects on perceived usefulness (E = 0.476, t = 4.96), on attitude to seek answers (E = 0.299, t = 2.75), and on attitude to share ideas (E = 0.221, t = 2.03). Perceived usefulness has a positive significant effect on both attitude for sharing and attitude (E = 0.405, t = 3.38), and for seeking answers (E = 0.229, t = 2.75). Attitude for seeking answers has a positive effect on intention to seek answers (E = 0.679, t = 12.60), and similarly attitude for sharing has a positive effect on intention to share as well (E = 0.392, t = 3.81). Intention to seek has a positive effect on usage (E = 0.290, t = 2.26), while intention to share has no significant relationship with usage (E = -0.047, t = 0.3).

#: not significant; *: p < 0.05; **: p < 0.01; ***: p < 0.001 Figure 2. The structural model estimation.

The model shows that, perceived usefulness and perceived ease of use explain about 40% of the variance in attitude for getting answers, while perceived ease of use and perceived usefulness explain about 28% of the variance in attitude for sharing. Lastly, both intentions (to get answers, and to share) explain about 7% of the variance in usage.

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

377

4.3 Path Comparison To test all hypotheses, statistical analyses comparing path coefficients between seeking answers and sharing ideas are performed by similar procedure found in [19]. The results show that there is a significant difference between the effects of perceived ease of use on attitude to seek answers, and on attitude to share (t = 5.425, p < 0.001). On the other hand, there is no significant difference between the effects of perceived usefulness on attitude to seek answers, and on attitude to share (t = 1.536, p > 0.1). Therefore hypothesis H1 is rejected, while hypothesis H2 is supported by the data. Figure 2 shows that there is no need to further test hypothesis H3, because the effect from intention to share ideas to usage is not significant, while the effect from intention to seek answers to usage is significant. Therefore, hypothesis H3 is supported by the data.

5. Discussion and Conclusion This paper examines two activities that students could do in the discussion forum: share ideas and seek answers. Based on the social exchange theory and the theory of technology acceptance model, we formulated three hypotheses. First, there is no difference in the paths from perceived ease of use to both attitude toward sharing ideas and seeking answers. The result, however, does not support our hypothesis. It shows there is a significant difference between these two paths. In Figure 2, the path from perceived ease of use to seeking answers is much stronger than from perceived ease of use to share ideas. Perhaps, from the users’ perspective, seeking answers (browsing) is easier to use than contributing ideas (typing). In our second hypothesis, we hypothesize that there is no difference in the paths from perceived usefulness to both attitude toward contributing ideas and seeking answers, and this hypothesis is supported by the data, and lastly, there is a significant difference in path coefficients between sharing ideas and seeking answers, specifically, the effect of seeking answers is stronger than sharing ideas. The data also supports our hypothesis. This result is consistent with many knowledge management researches, which ask people to contribute ideas by giving incentives. It seems that the user may need some incentives to share ideas. The future study should be directed to study what incentives should be given or provided to the students, in particular, to have a better attitude toward sharing ideas. References [1] [2] [3] [4]

[5]

[6]

[7]

Adams, D.A., Nelson, R.R. and Todd, P.A. (1992) “Perceived Usefulness, Ease of Use, and Usage of Information Technology: A Replication,” MIS Quarterly, 16, pp. 227-247. Ajzen, I. (1991) “The Theory of Planned Behavior,” Organizational Behavior and Human Decision Processes, 50(2), pp. 179-211. Bagozzi, R.P. (1975) “Marketing as Exchange,” Journal of Marketing (39), October, pp. 32-39. Cheney, P. H. (1984) “Effects of Individual Characteristics, Organizational Factors and Task Characteristics on Computer Programmer Productivity and Job Satisfaction,” Information and Management, July, pp. 209-214. Chin, WW. (1998) “The Partial Least Square Approach to Structural Equation Modeling,” in G.A. Marcoulides (Ed.) Modern Methods for Business Research, Mahwah, NJ: Lawrence Erlbaum, pp. 295-336. Conklin, J. (1992) “Hypertext: An Introduction and survey study” in Marca, D. and Bock, G. (Eds.) Groupware: Software for Computer-Supported Cooperative Work, IEEE Computer Society Press, Los Alamitos California, pp. 236-261. Cook, K.S. and Whitmeyer, J.M. (1992) “Two Approaches to Social Structure: Exchange Theory and Network Analysis,” Annual Review of Sociology (18), pp. 109-127.

378

[8]

[9] [10] [11] [12] [13] [14]

[15]

[16]

[17] [18]

[19] [20] [21] [22] [23] [24] [25] [26]

[27] [28] [29]

[30] [31] [32] [33]

[34]

P.I. Santosa et al. / Understanding Students’ Online Forum Usage

Cook, K.S., Molm, L.D. and Yamagishi, T. (1993) “Exchange Relations and Exchange Networks: Recent Developments in Social Exchange Theory,” in Berger, J. and Zeldith, Jr., M. (Eds.) Theoretical Research Programs, Studies in the Growth of Theory, Stanford University Press, Stanford, California. Davis, F.D. (1989) “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly, 13(2), pp. 319-340. Dishaw, M.T. and Strong, D.M. (1999) “Extending The Technology Acceptance Model with Task Technology Fit Constructs,” Information & Management, 36, pp. 9-21. Ekeh, P.P. Social Exchange Theory: The Two Traditions, Heinemann Educational Books Ltd., 1974. Fishbein, M. And Ajzen, I. (1975) Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research, Addison-Wesley Publishing Company. Gatignon, H and Robertson, T.S. (1986) “An Exchange Theory Model of Interpersonal Communication,” Advances in Consumer Research, 13(2), pp. 534-538. Gefen, D., Straub, D.W. and Boudreau, M-C. (2000) “Structural Equation Modeling and Regression: Guidelines for Research Practice,” Communication of the Association for Information Systems, (4:7), pp. 1-77. Hall, H. (2001) “Social Exchange for Knowledge Exchange,” International Conference on Managing Knowledge: Conversations and Critiques, University of Leicester Management Centre. Available at: http://www.bim.napier.ac.uk/~hazel/esis/hazel1.pdf. Accessed on 3 April 2004. Heijden, H.v.d., Verhagen, T., and Creemers, M. “Predicting Online Purchase Behavior: Replications and Tests of Competing Models,” Proceedings of the 34th Hawaii International Conference on System Sciences, 2001. Hendrickson, A.R., Massey, P.D., and Cronan, T.P. (1993) “On the Test-Retest Reliability of Perceived Usefulness and Perceived Ease of Use Scales,” MIS Quarterly, 17, pp. 227-230. Hong, W., Thong, J.Y.L., Wong, W-M., and Tam , K.Y. (2001-2002), “Determinants of User Acceptance of Digital Libraries: An Empirical Examination of Individual Differences and System Characteristics,” Journal of Management Information Systems, 18(3), pp. 97-124. Keil, M, Tan, B.C.Y., Wei, K.K., and Saarinen, T. (2000) “A Cross-Cultural Study on Escalation of Commitment Behavior in Software Projects,” MIS Quarterly, 24(2), pp. 299-325. Lin, J.C.C. and Lu, H. (2000) “Towards an Understanding of the Behavioral Intention to use a Website,” International Journal of Information Management, 20, pp. 197-208. Lu, J., Yu, C-S., Liu, C. and Yao, J.E. (2003) “Technology Acceptance Model for Wireless Internet,” Internet Research: Electronic Networking Applications and Policy, 13(3), pp. 206-222. Mantovani, G. (1996) New Communication Environments, From Everyday to Virtual, Taylor & Francis, Ltd. Mazzolini, M., and Maddison, S. (2003) “Sage, guide or ghost? The effect of instructor intervention on student participation in online discussion forums,” Computers & Education, 40, pp. 237–253 Meyer, J-A. (1998) “Information overload in marketing management”, Marketing Intelligence & Planning, 16(3), pp. 200-209. Molm, L.D. (1997) Coercive Power in Social Exchange, Cambridge University Press. Navarro-Prieto, R.; Scaife, M.; and Rogers, Y. (1999), “Cognitive strategies in Web Searching”, Proceedings of the Human Factors & the Web, June 1999 [WWW document], available at http://zing.ncsl.nist.govt/hfweb/proceedings/navarro-prieto/index. html. Pavri, F. N. (1988) “An Empirical Investigation of the factors contributing to dissertation,” University of Western Ontario, London, Ontario. Pena-Shaff, J.B. and Nicholls, C. (2004) “Analyzing student interactions and meaning construction in computer bulletin board discussions,” Computers & Education, 42, pp. 243–265. Rosman, M. H. (1999) “Successful online teaching using an asynchronous learner discussion forum,” Journal of Asynchronous Learning Network, 3(2). 1999. [WWW document] Available at: http://www.aln.org/alnweb/journal/Vol3_issue2/Rossman.htm. Accessed on 23 March 2005. Selim, H.M. (2003) “An Empirical Investigation of Student Acceptance of Course Websites,” Computers & Education, 40, pp. 343-360. Taylor, S. and Todd, P. (1995) “Assessing IT Usage: The Role of Prior Experience,” MIS Quarterly, 19(4), pp. 561-570. Thompson, R.L., Higgins, C.A. and Howell, J.M. (1991) “Personal Computing: Toward a Conceptual Model of Utilization,” MIS Quarterly, 15(1), pp. 125-143. Venkatesh, V. and Morris, M.G. (2000) “Why Don't Men Ever Stop to Ask for Directions? Gender, Social Influence, and Their Role in Technology Acceptance and Usage Behavior,” MIS Quarterly, 24(1), pp. 115-139. Weisskirch, R.S., and Milburn, S.S. (2003) “Virtual discussion: Understanding college students’ electronic bulletin board use,” Internet and Higher Education, 6, pp. 215–225.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

379

Use of Dataloggers in Science Learning in Singapore Schools SEAH Whye Choo, Daniel K.C. TAN, John G. HEDBERG, KOH Thiam Seng National Institute of Education, Nanyang Technological University Abstract. This paper reports the findings from an online survey administered to 593 science teachers including heads of science departments in 2004. This survey is part of a study initiated in December 2003 to examine the implementation, efficacy and use of dataloggers in Singapore science curricula. Dataloggers were introduced to all Singapore schools during the First Masterplan for IT in Education (1997-2002) to support science learning and experimentation. The aim of the survey is to find out how secondary schools and junior colleges have been supporting and integrating the use of these dataloggers in the teaching and learning of science. The findings from this survey address : x profile of schools and teachers using dataloggers x ways in which dataloggers are used in the science curriculum x teachers’ perceptions on whether pupils were able to interpret and analyse graphs x roles of teachers in preparing pupils and guiding them when using dataloggers in set experiments and inquiry based science experiments

Introduction The implementation of the First Masterplan for IT in Education (1997 – 2002) (MP1) in Singapore saw the provision of dataloggers to all secondary schools and junior colleges in 1997 and primary schools in 1998 to facilitate science learning and experimentation. The broad aims of this provision included exposing pupils to new technology for science learning and providing them with the opportunities to use them; providing a tool for pupils to engage in science inquiry in which pupils use higher order skills like planning, predicting, analyzing and reflecting; and promoting science learning in authentic settings. Training on the use of dataloggers for all science teachers which focused largely on technical know-how, was conducted mainly by vendors. Teachers were guided through set experiments as given in the instructional manuals supplied by the vendors. The Ministry of Education, through its Educational Technology Division, conducted workshops to help teachers integrate the use of dataloggers into the various disciplines in science, namely, physics, chemistry and biology. Fieldwork and projects, such as the Birds in Wetland Reserves, Mesocyclops Project1, were initiated to engage pupils in investigative activities in authentic settings using the dataloggers. Given the significant investment made in equipping schools with dataloggers, this study was carried out to find out the extent of usage, how the tool has been used to support science learning and what are some good practices among schools that could be shared with teachers in other schools. The survey was conducted in July 2004 and formed part of a larger study to understand how dataloggers could be used to support an inquiry approach 1

More details on the project can be obtained at http://www.moe.gov.sg/edumall/etd/meso_web/index.html

380

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

to science learning. This paper reports on some of the findings from the online survey on 593 science teachers (among them, 114 heads of science department) from 137 secondary schools and 14 junior colleges in Singapore. In particular, the findings shared will provide information on the profile of users of these dataloggers, the type of activities in which dataloggers are used, teachers’ perceptions of whether pupils were able to interpret and analyse the graphs and the role of the teacher in preparing and guiding pupils as they embarked on set experiments and inquiry based activities with dataloggers.

Background As we know, the “fundamental nature of science is embedded in inquiry-based learning” (Jarrett, 1999) and practical work in our science curriculum is one excellent platform to encourage inquiry based activities. With increasing emphasis on an investigative and discovery learning, it is imperative that teachers and pupils are provided with the appropriate tools to support them in this endeavour. Dataloggers or data logging systems (also known as microcomputer based laboratory) are one technology that presents great potential in facilitating the data gathering and analysis processes and also allows pupils greater responsibility in their science investigations (Newton 2000). Such technology has been shown to bring about significant benefits to an investigative and exploratory learning environment(Rogers 1994; Barton 1997a). Data logging is a term used to describe an activity in which data are collected by the use of sensors during an experiment and sent to a computer for processing and analysis. A data logging system comprises sensors or probes to detect changes in variables such as temperature, pH, light, an interface or datalogger to collect or store the information, a computer to display the information and a software program for further analysis of the data (Barton, 1997a). Such systems enable both instant capturing of data which can be displayed in real time in the form of tables and graphs on a computer screen and also remote logging which extends practical activities beyond the science laboratory into the field. They also allows for longer term monitoring of variable changes over hours, days or even weeks. Studies have shown that the use of dataloggers with an inquiry-based approach was found to be an effective way to acquire science content and develop understanding of the nature of science (Gipps, 2002). Data logging systems free pupils from the mundane chores of recording, tabulating and plotting data and hence make available time for productive science thinking(Rogers & Wild, 1994). Quality time could be spent on more demanding tasks like discussing, analyzing and interpreting data – asking questions about the data, making predictions, making comparisons, looking for trends, making links between experiment and results, and establishing relationships between variables (Rogers & Wild, 1994, 1996; Rogers, 1997; Barton, 1997a; Newton, 2000). However, for pupils to fully benefit from the time saved from data collection, the teacher plays a critical role in structuring the learning environment and equipping pupils with the necessary skills to explore the data (Rogers & Wild, 1994, 1996; Newton, 1997; Barton, 1997a). The teacher needs to set clear learning goals, design learning activities in which pupils assume greater ownership, guide pupils through probing and open ended questions to foster thinking, encourage and promote a questioning culture among pupils. Pupils must be equipped with skills to pose their own questions, think critically about their data, conduct effective discussions and tap the capabilities of the software to explore the data obtained. Teachers, thus, assume the multiple roles of guide, innovator, motivator and mentor in this technology supported learning environment(Crawford, 2000).

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

381

Despite the well established benefits of using data logging systems for science learning, schools in UK have noted a low adoption rate by science teachers (Rogers & Wild, 1996). The reasons cited for this low usage include access to the equipment, reliability of the equipment, the lack of support structures such as technical support, funds, training opportunities, the confidence level of the teachers in using these tools to facilitate science learning and limited awareness of the benefits of such tools (Rogers & Wild, 1994; Rogers & Wild, 1996). Through our online survey, we were interested to explore the adoption rate of dataloggers among the science teachers in our local context and examine the roles our teachers play as they facilitate investigative science activities supported by the use of these dataloggers.

The Survey The survey was carried out online with the Heads of Science Department (HODs) and science teachers. The questions in the survey covered the following aspects: x demographics on the background of teachers x whether they use dataloggers in their science lessons x the type of activities dataloggers were used with x the roles of pupils and teachers in the datalogging activities x how pupils were prepared to work with dataloggers x teachers’ perceptions on whether pupils were able to interpret and explain graphs x how pupils were guided in their interpretation of graphs x whether pupils were involved in inquiry based investigations x support structures deemed important to help teachers use dataloggers x type of training that would help teachers integrate use of dataloggers x teachers’ perceptions of the usefulness of dataloggers x the difficulties teachers faced when using the dataloggers

Implementation The items in the survey instrument were finalized after two pilot trials. Two versions of the earlier questionnaires were trialed and tested over six months from Jan to June 2004 to gather possible responses and obtain feedback on the clarity of questions. Trial 1, comprising mostly open ended questions, was carried out with 5 HODs and 17 teachers from 12 secondary schools and 2 junior colleges who were attending professional development graduate programs at the National Institute of Education. From the data gathered from the first trial, the instrument was refined to include more multiple-choice questions where options or choices were derived from the participants’ responses. The second version of the survey was tested with 2 HODs and 10 teachers from 4 secondary schools to get another round of feedback. In the final survey instrument, the respondents were allowed to select more than one choice for most of the multiple choice questions and an ‘others’ option was provided for respondents to state other responses not given in the options. A small number of free response questions were retained for teachers to respond to less defined issues like how teachers help pupils overcome difficulties in analyzing graphs, what they perceived were problems faced by pupils when carrying out inquiry-based investigations and how they helped them overcome the difficulties.

382

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

The actual survey was posted online for access by schools for a period of 5 weeks. 158 secondary schools, 16 junior colleges and 1 centralised institute were invited to respond to the online survey. Letters and emails were sent out to Principals and the Science HODs requesting for 6 teachers, including the science head from each school to take part. At the close of the survey, responses were received from 114 HODs (65.1% of an expected number of 175) and 479 science teachers (54.7% of an expected number of 875 teachers) from 151 schools. The overall response rate to the survey was 56.5%.

Survey Findings Teachers and HODs who took part in the survey came from 14 junior colleges (2 year programme which prepares 17-18 year olds for “A” level examination), 7 independent schools(4 year secondary education for 13-16 year olds which enjoy full autonomy in implementation of school programmes and administration), 19 autonomous schools (4-5 secondary education for 13-17 year olds which have some autonomy to provide a wider range of innovative and enrichment activities) and 111 other secondary schools (inclusive of government and government aided schools, which prepares 13-17 years old for “O” level examination). The highest response rates were from the independent schools (62.5%, 7 out of 8 independent schools) and junior colleges (66.7%, 14 out of 16 JCs).

Profile of users, past users and non-users In our analysis, the respondents were classified into 3 categories, namely the users (teachers who used the dataloggers in the recent 1-2 years), past users (those who used dataloggers more than 2 years ago), and non-users (those who never used dataloggers). Among the 593 respondents, 66.4% (394) were users, 15.3% (91) past users and 18.2% (108) never used dataloggers. The highest percentage of users came from the independent schools (90.0%, 27 out of 30 respondents), followed by junior colleges (76.5%, 52 out of 68 respondents) and the autonomous schools (75.6%, 59 out of 78 respondents). About 85% of 392 respondents indicated that they used dataloggers only in one to three topics in their science syllabi. Teachers with less than 5 years of teaching experience formed the largest group among the respondents (36.0%, 214 out of 593). However, it was interesting to note that teachers with more than 20 years of experience in science teaching formed the highest proportion of users (75.7% of 115 respondents) in contrast to the youngest group of teachers with less than 5 years of teaching experience who formed the smallest percentage of users (57.5% out of 214 respondents). One would expect younger teachers to be more IT savvy than their older colleagues and thus be more ready to experiment with technology. One HOD remarked that as new teachers, the less experienced teachers were probably adjusting to the demands of the schools and unable to find time to experiment with the use of technology. Unlike the younger teachers, the more experienced teachers were also trained in using the specific brand of dataloggers by vendors when the dataloggers were disseminated to schools, and many could also have attended workshops on integrating the use of these dataloggers into the various science disciplines. When asked why they stopped using dataloggers, the 91 past users cited the lack of time (47.1%) and equipment-related issues (35.1%) like insufficient notebook computers, the dataloggers are cumbersome to set up and involved too much troubleshooting, as the two main deterrents. Besides the need for time to complete syllabuses, many past users

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

383

expressed that designing the task, setting up the dataloggers, training pupils to use them and interpret data are all time consuming factors. Though they have never used dataloggers, 81.3% of the 108 non-users saw potential in the use of dataloggers for science learning, particularly, for demonstrations, in project work, fieldwork and enrichment.

Use of dataloggers Survey findings show all users (both current and past) used dataloggers mainly for demonstrations (59.4%) and set experiments from workbook and worksheets (56.3%). Usage is much less in activities like enrichment (19.6%), fieldwork/project work (19.0%), student initiated investigations (11.5%) and co-curricular activities (5.6%). This trend is common across users from the different types of school, among respondents with different years of science teaching experience and across the 3 science disciplines. Thus, it can be seen that the use of dataloggers in our science classroom is still largely teacher-directed and mainly for verification experiments. There is very little evidence to show that the potential of the tool is tapped to engage pupils in inquiry based activities.

Preparing pupils to work with dataloggers In preparing pupils to work with experiments using dataloggers, all users generally showed pupils how to set up and use the sensors (84.3%) and explained the functions of the different components e.g. sensors, logger, software (72.0%). As reflected from the data in Table 1, more teachers were concerned about familiarizing pupils with the technical skills of operating the dataloggers rather than training pupils to interpret and discuss their results. However, it was noted that a comparatively larger percentage of respondents from the junior colleges taught their pupils how to analyse and explain data as compared to their colleagues from the secondary schools. Table 1. Ways in which respondents from different types of schools prepare pupils to work with dataloggers Show how to Show how to Teach pupils Type of school Explain navigate use sensors to analyse and functions of components software explain graphs Autonomous (n=70) 50 (71.4%) 48 (68.6%) Independent (n=28) 23 (82.1%) 23 (82.1%) Other secondary (n=322) 226 (70.2%) 209 (64.9%) Junior College (n=62) 47 (75.8%) 48 (77.4%) All users (n=485) 349 (72.0%) 330 (68.0%) A respondent may tick more than one type of activities

57 (81.4%) 27 (96.4%) 263 (81.7%) 60 (96.8%) 409 (84.3%)

43 (61.4%) 22 (78.6%) 209 (64.9%) 51 (82.3%) 327 (67.4%)

Pupils’ interpretation of graphs A high percentage of teachers (82.9%) indicated that their pupils were able to interpret the graphs generated by the datalogging systems and also explain them. More current users (87.3%) perceived that their pupils were able to interpret graphs as compared to past users (63.7%). All respondents from independent schools as well as those teaching junior colleges showed greater confidence in their pupils’ ability to make sense of the graphs obtained.

384

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

Table 2. Distribution of respondents from different types of schools on pupils’ ability to interpret and explain graphs Type of school Yes/possibly No Autonomous (n=70) 60 (85.7%) 5 (7.1%) Independent (n=28) 28 (100.0%) 0 (0.0%) Other secondary (n=322) 255 (79.25%) 51 (15.8%) Junior College (n=62) 57 (91.9%) 5 (8.1%) All respondents (n=485) 402 (82.9%) 63 (12.8%)

To gain a better understanding of what led teachers to perceive that their pupils were able to interpret graphs, teachers were asked to cite one situation in which pupils exhibited such ability. The 285 positive responses could be broadly grouped into the following categories. In general, pupils were observed to be able to: x read off (e.g. maximum or minimum value, intercept values, pH) or derive values(gradient for velocity/acceleration, period, frequency) or identify segments ( plateau showing change of state) from the graphs and explain the conceptual meaning of these values (24.6%) x correlate experimental data with theoretical concepts (11.9%) x infer and derive conceptual understanding from the shape of the graph obtained (54.7%) x deduce the relationship between two variables (effect of the independent variable(s) on the dependent variable) (8.8%) Teachers’ beliefs that their pupils were able to interpret graphs were largely based on their observations though it is unclear how teachers ascertain the extent of pupil understanding or could it just be a general sense derived from choral responses. There are also quite many examples cited which(24.6%) merely require pupils to identify certain values which are significant to concepts learnt or calculate values of the desired variables from the graphs as required by the worksheets which hardly constitute interpretations. However, one observation from the examples cited is that with lesser time spent on recording data and plotting graphs which is one key affordance of the dataloggers, more time could be freed for teachers to discuss the significance of the graphs with pupils. Doing the experiment alongside with the graphs displayed instantly by the dataloggers enabled the pupils to better visualize and explain the phenomenon happening in the experiments. This could be one crucial step to close the perceived “gap” between theory and practical science among many of our pupils. Among the 61 teachers who indicated that their pupils were unable to interpret graphs, 51 (83.6%) respondents were from the `other secondary schools’. It could be gathered from the 55 responses on the common difficulties encountered by pupils were that they were not able to make sense from the graphs and explain the trends observed (23.6%), some were unable to make connections between the graphs with what they learnt in theory (21.8%) and there were those who are lacking in mathematical concepts in graphing (10.9%). To help pupils overcome these difficulties, steps taken by teachers included explaining to pupils the meaning of the graphs obtained, reinforcing the theory behind the graphs, asking guiding questions on what pupils can expect to see from the graphs and even showing them model answers. In cases where the pupils were weak in mathematical concepts, teachers also took time to explain basic graph concepts or resorted to using mainly the simplest graph functions available in the software.

Inquiry based investigations – pupil initiated experiments

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

385

Only 22.5% (out of 485 respondents) involved their pupils in planning and carrying out pupil initiated experiments. This low figure affirmed earlier findings on the use of dataloggers which showed that respondents involved their pupils mainly in verification type of experiments. However, it was noted that a higher percentage of junior college teachers involve their pupils in pupil initiated inquiry activities. That could be attributed to the greater emphasis on project work at the `A’ levels. More Biology teachers (31.8% of 110) involved their pupils in inquiry based activities than Chemistry(21.1% of 185) or Physics(21.2% of 245) teachers. The findings also showed a rising trend in the number of teachers involving pupils in inquiry based activities from the past to the current users across all 3 science disciplines. Respondents indicated that pupils were given the opportunity to plan their own experiments using dataloggers during project work, practical sessions on skill 4(SPA), science competitions, science fairs, enrichment modules for lower secondary, science research and co-curricular activities (CCA). Thus, it is evident that pupil directed activities happened predominantly in co-curricular activities. When planning these experiments, pupils were largely involved in selecting and control of variables and determining procedures as shown in Table 3. Table 3. Percentage of responses showing the ways in which pupils were involved in planning their own experiments Plan Select Craft aim/ Decide on Determine Generate hypothesis procedures variables the objective the type of data to apparatus required collect Number (n=109) 51 67 64 58 74 76 Percentage (%) 46.8 61.5 58.7 53.2 67.9 69.7 A respondent may tick more than one type of ways Table 4. Percentage of responses showing the guidance teachers gave to help pupil plan their experiments by different types of schools Type of school Advice on Explain the Guide them Guide them Provide safety steps on data to select feedback issues involved in collection and control on the planning of feasibility an variables of the experiment experiment Autonomous (n=18) 10 (55.6%) 12 (66.7%) Independent (n=7) 4 (57.1%) 3 (42.9%) Other secondary (n=65) 37 (56.9%) 34 (52.3%) Junior College (n=17) 10 (58.8%) 6 (35.3%) All respondents (n=109) 63 (57.8%) 57 (52.3%) A respondent may tick more than one way of guidance

16 (88.9%) 3 (42.9%) 39 (60.0%) 12 (70.6%) 72 (66.1%)

15 (83.3%) 2 (28.6%) 42 (64.6%) 12 (70.6%) 73 (67.0%)

15 (83.3%) 6 (85.7%) 43 (66.2%) 8 (47.1%) 74 (67.9%)

So as to gain a better awareness of the kind of scaffolding teachers provide for pupils in such activities, respondents were asked what kind of guidance they gave to their pupils. Findings in Table 4 showed that a high percentage of respondents provided feedback to their pupils on the feasibility of their experiment. Other forms of assistance included guiding pupils to select and control variables and advising them on the type of data to collect. Fewer teachers in junior colleges explained the steps involved in planning an experiment and gave feedback to their pupils. This could be because the older pupils in junior colleges are expected to manage on their own. Independent school teachers

386

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

appeared to provide less guidance in variable control, data collection and bringing pupils through the steps in planning as compared to their colleagues in other secondary schools. This is not unexpected as their pupils being intellectually more capable and likely to be more self driven. Analysing data and drawing conclusions When asked how they helped pupils analyse data and draw conclusions, responses from 60 teachers reflected that they helped to link what the pupils were doing in the experiments to the theory they had learnt previously (25%), examined data together with pupils, checked for consistency in data (33.3%), asked leading and probing questions to set pupils pondering and provided constructive feedback (31.7%). Some help rendered by the teachers were: “Elicit by pointing out salient points on graphs” “Guiding them to look for treads and patterns in results, discussing anomalous results” “Getting them to refer back to the earlier question in their problem definition” “Pointing pupils to relevant references, providing facts and background knowledge where appropriate, ask probing questions” “Carry out discussions based on data, look for trends in data and infer/explain”

Conclusion Despite the many potential benefits of dataloggers for supporting science inquiry processes, it is quite evident from the survey findings that teachers have not yet fully exploited the use of dataloggers to support inquiry-based learning of science. Teachers were using them mainly in demonstrations and pupils were largely engaged in laboratorybased verification type of experiments and had few opportunities to participate in inquirybased activities. Pupils were equipped mainly with the technical skills on how to use the various sensors and briefed on the functions of the different components. Comparatively, little was done to prepare them to examine and discuss graphs and to explore the data with the software. The survey also provided an indication that teachers were generally confident that their pupils were able to interpret and explain graphs. The findings reflected the very low percentage of teachers who involved their pupils in inquiry based activities and many of these happened outside the regular science curriculum. It was clear from the responses that teachers need to provide pupils with more scaffolding from the conceptualisation of the task through to interpreting data and drawing conclusions. In order to use dataloggers more effectively in science learning, the role of the teacher needs to be reformulated from one of an instructor/disseminator of knowledge to that of coach, resource procurer and mentor (Renzulli, 2004). Hence, to encourage more pupil initiated activities using dataloggers in science classrooms, in addition to helping teachers integrate the use of dataloggers in the curriculum context, it might be necessary to emphasize the advantages of dataloggers, share exemplary practices of effective use and explore with teachers, effective scaffolding structures and ways of engaging pupils in inquiry based activities. Thus, to encourage wider adoption of the tool, there is a need to routinely support teachers in their use of dataloggers, especially for the multiple roles they have to assume as they transit from being an instructor to a facilitator of science inquiry.

W.C. Seah et al. / Use of Dataloggers in Science Learning in Singapore Schools

387

References 1.

Barton, R. (1997a). "Does data-logging change the nature of children's thinking in experimental work in science?" In B.Somekh and N. Davis (eds), Using Information Technology Effectively in Teaching and Learning (London: Routledge): 63-72. 2. Crawford, B. (2000). "Embracing the Essence of Inquiry: New Roles for Science Teachers." Journal of Research in Science Teaching 37(9): 916-937. 3. Jarrett Denise, J. H. (1999). "Science Inquiry for the Classroom - A Literature Review." The Northwest Regional Educational Laboratory Program Report. 4. Gipps, J. (2002). "Data Logging and Inquiry Learning in Science." Australian Computer Society, Inc. 5. Renzulli S. Joseph, M. G. a. S. M. R. (2004). "A Time and Place for Authentic Learning." Educational Leadership : Teaching for Learning 62(1): 73-77. 6. Newton, L. (1997). "Graph talk: some observations and reflections on students' data-logging." School Science Review 79(287): 49-54. 7. Newton, L. R. (2000). "Data-logging in practical science: Research and Reality." International Journal of Science Education 22(12): 1247-1259. 8. Rogers, L. (1997). "New data-logging tools - new investigations." School Science Review 79(287): 6168. 9. Rogers, L. T., & Wild, P. (1994). "The use of IT in practical science - a practical study in three schools." School Science Review 75(273): 21-28. 10. Rogers, L. T., & Wild, P. (1996). "Data-logging: effects on practical science." Journal of Computer Assisted Learning 12(3): 130-145.

388

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

A Human Resource Model and Evidence Based Evaluation – Ontology for IT Skill Standards – Kazuhisa SETA*1, Mitsuru IKEDA*2, Kenji HIRATA*3, Yusuke HAYASHI*2 and Ken KURIYAMA*4 *1 Osaka Pref. University, Japan *2 JAIST, Japan *3 The Sanno Institute of Management, Japan *4 Digital Contents Division, Gakken Co. Japan [email protected], {ikeda, yusuke}@jaist.ac.jp, [email protected], [email protected]

Abstract. Putting the right person in the right post and planning human resource development (human resource management: HRM) are important decision making tasks for managing company. The basis for HRM is the accurate grasp of the competency of human resources as intellectual capital. Our research goals are (i) to carefully elucidate the principle criteria for HR assessment and systemize them as an ontology, and (ii) to create a human resource and KSAOs model based on the ontology. This paper introduces our ontological approach to realize those goals. Keywords: evidence based evaluation, IT skill ontology, human resource model, IT skill ontology

Introduction Putting the right person in the right post and planning human resource development (human resource management: HRM) are important decision making tasks for managing company. They strongly affect the future direction of an organization. The basis for HRM is the accurate grasp of the competency of human resources as intellectual capital. Nevertheless, we cannot deny that we tend to assess human resources (HR) based on evaluation bias such as image, halo or brand because causal relations are not explicit among performance, the competency that supports that performance, project outcomes, and criteria for evaluation of abilities. To control and change such tendencies in the current state of affairs, it is necessary to develop and share a methodology for evidence-based HR evaluation by establishing a system of knowledge, skill, ability, and other characteristics (KSAOs) as a principle criterion for conducting business affairs. In the ontological engineering field, much effort has been devoted to clarifying sharable principle knowledge as ontologies in various domains such as task ontologies for diagnosis, design, and education, and domain ontologies for biology, device, and function [1] [4]. In general, human perceptions fluctuate, say, in interpreting the functions of a device from its behaviors. Kitamura [3] and colleagues have attacked the problem and built a sophisticated device and functional ontology that serves principle knowledge to interpret a device’s functions embedded in a system from its behaviors: they proposed the concept of “functional

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

389

topping”, which specifies the axioms of interpretation of a device’s behaviors. In other words, it clarifies the principle knowledge that can explain humans’ abstraction mechanism in the artifact-engineering domain. Thereby, the validity of the interpretations (interpreted functions) of the observations (observed behaviors) based on that ontology is guaranteed. The HR evaluation task is no exception: there is fluctuation in the interpretation of an individual’s abilities from the observed human performance. It is especially difficult to interpret human ability accurately because the rule that governs the behavior is not clearly understood and human behavior is neither consistent nor stable. The fluctuation problem engenders other problems in HRM (described in detail in the next section). Therefore, it is necessary to clarify and accumulate principle knowledge to reduce the inherent fluctuation in HR evaluation. The “IT skill standard” (ITSS), which was drawn up by the Japan Ministry of Economy, Trade and Industry (METI), is a typical effort to eliminate such fluctuation. That standard was intended to produce explicit principle criteria for evaluating human abilities. The ITSS provides a system of IT profession and skills as a dictionary to grasp the abilities of individual IT professionals. The ITSS is intended to encourage adequate development of highly skilled persons in the IT industry through career development, employment, and replacement. The ITSS specification document [2] is appreciated: it specifies various ideological skill concepts by making the best use of representational advantages of the document medium. Notwithstanding, the builders’ intentions are missed because of the limitation of the representational characteristics of the document medium. Thereby, it is debatable whether ITSS can eliminate brand-based or image-based evaluation. The requirement of using ITSS adequately for employers, companies, industry, and some other users is quite strong: ITSS has a strong impact on the IT industry and market in Japan. Several difficulties exist, especially in the practical uses of the ITSS, which focus on tacit and latent skill concepts. It remains unclear: (a) how to infer and evaluate a human’s skill according to observed human performances; (b) to what indexes the assessor should refer; and (c) what kinds of information should be recorded as evidence for evaluation. Our research goals are (i) to carefully elucidate the principle criteria for HR assessment and systemize them as an ontology, and (ii) to create a human resource and KSAOs model based on the ontology. This paper introduces the concept of evidence-based evaluation. Then we clarify the intentions of the ITSS builders because it is important to reduce fluctuation and adequately use ITSS. Furthermore, we systemize a fundamental concept for evidence-based evaluation and a human resource model based on the ontology as a first step toward realizing our goals.

1. Evidence Based Evaluation 1.1 Issues of Current HR Assessment Table 1(a) illustrates issues in assessing an employee’s performance and causal side effects to human resource development and its utilization. Assessment results lack validity and reliability: (1) assessment activities are performed based on the HR image or brand; (2) results tend to be affected by the company’s achievement results and are made less in consideration of the relationships between individual performance in daily work and ability; (3) results are vulnerable to cognitive biases – halo effects, leniency tendencies, stringency tendencies, and central tendencies; and (4) HR growth information cannot be adequately accumulated and renewed. Consequently, it is crucial to clarify principle criteria to assess human ability, and a management mechanism for HR’s growing information.

390

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

Those problems propagate to human resource development (HRD) and human resource utilization (HRU) tasks. Assessment results are valuable information to perform HRD/U tasks. However, we cannot perform HRD/U tasks based on them because they do not reflect that person’s demonstrations in daily work and abilities. Consequently, an HR cannot develop individual abilities in performing daily work, grasp self-growing processes objectively, and draw self-career paths based on those results. Furthermore, it is not possible to hire or replace a worker adequately based on their ability information and project experiences because of less data expected for their personal history. 1.2 Evidence Based Evaluation We call assessment based on actual performance, ability, and fair assessment methods with clear measures “evidence based evaluation (eBase)”. Table 1(b) shows tasks to realize eBase and advantages of the eBase in HRD/A. We seek to reduce problems described in the previous section by systemizing an ontology that reduces fluctuation in evaluation and facilitates an eBase that is based on the ontology that (a) Issues in HR assessment

(c) Tasks from an ontological engineering viewpoint Contribution Tasks

HRD

HRD

HRU

HRU

Difficulties/Tendencies zAssessment results lack zTo clarify and produce validities and reliabilities because they tend to be made principle indicators for based on an image or brand of evidence based evaluation the individual HR z To make methods for zAssessment results tend to capturing the ability be affected by the company’s achievement concept explicit results z To make clear indicators zAssessment results tend to of abilities and the level be made less in concept of maturity consideration of the explicit relationships between HR’s z To produce sharable skill performances in daily work and abilities systems in industries zTo realize assessment zAssessment results tend to according to the results of methods based on clear be affected by the above two items indicators that specify level assessor’s cognitive biases: z To clarify causal relations definitions of maturities halo effect, leniency tendency, central tendency, among performances and and so on abilities zHR expertization information zTo realize the adequate method cannot be accumulated to record and update HR zTo build an information adequately growth information model for HR abilities based Side Effects Advantages on principle indicators zIt is not possible to use zIt is possible to use the z To build an HR model for assessment results as valuable assessment results as valuable accumulating ability information for HRD information for HRD information that reflects zIt is not possible for an HR zIt is possible for an HR to the facts of performance to develop self-abilities in develop self-abilities in daily work because his/her daily works because demonstration in daily assessment information assessment information works does not reflect one’s reflect her performances in z To build an HR model for performances in daily one’s daily works and works and abilities. abilities recording HR growth information zIt is not possible to grasp zIt is possible to grasp HR’s HR’s growing processes. growing processes. zTo build information systems zIt is not possible to draw zIt is possible to envision z To build an assessment one’s career paths. one’s career path. support system zIt is possible to use assessment zIt is not possible to use assessment results as valuable results as valuable information z To build an HR inforation information for human for human resource management system that resource arrangement. arrangement. maanges HR ability and zIt is not possible to hire or zIt not possible to hire or growth information. replace workes because of replace workers adequately z To build an assessment limited data of HR abilities based on evidence based expect for the personal information on HR abilities training system history. and project experiences. Table 1: Issues to realize evidence based evaluation

HR Assessment

HR Assessment

(b) Tasks to realize eBase and advantages of eBase Tasks for eBase zTo realize evidence based evaluation according to the facts of demonstrations of the HR abilities, clear indicators and fair evaluation methods zTo realize evaluation methods for separate assessment of achievements and HR abilities zTo realize assessment methods based on objective evidences on performance in daily work

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

391

satisfies the following requirements for HR assessment. i. HR assessment must be performed for individual viewpoints of outcomes and HR abilities. An HR assessment must be made based on the facts of demonstrated ability. ii. HR assessment must be done based on the fair and unbiased measurement methods. iii. We must correct tendencies of assessment dispersion resulting from cognitive biases: halo effect, the leniency tendency, the stringency tendency, and the central tendency. The halo effect describes the tendency to assign a positive or kind assessment to an HR who has one specific positive trait. Leniency and stringency tendencies are those that respectively assign positive and negative assessments to an HR based on traits of the assessor. A central tendency is one by which HR ability is assessed as “median” or “average”. iv. HR information must include the educational history (degrees awarded), educational experiences (completion), past job experiences (post, position, and technical) or certifications and licenses. Furthermore, the identity of the assessor must be recorded by referring to facts, measures and measurement methods. v. HR information must be changed and versioned adequately in cases of growing by learning and experience. For (i), (ii), and (iii), it is necessary to develop fair and clear principle measures for eBase and framework of information system supports. To realize (iv) and (v), it is necessary to develop an HR ability record model adequately to accumulate ability information of HR and their evidences both adequately and permanently. 1.3 Tasks from an ontological engineering viewpoint Table 1(c) lists tasks from an ontological engineering viewpoint to be carried out for eBase. Explicit Representation of Principle Indicators: We must produce explicit principle measures (criteria and level) to realize an eBase: how to capture ability concept and how to accumulate evidence information. Furthermore, we must develop a sharable skill system in industry and a causal model among competencies and performances based on those measures. Therefore, we have worked to develop ITSS ontology by clarifying builders’ intentions that are missed in the ITSS document and removing vagueness of terms that appear in the document. We will explain our ontology in greater detail in the next section. Building a Human Resource Model: We must build an HR model suitable for recording and organizing HR ability information (growing process, working performance information, and abilities) to perform HRD/A adequately. This paper explains an HR ability model based on the ITSS ontology. Information System Development: We can work toward realizing eBase by developing information systems such as (a) assessment support tools, (b) HR information management tools, and (c) HR assessment training tools. For example, it is useful to embed a document retrieval function in the assessment support tool: it aims to support assessors’ practical eBase processes by retrieving and showing adequate measures, performance information from a personal record and assessment methods of ITSS according to their assessment objectives. The HR ability information management tool manages the HR information of assessment and education and supports rational career path designs and human resource arrangement. The HR assessment training system contributes to diffusing the eBase concept properly. Various virtual agents appear in the IT performance assessment training system, each of which plays the role of an HR. Each agent has information related to individual project experiences and performance demonstrations. Learners can play roles of assessors (or assessment subjects, advisors, and so on) and gain virtual experiences of performing HR assessment. Then the system suggests issues for consideration on assessing methods or performance information that are referred if the learner, playing an assessor, inputs unreasonable evaluation results.

392

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

2. Conceptual Analysis for Evidence Based Evaluation – Taking ITSS as an example – 2.1. Basic Concepts for Evidence Based Evaluation We must elucidate and model the HR assessment guidelines that are implicit in ITSS specification for enabling eBase. ITSS ontology specifies the concepts used for modeling the builders’ implicit intention. It is important to capture the principle conceptual structure that is characteristic for constructing a conceptual model of HR records. For example, the term “human resource” appears frequently in the ITSS document corresponding to the different kinds of our conceptual perceptions as follows. x An individual human resource occupies a variety of job classifications and performs various kinds of projects [We have a permanent perception of a human resource and a changing, discrete perception of a human resource.]. x A human resource acquires experiences and displays abilities in performing a project [our perception of a human resource during a period of performing a project]. x A human resource continues to develop skills by performing various projects [our perception of a human resource as a subject of a continuous growing process]. When we model a human resource for eBase, it is noteworthy that plural conceptual structures of a human resource are generated from multiple viewpoints. Our study introduces the following concepts to disentangle them adequately. x Instance: represents the identity of an individual and captures our permanent perception. x Occurrence: represents our perception of an instance in space-time. x Version: represents a change history (continuous perception) of an instance Figure 2 shows an image of conceptual perception of a human resource that can be represented by those concepts. The instance represents that a human resource (I) occupies the project management job classification (PM). Perception of a human resource for the duration of performing a project is represented as a project occurrence of the human resource. Figure 1 represents that the performance result of an instance (I) in performing a project (o1) is “good” but skill-III was “not displayed (NG).” On the other hand, it represents that the performance result of (I) in another project (o2) is “no good” but skill-I and skill-III were “displayed” and skill-II was “not displayed.” The version concept captures the expertization processes by performing various projects properly. For example, the expertization processes of skillIII through performing projects o1 and o2 is representable as a version change. Furthermore, it can be represented that skill-III was displayed in performing project o3 as a result of the expertization. Building an ontology that systemizes the concepts that constitute our perception of HR, ITSS builders’ intention, which Figure 1: Relations between Instance, Occurrence and Version

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

393

are not describable in documents at all, can be clarified. 2.2 Key Performance Indicator and Maturity Skill systems are systemized for respective job classifications: careers of IT professionals are classified into 11 categories and 38 fields in ITSS. Here, we analyze career level and skill maturity level indicators in detail; those are closely related to eBase, based on the fundamental concepts in previous section. Two important viewpoints exist to assess HR performance. One is a viewpoint of assessing performance results (outcomes) of projects; the other is one of doing maturity level in performing various projects. In the ITSS document, career level and skill maturity level are specified as indicators for each viewpoint. Nevertheless, it remains unclear that each indicator captures what parts of our perception of HR. Our research defines each indicator based on the specification of conceptualization in the previous section. x Career level: A state concept for which the instance modifies a project occurrence. It stratifies levels of one’s performance and results of projects by key performances in the field and specifies each criterion. x Skill maturity level: A state concept for which the instance modifies a version of a skill. It stratifies the level of one’s skill maturity and specifies each criterion. Figure 2 shows systems of the career level and skill maturity level. The tick arrow represents an “is-a” relation. For example, the “PM career level” concept is a super concept of “System Integration Project Management (SI-PM) career level”. Symbol “s” attached to a concept definition represents that the concept is a state concept. A state concept represents an entity’s status that it modifies. Consequently, the state concept of “career level” represents a status of an “HR occurrence”: its entity represents a level of a performance result of a project that an individual HR performed. State membership represents conditions that the modified subject must satisfy. A skill level system is systemized for each job classification. The most upper-level concept – “skill maturity” – is specified to the “PM skill maturity” that corresponds to the project management job classification. It is also specified to more detailed skill maturity (integration management skill maturity). As a result, assessment criteria for individual skills are specified as the state membership at the bottom level. According to this specification of conceptualization, versioning axioms of HR information are specified explicitly as “pre-nest status relation”. Thereby, when an assessor assessed integration management skill that had evaluated level-6 at level-7, then the version of the integration [Conceptual Hierarchy of Career Level] s SI-PM Career Lev.-7 s PM Career Lev. management skill mod_target mod_target part-of part-of SI Project (o) Project (o) of the HR is state_membership state_membership axiom axiom SI Criteria#䋲 Constraint Concept updated according Constraint Concept s SI-PM Career Lev. to the axioms. s SI-PM Career Lev. 6 mod_target mod_target part-of part-of Consequently, SI Project (o) SI Project (o) state_membership state_membership axiom axiom expertization SI Criteria#1 Constraint Concept SI Criteria#䋳 Constraint Concept processes of HR s eBiz-PM Career Lev. skills are recorded [Conceptual Hierarchy of Skill Maturity Level] s PM Skill Lev. s IM Skill Maturity Lev.-7 continuously and mod_target mod_target part-of PM field part-of renewed to reflect IM Skill (v) Common Skill (v) prestate_membership next axiom state_membership axiom the latest state, Skill Constraint IM Criteria#2 Skill Constraint status s-is-a based on the s IM Skill Maturity Lev.-6 s IM Skill Maturity Lev. mod_target mod_target ontology. part-of part-of IM Skill (v) IM Skill (v) axiom

state_membership IM Criteria#1 Skill Constraint

axiom

state_membership IM Criteria#3 Skill Constraint

Figure 2: Conceptual Structure of Career Level and Skill Maturity Level

394

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

2.3 Assessment activity The ITSS specification document specifically addresses issues of career level and skill maturity level indicators, but does not clarify their relations with assessment methods. Therefore, we must consider the following issues and elucidate the meanings of assessment activities: (i) the purposes for which each assessment activity is performed; (ii) what part of our conceptual perception of HR an assessor should focus on; (iii) what kinds of information each activity should input; and (iv) what kinds of effects should occur. Figure 3 shows a conceptual structure of assessment activity that we adopted. We divide definitions of assessment activity into two types: career level assessment and skill maturity level assessment definitions. This division helps to explicate what part of the conceptual perception of HR each assessment activity should be referred to. We mainly explain the conceptual structure of assessment activity, especially examining the relationship between assessment action and assessment method. The definition, skill maturity level assessment action for integration management skill, specifies that: the assessor is a human subject of the action; the assessment target (in%target_skill), as input, is a version of an integration management skill (IM Skill(v)); and it refers (in%criteria) to a state membership as an assessment criterion. Furthermore, it specifies that the assessment action uses (in%methods) an assessment method that is specified as a skill maturity level assessment method for IM skill and outputs (out%gurantee) a version of integration management skill maturity (the skill modified by its maturity level) as the result. Figure 3(b) shows a conceptual definition of an assessment method for IM skill. The definition specifies that the assessment target (in%target_skill) is a version of an IM skill. It refers (in%reference) to a set of occurrences ($occurrence) and outputs (out%gurantee), a version of integration management skill maturity (the skill modified by its maturity level), as the measurement result. It is not represented explicitly in the figure. However, the axiom also stipulates that the input (output) of the assessment activity and the assessment method mutually correspond. Consequently, the meaning of each assessment activity can be clarified in terms of the conceptual perception of HR that forms a harmonious overall assessment activity. It is noteworthy that, by separating and integrating the assessment action and assessment methods, the assessment action can adopt (a) Conceptual Hierarchy of Assessment Actions is-a Maturity Assessment Action for IM Skill various assessment subject%assessor methods, each of part-of Human which can use in%target_skill part-of Assessment Methods A for IM Skill IM Skill (v) different HR in%target_skill in%criteria part-of part-of IM Skill (v) state membership for s information, even IM skill Maturity in%reference in%method part-of Assessment Method part-of $occurence though their criteria for IM Skill in%criteria out%gurantee part-of IM Skill Maturity part-of are mutually IM Skill (v) modified IM Criteria#4 Criteria by a Maturity Level out%gurantee compatible. part-of IM Skill (v) modified by a Maturity Level (b) Conceptual Hierarchy of Assessment Methods Thereby, we can is-a Assessment Method for IM Skill Assessment Method B for IM Skill specify separate in%target_skill in%target_skill part-of part-of IM Skill (v) IM Skill (v) measurement criteria in%reference in%reference part-of part-of by keeping the $occurence $occurence in%criteria in%criteria rationality to the part-of part-of IM Skill Maturity IM Skill Maturity IM Criteria#5 Criteria Criteria out%gurantee out%gurantee assessment criteria part-of part-of IM Skill (v) modified IM Skill (v) modified by a Maturity Level by a Maturity Level specified in the Figure 3: Conceptual Structure of Assessment Action and Method assessment activity.

K. Seta et al. / A Human Resource Model and Evidence Based Evaluation

395

3. Human Resource Model Based on Ontology Figure 4 shows an example of an ontology-based conceptual model of evidence information. It can record the following information based on the concept of occurrence and the version introduced in this paper: who gave an assessment result by referring to what information of the HR, and which assessment method the assessor used for assessment. In the model, “ver-of” links connect “HR#1” with “HR#1_v1” and “HR#1” with “HR#2_v2”. They represent both “HR#1_v1” and “HR#1_v2” as versions of “HR#1”, whereas the “occurrence-of” link connecting “HR#1” with “HR#1_o1” represents the “HR#1_o1”, which is an occurrence of “HR#1”. Information regarding performance in individual projects that were attended can be represented by the occurrence concept. The “occurrence-of” relation between “eBizPM#o1” (project management of e-business solution for CA company) and “eBizPM#1 instance” represents that “eBizPM#o1” is an information on “e-business project management for CA company” in the work of “e-business project management” job classification that “HR#1” attends. Furthermore, it is represented that “assessor#o1 (occurrence)” assessed (occurrence) “skill level maturity” at “level 6” using the “skill level assessment method Ma”, thereby, the version of the skill was renewed. This version-change mechanism is realized based on state axioms. Consequently, evidence information can be recorded according to the intentions of ITSS builders based on ontology as well as the perceptions, ability information in a project. In addition, expertization processes of an HR can be represented adequately. 4. Concluding Remarks This paper described an Figure 4: Ontology Based Human Resource Model overview of ITSS ontology, which explicate builders’ intention, and an ontology-based HR model to realize evidencebased evaluation. Our ontology is closely related to IMS’s RDCEO [5]. But our ontology clarifies more detailed perception of HR from the viewpoint of evidence-based evaluation and axioms to accumulate personal data. Thereby, we can build an HR model to record evidence information adequately. Recording HR information in accordance with the model facilitates adequate human resource development and utilization. Further research efforts will analyze the conceptual structure in more detail and establish a basis for evidence based evaluation. References [1] [2] [3] [4] [5]

A. Gomez-Perez et al. (2004) Ontological Engineering, Springer. ITSS: http://www.ipa.go.jp/jinzai/itss/, (in Japanese). Kitamura, Y. and Mizoguchi, R. (2004) Ontology-based systematization of functional knowledge, Journal of Engineering Design, Taylor & Francis, Vol. 15, No. 4, pp. 327-351. Mizoguchi, R. (2003) Tutorial on ontological engineering, Part 1, New Generation Computing, Ohmsha & Springer, Vol. 21, No. 4, pp. 365-384. IMS: Reusable Definition of Competency or Educational Objective - Information Model, http://www.imsproject.org/competencies/rdceov1p0/imsrdceo_infov1p0.html

396

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Human Factor Modeling for Development of Learning Support Systems Facilitating Meta-Cognition Kazuhisa Seta*1, Kei Tachibana*1, Motohide Umano*1 and Mitsuru Ikeda*2 Osaka Prefecture University, 1-1, Gakuen-cho, Sakai, Osaka 599-8531 Japan *2 Knowledge Sciences, JAIST, 1-1, Asahidai, Tatsunokuchi, Nomi, Ishikawa, 923-1292, Japan {seta, umano}@mi.s.osakafu-u.ac.jp, [email protected]

*1

Abstract. In the cognitive science field, many researchers have proposed a concept whereby meta-cognition plays an important role to acquire expertise and transfer it. Our research goal is to develop a methodology for building sophisticated learning systems which can effectively facilitate learners’ meta-cognition. The intelligent novice is good at observing knowledge objectively and producing an efficient plan of learning processes. On the other hand, it is not so easy for an ordinary novice to perform meta-cognitive activities. We must model these two types of learners for building learning systems that can support ordinary novices effectively. In this paper, we build cognitive models that capture human factors to perform meta-cognitive activities as a basis of learning systems development for facilitating meta-cognition based on the knowledge in cognitive system engineering and cognitive psychology fields. Then, we build an intelligent novice model that can explain the characteristics of them. Lastly, we clarify design rationales of our system called Kassist based on the models. Keywords: meta-cognition, human factor modeling, intelligent novice, learning environment

Introduction “The Unified Modeling Language User Guide” [1] describes reasons why we must perform the modeling process: we get to know the system in more detail. We can clarify system behaviors and structures through the modeling process. System developers are required to grasp the whole system and to represent it clearly in a way that is easy to understand for system users. System users are entitled to system-use based on the understanding of the whole system. Consequently, models play an important role as a basis to share common understanding of the system among system developers and users [7]. Our research goal is to develop a methodology for building sophisticated humancentered systems. We aim to clarify the roles of computers, especially for supporting learning activities of acquiring meta-cognitive abilities and where we should place them in the system. But, it is not easy to achieve this goal; it is hard to make general theories since the knowledge and human factors underlying meta-cognitive activities is quite tacit, vague, diverse and context dependent. Actually, developers build learning systems facilitating metacognition with groping based on the individual’s thinking, premises or assumptions. It is necessary that setting some kinds of assumptions at the current state of the art, however, we cannot accumulate the knowledge for building sophisticated learning systems without making them explicit. Model explicitly captures premises and assumptions of the system development. It provides a foundation to form shared understanding among learning system developers that facilitates communications among them and helps; (i) to examine design rationales of a learning system in detail, (ii) to provide viewpoints for checking the validity of the premises

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems

397

and assumptions by practice, (iii) to present a basis to accumulate knowledge for building sophisticated learning systems and (iv) to make theories detail [7]. Therefore, model based learning system development is a steady and promising approach to accumulate knowledge and to build a methodology of learning systems development. In this paper, we build cognitive models that capture human factors to perform metacognitive activities as a basis of learning systems development for facilitating meta-cognition based on the knowledge in cognitive system engineering and cognitive psychology fields. Then, we build an intelligent novice model that can explain the characteristics of them. Lastly, we examine design rationales of our system called Kassist based on the models. 1. Modeling Human Factors to Perform Meta-Cognitive Activities Most researchers recognize the necessity for integrating knowledge in educational engineering, psychology and information engineering fields for building effective learning support systems. We, however, have not obtained good results yet: premises and viewpoints of each theory and their relations are not explicit. Therefore, it is important for learning science’s progress that we develop a model to integrate and accumulate knowledge by making the premises explicit. Learners who can acquire knowledge effectively and solve problems even in strange subject areas are so-called “intelligent novices” in the educational psychology field. Many research results support that distinctive features of learners’ behavioral characteristics are that they can perform effective meta-cognitive activities in problem-solving and learning processes. Furthermore, many researchers hold that we can encourage development processes of meta-cognitive ability by providing effective supports because it is known that one’s metacognitive ability has been built up a posteriori. On the other hand, it is not easy to design an educational system that facilitates development of the meta-cognitive ability because building a framework to teach metacognitive knowledge that is vague, latent, tacit and highly context dependent is too difficult. In general, ability is trained by displaying the ability. Therefore, based on Piaget’s constructivism [8], we aim to develop a learning environment in which learners can construct meta-cognitive knowledge from their experiences of performing meta-cognitive activities that are facilitated by the system. To achieve this goal rationally, it is important to clarify differences of behavioral characteristics among “intelligent novices” and “ordinary novices”. Thereby, we produce a model that can explain their behaviors at the principle level. Subsequently, we can develop a system based on the model. This section presents a model as a reference for building learning support systems which facilitate meta-cognition by taking up the attention theory and COCOM that are constructed respectively from different viewpoints in the cognitive psychology and cognitive system engineering fields. This framework will constitute a foundation on which our “intelligent novice model” and “ordinary novice model”, described in next section, are based. Consequently, we can build an integrated model that explicitly represents the premises of respective theories. 1.1 Modeling Attention Control –Extracting and Defining Concepts on Attention Control– Attention control is the cognitive activity that distributes (collects) one’s attention resources to (from) target activities based on priority decision making of a person’s on-going activities. We cannot discuss meta-cognitive activities without issues of attention control because activities of monitoring and (re-)planning one’s own cognitive activities are necessarily accompanied by collection and (re-)distribution of attention resources [10]. But relations between meta-cognitive activity and attention control is not well-arranged by setting adequate viewpoints: we do not have shared concepts to discuss since various

398

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems Strategic control mode

complicated factors are related each other. We stress the attention of importance LAC (H, t1) distribution two separating RAC (H, t1, aj) viewpoints in Succeed Fall Adequate Inadequate Activity modeling attention AAC (H, t1) Assessment of Determination (aj) subjective control; one is from of outcome time a viewpoint of cognitive activities H that require the (a) Attention Control Model (b) COCOM attentional capacity LAC (H,ti) LAC (H,ti) LAC (H,ti) LAC (H,ti) of the human (CAV); the other is from a viewpoint of an available attention resource of the human who performs H H H H cognitive activities Strategic Control Scrambled Control Opportunistic Control Tactical Control (AAV). (c) Relations between LAC and control mode Figure 1(a) shows the model of Figure 1. Models of Attention theory and COCOM[6] and their relations attention control 1 . From the viewpoint of AAC, it should be modeled that the attention resource is recognized as energy for performing one’s activities: its total amount is limited [3]. We call the total amount of one’s attention “Limited Attentional Capacity” (LAC in Fig. 1(a)). The box height represents the limitation and amount of one’s attentional capacity2. On the other hand, from the CAV viewpoint, it is important to model that needed attention capacity for the human H to perform an activity. In other words, the attention capacity that an activity requires depends on its complexity and one’s experience of it [4]. We therefore model it using the notation of balance (in Fig. 1(a)) and call it (attentional capacity on the left side of the balance) the “Required Attentional Capacity” (RAC(H, ti, aj); where H, ti and aj respectively represent a learner performing an activity, time, and the activity that H performs). The model can capture that the larger amount of attentional capacity that an activity requires of the learner, the heavier the cognitive burden imparted on that learner. Furthermore, the model can explain that the greater expertise the learner acquires for performing an activity, the smaller amount of attentional capacity that the activity requires. If distributed attention resources (DAV(H, ti, aj)) to an activity (ai) are lower than the RAC of the activity, then the pointer of the balance turns right; this means the activity is not well performed because of the shortage of attention resources. By “Available Attentional Capacity” (AAC) we denote the attentional capacity that learner H can distribute at time t1 (AAC(H, ti), where H and ti respectively represent a learner and time) without collecting distributed attention resources. Thus the amount of AAC(H, ti) is LAC(H, tj) minus the total amount of distributed attention resources DAV(H, ti, aj=1,2,…,n) of on-going activities. We apply “attention exhausted” to the state in which AAC has dried up. In this situation, the learner is in confusion without any control. Clarifying these concepts contributes to constructing a bass to encourage smooth communications. Now, let us imagine the simplified situation in which only one task is given to a human (at first, AAC is equal to LAC in this situation) to consider the validity of separating two Tactical control mode

Opportunistic control mode

Scrambled control mode

Number of goals

Available plans

Mode of execution

Event horizon

1 We do not intend to take the internal mechanism of attention control, e.g. [3][15], into account because we cannot find any necessity for the learning system design. And, the arguments do not have the same opinion. 2 We model that one’s LAC fluctuates explicitly according the surrounding contexts, as described later in section 1.2. It is important to declare these kinds of premises explicitly for system development.

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems

399

viewpoints for modeling attentional capacity. From CAV, we can explicitly represent the fact in a model in which the RAC of an identical task becomes smaller according to the experiences of performing the activity by removing boxes representing RAC on the balance. From AAV, on the other hand, we can explicitly represent the fact in a model that LAC (AAC) fluctuates according to the situation surrounding the person by removing (adding) boxes representing LAC. This fact is easy to accept when we imagine relaxing, concentrating and tense situations. Consequently, we can adequately model attention control by separating two viewpoints. This is meaningful because the model plays an important role not only to capture attention control, but also to construct the intelligent novice model (described later) for reference of system development. 1.2 COCOM COntextual COntrol Model (COCOM) [6] is a model whose underlying concept is that an action sequence is controlled by context, i.e., how a person understands or interprets the situation, goals, demands, resources, restrictions, etc. It captures action sequences as constructed rather than as pre-defined. This model is suitable for modeling one’s metacognitive activities that control one’s own cognitive activities as triggered by meta-cognitive awareness (context-dependent information). Figure 1(b) depicts an overview of the model’s internal structure. Upper and lower series of images respectively represent main parameters and control modes that explain behavioral characteristics. Control modes that state relatively right (left) are higher-mode (lower-mode) and the reliability of behaviors at the higher-modes is more reliable than at the lower-ones. The combination of values of main parameters (number of goals, available plans, mode of execution, event horizon) characterizes the control modes. By clarifying the relationships among the parameters and control modes, the model can explain (i) in what situations a human can move from one control mode to another and (ii) Behavioral characteristics of humans at each control mode. Below, we briefly describe characteristics of respective control modes [6]. Scrambled Control: Choice of the subsequent action is unpredictable or random. The person seemingly has no useful internal model of the world in which they are taking action. Opportunistic Control: Opportunistic control corresponds to actions taken based on the current context. The current context has perceptually salient features or patterns as opposed to more fundamental constructs such as intentions or goals. Tactical Control: Tactical control pertains when operator performance is based on some kind of planning. This is behavior that is consistent with the rule-based levels of control identified by Rasmussen [9]. This mode is a so-called ordinary state. Strategic Control: Strategic control is that condition under which the human has a sufficiently accurate model of the controlled process and the environment in which that control is undertaken to support planning and prediction in support of high level goals that can be managed across a system of interruption. Determination of the outcome is different according to the control mode. Humans at the strategic control mode assess the feedback in detail by comparing the progress of the condition before execution whereas those at the scrambled control mode only assesses whether any change for the better occurs. Similarly, the assessment of subjective time is different at each mode. These behavioral characteristics naturally influence human reliability. 1.3 Integrating Attention Control Model and COCOM We produce a model that captures the relation between attention theory and the COCOM: we set the reasonable assumption from AAV that the higher (lower) the control mode becomes, the larger (smaller) LAC does (Fig. 1(c)).

400

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems

It is important to emphasize that the COCOM is constructed mainly to analyze human reliability for relatively short-term activities, e.g. emergency control of the nuclear plant. For that reason, it focuses on the controls of actions to operate in the real world (problem-solving states) and their internal factors to explain the mechanism of those controls. On the other hand, learning is a longer-term activity that explicitly operates from internal information in one’s mind, e.g., an internal model and knowledge of the problemsolving domains. Furthermore, meta-cognitive activities, especially in learning, monitor and control such kinds of internal information and learning activities. Consequently, COCOM is well-adapted to explain behaviors of (short-term) problem-solving processes but is not adaptable to behaviors and control mechanisms of (long-term) learning processes. Furthermore, it cannot explain behaviors of a person who can remain at the higher mode even in conditions shifted to a lower mode. Intelligent novices can remain at the higher mode by performing meta-cognitive activities in the learning processes. By integrating attention distribution model and COCOM based on the assumption described above (Fig. 1 (c)), human factors that characterize the differential behaviors of intelligent novices and ordinary ones can be clarified. This makes us understand the characteristics of each novice in detail.

2. Constructing an Intelligent Novice Model Our research is intended to build a learning environment in which learners can construct meta-cognitive activities by being facilitated to display meta-cognitive activities. Therefore, it is important to clarify human factors to display meta-cognitive activities for the basis of our system development. To achieve this goal, we conjecture that an ordinary novice and an intelligent novice both have equal understanding of the problem-solving domain and perform problem-solving oriented learning. Problem-solving oriented learning is a learning style by which a novice learner of the target domain tries not only to plan and perform problem-solving processes, but also to do learning processes [11]. Therefore, it requires excellent performance of metacognition to solve the problem well. It imparts heavy cognitive loads to learners. Figure 2 left and right images respectively represent models of an ordinary novice (ON) and an intelligent novice (IN); time progresses from top to bottom. Upper and lower parts at each time respectively illustrate a model of meta-cognitive activity and cognitive one (learning activity); basic structures are similar because meta-cognitive activity is also a cognitive one. In the model of the IN (ON) at time I-t1 (O-t1), it is represented that an IN (ON) tends to distribute much (less) attention resources to the meta-cognitive activity than the ON (IN) does, therefore tending to perform the meta-cognitive activity with a wider (narrower) range of event horizon. The area of the event horizon is represented as the size of the lamp on the activity. Activity with a wider range of event horizon requires more attention resources (RAC) than the one with narrower range of event horizon. Therefore, the amount of the attention resource on the left side of the balance at I-t1 is larger than that at O-t1. By performing meta-cognitive activities with a wide range of event horizon well (ill), the IN (ON) can make rational (irrational) learning plans. Consequently, a learner can (cannot) perform rational learning activities and build well organized schema (Fig. 2(I-t2, O-t2)). Then he (she) can (cannot) make available learning/problem-solving plans based on well (ill) organized schema. The number of goals that should be achieved can (cannot) be effectively reduced and the subjective time that IN (ON) can make use (represented as the length of the arrow that the human has) tends to be increased (decreased) (Fig. 2(I-t3, O-t3)). Consequently, because the IN can shift to the higher control mode whereas the ON falls into the lower mode, the LAC gets larger (smaller). This causes a positive (negative) loop. The IN (ON) at the higher (lower) control mode having greater (lesser) attention capacity can distribute much (less) attention capacity stably (unstably). Consequently, the

401

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems (I-t1)

(O-t1)

set

set t

t MCA

MCA

Learning Plan

available plans

Learning Plan

available plans

H

H Schema

Schema

Performing Meta-cognitive Activity

Performing Meta-cognitive Activity

t

t

H

H

Schema

Schema

(I-t2)

(O-t2)

t

t MCA

MCA

Learning Plan

Learning Plan

H

H

Performing cognitive Activity

Performing cognitive Activity

set

set t

t

CA (Learning)

CA (Learning)

H

H

Schema

Schema

(I-t3)

(O-t3)

(γ) t

set

set

t

t

MCA

MCA

Learning Learning Plan Plan

Learning Learning Plan Plan

H

H

Performing Meta-cognitive Activity

Performing Meta-cognitive Activity

t

t

H

Schema

H

Schema

Figure 2. Models of ordinary novices (left) and intelligent novices (right)

event horizon becomes wider (narrower) and the schema becomes (remains) organized (unorganized). We can make the roles and usefulness of meta-cognition in detail by constructing a model based on COCOM and attention theory and integrating learning activities that act on internal factors. Furthermore, we can understand this model is useful for building learning environments to facilitate meta-cognitive activities and making the design rationales of learning environments explicit. 3. Clarifying the Design Rationales of Kassist –Towards Constructing a Basis for Accumulating Knowledge to Develop Learning Systems– Providing a framework whereby learners can externalize and visualize their internal information is considered by most researchers as a basis for a computer based learning

402

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems

environment for facilitating meta-cognition. But design rationales of such systems are not clarified at the principle level. We also have developed an interactive open learner modeling environment called Kassist in which many support functions intended to facilitate meta-cognition are realized. In this section, we clarify the design rationales of support functions in Kassist based on the model, toward building a basis of accumulating the knowledge to develop learning systems which can effectively facilitate learners’ meta-cognition. A detailed explanation of the support functions is available in some references [12][13][14]. Externalization/visualization environment for describing information in a learner’s mind: It is important to realize a framework that can lighten learners’ cognitive load (RAC) of meta-cognition for facilitating construction of meta-cognitive ability: an activity with larger amount of RAC is not well-displayed in the situation that learner has smaller amount of AAC whereas ability is generally not nurtured without displaying it. The Kassist system provides learners with an environment in which they can visually represent their states of problem solving, learning, and understanding. The design rationales of this environment are as follows: the environment aims to retain distributable attention resources (AAC) for meta-cognition by lightening RAC for memorizing and operating on internal information; thereby, it facilitates their spontaneous meta-cognition (awareness) at higher control modes with wider event horizon by providing a visual representation of internal information that is easily monitored. Environment for Awareness of cognitive goals: Visualizing and externalizing one’s internal information encourages awareness of meta-cognition. Furthermore, it is also important for the learners to recognize their own cognitive goals. Meta-cognitive awareness is that one becomes aware of the gaps between goals and intentions of an activity and the activity’s results (Fig. 2(iii)). We divide goals into performance goals of the activity (Fig. 2(ii))(e.g. problem-solving goals and learning goals) and cognitive goals that are modes of performance (Fig. 2(i))(e.g., performing high quality decision making, performing problem-solving effectively). The recognition of cognitive goals encourages one’s awareness of the modes of performing activities. But, learners, especially at the lower control mode, tend to focus on performance goals rather than cognitive goals. Thus it is important to compel the learner recognize not only performance goals but also cognitive goals to encourage their meta-cognitive awareness. Thus, design rationales of the environment for describing both goals are for learners to recognize both goals and for facilitating learners’ spontaneous meta-cognition (awareness). Information focus function: Facilitating meta-cognitive activities with a wide range of event horizons does not necessarily yield good results. Meta-cognitive activity necessarily requires large amounts of attention capacity (RAC) because it requires the learner to grasp information that is inherent in the cognitive activities. If the learner tries to expand the event horizon (distribute large amount of attention resources) excessively to perform high-quality meta-cognition, performance suffers because of exhaustion of the AAC. As a result, the learner goes to the lower control mode. Therefore, it is important to perform rational metacognitive activities. The information focus function in Kassist highlights the information related to each selected problem-solving/learning processes and domain knowledge according to the learner’s request. Consequently, the design rationales of the function are for inhibiting un-reasonable excessive expansion of the event horizon and for keeping the higher control modes by making the learner focus on the meaningful information. Navigation function of meta-cognitive processes: Meta-cognitive activities are quite context-dependent ones. It is important to construct a framework that can capture the contexts in which the learner works and navigates meta-cognitive processes. With such a framework, a learner can learn a method of performing meta-cognitive activities effectively and progress to construct meta-cognitive knowledge. The navigation function guides learners’ meta-cognitive processes by providing basic (but limited) context dependent information along Rasmussen’s cognitive activity model [9]. Design rationales of the function in Kassist are for this purpose; the function contributes to retaining a learner’s AAC for constructing his meta-cognitive

K. Seta et al. / Human Factor Modeling for Development of Learning Support Systems

403

knowledge since the system looks up and presents the meaningful information for them along their meta-cognitive processes. By clarifying this design rationales based on the model as described above, learning system developers can not only make use of them as a reference for system development but also do as a basis for sharing and accumulating knowledge. Furthermore, learners are entitled to system-use based on the understanding of the design rationales. 4. Concluding Remarks This paper proposed novel models to integrate knowledge acquired in cognitive psychology and cognitive system engineering for development of a learning support system. By constructing the models, we can also model and clarify intelligent novices’ behavioral characteristics in detail. The models provide new viewpoints for learning system developers to design learning systems and they contribute to clarifying human factors for facilitating meta-cognitive activities and design rationales of the developed system. It is also quite important to clarify the premises of respective learning systems, not only for building highquality learning systems, but also for developing a methodology. We hope our models as a basis of communications contribute to sharing and accumulating knowledge for learning systems development. Acknowledgement This research was supported in part by a grant from the Telecommunications Advancement Foundation (TAF). References [1] [2]

[3] [4] [5] [6] [7] [8] [9] [10] [11]

[12] [13]

[14] [15]

Booch, G. et al., (1995) The Unified Modeling Language User Guide, Addison-Wesley. Okamoto, M. (2002) Review of Metacognitive Research, from educational implications to teaching methods, Transactions of Japanese Society for Information and Systems in Education, Vol. 19, No. 3, 2002 (in Japanese) Kahneman, D. (1973) Attention and effort, Prentice-Hall. Kawaguchi, J. (1995) Attention, in Y., Kohno (Eds.) Memory, pp. 29–48, University of Tokyo Press,(in Japanese) Hirashima, T. et al., (2004) Error-visualization for learning from mistakes, Transactions of Japanese Society for Information and Systems in Education, Vol. 21, No. 3, pp. 178–186. (in Japanese) Hollnagel, E. (1998) Cognitive Reliability and Error Analysis Method, Elsevier. Ikeda, M. et al., (2004) Ontology based modeling of learning activities, Transactions of Japanese Society for Information and Systems in Education, Vol. 21, No. 3, pp. 168–177. (in Japanese) Piaget, J. (1970) Piaget’s theory: In Carmichael’s Manual of Child Psychology, P. H. Musen, ed. New York: Wiley. Rasmussen, J. (1986) A Framework for Cognitive Task Analysis, Information Processing and HumanMachine Interaction: An Approach to Cognitive Engineering, North-Holland, New York, pp. 5–8. Sanmiya, M. (2001) Meta-cognition and attention in thinking, in S. Ichikawa (Eds.) Thinking, pp. 157– 180, University of Tokyo Press. (in Japanese) Seta, K. et al., (2002) A support system for planning problem solving workflow, Proc. of the International Conference on Computers in Education (ICCE-02), Workshop on Concepts and Ontologies for Webbased Educational Systems, Auckland, New Zealand, pp. 27–33. Seta, K. et al., (2004) Task ontology based reflection support in problem solving oriented learning, Proc. of the International Conference on Computers in Education, pp. 1781–1793. Seta, K. et al., (2004) An ontological approach to interactive navigation for problem-solving oriented learning processes, International Journal of Interactive Technology and Smart Education, Vol. 1, No. 3, pp. 185–193. Seta, K. et al., (2004) A support system for problem-solving oriented learning, International Journal of Continuing Engineering Education and Life Long Learning, Vol. 14, No. 3, pp. 246–259. Shallice, T. (1988) From Neuropsychology to mental structure, Cambridge University Press.

404

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Digital Information Literacy: Explorations of History Tasks in Singapore Schools a

Sunita SHANKAR a, John G HEDBERG b Centre for Research in Pedagogy and Practice, National Institute of Education, Nanyang Technological University, Singapore b Macquarie University, Australia [email protected] Abstract. Pedagogy of the 21st century is focusing on visible thinking of the students. Educators today are creating learning environments and processes where students’ learning is self regulated, active, collaborative and self-directed. The visibility of students’ cognition is important for effective mediation and facilitation by the teacher whose role is more of a coach. With the emergence of the internet, new possibilities for learning and teaching have evolved (Tan, 2004). The internet provides access to a plethora of information which is multi faceted and multimodal. Digital Information Literacy promotes ‘self-regulated’, self-directed’ and ‘exploratory’ learning. It also focuses on information problem solving strategies, the research process and the related skills that make up these information-seeking processes. The Digital curricular Literacies project (DCL) focuses on the key initiatives of the Singapore Ministry of Education (MOE) on the use of ICTs (Information and Communication Technologies) in school learning. This paper is based on the data from one section of the larger study, which looked at some of the students’ online usage skills when working on history and science tasks. Secondary 1 students from 6 Singapore local schools participated in this project. 13 pairs of students worked on the digital experiment. Each computer was equipped with the screen capturing software Snapzpro that produced a real time video of the students working on the task. Audio recorders were used to capture the discussion between the pairs of students as they worked to complete the task. This paper is a case study of 7 pairs of students working on history tasks. Kuhlthau’s (1993), Pitts/Stripling’s (1988) and Eisenberg/Berkowitz’s (1996) models of information literacy are compared. The data of this case study was analyzed qualitatively using the framework of Eisenberg/Berkowitz’s Big6 model. It was found that the students were making conscious choices on selection of websites. They spent most of their time thoughtfully searching and scanning for relevant information on the topic in a broad sense. In most cases they used phrases from the task. They did not spend time to evaluate the sources they had selected for authority, the credentials of the author and the accuracy of the information. The students’ artifacts produced did show evidence of multi-modality with a combination of images and text and the resulting artifacts were good examples of exploratory, self-regulatory and collaborative learning. Keywords: Digital Literacy, Digital Information Literacy, Problem Solving, Critical Thinking, Exploratory Learning

Introduction Information seeking styles have evolved gradually. Changes have been influenced by learning styles, the evolving expectations of learners, and developments of technology. Theorists have developed their theories and paradigms in response to changes that are happening in different learning contexts and society at large. No theory would gain

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

405

acceptance if it did not have any topical relevance and applicability. Information Literacy goes beyond the ability to read books and to turn on a computer. Problem solving, decision making, critical thinking and sense making are key attributes that define information literacy. Pedagogy of the 21st century is focusing on visible thinking of the students. In response to these changes in educational theory, educators today are creating learning environments and processes where students’ learning is self regulated, active, collaborative and selfdirected. The teacher is more a coach to help to students to recognize their levels of thinking and to challenge them with various problem-solving situations. Thus the visibility of students’ cognition is important for effective mediation and facilitation by the teacher. With the emergence of the internet, new possibilities for learning and teaching have evolved (Tan, 2004). The internet provides access to a plethora of information which is multi faceted and multimodal. It also gives students more choices it helps in developing their critical, creative and complex thinking skills (Jonassen, 1996). Thus students require digital information literacy to sift through this vast amount of information and select the appropriate information to make meaningful artifacts. Digital Information Literacy promotes ‘self-regulated’, self-directed’ and ‘exploratory’ learning.

Information Literacy Information literacy focuses on information problem solving strategies, the research process and the related skills that make up these information-seeking processes. These strategies apply to digital and internet information literacy. Doyle’s research (1994) was seminal in describing the need for information literacy. Later several theories and models on information literacy emerged. These models have many similarities and employ similar principles but emphasize different aspects of the information seeking process. Models of Information Literacy Kuhlthau’s (1993) research on information literacy concentrated on the students’ affective domain of’ information seeking. Kuhlthau did a series of studies where she made students log the actions they took to solve information problems and also the feelings they experienced during this process. Kuhlthau created the seven-stage process of learning from information. This process maps the emotions that people experience during the information search process – uncertainty, clarity, confidence ending with satisfaction or disappointment. Kuhlthau’s model applied constructivist principles beginning with uncertainty, moving through optimism, confusion, building knowledge on prior learning (Shannon, 2002). An alternative model of information literacy is by Stripling and Pitts (1988). They emphasized more on the cognitive domain of the information seeker. They highlighted that the students must be taught to stop and reflect on what is being learned through their research especially when they are investigating on a new topic. These recursive reflection points allow the student to gain a broader understanding of a topic before they narrow the scope of their research. Stripling and Pitts identified six levels of student research. These categories are somewhat similar to the dated Bloom’s Taxonomy, and they identify six responses to research endeavors entitled REACTS (Recalling, Explaining, Analyzing, Challenging, Transforming, and Synthesizing). It can also be compared to the revised Bloom’s theory by Anderson and Krathwohl. They are to Retrieve relevant knowledge

406

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

(Remember by Recognizing and Recalling), Understand, Apply, Analyze, Evaluate and Create. Anderson and Krathwohl also divided the knowledge dimension into Factual, Conceptual, Procedural, and Metacognitive (Anderson and Krathwohl, 2001). The Big 6 model which is a widely used model represents a non-linear process that is applicable to a variety of information problem-solving situations. Eisenberg and Berkowitz (1996) found the best way to teach information literacy skills in a curriculum context is through the collaboration of classroom teachers and library media specialists. Six Steps: 1.Task definition: Defining the problem and identifying its information requirements. 2.Information-Seeking Strategies: Determining possible sources and evaluating their priority. 3.Location and Access: Locating the sources and then locating information in them. 4.Information Usage: Reading information and then extracting details. 5.Synthesis: Organizing and presenting information. Integrates new information into an existing body of knowledge. 6.Evaluation These theories have quite a few similarities with each other and represent aspects of the cognition, affective and psychomotor domains of the individual. Kuhlthau’s theory focuses more on the responses of the students during the information seeking process. However, the Big 6 Model of Eisenberg and Berkowitz emphasizes on the cognition domain of the individual during the information problem solving process.

How does information literacy relate to critical thinking? Both Bloom and later, Anderson and Krathwohl’s taxonomies describe learning on a hierarchy of skills. Memorizing facts or understanding the meaning of material is at the low end of the cognitive ladder, requiring only lower order skills. Critical or higher order thinking requires the ability to analyze or separate material into its component parts, to synthesize information or put its parts together to create a new whole, and to evaluate or judge material for a given purpose. These tasks require the learner to go beyond what is written (Anderson and Krathwohl, 2001). Information skills are tools for inquiry. Dewey also was in favor of an inquiry based and learner-centered education. ‘Inquiry is an approach to learning that involves a process of exploring the natural or material world, that leads to asking questions and making discoveries in search for new understandings” (Inquiry page, http://www.inquiry.uiuc.edu/inquiry/definition.php). Some basic criticalthinking skills are below. x Making a distinction between facts and claims x Determining the credibility of a source x Determining accuracy and relevance of information x Determine the strength of an argument (Beyer in Crane 45) The popular Big6 model supports critical thinking skills and is also based on the six levels of the revised Bloom taxonomy by Anderson and Krathwohl: Task definition can be compared to retrieving of relevant knowledge; information-seeking strategies can be compared to comprehension or understanding of the information; location and access can be compared to application, and so on.

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

407

Digital Curricular Literacies 2 Project The Digital curricular Literacies project (DCL) focuses on the key initiatives of the Singapore Ministry of Education (MOE) on the use of ICTs (Information and Communication Technologies) in school learning. The term Digital Curriculum Literacies refers to the integrated use of ICTs within the various curriculum areas (Freebody, P. R., Hedberg, J. G. & Guo, L. B. (2003).This paper is based on the data from one section of the larger study, which looked at some of the students’ online usage skills when working on history and science tasks. The research questions were x How do students engage in on-line tasks? How can that be improved? x When students work on Internet-based inquiry tasks how do they navigate and search for information? What strategies are used and how do students integrate and synthesize information into an artifact?

Research Design of DCL Project Secondary 1 students from 6 Singapore local schools participated in this project. 13 pairs of students worked on the digital experiment. In this project the students were divided into pairs. The pairs of students were given the option to choose one of four tasks related to the curriculum in the history and science disciplines for secondary 1 level. Each group was provided with one Macintosh computer to complete the task. The Macintosh had access to the internet and the following search engines: Google (www.google.com); Ask Jeeves (www.ask.com); Mamma.com (www.mamma.com); Yahoo (www.yahoo.com). They also had the option to use another search engine or type in a URL for a web site in the address bar in the Safari browser to navigate to a specific web site. At the end of the task they were to produce a final artifact based on the task they had chosen. To produce the artifact they could choose to use Microsoft Word or Power Point. The students were given 20 minutes to complete the task.

Data Collection Each computer was equipped with the screen capturing software Snapzpro that produced a real time video of the student working on the task. Audio recorders were used to capture the discussion between the pairs of students as they worked to complete the task.

Case Study: History tasks

Methodology This paper focuses on the early Digital Information Literacy Skills of secondary one student who participated in this project and worked on History tasks. For this case study a sample of seven histories screen capture videos were selected. Data were analyzed qualitatively using the big6 model framework by content analysis.

408

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

Tasks History Task: Early Civilizations Instructions for students 1.Select 1 set only. Your work will not be graded. We are interested in how you go about task and how you present the information you find. 2. For each task you will have access to the following Search Engines in the Bookmark Bar: Google (also in the main bar) www.google.com; Ask Jeeves www.ask.com; Mamma.com www.mamma.com; Yahoo www.yahoo.com but you may select another search engine. Unit: Early civilizations Based on what you have learned about the factors which led to the rise of early civilizations (like suitability of climate and closeness to water), do an Internet-search for one set of the early civilizations: 1) Set one: Mesopotamia and Sri Vijaya ƒ What similarities and differences can you see in the civilization compared to the factors you've studied? ƒ Showcase your findings using a graphic organizer (e.g., chart, table, diagram) to give concrete support for each of the features identified. You might care to use MS Word or PowerPoint. OR 2) Set Two: Egypt and Greece ƒ What similarities and differences can you see in the civilization compared to the factors you've studied? ƒ Showcase your findings using a graphic organizer (e.g., chart, table, diagram) to give concrete support for each of the features identified. You might care to use MS Word or PowerPoint.

Discussion This discussion based on the findings of the qualitative analysis will illustrate the information literacy skills of these students based on the Big 6 model framework.

Task definition This refers to defining the problem and identifying its information requirements. Most students based their searches on the task and used phrases mostly from the task. However, the students did not spend much time understanding the task. Since most of the students did not understand and define the needs of the task they were on the wrong track.

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

409

Information-Seeking Strategies This refers to determining possible sources and evaluating their priority. .It was noticed that most of the students followed similar navigation patterns when searching for information. They would start with the prescribed search engines (Google, Yahoo, Mamma, and Ask Jeeves) and use short sentences to search for information from the search engines. They rarely searched for popular websites directly from the browser. Majority of the students did not use keywords for their searches. Some examples of the sentences they used were, ‘ancient greek civilization’ ‘occupation of ancient egypt’, ‘civilization and religion of Egypt’. Most students did not use Boolean operators and advance search strategies to narrow down the searches. In fact even in basic search their searches were very broad. For example ‘Egypt’, ‘ancient greek civilization, ‘about greek civilisation’’. However they tried ‘related sites’ where they were quite successful to get information. They searched several times using different search sentences on Google, Yahoo and sometimes Mama and Ask Jeeves.

Location and Access This aspect refers to locating the sources and then locating information in them. The students were a little critical while selecting information. It seemed they were exploring for information. They always scanned through the page of hits carefully and seemed as they were reading the abstracts. Since they were working in pairs they would discuss with each other on the appropriate choice. In most cases they would not automatically click on the first hit, which is supposed to be a common trend with most novice information searchers. They were engaged in some amount of critical thinking and decision making before clicking on any hit after a search. They would scan and read through the abstracts of the first hit page. It seemed that in the beginning the students were uncertain on information seeking strategies for the topic. After working at the search process for sometime they seemed to be more optimistic although still confused after they were encountered with a lot of information. They used several variations in search terms mostly in form of short sentences. Most students modified the search sentences. This confirms that they were active searchers and not just aimlessly browsing. They did not click into many websites. If they were not satisfied with the content of the results page they would start a new search or a different search engine. However, if they used some information from a source they would not compare the information with another source for accuracy and depth of the information. They seemed to be actively sifting through the information. Since they had studied these topics in the classroom following the constructivist theory they were more focused on their needs for information on the topic. However, since many students did not spend time in defining and understanding the needs of the task before their search process they found difficulty gathering relevant information for the task. The students mainly visited and used .com sites. However, they also visited and used .edu, .org and .net sites. However there were some exceptions also. An example of one pair of students is illustrated in figure 1. This pair was on the right track with their search, which was not the case with most of the other student pairs. They first used the search term ‘Civilization of Egypt and Greece’. They clicked the first hit ‘history for kids’. Then they clicked Egypt and after much thought and pausing clicked the link on ‘environment’. Not satisfied with

410

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

the information they searched for information on ‘Greece’. After scanning through the page several times they clicked the link on environment. They seemed to have been lucky as this page had information on the differences between Egypt and Greece. They extracted some information from this source into their artifact.

Figure 1: Students finding relevant information and integrating elements into an artifact. Sometimes the pairs went into links and sub links. Some students lost track of their search and went into advertisements and commercial sites not related to the task. A pair of students went into a telecommunications company home page and within the website did a search on ‘Ancient Greece and Egypt’. They were unsuccessful and had to try a new search. A pair of students also went into a commercial store’s home page and selected Egyptian Archaeology Kit sold for $22.95.

Information Usage This refers to reading information and then extracting details. Although they were actively searching for relevant information and spent a lot of time sifting and selecting the information, the information they finally selected were mostly low in relevance to the task. However, they were not actively evaluating the credentials of the authors of the websites they visited. They did not compare the information with another site to check the validity and accuracy of the information. They extracted some information from these sources into a word or PowerPoint artifact. These students used images in their artifacts, for example a pair of students used the maps of Egypt and Greece and the Flags of both these countries in their artifact. It was evident from their artifact that they did not understand the task. An example below of another pair of students’ artifact shows that they understood the question as the Similarities and Differences in the information on the Internet compared to what they have learnt in class. Also some students were not successful in finding information and did not prepare an artifact. A pair of students just had some points listed on a sheet of paper at the end of the twenty minute search. They were very confused with the question and did not prepare a digital artifact.

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

411

EgYpT In the internet, we learned about mummies, sphinx, pyramids, Egyptian writing and what they call their rulers. During lessons, we learnt about mummies and pyramids only. 1. Difference Learn about sphinx (internet) Learn about writing (internet) Learn about rulers were called. (Internet) Learnt about transport (studied) Learnt about technology (studied) 2. Similarities Learn about mummies and pyramids from both. Figure 2: Example student artifact

Synthesis This refers to organizing and presenting information and integrates new information into an existing body of knowledge. Since the students did not have much time to complete their task this phase of the information literacy cycle was not very complete. As mentioned by Stripling and Pitts (1995), reflection is an important factor for synthesis and knowledge construction. Most of the students extracted the information from the websites into a power point or word document. The information extracted was a combination of text and images. Some students created titles in their document. However, they did not integrate any new synthesized information into the artifact.

Evaluation. This phase was absent in the baseline data as the students did not have time to look over their artifacts. It seemed as most students were not satisfied with their work.

Conclusion In all tasks, the students were making conscious choices on selection of websites. They spent most of their time thoughtfully searching and scanning for relevant information on the topic in a broad sense and only clicked on a few websites. They made their selection of websites after reading the abstracts in the hits list. However, surprisingly the sites they visited and selected were mostly low in relevance to the task. This was probably due to the fact they did not spend time understanding the task before they started the search. In most cases they used phrases from the task. They rarely used keywords or narrowed their searches. They did not spend time to evaluate the sources they had selected for authority, the credentials of the author and the accuracy of the information. They did not compare nor verify the information with another source. Scaffolding in the task and mediation

412

S. Shankar and J.G. Hedberg / Digital Information Literacy: Explorations of History Tasks

during the process might improve the pair’s results markedly. The students artifacts produced did show evidence of multi-modality with a combination of images and text and the resulting artifacts were good examples of exploratory, self-regulatory and collaborative learning. This study closely analyzed the cognitive processes of the students when searching for information and presents an initial evaluation of the information searching navigational processes of the students. The artifacts show evidence of the cognitive processes and levels of critical thinking of the students. The poor initial search strategies and questioning patterns when pairs initiated a search does indicate that some further work and support is necessary if the time spent on internet tasks is to result in greater useful information efficiently found and well used to support arguments.

Acknowledgements The authors would like to acknowledge the DCL team from the Centre for Research in Pedagogy and Practice, National Institute of Education, Nanyang Technological University, Singapore for data collection. References [1] Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E Pintrich, P.R., Raths, J. & Wittrock, M.C. (2001). A Taxonomy for Learning Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. New York: Longman. [2] Crane, B.E, (2000). Teaching with the Internet: Strategies and Models for K-12 Curricula, Neal Schuman, New York.. [3] Doyle, C. S. (1994). Information Literacy in an information Society: A concept for the Information Age. Syracuse, NY: Clearinghouse on Information & Technology, Syracuse University. [4] Eisenberg, M. and Berkowitz, R. (1996). Information Problem Solving: The Big Six Skills Approach to Library and Information skills instruction. Norwood, NJ: Ablex. [5] Freebody, P. R., Hedberg, J. G. & Guo, L. B. (2003). Digital curricular literacies: Case for Support. Singapore: Centre for Research in Pedagogy and Practice, National Institute of Education. [6] Inquiry page Retrieved May 24th 2005 from http://www.inquiry.uiuc.edu/inquiry/definition.php [7] Jonassen, D. H. (1996) Computers in the classroom: Mind tools for critical thinking. Englewood Cliffs, New Jersey: Prentice-Hall. Kuhlthau, C.C. A Principle of Uncertainity for Information Seeking (1993). The Journal of Documentation, (19)4,339-355. [8] Shanon, D. (2002). Kuhlthau’s Information Search Process. School Library Media Activities Monthly, (19)2, 19-23. [9] Stripling B. (1995). Learning-Centered Libraries: Implications from Research. School Library Media Quarterly, (23) 3, 163-70. Stripling, B. and Pitts, J. (1988). Brainstorms and Blueprints: Teaching Library Research as a Thinking Process. Littleton, CO: Libraries Unlimited. [10]Tan, O S. (2004). Cognition, Metacognition, and Problem-based Learning in Enhancing Thinking through Problem-based learning approaches. Singapore: Thomson. [11]Washington Library Media Association. Process models for Information Literacy. Retrieved 25th may 2005 from http://www.wlma.org/Instruction/processmodels.htm.

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

413

The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns in a Computer-Mediated Learning Environment Hyo-Jeong SO Learning Sciences and Technologies Academic Group, National Institute of Education, Nanyang Technological University, Singapore [email protected]

Abstract. The purpose of the present study was to examine social presence and collaborative learning behavior patterns in asynchronous discussion forums using the content analysis method which has become a popular qualitative research approach in distance education. Two online discussion forums of students who enrolled in a distance course at a major university in the United States were analyzed by two coders using coding schemes. The inter-coder reliability measures were .87 for Holsti’s CR and .91 for Cohen’s Kappa. The results revealed that students showed affective, interactive, and cohesive behaviors of social presence to help each other in the process of collaborative learning. Additionally, this study found that while students actively exchanged resources and feedback, there was a lack of elaborating and challenging behaviors. In conclusion, it was argued that distance educators should consider instructional methods to increase the level of social presence in order to facilitate collaborative learning processes. This study will be beneficial to those who are interested in examining the effect of the social dimensions of learning in a distance learning environment, and also those who are interested in employing qualitative content analysis methods. Keywords: Social presence, collaborative learning, content analysis

Introduction The new generation of distance education is characterized as the use of interactive computer technology which enables students to interact synchronously or asynchronously with their teachers and classmates. In particular, two-way computer-mediated communication (CMC) technology has made online collaborative learning possible among students in geographically different areas. With the help of CMC technology, educators in higher education have tried to incorporate collaborative learning methods in their distance education courses with the belief that increased interaction among students could enhance learning outcomes and student satisfaction [1-3]. Despite the popular support among educators for collaborative learning approaches, prior research studies have suggested that students were often dissatisfied and frustrated with their collaborative learning experiences [4-5]. It is important to understand that collaborative learning does not occur simply because students are assigned to work together. A number of distance education researchers have argued that the feeling of connectedness or closeness among group members plays an important role in collaborative learning processes [6-8]. The current study, therefore, sought to examine how social presence behaviors, defined as the

414 H.-J. So / The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns

degree to which a person is perceived by and interact with others in a CMC learning environments, were presented in collaborative learning processes.

1. Theoretical Bases 1.1 Collaborative Learning Social constructivism proposes that people construct their own knowledge through meaningful interaction with others. According to Vygotsky [9], a person’s cognitive development is highly dependent on their relationship with others. Vigotsky’s idea on the Zone of Proximal Development (ZPD) –“the distance between actual or independent problem solving and performance when provided with learning assistance from adults or more capable peers” [10, p36] – suggests that people construct their knowledge through social interaction and collaboration with others. As an example, students with low ability levels may be able to reach their ZPD with a help of advanced and high-achieving peers. Collaborative learning approaches provide students with opportunities to share their knowledge and to engage in a social knowledge construction process in order to accomplish a shared goal.

1.2 Social Presence There has been a growing interest in social presence and its relationship with students’ learning experiences, as researchers have recognized the importance of social dimension in online education. Social presence theory has its origin in communication research and educational psychology. Short, Williams, and Christie [11] define social presence as “the degree of salience of the other person in the interaction and the consequent salience of the interpersonal relationships.” Intimacy, such as eye contact, smiling, and physical proximity, and immediacy, such as psychological distance, are known to enhance an individual’s perceived social presence and vice versa. Then, online learning contexts—in particular, asynchronous learning contexts—have fundamental problems due to the lack of visual or verbal cues and the nature of delayed communication.

2. The Present Study 2.1 Setting The purpose of the present study was to examine social presence and collaborative learning behavior patterns manifested in asynchronous discussion forums using the content analysis method which has become a growing qualitative research method in the field of distance education. For the current study, the researcher selected two groups of students who enrolled in a distance course in health education at a major university in the United States. This course can be considered as a blended distance course where students mainly communicate via online with classroom meetings. As part of course assignments, students were required to work on a collaborative group project of developing a comprehensive HIV-AIDS prevention community planning paper. For the collaborative group project, students were divided into the following groups: a) behaviors, b) resources, c) epidemiology, and d) intervention committees. The course instructor set up online discussion forums to facilitate the collaborative learning project.

H.-J. So / The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns 415

2.2 Method Two groups, the Resources Committee (n=5) and the Epidemiology Committee (n=4) were selected for this study because the online postings in these two groups were greater than other groups. Two coders participated in coding entire transcripts of online discussion forums using two coding schemes. For the collaborative learning behaviors, the coding scheme constructed by Curtis and Lawson (2001) was used. The behavior categories consist of a) planning, b) contributing, c) seeking input, d) technology, and e) social interaction. For the social presence behaviors, the same transcripts were analyzed using the coding scheme developed by Rourke, Anderson, Garrison, and Archer (1999). This code scheme includes three behavior categories: a) affective, b) interactive, and c) cohesive. During the training period, two coders, however, found that it was necessary to clarify and modify the original coding schemes in order to apply them in the current setting. Therefore, slight modifications were made through discussions between the two coders.

2.3 Coding Rules A unit of analysis is a meaning unit that conveys a single “function” or “purpose.” It is generally regarded that a paragraph conveys a single function. In some cases, however, a paragraph could have more than one unit of meaning. In that case, a paragraph was divided into meaningful sub-sections. The maximum unit of analysis was a paragraph and the minimum unit was a completed or uncompleted utterance. Only a single behavior category and code was assigned to each unit of analysis. Any disagreements were discussed by the two coders until an acceptable percentage of agreement was reached.

2.4 Inter-coder Reliability To ensure reliability, two coders independently coded online postings and their analyses were compared to identify places of agreement and disagreement. Two types of the inter-coder reliability coefficient were calculated: Holsti’s CR and Cohen’s Kappa. The following are formulas for calculating these inter-coder reliability measures. 1) Holsti’s C.R. = 2M / N1 + N2 Where M = the number of coding decisions on which the two coders are in agreement. N1 and N2 = the number of coding decisions made by coders 1 and 2, respectively. 2) Cohen’s Kappa = Po – Pc / 1 – Pc (Generally, a Kappa > .70 is considered satisfactory) Where Po = the proportion of agreement between coders Pc = the proportion of expected agreements by chance

As shown in the table below, the means of inter-coder reliability measures were .87 for Holsti’s CR, and .91 for Cohen’s Kappa, and these figures were satisfactory when considering the complexity of the content analysis method using coding schemes. Table 1. Inter-coder Reliability

Holsti’s CR Cohen’s Kappa

Collaborative Learning .87 .90

Social Presence .87 .92

Average .87 .91

416 H.-J. So / The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns

3. Results Overall, participants in the Epidemiology Committee made a total of 128 postings which was considerably higher than 82 postings in the Resources Committee. Online collaborative learning behaviors in the Resources Committee include 21.20% planning, 45.62% contributing, 14.29% seeking input, 0.92% technology, and 17.97% social interaction. The Epidemiology group shows similar behavior patterns: 21.04% planning, 46.99% contributing, 14.21% seeking input, 5.19% technology, and 12.57% social interaction. Figure 4.1 below illustrates the percentage of collaborative learning behaviors exchanged by the Resources and Epidemiology groups. In general, behaviors coded frequently in both groups were organizing work (OW), reporting progress (RP), sharing task (ST), feedback seeking/giving (FBS/FBG), and group cohesion (GC). However, there were low frequencies of challenging (CH) and explaining/elaborating (EX) behaviors, and advocating effort (EF) was not identified in transcripts. Social

Seeking Planning

Contributing

Tech. Interaction

Input

18 16 14

Percent (%)

12 10 8 6 4 2

Resources

C G

SC

CT

S

EF

S

FB

ST

HE

RP

EX

CH

SK

G

G

ER

FB

SM

HE

O

W

0

Epidemiology

Figure 1. Online Collaborative Learning Behaviors by Group Both groups showed high frequencies of planning activities. It was clear that group members exchanged online postings to clarify their project’s scope and to divide tasks. The Resources Committee, for example, exchanged several postings at the beginning to discuss the definition of the term “resources”, and to decide regional areas that they needed to research for the project. Additionally, group members used online discussion forums to discuss times and locations for face-to-face meetings. It appeared that the following set of behaviors was often sequentially coded in the collaborative working process: Member A: Sharing task (ST) - Reporting progress (RP) -Feedback seeking (FBS) Member B: Feedback giving (FBG). Group members actively exchanged their work and sought inputs from other group members. There were high instances of reporting what he or she had done with other group members,

H.-J. So / The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns 417

and this reporting behavior was often accompanied with attaching a document in progress and with seeking feedback from others. The frequency of social interaction was high due to the number of group cohesion codes. Group members used phrases such as “Good job everyone!”, “Go team!”, and “Thanks for your hard work” in order to complement and encourage each other. Social conversation unrelated to the group project was rarely found in the online discussion forum of the Epidemiology Committee. In the Resources Committee, in contrast, group members exchanged social messages to congratulate the completion of the project and to disclose personal information, such as the following: To further explore how group members expressed social presence behaviors in the process of collaborative learning, and whether there were any differences of behavior patterns between the compared groups, transcripts were analyzed using a coding scheme by Rourke et al. (1999), which included 11 indicators of social presence in affective, interactive, and cohesive categories. Both groups showed similar patterns of social presence behaviors. Cohesive behaviors were the most frequent type of social presence, accounting for more than 50% of all the social presence codes. Online discussions of the Resources Committee included 15.41% affective responses, 27.05% interactive responses, and 57.53% cohesive responses. The Epidemiology group showed 12.36% affective responses, 23.86% interactive responses, and 63.77% cohesive responses. Transcripts of the Epidemiology Committee contained considerably higher instances of social presence behaviors than those of the Resources Committee (n=461, 292 respectively). This was an expected result since group members in the Epidemiology Committee exchanged more postings than the Resources Committee did. The Social Presence Density (SPD) was calculated to quantitatively measure the extent of social presence behaviors for each group. The social presence density (Rourke et al., 1999) is an index representing a unit of social presence instances per 1000 words, and this measure allows researchers to conduct group comparisons which are not affected by the number of postings. Below is the formula for computing the social presence density: Total number of social presence indicators Social Presence Density u 1000 Total number of words in transcript The total numbers of words were 4965 for the Resources Committee and 7995 for the Epidemiology Committee, and thus, the social presence density for each group was calculated as follows: 292 SPD u 1000 58.81 4965 Resources Committee:

SPD Epidemiology Committee:

461 u 1000 7995

57.66

The social presence density measures for the Resources and Epidemiology groups were 58.81 and 57.66 respectively, indicating that there was no prominent difference in terms of the frequency of social presence behaviors. The interpretation of these SPD measure is that approximately 58 or 59 social presence behaviors appeared per 1000 words in both discussion forums. Next, the social presence density was calculated for each code in order to examine specific differences among affective, interactive, and cohesive categories (see Figure 2). The Resources Committee had higher measures of the social presence density for 9 indicators than the Epidemiology Committee.

418 H.-J. So / The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns Interactive

Affective

Cohesive

Social presence den

25

20

15

10

5

0 EM

UH

SD

CT

QM

RM

Resources

AQ

CM

AG

VC

WE

PS

Epidemiology

Figure 2. Social Presence Density by Group One noticeable difference was found in the vocative code (VC) where the social presence density for the Epidemiology Committee was considerably higher than one for the Resources Committee. This indicates that group members in the Epidemiology Committee often interacted by addressing each other’s name.

4. Conclusion

Although collaborative learning has been used to increase opportunities of learner-learner interaction, prior research has shown that collaborative learning in distance education contexts is a difficult approach to implement due to the lack of face-to-face interaction, and students were often dissatisfied and frustrated with the process of collaborative learning. Recognizing the importance of social interaction and psychological distance, recently the concept of social presence has received much attention by distance education researchers. Described as a degree of perceptions and relatedness to other people, social presence has been suggested to be a strong factor affecting students’ levels of satisfaction and interaction in distance courses. This study, therefore, examined what types of social presence and collaborative learning behaviors were presented in the process of working on the group project in a computer-mediate learning environment. The qualitative content analyses of two online discussion forums revealed that students showed affective, interactive, and cohesive behaviors of social presence to encourage and help each other in the process of collaborative learning. Additionally, this study found that while students actively exchanged resources and provided feedback each other, there was a lack of elaborating and challenging behaviors. In conclusion, it is argued that social presence played an important role in collaborative learning, and that distance educators should consider instructional methods to increase the level of student perceptions of social presence in order to facilitate collaborative learning processes.

H.-J. So / The Content Analysis of Social Presence and Collaborative Learning Behavior Patterns 419

References [ 1] Curtis, D. & Lawson, M. (2001). Exploring collaborative online learning. Journal of Asynchronous Learning Networks. 5, 21-34 [ 2] Graham, M., Scarborough, H., & Goodwin, C. (1999). Implementing computer mediated communication in an undergraduate course- a practical experience. Journal of Asynchronous Learning Network, 3(1), 32-45. [ 3] McAlpine, I. (2000). Collaborative learning online. Distance Education, 21(1), 66-80. [ 4] Kitchen, D., & McDougall, D. (1999). Collaborative learning on the Internet. Journal of Educational Technology Systems, 27, 245-258. [ 5] Ragoonaden, K., & Bordeleau, P. (2000). Collaborative learning via the internet. Educational Technology & Society, 3(3). [Online] http://ifets.ieee.org/periodical/vol_3_2000/d11.html [ 6] Gunawardena, C. N., & Zittle, F. J. (1997). Social presence as a predictor of satisfaction within a computer-mediated conferencing environment. The American Journal of Distance Education, 11, 8-26. [ 7] Rourke, L. Anderson, T., Garrison, D.R., & Archer, W. (1999). Assessing social presence in asynchronous text-based computer conferencing. Journal of Distance Education, 14(2), 51-70 [ 8] Tu, C. H., & McIsaac, M. (2002). The relationship of social presence and interaction in online classes. American Journal of Distance Education, 16, 131-150. [ 9] Vygotsky, L. (1978). Mind in society: the development of higher psychological process. Cambridge, MA: Harvard University Press. [ 10] Bonk, C. J., & Cunningham, D. J. (1998). Searching for learner-centered, constructivist, and sociocultural components of collaborative educational learning tools. In C. J. Bonk & K. S. King (Eds.), Electronic collaborators (pp. 25-50). Mahwah, New Jersey: Lawrence Erlbaum Associates. [ 11] Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications. London: John Wiley & Sons.GG

420

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

An Intelligent Agent that Learns How to Tutor Students: Design and Results Leen-Kiat Soh and Todd Blank Department of Computer Science and Engineering University of Nebraska, Lincoln, NE 68588-0115 USA { lksoh, tblank }@cse.unl.edu Abstract. In this paper, we describe an intelligent agent that presents different learning content such as tutorials, examples, and problems adaptively to individual students and learns from its interaction with the students how to improve its performance. We have built an end-to-end intelligent tutoring system, premised on the above goal, with a graphical user interface (GUI) front-end, an agent powered by case-based reasoning (CBR), and a mySQL database backend. We use a casebase to store the pedagogical strategies, embedded in the individual cases and the similarity retrieval and adaptation heuristics. Each case has situation, solution and outcome parameters. The situation parameters include the students' static and dynamic profiles and the instructional content’s characteristics while the solution parameters specify the characteristics of the example or problem to be delivered to the student. We developed a set of CS1 content that includes five topics and deployed our system in the laboratories. Our results show that when the machine learning mechanism is activated, our agent is able to learn to tutor students more efficiently.

Introduction There exists no universal agreement on which abilities are needed or present in intelligent tutoring systems (ITSs) [12]. ITSs may possess a combination of the following attributes [12]: (1) Generative in which an ITS dynamically generates instructional material including problems, hints, and help on student performance; (2) Able to model students in which an ITS assesses the current state of a student’s knowledge and does something instructionally useful based on the assessment; (3) Able to model expert performance in which an ITS detects expert performance and derive an instructionally useful model; (4) Able to change pedagogical strategies in which an ITS changes its instructional style based on the changing state of the student model, prescriptions of an expert model, or both; (5) Mixedinitiative in which an ITS changes its presentation schemes to improve human-computer interaction; and (6) Self-improving in which an ITS has the capacity to monitor, evaluate, and improve its own teaching performance as a function of experience. Based on the above, a truly intelligent ITS should be adaptive, leading to customized tutoring and learning for individual students. Further, for any intelligent system to be adaptive, it must be able to (1) monitor and evaluate its own interaction with the students, (2) modify its knowledge if the evaluation result is below par, and (3) subsequently apply the modified knowledge. An ITS’ knowledge generally consists of instructional content (tutorials, examples, problems, etc.), modeling of students, and pedagogical strategies. Instructional content might not be always appropriate (for example, a particular problem may be appropriate for one student but too easy for another student), modeling of students might not be accurate (for example, a student who fails three problems in a row could be perceived as lacking understanding of the topics involved or lacking motivation to study the

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

421

problems carefully), and in some instances, pedagogical strategies might not be applied correctly. Thus, when an ITS modifies its knowledge, it could modify its instructional content, its modeling of a particular student, or its confidence in a particular pedagogical strategy In this paper, we propose an ITS with machine learning capabilities to refine its performance over time. The ITS, named Intelligent Learning Materials Delivery Agent (ILMDA), models students based on how they interact with the system (not just questionanswer assessment), changes its pedagogical strategies, and self-improves. ILMDA’s machine learning capabilities are powered by case-based reasoning (CBR) [8]. We use a casebase to store the pedagogical strategies, embedded in the individual cases and the similarity retrieval and adaptation heuristics. In the following, we first discuss some related work in the area of intelligent tutoring systems. Then, we present our ILMDA project, its goals and framework. Subsequently, we outline our use of the CBR methodology. We then describe the implementation of ILMDA briefly. We have built an end-to-end ILMDA infrastructure, with an interactive GUI frontend, an agent powered by CBR and capable of learning, and a mySQL multi-database backend. We have deployed ILMDA in a CS1 course in the Fall semester of 2004. The results, though preliminary, are encouraging, indicating that ILMDA is able to learn to deliver more appropriate examples and problems based on student behavior.

1. Related Work There have been successful ITSs such as PACT [7], ANDES [4], AutoTutor [5], and SAM [3]. Though ITSs have been tested on students and have been proven to facilitate learning, Graesser et al. [5] pointed out the weaknesses of the current state of tutoring systems: First, it is possible for students to guess and find an answer and such shallow learning will not be detected by the system. Second, ITSs do not involve students in conversations so students might not learn the domain’s language. Third, to understand the students’ thinking, the GUI of the ITSs encourages students to tend to focus on the details instead of the overall picture of a solution. Furthermore, these systems do not generally adapt to new circumstance, do not self-evaluate and self-configure their own strategies, and do not monitor the usage history of the instructional content being delivered or presented to the students. A closely-related work to ILMDA is AnimalWatch. AnimalWatch [12] is an arithmetic tutor tailored for female elementary school students. It has a reinforcement learning component that automatically computes an optimal teaching policy, such as reducing the amount of mistakes made or time spent on each problem by using a “two-phase” learning algorithm: (1) model how a student interacted with the tutor, and (2) construct a tutoring policy. The reinforcement learning mechanism takes as input as set of abilities that describe the current tutoring situation and output a recommended action (e.g., which problems to generate and what type of hints to show). The tutor then observes whether the student’s response is correct and how much it takes the student to generate this response. By keeping these observations over time, AnimalWatch is able to make predictions about “an average student with a proficiency of x who has made y mistakes so far on the problem.” The authors report that students using the machine learning version averaged 27.7 seconds to solve a problem, while students using the classic version averaged 39.7 seconds. Our ILMDA design is similar to AnimalWatch in terms of the underlying concept—observing student interaction with the system and reinforcing decisions based on the observed outcomes. However, our machine learning component is comprehensive: it refines the existing pedagogical strategies that it has in its repository, learns new

422

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

pedagogical strategies, learns how to apply them, learns when to use them; it also learns to quality-tag the system’s instructional content.

2. Design Our ILMDA system is based on a three-tier design. It consists of a graphical user interface (GUI) front-end application, a database backend, and the intelligent agent reasoning in between. A student user accesses the content set via the GUI. The agent captures the student’s interactions with the GUI and provides the ILMDA reasoning module with a parametric profile of the student and environment. The ILMDA reasoning module performs CBR to obtain a search query (a vector of search keys), and then uses the query to retrieve the most appropriate example or problem from the database. The agent then delivers the example or problem in real-time back to the user through the interface.

2.1 Case Each instructional content set consists of three parts: (1) a tutorial, (2) a set of related examples, and (3) a set of exercise problems to assess the student’s understanding of the topic. Based on how a student progresses through the content set and based on his or her profile, ILMDA chooses the appropriate examples and exercise problems for the student. How to choose the appropriate examples and exercise problems is based on case-based reasoning (CBR). ILMDA has a casebase of cases. Each case is composed of three parts: situation, solution, and outcome parameters. The situation parameters include the student static and dynamic profiles and the instructional content’s characteristics. The student static and dynamic profiles form the basis for the ILMDA’s learner model. A learner model tells us the metacognitive level of a student by looking at the student’s behavior as he or she interacts with the instructional content set. We achieve this by profiling a learner/student along two dimensions: student static background and dynamic student activity. The background of a student stays relatively static and consists of the student’s last name, first name, major, GPA, goals, affiliations, aptitudes, and competencies. It also includes self-reported self-efficacy and motivation, based on a survey taken before a student processes a content set. The dynamic student profile captures the student’s real-time behavior and patterns. It consists of the student’s online interactions with the GUI module of ILMDA including the number of attempts on the same item, number of different modules taken so far, average number of mouse clicks during the tutorial, average number of mouse clicks viewing the examples, average length of time spent during the tutorial, number of quits after tutorial, number of successes, and so on. For details on these parameters, please refer to [1]. The solution parameters specify the characteristics of the example or problem to be delivered to the student. Each example or problem is meta-tagged with a set of attributes: length, interest, Bloom’s taxonomy, level of difficulty, degree of scaffolding, the number of times viewed, the average time per use, average number of clicks per use, and so on. Bloom et al. [2] identified three domains of educational activities. The three domains are cognitive, affective, and psychomotor. Cognitive is for mental skills (Knowledge), affective is for growth in feelings or emotional areas (Attitude), while psychomotor is for manual or physical skills (Skills). There are six major categories, starting from the simplest behavior to the most complex: knowledge, comprehension, application, analysis, synthesis, and evaluation. Currently, most of our examples and problems are at first four of Bloom’s levels.

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

423

Instructional scaffolding [11] is support for learning, and the level of scaffolding varies for different students and scenarios. In ILMDA, we use scaffolding similar those proposed in [6]. Specifically, we use cues, hints, references, and elaborations. Cues are highlighted phrases. A hint poses “what if” and “think about this” statements. A reference points the student to a particular item in the content set. An elaboration explains the steps or partial solutions. We use highlighted phrases, diagrams, and figures as cues; “What if” and “Think about this” questions as hints and prompts; and stepwise, spelled-out solutions as partial solutions. In addition, references will be used to point students back to certain pages of a tutorial or an example. The outcome parameters are based on the usage history of a case, which includes the number of times the case has been used, the number of times the case has been used successfully, whether the student quit the example or problem, whether the student answers the problem correctly, and so on.

2.2. Case-Based Reasoning (CBR) and Learning Briefly, a CBR system works as follows. When a new situation arises, the system searches its casebase for cases whose situations are similar to the new situation. This is termed similarity-based retrieval. Usually, the system retrieves the most similar case as the best case. Given the best case, the system now has in its reasoning process a solution. However, this solution is one that corresponds to the best case’ situation. Thus, the system has to adapt the solution to match the new situation. Usually, the system modifies the solution based on the differences between the best case’s situation and the new case's situation. After adaptation, the solution is applied. After application, the new situation and the adapted solution, together with the outcome, become a new case. If this new case is sufficiently different from all existing cases in the casebase, the system adds it to the casebase. The addition of a new case is case-based learning (CBL). Each case documents a unique pedagogical strategy: given the situation, what is the appropriate solution? ILMDA forms the situation based on its observation of the student. For example, for students who have higher motivation, self-efficacy and aptitude, ILMDA will give lower level of scaffolding such as cues; and vice versa. With this strategy, ILMDA will gradually withdraw the scaffolds as a student is observed to have improved his/her performance. In our agent design, we have three case-based learning modules. First, there is a case learning module that learns about good and bad cases. Secondly, we have a similarity learning module to learn about the significance of situation parameters such as GPA and the time spent in the tutorial. This allows us to retrieve most weighted similar cases as best cases, taking into consideration that some parameters are more important than the others. Finally, we have an adaptation learning module that learns about the importance or quality of the adaptation heuristics. The case learning module uses traditional CBL and a small amount of reinforcement learning. The other two modules meta-learn how to better retrieve from the database, evaluate the similarity between two cases, and adapt the solution of the best case to the current problem. For details on these modules, please refer to [1]. Briefly, by recording the success rate of each case, ILMDA refines its existing pedagogical strategies. Through the case learning module, ILMDA learns new pedagogical strategies. Through the similarity learning module, ILMDA learns when to use them. Through the adaptation learning module, ILMDA learns how to apply them. For detailed assumptions behind our agent learning and case adaptation, please refer to [9].

424

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

3. Implementation We have implemented an end-to-end prototype ILMDA system with a front-end, appletbased Graphical User Interface (GUI) that captures all user mouse activities, a CBRpowered agent, and a backend database system containing a set of learning materials. The ILMDA system is written in Java (SDK version 1.4.2). Both integrated development environments were run under Windows 2000/XP operating systems. The backend database is a MySQL database (Version 3.23). The system connects to the database using the JDBC drivers for Java. We have also built a comprehensive simulator [10] to provide us with virtual students to use ILMDA. This simulator allows us to test the correctness and feasibility of the ILDMA before deployment.

4. Results In Fall 2004, we deployed ILMDA to CSCE155 at the Department of Computer Science and Engineering at the University of Nebraska. CSCE155 is the first core course for the Computer Science majors. Typically, it has about 150 students per year, with a diverse group of students from majors such as CS, Math, Engineering, and so on. Further, the programming background of these students is highly diverse. Some incoming freshmen have had some programming in their high schools; some have had none. Thus, it is important for the course to be able to adapt to the different student aptitude levels and motivations. This course is thus well-suited for evaluating ILMDA. The course has a 2-hour structured lab component once a week. We used ILMDA in four CS1 labs: (1) File I/O, (2) Exceptions, (3) Inheritance and Polymorphism, and (4) Recursion. For each topic, the content set had a tutorial, a set of 3-4 examples, and a set of 20-25 problems. Students reviewed these materials through ILMDA prior to their labs. For our evaluation, we used two types of agents: learning and non-learning. Our nonlearning ILMDA did not refine its adaptation heuristics, did not learn new cases, and did not adjust it similarity weights. It basically disabled all its learning capabilities. The learning ILMDA enables all its machine learning mechanisms. We set up the ILMDA system such that it toggled on and off the learning mechanisms alternatively for each login.

4.1 Case-Based Learning Results: Impact on Pedagogical Strategies Initially, at the beginning of the Fall 2004 semester, we set the weights on the situation parameters to 1.0; that is, all situation parameters are considered equally important. At the end of the semester, however, ILMDA modified these weights based on its observation of how many times a retrieved case was useful. Briefly, we found that GPA was not that important (dropping from 1.0 to 0.46), quitting a problem before answering it was important (increasing from 1.0 to 2.54), the number of times a student went back-and-forth between an example and the tutorial was quite important (0.80) but not as important as the back-and-forth between a problem and the tutorial (0.99), and so on. These results are empirical data, key to the self-evaluation capabilities of ILMDA to meta-learn about when to use which pedagogical strategies. We also observe learning results in adaptation heuristics—there are 9 sets of heuristics, and each heuristic is triggered by a subset of about 20 situation parameters. An example of adaptation heuristics is as follows: To adapt the solution parameter BLOOM (i.e., the Bloom’s taxonomy), we look at the differences between the new situation and the best case’ situation. Initially, at the beginning of the semester, we set the weights of each situation

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

425

parameters on the adaptation heuristics to be 1.0. Take the number of questions answered correctly by a student (extracted from the student’s profile) as “successes”. If “successes” of the new situation is greater than “successes” of the best case’ situation, then increase the Bloom’s taxonomy level of the problem to be selected next. At the end of the semester, this number increased to 9.0. We also observe a significant change in the weight of “exmpQuits” (the number of quits during an example), going from 1.0 to -11.76. This indicates that the more often a student has quit during the viewing of an example, the lower the Bloom’s taxonomy level is for the problem to be selected. Another example is to adapt the solution parameter SCAFFOLDING (i.e., the degree of scaffolding). Initially, we expected that if a student is observed to have spent more time on a tutorial, example, or problem, or if a student is observed to go back-and-forth between example and tutorial, or problem and example, then ILMDA should provide a higher degree of scaffolding. However, after a semester of using ILMDA, our expectation was not met for “aveExmpTime” (average time spent on an example), “expToTtrl” (number of times going from example to tutorial, and “probToExmp” (number of times going from problem to example). We suspect that this was due to the following: When students were found to spend more time on examples, or were found to refer back to the examples, they were more likely to quit ILMDA before answering questions correctly. As a result, ILMDA adjusted its adaptation heuristics to reduce the degree of scaffolding, thinking that this would improve its performance.

4.2 A Case Study: Recursion Here we report a case study on the Recursion topic: x The average numbers of examples and problems that a student read when interacting with the learning ILMDA were 2.216, and 10.946, respectively and that with the non-learning ILMDA were 3.047 and 15.116, respectively. This indicates that the learning ILMDA system was able to deliver fewer examples and problems. x The average percentage of problems answered correctly by a student indicates how well a student did as part of our assessment. When interacting with the learning ILMDA system, the percentage was 66.2%. When interacting with the non-learning ILMDA system, it was 60.2%. This is encouraging. This hints that the learning ILMDA was able deliver more appropriate examples and problems. Combining these two observations, we see that the learning ILMDA seems to be effective (delivering more appropriate examples and problems) and efficient (delivering fewer examples and problems). x Table 1 shows the average number of seconds spent in each section. Students interacting with the learning ILMDA spent more time than students interacting with the non-learning ILMDA. This could be due to the learning ILMDA providing more interesting and appropriate examples and problems, engaging the students to invest more time and effort into reading the materials. Section Tutorial Per Example Per Problem 457 64 74 Learning 276 61 30 Non-Learning Table 1: Average time spent on each section in seconds for learning and non-learning ILMDA.

4.3 Efficiency and Effectiveness To further measure the efficiency and effectiveness of ILMDA’s machine learning capabilities, we devise a scheme as follows. We first identify a level of topical

426

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

comprehension for students to attain: answering one of the most difficult problems correctly. To be effective, ILMDA should learn to guide students to reach that level. To be efficient, ILMDA should learn to guide students to reach that level with a small number of problems. Tables 2 and 3 show the results. From Table 2, for File I/O, Exceptions, and Inheritance, the learning ILMDA consistently used fewer problems to bring a student to the level of topical understanding. However, the learning ILMDA did more poorly for the Recursion topic—requiring more than 2 additional problems on average to achieve a level of topical understanding. Comparing this observation with that in the above case study, we conclude this: Although the learning ILMDA increased the percentage of correct answers from the students, it did not bring them towards answering difficult problems efficiently. We suspect that the reason for this learning failure is due to the recursion topic. Unlike the other three topics which are closely related to programming, recursion is more closely related to problem solving. Upon closer analysis, we realized that ILMDA gave students difficult questions more often than easy questions in the recursion topic, with a correlation of 0.11 between the number of times a question is given and its difficulty level, whereas ILMDA gave students easy questions more often in other topics (correlations = 0.2–0.56). We will investigate this further in our next round of evaluation.

Deg. Diff. 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0

File I/O No Learning 8.78 8.78 12.00 12.00

Learning 6.50 6.50 11.36 11.36

Exceptions No Learning 8.75 8.75 15.28 25.66

Learning 4.10 4.10 15.80 14.50

Inheritance No Learning 3.57 3.57 4.00 6.60 8.2 8.2

Learning 2.50 2.50 2.50 2.75 7.25 7.25

Recursion No Learning 4.34 4.34 7.44 7.44 8.00 8.08 13.13 13.13

Learning 6.67 6.67 10.33 10.33 11.28 11.28 15.10 15.10

Table 2: Average number of problems given by ILMDA to a student to eventually correctly answer a problem with a certain degree of difficulty. Empty cells are due to that there are not problems of that degree of difficulty. Also, we did not include the data for problems with degree of difficulty smaller than 5.5.

Table 3 shows the percentages of failed sessions. A failed session is one in which a student quits before reaching one of the most difficult problems. We see that the results are not conclusive. That means we do not have evidence that learning ILMDA is more effective than the non-learning ILMDA. Upon closer analysis, we realize that the cases in our casebase were initially tuned to minimize the number of problems given to the students, thus prompting ILMDA to be more aggressive in giving more difficult problems but more conservative in giving easier problems. Further, in a student profile, a student usually answers questions correctly more often than not since there are more easy questions. Thus, an incorrect answer might not cause ILMDA to change its perception of the student drastically. Combining the results of Table 3 with that of the case study on recursion, we conclude this: Even though the learning ILMDA was able to increase the percentage of correct answers from the students, it was not able to engage students consistently longer for them to stay on through the set of problems.

5. Conclusions and Future Work We have described an intelligent tutoring system that learns how to tutor students. The system, called ILMDA, uses case-based reasoning to apply pedagogical strategies, and

L.-K. Soh and T. Blank / An Intelligent Agent that Learns How to Tutor Students

427

learn new ones, learn how and when to apply them. By storing the pedagogical strategies in cases, our ILMDA could be applied to different student groups and content topics by replacing the cases. This “plug-and-play” feature makes ILMDA a potentially useful testbed and knowledge-discovery platform. Results from our initial deployment in Fall 2004 are mixed. There are encouraging indicators that the learning ILMDA is efficient. Yet, there lacks evidence for the learning ILMDA being effective. Our current and future work include the continuous deployment in Spring 2005 and more comprehensive analyses.

Deg. Diff. 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0

File I/O No Learning 41.94% 41.94% 45.16% 45.16%

Learning 31.43% 31.43% 37.14% 37.14%

Exceptions No Learning 66.67% 66.67% 70.83% 75.00%

Learning 40.00% 40.00% 66.67% 66.67%

Inheritance No Learning 53.33% 53.33% 60.00% 66.67% 66.67% 66.67%

Learning 69.23% 69.23% 69.23% 69.23% 69.23% 69.23%

Recursion No Learning 40.91% 40.91% 43.18% 43.18% 45.45% 45.45% 50.00% 50.00%

Learning 51.35% 51.35% 59.46% 59.46% 62.16% 62.16% 72.97% 72.97%

Table 3: Percentage of failed sessions at various degrees of difficulty for the four topics. For the InheritanceLearning column, students who quit all quit at level 5.5.

6. Acknowledgement This work is supported in part by CSE Department, the Great Plains Software Technology Initiative, and an Enhancing Teaching and Learning Grant, all at the University of Nebraska. The authors thank L.D. Miller for his contribution to the project. References [1] Blank, T., Miller, L.D., Soh, L.-K., & Person, S. (2004). Case-Based Learning Mechanisms to Deliver Learning Materials. Proc. ICMLA’2004, Louisville, KY, USA, December 16-18, pp. 423-428. [2] Bloom, B. S., Mesia, B. B., & Krathwohl, D. R. (1964). Taxonomy of Educational Objectives (two vols: The Affective Domain & The Cognitive Domain). New York: David McKay. [3] Cassell, J., Annany, M., Basur, N., Bickmore, T., Chong, P., Mellis, D., Ryokai, K., Smith, J., Vilhjálmsson, H., & Yan, H. (2000). Shared Reality: Physical Collaboration with a Virtual Peer, ACM SIGCHI Conf. on Human Factors in Computer Systems, April 1-6, The Hague, The Netherlands. [4] Gertner, A. S. & VanLehn, K. (2000). ANDES: A Coached Problem-Solving Environment for Physics. Proc. ITS’2000, pp. 133-142. [5] Graesser, A. C., VanLehn, K., Rosé, C. P., Jordan, P. W., and Harter, D. (2001). Intelligent Tutoring Systems with Conversational Dialogue, AI Magazine, 22(4):39-51. [6] Hartman, H. (2002). Scaffolding and Cooperative Learning, Human Learning and Instruction, pp. 23-69. New York: City College of City University of New York.

[7] Koedinger, K. R., Anderson, J. R., Hadley, W. H., and Mark, M. A. (1997). Intelligent Tutoring Goes to School in the Big City, Journal of Artificial Intelligence in Education, 8(1):30-43. [8] Kolodner, J. (1993). Case-Based Reasoning. Morgan Kaufmann. [9] Soh, L.-K., Blank, T., and Miller, L. D. (forthcoming). Intelligent Agents that Learn to Deliver Online Materials to Students Better: Agent Design, Simulation, and Assumptions. In C. Chaoui, M. Jain, V. Bannore, and L. C. Jain (eds.) Studies in Fuzziness and Soft Computing: Knowledge-Based Virtual Education, Chapter 3, Volume 178, May 2005, pp. 49-80. [10] Soh, L.-K., Blank, T., Miller, L. D., & Person, S. (2005). ILMDA: An Intelligent Learning Materials Delivery Agent and Simulation. Proc. IEEE Int. EIT. Conf., Lincoln, NE, USA, May 22-25. [11] Vygotsky, L. S. (1978). Mind in Society, Cambridge, MA: Harvard University Press. [12] Woolf, B. P., Beck, J., Eliot, C., & Stern, M. (2002). Growth and Maturity of Intelligent Tutoring Systems: A Status Report. In Forbus, K. D. and P. J. Feltovich (2002). (Eds.) Smart Machines in Education, Menlo Park, CA: AAAI Press, pp. 99-144.

428

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Computer-Supported Structured Cooperative Learning Leen-Kiat Soh, Nobel Khandaker, Xuli Liu, and Hong Jiang Computer Science and Engineering University of Nebraska-Lincoln 256 Avery Hall, Lincoln, NE 68588-0115 {lksoh, knobel, xuliu, jiang }@cse.unl.edu Abstract: In this paper we describe computer-supported structured cooperative learning, applied to the laboratories of an introductory CS1 course. We extended IMINDS, a multiagent system that supports student-teacher and student-student realtime interactions, to support a cooperative learning paradigm called Jigsaw. Jigsaw considers of several phases of activities, including dividing students into different groups for focused exploration, reporting and reshaping, and integration and evaluation. I-MINDS supports the formation of teams and monitors the interaction. We have deployed I-MINDS in the hands-on programming laboratories in CS1, comparing student performance in the lab using only Jigsaw (where students carried out the Jigsaw process with face-to-face interactions) and in the lab using I-MINDS supported Jigsaw (where students could interact only through I-MINDS). We report the results of our study on the design of I-MINDS to support Jigsaw and the impact of IMINDS in student performance.

Introduction Most computer-supported collaborative learning (CSCL) systems (e.g., [2-4], [9], [11]) have in general three key weaknesses. First, the collaboration among students are not monitored, measured and analyzed automatically. Vital information on an individual student’s collaborative performance and behavior is lost. Further, even when the students’ collaborative activities are tracked, real-time analyses are not available automatically to support students or teachers. Second, the collaboration is free-formed and thus not structured. Often times, teachers design various cooperative learning activities using different models to structure how students should collaborate and to target different collaborative strategies to measure. Third, the collaboration among students is not actively supported: the CSCL system does not alert the instructor about a student of his or her lack of participation, does not encourage a student to increase his or her participation, and so on. In our research, our long-term goal is to support structured cooperative learning using an intelligent multiagent system where students and instructors may or may not have face-toface interactions. Cooperative learning is basically an instructional strategy where students form small groups or teams and work together to maximize their own and each other’s learning [6]. The system will track vital data on student activities to help the instructor monitor and evaluate student progress and performance more accurately and more conveniently. In structured cooperative learning, students are guided within a set of prescribed activities so that each activity has a set goal and measurable outcomes. An agent is a software module that observes and receives input stimuli from its environment, makes autonomous decisions based on these stimuli, and actuates actions to carry out these decisions, which, in turn, changes the environment and a multiagent system is one in which agents work competitively or cooperatively to accomplish task [12]. The advancement of multi-

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

429

agent systems presents a major opportunity to develop an infrastructure in which agents communicate, exchange experiences, and cooperate to better serve the instructors and students. In the following, we first describe our I-MINDS prototype, which is a CSCL system powered by multiagent intelligence [7]. Then we discuss a structured cooperative learning paradigm called Jigsaw [5]. Subsequently, we present the extension to I-MINDS to support Jigsaw and our study setup. In Section 4, we discuss and analyze the results of our initial deployment of I-MINDS-supported Jigsaw in our introductory CS1 course, specifically in our hands-on programming labs. Finally, we conclude.

1. I-MINDS I-MINDS (Intelligent Multiagent Infrastructure for Distributed Systems in Education) employs a system of intelligent software agents, representing individual students and the instructor (or teaching resource in the case of an asynchronous course or lesson). The rationale behind using multiagent intelligence is the agent’s persistence in tracking and monitoring its environment (student and instructor activities), autonomy in decision making, and responsiveness in providing services to both students and instructors. These are properties that are useful for distance learning and large classrooms. Briefly, in I-MINDS, each student has a personal agent (a student agent), and each instructor as a personal agent (a teacher agent). All these agents interact with their respective users as well as among themselves. These agents exchange information, coordinate their actions, and track inter-agent activities behind the scene. Currently, a teacher agent, for example, automatically ranks student questions and sorts them for the instructor. A student agent automatically monitors a student’s peer group and continuously refine the peer group (by removing and adding new peers) based on the student profiles that the student agent obtain from other agents. Details of the prototype are available from [7-8]. At the current stage, our I-MINDS technology is matured. It has multimedia capability—real-time audio and video streaming such that each student/instructor is able to transmit audio and video through the I-MINDS server. The I-MINDS agents are equipped with (1) tracking capabilities—recording the messages communicated between student agents (or students), the length of each message, the time stamp of each message, each question asked by a student to an instructor or another student, and so on, (2) machine learning capabilities—adjusting weights based on student and instructor actions that feed back to the decision making modules (e.g., how to rate a question, when to make an alert, and so on), and (3) collaborative environments such as chatrooms, token-based digital whiteboards and automated digital archival of notes and lectures. In addition, I-MINDS can be easily downloaded and installed via the Web. When a lecture starts, the teacher opens a session. Students logging in through the Web then join the session accordingly.

2. Jigsaw 2.1 Jigsaw Cooperative Learning Model The Jigsaw cooperative learning model was first introduced by [1]. This procedure works as follows. First, assign the students into groups. Second, the instructor divides a problem into different parts (or sections). Third, the instructor assigns a part/section for every student such that members of the same group will have different sections to solve. The students who are responsible for the same section then work together to come up with solu-

430

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

tions to the section to which they have been assigned and develop a strategy for teaching the solutions to their respective group members. Clarke [5] further refined the Jigsaw structure into stages. These stages are (1) Introduction of the topic to the class as a whole, (2) Focused Exploration: The focus groups explore issues pertinent to the section that they have been assigned, (3) Reporting and Reshaping: The students return to their original groups and instruct their teammates based on their findings from the focus groups, and (4) Integration and Evaluation: The team connects the various pieces generated by the individual members, address new problems posed by the instructor, or evaluates the group product.

2.2 I-MINDS Supported Jigsaw Cooperative Learning Model Following the four stages of the Jigsaw structure [5], we have added to the current IMINDS prototype capabilities that specifically support the last three stages: Focused Exploration, Reporting and Reshaping, and Integration and Evaluation. The current IMINDS prototype fully supports the Introduction stage with multimedia capabilities. At present I-MINDS assigns an intelligent agent to every student who is in the classroom and assigns a teacher agent to the instructor. These agents help the instructor monitor the student activities and performance. During the introduction phase the teacher agent monitors the performances of all the students in the classroom by classifying and ranking their questions. After the introduction phase the instructor declares the next task and initiates the group formation process. When signaled by the teacher agent, the student agents prompt their assigned students to fill out the Self Efficacy Questionnaire. The Self Efficacy Questionnaire is a survey that measures how competent the student thinks he or she is to do the assigned task. After the students have posted their Self Efficacy Questionnaire the teacher agent chooses the n best students from the classroom. These n students become the first members of the n buddy groups. Here n is determined by the number of subtasks the assigned task contains. Then the teacher agent initiates an auction where the remaining student agents bid to join their favorite group. Once the main groups are formed, the teacher agent chooses one student from each group randomly and forms the focus groups. The number of focus groups is equal to the number of subtasks the assigned task consists of. For details on the questionnaires, please refer to [10]. Focused Exploration To support this stage, we have developed focus group agents (FGAs) that, given a particular subtask assigned by the instructor, monitors and support the focus groups. During the focused exploration stage, the goal of the cooperative learning is to encourage the students to explore a particular topic in a focused manner. Thus, active interactions among the students, an increasing focus in their discussions, and a tangible conclusion are expected of such as an activity. Each FGA have the following capabilities to support its focus group: (1) tracking the messages exchanged between the focus groups (2) allowing the focus group to ask questions to the instructor, and (3) monitoring the activities of the exploration and reporting to the instructor of its observation of the progress of the exploration. As part of our continuous enhancement of I-MINDS, we plan to have FGAs that model each student’s interaction with his or her focus groups to identify, for example, over-dominating members, sporadic discussions, discussions that are not fruitful, and to prompt discussions by automatically injecting questions or hints. Reporting and Reshaping After exploration, the students return to their original groups and instruct their teammates based on their findings from the focus groups. Here, we have built buddy group agents (BGAs) to support the reporting and reshaping process. During this stage, the reporting involves sequences of questions and answers (Q&A), with the questions from the teammates and the answers provided by the topic/section expert,

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

431

which leads to refinement of the topics/sections. Each “buddy” in the group has a chance to be the expert on a particular topic/section and report to his/her teammates. Thus, each BGA have very similar capabilities as those focus group agents. Further, since for each assignment or task, there are multiple sections/topics (and the same number of members per group), there are multiple reporting and reshaping sessions. Each BGA compares the activities of each student as an expert and as a teammate in the buddy group and monitor the student’s interaction with his/her buddy group members. At the end of the reporting and reshaping, each team or buddy group prepares a written document of solutions to the assigned task. Integration and Evaluation At this stage, the team or the buddy group connects the various pieces generated by the individual members. Once the teams have completed the solution to the problem, the student agent prompts the students to fill out a Team-based Efficacy Questionnaire to evaluate the performance of their team and a Peer Rating Questionnaire to evaluate the performance of their buddy group members. This information is stored in the database and is used by the student agent in future group formation processes. After the prepared solutions are submitted, the instructor evaluates the integrated results. In future we will expand the teacher agent (TA) in the I-MINDS prototype with the following capabilities: (1) collecting and comparing the solutions to the results of the focus groups—for example, solutions that are too similar to the results of the focus groups indicate that the team has not performed a good job in reshaping the solution, (2) allowing the instructor to grade and score digitally the solutions, (3) allowing the instructor to administer online quizzes and scoring the quizzes, and (4) providing both a student-centric database and a team-centric database how each unit (student or team) performs during the last three stages of the Jigsaw process. Note that for now, we have simplified the role of the agents at this stage and rely on the instructor to be the ultimate evaluator of each group and student. The agents’ role here is to provide data and information for the instructor.

3. Implementation We have extended the original I-MINDS prototype to support structured cooperative learning, specifically Jigsaw. At this stage of our development, I-MINDS+Jigsaw has the following features. The teacher agent has features for specifying a task and its sub-tasks, announcing the task to all student agents, forming buddy main groups and focus groups using a multiagent bidding policy, initiating and ending different phases of Jigsaw, and reporting on the activities of each phase. The student agent has features for responding to the three questionnaires outlined in Section 2 (for details of these questionnaires, please refer to (Soh 2004)), and interacting with different groups of peers. There are also rudimentary versions of the proposed buddy group agents (BGAs) and focus group agents (FGAs). Currently, our focus group agents are able to track the number of messages sent among students in a focus group and score each student based on their performance in a group. All questionnaires, activities, and messages are logged and stored in a mySQL database. We have also implemented a Web-based query interface to conveniently access the database for real-time monitoring and off-line analysis. Ultimately, we aim to incorporate this interface directly into the teacher agent interface as one single package. We have installed I-MINDS on a server, where an I-MINDS manager (that takes care of student and teacher login and classroom initialization) and the teacher agent reside. Students login on their computers and download and install the student agent on their respective sites. They invoke the student agent executable and log in by supplying the IP address of the server and their username and password. Once everybody is logged in, then the IMINDS-supported cooperative learning session can begin.

432

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

4. Results Our initial study was designed to (1) compare I-MINDS-supported Jigsaw with simply Jigsaw in terms of student performance, and (2) evaluate how I-MINDS+Jigsaw supports structured cooperative learning. The course chosen for our study was CSCE 155 Introduction to Computer Science, which is the required CS1 core course for computer science students. This course has 3 hours of lectures and 2 hours of lab each week. We deployed our study in the lab. For our CSCE155 course, there were three lab sections, each with about 15 students.

4.1 Experiment Setup In our study, we utilized two sections. Section one is the “control” section where Jigsaw was used and students were allowed to move around in the lab to have face-to-face discussions during the Jigsaw phases. Section two is the “treatment” section where IMINDS+Jigsaw was used and students could only interact through I-MINDS. Students were monitored by the lab instructor, as well as two of the co-authors of this paper, to forbid them from interacting directly (face-to-face) during the lab. For each lab, the students were given a lab handout with a list of activities—thus, a lab is a task and its activities are the subtasks. For each activity, the students were required to answer some questions. At the end of the lab, each student was required to take a 10-minute post-test individually. This 10-minute post-test score is our measure of student performance in terms of understanding the topic of the lab. We ran the above study for three lab sessions: (1) debugging and testing, (2) Unified Modeling Language (UML), and (3) recursion. Questionnaire results and post-test scores of each lab were used to assign the buddy main groups and focus groups in the subsequent lab. In the “Control” section, we did this through manual computation and assignment. In the “Treatment” section, the instructor agent of I-MINDS+Jigsaw automatically carried out this task. Students were assigned to groups such that each group has a balanced mixture of high- and low-performing students.

4.2 Analysis Table 1 shows the post-test scores of the three sessions for the “Control” (i.e., Jigsaw without I-MINDS) and “Treatment” (i.e., Jigsaw with I-MINDS) sections. The results indicate that students using I-MINDS for the Jigsaw activities were able to obtain comparable and even better post-test scores. Students in the Treatment section performed better than the students in the Control section in the later two sessions. Further, students in the Treatment section also achieved better standard deviation—meaning that these students’ post-test scores were more tightly clustered than those of the Control section. This is very encouraging: without face-to-face interactions, students carried out their I-MINDS-supported Jigsaw tasks and performed relatively well in individual post-tests. However, we will need more data and more in-depth analysis to validate the significance of this finding. Post-Test Scores (max. 10)

Jigsaw without I-MINDS Jigsaw with I-MINDS

Session 1 Average Std. Dev. 7.06 1.83 6.10 1.79

Session 2 Average Std. Dev. 5.00 2.41 7.63 1.72

Session 3 Average Std. Dev. 8.83 2.85 9.00 1.50

Table 1. Student performance for the Control (Jigsaw without I-MINDS) and Treatment (Jigsaw with IMINDS) sections.

433

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

Table 2 compares the Control section and the Treatment section in terms of self-efficacy, peer rating, and team-based efficacy. • The self-efficacy questionnaire was conducted at the beginning of each lab session. It is a self-report of how a student feels about his or her ability about understanding topic and carrying out the activities of the lab. In general, students in the Treatment section had a slightly lower self-efficacy. This is interesting as these were the same students who scored better than the students from the Control section in Sessions 2 and 3. • The peer rating average is the average peer rating scores that each student gave to his or her group members. As evidenced in the scores, students in the Control section rated their peers better than the students in the Treatment section. This is possibly due to the face-to-face interaction. After all, students interacting through I-MINDS could not enjoy the advantages of face-to-face interactions such as facial expressions, the spontaneous free-flowing of ideas, and more immediate feedback in their discussions. Also, with students sitting nearby in a group, it was more engaging. This observation shows that to rival the more intuitive interactions in the face-to-face interactions, I-MINDS would need improvements in its graphical user interface and multimedia features. • The team-based efficacy was collected after each lab based on a set of questions. It measures how a student views how well its group has performed. Once again, students in the Control section approved of their team-based activities more than the students in the Treatment section. Session 1 Average Std. Dev. Self Efficacy (max. 45) Jigsaw without I-MINDS Jigsaw with I-MINDS Peer Rating (max. 45) Jigsaw without I-MINDS Jigsaw with I-MINDS Team-Based Efficacy (max. 35) Jigsaw without I-MINDS Jigsaw with I-MINDS

Session 2 Average Std. Dev.

Session 3 Average Std. Dev.

33.87 32.36

4.41 4.00

35.00 33.40

3.30 5.35

34.83 33.77

2.28 3.80

42.02 35.60

2.68 5.97

39.06 33.30

7.05 8.65

39.90 38.59

4.80 6.34

29.87 27.72

2.51 5.08

31.86 29.20

3.48 2.78

30.08 28.25

3.02 4.02

Table 2. Questionnaire Results: Self-Efficacy is pre-Jigsaw activities; Peer Rating and Team-Based Efficacy are post-Jigsaw activities.

Table 3 presents a sample of the individual students’ rating of their peers in the Treatment section, averaged over the three lab sessions. Computing correlations of each evaluation vs. the post-test scores, we observe that only two evaluations are highly correlated with the post-test scores. First, average peer rating that a student gave to his or her peers is negatively correlated to his or her post test score with a correlation of -0.63. We suspect that when a student is critical of his or her peers, the student is more aware of the different levels of abilities and is more confident of him- or herself. This means that the student is also more likely to do well in the post-test. Second, though the number of messages that a student sent to his or her group is not correlated to the student’s post-test score (only 0.03), the length of messages sent is (0.51). This provides a valuable insight to how we should measure student contribution to a group in the future: students who do well in the post-tests tend to send long messages. Table 4 shows a sample of the group members of a student rated the student or the student’s group. Once again, the average length of messages sent within the group is highly correlated with the student’s post-test score (0.62). This correlation is actually higher than that found in Table 3, hinting that how the group behaves could help improve the student’s

434

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

post-test score. Further, how the group rated the peers only slightly correlates with the post-test score (-0.24). In addition, the correlations of other evaluations vs. the post-test scores are greater than their counterparts in Table 3. This gives positive hint that group activities could help with a student’s performance, and our questionnaires and the messages tracked could be used as indicators. This encourages us to extend our analysis and tests in this direction for our future work. Table 5 shows the average number of messages sent during each Jigsaw Phase. The Reporting and Reshaping (Phase 3) phase yielded the most messages sent while the Integration and Evaluation (Phase 4) phase the least. The drop-off in the number of messages sent during Phase 4 makes sense as students tend to relax after the intensive Phase 2 and Phase 3 discussions. Student ID

s6 s10 s12

Average Peer Rating (max. 45)

34.8 41.4 38.5

Average TeamBased Rating (max. 35) 28.0 17.0 28.5

Average % of Work Done by Team 37.5% 5.0% 55.0%

Average % of Work Done by Student 62.5% 95.0% 45.0%

Average Number of Messages Sent

31.7 19.7 9.0

Average Length of Messages Sent (chars) 94.5 28.8 22.2

Student’s Average Post-Test Scores

8.0 7.0 7.0

Table 3. A sample of individual student’s perception or evaluation of his or her group and the activities within the group. All numbers are based on a student’s viewpoint of his or her group members or a student’s actions to his or group members (except for the last column). Student ID

s6 s10 s12

Group’s Peer Rating of Student (max. 45) 39.4 36.7 37.6

Group’s TeamBased Rating (max. 35) 27.6 29.6 26.3

Group’s view of %Work Done by Team 46.1% 31.6% 50.8%

Group’s view of %Work Done individually 53.9% 68.4% 49.2%

Group’s Average Total # Messages Sent 43.2 34.0 40.5

Group’s Average Length of Messages Sent (chars) 65.6 78.4 56.8

Student’s Average Evaluation

8.0 7.0 7.0

Table 4. A sample of group members’ perception or evaluation of a student’s group and the activities within the group. All numbers are based on the average group’s viewpoint or the aggregate group’s actions (except of the last column). Average Number of Messages

Jigsaw Phase 2 113

Jigsaw Phase 3 160.33

Jigsaw Phase 4 62.33

Table 5: Average number of messages for each Jigsaw phase.

5. Conclusions and Future Work We have described our extension of I-MINDS, a computer-supported collaborative Learning system, to support structured cooperative learning. Specifically, we have added features to support the Jigsaw procedure. We have deployed the I-MINDS+Jigsaw system three lab sessions of CS1. Initial results are encouraging. We have found out that students, without the benefit of face-to-face interactions, were able to make use of I-MINDS+Jigsaw, and performed as well as students with face-to-face interactions in the post-test of each lab. However, we have also observed that students rated their team activities more highly in face-to-face interactions. Further, we have also found significant correlations between the length of messages sent and a student’s performance. There are also indications that when a student is critical of his or her peers, this student’s post-test score is likely to be higher. Also, there seems to be hints that group activities do help improve students’ post-test scores. We will continue to enhance I-MINDS to fully incorporate the Jigsaw procedure, and improve the intelligence of the group agents. Our future work also includes further

L.-K. Soh et al. / Computer-Supported Structured Cooperative Learning

435

analysis, tying student performance to classroom achievement and individual group performance.

6. Acknowledgement This work was supported in part by the CSE Department and an NCITE Seed Grant at the University of Nebraska, Lincoln, NE, USA. References [1] Aronson, E., N. Blaney, J. Sikes, C. Stephan, and M. Snapp (1978). The Jigsaw Classroom, Beverly Hills, CA: Sage. [2] Betbeder, M.-L., P. Tchounikine, and A. Laënnec (2003). Symba: A Framework to Support Collective Activities in an Educational Context. Proc. ICCE’03, Hong Kong, China, pp. 189-196. [3] Chan, S. C.-F., C. W.-K. Leung, and V. T.-Y. Ng (2003). GroupUML: A PDA-Based Graphical Editor to Support Real-Time Collaboration in Student Group Projects. Proc. ICCE’03, Hong Kong, China, pp. 221223. [4] Chang, C.-C. (2003). Implications and Issues of Building a Distributed Web-Based Learning Community. Proc. ICCE’03, Hong Kong, China, pp. 224-233. [5] Clarke, J. (1994). Pieces of the Puzzle: The Jigsaw Method. In S. Sharan (ed.) Handbook of Cooperative Learning Methods, Westport, CT: Greenwood Press. [6] Johnson, D. W., R. T. Johnson, and K. A. Smith (1991). Cooperative Learning: Increasing College Faculty Instructional Productivity, ASHE-ERIC Higher Education Report No. 4, George Washington University. [7] Liu, X., X. Zhang, L.-K. Soh, J. Al-Jaroodi, and H. Jiang (2003). A Distributed, Multiagent Infrastructure for Real-Time, Virtual Classrooms. Proc. ICCE’2003, Hong Kong, China. [8] Liu, X., X. Zhang, J. Al-Jaroodi, P. Vemuri, H. Jiang, and L.-K. Soh (2003). I-MINDS: An Application of Multiagent System Intelligence to On-Line Education. Proc. IEEE Int. Conf. Systems, Man, & Cybernetics, Washington, D.C., pp. 4864-4871. [9] Salcedo, R. M., Y. Yano, Y. Miyoshi, and H. Ogata (2003). Collaborative Spaces in a Distributed Digital Library. Proc. ICCE’03, Hong Kong, China, pp. 136-129. [10] Soh, L-K. (2004). On Cooperative Learning Teams for Multiagent Team Formation, in Technical Report WS-04-06 of the AAAI’s 2004 Workshop on Forming and Maintaining Coalitions and Teams in Adaptive Multiagent Systems, San Jose, CA, pp. 37-44. [11] Sridharan, B., Kinshuk, and H. Hong (2003). Agent Based System to Capture, Discover and Retrieve Knowledge from Synchronous and Asynchronous Modes of Learning. Proc. ICCE’03, Hong Kong, China, pp. 759-760. [12] Weiss, G. (ed.) (1999). Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence, MIT Press.

436

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Sustaining Online Collaborative Problem Solving with Math Proposals Gerry STAHL Drexel University Abstract. Learning takes place over long periods of time that are hard to study directly. Even the learning experience involved in solving a challenging math problem in a collaborative online setting can be spread across hundreds of utterances during an hour or more. Such long-term interactions are constructed out of utterance-level interactions, such as the strategic proposing of a next step. This paper identifies a pattern of exchange of utterances that it terms math proposal adjacency pair, and describes its characteristics. Drawing on the methodology of conversation analysis, the paper adapts this approach to mathematical problemsolving communication and to the computer-mediated circumstances of online chat. In particular, a failed proposal is contrasted with successful proposals in the log of an actual chat. Math proposal adjacency pairs constitute the collaborative group as a working group, give direction to their problem solving and help to sustain their interaction.

1. Doing mathematics together online Computers offer many opportunities for innovation in education. One of the major avenues is by supporting the building of collaborative knowledge [1]. For instance, it is now possible for students around the world to work together on challenging math problems. Through online discussion, they can share problem-solving experiences and gain fluency in communicating mathematically. In a research project at the Math Forum @ Drexel (http://mathforum.org), we have begun to invite middle school students to participate in online chats about interesting problems in beginning algebra and geometry. The following problem, discussed in the example in this paper, is typical: If two equilateral triangles have edge-lengths of 9 cubits and 12 cubits, what is the edge-length of the equilateral triangle whose area is equal to the sum of the areas of the other two? We rely on a variety of methods from the learning sciences to guide our research and to analyze the results of our trials. In particular, we use conversation analysis [2; 3; 4; 5; 6] to interpret the interactions that take place in the student chats. In this paper, we adapt the findings of conversation analysis to math chats and develop a specific form of adjacency pairs that seem to be important for math chats. Before presenting this, it may be useful to describe briefly how the notion of adjacency pairs differs from naïve conceptions of conversation. There is a widespread common-sense or folk-theory [7; 8] view of conversation as the exchange of propositions [9]. This view was refined and formalized by logicians and cognitive scientists as involving verbal expression in meaningful statements by individuals, based on their internal mental representations. Speech served to transfer meanings from the mind of a speaker to the mind of a listener, who then interpreted the

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

437

expressed message. Following Wittgenstein [10] in critiquing this view, speech act theory [11; 12] argued that the utterances spoken by individuals were ways of acting in the world, and were meaningful in terms of what they accomplished through their use and effects. Of course, the expression, transmission and interpretation of meaning by individuals can be problematic, and people frequently have to do some interactional work in order to reestablish a shared understanding. The construction of common ground has been seen as the attempt to coordinate agreement between individual understandings [13]. Conversation analysis takes a different view of conversation. It looks at how interactional mechanisms, like the use of adjacency pairs [14; 15], co-construct intersubjectivity. Adjacency pairs are common sequences of utterances by different people— such as mutual greetings or question/answer interchanges—that form a meaningful speech act spanning multiple utterances that cannot be attributed to an individual or to the expression of mental states. We are interested in what kinds of adjacency pairs are typical for math chats. Online math chats differ from ordinary informal conversation in a number of ways. They are focused on the task of solving a specific problem and they take place within a somewhat formal institutional setting. They involve the doing of mathematics [16]. And, of course, they are computer-mediated rather than being face-to-face. The approach of conversation analysis is based on ethnomethodology [17], which involves the study of the methods that people use to accomplish what they are doing. So, we are interested in working out the methods that are used by students in online math chats. In this paper we discuss a particular method of collaboration in math chats that we have elsewhere called exploratory participation: participants engage each other in the conjoint discovery and production of both the problem and possible solutions [18]. The medium of online chat has its own peculiarities. Most importantly, it is a textbased medium, where interaction takes place by the sequential response of brief texts to each other [19; 20]. As a quasi-synchronous medium [21], chat causes confusion because several people can be typing at once and their texts can appear in an order that obscures what they are responding to. Furthermore, under time pressure to submit their texts so that they will appear near what they are responding to, some chat participants break their messages into several short texts. Because of these peculiarities of chat, it is necessary for researchers to carefully reconstruct the intended threading of texts that respond to each other before attempting to interpret the flow of interaction [22; 23]. 2. Math proposal adjacency pairs In order to begin to analyze the methods that students use in math chats, we take a close look at an excerpt from an actual chat. Figure 1 shows an excerpt from near the beginning of the log of one of our first online collaborative math problem-solving sessions. Three students—named Avr, Sup and Pin—have just entered the chat room, said hello to each other and read the problem involving three triangles. The first thing to notice here is a pattern of proposals, discussions and acceptances similar to what takes place in face-to-face discourse. Proposals about steps in solving the math problem are made by Avr in lines 1, 3, 8, 17 and by Pin in lines 20, 27. These proposals are each affirmed by someone else in lines 2, 6, 10, 19, 22, 28, respectively. To avoid chat confusion, note that line 21 responds to line 19, while line 22 responds to line 20. The timestamps show that lines 20 and 21 effectively overlapped each other chronologically: Avr was typing line 21 before she saw line 20. Similarly, lines 24 and the following were responses to line 20, not line 23. We will correct for these confusions in Figure 2, which reproduces a key passage in this excerpt.

438

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

1. Avr (8:21:46 PM): Okay, I think we should start with the formula for the area of a triangle 2. Sup (8:22:17 PM): ok 3. Avr (8:22:28 PM): A = 1/2bh 4. Avr (8:22:31 PM): I believe 5. pin (8:22:35 PM): yes 6. pin (8:22:37 PM): i concue 7. pin (8:22:39 PM): concur* 8. Avr (8:22:42 PM): then find the area of each triangle 9. Avr (8:22:54 PM): oh, wait 10. Sup (8:23:03 PM): the base and heigth are 9 and 12 right? 11. Avr (8:23:11 PM): no 12. Sup (8:23:16 PM): o 13. Avr (8:23:16 PM): that's two separate triangles 14. Sup (8:23:19 PM): ooo 15. Sup (8:23:20 PM): ok 16. Avr (8:23:21 PM): right 17. Avr (8:23:27 PM): i think we have to figure out the height by ourselves 18. Avr (8:23:29 PM): if possible 19. pin (8:24:05 PM): i know how 20. pin (8:24:09 PM): draw the altitude' 21. Avr (8:24:09 PM): how? 22. Avr (8:24:15 PM): right 23. Sup (8:24:19 PM): proportions? 24. Avr (8:24:19 PM): this is frustrating 25. Avr (8:24:22 PM): I don't have enough paper 26. pin (8:24:43 PM): i think i got it 27. pin (8:24:54 PM): its a 30/60/90 triangle 28. Avr (8:25:06 PM): I see 29. pin (8:25:12 PM): so whats the formula Figure 1. Excerpt of 3½ minutes from a one-hour chat log. Three students chat about a geometry problem. Line numbers have been added and screen-names anonymized; otherwise the transcript is identical to what the participants saw on their screens.

In Figure 1, we see several examples of a three step pattern: 1. A proposal is made by an individual for the group to work on: “I think we should ….” 2. An acceptance is made on behalf of the group: “Ok,” “right” 3. There is an elaboration of the proposal by members of the group. The proposed work is begun, often with a secondary proposal for the first sub-step. This suggests that collaborative problem-solving of mathematics may often involve a particular form of adjacency pair. We will call this a math proposal adjacency pair. Many adjacency pairs allow for insertion of other pairs between the two parts of the original pair, delaying completion of the pair. For instance, a question/answer pair may be interrupted by utterances seeking clarification of the question; the clarification interaction may itself consist of question/answer pairs, possibly with their own clarifications—this may continue recursively. With math proposal adjacency pairs, the subsidiary pairs seem to come after the completion of the original pair, in the form of secondary proposals, questions or explanations that start to do the work that was proposed in the original pair.

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

439

Proposals seem to lead to some kind of further mathematical work as a response to carrying out what was proposed. Often—as seen in the current example—that work consists of making further proposals. It is striking that the proposed work is not begun until there is agreement with the proposal. This may represent consent by the group as a whole to pursue the proposed line of work. Of course, it is not so clear in the current example, where there are only three participants and the interaction often seems to take place primarily between pairs of participants. As confirmed by other chat examples, however, the proposal generally seems to be addressed to the whole group and opens the floor for participants other than the proposer to respond. The use of “we” in “we should” or “we have to” (stated or implied) constitutes the multiple participants as a plural subject, an effective unified group [24]. Any one other than the proposer may respond on behalf of the group. Moreover, there seems to be what in conversation analysis is called an interactional preference [25] for acceptance of the proposal. That is, if one accepts a proposal, it suffices to briefly indicate agreement: “ok.” If one wants to reject a proposal, then one has to account for this response by giving reasons. We would like to characterize in more detail the method of making math proposal adjacency pairs. Often, the nature of an interactional method is seen most clearly when it is breached [17]. Methods are generally taken for granted by people; they are not made visible or conducted consciously. It is only when there is a breakdown [26] in the smooth, tacit performance of a method that people focus on its characteristics in order to overcome the breakdown. The normally transparent method becomes visible in its breach. We can interpret Sup’s posting in line 23 as a failed proposal. Given the mathematics of the triangle problem, a proposal like Sup’s related to proportionality might have been fruitful. However, in this chat, line 23 was effectively ignored by the group. While its character as a failed proposal did not become visible to the participants, it can become clear to us by comparing it to successful proposals in the same chat and by reflecting on its situation in the chat in order to ask why it was not a successful proposal. 3. A failed proposal Let us look at line 23 in its immediate interactional context in Figure 2. We can distinguish a number of ways in which it differed from successful math proposals that solicited responses and formed math proposal adjacency pairs. 17, 18. Avr (8:23: 29 PM): i think we have to figure out the height by ourselves … if possible 19. pin (8:24:05 PM): i know how 21. Avr (8:24:09 PM): how? 20. pin (8:24:09 PM): draw the altitude' 22. Avr (8:24:15 PM): right 24. Avr (8:24:19 PM): this is frustrating 23. Sup (8:24:19 PM): proportions? Figure 2. Part of the chat log excerpt in figure 1, with order revised for threading.

(a) All the other proposals (1, 3, 8, 17, 20, 27) were stated in relatively complete sentences. Additionally, some of the proposals were introduced with a phrase to indicate that they were the speaker’s proposal (1. “I think we should …,” 17. “I think we have to …,” 20. “i know how …” and 27. “i think i got it …”). The exceptions to these were simply continuations of previous proposals: line 3 provided the formula proposed in line 1

440

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

and line 8 proposed to “then” use that formula. Line 23, by contrast, provided a single word with a question mark. There was no syntactic context (other than the question mark) within the line for interpreting that word and there was no reference to semantic context outside of the line. Line 23 did not respond in any clear way to a previous line and did not provide any alternative reference to a context in the original problem statement or elsewhere. For instance, Sup could have said, “I think we should compute the proportion of the height to the base of those equilateral triangles.” (b) The timing of line 23 was particularly unfortunate. It exactly overlapped a line from Avr. Since Avr had been setting the pace for group problem solving during this part of the chat, the fact that she was involved in following a different line of inquiry spelled death for an alternative proposal at the time of line 23. Pin either seemed to be continuing on his own thread without acknowledging anyone else at this point, or else he was responding too late to previous postings. So a part of the problem for Sup was that there was little sense of a coherent group process—and what sense there was did not include him. If he was acting as part of the group process, for instance posing a question in reaction to Pin and in parallel to Avr, he was not doing a good job of it and so his contribution was ignored in the group process. It is true that a possible advantage of textbased interaction like chat over face-to-face interaction is that there may be a broader time window for responding to previous contributions. In face-to-face conversation, turn-taking rules may define appropriate turns for response that expire in a fraction of a second as the conversation moves on. In computer-based chat, the turn-taking sequence is more open. However, even here if one is responding to a posting that is several lines away, it is important to make explicit somehow to what one is responding. He could have said, “I know another way to find the height – using proportions.” Sup’s posting does not do anything like that; it relies purely upon sequential timing to establish its context, and that fails in this case. (c) Sup’s posting 23 came right after Pin’s proposal 20: “draw the altitude.” Avr had responded to this with 22 (“right”) but Pin seems to have ignored that. Pin’s proposal had opened up work to be done and both Avr and Pin responded after line 23 with contributions to this work. So Sup’s proposal came in the middle of an ongoing line of work without relating to it. In conversational terms, he made a proposal when it was not time to make a proposal. It is like trying to take a conversational turn when there is not a pause that creates a turn-taking opportunity. Now, it is possible—especially in chat—to introduce a new proposal at any time. However, to do so effectively, one must make a special effort to bring the on-going work to a temporary halt and to present one’s new proposal as an alternative. Simply saying “proportions?” will not do it. Sup could have said, “Instead of drawing the altitude, let’s use propostions to find it.” (d) To get a response to a proposal, one must elicit at least an affirmation or recognition. Line 23 does not really solicit a response. For instance, Avr’s question, 21: “how?” called for an answer—that was given by Pin in line 20, which actually appeared in the chat window just prior to the question and with the same time stamp. But Sup’s posting does not call for a specific kind of answer. Even Sup’s own previous proposal in line 10 ended with “right?”—requiring agreement or disagreement. Line 10 elicited a clear response from Avr, line 11(“no”) followed by an exchange explaining why Sup’s proposal was not right. (e) Other proposals in the excerpt are successful in contributing to the collaborative knowledge building or group problem solving in that they open up a realm of work to be done. One can look at Avr’s successive proposals on lines 1, 3, 8 and 17 as laying out a work strategy. This elicits a response from Sup trying to find values to substitute into the formula and from Pin trying to draw a graphical construction that will provide the values for the formula. Sup’s proposal in line 23, however, neither calls for a response nor opens

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

441

up a line of work. There is no request for a reaction from the rest of the group and the proposal is simply ignored. Since no one responded to Sup, he could have continued by doing some work on the proposal himself. He could have come back and made the proposal more explicit, reformulated it more strongly, taken a first step in working on it, or posed a specific question related to it. But he did not—at least not until much later—and the matter was lost. (f) Another serious hurdle for Sup was his status in the group at this time. In lines 10 through 16, Sup had made a contribution that was taken as an indication that he did not have a strong grasp of the math problem. He offered the lengths of the two given triangles as the base and height of a single triangle (line 10). Avr immediately and flatly stated that he was wrong (line 11) and then proceeded to explain why he was wrong (line 13). When he agreed (line 15), Avr summarily dismissed him (line 16) and went on to make a new proposal that implied his approach was all wrong (lines 17 and 18). Then Pin, who had stayed out of the interchange, re-entered, claiming to know how to implement Avr’s alternative proposal (lines19 and 20) and Avr confirmed that (line 22). Sup’s legitimacy as a source of useful proposals had been totally destroyed at precisely the point just before he made his ineffective proposal. Less than two minutes later, Sup tries again to make a contribution, but realizes himself that what he says is wrong. His faulty contributions confirm repeatedly that he is a drag on the group effort. He makes several more unhelpful comments later and then drops out of the discourse for most of the remaining chat. The weaknesses of line 23 as a proposal suggest some characteristics for successful proposals: (a) a clear semantic and syntactic structure, (b) careful timing within the sequence of postings, (c) a firm interruption of any other flow of discussion, (d) the elicitation of a response, (e) the specification of work to be done and (f) a history of helpful contributions. In addition, there are other interaction characteristics and mathematical requirements. For instance, the level of mathematical background knowledge assumed in a proposal must be compatible with the expertise of the participants, and the computational methods must correspond with their training. Other characteristics will become visible in other examples of chats. At this time, the notion of math proposal adjacency pairs is just a preliminary proposal based on a single chat log excerpt. It calls for extensive conversation analysis of a corpus of logs of collaborative online math problem solving to establish whether this is a fruitful way of interpreting the data. If it turns out to be a useful approach, then it will be important to determine what interactional methods of producing such proposals are effective (or not) in fostering successful knowledge building and group cognition. An understanding of these methods can guide the design of activity structures for collaborative math. As we are collecting a corpus of chat logs, we are evolving computer support through iterative trials and analyses. 4. Designing computer support If the failure of Sup’s proposal about proportions is considered deleterious to the collaborative knowledge building around the triangles problem, then what are the implications of this for the design of educational computer-based environments? One response would be to help students like Sup formulate stronger proposals. Presumably, giving him positive experiences of interacting with students like Avr and Pin, who are more skilled in chat proposal making, would provide Sup with models and examples from which he can learn—assuming that he perseveres. Another approach to the problem would be to build functionality into the software and structures into the activity that scaffold the ability of weak proposals to survive. As

442

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

students like Sup experience success with their proposals, they may become more aware of what it takes to make a strong proposal. [16] Professional mathematicians rely heavily upon inscription: the use of specialized notation, the inclusion of explicit statements of all deductive steps and the format of the formal proof to support the discussion of math proposals—whether on an informal whiteboard, a university blackboard or in an academic journal. Everything that is to be indexed in the discussion is labeled unambiguously. To avoid ellipsis, theorems are stated explicitly, with all conditions and dependencies named. The projection of what is to be proven is encapsulated in the form of the proof, which starts with the givens and concludes with what is proven. Perhaps most importantly, proposals for how to proceed are listed in the proof itself as theorems, lemmas, etc.—organized sequentially. One could imagine a chat system supplemented with a window containing an informal list of proposals analogous to the steps of a proof. After Sup’s proposal, the list might look like Figure 3. When Sup made a proposal in the chat, he would enter a statement of it in the proof window in logical sequence. He could cross out his own proposal when he felt it had been convincingly argued against by the group. [1] Given: 2 equilateral triangles of edge-length 9 cubits and 12 cubits [2] [3] formula for a triangle: A = 1/2bh [4] Area of each triangle = ? [5] b, h = 9, 12 [6] draw the altitude [7] use proportions for ratio of altitude to base [8] [9] Find: The edge-length of the equilateral triangle whose area is equal to the sum of the areas of the other two triangles

Figure 3. A list of proposals

The idea is that important proposals that were made would be retained in a visible way and be shared by the group. Of course, there are many design questions and options for doing something like this. Above all, would students understand this functionality and would they use it? The design indicated in Figure 3 is only meant to be suggestive. Another useful tool for group mathematics would be a shared drawing area. In the chat environment used by Sup, Pin and Avr, there was no shared drawing, but a student could create a drawing and send it to the others. Pin did this twelve minutes after the part of the interaction shown in the excerpt. Before the drawing was shared, much time was lost due to confusion about references to triangles and vertices. For math problems involving geometric figures, it is clearly important to be able to share drawings easily and quickly. Again, there are many design issues, such as how to keep track of who drew what, who is allowed to erase, how to point to items in the drawing and how to capture a record of the graphical interactions in coordination with the text chatting. References [1] Stahl, G. (2006) Group cognition: Computer support for building collaborative knowledge, MIT Press, Cambridge, MA. Available at: http://www.cis.drexel.edu/faculty/gerry/mit/. [2] Sacks, H., Schegloff, E. A., & Jefferson, G. (1974) A simplest systematics for the organization of turntaking for conversation, Language, 50 (4), pp. 696-735. Available at: www.jstor.org.

G. Stahl / Sustaining Online Collaborative Problem Solving with Math Proposals

443

[3] Pomerantz, A. & Fehr, B. J. (1991) Conversation analysis: An approach to the study of social action as sense making practices. In T. A. van Dijk (Ed.) Discourse as social interaction: Discourse studies, A multidisciplinary introduction, volume 2, Sage, London, UK, pp. 64-91. [4] Psathas, G. (1995) Conversation analysis: The study of talk-in-interaction, Sage, Thousand Oaks, CA. [5] ten Have, P. (1999) Doing conversation analysis: A practical guide, Sage, Thousand Oaks, CA. [6] Sacks, H. (1992) Lectures on conversation, Blackwell, Oxford, UK. [7] Bereiter, C. (2002) Education and mind in the knowledge age, Lawrence Erlbaum Associates, Hillsdale, NJ. [8] Dennett, D. C. (1991) Consciousness explained, Little Brown and Company, Boston, MA. [9] Shannon, C. & Weaver, W. (1949) The mathematical theory of communication, University of Illinois Press, Chicago, Il. [10] Wittgenstein, L. (1953) Philosophical investigations, Macmillan, New York, NY. [11] Searle, J. (1969) Speech acts: An essay in the philosophy of language, Cambridge University Press, Cambridge, UK. [12] Austin, J. (1952) How to do things with words, Harvard University Press, Boston, MA. [13] Clark, H. & Brennan, S. (1991) Grounding in communication. In L. Resnick, J. Levine, & S. Teasley (Eds.), Perspectives on socially-shared cognition, APA, Washington, DC, pp. 127-149. [14] Duranti, A. (1998) Linguistic anthropology, Cambridge University Press, Cambridge, UK. [15] Schegloff, E. (1991) Conversation analysis and socially shared cognition. In L. Resnick, J. Levine, & S. Teasley (Eds.), Perspectives on socially shared cognition, APA, Washington, DC, pp. 150-171. [16] Livingston, E. (1986) The ethnomethodological foundations of mathematics, Routledge & Kegan Paul, London, UK. [17] Garfinkel, H. (1967) Studies in ethnomethodology, Prentice-Hall, Englewood Cliffs, NJ. [18] Zemel, A., Xhafa, F., & Stahl, G. (2005) Analyzing the organization of collaborative math problemsolving in online chats using statistics and conversation analysis, In: Proceedings of CRIWG International Workshop on Groupware, Racife, Brazil. [19] Zemel, A. (2005) Texts-in-interaction: Collaborative problem-solving in quasi-synchronous computermediated communication, In: Proceedings of International Conference of Computer-Supported Collaborative Learning (CSCL 05), Taipei, Taiwan. [20] Livingston, E. (1995) An anthropology of reading, Indiana University Press, Bloomington: IN. [21] Garcia, A. & Jacobs, J. B. (1999) The eyes of the beholder: Understanding the turn-taking system in quasi-synchronous computer-mediated communication, Research on Language and Social Interaction, 34 (4), pp. 337-367. [22] Cakir, M., Xhafa, F., Zhou, N., & Stahl, G. (2005) Thread-based analysis of patterns of collaborative interaction in chat, In: Proceedings of international conference on AI in Education (AI-Ed 2005), Amsterdam, Netherlands. [23] Strijbos, J.-W. & Stahl, G. (2005) Chat-based problem solving in small groups: Developing a multidimensional coding scheme, In: Proceedings of Eleventh Biannual Conference of the European Association for Research in Learning and Instruction (EARLI 2005), Nicosia, Cyprus. [24] Lerner, G. (1993) Collectivities in action: Establishing the relevance of conjoined participation in conversation, Text, 13 (2), pp. 213-245. [25] Schegloff, E. A., Jefferson, G., & Sacks, H. (1977) The preference for self-correction in the organization of repair in conversation, Language, 53 (2), pp. 361-382. Available at: www.jstor.org. [26] Heidegger, M. (1927/1996) Being and time: A translation of Sein und Zeit, (J. Stambaugh, Trans.), SUNY Press, Albrany, NY.

444

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

CHALLENGE FRAP - A Combination Guide and Reporting Tool for Problem-based Exercises Terry M. Stewarta, William R. MacIntyreb, Victor J. Galeac a

b

Institute of Natural Resources, Massey University, New Zealand Department of Technology, Science and Mathematics Education, Massey University, New Zealand c School of Agronomy and Horticulture, University of Queensland, Australia [email protected] Abstract. A freeware editing tool, CHALLENGE FRAP, has been developed for use as both a guide and a reporting device for students undertaking an investigative, problem-solving task. This paper describes the tool, and presents two case studies from different subject domains, showing its use. Keywords. Problem-based learning, scenario-based learning, e-learning tool, constructivist tool, recording investigations, guidance tool, reporting tool, educational software

Introduction One of the basic fundamentals of constructivism is that students learn best when they actively construct their own knowledge and understanding [5]. A common constructivist exercise which facilitates problem-based learning is to set students an investigative task, designed to solve a complex but authentic problem [1], [4]. Students undertake this task, perhaps in groups, and report back to the tutor at several stages through the task or at completion. Reporting can take many forms from seminars through to written documents. One thing a tutor looks for when assessing a student’s effort on such a task, is a sensible investigative pathway. The activities carried out need to be specified, as do any results and reflections. To assist with this, the tutor may elect to set an example investigative pathway for the student to follow, to get them started or at least to indicate some of the common tasks expected of them. CHALLENGE FRAP (Form for the Recording of the Analysis of Problems) is a freeware program which provides the ability to both guide the students in an investigative exercise, and record their observations, reflections and conclusions. Input to the program can be saved as a data file known as a FRAP form. This electronic form can be treated like a living document, to be shared between group members, and sent to the tutor at various stages during the exercise for comment. A special part of the form can carry date-stamped comments from the tutor or student for feedback and discussion. Furthermore, this electronic document may initially take the form of a template, with the tutor having already suggested various actions and the content required, which the students can add or subtract to, or paste over respectively.

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

445

The software was developed as a derivative of the CHALLENGE scenario-based authoring tool [6]. However, unlike the latter, CHALLENGE FRAP is not designed to author and display a problem-based scenario as a “game”, but rather to document the solution to a real problem, while also providing guidance as to the process. This paper describes the software and reports on two examples where CHALLENGE FRAP has been used to assist with investigative exercises. 1. Description The open screen of CHALLENGE FRAP gives the user the choice of either starting a new FRAP document, or loading an existing document. A new FRAP form simply starts up with single activity node at the root and a blank editing page, while an existing document may be a partially completed student record or a template developed by the tutor, to guide the student through the exercise. After the user makes the appropriate selection, the program then moves to the main screen. (Figure 1).

Figure 1. A FRAP template showing a plant diagnostic pathway.

The authoring window is split into three parts. The left hand side is reserved for a series of nodes. These nodes can be pre-existing if a tutor-supplied template is being used or can be created by the students as they work through a task. Nodes can be represented by a series of in-built icons. They can represent any entity (object, location, action or theme) the student thinks is appropriate. Nodes can be organised in a hierarchies so their relationship to one another is obvious. In essence, they show a pathway of related activities that may take place (or have taken place) during the problem-solving exercise. First-level

446

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

nodes can represent main activities while second and third level nodes can represent subactivities off these main ones. The main right pane is an edit screen, and this holds the HTML content associated with each node. If a FRAP template is supplied, as shown in Figure 1, this may hold tutorwritten information (suggestions, hints, directions) pertaining to the activity represented by the node. The student would replace this with their own content (showing results and reflections) once they have undertaken the task themselves. Pictures, text and hyperlinks either to the web or a local resource are all accepted. The top left-hand screen contains a Properties tab and a Discussion and Feedback one. The latter allows input by the student and the tutor, pertaining to particular node content. All input is sequenced and date-stamped so a clear record is kept of the feedback. FRAP files can be exchanged between teacher and student or other members of a student team for additions and comments during the course of the investigation. 2. Case study 1 – A distance learning problem solving exercise 1.1 Description of the Project During 2004, two hundred and two students enrolled in Curriculum Integration: Science and Technology – a compulsory third year paper for teacher trainees at Massey University College of Education, New Zealand. Ninety-seven of these students studied this paper online. Three lectures were involved with teaching the course. Students worked in groups of three and were given a choice of four scenarios that represented an ill-structured, undefined problem or issue in society. The intention of this assignment was to immerse student trainees in the process of problem-based learning (PBL) by carrying out science and technology investigations. The learners negotiated their way (as active participants) through the process by: x x x x x x x

determining what information they know already, determining what information they need, determining how the information is obtained (via science investigations or technology design process or internet etc.) - often without a tutor. determining what information is relevant - often without a tutor, applying new information to the problem, presenting problems, processes and solutions to tutors/class for scrutiny and evaluating and reflecting on the process.

Each group was asked to record the ‘process’ (i.e. higher order thinking, discussions, questioning, key decisions made etc) involved in PBL as well as the science and technology investigations carried out. Each group had a meeting at the end of the second week / beginning of the third week with their tutor. This formal checkpoint was for formative assessment purposes - to ensure that each group was on the right path and had the necessary skills and time to complete the assessed aspects of the project. 1.2 Student Assessment The assessed components and sub-components of the PBL activity were: x x

Science: Overall Investigation Science: Exploring the situation

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

x x x x x x x x x

447

Science: Understanding / knowledge Science: links to SiNZC document Technology: Societal knowledge Technology: Knowledge / understanding Technology: Technical capability Technology: links to Technology in the New Zealand Curriculum (TiNZC) Information Technology - presentation Thinking Skills in PBL Process Self reflection / Evaluation skills in PBL

Each component had five progressive statements (indicators) of the expected standard of work, which corresponded to marks from 1 to 5. A ‘digital product’ of the collaborative work was to be submitted at the end of a five week block. The students were provided with various templates – WORD, PowerPoint and CHALLENGE FRAP. Tutors were available throughout the project as mentors and became involved in coaching groups early in their projects. Those groups working on-line submitted partially completed ‘digital products’ for comment and peer review as they worked their way towards a solution. Ten of the seventy groups choose to present their ‘digital product’ as a CHALLENGE FRAP file. Six of the ten groups were students involved in on-line learning. Figure 2 demonstrates the starting point for a group (involving two students), as well as the lecturer’s comment in the ‘Lecturer/Tutor Comment’ box.

Figure 2. Screen-shot showing part of a student’s FRAP ‘digital product’

448

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

1.3 Evaluation of CHALLENGE FRAP Student evaluations of CHALLENGE FRAP were collected after the submission of their final FRAP file. Only 8 students replied to the two given questions out of a total number of 28, but their results are included here for completeness. In answer to the question, “How did you find Challenge FRAP as a recording/feedback template? (1 = not useful at all to 5= extremely useful)”, three students responded with a 5, two with a 4 and two with a 3. Students were also asked an open-ended question regarding the usefulness of FRAP for primary school children to log their thinking on projects. The result was overwhelmingly positive. The FRAP template not only allowed the students to capture and record their PBL process, but it also provided direction for creating the final ‘digital product’. Staff assessing the submissions found the students files easy to navigate and assess. The ‘node tree’ was very useful for providing an overview of the group’s work at a quick glance. There was also anecdotal evidence that the FRAP students did not develop the ‘software anxiety’ that students using PowerPoint for the first time had as the project drew to an end with the final submission. Some deficiencies were noted by users. Hyperlinking to other nodes could have been more intuitive and the “Lecturer/Tutor Comment’ box” although very useful, was small and did not allow for the rich text available in the main edit window. 2. Case study 2 - Teaching of plant disease diagnosis. 2.1 Description of the Project In 2004, ten students enrolled in Plant Protection at the University of Queensland were asked to select a plant disease case from 14 problems submitted for consideration by a range of horticultural clients from south east Queensland. Each student was provided with brief details on their selected case along with the contact details of their client. Students were also provided with a CHALLENGE FRAP diagnostic template and had been previously exposed to some laboratory diagnostic cases. The FRAP template (seen in Figure 1) had a conventional diagnostic pathway already outlined via nodes and the node contents contained suggestions and the significance of what they might observe. The elements identified in this flowchart and their logical inter-relationship, are essential to the completion of a complete diagnostic case. Students were given access to all laboratory and glasshouse facilities required to carry out their individual tasks and were able to consult with the academic, and where necessary, receive guidance and relevant training on techniques to assist with their case. Where required, access to digital photography and photomicrography were also provided. 2.2 Student Assessment Students were invited to submit a draft of their template (assignment) to allow for constructive feedback and guidance by the academic. Students were able to use the discussion / feedback box available for each screen to raise questions or concerns about individual components of their diagnostic case. Constructive feedback on these and other issues could then be given by the academic to guide the students towards a more polished outcome. After consideration of the feedback from the academic, and if required, further investigation of the problem, students submitted a final version of their diagnostic case FRAP file. An example of a part of a submission is shown in Figure 3.

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

449

Figure 3. Screen-shot showing part of a student’s FRAP diagnostic assessment

This final submission was then assessed by the academic using a specifically designed set of assessment criteria 2.3 Evaluation of CHALLENGE FRAP by students Student attitudes towards this particular learning approach were investigated through the use of two questionnaires. The first, containing nine open-ended questions examined student attitudes towards this exercise, its conduct and resource issues was conducted by the students upon submission of the draft diagnostic case FRAP template. The second, a more searching examination of value of the learning exercise, measured the success of the template as a mechanism for case development and the overall benefit of this case study approach was a mixture of qualitative and open-ended questions. The opportunity for students to provide constructive feedback on the mechanisms used in this exercise was also given. This questionnaire was completed by nine of the ten students upon submission of the case template. The full results of the student evaluation will be published elsewhere. As this paper is primarily concerned with the use of CHALLENGE FRAP as a tool to facilitate this exercise, only the student responses relating specifically to the software will be presented here. This data can be seen in Table 1 As can be seen from the responses, the FRAP software and the Diagnostic template were viewed very favourably. A second, open question was also asked i.e. “How useful to you was the FRAP template?” The answer to this was very positive.

450

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

Table 1. Student response to the Diagnostic FRAP template Question The template provided a logical structure to this project The template provided a useful way to record my observations and thoughts during the project The structure within the template served as a model of common tasks and procedures which assisted me with my investigation The template assisted me (helped me focus) when seeking information from the client The comments and guidelines initially provided within the template were useful to me The feedback/discussion feature was useful to me The multimedia capabilities allowed me to better document the problem. The fact that the template structure could be altered to reflect my own investigation was a good feature The template was easy to use

Strongly Agree 6

Agree

Uncertain

Disagree

3

0

6

1

5

p1

0

Strongly Disagree 0

< 0.00

2

0

0

0.01

3

1

0

0

0.03

4

4

1

0

0

0.05

5

3

1

0

0

0.03

4

4

1

0

0

0.05

3

5

1

0

0

0.03

8

1

0

0

< 0.00

6

3

0

0

< 0.00

1. Based on One-Way Chi-Square

It was evident from the case study templates submitted by students that they succeeded in embracing the philosophy and approach to conducting diagnostic evaluations of plants with diseases which were previously unknown to them. As an aid to this, the FRAP template not only captured a record of their work, but also provided guidance to the diagnostic procedure and allowed teacher feedback. 3. Summary Students learn well, when the learning experience is guided and personal [3]. CHALLENGE FRAP is editing software which allows the production of an electronic report template that can both guide the students through a problem-solving task along with recording their observations, progress and reflections. The tree-structure of the activity nodes shows how tasks relate to one another, and functionality is also included which allows teacher-student asynchronous dialogue during the course of the investigation. Files can be easily passed between teacher and student and student and student facilitating feedback and teamwork. The FRAP software, as illustrated by the case studies above, support the five major teaching strategies consistent with constructivism as reported by Driscoll [2] .i.e.:

T.M. Stewart et al. / CHALLENGE FRAP - A Combination Guide and Reporting Tool

1. 2. 3. 4. 5.

451

Reasoning, critical thinking and problem-solving, Retention, Understanding and use, Cognitive flexibility, Self-regulation and mindful reflection and epistemic flexibility.

The CHALLENGE FRAP program and example templates are available free of charge from the website http://challenge.massey.ac.nz Acknowledgements The authors would like to thank Madhumita Bhattacharya and Lindsay Brears for their assistance in Case Study 1, as reported in this paper. References [1]

Boud, D., and Feletti, G. 1991. The Challenge of Problem-based Learning. London: Kogan Page.

[2]

Driscoll, M.P. 2000. Psychology of Learning for Instruction. Allyn and Bacon, Mass.

[3]

Gell-Mann, M. 1996. A commentary to R. Schank. In J. Brockman (Ed.) The third culture. Beyond the scientific revolution. New York. Touchstone Books, 167-180

[4]

Kain, D.L. 2003. Problem-Based Learning for Teachers, Grades 6-12. Boston, MA: Pearson Education.

[5]

Santrock J.W. 2001. Educational Psychology. McGraw-Hill, NY

[6]

Stewart, T.M., and Bartrum, P. 2002. CHALLENGE. An authoring tool for Problem-Based Scenarios delivered alone, or over the WWW. IEEE International Conference on Computers in Education, Auckland, N.Z. Los Alamitos, Calif. : IEEE Computer Society: 197-201

452

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Malaysian Female Pre-service Teachers Online: Exploring Their Internet Use and Attitudes WONG Su Luana* , NG Siew Fungb, TANG Sai Hongc Universiti Putra Malaysia, 43400 Serdang, Selangor, Malaysia. b Cheras Secondary School, 56000 Kuala Lumpur, Malaysia

a,c

[email protected]

Abstract. This paper discusses the Internet use and attitudes of female pre-service teachers with differing levels of Internet experience. This study was conducted at Universiti Putra Malaysia, one of the premier universities in Malaysia and is an analysis of early returns of a large scale study. The study is exploratory in essence and seeks to ascertain the comfort level of female pre-service teachers where the Internet is concerned. Keywords: Internet use, Internet attitudes, Malaysia pre-service teachers

Introduction The Internet has revolutionized the way students learn and how teachers teach in the classrooms. This new technology has created the opportunities for students to be active learners and allowed instructors to be facilitators in the classroom [1]. The use of the Internet, therefore, for teaching and learning purposes at the primary, secondary and higher education levels in Malaysia is considered pertinent. Since the Internet was introduced into education, many international studies reported females to use this new technology less extensively than males do [2]. Others reported females to use the Internet for different purposes [3] and had less positive attitudes toward the Internet compared to males [2,4]. Females seemed to prefer the Internet for interpersonal communication while males had stronger motives for information [5]. Jackson et al. [5] further added that females were more anxious while online compared to males. From the studies above, it appear that females do not share the same comfort zone as males do when they are online. The Internet environment does seem to be more masculine oriented and Woodfield [6] warned that this can lead females to feel disempowered while online. It would, therefore, be interesting to find out how female Malaysian pre-service teachers respond to the Internet especially now that Internet activities have become a central feature in most higher education curricula

1. Objectives of the Study The specific objectives of this study are to a. explore the sources of Internet use and types of Internet services preferred by female pre-service teachers; b. explore female pre-service teachers’ attitudes toward the Internet;

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

c. d.

453

determine if differences exist in Internet service preference between female pre-service teachers with differing levels of Internet experience; determine if differences exist in attitudes toward the Internet between female pre-service teachers with differing levels of Internet experience.

2. Instrumentation, Validation and Reliability The instrument developed was in the national language (Malay Language). Several items were adapted from Hills and Argyle [7] and Tsai, Lin and Tsai [4]. Items representing Internet use were measured by a five point Likert-type scale, ranging from never, rarely (average of 15 minutes per day), sometimes (average of between 15 minutes and 1 hour per day), frequently (average of between 1 and 3 hours per day) and very frequently (average of more than 3 hours per day). The participants were also required to indicate the average number of hours per week spent on the Internet. Items representing attitudes toward the Internet was measured against a five point Likert-type scale ranging from strongly disagree (5) to strongly agree (1). This meant that the lower the mean scores, the more positive participants’ attitudes were as the scores given for the items were in reversed order. Three content experts validated all the items in the questionnaire and found the items to be suitable in the Malaysian context. A double back translation was carried out on items adapted from Hills and Argyle (2003) and Tsai et al. (2001) to ensure that the items in the Malay Language were equivalent to the original version. The Cronbach’s alphas for items measuring Internet use and attitudes toward the Internet in the actual study were recorded at .81 and .78.

2. Participants The participants for this study consisted of 238 female pre-service teachers from the Faculty of Educational Studies, Universiti Putra Malaysia (UPM). These were the initial returns from a larger survey involving both female and male pre-service teachers from UPM. The mean age of the participants was 22.67 (S.D.= 3.69).

3. Results 3.1 Participants’ Demographic Background, Internet Experience and Sources of Internet Knowledge The participants reported spending an average of 3.32 hours (S.D.= 2.63) per week on the Internet. The majority of them did not have Internet access at home (n= 150) while the rest had (n= 88). Those with such access spent an average of 3.89 hours (S.D.= 2.76) surfing per week while those without such access spent an average of 2.98 hours (S.D.= 2.51) per week. Table 1 shows the various sources of learning the Internet against the number of participants. The most common source was learning through friends.

454

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

Table 1. Sources of Internet learning Sources Learning through friends Learning on their own Learning while in school Learning through courses while in university Learning from commercial computer centres Others

Number of participants 161 149 95 96 73 15

Percentage (%) 67.6 62.6 39.9 40.3 30.7 6.3

3.2 Use of Internet Services Table 2 shows that the majority of the participants used the Internet mostly for information and communication related purposes. The two least popular activities among them were online shopping and banking. It also shows that they treated the Internet as a useful learning tool and see it as a medium for communicating with friends or for academic purposes. Table 2. Types of Internet services Service Getting information for studiesi Getting information in generali E-mail for studiesi E-mail to friendss Downloading free softwares Accessing online newspaperi Online gamesl Online discussionl E-mail to familys Online bankings Online shoppings

Percentage of users 98.7 97.5 86.6 83.2 68.9 61.8 45.0 38.2 35.7 13 3.8

Mean frequency of use 2.58±1.09 3.82±.97 2.94± 1.19 2.58±1.09 2.39±1.23 2.19±1.19 1.81±1.09 1.71±1.08 1.54±.82 1.24±.71 1.07±.40

i=information, s=social, l=leisure

3.3 Attitude towards the Internet Attitude towards the Internet was measured in terms of perceived usefulness, affection and perceived control by 15 items (Table 3). The data in Table 3 indicated that most of the female participants disagreed strongly that they do not use the Internet to avoid looking stupid. The next four lowest mean were items related to the usefulness of the Internet. They strongly agreed that the new technology enlarges their scope and makes society more advanced. They also strongly agreed that the Internet is useful for them to acquire information but disagreed that they felt uncomfortable using it. The highest mean score was related to perceived control. The female participants disagreed that they do not need someone to guide them on how to maximize the potential of the Internet. The items were later summed up according to the respective subscales. Table 4 presents the participants’ means scores with the standard deviations of the three subscales. The participants scored the lowest on the affection subscale (an average of 1.88 per item) followed by the perceived usefulness subscale (an average of 1.95 per item) and lastly the perceived control subscale (an average of 2.44 per item). This means that the participants showed feelings that were more positive and perceived the Internet as useful. The results, however, show that they did not perceive

455

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

themselves to be in control when using the Internet. This suggests that the female participants feel unsure of themselves while navigating through the Internet. Table 3. Attitudes toward the Internet Items I could probably teach myself most of the things I need to know about the Internetc I hesitate to use the Internet in case I look stupid*a The Internet can allow me to do more imaginative worku I need an experienced person nearby when I use the Internet*c If given the opportunity to use the Internet, I am afraid that I might damage it in some way*a The Internet makes society more advancedu The Internet makes me feel uncomfortable*a The Internet enlarges my scopeu I do not need someone to tell me the best way to use the Internetc I feel bored toward using the Internet*a The Internet makes a great contribution to human lifeu I can use the Internet independently, without the assistance of othersc When using the Internet, I am not quite confident about what I am doing*a The Internet helps me acquire relevant information I needu If I get problems using the Internet, I can usually solve them one way or the otherc * = negative items; c= confidence; u= usefulness; a= affection

Females (n=238) Mean of SD attitude 1.78 .653 1.35 .644 2.52 .992 3.00 1.147 2.20 1.019 1.56 1.74 1.46 3.66 1.84 1.71 2.23 2.26 1.78 2.26

.803 .730 .755 .866 .761 .760 .882 .820 .708 .837

Table 4. Scores of perceived usefulness, affection and perceived control Subscale

Perceived usefulness Affection

Number Items 5

5

of

Possible range

Actual range

Mean

S.D.

5-25

5-24

9.73

2.63

5-25

5-25

9.39

2.76

12.22

2.52

Perceived 5 5-25 6-19 control The lower the mean scores, the more positive the attitudes

3.4 Internet experience and preferences of Internet services A one-way between groups multivariate analysis of variance was performed to investigate whether participants with differing levels of Internet experience had any preferences for different types of Internet services. Three dependent variables (types of Internet services) were measured: social, information and leisure. The independent variable was Internet experience that was categorized into two levels; inexperienced and experienced users. The mean score of the average time spent online was used as the cutoff point to differentiate between experienced and inexperienced users. Those whose average online time per week fell below the mean score of 3.32 were categorized as inexperienced users (Group 1) and the rest were categorized as experienced users (Group 2). Preliminary assumption testing was conducted to check for normality, linearity, univariate and multivariate outliers, homogeneity of variance-covariance matrices and multicollinearity. No serious violation was noted except for the equality of variance for the

456

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

dependent variable (leisure) which was not met. The authors decided to use the most stringent alpha level, .01 for the entire analysis because of the aforesaid violation. There was a statistical significant difference between Groups 1 and 2 on the combined dependent variables: F(3,3234)= 12.21, P< .0005; Wilks’ Lambda= 0.865, partial eta squared= 0.135. When the results for the dependent variables were considered separately using the Bonferroni adjusted alpha level of .003, the mean scores for all three variables reached statistical significance (Table 5). This suggests that participants more experienced Internet users used the Internet more for social, information and leisure purposes. Table 5: Differences between Groups 1 and 2 in terms of types of Internet use Group 1 S.D.

Group 2 S.D.

F

p

2.77

21.23

.000*

Partial Eta Squared .083

13.93

3.06

30.18

.000*

.113

4.00

2.06

15.61

.000*

.062

Mean

Dependent variables Social

Mean

8.10

3.30

8.94

Information

11.89

2.67

Leisure * p< .003

3.09

1.52

4.5 Internet experience and attitude towards the Internet A one-way between groups multivariate analysis of variance was also performed to determine whether participants with differing levels of Internet experience differed in terms of their attitudes toward the Internet. The three dependent variables (attitudes) were perceived usefulness, perceived control and affection. Preliminary assumption testing was conducted to check for normality, linearity, univariate and multivariate outliers, homogeneity of variance-covariance matrices and multicollinearity. No serious violation was noted except for the presence of multivariate outliers. Thus, three cases needed to be deleted. This left 235 participants as samples. The alpha level, .01 was still used for this second MANOVA analysis. There was a statistical significant difference between Groups 1 and 2 for attitudes: F(3,231)= 8.59, P< .0005; Wilks’ Lambda= 0.900, partial eta squared= 0.100. Using the Benferrroni method, each ANOVA was tested at the adjusted alpha level of .003. The results are shown in Table 6. Table 6: Differences between Groups 1 and 2 in terms of attitudes

Dependent variables Perceived usefulness

Mean

Group 1 S.D.

Mean

Group 2 S.D.

F

p

9.96

2.21

9.30

2.47

4.65

.032

Partial Eta Squared .020

Affection

9.58

2.60

8.97

2.39

3.46

.064

.015

Perceived control * p< .003

12.93

2.36

11.38

2.39

25.08

.000*

.097

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

457

When the results for the dependent variables were considered separately, the only difference to reach statistical significance using a Bonferroni adjusted level of .003, was perceived control. An inspection of the mean scores indicated that experienced female Internet users had better control of the Internet than inexperienced female users.

5. Discussion The majority of female participants involved in this study indicated that they learned to use the Internet through several sources. Learning through friends and on their own were the two most common choices. These findings are consistent with those by Duggan et al. [8] and Wei He and Jacobson [9] who found self-taught or learning from peers to be the most common methods. This means that many of them emphasized the importance of informal over formal learning. The female participants spent an average of 3.32 hours per week when asked to indicate how long they used the Internet either at home or in campus. The poor online time among the participants in this study may be due to insufficient Internet facilities in the university concerned. The computer laboratories in the university are usually fully booked and utilized for practical sessions. They are seldom available to students to surf the Internet during their free time. Those with Internet access at home were online longer (M= 3.89 hours) than those without such access (M= 2.98 hours). This poor level of use may be due to the high Internet subscription rates in Malaysia where users pay by the minute. The longer they used the Internet, the more they had to pay. This would certainly discourage users from using the Internet for a long time. It is possible that participants who did not have Internet access at home were deterred from subscribing to any Internet services because of the high rate. When the female participants were online, a high percentage of them used the Internet for searching information that was mostly for educational and general purposes. They, however, seemed to spend longer hours online searching for general related information. They also spent more time using the e-mail for educational reasons compared to social reasons. The use of the e-mail for their studies was prevalent among the participants. The results concur with the results reported by Li, Kirkup and Hodgson [10] that female undergraduates used the Internet to satisfy the need for information; and they also used the new media mostly for communication [5]. The least used service was online shopping. One probable reason that females do not buy online is because of the lack of contact with the goods purchased in such environment [11]. In addition, online shopping is usually payable by credit cards, not many students would be able to carry out such activities. Since the introduction of online banking in this country several years ago, many banking customers still express distrust at such services because of Internet fraud. Generally, the female participants in this study had positive feelings toward the Internet and found it useful to them as undergraduate students in a higher institution of learning. These suggest that they were confident and very unlikely to be phobic to future use since they saw the benefits of the Internet to them. The relatively higher scores on the perceived control subscale suggest a lack of skill while online. The findings of this research supported previous findings that females faced difficulties using the Internet mainly when searching for information ([10,12]. It can be assumed that they tended to get lost when using the Internet because of too much irrelevant material online. The results from MANOVA revealed that those with differing levels of Internet experience differed significantly in their preferences of the Internet services. Participants with more experience used the Internet more for information, relaxation and socialization.

458

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

The results from the second MANOVA also found that there was a statistical significant difference between those with differing levels of Internet when measured in terms of their attitudes toward the Internet. Those who had more experience using the Internet were in better control while online compared to those with less experience. The experienced participants appeared more independent while online and tended to be certain of what they were looking for. 6. Conclusion This research shows that female pre-service teachers in this public university used the Internet for many reasons and had positive attitudes toward this technology. This finding proves encouraging as studies have clearly shown that the success of Internet utilization was very much related to the users’ attitudes toward the Internet [13,14,15]. The tendency of those who have more positive attitudes, to use and integrate the Internet in their teaching-learning process in the classrooms is very high. It is likely that they will integrate the new technology effectively in their classroom instruction as they are convinced of the benefits of the Internet either as a teaching tool or as a learning medium. The research also suggests that Internet experience can increase Internet use and encourage positive Internet attitudes. Experienced female Internet users tended to remain online longer than those with lesser experience. The more experienced they were, the more likely they would use it for information, entertainment and social reasons. Female pre-service teachers with longer Internet exposure had better control over the Internet.

Acknowledgments The authors would like to thank Dr. Peter Hills and Dr. Tsai Chin-Chung for granting permission to use their instrument in this study. References [1]

Anderson, D. K. and Reed, W. M. (1998). The effects of Internet instruction, prior computer experience, and learning style on teachers' Internet attitudes and knowledge. Journal of Educational Computing Research, 19 (3), 227-246.

[2]

Sherman, R.C., End, C., Kraan, E., Cole, A., Campbell, J. Birchmeier, Z. and Klausner, J. (2000). The Internet Gender Gap Among College Students: Forgotten But Not Gone? CyberPsychology & Behavior 3(5), 885-894.

[3]

Odell, P., Korgen, K., Schumacher, P. & Delucchi, M. (2000). Internet use among female and male college students. CyberPsychology & Behavior 3(5), 855-862.

[4]

Tsai, C. C., Lin, S. J. & Tsai, M. J. (2001). Developing an Internet attitude scale for high school students. Computers & Education 37, 41 - 51.

[5]

Jackson, L.A., Ervin, K.S., Gardner, P.D. & Schmitt, N. (2001). Gender and the Internet: women communicating and men searching. Sex Roles, 44, 363-379.

[6]

Woodfield, R. (2000). Women, work and computing. Cambridge, UK: Cambridge Universiti Press.

[7]

Hills, P. & Argyle, M. (2003). Uses of the Internet and their relationships with individual differences in personality. Computers in Human Behavior 19, 59-70.

[8]

Duggan, A., Hess, B., Morgan, D., Kim, S., and Wilson, K. (2001). Measuring students’ attitudes toward educational use of the Internet. Journal of Educational Computing Research, 25(3), 267-281.

S.L. Wong et al. / Malaysian Female Pre-Service Teachers Online

459

[9]

Wei He, P. & Jacobson, T.E. (1996). What are they doing with the Internet? A study of user information seeking behaviours. Internet Reference Services Quarterly 1,31-51.

[10]

Li, Na, Kirkup, G., Hodgson, B. (2001). Cross-cultural comparison of women students’ attitudes toward the Internet and usage: China and the United Kingdom. CyberPsychology & Behavior 3(4), 415-426.

[11]

Li, H., Kuo, C., & Russell, M.G. (1999). The impact of perceived channel utilities, shopping orientations and demographics on consumer’s online buying behaviour. Journal of Computer-Mediated Communication, 5. Retrieved from http://www.ascusc.org/jcmc/vol5/issue2/hairong.html

[12]

Kirkup,G. (1999). The potential of the Internet for women’s education. Paper presented in Workshop Ferauen and neue, Median, Germany.

[13]

Liaw, S. S. (2002). An Internet survey for perceptions of computers and the world wide web: Relationship, prediction, and difference. Computers in Human Behaviour, 18, 17-35.

[14]

Moon, J. W., & Kim, Y. G. (2001). Extending the TAM for a world wide web context. Information & Management, 38, 217-230.

[15]

Johnson, R. A., & Hignite, M. A. (2000). Student usage of the world wide web: A comparative study. Journal of Computer Information Systems, 93-97.

460

Towards Sustainable and Scalable Educational Innovations Informed by the Learning Sciences C.-K. Looi et al. (Eds.) IOS Press, 2005 © 2005 The authors. All rights reserved.

Automatic Extraction of Learning Object Metadata (LOM) from HTML Web Pages Wai Yuen TANG and Lam For KWOK The Department of Computer Science, City University of Hong Kong [email protected], [email protected]

Abstract: It is difficult to locate learning resources on the Internet due to the loose structure of the web. Even with the help of search engines, there are simply too many search results with poor relevancy. In order to solve this problem, learning technology standards such as ADL SCORM, Dublin Core, IMS Specification, IEEE LOM, etc are emerged to provide a standard to identify and describe learning resources. Learning technology standards provide a structured index in describing learning objects and thus, help users in identifying a learning object with higher relevancy. However, the IEEE LOM standard contains too many attributes, making authors reluctant to use the standard. This paper discusses the difficulties of adapting to learning technology standards and describes a framework that automatically extracts important information from HTML web pages and maps them with attributes of the IEEE LOM standard. Keywords: Learning Technology Standards, IEEE LOM, Automatic Extraction

1.

Introduction

With the rapid growth of the Internet, the number of electronic learning resources is increasing exponentially, making it difficult to select an appropriate one to use. Due to the loose structure of the Internet; it is also difficult to locate useful resources for learning. Even with the help of search engines, the search results are usually of low quality and relevancy. A simple keyword search in Yahoo may result in millions of web sites, leading to information explosion. A simple search engine is not enough to satisfy our needs. On the other hand, a domain specific search engine [1, 2, 3] can provide results with higher relevancy because it does not try to cover the whole web. However, it is still not easy to identify relevant learning objects because different people may have different perceptions on the same word, leading to ambiguity during searching. Therefore, we need a well-defined index for describing and organizing the learning resources so that we can easily identify the relevant learning objects. How to better describe learning resources and how to locate and make good use of them becomes a challenge. Learning technology standards and well-designed learning resource management systems are critical in helping

W.Y. Tang and L.F. Kwok / Automatic Extraction of LOM from HTML Web Pages

461

one to find useful and relevant learning resources. There are some major organizations in defining standards for learning resources such as IEEE-LTSC [4], IMS Global Learning Consortium [5], Dublin Core [6] and Advanced Distributed Learning [7], etc. The IEEE Learning Object Metadata (LOM) standard defines a learning object as any entity, digital or non-digital, that may be used for learning, education or training. It aims to facilitate the searching, evaluation, acquisition and using of learning objects, by learners, instructors or automated software processes. Metadata here is data about learning object. With the use of metadata, users can obtain relevant learning resources easily. The sharing of learning resources is more convenient and practical with the use of metadata. However, the use of learning technology standards is rare in describing learning resources. Authors and developers are reluctant to use LOM because it is hard and tedious to fill in about 80 fields in LOM. The current learning resource management systems and search engines do not support the use of learning technology standard. Moreover, some of the fields in LOM are not well defined. For example, it is quite subjective to determine whether the content of a learning object is difficult and how difficult and whether a learning object is highly interactive or moderately interactive. We, thus, are investigating in this research on how to automatically extract LOM elements from one of the most popular type of learning resources, the HTML web pages, in order to solve the above problems. With the automatic extraction of LOM attributes from HTML web pages, a lot of metadata can be obtained and generated within a short time and authors do not need to fill in too many fields in LOM. The subjective judgment problems such as deciding the level of interactivity of learning objects are handled by some heuristic rules. The automatic extraction provides a standardized structure and values for LOM elements so that authors will not be frustrated with what values they should use. In this paper, we analyze the LOM attributes and data in HTML web pages, introduce the rules regarding the mapping of LOM attributes with HTML data, propose an automatic extraction framework to extract LOM elements from HTML web pages, explain how information extracted from HTML web pages can be mapped to LOM through direct mapping and heuristic rules and discuss some preliminary results.

462

2.

W.Y. Tang and L.F. Kwok / Automatic Extraction of LOM from HTML Web Pages

Analysis of LOM Attributes and HTML Web Pages Mapping of LOM attributes with HTML data can only be done after analyzing the

nature of LOM attributes and investigating what information can be found in HTML web pages. In this section, we classify LOM attributes according to their nature (Figure 1), study the information that can be extracted from HTML web pages (Table 1) and introduce the rules regarding the mapping of LOM attributes with HTML data. LOM

Depends Upon Measures and Control from Others

Depends Upon Input from Others

Relate To Physical Properties

1. General

3. M eta-Metadata

6. Rights

8. Annotation

7. Relation

Relate to Content

1. General

1.1 Identifier

Relate to Contextual Use

4. Technical 4.4 Requirement

1.2 Title

1.1.1 Catalog

1.4 Description

1.1.2 Entry

1.5 Keyword

4.4.1.1 Type

1.6 Coverage

4.4.1.2 Name

1.7 Structure

4.4.1.3 M inimum Version

1.3 Language

4.4.1 OrComposite

1.8 Aggregation Level

2. Life Cycle

4.4.1.4 M aximum Version 4.5 Installation Remarks

5. Educational

2.1 Version

4.6 Other Platform Requirements

2.2 Status

4.7 Duration 5.1 Interactivity Type

2.3 Contribute

5.2 Learning Resource Type

2.3.1 Role 2.3.2 Entity 2.3.3 Date

5. Educational

5.3 Interactivity Level 5.4 Semantic Density

5.5 Intended End User Role

5.8 Difficulty

5.6 Context 5.7 Typical Age Range

4. Technical 9. Classification

5.9 Typical Learning Time

4.1 Format

5.10 Description

4.2 Size

9.1 Purpose

4.3 Location

9.2 Taxon Path

5.11 Language

9.2.1 Source 9.2.2 Taxon

Remarks: Elements Require Others to Fill

9.2.2.1 ID 9.2.2.2 Entry 9.3 Description 9.4 Keywords

Elements that Can be Automatically Extracted Elements Require Authors to Fill

Figure 1 – Classification of LOM Attributes

Table 1- Information Extracted Directly from HTML Web Pages Extraction Source

Extracted Information

HTTP Header

Content-Language, Content-Length, Content-Type, Last-Modified

HTML Tag

<Title>, , ,

2.1. Classification of LOM Attributes According to Their Nature We have classified LOM attributes into the following 5 categories according to their nature, which are summarized in Figure 1:

W.Y. Tang and L.F. Kwok / Automatic Extraction of LOM from HTML Web Pages

•

463

Elements that Depend upon Input from Others – This category refers to the LOM attributes that must be filled in by others but not authors, such as “Meta-Metadata” (LOM 3) and “Annotation” (LOM 8). Meta-Metadata is the information about the metadata instance itself and that is the data describing the data, requiring the system that generates the metadata to fill in. The annotation category provides comments on the educational use of the learning object requiring other users’ contribution.

•

Elements that Depend upon Measures and Control from Others – “Rights” (LOM 6) category groups the intellectual property rights and condition of use for the learning object, requiring some other measures or controls to safeguard it. “Relation” (LOM 7) defines the relationship between related learning objects in which authors may not be aware of the relation and need others’ support.

•

Elements that Relate to the Physical Properties of a Learning Object – This category refers to the LOM attributes that describe the physical properties of a learning object, such as “Contribute” (LOM 2.3) representing the authors and email address of a learning object; “Size” (LOM 4.2) representing the file size of the learning object, etc.

•

Elements that Relate to the Content of a Learning Object – This category refers to the LOM attributes that describe the Content of a learning object, such as “Title” (LOM 1.2) representing the title of the learning object; “Interactivity Level” (LOM 5.3) representing the level of interactivity of the learning object, etc.

•

Elements that Relate to the Contextual Use of a Learning Object – This category refers to the usage of a learning object that seldom exists in the content of the learning object, requiring authors to fill in, such as “Installation Remarks” (LOM 4.5) giving the description on how to install this web page; “Intended End User Role” (LOM 5.5) describing the principal users, a teacher or a student, for which this learning object was designed, etc.

2.2. Information that Can be Found in HTML Web Pages A lot of information exist in a HTML web page, for example, the title, keywords, size, content language, creation date, etc. This information can be collected either directly from the HTTP header, the HTML tag and the content, or requires some heuristic rules to obtain their values. For example, we can obtain the last modified date of the web page directly from the HTTP header “Last-Modified”, title of the web page from the HTML tag