Algorithms and Architectures for Parallel Processing: 10th International Conference, ICA3PP 2010, Busan, Korea, May 21-23, 2010. Workshops, Part II ... Computer Science and General Issues)

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Germany Madhu Sudan Microsoft Research, Cambridge, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

6082

Ching-Hsien Hsu Laurence T. Yang Jong Hyuk Park Sang-Soo Yeo (Eds.)

Algorithms and Architectures for Parallel Processing 10th International Conference, ICA3PP 2010 Busan, Korea, May 21-23, 2010 Workshops, Part II

13

Volume Editors Ching-Hsien Hsu Chung Hua University, Department of Computer Science and Information Engineering Hsinchu, 300 Taiwan, China E-mail: [email protected] Laurence T. Yang St. Francis Xavier University, Department of Computer Science Antigonish, NS, B2G 2W5, Canada E-mail: [email protected] Jong Hyuk Park Seoul National University of Technology Department of Computer Science and Engineering 172 Gongreund 2-dong, Nowon-gu, Seoul, 139-742, Korea E-mail: [email protected] Sang-Soo Yeo Mokwon University, Division of Computer Engineering Daejeon 302-729, Korea E-mail: [email protected]

Library of Congress Control Number: 2010926694 CR Subject Classification (1998): F.2, H.4, D.2, I.2, H.3, G.2 LNCS Sublibrary: SL 1 – Theoretical Computer Science and General Issues ISSN ISBN-10 ISBN-13

0302-9743 3-642-13135-2 Springer Berlin Heidelberg New York 978-3-642-13135-6 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2010 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper 06/3180

Preface

It is our great pleasure to present the proceedings of the symposia and workshops on parallel and distributed computing and applications associated with the ICA3PP 2010 conference. These symposia and workshops provide vibrant opportunities for researchers and industry practitioners to share their research experience, original research results and practical development experiences in the new challenging research areas of parallel and distributed computing technologies and applications. It was the first time that the ICA3PP conference series added symposia and workshops to its program in order to provide a wide range of topics that extend beyond the main conferences. The goal was to provide a better coverage of emerging research areas and also forums for focused and stimulating discussions. With this objective in mind, we selected three workshops to accompany the ICA3PP 2010 conference: • • •

FPDC 2010, the 2010 International Symposium on Frontiers of Parallel and Distributed Computing HPCTA 2010, the 2010 International Workshop on High-Performance Computing, Technologies and Applications M2A 2010, the 2010 International Workshop on Multicore and Multithreaded Architectures and Algorithms

Each of the symposia / workshops focused on a particular theme and complemented the spectrum of the main conference. All papers published in the workshops proceedings were selected by the Program Committee on the basis of referee reports. Each paper was reviewed by independent referees who judged the papers for originality, quality, contribution, presentation and consistency with the theme of the workshops. We deeply appreciate the tremendous efforts and contributions of the Chairs of individual symposia / workshops. Our thanks also go to all authors for their valuable contributions and to all the Program Committee members and reviewers for providing timely and in-depth reviews. Particularly, we thank the Local Arrangements Committee for exceptionally nice arrangements. We hope you will enjoy the proceedings.

May 2010

Laurence T. Yang Jong Hyuk Park J. Daniel Garcia Ching-Hsien (Robert) Hsu Alfredo Cuzzocrea Xiaojun Cao Kuo-Chan Huang Yu Liang

FPDC 2010 Foreword

We would like to welcome you to the proceedings of the 2010 International Symposium on Frontiers of Parallel and Distributed Computing (FPDC 2010) held in Busan, Korea, May 21–23, 2010. The FPDC 2010 symposium intended to bring together researchers from industry and academia, practitioners, scientists and engineers to discuss novel and innovative research activities, on-going research efforts, emerging parallel/distributed computing technologies and applications. Each paper in the FPDC 2010 symposium was reviewed by at least three Technical Program Committee members of the ICA3PP 2010 conference. After the reviewing process, 29 papers of high quality were invited from 110 submissions for presentation and publication in the FPDC symposium. The acceptance rate of the symposium is 26%. The selected papers cover various topics in parallel and distributed computing systems and technologies with focus on the following areas: -

Parallel Programming and Multicore Technologies Grid / Cluster Computing Parallel Algorithms and Architectures Bioinformatics and Application Mobile Computing and Web Services Distributed Operating Systems and P2P Computing Fault-Tolerant and Information Security

Many individuals contributed to the success of this symposium directly or indirectly. First of all, the symposium Program Co-chairs would like to thank the symposium General Chairs, Laurence T. Yang and Jong Hyuk Park, for their excellent guidance and continuous support. We are very grateful to the ICA3PP 2010 General Chair and Program Chair, Laurence T. Yang and Robert C. Hsu, who helped us in selecting papers for this symposium. Last but not least, we would like to thank all authors for accepting our invitation to publish their papers in this symposium. We hope you will enjoy the proceedings.

May 2010

Laurence T. Yang Jong Hyuk Park Ching-Hsien (Robert) Hsu Sang-Soo Yeo

HPCTA 2010 Foreword

It gives us great pleasure to introduce this collection of papers that were presented at the 2010 International Workshop on High-Performance Computing Technologies and Applications (HPCTA 2010), May 21–23, 2010, at the Busan Lotte Hotel, Busan, Korea. The Program Committee received 23 submissions, from which it selected 12 for presentation and publication. Each paper was evaluated by three referees. Technical quality, originality, relevance, and clarity were the primary criteria for selection. We wish to thank all who submitted manuscripts for consideration. We also wish to thank the members of the HPCTA 2010 Program Committee who reviewed all of the submissions.

Whey Fone Tsai Hsi-Ya Chang Ching-Hsien Hsu Kuo-Chan Huang

M2A2 2010 Foreword

It is with great pleasure that we present the proceedings of the 2010 International Workshop on Multicore and Multithreaded Architectures and Algorithms (M2A2 2010) held in conjunction with the 10th International Conference on Algorithms and Architectures for Parallel Processing (ICA3PP 2010) at Busan, Korea. In recent years, multicore systems are dominating the processor market, and it is expected that the number of cores will continue increasing in most of the commercial systems, such as high-performance, desktops, or embedded systems. This trend is driven by the need to increase the efficiency of the major system components, that is, the cores, the memory hierarchy, and the interconnection network. For this purpose, the system designer must trade off performance versus power consumption, which is a major concern in current microprocessors. Therefore new architectures or architectural mechanisms addressing this trade-off are required. In this context, load balancing and scheduling can help to improve energy savings. In addition, it remains a challenge to identify and productively program applications for these architectures with a resulting substantial performance improvement. The M2A2 2010 workshop provided a forum for engineers and scientists to address the resulting challenge and to present new ideas, applications, and experience on all aspects of multicore and multithreaded systems. This year, and because of the high quality of the submitted papers, only about 40% of papers were accepted for the conference. We would like to express our most sincere appreciation to everyone contributing to the success of this workshop. First, we thank the authors of the submitted papers for their efforts in their research work. Then, we thank the TPC members and the reviewers for their invaluable and constructive comments. Finally, we thank our sponsors for the support of this workshop.

Houcine Hassan Julio Sahuquillo

Reviewers

FPDC 2010 Jemal Abawajy Ahmad S. AI-Mogren Hüseyin Akcan Giuseppe Amato Cosimo Anglano Alagan Anpalagan Amnon Barak Novella Bartolini Alessio Bechini Ladjel Bellatreche Ateet Bhalla Taisuke Boku Angelo Brayner Massimo Cafaro Mario Cannataro Jiannong Cao Andre C.P.L.F. de Carvalho Denis Caromel Tania Cerquitelli Hangbae Chang Ruay-Shiung Chang Yue-Shan Chang Jinjun Chen Tzung-Shi Chen Zizhong Chen Allen C. Cheng Francis Chin Michele Colajanni Carmela Comito Raphaël Couturier Mieso Denko Bronis R. de Supinski Julius Dichter Der-Rong Din Susan K. Donohue Shantanu Dutt Todd Eavis

Deakin University, Australia AI Yamamah University, Saudi Arabia Izmir University of Economics, Turkey ISTI-CNR, Italy Universitá del Piemonte Orientale, Italy Ryerson University, Canada The Hebrew University of Jerusalem, Israel University of Rome "La Sapienza", Italy Alessio Bechini, University of Pisa, Italy ENSMA, France Technocrats Institute of Technology, India University of Tsukuba, Japan University of Fortaleza, Brazil University of Salento, Lecce, Italy University “Magna Græcia” of Catanzaro, Italy Hong Kong Polytechnic University, Hong Kong Universidade de Sao Paulo, Brazil University of Nice Sophia Antipolis-INRIA-CNRS-IUF, France Politecnico di Torino, Italy Daejin University, Korea National Dong Hwa University, Taiwan National Taipei University, Taiwan Swinburne University of Technology, Australia National University of Tainan, Taiwan Colorado School of Mines, USA University of Pittsburgh, USA University of Hong Kong, Hong Kong Universitá di Modena e Reggio Emilia, Italy University of Calabria, Italy University of Franche Comte, France University of Guelph, Canada Lawrence Livermore National Laboratory, USA University of Bridgeport, USA National Changhua University of Education, Taiwan The College of New Jersey, USA University of Illinois at Chicago, USA Concordia University, Canada

XIV

Reviewers

Giuditta Franco Karl Fuerlinger Jerry Zeyu Gao Jinzhu Gao Irene Garrigós Amol Ghoting Harald Gjermundrod Janice Gu Hyoil Han Houcine Hassan Pilar Herrero Michael Hobbs JoAnne Holliday Ching-Hsien Hsu Tsung-Chuan Huang Yo-Ping Huang Young-Sik Jeong Qun Jin Xiaolong Jin Soo-Kyun Kim Jongsung Kim Dan Komosny Gregor von Laszewski Changhoon Lee Deok Gyu Lee Yang Sun Lee Laurent Lefevre Casiano Rodriguez Leon Daniele Lezzi Jikai Li Keqin Li Keqin Li Keqiu Li Minglu Li Xiaofei Liao Kai Lin Jianxun Liu Pangfeng Liu Alexandros V. Gerbessiotis Yan Gu Hai Jiang George Karypis Eun Jung Kim Minseok Kwon Yannis Manolopoulos

University of Verona, Italy University of California, Berkeley, USA San Jose State University, US University of the Pacific, Stockton, CA, USA University of Alicante, Spain IBM T. J. Watson Research Center, USA University of Nicosia, Cyprus Auburn University, USA Drexel University, USA Universidad Politecnica de Valencia, Spain Universidad Politécnica de Madrid, Spain Deakin University, Australia Santa Clara University, USA Chung Hua University, Taiwan National Sun Yat-sen University, Taiwan National Taipei University of Technology, Taiwan Wonkwang University, Korea Waseda University, Japan University of Bradford, UK PaiChai University, Korea Kyungnam University, Korea Brno University of Technology, Czech Republic Rochester Institute of Technology, USA Hanshin University, Korea ETRI, Korea Chosun University, Korea INRIA, University of Lyon, France Universidad de La Laguna, Spain Barcelona Supercomputing Center, Spain The College of New Jersey, USA State University of New York, USA SAP Research, France Dalian University of Technology, China Shanghai Jiaotong University, China Huazhong University of Science and Technology, China Dalian University of Technology, China Hunan University of Science and Technology, China National Taiwan University, Taiwan New Jersey Institute of Technology, USA Auburn University, US Arkansas State University, US University of Minnesota, US Texas A&M University, US Rochester Institute of Technology, USA Aristotle University of Thessaloniki, Greece

Reviewers

Alberto Marchetti-Spaccamela Toma Margalef María J. Martín Michael May Eduard Mehofer Rodrigo Fernandes de Mello Peter M. Musial Amiya Nayak Leandro Navarro Andrea Nucita Leonardo B. Oliveira Salvatore Orlando Marion Oswald Apostolos Papadopoulos George A. Papadopoulos Deng Pan Al-Sakib Khan Pathan Dana Petcu Rubem Pereira María S. Pérez Kleanthis Psarris Pedro Pereira Rodrigues Marcel-Catalin Rosu Paul M. Ruth Giovanni Maria Sacco Lorenza Saitta Frode Eika Sandnes Claudio Sartori Erich Schikuta Martin Schulz Seetharami R. Seelam Erich Schikuta Edwin Sha Rahul Shah Giandomenico Spezzano Peter Strazdins Domenico Talia Uwe Tangen David Taniar Christopher M. Taylor Parimala Thulasiraman A. Min Tjoa Paolo Trunfio Jichiang Tsai Emmanuel Udoh

Sapienza University of Rome, Italy Universitat Autonoma de Barcelona, Spain University of A Coruña, Spain Fraunhofer Institute for Intelligent Systems, Germany University of Vienna, Austria University of Sao Paulo, Brazil University of Puerto Rico, USA University of Ottawa, Canada Polytechnic University of Catalonia, Spain University of Messina, Italy Universidade Estadual de Campinas, Brazil Ca' Foscari University of Venice, Italy Hungarian Academy of Sciences, Budapest, Hungary Aristotle University of Thessaloniki, Greece University of Cyprus, Cyprus Florida International University, US BRAC University, Bangladesh West University of Timisoara, Romania Liverpool John Moores University, UK Universidad Politecnica de Madrid, Spain The University of Texas at San Antonio, USA University of Porto, Portugal IBM, USA The University of Mississippi, USA Universitá di Torino, Italy Università del Piemonte Orientale, Italy Oslo University College, Norway University of Bologna, Italy University of Vienna, Austria Lawrence Livermore National Laboratory, USA IBM T.J. Watson Research Center, USA University of Vienna, Austria The University of Texas at Dallas, USA Louisiana State University, USA ICAR-CNR, Italy The Australian National University, Australia Università della Calabria, Italy Ruhr-Universität Bochum, Germany Monash University, Australia University of New Orleans, USA University of Manitoba, Canada Vienna University of Technology, Austria University of Calabria, Italy National Chung Hsing University, Taiwan Indiana University-Purdue University, USA

XV

XVI

Reviewers

Gennaro Della Vecchia Lizhe Wang Max Walter Cho-Li Wang Guojun Wang Xiaofang Wang Chen Wang Chuan Wu Qishi Wu Yulei Wu Fatos Xhafa Yang Xiang Chunsheng Xin Neal Naixue Xiong Zheng Yan Sang-Soo Yeo Eiko Yoneki Chao-Tung Yang Zhiwen Yu Wuu Yang Jiehan Zhou Sotirios G. Ziavras Roger Zimmermann

Gennaro Della Vecchia - ICAR-CNR, Italy Indiana University, USA Technische Universität München, Germany The University of Hong Kong, China Central South University, China Villanova University, USA CSIRO ICT Centre, Australia The University of Hong Kong, China University of Memphis, USA University of Bradford, UK University of London, UK Central Queensland University, Australia Norfolk State University, USA Georgia State University, USA Nokia Research Center, Finland Mokwon University, Korea University of Cambridge, UK Tunghai University, Taiwan Northwestern Polytechnical University, China National Chiao Tung University, Taiwan University of Oulu, Finland NJIT, USA National University of Singapore, Singapore

HPCTA 2010 Hamid R. Arabnia, USA Rajkumar Buyya, Australia Jee-Gong Chang, Taiwan Ruay-Shiung Chang, Taiwan Yue-Shan Chang, Taiwan Wenguang Chen, China Khoo Boo Cheong, Singapore Yeh-Ching Chung, Taiwan Chang-Huain Hsieh, Taiwan James J.Y. Hsu, Taiwan Suntae Hwang, Korea Hae-Duck Joshua Jeong, Korea Jyh-Chiang Jiang, Taiwan Hai Jin, China Pierre Kestener, France Chung-Ta King, Taiwan Jong-Suk Ruth Lee, Korea Ming-Hsien Lee, Taiwan Weiping Li, China Kuan-Ching Li, Taiwan

Chao-An Lin, Taiwan Fang-Pang Lin, Taiwan Pangfeng Liu, Taiwan Carlos R. Mechoso, USA Rodrigo Mello, Brazil Nikolay Mirenkov, Japan Chien-Hua Pao, Taiwan Depei Qian, China Gudula Ruenger, Germany Cherng-Yeu Shen, Taiwan Tony Wen-Hann Sheu, Taiwan Michael J. Tsai, Taiwan Cho-Li Wang, Hong Kong Jong-Sinn Wu, Taiwan Yongwei Wu, China Chao-Tung Yang, Taiwan Jaw-Yen Yang, Taiwan Chih-Min Yao, Taiwan Weimin Zheng, China Albert Y. Zomaya, Australia

Reviewers

M2A2 2010 Hideharu Amano, Japan Hamid R. Arabnia, USA Luca Benini, Italy Luis Gomes, Portugal Zonghua Gu, Hong Kong Rajiv Gupta, USA Houcine Hassan, Spain Seongsoo Hong, Korea Shih-Hao Hung, Taiwan Eugene John, USA

Seon Wook Kim, Korea Jihong Kim, Korea Chang-Gun Lee, Korea Yoshimasa Nakamura, Japan Hiroshi Nakashima, Japan Sabri Pllana, Austria Julio Sahuquillo, Spain Zili Shao, Hong Kong Kenjiro Taura, Japan Sami Yehia, France

XVII

Table of Contents – Part II

The 2010 International Symposium on Frontiers of Parallel and Distributed Computing (FPDC 2010) Parallel Programming and Multi-core Technologies Efficient Grid on the OTIS-Arrangment Network . . . . . . . . . . . . . . . . . . . . . Ahmad Awwad, Bassam Haddad, and Ahmad Kayed Single Thread Program Parallelism with Dataflow Abstracting Thread . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tianzhou Chen, Xingsheng Tang, Jianliang Ma, Lihan Ju, Guanjun Jiang, and Qingsong Shi

1

11

Parallel Programming on a Soft-Core Based Multi-core System . . . . . . . . Liang-Teh Lee, Shin-Tsung Lee, and Ching-Wei Chen

22

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors . . . . . . Yongqing Ren, Hong An, Tao Sun, Ming Cong, and Yaobin Wang

32

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eunji Lee, Sungju Lee, Yongwha Chung, Hyeonjoong Cho, and Sung Bum Pan A Paradigm for Processing Network Protocols in Parallel . . . . . . . . . . . . . Ralph Duncan, Peder Jungck, and Kenneth Ross Real-Time Task Scheduling on Heterogeneous Two-Processor Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chin-Fu Kuo and Ying-Chi Hai

42

52

68

Grid/Cluster Computing A Grid Based System for Closure Computation and Online Service . . . . . Wing-Ning Li, Donald Hayes, Jonathan Baran, Cameron Porter, and Tom Schweiger

79

A Multiple Grid Resource Broker with Monitoring and Information Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chao-Tung Yang, Wen-Jen Hu, and Bo-Han Chen

90

Design Methodologies of Workload Management through Code Migration in Distributed Desktop Computing Grids . . . . . . . . . . . . . . . . . . Makoto Yoshida and Kazumine Kojima

100

Dynamic Dependent Tasks Assignment for Grid Computing . . . . . . . . . . . Meddeber Meriem and Yagoubi Belabbas

112

XX

Table of Contents – Part II

Implementation of a Heuristic Network Bandwidth Measurement for Grid Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chao-Tung Yang, Chih-Hao Lin, and Wen-Jen Hu

121

Parallel Algorithms, Architectures and Applications An Efficient Circuit-Switched Broadcasting in Star Graph . . . . . . . . . . . . . Cheng-Ta Lee and Yeong-Sung Lin Parallel Domain Decomposition Methods for High-Order Finite Element Solutions of the Helmholtz Problem . . . . . . . . . . . . . . . . . . . . . . . . Youngjoon Cha and Seongjai Kim Self-Organizing Neural Grove and Its Distributed Performance . . . . . . . . . Hirotaka Inoue A Massively Parallel Hardware for Modular Exponentiations Using the m-ary Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marcos Santana Farias, S´ergio de Souza Raposo, Nadia Nedjah, and Luiza de Macedo Mourelle Emulation of Object-Based Storage Devices by a Virtual Machine . . . . . . Yi-Chiun Fang, Chien-Kai Tseng, and Yarsun Hsu Balanced Multi-process Parallel Algorithm for Chemical Compound Inference with Given Path Frequencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jiayi Zhou, Kun-Ming Yu, Chun Yuan Lin, Kuei-Chung Shih, and Chuan Yi Tang Harnessing Clusters for High Performance Computation of Gene Expression Microarray Comparative Analysis . . . . . . . . . . . . . . . . . . . . . . . . Philip Church, Adam Wong, Andrzej Goscinski, and Christophe Lef`evre

131

136 146

156

166

178

188

Mobile Computing/Web Services Semantic Access Control for Corporate Mobile Devices . . . . . . . . . . . . . . . Tuncay Ercan and Mehmet Yıldız A New Visual Simulation Tool for Performance Evaluation of MANET Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Md. Sabbir Rahman Sakib, Nazmus Saquib, and Al-Sakib Khan Pathan A Web Service Composition Algorithm Based on Global QoS Optimizing with MOCACO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Wang Li and He Yan-xiang

198

208

218

Table of Contents – Part II

XXI

Distributed Operating System/P2P Computing Experiences Gained from Building a Services-Based Distributed Operating System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrzej Goscinski and Michael Hobbs

225

Quick Forwarding of Queries to Relevant Peers in a Hierarchical P2P File Search System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tingting Qin, Qi Cao, Qiying Wei, and Satoshi Fujita

235

iCTPH: An Approach to Publish and Lookup CTPH Digests in Chord . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhang Jianzhong, Pan Kai, Yu Yuntao, and Xu Jingdong

244

Fault-Tolerant and Information Security Toward a Framework for Cloud Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . Michael Brock and Andrzej Goscinski

254

Cluster-Fault-Tolerant Routing in Burnt Pancake Graphs . . . . . . . . . . . . . Nagateru Iwasawa, Tatsuro Watanabe, Tatsuya Iwasaki, and Keiichi Kaneko

264

Edge-Bipancyclicity of All Conditionally Faulty Hypercubes . . . . . . . . . . . Chao-Ming Sun and Yue-Dar Jou

275

The 2010 International Workshop on High Performance Computing Technologies and Applications (HPCTA 2010) Session I Accelerating Euler Equations Numerical Solver on Graphics Processing Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pierre Kestener, Fr´ed´eric Chˆ ateau, and Romain Teyssier

281

An Improved Parallel MEMS Processing-Level Simulation Implementation Using Graphic Processing Unit . . . . . . . . . . . . . . . . . . . . . . Yupeng Guo, Xiaoguang Liu, Gang Wang, Fan Zhang, and Xin Zhao

289

Solving Burgers’ Equation Using Multithreading and GPU . . . . . . . . . . . . Sheng-Hsiu Kuo, Chih-Wei Hsieh, Reui-Kuo Lin, and Wen-Hann Sheu

297

Support for OpenMP Tasks on Cell Architecture . . . . . . . . . . . . . . . . . . . . . Qian Cao, Changjun Hu, Haohu He, Xiang Huang, and Shigang Li

308

XXII

Table of Contents – Part II

Session II A Novel Algorithm for Faults Acquiring and Locating on Fiber Optic Cable Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ning Zhang, Yan Chen, Naixue Xiong, Laurence T. Yang, Dong Liu, and Yuyuan Zhang A Parallel Distributed Algorithm for the Permutation Flow Shop Scheduling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Samia Kouki, Talel Ladhari, and Mohamed Jemni A Self-Adaptive Load Balancing Strategy for P2P Grids . . . . . . . . . . . . . . Po-Jung Huang, You-Fu Yu, Quan-Jie Chen, Tian-Liang Huang, Kuan-Chou Lai, and Kuan-Ching Li Embedding Algorithms for Star, Bubble-Sort, Rotator-Faber-Moore, and Pancake Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mihye Kim, Dongwan Kim, and Hyeongok Lee

318

328 338

348

Session III Performance Estimation of Generalized Statistical Smoothing to Inverse Halftoning Based on the MTF Function of Human Eyes . . . . . . . . . . . . . . . Yohei Saika, Kouki Sugimoto, and Ken Okamoto

358

Power Improvement Using Block-Based Loop Buffer with Innermost Loop Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ming-Yuan Zhong and Jong-Jiann Shieh

368

An Efficient Pipelined Architecture for Fast Competitive Learning . . . . . Hui-Ya Li, Chia-Lung Hung, and Wen-Jyi Hwang

381

Merging Data Records on EREW PRAM . . . . . . . . . . . . . . . . . . . . . . . . . . . Hazem M. Bahig

391

The 2010 International Workshop on Multicore and Multithreaded Architecture and Algorithms (M2A2 2010) Session I Performance Modeling of Multishift QR Algorithms for the Parallel Solution of Symmetric Tridiagonal Eigenvalue Problems . . . . . . . . . . . . . . Takafumi Miyata, Yusaku Yamamoto, and Shao-Liang Zhang

401

A Parallel Solution of Large-Scale Heat Equation Based on Distributed Memory Hierarchy System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tangpei Cheng, Qun Wang, Xiaohui Ji, and Dandan Li

413

Table of Contents – Part II

A New Metric for On-line Scheduling and Placement in Reconfigurable Computing Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maisam Mansub Bassiri and Hadi Shahriar Shahhoseini

XXIII

422

Session II Test Data Compression Using Four-Coded and Sparse Storage for Testing Embedded Core . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Zhang Ling, Kuang Ji-shun, and You zhi-qiang Extending a Multicore Multithread Simulator to Model Power-Aware Hard Real-Time Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jos´e Luis March, Julio Sahuquillo, Houcine Hassan, Salvador Petit, and Jos´e Duato

434

444

Real-Time Linux Framework for Designing Parallel Mobile Robotic Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Joan Aracil, Carlos Dom´ınguez, Houcine Hassan, and Alfons Crespo

454

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

465

Table of Contents – Part I

Keynote Papers Efficient Web Browsing with Perfect Anonymity Using Page Prefetching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shui Yu, Theerasak Thapngam, Su Wei, and Wanlei Zhou

1

InterCloud: Utility-Oriented Federation of Cloud Computing Environments for Scaling of Application Services . . . . . . . . . . . . . . . . . . . . . Rajkumar Buyya, Rajiv Ranjan, and Rodrigo N. Calheiros

13

Parallel Algorithms Scalable Co-clustering Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bongjune Kwon and Hyuk Cho

32

Parallel Pattern Matching with Swaps on a Linear Array . . . . . . . . . . . . . . Fouad B. Chedid

44

Parallel Prefix Computation in the Recursive Dual-Net . . . . . . . . . . . . . . . Yamin Li, Shietung Peng, and Wanming Chu

54

A Two-Phase Differential Synchronization Algorithm for Remote Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yonghong Sheng, Dan Xu, and Dongsheng Wang

65

A New Parallel Method of Smith-Waterman Algorithm on a Heterogeneous Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bo Chen, Yun Xu, Jiaoyun Yang, and Haitao Jiang

79

Improved Genetic Algorithm for Minimizing Periodic Preventive Maintenance Costs in Series-Parallel Systems . . . . . . . . . . . . . . . . . . . . . . . . Chung-Ho Wang and Te-Wei Lin

91

A New Hybrid Parallel Algorithm for MrBayes . . . . . . . . . . . . . . . . . . . . . . Jianfu Zhou, Gang Wang, and Xiaoguang Liu Research and Design of Deployment Framework for Blade-Based Data Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Haiping Qu, Xiuwen Wang, Lu Xu, Jiangang Zhang, and Xiaoming Han Query Optimization over Parallel Relational Data Warehouses in Distributed Environments by Simultaneous Fragmentation and Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ladjel Bellatreche, Alfredo Cuzzocrea, and Soumia Benkrid

102

113

124

XXVI

Table of Contents – Part I

Parallel Architectures Function Units Sharing between Neighbor Cores in CMP . . . . . . . . . . . . . . Tianzhou Chen, Jianliang Ma, Hui Yuan, Jingwei Liu, and Guanjun Jiang

136

A High Efficient On-Chip Interconnection Network in SIMD CMPs . . . . . Dan Wu, Kui Dai, Xuecheng Zou, Jinli Rao, and Pan Chen

149

Network-on-Chip Routing Algorithms by Breaking Cycles . . . . . . . . . . . . . Minghua Tang and Xiaola Lin

163

A Fair Thread-Aware Memory Scheduling Algorithm for Chip Multiprocessor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Danfeng Zhu, Rui Wang, Hui Wang, Depei Qian, Zhongzhi Luan, and Tianshu Chu

174

Efficient Partitioning of Static Buses for Processor Arrays of Small Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Susumu Matsumae

186

Formal Proof for a General Architecture of Hybrid Prefix/Carry-Select Adders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Feng Liu, Qingping Tan, Xiaoyu Song, and Gang Chen

193

An Efficient Non-Blocking Multithreaded Embedded System . . . . . . . . . . Joseph M. Arul, Tsung-Yun Chen, Guan-Jie Hwang, Hua-Yuan Chung, Fu-Jiun Lin, and You-Jen Lee A Remote Mirroring Architecture with Adaptively Cooperative Pipelining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yongzhi Song, Zhenhai Zhao, Bing Liu, Tingting Qin, Gang Wang, and Xiaoguang Liu

205

215

SV: Enhancing SIMD Architectures via Combined SIMD-Vector Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Libo Huang and Zhiying Wang

226

A Correlation-Aware Prefetching Strategy for Object-Based File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Julei Sui, Jiancong Tong, Gang Wang, and Xiaoguang Liu

236

An Auxiliary Storage Subsystem to Distributed Computing Systems for External Storage Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MinHwan Ok

246

Table of Contents – Part I

XXVII

Grid/Cluster Computing Checkpointing and Migration of Communication Channels in Heterogeneous Grid Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . John Mehnert-Spahn and Michael Schoettner On-Line Task Granularity Adaptation for Dynamic Grid Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nithiapidary Muthuvelu, Ian Chai, Eswaran Chikkannan, and Rajkumar Buyya

254

266

Message Clustering Technique towards Efficient Irregular Data Redistribution in Clusters and Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shih-Chang Chen, Tai-Lung Chen, and Ching-Hsien Hsu

278

Multithreading of Kostka Numbers Computation for the BonjourGrid Meta-desktop Grid Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Heithem Abbes, Franck Butelle, and Christophe C´erin

287

Adaptable Scheduling Algorithm for Grids with Resource Redeployment Capability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Cho-Chin Lin and Chih-Hsuan Hsu

299

Using MPI on PC Cluster to Compute Eigenvalues of Hermitian Toeplitz Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fazal Noor and Syed Misbahuddin

313

Cloud Computing/Virtualization Techniques idsocket: API for Inter-domain Communications Base on Xen . . . . . . . . . . Liang Zhang, Yuein Bai, and Cheng Luo Strategy-Proof Dynamic Resource Pricing of Multiple Resource Types on Federated Clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marian Mihailescu and Yong Meng Teo

324

337

Adapting Market-Oriented Scheduling Policies for Cloud Computing . . . Mohsen Amini Salehi and Rajkumar Buyya

351

A High Performance Inter-VM Network Communication Mechanism . . . . Yuebin Bai, Cheng Luo, Cong Xu, Liang Zhang, and Huiyong Zhang

363

On the Effect of Using Third-Party Clouds for Maximizing Profit . . . . . . Young Choon Lee, Chen Wang, Javid Taheri, Albert Y. Zomaya, and Bing Bing Zhou

381

A Tracing Approach to Process Migration for Virtual Machine Based on Multicore Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Liang Zhang, Yuebin Bai, and Xin Wei

391

XXVIII

Table of Contents – Part I

GPU Computing and Applications Accelerating Dock6’s Amber Scoring with Graphic Processing Unit . . . . . Hailong Yang, Bo Li, Yongjian Wang, Zhongzhi Luan, Depei Qian, and Tianshu Chu

404

Optimizing Sweep3D for Graphic Processor Unit . . . . . . . . . . . . . . . . . . . . . Chunye Gong, Jie Liu, Zhenghu Gong, Jin Qin, and Jing Xie

416

Modular Resultant Algorithm for Graphics Processors . . . . . . . . . . . . . . . . Pavel Emeliyanenko

427

A Novel Scheme for High Performance Finite-Difference Time-Domain (FDTD) Computations Based on GPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tianshu Chu, Jian Dai, Depei Qian, Weiwei Fang, and Yi Liu

441

Parallel Programming, Performance Evaluation A Proposed Asynchronous Object Load Balancing Method for Parallel 3D Image Reconstruction Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jose Antonio Alvarez-Bermejo and Javier Roca-Piera

454

A Step-by-Step Extending Parallelism Approach for Enumeration of Combinatorial Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hien Phan, Ben Soh, and Man Nguyen

463

A Study of Performance Scalability by Parallelizing Loop Iterations on Multi-core SMPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Prakash Raghavendra, Akshay Kumar Behki, K. Hariprasad, Madhav Mohan, Praveen Jain, Srivatsa S. Bhat, V.M. Thejus, and Vishnumurthy Prabhu Impact of Multimedia Extensions for Different Processing Element Granularities on an Embedded Imaging System . . . . . . . . . . . . . . . . . . . . . Jong-Myon Kim

476

487

Fault-Tolerant/Information Security and Management Reducing False Aborts in STM Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daniel Nic´ acio and Guido Ara´ ujo

499

Fault-Tolerant Node-to-Set Disjoint-Path Routing in Hypercubes . . . . . . Antoine Bossard, Keiichi Kaneko, and Shietung Peng

511

AirScope: A Micro-scale Urban Air Quality Management System . . . . . . . Jung-Hun Woo, HyungSeok Kim, Sang Boem Lim, Jae-Jin Kim, Jonghyun Lee, Rina Ryoo, and Hansoo Kim

520

Table of Contents – Part I

XXIX

Wireless Communication Network Design of a Slot Assignment Scheme for Link Error Distribution on Wireless Grid Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junghoon Lee, Seong Baeg Kim, and Mikyung Kang

528

Wireless Bluetooth Communications Combine with Secure Data Transmission Using ECDH and Conference Key Agreements . . . . . . . . . . . Hua-Yi Lin and Tzu-Chiang Chiang

538

Robust Multicast Scheme for Wireless Process Control on Traffic Light Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Junghoon Lee, Gyung-Leen Park, Seong-Baeg Kim, Min-Jae Kang, and Mikyung Kang

549

A Note-Based Randomized and Distributed Protocol for Detecting Node Replication Attacks in Wireless Sensor Networks . . . . . . . . . . . . . . . . Xiangshan Meng, Kai Lin, and Keqiu Li

559

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

571

Efficient Grid on the OTIS-Arrangment Network Ahmad Awwad1, Bassam Haddad2, and Ahmad Kayed1 1

Department of Computer Science, Faculty of Computing Fahad Bin Sultan University, Tabuk - Saudi Arabia [email protected], [email protected] 2 Department of Computer Science, Faculty of Information Technology University of Petra, Amman-Jordan [email protected]

Abstract. Many recent studies have revealed that the Optical Transpose Interconnection Systems (OTIS) are promising candidates for future highperformance parallel computers. In this paper, we present and evaluate a general method for algorithm development on the OTIS-Arrangement network (OTIS-AN) as an example of OTIS network. The proposed method could be used and customized for any other OTIS network. Furthermore it allows efficient mapping of a wide class of algorithms into the OTIS-AN. This method is based on grids as popular structure that support a vast body of parallel applications including linear algebra, divide-and-conquer type of algorithms, sorting, and FFT computation. This study confirms the viability of the OTIS-AN as an attractive alternative for large-scale parallel architectures.

1 Introduction The choice of network topology for parallel systems is a critical design decision that involves inherent trade-offs in terms of efficient algorithms support and network implementation cost. For instance, networks with large bisection width allow fast and reliable communication. However, such networks are difficult to implement using today’s electronic technologies that are two dimensional in nature [19]. In principle, free-space optical technologies offer several fronts to improve this trade-off. The improved transmission rates, dense interconnects, power consumption, and signal interference are few examples on these fronts [1, 2, 6, 7, 10, 13, 14]. In this paper, we focus on Optical Transpose Interconnection Systems Arrangement Networks-(OTIS-AN) which was proposed by Al-Sadi [21] that can be easily implemented using free-space optoelectronic technologies [1]. In this model, processors are partitioned into groups, where each group is realized on a separate chip with electronic inter-processor connects. Processors on separate chips are interconnected through free space interconnects. The philosophy behind this separation is to utilize the benefits of both the optical and the electronic technologies. The advantage of using OTIS as optoelectronic architecture lies in its ability to manoeuvre the fact that free space optical communication is superior in terms of C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 1–10, 2010. © Springer-Verlag Berlin Heidelberg 2010

2

A. Awwad, B. Haddad, and A. Kayed

speed and power consumption when the connection distance is more than few millimetres [6]. In the OTIS-AN, shorter (intra-chip) communication is realized by electronic interconnects while longer (inter-chip) communication is realized by free space interconnects. Extensive modelling results for the OTIS have been reported in [7]. The achievable Terra bit throughput at a reasonable cost makes the OTIS-AN a strong competitive to the to its factor network [1, 6, 7, 11, 12]. These encouraging findings prompt the need for further testing of the suitability of the OTIS-AN for real-life applications. A number of recent studies have been conducted in this direction [3, 4, 5, 8, 15, 17, 18]. Sahni and Wang [3, 4] have presented and evaluated various algorithms on OTISnetworks such as basic data rearrangements, routing, selection and sorting. They have also developed algorithms for various matrix multiplication operations [18] and image processing [17]. Zane et al. [20] have shown that the OTIS-mesh efficiently embeds four-dimensional meshes and hypercubes. Aside from the above mentioned works, the study of algorithms on the OTIS is yet to mature [16]. In this paper we contribute towards filling this gap by presenting a method for developing algorithms on the OTIS-AN. This method is based on grid as popular structure that supports a vast body of applications ranging from linear algebra to divide-and-conquer type of algorithms, sorting, and FFT computation. The proposed method is discussed in the sequel, but first we give the necessary definitions and notation.

2 Preliminary Notations and Definitions Let n and k be two integers satisfying 1≤ k ≤ n-1 and let us denote = {1, 2,…., n} and = {1, 2,…, k}. Let

Pkn taken k at a time, the set of arrangements of k ele-

ments out of the n elements of . The k elements of an arrangements p are denoted p1, p2,…, pk. Definition 1 (Arrangement Graph) The (n,k)-arrangement graph An,k = (V, E) is an undirected graph given by: V= { p1 p2… pk⎪pi in and pi≠pj for i≠j} =

Pkn , ….

(1)

and E = {(p,q) ⎪ p and q in V and for some i in , pi≠qi and pj = qj for j≠i}. … (2) That is the nodes of An,k are the arrangements of k elements out of n elements of , and the edges of An,k connect arrangements which differ exactly in one of their k positions. For example in A5,2 the node p=23 is connected to the nodes 21, 24, 25, 13, 43, and 53. An edge of An,k connecting two arrangements p and q which differ only in one position i, it is called i-edge . In this case, p and q is called the (i,q)-neighbour of p. An,k is therefore a regular graph with degree k(n-k) and n!/(n-k)! nodes. As an example of this network figure 1 shows A4,2 arrangement with size of 12 nodes and a symmetric degree of 4.

Efficient Grid on the OTIS-Arrangment Network

3

Since OTIS-networks are basically constructed by "multiplying" a known topology by itself. The set of vertices is equal to the Cartesian product on the set of vertices in the factor network. The set of edges consists of edges from the factor network and new edges called the transpose edges. The formal definition of OTIS-networks is given below. 42

12

32 14

34

24

13

31

43 23

21

41

Fig. 1. The arrangement graph A4,2

Definition 2 (OTIS-Network) Let G0 = (V0, E0) be an undirected graph representing a factor network. The OTIS-G0 = (V, E) network is represented by an undirected graph obtained from G0 as follows V = {〈x, y〉 | x, y ∈ V0} and E = {(〈x, y〉, 〈x, z〉) | if (y, z)∈E0} ∪ {(〈x, y〉, 〈y, x〉) | x, y ∈ V0 and x ≠ y}. The set of edges E in the above definition consists of two subsets, one is from G0, called G0-type edges, and the other subset contains the transpose edges. The OTISAN approach suggests implementing Arrangement-type edges by electronic links since they involve intra-chip short links and implementing transpose edges by free space optics. Throughout this paper the terms “electronic move” and the “OTIS move” (or “optical move”) will be used to refer to data transmission based on electronic and optical technologies, respectively.

4

A. Awwad, B. Haddad, and A. Kayed

Definition 3 (Cross Product) The cross product G = G1⊗G2 of two undirected connected graphs G1=(V1, E1) and G2=(V2, E2) is the undirected Graph G=(V, E), where V and E are given by: V={〈x1, y〉 | x1∈V1 and y∈V2} and E={(〈x1, y〉, 〈y1, y〉) | (x1, y1) ∈ E1} ∪ {(〈x, x2〉,〈x, y2〉) | (x2 ,y2) ∈ E2} . So for any u =〈x1, x2〉 and v =〈y1, y2〉 in V, (u, v) is an edge in E if, and only, if either (x1, y1) is an edge in E1 and x2 = y2, or (x2, y2) is an edge in E2 and x1 = y1. The edge (u, v) is called a G1-edge if (x1, y1) is an edge in E1, and it is called G2-edge if (x2, y2) is an edge in E2 [9]. The size, degree, diameter and number of links of the cross product of two networks are defined next. Definition 4 (Topological properties of cross product networks) If G1 and G2 are two undirected connected graphs of respective size s1 and s2 and have respective diameters δ1 and δ2, then [8, 9]:

1) G1⊗G2 connected. 2) The diameter δ of G1⊗ G2 is δ = δ1+δ2. 3) The size s of G1⊗ G2 is given by s = s1.s2

4) The degree of a node u =〈x1, x2〉 in G1⊗G2 is equal to the sum of the degrees of vertices x1 and x2 in G1 and G2, respectively. 5) Number of links for the product network, is given by (size⋅degree)/2.

3 Topological Properties of OTIS-AN This section reviews some of the basic topological properties of the OTISArrangement network including size, degree, diameter, number of links, and shortest distance between 2 nodes [7, 21]. The topological properties of the OTISArrangement network along with those of the Arrangement network are discussed below. We will refer to g as the group address and p as the processor address. An intergroup edge of the form (〈g, p〉, 〈p, g〉) represents an optical link and will be referred to as OTIS or optical move. Note that also we will be using the following notations are defined: • • • • • •

| An,k | = size of the graph An,k.. |OTIS-An,k | = size of the graph OTIS-An,k. Deg. An,k ( p) = Degree of the graph An,k at node p. Deg. OTIS-An,k (g, p) = Degree of the graph OTIS- An,k at node address . Dist-An,k (p1, p2) = The length of a shortest path between the two nodes p1 and p2 in Arrangement graph. Dist. OTIS-An,k (p1, p2) = The length of a shortest path between the two nodes and in OTIS-Arrangement.

Efficient Grid on the OTIS-Arrangment Network

5

In the OTIS-Arrangement the notation 〈g, p〉 is used to refer to the group and processor addresses respectively. Figure 2 shows that as an example of OTIS-A3,2. The figure shows that two nodes 〈g1, p1〉 and 〈g2, p2〉 are connected if and only if g1 = g2 and (p1, p2)∈E0 (such that E0 is the set of edges in Arrangement network) or g1 = p2 and p1 = g2, in this case the two nodes are connected by transpose edge. The distance in the OTIS-Arrangement is defined as the shortest path between any two processors, 〈g1, p1〉 and 〈g2, p2〉, and involves one of the following forms: 1. When g1 = g2 then the path involves only electronic moves from source node to destination node. 2. When g1 ≠ g2 and if the number of optical moves is an even number of moves and

more than two, then the paths can be compressed into a shorter path of the form: 〈g1, E

E

O

p1〉 ⎯⎯→ 〈g1, p2〉 ⎯⎯→ 〈p2, g1〉 ⎯⎯→ 〈p2, g2〉 ⎯⎯→ 〈g2, p2〉 here the symbols O and E stand for optical and electronic moves respectively. 3. When g1 ≠ g2, and the path involves an odd number of OTIS moves. In this case the paths can be compressed into a shorter path of the form: O

E

O

E

〈g1, p1〉 ⎯⎯→ 〈g1, g2〉 ⎯⎯→ 〈g2, g1〉 ⎯⎯→ 〈g2, p2〉. The following are the basic topological properties for the OTIS-Arrangement. For instance if the factor Arrangement network is of size n!/(n-k)!|, degree is n-1 and diameter is ⎣1.5 k⎦ [7]. Then the size, the degree, the diameter, number of links, and the shortest distance of OTIS-Arrangement network are as follows: • • • • •

Size of |OTIS-An,k | = |n!/(n-k)!|2. Degree of OTIS-An,k= Deg.(An,k),if g= p. Deg. (An,k) + 1, if g ≠ p. Diameter of OTIS-An,k = 2⎣1.5 k⎦ +1. Number of Links: Let N0 be the number of links in the An,k and let M be the number of nodes in the An,k. The number of links in the OTIS-An,k =

( M 2 − M ) / 2 + N 02 . For instance, the number of links in the OTIS-A4,2 consisting of 144 processors is = (12 •

2

− 12) / 2 + 2302 = 595

Dist. of OTIS-An,k =

Theorem 1 The length of the shortest path between any two processors 〈g1, p1〉 and 〈g2, p2〉 in OTIS-Arrangement is d(p1, p2) when g1 = g2 and min{d(p1, p2) + d(g1, g2) +2, d(p1, g2)

+ d(g1, p2) + 1} when g1 ≠ g2, where d(p, g) stands for the shortest distance between the two processors p and g using any of the possible shortest paths as seen in the above forms 1, 2 and 3 [15].

6

A. Awwad, B. Haddad, and A. Kayed

It is obvious from the above theorem that when g1 = g2, then the length of the path

between the two processors 〈g1, p1〉 and 〈g2, p2〉 is d(p1, p2). From the shortest path construction methods in (2) and (3) above, it can be easily verified that the length of the path equal min {d (p1, p2) + d(g1, g2) +2, d(p1, g2) + d(g1, p2) + 1 when g1 ≠ g2}.

To send a message M from the source node 〈g1, p1〉 to the destination node 〈g2, p2〉 it must follow a route along one of the three possible paths 1, 2, and 3. The length of the shortest path between the nodes 〈g1, p1〉 and 〈g2, p2〉 is one of the forms: ⎧ d(p1, p2)

if g1 = g2

(3) Length = ⎨ ⎩ min( d(p1, g2) + d(g1, p2) + 1, d(p1, p2) + d(g1, g2) + 2) o.w. Where d(p1, p2) is the length of the shortest path between any two processors 〈g1, p1〉 and 〈g1, p2〉. If δ0 is the diameter of the factor network An,k then from (1I) it follows that

the diameter of the OTIS-An,k is 2δ0 + 1.The diameter of OTIS-An,k is the Max (δ0, 2δ0 + 1) which is equal to 2δ0 + 1. The proof of the above theorem is a direct result from (I).

Fig. 2. OTIS-A3,2 network

4 Hierarchical Decomposition for the OTIS-Networks In this section the hierarchical structure of the OTIS-AN is discussed. The properties of a new decomposition method for the OTIS-AN presented and proved. These

Efficient Grid on the OTIS-Arrangment Network

7

properties are then used in the subsequent sections to develop grids and pipelines as methods for developing various parallel algorithms on the OTIS-AN. An OTIS-AN based computer contains N2 processors partitioned into N groups with N processors each. A processor is indexed by a pair 〈x, y〉, 0≤x, y
single vertex labeled by i and having a link between i and j if Ψi and Ψj share a prefect matching, i.e. VGΨ = V0 and EGΨ = {(i, j) ⎟ Ψi perfectly matches Ψj}. Theorem 2 The two Ψ and Φ decomposition methods of the OTIS-AN0 have the following properties: 1. 2. 3. 4.

Ψi is isomorphic to AN0. VΨi ∩ VΦj = {〈i, j〉}.

Ψi and Φi share perfect matching for all i values. Ψi and Ψj share perfect matching for all i and j values and hence GΨ is a complete graph. (Fig. 3)

8

A. Awwad, B. Haddad, and A. Kayed

U

<1

<2

)~v0~

)2

)1

U

11

21

«¢¢ 1,*²²

U12

U22

«¢¢ 2,*²²

U1~~v0~~

U2~~v0 ~

¢*,1²²

< ~v0~~

U v ~ 0~ ~1

¢*,~ ~v ~²

¢*,2²²

U v ~ 0~ ~2

0

«¢¢ ~v0~,*²²

Uv v ~ 0~ ~~ 0~ ~

Fig. 3. The grid structural for OTIS-AN

Proof Property 1 is a direct consequence of Definition 7. The function ρ maps nodes from VΨi to V0. In fact, the set {ρ(u) | u∈Ψi} is equal to V0 for any i. Since any two neighboring nodes u and v in Ψi should have γ(u)=γ(v) and since (ρ(u), ρ(v)) is an edge in E0; the subgraph Ψi is isomorphic to AN0. Property 2 states that for any two labels i and j from V0, the two subgraphs Ψi and Φj have exactly one node in common. Since, VΨi = {〈i, x〉 | x ∈ V0} and VΦj = {〈x, j〉 | x

∈ V0}, the intersection VΨi ∩ VΦj contains only the node 〈i, j〉.

Let fi : VΨi → VΦi be a function that maps nodes form Ψi into Φi for all i values de-

fined as follows: fi(〈x, y〉) = 〈y, x〉. First we have |VΨi| = |VΦj| for all i and j. For any two

distinct nodes u and v in VΨi we have fi(〈γ(u), ρ(u)〉) = 〈ρ(u), γ(u)〉 ≠ fi(〈γ(v), ρ(v)〉) =

〈ρ(v), γ(v)〉; because ρ(u) ≠ ρ(v). Hence the function fi is on-to-one and onto. Thus property 3 follows.

Efficient Grid on the OTIS-Arrangment Network

9

Let tij: VΨi → VΨj be a function that maps nodes form Ψi into Ψj, for any i and j, as follows: tij(〈i, x〉)=〈j, x〉. For any two distinct nodes u and v from VΨi we have tij(〈i,

ρ(u)〉) = 〈j, ρ(u)〉 ≠ tij(〈i, ρ(v)〉) = 〈 j, ρ(v)〉. Since |VΨi| = |VΨj| it follows that Ψi and Ψj share perfect matching for all i and j values and hence GΨ is a complete graph. Lemma 1: GΨ can be embedded into OTIS-AN0 with dilation δAN0 + 2.

Proof Since GΨ is complete, any two distinct nodes i and j in VANΨ are neighbors. The “vir-

tual” path between 〈i, x〉 and 〈j, x〉 in OTIS-AN0 that corresponds to the edge (i, j) in EANΨ is constructed as follows: 〈i, x〉 → 〈x, i〉 || πG0(i, j) || 〈x, j〉 → 〈j, x〉. An arrow represents an edge connecting the two nodes and the operation “||” means appending two paths (i.e. connecting the last node in the left path to first node in the right path). Notice that the choice of x from V0 does not affect the construction of this path nor its length. The path segment πG0(i, j) is an isomorphic copy to the optimal length path from i to j in AN0. It can be verified that the above constructed path is of optimal length equal to dAN0(i, j)+2. Hence, the longest such path cannot exceed δAN0 + 2.

5 Conclusion The study of algorithms on the Optical Transpose Interconnection Systems (OTIS) is still far from being matured. In this paper, we have contributed towards filling this gap by proposing a method for algorithm development on OTIS-AN network. This method is based on the grid structure as popular framework for supporting vast body of important real-world parallel applications. Utilizing this method to develop parallel algorithms for linear algebra will be discussed as a future case study. Several topological properties including size, degree, diameter, number of links and shortest distance between any two nodes have been discussed. The proposed OTIS-AN shown to be an attractive alternative for its factor network in terms of routing by utilizing both electronic and optical technologies. As a future research work we could utilize the proposed framework in solving real life problems on OTIS-AN including Matrix problems and Fast Fourier transforms.

References 1. Agelis, S.: Optoelectronic router with a reconfigurable shuffle network based on microoptoelectromechanical sys. Journal of Optical Networking 4(1), 1–10 (2005) 2. Akers, S.B., Harel, D., Krishnamurthy, B.: The Star Graph: An Attractive Alternative to the n-Cube. In: Proc. Intl. Conf. Parallel Processing, pp. 393–400 (1987) 3. Al-Sadi, J., Awwad, A., AlBdaiwi, B.: Efficient Routing Algorithm on OTIS-Star Network. In: Proceedings of the IASTED International Conference on Advances in Computer Science and Technology, pp. 157–162. ACTA Press (2004)

10

A. Awwad, B. Haddad, and A. Kayed

4. Awwad, A., Al-Ayyoub, A., Ould-Khaoua, M., Day, K.: Solving Linear Systems Equations Using the Grid Structural Outlook. In: Proceedings of the 13th IASTED Parallel and Distributed Computing and Systems (PDCS 2001), USA, pp. 365–369 (2001) 5. Chatterjee, S., Pawlowski, S.: Enlightening the Effects and Implications of Nearly Infinite Bandwidth. Communications of the ACM 42(6), 75–83 (1999) 6. Dally, W.J.: Performance Analysis of k-ary n-cubes Interconnection Networks. IEEE Trans. Computers 39(6), 775–785 (1990) 7. Day, K., Tripathi, A.: Arrangement Graphs: A Class of Generalised Star Graphs. Information Processing Letters 42, 235–241 (1992) 8. Day, K., Al-Ayyoub, A.: Topological properties of OTIS-Networks. IEEE Transactions on Parallel and Distributed Systems 13(4), 359–366 (2002) 9. Day, K., Al-Ayyoub, A.: The Cross Product of Interconnection Networks. IEEE Trans. Parallel and Distributed Systems 8(2), 109–118 (1999) 10. Hendrick, W., Kibar, O., et al.: Modeling and Optimisation of the Optical Transpose Interconnection System. In: Optoelectronic Technology Centre, Program Review, Cornell University (September 1995) 11. Krishnamoorthy, A., Marchand, P., Kiamilev, F., Esener, S.: Grain-size Considerations for Optoelectronic Multistage Interconnection Networks. Applied Optics 31(26), 5480–5507 (1992) 12. Marsden, G., Marchand, P., Harvey, P., Esener, S.: Optical Transpose Interconnection System Architecture. Optics Letters 18(13), 1083–1085 (1993) 13. Wang, C., Sahni, S.: Matrix Multiplications on the OTIS-Mesh Optoelectronic Computer. IEEE Trans. Computers 40(7), 635–646 (2001) 14. Yayla, G., Marchand, P., Esener, S.: Speed and Energy Analysis of Digital Interconnections: Comparison of on-chip, off-chip, and Free-space Technologies. Applied Optics 37(2), 205–227 (1998) 15. Sahni, S., Wang, C.: OTIS-star an attractive alternative network. In: Proceedings of the 4th WSEAS International Conference on Software Eng., Parallel & Distributed Systems (2005) 16. Sahni, S.: Models and Algorithms for Optical and Optoelectronic Parallel Computers. In: Proceedings of the International Symposium on Parallel Algorithms and Networks, pp. 2– 7. IEEE Computer Society Press, Los Alamitos (1999) 17. Sahni, S., Wang, C.: BPC Permutations on the OTIS-mesh Optoelectronic Computer. Technical Report 97-008, CISE department, University of Florida (1997) 18. Wang, C., Sahni, S.: Image Processing on the OTIS-mesh Optoelectronic Computer. IEEE Trans. Parallel and Distributed Systems 11(2), 97–109 (2000) 19. Wang, C., Sahni, S.: Computational Geometry On The OTIS-Mesh Optoelectronic Computer. In: Proceedings International Conference on Parallel Processing, pp. 501–507 (2002) 20. Zane, F., Marchand, P., Paturi, R., Esener, S.: Scalable Network Architecture Using the Optical Transpose Interconnection System (OTIS). Journal of Parallel and Distributed Computing 60, 521–538 (2000) 21. Al-Sadi, J., Awwad, A.: A New Efficient Interconnection Network. The International Arab Journal of Information Technology (accepted, 2009) (to appear)

Single Thread Program Parallelism with Dataflow Abstracting Thread Tianzhou Chen, Xingsheng Tang, Jianliang Ma, Lihan Ju, Guanjun Jiang, and Qingsong Shi College of Computer Science, Zhejiang University, Hangzhou, Zhejiang, 310027, China {tzchen,tang1986,majl,lhju,libbug,zjsqs}@zju.edu.cn

Abstract. CMP brings more benefits comparing with uni-core processor, but CMP is not fit for legacy code well because legacy code bases on uni-core processor. This paper presents a novel Thread Level Parallelism technology called Dataflow Abstracting Thread (DFAT). DFAT builds a United Dependence Graph (UDG) for the program and decouples single thread into many threads which can run on CMP parallelly. DFAT analyzes the program’s data-, control- and anti-dependence and gets a dependence graph, then dependences are combined and be added some attributes to get a UDG. The UDG decides instructions execution order, and according to this, instructions can be assigned to different thread one by one. An algorithm decides how to assign those instructions. DFAT considers both communication overhead and thread balance after the original thread division. Thread communication in DFAT is implemented by producer-consumer model. DFAT can automatically abstract multi-thread from single thread and be implemented in compiler. In our evaluation, we decouple single thread into at most 8 threads with DFAT and the result shows that decoupling single thread into 4-6 threads can get best benefits.

1 Introduction As there are many problems arising from uni-core processor, nowadays multi-core processor is becoming more and more popular. Multi-core processor is also called as Chip Multi-Processor (CMP). It is no doubt that CMP can bring many benefits for parallel program or multi-thread program, but most of current application is based on the uni-core processor, which usually are single thread program. CMP is not good for single thread program because of the chip frequency declining. Automatic transferring single thread to multi-thread program is an ideal method, there are many researchers devoting themselves into this area. Nowadays the main idea for single thread parallelism in CMP is exploring the parallel part of code. Instruction Level Parallelism (ILP), Thread Level Parallelism (TLP) and Thread Level Speculation (TLS) are usual methods used in parallelism. ILP is a fine-grained way for parallelism, which realizes instruction parallelism execution. TLP, also called software level parallelism, abstracts multi-thread from single thread program and every abstracted thread is assigned to cores to accomplish some jobs. Few of abstracted threads comprise a kind of software level pipelining. Speculation is C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 11–21, 2010. © Springer-Verlag Berlin Heidelberg 2010

12

T. Chen et al.

another parallelism method, which can be considered as a kind of TLP. It can execute some parts of code beforehand according to some special algorithm. Speculation is an important direction for current research and there are many implementations of speculation, but if it is a wrong speculation, it cannot improve program performance and maybe lose some performance. Whatever people did in parallelism area, the dependence is the basic constraint for program parallelism. So every research current done is based on dependence. Program dependence has two kinds, control dependence and data dependence. But if a sequence code slice is executed parallelism, anti-dependence is also a problem which should be considered. If we consider branch code slice as value dependence of branch condition, control dependence is one kind of data dependence, adding the original data dependence, data dependence and control dependence link instructions into chains, which are called as data-flows. Usually there are many data-flows existing in a program at the same time and those data-flows interweave together combining the whole program data-flow. Adding the anti-dependence to the whole program dependence, it can be definitely decided that some instructions must be executed before some others and some be later than some else, and some can be executed at the same time. Based on such truth, a graph with instruction execution order is constructed. This graph indicates some instructions can be run at the same time, so all instructions in the code slice can be divided into groups according to the graph. Instructions with strong contact will be assigned to the same group and some with less contact assigned to the different group. The grouping can separate one slice into two or more slices so that code can be run parallelly in CMP. This paper tries to abstract program parallelism with Dataflow Abstracting Thread (DFAT) in compiling period. DFAT is an independent method for the threads parallelism and it is compatible with other parallel technology. DFAT analyzes the program’s data-, control- and anti-dependence and gets a dependence graph, then dependences are combined and be added some attributes to get a United Dependence Graph (UDG). The UDG can decide all instructions execution order in the graph. According to the order, instructions can be assigned to different thread one by one. An algorithm decides how to assign those instructions. DFAT considers both communication overhead and thread balance after the original thread division. Thread communication in DFAT is implemented by producer-consumer model. DFAT can automatically abstract multi-thread from single thread and be implemented in compiler.

2 Related Work As single core processor draws many problems that are difficult to be solved, CMP born in an appropriate time. CMP architecture has many benefits over than single core processor, and the most important advantage brought by CMP is that CMP can run the different threads at the same time and achieve the real parallelism but not concurrent. ILP tries to explore the parallelism probability of instructions. It is a fine-grained parallelism and can be supported by compiler. But most usually, ILP is supported by substrate hardware and is unconscious to the program. Kumar et al proposed an

Single Thread Program Parallelism with Dataflow Abstracting Thread

13

architecture call Carbon to support efficient ILP[8]. Carbon has relatively simple hardware and devotes to accelerate dynamic task scheduling on scalable CMPs. Ottoni et al uses a global instruction scheduler GREMIO to assign different instruction into cores[11]. GREMIO uses control dependence analysis to extract nonspeculative thread-level parallelism from sequential codes. It fits for parallel control application, but if the control dependences are sequence, the assignment will be constrained heavily. Sampson et al exploit the potential of CMPs for fine-grained data parallel tasks[16]. They present barrier filters, a mechanism for fast barrier synchronization on chip multi-processors to enable vector computations to be efficiently distributed across the cores of a CMP. The barrier can guarantee the data synchronization, but if the different parallel parts are not at the close length, the waiting overhead can take a huge overhead, especially, when one part need more time than others, this can cause a lot of latency. ILP has been proposed for a long time and many modern processors take the advantages of ILP. There are many researches about ILP such as [19][20]. Those researches have some similarities, which are the characters of ILP. Usually, ILP is implemented by substrate hardware and is invisible for compiler and programmer, but compiling optimization can make ILP more efficient. The benefit of ILP is great, but meanwhile, ILP brings complexity of hardware design. Substrate hardware should provides faster data transmission and need more energy assumption, besides, ILP brings more resources contention. For the most of programs, they are compute-intensive and have abundant threadlevel parallelism, and are therefore good targets for running on a CMP. Thies et al focuses on the streaming application of legacy C code and proposes a new and pragmatic approach to leveraging coarse-grained pipeline parallelism[1]. It inserts some annotations to the legacy code to indicate the pipeline boundaries and tracks communication across those boundaries. The difficulty of the proposed method is the programmer has to know where the annotations should be inserted. For a lot of legacy code, the parallel parts are implicit and it is not easy for programmer to find those parts. If there is an automatic method to insert the annotations, it will be good. DSWP is another way of thread parallelism for the legacy code and legacy code is extracted to multi-thread program automatically by compiler[4]. It works on loops and analyses the dependences of the loops to construct a dependences graph. According to the graph, some instructions will be combined to the new node so that all dependence in the graph is ordinal. If the loop is nested, the inner loop’s balance is a challenge of DSWP. The benefit of DSWP is that thread abstraction can be done by compiler and without manually helping. The disadvantage is that it brings much communication overhead and cannot deal cyclic block. Speculation can be used in ILP and TLP, but for the CMP, speculation is used as TLP usually, sometimes called thread-level speculation. Hammond [14] describes the complete implementation of the support for thread level speculation on the Hydra chip multiprocessor, which is good for improving performance when there is a substantial amount of medium-grained loop-level parallelism. The data path may be the constraint of whole system. Actually, speculation can be adopted in a more range area; speculative thread parallelism in SPEC2000 is exposed[10], which manually parallelizes several SPEC2000

14

T. Chen et al.

benchmarks to discuss how and where parallelism was located within the application. Those work can guides the future advanced TLS compiler design. The difficulty of speculation in the paper is that some of them are hard to implement. The common case is that people can analyze the program characters according to the program structure and then use efficient speculation to them; the compiler cannot do that so intelligently. TLP and speculation have not a clear distinction and many researches implement speculation with TLP. Speculation can make code block execution pipelining[5][6]. TLP needs compiler support to distract parallel parts form sequence codes. The keys of TLP are threads communication and balance. Speculation also needs compiler support usually. Meanwhile, speculation needs hardware support sometimes. Speculation policy decides the performance of speculation.

3 DFAT 3.1 Motivation There is not any simple and efficient multi-threads program design technology nowadays. Because of the variety of CMP, it needs a long time to build a general multithread program model. For the current existing parallel program model, such as OpenMP and MPI, can use the advantages of CMP and improve program performance, but those technologies need programmers have a clear understanding of program parallel parts and set the execution mode of those parts. This needs programmers learn new skills of writing parallel program, but for some parallel parts which are implicit, programmers might not find. In this section, we propose a technology called DFAT to improve single thread application parallelism in CMP. This technology cans automatically analyze the dependences among instructions, and group parallel instructions into new threads so that different cores in CMP can run those threads parallelly. DFAT is an automatic thread decoupling technology and completed by compiler, it can keep the original single thread application’s semantic when decoupled thread is executed on CMP. DFAT is not exclusive and can be inclusive with ILP, software level pipelining and speculation. The in-order execution program has two kinds of dependences, data dependence and control dependence. Those two dependences decide the process the program runs, but those dependences bring constraints for parallelism. If we want to fall this single thread program into two or more parts and run different parts in parallel, there will introduces another new dependence — anti-dependence, which is like the situation Write after Read (WAR) in out of order processor. Namely, the values of some variables have been changed by some instructions which should be executed later than the instruction using those variables in the original program. In this paper, we will mix mentioned three dependences and call the combined dependences as United Dependence (UD). UD decides the instruction execution order and is the key point of our analysis. Figure 1 is an example for instruction parallelism. Figure 1(a) is a code slice from SEPC2000 twolf which is a basic block. (b) is the UDG of (a). How to get a UDG is described at following section. (c) and (d) are two divisions of original code, and the

Single Thread Program Parallelism with Dataflow Abstracting Thread

15

Fig. 1. (a) Code slice from twolf SPEC2000; (b) UDG of (a); (c) Decoupling (a) into two groups; (d) Communication overhead with different decoupling; (e) Add attributes to UDG

Fig. 2. An example of control value transmission

dashed line is the value transmission. (c) is a ideal division, and is the result of DFAT. (d) exchanges instructions 4 and 5 bringing more communication. As for basic block, division is simple, but in loop or branch, control value should be transmitted to each group. Figure 2 is an example of branch value transmission. When the original branch is decoupled into more than one branch, the original branch value should be sent to each new branch. This is implemented by thread communication. Loop condition is the same as branch. 3.2 DFAT Algorithm We can consider those dependences as just one kind because what the three dependences definitely do is to decide the instructions execution order. So we mix three dependences into one dependence called United Dependence. The graph base on United Dependence is called United Dependence Graph (UDG), so UDG is also a graph of instructions execution order. The rest of this section will present the algorithm of DFAT. Figure 3 is the pseudo code of DFAT. The DFAT algorithm is a recursive algorithm. It bases on the un-nested loop. Unnested loop means the loop without containing any loops inside and including basic block and branch only. After finding such loops, abstract the sequence block (all instructions in the loop except the loop condition) and analyze the dependences of instructions and get a Dependences Graph (DG) with control, data and anti-dependences. If any two nodes in DG have more than one edge, eliminate redundant edges and just left one. This one edge is United Dependence and decides the instruction execution order. When all redundant edges are eliminated, DG is changed to UDG.

16

T. Chen et al.

Fig. 3. Pseudo code of DEAT algorithm

Considering the communication overhead and balance among decoupled threads, Communication scale and node latency are added to edges and nodes. Because of data and control dependences need value transmission, the edge with value transmission should add a value to represent the number of values transmitted, and the node should add a value of the estimated node execution time. Figure 1(e) is the UDG with adding attributes. The following step of DFAT is dividing UDG into different groups, the number of groups is the number of thread and set previously by system, compiler or user, and the instructions in the same group will construct a thread. The goal of division is to achieve the best thread performance in CMP after decoupling. There has best solution for every UDG division, but it is a NP problem. In this paper, we use a heuristic algorithm to reduce the time and space consumption. According to the evaluation, the heuristic algorithm can get close performance comparing with the best solution. In our division, we consider the communication overhead first, namely instruction with value dependence should be assigned into the same thread, but meanwhile the thread balance is important and would be guaranteed. As for the thread balance, we set a threshold and the biggest time difference of threads should not be bigger than the threshold. There are many ways to set threshold, it could be a constant, a random number among a range or a ratio of the longest thread running time. We chose the ratio of original code running time as the threshold in this paper. The detailed division is like this. Firstly, it gets a free node from UDG. Free node is the node without depending on other nodes. The free node will try to be put into every group, and for each group, DFAT will calculates the communication overhead

Single Thread Program Parallelism with Dataflow Abstracting Thread

17

and judges whether it is balance. If it is not balance, the node could not be put into that group. The free node would be put into the group that has the minimum communication overhead at last. But if the communication overhead is same when the free node is put into different groups, the more balance group would be the choice. Namely, estimated running time difference is smallest. The communication overhead is the total number of value transmission. The balance is judged by the threshold, and estimated thread running includes two parts — instruction execution time and estimated thread waiting time which is calculated according to previous instruction division. If the biggest estimated running time difference of threads is bigger than the threshold, threads are unbalance. After the node putting, this node would be deleted from UDG, and the related edges are also deleted. That would produce some new free nodes. Then get a new free node and try to put it into groups until the UDG is empty. After the division of un-nested loop, a group is considered as a node in original codes. DFAT estimates running time of all groups and calculates the live-in and liveout of each group. When estimating running time, the loop is assumed that it executes only once. Because we believe that every group has some instructions of loop, whatever how many iterations the loop has, groups are balance.

…

1j= ; 2 while or if (condition) { 3 some j = , = j or not; 4 j= ; 5 some j = , = j or not; } 6 = j;

…

… …

…… … …… …

Fig. 4. A special case should be considered

When all instructions are grouped, producer and consumer instructions are inserted into threads. There is a special case which should be considered. Figure 4 is a presentation of special case. In the figure, the values of variable j in instruction 6 may be from instruction 1, and may be from the inside of loop or branch. In such case, instructions 1 and 4 like would be adjusted in the previous division and put into the same thread. Then the producer instruction would be put behind the right bracket, if j is used in other threads. 3.3 Communication We use producer-consumer mode to deal with thread communication. Some modifications should be added to compiler. Compiler needs to add two instructions, producer and consumer respectively. producer’s format is operation code, register and memory address. Register is the value which should be delivered and memory address is a point to store value. consumer’s format is also operation code, register and memory address. The difference of producer and consumer is write value to and read value from memory.

18

T. Chen et al.

4 Evaluation 4.1 Test Cases and Simulator Table 1. The baseline microarchitecture of every core I cache D cache L2 cache Branch Pred Execution Function Unit Size(insts) Width(insts/c)

16K 4-way set-associative, 32 byte blocks, 1 cycle latency 16K 4-way set-associative, 32 byte blocks, 1 cycle latency 1024K 8-way set-associative, 128 byte blocks, 6 cycle latency hybrid Out-of-Order issue, Out-of-Order execution, In-Order commit 2 integer ALU, 1 integer MULT/DIV, 2 load/store units, 2 FP ALU, 1 FP IQ:8, RUU:32, LSQ:8 decode:2, issue:2, commit:2

In this section, we make an evaluation to the proposed idea. We choose benchmarks from SEPC2000, which are mentioned and tested in [10]. We do not evaluate the whole benchmark and only implement the important parts indicated in [10], because those parts almost take all running time of whole benchmark. We decouple chose parts into two to eight threads and simulate them on CMP. The simulation only implements the proposed method of this paper and does not use any other parallelism technology. The simulation is implemented in Simplescalar. The baseline microarchitecture is showed in Table 1. We modified Simplescalar heavily and merged many cores in Simplescalar. Cores are homogeneous. Instruction execution latency, cache construction, memory hiberarchy is the same as Simplescalar default set. Thread communication is solved through producer-consumer mode and two new instructions (producer and consumer) are added to the Simplescalar instruction set. L1 cache is private for cores, and L2 cache is shared between cores. Cache coherence is guaranteed by snoop. Table 2. The value of threshold set in this paper # of thread Threshold

2 0.4

3 0.5

4 0.4

5 0.3

6 0.3

7 0.2

8 0.2

The balance of thread is based on the threshold. There are many ways to define threshold, meanwhile every threshold can be tested through running a set of program. The changeable threshold provides the flexibility of the system. In our evaluation, we took the ratio of original program running time as the threshold and we tested the reasonable threshold for different number threads. Table 2 is the result of our test. 4.2 Performance Evaluation Compared to Baseline The baseline performance is the original single thread program running under the baseline microarchitecture showed in Table 1. Figure 5 is the normalized performance improvement compared with the baseline performance. In the figure, performance

Single Thread Program Parallelism with Dataflow Abstracting Thread

gzip gcc

2.4

parser votex

bzip art

Waitting time

Normalized Performance

2.2 2 1.8 1.6 1.4 1.2 1 2

3

4

5

6

Number of Threa ds

7

8

Fig. 5. Speedup with multi-threads

gzip gcc

90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 2

3

parser votex

4 5 6 Number of Threads

19

bzip art

7

8

Fig. 6. The ratio of waiting time

improvement is close when the number of thread is equal or smaller than 4. But when it is bigger than 4, the performance improvement for different program is various. Some programs still get performance improvement, but some get performance degradation. The reason is the constraint of program dependence and threads spends lot of time on waiting. If the overhead of thread communication is zero, program performance after decoupling would never degrade when the number of thread increase and the worst case is performance improvement is constant when the number of threads is bigger than a value. But threads communication cause more memory access and cache coherence, so when number of threads increases, communication overhead kill the performance improvement. Decoupling 4 to 6 threads is better in DFAT. But there also has some exception. votex still get improvement when decoupled into 8 threads. Figure 6 is percentage of average waiting time of each core compared the thread running time. First of all, all programs have the same trend that core spends more time on waiting when the number of thread increases. But some programs speed more time than others when they have the same number of thread, so programs get different improvement at the same number of thread. Secondly, the percentage of average waiting time is close, but the performance improvement is discrete, especially when number of thread is more than 6. The important reason is the percentage indicates the ratio of waiting only, but different programs have different running time after decoupling. Programs with same waiting percentage have different waiting time.

Acknowledgement This paper is supported by the Research Foundation of Education Bureau of Zhejiang Province under Grant No. Y200803333, the State Key Laboratory of High-end Server & Storage Technology(No. 2009HSSA10), National Key Laboratory OS Science and Technology on Avionics System Integration, the Special Funds for Key Program of the China No. 2009ZX01039-002-001-04, 2009ZX03001-016, 2009ZX03004-005.

20

T. Chen et al.

References 1. Thies, W., Chandrasekhar, V., Amarasinghe, S.: A Practical Approach to Exploiting Coarse-Grained Pipeline Parallelism in C Programs. In: Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture (December 2007) 2. Kaul, M., Vemuri, R., Govindarajan, S., Ouaiss, I.: An automated temporal partitioning and loop fission approach for FPGA based reconfigurable synthesis of DSP applications. In: Proceedings of the 36th ACM/IEEE conference on Design automation (June 1999) 3. Bondhugula, U., Ramanujam, J., Sadayappan, P.: Automatic mapping of nested loops to FPGAS. In: Proceedings of the 12th ACM SIGPLAN symposium on Principles and practice of parallel programming (March 2007) 4. Ottoni, G., Rangan, R., Stoler, A., August, D.I.: Automatic Thread Extraction with Decoupled Software Pipelining. In: Proceedings of the 38th annual IEEE/ACM International Symposium on Microarchitecture (November 2005) 5. Bhowmik, Franklin, M.: A general compiler framework for speculative multithreading. In: Proceedings of the 14th ACM Symposium on Parallel Algorithms and Architectures, pp. 99–108 (2002) 6. Johnson, T.A., Eigenmann, R., Vijaykumar, T.N.: Min-cut program decomposition for thread-level speculation. In: Proceedings of the ACM SIGPLAN 2004 conference on Programming language design and implementation, pp. 59–70 (2004) 7. Gordon, M.I., Thies, W., Amarasinghe, S.: Exploiting coarse-grained task, data, and pipeline parallelism in stream programs. ACM SIGPLAN Notices 41(11) (November 2006) 8. Kumar, S., Hughes, C.J., Nguyen, A.: Carbon: architectural support for fine-grained parallelism on chip multiprocessors. In: Proceedings of the 34th annual international symposium on Computer architecture (June 2007) 9. Vachharajani, N., Iyer, M., Ashok, C., Vachharajani, M., August, D.I., Connors, D.: Chip multi-processor scalability for single-threaded applications. ACM SIGARCH Computer Architecture News 33(4) (November 2005) 10. Prabhu, M.K., Olukotun, K.: Exposing speculative thread parallelism in SPEC 2000. In: Proceedings of the tenth ACM SIGPLAN symposium on Principles and practice of parallel programming (June 2005) 11. Ottoni, G., August, D.: Global Multi-Threaded Instruction Scheduling. In: Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture (December 2007) 12. Chen, J., Juang, P., Ko, K., Contreras, G., Penry, D., Rangan, R., Stoler, A., Peh, L.-S., Martonosi, M.: Hardware-modulated parallelism in chip multiprocessors. ACM SIGARCH Computer Architecture News 33(4) (November 2005) 13. Chu, M., Ravindran, R., Mahlke, S.: Data Access Partitioning for Fine-grain Parallelism on Multicore Architectures. In: Proceedings of the 40th Annual IEEE/ACM International Symposium on Microarchitecture (December 2007) 14. Hammond, L., Willey, M., Olukotun, K.: Data speculation support for a chip multiprocessor. ACM SIGOPS Operating Systems Review 32(5) (December 1998) 15. Sankaralingam, K., Nagarajan, R., Liu, H., Kim, C., Huh, J., Burger, D., Keckler, S.W., Moore, C.R.: Exploiting ILP, TLP, and DLP with the polymorphous TRIPS architecture. In: Proceedings of the 30th annual international symposium on Computer architecture (June 2003) 16. Sampson, J., Gonzalez, R., Collard, J.-F., Jouppi, N.P., Schlansker, M., Calder, B.: Exploiting Fine-Grained Data Parallelism with Chip Multiprocessors and Fast Barriers. In: Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture (December 2006)

Single Thread Program Parallelism with Dataflow Abstracting Thread

21

17. Rangan, R., Vachharajani, N., Vachharajani, M., August, D.I.: Decoupled Software Pipelining with the Synchronization Array. In: Proceedings of the 13th International Conference on Parallel Architectures and Compilation Techniques (September 2004) 18. Ferrante, J., Ottenstein, K.J., Warren, J.D.: The Program Dependence Graph and Its Use in Optimization. ACM Transactions on Programming Languages and Systems 9(3), 319–349 (1987) 19. Colwell, R.P., Nix, R.P., O’Donnell, J.J., Papworth, D.B., Rodman, P.K.: A VLIW architecture for a trace scheduling compiler. In: Proceedings of the 2nd International Conference on Architectural Support for Programming Languages and Operating Systems, April 1987, pp. 180–192 (1987) 20. Lam, M.S.: Software pipelining: An effective scheduling technique for VLIW machines. In: Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, June 1988, pp. 318–328 (1988)

Parallel Programming on a Soft-Core Based Multi-core System Liang-Teh Lee, Shin-Tsung Lee, and Ching-Wei Chen Department of Computer Science and Engineering, Tatung University Taipei City 10452, Taiwan [email protected]

Abstract. Soft-core system allows designers to modify the components which are in the architecture they designed conveniently. In some systems, uni-core processor can not provide enough computing power to support a huge amount of computing for specific applications. In order to improve the performance of a multi-core system, in addition to the hardware architecture design, parallel programming is an important issue. The current parallelizing compilers are hard to parallelize the programs effectively. The programmer must think about how to allot the task to each processor in the beginning. In this paper, we present a software framework for designing parallel program. The proposed framework provides a convenient parallel programming environment for programmers to design the multi-core system’s software. From the experiments, the proposed framework can parallelize the program effectively by applying the provided functions.

1 Introduction Recently, the applications of the embedded system are becoming more popular, and the computing speed requirement of the embedded system is also increasing. Usually the operating system used in the embedded system is an RTOS (Real-Time Operating System) with the time constrain. To improve the performance of an embedded system, in addition to modifying the scheduling algorithm, the other possible way is to modify the hardware architecture. In some systems, uni-core processor can not provide enough computing power to support a huge amount of computing for specific applications. For instance, in the image processing [1], the graphics card uses multi-core processors to process the images [2]. For this reason, multi-core system development has become more important. In order to improve the performance of a multi-core system, in addition to the hardware architecture and task scheduling [3], parallel programming is an important issue. The parallelizing compiler in nowadays are hard to parallelize the programs effectively. The programmer must think about how to allot the task to each processor in the beginning. In general computer systems, there are many parallel programming languages like MPI [4], OpenMP [5], CUDA [6], and OpenCL [7] and so on. But in the soft-core based embedded system, there is not a parallel programming language for use. So, we expect C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 22–31, 2010. © Springer-Verlag Berlin Heidelberg 2010

Parallel Programming on a Soft-Core Based Multi-core System

23

to provide a library-based parallel programming framework for designing algorithms on the soft-core based embedded system.

2 Background 2.1 Multi-core Architectures Compared with the uni-core system, the multi-core system is a system with two or more cores on a single chip. In the heterogeneous multi-core system, it is used to program on MPMD model. In the homogeneous multi-core system, it is used to program on SPMD model. In the MPMD model, each core runs a different program and works on different sets of data. In the SPMD model, the programs running on different core are identical. Each core has their own memory space and work on different sets of data. In this paper, we use the homogeneous multi-core system and SPMD model to construct our environment. Under this model, we can simplify the system architecture design. In the communication between each core, there are two major methods be proposed -shared memory and message passing. In the system with shared memory, several different cores access the same memory element. So they must access the data on shared memory in the same formula. Message passing is the interprocess communication between each core in multi-core systems. Communication is made by sending messages to recipients. The passed messages have to include function invocation, signals, and data packets. Message passing do not need to share anything as long as the cores which want to pass the message are connected. 2.2 Multi-core Architecture with Soft-Core System The soft-core system has flexibility for designing the system architecture. The designers can design the desired hardware architecture conveniently and easy by modifying the existing architecture. In the existing architecture, it uses the shared memory for data sharing. However, using the shared memory has to solve the problem of data corruption. If a core is writing the data to the shared memory and another core is reading or writing at the same time, data corruption will occur. In addition, synchronization and memory consistency issues must be solved in the shared memory system. To keep the design simply, we provide the Mailbox in the constructed multi-core architecture for message passing. Mailbox is easy to use and can avoid the data corruption, synchronization and memory consistency problems in the shared memory. If the updated data have not sent to the Mailbox, other cores can not get the data for further processing. 2.3 Parallel Programming In general computer systems, there are many parallel programming languages like MPI, OpenCL, CUDA, SWARM [8] and so on. Message Passing Interface (MPI) is a language-independent communications protocol used to program parallel computers. In

24

L.-T. Lee, S.-T. Lee, and C.-W. Chen

the communication it supports point-to-point and collective communication. MPI has the advantages of portability and speed. Because MPI is implemented for message passing, it almost can use in every distributed memory architecture. SWARM (SoftWare and Algorithms for Running on Multi-core) is a parallel programming framework for multi-core processors. It is based on homogeneous multi-core systems and communicates by shared memory architecture. SWARM offers open source library so we can use their function and read their source code to understand the function’s processing flow.

3 Parallel Programming Environment In this section, the proposed soft-core based multi-core system will be presented. By referring to other resources [9] [10], we construct a hardware architecture which consists of three cores and other necessary components. The multi-core architecture with the simple image in our design should include the CPU, memory, I/O, MUTEX, and a bus which connects all the components together. The architecture as shown in Fig.1 is the simple one. There are some important components should be discussed in detail, like CPU, memory, bus, MUTEX, Mailbox, SystemID, and I/O.

Fig. 1. A Simple Multi-core Hardware Architecture

3.1 Hardware Architecture A basic multi-core system includes several processors, and is connected with system bus to the memory and peripherals [9]. In this paper we implement the multi-core system with Altera’s NEEK FPGA platform. The number of logic elements in NEEK can implement the multi-core system of three cores. Fig.1 is the system architecture. In SPMD model, each core has its memory space to store the program instruction and data. Between CPUs and memories, we use bridge as connector. When each CPU wants to read instruction or data, they must access through the bridge. In this model, the bridge’s bandwidth will be the bottleneck of system performance. So we dispose the instruction cache in each CPU to decrease the access time of memories. At the communication between the CPUs, we offer the shared memory and mailbox to send the data. On the shared resources, we offer the hardware MUTEX to protect the shared data and keep the correctness of the data.

Parallel Programming on a Soft-Core Based Multi-core System

25

The System ID component is used to allocate the serial numbers to every CPU. The compiled program will be loaded to run on the target CPU with the System ID number to assure that the program will be executed correctly. In the program operation, we used the SPMD model to implement our parallel programming environment. In this model each CPU runs the same program. If we want to divide the specific code block to different CPU, we can use the System ID number to decide which CPU should do the section. At I/O device part, in order to the convenience of management, each I/O device is controlled by CPU1. If other CPUs need to access the I/O device, they must send the instruction to CPU1. Once CPU1 received the instruction, CPU1 will control the operation of I/O device. 3.2 Software Environment Our programming framework is a library-based parallel programming framework, and consists of many functions which will be used in the parallel programming. Using these functions, programmer can parallelize their program by doing a little change. Before using our function, programmers need to include a head file parallel.h. A program using our framework is structured as follows: #include “parallel.h” int main ( ) { p_Init( );/* the routine you want to parallelize*/ p_Finalize(); } In the main function, we need to use the p_Init() function to initialize the environment at first. In the initialization setting, every thread can initialize some components like the hardware mutex, shared memory and some useful variables we defined. THREADS is defined to be the total number of cores in this system. MYTHREAD is defined to be the id number of this core and starts from 0. THREADED is defined to be a structure that contains the required information for running the parallel routine. After initialization, we can use the offered function and write the program we want to process in parallel. At last, we need to call the p_Finalize( ) function to release the allocated memory. 3.3 Partition Function 3.3.1 Pardo In a program, if we need to use the loop to compute, it will take a lot of time to execute the loop. We design a loop partition function “pardo” to execute the loop concurrently on each core. The pardo can partition the loop into several parts according to the cpu numbers. When the calculated data are independent, using pardo to execute the loop will get a good performance. Pardo uses macro to replace the instruction with a “for” loop and cut the loop into several parts.

26

L.-T. Lee, S.-T. Lee, and C.-W. Chen

3.3.2 On_one_thread, On_thread() In program control, we provide two functions “on_one_thread” and “on_thread(thread_id).” When programmers want to distribute a code block to an allocated thread, they can use these two functions. “on_thread(thread_id)” function allocates code block which is surrounded by the brace characters to the assigned thread. When you insert the thread number to the “thread_id” location, the cpu will decide that whether it should do the code block according to the thread number. “on_one_thread” is defined using macro and it will be replaced by if statement. “on_one_thread” is replaced by “if(MYTHREAD == 0)”, and “on_thread(thread_id)” will be replaced by”if(MYTHREAD ==(thread_id))”. 3.4 Synchronization Function Because the operation amount is not equal after allocating each operation block to different threads, the threads running on different cores are asynchronous. When we want to synchronize each thread, we can use the barrier function to synchronize all the threads in the system. If we use the barrier function in the program, when a thread reaches the barrier, it will check whether all threads reach the barrier or not. If all threads do not reach the barrier, the reached threads will wait until all threads reach the barrier. To implement this function, we need a mutex core and a shared memory buffer to count the reached thread’s amount. When a thread reaches the barrier, it will add the counter after locked the mutex and check the counter value. If the counter is equal to THREADS, this thread will set the counter to zero and unlock the mutex. If the counter is not equal to THREADS, this thread will unlock the mutex and into a busy waiting loop. In this loop, threads read the counter value and check the counter value is 0 or not. If the value is 0, threads will jump out the loop. The flowchart of barrier function is shown in Fig.2. 3.5 Collective Communication Function (Broadcast and Reduce) In the SPMD (Single Program Multiple Data) architecture, each core runs the same program on their own memory space. When the variables in each CPU need to

Fig. 2. The flowchart of barrier function

Parallel Programming on a Soft-Core Based Multi-core System

27

participate in operation synchronously, we can use the collective communication function to operate the variables. Broadcast function can send the variable value in one thread to the others. It can synchronize the value in the same variable on different cores. In the broadcast function, we declare a shared memory buffer to store the broadcasted value. In the master threads, it will store the broadcasted value into the shared memory buffer when it runs the instruction. In other threads, it will receive the broadcasted value into the local variable space when it runs the instruction. Reduce function is able to calculate the partial values on each core to obtain addition, maximum or minimum of values. Different arguments put into the function make different operations. SUM is to calculate the sum of the partial values which are provided by each thread. MAX is to find the maximum value in the values provided by each thread. MIN is to find the minimum value in the values provided by each thread. 3.6 Memory Management Function (p_malloc, p_free) To use the shared memory, we provide two functions to allocate and release the memory space. “p_malloc” is used to allocate the memory space on the shared memory. “p_free” is used to release the allocated memory space. At the start of the program, we will get the base address of the shared memory. So we can use the address to manage the shared memory with pointer. In the example, we declare an integer pointer A. After calling the “p_malloc” function, A will be stored an address of the shared memory which the function allocated.When we do not need to use the memory, we should call the “p_free” to release the allocated memory space.

4 Experiment and Discussion 4.1 Experimental Environment In this experiment, we use Altera’s NEEK FPGA platform to construct our environment. According to the number of logic elements in NEEK, we can build a three-core multi-core system at most. Each core is 32-bit RISC CPU and the frequency is 100MHz. It has 4 kbytes instruction cache and includes Branch Prediction, Hardware Multiply and Hardware Divide in it. The memory subsystem contains two kinds of memory component. First is a 3MBytes DDR SDRAM with 16 bits wide, and the second is a 16 Mbytes Flash memory. SDRAM is used to execute the process. Flash is used to store the program. In these two memories each core has different space for use. System Bus is a 32 bits width bridge to connect the cores and memories. In software, we provide two programs in our experiments. The first is the matrix multiplication program. The second is Livermore loop 1 program. 4.2 Matrix Multiplication In matrix multiplication program, we make two matrices and give them initial values using two “for” loops. At last, we multiply these two matrices and store the result into another matrix. In this method, the data calculation is independent so we can distribute the program in high parallelism. After parallelizing, we can get the program like:

28

L.-T. Lee, S.-T. Lee, and C.-W. Chen

#include “parallel.h” .... int main() { .... p_Init(); z=(matrix*)p_malloc(SIZE*SIZE*sizeof(double)); .... pardo(i,0,SIZE,1){ for(j=0;j<SIZE;j++){ for(k=0;k<SIZE;k++){ z->h[i][j]+=x[i][j]*y[k][j];}} }p_Finalize(); } We use the pardo to divide the “for” loop and use the “p_malloc” to declare the matrix in the shared memory component. By using the shared memory, we can decrease the control of data transfer. In this experiment, we set the problem size as (i) 45 x 45 matrix (ii) 60 x 60 matrix. Then we use one core, two cores and three cores to run the program and then measure and record the execution time. 4.3 Livermore Loop Livermore loop is a standard benchmark for testing system performance. In this section, we use the Livermore loop 1 as an experiment in our environment. In the beginning, we declare three one-dimensional arrays and give them the initial number by a “for” loop. Then we calculate the floating point multiplication in the second “for” loop. After parallelized program of Livermore Loop as: #include “parallel.h” .... int main() { .... p_Init(); z=(array*)p_malloc(SIZE*SIZE*sizeof(double)); .... pardo(i,0,LOOP,1){ for(k=0;k<SIZE;k++){ x->h[k]=q+y[k]*(r*z[k+10]+t*z[k+11]);} }p_Finalize(); } The array x is declared in the shared memory. The second “for” loop is rewritten by pardo to be executed concurrently on each core. In this experiment, we also set two problem sizes. One is (i) loop 30 with array size 720 and the other is (ii) loop 90 with array size 1440. Then we use one core, two cores and three cores to run the program and then measure and record the execution time.

Parallel Programming on a Soft-Core Based Multi-core System

29

4.4 Experimental Results When all experimental programs are completed, we can get the execution time on the different conditions. is the experimental result of parallel programming on multi-core system. It shows the Matrix Multiplication and Livermore loop1 programs’ execution time in different core numbers with different problem sizes. Referring to this Table 1, we can find that at these two cases, programs can distribute equally among these cores. So when we parallelize the task on two or three cores, the processing time will be near to half or one third. In Fig.3 and Fig.4, we show the execution time and speedup in graphs. Table 1. The Experimental Result Experimental Environment (ms) Number(s) of core 1 2 3

Experimental Program and Problem Size Livermore loop1 Matrix Multiplication Loop=30 Size=720

Loop=90 Size=1440

45x45

60x60

7142.870 3681.697 2499.206

44717.277 23036.129 15015.596

5840.373 3018.139 2090.792

15556.318 8086.112 5431.313

Fig. 3. Execution Time of Each Program. X-axis is the Number of Cores and Y-axis is Execution Time (ms).

By this framework, we can change the existent sequential process into the parallel process with some modifications. In the experiment, parallel programming can reach the performance we expected.

30

L.-T. Lee, S.-T. Lee, and C.-W. Chen

Fig. 4. Speedup of Each Program. X-axis is the Number of Cores and Y-axis is Speedup.

5 Conclusions and Future Work Parallel programming is an important issue in multi-core systems. A well parallelized program can speed up the operation in the multi-core system and provide better performance. To make the software design faster, a convenient parallel programming environment is the development goal. We construct the Mailbox that provides the mechanism for multi-core architecture to pass data. We propose a software framework for designing parallel programs. In this framework, it contains several basic parallelizing functions for multi-core programming, such as partition function, synchronization function, collective communication function and memory management function. By this framework, we can change the existent serial process into the parallel process with some modifications. The experimental results show that by applying the proposed framework in a soft-core based multi-core system, the parallel programming can reach the performance we expected positively. In this framework, we just implement some basic functions in parallel programming. In future, more useful functions will be augmented in this framework to support complicated operations so as to obtain the higher programmability in parallel programming.

Acknowledgements Financial support for this work was provided by the National Science Council, Taiwan, R.O.C., under the contract of NSC 98-2221-E-036-034.

References 1. Huang, Q., Huang, Z., Werstein, P., Purvis, M.: GPU as a General Purpose Computing Resource. In: Ninth International Conference on Parallel and Distributed Computing, Applications and Technologies, pp. 151–158 (2008)

Parallel Programming on a Soft-Core Based Multi-core System

31

2. Park, J., Ha, S.: Performance Analysis of Parallel Execution of H.264 Encoder on the Cell Processor. In: IEEE/ACM/IFIP Workshop on Embedded Systems for Real-Time Multimedia, pp. 27–32 (2007) 3. Chen, J.J., Yang, C.Y., Kuo, T.W., Shih, C.S.: Energy-Efficient Real-Time Task Scheduling in Multiprocessor DVS Systems. In: Design Automation Conference 2007, ASP-DAC’07, Asia and South Pacific, pp. 342–349 (2007) 4. Message Passing Interface Forum, http://www.mpi-forum.org/ 5. OpenMP: Simple, Portable, Scalable SMP Programming, http://openmp.org/wp 6. NVIDIA, CUDA Programming Guide, Version 2.2.1 (2009), http://www.nvidia.com/object/cuda_develop.html 7. OpenCL: The open standard for parallel programming of heterogeneous systems, http://www.khronos.org/opencl/ 8. Bader, D.A., Kanade, V., Madduri, K.: SWARM: A Parallel Programming Framework for Multicore Processors. In: International Conference on Parallel and Distributed Processing Symposium 2007, pp. 1–8 (2007) 9. Jose, S.: Creating Multiprocessor Nios II Systems, Altera Co. (2007) 10. Sun, W.T., Salcic, Z.: Modeling RTOS for Reactive Embedded Systems. In: 20th International Conference on VLSI Design, pp. 534–539 (2007)

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors Yongqing Ren, Hong An, Tao Sun, Ming Cong, and Yaobin Wang School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China, 230027 [email protected], [email protected], {renyq,mcong,wyb1982}@mail.ustc.edu.cn

Abstract. Technology evolving has forced the coming of chip multiprocessors (CMP) era, and enabled architects to place an increasing number of cores on single chip. For the abundance of computing resources, a fundamental problem is how to map application on it, or how many cores should be assigned for each application. As the available concurrency varies widely for diverse applications or different execution phases of an individual program, the number of resource allocated should be adjusted dynamically for high utilization rate while not compromising performance. In this paper, aiming at resource management in flexible architecture, an implementation of confidence predictor, referred as speculative depth estimator (SDE), is introduced, which is able to conduct the real-time resource tuning. By applying the speculative depth estimator to dynamic resource tuning, the experiments results show that a good trade-off between concurrency exploitation and resource utilization is achieved. Keywords: Flexible Core Chip Multiprocessors, Resource Tuning, Concurrency Exploitation, Resource Utilization.

1 Introduction With the evolving of semiconductor technology, several physical constraints (power, wire delay, etc) force the processor design entering the multicore era, and in the near future, the number of cores available on single chip is able to reach more than one thousand [1]. On the one hand, it means almost infinite computing capability, but on the other, how to make use of so many resources is an open challenge. As the amount of available instruction level parallelism (ILP) varies widely for diverse applications [4] and different execution phases of an individual program [3], resource allocated for applications should be adjusted dynamically for high utilization rate while not compromising performance. Besides, for system with multiple tasks scheduling, dynamic resource tuning mechanism also result in highly efficient system performance, because resource-intensive applications will have more chances to obtain extra computing power for concurrency exploiting. But modern commercial CMPs are designed to exploit instruction-level parallelism within processors and thread-level parallelism across processors, which is not suitable C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 32–41, 2010. © Springer-Verlag Berlin Heidelberg 2010

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors

33

for dynamic resource tuning to each individual application. Some recently proposed alternatives to these fixed-core multiprocessors are Flexible core Chip MultiProcessers (FCCMP) [5] [6] [7], in which the number of cores attached to each program is determined at run time. For the flexible architectures, the decision mechanism of resource allocation becomes a major challenge. Some mechanisms for resource tuning [2] [8] have been proposed recently through OS or runtime controlling with profile-based or dynamic scheduling algorithms. As they are dependent heavily on software, the overhead of resource reconfiguration is greatly high. In this paper, we borrow ideas from the success of confidence estimation [9] in system resource control, and introduce a scheme to estimate the optimal number of cores allocated for different applications or execution phases. The confidence estimator, referred as speculative depth estimator (SDE), is able to generate quantitative estimations on the maximum size of instruction window for different execution phases. Based on a flexible architecture, TFlex [6], a “speculative depth table” is added to record speculative depth history patterns and predict the estimation result. Although the initial design of speculative depth estimator is insufficient, after introducing large value filter and eliminating pseudo mis-predictions, remarkably improved accuracy is obtained. By applying this predictor to actual resource tuning, results demonstrate that it achieves a good trade-off between concurrency exploitation and resource utilization. While greatly increasing the size of instruction window to the fixed 8 core allocation, the dynamic scheme improves the resource utilization rate from 47.7% to 71.8% compared to the fixed 32 core configuration. The rest of this paper is organized as follows. Section 2 is related work and the original idea. Section 3 introduces the initial implementation of the speculative depth estimator. Then in section 4, evaluation and optimizations are described. After that, in section 5, the predictor is applied to resource dynamic tuning. And at the end, section 6 are conclusions.

2 Related Work and Original Idea 2.1 Confidence Estimation Speculation plays a vital role in exploiting instruction level parallelism, such as branch prediction, value prediction, and thread level speculative execution. As the excessive use of speculation, confidence estimation [9] [10] is proposed to balance the benefit of speculation against to other overheads. A confidence estimator attempts to corroborate or assess the prediction made by branch predictors, value predictors, or thread predictors. Each prediction is eventually determined to have been predicted correctly or incorrectly. For each prediction, the confidence estimator assigns a “high confidence” or “low confidence” to the prediction. With this information, some possible applications are listed in [9], such as thread switch decision for multithreading CPU, the number of instruction issue control for individual thread in SMT, speculation control for energy and resource reduction [12] [13], and accuracy improvement of branch predictors [10].

34

Y. Ren et al.

2.2 Flexible Core Chip Multiprocessors Aiming at the technique inefficiency of modern commercial CMPs, recently proposed alternatives are some flexible core architectures [5] [6] [7], in which the cores attached to each program are determined by runtime configuration. These architecture designs provide a large optimization space to tune resource for applications or execution phases in order to achieve an ideal match between concurrency characteristics and resource utilization. Core fusion [5] and Federation [7] provides mechanism to enable multiple out-of-order/in-order cores to be fused into a single more powerful core. TFlex [6] exploits hyperblock based execution model, and in actual execution, several small cores group together to form a powerful logic core, on which one nonspeculative hyperblock and multiple speculative ones can be dispatched. As TFlex provides several novel and potential technique trends, we choose it as the fundamental platform of our work, and to explore the feasibility of dynamic resource tuning. 2.3 Confidence Estimation: Speculative Depth For sequential program, only one non-speculative hyperblock is determined at any time. To achieve a huge instruction window, an aggressive branch predictor for hyperblock is required. Thereby, besides the non-speculative hyperblocks, several speculative hyperblocks construct a huge instruction window, emerging high potential for performance exploiting.

Fig. 1. Description of Speculative Depth

To quantify the capability of speculative execution in hyperblock based program, we recur to the conception of confidence estimation, named as “speculative depth” in our work, which means how many correct predictions can be obtained on consecutive hyperblocks from a point of execution. In figure1, for a non-speculative hyperblock (HB0), the following hyperblocks can be predicted with branch predictor. The second one (HB1) is obtained based on global history and the address base of HB0. The third one (HB2) could be predicted based on the address of HB1 and the global history including predicting HB1, assuming it is a correct prediction. The same behavior repeats and a huge instruction window will be constructed by one non-speculative hyperblock (HB0) and several speculative ones (HB1, HB2…). But during the later execution, the prediction of HB6 is proved as a mis-prediction. For this execution phase, from HB0 to HB5, the speculative depth is considered as 5, which is a rough estimation about the capability of speculative execution in this code region. When execution enters this region again later, the value 5 could be used to conduct the resource allocation.

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors

35

3 Speculative Depth Estimator (SDE) Based on the consideration, some tables could be introduced to trace and predict the number of consecutive predictable hyperblocks in different phases of executing program. Once entering a phase with huge concurrency, the estimator identifies the state of affairs and conducts the processor to execute more aggressively with a more powerful logic core. Otherwise, less speculative hyperblocks should be fetched. The concurrency here means the number of hyperblocks predicted correctly, which determines the size of instruction window.

Mis-prediction

Fig. 2. Diagram of the SDE

Similar as the history table in conventional branch predictor, a depth table records the number of correct consecutive predictions from each address of start hyperblock in a phase. For example, in figure 2, hyperblock HB0 is the start block of an execution phase. The two basic operations of the SDE are: Request: Once encountering a mis-prediction, a request is sent to the SDE. In figure 2, a mis-prediction occurs when predicting the next hyperblock at the execution point of HBn. Then, HBn is considered as a new phase start, like HB0, and a request is sent to the depth table with the address base of HBn. The value in the entry of depth table indexed by the address of HBn is read out, which could be used to conduct the resource allocation of next execution phase. In figure 2, the value y from depth table indicates the reasonable number of speculative hyperblocks is y, and about y cores matches the concurrency requirement. Update: After an execution phase, in other words, encountering a mis-prediction, the information related to this region should be updated in the depth table. In figure 2, the entry indexed by start hyperblock (HB0) in depth table is updated, replacing x with value n. In addition, considering actual implementation, the value of speculative depth can not be too large. For example, for loops with great amount of iterations, it is not reasonable to spawn hundreds of instances, otherwise it will result in complex scheme to maintain the sequential semantics of program, and also cause huge amount of communication overhead for the distributed chip multiprocessors. Thereby, a maximal

36

Y. Ren et al.

execution phase can comprise up to 32 hyperblocks, and saturate the counter up to 32, which is referred the maximum number of cores in TFlex [6].

4 Evaluation and Optimizations Similar as conventional branch predictor, for the SDE, the basic metrics is accuracy. Higher accuracy rate means more opportunities to saving resource while not compromising performance. In this section, the initial SDE is evaluated, and two optimizations are introduced. 4.1 Simulation and Benchmarks In this work, we focus on the accuracy of SDE. Each correct branch prediction means the speculative depth adding 1, and branch mis-prediction will result in request and update to SDE, as described in 2.3. Six benchmarks from SPEC CPU2K are compiled and executed correctly with the TFlex compiler and simulator, and used in the following evaluation (3 spec-int: 164.gzip, 181.mcf, 255.vertex and 3 spec-fps: 179.art, 183.equake, 188.ammp).

Accuracy Rate

Initial

Filter

Eliminating

100% 90% 80% 70% 60% 50% 40% 30% 20%

A

ve ra

ge

x

ke

5. vo rt e 25

p

3. eq ua 18

.a rt

ip

8. am m 18

17 9

18

16 4. gz

1. m cf

10% 0%

Fig. 3. Accuracy rate of the SDE. The three bars correspond to results of initial design (4.1), introducing large value filter (4.2), and eliminating pseudo mis-predictions (4.3).

4.2 Initial Design of SDE If the predicted depth equals the actual number of correct consecutive predictions, it is considered as a correct prediction. Otherwise, a mis-prediction occurs. In figure3, the white bar indicates the accuracy rate of the simple SDE with these six selected benchmarks from SPEC CPU2000. The average accuracy rate is only about 25%, a very disappointing result. No one is greater than 35%, and for the worst one, 183.equake, the accuracy rate is only a little greater than 10%. Obviously, the accuracy ratio is unacceptable. In execution, diverse execution phases can be classified into two categories: one is these with intricate control flows and hard to predict for the branch predictor. For these code regions, the actual depth is usually small and varies widely for distinct execution instances. Another is these execution phases with highly accuracy rate of branch prediction, like huge loops, in which consecutive hyperblocks will be divided to sub-phases comprising 32 ones, as the upper size of an execution phase is 32. Because the execution phases with high accuracy rate of branch prediction are the major

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors

37

source of huge instruction window and concurrency exploiting, they are the major target of SDE. 4.3 Optimized with Large Value Filter Based on the analysis in last section, a value filter can be inserted into the SDE in order to eliminating these prediction outcomes which are extremely small. In implementation, the threshold value is set as 16, which means that only the depth value larger than or equal to 16 is counted and passed to the processor for resource allocation. If the value predicted is less than 16, then a default value 8 is used, and it is not included in the calculation of accuracy rate. The results with large value filter are plotted in gray bar of figure 4. It is much better than the initial SDE without any optimization. For the six SPEC benchmarks, the average rate increases to 70%, much better than the previous 25%. Even the worst case, 164.gzip, is in excess of 50%, and the best case, 255.vortex, is greater than 90%. 4.4 Eliminate Pseudo Mis-Prediction The large value filter in last section seems too strict, and some useful information is hidden: While erasing small values is reasonable, some large values may be considered as mis-prediction and filtered too. For example, assuming the prediction value from SDE is 24, but in actual execution is 28, and then it is considered as a misprediction in our evaluation. Actually, the prediction result 24 is a reasonable value, although a little less than 28. A resource tuning according to the value of 24 is also beneficial for performance improvement, even if a little potential performance is lost. Consequently, it is unreasonable to consider an absolute equivalent as a correct prediction between prediction and actual depth. During execution, the branch predictor is continuing training itself. While an execution phase ends with a depth of 24, the exit predictor also updates its content and expects a better prediction later. Once execution enters the same phase again, the speculative depth is expected larger than before. So we modify the large value filter to eliminate these pseudo mis-predictions. If the prediction result is reasonable, in other words, in excess of 15, and the actual depth is larger than the value predicted, then it is considered as correct prediction too. But if the predicted value is larger than the actual value, it is still considered as a misprediction, the same as former. The black bars in figure3 graph the results after considering this implementation. We can find the accuracy rate of these SPEC programs increases to more than 90%. It is much better and suitable to put in use.

5 Apply the SDE to Resource Allocation In this section, the efficiency of applying the SDE to resource allocation is explored. Firstly, the resource allocation scheme will be introduced. According the result of SDE, the predicted value about consecutive hyperblocks in control flow path is used to conduct the resource allocation. Assuming each hyperblock consuming a processor core, the default configuration of MultiScalar and TFLEX, the number of processor cores allocated for a program is determined in this way: if the value returned from SDE is larger than or equal to 16, then the number of cores allocated equals the value.

38

Y. Ren et al.

Fix-32

Cores in use

Dynamic 32

20

28

Number of Cores

Cores in use

Fix-8 24

16 12 8 4

(a)

ag e av er

17 9. ar t 18 8. am m p 18 1. m c f 18 3. eq ua ke

16 4. gz ip 25 5. vo rt ex

0

Cores Allocated

24 20 16 12 8 4 0

Fix-8

Fix-32

Dynamic

(b)

Fig. 4. (a) Average granularity of actual execution phase, in the terms of the number of consecutive hyperblocks predicted correctly on the configuration of fixed 8 cores, 32 cores, and dynamic ones with the help of SDE. (b) Resource utilizations of three different configurations. The left means the actual depth, or the number of consecutive hyperblocks predicted correctly, and the right one indicates how many cores are allocated.

Otherwise, the core number maintains fixed 8, which is same as the default configuration of TRIPS [17], the precursor of TFlex. Then what we concerned is the size of instruction window constructed and resource utilization rate with the help of SDE for resource allocation. The former one can be calculated by the average number of consecutive hyperblocks predicted correctly, and the latter one is the average cores allocated to the program. While encountering a mis-prediction, two status counters for instruction window and resource utilization change once together. The actual speculative depth is added up to the instruction window statistics, and the predicted depth is considered as the resource allocated. For the two counters, several increasing rules are put in use. In the following, RA means Resource Allocated though SDE, and IW means the size of Instruction Window, counted with the average number of consecutive hyperblocks of each phase in actual execution. (1) 15 < RA < IW: Counter_RA += RA, Counter_IW += RA: If instruction window is larger than number of cores allocated and the latter greater than 15, than the counter of RA and IW both increase by RA, as the instruction window is constrained by resource. (2) 15 < RA && IW ≤ RA: Counter_RA += RA, Counter_IW += IW: If the number of cores allocated is greater than 15 and also IW, then the Counter_RA increases by RA, and Counter_IW increases by IW. (3) RA ≤ 15 && IW > 8: Counter_RA += 8, Counter_IW += 8: If the speculative depth predicted is less than or equals to 15, only 8 cores are allocated to the execution phase. So even if the actual speculative depth is greater than 8, the actual number of resource and instruction window are 8 (cores and hyperblocks). (4) RA ≤ 15 && IW ≤ 8: Counter_RA += 8, Counter_IW += IW: Similar as the former rule, if the actual depth is extreme low, the actual IW will be less than 8. For comparison, two static schemes of resource allocation, fixed 8 and 32 cores, are also evaluated. Figure 4 shows the results. Fix-8 means allocating 8 cores statically, like TRIPS and Fix-32 indicates holding the number of 32 cores unchangeably, just like the extreme configuration of TFLEX for an individual program. The label “Dynamic” means resource allocation is dynamically determined by SDE. Result shows that fixed 8 cores is insufficient for concurrency exploiting, as for fixed 32

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors

39

cores, the average size of instruction window achieves up to 15.2 hyperblocks. But for the latter, the obvious shortcoming is too many resources wasting, in which only half of cores are used efficiently on average. After introducing SDE for resource allocation, the average number of cores allocated decreases to 18.9. Although a little concurrency exploitation loss (the size of instruction window decreases from 15.2 to 13.6), the average number of cores allocated for the benchmarks are reduced from 32 to 18.9, indicating a resource utilization rate promotion from 47.7% to 71.8%. With SDE, a simple model of resource allocation for flexible architecture is evaluated and remarkable improvement is achieved. Two factors lead to the efficiency of prediction on control flows: temporal locality and steady branch behavior, but also cause some troubles. For example, the granularity of execution phase may vary for different instances of the same code region, but the branch predictor trains itself gradually through execution, so the granularity of execution phase will grow coarser along with the training. A possible optimization is going through the similar process of branch predictors, such as adding global history, tournament predictor, etc. But it is more complex than branch predictor to implement these structures on SDE, and we will explore the feasibility in future work. Besides, although high accuracy of branch predictor means more opportunities of constructing a huge instruction window, and potential of performance improvement, the actual situation is more complicated. Besides control flow transfer, data dependence, communication mode, and organization of code are all critical factors for the final performance. In general, the SDE can be considered as a technique of performance tuning for the whole system. Some similar works are also proposed before [19]. Besides, more complicated methods like machine learning [15] also could be applied to highly accurate performance tuning.

6 Conclusions Flexible Core Chip Multiprocessors emerge many potential benefits compared to the current design. To support the adaptive configuration for diverse applications or execution phases, an efficient resource allocation mechanism is the first challenge, as the balance between concurrency exploitation and resource utilization is hard to determine with static methods. Some software schemes based on OS or runtime controller are inefficient for dynamic real-time resource tuning. In this paper, based on the idea of confidence estimation, a simple and efficient prediction scheme for resource tuning is proposed. The confidence estimator, named SDE, produces the estimation about the capability of speculative execution, so predicts how large an instruction window can be constructed. Although the initial design is insufficient, after introducing large value filter and eliminating some pseudo mispredictions, the SDE achieves an average accuracy rate of more than 90% on the SPEC benchmarks. By applying SDE to dynamic resource tuning, the experiments results show that it achieves a good trade-off between concurrency exploitation and resource utilization. While greatly increasing the size of instruction window compared to the fixed 8 core allocation, the dynamic resource tuning mechanism improves the resource utilization rate from 47.7% in the fixed 32 core configuration to 71.8% in adaptive allocation.

40

Y. Ren et al.

This paper is an attempt to solve the anguished problem of resource tuning on flexible architecture, and many open issues still exist. In the next step, more design space about the SDE will be explored and some more factors affecting performance will be considered.

Acknowledgement This work is supported financially by the National Basic Research Program of China under contract 2005CB321601,the National Natural Science Foundation of China grants 60633040 and 60970023 and 60736012, the National Hi-tech Research and Development Program of China under contracts 2006AA01A102-5-2 and 2009AA01Z106, the Important National Science & Technology Specific Projects 2009ZX01036-001-002 the China Ministry of Education & Intel Special Research Foundation for Information Technology under contract MOE-INTEL-08-07.

，

References 1. Asanovic, K., et al.: The Landscape of Parallel Computing Research: A View from Berkeley. Technical report, EECS, Department, University of California at Berkeley, UCB/EECS-2006-183 (2006) 2. Gulati, D., Kim, C., Sethumadhavan, S., Keckler, S.W., Burger, D.: Multitasking workload scheduling on flexible-core chip multiprocessors. In: Moshovos, A., Tarditi, D., Olukotun, K. (eds.) PACT, pp. 187–196. ACM, New York (2008) 3. Sherwood, T., Perelman, E., Hamerly, G., Sair, S., Calder, B.: Discovering and Exploiting Program Phases. IEEE Micro 23(6), 84–93 (2003) 4. Wall, D.W.: Limits of instruction-level parallelism. In: International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 176–188 (1991) 5. Ipek, E., Kirman, M., Kirman, N., Martínez, J.F.: Core fusion: accommodating software diversity in chip multiprocessors. In: International Symposium on Computer Architecture, pp. 186–197 (2007) 6. Kim, C., Sethumadhavan, S., Govindan, M., Ranganathan, N., Gulati, D., Keckler, S.W., Burger, D.: Composable lightweight processors. In: International Symposium on Microarchitecture (2007) 7. Tarjan, D., Boyer, M., Skadron, K.: Federation: Out-of-order execution using simple inorder cores. Technical Report, University of Virginia, Department of Computer Science (2007) 8. Corbalan, J., Martorell, X., Labarta, J.: Performance-driven processor allocation. IEEE Transactions on Parallel and Distributed Systems 16(7), 599–611 (2005) 9. Grunwald, D., Klauser, A., Manne, S., Pleszkun, A.: Confidence estimation for speculation control. In: Proceedings 25th Annual International Symposium on Computer Architecture, SIGARCH Newsletter, Barcelona, Spain (1998) 10. Jacobsen, E., Rotenberg, E., Smith, J.E.: Assigning Confidence to Conditional Branch Predictions. In: International Symposium on Microarchitecture, pp. 142–152 (1996) 11. Mahlke, S.A., et al.: Effective compiler support for predicated execution using the hyperblock. In: Proceedings of the 25th annual international symposium on Microarchitecture, Portland, Oregon, United States, December 01-04, pp. 45–54 (1992)

Dynamic Resource Tuning for Flexible Core Chip Multiprocessors

41

12. Manne, S., Klauser, A., Grunwald, D.: Pipeline gating: speculation control for energy reduction. In: Proc. ISCA-25, pp. 132–141 (1998) 13. Vandierendonck, H., Seznec, A.: Fetch gating control through speculative instruction window weighting. In: Stenström, P. (ed.) Transactions on High-Performance Embedded Architectures and Compilers II. LNCS, vol. 5470, pp. 128–148. Springer, Heidelberg (2009) 14. Franklin, M.: The Multiscalar Architecture. Ph.D. Dissertation. University of WisconsinMadison (1993) 15. Ganapathi, A., Datta, K., Fox, A., Patterson, D.A.: A Case for Machine Learning to Optimize Multicore Performance. In: HotPar09, Berkeley, CA (2009) 16. Ranganathan, N., Nagarajan, R., Burger, D., Keckler, S.W.: Combining hyperblocks and exit prediction to increase front-end bandwidth and performance. Technical Report TR-0241, Department of Computer Sciences, The University of Texas at Austin (September 2002) 17. Sankaralingam, K., et al.: TRIPS: A polymorphous architecture for exploiting ILP, TLP, and DLP. ACM Transactions on Architecture and Code Optimization (TACO) 1(1), 62–93 (2004) 18. Luo, Y., Packirisamy, V., Hsu, W.-C., Zhai, A., Mungre, N., Tarkas, A.: Dynamic performance tuning for speculative threads. In: Keckler, S.W., Barroso, L.A. (eds.) ISCA, pp. 462–473 (2009)

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms Eunji Lee1, Sungju Lee1, Yongwha Chung1,∗, Hyeonjoong Cho1, and Sung Bum Pan2 1

Department of Computer and Information Science, Korea University, Korea {achee,peacfeel,ychungy}@korea.ac.kr http://algolab.korea.ac.kr/ 2 Dept. of Control, Instrumentation, and Robot Engineering, Chosun Univ., Korea [email protected]

Abstract. SECMPEG selectively encrypts data to protect MPEG video streams with minimum overhead. However, this scheme is known to provide only confidentiality but not integrity for data. In this paper, we apply AES-CCM to SECMPEG for ensuring integrity as well as confidentiality for multimedia data. Furthermore, in order to satisfy the real-time requirement, we use a multi-core processor. After analyzing the computational characteristics of SECMPEG/AES-CCM on a multi-core processor, we parallelize SECMPEG/AESCCM with both static and dynamic task assignments. Our experimental results show that a near-optimal speed-up could be obtained using on 4-core processor with ensuring both confidentiality and integrity of multimedia data.

1 Introduction With the advance on the multimedia technology, the multimedia contents can be delivered to users having portable devices such as mobile phones. Multimedia data delivery raises issues of the content ownership and the privacy and thus protecting the multimedia data becomes important in multimedia applications. Since most multimedia data have huge sizes, we need an efficient multimedia encryption scheme for protecting the multimedia contents while satisfying the real-time requirement. Recently, SECMPEG(Secure MPEG)[1-2] - a tightly-coupled encryption and compression scheme - has been proposed in order to reduce the computational workload with keeping an appropriate level of security. However, such multimedia encryption scheme is still computationally heavy for embedded processors to satisfy the real-time requirement. Furthermore, SECMPEG ensures only confidentiality, but not integrity. As multi-core processors have been increasingly used for embedded systems, the parallel processing techniques on multi-core processors becomes attractive in order to satisfy the real-time requirement for embedded systems. Many parallel approaches have been proposed for video compression using parallel computers[3]. Those ∗

Corresponding Author.

C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 42–51, 2010. © Springer-Verlag Berlin Heidelberg 2010

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms

43

approaches are divided into two categories, static and dynamic. Static approaches assign each task into each processor(i.e., divide the data space equally) at the beginning of the computation to reduce heavy communication overhead across the processors. This static task assignment is easy to implement and works very well with regularly-structured computations such as matrix multiplication. However, some of the MPEG computation is irregularly-structured(i.e., computational workload depends on data content, rather than data size), and such static assignment may cause the load balancing problem[4-6]. On the contrary, assigning tasks dynamically based on the relative workload of each processor can solve the load unbalance problem, at the expense of the heavy communication overhead. Therefore, one of the major research issues on dynamic task assignments on parallel computers is to minimize the communication overhead across the processors while efficiently providing load balancing. Especially, multi-core processors chips designed to have low communication overhead between cores are of our interest. In this paper, we propose to apply AES-CCM[7-10] to SECMPEG(i.e., video compression and selective encryption) in order to ensure both confidentiality and integrity for multimedia data. In addition, we parallelize the proposed scheme on multi-core processor with both the static and the dynamic task assignment techniques. More specifically, we first analyze the computational characteristics of SECMPEG on a multi-core processor. Then, the static and the dynamic task assignments are applied to the regularly- and the irregularly-structured computation in compression and encryption, with the minimum number of synchronizations. Our experimental results showed that a near-optimal speed-up could be obtained on a 4-core processor with ensuring both confidentiality and integrity of multimedia data The rest of the paper is structured as follows. Section 2 describes SECMPEG and how to provide data integrity. Section 3 explains the parallel SECMPEG with AESCCM for a multi-core processor. Finally, Section 4 and 5 describe the experimental results and conclusion, respectively.

2 Background 2.1 Multimedia Data Protection Generally, the MPEG video encoding[1] scheme represents the video signal using repetition of Group of Pictures(GOPs), and each GOP is a sequence of selected I-, Pand B-frames. I-frames are encoded as standard JPEGs, whereas P- or B-frame are encoded only for the frame difference between the close frames based on motion compensation. Also, when the motion compensation cannot find the approximate matching block in the close frames, that block is encoded as intra-coded mode, where this macro block is called I-block. Finally, an I-block which has large amount of motion in P- or B-frame is encoded as standard JPEGs. When the real-time performance is required for embedded systems, encrypting the full video data may not be adequate. To reduce the computational workload while providing an appropriate level of security, SECMPEG[2] - a tightly-coupled encryption and compression scheme has been proposed. Using the characteristics of the MPEG video compression, SECMPEG selectively encrypts data such as I-frame and I-block.

44

E. Lee et al.

SECMPEG provides five levels of the security where level 4 represents full encryption and level 0 represents no encryption. At the security level 1, it encrypts the headers from the sequence layer down to the slice layer. The motion vectors and the DCT-coded blocks are unaltered. At the security level 2, SECMPEG encrypts parts of the I-blocks in addition to the encryption performed in the level 1. At the security level 3, SECMPEG encrypts I-frames and all I-blocks. Table 1 shows the encryption part and the data size of each security level. Note that we focus on SECMPEG security level 3. Table 1. Security level of secmpeg[2] Security Level 0

1

2 3 4

Encryption Part X Sequence Header GOP Header Picture Header Slice Header Security Level 1 Relevant I-block I-Frame Whole I-blocks in P-/B-Frame Encoded Video Sequence

Data Size X 40byte 8byte/GOP 16byte/Frame 6byte/Slice Security Level 1 384byte Variable Variable Variable

2.2 Ensuring Data Integrity In cryptography, Hash-based Message Authentication Code(HMAC) is a specific cons truction for calculating a Message Authentication Code(MAC) involving a cryptograp hic hash function in combination with a secret key. The cryptographic strength of the HMAC depends upon the size of the secret key used. The most common attack agains t HMACs is the brute force attack to uncover the secret key. Block cipher modes of operation have used with block cipher such as AES[7]. For ensuring confidentiality and integrity of the data, National Institute of Standards and Technology(NIST) has proposed five modes of operations[8]. One of the five modes, Counter(CTR) mode can be used as a stream cipher, and encrypt data very fast. Another method to verify integrity is Cipher Block Chaining-Message Authentication Code(CBC-MAC), which uses the final block for verifying of the message integrity. AES-Counter with CBC-MAC(AES-CCM)[9,10] is a combined encryption and aut hentication block cipher mode. The CCM mode specification supports either both aut hentication and encryption or encryption only. AES-CCM algorithm consists of two p rocesses.: (1) CCM requires two block cipher encryption operations per each block of encrypted and authenticated message; and (2) one encryption per each block of associ ated authenticated data. Fig. 1 shows the block diagram of AES-CCM.

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms

45

Input : Nonce, Associated Data, Plain Text CBC

Block Formatting function Block0

Block1

Block2

AES (Block0)

AES (Block1)

AES (Block2)

Increase

CTR

Increase

Counter0

Counter1

Counter2

AES (Counter0)

AES (Counter1)

AES (Counter2)

Plain Text

Plain Text Cipher Text

AES(Block2)

Plain Text MAC

AES(Counter0)

MAC

Fig. 1. Block Diagram of AES-CCM[9,10]

3 Parallel SECMPEG/AES-CCM In this section, we first apply AES-CCM to SECMPEG security level 3(denoted as SECMPEG/AES-CCM). Then, we analyze the computational characteristics of SECMPEG/AES-CCM on a multi-core processor. Finally, we assign tasks both statically and dynamically based on the relative workload of each processor for providing load balancing. 3.1 SECMPEG/AES-CCM At SECMPEG security level 3, only I-frames and I-blocks of P or B-frames are encrypted, instead of whole multimedia data. In this paper, we use AES-CCM combining CTR and CBC. Note that, AES-CCM ensures confidentiality and integrity of the data at the same time. In general, CMAC is generated with a block cipher mode such as CBC after applying full data encryption. However, only 2% of whole multimedia data were encrypted in SECMPEG security level 3. To apply AES-CCM to SECMPEG, we define V-MAC(Video MAC) and S-MAC(Small MAC). As shown in Fig. 2, V-MAC that is generated by all S-MACs can ensure the all video data integrity. Also, S-MAC can detect the forgery in selective data such as I-frame or I-block. Fig. 3 shows that comparison of full data encryption and modified SECMPEG with AES-CCM(i.e., SECMPEG/AES-CCM). Although this approach causes some redundant memory space, it can reduce the execution time by the selective encryption.

46

E. Lee et al.

Fig. 2. AES-CCM for SECMPEG Security Level 3

Fig. 3. Comparison of Full Encryption and Selective Encryption

Moreover, the total size of V-MAC and S-MAC is negligible, compared to that of the compressed multimedia data. 3.2 Parallel SECMPEG/AES-CCM In order to satisfy the real-time requirement, we need to apply parallel processing techniques on a multi-core processor. To do so, we use ALPBench[3] that is a parallel benchmark code for MPEG-2. We first apply AES-CCM to ALPBench and analyze Thread-Level Parallelism(TLP) of ALPBench which assigns the macro blocks statically to each core before run-time. As shown in Fig. 4, the static assignment cannot guarantee the balance of the workload across cores. This is because the MPEG-2 involves many branch and comparison operations. Note that, AES-CCM workload depends on the MPEG-2 computational workload. We also observed that the load unbalance problem might happen across cores in both B- and P-frames. This is because the computational workload of B- and P-frame depends on the characteristic of macro block(i.e., intra, forward, backward encoding). To solve this problem, we need to assign the workload to cores dynamically.

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms

47

Fig. 4. Execution Time of MPEG2 at Each Core with 25 Frames(4-Core AMD Phenom II)

3.3 Load Balancing SECMPEG/AES-CCM Static task assignment is easy to implement and works well with regularly-structured computation such as matrix multiplication. However, some of the MPEG computation is irregularly-structured, and such static assignment may cause the load balancing problem. On the contrary, assigning tasks dynamically based on the relative workload of each processor can solve the load unbalance, at the expense of the heavy communication overhead. Since “motion estimation” with P- or B-frame is the most time consuming and unpredictable workload, we need to assign the motion estimation tasks dynamically to minimize computational time and provide load balancing across the cores. For parallelizing SECMPEG/AES-CCM on a multi-core processor, an efficient data assignment needs to be considered for each I-, B-, and P-frames. Data assignment focuses on data distribution among the cores for balancing the workload. Iframes whose workload can be predicted accurately based on the data size encrypted are assigned statically into each core(See Fig. 5). Then, each core encrypts the assigned frame and performs the synchronization for the correct execution.

Fig. 5. Static Assignment for I-frame

48

E. Lee et al.

As an I-block depends on the motion in the video contents, its frequency cannot be predicted accurately. Therefore, in case of compressing an I-block, the dynamic assignment is applied. As shown in Fig. 6, each core dynamically assigns macro blocks in the variable length coding(i.e., “putpict”).

Fig. 6. Parallel Processing for SECMPEG/AES-CCM of B- and P-frame

Fig. 7. Static and Dynamic Task Assignment for SECMPEG/AES-CCM

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms

49

Fig. 7 shows the dynamic assignment of the motion estimation and the static assignment of the remaining tasks for B- and P-Frame. Note that, in this dynamic assignment, multiple cores can share the memory resources and thus the data consistency problem should be solved. To solve the consistency problem, we use the synchronization technique(i.e., mutex) in executing the critical section to maintain the consistency of data. Also, we need to minimize the number of synchronizations due to the relatively high overhead. In the proposed approach, every core performs the synchronization after executing the motion estimation task.

4 Experimental Results For evaluating the proposed approaches, we used AMD Phenom II X4 955 Processor having 4-cores(3.2 GHz) and RAM 3.0 GB, and the code was parallelized with Pthread. We also used a data set of 25 YUV frames composed of three frames per one video sequence. The size of experimental video image was 352×288, and the size of one pixel was 24 bit. First, we measured the execution time for AES-CCM with various data size. The execution time of AES-CCM was 0.03 sec/MB. Since AES-CCM encrypts the Iblocks(i.e., about 0.0002%), I-frames(i.e., about 10%), and S-MACs(i.e., about 0.01%) only, the execution time of SECMPEG/AES-CCM was 0.002 sec/MB. Since the multimedia data include a few numbers of I-blocks, I-frames, and S-MACs, the SECMPEG security level 3 is very effective in reducing the computational overhead of resource-constrained platforms such as embedded systems. Fig. 8 also shows the effect of SECMPEG/AES-CCM, compared with full encryption. The execution time of selective encryption is shorter than that of full encryption in SECMPEG/AESCCM over all ranges of data size, and thus SECMPEG security level 3 can protect a large amount of multimedia data efficiently.

Fig. 8. Effect of Selective Encryption with SECMPEG/AES-CCM

Finally, we measured execution times of sequential and parallel approaches. To satisfy the real-time requirement, MPEG2 encoder should be performed in 7.3MB/sec=(25 frames/sec 101,376 pixels/frame 3 B/pixel). As shown in Table 2,

ⅹ

ⅹ

50

E. Lee et al.

we confirmed that the sequential approach cannot satisfy the real-time requirement even with SECMPEG(i.e., only provides an execution time of 3.55MB/sec). However, the parallel approach can achieve a speedup of 3.5~3.8 with a 4-core processor, and can satisfy the real-time requirement(i.e., provides an execution time of 13.55MB/sec). For balancing the workload with minimum overhead, we applied not only the static but also the dynamic task assignment techniques carefully for encrypting I-blocks of P- or B-frames. Table 2. Execution Time Comparison with 500MB Data

AES-CCM MPEG2 Encoding Total Throughput Real-Time Satisfaction

Sequential Approach with AES-CCM Full SECMPEG Encryption 15.47 sec 1.46 sec 139.5 sec 139.5 sec 154.97 sec 140.96 sec 3.23 MB/sec 3.55 MB/sec NO

NO

Parallel Approach with SECMPEG/AES-CCM Static Task Static + Dynamic Assignment Task Assignment 0.39 sec 0.39 sec 49.0 sec 36.5 sec 49.39 sec 36.89 sec 10.12 MB/sec 13.55 MB/sec YES

YES

5 Conclusion In this paper, for ensuring both the confidentiality and the integrity multimedia data, we applied AES-CCM to SECMPEG. Also, in order to satisfy the real-time requirement, we parallelized SECMPEG/AES-CCM on a multi-core processor. After analyzing the characteristics of the computation on a multi-core processor, the static and the dynamic load balancing techniques were applied to the regularly- and the irregularlystructured computation, with the minimum number of synchronizations. Our experimental results confirmed that the speedup of up to 3.8 could be obtained with a 4-core processor. To the best of our knowledge, this is the first result ensuring both confidentiality and integrity of multimedia data in real-time using a multi-core processor with a portable Pthread program. We will extend this research to video surveillance applications in order to provide a secure and cost-efficient solution.

Acknowledgement This work was supported by the Korea Science and Engineering Foundation(KOSEF) grant funded by the Korea government(MEST) (No. 2009-0086148).

References 1. Watkinson, J.: The MPEG Handbook. Elsevier, Amsterdam (2004) 2. Furth, B., Kirovshi, D.: Multimedia Security Handbook. CRC Press, Boca Raton (2005) 3. Ahmad, I., He, Y., Liou, M.: Video Compression with Parallel Processing. Parallel Computing 28(7), 1039–1078 (2002) 4. Hennessy, J., Patterson, D.: Computer Architecture. Elsevier, Amsterdam (2007)

Ensuring Confidentiality and Integrity of Multimedia Data on Multi-core Platforms

51

5. Akhter, S., Roberts, J.: Multi-Core Programming - Increasing Performance through Software Multi-Threading. Intel Press (2006) 6. Park, K., et al.: On-Chip Multiprocessor with Simultaneous Multithreading. ETRI Journal 22(4), 13–24 (2000) 7. Daemen, J., Rijmen, V.: Advanced Encryption Standard (AES) (November 26, 2001) 8. NIST Special Publication 800-38A, Recommendation for Block Cipher Modes of Operation - Methods and Techniques, U.S. DoC/NIST (2001) 9. Jonsson, J.: On the Security of CTR+CBC-MAC. In: Nyberg, K., Heys, H.M. (eds.) SAC 2002. LNCS, vol. 2595, pp. 76–93. Springer, Heidelberg (2003) 10. Dworkin, N.: Recommendation for Block Cipher Modes of Operation: The CCM Mode for Authentication and Confidentiality. NIST Special Publication 800-38C (2002) 11. AMD (2009), http://multicore.amd.com/Products/

A Paradigm for Processing Network Protocols in Parallel Ralph Duncan, Peder Jungck, and Kenneth Ross CloudShield Technologies, an SAIC company 212 Gibraltar Drive, Sunnyvale, CA 94089 USA [email protected]

Abstract. Network packet processing applications increasingly execute at speeds of 1-40 Gigabits per second, often running on multi-core chips that contain multithreaded network processing units (NPUs) and a general-purpose processor core. Such applications are typically programmed in a language that exposes NPU specifics needed to optimize low-level thread control and resource management. This facilitates optimization at the cost of increased software complexity and reduced portability. In contrast, our approach provides portability by combining coarse-grained, SPMD parallelism with programming in the packetC language’s high-level constructs. This paper focuses on searching packet contents for packet protocol headers. We require the host system to locate protocol headers for layers 2, 3 and 4, and to encode their offsets data in a packet information block (PIB). packetC provides descriptors, C-style structures superimposed on the packet array at runtime-calculable, user or PIBsupplied offsets. We deliver state-of-the-practice performance via an FPGA for locating layer offsets and via micro-coded interpretation that treats PIB layer offsets as a special addressing mode.

1 Introduction Pressure for faster network packet processing continues to increase as transmission mediums become faster (e.g., those specified by SONET/SDH [2, 3] and 10GbE [4]) offer speeds in the 10-40 Gigabits per second range) and the volume of data to be transmitted continues its own relentless increase. Packets contain protocol headers for communications standards, such as IPv4. A header is a contiguous set of fields with routing, service and standards data. Many protocols exist, each with distinctive content. Since multiple headers may be present and since their relative offset from the packet’s start may vary from packet to packet, a key aspect of packet processing is to determine which headers are present and where they are. Header searching is a common task for network applications, which typically run in a multithreaded environment where they are partitioned into light-weight threads that swap themselves out for each memory access. This programming style exploits low-level machine features to optimize performance. However, the resulting C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 52–67, 2010. © Springer-Verlag Berlin Heidelberg 2010

A Paradigm for Processing Network Protocols in Parallel

53

machine-specific code can require extensive redesign and recoding when the application is ported.

2 Our Approach and Contributions Our approach to packet processing as a whole has three major elements: a model of parallel packet processing, a specialized language to express the model and an ensemble of heterogeneous processors to implement the language. In this paper, we focus on features in all three that are involved in header processing. The model’s key characteristics for protocol processing are as follows: • Task granularity is at the level of a complete program that processes a packet. Thus, the model’s parallelism embodies the Single Program Multiple Data (SPMD) paradigm). • The host system locates protocol headers in a packet before a copy of the program is executed on that packet. • Each program copy works with a current packet and system-provided data on the presence and location of layer offsets.

System-provided Ingress Processing

Input Packets

packet data

packet data

packet data

Packet

Packet

Packet

Program

Program

Program

Session and search term data Shared memory

System-provided Egress Processing

Processed Packets

Fig. 1. A parallel packet processing model Copyright CloudShield Technologies, 2009

This model is expressed through packetC [5], a C-style language that takes C99 [6] as its point of departure and provides features for packet operations and protocol header processing: • Each program copy works on a single packet stored as a byte array in big endian order (matches network order). • The protocol information predetermined by the system, such as layer locations and protocol flags, is provided in a C-style structure, termed the packet information block (PIB). • Protocols can be represented by a descriptor, a special kind of structure that is superimposed on the packet at a user-specified, runtime-calculable offset location.

54

R. Duncan, P. Jungck, and K. Ross

• packetC redefines structure bitfields in a manner calculated to remove ambiguity and to allow descriptors to overlay protocol fields with non-standard bit widths in a predictable fashion. Finally, implementation support for header processing includes using an FPGA to predetermine the presence and offsets of L2, L3 and L4 headers. Thus, we mitigate foregoing thread-level parallelism for protocol header processing by executing optimized header processing on a dedicated processor, while micro-coded interpreters on NPUs are running other application tasks in end-to-end fashion on each packet. The next section presents the model in more detail.

3 A Parallel Packet Processing Model The model provides an intuitive view of parallel packet processing to the developer, one in which task management mechanics are hidden, memory is partitioned in a straightforward way and protocol detection and analysis are performed by the system as part of setting up each execution run of a program copy. The principal aspects of the model (Fig. 1) are listed below: • Concurrency is provided by copies of a small program that each completely process one packet each time they are run. • A ‘global’ memory is shared by all the program copies and private memories for each program copy. • The host system manages the program copies and routes packets to and from program copies. • The host system ensures that a program copy has two kinds of pre-processed data when it is triggered for a packet (Fig. 2). • A copy of the packet, itself, in the form of an array of unsigned bytes (in big endian order).

System-provided Ingress Processing

Current Packet Array

Packet Info Block • L2_offset = 0 • L3_offset = 14 • L4_offset = 34 …

Input Packet

Fig. 2. System-calculated protocol layer offsets for a packetC program copy Copyright CloudShield Technologies, Inc., 2009

A Paradigm for Processing Network Protocols in Parallel

55

• A collection of values that indicate whether protocol headers for network layers [7] 2, 3 and 4 are present in the packet and, if they are, where they are located (in terms of offsets from the packet array’s start.). These values are assembled in a packet information block (PIB), along with checksum information and protocol type information. In this paper we are most interested in the model’s requirements for the host system to predetermine certain protocol layer offsets and assemble them in the PIB. Operations to determine protocol headers’ presence and their offsets could be done via run-time support software routines. However, requiring the host to perform this function facilitates using a specialized processor or offload engine dedicated to effectively performing this function. Much of the PIB data’s usefulness is a consequence of how its layer offset values can be used with packetC descriptor constructs and how the implementation rapidly fetches packet data via those offset values. The next section sketches the packetC language as a whole.

4 Packetc Language Overview The packetC language’s [1] high-level characteristics include: • Emphasizing strong typing by removing implicit type casts, coercions or promotion, • Promoting real-time reliability by eliminating dynamic memory allocation, as well as pointer and address manipulations, • Using a C-style program main as the unit of SPMD parallelism, • Providing extended data types and operators to support classic packet-processing capabilities, such as matching masked database and searching for packet payload contents that match user-defined strings and regular expressions. For high-speed protocol processing, these language features are most relevant: • Presenting the PIB information in the form of a C-style structure, • Providing a descriptor construct that superimposes a C-style definition of a protocol header on the packet array at a user-specified offset expression (which may contain a PIB offset value and runtime-variable calculations), • Defining a container-based alternative to C-style bit fields to ensure that protocol header structures will match corresponding data in the packet byte array. These elements are discussed in detail below.

5 The PIB as C-Style Structure A key aspect of the PIB’s header offset values for network layers 2, 3 and 4 is that these values are determined afresh for each packet, i.e., each time a parallel copy of the packetC main program is executed. Thus, the results are always packet-specific, since the location of a particular kind of protocol header may vary from one packet to another. The PIB definitions are shown below with some omissions for readability.

56

R. Duncan, P. Jungck, and K. Ross

Packet Info Block

pib.L3_offset = 14 … pib.L3_type = L3TYPE_IPv4

Packet Array

Descriptor descriptor ipv4Descr { bits byte { version : 4; headerLength: 4; } … int ipv4_payload; } ipv4Head at pib.L3_offset;

Fig. 3. Positioning descriptors on the basis of header offset values in the PIB Copyright CloudShield Technologies, 2009

A Paradigm for Processing Network Protocols in Parallel

57

The PIB provides packetC applications with easily accessed information about whether given protocols exist and (if they do) where they are. To exploit that information, applications must be able to define those protocol headers and apply those definitions to the packet array where the data resides. packetC provides this capability through the descriptor construct described below.

6 The Descriptor Construct packetC’s descriptor construct is a structure that corresponds to a portion of the packet array with the same size. In a sense, it is an alias for an array-slice within the packet.

A descriptor declaration consists of its structure base type, its name and its location – an integer expression that defines the descriptor’s starting point in terms of an index into the packet array. This location specification, or at-clause, may contain three kinds of elements: compile-time constants, variables with values known only at run-time and PIB layer-offset fields. By combining a descriptor’s structure definitions with an offset location based on a PIB layer offset value we can create a precise, high-level descriptor of a protocol header that gravitates to the correct packet array location each time a new packet is prepared for a packetC program (Fig. 3). descriptor ipv4Descr { bits byte { version : 4; headerLength: 4; } byte typeOfService; short totalLength; short ipv4_identification; short ipv4_fragmentOffset; byte ipv4_ttl; byte ipv4_protocol; short ipv4_checksum; int ipv4_sourceAddress; int ipv4_destaddress; int ipv4_payload; } ipv4Header a t pib.L3_offset;

58

R. Duncan, P. Jungck, and K. Ross

Consider the IPv4 protocol shown above. First, the descriptor defines a structure that matches the fields of an IPv4 header. The at-clause then states that it will always be found at the packet’s layer 3 offset (when a valid layer 3 header is present). Descriptor at-clauses can be constant or run-time determined, simple or arbitrarily complex. Complex at-clause expressions are especially relevant when the start of one header depends on the presence of optional fields in a preceding header. For example, if we did not provide Layer 4 offsets, it would be possible to calculate them in terms of an IPv4 Layer 3 header as follows:

The next section presents the model in more detail. The descriptor construct is useful for describing a stack of protocols, i.e., a sequence of consecutive protocol headers that appear in a packet as a group, as shown below and in Fig. 4. • Layer 2: • Layer 3: • Layer 4:

Ethernet IP (e.g., IPv4 or IPv6) TCP or UDP

The flexibility of the descriptor’s at-clause construct allows programmers to specify stacks (i.e., to link a sequence of protocols) when the involved protocol headers • Have layer offsets other than those available in the PIB. • Have a variable size (e.g., because the header has optional fields). Handling these characteristics is particularly valuable when the developer is dealing with custom or proprietary protocols. The descriptor construct’s at-clause provides a means to flexibly superimpose a Cstyle structure onto the packet array. However, in order to use structures to represent protocol headers, we have to address fundamental problems with C bit-fields, as the next section describes.

7 Container-Based Bit Fields Given an entire descriptor’s starting location in terms of an offset into the packet array (via the at-clause’s value) we should know which portions of the packet array correspond to individual fields of the descriptor. However, many standard protocols have fields smaller than typical integer storage units (32, 16, 8 bits) or do not take up an integral number of bytes. C’s bit field construct is not adequate because the implementation freedom it bestows creates a variety of uncertainties (See section 6.7.2.1 of the C99 standard [6]).

A Paradigm for Processing Network Protocols in Parallel

59

struct structTag { unsigned int notAbitField; unsigned char a: 4; unsigned int b: 2; unsigned int c: 4; } myStruct; Packet Info Block

Descriptors

Packet Array

descriptor ethermetDescr {…} ethernetHeader at pib.L2 offset

pib.L2_offset = 0 pib.L3_offset = 14

descriptor ipv4Descr {…} Ipv4Header at pib.L3 offset pib.L4_offset = 34

•••

descriptor tcpDescr {…} tcpHeader at pib.L4_offset

Fig. 4. A protocol ‘stack’ with its corresponding PIB and descriptor information Copyright CloudShield Technologies, 2009

• ‘Straddle’ behavior – The entire field named c cannot fit in a byte allocated for a and b. Some compilers let it ‘straddle’ bytes, with 2 bits in the byte allocated for the first two fields and the remaining bits in a trailing byte but others do not. The C99 specification comments: “if insufficient space remains, whether a bit-field that does not fit is put into the next unit or overlaps adjacent units is implementationdefined.” • Container size – Similarly, the compiler may or may not heed user specifications of the storage unit to use for the bit fields. The specification says an implementation can use “any addressable storage unit large enough” to hold the bit field. • Bit field layout – Finally, we cannot be certain whether the compiler allocates the topmost fields in the declaration to the least significant bytes of the corresponding memory or how the containing unit is aligned: “the order of allocation of bits-fields within a unit is implementation-defined. The alignment of the addressable storage unit is undefined.” It is highly desirable to port a packetC application to new processors or compilers without recoding it to reflect new bit field implementation peculiarities. Thus, the packetC bit field syntax and conventions discussed below strictly control implementation. Example code below shows the packetC equivalent of the C99 structure shown above.

60

R. Duncan, P. Jungck, and K. Ross

struct structTag { int notAbitField; bits short {1 a: 4;2 b: 2; c: 4; pad: 6;3 } containerName;4 } myStruct;

1. Related bit fields are organized inside a container, which has one of packetC’s 4 unsigned integer types. 2. Since a bit field is always part of a container that has a type, each bit field declares a name and a size, not a type. 3. Pad fields are declared explicitly and cannot be accessed for test or set operations. 4. The container name can be used to access and manipulate the bit field collection as a whole. This approach removes container size, straddling and boundary uncertainties. To achieve portability we must also manage byte allocation order. C structures do not use the same byte allocation order when they are compiled on big-endian and littleendian processors. User operations on fields that match whole integer storage units do not show host-specific endianness effects but operations on bit fields, which can be sub-elements of integer storage units or straddle them, do show such effects. Big endian machines store the most significant bits of a word at the lowest byte address, while little endian machines store the least significant bits starting at the lowest byte address. packetC packet array and descriptors are required to be in bigendian order to ensure that they are portable. Although CloudShield’s products are all big-endian platforms, little-endian processors could implement packetC descriptors and packet arrays via compiler adjustments and byte swapping. Having reviewed portability matters, we turn to performance issues in the next section.

8 Current Hardware Implementation The packetC language specifies that the host system predetermines specific layer offsets for certain PIB fields but does not dictate how those calculations are done. In practice, CloudShield Technologies’ products, such as the CS-2000 [8], use multi-core NPUs much as our rivals do. However, our approach uses micro-coded interpreters running on NPUs to interpret user programs and, when appropriate, to push data to specialized processors, as indicated below. • Microcode running on some of the NPU cores works with FPGAs to control the packet pipeline and pre-locate headers. • An ensemble of NPUs, each running multiple contexts, executes the SPMD program copies and system programs.

A Paradigm for Processing Network Protocols in Parallel

61

• A custom FPGA provides the shared memory for scalars and C-style aggregates. • T-CAM (Ternary Content Addressable Memory) chips and Regular Expression processing chips implement operations on packetC’s database and searchset types, respectively [1].

Fig. 5. Layer offsets test. (a) Output speed at 1, 2, 5, 10G; (b) Details at 10Gbps Copyright CloudShield Technologies, 2010

Two system hardware aspects stand out for this discussion and support the performance encountered in the next section: • A Xilinx Virtex-5 family FPGA is the ingress processor that locates offsets for layer 2, 3 and 4 protocol headers (Fig. 3). • The NPU-based interpreters efficiently cache PIB data in registers and effectively treat PIB layer offsets as an optimized addressing mode for packet access.

9 PIBs, Descriptors and Performance CloudShield’s fundamental approach is to combine several factors in order to mitigate foregoing the performance benefits of user-directed, fine-grained parallelism and lowlevel machine resource control. Those factors consist of using: • Specialized processors to speed-up key operations. • High-level programming constructs that neither preclude nor dictate specialized hardware implementation. • Mainstream NPU parallelism but with distinctive elements of program interpretation and packet-level SPMD parallelism. In the case of applications that primarily manipulate protocol headers, these factors involve using:

62

R. Duncan, P. Jungck, and K. Ross

• A Xilinx Virtex-5 family FPGA as an off-load engine to pre-calculate often-used layer offsets, • The packetC descriptor construct to define and access protocol header fields in terms of run-time calculable values and PIB layer offsets, • Support for interpreters handling PIB layer offsets as a packet array addressing mode. Thus, the data reported here shows that for network traffic with relatively large packets, our approach lets small programs with multiple accesses to PIB layer offsets and descriptor fields run ‘at wire-speed’ for a spectrum of 1-10 Gbps input speeds. We produced the performance metrics described below, using a CloudShield CS-2000 model that uses a DPPM-800 (Deep Packet Processing Module) to execute parallel packetC programs. We used CloudShield’s CPOS 3.0.2 operating system and the packetC compiler available as part of the PacketWorks IDE (Integrated Development Environment) 3.1. The Network traffic for the tests was produced by an IXIA 1600T[9] with 10G interface and line-rate processor. Test results were produced by selecting packet size and line utilization (i.e., utilizing 50% of the 10G interface to model a 5G link). After several wall-clock seconds of execution, the experiment would be manually halted and an output metric would be calculated in terms of the total number of output bytes that the application produced, divided by the total number of bytes that the IXIA had pumped into the application. The first test centers on four PIB accesses; it consists of accessing PIB layer 3 and 4 information and forwarding the packet if it contains IPv4 TCP protocol header data. packet module layerAccess_test; #include "protocols.ph" void main( $PACKET curpacket ) { if ( p ib.l3Offset && p ib.l3Type == L3TYPE_IPV4 ) { if ( p ib.l4Offset && p ib.l4Type == L4TYPE_TCP ) { pib.action = FORWARD_PACKET; } } }

As Fig. 5 shows, the CS2000 runs the test at wire-speed with output speeds matching input speed at 1, 2 and 5 Gbps. Differences based on packet size emerge at 10Gbps, although the greatest lag (at a 500-byte packet size) still produces an output speed that is 85.8% of the input. For this particular platform and system software release, the ‘sweet spot’ is found near a packet size of 1500 bytes, where output is 97.2% of the input speed. The second test takes the four PIB accesses from the initial test and adds 8 descriptor field accesses (4 reads and 4 writes). This spoofing program simply swaps the source and destination IP addresses, as well as the source and destination ports, before sending the otherwise unaltered packet back to its sender.

A Paradigm for Processing Network Protocols in Parallel

63

Output as % of input at 10Gbps

Output (% of input)

100%

99.5%

100% 96%

80%

95.4% 93.2% 91.2%

92% 60% 88%

40%

84%

Input Gbps

1

2

5

10

Packet size (bytes)

500

1000 1500 2000

Fig. 6. Spoofing test. (a) Output speed at 1, 2, 5, 10G; (b) Details at 10Gbps Copyright CloudShield Technologies, 2010

// protocol.ph defines descriptor ipv4 below void main( $PACKET curpacket ) { int myIpSrc, myIpDest; short mySrcPort, myDestPort; if ( pib.l3Offset && pib.l3Type == L3TYPE_IPV4 ) { if ( pib.l4Offset && pib.l4Type == L4TYPE_TCP ) { myIpSrc = i pv4.sourceAddress; myIpDest = i pv4.destinationAddress; mySrcPort = t cp.sourcePort; myDestPort = t cp.destinationPort; // Swap addresses ipv4.sourceAddress = myIpDest; ipv4.destinationAddress = myIpSrc; // Swap Ports tcp.sourcePort = myDestPort; tcp.destinationPort = mySrcPort; pib.action = FORWARD_PACKET; } } }

The results are akin to those of the previous test, although everything runs slightly faster (Fig. 6). Again, the program keeps up with its input until the input speed is around 10G, at which time it falters very slightly. The marginally better numbers for a more complex program are possibly the result of the buffering and scheduling software favoring larger programs.

64

R. Duncan, P. Jungck, and K. Ross

The results indicate that small packetC programs with roughly a dozen or more descriptor and PIB accesses can run at wire-speed up to roughly a speed of 10G on the CS2000/DPPM800 platform (with its current configuration and software). The results also have broader implications. Most importantly, the results show that the benefits of the CloudShield implementation, which include using an FPGA off-load engine for layer calculations and using a fast PIB offset addressing mode, compensate for any advantages of having users implement layer detection via low-level, machine-specific tasking. In sum, the high-level, portable programs are running at state-of-the-practice speeds. Second, the application code for protocol definitions and manipulation can be coded at a high-level that is succinct and readily comprehensible. For example, both test programs fit within a paragraph or two’s space and, even for a reader unfamiliar with network programming, are immediately accessible. The next section reviews other languages for parallel network processing in general and protocol processing in particular.

10 Related Work L. George and M. Blume describe the NOVA language for the IXP network processor in [10]. NOVA features include record and tuple aggregates, familiar control-flow constructs, functions and exceptions. NOVA does not automate protocol header discovery but it provides ample means for precisely specifying protocol representation through a layout construct for unambiguously describing protocol header bit fields. A layout can describe a given bit field in two forms: packed and unpacked. The packed form approximates a C bit field description. NOVA’s overlay construct provides the capability to define alternative organizations for a given bit range within a layout.. The unpacked form accords a word of storage or a nested unpacked form to each field. NOVA provides pack and unpack operations to manage the two forms. Intel Corporation’s Microengine C is targeted to its IXP1200 network processor family [11,12]. The language is a C dialect that omits features that embedded network applications are unlikely to use (the floating-point data type, function pointers and a runtime stack). The IXP architecture features a general-purpose processor core and multiple microengines, RISC processors optimized for packet processing. The processing model involves breaking programs into multiple threads that are partitioned across multiple microengines. The user manages communications among threads and processes and typically swaps out a task that reads from or writes to memory. Microengine C provides access to many machine-specific IXP features, such as associating variables with one of three memory classes, with FIFOs or with five register classes. In addition, the language facilitates exploiting machine specifics through an extensive battery of intrinsics and support for inlining. NetPDL (Network Protocol Description Language) [13] is an XML-based language aimed at developing a database of standard network protocols, i.e., header descriptions for standard protocols. The designers appear primarily interested in providing a reusable repository of protocol definitions, rather than in defining a programming language as such, although the language does include provisions for expressions and application development.

A Paradigm for Processing Network Protocols in Parallel

65

J. Wagner and R. Leupers describe a C dialect and compiler for the Infineon network processor [14]. Language extensions provide capabilities for mapping packet protocol contents to special registers and accessing arbitrary bit-width operands without typical alignment restrictions. The compiler maps C arrays with the register qualifier to a special register file. Programmers can specify operands in the register file by using a bit-field pointer and bit-width arguments. Such arguments can have arbitrary sizes and alignments; they can also span multiple registers. A set of compiler intrinsics (compiler-known functions) are the key language extension that provides users with a mechanism to employ these capabilities at a high language level. packetC shares precise bit field specification with NOVA and shares protocol header support with NOVA and the Infineon C dialect. However, only packetC requires off-loading header detection. NOVA, microengineC and Infineon C are all geared to specific NPUs and reflect machine specifics. In contrast, packetC hides processor-specifics, although it is currently implemented with the same NPU family as the first two languages. packetC’s machine-independence goals and its current implementation’s exploitation of familiar NPU technology drive our conclusions about its portability and performance.

11 Conclusions This paper is one of several related papers: each isolates a machine-independent packetC language construct, then describes CloudShield’s specialized hardware implementation that delivers high-speed performance for that construct. In this paper, we concentrate on protocol header processing, so the relevant features in each area are as follows. • Relevant language constructs are the descriptor type and PIB definitions. • Relevant implementation specifics are the ingress processing FPGA and the use of PIB layer offsets for a fast addressing mode during user program interpretation. • Relevant performance experiments are focused on reading and writing protocol header information. Our general intent is to demonstrate that it is possible for packet processing applications to enjoy both portable, high-level language programming and state-of-thepractice packet processing performance (in the 1-10 Gbps range). CloudShield’s approach to maintaining portability while providing packet processing parallelism rests on adopting coarse-grained, SPMD parallelism and packetC programming constructs that hide host machine specifics. Consider the test programs used in section 9. Although simple, the example is a complete network program, reduced to a paragraph’s size. Esoteric system data structures have been replaced by references to predefined structure fields and enumeration literals. The SPMD-level parallelism relieves the programmer of responsibility for tasking at the everymemory-access level. In sum, much of the more arcane aspects of network programming are replaced by familiar, comprehensible constructs. Our overall approach to performance compensates for the lack of user-specified, low-level thread optimizations by employing specialty chips and processors and by

66

R. Duncan, P. Jungck, and K. Ross

exploiting classic NPU multi-core parallelism in the service of an interpreted virtual machine with domain-specific operations. In protocol header processing the CloudShield approach is manifested by using an FPGA off-load engine and by having the interpreters cache and exploit PIB layer offset information for rapid packet access. The CS2000 system used for these tests is typically deployed as a deep packet inspection platform. Our tests are essentially using it as a network router, which is appropriate for this discussion’s focus on protocol header operations. Not surprisingly, the CS2000’s observed performance exceeds that of many commercial routers. The system easily handles the test programs at 1, 2 and 5 Gbps but falters slightly at 10 Gbps. In summary, this language and system implementation approach does deliver concise, portable network programs that operate in the 1-10 Gbps range.

Acknowledgements Peder Jungck, Dwight Mulcahy and Ralph Duncan are the co-authors of the packetC language. Gary Oblock and Matt White also contributed to the language and to compiler development. Andy Norton, Greg Triplett, Kai Chang, Mary Pham, Alfredo Chorro-Rivas and Minh Nguyen provided valuable help. Professors Rajiv Gupta and Rainer Leupers provided expert advice.

References 1. Duncan, R., Jungck, P.: packetC Language for High Performance Packet Processing. In: Proceedings of the 11th IEEE International Conference on High Performance Computing and Communications, Seoul, South Korea, pp. 450–457. IEEE, Los Alamitos (June 2009) 2. ANSI T1.105: SONET - Basic Description including Multiplex Structure, Rates and Formats 3. ANSI T1.119/ATIS PP 0900119.01.2006: SONET - Operations, Administration, Maintenance, and Provisioning (OAM&P) - Communications 4. IEEE Std 802.3ae-2002. To be superseded by more recent but not finalized IEEE Std 802.3-2008 5. CloudShield Technologies. packetC Programming Language Specification. Rev. 1.128 (October 10, 2008) 6. ISO/IEC 9899:1999. Standard for the C programming language, version (‘C99’) (May 2005) 7. International Organization for Standardization (ISO) 7498. Open Systems Interconnections (OSI) reference model (1983) 8. CloudShield Technologies. CS-2000 Technical Specifications. Product datasheet available from CloudShield Technologies, 212 Gibraltar Dr., Sunnyvale, CA, USA 94089 (2006) 9. IXIA. IXIA 1600T/400T. Datasheet (retrieved 11/29/2009), http://www.ixiacom.com/pdfs/datasheets/ch_1600t_400t.pdf 10. George, L., Blume, M.: Taming the IXP network processor. In: Proceedings of the ACM SIGPLAN ’03 Conference on Programming Language Design and Implementation, San Diego, California, USA, pp. 26–37. ACM, New York (June 2003) 11. Intel Microengine C Compiler Language Support: Reference Manual. Intel Corporation, order number 278426-004 (August 10, 2001)

A Paradigm for Processing Network Protocols in Parallel

67

12. Intel Microengine C Networking Library for the IXP1200 Network Processor: Reference Guide. Intel Corporation (December 2001) 13. Risso, F., Baldi, M.: NetPDL: an extensible XML-based language for packet header description. Computer Networks. The International Journal of Computer and Telecommunications Networking 50(5) (2006) 14. Wagner, J., Leupers, R.: C compiler design for a network processor. IEEE Trans. on CAD 20(11), 1–7 (2001)

Real-Time Task Scheduling on Heterogeneous Two-Processor Systems Chin-Fu Kuo and Ying-Chi Hai Department of Computer Science and Information Engineering National University of Kaohsiung, Kaohsiung, Taiwan 106, ROC [email protected]

Abstract. A heterogenous multiprocessor system is usually composed of one general purpose processor and one or more specific purpose computing component. While heterogenous multiprocessor system becomes more popular, more and more researches pay lots of attention to this domain. In such a system, tasks often need to be processed at multiple different functional processing units. Therefore a task is usually divided into several subtasks according to its execution requirement, each of which is executed at particular processing unit with precedence constraints. In this paper, we present an EDF-based algorithm to schedule the tasks in the heterogenous multiprocessor system and propose the scheduability analysis. A series of simulation experiments are conducted to verify the analytic results and to show the capability of the proposed algorithm.

1

Introduction

The applications, which need higher computational functionality while meeting the real-time constraints in the embedded system at the same time, encounter much more rigor with the growth of application complexity. As the applications of multimedia system likewise, the processor also demands enough power to handle the video or audio streams, such as the decoding or encoding of the high-quality video compression format H.264 [1] or the Advanced Audio Coding (AAC ) [2]. For the higher computational demands of real-time applications, there are two main fashion to achieve the goal. The first fashion is to speed the processor up by means of deeper pipeline and the advance of IC technology. However, tasks are preempted with the asynchronous events to meet the real-time constraints such that the system not only losses the benefits of pipeline but also impacts additional overhead. Moreover, although the processor speed becomes faster with the advance of IC technology, it causes the problems of higher power consumption and thermal discharge too. It is not a trustworthy fashion to some devices which need to limit the power supply. The second fashion is to cope with data simultaneously on the strength of extra processors to support the system more computational capacity. The products of Intel CoreT M 2 duo and Intel CoreT M 2 quad nowadays are on the basis of this concept [3]. Besides, the uniprocessor architecture development not only suffers the bottleneck of IC processes C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 68–78, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Real-Time Task Scheduling on Heterogeneous Two-Processor Systems

69

but has the more cost than multiprocessor architecture. Thereby, the multiprocessor architecture platform becomes more and more popular to the applications with higher computational capacity for meeting the fast-grown demand of applications and increasing the performance of whole system. Besides, heterogenous multiprocessor platforms has become the major trend. For example, ”TMS320DM6446” of Texas Instruments is a powerful embedded system platform to support highly-computing requirement of multimedia applications [4]. The multiprocessor system can be typically classified to two kinds of architectures: homogenous and heterogenous. The homogenous multiprocessor system as implied by the name has serval identical processors. On the contrary, the heterogenous multiprocessor system has different characteristics processors including the general purpose processors (GPP) and the special purpose processors (DSP). Among this two kinds of processors, the DSPs are dedicated to handle the complex large mathematic equations and the decoding or encoding of multimedia streams. Moreover, because the DSPs have large amount of registers to speed the computation up, the operation fashions of them are non-preemptable to avoid the overhead of unnecessary context switch. In opposition to DSPs, due to the GPPs are not dedicated to special purpose, the frequency of GPPs must be much higher than DSPs to be evenly matched with handling some special applications. Hence the homogenous multiprocessor systems lose their predominance under the consideration of cost. Furthermore, the architecture of DSPs is provided with predictable execution time, it has more advantage in real-time system. Consequently, the application of heterogenous multiprocess systems become noteworthiness in industry, such as automotive systems, communication network, consumer electronics and mobile handhelds. Those researches mentioned above are preemptive algorithms. On the other hand, some scholars develop their schedulability tests based on non-preemptive algorithms, such as: Sanjoy purposes a feasibility analysis based on non-preemptive earliest-deadline first algorithm upon several identical processors in [5]. An on-line non-preemptive scheduling algorithm that is easy to be implemented for both, underload and overload upon the homogenous multiprocessor system is investigated in [6]. On the contrary, it is relative to be scanty of researches in heterogenous multiprocessor systems. One of the most significant reason is that the problems of real-time system are NP-hard [7] and the characteristics of heterogenous multiprocessor system lead the problems to be more complex. However, our paper focuses the previous research problem model in heterogenous multiprocessor system [8, 9, 10]. Gai et al, first points out that when the traditional scheduling algorithm such as [10] schedules tasks which need to be executed on DSP, a hole within each job is generated in the schedule of the GPP, and then they analyze the scenario when tasks are scheduled via Rate-Monotonic scheduling algorithm (RM) [11] to make more accurate the schedulability bound [9]. Due to the system model in [9] has two waiting queues (one for GPP and the other for DSP). When the DSP is active, the scheduler selects tasks from the GPP queue only. Nevertheless, the task which needs to be executed on DSP still should have the

70

C.-F. Kuo and Y.-C. Hai

right to be executed on GPP when its partial work which needs to be executed on GPP doesn’t finish. Previous problem model is improved in [8]. Under traditional real-time applications, the system decides the execution priorities of tasks according to their periods or deadlines. Many researches of real-time system are on the foundation of this basic phenomenon over the years, such as network, database and disk I/O. Even so, with the growth of complexity of real-time applications in embedded system, tasks often need to be split to execute on different processor environments, i.e., the heterogenous multiprocessor architecture (GPPs and DSPs), to satisfy the performance and real-time constraints. For instance, a video decode program must decode 30 frames per second. During the digital processing , GPP reads the data of video from disks in the first stage. The second stage is to call DSP to decode the data and then GPP shows the manipulated data on the monitor in the final stage. However, this action influences the efficiency of traditional real-time scheduling algorithms. In our paper, we adopt the previous two-queue architecture as our strategy but the DSP queue doesn’t break off when DSP is active against [9]. Furthermore, in order to improve the upper bound limit, we choose Earliest-Deadline-First (EDF) scheduling algorithm as our basis algorithm instead of RM algorithm. Note that the schedulability tests of prior researches just calculate the total utilization of GPP, but we count both the DSP and GPP utilizations separately in our schedulability tests. It give a more precise analysis than previous researches. Finally, we evaluate the effect of the proposed algorithms by a series of simulations. Our paper offers a precise schedulability test analysis for task scheduling in heterogenous multiprocessor systems. In Section 2, we first introduce the architecture of a heterogenous multiprocessor system. And in Section 3, we then propose our deadline-assignment method considering that the platform has different processors and tasks have non-preemptive execution requirements on the DSP. We illustrate the simulation and propose the evaluation results in Section 4. Finally, Section 5 presents an inductive conclusion of this paper and our future work.

2 2.1

System Model Processor Model

In this section, we will describe the processor architecture in this paper. The system architecture is composed of two different kinds of computational units named general purpose processor (GPP) and digital signal processor (DSP), respectively, is built in the same chip. Hence, the heterogenous multiprocessor system can exchange message to each other via sharing a common bus. Figure 1 illustrates the abstract of heterogenous dual-core architecture. For instance, in the playback upon TMS320C64x of Texas Instruments [4], the general purpose processor (ARM9) saves the captured raw frame data from the video capture device to the common memory shared by the GPP and the DSP, and then invokes the DSP to begin encoding those data using a video

Real-Time Task Scheduling on Heterogeneous Two-Processor Systems

71

Fig. 1. Architecture configuration of the dual-core system

Fig. 2. Execution of a DSP task

encoder algorithm. After above steps, the GPP writes those handled data to the file system. The functionality of host port interface (HPI) in Figure 1 lets the GPP can directly access the cache of the DSP and thereby speed the communication up. Furthermore, the two processors are built on the same chip and both can share the memory address space of the architecture, we can assume that the communication overheads between the GPP and the DSP can be considered as negligible. Even if some possible communication overheads still exist when tasks invoke requests on the GPP or migrate to another core, we can put it into account in priori. Hence, we can classify the system model as similar as systemon-a-chip (SoC) architecture. Note that the GPP not only communicates with the DSP but also directly controls the other peripherals. In the system, all the jobs launching on a DSP are invoked by a remote procedure call (RPC) and are executed in a non-preemptive fashion. Such RPC is also called DSP activity and scheduled by the GPP. Therefore, the GPP has the responsibility of the real-time kernel to prevent that a task invokes a DSP request when there is at least one task on the DSP. 2.2

Task Model

Figure 2 illustrates the considered real-time task model on the two heterogeneous processor system. Each task τi is a stream of job instances generated in runtime. For example, a decoder task needs to decompress about 30 encoded frames per second. The task reads the video file on the GPP, decompress the data on the DSP, and then display the frame on the GPP. For frame encoding, the execution sequence is similar to decoding procedure [12]. Therefore, each job τi,j which

72

C.-F. Kuo and Y.-C. Hai

arrives at time ri,j needs ci,gpp = ci,pre +ci,post unit of time on the GPP. ci,pre and ci,post are for the data reading and frame displaying, respectively, in a decoder task. The job also need ci,dsp unit of time and on the DSP. Besides, we assume that there is at most one DSP request invoked by each job during its execution after ci,pre unit of time. When a job finishes its DSP activity it then executes for ci,post unit of time, so that ci,pre + ci,post = ci,gpp . Definition 1. If a task needs to be executed on the DSP, it is regarded as a DSP task, otherwise it is regarded as a regular task. Moreover, jobs should arrive periodically (i.e., ri,j −ri,j−1 = Ti , where Ti denotes the task’s period) or sporadically (that is ri,j − ri,j−1 ≥ Ti , where Ti means minimum interarrival time) and end before the absolute deadline adi,j . Therefore, the relative deadline di = adi,j − ri,j . A regular task, in fact, is equal to a DSP task with ci,dsp = 0 but for the sake of simplicity, we disrepute regular tasks as DSP tasks when their ci,dsp = 0. Finally, ui denotes the utilization of task τi , ci where ui = min(d . i ,Ti )

3

Real-Time Task Scheduling on Two Heterogenous Processor System

In this section, we propose the on-line EDF-based scheduling algorithm and apply the deadline assignments for off-line schedulability test and on-line admission control. 3.1

Deadline Assignment for Subtasks on Processors

In order to guarantee the real-time requirements of tasks on the system with two heterogenous processors, the idea behind our proposed approach is to assign the relative deadlines to the subtasks of a DSP task and do the off-line schedulability analysis. In runtime, when the processor (i.e., GPP or DSP) is idle, the global scheduler dispatchs the subjob with the most urgent absolute deadline in the corresponding ready queues to the processors. As forementioned, the system maintains a global scheduler that has the responsibility to dispatch subjobs to

Fig. 3. Illustration of schedulers on dual-core systems

Real-Time Task Scheduling on Heterogeneous Two-Processor Systems

73

appropriate processors, as Figure 3 shown, against local scheduler that only deals with its own processor such as GPP or DSP. Furthermore, the scheduling strategy of the scheduler is EDF [11]. In this section, first we discuss how to assign the relative deadlines to the subtasks of a DSP task. A DSP task is composed of three subtasks that carry out on different processors according to their execution requirements. Each subtask can be considered as a stream of subjobs. The subtasks of task τi can be viewed as individual and independent tasks with equal period but different absolute deadlines. Let di,pre , di,dsp and di,post represent the relative deadline of three subtasks, i.e., τi,pre , τi,dsp , and τi,post of task τi , respectively. If a job τi,j arrives at ri,j , its corresponding subjobs are τi,j,pre , τi,j,dsp , and τi,j,post . Let adi,j,pre , adi,j,dsp , and adi,j,post denote the corresponding absolute deadlines of the subjobs. The absolute deadline of subjob τi,j,pre /(τi,j,dsp ) is the ready time of subjob τi,j,dsp /(τi,j,post ). The absolute deadlines are derived by the following equations: adi,j,pre = ri,j + di,pre , adi,j,dsp = adi,j,pre + di,pre , and adi,j,post = adi,j,dsp + di,post .

The subjobs τi,j,pre and τi,j,post are then saved in the corresponding ready queue for the GPP while subjob τi,j,dsp is saved in that for the DSP. The scheduler schedules the subjobs in the queues based on their absolute deadlines. The relative deadlines of subtasks can be assigned according to different deadline assignment strategies such as Effective Deadline (ED), Equal Slack (EQS), Equal Flexibility (EQF) in [13] and Equal Deadline (EQD). In the paper, we adopt the EQF deadline assignment strategy. The formulas of the strategy are shown in the following: c

c

c

i,dsp i,post di,pre = Ti×( ci,gppi,pre +ci,dsp ), di,dsp = Ti×( ci,gpp+ci,dsp ), and di,post = Ti×( ci,gpp+ci,dsp ).

For instance, a DSP task τi with di = 9, Ti = 10, ci,pre = ci,dsp = ci,post = 1, under the deadline assignment strategy EQF. τi,j,pre arrives at time 0, for the reason that we can visualize the subjob of τi,pre of τi as an independent and individual task τk with dk = 3, Tk = 10, ck,gpp = 1, ck,dsp = 0 and adk = adi,j,pre = 3. The subjobs of τi,j,dsp and τi,j,post have the properties s similar as τi,j,pre . The period of the three subtasks are all equal to 10. The most difference is adi,j,dsp = adi,j,pre + dk = 3 + 3 = 6 and adi,j,post = adi,j,dsp + di,pre = 6 + 3 = 9. Note that The different color boxes represent the subtasks of the DSP task. 3.2

Online EDF-Based Scheduling Algorithm

The proposed Online EDF-based Scheduling Algorithm (OESA) is based on EDF scheduling algorithm [11], that always assigns the highest priority to the job with the absolute deadline nearest to the current time. The most significant difference about the scheduling strategy in this paper is the scheduler takes no absolute deadlines of jobs but absolute deadlines of subjobs as the foundation of priority assignment in runtime, i.e., the subjob whose absolute deadline is the closest to the current time of all subjobs of all ready jobs has the

74

C.-F. Kuo and Y.-C. Hai

Algorithm 1. Online EDF-based Scheduling Algorithm (OESA) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

/* Deals with the operations while a subjob of job τi,j is finished or job τi,j arrives */ if subjob of job τi,j finishes then if the status of job τi,j == pre then if ci,dsp != 0 then The status of job τi,j := dsp; Insert nowGPP into D-queue; nowGPP := Null; else

if the status of job τi,j == dsp then if ci,post != 0 then The status of job τi,j := post; Insert nowDSP into G-queue; nowDSP := Null; else

if the status of job τi,j := post then nowGPP := Null;

16 if job τi,j arrives then 17 Insert job τi,j into G-queue; /* Checking the urgency of jobs */ 18 if the first element of G-queue has a more urgent deadline than the deadline of nowGPP then 19 Insert nowGPP into G-queue; nowGPP := the first element of G-queue; 20 if nowDSP := Null then 21 nowDSP := the first element of D-queue; 22 Execute the corresponding operation of nowGPP and nowDSP.

highest priority to be executed. If a job finishes its partial works on one processor GPP(/DSP)and it still needs to be executed on the other processor DSP(/GPP) whose characteristics is different from the previous one, the job then is migrated to the other processor. Algorithm 1 describes the scheduling situation in detail. Whenever a scheduling event occurs such as subjob finishes and new job released, etc, the global scheduler is invoked. The scheduler has the two queues, G-queue and D-queue to maintain job information. The elements in the queues are in the order of non-decreasing absolute deadlines. Besides, we use two variables named nowGPP and nowDSP to keep the information of jobs running on the GPP and the DSP, respectively. From steps 1 to 15, the scheduler deals with the operations after subjobs completion by checking the statuses of jobs, such as changing statuses or clearing nowGPP and nowDSP. There are three kinds of statuses of job named pre, dsp, and post, respectively, and the order of them takes turns. Each status represents the execution requirement of job now. Hence, a job has no status of dsp if it is a instance of a regular task. From steps 16 to 19, the scheduler checks if the subjob of the new arrival job is more urgent than the current running subjob. If the result is true, the running subjob will be preempted. Finally, because of the non-preemptive fashion of DSP, the scheduler must check whether DSP is active or not before assigning subjob to it in step 20.

Real-Time Task Scheduling on Heterogeneous Two-Processor Systems

3.3

75

Off-Line Schedulability Test Analysis

In our study, we propose a off-line schedulability test analysis to determine whether the system with n tasks is schedulable or not, which is divided into two parts. The first part is used for the consideration of the GPP and the other is used for the DSP. As long as one of them is failed, the system is unschedulable. On the General Purpose Processor. As we mention in Section 3.2, the scheduling algorithm we adopt is on the basis of EDF algorithm. Hence, the utilization upper bound which determines a system is feasible or not is equal to 1. Theorem 1 gives a complete depiction. Theorem 1. [14] A system of n independent, preemptable tasks is schedulable upon one processor via the EDF algorithm if the total utilization of the tasks is not more than 1. Theorem 1 is applied to uniprocessor system without considering the existence of DSP tasks. However, as forementioned visualization, the subtasks of the same task are viewed as individual and independent tasks with equal period but different deadlines. Therefore the utilization ui,pre of subtask τi,pre is equal to ci,pre min(di,pre ,Ti ) . The deriving of ui,dsp and ui,post are similar to ui,pre . Besides, because the relative deadlines of subtasks are smaller than period, we can therec fore reduce the utilization ui,pre to di,pre for subtask τi,pre . Before we extend i,pre Theorem 1 to the system we study, we define some notations first: UT is the total utilization of task set T in system. Φ is the task set whose subtasks need to execute on DSP in system. ui,dsp denotes the utilization of τi,dsp , c where ui,dsp = ui,dsp . ui,post denotes the utilization of τi,post , where ui,post = i,dsp ci,post di,post . Because of the characteristics of splitting jobs, we then conclude the following corollary that deal with the GPP schedulability. Corollary 1. A system with the task set T , where T consists of n independent and preemptable tasks, is schedulable via the EDF algorithm if   UT = (ui,pre + ui,post ) + ui ≤ 1. (1) τi ∈Φ

τi ∈T −Φ

Proof. The proof is a direct consequence of the point of view of visualization mentioned behind Theorem 1.  On the Digital Signal Processor. We show the schedulability test on the DSP in this subsection. First, we define some basic notations. Cmax is the maximal blocking time of all non-preemptable subtasks on the DSP, i.e., Cmax = max∀τi ∈Φ {ci,dsp }. Dmin is the minimal relative deadline of all non-preemptable subtasks on the DSP, that is Dmin = min∀τi ∈Φ {di,dsp }.

76

C.-F. Kuo and Y.-C. Hai

Theorem 2. [15] For a system with non-preemptable subtasks is schedulable via the EDF algorithm if  Cmax ui,dsp ≤ 1 − . (2) Dmin τi ∈Φ

Therefore, we follow Theorem 2 as our scheduling test on DSP. Due to take consideration of different processors, the advantages is not only to enhance the schedulability but to relax the total utilization UT .

4

Simulation and Performance Evaluations

The purpose of this section is to evaluate the performance of the proposed scheduling scheme. We develop a simulation model for the scheme, and use some numerical examples to show the capabilities of our scheme. We investigate two kinds of experiments. First we examine the task sets in the heterogenous multiprocessor system to be scheduled or not. 4.1

Experiment Environment Setting

In our experiments, the subtasks on each processor are scheduled via EDF algorithm [11]. We design a task generator to produce end-to-end periodic tasks and their workload for each set of the experiments. The performance of our scheduling test analysis is inspected on a large number of task sets by the schedulability test equations (i.e., Equation 2 and Equation 1), each simulation result is an average over twenty independent simulation runs. The execution times of tasks in a task set are generated by random variables with uniform distribution while the utilization of a task is assigned with poisson distribution. The task numbers are 10, 20, 30, 40 and 50. The total utilization of the system is first randomly generated between 0 and 1, and then the average utilization of each task is calculated as the seed of poisson distribution. We assign the utilization of each tasks via poisson distribution variable with mean equal to the average utilization of each task. In each experimental task set, the proportion of DSP tasks to total tasks is 0.9. The ci,pre and ci,post are chosen between 1 to 10 with uniform probabilities, and the ci,dsp is assigned in the range between 0.5 to 0.8 of ci,gpp . The c +c +ci,post Ti is derived by i,pre i,dsp . ui We first examine the performance and correctness of our off-line schedulability test. The performance metric is the average schedulability ratio. The schedulability ratio is the feasible solution number divided by the number of task sets. We compare the evaluation of three different approaches: MDSS [9], RSHD [8] and DPCP [10], as description in Section 1, against our proposed method. 4.2

Simulation Result

Figure 4(a) shows the schedulability ratios under different compared algorithms. We can observe that when the system load is light (i.e., the total utilization

Real-Time Task Scheduling on Heterogeneous Two-Processor Systems

(a) Task number is 30

77

(b) Task number is 50

Fig. 4. Performance of different schedulability test methods

 ci,pre +ci,dsp +ci,post is less or equal to 0.4, which is defined as ), the results Ti of OESA are similar to that of DPCP, MDSS and RSHD. The schedulability ratios start to drop when the total utilizations are equal to 0.4, 0.5, 0.6 and 0.8 under DPCP, OESA, MDSS and RSHD, respectively. We can also observe that the performance of OESA is better than MDSS and DPCP in most situation. However, the performance of OESA algorithm is worse than that of RSHD when the total utilization of the system is less than 0.9. That is because Equation 1 of our OESA algorithm lets the utilization of DSP tasks become twice lager against the other algorithms. Therefore, the performance of OESA algorithm decays when the total utilization of the system is about equal to 0.5. However, although the point, where the schedulability ratios decreasing of our OESA algorithm is earlier than RSHD and MDSS, the decay ratio of our OESA algorithm is smoother and outperforms the others when the total utilization of the system is larger than 0.9. The reason can be concluded that because we use EDF-based scheduling algorithm as our method which is different from the compared RMbased algorithms, the schedulability upper bound of EDF-based algorithm is higher than that of RM-based algorithm. Moreover, the previous researches only take the total utilization including DSP activities as blocking time on GPP into account. Our proposed method separates the utilization calculation for the GPP and the DSP, respectively. Hence, our method is better when the system has a higher load. When the task number is equal to 50, the similar results can be obtained in Figure 4(b).

5

Conclusion and Future Work

In this paper, we propose an online EDF-based scheduling algorithm, named OESA, for a heterogenous multiprocessor system with a GPP and a DSP. It is motivated by the needs in which the system must run real-time tasks in a real-time fashion. The schedulability test is proposed to determine whether the system can schedule the tested task set without any deadline missing. The capability of the proposed algorithm is evaluated by a series of experiments. It was shown that the proposed algorithm can have better performance compared to other related approaches.

78

C.-F. Kuo and Y.-C. Hai

More than our features mentioned before, there are still some appealing problems which are worthy to be investigated in the future. For example, we can derive a more accurate schedulability bound in this dual-cores heterogenous multiprocessor system and extend our OESA algorithm to deal with the system where there are multiple GPP’s and DSP’s.

References [1] Wiegand, T., Sullivan, G.J., Bjontegaard, G., Luthra, A.: Overview of the h.264/avc video coding standard. IEEE Transactions on Circuits and Systems for Video Technology 13, 560–576 (2003) [2] Bosi, M., Brandenburg, K., Quackenbush, S., Fielder, L., Akagiri, K., Fuchs, H., Dietz, M., Herre, J., Davidson, G., Oikawa, Y.: ISO/IEC MPEG-2 advanced audio coding. Journal of the Audio Engineering Society 45, 789–814 (1997) [3] Intel, http://www.intel.com [4] TMS320DM6446 Digital Media System-on-Chip – Datasheet. Texas Instruments (March 31, 2008) [5] Baruah, S.K.: The non-preemptive scheduling of periodic tasks upon multiprocessors. Real-Time System 32, 9–20 (2006) [6] Dolev, S., Keizelman, A.: Non-preemptive real-time scheduling of multimedia tasks. In: Proc. Third IEEE Symposium on Computers and Communications ISCC ’98, June 30-July 2, pp. 652–656 (1998) [7] Lee, C., Lehoczky, J., Siewiorek, D., Rajkumar, R., Hansen, J.: A scalable solution to the multi-resource qos problem. In: Proc. 20th IEEE Real-Time Systems Symposium, December 1-3, pp. 315–326 (1999) [8] Kim, K., Kim, D., Park, C.: Real-time scheduling in heterogeneous dual-core architectures. In: Proc. 12th International Conference on Parallel and Distributed Systems ICPADS 2006, vol. 2, p. 6 (July 12-15, 2006) [9] Gai, P., Abeni, L., Buttazzo, G.: Multiprocessor dsp scheduling in system-on-achip architectures. In: Proc. 14th Euromicro Conference on Real-Time Systems, pp. 231–238 (June 19–21, 2002) [10] Sha, L., Rajkumar, R., Lehoczky, J.P.: Priority inheritance protocols: an approach to real-time synchronization. IEEE Transactions on Services Computing 39, 1175– 1185 (1990) [11] Liu, C.L., Layland, J.W.: Scheduling algorithms for multiprogramming in a hardreal-time environment. Journal of the ACM (JACM) 20(1), 46–61 (1973) [12] Paulin, P.G., Pilkington, C., Langevin, M., Bensoudane, E., Benny, O., Lyonnard, D., Lavigueur, B., Lo, D.: Distributed object models for multi-processor soc’s, with application to low-power multimedia wireless systems. In: DATE ’06: Proceedings of the conference on Design, automation and test in Europe, pp. 482–487 (2006) [13] Kao, B., Garcia-Molina, H.: Deadline assignment in a distributed soft real-time system. IEEE Transactions on Parallel and Distributed Systems 8, 1268–1274 (1997) [14] L´ opez, J.M., D´ıaz, J.L., Garc´ıa, D.F.: Utilization bounds for edf scheduling on real-time multiprocessor systems. Real-Time System 28(1) (2004) [15] Liu, J.W.S.: Real-Time Systems. Prentice-Hall, Englewood Cliffs (2000)

A Grid Based System for Closure Computation and Online Service Wing-Ning Li1 , Donald Hayes1 , Jonathan Baran1 , Cameron Porter2, and Tom Schweiger2 1

Department of Computer Science and Computer Engineering University of Arkansas, Fayetteville, AR 72701 USA [email protected] 2 Acxiom Corporation, 301 Ward Ave., Conway, AR 72032

Abstract. Record linkage deals with finding records that identify the same real world entity, such as an individual or a business, from a given file or set of files and has many applications. This problem is also referred to as the entity resolution or record recognition problem. To locate those records identifying the same real world entity, in principle, pairwise record analyses have to be performed among all records. Analytical operations are complex and take a lot of time. The number of such analyses is quadratic in terms of the number of records given and therefore is very time consuming. To reduce the number of pairwise record comparisons, blocking techniques are introduced to partition the records into blocks and records in each block are analyzed against one and another. One of the effective blocking methods is the closure approach. In this paper, we describe the design and implementation of a parallel and distributed closure prototype system running in an enterprise grid. The system can either produce all closures to a file in a batch fashion or run as a service where upon receiving a record it returns the closure of that record. Preliminary experiment indicates the approach is efficient and scalable.

1

Introduction

Determining records that represent the same real world entity is an important and challenging problem with many applications. For instance, it addresses data quality issues such as “data accuracy, redundancy, consistency, currency and completeness [1]. Ensuring data quality is becoming a critical issue that impacts organizational performance [2,3,4,5]. This problem is also referred to in the literature as record linkage problem [6], data cleaning problem [7], object identification problem [8], or entity resolution problem [9]. All these research efforts deal with the fundamental question of how to effectively identify record ”duplicates” when unique identifiers are unavailable or do not exist in records. The main idea is to rely on matching of other fields in records such as name, address, and so on. The set of fields chosen is application dependent and is often referred to as keys. The most basic application is to identify duplicates within a single file or between two files. In the single file situation, in principle, each record must be C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 79–89, 2010. c Springer-Verlag Berlin Heidelberg 2010 

80

W.-N. Li et al.

checked against every other record in the same file in order to find its duplicates. Similarly, in the two files scenario, each record in one file must be compared against every record in the other file. Both schemes amount to carrying out all pairwise analyses among records and have a time complexity that is quadratic to the number of records (the input size of an algorithm). Since analytical tools are complex and time consuming, each pairwise analysis takes much more time than that of a simple instruction. For large files having hundreds of millions to billions of records, the performance of such a scheme is unacceptable. To overcome the poor performance, the total number of pairwise record analyses must be reduced. To understand how this could be done, let us consider the case where records are in a single file. Conceptually each record may be viewed as being associated with a potential set of records from which to find its duplicates. Records not in the potential set are guaranteed not to be duplicates, and therefore need not be compared with. Hence, pairwise analyses are only needed between the records in each potential set. For a record, a straightforward way of defining its potential set is to let all other records in the file to be its potential set. This leads to the quadratic pairwise comparisons. Now imagine that a scheme exists that reduces the potential set from the whole file to a small fraction of that and this scheme can be implemented efficiently. What the scheme does is that it effectively partitions the records in the input file into many relatively small groups within which all pairwise record analyses are needed. The scheme that reduces the record pairs to be compared is called blocking in the literature [10]. Closure operation, of which the definition is given in the preliminary section, is one of the blocking schemes. Closure operation is not only useful in reducing the number of pairwise comparisons in single file record linkage applications but also in applications that involve multiple files. For instance, sometimes two files are provided as input, where file A has the distinct records (no duplicates) and the file B contains potential ”duplicate” records of file A, and the goal is to find the ”duplicate” records in file B for each record in file A. To apply the same idea introduced earlier, for each record in file A, we need to know its potential set or closure in file B. One way to solve this problem is to combine the two files into a single file and then perform the closure (blocking or partitioning) operation. Once this is done, each record in file A is associated with a closure (a potential set) that could contain some records from file B. Notice that records in file B are partitioned into closures and each closure could be associated with some record in file A. To determine its duplicates in file B, a record of file A only needs to be checked with those records of file B that are in the same closure in which the record of file A also belongs. Other record linkage applications that involve multiple files can use the closure idea to reduce the number of record analyses in a similar way. Since the primary objective of a blocking scheme such as closure is to reduce the time and space complexity of an overall process of record linkage, the blocking scheme must be implemented efficiently. To this end, it is necessary to investigate the question of how to design efficient algorithms and their implementations to realize any proposed blocking method. For the closure method,

A Grid Based System for Closure Computation and Online Service

81

efficient sequential algorithms are proposed and empirical studies are carried out in [1]. A Parallel and distributed algorithm is proposed and implemented in MPI [11,12,13]. A graph theoretic view of the closure problem is also introduced in [1,11]. This paper adopts the parallel and distributed closure algorithm proposed in [11] to a service based grid environment. In this paper, a grid based closure prototype system is reported. The system is developed in C++ with pthread and Corba and runs in an enterprise grid. The system can either produces a closure file in a batch fashion or runs as a service where upon receiving a record it returns the closure of that record. Preliminary experiment indicates the approach is efficient and scalable.

2

Preliminary and Background

The development of the prototype system is based on previous research efforts on the closure problems [1,11,12,13], where the reader will find more thorough discussions of the transitive closure problem and the proposed parallel and distributed algorithm that has been adopted and implemented in grid architecture. A brief description is provided later to make the paper self-contained. 2.1

The Closure Problem

Since the application of a blocking scheme such as Transitive Closure is basically the same for a single file or multiple files, in the sequel, a single file is assumed. A file contains a sequence of records. Each record contains a sequence of fields. Certain fields or some combinations of fields are chosen as keys. The selection of keys depends on the types of record linkage and could vary from applications to applications. Once keys are selected, two records are defined as directly related if for some key the two records have the same value. For example, if name and address fields are the keys, two records A and B with the same name values ”John Doe” and different address values ”723 main street” and ”123 main street”, respectively, are directly related because of same value ”John Doe” in the name key. Two records X and Y are defined as transitively related if there is a sequence of records of which each adjacent pair is directed related and X and Y are directly related to the first and last records in the sequence respectively. Let us expand the above example by introducing records C, where C has name value ”Jane Doe” and address value ”123 main street”. In the example, A and C are transitively related because A and B are directly related (through name key with value ”John Doe”) and B and C are directly related (through address key with value ”123 main street”). In this simple example, the sequence in the definition of transitively relatedness has only one record (record B) even though in reality the sequence may contain many records. Two records are defined as related if and only if either they are directly related or transitively related. Based on the relatedness just introduced, records in the input file are partitioned into group in the closure (transitive closure) computation, where all related records are in one group, which is called a closure . Typically, a closure is a set of record identifiers

82

W.-N. Li et al.

instead of a set of physical records with all the fields. Record identifier and record are used interchangeably in the sequel. Given a file of records and chosen keys as the input to the transitive closure problem, in the batch-processing mode, the output is a set of closures. In a service-oriented mode, a file of records and chosen keys are given once to set up the closure service. Afterwards each record in the file may be sent to the service. Upon receiving a record, the service returns the closure associated with the record. The design and implementation of the closure software to be presented support both modes of operations. Even though the transitive closure problem is defined in terms of a file of records containing many fields of which some are chosen as keys, the input to the prototype system is not a file of records. Instead it is a file of pairs of record identifiers. Each pair of record identifiers signifies that the corresponding records are directly related. For any two directly related records at least one pair of the corresponding record identifiers exists in the input file. The reason for the change of input is as follows. It has been observed in [11,13], the process of parallel and distributed computation of closure may be broken into two steps: in step one all directly related pair of records are computed; in step two upon getting the result from step one all closures are computed. Since step one can be computed efficiently using a parallel and distributed processing algorithm that is scalable [11,13], the focus of this research effort aims at step two. For that reason, the input to the prototype is a file of pairs of record identifiers. Conceptually the two parallel and distributed solutions of steps one and two can be easily combined into a single system where the input is a file of records. 2.2

The Basic Parallel and Distributed Algorithm

The basic parallel and distributed algorithmic idea and algorithm that the prototype uses are provided in [11,12,13,14]. The main ideas are recapped as follows. First hashing is used to evenly assign records to processors conceptually. A record identifier is an alphanumeric string. The string’s hash value modular of the number of processors is the ID of the processor that owns the record. Records owned to a processor are called local records with respect to that processor. Once this is done, record pairs (directly related records) in the input file are classified with respect to each processor as local pairs, global pairs, and irrelevant pairs. A record pair is said to be a local pair of a processor if both records in the pair are local to the processor, a global pair if only one of the records is local to the processor, and an irrelevant pair if none are local to the processor. In parallel, processors read the record pairs from the input file, where local pairs are processed immediately using disjoint set find and union data structures [15](the two records are directly related and should be in the same closure or the same set), global pairs are stored so they may be processed after all the pairs are read, and irrelevant pairs are ignored (note an irrelevant pair to one processor is always a local pair or a global pair to some other processor). The disjoint set data structures maintain that the local records that belong to the same closure

A Grid Based System for Closure Computation and Online Service

83

are grouped together as a set, called a local cluster. Once the reading of the input file is finished, in parallel each processor goes through its global pairs to generate new local pairs and new global pairs for other processors. Then in parallel the processors interchange new local pairs and global pairs among themselves. The new local edges are processed first using the disjoint set data structures that allow more local records in a closure to be grouped together (combining local clusters determined in the previous iteration together). Once the new local pairs are processed, the new global pairs are examined to generate new local and global pairs for the next phase in the iteration. After that global pairs are collapsed or reduced. Each phase is synchronized and within a phase communication and computation among the processors are asynchronous or concurrent. The number of iteration is logarithmically related to the number of processors [11]. Note that after processing the local pairs, no local pair is stored. And after processing the global pairs, each local cluster has at most one global pair from its processor to another processor. The rest of the global pairs are thrown away. The existence of global pairs indicates that records in a closure are assigned to different processors by the hashing scheme. At the end, all local records belonging to a closure form a single local cluster. And local clusters in different processors are linked to each other by global pairs to establish a closure. The disjoint sets (local clusters) and the associated global pair structures allow the generation of closure file and support the closure query service. The above algorithm is realized in the prototype of which the design is given next. For a more detailed description and examples of the algorithm the reader is referred to [11,12,13,14]. One more thing should be noted before describing the design. The efficient implementation of disjoint set find and union requires that integer values be the members of the set [1,15]. On the other hand record identifiers are alphanumeric strings in general, and this is particularly true for the real data files where record identifiers contain both letters and digits. To bridge the gap, a mapping scheme is needed that maps each alphanumeric string to an integer. The mapping scheme is also used to convert pairs of strings to pair of integers. Once the conversion is done, the computation of closures is based on integer pairs, which have much smaller memory foot print and reduce computation time as well as communication time of the parallel and distributed algorithm outlined earlier. An empirical study of the time reduction in computation and communication by using integer pairs is conducted in [11]. Even though the mapping operation is expensive, as shown in experimentation (Setup pair), the time and space reduction it brings to the overall computation outweighs its initial cost.

3

Architecture Design

The system runs on a set of grid nodes, which may be thought of as processors in a parallel computer conceptually. Multiple threads are used to carry out the processing in each grid node. The system uses Client-Server architecture to implement the parallel and distributed closure algorithm. In particular, the architecture addresses the no blocking communication between processors so that

84

W.-N. Li et al.

computation and communication can be concurrent within a phase of the algorithm. The client and server are separate threads in the same grid node so that computation and communication are carried out concurrently. The current design uses one thread for the client and one thread for the server. Each grid node has a server, which is a remote object to clients running on other grid nodes. The server provides remote procedure calls for clients. These procedures allow a remote client to send the new local and global pairs derived in each iteration (push the data to grid node running the server) and to query the string to integer mapping information of the local records of the gird node. Each grid node spawns a client thread and keeps the main thread running as a server. The client thread does the majority of processing and makes remote procedure calls to serves running on other grid nodes. The server thread handles incoming requests from remote clients and delivers the data to the client thread via shared memory between the two threads. Each gird node has three major data structures for mapping between local records and integers from 1 to n (n is the total number of records), for supporting disjoint set find and union, and for relating global pairs and local clusters. The current design delegates the mapping structures to the server thread, and the other two structures to the client thread. It is conceivable that the mapping operation may be separated as a grid service to reduce the memory requirement of each grid node in closure computation. The current design makes that transition easier to implement. For batch processing, the system has three phases to go through: the assignment phase, the closure computation phase, and output phase. For service processing, the only change is to have the output phase replaced by a repeated query-processing phase. In the assignment phase, the local records of each grid node are determined and mapped to integers. At the end of the assignment phase, a single communication occurs in which all grid nodes communicate the largest integer used in the local map. The information is used to establish a global map consisting of all the local maps (each grid node adds an offset to its local map). The global map is used to map string pairs to integer pairs. The information is needed so that remote records (an integer) can be mapped back to the grid node to which they are assigned. The client thread of each grid node sends the largest integer used information to all servers (server threads) on other grid nodes. Once a grid node receives this communication from all other grid nodes, it has the assignment information and the next phase begins. During closure computation phase, in each grid node, the client thread goes through the record pairs (first local pairs then global pairs) of the current iteration by combining local clusters, generating local and global pairs for other grid nodes, and sending new pairs generated to other grid nodes (remote servers) for the next iteration. The server thread receives record pairs from remote clients and gathers all the pairs received so that pairs are delivered when the client is ready for the next iteration.

A Grid Based System for Closure Computation and Online Service

85

The problem of handling communication between multiple, simultaneously running programs in each grid node is solved by using multiple threads. The client thread is used for processing while the server thread is used to handle incoming messages. The server thread handles incoming messages by storing the data (pairs) in a temporary buffer and then writing to a shared data structure at the end of a closure computation phase. This allows the client thread to continue processing while information is being received. This threading approach realizes a significant practical improvement from the single threaded MPI implementation in [11]. The communication between grid nodes is logically a fully connected network; each grid node communicates with every other grid node. More detailed discussions of the design are given in [14].

4

System Implementation

The software is implemented in C++ programming language using a multithreaded client-server architecture. Each grid node runs the same copy of the program. POSIX Pthread library is used to implement multithreading and CORBA library is used to implement remote objects allowing communication between servers and clients running on different grid nodes. Each grid node has a client thread that does the bulk of the processing. A separate server thread is started in each grid node in order to asynchronously receive data from remote grid nodes. The specific communication API used is CORBA. CORBA was selected in order to allow the software to conform to the current grid architecture of the project sponsor and to be more easily adapted to a service-oriented architecture in the future (for instance closure query service). The total number of grid nodes and the ID of the current grid node must be provided to the program as a command line parameter. A potential improvement would be to move this information to a configuration file. In either cases, the number of running grid nodes is known. The total number of grid nodes and the ID of the current grid node running the program are required by the program for the implemented algorithm to perform correctly. MD5 is used as the hashing function in the assignment phase. The mapping structure is implemented using C++ map class in the standard template library. The mapping object maps the alphanumeric strings (record identifiers) to integers. Records may be inserted into the object. After all initial records are inserted, the object must be finalized. This finalization is done for performance reasons, as the object is read only after finalization and does not require further locking. The local server reads this object in order to map incoming global records from strings to integers. During the output phase, a reverse map is also constructed to allow the string records to be output after completion of the entire algorithm or to support closure service. A two-pass scan of the input file is used in this particular implementation. The first scan assigns records to grid nodes and establishes the mapping object in each grid node. The second scan converts string pairs from the input file to an integer pairs. The mapping object of each grid node, which is established in

86

W.-N. Li et al.

the first scan, is used to realize the conversion step. To be more specific, the local mapping object will be consulted twice to convert a local pair. The local mapping object and a remote mapping object are consulted to convert a global pair. Since consulting remote mapping objects involve remote procedure calls, bulk lookup was implemented and used to improve efficiency. In bulk lookup an array of strings is provided as input and an array of integers is returned as output. More detailed discussions of the implementation are given in [14].

5

Experimental Results

Preliminary tests are conducted for the system. The tests were run on dual core Intel Pentium III 1.266 Ghz dual core machines with 4 GB of memory using 100 Mbit Ethernet for grid node connection. The operating system was CentOS 4.5 with the 2.6.9-55.0.2.ELsmp linux kernel. Two grid nodes were run on each machine. In Tables 1 to 3, each column except the first one identifies a grid node and shows the resources used by the grid node, in various steps in the computation. The second row shows the peak memory usage. For example, in Table 1, memory at 8M means the global pairs array was allocated to contain 8 million record pairs and node 0 peak memory usage is 276 MB. As mentioned in [14], CORBA does not define or make available its memory allocation and deallocation routines. As such, the memory usage is more indicative of the implementation than of the algorithm. All times are in seconds. ”Max pair” is the largest number of global pairs that existed at any point in the processing, though this usually occurred during the first pass. ”Scan time” gives the time to complete the initial pass of reading the file. ”Setup pair” is the time to convert string pairs to integer pairs and to process the initial local pairs. ”Compute Closure” is the time to process initial global pairs, to generate and send new local and global pairs in all iterations. Table 1. Synthetic data of 7.3 million pairs Node ID

0

1

2

3

4

5

Memory at 8M Max Pair Scan Time Pair Set-up Computer Closure Total Time

276 MB 2,130,875 14 27 11 52

274 MB 2,044,955 13 28 10 51

281 MB 2,204,346 14 28 11 53

268 MB 1,837,045 16 25 14 55

280 MB 2,317,816 16 30 9 55

271 MB 1,968,969 16 27 12 55

Table 1 contains the results of a synthetic data set running on six grid nodes. Tables 2 and 3 contain the results from running a real data set on two and six grid nodes. The average runtime on two grid nodes is 423 seconds, compared to 345 seconds on six grid nodes. The memory usage for two grid nodes is 1.8 GB

A Grid Based System for Closure Computation and Online Service

87

Table 2. Real data of 26.9 million pairs on 2 nodes Node ID

0

1

Memory at 14M Max Pair Scan Time Pair Set-up Computer Closure Total Time

1.8 GB 13,549,308 159 225 39 423

1.8 GB 13,549,308 159 225 39 423

of memory, but only 1.1GB of memory when using six grid nodes. This shows the speed and memory benefits of parallelism. Table 3. Real data of 26.9 million pairs on 6 nodes Node ID

0

1

2

3

4

5

Memory at 14M Max Pair Scan Time Pair Set-up Computer Closure Total Time

1.1 GB 7,482,705 101 179 62 342

1.1 GB 7,481,988 101 178 66 345

1.1 GB 7,488,032 102 173 70 345

1.1 GB 7,474,246 102 179 67 348

1.1 GB 7,477,869 98 180 64 342

1.1 GB 7,489,096 99 183 64 346

Note that the large portion of the time is related to alphanumeric string pair to integer pair conversion process in all tests. Note also that in the two grid node test, both grid nodes are in the same machine and the measured time (communication portion) could be shorter than that of the two grid nodes running on different machines. The closure query service uses the results and structures of the closure computation and has two major steps. Upon receiving a record identifier for closure service, the service grid node uses the hash function to determine which grid node owns the record as its local record. Once the grid node is determined, the service grid node performs an initial fetch by sending the record identifier to the grid node, which returns a list of processor ID and remote key pairs. The processor ID tells us the processor that has some of the records in the closure as its local records. The remote key is the corresponding mapped integer value of one local record in the closure. The list is derived from the mapping object and the global pairs of the local cluster to which the queried record identifier belongs. Next for each pair in the list, the service grid node sends the remote key to the processor, identified by the processor ID, which returns all the record identifiers of the local cluster to which the remote key belongs. The returned information is derived from the disjoint set and mapping object structures. When every queried processor returns, the service grid node has all the records in the closure corresponding to the query record identifier and performs a return.

88

W.-N. Li et al.

The closure query service is evaluated by a synthetic data set; the same data set that table 1 is based. It has about two millions of records and 7.3 million pairs. This data was processed by the transitive closure solution, and subsequently queried by the service addition to select a known closure of 1,472 members for the purpose of timing comparison. The timing method was composed of repeating a request for an entry 1,000 times and recording the average request timer for the fetch process. This was performed for multiple members of the closure, and using one to six processors to maintain the distributed closure. The process was timed for both the initial process fetch that returns the processor id and the remote key pairs for a given record identifier, and for the subsequent queries of transitive closure components. The initial fetch was noted as negligible, as it did not require more than one value search into its local data, and returned at most 2n integers, with n being the number of processes (nodes in the grid) in the system. The time spent for a query to each processor was averaged for the 1000 attempts, and then grouped by processor count to record the average amount of time that was spent on each processor. Table 4. Closure queries to 2 million records (1472 records retrieved) Nodes to maintain closure

1

2

3

4

5

6

Average fetch time 0.0420 0.0730 0.0720 0.0680 0.0560 0.608 0.0420 0.0365 0.0240 0.0170 0.0112 0.0101 Average busy time per node 0.0055 0.0125 0.0070 0.0058 0.0011 Reduced work time (tn − tn−1 ) −

As shown in Table 4, independent to the number of grid nodes used, the average fetch time remains more or less the same. This is expected as the service node runs a sequential process, which is the bottleneck. However, the average busy time per process decreases as more nodes are used. It suggests that nodes in the closure solution are not very busy when more nodes are use. It seems that this could support multiple service nodes to query the closure solution in parallel and maintain average fetch time for each service node.

6

Conclusions

A parallel and distributed transitive closure prototype system running in an enterprise grid was developed. The system could speed up various record linkage applications. The design and implementation is based on a novel parallel and distributed closure algorithm, multithreading, and Corba-based client and server model. The system can either produce all closures to a file in a batch fashion or run as a service where upon receiving a record it returns the closure of that record. Preliminary experiment results are encouraging and seem to indicate the approach is efficient and scalable.

A Grid Based System for Closure Computation and Online Service

89

A more thorough evaluation of the prototype is needed in the future so it may be fine tuned and enhanced for efficiency and scalability. Acknowledgment. This research was supported in part by Acxiom Corporation through the Acxiom Laboratory for Applied Research.

References 1. Li, W., Zhang, J., Bheemavaram, R.: Efficient algorithms for grouping data to improve data quality. In: Proc. 2006 International Conference on information and knowledge engineering, Las Vegas, pp. 149–154 (2006) 2. Ballou, D., Wang, H., Pazer, G.: Modeling information manufacturing systems to determining information product quality. Management Science 44(4), 462–484 (1998) 3. Ballou, D.: Enhancing data quality in data warehousing environment. Comm. ACM 42(1), 73–78 (1999) 4. Delone, W., Mclean, E.: Information systems success: The quest for the independent variable. Information Systems Research 3(1), 60–95 (1992) 5. Redman, T.: The impact of poor data quality on the typical enterprise. Comm. ACM 41(2), 79–82 (1998) 6. Fellegi, I., Sunter, A.: A theory for record linkage. Journal of the American Statistical Association 64, 1183–1210 (1969) 7. Do, H., Rahm, E.: Coma - a system for flexible combination of schema matching approaches. In: Proc. ACM SIGKDD ’02 (2002) 8. Tejada, S., Knoblock, C.A., Minton, S.: Learning domain-independent string transformation for high accuracy object identification. In: Proc. Very Large Data Bases 2002, pp. 610–621 (2002) 9. Benjelloun, O., Garcia-Molina, H., Su, Q., Widom, J.: Swoosh: A generic approach to entity resolution. Tech. rep., Stanford University (2005) 10. Baxter, R., Christen, P., Churches, T.: A comparison of fast blocking methods for record linkage. In: ACM SIGKDD 2003 Workshop on Data Cleaning, Record Linkage, and Object Consolidation, Washington, DC, pp. 25–27 (2003) 11. Bheemavaram, R.: Parallel and distributed grouping algorithms for finding related records of huge data sets on cluster grid. M. s. thesis, University of Akansas (2006) 12. Li, W., et al.: Paralle and distributed grouping algorithms for finding related records of huge data sets on cluster grids. In: Proc. ALAR conference on Applied Research in Information Technology, Conway (2007) 13. Li, W., Bheemavaram, R., Zhang, J.: Transitive closure of data records. In: Chan, Y., Talburt, J., Talley, T. (eds.) Data Engineering: Mining, information and Intelligence, pp. 39–74. Springer, New York (2010) 14. Hayes, D.: A corba-based distributed and multithreaded algorithm finding related records in a large data set. M. s. thesis, University of Akansas, Fayetteville, Arkansas (2008) 15. Cormen, T., Leiserson, C., Rivest, R., Stein, C.: Introduction To Algorithms. McGraw-Hill Book Company, Cambridge (2002)

A Multiple Grid Resource Broker with Monitoring and Information Services∗ Chao-Tung Yang∗∗, Wen-Jen Hu, and Bo-Han Chen Department of Computer Science, Tunghai University, Taichung, 40704, Taiwan (ROC) [email protected]

Abstract. Grid computing is now in widespread use, which integrates geographical computing resources across multiple virtual organizations to achieve high performance computing. We build a resource broker on multiple grid environments, which integrates a number of single grids from different virtual organizations without the limit of cross-organization. We can efficiently use the multiple grid resources to avoid the waste of resources. In addition, we proposed a Multi Grid Resource Selection Strategy for the resource broker to select the better allocation of resources before submitting job for avoiding network congestion caused by the decrement of performance. Keywords: Multi-Grid, Resource Broker, MGRSS.

1 Introduction In the Grid environment, applications make use of shared Grid resources to improve performance. The target function usually depends on many parameters, e.g., the scheduling strategies, the configurations of machines and links, the workloads in a Grid, the degree of data replication, etc [3, 4]. In this work, we examine how those parameters may affect performance. We choose an application’s overall response time as the object function and focus on dynamically scheduling independent tasks. We define the job, the scheduler, and the performance model of a Grid site and conduct experiments on the Multi-Grid platform [1]. We propose a multi-grid resource selection strategy, called “MGRSS”. The strategy helps user to select the better performance of machines in order to shorten the execution time of programs, and furthermore we adjusted each grid user has the quota of resources. Experimental results show that MGRSS exhibits a better performance than other strategies [7].

2 System Design and Implementation 2.1 Resource Broker The system architecture of the resource broker is shown in Fig. 1. Users could easily make use of our resource broker through a common Grid portal [5, 6, 7-11]. The ∗

This work is supported in part by National Science Council, Taiwan R.O.C., under grants no. NSC 96-2221-E-029-019-MY3 and NSC 98-2218-E-007-005. ∗∗ Corresponding author. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 90–99, 2010. © Springer-Verlag Berlin Heidelberg 2010

A Multiple Grid Resource Broker with Monitoring and Information Services

91

primary task of the Resource Broker is to compare requests of users and resource information provided by the Information Service [2, 13]. After choosing the appropriate job assignment scheme, Grid resources are assigned and the Scheduler is responsible to submit the job. The results are collected and returned to Resource Broker. Then, Resource Broker records results of the execution in the database of Information Center through the Agent of Information Service. The user can query the results from the Grid portal. Our resource broker architecture includes the four layers, web portal, resource brokering subsystem, multi-grid manager center, and multi-grid resource. The multi-grid resource consists of many single grid environments

Fig. 1. Resource Broker system architecture

2.2 Cross Grid Authentication Service The Globus Toolkit authentication is issued by the certificate of GSI. Each user and service is certificated to identify and authenticate the trust between users or services [12, 17]. If two parties have the certificate and both of them trust the CAs, then these two parties could trust each other. This is known as the mutual authentication. A subject name, which identifies the person or object that the certificate represents. The cross grid authentication service manages the several certificates and the subject of grid, and we know these messages from the registered information via the web portal.

92

C.-T. Yang, W.-J. Hu, and B.-H. Chen

All of nodes on the multi-grid environment should setup the cross grid service tool, which written in the shell script program, as shown in Fig. 2. The cross grid service tool contains some procedures to regular automatic updates with a multi-grid manager center, such as the IP and domain of host list, the certificates, and the subject. Any single grid user on our multi-grid environment makes use of the multi-grid resource through the tool to update with a multi-grid manager center. The primary task of cross grid information is to gather the IDL file, which contains the several attributes of hosts on the grid, the detail is described in next section.

Fig. 2. Multi-Grid Manager Center

2.3 Cross Grid Information Service The CGIS consists of three layers: the Core Service Layer, the Translator Layer, and the Resource Layer, as shown in Fig. 3. The Core Service Layer contains the Agent, Filter, Getter and Setter, and Gather. These components are installed in every grid environment for information gathering and maintenance. The Translator Layer supports a variety of format conversion monitor tools, such as Ganglia, Cacti, and MDS. The information in different monitor tools is transformed into the IDL format. The Resource Layer describes resource information from different grid environments. In this study, it includes Tiger Grid, Medical Grid, and Bio Grid [14, 15, 16]. We define a new information format to translate different formats. The proposed format is called the IDL (Information Describing Language). It is responsible to exchange and translate resource information among grids, and to perform the posttransfer filtering to ensure that only necessary information is passed to clients, end users, and software components.

A Multiple Grid Resource Broker with Monitoring and Information Services

93

Fig. 3. Cross Grid Information Service architecture

2.4 Web Portal The user could execute a parallel program via the resource broker and choose the required for the execution of the host after submitting the job, as shown in Fig. 4, and get the result form the Job Monitor Service. The page shows the many detail messages in the execute time, such as scheduling log, machine list, result message, running log, debug message, and turnaround time, as shown in Fig. 5. We enhance the graphical display of monitor function, uses JFreechart [18] tool draw more meaningful graphics, including Job Monitor, Login Info, and Utilization of Resource. JFreeChart is a free 100% Java chart library that makes it easy for developers to display professional quality charts in their applications.

Fig. 4. Job Monitor

94

C.-T. Yang, W.-J. Hu, and B.-H. Chen

Fig. 5. Job Monitor

3 Multi-Grid Resource Selection Strategy 3.1 Parameters and MGRSS Algorithm In this section, the parameters used in the algorithm are listed and explained in the following: y

y y y y y y y

jobi: The ith job dequeued from the job queue, where i = 1~n. The job contains some information such as job name, program name, program arguments, input data, and number of processors to run on. The program is usually a parallel program written by MPI library and compiled by MPICH-G2 that is a Gridenabled implementation of the MPI standard. The Resource Broker will allocate resources according to the information provided by the job. NPmGrid : The total number of processors on the multiple grid NPlGrid : The total number of processors on the local grid NPmf : The total number of processors with idle status on the multiple grid NPlf : The total number of processors with idle status in the local grid NPmax : The maximum processors on the multiple grid NPvalid : The available processors on the multiple grid NPreq: The processors used for executing jobi, if the Resource Broker dispatches the jobi successfully, the resources distributed over several Nodes or Sites used for jobi will be locked for a while until the jobi finishes.

A Multiple Grid Resource Broker with Monitoring and Information Services

y y

Smax : The higher scores of the machines NPvalid : How many number of processor user can apply. •

y

95

·

·

(1)

Loadi: It is a sum of the Load1min, Load5min, and Load15min of Nodei, the expression of Loadi is shown as follows: 3

2

,

6

(2)

where Load1min is the average of load in one minute, Load15min is the average of load in five minute, and Load15min is the average of load in fifteen minute. For each parameter has different proportion value, the time is closer, the value is higher. y PEi: The computing performance efficiency of host, the expression is shown as follows: (3)

,

where HPLi is the benchmarking value of host obtained by the approach of benchmarking. y Flowi : The average network flow of host, the expression is shown as follows: ∑

_

_

,

(4)

where Byte_in and Byte_out get from the IDL, estimated the network flow influence through these two parameters. We also considered that same domain of other machines will affect the flow of the entire site, where N is the number of nodes in same domain. Therefore we use the average of all nodes in same domain. Finally, the Flowi mean Megabyte per second after divided by square of 1024. y TPi : Total performance power, the expression is shown as follows: ·

,

(5)

where β is the effect ratio used to regulate the percentage of PE and Flow. The main policy of MRGSS, a single grid select resource itself when that possesses sufficient resources, otherwise selects the multiple grid resource. The number of available resources is variation, which according to their own resources of local grid. 3.2 MGRSS Flowchart Job submission is the most important part of Resource Broker. Among these resources, choosing the appropriate resource to do assignment is a very important topic, because the fine resource selection strategy can shorten the execution time of jobs, and upgrade the overall performance on the Multi-Grid environment. For above purposes, we provide a Multi-Grid Resource Selection Strategy, called MGRSS, as shown in Fig. 6.

96

C.-T. Yang, W.-J. Hu, and B.-H. Chen

Fig. 6. Multi-Grid resource selection strategy flowchart

First, a job is dequeued from the job queue for scheduling. Second, it gets the latest information from an IDL file of multi-grid information server to determine, whether the number of processors of requests is smaller than the total number of processors of the Multi-Grid. If it is false, scheduler discards jobs. If it is true, scheduler determining use the single grid or multi-grid strategy by the processors of request. For the single grid strategy, the processors of requests should be smaller than the free processors of a local grid; otherwise, that enqueue the job and waiting for five minutes. Our schedule will resubmit the job when the waiting time is exceeded five minutes. If the number of processors of requests is still smaller than that of the free processors of the local grid, we adopt the multiple grid strategy. For the multiple grid strategy, if the number of the processors of requests is larger than that of the valid processors of multiple grids, then scheduler discard job. Otherwise if the processors of request smaller than free processors of multiple grid, then we adopt the multiple grid strategy.

4 Experimental Results In this section, we have the four experimental results. Among the first two experiment respectively measurement the execute time for different specific program which use the different metric in five strategy environment. In the third experiment, we execute the entire parallel programs. The last experiment randomly selected twenty different types of parallel programs and uses the different number of processors to execute. The above-mentioned results are all order to compare the execution time between different resource selection strategies. We calculate the value of performance and network for each machine, and according the score to sort the list of machines which represents the resource selection priority and previous section mention the MGRSS algorithm. We divided MGRSS into three levels from level one to level three by different weighted valueβ. The maximum weighted value of the network is MGRSS3(β=0.7);

A Multiple Grid Ressource Broker with Monitoring and Information Services

97

the medium weighted valuee(β=0.5); the maximum weighted value of the performaance is MGRSS1(β=0.3). First experiment shows the total time of bucket-sort MPI programs total in ten times and the program parrameter size using the 512, 1024, 4096 respectively. T The bucket-sort MPI program uses u a small number of resources. In this case, the raapid transmission normally will decide the time of program execution. The MGRSS2 and MGRSS3 have better strateegies in this experiment, because that has more proporttion of network in three level off MGRSS algorithm. Experiment results are shown in Fiig.7 and 8.

Fig. 7. Result fxor Bucketsorrt_MPI in different parameters sequence

Fig. 8. Result for Bucketsort_MPI in diffeerent strategies sequence

Second experiment show ws the total time of matrix multiplication MPI program ms in ten times, and the program parameter size using the 256, 512, 1024 respectively. T The matrix multiplication requiires a large amount of computing power and network, but the network flow is still mo ore proportion. So the MGRSS3 spent the less time execcuting the jobs. Experiment ressults are shown in Fig. 9 and Fig. 10.

Fig. 9. Result for Mmd_M MPI in different parameters sequence

Fig. 10. Result for Mmd _MPI in different strategies sequence

We execute the nine paarallel programs which are described in previous sectiion. Similarly, we compared the t MGRSS and other differences strategy again. T The different characteristics of the program, so using the different strategies will prodduce different results between th hese situations. Overall, three levels of MGRSS strateggies

98

C.-T. Yang, W.-J. Hu u, and B.-H. Chen

are still better than other strrategies. Experiment results are shown in Fig. 11. The last experiment we randomly selected s twenty different types and parameter of paraallel programs to execute the diifferent strategies in twenty times, as shown in Fig. 122. In some cases, the speed only y and the network only are better than the MGRSS, but MGRSS is better than otherr two strategies in the most of case.

Fig. 11. Result for all mp pi programs

ms Fig. 12. Result for variety of mpi program

5 Conclusion and Fu uture Work In this paper, we propose a multi-grid resource selection strategy, which helps userrs to select the better performan nce of machines in order to shorten the execution timee of programs, and furthermoree we adjusted each grid user has the quota of resourcces. Experimental results show that MGRSS exhibits a better performance than that applied other strategies. Overrall, we construct a multi-grid platform, which integraates four single grid systems inccluding the Tiger Grid, the Medical Grid, the Bio Grid, and GCA Grid and, furthermorre, design and implement a Multi-Grid Resource Brooker with multi-grid resource selection strategy. In the future, we will continue to enhaance and improve the function of resource brokers and the multi-grid resource selecttion strategy.

References 1. Foster, Kesselman, C.: Globus: G A Metacomputing Infrastructure Toolkit. Internatioonal Journal of Supercomputerr Applications 11, 115–128 (1997) 2. Czajkowski, K., Fitzgeraald, S., Foster, I., Kesselman, C.: Grid Information Servicess for Distributed Resource Sharing. In: Proceedings of the Tenth IEEE International Sym mposium on High-Performancce Distributed Computing, pp. 181–194 (2001) 3. Foster, Karonis, N.: A Grrid-Enabled MPI: Message Passing in Heterogeneous Distribuuted Computing Systems. In: Proceedings P of 1998 Supercomputing Conference, p. 46 (19988) 4. Tang, J., Zhang, M.: An Agent-based Peer-to-Peer Grid Computing Architecture. In: F First International Conference on Semantics, Knowledge and Grid, SKG ’05, p.57 (2005) 5. Aloisio, G., Cafaro, M.: Web-based access to the Grid using the Grid Resource Brooker mputation: Practice and Experience 14, 1145–1160 (2002) portal. Concurrency Com

A Multiple Grid Resource Broker with Monitoring and Information Services

99

6. Krauter, K., Buyya, R., Maheswaran, M.: A taxonomy and survey of grid resource management systems for distributed computing. Software Practice and Experience 32, 135– 164 (2002) 7. Yang, C.T., Lai, C.L., Shih, P.C., Li, K.C.: A Resource Broker for Computing Nodes Selection in Grid Environments. In: Jin, H., Pan, Y., Xiao, N., Sun, J. (eds.) GCC 2004. LNCS, vol. 3251, pp. 931–934. Springer, Heidelberg (2004) 8. Yang, C.T., Shih, P.C., Li, K.C.: A high-performance computational resource broker for grid computing environments. In: 19th International Conference on Advanced Information Networking and Applications, AINA 2005, vol. 2, pp. 333–336 (2005) 9. Yang, C.T., Li, K.C., Chiang, W.C., Shih, P.C.: Design and Implementation of TIGER Grid: an Integrated Metropolitan-Scale Grid Environment. In: Proceedings of the 6th IEEE International Conference on PDCAT’05, pp. 518–520 (2005) 10. Yang, C.T., Lin, C.F., Chen, S.Y.: A Workflow-based Computational Resource Broker with Information Monitoring in Grids. In: Proceedings of Fifth International Conference on Grid and Cooperative Computing (GCC’06), pp. 199–206 (2006) 11. Yang, C.T., Chen, S.Y., Chen, T.T.: A Grid Resource Broker with Network BandwidthAware Job Scheduling for Computational Grids. In: Cérin, C., Li, K.-C. (eds.) GPC 2007. LNCS, vol. 4459, pp. 1–12. Springer, Heidelberg (2007) 12. Ian, F., Carl, K.: Globus: A Metacomputing Infrastructure Toolkit. International Journal of Supercomputer Applications 11, 115–128 (1997) 13. Kim, D.H., Kang, K.W.: Design and Implementation of Integrated Information System for Monitoring Resources in Grid Computing. In: Computer Supported Cooperative Work in Design, pp. 1–6 (2006) 14. Tiger Grid, http://gamma2.hpc.csie.thu.edu.tw/ganglia/ 15. Medical Grid, http://eta1.hpc.csie.thu.edu.tw/ganglia/ 16. Bio Grid, http://140.128.98.25/ganglia/ 17. Globus, http://www.globus.org/ 18. JFreeChart, http://www.jfree.org

Design Methodologies of Workload Management through Code Migration in Distributed Desktop Computing Grids Makoto Yoshida and Kazumine Kojima Information and Computer Engineering, Okayama University of Science, 1-1 Ridai-cho, Okayama, 700-0005, Japan {yoshida,kojima}@ice.ous.ac.jp

Abstract. Large scale loosely coupled PCs can organize clusters and form desktop computing grids on sharing each processing power; power of PCs, transaction distributions and load balancing characterize the performance of the computing grids. This paper describes the design methodologies of workload management in distributed desktop computing grids. Based on the prototype experiment, several simulations were performed; several centralized and decentralized algorithms for location policy were examined, and the design methodologies for distributed desktop computing grids are derived from the simulation results. The methodologies for domains, language and control algorithms for computing grids are described. The language for distributed desktop computing is designed to accomplish the design methodologies.

1 Introduction Grid system is a type of distributed systems, and it can be categorized into three classes: computing grid, data grid, and service grid [3, 6]. This paper deals with the computing grid. The purpose of the computing grid in this paper is to use the idle computing powers effectively; those scattered around among one or more organizations. Several hundreds of PSs, such as Note PC, Desktop PC and WS, are assumed. User jobs can be executed on the local or remote computer systems through migration [1]. To work the desktop grids efficiently, the transfer policy that determines the migration decision, and the location policy that determines the site to be migrated, must be provided adequately depending upon the application environment [8]. However, little work has been associated with their application environment. We need some methodologies to design the workload of transactions in the application environment. In the previous paper [2, 7], we compared and evaluated several centralized and decentralized algorithms for location policy. We summarize these results in this paper with some new results for scalability, and design methodologies for desktop computing grids derived from several simulation results are described. The language for distributed computing is designed to accomplish the methodologies derived. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 100–111, 2010. © Springer-Verlag Berlin Heidelberg 2010

Design Methodologies of Workload Management through Code Migration

101

Distributed Computing

Grid System

Data Grid

Computing Grid

Service Grid

Distributed File Computing Sharing

Platform Collaboration

Load Balancing

Scheduling

Transfer Policy State Migration Table

P2P System

Location Policy

Active Method Pull Push

Evaluation Method Centralized Decentralized

Static Dynamic

Non Migration CPU based Workload based Working Buffer based

Fig. 1. Taxonomy of Distributed Computing System

The rest of paper is organized as follows. Section 2 and 3 summarizes the simulation results. Section 2 describes the grid computing model and the prototype experiment, and section 3 describes the results of the simulations. Section 4 describes some design methodologies. The language designed for workload sharing in distributed desktop computing is also introduced. Section 5 concludes with the future works.

2 Computing Model Figure 1 shows the taxonomy of the distributed computing systems [4, 5]. The shadow area is the objective area in this paper. The transfer policy and the location policy must be adequately defined for computing grids [8]. The transfer policy in our model is determined by the migration state table, which was obtained by implementing and evaluating the prototype experiment [2, 7]. To satisfy the location policy, the algorithm to detect the appropriate site for object migration must be provided [8]. Many load balancing algorithms have been proposed; pull and push approach for active method, and several algorithms for evaluation method exist, as is shown in Figure 1 [3, 6]. We compared several centralized algorithms [7], and the decentralized algorithm [2], by varying transaction patterns, and derived some design methodologies. 2.1 Centralized Algorithms There exists several centralized location policy algorithms [7]: the random based selection algorithm, the CPU power based selection algorithm, the traffic based selection algorithm and the working buffer based selection algorithm. The random based algorithm (RB) selects the site randomly from available sites.

102

M. Yoshida and K. Kojima

SITE i (Si) SENDER Process Send (MigrateRequest) to all_member; Receive(Ack_with_Timestamp Txo) frommembers; /* Txo stands for the timestamp T for object o on site x/ Select the site_X that has the earliest timestamp Txo from Received messages;

Move_Transaction with Txo) to the members; Receive(Result); Send (

RECEIVER Process if Receive(MigrateRequest) then Statecheck(); /* Check Migration State Table/ if state=idle then Reserve&Enque_Request with timestamp Tio; /*Tio stands for the Timestamp T for object on site i/ Return(Ack_with_TimestampTio);

if

Receive(Move_Transaction) then /*Compare the timestamps/ if (It has sent the Ack with Timestamp Tio) & ( Tio= Txo) Then Migrate_Code & Execute_Code;

Return (Result); Else

Cancel&Deque the request that has the timestamp Tio;

Fig. 2. Decentralized Algorithm (DC Algorithm)

The CPU based algorithm (CB) selects the site by the order of CPU powers, and the traffic based algorithm (TB) selects the site by the order of workloads at each site. The working buffer algorithm (WB) assigns the number of transactions affordable at each site to be migrated the transactions from remote sites, and the algorithm selects the site by the largest value of the numbers. If the site is selected for migration, the number is decreased. The number which defines the affordable power of processing for other sites must be assigned properly [7]. These location policy algorithms are compared to the non-migration policy (NM). 2.2 Decentralized Algorithm Figure 2 is the autonomous decentralized algorithm adopted at each site [2]. Each site determines the migration site autonomously by using the DC algorithm described in Figure 2. Both the sender and the receiver processes shown in the Figure 2 exist at each site. The sender broadcasts the migration request to the members of domains when the state is not in the idle state, and waits the acknowledgements from the members. If several acknowledgements are returned, it selects the site that has the earliest timestamp attached, and migrate the code to that site. On the other hand, if the receiver receives the migration request and if its state is in idle state, it returns the acknowledgement with its current timestamp. After acknowledgement, if it has received Move_Transaction message with more earlier timestamp than owns or it has not received Move_Transaction message, then it cancels the migration request received. 2.3 Prototype Experiment A middleware prototype system that migrates objects among PCs was implemented. And, the performance of the two systems; the one that does not migrate object and the

Design Methodologies of Workload Management through Code Migration

103

100% ) et is a f o n iot az ili t u U P C( t in op ld o hs re h T

90% 80%

Busy

70% 60%

Normal

50% 40% Idle

30% 20% 10% 0% 0%

10%

20%

30%

40% 50% 60% T ransaction w orkload

70%

80%

90%

100%

Fig. 3. Migration State Table Arrival (from users & other sites)

Transaction Manager

Queue Migration (to other sites)

Agent

CPU Processing

Departure (end of transaction)

Migration State Table

Fig. 4. Simulation Model of Each Site

other that always migrates object to the remote site was compared [7], and the threshold points were obtained by varying the CPU utilization and the transaction workload. Figure 3 plotted the threshold points with the axis X the workload of a transaction and the axis Y the average CPU utilization of other transactions in the site. We call the table shown in Figure 3 as the migration state table (MST). It was used to provide the mechanism for the transfer policy in the following simulations. The migration state table is divided into three states [9]: busy state, normal state and idle state, through setting some widths around the threshold points. When a transaction comes in a site and the current state is in the normal or busy states, the transaction can be moved to the remote site whose state is in the idle state, provided the mechanism for location policy.

3 Simulation A simulation model of each site of the system is modeled based on the prototype experiment, and it is shown in Figure 4. Each site runs transaction manager that directs a transaction which has just originated from that site. If the processor is idle,

104

M. Yoshida and K. Kojima

a transaction is processed at that site. Otherwise, a transaction must wait in the queue to finish the other transactions, or must move to the other site to finish computing. It is determined by checking the Migration State Table located at each site. The agent negotiates with the information server for the movement of the transactions if it were the centralized control. The simulation is carried out by using the event based algorithm. The simulation model, shown in Figure 4, was organized and several simulations are carried out. The parameters we selected for the model are the following: 1) Site number: 50 to 500 sites for computing are assumed, and 1 site for information server is assumed for centralized control. 2) Average arrival time: The arrival transaction patterns at each site follow one of the three different distributions; a Normal distribution, a Poisson distribution, and a Uniform distribution. 1000 to 15000 transactions at each site are assumed. 3) Processing time: The processing time of transactions at each site is assumed to follow a normal distribution. The average processing time is assumed to have 2 simulation units. 4) CPU power: The CPU power at each site follows a normal distribution. 5) Transmission time: The time taken to transmit a message from one site to another for migration is assumed to vary from 1 unit to 20 simulation units. 6) Object migration time: The time taken to migrate object code of a transaction to the remote site is assumed to be constant. 10 simulation units are assumed. 7) Transfer policy: The migration state table given in Figure 3 was provided at each site. 8) Location policy: Six location policy algorithms are compared; these are the random based algorithm (RB), the CPU power based algorithm (CB), the traffic based algorithm (TB), the working buffer based algorithm (WB), the decentralized algorithm (DC), and the non-migration algorithms (NM). The resulting statistics collected are the average response time, the average throughput, the remote processing rate, the rejected transaction rate, and the average message transferred. 3.1 Simulation1: Performance Comparison of Various Load Balancing Algorithms The algorithms for location policy were examined in the simulations. The result for the response time was as follows [2, 7]: (good) DC > WB > {TB, CB} > RB > NM (bad) In this paper, we picked up the two best algorithms from the simulation results [2, 7]. The WB which is the best centralized algorithm for response time, and the DC which is decentralized and the best algorithm in all algorithms simulated, are selected and compared to the non-migration method.

Design Methodologies of Workload Management through Code Migration 400

NM DC WB

350 300 e) m tit niu 250 ( e im t 200 es no 150 spe R 100 50 0

4000

5000

6000 7000 8000 work load (number of transaction)

9000

10000

Fig. 5. Comparison of the Response Time (Normal Distribution) 400

NM DC WB

350

)e 300 tim itn 250 u( e tim200 es no 150 ps Re 100 50 0

4000

5000

6000 7000 8000 work load (number of transaction)

9000

10000

Fig. 6. Comparison of the Response Time (Poisson Distribution) 400

NM DC WB

350 300 e) im tti 250 nu ( e im t 200 es no 150 spe R 100 50 0

4000

5000

6000 7000 8000 work load (number of transaction)

9000

10000

Fig. 7. Comparison of the Response Time (Uniform Distribution)

105

106

M. Yoshida and K. Kojima Transaction patterns

Uniform distribution

NM

Poisson distribution

DC/WB

NM

Normal distribution

DC/WB

NM

Non Applicable

DC/WB

4,000

5,000

Non Applicable

6,000 7,000

8,000

9,000

Work load ( number of transactions)

10,000

Fig. 8. Migration Domain

3.2 Simulation2: Performance Comparison of Response Time Varying Transaction Patterns The distribution of transactions at each site was varied and evaluated [2, 7]. Three distributions, such as a Normal distribution, a Poisson distribution, and a Uniform distribution, are provided, and the response time was compared. Figure 5, 6 and 7 show the comparison of the response time. The effectiveness of the traffic patterns can be ordered as follows: (good) Uniform > Poisson > Normal (bad) The non migration policy get worsen the response time immediately when the traffic increases. The DC and WB have a threshold at which the response time gets worsen, around 7000 in the normal distribution, and around 9000 in the Poisson distribution. Figure 8 shows the application domain of each migration algorithms with the transaction patterns. If transactions arrive as a normal distribution pattern, and the number of transactions are between 4000 and 7000, the DC and WB algorithms, work efficiently. It could have about 9 times better response time than the NM, when the workload is 7000. When the workload exceeds 7000, both algorithms get worsen exponentially, so it had better not to migrate transactions. When transactions arrive as a Poisson distribution pattern, it has also two threshold points at 4500 and 9000. Between them, the migration algorithms work efficiently. It works about 15 times better than the NM. When transactions arrive as a uniform distribution pattern, the migration algorithms have tremendous effect. Though the NM has the threshold point at 8500, the migration methods do not have the threshold point. It works over the threshold point continuously. 2000

NM RB DC WB

1800 1600

e)im 1400 tit n(u 1200 em it 1000 sen 800 poes R 600 400 200 0

1

5

9

13

17

21

25 29 Site number

33

37

Fig. 9. Response Time (50 sites)

41

45

49

Design Methodologies of Workload Management through Code Migration 2000

107

NM RB DC WB

1800 1600

)e 1400 im ttni 1200 u( em it 1000 ens osp 800 eR 600 400 200 0

1

46

91

136

181

226 271 Site Number

316

361

406

451

496

Fig. 10. Response Time (500 sites) 12

)e 10 im tt in 8 u( em 6 tie sn op 4 esR 2 0

NM RB CB TB WB

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Network delay (unit time)

Fig. 11. Response Time (4000 Transaction /site Assumed) 24 22 e)m tit 20 in u( 18 em it sen 16 pos 14 eR

NM RB CB TB WB

12 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Network delay (unit time)

Fig. 12. Response Time

（5000 Transactions/site Assumed)

3.3 Simulation3: Performance Comparison of Response Time Varying Grid Scales The computing sites are scaled up to 500 sites from 50 sites, and the same simulations are performed. Figure 9 shows the response time of each site in the case of 50 sites. Figure 10 shows the response time scaling up the sites to 500. It can be observed that the increase of sites does not affect the response time at all. Scaling up sites did not affect the response time of each site. Even though the sites increase, the

108

M. Yoshida and K. Kojima

average response time in the grids will not change, if the computing scales were in the order of hundreds. 3.4 Simulation4: Performance Comparison of Response Time Varying Network Delay To observe the influence of network delay, the transmission time is changed, and the response time is observed. The ratio of transmission time to processing time is ranged from 1:1 to 1:20. Figure 11 and 12 show the response time changing both the transmission time and the number of transactions. The intersections of the NM and the others are shifting to the right by the increase of the transactions. Though the response gets worsen by increasing the network delay, we could observe that the more the transmission delay, the more we could have extend the migration domain. When the transactions are 4000 and the network delay is less than the 10 times of the processing time, the migration works effectively. When the transactions are 5000, the migration works effectively if the network delay is less than the 20 times of the processing time of a transaction. As the network delay increases, it get worsen the response time lineally. The equation for the response time graph can be described as follows: / 2 + C

(1)

Network delay is the relative time unit compared to the processing time. C is the constant value determined by the workload of transactions. Given the network delay and C, the upper bound of the migration domain can be defined by the equation. If the value by the equation (1) is greater than the average non-migrating response time, then migration should not be performed, or the constant value C has to be adjusted. Given the network delay and the value of C, we can calculate the response time and compare to the anticipated response time. 3.5 Simulation Results The results of the simulations are summarized as follows: 1) Several load balancing algorithms were evaluated and compared to the non migration policy (NM). The effective order of the location policy algorithms for response time was observed. The migration ratio converged to a certain different point depending upon the location policy algorithms. 2) A transaction pattern affects the response time of the system tremendously. The order of effective response time for transaction patterns can be observed, and the location policy algorithms of the application domains can be defined. 3) The increase of sites does not affect the response time of each site when the grid scales were in the order of hundreds. The results we obtained are effective to the grid scales that were in the order of hundreds. 4) When the network delay is very large comparing to the processing time of a transaction, it is useless to migrate the transaction. The relationship of the network delay and response time was obtained. It sets down the upper bound of the transaction workloads.

Design Methodologies of Workload Management through Code Migration

109

4 Design Methodologies The language and the design methodologies for the desktop computing grids are described. 4.1 Language Designed We designed the language that specifies the domains of the system for load balancing. The domain defines the applicable area of the migration, and can be used not only for the purpose of load balancing but also for fault tolerant or any other purposes. Figure 13 (a) shows the syntax of the language, and (b) shows the BNF notation. In the language, one of the four parameters:” ALL”, “domain names”,” direct address” or “NULL”, must be selected. If “ALL” is selected, the migration request is notified to all the sites of desktop PCs in the grids. If “domain names” is selected, the migration request is multicasted to the sites that the domain involves. The address of the sites must be included into the domain name in advance. If “direct address” is selected, the request is sent to the address directly described. If “NULL” is selected, the request is not sent to anywhere, and be executed in its own site. 4.2 Design Methodologies This section describes the design methodologies obtained from the simulation results, assuming the following strategies: 1) The Migration State Table must be constantly observed by the SMT. It defines the transfer policy for migration. 2) Transactions had better be distributed uniformly. 3) The relationship between the network delay and processing time must be observed to define the upper bound of the migration domain. The simulation result shows that the network delay should be less than the 20 times of the processing time when the number of sites is in the order of hundreds. 4) If the number of grid sites is in the order of hundreds and the heavy traffics are in the migration domain, the distributed algorithm for decentralized control and the working buffer algorithm for centralized control are the best solution to apply. The following is the design methodologies we learned from the simulation results: 1) Before designing the migration, trace the workload of transactions and analyze the average workload of a transaction and the pattern of transactions at each site. 2) Obtain the Migration State Table for transfer policy, described in section 2.3 by tracing or calculating the workload of transactions.

110

M. Yoshida and K. Kojima

3) Define the upper bound of the migration domain using the equation (1) in section 3.4. Given the network delay, workload of transactions, or anticipating response time, we can define the upper bound of transactions for migration. Migration effectively works within the migration domain. 4) Design the domain of each site. Try to group each site to the domains as each domain to have the uniform distribution of the workloads, in the migration domain described in Figure 8. 5) Define the sites of the domains by the language described in section 4.1. 6) Select the migration algorithms for location policy. Either DC algorithm for distributed control or WB algorithm for centralized control can be selected. If it does not have powerful machines for servers, select the DC algorithm. However, the WB algorithm has an advantage for controlling the response time at each site to be uniformly distributed. #DIST

ALL <domain names> NULL

## Follows the original Java source code

…………… ……………

(a) DIST Syntax DIST::= #DIST {ALL | <domain names> | | NULL} <domain names>::= <domain name> | <domain name>,<domain names> <domain name>::= | ,<domain name> ::= | <symbolic address> <symbolic address>::= | , <symbolic address>

(b) BNF Notation Fig. 13. Language Syntax for Distributed Computing

5 Conclusion This paper summarized the performance evaluation of load balancing control [2, 7], and described the design methodologies derived from the simulation results in distributed desktop computing grids. The domain, the language and the algorithms for computing grids are described. The distributed language which supplements the design methodologies was introduced. The distributed language for distributed desktop computing grids must be used effectively within a company or within a university. The validation of the effectiveness of the system we described is our next research topic. We are now implementing the platform of the distributed desktop computing grids.

Design Methodologies of Workload Management through Code Migration

111

References 1. Coulouris, G., Dollimore, J., Kindberg, T.: Distributed Systems concepts and Design, 4th edn. Addison Wesley, Reading (2005) 2. Yoshida, M., Sakamoto, K.: Performance Comparison of Decentralized Workload Control through Code Migration in Distributed Desktop Computing Grids. In: The 5th IEEE International Symposium on Embedded Computing (2008) 3. Krauter, K., Buyya, R., Maheswaran, M.: A Taxonomy and Survey of Grid Resource Management Systems for Distributed Computing. Software-Practice and Experience 32 (2002) 4. Venugopal, S., Buyya, R., Ramamohanarao, K.: A Taxonomy of Data Grids for Distributed Data Sharing, Management and Processing. ACM Computing Surveys 28 (2006) 5. Theotokis, S.A., Spinellis, D.: A Survey of Peer-to-Peer Content Distribution Technologies. ACM Computing Sueveys 36(4) (2004) 6. Choi, S., Buyya, R., et al.: A Taxonomy of Desktop Grids and its Mapping to State-of –the Art Systems, Technical Report, GRIDS-TR-2008-3, The University of Melbourne, Australia (2008) 7. Yoshida, M., Sakamoto, K.: Performance Comparison of Load Balancing Algorithms through Code Migration in Distributed Desktop Computing Grids. In: The 3rd IEEE Asia Pacific Services Computing Conference (2008) 8. Shah, R., Veeravalli, B., Misra, M.: On the Design of Adaptive and Decentralized LoadBalancing Algorithms with Load Estimation for Computational Grid Environments. IEEE Trans. On Parallel and Distributed Systems 18(12) (2007) 9. Alonso, R., Cava, L.: Sharing Jobs Among Independently Owned Processors. In: Proc. of the 8th ICDCS (1988)

Dynamic Dependent Tasks Assignment for Grid Computing Meddeber Meriem1 and Yagoubi Belabbas2 1

Department of Computer Science, University of Mascara, Algeria [email protected] 2 Department of Computer Science, University of Oran, Algeria [email protected]

Abstract. In a grid computing, task execution time is dependent on the machine to which it is assigned and task precedence constraints represented by a directed acyclic graph. In this paper we propose a hybrid assignment strategy of dependent tasks in Grids which integrated static and dynamic assignment technologies. We consider that a grid computing is a set of clusters formed by a set of computing elements and a cluster manager. Our main objective is to arrive at task assignment that could achieve minimum response time and to reduce the transfer cost inducing by the tasks transfer respecting the dependency constraints.. ...

1

Introduction

Grid computing originated from a new computing infrastructure for scientific research and cooperation and is becoming a mainstream technology for largescale resource sharing and distributed system integration. Current efforts towards making the global infrastructure a reality provide technologies on both grid services and application enabling[1]. A task is defined to be a program segment that can be individually scheduled. A grid computing element is defined to be any processor that can receive tasks from a central scheduler and may be a single processor node or one of the processors within a multi-processor node. The problem of obtaining an optimal matching of tasks to machines in any distributed system is well known to be NP-hard even when the tasks are independent. The problem is much more difficult when the tasks have dependencies because the order of task execution as well as task-machine pairing affects overall completion time[2]. A precedence relation from task i to task j means that j needs data from i before being started. If these two tasks are not assigned to the same computing element, a delay cij must be considered between the completion of i and the beginning of j to transfer the data. Dynamic tasks assignment assumes a continuous stochastic stream of incoming tasks. Very little parameters are known in advance for dynamic tasks assignment. Obviously, it is more complex than static tasks assignment for implementation, C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 112–120, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Dynamic Dependent Tasks Assignment for Grid Computing

113

but achieves better throughput. Also it is the most desired because of the application demand[3]. In this paper, we propose a hybrid assignment strategy of dependent tasks in Grids which integrated static and dynamic assignment technologies. This strategy meets the following objectives: (i) reducing, whenever possible, the average response time of tasks submitted to the grid, (ii) respecting the constraints of dependency between tasks, and, (iii) reducing communication costs by using a static tasks placement based on the connected components algorithm to minimize the delay cij between task i and task j and by favoring a dynamic tasks placement within the cluster rather than the entire grid. The rest of this paper is organized as follows. We begin with the overview of some related works in Section 2. Section 3 presents the tasks assignment problem. In Section4 we will presents our system model. Section 5 describes the main steps of the proposed assignment strategy. We evaluate the performance of the proposed scheme in Section 6. Finally, Section 7 concludes the paper.

2

Related Work

There have been many heuristic algorithms proposed for the static and dynamic tasks assignment problem. Many of these algorithms apply only to the special case where the tasks are independent i.e. there are no precedence constraints[4,5,6]. Many heuristic algorithms have been proposed for static scheduling of dependent tasks where task precedence constraints are modelled as a directed acyclic graph (DAG). In [7], Yang Qu and al, target dependent task models and propose three static schedulers that use different problem solving strategies. The first is a heuristic approach developed from traditional list based schedulers. It presents high efficiency but the least accuracy. The second is based on a full-domain search using constraint programming. It can guarantee to produce optimal solutions but requires significant searching effort. The last is a guided random search technique based on a genetic algorithm, which shows reasonable efficiency and much better accuracy than the heuristic approach. Wayne F. Boyer and al [2], propose a non-evolutionary random scheduling (RS) algorithm for efficient matching and scheduling of inter-dependent tasks in a DHC system. RS is a succession of randomized task orderings and a heuristic mapping from task order to schedule. Randomized task ordering is effectively a topological sort where the outcome may be any possible task order for which the task precedent constraints are maintained. However static tasks assignment is performed offline, or in a predictive manner and can be used whenever the task information is known a priori such as at compile time of a parallelized application. Although the good results that this approaches provides, they stay limited to a static assignment. Large and non dedicated computing platforms as grids may require dynamic task assignment methods to adapt to the run-time changes such as increases in the workload or resources, processor failures, and link failures [8]. In this paper, we address these issues.

114

3

M. Meriem and Y. Belabbas

Tasks Assignment

As Grid is a distributed system utilizing idle nodes scattered in every region, the most critical issue pertaining to distributed systems is how to integrate and apply every computer resource into a distributed system, so as to achieve the goals of enhancing performance, resource sharing, extensibility, and increase availability. Tasks assignment is very important in a distributed environment. In distributed systems, every node has different processing speed and system resources, so in order to enhance the utilization of each node and shorten the consumption of time, tasks assignment will play a critical role. On the other hand, in distributed systems, the policies and methods for keeping a tasks assignment will directly affect the performance of the system. In addition, the tasks assignment policies for distributed systems can be generally categorized into static tasks assignment policies and dynamic tasks assignment policies[9]. 3.1

Static Tasks Assignment

Static tasks assignment policies use some simple system information, such as the various information related to average operation, operation cycle, and etc., and according to these data, tasks are distributed through mathematical formulas or other adjustment methods, so that every node in the distributed system can process the assigned tasks until completed. The merit of this method is that system information is not required to be collected at all times, and through a simple process, the system can run with simple analysis. However, some of the nodes have low utilization rates. Due to the fact that it does not dynamically adjust with the system information, there is a certain degree of burden on system performance. 3.2

Dynamic Tasks Assignment

Dynamic tasks assignment policies refer to the current state of the system or the most recent state at the system time, to decide how to assign tasks to each node in a distributed system. If any node in the system is over-loaded, the over-loading task will be transferred to other nodes and processed, in order to achieve the goal of a dynamic assignment. However, the migration of tasks will incur extra overhead to the system. It is because the system has to reserve some resources for collecting and maintaining the information of system states. If this overhead can be controlled and limited to a certain acceptable range, in most conditions, dynamic tasks assignment policies out perform the static tasks assignment policies.

4 4.1

System Model Grid Model

In our study we model a Grid as a collection of n clusters with different computational facilities. Let G = (C1 , C2 ,..., Cn ) denotes a set of clusters, where each cluster Ci is defined as a vector with four parameters : Ci = (N CEi , Mi , Bandi ,

Dynamic Dependent Tasks Assignment for Grid Computing

115

Spdi ), where N CEi is the number of computing elements, Mi is the Manager node of the cluster Ci , Bandi is the bandwidth of the network, Spdi correspond to the cluster capability. Every cluster is connected to the global network (WAN). In this model, a cluster represents a set of R homogeneous computing elements connected by a local network (LAN) and located geographically in the same organization. Ci = (CEi1 , CEi2 ,..., CEir ), where each computing element CEij , have it own capability. The cluster manager CMi uses the following equation to calculate Spdi :  Spdij (1) Spdi = j∈N CEi

Figure1 shows the Grid system model. In highly distributed systems, centralized work tasks assignment approaches become less feasible because it make use of a high degree of information, which causes a high work tasks assignment overhead. That is why we chose to develop a hybrid load balancing model that is centralized intra-cluster, but distributed inter-clusters. Each cluster in the Grid has a manager, which assign tasks to the cluster computing element.

Cluster 2 1111 Manager 1111 0000 0000 0000000 1111 0000 111 1111 0000 00001111 1111 ... 0000 1111

Cluster 1

0110

CE1,2

00 11 11 00 00 11 00 11 00 11111 0110 000 00 11

Network

Broker

11 00

T1 T2 T3

.....

Tasks arrival

CEn,i

Cluster n Fig. 1. Grid model

We assume that in the grid under study there is a central resource broker (CRB ), to which every Cluster Manager (CM ) connects and the grid clients send their tasks to the CRB. The CRB is responsible for scheduling tasks among CMs. 4.2

Application Model

DAG model. An application can be represented by a directed acyclic graph (DAG) D= (V, E), where V is a set of v nodes and E is a set of directed e

116

M. Meriem and Y. Belabbas

edges. A node in the DAG represents a task which in turn is a set of instructions which must be executed sequentially without pre-emption in the same processor. The edges in the DAG, each of which is denoted by (ni , nj ), correspond to the precedence constraints among the nodes. The weight of an edge is called the communication cost of the edge and is denoted by Cij . The source node of an edge is called the parent node while the sink node is called the child node. A node with no parent is called an entry node and a node with no child is called an exit node[10].

T1

T2

T4

T3

T5

Fig. 2. Task precedence graph

Task model. Tasks arrive randomly with a random computation length, an arrival time and precedence constraints. In our work, we generate randomly precedence constraints between tasks. Also, we believe that tasks can be executed on any computing element and each CE can only execute one task at each time point, the execution of a task cannot be interrupted or moved to another CE during execution. We also assume that a task cannot start execution before it gathers all of the messages from its parent tasks. The communication cost between two tasks assigned to the same processor is supposed to be zero.

5

Proposed Strategy

In order to reduce the global response time of the system and respect the tasks dependencies, this study proposed a hybrid tasks assignment policy, consisting of static and dynamic tasks assignment strategies. In the static case, when a user sends his tasks, they will be assigned to appropriate computing elements to achieve the goal of placement. In addition, in the dynamic case, the system will be adjusted dynamically according to the clusters workload. 5.1

Static Tasks Placement Strategy

Central resource broker. The role of the CRB in the system is to assign statically tasks placed in the task queue. For that we propose the following steps that will be executed periodically:

Dynamic Dependent Tasks Assignment for Grid Computing

117

– Partition all tasks waiting in the queue to x connected component by executing the connected component algorithm. A connected component is defined as a collection of dependent tasks with inter-task data dependencies. The following figure shows a set of waiting tasks composed of three connected component.

T1 T3 T6

T2

T7 T4

T8

T5

Fig. 3. Example with three connected components

– Send each connected component CCk to a cluster manager CMi , using a round robin strategy, as follows : (CC1 , CM1 ), (CC2 , CM2 ),..., (CCP , CMn ), (CCp+1 ,CM1 ),...,(CCx , CMj ) – Send tasks associated to the connected component CCk , to cluster manager CMi . Cluster manager. Once the manager receives the connected component, it affects them to the computing elements composing the cluster: – Randomly, or, – Using a round robin strategy : (CC1 , CE1 ), (CC2 , CE2 ),..., (CCx , CEj ) Then the cluster manager sends tasks composing each connected component CCj to the same computing element as CCj . 5.2

Dynamic Tasks Placement Strategy

Computing element. The computing element, perform these steps while its tasks queue is not empty: – Run the first entry task Tj (with no precedence constraints) of its tasks queue. – Updates the connected component CCj associated to task Tj . – Executes the connected component algorithm on CCj to obtain the new entry tasks. Figure4 shows an example of a connected component with one entry task. After the end of execution of Tj , the CCj is divided on three connected components.

118

M. Meriem and Y. Belabbas T1

T2

T4

T3

T5

T3

T2

T6

T4

T7

T5

T6

T7

Fig. 4. Example with one entry task

The computing element executes the following steps periodically: – Computes its execution time texij as follows:    lenghtc,k T exij = Spdij

(2)

c∈CCNij l∈L k∈P

Where, CCN is the connected component number assigned to the computing element. L is the level number of connected components, and P is the tasks number of level k. – Sends its execution time to the cluster manager and all computing element of the cluster. – We define a threshold α, from which a resource CEij can say that it is more loaded than another. If (T exij > T exik +α ) then transfer some connected components from CEij to CEik until T exij ≤ T exik + α. – Inform the cluster manager about the tasks movement. Cluster manager. The cluster manager receives periodically the execution time of each resource of the cluster and performs the next steps: – Computes the execution time of cluster as follow:  T exi = T exij

(3)

j∈R

– We define a threshold β, such as: If (T exi >T exk + β) then transfer some connected components from Ci to Ck until T exi ≤ T exk + β.

6

Simulation Results

To test and evaluate the performance of our model, we developed our strategy under the GridSim[11] simulator written in Java, we can: (i) Generate the grid configuration file (clusters number, computing element number, their characteristics, period sending load information, bandwidth ...),

Dynamic Dependent Tasks Assignment for Grid Computing

119

(ii) Generate a set of tasks with all associated data (submission time, computation length, precedence constraints...). As performance measures, we are interested in average response time of tasks. To obtain results that are as consistent as possible, we repeated the same experiments more than ten 5 times. All these experiments were performed on a PC 3 GHz Pentium IV, with 1GB of memory and running on Linux Redhat 9.0. The first results obtained for response time are shown in figure5.

Fig. 5. Response time results (4 clusters)

We observe that our strategy reduce considerably the average response time of tasks submitted to the system. When increasing the tasks number, response time benefits increase. The lower benefit is obtained with 500 tasks, it is due to an under loaded state of the grid.

7

Conclusion

In this paper we have proposed a hybrid assignment strategy of dependent tasks in Grids which integrated static and dynamic assignment technologies for solving the placement problem. A tasks placement strategy is introduced; it has the advantage of being able to divide the input task graph into set of connected component in order to reduce the response time of system application. To test and evaluate the performance of our model, we developed our strategy under the GridSim simulator written in Java. We have randomly generated clusters with different characteristics and a set of dependent tasks. The first experimental results are encouraging since we can significantly reduce the average response time. To measure the efficiency of the strategy, we plan to compare its performance with other grid simulators such as SimGrid[12]. We plan also to integrate our strategy to the middleware GLOBUS[13].

References 1. Cao, J., Spooner, D.P., Jarvis, S.A., Nudd, G.R.: Grid load balancing using intelligent agents. Future Generation Comp. Syst. 21, 135–149 (2005) 2. Boyer, W.F., Hura, G.S.: Non-evolutionary algorithm for scheduling dependent tasks in distributed heterogeneous computing environments. J. Parallel Distrib. Comput. 65, 1035–1046 (2005)

120

M. Meriem and Y. Belabbas

3. Vidyarthi, D.P., Sarker, B.K., Tripathi, A.K., Yang, L.T.: Scheduling in Distributed Computing Systems Analysis, Design and Models. A research monograph. Springer, New York (2009) 4. Leal, K., Huedo, E., Llorente, I.M.: A decentralized model for scheduling independent tasks in Federated Grids. Future Generation Comp. Syst. 25, 840–852 (2009) 5. Salcedo-Sanz, S., Xu, Y., Yao, X.: Hybrid meta-heuristics algorithms for task assignment in heterogeneous computing systems. Computers and OR 33, 820–835 (2006) 6. Maheswaran, M., Ali, S., Siegel, H.J., Hensgen, D., Freund, R.F.: Dynamic mapping of a class of independent tasks onto heterogeneous computing systems. J. Parallel Distrib. Comput. 59, 107–121 (1999) 7. Qu, Y., Soininen, J.-P., Nurmi, J.: Static scheduling techniques for dependent tasks on dynamically reconfigurable devices. Journal of Systems Architecture 53, 861–876 (2007) 8. Uar, B., Aykanat, C., Kaya, K., Ikinci, M.: Task assignment in heterogeneous computing systems. J. Parallel Distrib. Comput. 66, 32–46 (2006) 9. Yan, K.Q., Wang, S.C., Chang, C.P., Lin, J.S.: A hybrid load balancing policy underlying grid computing environment. Computer Standards and Interfaces 29, 161–173 (2007) 10. Kwok, Y.-K., Ahmad, I.: Static scheduling algorithms for allocating directed task graphs to multiprocessors. ACM Computing Surveys 31, 406–471 (1999) 11. Buyya, R., Murshed, M.: Gridsim: A toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. The Journal of Concurrency and Computation: Practice and Experience (CCPE) 14, 13–15 (2002) 12. Casanova, H.: Simgrid: a toolkit for the simulation of application scheduling. In: Proceedings of the IEEE International Symposium on Cluster Computing and the Grid (CCGrid’01), Brisbane, Australia, pp. 430–437 (2001) 13. Foster, I.: Globus toolkit version 4: Software for service oriented systems. In: IFIP: International Conference on Network and Parallel Computing, Beijing, China, pp. 2–13 (2005)

Implementation of a Heuristic Network Bandwidth Measurement for Grid Computing Environments∗ Chao-Tung Yang∗∗, Chih-Hao Lin, and Wen-Jen Hu Department of Computer Science, Tunghai University, Taichung, 40704, Taiwan (ROC) [email protected]

Abstract. Grid computing technique is more and more popular. In general, Ganglia and NWS are applied to monitor Grid nodes’ status and networkrelated information, respectively. Comprehensive monitoring and effective management are criterions to archive higher performance of grid computation. Unfortunately, owing to diverse user requirements, information provided by Ganglia and NWS services is not sufficient in real cases, especially for application developers. In addition, NWS services that deployed based on “Domainbased Network Information Model” could greatly reduce overheads caused by unnecessary measurements. This study proposes a heuristic QoS measurement architecture, which constructed with a domain-base model, to provide effective information to meet user requirements, especially for application developers. Keywords: Grid Computing, Heuristic, QoS, Network Information Model.

1 Introduction As we known, Grid computing technique is more and more popular adopted by organizations to obtain high performance computing and heterogeneous resources sharing. Since all computing nodes in grid environments are connected by means of the network, all tasks that executed in grid environments will be influenced by network status due to complicated and numerous communications between computing resources [1, 2]. While we design algorithms for specific usages or assign tasks into grid environments, we have to evaluate the network performance and to adjust algorithms to attain optimal performance in real-time execution [9, 11]. The best scenario is that our grid environments have some mechanisms to retrieve network status and to evaluate performance automatically [3, 4]. Thus, applications or web service agents could provide higher performance due to dynamic parameter adjustment and algorithms optimization. While grid computing becomes popular, it brings about a new issue, i.e., how to manage and monitor numerous resources of grid computing environments. In most cases, we use Ganglia and NWS to monitor machines’ status and network-related information, respectively. Owing to diverse user requirements, information provided by ∗ This work is supported in part by National Science Council, Taiwan R.O.C., under grants no. NSC 96-2221-E-029-019-MY3 and NSC 98-2622-E-029-001-CC2. ∗∗ Corresponding author. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 121–130, 2010. © Springer-Verlag Berlin Heidelberg 2010

122

C.-T. Yang, C.-H. Lin, and W.-J. Hu

these services is not sufficient. According to the mechanism that we designed in previous work [10], we could retrieve relative network information in real-time manner; even advanced customization for special purposes is available. With the customized shell scripts that we wrote for NWS services’ deployment, we could easily and quickly deploy NWS services to each grid node and fetch network-related information in a regular time interval. Besides, we could obtain extra statistics for job-scheduling in our grid environments. Except job-scheduling, statistics is also helpful in many respects.

2 Network Information Provider The NWS (Network Weather Service) [6, 7] is a distributed system that detects network status by periodically monitoring and dynamically forecasting over a given time interval. This service operates by a distributed set of performance sensors (network monitors, CPU monitors, etc.) from which they gather system information. It uses numerical models to generate forecasts of what the conditions will be produce a given time period. The NWS system includes sensors for end-to-end TCP/IP performance (bandwidth and latency) [5], available CPU percentage, and available non-paged memory. The sensor interface, however, allows new internal sensors to be configured into the system. We primarily use NWS for end-to-end TCP/IP measurements. As Rich Wolski said [7], NWS is designed to maximize four possible conflicting functional characteristics. It must meet these goals despite the highly dynamic execution environment and evolving software infrastructure provided by shared metacomputing systems.

Fig. 1. NWS services integrated with Ganglia web portal

Implementation of a Heuristic Network Bandwidth Measurement

123

Fig. 2. Network statistics produced by NWS measurements demonstrated in web portal

• • • •

Predictive Accuracy Non-intrusiveness Execution longevity Ubiquity

We have successfully developed a number of shell scripts for the automatic NWS deployment. And these scripts form the basis of NWS services’ management. And we have successfully integrated NWS services with the Ganglia web portal.

3 Heuristic QoS Measurement QoS (Quality of service) [13, 14] is the ability to provide different priority to different applications, users, or data flows, or to guarantee a certain level of performance to a data flow. It was widespread adopted in the field of computing networking, and we use it as a quality measurement of grid environments. In our previous project, we have built an integrated grid environment including a web portal composed of Ganglia and NWS services. Afterward, we start another project about PACS (Picture Archive and Communication System) [8] and most experiments were done in the same platform. The primary mission in this project is to exchange medical images efficiently with specific applications developed by our team. The application, named "Cyber [12]", has successfully integrated eight algorithms. For exchanging medical images efficiently with these algorithms integrated in Cyber, we have to configure a lot of parameters before tasks submitted. Unfortunately, we have no idea what's best combination of parameters we should take in advance. Therefore, we adopt "trial and error method" unavoidably. But it's definitely not

124

C.-T. Yang, C.-H. Lin, and W.-J. Hu

practical for most conditions. For this reason, we expect to establish an automatic parameter-self-optimization method. To guarantee the degree of QoS, we regard user requirements as constraints of tasks. With these constraints and heuristic QoS measurements we proposed in this paper, we could provide more QoS to meet user requirements. 3.1 Deploy Flowchart We regard several grid nodes as a group, and each group has a header to deploy nameserver and memoryserver. A simple NWS services deployment procedure that we used is divided into 3 steps: 1. Clean all NWS process. 2. Load NWS services. 3. Register NWS clique. And the standard procedure we wrote in shell scripts is shown as Fig. 3. Owing to the non-intrusiveness characteristic of NWS, these shell scripts could be executed without root privilege.

Fig. 3. Procedure of NWS services deployment

Fig. 4. The flowchart of gathering network information

Implementation of a Heuristic Network Bandwidth Measurement

125

Fig. 4 has shown a simple flowchart. In this paper, we have edit crontab(Linux Scheduler) to schedule some routines for inserting NWS information into database automatically and backing up raw data as plain text files locally. While the procedure routines that we scheduled in crontab are invoked, customized shell scripts are executed. The first step of the shell script is to get host groups from the database for NWS information gathering. Each host group is pre-defined in the database and is assigned a clique for measuring network status. After the clique is created, it will measure network information in an equal time interval, for example, 30 seconds. Then the script extracts bandwidth and latency from the NWS clique respectively. If it successes, it will insert bandwidth and latency information into the database. The second routine keep raw data as plain text files locally is designed for the future use. Currently, it just provides a different storage than a database to keep raw information of NWS services. 3.2 Heuristic Architecture We collected historical network information of grid environments and found an approach to evaluate QoS. We could give applications dedicated parameters in a simple manner by means of database operations. Couples of functions have been designed for analyzing historical information of network performance. Statistics is helpful in many fields, especially for prediction. Some researchers have used statistical method to monitor and predict bandwidth for QoS sensitive tasks [13]. All network relative information was periodically categorized to most used statistics. Besides, we have planned an innovative method to obtain the real-time network state that worked with Dynamic Domain-based Network Information Model, i.e. dynamically deploying clique into dedicated nodes, measuring network states, and then reporting results to a database, users, or applications. The enhanced version of current work which supports Dynamic Domain-based Network Information Model is currently under development. We have designed a simple model for the integration of Ganglia, NWS, and NINO. Ganglia and NINO provide the UI for users to manage and monitor grid environments. NWS and Ganglia collect related information from hosts and network regularly. And “Smart Broker” provides parameters to applications like Cyber. Smart Broker is the key component for us to evaluate QoS. Our previous work [8, 12] has provided users an interface for tuning up parameters which is shown as shown in Fig. 5. But most parameters used by this application, Cyber, must be set manually and it’s very inconvenient. We developed “Smart Broker” to help us to achieve automation of parameters self-optimization in diverse scenarios. Smart Broker works as the evaluation layer between applications and the information collection layer. We have pre-defined 4 task types that perform QoS measurement in various ways. • • • •

Download Upload Computational Hybrid

126

C.-T. Yang, C.-H. Lin, and W.-J. Hu

Fig. 5. Smart Broker Model

4 Experimental Results In this paper, we have chosen 4 grid nodes as “Header”, which is called “border” in the domain-based network information model, to register specific NWS services – the clique for gathering inter-domain network performance. Except these headers, we also registered a NWS clique named “cross-domain” to measure network performance between these headers. Information collected by NWS services is our basis to evaluate QoS. Hence, we have to ensure the NWS services deployment we performed is applicable. We have adopted a pull-based model to collect network information measured by NWS services as shown in Fig. 6. All grid nodes were deployed with NWS sensors, and zeta1, beta2, delta2, and eta4 were deployed both NWS nameserver and memoryserver. Zeta1 was deployed a routine to collect (pull) all network information measured by NWS services and to load these raw data into the database locally. The versions of Operation systems in these grid nodes are different, but they don’t influence our work.

Implementation of a Heuristic Network Bandwidth Measurement

Fig. 6. NWS cliques deployed in our Grid environments

Fig. 7. NWS measurements as our basis for QoS evaluation

127

128

C.-T. Yang, C.-H. Lin, and W.-J. Hu

We could easily found that the measurements of the NWS clique may be uneven. For example, eta7-eta9 has minimum measurements, 615, but eta1-eta3 has maximum measurements, 754. Uneven measurements may influence the accuracy of our model while evaluating QoS with statistical approaches. The NWS services have the ability to avoid collision which may cause inaccuracy of measurement, and this advantage is restricted in the same nameserver. In our test-bed, we found that the collision influences the accuracy frequently. We could find that network performance has a great variation due to collision of the NWS measurement. Fig. 8 has shown NWS measurements of our test-bed. Although the QoS evaluation model we adopted in this paper could not absolutely predict the real performance for real-time tasks execution. We still could pick out best selection of resources by means of the QoS evaluation model. To verify the usability of this QoS evaluation approach, we have also performed a simple experiment of the file transmission. And the result is the same as that of our predication using the QoS evaluation model.

Fig. 8. NWS measurements as our basis for QoS evaluation

5 Conclusions In this paper, we propose a heuristic QoS measurement constructed with a domainbased information model and by using a Relational Database Management System. According to this scheme, we could retrieve both real-time and historical network information. With customized shell scripts, NWS services could be quickly deployed to grid machines and fetch network information regularly. And with RDBMS, we could not only keep historical information, but also design some statistical analysis as we need. Statistics is helpful in many fields, for example, job dispatching or replicas

Implementation of a Heuristic Network Bandwidth Measurement

129

selection. We are planning to refer some approaches proposed by other researcher to reduce measurements in the near future. And this evaluation approach should be adjusted to meet the requirements of other 3 task types, i.e., Upload-oriented, Computational and Hybrid. We’ll make a study of these kinds of tasks before long.

References 1. A taxonomy and survey of grid resource management systems for distributed computing. Softw. Pract. Exper. 32(2), 135–164 (2002) 2. Cao, J., Jarvis, S.A., Saini, S., Kerbyson, D.J.: Performance prediction and its use in parallel and distributed computing systems. Future Generation Computer Systems 22(7), 745– 754 (2006)doi: 10.1016/j.future, 02.008 3. Chung, W., Chang, R.: A new mechanism for resource monitoring in Grid computing. Future Generation Computer Systems 25(1), 1–7 (2009) doi: 10.1016/j.future.2008.04.008 4. Krefting, D., Vossberg, M., Tolxdorff, T.: Simplified Grid Implementation of Medical Image Processing Algorithms using a Workflow Managment System. Presented at the MICCAI-Grid Workshop, New York (2008), http://www.i3s.unice.fr/~johan/MICCAI-Grid/website.html (re trieved) 5. Legrand, A., Quinson, M.: Automatic deployment of the Network Weather Service using the Effective Network View. In: Proceedings of 18th International Parallel and Distributed Processing Symposium (2004) 6. Network Weather Service. Network Weather Service, http://nws.cs.ucsb.edu/ewiki/ (retrieved) 7. Wolski, R., Spring, N., Hayes, J.: The network weather service: A distributed resource performance forecasting service for metacomputing. Future Generation Computer Systems 15(5-6), 757–768 (1999) 8. Yang, C.T., Chen, C.H., Yang, M.F., Chiang, W.C.: MIFAS: Medical Image File Accessing System in Co-allocation Data Grids. In: IEEE Asia-Pacific Services Computing Conference, APSCC’08, pp. 769–774 (2008) 9. Yang, C., Chen, S.: A Multi-site Resource Allocation Strategy in Computational Grids. In: Advances in Grid and Pervasive Computing, pp. 199–210 (2008), http://dx.doi.org/10.1007/978-3-540-68083-3_21 (retrieved July 29, 2009) 10. Yang, C., Chen, T., Tung, H.: A Dynamic Domain-Based Network Information Model for Computational Grids. In: Future Generation Communication and Networking, vol. 1, pp. 575–578. IEEE Computer Society, Los Alamitos (2007), http://doi.ieeecomputersociety.org/10.1109/FGCN.2007.9 11. Yang, C., Shih, P., Chen, S., Shih, W.: An Efficient Network Information Model Using NWS for Grid Computing Environments. In: Grid and Cooperative Computing - GCC 2005, pp. 287–299, http://dx.doi.org/10.1007/11590354_40 (retrieved July 29, 2009) 12. Yang, C., Yang, M., Chiang, W.: Implementation of a Cyber Transformer for Parallel Download in Co-Allocation Data Grid Environments. In: Proceedings of the 2008 Seventh International Conference on Grid and Cooperative Computing, pp. 242–253. IEEE Computer Society, Los Alamitos (2008), http://portal.acm.org/citation.cfm?id=1471431 (retrieved July 29, 2009)

130

C.-T. Yang, C.-H. Lin, and W.-J. Hu

13. Yu, Y., Cheng, I., Basu, A.: Optimal adaptive bandwidth monitoring for QoS based retrieval. IEEE Transactions on Multimedia 5(3), 466–472 (2003) doi: 10.1109/TMM.2003.814725 14. Cheng, Z., Du, Z., Zhu, S.: A Service Level QoS Mechanism and Algorithm for Data Distribution and Backup in an Grid Based Astronomy Data Management System. Presented at the Sixth International Conference on Grid and Cooperative Computing, GCC 2007, pp. 430–436 (2007) doi: 10.1109/GCC.2007.25

An Efficient Circuit–Switched Broadcasting in Star Graph Cheng-Ta Lee and Yeong-Sung Lin Department of Information Management, National Taiwan University No. 1, Sec. 4, Roosevelt Road, Taipei, 10617 Taiwan {d90001,yslin}@im.ntu.edu.tw

Abstract. In this paper, we propose an algorithm for broadcasting in star graph using circuit-switched, half-duplex, and link-bound communication. By using the algorithm, we show that the broadcasting in an n-dimensional star graph can be done in n-1 time steps. We also study the lower bound of time steps of the circuit-switched broadcasting in star graph, and we prove that the optimal broadcasting time steps in an n-dimensional star graph is ⎡lognn!⎤. Finally, the computational results showed that the proposed algorithm gets nearly optimal solutions. Keywords: Broadcasting, interconnection network, star graph, circuit-switched routing, link-bound.

1 Introduction The star graph interconnection network, since being proposed in [1], [2], is receiving increasing attention in the literature. It has been considered as an attractive alternative to the popular hypercube as the network architecture for parallel processing. Part of the reason is its symmetric and recursive nature, and superior (lower) node degree and comparable diameter as opposed to the hypercube [14]. Many references can be found in studying the star graph regarding such as its topological properties [3], [4], [13], embedding capability [5], [6], fault-tolerant capability [7], [8], [9], and even the construction of incomplete stars [10]. Among the efforts of studying the star graph, one of the central issues is around the various versions of broadcasting problem, broadcasting refers to the process by which a data set is sent from one node to all other nodes. Results about broadcasting are summarized in papers by Hedtniemi et al. [11] and Fraigniaud et al. [12]. In this paper, we considered the problem of broadcasting in star graph using circuit-switched, half-duplex, and link-bound communication. We propose an Efficient Circuit-Switched Broadcasting (ECSB) algorithm for an n-dimensional star graph with n! nodes. By using this algorithm, we showed that the broadcasting for an ndimensional star graph is done in n-1 time steps. The rest of this paper is organized as follows. In Section 2, we describe our communication model. In section 3, we discuss lower bounds on the optimal circuit-switched C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 131–135, 2010. © Springer-Verlag Berlin Heidelberg 2010

132

C.-T. Lee and Y.-S. Lin

broadcasting time steps. An efficient circuit-switched broadcasting algorithm is presented in Section 4. Finally, we give our concluding remarks in Section 5.

2 Communication Model An n-dimensional star graph, also referred as n-star or Sn, is an undirected graph consisting of n! nodes (or vertices) and (n-1)n!/2 edges. Each node is uniquely assigned a label x0x1…xn-1, which is the concatenation of a permutation of n distinct symbols {x0x1…xn-1}. Without loss of generality, let these n symbols be {0,1,…,n-1}. Given any node label x0…xi…xn-1, let function gi, 1 ≤ i ≤ n-1, be such that gi(x0…xi…xn1)=xi…x0…xn-1 (i.e., swap x0 and xi to keep the rest symbols unchanged). In Sn, for any node x, there is an edge joining x and node gi(x), and this edge is said to e along dimension i. It is known that Sn is node- and edge-symmetric and has a diameter of Dn=⎣3(n-1)/2⎦. In circuit-switched model, a node x sends its message to a node y via a directed path. Between two neighbor nodes in star graph, there exists exactly one link which can be used for both directions (but only one direction at one time), i.e., half-duplex link, and the link-bound communication is assumed, i.e., a node can use all of its links at the same time. Fig. 1 shows an example of the circuit-switched broadcasting in S3 under our communication model. In Fig. 1(a), a source node 012 sends a message to nodes 210 and 201 during the first time step. Fig. 1(b) shows that the source node 012 and informed nodes, 210 and 201, send messages to the remaining three nodes during the next time step. 012 g2

012 g1

g1

210

102 g2

210 g1

120

201

120

102

201 g1

021

021

(a)

(b)

Fig. 1. An example of the circuit-switched broadcasting in S3

3 Lower Bound of the Optimal Broadcasting Time Steps In this section, we study the lower bound of time steps of the circuit-switched broadcasting in star graph. Theorem 1. The optimal broadcasting time steps for an n-dimensional star graph with n! nodes is ⎡lognn!⎤.

An Efficient Circuit–Switched Broadcasting in Star Graph

133

Proof: The proof of Theorem 1 follows from the observation that each node can send the message to at most n-1 uninformed nodes at each time step in Fig. 1, because each node has n-1 degrees in Sn. For rapid broadcasting, source node and informed nodes must inform exactly n-1 other nodes at each time step except the last time step. Therefore, lower bound of the optimal broadcasting time steps is ⎡lognn!⎤. …

4 An Efficient Circuit–Switched Broadcasting Algorithm In this section, we present an efficient circuit-switched broadcasting algorithm in an n-dimensional star graph with broadcasting time steps n-1, n≥3. In order to facilitate our discussion, we introduce the following definitions. Definition 1. The generator g0 is defined by vg0 = v, where v is a source node or a node of informed nodes in Sn. Definition 2. We define a function send[v, gx gx …gx ] to send the message from node v to the node located by gx gx …gx function for c≥0, xi≥0, where v is a source node or a node of informed nodes in Sn. The proposed ECSB algorithm is shown in Fig. 2. 0

0

1

1

c

c

Algorithm. Efficient Circuit-Switched Broadcasting Input: Sn and source node Output: Broadcast to all the nodes 1: begin 2: for i=n-1 to 2 do 3: for j=i to 1 pardo 4: send[v, gj-1gi]; 5: end for 6: end for 7: send[v,g1]; 8: end Fig. 2. The efficient circuit-switched broadcasting algorithm

Lemma 1. The efficient circuit-switched broadcasting algorithm for an n-dimensional star graph with n! nodes can be done in n-1 time steps. Proof: In the broadcasting algorithm, from step 2-6 execute n-2 time steps. Step 7 executes 1 time step. Hence, the algorithm executes n-1 time steps. … Lemma 2. The algorithm can broadcasting from source node to all other nodes. Proof: In efficient circuit-switched broadcasting algorithm, the steps 2-6 can send messages to other substar by each iteration, because v can send messages to other substar by gj-1gi function. For example, v=0123 by g2g3 functions=3102, by g1g3 functions=3021, and by g0g3 functions=3120 in S4. Step 7 can send messages to neighbor node by function g1 in the last time step. Hence, the algorithm can broadcasting from the source node to all the other nodes. …

134

C.-T. Lee and Y.-S. Lin

g3

0123 g2

c

3120

g1

a

2103

1023

2130

1320

1203

2013

1230 b

2310

d 0213

3210 g3

2301

1302 g3

a

c

3201

0321

3102

0312

0231

3021

0132

3012

b

d

2031

1032

(a) 1st time step 0123

0123

3120

c 2103

a g2

g1

1023

g2

2130

g1

g2 1203

1320

d

1230 b

a g1

2103

2310

a

c

0321

3102

2031

g2

g1

0312

3201

2310

3210

1302

g1

g2 3021

g1

2301

1302

g2

1320

g3

g1

b

1230 b

0213

3210

2301

0231

2013

g1 g1

g1

1203

g3

g2

2130

d

0213

3201

1023

g1

g2 2013

3120

c

a

c

0321

3102

g1

0132

3012 d 1032

(b) 2nd time step

0231 b

g1

0312 g1

g1 3021

0132

g1

3012 d

2031

1032

(c) 3rd time step

Fig. 3. An example of the efficient circuit-switched broadcasting in S4 Table 1. The comparison of the time steps of our proposed algorithm and lower bound time steps in n-dimensional star graph

Star size: n 3 4 5 6 7 8 9 10

Number of nodes: n! Lower bound: ⎡lognn!⎤ 6 2 24 3 120 3 720 4 5,040 5 40,320 6 362,880 6 3,628,800 7

ECSB algorithm: n-1 2 3 4 5 6 7 8 9

An Efficient Circuit–Switched Broadcasting in Star Graph

135

Fig. 3 shows an example of the broadcasting in S4. Figs. 3(a), 3(b), and 3(c) are the 1st time step, 2nd time step, and 3rd time step, respectively.

5 Conclusion We considered the problem of broadcasting in n-dimensional star graph by using circuit-switched, half-duplex, and link-bound communication. The results showed that the broadcasting algorithm can be done in the star graph in nearly optimal time steps. The comparison of the time steps of our proposed algorithm and lower bound time steps in star graph is listed in Table 1.

References 1. Akers, S.B., Harel, D., Krishnameurthy, B.: The star graph: an attractive alternative to the n-cube. In: International Conference on Parallel Processing, pp. 393–400 (1987) 2. Akers, S.B., Krishnameurthy, B.: A group-theoretic model for symmetric interconnection networks. IEEE Transactions on Computers 38(4), 555–566 (1989) 3. Day, K., Tripathi, A.: A comparative study of topological properties of hypercubes and star graphs. IEEE Transactions on Parallel and Distributed Systems 5(1), 31–38 (1994) 4. Qiu, K.: On some properties and algorithms for the star and pancake interconnection network. Journal of Parallel and Distributed Computing, 16–25 (1994) 5. Jwo, J.S., Lakshmivarahan, S., Dhall, S.K.: Embeddings of cycles and grids in star graphs. In: IEEE International Symposium on Parallel and Distributed Processing, pp. 540–547 (1990) 6. Nigam, M., Sahni, S., Krishnamurthy, B.: Embedding hamiltonians and hypercubes in star interconnection graphs. In: International Conference on Parallel processing, vol. 3, pp. 340–343 (1990) 7. Bagherzadeh, N., Nassif, N., Latifi, S.: A routing and broadcasting scheme on faulty star graphs. IEEE Transactions on Computers 42(11), 1398–1403 (1993) 8. Jovanovic, Z., Mišic, J.: Fault tolerance of the star graph interconnection network. Information Processing Letters 49(3), 145–150 (1994) 9. Latifi, S.: On the fault-diameter of the star graph. Information Processing Letters 46(3), 143–150 (1993) 10. latifi, S., Bagherzadeh, N.: Incomplete star: an incrementally scalable network based on the star graph. IEEE Transactions on Parallel and Distributed Systems 5(1), 97–102 (1994) 11. Hedtniemi, S.M., Hedetniemi, S.T., Liestman, A.L.: A survey of gossiping and broadcasting in communication networks. IEEE Networks 18(4), 319–349 (1988) 12. Fraigniaud, P., Lazard, E.: Methods and problems of communication in usual networks. Discrete Applied Mathematics 53(1-3), 79–133 (1994)

Parallel Domain Decomposition Methods for High-Order Finite Element Solutions of the Helmholtz Problem Youngjoon Cha1 and Seongjai Kim2 1 Department of Applied Mathematics Sejong University, Seoul, 143-747, South Korea [email protected] 2 Department of Mathematics & Statistics Mississippi State University, Mississippi State, MS 39762 USA [email protected]

Abstract. The article is concerned with a parallel iterative domain decomposition algorithm for high-order finite element solutions of the Helmholtz wave equation. The iteration is performed in a block-Jacobi manner. For the interface operator, a Robin interface boundary condition is employed in a modified form which allows possible discontinuities of the discrete normal flux on the subdomain interfaces. The convergence of the algorithm is analyzed using energy estimates. Numerical results are given to show the effectiveness and parallel efficiency of the algorithm for the simulation of high-frequency waves in heterogeneous media in the two-dimensional space. The algorithm is carried out on a 16-node Linux cluster; it has been observed more than 97% parallel efficiency for all tested problems.

1

Introduction

Let Ω ⊂ IRd , d = 2 or 3, be a logically rectangular/cubic domain with its boundary Γ = ∂Ω. Consider the following Helmholtz problem (a) −Δ u − K(x)2 u = S(x), x ∈ Ω, ∂u + iα(x) u = 0, x ∈ Γ, (b) ∂ν

(1)

where i is the imaginary unit, ν is the unit outward normal to Γ , and the coefficients K(x) and α(x) satisfy K(x)2 = p(x)2 − iq(x)2 , 0 < p0 ≤ p(x) ≤ p1 < ∞, 0 ≤ q0 ≤ q(x) ≤ q1 < ∞, α = αr − iαi , αr > 0, αi ≥ 0,

(2)

and are sufficiently regular so that (1) admits a unique solution lying in H 1 (Ω). Here (1b) is an absorbing boundary condition (ABC); for example, one can select C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 136–145, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Parallel Domain Decomposition Methods

137

α appropriately so that (1b) represents a first-order ABC that allows normally incident waves to pass out of Ω transparently [1]. The Helmholtz problem is difficult to solve numerically, in particular when 0 ≤ q  p. In addition to having a complex-valued solution, it is neither Hermitian symmetric nor coercive. As a consequence, most standard iterative methods either fail to converge or converge so slowly. In many applications (e.g., the geophysical wave simulation and seismic velocity inversion), it is often required to reproduce waves up to 50-100 wavelengths in heterogeneous media. It is known that second-order discretization methods need to select at least 10-12 points per wavelength (2π/p) for stability reasons [2,3]. However, in practical high-frequency applications, one should choose at least 20-25 grid points per wavelength for a reasonable accuracy [2,3,4]. Thus the algebraic system for the numerical solution of the Helmholtz problem becomes very huge for realistic applications, besides being poorly-conditioned. As Zienkiewicz [5] pointed out, “the problem remains unsolved and a completely new method is needed.” Most of computational methods in the literature have been suggested for lower-order numerical solutions of constant-coefficient Helmholtz problems. For general coefficient problems, Kim [6,7,8,9] studied nonoverlapping DD methods for solving the Helmholtz problem by finite difference and linear finite element (FE) methods; see also [2]. For the simulation of high-frequency waves in heterogeneous media, Kim et al. [10] suggested the so-called high-frequency asymptotic decomposition method, in which the wavefield was decomposed into two parts (the phase and the cumulative amplitude) and the solution could be simulated by solving two easier-to-solve equations. This article develops a parallel iterative DD method for high-order FE solutions of the Helmholtz problem of variable coefficients. DD methods can combine iterative methods at the interface level and direct algorithms at the subdomain level so that they are attractive for poorly-conditioned large problems such as the Helmholtz problem. We will consider a nonoverlapping algorithm incorporating a Robin interface boundary condition (RIBC). Note that RIBCs impose the continuity of both the discrete solution uh and its normal flux on the subdomain interfaces, while most conforming FE methods admit discontinuities in the normal flux on the element interfaces. Thus, the RIBC should be modified appropriately in order for the DD method to converge to the original discrete solution.

2

Preliminaries

This section begins with a brief review for the existence and uniqueness of the weak solution of (1). We then present convergence properties for the FE solution of the Helmholtz problem. In the following, L2 (D) denotes the space of all square-integrable functions f on a domain D; (·, ·)D and  · 0,D are the corresponding inner product and norm, respectively. Analogously, H m (D) is the usual m-th order Sobolev spaces on D with the norm  · m,D for a positive integer m.

138

2.1

Y. Cha and S. Kim

Existence and Uniqueness of the Solution

The weak formulation of (1) is given by seeking u ∈ V = H 1 (Ω) such that (∇u, ∇v)Ω − ( K 2 u, v)Ω + iαu, vΓ = (S, v)Ω , ∀ v ∈ V, 

where (f, g)Ω =

(3)

 f gdx,

f, gΓ =

Ω

f gdσ. Γ

For a simpler presentation, we define the following bilinear form L(u, v; D) = (∇u, ∇v)D − ( K 2 u, v)D + iαu, v∂D∩Γ ,

D ⊂ Ω.

We cite the following lemma. Lemma 1. [11] The weak formulation (3) of the Helmholtz problem admits a unique solution u ∈ H 1 (Ω) for S ∈ L2 (Ω). 2.2

The Discrete Solution

Given a FE subspace V h ⊂ V ∩ Qhr , where Qhr is the space of the r-th order splines corresponding to the set of finite elements T h , r = 1, 2, · · ·, the FE approximation of the weak solution u of (3) is the function uh ∈ V h such that L(uh , v; Ω) = (S, v)Ω , ∀ v ∈ V h .

(4)

Let the approximation error eh = u − uh , where u is the solution of (3) and uh is the solution of (4). It is known [12] that (4) has a unique solution uh ∈ V h for p2 h sufficiently small and that eh 0,Ω = O(pr+2 hr+1 ),

eh 1,Ω = O(pr+1 hr ),

(5)

for certain classes of data S, e.g., Sr−1,Ω ≤ Cr u0,Ω , where Cr is independent of p. The algebraic system for (4) can be written as Auh = b.

(6)

It is extremely difficult to solve (6) for non-attenuate (q = 0) or slightly-attenuate (q is small) waves. The attenuation coefficient q is negligible for certain cases, e.g. ocean acoustics and optical waves in a vacuum. For the case that 0 ≤ q  p, it has been verified that relaxation methods such as Jacobi and SOR iterations does not converge and that nonsymmetric Krylov subspace algorithms (GCR [13], GMRES [14], etc.) either converge very slowly or have possible breakdowns [15,16,17]. The existence of a convergent nonsymmetric Krylov subspace algorithm for (6) is equivalent to the positive definiteness of the imaginary part of A (i.e., q(x) ≥ q0 > 0) [17].

Parallel Domain Decomposition Methods

139

Fig. 1. The domain decomposed into two nonoverlapping subdomains

3

The Domain Decomposition Method

This section introduces a nonoverlapping, iterative DD method for high-order FE solutions of the Helmholtz problem (1)-(2) of which the convergence can be analyzed by applying energy estimates [8]. For simplicity, we will consider 2D problems and choose a rectangular reference finite element with the shape functions being tensor products of 1D quadrature-based shape functions. Thus the arguments to be presented are applicable to the 3D space with a minor modification. For simplicity, we will present the DD algorithm in the algebraic formulation, rather than the variational formulation. 3.1

The DD Method in the Algebraic Formulation

Consider a rectangular domain of a uniform mesh, with two rectangular subdomains (Ω1 and Ω2 ); see 1. Let its degrees of freedom be ordered by Ω1 \Γ12 first, the interface Γ12 next, and then Ω2 \Γ12 . Then, the algebraic system (6) corresponding to the ordering can be written as ⎡ ⎤⎡ h⎤ ⎡ ⎤ A11 A12 0 u1 b1 ⎣ A21 A22 A23 ⎦ ⎣ uh2 ⎦ = ⎣ b2 ⎦ . (7) 0 A32 A33 b3 uh3 However, when a DD method is considered, the degrees of freedom on the interface Γ12 will be counted twice, one for each of Ω1 and Ω2 . Thus, the algebraic system corresponding to the DD method reads  u  h = b, A where

⎡

A11 ⎢ A21 =⎢ A ⎣ A21 0 Here uh12 = uh21 = uh2 .

A12 A22 0 0

0 0 A22 A32

⎤ 0 A23 ⎥ ⎥, A23 ⎦ A33

(8)

⎤ uh1 ⎢ uh12 ⎥ ⎥ h = ⎢ u ⎣ uh21 ⎦ , uh3 ⎡

⎡

⎤ b1 ⎢ ⎥  = ⎢ b2 ⎥ . b ⎣ b2 ⎦ b3

140

Y. Cha and S. Kim

Fig. 2. An example of nonoverlapping domain decomposition

The above system can be solved in a block-Jacobi manner, parallelized, with each subdomain problem being solved by a direct method such as the LU factorization. The following iterative procedure for (8) is formulated via a matrix split h,0 , find u  h,n , n = 1, 2, · · ·, ting, incorporating a stabilization term: For a given u by recursively solving +R u u  h,n = b  h,n−1 , P (9) where

⎡

A11 A12 0 ⎢ A21 A22 + iβD12 0  P =⎢ ⎣ 0 0 A22 + iβD21 0 0 A32

⎤ 0 0 ⎥ ⎥, A23 ⎦ A33

⎡

⎤ 0 0 0 0 ⎢ 0 iβD12 −A23 ⎥ =⎢ 0 ⎥. R ⎣ −A21 iβD21 0 0 ⎦ 0 0 0 0 Here Djk are diagonal matrices with their main diagonal elements being positive; one may choose D12 = D21 = diag{1, 1, · · · , 1}. The quantity β denotes a complex-valued relaxation parameter, β = βr − iβi , βr > 0, βi ≥ 0. Such a choice of β not only introduces a convergent sequence of the iterates { uh,n } but h,n also imposes the continuity of the iterates on the interface, i.e., u12 → uh,n 21 , as −R  is equivalent to A  when uh12 = uh21 . n → ∞. Note that P The above algorithm can be applied for a general number of subdomains M in a similar way; see Figure 2. The difference in the algebraic formulation (9) is  and R  consist of M blocks rather than two. that P Theorem 1. Assume q(x) ≥ q0 > 0 and let the relaxation parameter β = βr − iβi be given as C1 h−1 q2 ≤ βi < βr 02 . 4 p1

Parallel Domain Decomposition Methods

141

where C1 is a positive constant. Then the iterates uh,n of algorithm (9) of M subdomains, M ≥ 2, converge to the original discrete solution of (4), uh ∈ V h , in the following sense: uh,n |Ωj → uh |Ωj in L2 (Ωj ),

j = 1, · · · , M.

Furthermore, let the relaxation parameter β = βr − iβi be chosen as βi =

C1 h−1 , 4

βr = ξ βi

p21 , q02

h2 p2 q 4 1/2 ξ =1+ 1+ · 41 04 . 4C1 p1 + q0

Then the spectral radius of the iteration matrix of algorithm (9) is minimized and bounded as 4  −1 R)  ≤ 1 − C2 q0 h2 , ρ(P (10) p21 for some C2 > 0 independent of h, p, and q. For general cases, i.e., q ≥ q0 ≥ 0, we do not know of any convergence analysis for the DD algorithm (9). As mentioned in Section 2, the positivity of q0 is equivalent to the existence of a convergent nonsymmetric Krylov subspace algorithm [17,18]. However, the right-hand side of (10) is most sensitive to the attenuation coefficient q. When q increases (a little, though), the convergence would be significantly improved. This is the motivation of the following artificial damping iteration (ArtDI): Given a constant η > 0 and an initial guess uh,0 ∈ V h , find uh, ∈ V h ,  ≥ 1, by recursively solving L(uh, , v; Ω) + (iη 2 uh, , v)Ω = (S, v)Ω + (iη 2 uh,−1 , v)Ω , ∀ v ∈ V h .

(11)

One can show that the above iteration converges when the imaginary parts of all eigenvalues of A is nonnegative, which also can be proved for p21 h2 sufficiently small. Each step of the ArtDI algorithm (11) is a perturbation of (4), with the wave where K 2 = p2 − i(q 2 + η 2 ). Thus number K being changed by a new one K, one can solve each step by applying the DD method (9), as an inner iteration. The ArtDI algorithm in the algebraic formulation can be expressed as follows:  h,0 ; Set η > 0, ε >, and u For⎡ = 1, 2, · · ·  h,,0 = u  h,−1 ; u ⎢ For n = 1, 2, · · · , n∗ ⎢ ⎢  + iη 2 D  + iη 2 D) u u  h,,n−1 ;  h,,n = b ⎢ (P uh,−1 + R ⎢ h, h,,n ∗ ⎣u  =u  ;  h,−1 0 < ε, stop; If  uh, − u

(12)

where D is a diagonal matrix which is obtained from the numerical integration of the square of the shape functions for each of nodal points. In the above ArtDI algorithm, the inner iteration has been carried out incompletely, stopping in n∗ iterations; for n∗ sufficiently large, ArtDI would converge.

142

4

Y. Cha and S. Kim

Numerical Results

In this section, we verify accuracy and efficiency of algorithms (9) and (12) for solving the Helmholtz equation in 2D media by FE methods, for various choices of h, r, η, and n∗ . Set the domain Ω = (0, 1)2 ; we consider uniform quadrilateral elements of the edge length h = 1/np , for np > 0, and the finite element methods incorporating Legendre-Gauss-Lobatto splines of order r. The algorithm is implemented in C++ for the main function and FORTRAN for the others and carried out on a 16-node cluster of 2.4 Ghz Pentium 4 processors, with 512MB RAM memory each. The wave number is selected as p(x) = ω/v(x), where ω (:= 2πf ) denotes the angular frequency. Here f is the frequency. The wave speed v(x) is chosen as follows: v1 (x, y) ≡ 1, v2 (x, y) = 1.6 + | sin 3πx cos 4πy|,  2, if (x, y) ∈ [0.45, 0.75] × [0.55, 0.75], v3 (x, y) = 1, otherwise. Note that v2 is continuous (but not smooth) and v3 is piecewise constant. For the ABC, we set α(x) = p(x). Since we are interested in the propagation of waves in slightly-attenuate or non-attenuate media, the quality factor can be defined by Q :=

ω2 p2 = 2 2 ∈ (0, ∞], 2 q v q

(13)

where Q = ∞ for q = 0. The quality factor is known to be between 50 and 300 in most earth media. The higher it is, the less attenuate and therefore the harder to solve. In this article the attenuation coefficient q(x) will be determined from the selected values of Q, ω, and v(x) and utilizing (13). Let St (x) be the source that corresponds to the following true solution u(x) =

φ(x) · φ(y) , ω2

(14)

where φ(x) = eiω(x−1) + e−iωx − 2, and Sx0 (x) = δ(x − x0 ), for some x0 ∈ Ω. One can decompose the domain in various ways. However, for simplicity, we consider the element-wise decomposition; for parallelism, strips of subdomains are equally divided and assigned to the processors; see Figure 2. The wall-clock time is denoted by CPU (in seconds) and the iteration is stopped when the iterates satisfy the following stopping criterion uh,m − uh,m−1 ∞ < tol, uh,m ∞

Parallel Domain Decomposition Methods

143

Table 1. Effectiveness of ArtDI. Set S = St , v = v1 , Q = ∞, and tol =1.0e-04. 1/h = 160, r = 1, f = 5 η n∗ N CPU r0N 0 1 880 1.9 1.91e-02 10 10 770 1.3 1.90e-02 10 20 520 0.8 1.90e-02 20 20 800 1.5 1.90e-02

1/h = 320, r = 2, f = 30 η n∗ N CPU r0N 0 1 diverges 30 25 diverges 50 25 925 23.5 3.02e-03 50 50 1100 26.8 3.20e-03

where tol is the tolerance, m is n or  which is the iteration index respectively for (9) or (12). The tolerance will be chosen as tol = 10−γ , γ = 4 or 8, depending on the solution accuracy. Zero initial values are given: uh,0 ≡ 0, for all examples dealt in this article. The total number of DD iterations is denoted by N and therefore N = m for (9) and N = m · n∗ for (12). For S = St , the numerical error is measured by the relative L2 norm r0N =

uh,N − u0 , u0

where u is the true solution in (14). In table 1, we first verify effectiveness of the ArtDI. We have tested the algorithm for various choices of η and n∗ . As one can see from the table, the choice of (η, n∗ ) is important for the simulation of non-attenuate waves (Q = ∞). Note that the ArtDI method (12) with the choice (η, n∗ ) = (0, 1) becomes the DD algorithm (9). As shown in the convergence analysis, the DD algorithm (without ArtDI) fails to converge for most of high-frequency solutions in non-attenuate media. The iteration count n∗ must be set large enough to incorporate the ArtDI appropriately and effectively. It has been numerically verified that n∗ can be selected solely depending on the grid size h, while the selection of η is more complicated due to its dependence on both h and the wave number (ω/v). However, the quality factor (Q) and the degree of basis functions (r) have shown little effect on the selection of the parameters η and n∗ . For example, one may choose n∗ and η as follows: n∗ = 1/(κh),

κ = 8 ∼ 16,

η 2 = p2 /Qa ,

Qa = 10 ∼ 20,

(15)

where p denotes the L2 -average of p = ω/v over the domain and Qa is the artificial quality factor. Table 2 shows the performance of the ArtDI (12) for various spline orders r. Set S = St , Q = ∞, and tol =1.0e-08 for all the results. Note that r/h is set to be the same as 384 for the first part (f = 10) and 768 for the second part (f = 30), which implies that the number of grid points is the same in each part. As one can see from the table, the case r = 2 gives the smallest CPU time, while higher-order ones (r ≥ 3) produce more accurate results. When the velocity is

144

Y. Cha and S. Kim Table 2. Accuracy-efficiency test. Set S = St , Q = ∞, and tol = 1.0e − 08. f = 10 1/h r 384 1 192 2 128 3 96 4 f = 30 1/h r 768 1 384 2 256 3 192 4

N 2842 1140 840 720 N 6390 2117 1606 1380

v = v1 CPU r0N 28.8 2.59e-02 10.0 9.22e-05 11.7 1.19e-06 12.2 7.76e-08 v = v1 CPU r0N 263.8 1.72e-01 77.9 1.39e-03 91.3 1.98e-05 97.3 9.31e-07

N 3219 1580 1060 920 N 4410 2088 1430 1280

v = v2 CPU r0N 32.6 2.88e-03 13.8 1.56e-05 14.7 1.06e-06 15.7 7.71e-08 v = v2 CPU r0N 182.2 7.21e-03 77.2 7.71e-05 81.9 8.11e-06 90.1 8.70e-07

N 6815 1320 6180 9860 N 9135 3393 11572 12240

v = v3 CPU r0N 69.3 2.44e-02 11.6 8.65e-05 85.2 1.18e-06 167.3 8.55e-08 v = v3 CPU r0N 376.7 1.62e-01 125.4 1.31e-03 659.1 1.89e-05 860.6 9.30e-07

continuous (v = v1 and v = v2 ), the higher-order FE methods (r ≥ 3) turn out to be slightly more expensive computationally, while they improve accuracy a lot. One can easily expect that higher-order FE methods (r ≥ 3) may result in a more efficient method for a fixed accuracy, when the medium is smooth enough. On the other hand, for the discontinuous velocity (v = v3 ), the higher-order methods cost much more for a relatively small accuracy improvement over the quadratic FE method. We recommend to employ the FE method of quadratic splines for the simulation of waves in discontinuous media. We have tested the parallel efficiency, which turns out to be larger than 97% for all tested examples. Such a high efficiency is due to the application of LU factorization for subproblems.

5

Conclusions

A domain decomposition (DD) iterative procedure for solving the Helmholtz wave problem by high-order finite element (FE) methods has been considered. We have chosen nonoverlapping subdomains and employed a modified Robin interface boundary condition. Under certain assumptions on the mesh and the quadrature rule, we have proved the convergence of the algorithm for attenuate waves. For non-attenuate waves, we have introduced the artificial damping iteration (ArtDI) as an outer iteration of the DD method, the convergence of which can be proved when the inner iteration is solved accurately enough. The resulting algorithm combining the ArtDI and DD iterations has been tested for the numerical solution of the Helmholtz problem in 2D media for various spline orders r and diverse frequencies f . The FE method of quadratic splines is recommended for the simulation of waves in heterogeneous media; no apparent phase lag has appeared in the numerical solution for 10-15 grid points per wavelength.

Parallel Domain Decomposition Methods

145

Acknowledgment The work of S. Kim is supported in part by NSF grant DMS-0609815.

References 1. Clayton, R., Engquist, B.: Absorbing boundary conditions for acoustic and elastic wave calculations. Bull. Seismol. Soc. Amer. 67, 1529–1540 (1977) 2. Kim, S., Kim, S.: Multigrid simulation for high-frequency solutions of the Helmholtz problem in heterogeneous media. SIAM J. Sci. Comput. 24, 684–701 (2002) 3. Shaidurov, V., Ogorodnikov, E.: Some numerical methods of solving Helmholtz wave equation. In: Cohen, G., Halpern, L., Joly, P. (eds.) Mathematical and Numerical Aspects of Wave Propagation Phenomena, pp. 73–79. SIAM, Philadelphia (1991) 4. Douglas Jr., J., Hensley, J.L., Roberts, J.E.: An alternating-direction iteration method for Helmholtz problems. Appl. Math. 38, 289–300 (1993) 5. Zienkiewicz, O.C.: Achievements and some unsolved problems of the finite element method. Internat. J. Numer. Methods Engrg. 47, 9–28 (2000) (Richard H. Gallagher Memorial Issue) 6. Kim, S.: A parallelizable iterative procedure for the Helmholtz problem. Appl. Numer. Math. 14, 435–449 (1994) 7. Kim, S.: Parallel multidomain iterative algorithms for the Helmholtz wave equation. Appl. Numer. Math. 17, 411–429 (1995) 8. Kim, S.: Domain decomposition iterative procedures for solving scalar waves in the frequency domain. Numer. Math. 79, 231–259 (1998) 9. Kim, S., Lee, M.: Artificial damping techniques for scalar waves in the frequency domain. Computers Math. Applic. 31(8), 1–12 (1996) 10. Kim, S., Shin, C., Keller, J.: High-frequency asymptotics for the numerical solution of the Helmholtz equation. Appl. Math. Letters 18, 797–804 (2005) 11. Douglas Jr., J., Santos, J.E., Sheen, D.: Approximation of scalar waves in the space-frequency domain. Math. Models Methods Appl. Sci. 4, 509–531 (1994) 12. Bayliss, A., Goldstein, C., Turkel, E.: On accuracy conditions for the numerical computation of waves. J. Comput. Phys. 59, 396–404 (1985) 13. Eisenstat, S., Elman, H., Schultz, M.: Variational iterative methods for non– symmetric systems of linear equations. SIAM J. Numer. Anal. 20, 345–357 (1983) 14. Saad, Y., Schultz, M.: GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 7, 856–869 (1986) 15. Bayliss, A., Goldstein, C., Turkel, E.: An iterative method for the Helmholtz equation. J. Comput. Phys. 49, 443–457 (1983) 16. Faber, V., Manteuffel, T.: Necessary and sufficient conditions for the existence of a conjugate gradient method. SIAM J. Numer. Anal. 21, 352–362 (1984) 17. Joubert, W., Young, D.: Necessary and sufficient conditions for the simplification of the generalized conjugate–gradient algorithms. Linear Algebra Appl. 88/89, 449– 485 (1987) 18. Freund, R.W.: Conjugate gradient–type methods for linear systems with complex symmetric coefficient matrices. SIAM J. Sci. Stat. Comput. 13, 425–448 (1992)

Self-Organizing Neural Grove and Its Distributed Performance Hirotaka Inoue Department of Electrical Engineering and Information Science, Kure National College of Technology, 2-2-11 Agaminami, Kure-shi, Hiroshima, 737-8506 Japan [email protected]

Abstract. In this paper, we present the improving capability of accuracy and the parallel efficiency of self-organizing neural groves (SONGs) for classification on a MIMD parallel computer. Self-generating neural networks (SGNNs) are originally proposed on adopting to classification or clustering by automatically constructing self-generating neural tree (SGNT) from given training data. The SONG is composed of plural SGNTs each of which is independently generated by shuffling the order of the given training data, and the output of the SONG is voted all outputs of the SGNTs. We allocate each of SGNTs to each of processors in the MIMD parallel computer. Experimental results show that the more the number of processors increases, the more the classification accuracy increases for all problems.

1

Introduction

Neural networks have been widely used in the field of intelligent information processing such as classification, clustering, prediction, and recognition. Generally, these neural networks have to be decided the network structure and some parameters by human experts. It is quite tricky to choose the right network structure suitable for a particular application at hand. Concerning the design of the network structure, the following must be decided, (i) the number of the network layers, (ii) the number of the neurons of each layer, (iii) the weights on connection between consequent layers. During learning iterations, the weights on connections of the given networks are updated so as to converge to target value conserving the initially decided static network structure. Consequently, obtaining the right structure of each network is the most important factor in learning and also the most difficult problem in the design of neural networks. In order to avoid these tricky and difficult situations, Self-generating neural networks (SGNNs) are focused an attention because of their simplicity on networks design [1]. SGNNs are some kinds of extension of the self-organizing maps (SOMs) of Kohonen [2] and utilize the competitive learning algorithm which is implemented as self-generating neural tree (SGNT). The SGNT algorithm is proposed in [3] to generate a neural tree automatically from training data directly. In our previous study concerning the performance C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 146–155, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Self-Organizing Neural Grove and Its Distributed Performance

147

analysis of the SGNT algorithm [4], we showed that the main characteristic of this SGNT algorithm was its high speed convergence in computation time but it was always not best algorithm in its accuracy comparing with the existing other feed-forward neural networks such as the back-propagation (BP). In order to improve the generalization capability of SGNNs, we proposed ensemble self-generating neural networks (ESGNNs) for classification [5]. ESGNNs apply ensemble averaging [6] to SGNNs and fully utilize the high speed convergence characteristics of the SGNT algorithm. Although ESGNNs are improved the accuracy by using various SGNTs, the computation time and the memory capacity are increased in proportion to increase the number of SGNTs. Therefore, we proposed a novel pruning method for the structure of the ESGNNs to reduce the computation time and the memory capacity and we called this model as self-organizing neural grove (SONG) [7]. Ensemble learning has been studied many AI and neural network researchers. Breiman proposed bagging predictors to improve the accuracy of CART [8] and investigated bagging performance on CART and other methods for classification and regression problems in [9]. Since ensemble learning is a variance-reduction technique, it is well known that ensemble learning tends to work well for methods with high variance such as neural networks and tree-based methods. In this paper, we present the improving capability of accuracy and the parallel efficiency of the SONG for classification on a MIMD parallel computer. We apply to three problems in the UCI repository [10] which are given as benchmark.

2

Self-Organizing Neural Grove

In this section, we describe how to prune redundant leaves in the SONG. First, we mention the on-line pruning method in learning of SGNT. Second, we show the optimization method in constructing the SONG. 2.1

On-Line Pruning of Self-Generating Neural Tree

SGNT is based on SOM and implemented as a competitive learning. The SGNT can be constructed directly from the given training data without any intervening human effort. The SGNT algorithm is defined as a tree construction problem of how to construct a tree structure from the given data that consist of multiple attributes under the condition that the final leaves correspond to the given data. Before we describe the SGNT algorithm, we denote some notations. – – – – – –

input data vector: ei ∈ IRm . root, leaf, and node in the SGNT: nj . weight vector of nj : w j ∈ IRm . the number of the leaves in nj : cj . distance measure: d(ei , wj ). winner leaf for ei in the SGNT: nwin .

148

H. Inoue

Input: A set of training examples E = {e_i}, i = 1, ... , N. A distance measure d(e_i,w_j). Program Code: copy(n_1,e_1); for (i = 2, j = 2; i <= N; i++) { n_win = choose(e_i, n_1); if (leaf(n_win)) { copy(n_j, w_win); connect(n_j, n_win); j++; } copy(n_j, e_i); connect(n_j, n_win); j++; prune(n_win); } Output: Constructed SGNT by E. Fig. 1. SGNT algorithm Table 1. Sub procedures of the SGNT algorithm Sub procedure copy(nj , ei /w win ) choose(ei , n1 ) leaf (nwin ) connect(nj , nwin ) prune(nwin )

Specification Create nj , copy ei /w win as w j in nj . Decide nwin for ei . Check nwin whether nwin is a leaf or not. Connect nj as a child leaf of nwin . Prune leaves if the leaves have the same class.

The SGNT algorithm is a hierarchical clustering algorithm. The pseudo C code of the SGNT algorithm is given in Fig. 1. In Fig. 1, several sub procedures are used. Table 1 shows the sub procedures of the SGNT algorithm and their specifications. In order to decide the winner leaf nwin in the sub procedure choose(e i,n 1), the competitive learning is used. If an nj includes the nwin as its descendant in the SGNT, the weight wjk (k = 1, 2, . . . , m) of the nj is updated as follows: wjk ← wjk +

1 · (eik − wjk ), cj

1 ≤ k ≤ m.

(1)

After all training data are inserted into the SGNT as the leaves, the leaves have each class label as the outputs and the weights of each node are the averages of the corresponding weights of all its leaves. The whole network of the SGNT reflects the given feature space by its topology. For more details concerning how to construct and perform the SGNT, see [3]. Note, to optimize the structure

Self-Organizing Neural Grove and Its Distributed Performance

149

Input T

SGNT 1

SGNT 2

o1

...

o2

SGNT K

oK

Combiner

Σ

Output

o

Fig. 2. An MCS which is constructed from K SGNTs. The test dataset T is entered each SGNT, the output oi is computed as the output of the winner leaf for the input data, and the MCS’s output is decided by voting outputs of K SGNTs.

of the SGNT effectively, we remove the threshold value of the original SGNT algorithm in [3] to control the number of leaves based on the distance because of the trade-off between the memory capacity and the classification accuracy. In order to avoid the above problem, we introduce a new pruning method in the sub procedure prune(n win). We use the class label to prune leaves. For leaves connected to the nwin , if those leaves have the same class label, then the parent node of those leaves is given the class label and those leaves are pruned. In the next sub-section, we describe how to optimize the structure of the SGNT in the SONG to improve the classification accuracy. 2.2

Optimization of the SONG

The SGNT has the capability of high speed processing. However, the accuracy of the SGNT is inferior to the conventional approaches, such as nearest neighbor, because the SGNT has no guarantee to reach the nearest leaf for unknown data. Hence, we construct an MCS by taking the majority of plural SGNT’s outputs to improve the accuracy (Figure 2). Although the accuracy of the SONG is comparable to the accuracy of conventional approaches, the computational cost increases in proportion to increase in the number of SGNTs in the SONG. In particular, the huge memory requirement prevents the use of the SONG for large datasets even with latest computers. In order to improve the classification accuracy, we propose an optimization method of the SONG for classification. This method has two parts, the merge phase and the evaluation phase. The merge phase is performed as a pruning algorithm to reduce dense leaves (Figure 3). This phase uses the class information and a threshold value α to decide which subtree’s leaves to prune or not. For leaves that have the same parent node, if the proportion of the most common class is greater than or equal to the threshold value α, then these leaves are pruned and the parent node is

150

H. Inoue 1 begin initialize j = the height of the SGNT 2 do for each subtree’s leaves in the height j 3 if the ratio of the most class ≥ α, 4 then merge all leaves to parent node 5 if all subtrees are traversed in the height j, 6 then j ← j − 1 7 until j = 0 8 end. Fig. 3. The merge phase 1 begin initialize α = 0.5 2 do for each α 3 evaluate the merge phase with 10-fold CV 4 if the best classification accuracy is obtained, 5 then record the α as the optimal value 6 α ← α + 0.05 7 until α = 1 8 end. Fig. 4. The evaluation phase

given the most common class. The optimum threshold values α of the given problems are different from each other. The evaluation phase is performed to choose the best threshold value by introducing 10-fold cross validation (Figure 4). 2.3

Simple Example of the Pruning Method

We show an example of the pruning algorithm in Figure 5. This is a twodimensional classification problem with two equal circular Gaussian distributions that have an overlap. The shaded plane is the decision region of class 0 and the other plane is the decision region of class 1 by the SGNT. The dotted line is the ideal decision boundary. The number of training samples is 200 (class0: 100,class1: 100) (Figure 5(a)). The unpruned SGNT is given in Figure 5(b). In this case, 200 leaves and 120 nodes are automatically generated by the SGNT algorithm. In this unpruned SGNT, the height is 7 and the number of units is 320. In this, we define the unit to count the sum of the root, nodes, and leaves of the SGNT. The root is the node which is of height 0. The unit is used as a measure of the memory requirement in the next section. Figure 5(c) shows the pruned SGNT after the merge phase in α = 1. In this case, 159 leaves and 107 nodes are pruned away and 54 units remain. The decision boundary is the same as the unpruned SGNT. Figure 5(d) shows the pruned SGNT after the merge phase in α = 0.6. In this case, 182 leaves and 115 nodes are pruned away and only 23 units remain. Moreover, the decision boundary is improved more than the unpruned SGNT because this case can reduce the effect of the overlapping class by pruning the SGNT.

Self-Organizing Neural Grove and Its Distributed Performance 1

class0 class1

class0 class1 node

Height

0.8

151

7 6 5 4 3 2 1 0

x2

0.6

0.4

1 0.8

0.2 0

0.6 0.2

0 0

0.2

0.4

0.6

0.8

0.4 x1

0.4 0.6

x2

0.2

0.8

1 0

1

x1

(a)

(b) class0 class1 node

Height

class0 class1 node

Height

7 6 5 4 3 2 1 0

7 6 5 4 3 2 1 0 1

1

0.8 0

0.8

0.6 0.2

0.4 x1

0.4 0.6

0

x2

0.2

0.8

1 0

0.6 0.2

0.4 x1

(c)

0.4 0.6

x2

0.2

0.8

1 0

(d)

Fig. 5. An example of the SGNT’s pruning algorithm, (a) a two dimensional classification problem with two equal circular Gaussian distribution, (b) the structure of the unpruned SGNT, (c) the structure of the pruned SGNT (α = 1), and (d) the structure of the pruned SGNT (α = 0.6). The shaded plane is the decision region of class 0 by the SGNT and the doted line shows the ideal decision boundary. 1

1

class0 class1

0.6

0.6 x2

0.8

x2

0.8

class0 class1

0.4

0.4

0.2

0.2

0

0 0

0.2

0.4

0.6 x1

(a)

0.8

1

0

0.2

0.4

0.6

0.8

1

x1

(b)

Fig. 6. An example of the MCS’s decision boundary (K = 25), (a) α = 1, and (b) α = 0.6. The shaded plane is the decision region of class 0 by the MCS and the doted line shows the ideal decision boundary.

To show how well the MCS is optimized by the pruning algorithm, we show an example of the MCS in the same problem used above. Figure 6(a) and Figure 6(b) show the decision region of the MCS in α = 1 and α = 0.6, respectively. We set

152

H. Inoue

the number of SGNTs K as 25. The result of Figure 6(b) is a better estimation of the ideal decision region than the result of Figure 6(a). We investigate the optimization method for more complex problems in the next section.

3

Distributed Processing

Because each SGNT of the SONG can train and test independently, the SONG has a possibility of the parallel computation at the training process and the testing process. Hence, we allocate each of SGNTs to each of processors on the MIMD computer. The procedure of the parallelization of the SONG is presented as follows: Step1: In a master processor, read the training set D and the test set T in the disk. Step2: In the master processor, broadcast D and T for all K−1 slave processors. Step3: In all processors, generate the SGNT from D, then test the SGNT using T , and compute the ok independently. Step4: In all processors, each output ok for T is collected in the master processor by all to one communication. Step5: In the master processor, compute o by voting and write to the disk. Because the number of the communications between the master processor and each slave processor is only two times (Step2 and Step4), the parallel efficiency is approximately expected the linear speedup. In our case, all computations are performed on the Intel Paragon (Paragon XP/S15). This is a distributed memory multicomputer, and the architecture is multiple instruction multiple data (MIMD). The Paragon we use has 296 processors. Each processor is Intel i860XP (50MHz). The network topology of the Paragon is adopted the twodimensional mesh.

4

Experimental Results

We allocate each of SGNTs to each of processors on the Paragon, and compute 100 trials for each single/ensemble model. The number of processors (SGNTs) K for the ensemble averaging is changed from 1 to 201 (1,3,5,7,9,15,25,51,101,151 and 201), and the threshold value α is 1 for each SONG. In order to reduce the redundant execution, we repeated 100 trials from Step3 to Step5 in prior section continuously. In order to investigate the parallel performance of the SONG, we select three typical classification problems, which are given as benchmark problems in UCI repository [10]. Next, we describe the brief explanation of these problems. breast-cancer-wisconsin: This problem is a binary classification task for classify a tumor as either benign or malignant based on cell descriptions gathered by a microscopic examination. Input attributes are:

Self-Organizing Neural Grove and Its Distributed Performance

153

– the clump thickness, – the uniformity of cell size, – cell shape, – the amount of magical adhesion, – the frequency of bare nuclei, etc. This problem has 9 attributes, 699 examples. Each attribute consists of continuous real value. ionosphere: This problem is a binary classification task for a radar as either good or bad based on the complex electromagnetic signals. The targets were free electrons in the ionosphere. “Good” radar returns are those showing evidence of some type of structure in the ionosphere. “Bad” returns are those that do not; their signals pass through the ionosphere. This problem has 34 attributes, 351 examples. Each attribute consists of continuous real value. letter-recognition: The objective is to identify each of a large number of blackand-white rectangular pixel displays as one of the 26 capital letters in the English alphabet. The character images were based on 20 different fonts and each letter within these 20 fonts was randomly distorted to produce a file of 20,000 unique stimuli. Each stimulus was converted into 16 primitive numerical attributes (statistical moments and edge counts), which were then scaled to fit into a range of integer values from 0 through 15. In this paper, we use below defined classification accuracy. classification accuracy =

number of correct . number of test data

(2)

We evaluate the classification accuracy using 10-fold cross-validation [11] for above problems. Figure 7(a), (b), and (c) show the influence of the number of processors on classification accuracy for breast-cancer-wisconsin, ionosphere and letter-recognition problems respectively. Classification accuracies are improved by computing the ensemble averaging of various SGNTs for all problems. Here, each classification accuracies shows the average of 100 trials and its error-bar. It is shown that the classification accuracies are improved by computing the ensemble averaging of various SGNTs for all problems. Especially, the minimum classification accuracies are largely improved for all problems. The improvement ability is obtained from small K most effectively. The classification accuracy of larger than 51 SGNTs is convergence for all problems. Figure 8(a),(b), and (c) show the relation between the number of processors and the execution times for breast-cancer-wisconsin, ionosphere, and letterrecognition problems respectively. The execution times are gradually saturated as the number of processors increase. As the scale of the dataset grows, the proportion of the communication time, i.e. the difference of the total time and the training time + the testing time, for the total time is decrease. This means that this method have an approximately linear speedup for large-scale datasets.

154

H. Inoue 0.98

0.92

breast-cancer-wisconsin

0.97

ionosphere

0.96 0.955 0.95 0.945

Classification accuracy

0.965

letter

0.96

0.9

0.97

Classification accuracy

Classification accuracy

0.975

0.88 0.86 0.84 0.82

0.94

0.95 0.94 0.93 0.92 0.91 0.9 0.89

0.8

0.935 0.93

0.88

0.78 0

50

100 # of processors

150

0.87

200

0

50

(a)

100 # of processors

150

200

0

50

(b)

100 # of processors

150

200

(c)

2.4

230

1.7

2.3

225

1.6 1.5 1.4 1.3 1.2 Total time Training time + Testing time Training time

1.1 1 0

50

100 # of processors

150

Computation time (sec.)

1.8

Computation time (sec.)

Computation time (sec.)

Fig. 7. Influence of the number of processors on classification accuracy for (a) breastcancer-wisconsin, (b) ionosphere, and (c) letter-recognition

2.2 2.1 2 1.9 1.8 Total time Training time + Testing time Training time

1.7 1.6 200

(a)

0

50

100 # of processors

(b)

150

220 215 210 205 200 195 Total time Training time + Testing time Training time

190 185 200

0

50

100 # of processors

150

200

(c)

Fig. 8. Relation between the number of processors and execution time (seconds) for (a) breast-cancer-wisconsin, (b) ionosphere, and (c) letter-recognition

Consequently, a parallel distributed computing using SONG can obtain more higher classification accuracy than the single SGNT by allocating each of SGNTs to each of processors, go on maintaining the high speed processing property of the single SGNT.

5

Conclusions

In this paper, we presented the distributed computing with SONG to obtain more effective implementation for classification on the MIMD parallel computer. From the experimental results the following conclusions can be drawn: – Distributed computing with SONG can improve the classification accuracy using various SGNTs which are allocated processors on the MIMD computer. – Distributed computing with SONG can perform a task with the high parallel efficiency by allocating each of SGNTs to each of processors on the MIMD computer. In the future work, we will study an incremental learning of SONG for large-scale data mining.

Acknowledgements The authors would like to thank the Information Processing Center in Okayama University of Science for using the Paragon.

Self-Organizing Neural Grove and Its Distributed Performance

155

References 1. Wen, W.X., Pang, V., Jennings, A.: Self-generating vs. self-organizing, what’s different? In: Simpson, P.K. (ed.) Neural Networks Theory, Technology, and Applications. IEEE Technology Update Series, pp. 210–214. IEEE Technical Activities Board, Piscataway (1996) 2. Kohonen, T.: Self-Organizing Maps. Springer, Berlin (1995) 3. Wen, W.X., Jennings, A., Liu, H.: Learning a neural tree. In: The International Joint Conference on Neural Networks, Beijing, China, vol. 2, pp. 751–756, November 3-6 (1992) 4. Inoue, H., Narihisa, H.: Efficiency of Self-Generating Neural Networks Applied to Pattern Recognition. Int. J. of Mathematical and Computer Modelling 38(11-13), 1225–1232 (2003) 5. Inoue, H., Narihisa, H.: Improving generalization ability of self-generating neural networks through ensemble averaging. In: Terano, T., Liu, H., Chen, A.L.P. (eds.) PAKDD 2000. LNCS (LNAI), vol. 1805, pp. 177–180. Springer, Heidelberg (2000) 6. Haykin, S.: Neural Networks: A comprehensive foundation, ch. 7, 2nd edn. PrenticeHall, Upper Saddle River (1999) 7. Inoue, H.: Self-organizing neural grove: Efficient multiple classifier system with pruned self-generating neural trees. In: Proc. The 18th International Conference on Artificial Neural Networks, Part I, Prague, Czech Rep., pp. 613–622, September 3-6 (2008) 8. Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth, California (1984) 9. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996) 10. Asuncion, A., Newman, D.J.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine (2007), http://www.ics.uci.edu/~ mlearn/MLRepository.html 11. Stone, M.: Cross-validation: A review. Math. Operationsforsch. Statist., Ser. Statistics 9(1), 127–139 (1978)

A Massively Parallel Hardware for Modular Exponentiations Using the m-ary Method Marcos Santana Farias1 , S´ergio de Souza Raposo1 , Nadia Nedjah1 , and Luiza de Macedo Mourelle2 1

Department of Electronics Engineering and Telecommunications 2 Department of System Engineering and Computation, Engineering Faculty, State University of Rio de Janeiro, Brazil

Abstract. Most of cryptographic systems are based on modular exponentiation. It is performed using successive modular multiplications. One way of improving the throughput of a cryptographic system implementation is reducing the number of the required modular multiplications. Existing methods attempt to reduce this number by partitioning the exponent in constant or variable size windows. In this paper, in the purpose of further accelerating the computation of modular exponentiation, a concurrent novel approach is proposed along with hardware implementation of the concurrent m-ary method. We compare the proposed method to the sequential implementation. Keywords: Modular exponentiation, m-ary method, concurrency.

1 Introduction Modular exponentiation is a very important operation for the public key cryptography systems such as RSA [9], which is an encryption algorithm considered one of the safest. The algorithm encrypts and decrypts information performing the operation C = T E mod M , wherein E is called exponent and M modulus is the module chosen from the product of two primes. Note that the larger the prime numbers are, the more secure the process is. There are many efficient algorithms [2,1,6] that permit the computation of a modular power given the necessary data, i.e. T , E and M . Some of these algorithms [1,6] schedule the underlaying modular multiplications in parallel in order to improve the processing time. Here, we propose a parallel implementation of the m-ary modular exponentiation method. The m-ary methods for exponentiation [10] may be thought of as a three steps procedure: (a) partitioning the exponent E in w d-bit windows; (b) pre-computing all possible powers of a windows; (c) iterating the squaring of the partial result and then multiplying it by the power in the next window. The m-ary methods scans the digits of E from the less significant to the most significant digit and groups them into partitions of equal length. The most popular method is the binary exponentiation as it uses the binary representation of the exponent E = en−1 en−2 . . . e1 e0 as described in Algorithm 1. A generalization of Algorithm 1 to using groups of d bits instead of a single bit, called the m-ary method, is given in Algorithm 2. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 156–165, 2010. c Springer-Verlag Berlin Heidelberg 2010 

A Massively Parallel Hardware for Modular Exponentiations

157

Algorithm 1. Binary Method [3] Require: T , M , E; Ensure: C = T E mod M ; C := T en−1 mod M ; for i := n − 2 downto 0 do C := C 2 mod M ; if ei = 0 then C := C × T ei mod M ; end if end for return C;

Algorithm 2. m-aryMethod [3] Require: T , M , E, d = log2 m; Ensure: C = T E mod M ; for i := 2 to 2d − 1 do Store T i mod M ; end for C := T vw−1 mod M ; for i := w − 2 downto 0 do d C := C 2 mod M ; if vi = 0 then C := C × T vi mod M ; end if end for return C;

There exists many hardware implementation of the modular exponentiation [3,8,7]. However, these are all sequential. The solution developed in this paper seeks the parallel implementation of the exponentiation T E mod M using several multipliers. As in the m-ary method [10], the exponent E is divided into w partitions or windows with d bits. The computation can be described as in (1). (w−1)d

T E = (T pw−1 )2

× . . . × (T pi )2 × . . . × (T p1 )2 × T p0 , id

d

(1)

wherein exponent E is viewed as in (2). E = pw−1 × 2(w−1)d + . . . + pi × 2id + . . . + p1 × 2d + p0

(2)

From (1), one can envision the computation of all (T pi )2 in parallel once the precomputation of (T pi ) mod M has been completed. Note that the number of modular id multiplications required in the computation of (T pi )2 mod M is always one multipli(i−1)d cation shorter than that needed in (T pi−1 )2 mod M for all i ≥ 1. Therefore, once d the final result corresponding to the computation of (T p1 )2 mod M has been yield and, as the result of T p0 is already available (from pre-computation), then one can immed diately proceed with the computation of the modular multiplication C = (T p1 )2 × id

158

M. Santana Farias et al.

T p0 mod M . Concurrently with that multiplication, the last one in the exponentiation 2d 2d (T p2 )2 mod M can occur. Hence, the computation of C = C ×(T p2 )2 mod M may id occur in the sequel. Similarly, we can infer that, as soon as power (T pi )2 mod M is id obtained, one can proceed with the modular multiplication C × (T pi )2 mod M , yielding a new partial result and so forth until the final power is obtained. Note that no extra delays are needed to synchronize the squaring and the multiplications steps. The parallelization of the computation of T E mod M is formally described using the Petri net model of Fig. 1, wherein a transition represents the computation indicated by its label and the places with a token symbolize the availability of the result. The net shows the main stages of the proposed concurrent modular exponentiator. The upper place and transition refer to the pre-computation of the required powers of T . Hereafter, this is called the pre-computation stage. Ideally, the end of this stage would trigger the start of all squaring (squaring stage), whose end would, in turn, triggers the prescribed modular multiplication (multiplication stage). However, in an implementation with a single-port shared power memory, this would not be the case as the pre-computed powers are all kept in that memory. Among other improvements, the proposed hardware architecture avoids this kind of delays to further improve the performance of the parallel modular exponentiator. The rest of this paper is organized as follows: first, in Section 2, we describe the proposed architecture for a modular exponentiator and explain its underlaying operation; thereafter, in Section 3, we present some performance results to assess the efficiency of the proposed implementations; last but not least, in Section 4, we draw some conclusions and point out some directions for future work.

2 The Proposed Architecture Fig. 2 shows the macro-architecture of the proposed modular exponentiator. It includes the power memory (PMEM), a scalable number of modular multipliers (MMULTs), the main controller (MCTRL), wherein MMULTs receive their operands from PMEM via the shared data bus (DBUS). Memory PMEM is a shared RAM, which stores the repository of the pre-computed powers of T . Modular multiplier MMULT implements a modular multiplication using the Montgomery algorithm [10]. The modular multiplier details can be found in [5]. Each MMULT performs iteratively in order to provide a given power of one of the pre-computed values stored in PMEM. Controller MCTRL supervises the operation of the MMULTs by controlling several signals, which synchronize the data-path components through the three stages of the modular power computation. Binary-coded exponent is split into w ≥ 2 partitions. Each partition comprises d ≥ 2 bits. The number of MMULTs coincides with the number of partitions. In the following, we describe the three processing stages of the proposed modular exponentiator: pre-computation; squaring and multiplication. 2.1 Pre-computation Besides the basic operation that is performed by all modular multipliers MMULTs, the highest order MMULT(w−1) also pre-computes 2d −1 modular powers of T : T 2 mod

A Massively Parallel Hardware for Modular Exponentiations

159

Fig. 1. Petri net model for the parallelization of the computation of T E

M, . . . , T 2 −1 mod M , and store them in PMEM. Multiplier MMULT(w−1) performs these pre-computations thanks to a slight adjustment using three extra multiplexers Mux3, Mux4 and MuxE, and two registers RegE and Cnt1, as shown in Fig. 2 and Fig. 3. In this stage, MuxE forwards the signal from Cnt1, initially set to constant 1. This counter always points to the PMEM address, wherein the result of the multiplication would be recorded. During the computation of the first modular multiplication, Mux3 switches to T and Mux4 to 1. Then, multiplexers Mux1 and Mux2 that are associated with MMULT(w−1) forward Mux3 and Mux4 outputs, respectively. In addition to MCTRL, MMULTs require local controllers to operate concurrently. Yet during first multiplication, the MMULT controller signals to Mux1 and Mux2 as to forward the data coming from Mux3 and Mux4 outputs. The MMULT controller, through a tri-state buffer, writes data on DBUS when its down-counter reaches 0. At the present stage, this counter stores value 1 and on every multiplication done the result is written on DBUS. When first multiplication is ended up, data is put on DBUS (T mod M ) and MCTRL enables PMEM to write (RD = 0) at position 1, provided by Cnt1. Counter Cnt1 is incremented and now Mux4 switches to data on DBUS. The modular product obtained in each iteration is stored at the proper PMEM address, e.g., T 2 mod M is stored at address 2, T 3 mod M at address 3 and so on. Note that counter Cnt1 has d bits. When d

160

M. Santana Farias et al.

all bits are high, an AND gate (not represented in Fig. 3 for readability reasons only) signals to MCTRL that pre-computation is through and, so squaring must commence. M 1

A SelMux3

Mux _1

Mux _2

Mux _1

Mux _4

B

A

MMult 1

T

Mux _2

B

SelMux4

( w -1)

P

Mux _1

A

B

Mux _3 MMult

Mux _2

ZeroDC

( w -2)

MMult

0

P ZeroDC

P ZeroDC

:

:

REG_Final

DBUS

ZeroDC ZeroDC

. . .

. . . ZeroDC ZeroDC

E

REG_ w

( w -1)

Data Addr

( w -2)

CTRL 1 0

MMStart

Addr

OE

OE

RD

RD

PMEM

( w -1)

MMStartFinal SelMux3 SelMux4

Fig. 2. Macro-architecture of the parallel exponentiator

2.2 Squaring In this stage, controller MCTRL permits for MMULTs to be fed with the pre-computed powers stored in PMEM. Multiplier MMULTi is provided with the word content whose address consist to the d bits of the corresponding partition in exponent E. Exponent E is stored in REGE, which is a shift register of d bits at a time. The most significant d bits of REGE are fed into PMEM address, which allows the exponentiator to retrieved the necessary pre-computed power for each one of the MMULTs before squaring starts. After each PMEM read cycle, MCTRL performs d left-shifts to get the next power address in PMEM. The launch of all MMULTs operation is controlled by a single wbit wide right-shift register, shown are in Fig. 3 as REGw. This register starts with value 2(w−1) , i.e. 1 in the most significant bit and 0 in the remaining d − 1 bits. Rightshifting this register, MCTRL automatically signals to the corresponding MMULTi , setting bit i of REGw, to start operating. When REGw gets zeroed, MCTRL knows that all MMULTs have been initialized. Now it is time to move to next stage: multiplication. 2.3 Multiplication This stage multiplies the partial results obtained by MMULTs. Multiplier MMULT0 is elected to perform the necessary multiplications. It was chosen to run this task because it becomes idle first. Its controller waits for bit 0 of REGw to be set so it loads 1 in one of the inputs of MMULT0 and the other with value on DBUS which is T mod M , retrieved form PMEM. Afterwards, whenever a MMULT completes squaring and flags via signal ZeroDCi , MMULT0 uses DBUS data to obtain the other input. When the this done for all MMULTi , register REGFinal has the exponentiation final result. Each MMULT, as described in Fig. 4, includes a modular multiplier (MM) that implements Montgomery multiplication algorithm [5], a down-counter, which used to control the number of times the power, read from PMEM, is squared, and an independent controller. Fig. 5 presents the state machine of the main controller. The description of each state is presented as follows.

A Massively Parallel Hardware for Modular Exponentiations

ExpStart

LER1

MMDone

Inc_ CNT1 LERE

End_Pre_comp

MMStart RD

1

REGE

CNT1

d

ZeroRegW MMDone _Final

E

d

LShift _RE

Main Controller State Machine

SelMuxE MuxE OE ExpDone

SelMux3

RShift _ RegW

SelMux4

LERW

d

ADDR

REGw ZeroDC ( w -2) . . . ZeroDC 1

MMStart _Final

w

Regw

Fig. 3. Main controller (MCTRL) details

– S0 : initialize system; if start signal is asserted then go to S1 ; – S1 : start MMULT(w−1) for pre-computation; go to S2 ; – S2 : if MM has finished then go to S3 ; – S3 : start result write into PMEM; go to S4 ; – S4 : finish result write into PMEM; go to S5 ; – S5 : increment counter Cnt1; go to S6 ; – S6 : if pre-computation complete then go to S12; else go to S7 ; – S7 : start MMULT(w−1) with Mux4 forwarding data from DBUS; – S8 : if MM has finished then go to S9 ; – S9 : start result write into PMEM; go to S10 ; – S10 : finish result write into PMEM; go to S11 ; – S11 : increment counter Cnt1; go to S6 ; – S12 : MuxE switches to RegE; Mux3 and Mux4 forward data from DBUS; start MMULT(w−1) to perform squaring; – S13 : read PMEM at most significant partition; – S14 : left-shift REGE d bits; right-shift REGw bit; – S15 : start MMULTi corresponding to bit i, set in REGw; – S16 : if all MMULTs have been initiated then go to S17 ; else go to S13

161

162

M. Santana Farias et al.

A

B

M

Sel _ mux2

Sel _ mux1 RegA

RegM

LEABM

RegB

MMStart MMult CRTL ZeroDC

LEP

clk

MMDone MM Down Down Counter RegP

ZeroDC

Fig. 4. MMULT details

S1

S0

S2

S3

S4

S5

Start = 0 MMDone0

S11

=0

S10

End of Pre-computation = 0

S9

S8

MMDone0

S7

S6

=0

End of Pre-computation =1 Zero_REG_ w = 0

S16

S19

S15

S14

Zero_REG_ w = 1

S18

MMDoneFinal

S17

=0

MMDone0

=0

Fig. 5. State Machine to the main controller

S13

S12

A Massively Parallel Hardware for Modular Exponentiations

163

ExpStart ExpDone CLK T 4759

21397

37783

M 5073

977

9169

E 191

60

Adress 01

10

0 4...

Data bus

11

... 11

00

2...

01

114 10

898

11

00 11

00

01

880 616

10

1

1...

11

... 11 00 10 00

5...

01 3330

Data bus enable Final result 0

4432

157

1126

end of pre-computation MMult (w-1) - Start MMult (w-1) - done MMult (w-1) - end MMult (w-2) - start MMult (w-2) - done MMult (w-2) - end MMult (w-3) - Start Multiplier (w-3) - done Multiplier (w-3) - end Final Multiplier - Start Final Multiplier - Done

0 ns

200 ns

100 ns

400 ns

300 ns

500 ns

600 ns

Entity:teste_exponenc Architecture:test_exp Date: Wed Oct 21 20:50:36 E. South America Daylight Time 2009 Row: 1 Page: 1

Fig. 6. Simulation result

– S17 : if MMULT(w−1) has finished then go to S18 ; – S18 : start MMULT0 ; if it has finished then go to S19 ; – S19 : go to S0 ;

3 Performance Results A parametrized VHDL [4] code was written and simulated on ModelSim XE III 6.4 [11]. Fig. 6 shows that the modular exponentiator takes about 40 clock cycles to yield one result. Signal ExpStart in high state flags an exponentiation start. Signal ExpDone is set when a run is through, and power T E is available in Final result. Table 1 presents a summary of the resources requirements of an Spartan-3E FPGA [11] for three different exponent sizes compared to the sequential hardware, which uses a single modular multiplier to perform all the multiplication required to yield the final exponentiation result Table 1. Estimated area for the proposed method vs. the sequential method [3] Size of E (bits) Slice Registers Slice LUT’s Used LUT-FF Response time

Parallel hardware 4 8 16 113 205 373 225 391 703 103 193 361 40 cycles

Sequential hardware 4 8 16 47 101 192 89 166 395 34 71 153 289 cycles

164

M. Santana Farias et al.

[3]. It is clear that the sequential implementation, as expected uses far less hardware resources than the proposed implementation. However, the gain in the computation time suggests that the latter is 7 times more efficient that the former. The use of the proposed hardware within security chips allows us to have a much higher throughput, which is a pre-requisite in most nowadays cryptosystems implementations.

4 Conclusion This paper describes a massively parallel implementation of modular exponentiation. It is a practical demonstration that parallelized modular exponentiation remarkably speeds up modular exponentiations, which is a critical stage on encryption/decryption relatedcomputation in most public-key cryptosystems. Performance depends on parameters w, d and the number of bits in E. We compared the performance of the proposed implementation to that of [3], which is a sequential implementation of the m-ary method. It is obvious that the sequential implementation, as one can expect, uses far less hardware resources than the proposed implementation. However, the gain in the computation time suggests that the latter is 7 times more efficient that the former. The use of the proposed hardware within security chips allows us to have a much higher throughput, which is a pre-requisite in most nowadays cryptosystems implementations. Wider windows (exponent share) extends pre-computation stage, requires larger PMEM, cuts down the number of MMULTs, reduces accesses to DBUS and demands less iterations from the highest order MMULT. In contrast, narrower windows yield the reduce the pre-computation step, requires less memory to store pre-computed powers but needs more modular multiplier. The best configuration should find a balance regarding hardware area and response time. Enhancements to decrease clock cycles per run through a better management of data bus accesses, normal exponentiation bypassing when the handled window is zero, multiple feeds of MMULTs after pre-computation stage using a many-port memory are some possible issues to be developed in an improved future implementation.

References 1. Wu, C.L., Lou, D.C., Lai, J.C., Chang, T.J.: Fast parallel exponentiation algorithm for RSA Public-Key Cryptosystem. Informatica 17, 445–462 (2006) 2. Knuth, D.E.: The art of computer programming: seminumerical algorithms, 2nd edn. Addison-Wesley, Reading (1981) 3. Mourelle, L.M., Nedjah, N.: Fast reconfigurale hardware for the m-ary modular exponentiation. In: Proc. Symposium on Digital System Design: Architectures, Methods and Tools, pp. 516–523. IEEE Computer Society, Los Alamitos (2004) 4. Navabi, Z.: VHDL – Analysis and modeling of digital systems, 2nd edn. McGraw Hill, New York (1998) 5. Nedjah, N., Mourelle, L.M.: Two hardware implementations for the Montgomery multiplication: sequential vs. parallel. In: Proc. of the 15th. SBCCI, pp. 3–8. IEEE Computer Society, Los Alamitos (2002)

A Massively Parallel Hardware for Modular Exponentiations

165

6. Nedjah, N., Mourelle, L.M.: Efficient Parallel Modular Exponentiation Algorithm. In: Yakhno, T. (ed.) ADVIS 2002. LNCS, vol. 2457, pp. 405–414. Springer, Heidelberg (2002) 7. Nedjah, N., Mourelle, L.M.: Efficient hardware for modular exponentiation using the slidingwindow method with variable-length partitioning. In: Proc. ICYCS, pp. 1980–1985 (2008) 8. Nedjah, N., Mourelle, L.M.: High-performance hardware of the sliding-window method for parallel computation of modular exponentiations. International Journal of Parallel Programming 37(6), 537–555 (2009) 9. Rivest, R.L., Shamir, A., Adleman, L.: A method for obtaining digital signature and publickey cryptosystems. Communication of ACM 21(2), 120–126 (1978) 10. Montgomery, P.L.: Modular multiplication without trial division. Mathematics of Computation 44, 519–521 (1985) 11. Xilinx, Inc. Foundation Series Software (2009), http://www.xilinx.com

Emulation of Object-Based Storage Devices by a Virtual Machine Yi-Chiun Fang, Chien-Kai Tseng, and Yarsun Hsu Department of Electrical Engineering, National Tsing Hua University, Hsinchu, 30013, Taiwan [email protected], [email protected], [email protected]

Abstract. This work presents a new emulation framework for objectbased storage devices (OSDs). The framework integrates an object-based storage emulator with a virtual machine. It resolves the limitations of host CPU resource consumption and network communication overhead found in other frameworks. The storage emulator in the virtual machine runs independently of the emulated CPU, and no network messages are required for communication. The OSD model is compared to a tradition file system on top of a block device. The benchmark results demonstrate an I/O performance improvement of 436% in the best case. The results suggest that the object-based storage architecture is an ideal choice for throughput machines, especially under heavy workloads.

1

Introduction

This work sets out to establish a new framework for software OSD emulations. The virtual storage emulation framework utilizes a virtual machine to implement the object-based storage system model. QEMU, chosen as the virtual machine, is an ideal choice to perform early emulations of new system architectures. The OSD model in this work avoids the common problems with conventional storage emulation frameworks. Early emulation before a real product is developed is a challenging task. The conventional frameworks for software storage emulation use either local or remote storage emulation [1]. Local storage emulation uses a device driver to redirect the requests to the user-space storage emulator. This framework provides simple implementation. One of its drawbacks is that the emulator consumes the resources of the host CPU. The other drawback is the context switch overhead between the driver and the emulator. Remote storage emulation runs the emulator on another computer and connects the computers by a network. This separates the emulator from the host CPU but introduces significant network overhead and a thick layer between the emulator and its underlying operating system. The overhead introduced by both frameworks decreases the accuracy of the system analysis. 

This work is supported by the National Science Council (NSC) of Taiwan under grant 96-2221-E-007-131-MY3.

C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 166–177, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Emulation of Object-Based Storage Devices by a Virtual Machine

167

Virtual storage emulation utilizes the open-source virtual machine QEMU. Its internal design makes it suitable for system architecture explorations. The emulated devices function independently of the emulated CPU, and network messages are not required for the communication. Operating systems can be run inside QEMU to provide a realistic workload and a more in-depth analysis of the system.

2

Related Work

Other researchers have analyzed and implemented the object-based interface. Ceph [2] uses an object-interface file system in its object storage cluster. The prototype object store Antara [3] uses its own network protocol to provide its functionalities. Lustre [4] uses an object store as its storage infrastructure for improved scalability. An implementation of the Parallel Virtual File System (PVFS) integrated with a software OSD emulator is presented in a project by Ohio Supercomputer Center [5]. All of the OSDs in the research mentioned above are communicated through network protocols which introduce significant communication overhead. They focus more on the performance of the overall storage architecture rather than the analysis of the OSD itself. Research has been conducted on software storage emulation. A timing-accurate storage emulator is presented by the Parallel Data Lab in Carnegie Mellon University [1]. The emulator allows experiments to be performed with not-yet-existing storage components in the context of real systems executing real applications. An instructional disk drive simulator with statistical disk models is presented in [6], retaining simplicity while providing timing statistics similar to that of real disk drives. The emulators are all based on the traditional block device and do not provide the object interface. They are implemented as standalone subsystems instead of emulated devices in a virtual machine. The result of these implementations is the consumption of the resources of their host CPUs or the network communication overhead. Other researchers also use QEMU for the emulation of system behaviors and communication protocols. The Virtual 802.11 Fuzzing Project [7] utilizes QEMU to provide a framework to assess wireless communication software inside a virtual environment. It focuses on the communication system instead of the storage system.

3

Implementation Details

Most of the implementations are done in the SCSI subsystem and I/O scheduler in QEMU. QEMU consists of several other subsystems besides the CPU emulator to perform full-system emulation. An emulated device emulates the behavior of a device and calls its attached generic device for access to the underlying host device. The generic device subsystem provides an abstraction of the host devices and is capable of accessing them.

168

Y.-C. Fang, C.-K. Tseng, and Y. Hsu

The overall scheme of the virtual storage emulation is depicted in Fig. 1. The Linux kernel is used as the operating system kernel on top of the emulated CPU. The sg (SCSI generic) driver and sym53c8xx 2 driver are used as the SCSI upperlevel and lower-level drivers inside the kernel. SCSI commands are passed from the sym53c8xx 2 driver to the modified LSI53C895A SCSI Controller emulated in QEMU. The modified LSI53C895A SCSI Controller attaches a SCSI OSD Disk in the emulated device layer of QEMU. The SCSI OSD Disk is implemented with SCSI-3 standards with minor modifications and extensions to the SPC-2 standard [8]. The block-osd generic device is used as the object interface for the emulated devices to access the underlying software storage emulator. The software storage emulator, EBOFS, is a part of the Ceph network file system [2]. EBOFS exports an object interface to its users, and manages block layouts and object caching internally. It has access to an underlying disk in the host computer.

Fig. 1. The overall scheme of the software OSD emulation

Emulation of Object-Based Storage Devices by a Virtual Machine

3.1

169

QEMU SCSI OSD Disk

The QEMU SCSI OSD Disk is implemented based on the emulated SCSI disk. The disk exports methods to its SCSI controller. The controller calls the registered methods when it needs to process the SCSI request. The QEMU SCSI OSD Disk is responsible for parsing the SCSI command and preparing the request for the underlying EBOFS. The actual object data transfer is done in EBOFS. The SCSI OSD Disk uses the SPC-2 commands with some extensions. The original SCSI-3 command set for SCSI disks only uses fixed-length CDBs. The OSD draft established by the T10 Technical Committee uses the variable-length CDB format for the commands. The format is also supported in the SPC-2 standard [8]. All OSD CDBs are 200-byte long even though they use the variablelength CDB format. The fields in the CDB are organized so that the same field is in the same location in all OSD CDBs. The adapted service action specific fields exclude the fields in the original CDB unrelated to this work, such as security parameters. The send command method is responsible for parsing the SCSI command and preparing the internal request. Its internal request structure includes the command tag, the parameters needed for the OSD request, the total data transfer length, an internal buffer for the data transfer, the request status, and a callback structure used by the generic device layer. This method sets up the request structure and returns the length of the data expected by the command. A positive number will be returned for data transfers from the device. A negative number will be returned for transfers to the device. Zero will be returned if the command does not require any data transfer. The read data and write data methods are responsible for issuing read and write requests to the underlying generic device for I/O scheduling. The SCSI OSD Disk uses an internal buffer of 128 KB for each request. Data transfers with transfer lengths larger than 128 KB will be split into multiple transfers. The request structure previously set up by the send command method is located using the command tag. The request is then sent to the underlying generic device block-osd. The request is put into an internal queue for scheduling and done asynchronously. The registered callback function is called by the generic device layer after the completion of the request. The callback function determines whether the request needs more transfers and notifies the SCSI controller of the finished transfer. The SCSI controller will issue more read or write transfers with the same command tag if they are required. 3.2

QEMU OSD Generic Device

The QEMU OSD Generic Device block-osd is implemented based on the blockraw generic device. It exports its methods to and accepts requests from the emulated devices. It also communicates with the underlying EBOFS when an OSD request is scheduled. The generic device interface in QEMU is extended with a new set of methods to adapt to the object interface. Only the block-osd generic device implements the new methods.

170

Y.-C. Fang, C.-K. Tseng, and Y. Hsu

Fig. 2. The overall sketch of the asynchronous request processing

The overall sketch of the asynchronous request processing is illustrated in Fig. 2. The block-osd generic device uses an internal queue for asynchronous reads and writes. It initializes the asynchronous I/O scheduler during system initialization. The scheduler initialization function sets up a pipe for complete tokens and registers the signal SIGUSR2 with its handler. The signal is fired when an OSD request is completed. The QEMU SCSI OSD Disk issues OSD requests by calling the exported methods from the generic device interface. The methods receive an OSD request set up by the emulated device, the address of the internal buffer in the emulated device, and a callback function as the parameters. An asynchronous request structure is initialized, set up and inserted into the asynchronous request queue. The asynchronous I/O scheduler is awakened, and the issued request returns with the address of the asynchronous request structure. The scheduler fetches an asynchronous request from the request queue, marks it as active, and calls the exported methods from EBOFS for the data transfer. Data will be read from or written to the buffer address given by the issuing emulated device, based on the given object ID, offset and transfer length. The scheduler records the return value from the EBOFS call in the asynchronous request structure and fires the signal SIGUSR2 after the transfer completion.

Emulation of Object-Based Storage Devices by a Virtual Machine

171

The signal handler, catching the signal, writes a token byte to the pipe to indicate a request completion. The asynchronous I/O scheduler keeps fetching and processing the requests until the request queue becomes empty. An I/O handling function is called every time the QEMU CPU emulator finishes one iteration. This function searches for the I/O handlers and calls the corresponding handling functions. The asynchronous I/O handling function reads the tokens from the pipe and searches for the completed asynchronous requests. The handling function calls the callback function registered in the completed asynchronous request and frees the request. The callback function is registered by the emulated SCSI OSD Disk. The emulated device is responsible for handling the callback and performing further actions.

4

Performance Evaluation

The performance of the proposed SCSI OSD model is compared to that of the traditional file interface. The file system ext2 is used for the benchmarks because of its popularity, speed and support for direct I/O. The computer system developed in this work lies within the QEMU virtual machine. The emulated SCSI OSD Disk can only be seen and accessed by the operating system running on top of the QEMU CPU emulator. The benchmark programs used for the evaluations are all run by the operating system inside QEMU. Debian [9] 5.0.1 with Linux kernel [10] official release 2.6.26.2 is used as the operating system running in the virtual machine. The Linux kernel is applied with the modifications described in Section 4.3. The host computer uses an IBM eServer with a 3.40 GHz Intel Pentium 4 CPU, 3 GB DDR133 RAM, and a 70 GB IBM-ESXS VPR073C3ETS10FN SCSI disk. The SCSI disk is attached to QEMU. Ubuntu [11] 7.04 with Linux kernel 2.6.20-16 is used as the host operating system. Bonnie++ [12] is chosen as the benchmark program. It is a benchmark suite that performs a number of simple tests on hard drive and file system performance. It is widely used for file system performance evaluation such as the project on Virtual Storage Allocation [13]. Bonnie++ is originally designed to use the file interface. SCSI commands need to be prepared and processed using the SG IO ioctl() interface in order to access the sg device. Code segments from the Linux sg3 utils package [14] are adapted to Bonnie++, enabling it to access the SCSI OSD Disk by the sg interface. The page cache subsystem in the Linux kernel caches the inodes, directory entries, and data recently accessed using the VFS interface. This subsystem cannot be completely eliminated from the underlying file system. The SCSI OSD Disk, implemented as a SCSI device, has no caching support in the operation system. This makes the evaluation unfair for the SCSI OSD Disk, since the effect of offloading the computation efforts of the storage component fails to present. Direct I/O and cache eviction are used in the ext2 benchmark to suppress the effects of the page cache subsystem. Direct I/O allows the VFS layer to minimize cache effects of the I/O to and from the file. Cache eviction is accomplished by performing the posix fadvise() system call with the POSIX FADV DONTNEED

172

Y.-C. Fang, C.-K. Tseng, and Y. Hsu

option and by writing a 3 to the file /proc/sys/vm/drop caches. The system call fsync() is required before calling posix fadvise() to guarantee the effect. The time spent in cache evictions is not taken into account in the evaluations. 4.1

Single-File Performance

Three types of evaluation are performed in the single-file evaluation. The Sequential Write test is run at the beginning of the evaluation. The test writes to a new file or object, sequentially and one block at a time, until the given total length is reached. The length of the data block is given as a program argument. The Rewrite test then reads a block from the file or object, changes a byte in the block, and overwrites the block back to the previous location. The rewriting is done sequentially until all the blocks are processed. The Sequential Read test starts after the Rewrite test. It reads from the file or object, sequentially and one block at a time, until the total length is reached. The file name and object ID are randomly generated. The bonnie++ benchmark suggests using the total data length at least twice the size of the main memory. This evaluation sets the total data length to 300 MB, since QEMU uses 128 MB of RAM from the host computer as its main memory. Four different configurations are evaluated. The SCSI OSD Disk is evaluated with and without direct I/O (the SG FLAG DIRECT IO flag). Direct I/O for the sg device only supports up to transfer length of 256 KB per system call. The ext2 file system is evaluated with direct I/O and cache eviction, and without any of the two. The result of the Sequential Write test is shown in Fig. 3a. The SCSI OSD Disk, without the file system management, outperforms the ext2 file system with cache suppression in all block sizes. The ext2 file system and the SCSI OSD Disk both benefit from the larger DMA size with larger data blocks. The increasing overhead of space management in ext2, however, results in a much slower throughput growth rate. The throughput difference between the SCSI OSD Disk and the cache-suppressed ext2 file system increases as the block size increases. The overhead of the file system allocator increases as the block size increases. The allocator spends more time allocating space for larger blocks, and space fragmentation is more likely to occur. The SCSI OSD Disk shows a performance improvement of 330% over the cache-suppressed ext2 file system with block size 1 MB. The direct I/O feature in the sg driver avoids extra kernel buffer copies, but it introduces significant per-command overhead. The feature is a performance win for SCSI commands only with large data payloads. Direct I/O benefits from avoiding buffer copies with block sizes larger than 32 KB. The performance is improved by 30% compared to disabling the feature, and by 436% compared to the cache-suppressed ext2 file system with block size 256 KB. The normal ext2 file system, with the benefits from the page cache subsystem, outperforms the rest of the configurations with block sizes smaller than 32 KB. The SCSI OSD Disk still outperforms the normal ext2 file system with block sizes larger than 32 KB, and the throughput difference increases as the block size increases. The throughput is relatively stable in terms of the block size. It

Emulation of Object-Based Storage Devices by a Virtual Machine

173

is 14% smaller, however, than its cache-suppressed counterpart when the block size reaches 1 MB. This demonstrates the drawback of memory caches with large data transfers. The overhead of preparing buffers and doing buffer copies degrades the overall performance. The result of the Rewrite and Sequential Read test are shown in Fig. 4 and Fig. 3b, respectively. The results are similar to the Sequential Write test, with the exception of the normal ext2 file system performing much better at throughput in the Sequential Read test. This is a result of the caching effect in the page cache subsystem and prefetching in the file system. The normal ext2 file system outperforms the rest of the configurations with block sizes smaller than 64 KB, and outperforms its cache-suppressed counterpart in all cases. The SCSI OSD Disk, compared with the cache-suppressed ext2 file system, shows a performance improvement of 13% in the Rewrite test and of 172% in the Read Sequential test with block size 1 MB.

(a) Write

(b) Read

Fig. 3. The result of Sequential Read/Write test

4.2

Multiple-File Performance

Three types of evaluation are performed in the multiple-file evaluation. Each type of evaluation includes a small-file and a large-file test. The small-file tests use 8 KB data blocks, and the large-file tests use 1 MB data blocks. The Create test creates the given number of files or objects, and writes a block after each creation. The Read test reads the block from each file or object after the creation. The Delete test deletes all of the created files or objects. The throughput in this evaluation is defined as operations per second. The number of manipulated files or objects range from 1 K to 256 K. Four different configurations are evaluated. The sequential and random configurations are used in both the SCSI OSD Disk and the ext2 file system. The sequentia configuration manipulates in an in-order fashion in terms of the data structure storing the file names or the object IDs. The random configuration manipulates in a random fashion. The SCSI OSD Disk disables the direct I/O feature in small-file tests,

174

Y.-C. Fang, C.-K. Tseng, and Y. Hsu

Fig. 4. The result of the Rewrite test

and enables it in large-file tests to accelerate the driver processing. The ext2 file system uses cache suppression to provide a fair comparison to the SCSI OSD Disk. The results of the Create test are shown in Fig. 5a and Fig. 5b. The effort of allocation increases as the number of manipulating files increases, resulting in the throughput degradation. The sequential and random configurations show similar results due to the early returns of write operations. Write operations return after writing data into the memory cache. Data is written back to the disk asynchronously. All of the sequential configurations slightly outperform their random counterparts due to the increased overhead of allocation in random configurations. The random allocation overhead of the SCSI OSD Disk decreases the performance by 10% with 256 K files. The SCSI OSD Disk underperforms the ext2 file system in all cases with small files, but outperforms the ext2 file system in all cases with large files. This is the result of the file system performing inmemory storage management and of the larger overhead the sg driver introduces. In-memory storage management shows its advantage with small I/O transfer size. The effort of allocation is relatively small, and the operation returns after the memory copy from the request to the page cache is completed. The generic design and blocking wait of the sg driver increases the driver overhead of the SCSI OSD Disk. The overhead increases the performance gap between the two models. The in-memory storage management becomes a burden for the system with large transfer size. The overhead of buffer management degrades the performance, resulting in the SCSI OSD Disk outperforming the ext2 file system in all cases. The results of the Read test are shown in Fig. 6a and Fig. 6b. The sequential small-file reads in the ext2 file system benefit from the in-memory storage management and outperform the rest of the small-file configurations in all cases. The sequential reads utilize the underlying disk by performing reads from neighboring sectors. The random small-file reads in the ext2 file system underperform the rest of the configurations in all cases. The random reads do not have the benefit of neighboring sector reads and spend more time resolving inodes. The SCSI OSD Disk, without the in-memory storage management, outperforms the ext2 file system in all large-file read cases. Its sequential read shows a performance

Emulation of Object-Based Storage Devices by a Virtual Machine

(a) small-file

175

(b) large-file

Fig. 5. The result of Create test

(a) small-file

(b) large-file

Fig. 6. The result of Read test

improvement of 232% over the ext2 file system with 4 K files. Its random reads underperform the sequential counterparts due to more time spent in locating data in the disk. The throughput of sequential reads in the SCSI OSD Disk increases as the number of objects increases in some cases. This results from the underlying I/O scheduler in EBOFS. The I/O scheduler uses the elevator algorithm to search for pending requests and combines neighboring sector reads into one large read, thereby reducing the read overhead. The allocation becomes more fragmented as the number of objects keeps increasing, and the throughput drops due to more effort in resolving data locations and more read overhead. The results of the Delete test are shown in Fig. 7a and Fig. 7b. Deletion in the SCSI OSD Disk requires the commands to be sent to the device. Deletion in file systems only requires marking the corresponding in-memory inode. Sequential deletes in the ext2 file system outperform the rest of the configurations in all cases. This results from less effort in resolving inodes and the benefit of inmemory inode processing. The effort of resolving inodes with random deletes

176

Y.-C. Fang, C.-K. Tseng, and Y. Hsu

increases dramatically as the number of files increases. Random small-file deletes with more than 16 K files underperform the random small-object deletes. The SCSI OSD Disk outperforms the ext2 file system by 169% with 64 K small-object random deletes.

(a) small-file

(b) large-file

Fig. 7. The result of Delete test

5

Conclusion

This work offers a new simulation framework for software OSD emulations. It provides a new model for the object-based storage system by utilizing a virtual machine. It addresses the drawbacks of other frameworks. The emulator does not consume the resources of the host CPU and no network overhead exists. This framework allows for more in-depth and accurate analyses to early system emulations at a reasonable speed. An object-based storage emulator is integrated into QEMU to implement the model which allows realistic workloads to be applied to the storage system. The results of the OSD model demonstrate the advantages of an object-based interface. The offloading of storage management into the device dramatically increases the throughput of the storage system. The throughput of the OSD model outperforms that of a local file system in most cases, especially under heavy workloads. The results suggest that the object-based storage architecture is an ideal choice for throughput machines.

References 1. Griffin, J.L., Schindler, J., Schlosser, S.W., Bucy, J.S., Ganger, G.R.: TimingAccurate Storage Emulation. In: Proceedings of the Conference on File and Storage Technologies, Janaury 2002, pp. 75–88 (2002) 2. Weil, S.A., Brandt, S.A., Miller, E.L., Long, D.D.E., Maltzahn, C.: Ceph: A Scalable, High-Performance Distributed File System. In: Proceedings of the 7th symposium on Operating systems design and implementation, pp. 307–320 (2006)

Emulation of Object-Based Storage Devices by a Virtual Machine

177

3. Azagury, A., et al.: Towards an Object Store. In: Proceedings of the 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, pp. 165–176 (2003) 4. Schwan, P.: Lustre: Building a File System for 1,000-node Clusters. In: Proceedings of the Linux Symposium, July 2003, pp. 380–386 (2003) 5. Devulapalli, A., Dalessandro, D., Wyckoff, P., Ali, N., Sadayappan, P.: Integrating Parallel File Systems with Object-Based Storage Devices. In: Proceedings of the 2007 ACM/IEEE conference on Supercomputing, vol. 00(27) (2007) 6. DeRosa, P., Shen, K., Stewart, C., Pearson, J.: Realism and Simplicity: Disk Simulation for Instructional OS Performance Evaluation. In: Proceedings of the 37th SIGCSE technical symposium on Computer Science Education, pp. 308–312 (2006) 7. Virtual 802.11 Fuzzing, http://www.iseclab.org/projects/vifuzz/ 8. SCSI Primary Commands - 2 (SPC-2), T10 Project 1236-D (July 2001) 9. Debian - The Universal Operating System, http://www.debian.org/ 10. The Linux Kernel Archives, http://www.kernel.org/ 11. Ubuntu, http://www.ubuntu.com/ 12. Bonnie++, http://www.coker.com.au/bonnie++/ 13. Kang, S., Reddy, A.L.: An Approach to Virtual Allocation in Storage Systems. ACM Transactions on Storage 2(4), 371–399 (2006) 14. The Linux sg3 utils package, http://sg.danny.cz/sg/sg3_utils.html

Balanced Multi-process Parallel Algorithm for Chemical Compound Inference with Given Path Frequencies∗ Jiayi Zhou1, Kun-Ming Yu2,∗∗, Chun Yuan Lin3, Kuei-Chung Shih1, and Chuan Yi Tang1 1

Department of Computer Science, National Tsing Hua University, Hsinchu 300, Taiwan [email protected], [email protected], [email protected] 2 Department of Computer Science and Information Engineering, Chung Hua University, Hsinchu 300, Taiwan [email protected] 3 Department of Computer Science and Information Engineering, Chang Gung University, Taoyuan 333, Taiwan [email protected]

Abstract. To enumerate chemical compounds with given path frequencies is a fundamental procedure in Chemo- and Bio-informatics. The applications include structure determination, novel molecular development, etc. The problem complexity has been proven as NP-hard. Many methods have been proposed to solve this problem. However, most of them are heuristic algorithms. Fujiwara et al. propose a sequential branch-and-bound algorithm. Although it reaches all solutions and avoids exhaustive searching, the computation time still increases significantly when the number of atoms increases. Hence, in this paper, a parallel algorithm is presented for solving this problem. The experimental results showed that computation time was reduced even when more processes were launched. Moreover, the speed-up ratio for most of the test cases was satisfactory and, furthermore, it showed potential for use in drug design. Keywords: Branch-and-bound algorithm, load-balancing, chemical compound inference, drug design.

1 Introduction The enumeration of chemical compounds that has the same characteristics is one of the fundamental issues in Chemo- and Bio-informatics. Its applications include structure determination using mass-spectrum [1-2], reconstructing molecular structure with given signatures [3-4], classification of compounds [5], etc. In an effort to improve on existing algorithms, many studies have proposed different ways of dealing with the ∗ This work is partially supported by Nation Science Council. (NSC98-2221-E-216-023 and NSC97-2221-E-216-020). ∗∗ Corresponding author. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 178–187, 2010. © Springer-Verlag Berlin Heidelberg 2010

Balanced Multi-process Parallel Algorithm for Chemical Compound Inference

179

enumeration of constraints for other purposes, such as the virtual exploration of chemical universe [6-7]. However, none can guarantee that all solutions can be found since they are based on heuristic algorithms and require additional operations to avoid generating isomorphic results. The Kernel Methods (KMs) approach maps the data in the input space into a high dimension feature space. The data are computed as points in the feature space where each coordinate represents one feature of the data. In applying KMs to chemical compounds, all compounds are mapped to feature vectors in the feature space. The definition of feature vector is widely based on frequencies of labeled paths [8] or frequencies of small fragments [5]. In this study, the pre-image problem was focused on by enumerating all chemical compounds with the same path frequency. The desired object was computed as a point in the feature space, and then the point was mapped back to the input space. This point was called the pre-image of the point. A given point in the feature space (frequency path) was then mapped back to the point in the input space (chemical compound). Let ψ be the mapping function, then the preimage is defined as follows: given a point y in the feature space, to find all x in the input space such that y = ψ(x ) . Therefore, the chemical compound inference (CCI) is defined as follows: given a target compound c and computed path frequency ψ(c ) . The objective is to infer all compounds c1 …cn such that ψ(ci ) = ψ(c ) for

i = 1, …, n . Since the solution for CCI can be applied to new chemical compounds with the same path frequencies, it may be useful in drug design. Akutsu and Fukagawa have proved that to enumerate chemical compounds with given constrains is an NP-hard problem [9]. The branch-and-bound algorithm is generally used to solve a wide variety of NP-hard problems [10], such as Traveling Salesman, Knapsack, Vertex Covering, avoiding exhaustive searches. For CCI problem, Fujiwara et al. have proposed an efficient branch-and-bound algorithm [11]. The branching and bounding strategies are based on the path frequency and the valence constrains of atoms, respectively, to avoid the generation of an invalid tree, thus reducing the search space. Although this algorithm outperforms the exhaustive search algorithms, the computation time increases significantly when the number of atoms in a compound grows. A balanced multi-process parallel branch-and-bound algorithm for CCI (BMPBBCCI) was designed in this study. In BMPBB-CCI, the searching space is dynamic divided into p subspaces with p processors. Since each subspace is independent, it can be processed by a processor and then applied to a sequential branch-and-bound algorithm. Hence, it minimizes the parallel communication cost between processors since only local communication is required. Moreover, two types of queuing, global queue (GQ) and local queue (LQ), are used for load-balancing in branching. A hashable and a serial data structure were used to store the path frequencies instead of the Trie structure because it can be transferred between processors as a plain string. The development trend in computers is one of multi-core processors. Adding more cores to a computer makes it faster, but it also leads to difficulties in designing programs. Although OpenMP [12] is a standard and easy to use multi-threading library that gives the performance of multi-core processors, it is not flexible enough for

180

J. Zhou et al.

delicate operations or for designing complicated parallel algorithm. Therefore, BMPBB-CCI was implemented using multi-process architecture (MPA). Different from multi-thread architecture, for long running programs, MPA obtains memory protection and access control benefits of operating systems. In addition, when a process crashes it does not affect the remaining running processes. Moreover, it also could be easily extended to distributed memory multi-processors system, e.g. a multinode cluster. The experimental results showed that BMPBB-CCI found the optimal solution for chemical compounds in a short time. In addition, it also had a satisfactory speed-up ratio.

2 Preliminaries Being different from a simple graph, a graph that allows multiple edges is called a multigraph. A multigraph without self-loop and cycle is called a multitree. Let Σ denote a label set with each label representing a symbol of an atom, e.g. Σ = {C ,O, H } . A Σ -labeled multitree can be denoted as T = (V , E ) with a vertex set V and edge set E . A valence function, val : Σ → + , is introduced to be the maximum number of bonds an atom can hold, e.g. val (C ) = 4 and val (O ) = 2 . For any vertex v in T , the valence of v is val (l (v )) , where l is the function returning label ∈ Σ of vertex v . A chemical compound can be treated as a (Σ, val ) -labeled multitree, where the number of edges represents the chemical bonds between two vertices (atoms). The path frequencies of the (Σ, val ) -labeled multitree are defined. Let P = (v 0 , …vs ) be the path in multitree T and l (P ) = (l(v0 ), …l(vs )) be the labeled sequence of the path. For a labeled sequence t , let occ(t,T ) stand for the number of paths of t in multitree T , where the multi-edge is treated as a single edge. Then Σk denotes the set of sequences with k labels, and Σ≤k = ∪kj =1 Σk . The path frequencies with level K are defined as fK (T ) = occ(t,T )t ∈ΣK +1 . Fig. 1 illustrates the path frequencies of un-rooted (Σ, val ) -labeled multitree (a chemical compound, C2O2H4) with level K =1. Where all paths are treated as "directed", thus occ(OH ,T ) = occ(HO,T ) =1 and so on. CCI is further defined in the following [11]. Given a finite label K ≥ 1 , a target with path frequencies g , and a valence function val : Σ →

+

. Find all (Σ, val ) -

labeled multitree T such that fK (T ) = g and deg(v ) = val (l (v )) for each vertex v in multitree T = (V , E ) . However, the main concern with CCI is to avoid the generation of isomorphic chemical compounds. Many methods [13-14] propose to solve this problem by choosing a unique vertex or a unique pair of adjacent vertices as a root. Fujiwara et al. [11] applied centroid-rooted [15] left-heavy properties (Theorem 1) to avoid the generation of isomorphic chemical compounds.

Balanced Multi-process Parallel Algorithm for Chemical Compound Inference

181

Fig. 1. Example of (Σ, val ) -labeled multitree T

Theorem 1. For any tree with n vertices, either there is a unique vertex v * such that each subtree obtained by removing v * to contain at most ⎢⎣⎢(n − 1) / 2⎥⎦⎥ vertices, or there is a unique edge e * such that both of the subtrees obtained by removing e * contain exactly n / 2 vertices. Vertex v * is identified as a unicentroid of the tree, or a bicentroid for e * . In order to introduce left-heavy properties, multitree T is indexed using a depth-first search (DFS) order. In general, the vertex sequence, v0 , v1, …vn , is labeled by DFS from the root vertex. The depth function d (v ) of vertex v is the number of edges in P(v ) , where the depth of the root vertex is 0. The depth label sequence of T is defined as DL(T ) = (d(vo ), l(vo ), …, d(vn−1 ), l(vn−1 )) . T(v ) denotes a subtree rooted at vertex v and all of its descendants. The left-heavy properties are defined as follows. T is left-heavy if i < j implies DL(T(vi )) ≥ DL(T(v j )) for any two siblings vi and v j .

For any two depth label sequences DL(T1 ) = (d1,0 , l1,0 , d1,1, l1,1 …, d1,n , l1,n ) and 1

1

DL(T2 ) = (d2,0 , l2,0 , d2,1, l2,1 …, d2,n , l2,n ) , DL(T1 ) > DL(T2 ) means that there is a 2

2

i ∈ [1, min(n1 − 1, n2 − 1)] such that d1, j = d2, j and l1,j = l2, j for j = 0, …i − 1 . In addition to either (1) d1,i > d2,i , or (2) d1,i = d2,i and l1,i > l2,i . Fujiwara et al. [11] propose a branch-and-bound algorithm to enumerate all treelike chemical compounds with given path frequencies. It starts from an empty multitree, then iteratively creates a multi-tree rooted in atom v , where l (v ) ∈ Σ . After that, new children multitree offspring can be obtained by inserting vertex v ' where l (v ') ∈ Σ at the right-most path. If T violates (1) centroid-rooted constraints, (2)

fK (T ) ≤ g , and (3) deg(v ) ≤ val(l (v )) for each v ∈ T then candidate T is bounded immediately.

3 BMPBB-CCI BMPBB-CCI was designed on shared memory multi-core computers. However, the built-in shared memory facilities of the operating system (OS) somewhat restricted

182

J. Zhou et al.

portability to another OS. In addition, this facility allows fewer types of data structures. Therefore, a socket-based manager process was implemented which held various types of data structures. In addition, the manager also could be extended to distributed memory computing architecture, such as cluster systems. Fig. 2 shows the framework of the BMPBB-CCI, where the manager module of the manager process implemented a socket-based communication object which supported the data structure, such as list, double-ended queue (dequeue) and dictionary. A Global Queue (GQ) was also implemented on the manager process as well as a Local Queue (LQ) on each computing process to balance the workload among processors and to reduce inter-process communication. There are three different processes, main process (MP), computing process (CP) and manager process (MgP). Since the valence of the H atom is 1 and always attached in the leaf node, the H atom can be removed during branching operations. The path frequency with the H atom is computed in step 2. After that, in steps 3 and 4, the MgP is created and started, and the required shared objects are also allocated in this step. In steps 5 and 6, the CP is created and launched at each computing core. Moreover, the obtained Id of created shared objects is passed to each CP. Finally, the MP joins all launched CPs until it is terminated and then MP writes the results to disk. Manager Process (MgP)

Manager Module

Main Process (MP) Sub-process Launcher Sub-process Synchronizer

Global Queue (GQ) Computing Process (CP1) Balanced Branchand-Bound Module Local Queue (LQ)

. . . Computing Process (CPn) Balanced Branchand-Bound Module Local Queue (LQ)

Fig. 2. Multi-process framework of BMPBB-CCI

The branch and bound operations are done in CP. Moreover, CP also uses the shared object to balance the workload. An overview of the algorithm of CP, show that it is an infinite loop. It loops the algorithm over until all solutions are found. From steps 2 to 6, the compound in LQ is chosen first. If the LQ is empty, then the compound in the GQ is chosen if that is not empty. Otherwise, the new atom in insertAtomQueue is selected as a new root of the candidate compound in step 5. The

Balanced Multi-process Parallel Algorithm for Chemical Compound Inference

183

bounding operations are applied in steps 7 to 13. H atoms are inserted back to comp to check the feature vector constrains and valence constrains in steps 8 and 9. The centroid-rooted and left-heavy properties are verified in step 10 to avoid generation of isomorphic chemical compounds. If the comp passes all verifications, it is inserted into resultQueue in step 11. The comp is dropped immediately if its path frequency fK (comp) greater than gnoH (step 13). Steps 14 to 17 are branching operations. In step 16, the potentially new pair of atoms are verified in the gnoH . If the pair is presented in gnoH then the new candidate compound newComps is generated. Moreover, borrow a single bond transformation as proposed by Fujiwara et al. [11]. The number of bonds of the attached new vertex is limited by the maximum number of bonds between pairs of atoms in the target compound. Therefore, the searching spaces can be significantly reduced to save on computation time. In order to balance the workload, when there are too few compounds in the GQ, the newComps are appended to GQ, otherwise the newComps are appended to LQ (step 17). Since the GQ is filled during branching operations, the CP can immediately acquire the candidate compounds from GQ without waiting for other CPs to transfer candidates. MgP creates and obtains a shared object via socket-based connections. The benefits of the shared object facilities are (1) they deal with various types of data structures, (2) they have built-in synchronization facility, and (3) they share objects from different computers. The algorithm for BMPBB-CCI is given below. Algorithm. BMPBB-CCI Input: Target compound and valence function Output: Chemical compound which confirms to path frequencies of given compound Environment: Multi-core architecture computers Main Process (MP) Step 1: Remove H atoms from given target compound and compute its path frequency gnoH . Step 2:

Insert the H atoms back to the given target compound and compute its path frequency g H .

Step 3: Step 4: Step 5: Step 6:

Create and start MgP and create a global queue on MgP. Allocate shared object form MgP, resultQueue and insertAtomQueue. Create CP on each computing core in processor. Start CP with following parameters: gnoH , g H ,

Step 7: Step 8:

resultQueue, insertAtomQueue. Wait all started CPs terminated. Write the results to disk.

Computing Process (CP) Step 1: while True: Step 2: if local queue is not empty: comp = pop last item of local queue Step 3: else: if global queue is not empty: comp = pop last item of global queue else:

184

J. Zhou et al.

Step 4: Step 5: Step 6: Step 7: Step 8: Step 9:

if resultQueue is not empty: break else if insertAtomQueue is not empty: comp = pop a atom from insertAtomQueue and create new compound else: break if num of atom of comp = target compound w/o H: Add H atom to comp continue if fK (comp) != g H :

Step 10: Step 11: Step 12:

Check the centroid-rooted and left-heavy properties If pass the checks, add the comp to resultQueue else: Check if fK (comp) <= gnoH

Step 13: Step 14:

for v in right most path of comp: if v is the last atom on right most path: insertAtom = Σ else: insertAtom = s for all s < l (next vertext of v ) for s in insertAtom: if (s, v ) in g H :

Step 15:

Insert s on v to generate new candidate newComp0 , …, newCompq such that bond

≤ the maximum number of bond of pair .. in target compound Step 16:

if len(GQ ) ≤ (numberOfProcessors )2 / 2 Insert newCompi to GQ for i = 0, …, q else: Insert . to LQ for i = 0, …, q .

Manager Process (MgP) Step 1: Create a socket and waiting for the request. Step 2: if incoming request is shared object creation: Create a shared object of request data type return shared object Id Step 3: if incoming request is shared object read: Read the corresponding shared object of given Id return the read value Step 4: if incoming request is shared object write: Acquire a lock object Write the given data to shared object of given Id Release a lock object

4 Experimental Results BMPBB-CCI was implemented in Python language 2.6 on IBM System x 3650 T consisting of 2 Intel Xeon 3.20GHz CPUs (8 computing cores, 4 cores per CPU). Two data sets were used to verify BMPBC-CCI. One was KEGG LIGAND database [16], the other was a set of 22 compounds for neuraminidase (NA) inhibitors of the influenza A virus from Zhang et al. [17]. The chemical compounds in KEGG LIGAND’s database were used to verify the performance of BMPBB-CCI, and path frequencies were computed for levels 1 and 2. Due to page limitation, selected compounds and their properties are shown in Table 1, where (1) C00064, C00073, and C00077 are the entries for L-Glutamine, LMethionine and L-Ornithine in the KEGG LIGAND database, (2) n1 is the number of atoms of an entry and n2 is the number of atoms which removed H atoms, (3) fs is the number of feasible solutions found. Fig. 3 (a)-(c) shows the computation time of

Balanced Multi-process Parallel Algorithm for Chemical Compound Inference

185

BMPBB-CCI with a different number of processes. It was found that a large number of processes (computing process, CP) reduced computation time. Moreover, smaller K values had fewer constraints and more feasible solutions were found (see Table 1). Therefore, there were more candidate compounds in branch-and-bound operations, leading to larger solution spaces to be traversed. The computation time was long when level K was small. Fig. 3 (d) illustrates the speed-up ratio of BMPBB-CCI from (a)-(c). The speed-up ratio increased when the number of processes increased. Moreover, the speed-up ratios were satisfactory even for 8 processes. This result showed that BMPBB-CCI was scalable. Table 1. Properties of selected compounds Entry Formula C00064 C5H10N2O3 C00073 C5H11NO2S C00077 C5H12N2O2

n1

n2

K

fs

20

10

21

9

21

9

1 2 1 2 1 2

274 3 339 2 236 3

Computation time of different quantities of processes (C00064)

Computation time of different quantities of processes (C00073) 90

250

80 70

150 K=1

100

Time (sec.)

Time (sec.)

200

K=2

60 50 40

K=1

30

K=2

20

50

10 0

0 1

2

4

1

8

Number of processes

2

(a)

8

(b)

Computation time of different quantities of processes (C00077)

Speed-up ratio of dfiirent entries

70

4.5

60

4

30

K=1

20

K=2

Speed-up ratio

40

1 proc. 2 proc. 4 proc. 8 proc.

3.5

50

Time (sec.)

4

Number of processes

3 2.5 2 1.5 1

10

0.5

0 1

2

4

Number of processes

8

0 C00064,K=1 C00064,K=2 C00073,K=1 C00073,K=2 C00077,K=1 C00077,K=2 Entries

(c)

(d)

Fig. 3. Computation time and speed-up ratio

BMPBB-CCI was used to infer novel chemical compounds for the second data set to show that it had potential for drug design. A pharmacophore model consists of a 3D arrangement of a collection of features necessary for the biological activity of ligands (compounds). At first, the pharmacophore model (by Accelrys DiscoveryStudio) fwas

186

J. Zhou et al.

constructed for NA of an influenza A virus from Zhang et al. [17]. Then the model was tested with Tamiflu and Zanamivir. The results (Table 2) showed that it was reliable since the predicted IC50 and actual IC50 were on the same quantity level. Finally, the model was used to test new chemical compounds inferred by BMPBB-CCI. Due to page limitation, only the compound Cpd was used as an example (Table 2). The new compound Cpd-Reb had a better predicted IC50 and Fit value than Cpd-Ori. DiscoveryStudio CDOCKER’s docking program was also used to compute the interactions between compound and protein. The best CDOCKER interaction energy for Cpd-Ori was 42.565. The best CDOCKER interaction energy for Cpd-Reb was 45.324. The docked poses are given in Fig. 4. These results showed that the novel compound CpdReb may be a candidate for NA inhibiting of the influenza A virus. Table 2. Results for test compounds in the pharmacophore model Fit Mapped feature Actual IC50 Estimated IC50 value HD1* HD2* HY* NI* (nM) (nM) Tamiflu 1 8.502 10.95 + + + Zanamivir 1.3 4.848 11.194 + + + # Cpd-Ori 6300 5639.67 8.129 + + # Cpd-Reb NA 6.577 11.062 + + + + * HD1: hydrogen-bond donor 1; HD2: hydrogen-bond donor 2; HY: hydrophobic group; NI: negative ionizable group; PI: positive ionizable group. # Cpd-Ori: original compound Cpd; Cpd-Reb: novel compound inferred by BMPBB-CCI. Compound

(a) Cpd-Ori

PI* + + +

(b) Cpd-Reb

Fig. 4. The Cpd-Ori and Cpd-Reb docked pose into NA protein (PDB code: 2hu4)

5 Conclusions An algorithm (BMPBB-CCI) was designed and verified on multi-core computers. From the experimental results, it was observed that BMPBB-CCI reduced computation time with more processors in the case of the KEGG LIGAND database. It also achieved a satisfactory speed-up ratio for most of the test cases. Moreover, it showed potential for drug design.

Balanced Multi-process Parallel Algorithm for Chemical Compound Inference

187

Acknowledgement We are grateful to the National Center for High-performance Computing for computer time and facilities.

References 1. Buchanan, B., Feigenbaum, E.: DENDRAL and Meta-DENDRAL, Their Applications Dimension. Artif. Intell. 11, 5–24 (1978) 2. Funatsu, K., Sasaki, S.: Recent advances in the automated structure elucidation system, chemics. Utilization of two-dimensional nmr spectral information and development of peripheral functions for examination of candidates. J. Chem. Inf. Comput. Sci. 36(2), 190– 204 (1996) 3. Faulon, J., Churchwell, C., Visco Jr., D.: The signature molecular descriptor. 2. Enumerating molecules from their extended valence sequences. J. Chem. Inf. Comput. Sci. 43(3), 721–734 (2003) 4. Hall, L., Dailey, R., Kier, L.: Design of molecules from quantitative structure-activity relationship models. 3. Role of higher order path counts: path 3. J. Chem. Inf. Comput. Sci. 33(4), 598–603 (1993) 5. Deshpande, M., Kuramochi, M., Wale, N., Karypis, G.: Frequent substructure-based approaches for classifying chemical compounds. IEEE Trans. Knowl. Data Eng. 17(8), 1036–1050 (2005) 6. Fink, T., Reymond, J.: Virtual Exploration of the Chemical Universe up to 11 Atoms of C, N, O, F: Assembly of 26.4 Million Structures (110.9 Million Stereoisomers) and Analysis for New Ring Systems, Stereochemistry. J. Chem Inf. Model. 47(2), 342–353 (2007) 7. Mauser, H., Stahl, M.: Chemical fragment spaces for de novo design. J. Chem. Inf. Model. 47(2), 318–324 (2007) 8. Kashima, H., Tsuda, K., Inokuchi, A.: Marginalized kernels between labeled graphs. In: ICML, pp. 321–328 (2003) 9. Akutsu, T., Fukagawa, D.: Inferring a Graph from Path Frequency. In: Apostolico, A., Crochemore, M., Park, K. (eds.) LOPSTR 2004. LNCS, vol. 3573, pp. 371–382. Springer, Heidelberg (2005) 10. Yu, C., Wah, B.: Efficient branch-and-bound algorithms on a two-level memory system. IEEE Trans. Softw. Eng. 14(9), 1342–1356 (1988) 11. Fujiwara, H., Wang, J., Zhao, L., Nagamochi, H., Akutsu, T.: Enumerating Treelike Chemical Graphs with Given Path Frequency. J. Chem. Inf. Model. 48(7), 1345–1357 (2008) 12. OpenMP, http://openmp.org/ 13. Nakano, S., Uno, T.: Generating colored trees. In: Kratsch, D. (ed.) WG 2005. LNCS, vol. 3787, pp. 249–260. Springer, Heidelberg (2005) 14. Wright, R., Richmond, B., Odlyzko, A., McKay, B.: Constant time generation of free trees. SIAM J. Comput. 15, 540 (1986) 15. Jordan, C.: Sur les assemblages de lignes. J. Reine Angew. Math. 70(185), 81 (1869) 16. KEGG Ligand database, http://www.genome.jp/kegg/ligand.html 17. Zhang, J., Yu, K., Zhu, W., Jiang, H.: Neuraminidase pharmacophore model derived from diverse classes of inhibitors. Bioorg. Med. Chem. Lett. 16, 3009–3014 (2006)

Harnessing Clusters for High Performance Computation of Gene Expression Microarray Comparative Analysis Philip Church1, Adam Wong2, Andrzej Goscinski1, and Christophe Lefèvre3,4 1

School of Information Technology, Deakin University, Geelong, Australia {pcc,ang}@deakin.edu.au 2 Victorian Partnership for Advanced Computing (VPAC) [email protected] 3 Institute for Technology Research and Innovation (ITRI), BioDeakin, Deakin University 4 Victorian Bioinformatics Consortium, Monash University [email protected]

Abstract. Gene Expression Comparative Analysis allows bio-informatics researchers to discover the functional regulation of genes. This is achieved through comparisons between data-sets representing the quantities of substances in a biological system. Unnatural variations can be introduced during the data collection and digitization process so normalization algorithms must be applied to data before any accurate comparison can be made. There exist many different normalization methods each of which gives a different result. Comparing differently normalized datasets can allow for discovery of crucial regulated genes that may be otherwise hidden due to errors in a single normalization study. In this paper we introduce a web-based software package called EXPPAC which makes use of a high performance computing platform of computer clusters to run multiple normalization methods in parallel. By generating multiple normalized datasets concurrently, we allow researchers the ability to improve the accuracy of their research with almost no extra time-cost. Keywords: Gene Expression, Normalization, Clusters, Statistical Algorithms.

1 Introduction Gene Expression Comparative Analysis is a field of bioinformatics that allows researchers to discover the function and regulation of genes. Its basic principle is to group datasets with similar genes and isolate the components that are different. By comparing the expressed genes of similar components with different features, we can then determine the functionality of genes. For example, the mother wallaby produces two different types of milk over an eight month period. The first milk has low levels of fat and protein and elevated levels of carbohydrate. In the second milk, concentration of carbohydrate declines, whereas the concentrations of both protein and fat increase. Research has shown that the development of pouch young when fed the second type of milk develop quicker and are stronger than those fed the first type of milk [1]. By comparing the expressed genes of the two milks, we can isolate the components and quantities that are responsible for this improved growth. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 188–197, 2010. © Springer-Verlag Berlin Heidelberg 2010

Harnessing Clusters for High Performance Computation

189

The microarray approach [2] is the most common method of collecting gene expression data currently being used in bioinformatics. Data from microarray experiments must be stored digitally using one of the many gene expression file formats before being analysed using statistical algorithms. Normalization is a key part of gene expression microarray analysis since unnatural variations can be introduced during the data collection and digitization process. Thus, this data must be corrected and standardized with other arrays before being compared and analysed. There are many normalization algorithms which balance accuracy of ratios and correctness of the returned probes. Comparison of differently normalized datasets can uncover expressed genes once overlooked however the time required to generate normalized datasets mean these additional comparisons are not often done. In this paper we study the use of high performance computing platforms, such as clusters, in running multiple gene expression microarray normalization algorithms concurrently. Presented is a web based package called EXP-PAC that makes use of high performance computing to ready multiple gene expression datasets with common normalization methods in a reduced time. Using this package it is hoped gene expression researchers will be able to improve the accuracy of their results with minimal time and effort. The rest of this paper is organized as follows. Section 2 gives a background of gene expression comparative analysis in bioinformatics. Section 3 explains the role and mechanisms used in normalization of gene expression data. Section 4 describes the implementation of our high performance normalization method through EXP-PAC [3], a gene expression comparative analysis framework. The experiments and performance evaluation of the high performance normalization method is presented in Section 5. Finally, Section 6 concludes by noting the importance of high performance computing in comparative microarray gene expression analysis.

2 Gene Expression Comparative Analysis Gene expression comparative analysis is usually performed in the following three steps as illustrated in Fig. 1. 1. Data is collected in a wet-lab using a gene expression platform (cDNA, Microarrays, etc.). 2. Collected data is converted to a digital format and any un-natural variation is removed. 3. Data analysis is used to group together similar datasets to locate components responsible for biological functions.

Fig. 1. The three stages of gene expression comparative analysis

190

P. Church et al.

Microarrays are the most common gene expression collection platform. They are nylon boards that contain thousands of probes that react when exposed to a biological sample. A typical microarray experiment consists of many arrays. To be recognized by computers, the surface area of each array is scanned and converted to an image file. Image files are then converted to numerical datasets, the colour intensity of each probe is often represented by a ratio. Statistical algorithms can be applied in order to determine the numerical probability of similarity between datasets.

3 Normalization of Gene Expression Data Microarray data collection is not perfect; un-natural variations can arise due to differences in sample preparation, production of arrays and the processing of the arrays. Normalization algorithms are applied to collected data in order to standardize intensities and remove un-natural variations. Common normalization approaches include background subtraction, construction of artificial reference arrays and quantile normalization. • Background Subtraction [4] removes variation introduced from background noise during the scanning process. This method involves subtracting the background from the values of the probes. This method is reliant on having space between probes as background noise is usually not linear. • An artificial reference array [5] is a model based on the median of gene expression levels over all other arrays. Using this constructed model it is possible to normalize values across microarrays. • Quantile normalization [5] is another popular method for normalization across arrays based on transforming each array-specific distribution of intensities so they have the same values at specified quantiles, for example centring arrays on their mediums. A number of algorithms combine these methods to provide normalization. These algorithms are balanced between accuracy of ratios and correctness of the returned probes [6]. It is reasonable to expect that each method returns different results. Commonly used algorithms include: • RMA measure [7] normalizes data using an artificial reference array created through quantile normalization of probes. • Gcrma [8] performs a background adjustment based on sequence information of each probe. • Mas5 [9] makes use of the mismatched probes (MM) designed to remove similar targeted genes increasing the accuracy of microarray data. Normalization uses the robust average of the (logged) probe-MM values. • Plier [10] is similar to the Mas5 method however it assumes variations between perfect match (PM) probes are more accurate then the MM probes. • Qspline [11] fits splines to the quantiles of a target array, normalization is performed using these splines.

Harnessing Clusters for High Performance Computation

191

• Invariantset [12] normalize arrays by selecting invariant sets of genes (or probes) and then using them to fit a non-linear relationship between “treatment" and “baseline" arrays. Normalization using different algorithms can be applied to a dataset in multiple ways in a single experiment. Normalization algorithms such as RMA [7], MAS5 [9], gcrma [8] and plier [10] differ in their approach; some focus on delivering accurate data (value of ratios) and others correctness of data (filtering expressed genes) [6]. Performing normalization using different methods can uncover different sets of expressed genes, increasing the accuracy of results from comparative analysis. Most researchers do not take advantage of this method which allows for improved accuracy of their results due to the time necessary to run this extra analysis. Using high performance computing, multiple normalization methods can be run in parallel allowing many normalized datasets to be collected without the disadvantage of an increased processing time. This method of high performance normalization has been applied to a software package called EXP-PAC.

4 EXP-PAC EXP-PAC is a web based system developed for gene expression comparative analysis. The EXP-PAC system combines the features of two pre-existing software packages, MammoSapians [13] and EST-PAC [14] (see Fig. 2.). MammoSapians is a tool for gene expression analysis providing a unique method of post-analysis based on SQL queries. EST-PAC is a sequence analysis framework which provides data storage and management, security and sequence analysis through embedded tools. EXP-PAC provides users with the ability to upload a number of gene expression file formats (raw microarray data , SOFT [15], MAGE-tab [16], etc.). Uploaded raw microarray data files (also called CEL files) are normalized using the R statistical scripting language. EXP-PAC supports normalization through a distributed platform which uses the Sun Grid Engine [17] in order to speed up microarray data analysis. Normalized data can be linked to results from statistical analysis which is uploaded in a tab-delimited format. Microarray data uploaded to the system can then be queried using an interface dynamically generated from the uploaded microarray and statistical data. In addition, through creation of a sequence to probe ID map, it is possible for a user to perform comparisons on multiple species. Unique to EXP-PAC is cross-species analysis and high performance normalization. Cross Species analysis combines sequence and gene expression data to compare biological systems. High performance normalization, allows users to apply multiple normalization algorithms to different uploaded datasets. Different normalization tasks are constructed by the EXP-PAC system and each of them encapsulates the computation of gene expression data of a particular dataset with a particular normalization algorithm. Thus, multiple of such compute-tasks are submitted to the batch job scheduler of a high performance computing system which would then be scheduled to run concurrently. Multiple normalized datasets can be compared using the crossspecies analysis feature, highlighting differences between the different normalization methods.

192

P. Church et al.

Fig. 2. The structure of the EXP-PAC system

Fig. 3. Architecture showing EXP-PAC interacting with a high performance computing system

Harnessing Clusters for High Performance Computation

193

Accuracy of microarray comparative analysis can benefit from having access to multiple normalized datasets. The normalization methods described in Section 3 require each array be first loaded into memory in order to generate a standardized dataset. Implementing truly parallel normalization methods requires that new normalization techniques be developed. However, common methods of normalization can take advantage of embarrassing parallel methods to improve the speed of data generation. Each normalization job can be run on an individual compute node of a cluster with no modification to the base algorithms and implementation. Our system (see Fig. 3) consists of a bio-server which hosts the EXP-PAC platform; through this interface users can submit normalization jobs and the raw microarray data to an unoccupied cluster node. Datasets are made available as they are generated.

5 Experiment and Results Normalization of microarray data was performed on three CEL files which were downloaded from the Array Express [18] database. The CEL files were taken from experiments E-GEOD-8191 [19], E-GEOD-14764 [20] and E-MEXP-1594 [21]. Starting at 40 files, these experiments double in size and file quantity until a maximum of 160 files is reached. Choosing this range of CEL file sizes allows performance of normalization algorithms to be measured as file size increases. Implementation of the normalization algorithm were provided through the R statistical language [22] and bio-conductor [23] libraries. R scripts were written for the six most commonly used normalization methods described in Section 3; RMA, mas5, gcrma, justPlier, qspline and invariantset. Performance testing of EXP-PAC’s normalization algorithms made use of a high performance computer cluster at Deakin University. This cluster contains 20 compute-nodes connected using an InfiniBand network. Each compute-node has two Intel quad-core processors (a total number of 160 CPUs) running at 1.6 Ghz and 8 GB of ram. The CentOS Linux operating system is run on each node of the cluster. The Sun Grid Engine [17] middleware is used for resource management of the cluster (see Fig. 3). Computations to be performed on the cluster were submitted through the EXPPAC system to the Sun Grid Engine scheduler. Jobs submitted to the scheduler are queued and distributed across nodes of the cluster with processing power available. Normalization tasks carried out on single node and multi node (i.e., a computer cluster) platforms were implemented and compared as illustrated in Fig. 4. For each file size, single node normalization involved a sequential execution of six R scripts each representing a commonly used normalization method. A normalized gene expression data file is generated for each R script run. Conversely, multi node processing speeds up this normalization process by submitting the same six R scripts to a computer cluster which are then executed concurrently. The performances of six normalization methods (RMA, mas5, gcrma, justPlier, qspline and invariantset) were examined on each downloaded gene expression

194

P. Church et al.

Fig. 4. Implemented single and multi node normalization methods

experiment through both single node and multi mode platforms. Each method was run three times to ensure accurate performance results. Single node normalization showed major differences between the computation times of different normalization methods (see Fig. 5.). RMA measure was the quickest of the six tested methods; results followed a linear increase in time to file size. The qspline normalization method was the second fastest algorithm, completing normalization of forty files in two minutes and twenty three seconds. The gcrma method has a stable computational time when processing small files, while the one hundred and sixty file dataset required a great amount of processing time. Invarientset normalization took longer then gcrma, however it shows a linear increase in computation time. JustPlier is quicker when processing small amounts of data but shows a logarithmic increase in process time as the amount of input files increase. The Mas5 method is the most time consuming of the tested methods. In multi node processing, all of the 16 normalization tasks (3 file sizes, 6 normalization algorithms) were run concurrently (one task per CPU). The amount of normalization tasks is less than the twenty nodes available on our cluster. Therefore all normalization tasks were completed within the time taken for the largest file set and the slowest normalization method which is the 160 Files set and the justPlier normalization method. Figure 6 shows that a dramatic improvement in the computation time for normalization was achieved when it was moved from single node execution (a total time of 226 minutes) to multi node execution (a total time of 46 minutes). Comparison of these execution times shows that a speed-up of nearly five has been achieved.

Harnessing Clusters for High Performance Computation

195

Fig. 5. Average performance of pre-processing methods on a single node computer

Fig. 6. Comparison of single node and multi node computational time for normalization

6 Conclusion In this paper, we have presented the study of how high performance computing platforms such as clusters can be employed for running multiple gene expression microarray analysis normalization methods in parallel. We have presented a system called EXP-PAC that provides a framework for gene expression research. Compared to available gene expression software packages, EXPPAC is unique in that it provides a method of cross-species gene expression analysis and a method to apply high performance computing to common normalization methods. We have performed an experiment to parallelize the normalization of gene expression data using three different sizes of data file sets and six different normalization methods as a case study. The speed-up of the normalization process from single node execution to multi node execution is nearly five. Our result has demonstrated

196

P. Church et al.

that significant computational improvement in normalization of gene expression data can be achieved especially when large files and multiple normalization methods are applied. By generating these six normalized datasets in the time taken to produce one, researchers can compare the key differences in expressed genes efficiently.

References 1. Trott, J.F., Simpson, K.J., Moyle, R.L.C., Hearn, C.M., Shaw, G., Nicholas, K.R., Renfree, M.B.: Maternal Regulation of Milk Composition, Milk Production, and Pouch Young Development During Lactation in the Tammar Wallaby (Macropus eugenii). Biol. Reprod. 68, 929–936 (2003) 2. Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., Aach, J., Ansorge, W., Ball, C.A., Causton, H.C., Gaasterland, T., Glenisson, P., Holstege, F.C., Kim, I.F., Markowitz, V., Matese, J.C., Parkinson, H., Robinson, A., Sarkans, U., Schulze-Kremer, S., Stewart, J., Taylor, R., Vilo, J., Vingron, M.: Minimum information about a microarray experiment (MIAME)-toward standards for microarray data. Nat. Genet. 29, 365–371 (2001) 3. Church, P., Goscinski, A., Wong, A., Lefevre, C.: Exp-Pac: A Web Based Package For The Comparitive Analysis Of Microarray Data. Bioinformatics Australia, Melbourne (2009) 4. Yang, Y.H., Buckley, M.J., Speed, T.P.: Analysis of cDNA microarray images. Briefings in Bioinformatics 2, 341–349 (2001) 5. Dudoit, S., Yang, Y.H., Callow, M.J., Speed, T.P.: Statistical methods for identifying differentially expressed genes in replicated cDNA microarray experiments. Statistica Sinica 12, 111–140 (2002) 6. Irizarry, R.A., Wu, Z., Jaffee, H.A.: Comparison of Affymetrix GeneChip expression measures. Bioinformatics 22, 789–794 (2006) 7. Irizarry, R.A., Hobbs, B., Collin, F., Beazer-Barclay, Y.D., Antonellis, K.J., Scherf, U., Speed, T.P.: Exploration, normalization, and summaries of high density oligonucleotide array probe level data. Biostat. 4, 249–264 (2003) 8. Wu, Z., Irizarry, R., Gentleman, R., Martinez-Murillo, F., Spencer, F.: A Model-Based Background Adjustment for Oligonucleotide Expression Arrays. J. Am. Stat. Assoc. 99, 909 9. Hubbell, E., Liu, W.-M., Mei, R.: Robust estimators for expression analysis. Bioinformatics 18, 1585–1592 (2002) 10. Affymetrix, I.: Technical note: guide to probe logarithmic intensity error (PLIER) estimation (2005) 11. Workman, C., Jensen, L., Jarmer, H., Berka, R., Gautier, L., Nielser, H., Saxild, H.-H., Nielsen, C., Brunak, S., Knudsen, S.: A new non-linear normalization method for reducing variability in DNA microarray experiments. Genome Biol. 3 (2002) research0048.0041 research0048.0016 12. Li, C., Wong, W.H.: Model-based analysis of oligonucleotide arrays: Expression index computation and outlier detection. Proc. Natl. Acad. Sci. U.S.A. 98, 31–36 (2001) 13. Lefèvre, C., Nicholas, K.R., Kumar, A., Strahm, Y., Powell, D., Seemann, T., Daly, K.A., Brennan, A., Menzies, K., Sharp, J., Digby, M.: MammoSapiens: eResearch of the lactation program. Building online facilities for collaborative molecular and evolutionary analysis of lactation and other biological systems from gene sequences and gene expression data: eResearch Australasia, Sebel and Citigate Hotels, Albert Park in Melbourne, Australia (2008)

Harnessing Clusters for High Performance Computation

197

14. Strahm, Y., Powell, D., Lefevre, C.: EST-PAC a web package for EST annotation and protein sequence prediction. Source Code Biol. Med. 1, 2 (2006) 15. Barrett, T., Suzek, T.O., Troup, D.B., Wilhite, S.E., Ngau, W.-C., Ledoux, P., Rudnev, D., Lash, A.E., Fujibuchi, W., Edgar, R.: NCBI GEO: mining millions of expression profiles– database and tools. Nucl. Acids Res. 33, D562–D566 (2005) 16. Rayner, T., Rocca-Serra, P., Spellman, P., Causton, H., Farne, A., Holloway, E., Irizarry, R., Liu, J., Maier, D., Miller, M., Petersen, K., Quackenbush, J., Sherlock, G., Stoeckert, C., White, J., Whetzel, P., Wymore, F., Parkinson, H., Sarkans, U., Ball, C., Brazma, A.: A simple spreadsheet-based, MIAME-supportive format for microarray data: MAGE-TAB. BMC Bioinformatics 7, 489 (2006) 17. Gentzsch, W.: Sun Grid Engine: Towards Creating a Compute Power Grid. In: Proceedings of the 1st International Symposium on Cluster Computing and the Grid. IEEE Computer Society, Los Alamitos (2001) 18. Brazma, A., Parkinson, H., Sarkans, U., Shojatalab, M., Vilo, J., Abeygunawardena, N., Holloway, E., Kapushesky, M., Kemmeren, P., Lara, G.G., Oezcimen, A., Rocca-Serra, P., Sansone, S.-A.: ArrayExpress–a public repository for microarray gene expression data at the EBI. Nucl. Acids Res. 31, 68–71 (2003) 19. Anderson, S., Rudolph, M., McManaman, J., Neville, M.: Key stages in mammary gland development. Secretory activation in the mammary gland: it’s not just about milk protein synthesis! Breast Cancer Res. 9, 204 (2007) 20. Denkert, C., Budczies, J., Darb-Esfahani, S., Györffy, B., Sehouli, J., Könsgen, D., Zeillinger, R., Weichert, W., Noske, A., Buckendahl, A.-C., Müller, B.M., Dietel, M., Lage, H.: A prognostic gene expression index in ovarian cancer - validation across different independent data sets. The Journal of Pathology 218, 273–280 (2009) 21. Ayroles, J.F., Carbone, M.A., Stone, E.A., Jordan, K.W., Lyman, R.F., Magwire, M.M., Rollmann, S.M., Duncan, L.H., Lawrence, F., Anholt, R.R.H., Mackay, T.F.C.: Systems genetics of complex traits in Drosophila melanogaster. Nat. Genet. 41, 299–307 (2009) 22. Ihaka, R., Gentleman, R.: A Language for Data Analysis and Graphics. Journal of Computational and Graphical Statistics 5, 299–314 (1996) 23. Gentleman, R., Carey, V., Bates, D., Bolstad, B., Dettling, M., Dudoit, S., Ellis, B., Gautier, L., Ge, Y., Gentry, J., Hornik, K., Hothorn, T., Huber, W., Iacus, S., Irizarry, R., Leisch, F., Li, C., Maechler, M., Rossini, A., Sawitzki, G., Smith, C., Smyth, G., Tierney, L., Yang, J., Zhang, J.: Bioconductor: open software development for computational biology and bioinformatics. Genome Biol. 5, R80 (2004)

Semantic Access Control for Corporate Mobile Devices Tuncay Ercan1 and Mehmet Yıldız2 1

Department of Computer Engineering, Yasar University, Universite cad. No:35-37, 35100 Bornova, Izmir, Turkey [email protected] 2 Global Technology Services, IBM Australia, Melbourne, Australia [email protected]

Abstract. Many of the mobile business applications are executed in different domains as business-to-consumer (B2C) and business-to-business (B2B) operations. Computing environments of mobile wireless devices owned by individuals or organizations have become totally distributed between peers and partners. Designing applicable access control mechanisms in this environment is difficult from the point of traditional security measures. Semantic web technologies offer appropriate access opportunities to the corporate resources by using related context in user devices and servers under the trust philosophy. Semantic web technology combines user requests and service descriptions for an efficient matchmaking. This paper examines various access control mechanisms and semantically analyzes them. The purpose of this paper is to present a more secure access control mechanism for mobile corporate devices. This model can be used as an additional security framework that enforces access control mechanisms in the organizations.

1 Introduction Current web technologies have brought a high benefit to pervasive users and their mobile devices. Together with the development of Information Technology (IT) products, employees working in different industrial sectors have been using internet. Both people and the organizations tend to use mobile devices such as laptops, personal digital assistants (PDAs), and smart mobile phones and have a strong demand for their networking and computing with each other. Their widespread use and characteristics make them ideal devices for different applications and allow employees to keep current with organizational activities while working away from the office. Typical users are sales, marketing people, and technicians. They place orders or check the availability of organizational resources like products, different data types, and maintenance information. They want easy and secure access to corporate processes. A combination of wired and wireless hardware technologies creates new advantages for business applications on the Internet. However, the increasing number of mobile devices causes confusion and difficulty. B2B and B2C companies offer their business services to customers by delegating necessary access rights for their corporate resources through the web services. While initial user authentications check the C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 198–207, 2010. © Springer-Verlag Berlin Heidelberg 2010

Semantic Access Control for Corporate Mobile Devices

199

validity of the users’ credentials, service providers check the identities and users’ privileges in the targeted company domain. Access decision is given by the company servers and based on the assessment of security policies associated with mobile device functions and user requirements. Web 2.0 is the current World Wide Web (WWW) technology and enhances users’ functionality, information sharing and collaboration with each other. Since this type of environment is used by mobile users, both their needs and services should meet in a common point where overall corporate management could be smoothly handled. Using semantic web technologies is the only approach for an efficient and effective access control mechanism. Mobile ontology describes a mobile domain for mobile networks, mobile applications, mobile users, and mobile devices [1]. Mobile context changes are related with different mobile devices, physical locations, applications executed by the users, and different service requests from the service providers in the internet. Since the importance of the interoperability for mobile networking services, semantic ontology may facilitate how each device is suitable for different types of information and interaction demands. Kalaoja et al [2] presented an analysis for the ontology in different functional domains to cope with the heterogeneity of service descriptions like discovery, integration and composition. Mobile devices combine different features like telephone, email, multimedia message, and web browsing to enable employees to keep current with organizational activities. However, this causes a problem of delegating the right access decision for the corporate resources. Traditional access control policies follow different rules to manage resource access between users and companies. However, in a pervasive environment, dynamic and spontaneous access rights among the peers and service providers should be context dependant and specified by the company managements. This paper reviews different access control mechanisms, analyzes them semantically, and offers a combined approach for secure access. The remainder of this paper is organized as follows. We first provide a detailed background through a comprehensive review of previous studies in the area. We present traditional access control policies and semantic approach in section 3. We propose our model in section 4 and finally draw conclusions in section 5.

2 Literature Review Since mobile communications and computing tools have been used for the last 5-6 years, best practices and standards in personal and corporate use for mobility must be identified. Considerable research on how to design different and robust security mechanisms for mobile devices and what the right mix of technology are summarized with the following topics. − − − − − − −

Semantic web technologies and mobile devices Access control applications and models Role-based access control (RBAC) Activity-based access Control Model (ACM) Context-based services and Information Retrieval Friend-of-a-friend (FOAF) model Attribute-based access control (ABAC)

200

T. Ercan and M. Yıldız

As mobile devices are smaller and more sophisticated, and become easier to obtain, their use has made them common tools for all users. Mobile devices act as both requestors and providers of data. Doulkeridis and Vazirgiannis [3] argued that context for mobile web services would play an important role in service discovery and focus on the semantic matching of static attributes. Some of the previous works have examined how to overcome main problems of mobile devices due to their well-known drawbacks. Jou [4] designed a semantics-based web content adaptation framework to provide contents to different mobile devices. Mobile devices run with the context-aware applications based on the context information like relative position of users, user preferences, device capabilities and available resources. Semantic languages are well suited for the proper usage of context information and facilitate knowledge sharing and interoperability among heterogeneous devices. Semantic languages require complex and heavyweight features that may not fit the capabilities of all user devices [5-8]. The highly dynamic and contextdependent requirements of corporate services in distributed environments motivate and recommend the use of ontology-based techniques to combine user requests and service descriptions. Ontology-based approach for Mobility Information Systems in [9] provides different companies and individual consumers with the information to facilitate their mobility in a metropolitan area. Main advantages are semantic user queries that are independent of the system implementation and the possibility of delegating user tasks to mobile software agents. Reference [10] and [11] respectively defines another automatically generated access mechanism based on a set of different access patterns in ontology-based systems and a role signature based on generic role identifiers to verify authorization by the roles. In order to handle problems with mobile agents and object frameworks, new features of firewalls enhanced flexible access control perimeters. Domain and Type Enforcement (DTE) firewall runs application-level proxies in restrictive domains to increase security [12]. A predefined criterion-based multilayer access control (CBMAC) approach extracted from authorization rules and enhanced existing access control models such as Role-Based, Mandatory, and Discretionary Access Control models to support multilayer access control was presented in [13]. Role-based access control (RBAC) is a widely used solution to cope with these problems. However, as the size of the organization increases, users’ attributes such as membership class or job position become more complicated. It is difficult that a centralized role server manages the large and complex role hierarchy. A distributed role hierarchy that can manage the role hierarchy effectively and practically in [14] ensures web security. Yamazaki et al [15] discusses that how an access control system enables managing dynamic security policies by using the Role-Based Access Control (RBAC) that the agents decide access rights dynamically by using context-enabled rules and an inference engine for the role defined. Reference [16-20] describe different models to assign dynamically users to roles by setting out attributes that are not accessible according to the specified access control policies. Hung et al [21] proposed Activity-based access Control Model (ACM) leveraging user’s activities to determine the access permissions for that user. In ACM, a user is assigned to perform a number of actions if s/he poses a set of satisfactory attributes and access permissions to hospital information are granted according to users’ actions. Jung et al [22] studied on a collected dataset of user activities in telecommunication

Semantic Access Control for Corporate Mobile Devices

201

industry and investigated how much the personal context of a certain person is interrelated with those of other people to build meaningful relationships by a semantic approach. Other information retrieval systems were proposed in [23-25] to enable semantic information to represent context and domain knowledge and aid interactions mediated by mobile devices in an easy and efficient way, anywhere and anytime for wireless devices. FOAF ontology schema evaluates the context query (semantically searchable), share ability among local and remote context-aware applications [26]. A trust mechanism that takes advantage of both the capabilities of the Semantic Web and mobile ad-hoc networks was described in [27], since mobile devices enable social interaction as a level of trust for another person. Attribute Based Access Control (ABAC) model as a new approach is based on subject, object, and environment attributes and supports both mandatory and discretionary access control (MAC-mandatory access control, DAC-Discretionary access control) [27]. ABAC provides a promising approach to define authorization over shared resources and is based on users’ attributes rather than their identities. However, user’s attributes are always asserted by different authorities that may not be accepted by the resource owner with the same degree of trust.

3 Traditional Access Control Policies and Semantic Approach Current information security mechanisms are insufficient and not well suited to afford the dynamic and ad hoc status in wireless and mobile environment. Access control mechanisms run the authentication and authorization decisions on an entity's (subject) access to corporate resources. Credentials are accepted as obligations to be fulfilled by the subject when the network connection is established. Environment can include everything around the user. It is independent from the subject (user or mobile device) and object (any kind of resource). Proper usage of the resources in a dynamic environment requires important and sensitive decisions. Some of the current access policies summarized below explain how to handle different access decisions. Discretionary access control (DAC) is the user centric and the ownership of every system resource is assigned one or more system users. System users control the use of objects under their control. This model is commonly used for being simple, flexible and ease of implementation. The resource access privileges in Mandatory Access Control (MAC) are determined by the system as distinct from DAC. Access to the data resources remains unchanged, since it has been predefined with administrative procedures [28]. Role Based Access Control (RBAC) emerged as an option to DAC and MAC policies. RBAC regulates access control action according to the subject's role. Thus, users inherit all the permissions associated with these roles and their order simplifies the definition of policies separately. Attribute-Based Access Control (ABAC) uses the attributes of subjects and objects for authorization instead of directly using the permissions between them. There are three different attributes: subject, resource and environment [29]. Organization Based Access Control (OrBAC) system has been developed because of the problems experienced in corporate access control policies. It works with the presence of three components that are subject, action and object. The policy describes how some subjects have permissions to perform some actions on

202

T. Ercan and M. Yıldız

some objects with the aim of controlling access. OrBAC mechanism gives permission to write policies regardless from the application [30]. New access control mechanisms, policies and languages have been developed for the definition and implementation of security requirements in real-world scenarios, such as internet usage and wireless pervasive networks. The representation of semantic knowledge in the access control mechanisms in the previous section has been partly covered. The resource owner (subject), the source of information (object) and the desired action (action) that would be performed on the source should be kept as ontological, in order to ensure the existence of semantic approach. When we examined the access control mechanisms, these ontological data (resources to access, entity that has access to the resources, and the action which will be held on the resource) would stand out. In this respect, the source, subject and actions in the access control mechanism must be represented as semantic to ensure the targeted semantic level. If we apply the proper corporate policies, a corporate policy decision maker can handle these components like subject, actions, conditions, domain ontology and policy object. By using the framework proposed in [31], particular pieces of information relevant to the corresponding users could be summarized and transferred again into the corporate mobile users’ environment by generating semantic templates as users’ input. Since the importance of the interoperability issues in heterogeneous mobile networking services, semantic ontologies may facilitate how each device is suitable for different kinds of information and different interaction demands. The advantages coming with the adaptation of semantic based models improve the efficiency and accuracy for selecting the right web services in the corporate environments. Many of the works in this area have been referenced in [32]. Even though the advancement in mobile and wireless technologies affects the demands for significant mobile services, discovering such services from any device is still a major challenge due to the variety of available mobile devices and the lack of service descriptions. Ontology-based approaches create alternatives to discover the mobile services with respect to user preferences and device capabilities. Niazi et al [33] provided a Delivery Context Ontology with the enriched profile-based service descriptions to discover, personalize, adapt and automatically execute of mobile services with maximum degree of user satisfaction.

4 Activity and Context Based Access Control When we looked at the previous works, we see that there are two types of security applications for mobile users. The first one is kept in the user device and processes automatically security-related mechanisms. Other is the rules of the system, and organization. Predetermined corporate policies ensure security and reliability for the usage of mobile devices at least for corporate applications and during working hours, if and only if a company employee in the work-related issues uses them. Employees deal with specific applications and data referred to the organizational activities. User activity-based access control mechanisms in the company side check for user activities to determine access rights for the user. Context-based applications handled with semantic web technologies have been increased in today’s web domains, in the area

Semantic Access Control for Corporate Mobile Devices

203

of e-commerce. Context data can be characterized as user profiles and rights, users’ networking environment, mobile devices, mobile applications and a particular time during the day.

Fig. 1. Additional classes

In our work, we tried to outline a combined policy of activity and context based access control approaches. Both use the web interfaces of the service provider domains. While activity-based application uses XML specifications, context based approach uses the creation of necessary ontology. Reference [34] presented a new method for searching documents, which have similar topics. It was designed to help mobile device users to search for documents in a peer-to-peer environment, which have similar topic to the ones on the users’ own device. The algorithms reasonably were designed for slower processors, smaller memory and small data traffic between the devices. These features allowed the application in an environment of mobile devices like phones or PDAs. Therefore, a common corporate ontology can be developed by using these aforementioned concepts and features. Figure 1 shows additional classes in dotted circles, with a few attributes determined by the corporate policy in a sample FOAF format. We can define the following steps in the creation of corporate policies. − Identification of corporate activities (beginning with the user access into the organization web site) related to the corporate policy. − Description of conditions (networking type, bandwidth, user device, user applications etc.) related with the activities by using the domain ontology − Determining the type of policy objects and association of policy objects described with the policy subject. Pre-determined corporate policies allow application-specific policies to move along with the logic of source, action and object in a distributed computing system. Figure 2 shows an example of application permission.

204

T. Ercan and M. Yıldız

Except the “companyA” employees, no one is allowed to practice “application A”. (Deny) “CompanyB” employees use “application A2” of “CompanyA”. (Allow) Fig. 2. Allow/Deny sentence

The context of an organizational services infrastructure is expected to support all mobile users, other business partners and remote users as well. Figure 3 illustrates a dynamic security model, which we proposed for the access solutions in a pervasive environment.

Fig. 3. Proposed Architecture

This model has been emerged through practical experience of authors in developing solutions for service-oriented projects and is based on personal and corporate attributes and user activities for the requested applications. Characteristics of the user’s environment taken into account allow them to deliver relevant information to the corporate policy database. It is imperative that any deployable technology be inserted into the policy database with the required attributes to allow/deny end users. This multilayer architecture different from the traditional 7 layer security process design can provide more powerful solution for the needs of the electronic commerce applications in different domains. The credentials are kept again in the AAA server. Our proposed model uses different aspects of semantic approaches in access control mechanisms. Our model uses existing traditional access control mechanisms and works strengthens the drawbacks of the wireless environment created by recent semantic approaches.

Semantic Access Control for Corporate Mobile Devices

205

5 Conclusion Defining the access control policies and the connectivity of mobile systems into the corporate web site will reduce the problems during service creation. Other partner organizations collaborating with the central company will also require similar type securities. Flexibility in allowing organizations to define their own security policies will assist in the growth of the secure collaborations through their own web sites. Mobile users are becoming more aware of computing and want the flexibility to initiate organizational processes on their portable devices in a safe mode. Dealing with new context information brings new access and privacy issues into the related organization. Facilitating a semantic approach in the context-based access control for a broad range of applications in different industrial sectors enables the benefit of different vertical markets. However, specific ontologies owned by private organizations are only feasible at the application or agency (company) level. There can be different ontologies and semantic models for a common consensus about business complexities affecting organizations. Semantic infrastructure should be integrated with today’s active security terms and standards such as rules, signature, encryption, proof, and trust. Computers at both end user and company side should have the required web scaled knowledge base of hyperlinked data in order to reason with the content between end user devices and company web servers. A web widget approach proposed in [35] allows adding semantic functionality to systems with just a few lines of code. One of the benefits of publishing widgets as centralized services (between the end user and the organization) is that updates in content and functionalities are instantaneously available for the users. This approach can easily be integrated into applications owned by the company.

References 1. Veijalainen, J.: Developing mobile ontologies; who, why, where, and how? In: Mobile Services oriented Architectures and Ontologies Workshop, MoSO 2007 (2007) 2. Kalaoja, J., Kantorovitch, J., Carro, S., Miranda, J.M., Ramos, A., INSTICC: The vocabulary ontology engineering for the semantic modelling of home services. In: 8th International Conference on Enterprise Information Systems (ICEIS 2006), May 23-27 (2006) 3. Doulkeridis, C., Vazirgiannis, M.: Querying and updating a context-aware service directory in mobile environments. In: WI 2004 IEEE/WIC/ACM International Conference on Web Intelligence, September 20-24 (2004) 4. Jou, C.: A semantics-based automatic web content adaptation framework for mobile devices. Web Information Systems and Technologies, 230–242 (2008) 5. Corradi, A., Montanari, R., Toninelli, A., IEEE Computer, S.: Adaptive semantic support provisioning in Mobile Internet environments. In: International Symposium on Applications and the Internet (SAINT 2005), Janaury 31-February 04 (2005) 6. Weissenberg, N., Gartmann, R., Voisard, A.: An ontology-based approach to personalized situation-aware mobile service supply. Geoinformatica 10(1), 55–90 (2006) 7. Drogehorn, O., Wust, B., David, K., IEEE: Personalised applications and services for a mobile user. In: International Symposium on Autonomous Decentralized Systems (ISADS 2005), April 4-8 (2005)

206

T. Ercan and M. Yıldız

8. Bianchini, D., De Antonellis, V., Melchiori, M., Salvi, D.: Lightweight ontology-based service discovery in mobile environments. In: Seventeenth International Conference on Database and Expert Systems Applications, Proceedings, pp. 359–364 (2006) 9. Faro, A., Giordano, D., Musarra, A., IEEE, I.: Ontology based intelligent mobility systems. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 1-5, pp. 4288–4293 (2003) 10. Villanueva, F.J., Villa, D., Barba, J., Rincon, F., Moya, F., Lopez, J.C.: Ontology access patterns for pervasive computing environments. In: Mikulecky, P., Liskova, T., Cech, P., Bures, V. (eds.) Ambient Intelligence Perspectives, pp. 236–244 (2009) 11. Crampton, J., Lim, H.W.: Role signatures for access control in open distributed systems. In: 23rd International Information Security Conference held at the 20th World Computer Congress, September 07-10 (2008) 12. Oostendorp, K.A., Badger, L., Vance, C.D., Morrison, W.G., Petkac, M.J., Sherman, D.L., Sterne, D.F., IEEE Comp. Soc.: Domain and type enforcement firewalls (1997) 13. Pan, L., Zhang, C.N.: A Criterion-Based Multilayer Access Control Approach for Multimedia Applications and the Implementation Considerations. ACM Transactions on Multimedia Computing Communications and Applications 5(2) (2008) 14. Lee, G.H., Yeh, H.J., Kim, W.I., Kim, D.K.: Web security using distributed role hierarchy. In: 2nd International Workshop on Grid and Cooperative Computing, December 07-10 (2003) 15. Yamazaki, W., Hiraishi, H., Mizoguchi, F., IEEE Computer, S.: Designing an agentbased RBAC system for dynamic security policy. In: 13th IEEE International Workshop on Enabling Technologies - Infrastructure for Collaborative Enterprises, June 14-16 (2004) 16. Al-Kahtani, M.A., Sandhu, R., IEEE Computer Society, I.C.S.: A model for attributebased user-role assignment. In: 18th Annual Computer Security Applications Conference, December 09-13 (2002) 17. Carminati, B., Ferrari, E., Tan, K.L., ACM: Enforcing Access Control Over Data Streams. In: 12th ACM Symposium on Access Control Models and Technologies, June 20-22 (2007) 18. Park, J.S., Ahn, G.J., Sandhu, R.: Role-based access control on the web using LDAP. Database and Application Security XV, 19–30 (2002) 19. Schwartmann, D.: An attributable role-based access control for healthcare. In: Bubak, M., DickVanAlbada, G., Sloot, P.M.A., Dongarra, J.J. (eds.), pp. 1148–1155 (2004) 20. Greenhalgh, C., Glover, K., Humble, J., Robinson, J., Wilson, S., Frey, J., Page, K., De Roure, D., IEEE: Combining System Introspection with User-Provided Description to Support Configuration and Understanding of Pervasive systems. In: 3rd International Conference on Pervasive Computing and Applications, October 06-08 (2008) 21. Hung, L.X., Lee, S., Lee, Y.K., Lee, H.: Activity-based access control model to hospital information. In: 13th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications, August 21-24 (2007) 22. Jung, J.J., Lee, H., Choi, K.S.: Towards Efficient Reality Mining with Contexts and Semantics: a Case Study of Telecommunication. In: 2nd International Symposium on Intelligent Information Technology Application, December 21-22 (2008) 23. Naudet, Y., Aghasaryan, A., Toms, Y., Senot, C.: An Ontology-based Profiling and Recommending System for Mobile TV. In: 3rd International Workshop on Semantic Media Adaptation and Personalization (SMAP 2008), December 15-16 (2008) 24. Martins, D.S., Santana, L.H.Z., Biajiz, M., do Prado, A.F., de Souza, W.L., ACM: Context-aware Information Retrieval on a Ubiquitous Medical Learning Environment. In: 23rd Annual ACM Symposium on Applied Computing, March 16-20 (2008)

Semantic Access Control for Corporate Mobile Devices

207

25. Mena, E., Illarramendi, A., Royo, J.A., Goni, A.: A Software Retrieval Service Based on Adaptive Knowledge-Driven Agents for Wireless Environments. ACM Transactions on Autonomous and Adaptive Systems 1(1), 67–90 (2006) 26. Hu, D.H., Dong, F., Wang, C.L., IEEE Computer, S.O.C.: A Semantic Context Management Framework on Mobile Device. In: 6th International Conference on Embedded Software and Systems, May 25-27 (2009) 27. Yuan, E., Tong, J., IEEE Comp. Soc: Attributed based access control (ABAC) for web services. In: IEEE International Conference on Web Services, Proceedings, July 11-15, vol. 1,2 (2005) 28. Benantar, M.: Access Control Systems Security, Identity Management and Trust Models. Springer Science Business Media, Heidelberg (2006) 29. Yuan, E., Tong, J.: Attribute Based Access Control - A New Access Control Approach for Service Oriented Architecture (SOA). In: New Challenges for Access Control Workshop (2005) 30. Cuppens, F., Miège, A.: Modelling Contexts in the Or-BAC Model. In: 19th Annual Computer Security Applications Conference (2003) 31. Jung, J.J., Park, S.B., Jo, G.S.: Semantic template generation based information summarization for mobile devices. In: Shimojo, S., Ichii, S., Ling, T.-W., Song, K.-H. (eds.) HSI 2005. LNCS, vol. 3597, pp. 135–143. Springer, Heidelberg (2005) 32. Lau, B.Y.S., Pham-Nguyen, C., Lee, C.S., Garlatti, S., IEEE: Semantic Web Service Adaptation Model for a Pervasive Learning Scenario. In: 2nd IEEE Conference on Innovative Technologies in Intelligent Systems and Industrial Applications, July 12-13 (2008) 33. Niazi, R., Mahmoud, Q.H., IEEE.: An Ontology-Based Framework for Discovering Mobile Services. In: 7th Annual Conference on Communication Networks and Services Research, May 11-13 (2009) 34. Csorba, K., Vajk, I.: Iterative Search for Similar Documents on Mobile Devices. In: Dengel, A.R., Berns, K., Breuel, T.M., Bomarius, F., Roth-Berghofer, T.R. (eds.) KI 2008. LNCS (LNAI), vol. 5243, pp. 38–45. Springer, Heidelberg (2008) 35. Makela, E., et al.: Enabling the semantic web with ready-to-use web widgets. In: Proceedings of the First Industrial Results of Semantic Technologies Workshop, ISWC 2007 (2007)

A New Visual Simulation Tool for Performance Evaluation of MANET Routing Protocols Md. Sabbir Rahman Sakib1, Nazmus Saquib1, and Al-Sakib Khan Pathan2 1

Department of Electrical and Electronic Engineering Department of Computer Science and Engineering BRAC University, 66 Mohakhali, Dhaka 1212, Bangladesh {srsakib,nsaquib,spathan}@bracu.ac.bd 2

Abstract. A new user-friendly visual simulation tool; ViSim is presented. ViSim could be useful for researchers, students, teachers in their works, and for the demonstration of various wireless network scenarios on the computer screen. It could make the task of simulation more exciting and enhance the interest of the users without going into complex command-only text interface. Using our simulation tool, we measured the performance of several Mobile Ad-hoc Network (MANET) routing protocols. In this paper, we present the performance analysis of three prominent MANET routing protocols; DSDV, DSR, and AODV using ViSim. The details of various features of ViSim, brief descriptions of the selected routing protocols and their comparisons, details about the performed experiments, and gained results are also presented. Keywords: Comparison, Graphical, ns-2, Performance, Routing, Simulation.

1 Introduction A Mobile Ad-hoc Wireless Network (MANET) is a collection of autonomous nodes that communicate with each other by forming a multi-hop network, maintaining connectivity in a decentralized manner [1], [2]. It consists of a set of mobile nodes communicating amongst themselves using wireless links, without the use of any centralized entity. Because of the mobility feature, any node may sometimes go out of the range of other nodes in the network. Therefore, MANET routing is difficult since mobility causes frequent network topology changes and requires more robust and flexible mechanism to search for and maintain the routes. Because of the challenging features of MANET routing, it has been under tremendous scrutiny and interest from the time of its emergence. In fact, it is one of most addressed topics. Though many routing protocols have already been proposed and well-accepted in the research community because of their given promise and performance, there still remains the necessity of a flexible, user-friendly simulation tool that can make the task of simulation and visualization of routing protocols easier. Many simulators can successfully simulate various routing protocols of MANET. However, there are only a few tools to handle the simulations with a user-friendly graphical interface. This particular fact has motivated us to design and develop our tool so that the users could be able to deal with complex simulation scenarios in a much easier way without getting involved into the command-only interface. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 208–217, 2010. © Springer-Verlag Berlin Heidelberg 2010

A New Visual Simulation Tool for Performance Evaluation

209

We have named our simulation tool ‘ViSim’. ViSim could help a network administrator to choose a particular ad-hoc routing protocol for a specific scenario through analyzing the graphs for different routing protocols. Alongside describing various aspects and features of ViSim, we have also analyzed the performances of the prominent routing protocols for MANET using our tool. The rest of the paper is organized as follows: Section 2 describes ViSim. Performance evaluations using our tool are presented in Section 3, Section 4 talks about some relevant works, and finally Section 5 concludes the paper.

2 ViSim: A Visual Simulation Tool 2.1 Building Blocks of ViSim We have used two software in Windows environment; ActiveTcl8.3.5 and Microsoft Visual Basic (VB) 6.0. ActiveTcl is an industry-standard Tcl distribution, available for Windows, Linux, Mac OS X, Solaris, AIX and HP-UX. This software creates an environment in Windows to run the ns-2 [3] simulations and .tcl scripts. It is capable of executing the simulation faster than cygwin [4]. For details of ActiveTcl, the readers are encouraged to visit; http://www.activestate.com/activetcl/ . Microsoft VB is a popular software that we have used for developing ViSim prototype so that it can connect the simulation related tasks with a user-friendly graphical interface. 2.2 Overview of ViSim Our graphical simulation tool, ViSim is built using Visual Basic 6.0 in order to make comparisons among various MANET routing protocols since there are very few prototypes available today for performing such type of task. Most of the available tools are somewhat not user-friendly. Hence, keeping that in mind, we built ViSim in such a way that any naive user can also be able to use this tool to visualize the background simulations done in ns-2 (that is run with the help of ActiveTcl in Windows operating system). ViSim runs associated .tcl files for all the three mentioned protocols (DSDV [5], DSR [6], AODV [7]) and extracts the required information from the trace files that are generated. Eventually the graphs are plotted for different performance indicators such as Throughput, Goodput, and Routing Loads. ViSim can make the task of a network administrator easy to decide which routing protocol would be better for a particular MANET scenario. 2.3 Different Working Areas Figure 1 (left) shows the ViSim prototype/tool when it is run in Windows environment for the first time. The graphical interface has some working areas and functionalities that should be known before using it for analysis of various parameters: (a) Simulation Area: In this area, three routing protocols are mentioned. Clicking on the names of each protocol gives the options of simulating three network scenarios using that particular protocol. (b) Comparison Area: This area has the options; Throughput vs Time, Goodput (Packets), Routing Load (Packets), Goodput (Bytes), and Routing Load (Bytes). All

210

Md.S.R. Sakib, N. Saquib, and A.-S.K. Pathan

these buttons are used to select the parameters that the user needs for the performance analysis and comparison among the routing protocols. (c) Scenarios and Protocols Area: This area specifies the options of three network scenarios (radio buttons) and three routing protocols (tick boxes). Also it has two buttons namely; ‘Simulate’ and ‘Create Graph’. ‘Simulate’ button is used for playing the simulations and ‘Create Graph’ is used to plot the comparison graphs. (d) Output Area: This is shown as a blank window area when ViSim is run for the first time. Based on the choice of various options, the outputs or further options are shown in this area. The graphs are also plotted on this area.

Fig. 1. (Left) ViSim Interface, version 1.0 (in Windows XP). (right) DSR simulation options.

2.4 Functionalities of ViSim with Examples Now, let us see the functionalities of ViSim with some practical examples. Let us suppose that we want to visualize the simulation for DSR for a particular network scenario. For this task, first we have to click DSR button under simulation area. After clicking DSR button, ViSim shows three more options (DSR Simulation 1, DSR Simulation 2, and DSR Simulation 3) on the output area as shown in Figure 1 (right). From these three options any one could be chosen. For our task, let us choose DSR Simulation 3. After clicking this button, ViSim calls ns-2 in its background, then reads .tcl file that specifies the simulation scenario 3, generates .nam and .tr files. Once the .nam and .tr files are generated, ViSim calls the NAM (Network Animator tool) in its background and reads the generated .nam files. Consequently, it shows a display for visual simulation [see Figure 2 (left)]. On the NAM screen, there are few buttons such as play, forward, backward, stop buttons available to control the simulation as these are done usually in Linux based environment with ns-2 and NAM. To see the visual simulation on the screen, the play button should be clicked. Like any other simulation using NAM, we can also change the step size of the simulation. Now, if we want to make comparisons among three different protocols for performance analysis, we have to choose a specific network scenario. In our case, let us select Scenario 1. Then we have to select three mentioned protocols (or, any two or

A New Visual Simulation Tool for Performance Evaluation

211

Fig. 2. (Left) The output after choosing DSR Simulation 3. (right) A sample output graph (Throughput vs Time) using network scenario 1, all three protocols are compared.

one) and side by side the performance indicators should be clicked from the five options in the comparison area. Once the simulations are performed by clicking the ‘Simulate’ button, we can use the generated results in the background for plotting comparison graphs. Basically, this ‘Simulate’ button facilitates performing various simulations with three protocols for a particular network scenario at the same time. This reduces the burden of doing the tasks repeatedly or selecting one protocol at a time under Simulation area. Once all the simulations are completed, the graph can be generated by clicking the ‘Create Graph’ button. By clicking ‘Create Graph’ button, we send the command to read the generated .tr files (trace files) and extract the required information/values from those. These values are used to plot the graphs for different protocols for a specific scenario and for different performance indicators. Figure 2 (right) shows a sample output of what we have done so far (it should be mentioned that each simulation and plotting of graph takes a bit time as required by ns-2; for example in our case, it took about 25 seconds to plot the graph). Let us talk about the working mechanism of ViSim buttons briefly. When the user selects the simulation option in order to view the simulation for a particular scenario corresponding to the selected ad hoc network protocol, ViSim calls up a .bat file which contains shell script. This shell script calls the ns-2 and feeds files or file having extension .tcl, according to the choice of simulation. Then ns-2 generates trace file (extension .tr) and nam file (extension .nam). After that NAM is called via shell script and using NAM the script feeds .nam file into NAM which gives a GUI (Graphical User Interface) popup and using it, a user can actually observe the simulation. Again, when the user selects the Comparison option and clicks Create Graph after performing simulations, ViSim gathers the .tr files according to the choice of protocol, reads those and according to the performance indicators, it filters the data and picks up important information to generate the graph. For ViSim, we used some given network specifications. Note that any specification can be modified in the .tcl files according to the requirements to simulate another network setting. Also, various parameters used in ViSim code could be given new values. A diagram of the operational flows of ViSim is presented in Figure 3.

212

Md.S.R. Sakib, N. Saquib, and A.-S.K. Pathan

Fig. 3. Diagram of the operational flows of ViSim

2.5 ViSim File Organization and Simulation Scenario Modification In this sub-section we briefly talk about the file organization that is used in our tool. After installing ViSim a particular folder appears in which several .tcl files are kept. For any .tcl file, the general format that is used is: [The naming convention: a + Adhoc protocol (X) + Scenario (Y).tcl]=axy.tcl, where x represents the protocol whose value ranges from 0 to 2 and y represents the scenario whose value also ranges from 0 to 2. Now, every value of x or y is associated with the file name. Let us have a look at Table 1 for a clear idea regarding the naming convention that is used. Table 1. File Naming Convention ([row+column] values in the cells)

0 (AODV) 1 (DSR) 2 (DSDV)

0 (Scenario 1)

1 (Scenario 2)

2 (Scenario 3)

00

01

02

10

11

12

20

21

22

Therefore, if a user needs to change a file’s specifications, say for example; the 2nd scenario of DSR routing protocol, he needs to open the a11.tcl file and do necessary changes. The user’s manual is also available in ViSim website (mentioned later) which tells more about these facilities. We have noted this information here to indicate that ViSim could be used both as a simulation demonstration tool and a simulator tool for MANET routing protocols.

A New Visual Simulation Tool for Performance Evaluation

213

3 Performance Evaluations and Results Table 2 shows the specifications and parameters that we used for our experiments: Table 2. Simulation Parameters Simulation Parameter Channel Type Radio-propagation model Network interface type MAC type Interface Queue Type Antenna model Number of Mobile nodes Ad Hoc Routing Protocol Simulation Area Simulation Time Traffic Type Nodal speed Packet size Total Number of different Scenarios

Value Wireless Channel Two Ray Ground Model Wireless Physical 802_11 b Drop Tail Primary Queue Omni Direction 3-10 DSDV, DSR, AODV 500m x 400m 150 ms TCP 3-10 m/s 1040 Byte (Data Packets) 40 bytes(Acknowledgement Packets) 60 Bytes (Routing Packets) 15

To evaluate the performances of various routing protocols we took three network scenarios; Scenario 1 with 3 nodes, Scenario 2 and 3 with 10 nodes with different mobility characteristics. Comparisons among different protocols were based on the aggregate of the performance metrics resulting from the simulations of 15 different scenarios that had been performed for each protocol separately. To measure the performances, we used the following metrics: Throughput. The total bytes received by the destination node per second (Data packets and Overhead). Goodput (In terms of Number of Packets). The ratio of the total number of data packets that are sent from the source to the total number of packets that are transmitted within the network to reach the destination. Goodput (In terms of Packet Size in Bytes). The ratio of the total bytes of data that are sent from the source to the total bytes that are transmitted within the network to reach the destination. Routing Load (In terms of Number of Packets). The ratio of the total number of routing packets that are sent within the network to the total number packets that are transmitted within the network to reach the destination. Routing Load ( In terms of Packet Size in Bytes). The ratio of the total bytes of routing packets that are sent within the network to the total number of bytes that are transmitted within the network to reach the destination. To illustrate our experimental results using our tool, we first present all the outputs of 15 different cases in the Figure 4 [(a) to (f)], Figure 5 [(a) to (c)], Figure 6 [(a) to (c)], and Figure 7 [(a) to (c)].

214

Md.S.R. Sakib, N. Saquib, and A.-S.K. Pathan

Figure 8 (left) shows the aggregated result for ‘Throughput vs Time’ where we analyzed the total bytes received by the destination node per second (Data packets and overhead). Based on the results that we see here, the following comments could be made: (a) AODV: starts off quickly and the data rate is more stable, (b) DSR: starts off quickly however we can see that there are lots of fluctuations in the data rate, (c) DSDV: takes time to start off but the data rate has lesser fluctuations.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 4. Throughput vs Time: (a) Scenario 1 (b) Scenario 2 (c) Scenario 3 [y axis represents Throughput (KB) and x axis represents Time (seconds)]. Goodput (packets): (d) Scenario 1 (e) Scenario 2 (f) Scenario 3.

(a)

(b)

(c)

Fig. 5. Routing Load (packets) (a) Scenario 1 (b) Scenario 2 (c) Scenario 3

We calculated Goodput in terms of number of packets and the packet size in bytes. Now, if we analyze the graph presented in Figure 8 (right), we can see that on an average, if 100 packets are transmitted in the network, 19 packets would be data packets for AODV, 16 for DSR, and 24 for DSDV. In term of bytes, on an average; if 100 bytes of packets are transmitted through the network, 36 bytes would be data

A New Visual Simulation Tool for Performance Evaluation

215

packets for AODV, 28 bytes for DSR, and 48 bytes for DSDV. From these data, we could deduce that; though DSDV takes time to converge, it actually is sending more data packets in number as well as in bytes than that of AODV and DSR. Now, the rest of the percentage of each individual graph will be the overheads that contain routing packets and acknowledgements.

(a)

(b)

(c)

Fig. 6. Goodput (bytes) (a) Scenario 1 (b) Scenario 2 (c) Scenario 3

(a)

(b)

(c)

Fig. 7. Routing Load (bytes) (a) Scenario 1 (b) Scenario 2 (c) Scenario 3

Fig. 8. (Left) Throughput vs Time. (right) Goodput for three MANET routing protocols.

We again calculated routing loads in terms of number of packets and the packets size in bytes. The results are presented in Figure 9. Again we can see that; though DSR has a better throughput, it actually contains more overhead for routing packets. However, DSDV has a relatively lower routing load than AODV and DSR.

216

Md.S.R. Sakib, N. Saquib, and A.-S.K. Pathan

4 Related Works Rosen et al. [8] describe a simulation tool, SHOPMET; used to study the creation and optimization of propagation maps for Node State Routing protocols within wireless ad hoc networks. Though the authors have tried to develop an all-in-one tool, this is too focused to work with a specific type of routing protocol and thus the scope of using it for various purposes is limited. In [9], the authors present their interactive ns-2 protocol and environment confirmation tool (iNSpect). Though iNSpect on the surface seems to be an elegant tool for visual simulation, it is not user-friendly and requires much effort to make the codes run properly. Other than these works, some good surveys on different simulation tools for MANET (or any other wireless networks) could be found in [10] and [11]. There are many works like [12], [13], [14], [15] available which have evaluated the performances of different MANET routing protocols. However, in our work we used our own developed tool to simplify the tasks of multiple simulations at a time. We also have considered various scenarios to run our simulations. We already have talked about various simulation tools and their features. Compared to all other works, ViSim gives much more user-friendliness, flexibility, functionalities, and ease of using complex simulation scenarios. The main difference between ViSim and any other simulation tool is that ViSim uses ns-2 simulations in the background but makes the tasks lot easier for the users. The plotting of graphs is also a great feature and this facility was not included in most of the previously developed graphical simulation tools. Our tool is developed in such a way that it can be used by all types of users. A naive user can use it for visualizing simulations or for simulation demonstration. An expert user can write his own tcl script and run it using ViSim tool and then can generate various performance comparison graphs without taking resort to other cumbersome programming methods.

Fig. 9. Routing loads for different experimented MANET protocols

5 Conclusions and Future Works In this paper, we have presented our user-friendly simulation tool/prototype which can ease the task of simulation of MANET routing protocols even in Windows based environments. Many users dealing with ns-2 simulations face troubles in setting up

A New Visual Simulation Tool for Performance Evaluation

217

Linux or other systems and environment. The use of ActiveTcl with graphical ViSim interface could really be beneficial for the research community in general. Using our simulation tool, we obtained different graphs and analyzed the results for different MANET routing scenarios. As our future works, we would like to add more functionalities to ViSim with easy access to the programming codes and parameter changes for various network scenarios. For the information about the official release of ViSim 1.0, the readers are encouraged to visit: http://faculty.bracu.ac.bd/~spathan/research/ visim.html Acknowledgments. Special thanks to Taufiq Abdur Rahman and Sadia Hamid Kazi for their cooperation in this work.

References 1. Pathan, A.-S.K., Hong, C.S.: Routing in Mobile Ad Hoc Networks. In: Misra, S., Woungang, I., Misra, S.C. (eds.) Guide to Wireless Ad Hoc Networks, pp. 59–96. Springer, Heidelberg (2009) 2. Marti, S., Giuli, T.J., Lai, K., Baker, M.: Mitigating routing misbehavior in mobile ad hoc networks. In: Proceedings of ACM MOBICOM, Boston, MA, USA, pp. 255–265 (2000) 3. The Network Simulator - ns-2, http://www.isi.edu/nsnam/ns/ 4. http://www.cygwin.com/ 5. Perkins, C.E., Bhagwat, P.: Highly Dynamic Destination-Sequenced Distance-Vector Routing (DSDV) for Mobile Computers. In: Proceedings of ACM SIGCOMM, pp. 234–244 (1994) 6. Broch, J., Johnson, D.B., Maltz, D.A.: The Dynamic Source Routing Protocol for Mobile Ad Hoc Networks. IETF Draft (1999), http://tools.ietf.org/html/draft-ietf-manet-dsr-03 7. Perkins, C.E., Royer, E.M., Chakeres, I.D.: Ad Hoc On-Demand Distance Vector (AODV) Routing. IETF Draft (2003), http://tools.ietf.org/html/draft-perkins-manet-aodvbis-00 8. Rosen, S.L., Stine, J.A., Weiland, W.J.: A MANET Simulation Tool to Study Algorithms for Generating Propagation Maps. In: Proc. of the 2006 IEEE WSC, pp. 2219–2224 (2006) 9. Kurkowski, S., Camp, T., Mushell, N., Colagrosso, M.: A Visualization and Analysis Tool for Wireless Simulations: iNSpect. In: Proc. of 13th IEEE Int. Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 503–506 (2005) 10. Hogie, L., Bouvry, P., Guinand, F.: An Overview of MANETs Simulation. Electronic Notes in Theoretical Computer Science 150(1), 81–101 (2006) 11. Rahman, M.A., Pakštas, A., Wang, F.Z.: Network Modelling and Simulation Tools. Simulation Modelling Practice and Theory 17, 1011–1031 (2009) 12. Das, S.R., Castaneda, R., Yan, J., Sengupta, R.: Comparative Performance Evaluation of Routing Protocols for Mobile Ad hoc Networks. In: Proc. of 7th IEEE ICCCN, pp. 153–161 (1998) 13. Rahman, A.H.A., Zukarnain, Z.A.: Performance Comparison of AODV, DSDV and I-DSDV Routing Protocols in Mobile Ad Hoc Networks. European Journal of Scientific Research 31(4), 566–576 (2009) 14. Schmidt, R.d.O.: MANETs Routing Protocols Evaluation in a Scenario with High Mobility MANET Routing Protocols Performance and Behavior. In: Proc. of IEEE Network Operations and Management Symposium (IEEE NOMS’08), pp. 883–886 (2008) 15. Boukerche, A.: Performance Evaluation of Routing Protocols for Ad Hoc Wireless Networks. In: Mobile Networks and Applications, vol. 9, pp. 333–342. Kluwer Ac. Publ., NL (2004)

A Web Service Composition Algorithm Based on Global QoS Optimizing with MOCACO Wang Li and He Yan-xiang Computer School, Wuhan University, Wuhan, P.R. China [email protected], [email protected]

Abstract. Web services composition has gained a considerable momentum as a means to create and streamline B2B collaborations within and across organizational boundaries. This paper focuses on the web services composition and provides a novel selection algorithm based on global QoS optimizing and Multi-objective Chaos Ant Colony Optimization (MOCACO). Firstly, the web services selection model with QoS global optimization is converted into a multi-objective optimization problem. Furthermore, the MOCACO is used to select the service and optimize QoS to satisfy the user constraints. During the optimizing procedure, the random and ergodic chaos variable is used to make an optimal search, it overcomes the problem of low efficiency and easily being in a partial optimization that ant colony algorithm brings. The simulation shows that the MOCACO is more efficient and effective than Multi-objective Genetic Algorithm (MOGA) applied to services composition.

1 Introduction Web services are self-contained, modular applications that can be described, published, located, and accessed over a network using open standards. With the web applications becoming more and more complex, many simple web service are combined together to complicate a complex task to meet practical needs, called web services composition. It has gained a considerable momentum as a means to create and streamline B2B collaborations within and across organizational boundaries. Especially in the composition procedure, we can get many services with the same function but different QoS. So, how to buildup a scientific way to select web services is a very important issue in web services composition [1]. And more, the web services composition based on QoS has been betaken greatly and researched deeply. There are a lot of studies on web services composition based on QoS. In [2], a predictive QoS model is presented to compute the QoS for workflows automatically based on atomic task QoS attributes and the multi-restriction parameters of service QoS are transformed into single-goal function by weighted linear method. Paper [3] buildups a QoS registry and presents a dynamic QoS computation model for web services selection. Paper [4] gives an online web services composition method based on the global QoS optimization which composites the modified simplex method and the heuristic enumerate method to solve the multi-objective global optimization problem. C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 218–224, 2010. © Springer-Verlag Berlin Heidelberg 2010

A Web Service Composition Algorithm Based on Global QoS Optimizing

219

Paper [5] proposes a global optimizing and multi-objective algorithm based on Multi-objective Genetic Algorithm and it simultaneously minimizes two objectives functions and two constraints. However, many of the research on service selection of web service composition are almost QoS local optimization or mono-objective, and will not be so suitable to resolve the problem of web services selection with QoS global optimization and multi-objective. The purpose of this paper is to focus on the web services composition and provides a novel and automatic composition algorithm based on global QoS optimizing and Multi-objective Chaos Ant Colony Optimization (MOCACO). Firstly, the web services selection model with QoS global optimization is converted into a multi-objective optimization problem. And then the MOCACO is used to select the service and optimize the different QoS parameters to satisfy the user constraints. Especially, in the optimizing procedure, the random and ergodic chaos variable is adopted and used to make an optimal search, it overcomes the problem of low efficiency and easily being in a partial optimization that ant colony algorithm brings. The remainder of this paper is organized as follows: Section 2 describes the web service composition model based on multi-objective optimization. Section 3 presents the Multi-Objective Chaos Ant Colony Optimization method to efficiently solve the multi-objective optimization. Section 4 gives out the simulation results. Finally, the conclusion is made in Section 5.

2 Web Service Composition Model Based on QoS Multi-objective Optimization 2.1 Web Service Composition Model From the view of the graph theory and set theory, the web service composition can be regarded as an oriented graph G =< V , E > which has multi-i/o. The web service can be presented as a node in it and all of them compose the vertex set V . The interaction of services can be described as the edges of the graph and all of them combine the edge set E . And especially, the QoS of the web service can be move from the node in the graph into the edge presenting the cost from one web service to another. So the web service composition problem will be transferred to the optimized combination of finding the most suitable direct acyclic path from the input to the output in the graph G to satisfy the global QoS constrain. Assume there is a set of composite service CS , consisting of m services, that is CS = {WS1 , WS 2 ,..., WS m } . Every service WSi consists of ni candidate services which compose the service group SGi = {WSi ,1 , WSi ,2 ,..., WSi , ni } , and the oriented graph G is shown in Fig. 1. In Fig.1, S presents the start, T presents the destination. The weights of connection from one web service to another present the QoS of selected services, so the composite services can be abstracted as an oriented graph with weights G =< V , E , QoS > . V =< S , V1 , V2 ,..., T > presents a set of web services and the

220

L. Wang and Y.-x. He

Fig. 1. An oriented graph of composite service

problem of combination service is equal to looking for an optimized way from in an oriented graph with weights.

S to T

2.2 Multi-objective Optimization for QoS Generally, the universal web services QoS attribute includes some important parts: cost, time, network, dependability and so on. Intuitively, the web service should provide service with lower cost, less time, high dependability and high- bandwidth network. Based on the oriented graph G =< V , E , QoS > , we can set P =< S , T > as the path from the source node S to destination node T . So the following parameters can be defined for the path P :

C os t ( P) = ∑ Cost (m) .

(1)

Time( P) = ∑ Time(m) .

(2)

Re liability ( P) = ∏ Re liability (m) .

(3)

m∈P

m∈P

m∈P

Based on the above definitions, the web services composition problem with QoS may be stated as a MOP [6] that tries to find an optimal/near-optimal service flow path P =< S , T > that simultaneously minimizes the cost and the time, maximizing its reliability and satisfies the user QoS restrictions. If we have two solutions path P and P′ from the source node S to destination node T : x = [Cost ( P ), Time( P ),1/ Re liability ( P )] . z = [Cost ( P′), Time( P′),1/ Re liability ( P′)] .

There is only one of the three conditions can be given: 1) x

z ( x dominates z ), if xi ≤ zi ∧ xi ≠ zi , ∀i ∈ {1, 2,3} ;

2) z x ( z dominates x ), if zi ≤ xi ∧ zi ≠ xi , ∀i ∈ {1, 2,3} ; 3) x ∼ z ( x and z are non-comparable), if zi xi ∧ xi zi , ∀i ∈ {1,2,3}

A Web Service Composition Algorithm Based on Global QoS Optimizing

221

P is non-dominated with respect to a set Q if: RCW ⋅QoS , P′ ∈ Q and the non-functional requirements for the concrete service flow is RCW ⋅QoS = [C , T ,1/ R ] , where C , T , R denote the global cost, global time, and global reliability constraints. When P is non-dominated with A decision vector

P

P′ ∧ P

respect to the user QoS requirements and the whole domain of feasible solutions, it is called an optimal Pareto solution; therefore, the Pareto optimal set Ptrue may be formally defined as:

Ptrue = { P ∈ X f | P is non-dominated with respect to X f }. The corresponding set of objectives Ptrue = f ( X true ) constitutes the Optimal Pareto Front. Here, this MOP problem can be resolved by the Multi-objective Chaos Ant Colony Optimization algorithm.

3 Multi-objective Chaos Ant Colony Optimization for Web Service Composition The Multi-objective Chaos Ant Colony Optimization (MOCACO) algorithm is based on the Multi-objective Ant Colony Optimization algorithm (MOACO) [6] and the chaos operator, bringing the advantage of them together. During the ant colony optimization procedure, the randomization, ergodicity, initial sensitivity and chaos operator are brought in to make the chaos variable linear mapping to a domain of optimization variable. This will avoid the search becoming a partial optimization and making up the shortage of an ant algorithm. As a result, it improves the diversity and general optimization of this algorithm. 3.1 MOCACO Optimization Procedure In the MOCACO algorithm used for web services composition problem, a colony of ants are used to construct m solutions P at every generation and the detailed procedure is following [7,8]: 1) Initialize source node S , destination node T , N r and ϕ . 2) Initialize pheromone matrix τ ij , τ ij (0) = C ( C is a constant). 3) For every ant, construct the solution P . 3.1) Set tabulist = φ and P = φ ; 3.2) Set N i as the nodes in the neighborhood of node i that the ant has not visited yet. For all the nodes in N i compute the chosen probability:

pij =

[τ ij ]α [ηij ]β

∑ [τ

s ⊂ Ni

is

]α [ηis ]β

.

(4)

And get P = P ∪ { j} , tabuList = tabuList ∪ { j} .In (4), ηij is the heuristic value of moving from node i to node j , and we can define it as:

222

L. Wang and Y.-x. He

ηij = 1 3.3) If

Cost 2 ( j ) + Time2 ( j ) + (1/ Re liability ( j )) 2

.

| P |< n then go to 3.1), else go to 4).

4) Get Pknow = Pknow ∪ P −{Py | P − Py }∀Py ∈ Pknow . Pknow is the known Pareto Front [9]. 5) Update pheromone matrix τ ij : m

τ ij (t + 1) = ρ • τ ij (t ) + ∑ Δτ ijk .

(5)

k =1

Where ρ ∈ (0,1] and the Δτ ij presents the pheromones ant iteration.  k

k left on route in this

6) If not convergent, then go to 3), otherwise we will get the optimized P . 3.2 Chaos Pheromone Update During the optimization procedure, the initial value of pheromone matrix τ ij , τ ij (0) = C ( C is a constant), the strength of pheromones on every route is equal that the possibility of every route is equal. It’s hard for ant colony to find an optimization route; and more, the convergent speed of algorithm is slow. Here we adopt the chaos operator for the pheromone matrix τ ij , it will increase the efficiency of it’s searching by the random and ergodic of chaos. The chaos variable coursed by Logistic mapping is:

λi +1 = μ • λi • (1 − λi ) .

(6)

In (6), i = 0,1, 2,... , μ is the control parameter, with the domain between (2, 4] , when μ = 4 , the Logistic map- ping are the full mapping in (0,1) , which are at a totally chaos status. The iteration will create an ergodic chaos sequence which can be used to solve the problem of optimization in space. After using chaos operator and chaos variable to update the pheromone trail strength, the formula (5) will become: m

τ ij (t + 1) = ρ • τ ij (t ) + ∑ Δτ ijk + Aλi .

(7)

k =1

4 Simulation In this section we will give out some web service composition simulation results based on the MOCACO presented in this paper, and the results based on Multi- objective Ant Colony Optimization(MOACO) [8] and Multi-objective Genetic Algorithm(MOGA) [6] are also given out in order to compare the solution quality and performance for all of the three algorithm. In this simulation, 3 web service groups are adopted and the details of service groups are presented in Table 1. It means that the composed web service will combine m web service and each web service has n services. All three algorithms, MOCACO, MOACO and MOGA, have been implemented on a computer with Intel Pentium Dual 2.40GHz

A Web Service Composition Algorithm Based on Global QoS Optimizing

223

Table 1. Web service group for composition test

Service group Group 1 Group 2 Group 3

m 10 20 20

n 5 10 20

processor, 2GB RAM, and the operating system Windows XP professional. The compiler used was Java. For each groups, the test run 2 times and 100 and 200 iterations. The services QoS values were achieved randomly from a gaussian distribution function. The user’s constraint for the composed service flow is ϕ = [C , T , R] = [1000,10, 0.5] . In each ant colony for MOCACO and MOACO, the number of ants is set to 20, and the parameters of algorithm are α = 1.0 , β = 1.0 , ρ = 0.3 . Table 2. gives out the number of solutions found for every web service group. From the result in Table 2, we can find that the MOCACO algorithm demonstrates better performance than the MOACO and MOGA algorithm, achieving more optimal P amount than both of others whether the group’s web services amount is large or small. Furthermore, the running time for all of the three algorithms is also compared in this simulation and the results are shown in Fig. 2. It can be seen that the time astringency of MOCACO algorithm is better than the others. Table 2. Number of optimal solution for each test group

Service group Iterations 100 Group 1 200 Group 2 100 200 100 Group 3 200 25

20

) S ( e m15 i T g n i10 n n u R

MOCACO 14 15 24 26 36 39

MOACO 12 13 21 22 33 37

MOCACO(It.=100) MOACO(It.=100) MOGA(It.=100) MOCACO(It.=200) MOACO(It.=200) MOGA(It.=200)

5

0 1

2

Service Group

3

Fig. 2. Running time comparison result

MOGA 9 10 17 19 31 34

224

L. Wang and Y.-x. He

5 Conclusion In this paper, it introduces a novel and automatic composition algorithm based on global QoS optimizing and Multi-objective Chaos Ant Colony Optimization (MOCACO) for the web services composition problem. This algorithm brings the advantage of multi-objective ant colony optimization and chaos operators together. With the random, ergodic chaos variable, the ant colony optimization’s problem of low efficiency and easy in partial optimization is overcome. And more, the blindness and efficiency of chaos search is improved by the positive feedback of multi-objective ant colony algorithm. From the simulation presented in the paper, we can find that the MOCACO is able to find more optimal solutions than MOACO and MOGA algorithm. Especially, the MOCACO also shows its feasible and efficient performance. As future work, we will focus on the adaptive parameter adjusting in this algorithm and compare the performance of different chaos mapping function in this algorithm.

Acknowledgement This paper is supported by the National High-Tech Research Development Program of China (863 program) under Grant No.2007AA01Z138 and the China Postdoctoral Science Foundation under grant No. 20090460978.

References 1. Kleijnen, S., Raju, S.: An Open Web services Architecture, pp. 38–46. ACM Press, NewYork (2003) 2. Jorge, C., Amit, S., John, M.: Quality of Service for workflows and Web Service Processes. Journal of Web Semantics 1(3), 281–338 (2004) 3. Liu, Y.T., Anne, H.H., Zeng, L.Z.: QoS Computation and Policing in Dynamic Web Services selection. In: Proc. WWW 2004, pp. 66–73. ACM, New York (2004) 4. Wan, L., Gao, C., Xiao, W., Su, L. (eds.): Global optimization method of Web services composition based on QoS, Computer Engineer And Applications, vol. 24 (2007) 5. Liu, S., Liu, Y., Jing, N., Tang, G., Tang, Y.: A Dynamic Web Service Selection Strategy with QoS Global Optimization Based on Multi-objective Genetic Algorithm. In: Zhuge, H., Fox, G.C. (eds.) GCC 2005. LNCS, vol. 3795, pp. 84–89. Springer, Heidelberg (2005) 6. Schaerer, M., Barán, B.: A Multi-objective Ant Colony System For Vehicle Routing Problem With Time Windows. In: Proc. IASTED International Conference on Applied Informatics, Innsbruck (2003) 7. Yang, H., Wang, H., Hou, L., Sun, X. (eds.): Application of Chaos Ant Colony Optimization in the Intelligent Transportation System and Its Algorithm. Journal Of Chengdu University (Natural Science Edition) 4 (2007) 8. Qiqing, F., Xiaoming, P., Qinghua, L., Yahui, H.: A Global QoS Optimizing Web Services Selection Algorithm based on MOACO for Dynamic Web Service Composition. In: 2009 International Forum on Information Technology and Applications, pp. 37–42 (2009) 9. Van Veldhuizen, D.A.: Multiobjective Evolutionary Algorithms: Classifications, Analyses and New Innovations. Ph. D. thesis Air Force Institute of Technology (1999)

Experiences Gained from Building a Services-Based Distributed Operating System Andrzej Goscinski and Michael Hobbs School of Information Technology, Deakin University Waurn Ponds, Victoria, 3217, Australia {ang,mick}@deakin.edu.au

Abstract. The goal of this paper is to present the experiences gained over 15 years of research into the design and development of a services-based distributed operating system. The lessons learnt over this period, we hope, will be of value to researchers involved in the design and development of operating systems that wish to harness the collective resources of ever-expanding distributed systems. Keywords: Distributed Operating System Design, Service Oriented Computing.

1 Introduction Research of a distributed operating systems based on a set of services was addressed by the first author in 1985, following a study of two major set of works by Cheriton [1] and Tanenbaum [2]. The study confirmed that the author’s idea of service based approach to building operating systems was sound. Second, the study was also a trigger to research the logical design of distributed operating systems [3] where interprocess communication is a major platform for cooperating modules of an operating system. This study formed the inception for a proof of concept project called RHODOS (Research Oriented Distributed Operating System) [4], where the idea of a service based operating system supported by a small microkernel (initially called a nucleus) was demonstrated. This project has reached its research orbit when Dr Hobbs teamed up with Dr Goscinski. The authors received a strong positive kick when Tanenbaum presented his support to microkernel-based architecture of operating systems in comp.os.minix in January 1992 [5]. That was the beginning of our journey into the research of service-based distributed operating systems. Traditional operating system research has its heritage based in monolithic architectures such as UNIX [6] and Linux [7], where all functionality and services are provided within a protected, consolidated set of kernel code. A major concern with this approach is the difficulty in building such systems and with addition of functionality, due to the tight coupling of the kernel code which can lead to errors in one component of the kernel to affect other non-faulty components (a situation that is difficult to debug). An alternative approach is to provide the core functionality or services of an operating system as processes that are supported by a minimalist kernel layer, termed microkernel architecture. Examples of microkernel systems include: Mach [8], QNX [9] and L4 [10]. The MIT exokernel [11] project took the microkernel paradigm C.-H. Hsu et al. (Eds.): ICA3PP 2010, Part II, LNCS 6082, pp. 225–234, 2010. © Springer-Verlag Berlin Heidelberg 2010

226

A. Goscinski and M. Hobbs

further by also reducing the number of interfaces that the operating supported, allowing users to define the services that were required. There are also a number of projects that looked at the designing and building an operating system that harnesses the collective resources of a distributed system; these include: Sprite [12], Plan9 [13] Amoeba [14], 2K [15] and SPIN [16]. Other distributed operating systems addressed usability, by making the complete distributed system appear as a single, large computer; these include: Mosix [17] and Kerrighed [18]. The goal of this report is to describe the authors’ experiences in designing, developing and implementing a distributed operating system based on service oriented principles. These experiences are drawn from over 15 years of research and development of distributed operating systems and will hopefully provide readers interested in developing future systems an insight into both the successes and problems we encountered. This report is structured as follows. In Section 2, the scope of our project are presented. Section 3 discusses our view on services and their relationship to resources managed by an operating system. Section 4 presents the resources managed by distributed operating systems through services. The often conflicting expectations of users of operating systems and the key responsibilities of operating systems are highlighted in Section 5. A description on how the computational resources of a distributed system are harnessed and exposed for the application of parallel processing is presented in Section 6. The broader role of user interface and execution environment of a services based distributed operating system is given in Section 7. Finally, Section 8 provides a summary of the important outcomes and experiences.

2 Project Scope In the 1980s one could identify two major research and development streams in the area of operating systems. One was very much user oriented and demonstrated itself in the development and perfecting of menu driven window environments. Another stream concentrated on issues such as architecture, development, maintenance and execution. We concentrated our effort on the latter, although we also devoted some projects on a command driven and window based interfaces. Computing systems in general and operating systems in particular depend on virtualization. If there is no virtualization there is no computing. Operating systems and their architectures reflect a need for and application of virtualization in a natural manner. We decided to provide virtualization by employing the service computing paradigm and identified some basic features of a distributed operating system and carried out our research toward their development. These features were as follows: Modularity: a variety of areas we have envisioned and problems encountered with Unix (e.g., modification, debugging) required avoiding monolithic architecture. This was one of the strongest factors that has influenced our research and led us to modularity based on service orientation. 2. Efficient / Adaptive / Flexible Communications: interprocess communication of distributed systems depends on interconnecting network including network protocols; and the design and implementation of communication primitives. 1.

Experiences Gained from Building a Services-Based Distributed Operating System 3. 4.

5. 6. 7.

227

Transparency: any real distributed system should be designed and implemented in such a way that distribution of all resources will be hidden. High Performance: any operating system, centralized or distributed, should manage resources in such a manner that the highest performance possible is offered (measured using response time, throughput, and system utilization). Dynamic Instantiation: an operating system should be able to automatically recognize and use new services provided by new resources. Single System Image (SSI): its provision means that the application developer sees the whole distributed system as a single powerful and reliable computer. Distribution and Parallelism Execution Management: an operating system should manage all resources automatically and transparently toward high performance / throughput execution without a direct involvement of programmers.

Our involvement in the provision of these features clearly demonstrated that transparency [3] is a subset of SSI [19]. The following dimensions were considered to offer SSI: transparency – distribution should be hidden; availability – that could be achieved if a virtual machine can be established automatically and dynamically – thus needing a resource discovery service that can identify computers and peripheral devices automatically and record their presence, load state and fault events; fault tolerance – as the scale of the distributed system increases, the probability that components will fail also increases – thus fault tolerance mechanisms of replication, checkpointing and recovery should be employed; communication paradigms – both message passing and (distributed) shared memory methods should be provided. Many of these features can also be generalized by the Autonomic Computing characteristics [20]. This important characteristic implies that there is a need for not only high performance but also ease of programming and use, reliability, availability through proper reaction to unpredictable changes and transparency. This could be achieved through the provision of autonomic computing characteristics. According to Horn [20], an autonomic computing system possesses at least the following characteristics: knows itself; configures and reconfigures itself under varying and unpredictable conditions; optimizes its working; performs something akin to healing; provides self-protection; knows its surrounding environment; exists in an open (non-hermetic) environment; and anticipates the optimized resources needed while keeping its complexity hidden. We addressed these characteristics in the Helos project [21].

3 Services vs. Resources The management and execution of user processes requires all resources, physical and logical, of a computer system to be exposed. We proposed a distributed operating system built as a set of cooperating services able to expose these resources. These services (for transparency reasons) communicate within a single computer as well as among remote computers using messages. We proposed a provision of well defined interfaces to invoke a service. This implies that messages must be designed having standardization in mind. For this purpose it was proposed that messages should have a distinct format that reflects the way a service consumes messages. Thus, the header being consumed first contains information that indicates what to do next.

228

A. Goscinski and M. Hobbs

A general model of the relationship between a resource; a service exposing this resource; and a client invoking the service; is shown in Fig. 1. Service

Resource

3. Operations

Client

1. Client Request 2. Load / Save State Data / State

Fig. 1. A view shows only a portion of the entire document

Initially, we believed that a simple client server model will satisfy our modularity requirements. We realized early that some operations (e.g., process migration, local and remote process creation), to achieve high performance, should be performed concurrently, leading to a more general service oriented approach. A service in a distributed operating system may, to complete a client’s request, be required to invoke additional services which could be on the same computer or more remote computers. Implemented correctly, this model of cooperating and distributed services enables a level of transparency in which the user is unaware of where services (and resources) are located. This provides the foundation on which many more desirable features can be built, such as a SSI and the provision of autonomic principles.

4 Services Exposing Resources The above analysis implies a need for the following basic system services that create virtual resources to hide and expose the major system resources and manage them: Process Service, Space Service, File Service, and Driver Services for I/O devices. One of the most crucial elements of decision making is the provision and exposure of a resource state. There are two different data states: internal data structures, and shared data structures. A process is an example of a resource that is maintained by a Process Service and due to its virtual nature requires its state to be shared. Conventionally known as a Process Control Block (PCB), it contains information that relates to both processes and resources such as space, I/O devices, and files. A structure of a PCB was proposed such that it could be shared with related services such as the Space Service, I/O Service, and File Service, respectively. This also follows the requirement of modularity and allows the associated services working on the PCB concurrently. This modularity requirement led us to the concept of a nucleus, later renamed microkernel for uniformity with other projects. The following data structures were proposed to be stored in the microkernel: clock data; page maps for process address space population and simple page fault handling; interrupt and exception tables; and processor registers for context switching to support local IPC, deal with interrupts and provide context switching, and basic memory (page) operations. The microkernel’s architecture was proposed to support portability. This implied a need to have a small component that was hardware dependent and a hardware independent component. They were made separate by a well defined interface.

Experiences Gained from Building a Services-Based Distributed Operating System

229

The microkernel and basic system services form a basic virtual computer system able to execute system and user processes. All these basic services were designed to run in the user space, as privileged user processes. These processes communicate among themselves using messages by invoking a required microkernel IPC primitive, send and receive. If the destination process executes on the source computer local message delivery is performed. “Locality” was proposed to be resolved by storing in the microkernel information regarding locally executing processes. Communication with remote processes was proposed and supported by an Interprocess Communication (IPC) Service. This service, invoked by a message from the microkernel, communicates over a network with a peer IPC Service running on a computer of a remote destination process (Fig. 2). This solution provided full transparency and very low communication overhead. Communication Facility User Process

Remote IPC

IPC Primitives: send() / recv()

Inter Process Communications Service

Transport Protocol IP / Ethernet Microkernel Local IPC

Network Service

Physical Layer

Fig. 2. IPC component interaction supporting transparent communication

The IPC Service was proposed to be made responsible to providing both an interface to the Network Service and information to support group communication. Thus, the IPC Service internally stores information about the most frequently accessed remote processes and communication groups.

5 Satisfying User Requirements and Resource Utilization Two sets of requirements, users and compute service providers, were addressed through the high performance and fault tolerance perspectives. Global Scheduling Service To achieve high performance execution of user applications there is a need for another service, a Global Scheduling Service [22, 23]. This service is responsible for the best possible allocation of compute resources to user processes to provide high execution performance and maintain an efficient (balanced) utilization of the computer systems’ resources. We proposed that the Global Scheduling Service be responsible to provide placement based on both: Static Allocation – at the beginning of running a distributed application and later when the computational load does not change frequently; and Dynamic Load Balancing – used in the case of load changing frequently. Provision of this service could be provided centrally or as a set of cooperating distributed services for large distributed system.

230

A. Goscinski and M. Hobbs

The quality of decisions made by the Global Scheduling Service depends directly on the validity of the input data. To address this problem a service called Resource Discovery Service [24] was proposed. This service, located on all computers of a distributed computer system, is responsible for collecting both: (i) static parameters – such as the number of processors and their characteristics, main memory size, disk size and I/O bandwidth; and (ii) dynamic parameters – such as current processor load, available memory, communication pattern and volume of each executing process. The execution of the decisions made by the Global Scheduling Service were proposed to be performed by additional services such as Local and Remote Process Creation Service, Local and Remote Process Duplication Service, and Process Migration Service (discussed in Section 6). Process Migration Service The Process Migration Service is a clear demonstration of advantages of servicesbased architecture. It was envisioned that this service will only coordinate process migration, and carry it out in a transaction fashion to only release resources on a source computer when the process on the destination computer was deployed. Global Scheduling Service

Migration-to-Migration Manager Coordination Messages

Migration Service

Migration Service Process Service Space Service IPC Service

Source Computer

Process State Spaces

Comms. Buffers

Process Service Space Service IPC Service

Destination Computer

Fig. 3. Service Cooperation to provide Process Migration

In migration of a process there is a need for transferring its state, address space memory, buffers, communication ports and open file information (Fig. 3). These components are stored in the process’ PCB. The Process Migration Service invokes relevant services, i.e., Process, Space, I/O, and File Services, to transfer their respective resources of the process and confirm completion of their operations [25]. The process migration service plays a critical role in the provision of the global scheduling service. It also confirms that through a service oriented approach, complex operating systems functions can be implemented transparently using sets of cooperating individual services. Checkpointing and Fault-Tolerance Service A failure of a computer or a process could lead to great computational loses as the process or the whole distributed application must be restarted from scratch. This implies a need for a service that could provide fault tolerance. For this purpose we proposed to employ two services, supported by the Resource Discovery Service: Checkpointing Service – records a state of a process (and of all processes of a distributed application)

Experiences Gained from Building a Services-Based Distributed Operating System

231

during a fault free execution of this process; and Recovery Service – restores a failed process (and of all processes of a distributed application) based on the last checkpointed state. Copies of both these services are provided on each computer of a distributed computer system [26, 27]. Dynamic Provision of Services All the services to satisfy user requirements and offer good resource utilization were designed in such a manner that they could be deployed dynamically [24]. For this purpose we proposed: (i) a resource that is not exposed through a service is not accessible by other services (it is not part of the resource pool); and (ii) services can be dynamically stopped and restarted (potentially at a different location). However, basic system services were designed to be deployed as replacements. For this purpose we ensured that: a fundamental service (such as a Process Service) cannot be shut down but it is possible to ‘hand-over’ to another instance of a service. Of course all these services to satisfy user requirements and offer good resource utilization were proposed and designed to run in the user space.

6 Services Supporting Parallel Computing Following an analysis of parallel applications and processes of a parallel application executing on a cluster we decided to introduce services, which were embraced by facility called Parallelism Management Facility [28] including the following services: • Local and Remote Group Process Creation Service – which creates a group of new processes based on code supplied by the Process Service and creates them on computers determined by the Global Scheduling Service; • Local and Remote Group Process Duplication Service – duplicates a specified process on a set of local and/or remote computers determined by the Global Scheduling Service; • Process Migration Service – which migrates a group of processes to the remote locations, specified by the Global Scheduling Service; • Distributed Shared Memory (DSM) Service – which provided an environment that supported shared memory programming paradigm; and • System Discovery Service – responsible for forming a virtual cluster out of available (lightly loaded) computers to support individual parallel applications. In the design, the System Discovery Service [24] provided aggregated information based on data provided by the Resources Discovery Services of the whole cluster directly to the Global Scheduling Service.

7 Programming and Execution Environment We decided that for the study of distributed operating systems and execution of distributed and parallel applications there is a minimum need for a user interface and execution environment. A decision was made to provide a shell that offers standard I/O redirection. This allowed us to execute cross-compiled programs written in the C

232

A. Goscinski and M. Hobbs

language. At one stage a Posix interface was created to make possible the execution of user programs/ applications running on Unix-like operating systems. Users wish to have an easy and commonly used environment to be provided to allow them to execute their programs/applications. In the majority of execution environments cases application developers did not have the opportunity to make a choice between the message passing (MP) and DSM communication paradigm. These paradigms and supporting them systems were developed independently of an operating system as separate services, rather than to be their integral parts. Two parallelization environments were considered in the area of message passing: at an early stage of the project a PVM environment was developed [29]. Later, a MPI environment was added [30]. As DSM addresses shared memory, an enhancement to the Space Service was made to provide a shared memory in a distributed system [31]. The proposed DSM environment required a copy of the enhanced Space Service deployed on each computer of a distributed system.

8 Concluding Remarks Our study of the design and implementation of service-based distributed operating systems has spanned over 15 years, during which a two key versions of the project have been developed: RHODOS [4] and following it GENESIS [32]. RHODOS focused more on the microkernel architecture, overall service based system design and implementation issues of a distributed operating system. Outcomes of this stage of the project included a demonstration that microkernel based architecture can be used to support a distributed operating system, where system level services such as process, memory and IO management are supported by a transparent message passing based IPC mechanisms [4, 33, 34]. These outcomes verified the set of features identified in Section 2 were valid and that common operating system functions could be supported by exposing them as services. GENESIS, also designed and implemented based on services and microkernel, involved a greater emphasis on higher level operating system functions and included: global scheduling, process migration, remote process management, fault tolerance and recovery, distributed shared memory, parallelism management, resource discovery and transparent remote resource use. These outcomes verified that complex autonomic characteristics are achievable through inter service cooperation. Over the lifetime of this project we have learned that service orientation forms an excellent basis of the design and implementation of distributed operating systems; although care needs to be taken to ensure that the performance of the system is maintained. Such systems could be easily configured, modified and debugged. Original solutions have also been achieved in the areas of parallel execution on non-dedicated clusters, provision of SSI and autonomic services, reliability and usability features supporting distributed systems. Two implemented systems demonstrated that service orientation allows for incremental development of complex systems, support changes in research directions, and achieve very good outcomes. We can confirm that research into operating systems is a very rewarding endeavor which, in our case, supported the completion of 10 PhD projects and contributed to knowledge in areas of operating system design, system services and service oriented

Experiences Gained from Building a Services-Based Distributed Operating System

233

computing. The challenges and overheads of such a large project are high in particular because PhD students have to complete their projects and leave, especially when undertaking it from the ground-up as we had done.

References 1. Cheriton, D.R.: The V Kernel: A Software Base for Distributed Systems. IEEE Software 1, 19–42 (1984) 2. Tanenbaum, A.S., Van Renesse, R.: Distributed operating systems. ACM Computer Surveys 17, 419–470 (1985) 3. Goscinski, A.: Distributed operating systems, the logical design. Addison-Wesley, Reading (1991) 4. De Paoli, D., Goscinski, A., Hobbs, M., Wickham, G.: The RHODOS Microkernel, Kernel Servers and Their Cooperation. In: IEEE 1st Intl. Conf. on Algorithms and Architectures for Parallel Processing, vol. 1, pp. 345–354. IEEE, Brisbane (1995) 5. Tanenbaum, A.S.: LINUX is obsolete. comp.os.minix, Google Groups (1992), http://groups.google.com/group/comp.os.minix/browse_thread/ thread/c25870d7a41696d2/ (last accessed 20/1/2010) 6. Ritchie, D.M., Thompson, K.: The UNIX time-sharing system. Communications of the ACM 17, 10 (1974) 7. Torvalds, L.B.: What would you like to see most in minix? comp.os.minix, Google Groups (1991), http://groups.google.com/group/comp.os.minix/msg/b813d52cbc5 a044b (last accessed 20/1/2010) 8. Golub, D.B., Julin, D.P., Rashid, R.F., Draves, R.P., Dean, R.W., Forin, A., Barrera, J., Tokuda, H., Malan, G., Bohman, D.: Microkernel operating system architecture and mach. In: Proc. USENIX Workshop on Micro-Kernels and Other Kernel Architectures, pp. 11–30 (1992) 9. Hildebrand, D.: An Architectural Overview of QNX. In: Proceedings of the Workshop on Micro-kernels and Other Kernel Architectures, pp. 113–126. USENIX Assoc., CA (1992) 10. Liedtke, J.: On μ-Kernel Construction. In: Proc. 15th ACM symposium on Operating Systems Principles (SOSP), pp. 237–250 (1995) 11. Engler, D.R., Kaashoek, M.F., O’Toole, J.: Exokernel: an operating system architecture for application-level resource management. In: Fifteenth ACM Symposium on Operating Systems Principles, pp. 251–266. ACM, New York (1995) 12. Ousterhout, J.K., Cherenson, A.R., Douglis, F., Nelson, M.N., Welch, B.B.: The Sprite Network Operating System. Computer 21, 23–36 (1988) 13. Presotto, D., Pike, R., Thompson, K., Trickey, H.: Plan 9, A Distributed System. In: Proceedings of the Spring 1991 EurOpen Conference (1991) 14. Mullender, S.J., Rossum, G.v., Tanenbaum, A.S., Renesse, R.